Spring Kafka producer requires restart
up vote
0
down vote
favorite
We have an issue where a Kafka partition outage (just 3 minutes) caused our producer to time out. We are producing with a 30 second timeout with 30 retries. After the partition outage ended, our Spring Kafka service producing to the topic required a service restart to start working. We are trying to understand a configuration missing or what issue did not allow our service threads to automatically resume the ability to publish to the topic without a service restart.
Below are details on how our producers look like. If you guys see something that you think we are doing wrong or can be done better, that will be helpful.
We are using kafka version 0.10
And the only custom config we are using is –
packEvent.producer.retries=30
packEvent.producer.acksConfig=1
Any thoughts on what was missing to not gracefully recover?
apache-kafka spring-kafka
edited Nov 9 at 6:36
Giorgos Myrianthous
3,61121233
asked Nov 8 at 19:30
Kris
1
add a comment |
up vote
0
down vote
favorite
We have an issue where a Kafka partition outage (just 3 minutes) caused our producer to time out. We are producing with a 30 second timeout with 30 retries. After the partition outage ended, our Spring Kafka service producing to the topic required a service restart to start working. We are trying to understand a configuration missing or what issue did not allow our service threads to automatically resume the ability to publish to the topic without a service restart.
Below are details on how our producers look like. If you guys see something that you think we are doing wrong or can be done better, that will be helpful.
We are using kafka version 0.10
And the only custom config we are using is –
packEvent.producer.retries=30
packEvent.producer.acksConfig=1
Any thoughts on what was missing to not gracefully recover?
apache-kafka spring-kafka
edited Nov 9 at 6:36
Giorgos Myrianthous
3,61121233
asked Nov 8 at 19:30
Kris
1
1
I suggest you turn on DEBUG logging fororg.apache.kafka; it that doesn't help; post the logs someplace like pastebin/github gist. You should also edit your question to show exactly how you are using the peoducer.
– Gary Russell
Nov 8 at 21:32
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
We have an issue where a Kafka partition outage (just 3 minutes) caused our producer to time out. We are producing with a 30 second timeout with 30 retries. After the partition outage ended, our Spring Kafka service producing to the topic required a service restart to start working. We are trying to understand a configuration missing or what issue did not allow our service threads to automatically resume the ability to publish to the topic without a service restart.
Below are details on how our producers look like. If you guys see something that you think we are doing wrong or can be done better, that will be helpful.
We are using kafka version 0.10
And the only custom config we are using is –
packEvent.producer.retries=30
packEvent.producer.acksConfig=1
Any thoughts on what was missing to not gracefully recover?
apache-kafka spring-kafka
edited Nov 9 at 6:36
Giorgos Myrianthous
3,61121233
asked Nov 8 at 19:30
Kris
1
We have an issue where a Kafka partition outage (just 3 minutes) caused our producer to time out. We are producing with a 30 second timeout with 30 retries. After the partition outage ended, our Spring Kafka service producing to the topic required a service restart to start working. We are trying to understand a configuration missing or what issue did not allow our service threads to automatically resume the ability to publish to the topic without a service restart.
Below are details on how our producers look like. If you guys see something that you think we are doing wrong or can be done better, that will be helpful.
We are using kafka version 0.10
And the only custom config we are using is –
packEvent.producer.retries=30
packEvent.producer.acksConfig=1
Any thoughts on what was missing to not gracefully recover?
apache-kafka spring-kafka
apache-kafka spring-kafka
edited Nov 9 at 6:36
Giorgos Myrianthous
3,61121233
asked Nov 8 at 19:30
Kris
1
edited Nov 9 at 6:36
Giorgos Myrianthous
3,61121233
asked Nov 8 at 19:30
Kris
1
edited Nov 9 at 6:36
Giorgos Myrianthous
3,61121233
edited Nov 9 at 6:36
Giorgos Myrianthous
3,61121233
edited Nov 9 at 6:36
Giorgos Myrianthous
3,61121233
3,61121233
asked Nov 8 at 19:30
Kris
1
asked Nov 8 at 19:30
Kris
1
asked Nov 8 at 19:30
Kris
1
1
1
I suggest you turn on DEBUG logging fororg.apache.kafka; it that doesn't help; post the logs someplace like pastebin/github gist. You should also edit your question to show exactly how you are using the peoducer.
– Gary Russell
Nov 8 at 21:32
add a comment |
1
I suggest you turn on DEBUG logging fororg.apache.kafka; it that doesn't help; post the logs someplace like pastebin/github gist. You should also edit your question to show exactly how you are using the peoducer.
– Gary Russell
Nov 8 at 21:32
1
1
I suggest you turn on DEBUG logging for
org.apache.kafka; it that doesn't help; post the logs someplace like pastebin/github gist. You should also edit your question to show exactly how you are using the peoducer.– Gary Russell
Nov 8 at 21:32
I suggest you turn on DEBUG logging for
org.apache.kafka; it that doesn't help; post the logs someplace like pastebin/github gist. You should also edit your question to show exactly how you are using the peoducer.– Gary Russell
Nov 8 at 21:32
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53214861%2fspring-kafka-producer-requires-restart%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
I suggest you turn on DEBUG logging for
org.apache.kafka; it that doesn't help; post the logs someplace like pastebin/github gist. You should also edit your question to show exactly how you are using the peoducer.– Gary Russell
Nov 8 at 21:32