Does KafkaConsumer.commitSync just commit "offsets returned on the last poll()" as JavaDoc claims, (which may miss some partitions not included in the last poll
result), or it is actually committing the latest positions for all subscribed partitions? Asking because the code suggests the latter, considering allConsumed
:
https://github.com/apache/kafka/blob/2.4.0/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L1387
@Override
public void commitSync(Duration timeout) {
acquireAndEnsureOpen();
try {
maybeThrowInvalidGroupIdException();
if (!coordinator.commitOffsetsSync(subscriptions.allConsumed(), time.timer(timeout))) {
throw new TimeoutException("Timeout of " + timeout.toMillis() + "ms expired before successfully " +
"committing the current consumed offsets");
}
} finally {
release();
}
}
It only commits the offsets that were actually polled and processed. If some offsets were not included in the last poll, then those offsets will not be committed.
It will not commit the latest positions for all subscribed partitions. This would interfere with the Consumer Offset management concept of Kafka to be able to re-start an application where it left off.
From my understanding, the allConsumed
is equivalent to all offsets included in the last poll
which the comment of the commitSync
also documents
Commit offsets returned on the last poll() for all the subscribed list of topics and partitions.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With