Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

What does KafkaConsumer.commitSync() actually commit?

Does KafkaConsumer.commitSync just commit "offsets returned on the last poll()" as JavaDoc claims, (which may miss some partitions not included in the last poll result), or it is actually committing the latest positions for all subscribed partitions? Asking because the code suggests the latter, considering allConsumed: https://github.com/apache/kafka/blob/2.4.0/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L1387

@Override
public void commitSync(Duration timeout) {
    acquireAndEnsureOpen();
    try {
        maybeThrowInvalidGroupIdException();
        if (!coordinator.commitOffsetsSync(subscriptions.allConsumed(), time.timer(timeout))) {
            throw new TimeoutException("Timeout of " + timeout.toMillis() + "ms expired before successfully " +
                    "committing the current consumed offsets");
        }
    } finally {
        release();
    }
}
like image 659
Anton Savelyev Avatar asked Oct 16 '25 04:10

Anton Savelyev


1 Answers

It only commits the offsets that were actually polled and processed. If some offsets were not included in the last poll, then those offsets will not be committed.

It will not commit the latest positions for all subscribed partitions. This would interfere with the Consumer Offset management concept of Kafka to be able to re-start an application where it left off.

From my understanding, the allConsumed is equivalent to all offsets included in the last poll which the comment of the commitSync also documents

Commit offsets returned on the last poll() for all the subscribed list of topics and partitions.

like image 105
Michael Heil Avatar answered Oct 17 '25 18:10

Michael Heil