Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Understanding of kafka auto-offset-reset

Tags:

apache-kafka

I read that the default experation time of kafka offsets are 7 days. Is my understanding correct for the following scenario?

  1. Producer produces message 1 and 2
  2. Consumer with auto-offset-reset=latest consumes messages 1 and 2 with commit

Nothings happens for 6 days

  1. Consumer crashes
  2. Producer produces message 3

Nothings happens for 2 days

  1. Consumer is starting but doesnt read message 3, because the last committed offset is deleted and auto-offset-reset=latest is triggered?
like image 642
JavaQuestioner123 Avatar asked Oct 14 '25 23:10

JavaQuestioner123


1 Answers

Offset expiration Additional Upgrade Notes

According to the new semantics, offsets of partitions in a group will not be removed while the group is subscribed to the corresponding topic and is still active (has active consumers). If group becomes empty all its offsets will be removed after default offset retention period (or the one set by broker) has passed (unless the group becomes active again)

The consumer group offset will be removed when there are no active consumers in that group for 7 days of period (default retention).

When consumer crashes, that means there are no active consumers in that group and if the consumer is crashed for 7 days then offset will be removed.

Once the offset data is deleted/not found, then if any consumer subscribes will take the configs provided. If the auto-offset-reset=latest then it will just consume the data published after the consumer is initialized.

auto.offset.reset docs

What to do when there is no initial offset in Kafka or if the current offset does not exist any more on the server (e.g. because that data has been deleted):

like image 135
Deadpool Avatar answered Oct 18 '25 08:10

Deadpool



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!