how to generate id for message when using Kafka stream?

Question

I want to have a try about implementing a normal chatting system after have read many artifles in confluent kafka. But I have met some problems when doing some structure design. When using mysql as my data's db, I can give id to every meaningful message, like user_id in user table, message_id for message table. After having id in model table, it is very convinient for client and server doing some comunication. But in Kafka stream, how can I give every meaningful model a unique id in KTable? Or is it really necessary for me to do this?

Answer 1

Maybe I can answer the question for myself.

In mysql, we can directly use sequenceId because all data will go to one place and then be auto allocated a new id. But when the table grows too large, we also need to split table to several little tables.In that case, we also should to regenerate the unique id for each record, because auto generated id in these tables is begun from 0.

Maybe it is the same in Kafka. When we only have one partition in kafka, we also can use the id from kafka generated id because all the message will go to only one place, so they will never be dumplicated. But when we want more partitions, we also have to be careful that these generated id from different partition is not global unique.

So what we should do is to generate id for ourself. UUID is a fast way to do this, but I we want to have a number, we can use a little algorithm to implement this. Maybe use the structure like this in a distributed enviroment:

[nodeid+threadId+current_time+auto_increased_number]