can MongoDB ChangeStream feature replace Pub/Sub technologies

Question

I was going through MongoDB ChangeStream , I understand it reduces the risk of tailing the oplog - we are Currently tailing the oplog to publish the data to Kafka.

Please help me to grasp so how can changestreams are better compared to the Pub/Sub technologies like Kafka or RabbitMQ

Answer 1

ChangeStreams should not be compared to Pub/Sub technologies - ChangeStreams are there to provide a safe way to enable real-time (data) change events to be captured and then processed (and as you rightly pointed out, previously you had to tail the oplog in MongoDB in order to achieve a similar outcome which had its own set of issues, risks and complexities which taxed the developer).

As I mentioned above, ChangeStreams provide a safe way to look at each data change event occuring in MongoDB, apply a filter to those events, and then process each qualifying event. ChangeStreams let you to replay previous events based on the timeframe that your oplog covers - for example, if your application that is implementing ChangeStreams fails, you have the ability to pick-up from the point the application failed.

Whilst ChangeStreams exhibit Pub/Sub-like behaviour from an event identification/processing perspective, that is where the likeness stops. A typical/common use case, where you are interested in capturing/identifying data-change events in MongoDB for downstream processing, is to create a Kafka Producer that utilises the MongoDB driver, instantiates a ChangeStream, and for each qualifying event occurring in MongoDB (made available via the ChangeStream) passes that onto Kafka.