How to make a spout wait on another spout?

Question

I have a use case where I want to wait for one spout to emit and start listening on kafka from other spout. Is this possible in Storm ?

For Example: Spout A does some processing and emits done, once I receive done message, Spout B should start listening on Kafka topic from the offset it stopped in last run and emit to some other bolts.

Any help would be appreciated.

Answer 1

Just choose how Spout A communicates with Spout B. You can:

1. Use another kafka topic. When Spout A finishes it then produces a message to a special topic and Spout B waits for that message on that topic before beginning to read from the primary topic.

2. Use a flag written to Zookeeper (or HDFS, HBase, etc) to indicate that Spout A has finished

When I say that Spout B waits, its nextTuple() method first tests to see whether the communication from A (above) has happened and simply returns without further action until it does.

Remember that your code that creates and submits the topology can also initialize the flag (if you're doing 2 above) or can create a unique message (for 1 above) and store it in the Configuration to be passed to Spouts A and B.