How to order a Kafka startup shell script for a docker container?

Question

I am trying to containerize a Kafka server. A normal start-up sequence for a Kafka server is like this:

A: start Zookeeper server
B: start Broker server
C: create topic

Item A and B are long running process. And C need to wait for B to come up and running.

So I wrote a Dockerfile, with ENTRYPOINT executing a shell script for the above sequence:

#!/bin/sh

$KAFKA_HOME/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties &
$KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &

$KAFKA_HOME/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test_topic

However, I encountered three problems when run the docker image built from this:

• Docker requires the ENTRYPOINT process to be long-running, while above script does not. (it exits as long as the topic-creation is done)
• The broker server statement (the 2nd one) is long-running process. Currently I have to put it as background process using ending &, because otherwise the statements after it are not executed at all. (But making it background also has problem: the topic creation is executed immediately, while the broker server is not ready yet.)
• I could not put the broker server statement as the last long-running process, because the topic creation statement has to come after the server creation.

What could be a good way to arrange this start-up sequence?

Answer 1

Basically you want to start ZK, then Kafka. Then somehow wait until Kafka is ready (that's the tricky part), do your job with kafka (e.g. topic creation in your case), and then wait until Kafka & ZK have finished (what happens on interrupt).

start-zookeeper &
ZK_PID=$!
start-kafka &
KAFKA_PID=$!

# that's the tricky part
wait_for_kafka
create-topic.sh

wait "${KAFKA_PID}"
wait "${ZK_PID}"

---

As mentioned, the Kafka-readiness might be tricky - the following ways might be helpful:

• waiting until Kafka responds to read requests (e.g. probing with kafka-topic.sh --list periodically)
• creating a pocket consumer / AdminClient (java kafka 0.11+) and getting metadata (similar to above point)
• checking existence of JMX beans for logs/controller etc.
• checking listening port availability