How to install connectors to the docker image of apache kafka connect

Question

I am using https://hub.docker.com/r/1ambda/kafka-connect/ to install docker kafka connect on ubuntu .I am able to run it but ,I am unable to install any more connectors in it.

What I have tried

1-I tried copying the connector files from my machine to the docker containers connector folder and restarting but the api ,http://localhost:8080/connectors give empty array.

2-Also added plugin path in connect-distributed.properties . How to do this any idea ??

Answer 1

Solution without using Confluent Hub

Step 1: Build your own Kafka-Connect image

Your directory should look like this:

my-directory /
  Dockerfile
  plugins /
    my-connector /
       <connector-jars>

Dockerfile:

FROM confluentinc/cp-kafka-connect-base:latest
USER root:root
COPY ./plugins/ /opt/kafka/plugins/
ENV CONNECT_PLUGIN_PATH="/opt/kafka/plugins"
USER 1001

Run the following

cd /my-directory
sudo docker build . -t my-connector-image-name

Step 2: Use the created image for your kafka-connect container

version: '3'
services:
        zookeeper:
                container_name: 'zookeeper'
                image: 'bitnami/zookeeper:latest'
                ports:
                        - '2181:2181'
                environment:
                        - ALLOW_ANONYMOUS_LOGIN=yes
        kafka:
                image: 'bitnami/kafka:latest'
                container_name: 'kafka'
                ports:
                        - '9092:9092'
                environment:
                        - KAFKA_BROKER_ID=1
                        - KAFKA_LISTENERS=PLAINTEXT://:9092
                        - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.178.70:9092
                        - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
                        - ALLOW_PLAINTEXT_LISTENER=yes
                depends_on:
                        - zookeeper
        kafka-connect:
                image: 'my-connector-image-name:latest'
                container_name: 'kafka-connect'
                ports:
                        - '8083:8083'
                environment:
                        - CONNECT_BOOTSTRAP_SERVERS=kafka:9092
                        - CONNECT_REST_PORT=8083
                        - CONNECT_GROUP_ID=quickstart
                        - CONNECT_CONFIG_STORAGE_TOPIC=quickstart-config
                        - CONNECT_OFFSET_STORAGE_TOPIC=quickstart-offsets
                        - CONNECT_STATUS_STORAGE_TOPIC=quickstart-status
                        - CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1
                        - CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1
                        - CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1
                        - CONNECT_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
                        - CONNECT_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
                        - CONNECT_INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter
                        - CONNECT_INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter
                        - CONNECT_REST_ADVERTISED_HOST_NAME=localhost
                depends_on:
                        - kafka

Step 3 Get available Connector Plugins

curl localhost:8083/connector-plugins | json_pp

Side note on my Kafka configuration: I'm hosting Docker inside a Linux virtual machine that I've assigned IP 192.168.178.70.