Kafka Connector - Packacking jars

Question

QUESTION

I am not a maven pro and I got stuck trying to package a Kafka Connector. There are two options for packaging it:

• Either you produce a folder with a jar that contains the connector + all the dependency jars - all the kafka-specific jars
• Or build a fat jar with all of the dependencies (and I also assume without the kafka-specific jars again, but it is not explicit in the docs).

I am following docs on confluent webpage and the connector I am trying to package is this one on github.

What I tried, after cloning the repo with git, is the following mvn clean package. But this seems to create only a single jar of the original project with the dependencies in mvn cache (~/.m2/repository/).

Google also has link on how to create a fat jar, but I would need somehow to specify which jars I want to exclude from the fat jar.

Thanks

UPDATE

Now I am running:

connect-standalone /etc/kafka/connect-standalone.properties /etc/kafka/connect-cdc-mssql-source.properties

Where /etc/kafka/connect-standalone.properties contains the following line:

plugin.path=/shared_win_files

And ls -al /shared_win_files contains the following:

kafka-connect-cdc-mssql-0.0.1-SNAPSHOT.jar

And jar tvf kafka-connect-cdc-mssql-0.0.1-SNAPSHOT.jar contains the following:

  6996 Thu Sep 07 14:47:24 BST 2017 com/github/jcustenborder/kafka/connect/cdc/mssql/MsSqlSourceConnector.class

where MsSqlSourceConnector.classis basically this class here which implements the Connector.

But when I try to run the connector with the command above, I get an error

Failed to find any class that implements Connector and which name matches com.github.jcustenborder.kafka.connect.cdc.mssql.MsSqlSourceConnector

It gives a massive list with all available plugins, but mine is not in there.

Answer 1

Currently, an easy way to package your connector with maven is to use maven-assembly-plugin. This basically entails two main steps:

1. Define one or more assembly descriptors and save them under src/assembly.

   Doc: http://maven.apache.org/plugins/maven-assembly-plugin/descriptor-refs.html

   Example: https://github.com/confluentinc/kafka-connect-elasticsearch/blob/master/src/assembly/package.xml

   In the descriptor, among other things, you may choose the packaging format of your archive, files and directories to include or exclude, as well as specific settings regarding your projects dependencies.

2. Include the plugin in your project's pom.xml

   Example: https://github.com/confluentinc/kafka-connect-elasticsearch/blob/master/pom.xml

   This mainly requires you to define the configuration and execution sections of the maven-assembly-plugin section. Additionally you can associate calls to specific assembly plugin descriptors with certain maven profiles that you may define.

Finally, stay tuned because packaging your Kafka Connect plugins (connectors, transforms, converters) might be significantly simplified soon.