Mirror Maker 2 doesn't follow compression.type from source

Question

I'm trying to mirror topics with compression.type = gzip but the messages arrive on target cluster without compression. I'm using kafka connector to run MM2.

I had tried those settings without success:

compression.type = gzip

producer.compression.type = gzip

target.compression.type = gzip

I'm checking the messages in the target cluster using:

/bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /path_to_log/00000000000000000000.log --print-data-log | grep compresscodec

Anyone is facing this issue with the MM 2? Or any tip?

Answer 1

I was able to turn on the GZIP compression to MM2 producers to the Target Cluster (Cluster2). I do not know if this is the only and correct way to do it, but I use a configuration like this:

#########################################################################
# specify any number of cluster aliases
clusters=Cluster1, Cluster2

# connection information for each cluster
# This is a comma separated host:port pairs for each cluster
# for e.g. "A_host1:9092, A_host2:9092, A_host3:9092"
Cluster1.bootstrap.servers=server1.mynetwork.net:9095
Cluster2.bootstrap.servers=server2.mynetwork.net:9093

# enable and configure individual replication flows
Cluster1->Cluster2.enabled=true

# regex which defines which topics gets replicated. For eg "foo-.*"
Cluster1->Cluster2.topics=.*

# Reverse
Cluster2->Cluster1.enabled=false
Cluster2->Cluster1.topics=.*

# Workers configs (Cluster2)
Cluster2.producer.compression.type=gzip
#########################################################################

In the MM2 log, we can see that these 2 producers have the GZIP compression option turned on:

[2021-01-27 14:52:19,006] INFO ProducerConfig values:
        acks = -1
        batch.size = 16384
        bootstrap.servers = [server2.mynetwork.net:9093]
        buffer.memory = 33554432
        client.dns.lookup = default
        client.id = **connector-producer-MirrorSourceConnector-0**
        compression.type = gzip
.
.
.
--
[2021-01-27 14:52:19,023] INFO ProducerConfig values:
        acks = -1
        batch.size = 16384
        bootstrap.servers = [server2.mynetwork.net:9093]
        buffer.memory = 33554432
        client.dns.lookup = default
        client.id = connector-producer-MirrorHeartbeatConnector-0
        compression.type = gzip
.
.
.

Dumping my log segment I got this:

baseOffset: 15190 lastOffset: 15190 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false isControl: false position: 1473759 CreateTime: 1611769946191 size: 110 magic: 2 compresscodec: GZIP crc: 1457322770 isvalid: true
| offset: 15190 CreateTime: 1611769946191 keysize: 20 valuesize: 10 sequence: -1 headerKeys: [] key:ClusterCluster1 payload: wD��O