Kafka transactionLog fails with NotEnoughReplicasException, despite correct config

Question

I'm having issues with reads from Kafka failing in our single-node dev environments

We are running a single-node kafka instance, and single-node Zookeeper -- as this is a dev env, we don't really care about replication.

We're using Kafka as an event store for Akka persistence. When we attempt to read messages (even from an empty topic) we're getting NotEnoughReplicasException raised:

[2018-05-03 16:57:00,880] ERROR [ReplicaManager broker=0] Error processing append operation on partition __transaction_state-7 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.NotEnoughReplicasException: Number of insync replicas for partition __transaction_state-7 is [1], below required minimum [2]
[2018-05-03 16:57:00,889] INFO [Transaction State Manager 0]: TransactionalId akka-journal-message-AkkaJavaSpring-4 append transaction log for TxnTransitMetadata(producerId=6000, producerEpoch=0, txnTimeoutMs=60000, txnState=Empty, topicPartitions=Set(), txnStartTimestamp=-1, txnLastUpdateTimestamp=1525366620855) transition failed due to COORDINATOR_NOT_AVAILABLE, resetting pending state from Some(Empty), aborting state transition and returning COORDINATOR_NOT_AVAILABLE in the callback (kafka.coordinator.transaction.TransactionStateManager)

I have configured the following settings:

"min.insync.replicas": 1
"offsets.topic.replication.factor": 1
"transaction.state.log.replication.factor": 1
"transaction.state.log.min.isr" : 1

which is confirmed in the output of the config logged on startup:

[2018-05-03 16:33:34,381] INFO KafkaConfig values: 
    ...snip...
    min.insync.replicas = 1
    num.io.threads = 8
    num.network.threads = 3
    num.partitions = 1
    ...snip...
    transaction.state.log.load.buffer.size = 5242880
    transaction.state.log.min.isr = 1
    transaction.state.log.num.partitions = 50
    transaction.state.log.replication.factor = 1
    transaction.state.log.segment.bytes = 104857600

However, querying the __transactionLog topic seems to report it's min.insync.replicas set to 2:

/usr/bin/kafka-topics --zookeeper zookeeper:2181 --describe

Topic:__transaction_state       PartitionCount:50       ReplicationFactor:1     Configs:segment.bytes=104857600,unclean.leader.election.enable=false,compression.type=uncompressed,cleanup.policy=compact,min.insync.replicas=2
        Topic: __transaction_state      Partition: 0    Leader: 0       Replicas: 0     Isr: 0
        Topic: __transaction_state      Partition: 1    Leader: 0       Replicas: 0     Isr: 0
        Topic: __transaction_state      Partition: 2    Leader: 0       Replicas: 0     Isr: 0
        Topic: __transaction_state      Partition: 3    Leader: 0       Replicas: 0     Isr: 0

I'm using Kafka 1.0.0, as shown in the logs:

 INFO Kafka version : 1.0.0-cp1 (org.apache.kafka.common.utils.AppInfoParser)

What is causing this, and what is the correct config to prevent these exceptions from occuring?

Answer 1

The config posted above turned out to be correct.

However, the Zookeeper instance had been incorrectly configured when the topic was initially created, and replaying messages from the log seemed to throw the error.

Deleting the Zookeeper instance and starting again from fresh resolved the error