How to fix error from embedded kafka that cannot find meta.properties

Question

I am trying to do integration test for app that using kafka, kafka-streams and cassandra. But when I am trying to setUp test class, i've got 2 errors: ERROR [main] BrokerMetadataCheckpoint: Failed to read meta.properties file under dir ERROR [main] KafkaServer: Fail to read meta.properties under log directory

I am using spring-boot-starter 2.1.2, spring-boot-starter-test 2.1.2, spring-kafka 2.2.0, spring-kafka-test 2.2.0, apache.kafka-streams 2.1.0

trying to change logs.dir and logs.dirs params. use @EnableKafka @EnableKafkaStreams

@RunWith(SpringRunner.class)
@SpringBootTest
@EmbeddedKafka(partitions = 3, controlledShutdown = false, count = 1, topics = {"zc.deviceposition"}, brokerProperties = {"listeners=PLAINTEXT://localhost:9092", "port=9092", "log.dir=/home/name/logs"})
@EmbeddedCassandra(timeout = 60000)
@CassandraDataSet(value = {"bootstrap_test.cql"}, keyspace = "statistics")
@ActiveProfiles("test")
@DirtiesContext
public class CassandraTripsAggregatorProcessorSupplierIntegrationTest {
  @Test
  public void someTest() {System.out.println("hello world");}
}

I excpect to run up context with embedded kafka, but for now I receive an error that meta.properties is no exist

Answer 1

Probably, you didn't shutdown Kafka gracefully in the last run which caused this. Always shut it down with kafka-server-stop.sh or Ctrl+C.

I had to format the whole storage using these commands to get it back working:

# WARNING: This will delete all your topics and kafka storage
# Generate UUID
kafka-storage.sh random-uuid

# Format storage
kafka-storage.sh format -t 2vi2WtHxQAOPyXb1Bj1Jvw -c ~/.kafka/kafka_2.13-3.7.1/config/kraft/server.properties

# Start Server
kafka-server-start.sh ~/.kafka/kafka_2.13-3.7.1/config/kraft/server.properties

Answer 2

In windows 10 0r 11, if you are facing the above error.

Follow the below steps.

It happened for me when I put the laptop to sleep for a few days and Kafka Producer crashed ( did not shutdown gracefully)

1. Delete the Log folder, In my case Kafka was run from d:drive, so my log folder was in D:\tmp\kraft-combined-logs, dont do this in production if your consumer hasnt read all messages.

2. Run the below command to recreate the logs folder

---

.\bin\windows\kafka-storage.bat format --standalone -t kafkastore -c .\config\server.properties

3. You should see something like this :

It does say complain about some error , but it creates the storage dir as needed.

4. Run the below command to get the server up and running

.\bin\windows\kafka-server-start.bat .\config\server.properties     

5. You will see something like this below