AWS MSK lambda concurrent consumers

Question

AWS Lambda Supports Parallelization Factor for Kinesis and DynamoDB Event Sources. But its not supported for MSK. Can we create a reserved concurrency of the Lambda function and would it help to concurrently consume from MSK topic

Answer 1

TL'DR Connecting lambda to kafka cluster using aws::event-source-mapping is limited to the amount of partitions you are having in the topics

I had the experience to setup a poc of

Custom Kafka Cluster Topic (1 Partition) > EventSourceMapping > Lambda

and after opening a discussion with AWS it looks like it is a limitation

Another approach that I didn't try is to setup a lambda sink (kafka connect) and setup a tasks.max which seems like it can solve this issue https://docs.confluent.io/kafka-connectors/aws-lambda/current/overview.html#lambda-sink-multiple-tasks