DynamoDB ProvisionedReadCapacity exceeded

Question

I am load testing a service with 3000 requests per second. Each request fetches data from a DynamoDB table. The table has a provisioned read capacity of 10,000 reads per second. However, I am getting the following exception:

com.amazonaws.services.dynamodbv2.model.ProvisionedThroughputExceededException: The level of configured provisioned throughput for the table was exceede d. Consider increasing your provisioning level with the UpdateTable API (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ProvisionedThroughputE xceededException; Request ID: KHOG5L1S83VU05CAOEJCCPAUFVVV4KQNSO5AEMVJF66Q9ASUAAJG)

My table description is as follows.

$ aws dynamodb describe-table --table-name my_table
{
"Table": {
    "TableArn": "arn:aws:dynamodb:us-east-1:188456577:table/my_table", 
    "AttributeDefinitions": [
        {
            "AttributeName": "username", 
            "AttributeType": "S"
        }
    ], 
    "ProvisionedThroughput": {
        "NumberOfDecreasesToday": 0, 
        "WriteCapacityUnits": 10000, 
        "LastIncreaseDateTime": 1462386432.633, 
        "ReadCapacityUnits": 10000
    }, 
    "TableSizeBytes": 289776, 
    "TableName": "my_table", 
    "TableStatus": "ACTIVE", 
    "KeySchema": [
        {
            "KeyType": "HASH", 
            "AttributeName": "username"
        }
    ], 
    "ItemCount": 81, 
    "CreationDateTime": 1458249331.208
    }
}

As can be seen, both read and write capacity units for the table have been set to 10,000.

I was monitoring the health of the beanstalk service while the load testing was in progress and the number of requests was indeed around 3K per second on an average. I don't understand why then the throughput capacity is exceeded. The table should also be getting 3,000 requests per second.

Answer 1

DynamoDB does not throttle on the whole table capacity. From the documentation:

The provisioned throughput associated with a table is also divided evenly among the partitions, with no sharing of provisioned throughput across partitions.

So if you have a read capacity of 10,000 per second and your table has say 10 partitions then each partition is allocated a read capacity of 1000 reads per second. If in your load test you are hitting keys in the same partition (uneven distribution of workload) then you will see ProvisionedThroughputExceededExceptions. You should try to have an even distribution of your partition key space so you can get the most of your provisioned throughput.

Read more on this topic here.