How to use pagination in Amazon DynamoDB using the AWS SDK for Java

Question

According to this documentation from AWS, it is possible to make paginated queries on Amazon DynamoDB, but It doesn't provide any example about how to do it. It only says to use the iterator method, that I found on PaginatedQueryList after using DynamoDBMapper.query().

Could anyone explain how does it work, how to use it correctly and if it really worth using it when you don't want to retrieve all results at once from dynamoDB, like the scan opeation does?

Answer 1

There is no fully native way to have pagination with page number using DynamoDB. Mainly because of distributed nature of DynamoDB - tables are stored in multiple partitions (mb even in multiple servers). Also, in DynamoDB, each record you retrieve from a query has a direct cost, so I bet you don't want to run something something similar to count query.

But if you want to achieve something like infinite scroll or "Show more effect" there is a way to do that. Following example is actual for AWS Java SDK v2 2.20.7. the example shows Enhanced Client usage.

Model:

@Data
@DynamoDbBean
@FieldNameConstants
public class Event {

  @Getter(onMethod_ = {@DynamoDbPartitionKey})
  private String id;
  private String name;
  private String description;
  @Getter(onMethod_ = {@DynamoDbSecondaryPartitionKey(indexNames = "GSI_CREATOR")})
  private String creatorId;
  @Getter(onMethod_ = {@DynamoDbAutoGeneratedTimestampAttribute})
  private Instant updatedAt;
}

Pagination example is provided in the form of JUnit 5 test.

Data preparation using Easy Random lib:

 private static final TableSchema<Event> TABLE_SCHEMA = TableSchema.fromBean(Event.class);

  private static final String EVENT_TABLE_NAME = "events-table";
  private static final String CREATOR = UUID.randomUUID().toString();

  @BeforeEach
  public void initDataModel() {

    final var eventFileDynamoDbTable = enhancedClient.table(EVENT_TABLE_NAME, TABLE_SCHEMA);

    final var factory = new EasyRandom();

    event1 = factory.nextObject(Event.class);
    event1.setCreatorId(CREATOR);

    eventFileDynamoDbTable.putItem(event1);

    event2 = factory.nextObject(Event.class);
    event2.setCreatorId(CREATOR);

    eventFileDynamoDbTable.putItem(event2);
  }

  @AfterEach
  public void cleanUp() {
    final var table = enhancedClient.table(EVENT_TABLE_NAME, TABLE_SCHEMA);

    table.deleteItem(event1);
    table.deleteItem(event2);
  }

Actual pagination process:

  @Test
  void testSecondaryIndexSearch() {

    final var table = enhancedClient.table(EVENT_TABLE_NAME, TABLE_SCHEMA);

    // Use global secondary index
    final var secIndex = table.index("GSI_CREATOR");

    // Create a QueryConditional object that's used in the query operation.
    final var queryConditional = QueryConditional
        .keyEqualTo(Key.builder().partitionValue(event1.getCreatorId()).build());

    // Get items in the table with limit 1
    SdkIterable<Page<Event>> pages = secIndex.query(QueryEnhancedRequest.builder()
        .queryConditional(queryConditional)
        .limit(1)
        .build());
    
    Page<Event> next = pages.iterator().next();

    assertEquals(1, next.items().size());

    // retrieve last processed key. this cursor can be passed to a client. 
    // so, it will be able to retrieve data from where it left off in the next query
    final var cursor = next.lastEvaluatedKey();

    assertNotNull(cursor);

    // Get items in the table with limit 1 using cursor
    // in real world case cursor will be received from the client
    SdkIterable<Page<Event>> newPages = secIndex.query(QueryEnhancedRequest.builder()
        .queryConditional(queryConditional)
        .limit(2) // trying to fetch more then we have
        .exclusiveStartKey(cursor)
        .build());
    Page<Event> nextWithLast = newPages.iterator().next();

    assertEquals(1, nextWithLast.items().size());

    final var cursorLast = nextWithLast.lastEvaluatedKey();

    // there is no more data to check. so, it's the end of "pagination"
    assertNull(cursorLast);
  }

Notices and in conclusion:

LastEvaluatedKey is a basically Primary Key of the record where a Query operation finished for now and returned as the last value of the result set. In this case - id and creatorID

LastEvaluatedKey is returned on two conditions:

• Query results have hit the upper limit (e.g., DynamoDB Query operation divides the data into 1MB of size).
• If you have specified a limit by adding the Limit parameter in the query, it returns a dataset with more records remaining to evaluate for the next page.

Therefore, continuous pagination (infinite scrolling) can be implemented in the such way.