Issues accessing DynamoDB Pagination Token in AWS SDK for .NET's AsyncSearch class

Question

I'm currently diving into the DynamoDB AWS SDK for .NET, particularly tinkering with the AsyncSearch<T> class to perform table queries. The goal is to incorporate pagination into my application, and according to AWS documentation, the pagination token resides within the underlying Search object. This object is accessible through the DocumentSearch property of the AsyncSearch<T> class, and it gets set by the GetNextSetAsync() method, as illustrated in this documentation.

You can find the AsyncSearch class documentation here.

However, there's a catch—I'm grappling with the fact that the DocumentSearch property is private, making it tricky to access the pagination token directly.

Here's a glimpse of my code:

public interface IAsyncSearch<T>
{
    bool IsDone { get; }
    string PaginationToken { get; }
    Dictionary<string, AttributeValue> NextKey { get; }

    Task<List<T>> GetRemainingAsync(CancellationToken token = default);
    Task<List<T>> GetNextSetAsync(CancellationToken token = default);
}

public class CustomAsyncSearch<T> : IAsyncSearch<T>
{
    private readonly AsyncSearch<T> _search;

    public CustomAsyncSearch(AsyncSearch<T> search)
    {
        _search = search;
    }

    async Task<List<T>> IAsyncSearch<T>.GetRemainingAsync(CancellationToken token)
    {
        return await _search.GetRemainingAsync(token);
    }

    async Task<List<T>> IAsyncSearch<T>.GetNextSetAsync(CancellationToken token)
    {
        return await _search.GetNextSetAsync(token);
    }

    public bool IsDone => _search.IsDone;
    public string PaginationToken => _search.DocumentSearch.PaginationToken?.ToString();
    
    public Dictionary<string, AttributeValue> NextKey => _search.DocumentSearch.NextKey;
}

I'm wondering if there's a proper way to fetch the pagination token from the AsyncSearch class without tweaking its internal structure. Any insights or suggestions on accessing the pagination token or ensuring correct pagination implementation with the AsyncSearch class would be immensely helpful.

EDIT: Here's AsyncSearch class base class in AWS SDK for .NET

namespace Amazon.DynamoDBv2.DataModel;

// Summary:
//     A strongly-typed object for retrieving search results (Query or Scan) from DynamoDB.
//
//
// Type parameters:
//   T:
public class AsyncSearch<T>
{
    private Search DocumentSearch { get; set; }

    private DynamoDBContext SourceContext { get; set; }

    private DynamoDBFlatConfig Config { get; set; }

    //
    // Summary:
    //     Flag that, if true, indicates that the search is done
    public bool IsDone => DocumentSearch.IsDone;

    internal AsyncSearch(DynamoDBContext source, DynamoDBContext.ContextSearch contextSearch)
    {
        SourceContext = source;
        DocumentSearch = contextSearch.Search;
        Config = contextSearch.FlatConfig;
    }

    //
    // Summary:
    //     Initiates the asynchronous execution to get the next set of results from DynamoDB.
    //     If there are more items in the Scan/Query, PaginationToken will be set and can
    //     be consumed in a new Scan/Query operation to resume retrieving items from this
    //     point.
    //
    // Parameters:
    //   cancellationToken:
    //     Token which can be used to cancel the task.
    //
    // Returns:
    //     A Task that can be used to poll or wait for results, or both. Results will include
    //     the next set of result items from DynamoDB.
    public async Task<List<T>> GetNextSetAsync(CancellationToken cancellationToken = default(CancellationToken))
    {
        List<Document> documents = await DocumentSearch.GetNextSetHelperAsync(cancellationToken).ConfigureAwait(continueOnCapturedContext: false);
        return SourceContext.FromDocumentsHelper<T>(documents, Config).ToList();
    }

    //
    // Summary:
    //     Initiates the asynchronous execution to get all the remaining results from DynamoDB.
    //
    //
    // Parameters:
    //   cancellationToken:
    //     Token which can be used to cancel the task.
    //
    // Returns:
    //     A Task that can be used to poll or wait for results, or both. Results will include
    //     the remaining result items from DynamoDB.
    public async Task<List<T>> GetRemainingAsync(CancellationToken cancellationToken = default(CancellationToken))
    {
        List<Document> documents = await DocumentSearch.GetRemainingHelperAsync(cancellationToken).ConfigureAwait(continueOnCapturedContext: false);
        return SourceContext.FromDocumentsHelper<T>(documents, Config).ToList();
    }
}

Answer 1

When using AsyncSearch, the PaginationToken (actually NextKey, which is the basis for creating the pagination token) is established and stored within the instance - this allows you to perform operations like GetNextSetAsync() multiple times without worrying about setting subsequent PaginationTokens. I assume this is the expected behavior of the higher-level API, which AsyncSearch appears to be.

If you want full and straightforward access to the pagination token, which is a very common scenario, especially when providing pagination to a client, then there's nothing stopping you from using a lower-level API, such as:

// context is IDynamoDBContext

var table = context.GetTargetTable<CarModel>(); // Adjust to your model
var result = table.Query(new QueryOperationConfig()
{
    Filter = new QueryFilter(), // Adjust your query
    Limit = 100,
    PaginationToken = paginationToken // Pass a PaginationToken
}); // 

var documents = await result.GetNextSetAsync();
var records = context
    .FromDocuments<CarModel>(documents)
    .ToList();

return new ResponseResult
{
    Records = records,
    NextPaginationToken = result.PaginationToken
};

As a paginationToken you pass your token from the request for example or null if it's a first query. You also need to adjust Filter to your needs of course.

You can get "next" pagination token from the result variable.

Essentially, you could say it's a simplified version of what happens when you use AsyncSearch.