"BadRequestError: cursor position is outside the range of the original query" when fetching the next batch of a query

Question

I have this handler to do some process on all users of our app. Basically, it takes 1 batch, processes the records of that batch, then queues a new task for the next batch.

QueueAllUsers(BaseHandler):
    FETCH_SIZE = 10
    FILTERS = [
        UserConfig.level != 0,
        UserConfig.is_configured == True
    ]
    ORDER = [UserConfig.level, UserConfig._key]

    def get(self):
        cursor_key = self.request.get('cursor')
        cursor = None
        if cursor_key:
            # if `cursor` param is provided, use it
            cursor = Cursor(urlsafe=str(cursor_key))
        q = UserConfig.query(*self.FILTERS).order(*self.ORDER)
        total = q.count()  # 31 total records
        logging.info(total)
        users, next_cursor, more = q.fetch_page(self.FETCH_SIZE,
                                                keys_only=True,
                                                start_cursor=cursor)
        self.process_users(users)
        if more:
            self.queue_next_batch(next_cursor)

    def queue_next_batch(self, next_cursor):
        # Call get() again but this time pass `cursor` param to process next batch
        logging.info(next_cursor.urlsafe())
        url = '/queue_all_users?cursor=%s' % (next_cursor.urlsafe())
        taskqueue.add(
            url=url,
            method='get',
            queue_name='cronjobs'
        )

    def process_users(self, users):
        logging.info(len(users))
        # trimmed

But when the task for the 2nd batch runs, NDB throws BadRequest error saying that the cursor is out of range.

I don't understand why it's out of range? I fetched 10 records out of a total of 31, so the cursor should still be valid.

Note that the error is thrown on the 2nd batch (i.e. records 11-20).

So the flow is like this:

1. Call /queue_all_users to process the first batch (no cursor). Everything works ok.
2. Step 1 creates a task for /queue_all_users?cursor=123456 for the next batch.
3. Call /queue_all_users?cursor=123456 (cursor provided). fetch_page throws BadRequestErrror.

EDIT: I tried setting FETCH_SIZE to 17, and fetching the 2nd batch worked! It seems anything below 17 causes the error, and 17 above works. So... what the heck?

Answer 1

I had same problem. When I make the first query everything goes fine and a cursor is returned. The second query using the cursor give me the error:

BadRequestError: cursor position is outside the range of the original query.

I tried your solution but isn't work for me. So I change my filters in my query and it works, I don't know why but maybe can be a solution for you and others.

My old query was:

page_size = 10
query = Sale.query(ancestor=self.key).filter(ndb.AND(
    Sale.current_status.status != SaleStatusEnum.WAITING_PAYMENT,
    Sale.current_status.status != SaleStatusEnum.WAITING_PAYMENT
    ).order(Sale.current_status.status, Sale._key)
query.fetch_page(page_size, start_cursor=cursor)

Then a I change all "!=" to IN operation, like this:

page_size = 10
query = Sale.query(ancestor=self.key).filter(
    Sale.current_status.status.IN([
        SaleStatusEnum.PROCESSING, 
        SaleStatusEnum.PAID, 
        SaleStatusEnum.SHIPPING, 
        SaleStatusEnum.FINALIZED,
        SaleStatusEnum.REFUSED])
    ).order(Sale.current_status.status, Sale._key)
query.fetch_page(page_size, start_cursor=cursor)