Elasticsearch's search_after doesn't work with date

Question

I have an Elasticsearch query that looks something like this,

{
  size: 25,
  query: {
    bool: {
      filter: ...,
      must: ...
    }
  },
  sort: [
    {
      created_at: {
        order: "desc
      }
    },
    {
      id: {
        order: "desc"
      }
    }
  ],
  from: 0,
  search_after: ["2018-10-25T18:04:13.488Z", "8"]
}

When I run this I get an error saying,

"reason": Failed to parse search_after value for field [created_at], caused_by":{"type":"number_format_exception","reason":"For input string: \"2018-10-23T21:41:03.167Z\"

I tried mapping the created_at property with type of date and format of date_time, but that didn't seem to help.

Can anyone shed some light on how I can solve this?

Answer 1

You need to pass the date as the number of milliseconds since epoch, i.e. instead of 2018-10-25T18:04:13.488Z try passing 1540491254488

{
  size: 25,
  query: {
    bool: {
      filter: ...,
      must: ...
    }
  },
  sort: [
    {
      created_at: {
        order: "desc
      }
    },
    {
      id: {
        order: "desc"
      }
    }
  ],
  from: 0,
  search_after: [1540491254488, "8"]
                       ^
                       |
                  change this

Note that you can continue indexing your documents as you do now, but when creating your query, you need to feed that number into search_after.

Answer 2

Additionally, the Elasticsearch documentation is misleading on this, as their example looks like:

    "search_after": [1463538857, "654323"],

which leads you to believe that it is seconds, rather than milliseconds, since the epoch.