Get every Nth result in Elasticsearch

Question

I have this large set of data and I want a sample that I can use in a graph. For this I don't need all of the data, I need every Nth item.

For instance if I have 4000 results, and I only need 800 results, I want to be able to get every 5th result.

So some like: get, skip, skip, skip, skip, get, skip, skip, skip,..

I was wondering if such a thing is possible in Elasticsearch?

Answer 1

You're better off using a scripted filter. Otherwise you're needlessly using the score. Filters are just like queries, but they don't use scoring.

POST /test_index/_search
{
  "query": {
    "filtered": {
      "filter": {
        "script": {
          "script": "doc['unique_counter'].value % n == 0",
          "params" : {
            "n" : 5
          }
        }
      }
    }
  }
}

You're also better off not using dynamic scripting in real world usage.

That said, you probably want to take a look at aggregations for graphing analytical information about your data rather than taking an arbitrary sample.

Answer 2

One way you could do it is with random scoring. It won't give you precisely every nth item according to a rigid ordering, but if you can relax that requirement this trick should do nicely.

To test it I set up a simple index (I mapped "doc_id" to "_id" just so the documents would have some contents, so that part isn't required, in case that's not obvious):

PUT /test_index
{
   "mappings": {
      "doc": {
         "_id": {
            "path": "doc_id"
         }
      }
   }
}

Then I indexed ten simple documents:

POST /test_index/doc/_bulk
{"index":{}}
{"doc_id":1}
{"index":{}}
{"doc_id":2}
{"index":{}}
{"doc_id":3}
{"index":{}}
{"doc_id":4}
{"index":{}}
{"doc_id":5}
{"index":{}}
{"doc_id":6}
{"index":{}}
{"doc_id":7}
{"index":{}}
{"doc_id":8}
{"index":{}}
{"doc_id":9}
{"index":{}}
{"doc_id":10}

Now I can pull back three random documents like this:

POST /test_index/_search
{
   "size": 3,
   "query": {
      "function_score": {
         "functions": [
            {
               "random_score": {
                  "seed": "some seed"
               }
            }
         ]
      }
   }
}
...
{
   "took": 1,
   "timed_out": false,
   "_shards": {
      "total": 1,
      "successful": 1,
      "failed": 0
   },
   "hits": {
      "total": 10,
      "max_score": 0.93746644,
      "hits": [
         {
            "_index": "test_index",
            "_type": "doc",
            "_id": "1",
            "_score": 0.93746644,
            "_source": {
               "doc_id": 1
            }
         },
         {
            "_index": "test_index",
            "_type": "doc",
            "_id": "10",
            "_score": 0.926947,
            "_source": {
               "doc_id": 10
            }
         },
         {
            "_index": "test_index",
            "_type": "doc",
            "_id": "5",
            "_score": 0.79400194,
            "_source": {
               "doc_id": 5
            }
         }
      ]
   }
}

Or a different random three like this:

POST /test_index/_search
{
   "size": 3,
   "query": {
      "function_score": {
         "functions": [
            {
               "random_score": {
                  "seed": "some other seed"
               }
            }
         ]
      }
   }
}
...
{
   "took": 1,
   "timed_out": false,
   "_shards": {
      "total": 1,
      "successful": 1,
      "failed": 0
   },
   "hits": {
      "total": 10,
      "max_score": 0.817295,
      "hits": [
         {
            "_index": "test_index",
            "_type": "doc",
            "_id": "4",
            "_score": 0.817295,
            "_source": {
               "doc_id": 4
            }
         },
         {
            "_index": "test_index",
            "_type": "doc",
            "_id": "8",
            "_score": 0.469319,
            "_source": {
               "doc_id": 8
            }
         },
         {
            "_index": "test_index",
            "_type": "doc",
            "_id": "3",
            "_score": 0.4374538,
            "_source": {
               "doc_id": 3
            }
         }
      ]
   }
}

Hopefully it's clear how to generalize this method to what you need. Just take out however many documents you want, in however many chunks make it performant.

Here is all the code I used to test:

http://sense.qbox.io/gist/a02d4da458365915f5e9cf6ea80546d2dfabc75d

EDIT: Actually now that I think about it, you could also use scripted scoring to get precisely every nth item, if you set it up right. Maybe something like,

POST /test_index/_search
{
   "size": 3,
   "query": {
      "function_score": {
         "functions": [
            {
               "script_score": {
                  "script": "if(doc['doc_id'].value % 3 == 0){ return 1 }; return 0;"
               }
            }
         ]
      }
   }
}
...
{
   "took": 13,
   "timed_out": false,
   "_shards": {
      "total": 1,
      "successful": 1,
      "failed": 0
   },
   "hits": {
      "total": 10,
      "max_score": 1,
      "hits": [
         {
            "_index": "test_index",
            "_type": "doc",
            "_id": "3",
            "_score": 1,
            "_source": {
               "doc_id": 3
            }
         },
         {
            "_index": "test_index",
            "_type": "doc",
            "_id": "6",
            "_score": 1,
            "_source": {
               "doc_id": 6
            }
         },
         {
            "_index": "test_index",
            "_type": "doc",
            "_id": "9",
            "_score": 1,
            "_source": {
               "doc_id": 9
            }
         }
      ]
   }
}