Limiting the nested fields in Elasticsearch

Question

I am trying to index json documents in ElasticSearch with dynamic mappings on. Some of the documents have unpredictable number of keys (nested levels) because of which I started getting this error from ES Java api.

[ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Limit of total fields [1000] in index [my_index] has been exceeded]]]failure in bulk execution

I was wondering if there is an option which can be configured at index level where I can define to scan for fields only till certain level (maybe 2) and store the rest of the document as a string or in flattened form. I did come across some settings like index.mapping.depth.limit but it seems if I set it to 2, this setting rejects the document if there are more levels. link

Answer 1

For Total Field

PUT <index_name>/_settings
{
  "index.mapping.total_fields.limit": 2000
}

For depth limit

PUT <index_name>/_settings
{
  "index.mapping.depth.limit": 2
}

https://www.elastic.co/guide/en/elasticsearch/reference/master/mapping.html