If you have ever worked with Elasticsearch, then you are probably familiar with one of the most important features of Elasticsearch - the Dynamic Field Mapping:
By default, when a previously unseen field is found in a document, Elasticsearch will add the new field to the type mapping.
Therefore, if you need to index documents with high similarity between their field names (dense data), including field types, then this default behaviour may be exactly what you want. However, if your documents have high variation of field names (sparse data), or have same field names but with different types, then you will need a different approach.
In this post I would like to show you how to create an Elasticsearch index that can be used to index arbitrary JSON data, including data with nested arrays and objects. All this, without exploding the index type mapping with arbitrary properties originating from the indexed data. Nevertheless, the indexed data will still be searchable by any of its fields, including any nested fields by specifying their path in a “dot” notation format. In addition, I will show how to use Elasticsearch aggregations to fetch a list of all of the available fields and their respective types of the indexed data.