Elasticsearch 1.5.2 deployment issue
- delete the empty indices
- for the 1.5 cluster the major usage of your heap is for fielddata - around 9.5GB for each node, 1.2GB for filter cache and around 1.7GB for segments files' metadata
- even if you have that snippet in your template to make the
strings asnot_analyzed, in 1.5 this doesn't automatically mean ES will usedoc_values, you need to specifically enable them. - if you enable
doc_valuesnow in 1.5.x cluster, the change will be effective with the new indices. For the old indices you need to reindex the data. Or if you have time-based indices (created daily, weekly etc) you just need to wait for the new indices to be created and the old ones to be deleted. - until the
doc_valueswill be predominant in your indices in the 1.5 cluster, what @Val suggested in the comments is the only option: limit the fielddata cache size or add more nodes to your cluster (and implicitly more memory) or increase the RAM on your nodes. Or manually clear the fielddata cache ;-) from time to time.
- even if you have that snippet in your template to make the
- not related to the memory issue entirely, but try to avoid using ttl. If you don't need some data anymore, simply delete the index, don't rely on
ttl, it is much more costly than simply deleting the index. The use ofttlcreates can potentially cause issues at search time and affect the overall performance of a cluster, as it deletes documents from indices, which means a lot of updates and a lot of merging to those indices. Since you probably have time-based indices (which means data from yesterday doesn't really change) using ttl brings unnecessary operations on data that should otherwise be static (and which can potentially be optimized).
If your heap is getting affected rapidly while querying, that means you're doing something really heavy in your query, like for example aggregations. Like val and Andrei suggested, the problem might be with your field data going unbounded. I'd suggest to check your mappings and use doc_values and not_analyzed properties wherever applicable to cut down query cost.