Removing old indices in elasticsearch

Curator would be an ideal match here.You can find the link here - https://github.com/elastic/curator

A command like below should work just fine -

curator --host <IP> delete indices --older-than 30 --prefix "twitter-" --time-unit days  --timestring '%Y-%m-%d'

You can keep in this in the CRON for removing the indices occasionally.

You can find some examples and docs here - https://www.elastic.co/guide/en/elasticsearch/client/curator/current/examples.html

If you are using elasticsearch version 5.x then you need to install the curator version 4.x.You can see the version compatibility and installation steps from the documentation

Once installed. Then just run the command

curator --config path/config_file.yml [--dry-run] path/action_file.yml

Curator provides a dry-run flag to just output what Curator would have executed. Output will be in your log file which you have defined in config.yml file. If not logging key defined in config_file.yml then currator will output to console. To delete the indices run the above command without --dry-run flag

The configuration file config_file.yml is

---client:  hosts:   - 127.0.0.1  port: 9200logging:  loglevel: INFO  logfile: "/root/curator/logs/actions.log"  logformat: default  blacklist: ['elasticsearch', 'urllib3']

The action file action_file.yml is

---actions:  1:    action: delete_indices    description: >-      Delete indices older than 7 days (based on index name), for logstash-      prefixed indices. Ignore the error if the filter does not result in an      actionable list of indices (ignore_empty_list) and exit cleanly.    options:      ignore_empty_list: True      timeout_override:      continue_if_exception: False      disable_action: False    filters:    - filtertype: pattern      kind: prefix      value: logstash-      exclude:    - filtertype: age      source: name      direction: older      timestring: '%Y.%m.%d'      unit: days      unit_count: 7      exclude:

If you want to delete the indices weekly, monthly, etc automatically. Then just write the bash script like

#!/bin/bash# Script to delete the log event indices of the elasticsearch weekly#This will delete the indices of the last 7 dayscurator --config /path/config_file.yml /path/action_file.yml

Put a shell script in one of these folders: /etc/cron.daily, /etc/cron.hourly, /etc/cron.monthly or /etc/cron.weekly and your job is done.

NOTE: Make sure to use the correct indentation in your configuration and action files. Otherwise it will not work.

I use a bash script, just change the 30 with the # of days you want to keep

#!/bin/bash# Zero padded days using %d instead of %eDAYSAGO=`date --date="30 days ago" +%Y%m%d`ALLLINES=`/usr/bin/curl -s -XGET http://127.0.0.1:9200/_cat/indices?v | egrep logstash`echoecho "THIS IS WHAT SHOULD BE DELETED FOR ELK:"echoecho "$ALLLINES" | while read LINEdo  FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | awk -F'-' '{ print $2 }' | sed 's/\.//g' `   if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]  then    TODELETE=`echo $LINE | awk '{ print $3 }'`    echo "http://127.0.0.1:9200/$TODELETE"  fidoneechoecho -n "if this make sence, Y to continue N to exit [Y/N]:"read INPUTif [ "$INPUT" == "Y" ] || [ "$INPUT" == "y" ] || [ "$INPUT" == "yes" ] || [ "$INPUT" == "YES" ]then  echo "$ALLLINES" | while read LINE  do    FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | awk -F'-' '{ print $2 }' | sed 's/\.//g' `    if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]    then      TODELETE=`echo $LINE | awk '{ print $3 }'`      /usr/bin/curl -XDELETE http://127.0.0.1:9200/$TODELETE      sleep 1      fi  doneelse   echo SCRIPT CLOSED BY USER, BYE ...  echo  exitfi