ElasticSearch disappearing / crashing -- cannot find log files or any other information

If you run a relatively new version of elasticsearch you should be able to find location of the log files by running the following command

curl -s "localhost:9200/_nodes/_local/settings?pretty=true" | fgrep path.logs

But if elasticsearch suddenly disappears, chances are you will not find anything in these logs. If you are running an older version of java (a year old or more), it's quite possible that your java process is crashing. When it happens the JVM typically leaves files named hs_err_pidXXXX.log on your hard drive (typically in /tmp/ on Linux. See if you can find any and try upgrading Java to the latest version.

The second possible reason for sudden disappearance of elasticsearch is oom-killer. It's a process that kicks in and kills the biggest process (elasticsearch in your case) when system runs out of memory. Check syslog and kern.log around the time elasticsearch last time disappeared. If you see messages like this one

Out of memory: Kill process 1234 (java) score 567 or sacrifice child

you might need to reduce elasticsearch heap size to leave more memory to OS or upgrade to a bigger instance.

I experienced same troubles :

• Updated JAVA did not help
• Moving from openJDK to oracle changed nothing
• Memory consumption was finally OK
• No error message anywhere, just JVM crashing randomly

My solution was to remove sigar lib :

cd    sudo mv /usr/share/elasticsearch/lib/sigar .sudo service elasticsearch restart

Since, everything goes smoothly, the only impact: I lost CPU monitoring.ElasticSearch 1.5.1 is running on a VPS with Ubuntu 14.04 (OpenVZ).

UPDATE 1: I still have to remove sigar lib with elasticsearch 1.5.2

UPDATE 2: to figure out, add these options when starting elasticsearch :

-XX:HeapDumpPath=./java_pid.hprof

Path to directory or filename for heap dump. Manageable.

-XX:-HeapDumpOnOutOfMemoryError

Dump heap to file when java.lang.OutOfMemoryError is thrown.

Will help for debugging