Why is Elasticsearch starting manually but not starting as a service on Ubuntu 16.04?

As @Suaro states: our problem is related to what is discussed in this question with nearly identical symptoms and this thread that reiterates the START_DAEMON solution in the first link. That is a good start, but for me that didn't offer a comprehensive fix. I looked deeper at permissions, heap size, and ownership to find a final solution.

These are the steps for Elasticsearch 6 on Ubuntu 16.04 installed per the Elasticsearch debian installation instructions as of Jan 2018:

1. Set START_DAEMON=true in /etc/default/elasticsearch and restart service.
2. If the system has 2GB of RAM (true in my case), set ES_HEAP_SIZE=1g
3. Check the permissions of the elasticsearch directory in /usr/share/elasticsearch. Chances are that root owns these, which is not ideal. If you don't know already, running any service as root exposes your infrastructure to exploitation by attackers.
4. The temptation in #3 is to set ES_USER=root and ES_GROUP=root which will solve your problem. Elasticsearch will start as a service (even though their product documentation alleges that ES won't operate as root). DON'T DO THAT.

5. Instead, check that the elasticsearch user exists locally and that the group of the same name exists too.

   $ cut -d: -f1 /etc/passwd$ cut -d: -f1 /etc/group

6. Then, change ownership of all elasticsearch folders and resources to the elasticsearch user and group.

   $ ~ $ > ll /usr/share/elasticsearch/total 8.0Kdrwxr-xr-x 2 elasticsearch 4.0K Jan 22 10:02 bin/lrwxrwxrwx 1 elasticsearch   18 Dec 24  2015 config -> /etc/elasticsearch/lrwxrwxrwx 1 elasticsearch   22 Dec 24  2015 data -> /var/lib/elasticsearch/lrwxrwxrwx 1 elasticsearch   22 Dec 24  2015 logs -> /var/log/elasticsearch/drwxr-xr-x 2 elasticsearch 4.0K Dec 24  2015 plugins/# ^-- Take note that the symlinked directories need to be adjusted too$ sudo chown -R elasticsearch:elasticsearch /usr/share/elasticsearch$ sudo chown -R elasticsearch:elasticsearch /etc/elasticsearch/$ sudo chown -R elasticsearch:elasticsearch /var/lib/elasticsearch/$ sudo chown -R elasticsearch:elasticsearch /var/log/elasticsearch/

7. Then, set the values ES_USER=elasticsearch and ES_GROUP=elasticsearch in /etc/default/elasticsearch if they aren't set that way already (in case you gave in to temptation per #4).

8. Try it again...

   ~ $ > sudo service elasticsearch status● elasticsearch.service - LSB: Starts elasticsearch   Loaded: loaded (/etc/init.d/elasticsearch; bad; vendor preset: enabled)   Active: active (exited) since Mon 2018-01-22 20:51:29 UTC; 2min 25s ago     Docs: man:systemd-sysv-generator(8)  Process: 3133 ExecStop=/etc/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)  Process: 3209 ExecStart=/etc/init.d/elasticsearch start (code=exited, status=0/SUCCESS)~ $ > sudo systemctl restart elasticsearch.service~ $ > sudo service elasticsearch status● elasticsearch.service - LSB: Starts elasticsearch   Loaded: loaded (/etc/init.d/elasticsearch; bad; vendor preset: enabled)   Active: active (running) since Mon 2018-01-22 20:54:05 UTC; 2s ago     Docs: man:systemd-sysv-generator(8)  Process: 3306 ExecStop=/etc/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)  Process: 3340 ExecStart=/etc/init.d/elasticsearch start (code=exited, status=0/SUCCESS)   CGroup: /system.slice/elasticsearch.service           └─3391 /usr/lib/jvm/java-8-openjdk-armhf/bin/java -Xms1g -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.pidfilJan 22 20:54:04 lego systemd[1]: Starting LSB: Starts elasticsearch...Jan 22 20:54:05 lego elasticsearch[3340]:  * Starting Elasticsearch ServerJan 22 20:54:05 lego elasticsearch[3340]:    ...done.Jan 22 20:54:05 lego systemd[1]: Started LSB: Starts elasticsearch.Jan 22 20:51:29 lego systemd[1]: Starting LSB: Starts elasticsearch...

Et Violà!

~ $ > curl -XGET 'localhost:9200/?pretty'{  "status" : 200,  "name" : "Svarog",  "cluster_name" : "elasticsearch",  "version" : {    "number" : "1.7.3",    "build_hash" : "NA",    "build_timestamp" : "NA",    "build_snapshot" : false,    "lucene_version" : "4.10.4"  },  "tagline" : "You Know, for Search"}

Possibly your problem is related to what is discussed in this question.

Can not start elasticsearch as a service in ubuntu 16.04

and

https://discuss.elastic.co/t/cant-start-elasticsearch-with-ubuntu-16-04/48730/9

If is your case, you must change START_DAEMON to true in /etc/default/elasticsearch and restart service.