How to initialize Splunk HTTP Event Collector via Docker Compose and use it with splunk logging driver How to initialize Splunk HTTP Event Collector via Docker Compose and use it with splunk logging driver docker docker

How to initialize Splunk HTTP Event Collector via Docker Compose and use it with splunk logging driver


The configuration of the new image (7.2.0) says that you can specify an HTTP Event Collector token with the environment variable https://github.com/splunk/docker-splunk/blob/48d5322bc574792a5bfbfe8f68769aa16e7688b7/documentation/ADVANCED.md#valid-enterprise-environment-variables

But I don't think it works for single instance after looking at https://github.com/splunk/splunk-ansible/search?q=set_as_hec_receiver.yml&unscoped_q=set_as_hec_receiver.yml - seems like that playbook will be executed only for heavy-weight-forwarder and indexer.

Alternatively, if you will look at the "legacy"/community supported image you will find a different way of doing that. As an example, you can take a look at the app-boilerplate that we use at Outcold Solutions for developing Splunk apps https://github.com/outcoldsolutions/splunk-app-boilerplate, where we:

  1. Map configurations https://github.com/outcoldsolutions/splunk-app-boilerplate/blob/master/Makefile#L23
  2. Copy it over https://github.com/outcoldsolutions/splunk-app-boilerplate/blob/master/Makefile#L26

To solve this issue "Another issue is that Splunk takes some time to start up, and before it starts listening the app service fails to build because the logging driver cannot connect." - please take a look on option splunk-verify-connection (see https://docs.docker.com/config/containers/logging/splunk/#splunk-options), in that way it will keep retrying to send the data over and over till the HTTP Event Collector will be available.

As alternative to splunk-verify-connection you can also use a different approach of forwarding logs to Splunk, by using Outcold Solutions collector, that forwards container logs from JSON logs. It is easy to install https://www.outcoldsolutions.com/docs/monitoring-docker/v5/installation/, and you will be able to use an application for monitoring your docker environments as well https://splunkbase.splunk.com/app/3723/