How to forward application logs to Splunk from docker container?
There are a lot of options to send logs from containers to Splunk.
For logs, sent to Standard Output and Error:
- Splunk Logging Driver https://docs.docker.com/v17.09/engine/admin/logging/splunk/
- Splunk Docker logging plugin https://github.com/splunk/docker-logging-plugin - an improved version of Splunk Logging Driver
For application logs (logs written inside of the container):
- Sidecars with UF
Our company (https://www.outcoldsolutions.com) offers one solution that can simply forward container (https://www.outcoldsolutions.com/docs/monitoring-docker/v5/) and application logs (https://www.outcoldsolutions.com/docs/monitoring-docker/v5/annotations/#application-logs) from the Docker hosts, and collect metrics. We also provide you with an application in Splunk for tracking the health and performance of your clusters https://splunkbase.splunk.com/app/3723/. Our application is not free, but cheap compared to the time you can spend building something similar.
Another option is using fluentd as an intermediary.
Fluentd exists as docker logging driver as well, but you can use it to redirect the logs to several backends (Splunk, Elasticsearch). You are not as tightly coupled to Splunk.
Additionally that's the way proposed by Openshift.
It looks like Docker has a logging driver that handles this