Send logs to ELK container from Docker containers (without Filebeat)

SEBP/ELK was the wrong tool to tackle this problem. Instead, I should have been using a project that spins up a container for each of the elements of the ELK stack: Elasticsearch, Logstash, and Kibana. I found just such a repository on GitHub.

The deviantony/docker-elk project combines the three ELK elements into a working set of containers. The great thing about this is that unlike the SEBP/ELK project, deviantony/docker-elk doesn't take an opinionated view about what features should be available and what should be closed off. In the SEBP/ELK project, the ability to write to port 5000 is removed and when you try to add it back via a custom logstash.conf file, the UDP listener ultimately fails. Conversely, the deviantony/docker-elk project just works.

Bonus points: This project also has a branch that includes X-Pack which adds a minimal layer of security out of the box.

Using filebeat in each container is against Docker's philosophy. It will be waste of resources, And have more management overhead.

You can use local log file via logstash.

Example config:

input {  file {    path => "/var/log/apache.log"    type => "apache-access"  # a type to identify those logs (will need this later)    start_position => "beginning"  }}

Now we have to make the log files local to logstash container:

If you are using bind mounts you can mount the same directory in logstash container.

sudo docker run -d -v /path/to/logs/:/path/to/logs/in/container logstash

If you are using volumes you can mount the same volume that contains logs to logstash too.

sudo docker run -d -v logvol:/path/to/logs/in/container logstash