How to define seperated indexes for different logs in Filebeat/ELK?
In your Filebeat configuration you can use document_type
to identify the different logs that you have. Then inside of Logstash you can set the value of the type
field to control the destination index.
However before you separate your logs into different indices you should consider leaving them in a single index and using either type
or some custom field to distinguish between log types. See index vs type.
Example Filebeat prospector config:
filebeat: prospectors: - paths: - /var/log/redis/*.log document_type: redis - paths: - /var/log/python/*.log document_type: python - paths: - /var/log/mongodb/*.log document_type: mongodb
Example Logstash config:
input { beats { port => 5044 }}output { # Customize elasticsearch output for Filebeat. if [@metadata][beat] == "filebeat" { elasticsearch { hosts => "localhost:9200" manage_template => false # Use the Filebeat document_type value for the Elasticsearch index name. index => "%{[@metadata][type]}-%{+YYYY.MM.dd}" document_type => "log" } }}
filebeat.yml
filebeat.prospectors:- input_type: log paths: - /var/log/*.log fields: {log_type: toolsmessage}- input_type: log paths: - /etc/httpd/logs/ssl_access_* fields: {log_type: toolsaccess}
in the logstash.conf.
input { beats { port => "5043" }}filter { if ([fields][log_type] == "toolsmessage") { mutate { replace => { "[type]" => "toolsmessage" } } } else if ([fields][log_type] == "toolsaccess") { mutate { replace => { "[type]" => "toolsaccess" } } }}output { elasticsearch { hosts => ["10.111.119.211:9200"] index => "%{type}_index" } #stdout { codec => rubydebug }}
In logstash you can define multiple input, filter or output plugins with the help of tags:
input { file { type => "redis" path => "/home/redis/log" } file { type => "python" path => "/home/python/log" }} filter { if [type] == "redis" { # processing ....... } if [type] == "python" { # processing ....... }}output { if [type] == "redis" { # output to elasticsearch redis index => "redis" } if [type] == "python" { # output to elasticsearch python index => "python" }}