Logstash - Use current date as timestamp date Logstash - Use current date as timestamp date elasticsearch elasticsearch

Logstash - Use current date as timestamp date


You can add a field with the part of the timestamp that is missing in your log and then concatenate with a variable that contains the hour and use it as your @timestamp field.

The filter below does something like this:

filter {    grok {        break_on_match => false        match => ["message","%{TIME:hour} %{GREEDYDATA:msg}"]        tag_on_failure => [ "_grokparsefailure"]        add_field => { "time" => "%{+YYYY-MM-dd}"}        add_field => { "timestamp" => "%{time} %{hour}" }    }    date {        target => "@timestamp"        match => ["timestamp", "YYYY-MM-dd HH:mm:ss.SSS"]    }}

First it will match your message with a grok pattern that will extract the hour and save it in a field name hour, and the rest will be saved in a field name msg, but you can parse the rest if you want.

Then it will add a field name time with the pattern YYYY-MM-dd, for example 2018-07-12.

After that, it will create a field named timestamp the field time with the field hour, which will result in 2018-07-12 4:00:19.675

The date filter is used to use your generated timestamp as the default timestamp field in elastic, which is @timestamp.

A logstash output for this filter is something like this:

{ "@timestamp":"2018-07-12T04:00:19.675Z", "message":"04:00:19.675 [ActiveMQ Task-9] INFO a.b.c.t.failover.FailoverTransport - Successfully reconnected to ssl://localhost:12345", "timestamp":"2018-07-12 04:00:19.675", "msg":"[ActiveMQ Task-9] INFO  a.b.c.t.failover.FailoverTransport - Successfully reconnected to ssl://localhost:12345", "time":"2018-07-12", "@version":"1", "hour":"04:00:19.675", "host":"logstash-hostname"}