Logstash - Use current date as timestamp date
You can add a field with the part of the timestamp that is missing in your log and then concatenate with a variable that contains the hour and use it as your @timestamp field.
The filter below does something like this:
filter { grok { break_on_match => false match => ["message","%{TIME:hour} %{GREEDYDATA:msg}"] tag_on_failure => [ "_grokparsefailure"] add_field => { "time" => "%{+YYYY-MM-dd}"} add_field => { "timestamp" => "%{time} %{hour}" } } date { target => "@timestamp" match => ["timestamp", "YYYY-MM-dd HH:mm:ss.SSS"] }}
First it will match your message with a grok pattern that will extract the hour and save it in a field name hour
, and the rest will be saved in a field name msg
, but you can parse the rest if you want.
Then it will add a field name time
with the pattern YYYY-MM-dd, for example 2018-07-12
.
After that, it will create a field named timestamp
the field time
with the field hour
, which will result in 2018-07-12 4:00:19.675
The date
filter is used to use your generated timestamp as the default timestamp field in elastic, which is @timestamp
.
A logstash output for this filter is something like this:
{ "@timestamp":"2018-07-12T04:00:19.675Z", "message":"04:00:19.675 [ActiveMQ Task-9] INFO a.b.c.t.failover.FailoverTransport - Successfully reconnected to ssl://localhost:12345", "timestamp":"2018-07-12 04:00:19.675", "msg":"[ActiveMQ Task-9] INFO a.b.c.t.failover.FailoverTransport - Successfully reconnected to ssl://localhost:12345", "time":"2018-07-12", "@version":"1", "hour":"04:00:19.675", "host":"logstash-hostname"}