Drop log messages containing a specific string Drop log messages containing a specific string elasticsearch elasticsearch

Drop log messages containing a specific string


To drop the message that does not contain the string xyz:

if ([message] !~ "xyz") {    drop { }}

Your grok pattern is not grabbing the date part of your logs.
Once you have a field from your grok pattern containing the date, you can invoque the date filter on this field.
So your grok filter should look like this:

grok {    match => {        "message" => '%{SYSLOG5424SD:loglevel}  <%{JAVACLASS:job}>       %{TIMESTAMP_ISO8601:Date} %{GREEDYDATA:content}'    }}

I added a part to grab the date, which will be in the field Date. Then you can use the date filter:

date {    match => [ "Date", "YYYY-mm-dd HH:mm:ss,SSS" ]    locale => en}

I added the ,SSS so that the format match the one from the Date field.The parsed date will be stored in the @timestamp field, unless specified differently with the target parameter.


to check if your message contains a substring, you can do:

if [message] =~ "a" {   mutate {      add_field => { "hello" => "world" }   }}

So in your case you can use the if to invoke the drop{} filter, or you can wrap your output plugin in it.

To parse a date and write it back to your timestamp field, you can use something like this:

date {    locale => "en"    match => ["timestamp", "ISO8601"]    timezone => "UTC"    target => "@timestamp"    add_field => { "debug" => "timestampMatched"}}

This matches my timestamp in:

  • Source field: "timestamp" (see match)
  • Format is "ISO...", you can use a custom format that matches your timestamp
  • timezone - self explanatory
  • target - write it back into the event's "@timestamp" field
  • Add a debug field to check that it has been matched correctly

Hope that helps,

Artur