Drop log messages containing a specific string
To drop the message that does not contain the string xyz
:
if ([message] !~ "xyz") { drop { }}
Your grok pattern is not grabbing the date part of your logs.
Once you have a field from your grok pattern containing the date, you can invoque the date filter on this field.
So your grok filter should look like this:
grok { match => { "message" => '%{SYSLOG5424SD:loglevel} <%{JAVACLASS:job}> %{TIMESTAMP_ISO8601:Date} %{GREEDYDATA:content}' }}
I added a part to grab the date, which will be in the field Date
. Then you can use the date filter:
date { match => [ "Date", "YYYY-mm-dd HH:mm:ss,SSS" ] locale => en}
I added the ,SSS
so that the format match the one from the Date
field.The parsed date will be stored in the @timestamp
field, unless specified differently with the target
parameter.
to check if your message contains a substring, you can do:
if [message] =~ "a" { mutate { add_field => { "hello" => "world" } }}
So in your case you can use the if to invoke the drop{} filter, or you can wrap your output plugin in it.
To parse a date and write it back to your timestamp field, you can use something like this:
date { locale => "en" match => ["timestamp", "ISO8601"] timezone => "UTC" target => "@timestamp" add_field => { "debug" => "timestampMatched"}}
This matches my timestamp in:
- Source field: "timestamp" (see match)
- Format is "ISO...", you can use a custom format that matches your timestamp
- timezone - self explanatory
- target - write it back into the event's "@timestamp" field
- Add a debug field to check that it has been matched correctly
Hope that helps,
Artur