Parse multiline JSON with grok in logstash

I think I found a working answer to my problem. I am not sure if it's a clean solution, but it helps parse multiline JSONs of the type above.

input {       file     {        codec => multiline        {            pattern => '^\{'            negate => true            what => previous                        }        path => ["/opt/mount/ELK/json/*.json"]        start_position => "beginning"        sincedb_path => "/dev/null"        exclude => "*.gz"    }}filter {    mutate    {        replace => [ "message", "%{message}}" ]        gsub => [ 'message','\n','']    }    if [message] =~ /^{.*}$/     {        json { source => message }    }}output {     stdout { codec => rubydebug }}

My mutliline codec doesn't handle the last brace and therefore it doesn't appear as a JSON to json { source => message }. Hence the mutate filter:

replace => [ "message", "%{message}}" ]

That adds the missing brace. and the

gsub => [ 'message','\n','']

removes the \n characters that are introduced. At the end of it, I have a one-line JSON that can be read by json { source => message }

If there's a cleaner/easier way to convert the original multi-line JSON to a one-line JSON, please do POST as I feel the above isn't too clean.

You will need to use a multiline codec.

input {  file {    codec => multiline {        pattern => '^{'        negate => true        what => previous    }    path => ['/opt/mount/ELK/json/mytestjson.json']  }}filter {  json {    source => message    remove_field => message  }}

The problem you will run into has to do with the last event in the file. It won't show up till there is another event in the file (so basically you'll lose the last event in a file) -- you could append a single { to the file before it gets rotated to deal with that situation.