multiline fluentd logs in kubernetes
There are at least two ways:
multiline
plugin
Thanks to @rickerp, he suggested multiline
plugin.
The multiline parser plugin parses multiline logs. This plugin is the multiline version of regexp parser.
The multiline parser parses log with formatN and format_firstline parameters. format_firstline is for detecting the start line of the multiline log. formatN, where N's range is [1..20], is the list of Regexp format for multiline log.
Unlike other parser plugins, this plugin needs special code in input plugin e.g. handle format_firstline. So, currently, in_tail plugin works with multiline but other input plugins do not work with it.
fluent-plugin-concat
plugin
As per fluentd
documentation, fluent-plugin-concat
solves this:
Concatenate multiple lines log messages
Application log is stored into
"log"
field in the records. You can concatenate these logs by using fluent-plugin-concat filter before send to destinations.
<filter docker.**>@type concatkey logstream_identity_key container_idmultiline_start_regexp /^-e:2:in `\/'/multiline_end_regexp /^-e:4:in/</filter>
Original events:
2016-04-13 14:45:55 +0900 docker.28cf38e21204: {"container_id":"28cf38e212042225f5f80a56fac08f34c8f0b235e738900c4e0abcf39253a702","container_name":"/romantic_dubinsky","source":"stdout","log":"-e:2:in `/'"}2016-04-13 14:45:55 +0900 docker.28cf38e21204: {"source":"stdout","log":"-e:2:in `do_division_by_zero'","container_id":"28cf38e212042225f5f80a56fac08f34c8f0b235e738900c4e0abcf39253a702","container_name":"/romantic_dubinsky"}2016-04-13 14:45:55 +0900 docker.28cf38e21204: {"source":"stdout","log":"-e:4:in `<main>'","container_id":"28cf38e212042225f5f80a56fac08f34c8f0b235e738900c4e0abcf39253a702","container_name":"/romantic_dubinsky"}
Filtered events:
2016-04-13 14:45:55 +0900 docker.28cf38e21204: {"container_id":"28cf38e212042225f5f80a56fac08f34c8f0b235e738900c4e0abcf39253a702","container_name":"/romantic_dubinsky","source":"stdout","log":"-e:2:in `/'\n-e:2:in `do_division_by_zero'\n-e:4:in `<main>'"}
With the plugin, you'll want to fix some regexes.