Serilog logs collected by Fluentbit to Elasticsearch in kubernetes doesnt get Json-parsed correctly Serilog logs collected by Fluentbit to Elasticsearch in kubernetes doesnt get Json-parsed correctly kubernetes kubernetes

Serilog logs collected by Fluentbit to Elasticsearch in kubernetes doesnt get Json-parsed correctly


Problem was bad fluentbit configmap. This works:

apiVersion: v1kind: ConfigMapmetadata:  name: fluent-bit-config  namespace: logging  labels:    k8s-app: fluent-bitdata:  # Configuration files: server, input, filters and output  # ======================================================  fluent-bit.conf: |    [SERVICE]        Flush         1        Log_Level     info        Daemon        off        Parsers_File  parsers.conf        HTTP_Server   On        HTTP_Listen   0.0.0.0        HTTP_Port     2020            @INCLUDE input-kubernetes.conf    @INCLUDE filter-kubernetes.conf    @INCLUDE output-elasticsearch.conf  input-kubernetes.conf: |    [INPUT]        Name              tail        Tag               kube.*        Path              /var/log/containers/*.log        Parser            docker        DB                /var/log/flb_kube.db        Mem_Buf_Limit     5MB        Skip_Long_Lines   On        Refresh_Interval  10  filter-kubernetes.conf: |    [FILTER]        Name                kubernetes        Match               kube.*        Kube_URL            https://kubernetes.default.svc:443        # These two may fix some duplicate field exception        Merge_Log           On        Merge_JSON_Key      k8s        K8S-Logging.Parser  On        K8S-Logging.exclude True  output-elasticsearch.conf: |    [OUTPUT]        Name            es        Match           *        Host            ${FLUENT_ELASTICSEARCH_HOST}        Port            ${FLUENT_ELASTICSEARCH_PORT}        Logstash_Format On        # This fixes errors where kubernetes.apps.name must object        Replace_Dots    On         Retry_Limit     False        Type            flb_type        # This may fix some duplicate field exception        Time_Key        @timestamp_es        # The Index Prefix:        Logstash_Prefix logstash_07  parsers.conf: |    [PARSER]        Name   apache        Format regex        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$        Time_Key time        Time_Format %d/%b/%Y:%H:%M:%S %z    [PARSER]        Name   apache2        Format regex        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$        Time_Key time        Time_Format %d/%b/%Y:%H:%M:%S %z    [PARSER]        Name   apache_error        Format regex        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$    [PARSER]        Name   nginx        Format regex        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$        Time_Key time        Time_Format %d/%b/%Y:%H:%M:%S %z    [PARSER]        Name   json        Format json        Time_Key time        Time_Format %d/%b/%Y:%H:%M:%S %z    [PARSER]        Name        docker        Format      json        #Time_Key    time        Time_Key    @timestamp        Time_Format %Y-%m-%dT%H:%M:%S.%L        Time_Keep   Off # on        # See: https://fluentbit.io/documentation/0.14/parser/decoder.html        # Command      |  Decoder | Field | Optional Action        # =============|==================|=================        # Decode_Field_As   escaped    log        # Decode_Field_As   escaped    log    do_next        # Decode_Field_As   json       log         [PARSER]        Name        syslog        Format      regex        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$        Time_Key    time        Time_Format %b %d %H:%M:%S