How we can filter namespace in filebeat kubernetes?

If you want Filebeat to only grab logs from certain namespaces you use a condition:

filebeat.yml:

    logging.level: error    logging.json: true    filebeat.config:      inputs:        # Mounted `filebeat-inputs` configmap:        path: ${path.config}/inputs.d/*.yml        # Reload inputs configs as they change:        reload.enabled: false      modules:        path: ${path.config}/modules.d/*.yml        # Reload module configs as they change:        reload.enabled: false    filebeat.autodiscover:      providers:        - type: kubernetes          templates:          - condition:              equals:                kubernetes.namespace: stage            config:              - type: container                paths:                 - /var/log/containers/*${data.kubernetes.container.id}.log                multiline.pattern: '^[[:space:]]'                multiline.negate: false                multiline.match: after                include_lines: ['^{']

Note this part:

          templates:          - condition:              equals:                kubernetes.namespace: stage

I do run a Filebeat as a Daemonset in each Namespace. It's a bit of extra overhead but Filebeat can be finicky so that does help us work out issues in other logical environments first.

how to drop some namespaces, i documented here: https://ezyforanykey.blogspot.com/2020/11/filebeat-exclude-kubernetes-namespace.html

example is below:

- type: container      paths:        - /var/log/containers/*.log      exclude_files:        - /var/log/containers/java.*      processors:        - add_kubernetes_metadata:            host: ${NODE_NAME}            matchers:            - logs_path:                logs_path: "/var/log/containers/"        - drop_event.when:            or:            - equals:                kubernetes.namespace: "kube-system"            - equals:                kubernetes.namespace: "calico-system"

I don't know how to filter filebeat (or even if it's possible), but you can filter on fields in the output part of your logstash configuration, using conditionals:

output {    if [kubernetes][namespace] == "fluentd" {        ...        Send to Elasticsearch        ...    } else {        ...    }}

This way you can choose different actions to take on each message, depending on the value of the kubernetes.namespace field.