Fluent Bit: Logstash_Prefix_Key is not working as expected with 'es' output plugin

Question

I am trying to lookup a key from a record and use it as logstash prefix in fluent bit. But that's not happening and Logstash_Prefix is not being replaced by Logstash_Prefix_Key even though the specified key exists in the enriched log from kubernetes filter.

The ideal behaviour of a kubernetes filter is to enrich the logs read from input path via input plugin with kubernetes data such as pod name, pod id, namespace name etc. And when the logs after applying the filter is pushed to output source via es output plugin. I used Logstash_Prefix_Key to get the key kubernetes.pod_name and gave Logstash_Prefix as pod_name. Even though I'm able to see kubernetes.pod_name key in Kibana, the logs are getting stored in the prefix pod_name (which means Logstash_Prefix_Key is not found tn log records so it uses Logstash_Prefix).

Code sample

input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     2GB
        Skip_Long_Lines   On
        Refresh_Interval  10

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc.cluster.local:443
        Merge_Log           Off
        K8S-Logging.Parser  On

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           kube.*
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}
        HTTP_User       ${FLUENT_ELASTICSEARCH_USER}
        HTTP_Passwd     ${FLUENT_ELASTICSEARCH_PASSWORD}
        Logstash_Format On
        Logstash_Prefix pod_name
        Logstash_Prefix_Key kubernetes.pod_name
        Retry_Limit     False

Since I am new to EFK stack, Could someone help me with this

Answer 1

UPD: it's now supported! https://github.com/fluent/fluent-bit/issues/421#issuecomment-766912018 Should be in Fluent Bit v1.7 release!

---

Dynamic ElasticSearch indexes are not supported in FluentBit at the moment. Here's a related issue: https://github.com/fluent/fluent-bit/issues/421. You can only specify a string (hardcoded) prefixes.

The workaround is to use a fluentd log collector instead, which supports dynamic indexes: https://docs.fluentd.org/output/elasticsearch#index_name-optional. There's a community chart for it: https://github.com/helm/charts/tree/master/stable/fluentd

UPD: it's now supported! https://github.com/fluent/fluent-bit/issues/421#issuecomment-766912018 Should be in Fluent Bit v1.7 release!