Skip to content

How to rename or copy and rename nested fields? #11324

@Ants5

Description

@Ants5

`

[INPUT]
    Name                tail
    Tag                 charging-us-prod-vmlogs.*
    Path                /var/log/containers/hes-hes-2030*_charging-us-prod_*.log
    Parser              cri
    DB                  /var/fluent-bit/state/charging-us-prod-vmlogs.db
    Mem_Buf_Limit       100MB
    Skip_Long_Lines     On
    Skip_Empty_Lines    On
    Refresh_Interval    5
    Rotate_Wait         20
    storage.type        filesystem
    #Read_from_Head      off

[FILTER]
    Name                kubernetes
    Match               charging-us-prod-vmlogs.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_Tag_Prefix     charging-us-prod-vmlogs.var.log.containers.
    Merge_Log           On
    Keep_Log            On
    namespace_labels    On
    K8S-Logging.Parser  On
    K8S-Logging.Exclude Off
    Labels              Off
    Annotations         Off
[FILTER]
    Name                nest
    Match               charging-us-prod-vmlogs.*
    Operation           lift
    Nested_under        kubernetes
    Add_prefix          kubernetes.
[FILTER]
    Name                record_modifier
    Match               charging-us-prod-vmlogs.*
    Record              area   usa
    Record              type   test
    Remove_key          kubernetes.pod_id
    Remove_key          kubernetes.pod_ip
    Remove_key          kubernetes.docker_id
    Remove_key          kubernetes.container_hash
    Remove_key          kubernetes.container_image
    #Remove_key          kubernetes.container_name
    #Remove_key          kubernetes.pod_name
    #Remove_key          kubernetes.namespace_name
[FILTER]
    Name                modify
    Match               charging-us-prod-vmlogs.*
    Rename              kubernetes.container_name   service_name
    Rename              kubernetes.namespace_name   namespace
    Rename              kubernetes.pod_name   pod
    Rename              kubernetes.host   nodename
    #Set                 cluster ${kubernetes_namespace.labels.cluster}
    #Set                 cluster ${kubernetes_namespace.labels.project}
    #Rename              kubernetes_namespace.labels.cluster  cluster
    #Rename              kubernetes_namespace.labels.project  project
    #Remove              kubernetes_namespace

#[FILTER]
#    Name                nest
#    Match               charging-us-prod-vmlogs.*
#    Operation           nest
#    Wildcard            kubernetes.*
#    Nest_under          kubernetes
#    Remove_prefix       kubernetes.

[OUTPUT]
    Name                stdout
    Match               charging-us-prod-vmlogs.*
[OUTPUT]
    Name                http
    Match               charging-us-prod-vmlogs.*
    host                localhost
    port                8427
    http_user           aaaaa
    http_passwd         123123
    header              AccountID 0
    header              ProjectID 0
    uri                 /insert/jsonline?_stream_fields=project,cluster,namespace,service_name,type&_msg_field=log
    format              json_lines
    json_date_format    iso8601
    compress            gzip

`

My data collection configuration is shown above. The Kubernetes plugin has namespace_labels enabled to retrieve labels from the namespace, but the retrieved labels are in nested JSON format. How can I rename kubernetes_namespace.labels.cluster to the cluster name without unnesting the tags? My log format is as follows.

charging-us-prod-vmlogs.var.log.containers.hes-hes-2030-795697479c-ps9dc_charging-us-prod_hes-hes-2030-102360dc7fd5eaab6fe5b984d1a742884ed36bb43864328b284e174a3d8da43c.log: [[1767168962.880809955, {}], {"stream"=>"stdout", "logtag"=>"F", "log"=>"{"@timestamp":"2025-12-31T08:16:02.880Z","caller":"util/httputil.go:130","content":"[HTTP_SUCCESS] DNSP:synergy, Method:POST, Href:/sep2/edev/144/sub, Body:\"[Subscription xmlns=\\\"urn:ieee:std:2030.5:ns\\\"]\\n [subscribedResource]/sep2/derp/1976/derc[/subscribedResource]\\n [encoding]0[/encoding]\\n [level]+S1[/level]\\n [limit]0[/limit]\\n [notificationURI]https://ankerpower-api.anker.com/charging_hes_2030_svc/ntfy/synergy/2F266B0212F96DF43FCD4AF2E51B007D00060981[/notificationURI]\\n[/Subscription]\", Response:, Elapsed:221ms, Status:201,Location:/sep2/sub/165","level":"info"}", "@timestamp"=>"2025-12-31T08:16:02.880Z", "caller"=>"util/httputil.go:130", "content"=>"[HTTP_SUCCESS] DNSP:synergy, Method:POST, Href:/sep2/edev/144/sub, Body:"[Subscription xmlns=\"urn:ieee:std:2030.5:ns\"]\n [subscribedResource]/sep2/derp/1976/derc[/subscribedResource]\n [encoding]0[/encoding]\n [level]+S1[/level]\n [limit]0[/limit]\n [notificationURI]https://ankerpower-api.anker.com/charging_hes_20_svc/ntfy/synergy/2F266B0212F43FCD4AF2EB007D00060981[/notificationURI]\n[/Subscription]", Response:, Elapsed:221ms, Status:201,Location:/sep2/sub/165", "level"=>"info", "kubernetes_namespace"=>{"name"=>"charging-us-prod", "labels"=>{"cluster"=>"aiot-us-prod", "dynakube.internal.dynatrace.com/instance"=>"aiot-qa", "field.cattle.io/projectId"=>"p-7w2zz", "kubernetes.io/metadata.name"=>"charging-us-prod", "project"=>"charging"}}, "pod"=>"hea-hes-22230-795697479c-ps9dc", "namespace"=>"charging-us-prod", "nodename"=>"ip-11-11-11-153.us-east-2.compute.internal", "service_name"=>"hea-hes-22230", "area"=>"usa", "type"=>"test"}]

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions