-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Description
`
[INPUT]
Name tail
Tag charging-us-prod-vmlogs.*
Path /var/log/containers/hes-hes-2030*_charging-us-prod_*.log
Parser cri
DB /var/fluent-bit/state/charging-us-prod-vmlogs.db
Mem_Buf_Limit 100MB
Skip_Long_Lines On
Skip_Empty_Lines On
Refresh_Interval 5
Rotate_Wait 20
storage.type filesystem
#Read_from_Head off
[FILTER]
Name kubernetes
Match charging-us-prod-vmlogs.*
Kube_URL https://kubernetes.default.svc:443
Kube_Tag_Prefix charging-us-prod-vmlogs.var.log.containers.
Merge_Log On
Keep_Log On
namespace_labels On
K8S-Logging.Parser On
K8S-Logging.Exclude Off
Labels Off
Annotations Off
[FILTER]
Name nest
Match charging-us-prod-vmlogs.*
Operation lift
Nested_under kubernetes
Add_prefix kubernetes.
[FILTER]
Name record_modifier
Match charging-us-prod-vmlogs.*
Record area usa
Record type test
Remove_key kubernetes.pod_id
Remove_key kubernetes.pod_ip
Remove_key kubernetes.docker_id
Remove_key kubernetes.container_hash
Remove_key kubernetes.container_image
#Remove_key kubernetes.container_name
#Remove_key kubernetes.pod_name
#Remove_key kubernetes.namespace_name
[FILTER]
Name modify
Match charging-us-prod-vmlogs.*
Rename kubernetes.container_name service_name
Rename kubernetes.namespace_name namespace
Rename kubernetes.pod_name pod
Rename kubernetes.host nodename
#Set cluster ${kubernetes_namespace.labels.cluster}
#Set cluster ${kubernetes_namespace.labels.project}
#Rename kubernetes_namespace.labels.cluster cluster
#Rename kubernetes_namespace.labels.project project
#Remove kubernetes_namespace
#[FILTER]
# Name nest
# Match charging-us-prod-vmlogs.*
# Operation nest
# Wildcard kubernetes.*
# Nest_under kubernetes
# Remove_prefix kubernetes.
[OUTPUT]
Name stdout
Match charging-us-prod-vmlogs.*
[OUTPUT]
Name http
Match charging-us-prod-vmlogs.*
host localhost
port 8427
http_user aaaaa
http_passwd 123123
header AccountID 0
header ProjectID 0
uri /insert/jsonline?_stream_fields=project,cluster,namespace,service_name,type&_msg_field=log
format json_lines
json_date_format iso8601
compress gzip
`
My data collection configuration is shown above. The Kubernetes plugin has namespace_labels enabled to retrieve labels from the namespace, but the retrieved labels are in nested JSON format. How can I rename kubernetes_namespace.labels.cluster to the cluster name without unnesting the tags? My log format is as follows.
charging-us-prod-vmlogs.var.log.containers.hes-hes-2030-795697479c-ps9dc_charging-us-prod_hes-hes-2030-102360dc7fd5eaab6fe5b984d1a742884ed36bb43864328b284e174a3d8da43c.log: [[1767168962.880809955, {}], {"stream"=>"stdout", "logtag"=>"F", "log"=>"{"@timestamp":"2025-12-31T08:16:02.880Z","caller":"util/httputil.go:130","content":"[HTTP_SUCCESS] DNSP:synergy, Method:POST, Href:/sep2/edev/144/sub, Body:\"[Subscription xmlns=\\\"urn:ieee:std:2030.5:ns\\\"]\\n [subscribedResource]/sep2/derp/1976/derc[/subscribedResource]\\n [encoding]0[/encoding]\\n [level]+S1[/level]\\n [limit]0[/limit]\\n [notificationURI]https://ankerpower-api.anker.com/charging_hes_2030_svc/ntfy/synergy/2F266B0212F96DF43FCD4AF2E51B007D00060981[/notificationURI]\\n[/Subscription]\", Response:, Elapsed:221ms, Status:201,Location:/sep2/sub/165","level":"info"}", "@timestamp"=>"2025-12-31T08:16:02.880Z", "caller"=>"util/httputil.go:130", "content"=>"[HTTP_SUCCESS] DNSP:synergy, Method:POST, Href:/sep2/edev/144/sub, Body:"[Subscription xmlns=\"urn:ieee:std:2030.5:ns\"]\n [subscribedResource]/sep2/derp/1976/derc[/subscribedResource]\n [encoding]0[/encoding]\n [level]+S1[/level]\n [limit]0[/limit]\n [notificationURI]https://ankerpower-api.anker.com/charging_hes_20_svc/ntfy/synergy/2F266B0212F43FCD4AF2EB007D00060981[/notificationURI]\n[/Subscription]", Response:, Elapsed:221ms, Status:201,Location:/sep2/sub/165", "level"=>"info", "kubernetes_namespace"=>{"name"=>"charging-us-prod", "labels"=>{"cluster"=>"aiot-us-prod", "dynakube.internal.dynatrace.com/instance"=>"aiot-qa", "field.cattle.io/projectId"=>"p-7w2zz", "kubernetes.io/metadata.name"=>"charging-us-prod", "project"=>"charging"}}, "pod"=>"hea-hes-22230-795697479c-ps9dc", "namespace"=>"charging-us-prod", "nodename"=>"ip-11-11-11-153.us-east-2.compute.internal", "service_name"=>"hea-hes-22230", "area"=>"usa", "type"=>"test"}]