From 4e84c15da18c387caf033feddad381381d120189 Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Wed, 21 May 2025 08:50:28 -0700 Subject: [PATCH] pipeline: filter: kubernetes: incorporating feedback form @gguillotte Signed-off-by: Lynette Miles --- pipeline/filters/kubernetes.md | 72 +++++++++++++++++----------------- 1 file changed, 36 insertions(+), 36 deletions(-) diff --git a/pipeline/filters/kubernetes.md b/pipeline/filters/kubernetes.md index dc85dff0f..8a33a9c0a 100644 --- a/pipeline/filters/kubernetes.md +++ b/pipeline/filters/kubernetes.md @@ -25,39 +25,39 @@ The plugin supports the following configuration parameters: | Key | Description | Default | | :--- | :--- | :--- | -| `Buffer_Size` | Set the buffer size for HTTP client when reading responses from Kubernetes API server. The value must be according to the [unit size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. A value of `0` results in no limit, and the buffer will expand as-needed. If pod specifications exceed the buffer limit, the API response is discarded when retrieving metadata, and some Kubernetes metadata will fail to be injected to the logs. | `32k` | +| `Buffer_Size` | Set the buffer size for HTTP client when reading responses from Kubernetes API server. The value must conform to the [unit size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. A value of `0` results in no limit, and the buffer will expand as-needed. If pod specifications exceed the buffer limit, the API response is discarded when retrieving metadata, and some Kubernetes metadata will fail to be injected to the logs. | `32k` | | `Kube_URL` | API Server end-point | [https://kubernetes.default.svc:443](https://kubernetes.default.svc:443) | | `Kube_CA_File` | CA certificate file | `/var/run/secrets/kubernetes.io/serviceaccount/ca.crt` | | `Kube_CA_Path` | Absolute path to scan for certificate files | _none_ | | `Kube_Token_File` | Token file | `/var/run/secrets/kubernetes.io/serviceaccount/token` | -| `Kube_Tag_Prefix` | When the source records come from `tail` input plugin, this option specifies the prefix used in `tail` configuration. | `kube.var.log.containers.` | -| `Merge_Log` | When enabled, it checks if the `log` field content is a JSON string map. If yes, it appends the map fields as part of the log structure. | `Off` | -| `Merge_Log_Key` | When `Merge_Log` is enabled, the filter tries to assume the `log` field from the incoming message is a JSON string message and make a structured representation of it at the same level of the `log` field in the map. If `Merge_Log_Key` is set (a string name), all the new structured fields taken from the original `log` content are inserted under the new key. | _none_ | +| `Kube_Tag_Prefix` | When the source records come from the `tail` input plugin, this option specifies the prefix used in `tail` configuration. | `kube.var.log.containers.` | +| `Merge_Log` | When enabled, check if the `log` field content is a JSON string map. If it is, append the map fields as part of the log structure. | `Off` | +| `Merge_Log_Key` | When `Merge_Log` is enabled, the filter assumes the `log` field from the incoming message is a JSON string message and attempts to create a structured representation of it at the same level of the `log` field in the map. If `Merge_Log_Key` is set (a string name), all the new structured fields taken from the original `log` content are inserted under the new key. | _none_ | | `Merge_Log_Trim` | When `Merge_Log` is enabled, trim (remove possible `\n` or `\r\`) field values. | `On` | | `Merge_Parser` | Optional parser name to specify how to parse the data contained in the `log` key. Recommended for developers or testing only. | _none_ | -| `Keep_Log` | When `Keep_Log` is disabled, the `log` field is removed from the incoming message once it has been successfully merged (`Merge_Log` must be enabled as well). | `On` | -| `tls.debug` | Debug level between `0 (no information) and 4 (all details). | `-1` | +| `Keep_Log` | When `Keep_Log` is disabled and `Merge_Log` enabled, the `log` field is removed from the incoming message once it has been successfully merged. | `On` | +| `tls.debug` | Debug level between `0` (no information) and `4` (all details). | `-1` | | `tls.verify` | When enabled, turns on certificate validation when connecting to the Kubernetes API server. | `On` | | `tls.verify_hostname` | When enabled, turns on hostname validation for certificates. | `Off` | -| `Use_Journal` | When enabled, the filter reads logs coming in `Journald` format. | `Off` | +| `Use_Journal` | When enabled, the filter reads logs in `Journald` format. | `Off` | | `Cache_Use_Docker_Id` | When enabled, metadata will be fetched from Kubernetes when `docker_id` is changed. | `Off` | | `Regex_Parser` | Set an alternative Parser to process record tags and extract `pod_name`, `namespace_name`, `container_name`, and `docker_id`. The parser must be registered in a [parsers file](https://github.com/fluent/fluent-bit/blob/master/conf/parsers.conf) (refer to parser `filter-kube-test` as an example). | _none_ | | `K8S-Logging.Parser` | Allow Kubernetes pods to suggest a pre-defined parser. | `Off` | | `K8S-Logging.Exclude` | Allow Kubernetes pods to exclude their logs from the log processor. | `Off` | | `Labels` | Include Kubernetes pod resource labels in the extra metadata. | `On` | | `Annotations` | Include Kubernetes pod resource annotations in the extra metadata. | `On` | -| `Kube_meta_preload_cache_dir` | If set, Kubernetes metadata can be cached or pre-loaded from files in JSON format in this directory, named `namespace-pod.meta` | _none_ | +| `Kube_meta_preload_cache_dir` | If set, Kubernetes metadata can be cached or pre-loaded from files in JSON format in this directory, named `namespace-pod.meta`. | _none_ | | `Dummy_Meta` | If set, use dummy-meta data (for test/dev purposes). | `Off` | -| `DNS_Retries` | DNS lookup retries N times until the network start working. | `6` | +| `DNS_Retries` | Number of DNS lookup retries until the network starts working. | `6` | | `DNS_Wait_Time` | DNS lookup interval between network status checks. | `30` | | `Use_Kubelet` | Optional feature flag to get metadata information from Kubelet instead of calling Kube Server API to enhance the log. This could mitigate the [Kube API heavy traffic issue for large cluster](kubernetes.md#optional-feature-using-kubelet-to-get-metadata). If used when any [Kubernetes Namespace Meta](#kubernetes-namespace-meta) fields are enabled, Kubelet will be used to fetch pod data, but namespace meta will still be fetched using the `Kube_URL` settings.| `Off` | | `Use_Tag_For_Meta` | When enabled, Kubernetes metadata (for example, `pod_name`, `container_name`, and `namespace_name`) will be extracted from the tag itself. Connection to Kubernetes API Server won't get established and API calls for metadata won't be made. See [Workflow of Tail + Kubernetes Filter](#workflow-of-tail--kubernetes-filter) and [Custom tag For enhanced filtering](#custom-tag-for-enhanced-filtering) to better understand metadata extraction from tags. | `Off` | -| `Kubelet_Port` | Kubelet port using for HTTP request, this only works when `Use_Kubelet` set to `On`. | `10250` | -| `Kubelet_Host` | Kubelet host using for HTTP request, this only works when `Use_Kubelet` set to `On`. | `127.0.0.1` | -| `Kube_Meta_Cache_TTL` | Configurable `TTL` for Kubernetes cached pod metadata. By default, it's set to `0` which means `TTL` for cache entries is disabled and cache entries are evicted at random when capacity is reached. To enable this option, set the number to a time interval. For example, set the value to `60` or `60s` and cache entries which have been created more than 60 seconds ago will be evicted. | `0` | +| `Kubelet_Port` | Kubelet port to use for HTTP requests. This only works when `Use_Kubelet` is set to `On`. | `10250` | +| `Kubelet_Host` | Kubelet host to use for HTTP requests. This only works when `Use_Kubelet` is set to `On`. | `127.0.0.1` | +| `Kube_Meta_Cache_TTL` | Configurable time-to-live for Kubernetes cached pod metadata. By default, it's set to `0` which means `TTL` for cache entries is disabled and cache entries are evicted at random when capacity is reached. To enable this option, set the number to a time interval. For example, set the value to `60` or `60s` and cache entries which have been created more than 60 seconds ago will be evicted. | `0` | | `Kube_Token_TTL` | Configurable time-to-live for the Kubernetes token. After this time, the token is reloaded from `Kube_Token_File` or the `Kube_Token_Command`.| `600` | | `Kube_Token_Command` | Command to get Kubernetes authorization token. Defaults to `NULL` uses the token file to get the token. To manually choose a command to get it, set the command here. For example, run `aws-iam-authenticator -i your-cluster-name token --token-only` to set token. This option is currently Linux-only. | `NULL` | -| `Kube_Meta_Namespace_Cache_TTL` | Configurable `TTL` for Kubernetes cached namespace metadata. Setting this to `0` will mean entries are evicted at random once the cache is full. | `900` (seconds) | +| `Kube_Meta_Namespace_Cache_TTL` | Configurable time-to-live for Kubernetes cached namespace metadata. If set to `0`, entries are evicted at random when capacity is reached. | `900` (seconds) | | `Namespace_Labels` | Include Kubernetes namespace resource labels in the extra metadata. See [Kubernetes Namespace Meta](#kubernetes-namespace-meta)| `Off` | | `Namespace_Annotations` | Include Kubernetes namespace resource annotations in the extra metadata. See [Kubernetes Namespace Meta](#kubernetes-namespace-meta)| `Off` | | `Namespace_Metadata_Only` | Include Kubernetes namespace metadata only and no pod metadata. When set, the values of `Labels` and `Annotations` are ignored. See [Kubernetes Namespace Meta](#kubernetes-namespace-meta)| `Off` | @@ -76,13 +76,13 @@ Kubernetes filter provides several ways to process the data contained in the `lo Time_Keep On ``` -For Fluent Bit v1.2 or greater, don't use decoders (`Decode_Field_As`) if you are using Elasticsearch database in the output to avoid data type conflicts. +To avoid data-type conflicts in Fluent Bit v1.2 or greater, don't use decoders (`Decode_Field_As`) if you're using Elasticsearch database in the output. To perform processing of the `log` key, you must enable the `Merge_Log` configuration property in this filter, then the following processing order will be done: - If a pod suggests a parser, the filter will use that parser to process the content of `log`. - If the `Merge_Parser` option was set and the pod didn't suggest a parser, process the `log` content using the suggested parser in the configuration. -- If no pod was suggested and no `Merge_Parser` is set, try to handle the content as JSON. +- If no pod was suggested and `Merge_Parser` isn't set, try to handle the content as JSON. If `log` value processing fails, the value is untouched. The order of processing isn't chained, meaning it's exclusive and the filter will try only one of the options, not all of them. @@ -93,15 +93,15 @@ Enable namespace meta using the following settings: - `Namespace_Labels` - `Namespace_Annotations` -Using any namespace meta requires the use of the Kube API. It can not be fetched directly from Kubelet. If `Use_Kubelet On` has been set, the Kubelet API will only be used to fetch pod metadata, while namespace meta is fetched from the upstream Kubernetes API. +Using any namespace meta requires the use of the Kube API. It can't be fetched directly from Kubelet. If `Use_Kubelet On` has been set, the Kubelet API will be used only to fetch pod metadata, while namespace meta is fetched from the upstream Kubernetes API. If collected, namespace meta will be stored in a `kubernetes_namespace` record key. -Namespace meta isn't be guaranteed to be in sync as namespace labels and annotations can be adjusted after pod creation. Adjust `Kube_Meta_Namespace_Cache_TTL` to lower caching times to fit your use case. +Namespace meta isn't guaranteed to be in sync since namespace labels and annotations can be adjusted after pod creation. Adjust `Kube_Meta_Namespace_Cache_TTL` to reduce caching times to fit your use case. - `Namespace_Metadata_Only` - Using this feature will instruct Fluent Bit to only fetch namespace metadata and to not fetch pod metadata at all. - Pod basic metadata like `container id` and `host` won't be added and the labels and annotations configuration options which are used specifically for pod Metadata will be ignored. + Pod basic metadata like `container id` and `host` won't be added, and the labels and annotations configuration options which are used specifically for pod Metadata will be ignored. ## Kubernetes pod annotations @@ -115,7 +115,7 @@ The following annotations are available: | Annotation | Description | Default | | :--- | :--- | :--- | | `fluentbit.io/parser[_stream][-container]` | Suggest a pre-defined parser. The parser must be registered already by Fluent Bit. This option will only be processed if Fluent Bit configuration (Kubernetes Filter) has enabled the option `K8S-Logging.Parser`. If present, the stream (stdout or stderr) will restrict that specific stream. If present, the container can override a specific container in a Pod. | _none_ | -| `fluentbit.io/exclude[_stream][-container]` | Request Fluent Bit to exclude or not the logs generated by the pod. This option will only be processed if Fluent Bit configuration (Kubernetes Filter) have enabled the option `K8S-Logging.Exclude`. | `False` | +| `fluentbit.io/exclude[_stream][-container]` | Define whether to request that Fluent Bit excludes the logs generated by the pod. This option will be processed only if the Fluent Bit configuration (Kubernetes Filter) has enabled the option `K8S-Logging.Exclude`. | `False` | ### Annotation examples in pod definition @@ -157,7 +157,7 @@ spec: image: edsiper/apache_logs ``` -The annotation value is boolean which can take a `"true"` or `"false"`. +The annotation value is Boolean which can take a `"true"` or `"false"`. Values must be quoted. ## Kubernetes owner references @@ -211,7 +211,7 @@ When Kubernetes filter runs, it tries to match all records that start with `kube Kubernetes filter doesn't care from where the logs comes from, but it cares about the absolute name of the monitored file. That information contains the pod name and namespace name that are used to retrieve associated metadata to the running pod from the Kubernetes Master/API Server. -If you have large pod specifications (can be caused by large numbers of environment variables), be sure to increase the `Buffer_Size` parameter of the Kubernetes filter. If object sizes exceed this buffer, some metadata will fail to be injected to the logs. +If you have large pod specifications, which can be caused by large numbers of environment variables, increase the `Buffer_Size` parameter of the Kubernetes filter. If object sizes exceed this buffer, some metadata will fail to be injected to the logs. If the configuration property `Kube_Tag_Prefix` was configured (available on Fluent Bit >= 1.1.x), it will use that value to remove the prefix that was appended to the Tag in the previous `Input` section. The configuration property defaults to `kube.var.logs.containers.` , so the previous tag content will be transformed from: @@ -225,15 +225,17 @@ to: apache-logs-annotated_default_apache-aeeccc7a9f00f6e4e066aeff0434cf80621215071f1b20a51e8340aa7c35eac6.log ``` -The transformation doesn't modify the original tag, it creates a new representation for the filter to perform metadata lookup. +Rather than modify the original tag, the transformation creates a new representation for the filter to perform metadata lookup. +Suggested change The new value is used by the filter to lookup the pod name and namespace, for that purpose it uses an internal regular expression: +The new value is used by the filter to lookup the pod name and namespace. For that purpose, it uses an internal regular expression: ```text (?[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_(?.+)-(?[a-z0-9]{64})\.log$ ``` -For more details, review the source code of that definition [here](https://github.com/fluent/fluent-bit/blob/master/plugins/filter_kubernetes/kube_regex.h#L26>). +For more details, review the [source code of that definition](https://github.com/fluent/fluent-bit/blob/master/plugins/filter_kubernetes/kube_regex.h#L26>). You can see on the [Rublar.com](https://rubular.com/r/HZz3tYAahj6JCd) website how this operation is performed. See the following demo link: @@ -268,11 +270,11 @@ One such use case involves splitting logs by namespace, pods, containers or cont Merge_Log On ``` -Now, the filter is able to gather the values of `pod_name` and `namespace`. With that information it will check in the local cache (internal hash table) if some metadata for that key pair exists. If it exists, it will enrich the record with the metadata value, otherwise it will connect to the Kubernetes Master/API Server and retrieve that information. +The filter can now gather the values of `pod_name` and `namespace`. With that information, it will check in the local cache (internal hash table) if some metadata for that key pair exists. If it exists, it will enrich the record with the metadata value. Otherwise, it connects to the Kubernetes Master/API Server and retrieves that information. ## Using Kubelet to get metadata -There is an [issue](https://github.com/fluent/fluent-bit/issues/1948) reported about kube-apiserver fall over and become unresponsive when cluster is too large and too many requests are sent to it. For this feature, fluent bit Kubernetes filter will send the request to kubelet /pods endpoint instead of kube-apiserver to retrieve the pods information and use it to enrich the log. Since Kubelet is running locally in nodes, the request would be responded faster and each node would only get one request one time. This could save kube-apiserver power to handle other requests. When this feature is enabled, you should see no difference in the Kubernetes metadata added to logs, but the Kube-apiserver bottleneck should be avoided when cluster is large. +An [issue](https://github.com/fluent/fluent-bit/issues/1948) about `kube-apiserver` suggests it will fail and become unresponsive when a cluster is too large and receives too many requests. For this feature, the Fluent Bit Kubernetes filter will send the request to the Kubelet `/pods` endpoint instead of `kube-apiserver` to retrieve the pods information and use it to enrich the log. Since Kubelet is running locally in nodes, the request response would be faster and each node would receive a request only one time. This could preserve `kube-apiserver` capacity to handle other requests. When this feature is enabled, you should see no difference in the Kubernetes metadata added to logs, but the `kube-apiserver` bottleneck should be avoided when the cluster is large. ### Configuration setup @@ -423,7 +425,7 @@ If you are in debug mode, you can see more: ## Troubleshooting -The following section goes over specific log messages you might receive. Learn how to solve them to ensure that the Fluent Bit Kubernetes filter is operating properly +Learn how to solve them to ensure that the Fluent Bit Kubernetes filter is operating properly. The following section describes specific log messages you might receive. - You can't see metadata appended to your pods or other Kubernetes objects @@ -435,11 +437,11 @@ The following section goes over specific log messages you might receive. Learn h [2020/10/15 03:48:57] [ warn] [filter_kube] could not get meta for POD ``` - - Potential fix #1: Check Kubernetes roles + - Potential fix 1: Check Kubernetes roles When Fluent Bit is deployed as a DaemonSet it generally runs with specific roles that allow the application to talk to the Kubernetes API server. If you are deployed in a more restricted environment ensure that all the Kubernetes roles are set correctly. - You can test this by running the following command (replace `fluentbit-system` with the namespace where your Fluent Bit is installed) + You can test this by running the following command. Replace `fluentbit-system` with the namespace where your Fluent Bit is installed. ```text kubectl auth can-i list pods --as=system:serviceaccount:fluentbit-system:fluentbit @@ -453,9 +455,7 @@ The following section goes over specific log messages you might receive. Learn h no - Azure does not have opinion for this user. ``` - If you have connectivity to the API server, but still `could not get meta for POD` - debug logging might give you a message with `Azure does not have opinion for this user`. Then the following `subject` might need to be included in the `fluentbit` `ClusterRoleBinding`: - - appended to `subjects` array: + If you have can connect to the API server, but still `could not get meta for POD` - debug logging might give you a message with `Azure does not have opinion for this user`. The following `subject` might need to be included in the `fluentbit` `ClusterRoleBinding`, appended to `subjects` array: ```yaml - apiGroup: rbac.authorization.k8s.io @@ -463,17 +463,17 @@ The following section goes over specific log messages you might receive. Learn h name: system:serviceaccounts ``` - - Potential fix #2: Check Kubernetes IPv6 + - Potential fix 2: Check Kubernetes IPv6 There might be cases where you have IPv6 on in the environment and you need to enable this within Fluent Bit. Under the service tag set the following option `ipv6` to `on` . - - Potential fix #3: Check connectivity to `Kube_URL` + - Potential fix 3: Check connectivity to `Kube_URL` - By default the Kube_URL is set to `https://kubernetes.default.svc:443`. Ensure that you have connectivity to this endpoint from within the cluster and that there are no special permission interfering with the connection. + By default the `Kube_URL` is set to `https://kubernetes.default.svc:443`. Ensure that you have connectivity to this endpoint from within the cluster and that there are no special permissions interfering with the connection. -- You can't see new objects getting metadata +- You can't see new objects getting metadata - In some cases, you might only see some objects being appended with metadata while other objects aren't enriched. This can occur at times when local data is cached and doesn't contain the correct id for the Kubernetes object that requires enrichment. For most Kubernetes objects the Kubernetes API server is updated which will then be reflected in Fluent Bit logs. In some cases for `Pod` objects this refresh to the Kubernetes API server can be skipped, causing metadata to be skipped. + In some cases, you might see only some objects being appended with metadata while other objects aren't enriched. This can occur when local data is cached and doesn't contain the correct ID for the Kubernetes object that requires enrichment. For most Kubernetes objects the Kubernetes API server is updated, which will then be reflected in Fluent Bit logs. In some cases for `Pod` objects, this refresh to the Kubernetes API server can be skipped, causing metadata to be skipped. ## Credit