Merge pull request #1446 from fluent/add-threading-info

Add threading info

Author: alexakreizinger

pipeline/inputs/collectd.md

@@ -11,6 +11,7 @@ The plugin supports the following configuration parameters:
 | Listen | Set the address to listen to | 0.0.0.0 |
 | Port | Set the port to listen to | 25826 |
 | TypesDB | Set the data specification file | /usr/share/collectd/types.db |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
 
 ## Configuration Examples <a id="config_example"></a>
 
@@ -31,4 +32,3 @@ Here is a basic configuration example.
 With this configuration, Fluent Bit listens to `0.0.0.0:25826`, and outputs incoming datagram packets to stdout.
 
 You must set the same types.db files that your collectd server uses. Otherwise, Fluent Bit may not be able to interpret the payload properly.
-

pipeline/inputs/cpu-metrics.md

@@ -4,13 +4,15 @@ The **cpu** input plugin, measures the CPU usage of a process or the whole syste
 
 The following tables describes the information generated by the plugin. The keys below represent the data used by the overall system, all values associated to the keys are in a percentage unit \(0 to 100%\):
 
-The CPU metrics plugin creates metrics that are log-based \(I.e. JSON payload\). If you are looking for Prometheus-based metrics please see the Node Exporter Metrics input plugin. 
+The CPU metrics plugin creates metrics that are log-based, such as JSON payload. For
+Prometheus-based metrics, see the Node Exporter Metrics input plugin.
 
 | key | description |
 | :--- | :--- |
 | cpu\_p | CPU usage of the overall system, this value is the summation of time spent on user and kernel space. The result takes in consideration the numbers of CPU cores in the system. |
 | user\_p | CPU usage in User mode, for short it means the CPU usage by user space programs. The result of this value takes in consideration the numbers of CPU cores in the system. |
 | system\_p | CPU usage in Kernel mode, for short it means the CPU usage by the Kernel. The result of this value takes in consideration the numbers of CPU cores in the system. |
+| threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). Default: `false`. |
 
 In addition to the keys reported in the above table, a similar content is created **per** CPU core. The cores are listed from _0_ to _N_ as the Kernel reports:
 

pipeline/inputs/disk-io-metrics.md

@@ -2,7 +2,8 @@
 
 The **disk** input plugin, gathers the information about the disk throughput of the running system every certain interval of time and reports them.
 
-The Disk I/O metrics plugin creates metrics that are log-based \(I.e. JSON payload\). If you are looking for Prometheus-based metrics please see the Node Exporter Metrics input plugin. 
+The Disk I/O metrics plugin creates metrics that are log-based, such as JSON payload.
+For Prometheus-based metrics, see the Node Exporter Metrics input plugin.
 
 ## Configuration Parameters
 
@@ -13,6 +14,7 @@ The plugin supports the following configuration parameters:
 | Interval\_Sec | Polling interval \(seconds\).  | 1 |
 | Interval\_NSec | Polling interval \(nanosecond\). | 0 |
 | Dev\_Name | Device name to limit the target. \(e.g. sda\). If not set, _in\_disk_ gathers information from all of disks and partitions. | all disks |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
 
 ## Getting Started
 
@@ -72,4 +74,3 @@ pipeline:
 Note: Total interval \(sec\) = Interval\_Sec + \(Interval\_Nsec / 1000000000\).
 
 e.g. 1.5s = 1s + 500000000ns
-

pipeline/inputs/docker-events.md

@@ -14,6 +14,7 @@ This plugin supports the following configuration parameters:
 | Key | When a message is unstructured \(no parser applied\), it's appended as a string under the key name _message_. | message |
 | Reconnect.Retry_limits| The maximum number of retries allowed. The plugin tries to reconnect with docker socket when EOF is detected. | 5 |
 | Reconnect.Retry_interval| The retrying interval. Unit is second. | 1 |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
 
 ### Command Line
 

pipeline/inputs/docker-metrics.md

@@ -15,6 +15,7 @@ The plugin supports the following configuration parameters:
 | Interval_Sec | Polling interval in seconds                     | 1       |
 | Include      | A space-separated list of containers to include |         |
 | Exclude      | A space-separated list of containers to exclude |         |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
 
 If you set neither `Include` nor `Exclude`, the plugin will try to get metrics from _all_ the running containers.
 

pipeline/inputs/dummy.md

@@ -18,6 +18,7 @@ The plugin supports the following configuration parameters:
 | Samples            | If set, the events number will be limited. For example, if Samples=3, the plugin generates only three events and stops. | _none_ |
 | Copies             | Number of messages to generate each time they are generated. | `1` |
 | Flush\_on\_startup | If set to `true`, the first dummy event is generated at startup. | `false` |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
 
 ## Getting Started
 

pipeline/inputs/elasticsearch.md

@@ -14,6 +14,7 @@ The plugin supports the following configuration parameters:
 | meta\_key | Specify a key name for meta information. | "@meta" |
 | hostname | Specify hostname or FQDN. This parameter can be used for "sniffing" (auto-discovery of) cluster node information. | "localhost" |
 | version  | Specify Elasticsearch server version. This parameter is effective for checking a version of Elasticsearch/OpenSearch server version. | "8.0.0" |
+| threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
 
 **Note:** The Elasticsearch cluster uses "sniffing" to optimize the connections between its cluster and clients.
 Elasticsearch can build its cluster and dynamically generate a connection list which is called "sniffing".

pipeline/inputs/exec-wasi.md

@@ -17,6 +17,7 @@ The plugin supports the following configuration parameters:
 | Wasm\_Stack\_Size | Size of the stack size of Wasm execution. Review [unit sizes](../../administration/configuring-fluent-bit/unit-sizes.md) for allowed values. |
 | Buf\_Size | Size of the buffer \(check [unit sizes](../../administration/configuring-fluent-bit/unit-sizes.md) for allowed values\) |
 | Oneshot | Only run once at startup. This allows collection of data precedent to fluent-bit's startup (bool, default: false) |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). Default: `false`. |
 
 ## Configuration Examples
 

pipeline/inputs/exec.md

@@ -25,6 +25,7 @@ The plugin supports the following configuration parameters:
 | Oneshot | Only run once at startup. This allows collection of data precedent to fluent-bit's startup (bool, default: false) |
 | Exit\_After\_Oneshot | Exit as soon as the one-shot command exits. This allows the exec plugin to be used as a wrapper for another command, sending the target command's output to any fluent-bit sink(s) then exiting. (bool, default: false) |
 | Propagate\_Exit\_Code | When exiting due to Exit\_After\_Oneshot, cause fluent-bit to exit with the exit code of the command exited by this plugin. Follows [shell conventions for exit code propagation](https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html). (bool, default: false) |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). Default: `false`. |
 
 ## Getting Started
 

pipeline/inputs/fluentbit-metrics.md

@@ -12,12 +12,13 @@ They can be sent to output plugins including [Prometheus Exporter](../outputs/pr
 **Important note:** Metrics collected with Node Exporter Metrics flow through a separate pipeline from logs and current filters do not operate on top of metrics.
 
 
-## Configuration 
+## Configuration
 
 | Key             | Description                                                                                               | Default   |
 | --------------- | --------------------------------------------------------------------------------------------------------- | --------- |
 | scrape_interval | The rate at which metrics are collected from the host operating system                                    | 2 seconds |
 | scrape_on_start | Scrape metrics upon start, useful to avoid waiting for 'scrape_interval' for the first round of metrics.  | false     |
+| threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
 
 
 ## Getting Started

pipeline/inputs/forward.md

@@ -20,6 +20,7 @@ The plugin supports the following configuration parameters:
 | Shared\_Key         | Shared key for secure forward authentication. |  |
 | Self\_Hostname      | Hostname for secure forward authentication.   |  |
 | Security.Users      | Specify the username and password pairs for secure forward authentication. | |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
 
 ## Getting Started
 
@@ -142,4 +143,3 @@ Copyright (C) Treasure Data
 [2016/10/07 21:49:40] [ info] [in_fw] binding 0.0.0.0:24224
 [0] my_tag: [1475898594, {"key 1"=>123456789, "key 2"=>"abcdefg"}]
 ```
-

pipeline/inputs/head.md

@@ -16,6 +16,7 @@ The plugin supports the following configuration parameters:
 | Key | Rename a key. Default: head. |
 | Lines | Line number to read. If the number N is set, in\_head reads first N lines like head\(1\) -n. |
 | Split\_line | If enabled, in\_head generates key-value pair per line. |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). Default: `false`. |
 
 ### Split Line Mode
 
@@ -84,7 +85,7 @@ pipeline:
 Output is
 
 ```bash
-$ bin/fluent-bit -c head.conf 
+$ bin/fluent-bit -c head.conf
 Fluent Bit v1.x.x
 * Copyright (C) 2019-2020 The Fluent Bit Authors
 * Copyright (C) 2015-2018 Treasure Data
@@ -162,4 +163,3 @@ pipeline:
 Note: Total interval \(sec\) = Interval\_Sec + \(Interval\_Nsec / 1000000000\).
 
 e.g. 1.5s = 1s + 500000000ns
-

pipeline/inputs/health.md

@@ -15,6 +15,7 @@ The plugin supports the following configuration parameters:
 | Alert | If enabled, it will only generate messages if the target TCP service is down. By default this option is disabled. |
 | Add\_Host | If enabled, hostname is appended to each records. Default value is _false_. |
 | Add\_Port | If enabled, port number is appended to each records. Default value is _false_. |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). Default: `false`. |
 
 ## Getting Started
 
@@ -87,4 +88,3 @@ Fluent Bit v1.8.0
 [2] health.0: [1624145990.306498573, {"alive"=>true}]
 [3] health.0: [1624145991.305595498, {"alive"=>true}]
 ```
-

pipeline/inputs/http.md

@@ -15,6 +15,7 @@ description: The HTTP input plugin allows you to send custom records to an HTTP
 | buffer_chunk_size        | This sets the chunk size for incoming incoming JSON messages. These chunks are then stored/managed in the space available by buffer_max_size. | 512K    |
 | successful_response_code | It allows to set successful response code. `200`, `201` and `204` are supported.                                                              | 201     |
 | success_header           | Add an HTTP header key/value pair on success. Multiple headers can be set. Example: `X-Custom custom-answer`                                  |         |
+| threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
 
 ### TLS / SSL
 

pipeline/inputs/kafka.md

@@ -15,6 +15,7 @@ This plugin uses the official [librdkafka C library](https://github.com/edenhill
 | poll\_ms | Kafka brokers polling interval in milliseconds. | 500 |
 | Buffer\_Max\_Size | Specify the maximum size of buffer per cycle to poll kafka messages from subscribed topics. To increase throughput, specify larger size. | 4M |
 | rdkafka.{property} | `{property}` can be any [librdkafka properties](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) |  |
+| threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
 
 ## Getting Started
 
@@ -45,7 +46,8 @@ In your main configuration file append the following _Input_ & _Output_ sections
 
 #### Example of using kafka input/output plugins
 
-The fluent-bit source repository contains a full example of using fluent-bit to process kafka records: 
+The Fluent Bit source repository contains a full example of using Fluent Bit to
+process Kafka records:
 
 ```text
 [INPUT]

pipeline/inputs/kernel-logs.md

@@ -7,6 +7,7 @@ The **kmsg** input plugin reads the Linux Kernel log buffer since the beginning,
 | Key | Description | Default |
 | :--- | :--- | :--- |
 | Prio_Level | The log level to filter. The kernel log is dropped if its priority is more than prio_level. Allowed values are 0-8. Default is 8. 8 means all logs are saved. | 8 |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
 
 ## Getting Started
 
@@ -60,4 +61,3 @@ pipeline:
 ```
 {% endtab %}
 {% endtabs %}
-

pipeline/inputs/kubernetes-events.md

@@ -31,6 +31,9 @@ Kubernetes exports it events through the API server. This input plugin allows to
 
 - _* As of Fluent-Bit 3.1, this plugin uses a Kubernetes watch stream instead of polling. In versions before 3.1, the interval parameters are used for reconnecting the Kubernetes watch stream._
 
+## Threading
+
+This input always runs in its own [thread](../../administration/multithreading.md#inputs).
 
 ## Getting Started
 

pipeline/inputs/memory-metrics.md

@@ -23,6 +23,11 @@ Fluent Bit v1.x.x
 [3] memory: [1488543159, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
 ```
 
+## Threading
+
+You can enable the `threaded` setting to run this input in its own
+[thread](../../administration/multithreading.md#inputs).
+
 ### Configuration File
 
 In your main configuration file append the following _Input_ & _Output_ sections:

pipeline/inputs/mqtt.md

@@ -11,6 +11,7 @@ The plugin supports the following configuration parameters:
 | Listen      | Listener network interface. | `0.0.0.0` |
 | Port        | TCP port where listening for connections. | `1883` |
 | Payload_Key | Specify the key where the payload key/value will be preserved. | _none_ |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
 
 ## Getting Started
 
@@ -53,4 +54,3 @@ In your main configuration file append the following _Input_ & _Output_ sections
     Name   stdout
     Match  *
 ```
-

pipeline/inputs/network-io-metrics.md

@@ -2,7 +2,8 @@
 
 The **netif** input plugin gathers network traffic information of the running system every certain interval of time, and reports them.
 
-The Network I/O Metrics plugin creates metrics that are log-based \(I.e. JSON payload\). If you are looking for Prometheus-based metrics please see the Node Exporter Metrics input plugin. 
+The Network I/O Metrics plugin creates metrics that are log-based, such as JSON
+payload. For Prometheus-based metrics, see the Node Exporter Metrics input plugin.
 
 ## Configuration Parameters
 
@@ -15,6 +16,7 @@ The plugin supports the following configuration parameters:
 | Interval\_NSec | Polling interval \(nanosecond\). | 0 |
 | Verbose | If true, gather metrics precisely. | false |
 | Test\_At\_Init | If true, testing if the network interface is valid at initialization. | false |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
 
 ## Getting Started
 
@@ -75,4 +77,3 @@ pipeline:
 Note: Total interval \(sec\) = Interval\_Sec + \(Interval\_Nsec / 1000000000\).
 
 e.g. 1.5s = 1s + 500000000ns
-

pipeline/inputs/nginx.md

@@ -12,6 +12,7 @@ The plugin supports the following configuration parameters:
 | Port | Port of the target nginx service to connect to. | 80 |
 | Status_URL | The URL of the Stub Status Handler. | /status |
 | Nginx_Plus | Turn on NGINX plus mode. | true |
+| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
 
 ## Getting Started
 
@@ -35,8 +36,8 @@ server {
 
 ### Configuration with NGINX Plus REST API
 
-A much more powerful and flexible metrics API is available with NGINX Plus. A path needs to be configured 
-in NGINX Plus first.
+Another metrics API is available with NGINX Plus. You must first configure a path in
+NGINX Plus.
 
 ```
 server {

pipeline/inputs/node-exporter-metrics.md

@@ -81,6 +81,10 @@ The following table describes the available collectors as part of this plugin. A
 | nvme              | Exposes nvme statistics from `/proc`.                                                            | Linux       | v2.2.0  |
 | processes         | Exposes processes statistics from `/proc`.                                                       | Linux       | v2.2.0  |
 
+## Threading
+
+This input always runs in its own [thread](../../administration/multithreading.md#inputs).
+
 ## Getting Started
 
 ### Simple Configuration File
@@ -114,7 +118,7 @@ In the following configuration file, the input plugin _node_exporter_metrics col
     host            0.0.0.0
     port            2021
 
-        
+
 ```
 {% endtab %}
 
@@ -201,4 +205,3 @@ docker-compose down
 Our current plugin implements a sub-set of the available collectors in the original Prometheus Node Exporter, if you would like that we prioritize a specific collector please open a Github issue by using the following template:\
 \
 \- [in_node_exporter_metrics](https://github.com/fluent/fluent-bit/issues/new?assignees=\&labels=\&template=feature_request.md\&title=in_node_exporter_metrics:%20add%20ABC%20collector)
-

pipeline/inputs/opentelemetry.md

@@ -20,6 +20,7 @@ Our compliant implementation fully supports OTLP/HTTP and OTLP/GRPC. Note that t
 | buffer_chunk_size | Initial size and allocation strategy to store the payload (advanced users only)    | 512K    |
 |successful_response_code | It allows to set successful response code. `200`, `201` and `204` are supported.| 201 |
 | tag_from_uri      | If true, tag will be created from uri. e.g. v1_metrics from /v1/metrics .                                                                      | true    |
+| threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
 
 Important note: Raw traces means that any data forwarded to the traces endpoint (`/v1/traces`) will be packed and forwarded as a log message, and will NOT be processed by Fluent Bit. The traces endpoint by default expects a valid protobuf encoded payload, but you can set the `raw_traces` option in case you want to get trace telemetry data to any of Fluent Bit's supported outputs.
 

pipeline/inputs/podman-metrics.md

@@ -13,6 +13,7 @@ description: The Podman Metrics input plugin allows you to collect metrics from
 | path.config       | Custom path to podman containers configuration file        | /var/lib/containers/storage/overlay-containers/containers.json |
 | path.sysfs        | Custom path to sysfs subsystem directory                   | /sys/fs/cgroup                                                 |
 | path.procfs       | Custom path to proc subsystem directory                    | /proc                                                          |
+| threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
 
 ## Getting Started