From 4d4004d246cc959279362540065cbe15da160e98 Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Wed, 23 Jul 2025 09:09:18 -0700 Subject: [PATCH 1/2] pipeline: output: tls tcp: style: Signed-off-by: Lynette Miles --- pipeline/outputs/tcp-and-tls.md | 78 +++++++++++++++++------------- vale-styles/FluentBit/Headings.yml | 1 + 2 files changed, 46 insertions(+), 33 deletions(-) diff --git a/pipeline/outputs/tcp-and-tls.md b/pipeline/outputs/tcp-and-tls.md index 15d02efe6..c14cff666 100644 --- a/pipeline/outputs/tcp-and-tls.md +++ b/pipeline/outputs/tcp-and-tls.md @@ -1,54 +1,60 @@ -# TCP & TLS +# TCP and TLS -The **tcp** output plugin allows to send records to a remote TCP server. The payload can be formatted in different ways as required. +The _TCP_ output plugin lets you send records to a remote TCP server. The payload can be formatted in different ways as required. -## Configuration Parameters +## Configuration parameters -| Key | Description | default | -|:-------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------| -| Host | Target host where Fluent-Bit or Fluentd are listening for Forward messages. | 127.0.0.1 | -| Port | TCP Port of the target service. | 5170 | -| Format | Specify the data format to be printed. Supported formats are _msgpack_ _json_, _json\_lines_ and _json\_stream_. | msgpack | -| json\_date\_key | Specify the name of the time key in the output record. To disable the time key just set the value to `false`. | date | -| json\_date\_format | Specify the format of the date. Supported formats are _double_, _epoch_, _iso8601_ (eg: _2018-05-30T09:39:52.000681Z_) and _java_sql_timestamp_ (eg: _2018-05-30 09:39:52.000681_) | double | -| workers | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `2` | +This plugin supports the following parameters: -## TLS Configuration Parameters +| Key | Description | Default | +|:--- |:----------- |:------- | +| `Host` | Target host where Fluent-Bit or Fluentd are listening for Forward messages. | `127.0.0.1` | +| `Port` | TCP Port of the target service. | `5170` | +| `Format` | Specify the data format to be printed. Supported formats: `msgpack`, `json`, `json_lines`, `json_stream`. | `msgpack` | +| `json_date_key`| Specify the name of the time key in the output record. To disable the time key, set the value to `false`. | `date` | +| `json_date_format` | Specify the format of the date. Supported formats: `double`, `epoch`, `iso8601`, `java_sql_timestamp` | `double` | +| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `2` | + +## TLS configuration parameters The following parameters are available to configure a secure channel connection through TLS: -| Key | Description | Default | -|:----------------|:--------------------------------------------------------------------------------------------------------------------------------------------------|:--------| -| tls | Enable or disable TLS support | Off | -| tls.verify | Force certificate validation | On | -| tls.debug | Set TLS debug verbosity level. It accept the following values: 0 \(No debug\), 1 \(Error\), 2 \(State change\), 3 \(Informational\) and 4 Verbose | 1 | -| tls.ca\_file | Absolute path to CA certificate file | | -| tls.crt\_file | Absolute path to Certificate file. | | -| tls.key\_file | Absolute path to private Key file. | | -| tls.key\_passwd | Optional password for tls.key\_file file. | | +| Key | Description | Default | +|:--- |:----------- |:------- | +| `tls` | Enable or disable TLS support. | `Off` | +| `tls.verify` | Force certificate validation. | `On` | +| `tls.debug` | Set TLS debug verbosity level. Allowed values: `0` (No debug), `1` (Error), `2` (State change), `3` (Informational), `4` (Verbose) | `1` | +| `tls.ca_file` | Absolute path to CA certificate file. | _none_ | +| `tls.crt_file` | Absolute path to Certificate file. | _none_ | +| `tls.key_file` | Absolute path to private Key file. | _none_ | +| `tls.key_passwd` | Optional password for tls.key_file file. | _none_ | -### Command Line +### Command line #### JSON format +This example specifies gathering [CPU](https://docs.fluentbit.io/manual/pipeline/inputs/cpu-metrics) usage metrics and send them in JSON lines mode to a remote endpoint using netcat service. + ```shell fluent-bit -i cpu -o tcp://127.0.0.1:5170 -p format=json_lines -v ``` -We have specified to gather [CPU](https://github.com/fluent/fluent-bit-docs/tree/16f30161dc4c79d407cd9c586a0c6839d0969d97/pipeline/input/cpu.md) usage metrics and send them in JSON lines mode to a remote end-point using netcat service. -Run the following in a separate terminal, `netcat` will start listening for messages on TCP port 5170. -Once it connects to Fluent Bit ou should see the output as above in JSON format: +Run the following in a separate terminal, `netcat` will start listening for messages on TCP port 5170. After it connects to Fluent Bit you should see the output in JSON format: ```shell -$ nc -l 5170 +nc -l 5170 +``` +Which returns results similar to: + +```text ... {"date":1644834856.905985,"cpu_p":1.1875,"user_p":0.5625,"system_p":0.625,"cpu0.p_cpu":0.0,"cpu0.p_user":0.0,"cpu0.p_system":0.0,"cpu1.p_cpu":1.0,"cpu1.p_user":1.0,"cpu1.p_system":0.0,"cpu2.p_cpu":4.0,"cpu2.p_user":2.0,"cpu2.p_system":2.0,"cpu3.p_cpu":1.0,"cpu3.p_user":0.0,"cpu3.p_system":1.0,"cpu4.p_cpu":1.0,"cpu4.p_user":0.0,"cpu4.p_system":1.0,"cpu5.p_cpu":1.0,"cpu5.p_user":1.0,"cpu5.p_system":0.0,"cpu6.p_cpu":0.0,"cpu6.p_user":0.0,"cpu6.p_system":0.0,"cpu7.p_cpu":3.0,"cpu7.p_user":1.0,"cpu7.p_system":2.0,"cpu8.p_cpu":0.0,"cpu8.p_user":0.0,"cpu8.p_system":0.0,"cpu9.p_cpu":1.0,"cpu9.p_user":0.0,"cpu9.p_system":1.0,"cpu10.p_cpu":1.0,"cpu10.p_user":0.0,"cpu10.p_system":1.0,"cpu11.p_cpu":0.0,"cpu11.p_user":0.0,"cpu11.p_system":0.0,"cpu12.p_cpu":0.0,"cpu12.p_user":0.0,"cpu12.p_system":0.0,"cpu13.p_cpu":3.0,"cpu13.p_user":2.0,"cpu13.p_system":1.0,"cpu14.p_cpu":1.0,"cpu14.p_user":1.0,"cpu14.p_system":0.0,"cpu15.p_cpu":0.0,"cpu15.p_user":0.0,"cpu15.p_system":0.0} ... ``` -#### Msgpack format +#### `msgpack` format Repeat the JSON approach but using the `msgpack` output format. @@ -56,9 +62,9 @@ Repeat the JSON approach but using the `msgpack` output format. fluent-bit -i cpu -o tcp://127.0.0.1:5170 -p format=msgpack -v ``` -We could send this to stdout but as it is a serialized format you would end up with strange output. -This should really be handled by a msgpack receiver to unpack as per the details in the developer documentation [here](../../development/msgpack-format.md). -As an example we use the [Python msgpack library](https://msgpack.org/#languages) to deal with it: +You could send this to stdout but as it's a serialized format you would end up with strange output. + +This should be handled by a [`msgpack`](../../development/msgpack-format.md) receiver to unpack. The following example uses the [Python `msgpack` library](https://msgpack.org/#languages) handle it: ```python #Python3 @@ -81,11 +87,17 @@ while True: print(unpacked) ``` +Run the following commands: + ```shell -$ pip install msgpack -$ python3 test.py + pip install msgpack + python3 test.py +``` +Which return results like: + +```text ... (ExtType(code=0, data=b'b\n5\xc65\x05\x14\xac'), {'cpu_p': 0.1875, 'user_p': 0.125, 'system_p': 0.0625, 'cpu0.p_cpu': 0.0, 'cpu0.p_user': 0.0, 'cpu0.p_system': 0.0, 'cpu1.p_cpu': 0.0, 'cpu1.p_user': 0.0, 'cpu1.p_system': 0.0, 'cpu2.p_cpu': 1.0, 'cpu2.p_user': 0.0, 'cpu2.p_system': 1.0, 'cpu3.p_cpu': 0.0, 'cpu3.p_user': 0.0, 'cpu3.p_system': 0.0, 'cpu4.p_cpu': 0.0, 'cpu4.p_user': 0.0, 'cpu4.p_system': 0.0, 'cpu5.p_cpu': 0.0, 'cpu5.p_user': 0.0, 'cpu5.p_system': 0.0, 'cpu6.p_cpu': 0.0, 'cpu6.p_user': 0.0, 'cpu6.p_system': 0.0, 'cpu7.p_cpu': 0.0, 'cpu7.p_user': 0.0, 'cpu7.p_system': 0.0, 'cpu8.p_cpu': 0.0, 'cpu8.p_user': 0.0, 'cpu8.p_system': 0.0, 'cpu9.p_cpu': 1.0, 'cpu9.p_user': 1.0, 'cpu9.p_system': 0.0, 'cpu10.p_cpu': 0.0, 'cpu10.p_user': 0.0, 'cpu10.p_system': 0.0, 'cpu11.p_cpu': 0.0, 'cpu11.p_user': 0.0, 'cpu11.p_system': 0.0, 'cpu12.p_cpu': 0.0, 'cpu12.p_user': 0.0, 'cpu12.p_system': 0.0, 'cpu13.p_cpu': 0.0, 'cpu13.p_user': 0.0, 'cpu13.p_system': 0.0, 'cpu14.p_cpu': 0.0, 'cpu14.p_user': 0.0, 'cpu14.p_system': 0.0, 'cpu15.p_cpu': 0.0, 'cpu15.p_user': 0.0, 'cpu15.p_system': 0.0}) ... -``` \ No newline at end of file +``` diff --git a/vale-styles/FluentBit/Headings.yml b/vale-styles/FluentBit/Headings.yml index 28978aca4..72327f49d 100644 --- a/vale-styles/FluentBit/Headings.yml +++ b/vale-styles/FluentBit/Headings.yml @@ -62,6 +62,7 @@ exceptions: - HTTP - I - InfluxDB + - JSON - Kafka - Kafka Producer - Kafka REST From 3a0cf5e15d91a2076bbab8d36b87b3cc1c3cc5e3 Mon Sep 17 00:00:00 2001 From: Lynette Miles <6818907+esmerel@users.noreply.github.com> Date: Wed, 23 Jul 2025 13:53:48 -0700 Subject: [PATCH 2/2] Apply suggestions from code review Co-authored-by: Craig Norris <112565517+cnorris-cs@users.noreply.github.com> Signed-off-by: Lynette Miles <6818907+esmerel@users.noreply.github.com> --- pipeline/outputs/tcp-and-tls.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/pipeline/outputs/tcp-and-tls.md b/pipeline/outputs/tcp-and-tls.md index c14cff666..e621160a0 100644 --- a/pipeline/outputs/tcp-and-tls.md +++ b/pipeline/outputs/tcp-and-tls.md @@ -8,11 +8,11 @@ This plugin supports the following parameters: | Key | Description | Default | |:--- |:----------- |:------- | -| `Host` | Target host where Fluent-Bit or Fluentd are listening for Forward messages. | `127.0.0.1` | +| `Host` | Target host where Fluent Bit or Fluentd are listening for Forward messages. | `127.0.0.1` | | `Port` | TCP Port of the target service. | `5170` | | `Format` | Specify the data format to be printed. Supported formats: `msgpack`, `json`, `json_lines`, `json_stream`. | `msgpack` | | `json_date_key`| Specify the name of the time key in the output record. To disable the time key, set the value to `false`. | `date` | -| `json_date_format` | Specify the format of the date. Supported formats: `double`, `epoch`, `iso8601`, `java_sql_timestamp` | `double` | +| `json_date_format` | Specify the format of the date. Supported formats: `double`, `epoch`, `iso8601`, `java_sql_timestamp`. | `double` | | `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `2` | ## TLS configuration parameters @@ -27,7 +27,7 @@ The following parameters are available to configure a secure channel connection | `tls.ca_file` | Absolute path to CA certificate file. | _none_ | | `tls.crt_file` | Absolute path to Certificate file. | _none_ | | `tls.key_file` | Absolute path to private Key file. | _none_ | -| `tls.key_passwd` | Optional password for tls.key_file file. | _none_ | +| `tls.key_passwd` | Optional password for `tls.key_file` file. | _none_ | ### Command line @@ -40,7 +40,7 @@ fluent-bit -i cpu -o tcp://127.0.0.1:5170 -p format=json_lines -v ``` -Run the following in a separate terminal, `netcat` will start listening for messages on TCP port 5170. After it connects to Fluent Bit you should see the output in JSON format: +Run the following in a separate terminal, `netcat` will start listening for messages on TCP port `5170`. After it connects to Fluent Bit you should see the output in JSON format: ```shell nc -l 5170