Skip to content

Pipeline: filters: nest: style updates #1693

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 30, 2025
Merged

Pipeline: filters: nest: style updates #1693

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2e9af2f
Pipeline: filters: nest: style updates
esmerel May 29, 2025
fe6b2b7
Apply suggestions from code review
esmerel May 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 22 additions & 24 deletions pipeline/filters/nest.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Nest

The _Nest Filter_ plugin lets you operate on or with nested data. Its modes of operation are:
The _Nest_ filter plugin lets you operate on or with nested data. Its modes of operation are:

- `nest` - Take a set of records and place them in a map.
- `lift` - Take a map by key and lift its records up.
- `nest`: Take a set of records and place them in a map.
- `lift` Take a map by key and lift its records up.

## Example usage for `nest`

Expand Down Expand Up @@ -59,7 +59,7 @@ Output:
}
```

## Configuration Parameters
## Configuration parameters

The plugin supports the following configuration parameters:

Expand All @@ -69,23 +69,21 @@ The plugin supports the following configuration parameters:
| `Wildcard` | FIELD WILDCARD | `nest` | Nest records which field matches the wildcard |
| `Nest_under` | FIELD STRING | `nest` | Nest records matching the `Wildcard` under this key |
| `Nested_under` | FIELD STRING | `lift` | Lift records nested under the `Nested_under` key |
| `Add_prefix` | FIELD STRING | ANY | Prefix affected keys with this string |
| `Remove_prefix` | FIELD STRING | ANY | Remove prefix from affected keys if it matches this string |
| `Add_prefix` | FIELD STRING | Any | Prefix affected keys with this string |
| `Remove_prefix` | FIELD STRING | Any | Remove prefix from affected keys if it matches this string |

## Getting Started
## Get started

To start filtering records, run the filter from the command line or through the
configuration file. The following example invokes the
[Memory Usage Input Plugin](../inputs/memory-metrics.md), which outputs the
following:
To start filtering records, run the filter from the command line or through the configuration file. The following example invokes the
[Memory Usage Input Plugin](../inputs/memory-metrics.md), which outputs the following:

```text
[0] memory: [1488543156, {"Mem.total"=>1016044, "Mem.used"=>841388, "Mem.free"=>174656, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
```

## Example 1 - nest

### Command Line
### Use `nest` from the command Line

Using command line mode requires quotes to parse the wildcard properly. The use
of a configuration file is recommended.
Expand All @@ -98,7 +96,7 @@ wildcard rule to the keys and nests the keys matching `Mem.*` under the new key
bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildcard=Mem.*' -p 'Nest_under=Memstats' -p 'Remove_prefix=Mem.' -m '*' -o stdout
```

### Configuration File
### Nest configuration file

{% tabs %}
{% tab title="fluent-bit.conf" %}
Expand Down Expand Up @@ -145,7 +143,7 @@ pipeline:
{% endtab %}
{% endtabs %}

### Result
### Nest result

The output of both the command line and configuration invocations should be identical and result in the following output.

Expand All @@ -154,12 +152,12 @@ The output of both the command line and configuration invocations should be iden
[0] mem.local: [1522978514.007359767, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Memstats"=>{"total"=>4050908, "used"=>714984, "free"=>3335924}}]
```

## Example 2 - nest and lift undo
## Example 2 - `nest` and `lift` undo

This example nests all `Mem.*` and `Swap.*` items under the `Stats` key and then
reverses these actions with a `lift` operation. The output appears unchanged.

### Example 2 Configuration File
### `nest` and `lift` undo configuration file

{% tabs %}
{% tab title="fluent-bit.conf" %}
Expand Down Expand Up @@ -220,19 +218,19 @@ pipeline:
{% endtab %}
{% endtabs %}

### Result
### `nest` and `lift` undo result

```text
[2018/06/21 17:42:37] [ info] [engine] started (pid=17285)
[0] mem.local: [1529566958.000940636, {"Mem.total"=>8053656, "Mem.used"=>6940380, "Mem.free"=>1113276, "Swap.total"=>16532988, "Swap.used"=>1286772, "Swap.free"=>15246216}]
```

## Example 3 - nest 3 levels deep
## Example 3 - `nest` 3 levels deep

This example takes the keys starting with `Mem.*` and nests them under `LAYER1`,
which is then nested under `LAYER2`, which is nested under `LAYER3`.

### Example 3 Configuration File
### Deep `nest` configuration file

{% tabs %}
{% tab title="fluent-bit.conf" %}
Expand Down Expand Up @@ -300,7 +298,7 @@ pipeline:
{% endtab %}
{% endtabs %}

### Result
### Deep `nest` Result

```text
[0] mem.local: [1524795923.009867831, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "LAYER3"=>{"LAYER2"=>{"LAYER1"=>{"Mem.total"=>4050908, "Mem.used"=>1112036, "Mem.free"=>2938872}}}}]
Expand All @@ -322,14 +320,14 @@ pipeline:
}
```

## Example 4 - multiple nest and lift filters with prefix
## Example 4 - multiple `nest` and `lift` filters with prefix

This example uses the 3-level deep nesting of _Example 2_ and applies the
This example uses the 3-level deep nesting of Example 2 and applies the
`lift` filter three times to reverse the operations. The end result is that all
records are at the top level, without nesting, again. One prefix is added for each
level that's lifted.

### Configuration file
### `nest` and `lift` prefix configuration file

{% tabs %}
{% tab title="fluent-bit.conf" %}
Expand Down Expand Up @@ -434,7 +432,7 @@ pipeline:
{% endtab %}
{% endtabs %}

### Result
### `nest` and `lift` prefix result

```text
[0] mem.local: [1524862951.013414798, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Lifted3_Lifted2_Lifted1_Mem.total"=>4050908, "Lifted3_Lifted2_Lifted1_Mem.used"=>1253912, "Lifted3_Lifted2_Lifted1_Mem.free"=>2796996}]
Expand Down