Skip to content

pipeline: filters: rewrite_tag: Style #1697

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 30, 2025
Merged

pipeline: filters: rewrite_tag: Style #1697

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e935f4e
pipeline: filters: rewrite_tag: Style
esmerel May 29, 2025
68d3a3b
Update pipeline/filters/rewrite-tag.md
esmerel May 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
117 changes: 58 additions & 59 deletions pipeline/filters/rewrite-tag.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,44 +4,43 @@ description: Powerful and flexible routing

# Rewrite Tag

Tags are what makes [routing](../../concepts/data-pipeline/router.md) possible. Tags are set in the configuration of the Input definitions where the records are generated, but there are certain scenarios where might be useful to modify the Tag in the pipeline so we can perform more advanced and flexible routing.
Tags make [routing](../../concepts/data-pipeline/router.md) possible. Tags are set in the configuration of the `INPUT` definitions where the records are generated. There are scenarios where you might want to modify the tag in the pipeline to perform more advanced and flexible routing.

The `rewrite_tag` filter, allows to re-emit a record under a new Tag. Once a record has been re-emitted, the original record can be preserved or discarded.
The _Rewrite Tag_ filter lets you re-emit a record under a new tag. Once a record has been re-emitted, the original record can be preserved or discarded.

## How it Works
The Rewrite Tag filter defines rules that match specific record key content against a regular expression. If a match exists, a new record with the defined tag will be emitted, entering from the beginning of the pipeline. Multiple rules can be specified and are processed in order until one of them matches.

The way it works is defining rules that matches specific record key content against a regular expression, if a match exists, **a new record with the defined Tag will be emitted, entering from the beginning of the pipeline.** Multiple rules can be specified and they are processed in order until one of them matches.
The new tag definition can be composed of:

The new Tag to define can be composed by:
- Alphanumeric characters
- Numbers
- The original tag string or part of it
- Regular expressions groups capture
- Any key or sub-key of the processed record
- Environment variables

* Alphabet characters & Numbers
* Original Tag string or part of it
* Regular Expressions groups capture
* Any key or sub-key of the processed record
* Environment variables

## Configuration Parameters
## Configuration parameters

The `rewrite_tag` filter supports the following configuration parameters:

| Key | Description |
| :--- | :--- |
| Rule | Defines the matching criteria and the format of the Tag for the matching record. The Rule format have four components: `KEY REGEX NEW_TAG KEEP`. For more specific details of the Rule format and it composition read the next section. |
| Emitter\_Name | When the filter emits a record under the new Tag, there is an internal emitter plugin that takes care of the job. Since this emitter expose metrics as any other component of the pipeline, you can use this property to configure an optional name for it. |
| Emitter\_Storage.type | Define a buffering mechanism for the new records created. Note these records are part of the emitter plugin. This option support the values `memory` \(default\) or `filesystem`. If the destination for the new records generated might face backpressure due to latency or slow network, we strongly recommend enabling the `filesystem` mode. |
| Emitter\_Mem\_Buf\_Limit | Set a limit on the amount of memory the tag rewrite emitter can consume if the outputs provide backpressure. The default for this limit is `10M`. The pipeline will pause once the buffer exceeds the value of this setting. For example, if the value is set to `10M` then the pipeline will pause if the buffer exceeds `10M`. The pipeline will remain paused until the output drains the buffer below the `10M` limit. |
| `Rule` | Defines the matching criteria and the format of the tag for the matching record. The Rule format has four components: `KEY REGEX NEW_TAG KEEP`. |
| `Emitter_Name` | Use this property to configure an optional name for the internal emitter plugin that handles filters emitting a record under the new tag. This emitter exposes metrics like any other component of the pipeline. |
| `Emitter_Storage.type` | Define a buffering mechanism for the new records created. These records are part of the emitter plugin. Supported values are `memory` (default) and `filesystem`. If the destination for the new records generated might face backpressure due to latency or slow network, Fluent Bit strongly recommends enabling the `filesystem` mode. |
| `Emitter_Mem_Buf_Limit` | Set a limit on the amount of memory the tag rewrite emitter can consume if the outputs provide backpressure. The default value is `10M`. The pipeline will pause once the buffer exceeds the value of this setting. For example, if the value is set to `10M` then the pipeline will pause if the buffer exceeds `10M`. The pipeline will remain paused until the output drains the buffer below the `10M` limit. |

## Rules

A rule aims to define matching criteria and specify how to create a new Tag for a record. You can define one or multiple rules in the same configuration section. The rules have the following format:
A rule defines matching criteria and specifies how to create a new tag for a record. You can define one or multiple rules in the same configuration section. The rules have the following format:

```text
$KEY REGEX NEW_TAG KEEP
```

### Key

The key represents the name of the _record key_ that holds the _value_ that we want to use to match our regular expression. A key name is specified and prefixed with a `$`. Consider the following structured record \(formatted for readability\):
The key represents the name of the _record key_ that holds the `value` to use to match the regular expression. A key name is specified and prefixed with a `$`. Consider the following structured record (formatted for readability):

```javascript
{
Expand All @@ -54,71 +53,72 @@ The key represents the name of the _record key_ that holds the _value_ that we w
}
```

If we wanted to match against the value of the key `name` we must use `$name`. The key selector is flexible enough to allow to match nested levels of sub-maps from the structure. If we wanted to check the value of the nested key `s2` we can do it specifying `$ss['s1']['s2']`, for short:
To match against the value of the key `name`, you must use `$name`. The key selector is flexible enough to allow to match nested levels of sub-maps from the structure. To capture the value of the nested key `s2`, specify `$ss['s1']['s2']`, for short:

* `$name` = "abc-123"
* `$ss['s1']['s2']` = "flb"
-`$name` = "abc-123"
-`$ss['s1']['s2']` = "flb"

Note that a key must point to a value that contains a string, it's **not valid** for numbers, booleans, maps or arrays.
A key must point to a value that contains a string. It's not valid for numbers, Booleans, maps, or arrays.

### Regex
### Regular expressions

Using a simple regular expression we can specify a matching pattern to use against the value of the key specified above, also we can take advantage of group capturing to create custom placeholder values.
Use a regular expression to specify a matching pattern to use against the value of the key specified previously. You can take advantage of group capturing to create custom placeholder values.

If we wanted to match any record that it `$name` contains a value of the format `string-number` like the example provided above, we might use:
To match any record that it `$name` contains a value of the format `string-number` like the previous example, you might use:

```text
^([a-z]+)-([0-9]+)$
```

Note that in our example we are using parentheses, this teams that we are specifying groups of data. If the pattern matches the value a placeholder will be created that can be consumed by the NEW\_TAG section.
This example uses parentheses to specify groups of data. If the pattern matches the value a placeholder will be created that can be consumed by the `NEW_TAG` section.

If `$name` equals `abc-123` , then the following **placeholders** will be created:
If `$name` equals `abc-123` , then the following placeholders will be created:

* `$0` = "abc-123"
* `$1` = "abc"
* `$2` = "123"
-`$0` = "abc-123"
-`$1` = "abc"
-`$2` = "123"

If the Regular expression do not matches an incoming record, the rule will be skipped and the next rule \(if any\) will be processed.
If the regular expression doesn't match an incoming record, the rule will be skipped and the next rule (if present) will be processed.

### New Tag
### `New_Tag`

If a regular expression has matched the value of the defined key in the rule, we are ready to compose a new Tag for that specific record. The tag is a concatenated string that can contain any of the following characters: `a-z`,`A-Z`, `0-9` and `.-,`.
If a regular expression has matched the value of the defined key in the rule, you can compose a new tag for that specific record. The tag is a concatenated string that can contain any of the following characters: `a-z`,`A-Z`, `0-9` and `.-,`.

A Tag can take any string value from the matching record, the original tag it self, environment variable or general placeholder.
A tag can take any string value from the matching record, the original tag it self, environment variables, or general placeholders.

Consider the following incoming data on the rule:

* Tag = aa.bb.cc
* Record = `{"name": "abc-123", "ss": {"s1": {"s2": "flb"}}}`
* Environment variable $HOSTNAME = fluent
- Tag = aa.bb.cc
- Record = `{"name": "abc-123", "ss": {"s1": {"s2": "flb"}}}`
- Environment variable $HOSTNAME = fluent

With such information we could create a very custom Tag for our record like the following:
With this information you can create a custom tag for your record like the following:

```text
newtag.$TAG.$TAG[1].$1.$ss['s1']['s2'].out.${HOSTNAME}
```

the expected Tag to generated will be:
the expected generated tag will be:

```text
newtag.aa.bb.cc.bb.abc.flb.out.fluent
```

We make use of placeholders, record content and environment variables.
This make use of placeholders, record content, and environment variables.

### Keep

If a rule matches a rule the filter will emit a copy of the record with the new defined Tag. The property keep takes a boolean value to define if the original record with the old Tag must be preserved and continue in the pipeline or just be discarded.
If a rule matches, the filter will emit a copy of the record with the new defined tag. The `keep` property takes a Boolean value to determine if the original record with the old tag must be preserved and continue in the pipeline or be discarded.

You can use `true` or `false` to decide the expected behavior. There is no default value and this is a mandatory field in the rule.
You can use `true` or `false` to decide the expected behavior. This field is mandatory and has no default value.

## Configuration Example
## Configuration example

The following configuration example will emit a dummy \(hand-crafted\) record, the filter will rewrite the tag, discard the old record and print the new record to the standard output interface:
The following configuration example will emit a dummy record. The filter will rewrite the tag, discard the old record, and print the new record to the standard output interface:

{% tabs %}
{% tab title="fluent-bit.conf" %}

```python
[SERVICE]
Flush 1
Expand All @@ -139,9 +139,11 @@ The following configuration example will emit a dummy \(hand-crafted\) record, t
Name stdout
Match from.*
```

{% endtab %}

{% tab title="fluent-bit.yaml" %}

```yaml
service:
flush: 1
Expand All @@ -160,10 +162,10 @@ pipeline:
- name: stdout
match: from.*
```

{% endtab %}
{% endtabs %}


The original tag `test_tag` will be rewritten as `from.test_tag.new.fluent.bit.out`:

```bash
Expand All @@ -180,21 +182,19 @@ Fluent Bit v1.x.x

## Monitoring

As described in the [Monitoring](../../administration/monitoring.md) section, every component of the pipeline of Fluent Bit exposes metrics. The basic metrics exposed by this filter are `drop_records` and `add_records`, they summarize the total of dropped records from the incoming data chunk or the new records added.
As described in the [Monitoring](../../administration/monitoring.md) section, every component of the pipeline of Fluent Bit exposes metrics. The basic metrics exposed by this filter are `drop_records` and `add_records`, which summarize the total of dropped records from the incoming data chunk or the new records added.

Since `rewrite_tag` emit new records that goes through the beginning of the pipeline, it exposes an additional metric called `emit_records` that summarize the total number of emitted records.
The `rewrite_tag` filter emits new records that go through the beginning of the pipeline, and exposes an additional metric called `emit_records` that summarize the total number of emitted records.

### Understanding the Metrics
### Understand the metrics

Using the configuration provided above, if we query the metrics exposed in the HTTP interface we will see the following:

Command:
Using the previously provided configuration, when you query the metrics exposed in the HTTP interface:

```bash
$ curl http://127.0.0.1:2020/api/v1/metrics/ | jq
curl http://127.0.0.1:2020/api/v1/metrics/ | jq
```

Metrics output:
You will see metrics output similar to the following:

```javascript
{
Expand Down Expand Up @@ -227,13 +227,12 @@ Metrics output:
}
```

The _dummy_ input generated two records, the filter dropped two from the chunks and emitted two new ones under a different Tag.

The records generated are handled by the internal Emitter, so the new records are summarized in the Emitter metrics, take a look at the entry called `emitter_for_rewrite_tag.0`.
The _dummy_ input generated two records, while the filter dropped two from the chunks and emitted two new ones under a different tag.

### What is the Emitter ?
The records generated are handled by the internal emitter, so the new records are summarized in the Emitter metrics. Take a look at the entry called `emitter_for_rewrite_tag.0`.

The Emitter is an internal Fluent Bit plugin that allows other components of the pipeline to emit custom records. On this case `rewrite_tag` creates an Emitter instance to use it exclusively to emit records, on that way we can have a granular control of _who_ is emitting what.
### The Emitter

The Emitter name in the metrics can be changed setting up the `Emitter_Name` configuration property described above.
The _Emitter_ is an internal Fluent Bit plugin that allows other components of the pipeline to emit custom records. On this case `rewrite_tag` creates an emitter instance to use it exclusively to emit records, allowing for granular control of who is emitting what.

Change the Emitter name in the metrics by adding the `Emitter_Name` configuration property described previously.