Skip to content

Commit 13c140a

Browse files
authored
Merge pull request #1477 from fluent/lynettemiles/sc-113606/update-pipeline-filters-record-modifier-for
2 parents cc6af37 + 182b122 commit 13c140a

File tree

1 file changed

+52
-40
lines changed

1 file changed

+52
-40
lines changed

pipeline/filters/record-modifier.md

Lines changed: 52 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -1,36 +1,40 @@
11
# Record Modifier
22

3-
The _Record Modifier Filter_ plugin allows to append fields or to exclude specific fields.
3+
The _Record Modifier_ [filter](pipeline/filters.md) plugin lets you append
4+
fields to a record, or exclude specific fields.
45

5-
## Configuration Parameters
6+
## Configuration parameters
67

7-
The plugin supports the following configuration parameters: _Remove\_key_ and _Allowlist\_key_ are exclusive.
8+
The plugin supports the following configuration parameters:
89

910
| Key | Description |
1011
| :--- | :--- |
11-
| Record | Append fields. This parameter needs key and value pair. |
12-
| Remove\_key | If the key is matched, that field is removed. |
13-
| Allowlist\_key | If the key is **not** matched, that field is removed. |
14-
| Whitelist\_key | An alias of `Allowlist_key` for backwards compatibility. |
15-
| Uuid\_key| If set, the plugin appends uuid to each record. The value assigned becomes the key in the map.|
12+
| `Record` | Append fields. This parameter needs a key/value pair. |
13+
| `Remove_key` | If the key is matched, that field is removed. You can this or `Allowlist_key`.|
14+
| `Allowlist_key` | If the key isn't matched, that field is removed. You can this or `Remove_key`. |
15+
| `Whitelist_key` | An alias of `Allowlist_key` for backwards compatibility. |
16+
| `Uuid_key` | If set, the plugin appends Uuid to each record. The value assigned becomes the key in the map. |
1617

17-
## Getting Started
18+
## Get started
1819

19-
In order to start filtering records, you can run the filter from the command line or through the configuration file.
20+
To start filtering records, run the filter from the command line or through a
21+
configuration file.
2022

21-
This is a sample in\_mem record to filter.
23+
This is a sample `in_mem` record to filter.
2224

2325
```text
2426
{"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>299352, "Swap.total"=>2064380, "Swap.used"=>32656, "Swap.free"=>2031724}
2527
```
2628

2729
### Append fields
2830

29-
The following configuration file is to append product name and hostname \(via environment variable\) to record.
31+
The following configuration file appends a product name and hostname to a record
32+
using an environment variable:
3033

3134
{% tabs %}
3235
{% tab title="fluent-bit.conf" %}
33-
```python
36+
37+
```python copy
3438
[INPUT]
3539
Name mem
3640
Tag mem.local
@@ -45,48 +49,50 @@ The following configuration file is to append product name and hostname \(via en
4549
Record hostname ${HOSTNAME}
4650
Record product Awesome_Tool
4751
```
52+
4853
{% endtab %}
4954

5055
{% tab title="fluent-bit.yaml" %}
51-
```yaml
56+
57+
```yaml copy
5258
pipeline:
5359
inputs:
5460
- name: mem
5561
tag: mem.local
5662
filters:
5763
- name: record_modifier
5864
match: '*'
59-
record:
65+
record:
6066
- hostname ${HOSTNAME}
6167
- product Awesome_Tool
6268
outputs:
6369
- name: stdout
6470
match: '*'
6571
```
72+
6673
{% endtab %}
6774
{% endtabs %}
6875
76+
You can run the filter from command line:
6977
70-
You can also run the filter from command line.
71-
72-
```text
73-
$ fluent-bit -i mem -o stdout -F record_modifier -p 'Record=hostname ${HOSTNAME}' -p 'Record=product Awesome_Tool' -m '*'
78+
```shell copy
79+
fluent-bit -i mem -o stdout -F record_modifier -p 'Record=hostname ${HOSTNAME}' -p 'Record=product Awesome_Tool' -m '*'
7480
```
7581

76-
The output will be
82+
The output looks something like:
7783

78-
```python
84+
```python copy
7985
[0] mem.local: [1492436882.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>299352, "Swap.total"=>2064380, "Swap.used"=>32656, "Swap.free"=>2031724, "hostname"=>"localhost.localdomain", "product"=>"Awesome_Tool"}]
8086
```
8187

82-
### Remove fields with Remove\_key
83-
84-
The following configuration file is to remove 'Swap.\*' fields.
88+
### Remove fields with `Remove_key`
8589

90+
The following configuration file removes `Swap.*` fields:
8691

8792
{% tabs %}
8893
{% tab title="fluent-bit.conf" %}
89-
```python
94+
95+
```python copy
9096
[INPUT]
9197
Name mem
9298
Tag mem.local
@@ -102,47 +108,51 @@ The following configuration file is to remove 'Swap.\*' fields.
102108
Remove_key Swap.used
103109
Remove_key Swap.free
104110
```
111+
105112
{% endtab %}
106113

107114
{% tab title="fluent-bit.yaml" %}
108-
```yaml
115+
116+
```yaml copy
109117
pipeline:
110118
inputs:
111119
- name: mem
112120
tag: mem.local
113121
filters:
114122
- name: record_modifier
115123
match: '*'
116-
remove_key:
124+
remove_key:
117125
- Swap.total
118126
- Swap.used
119127
- Swap.free
120128
outputs:
121129
- name: stdout
122130
match: '*'
123131
```
132+
124133
{% endtab %}
125134
{% endtabs %}
126135
127136
You can also run the filter from command line.
128137
129-
```text
130-
$ fluent-bit -i mem -o stdout -F record_modifier -p 'Remove_key=Swap.total' -p 'Remove_key=Swap.free' -p 'Remove_key=Swap.used' -m '*'
138+
```shell copy
139+
fluent-bit -i mem -o stdout -F record_modifier -p 'Remove_key=Swap.total' -p 'Remove_key=Swap.free' -p 'Remove_key=Swap.used' -m '*'
131140
```
132141

133-
The output will be
142+
The output looks something like:
134143

135144
```python
136145
[0] mem.local: [1492436998.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>295332}]
137146
```
138147

139-
### Remove fields with Allowlist\_key
148+
### Retain fields with `Allowlist_key`
140149

141-
The following configuration file is to remain 'Mem.\*' fields.
150+
The following configuration file retains `Mem.*` fields.
142151

143152
{% tabs %}
144153
{% tab title="fluent-bit.conf" %}
145-
```python
154+
155+
```python copy
146156
[INPUT]
147157
Name mem
148158
Tag mem.local
@@ -158,37 +168,39 @@ The following configuration file is to remain 'Mem.\*' fields.
158168
Allowlist_key Mem.used
159169
Allowlist_key Mem.free
160170
```
171+
161172
{% endtab %}
162173

163174
{% tab title="fluent-bit.yaml" %}
164-
```yaml
175+
176+
```yaml copy
165177
pipeline:
166178
inputs:
167179
- name: mem
168180
tag: mem.local
169181
filters:
170182
- name: record_modifier
171183
match: '*'
172-
Allowlist_key:
184+
Allowlist_key:
173185
- Mem.total
174186
- Mem.used
175187
- Mem.free
176188
outputs:
177189
- name: stdout
178190
match: '*'
179191
```
192+
180193
{% endtab %}
181194
{% endtabs %}
182195
183-
You can also run the filter from command line.
196+
You can also run the filter from command line:
184197
185-
```text
186-
$ fluent-bit -i mem -o stdout -F record_modifier -p 'Allowlist_key=Mem.total' -p 'Allowlist_key=Mem.free' -p 'Allowlist_key=Mem.used' -m '*'
198+
```shell copy
199+
fluent-bit -i mem -o stdout -F record_modifier -p 'Allowlist_key=Mem.total' -p 'Allowlist_key=Mem.free' -p 'Allowlist_key=Mem.used' -m '*'
187200
```
188201

189-
The output will be
202+
The output looks something like:
190203

191204
```python
192205
[0] mem.local: [1492436998.000000000, {"Mem.total"=>1016024, "Mem.used"=>716672, "Mem.free"=>295332}]
193206
```
194-

0 commit comments

Comments
 (0)