Skip to content

Commit e2efe28

Browse files
esmerelesmerel
authored
Merge pull request #1689 from fluent/lynettemiles/sc-136152/update-fluent-bit-docs-pipeline-filters-modify
2 parents c349252 + f0a2fb4 commit e2efe28

File tree

1 file changed

+68
-54
lines changed

1 file changed

+68
-54
lines changed

‎pipeline/filters/modify.md

Lines changed: 68 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,15 @@
11
# Modify
22

3-
The _Modify Filter_ plugin allows you to change records using rules and conditions.
3+
The _Modify Filter_ plugin lets you change records using rules and conditions.
44

55
## Example usage
66

7-
As an example using JSON notation to,
7+
As an example, use JSON notation to:
88

9-
* Rename `Key2` to `RenamedKey`
10-
* Add a key `OtherKey` with value `Value3` if `OtherKey` does not yet exist
9+
- Rename `Key2` to `RenamedKey`
10+
- Add a key `OtherKey` with value `Value3` if `OtherKey` doesn't yet exist
1111

12-
_Example \(input\)_
12+
Example (input):
1313

1414
```text
1515
{
@@ -18,7 +18,7 @@ _Example \(input\)_
1818
}
1919
```
2020

21-
_Example \(output\)_
21+
Example (output):
2222

2323
```text
2424
{
@@ -28,56 +28,60 @@ _Example \(output\)_
2828
}
2929
```
3030

31-
## Configuration Parameters
31+
## Configuration parameters
32+
33+
The Modify filter supports multiple parameters.
3234

3335
### Rules
3436

3537
The plugin supports the following rules:
3638

3739
| Operation | Parameter 1 | Parameter 2 | Description |
3840
| :--- | :--- | :--- | :--- |
39-
| Set | STRING:KEY | STRING:VALUE | Add a key/value pair with key `KEY` and value `VALUE`. If `KEY` already exists, _this field is overwritten_ |
40-
| Add | STRING:KEY | STRING:VALUE | Add a key/value pair with key `KEY` and value `VALUE` if `KEY` does not exist |
41-
| Remove | STRING:KEY | NONE | Remove a key/value pair with key `KEY` if it exists |
42-
| Remove\_wildcard | WILDCARD:KEY | NONE | Remove all key/value pairs with key matching wildcard `KEY` |
43-
| Remove\_regex | REGEXP:KEY | NONE | Remove all key/value pairs with key matching regexp `KEY` |
44-
| Rename | STRING:KEY | STRING:RENAMED\_KEY | Rename a key/value pair with key `KEY` to `RENAMED_KEY` if `KEY` exists AND `RENAMED_KEY` _does not exist_ |
45-
| Hard\_rename | STRING:KEY | STRING:RENAMED\_KEY | Rename a key/value pair with key `KEY` to `RENAMED_KEY` if `KEY` exists. If `RENAMED_KEY` already exists, _this field is overwritten_ |
46-
| Copy | STRING:KEY | STRING:COPIED\_KEY | Copy a key/value pair with key `KEY` to `COPIED_KEY` if `KEY` exists AND `COPIED_KEY` _does not exist_ |
47-
| Hard\_copy | STRING:KEY | STRING:COPIED\_KEY | Copy a key/value pair with key `KEY` to `COPIED_KEY` if `KEY` exists. If `COPIED_KEY` already exists, _this field is overwritten_ |
48-
| Move\_to\_start | WILDCARD:KEY | NONE | Move key/value pairs with keys matching KEY to the start of the message |
49-
| Move\_to\_end | WILDCARD:KEY | NONE | Move key/value pairs with keys matching KEY to the end of the message |
50-
51-
* Rules are case insensitive, parameters are not
52-
* Any number of rules can be set in a filter instance.
53-
* Rules are applied in the order they appear, with each rule operating on the result of the previous rule.
41+
| `Set` | `STRING:KEY` | `STRING:VALUE` | Add a key/value pair with key `KEY` and value `VALUE`. If `KEY` already exists, this field is overwritten. |
42+
| `Add` | `STRING:KEY` | `STRING:VALUE` | Add a key/value pair with key `KEY` and value `VALUE` if `KEY` doesn't exist. |
43+
| `Remove` | `STRING:KEY` | _none_ | Remove a key/value pair with key `KEY` if it exists. |
44+
| `Remove_wildcard` | `WILDCARD:KEY` | _none_ | Remove all key/value pairs with key matching wildcard `KEY`. |
45+
| `Remove_regex` | `REGEXP:KEY` | _none_ | Remove all key/value pairs with key matching regexp `KEY`. |
46+
| `Rename` | `STRING:KEY` | `STRING:RENAMED_KEY` | Rename a key/value pair with key `KEY` to `RENAMED_KEY` if `KEY` exists and `RENAMED_KEY` doesn't exist. |
47+
| `Hard_rename` | `STRING:KEY` | `STRING:RENAMED_KEY` | Rename a key/value pair with key `KEY` to `RENAMED_KEY` if `KEY` exists. If `RENAMED_KEY` already exists, this field is overwritten. |
48+
| `Copy` | `STRING:KEY` | `STRING:COPIED_KEY` | Copy a key/value pair with key `KEY` to `COPIED_KEY` if `KEY` exists and `COPIED_KEY` doesn't exist. |
49+
| `Hard_copy` | `STRING:KEY` | `STRING:COPIED_KEY` | Copy a key/value pair with key `KEY` to `COPIED_KEY` if `KEY` exists. If `COPIED_KEY` already exists, this field is overwritten. |
50+
| `Move_to_start` | `WILDCARD:KEY` | _none_ | Move key/value pairs with keys matching `KEY` to the start of the message. |
51+
| `Move_to_end` | `WILDCARD:KEY` | _none_ | Move key/value pairs with keys matching `KEY` to the end of the message. |
52+
53+
- Rules are case insensitive, but parameters aren't.
54+
- Any number of rules can be set in a filter instance.
55+
- Rules are applied in the order they appear, with each rule operating on the result of the previous rule.
5456

5557
### Conditions
5658

5759
The plugin supports the following conditions:
5860

5961
| Condition | Parameter | Parameter 2 | Description |
6062
| :--- | :--- | :--- | :--- |
61-
| Key\_exists | STRING:KEY | NONE | Is `true` if `KEY` exists |
62-
| Key\_does\_not\_exist | STRING:KEY | NONE | Is `true` if `KEY` does not exist |
63-
| A\_key\_matches | REGEXP:KEY | NONE | Is `true` if a key matches regex `KEY` |
64-
| No\_key\_matches | REGEXP:KEY | NONE | Is `true` if no key matches regex `KEY` |
65-
| Key\_value\_equals | STRING:KEY | STRING:VALUE | Is `true` if `KEY` exists and its value is `VALUE` |
66-
| Key\_value\_does\_not\_equal | STRING:KEY | STRING:VALUE | Is `true` if `KEY` exists and its value is not `VALUE` |
67-
| Key\_value\_matches | STRING:KEY | REGEXP:VALUE | Is `true` if key `KEY` exists and its value matches `VALUE` |
68-
| Key\_value\_does\_not\_match | STRING:KEY | REGEXP:VALUE | Is `true` if key `KEY` exists and its value does not match `VALUE` |
69-
| Matching\_keys\_have\_matching\_values | REGEXP:KEY | REGEXP:VALUE | Is `true` if all keys matching `KEY` have values that match `VALUE` |
70-
| Matching\_keys\_do\_not\_have\_matching\_values | REGEXP:KEY | REGEXP:VALUE | Is `true` if all keys matching `KEY` have values that do not match `VALUE` |
71-
72-
* Conditions are case insensitive, parameters are not
73-
* Any number of conditions can be set.
74-
* Conditions apply to the whole filter instance and all its rules. _Not_ to individual rules.
75-
* All conditions have to be `true` for the rules to be applied.
76-
* You can set [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) as `STRING:KEY` for nested key.
77-
78-
## Example \#1 - Add and Rename
79-
80-
In order to start filtering records, you can run the filter from the command line or through the configuration file. The following invokes the [Memory Usage Input Plugin](../inputs/memory-metrics.md), which outputs the following \(example\),
63+
| `Key_exists` | `STRING:KEY` | _none_ | Is `true` if `KEY` exists. |
64+
| `Key_does_not_exist` | `STRING:KEY` | _none_ | Is `true` if `KEY` doesn't exist. |
65+
| `A_key_matches` | `REGEXP:KEY` | _none_ | Is `true` if a key matches regex `KEY`. |
66+
| `No_key_matches` | `REGEXP:KEY` | _none_ | Is `true` if no key matches regex `KEY`. |
67+
| `Key_value_equals` | `STRING:KEY` | `STRING:VALUE` | Is `true` if `KEY` exists and its value is `VALUE`. |
68+
| `Key_value_does_not_equal` | `STRING:KEY` | `STRING:VALUE` | Is `true` if `KEY` exists and its value isn't `VALUE`. |
69+
| `Key_value_matches` | `STRING:KEY` | `REGEXP:VALUE` | Is `true` if key `KEY` exists and its value matches `VALUE`. |
70+
| `Key_value_does_not_match` | `STRING:KEY` | `REGEXP:VALUE` | Is `true` if key `KEY` exists and its value doesn't match `VALUE`. |
71+
| `Matching_keys_have_matching_values` | `REGEXP:KEY` | `REGEXP:VALUE` | Is `true` if all keys matching `KEY` have values that match `VALUE`. |
72+
| `Matching_keys_do_not_have_matching_values` | `REGEXP:KEY` | `REGEXP:VALUE` | Is `true` if all keys matching `KEY` have values that don't match `VALUE`. |
73+
74+
- Conditions are case insensitive, but parameters aren't.
75+
- Any number of conditions can be set.
76+
- Conditions apply to the whole filter instance and all its rules. _Not_ to individual rules.
77+
- All conditions must be `true` for the rules to be applied.
78+
- You can set [Record Accessor](../../administration/configuring-fluent-bit/classic-mode/record-accessor.md) as `STRING:KEY` for nested key.
79+
80+
## Example 1 - add and rename
81+
82+
To start filtering records, you can run the filter from the command line or through
83+
the configuration file. The following invokes the [Memory Usage Input Plugin](../inputs/memory-metrics.md),
84+
which outputs data similar to the following:
8185

8286
```text
8387
[0] memory: [1488543156, {"Mem.total"=>1016044, "Mem.used"=>841388, "Mem.free"=>174656, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
@@ -86,9 +90,9 @@ In order to start filtering records, you can run the filter from the command lin
8690
[3] memory: [1488543159, {"Mem.total"=>1016044, "Mem.used"=>841420, "Mem.free"=>174624, "Swap.total"=>2064380, "Swap.used"=>139888, "Swap.free"=>1924492}]
8791
```
8892

89-
### Using command Line
93+
### Using command line
9094

91-
> Note: Using the command line mode requires quotes parse the wildcard properly. The use of a configuration file is recommended.
95+
Using the command line mode requires quotes parse the wildcard properly. The use of a configuration file is recommended.
9296

9397
```text
9498
bin/fluent-bit -i mem \
@@ -105,10 +109,11 @@ bin/fluent-bit -i mem \
105109
-o stdout
106110
```
107111

108-
### Configuration File
112+
### Configuration file
109113

110114
{% tabs %}
111115
{% tab title="fluent-bit.conf" %}
116+
112117
```python
113118
[INPUT]
114119
Name mem
@@ -129,9 +134,11 @@ bin/fluent-bit -i mem \
129134
Rename Swap.total SWAPTOTAL
130135
Add Mem.total TOTALMEM
131136
```
137+
132138
{% endtab %}
133139

134140
{% tab title="fluent-bit.yaml" %}
141+
135142
```yaml
136143
pipeline:
137144
inputs:
@@ -153,12 +160,13 @@ pipeline:
153160
- name: stdout
154161
match: '*'
155162
```
163+
156164
{% endtab %}
157165
{% endtabs %}
158166
159167
### Result
160168
161-
The output of both the command line and configuration invocations should be identical and result in the following output.
169+
The output of both the command line and configuration invocations should be identical and result in the following output:
162170
163171
```text
164172
[2018/04/06 01:35:13] [ info] [engine] started
@@ -168,12 +176,13 @@ The output of both the command line and configuration invocations should be iden
168176
[3] mem.local: [1522980613.000122671, {"Mem.total"=>4050908, "MEMUSED"=>738068, "MEMFREE"=>3312840, "SWAPTOTAL"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Service1"=>"SOMEVALUE", "Service3"=>"SOMEVALUE3", "Mem.total2"=>"TOTALMEM2"}]
169177
```
170178
171-
## Example \#2 - Conditionally Add and Remove
179+
## Example 2 - conditionally add and remove
172180
173-
### Configuration File
181+
### Use a configuration file
174182
175183
{% tabs %}
176184
{% tab title="fluent-bit.conf" %}
185+
177186
```python
178187
[INPUT]
179188
Name mem
@@ -211,9 +220,11 @@ The output of both the command line and configuration invocations should be iden
211220
Name stdout
212221
Match *
213222
```
223+
214224
{% endtab %}
215225

216226
{% tab title="fluent-bit.yaml" %}
227+
217228
```yaml
218229
pipeline:
219230
inputs:
@@ -242,10 +253,11 @@ pipeline:
242253
- name: stdout
243254
match: '*'
244255
```
256+
245257
{% endtab %}
246258
{% endtabs %}
247259
248-
### Result
260+
### Add and remove result
249261
250262
```text
251263
[2018/06/14 07:37:34] [ info] [engine] started (pid=1493)
@@ -255,12 +267,13 @@ pipeline:
255267
[3] mem.local: [1528925858.000152319, {"cpustats"=>"UNKNOWN", "sourcetype"=>"memstats", "cpustats_more"=>"STILL_UNKNOWN"}]
256268
```
257269
258-
## Example \#3 - Emoji
270+
## Example 3 - emoji
259271
260-
### Configuration File
272+
### Emoji configuration File
261273
262274
{% tabs %}
263275
{% tab title="fluent-bit.conf" %}
276+
264277
```python
265278
[INPUT]
266279
Name mem
@@ -283,9 +296,11 @@ pipeline:
283296
Set ❄️ is_cold
284297
Set 💦 is_wet
285298
```
299+
286300
{% endtab %}
287301

288302
{% tab title="fluent-bit.yaml" %}
303+
289304
```yaml
290305
pipeline:
291306
inputs:
@@ -310,11 +325,11 @@ pipeline:
310325
- name: stdout
311326
match: '*'
312327
```
328+
313329
{% endtab %}
314330
{% endtabs %}
315331
316-
317-
### Result
332+
### Emoji example result
318333
319334
```text
320335
[2018/06/14 07:46:11] [ info] [engine] started (pid=21875)
@@ -324,4 +339,3 @@ pipeline:
324339
[3] mem.local: [1528926375.000090841, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
325340
[0] mem.local: [1528926376.000610974, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}]
326341
```
327-

0 commit comments

Comments
 (0)