Skip to content

Commit 472a679

Browse files
eschabelleschabell
authored
yaml: fixes to all relevant Processor docs sections across all linked doc pages (#1869)
* Adding warning to yaml configuration pipeline doc page for processors. Signed-off-by: Eric D. Schabell <[email protected]> * Adding warning to processors main doc page for yaml only configuration. Signed-off-by: Eric D. Schabell <[email protected]> * Fixes to standardize the yaml configuration pipeline section doc page. Part of issue #1867. Signed-off-by: Eric D. Schabell <[email protected]> * Fixes to standardize the yaml configuration and add yaml only warning to conditional processing doc page. Part of issue #1867. Signed-off-by: Eric D. Schabell <[email protected]> * Fixes to standardize the yaml configuration and add yaml only warning to content modifier processor doc page. Part of issue #1867. Signed-off-by: Eric D. Schabell <[email protected]> * Fixes to standardize the yaml configuration and add yaml only warning to labels processor doc page. Part of issue #1867. Signed-off-by: Eric D. Schabell <[email protected]> * Fixes to standardize the yaml configuration and add yaml only warning to conditional processing processor doc page. Part of issue #1867. Signed-off-by: Eric D. Schabell <[email protected]> * Fixes to standardize the yaml configuration and add yaml only warning to filters processor doc page. Part of issue #1867. Signed-off-by: Eric D. Schabell <[email protected]> * Fixes to standardize the yaml configuration and add yaml only warning to sql processor doc page. Part of issue #1867. Signed-off-by: Eric D. Schabell <[email protected]> * Fixes to standardize the yaml configuration and add yaml only warning to sampling processor doc page. Part of issue #1867. Signed-off-by: Eric D. Schabell <[email protected]> * Fixes to standardize the yaml configuration and add yaml only warning to opentelemetry envelope processor doc page. Part of issue #1867. Signed-off-by: Eric D. Schabell <[email protected]> --------- Signed-off-by: Eric D. Schabell <[email protected]>
1 parent 1e20ea0 commit 472a679

File tree

9 files changed

+947
-691
lines changed

9 files changed

+947
-691
lines changed

‎administration/configuring-fluent-bit/yaml/pipeline-section.md

Lines changed: 121 additions & 85 deletions
Original file line numberDiff line numberDiff line change
@@ -11,28 +11,42 @@ The `pipeline` section defines the flow of how data is collected, processed, and
1111

1212
## Example configuration
1313

14+
{% hint style="info" %}
15+
16+
**Note:** Processors can be enabled only by using the YAML configuration format. Classic mode configuration format
17+
doesn't support processors.
18+
19+
{% endhint %}
20+
1421
Here's an example of a pipeline configuration:
1522

23+
{% tabs %}
24+
{% tab title="fluent-bit.yaml" %}
25+
1626
```yaml
1727
pipeline:
18-
inputs:
19-
- name: tail
20-
path: /var/log/example.log
21-
parser: json
22-
23-
processors:
24-
logs:
25-
- name: record_modifier
26-
filters:
27-
- name: grep
28-
match: '*'
29-
regex: key pattern
30-
31-
outputs:
32-
- name: stdout
33-
match: '*'
28+
inputs:
29+
- name: tail
30+
path: /var/log/example.log
31+
parser: json
32+
33+
processors:
34+
logs:
35+
- name: record_modifier
36+
37+
filters:
38+
- name: grep
39+
match: '*'
40+
regex: key pattern
41+
42+
outputs:
43+
- name: stdout
44+
match: '*'
3445
```
3546
47+
{% endtab %}
48+
{% endtabs %}
49+
3650
## Pipeline processors
3751
3852
Processors operate on specific signals such as logs, metrics, and traces. They're attached to an input plugin and must specify the signal type they will process.
@@ -41,75 +55,91 @@ Processors operate on specific signals such as logs, metrics, and traces. They'r
4155
4256
In the following example, the `content_modifier` processor inserts or updates (upserts) the key `my_new_key` with the value `123` for all log records generated by the tail plugin. This processor is only applied to log signals:
4357

58+
{% tabs %}
59+
{% tab title="fluent-bit.yaml" %}
60+
4461
```yaml
4562
parsers:
46-
- name: json
47-
format: json
63+
- name: json
64+
format: json
4865
4966
pipeline:
50-
inputs:
51-
- name: tail
52-
path: /var/log/example.log
53-
parser: json
54-
55-
processors:
56-
logs:
57-
- name: content_modifier
58-
action: upsert
59-
key: my_new_key
60-
value: 123
61-
filters:
62-
- name: grep
63-
match: '*'
64-
regex: key pattern
65-
66-
outputs:
67-
- name: stdout
68-
match: '*'
67+
inputs:
68+
- name: tail
69+
path: /var/log/example.log
70+
parser: json
71+
72+
processors:
73+
logs:
74+
- name: content_modifier
75+
action: upsert
76+
key: my_new_key
77+
value: 123
78+
79+
filters:
80+
- name: grep
81+
match: '*'
82+
regex: key pattern
83+
84+
outputs:
85+
- name: stdout
86+
match: '*'
6987
```
7088

89+
{% endtab %}
90+
{% endtabs %}
91+
7192
Here is a more complete example with multiple processors:
7293

94+
{% tabs %}
95+
{% tab title="fluent-bit.yaml" %}
96+
7397
```yaml
7498
service:
75-
log_level: info
76-
http_server: on
77-
http_listen: 0.0.0.0
78-
http_port: 2021
99+
log_level: info
100+
http_server: on
101+
http_listen: 0.0.0.0
102+
http_port: 2021
79103
80104
pipeline:
81-
inputs:
82-
- name: random
83-
tag: test-tag
84-
interval_sec: 1
85-
processors:
86-
logs:
87-
- name: modify
88-
add: hostname monox
89-
- name: lua
90-
call: append_tag
91-
code: |
92-
function append_tag(tag, timestamp, record)
93-
new_record = record
94-
new_record["tag"] = tag
95-
return 1, timestamp, new_record
96-
end
97-
98-
outputs:
99-
- name: stdout
100-
match: '*'
101-
processors:
102-
logs:
103-
- name: lua
104-
call: add_field
105-
code: |
106-
function add_field(tag, timestamp, record)
107-
new_record = record
108-
new_record["output"] = "new data"
109-
return 1, timestamp, new_record
110-
end
105+
inputs:
106+
- name: random
107+
tag: test-tag
108+
interval_sec: 1
109+
110+
processors:
111+
logs:
112+
- name: modify
113+
add: hostname monox
114+
115+
- name: lua
116+
call: append_tag
117+
code: |
118+
function append_tag(tag, timestamp, record)
119+
new_record = record
120+
new_record["tag"] = tag
121+
return 1, timestamp, new_record
122+
end
123+
124+
outputs:
125+
- name: stdout
126+
match: '*'
127+
128+
processors:
129+
logs:
130+
- name: lua
131+
call: add_field
132+
code: |
133+
function add_field(tag, timestamp, record)
134+
new_record = record
135+
new_record["output"] = "new data"
136+
return 1, timestamp, new_record
137+
end
111138
```
112139

140+
{% endtab %}
141+
{% endtabs %}
142+
113143
Processors can be attached to inputs and outputs.
114144

115145
### How Processors are different from Filters
@@ -128,22 +158,28 @@ You can configure existing [Filters](https://docs.fluentbit.io/manual/pipeline/f
128158

129159
In the following example, the `grep` filter is used as a processor to filter log events based on a pattern:
130160

161+
{% tabs %}
162+
{% tab title="fluent-bit.yaml" %}
163+
131164
```yaml
132165
parsers:
133-
- name: json
134-
format: json
166+
- name: json
167+
format: json
135168
136169
pipeline:
137-
inputs:
138-
- name: tail
139-
path: /var/log/example.log
140-
parser: json
141-
142-
processors:
143-
logs:
144-
- name: grep
145-
regex: log aa
146-
outputs:
147-
- name: stdout
148-
match: '*'
170+
inputs:
171+
- name: tail
172+
path: /var/log/example.log
173+
parser: json
174+
175+
processors:
176+
logs:
177+
- name: grep
178+
regex: log aa
179+
outputs:
180+
- name: stdout
181+
match: '*'
149182
```
183+
184+
{% endtab %}
185+
{% endtabs %}

‎pipeline/processors/README.md

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,13 @@ creating a performance bottleneck.
88
Additionally, filters can be implemented in a way that mimics the behavior of
99
processors, but processors can't be implemented in a way that mimics filters.
1010

11+
{% hint style="info" %}
12+
13+
**Note:** Processors can be enabled only by using the YAML configuration format. Classic mode configuration format
14+
doesn't support processors.
15+
16+
{% endhint %}
17+
1118
## Available processors
1219

1320
Fluent Bit offers the following processors:
@@ -28,4 +35,4 @@ Fluent Bit offers the following processors:
2835
Compatible processors include the following features:
2936

3037
- [Conditional Processing](conditional-processing.md): Selectively apply processors
31-
to logs based on the value of fields that those logs contain.
38+
to logs based on the value of fields that those logs contain.

0 commit comments

Comments
 (0)