Skip to content

Commit bbed530

Browse files
eschabelleschabell
authored
Update the filter docs with YAML configuration examples, the first half of filters in list. Fixes #1872. (#1873)
* Adding YAML examples for aws metadata filter doc. Part of issue #1872. Signed-off-by: Eric D. Schabell <[email protected]> * Adding YAML examples for ecs metadata filter doc. Part of issue #1872. Signed-off-by: Eric D. Schabell <[email protected]> * Adding YAML examples for geoip2 filter doc. Part of issue #1872. Signed-off-by: Eric D. Schabell <[email protected]> * Adding YAML examples for kubernetes filter doc. Part of issue #1872. Signed-off-by: Eric D. Schabell <[email protected]> * Adding YAML examples for lua filter doc. Part of issue #1872. Signed-off-by: Eric D. Schabell <[email protected]> * Adding YAML examples for record modifier filter doc. Part of issue #1872. Signed-off-by: Eric D. Schabell <[email protected]> * Adding YAML examples for nightfall filter doc. Part of issue #1872. Signed-off-by: Eric D. Schabell <[email protected]> * Adding YAML examples for standard output filter doc. Part of issue #1872. Signed-off-by: Eric D. Schabell <[email protected]> * Adding YAML examples for throttle filter doc. Part of issue #1872. Signed-off-by: Eric D. Schabell <[email protected]> * Adding YAML examples for tensorflow filter doc. Part of issue #1872. Signed-off-by: Eric D. Schabell <[email protected]> * Adding YAML examples for multiline stacktrace filter doc. Part of issue #1872. Signed-off-by: Eric D. Schabell <[email protected]> --------- Signed-off-by: Eric D. Schabell <[email protected]>
1 parent d095766 commit bbed530

File tree

11 files changed

+836
-283
lines changed

11 files changed

+836
-283
lines changed

‎pipeline/filters/aws-metadata.md

Lines changed: 79 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -29,12 +29,12 @@ If you run Fluent Bit in a container, you might need to use instance metadata v1
2929
Run Fluent Bit from the command line:
3030

3131
```shell
32-
bin/fluent-bit -c /PATH_TO_CONF_FILE/fluent-bit.conf
32+
$ ./fluent-bit -c /PATH_TO_CONF_FILE/fluent-bit.conf
3333
```
3434

3535
You should see results like this:
3636

37-
```shell
37+
```text
3838
[2020/01/17 07:57:17] [ info] [engine] started (pid=32744)
3939
[0] dummy: [1579247838.000171227, {"message"=>"dummy", "az"=>"us-west-2c", "ec2_instance_id"=>"i-0c862eca9038f5aae", "ec2_instance_type"=>"t2.medium", "private_ip"=>"172.31.6.59", "vpc_id"=>"vpc-7ea11c06", "ami_id"=>"ami-0841edc20334f9287", "account_id"=>"YOUR_ACCOUNT_ID", "hostname"=>"ip-172-31-6-59.us-west-2.compute.internal"}]
4040
[0] dummy: [1601274509.970235760, {"message"=>"dummy", "az"=>"us-west-2c", "ec2_instance_id"=>"i-0c862eca9038f5aae", "ec2_instance_type"=>"t2.medium", "private_ip"=>"172.31.6.59", "vpc_id"=>"vpc-7ea11c06", "ami_id"=>"ami-0841edc20334f9287", "account_id"=>"YOUR_ACCOUNT_ID", "hostname"=>"ip-172-31-6-59.us-west-2.compute.internal"}]
@@ -44,7 +44,38 @@ You should see results like this:
4444

4545
The following is an example of a configuration file:
4646

47-
```python
47+
{% tabs %}
48+
{% tab title="fluent-bit.yaml" %}
49+
50+
```yaml
51+
pipeline:
52+
inputs:
53+
- name: dummy
54+
tag: dummy
55+
56+
filters:
57+
- name: aws
58+
match: '*'
59+
imds_version: v1
60+
az: true
61+
ec2_instance_id: true
62+
ec2_instance_type: true
63+
private_ip: true
64+
ami_id: true
65+
account_id: true
66+
hostname: true
67+
vpc_id: true
68+
tags_enabled: true
69+
70+
outputs:
71+
- name: stdout
72+
match: '*'
73+
```
74+
75+
{% endtab %}
76+
{% tab title="fluent-bit.conf" %}
77+
78+
```text
4879
[INPUT]
4980
Name dummy
5081
Tag dummy
@@ -68,6 +99,9 @@ The following is an example of a configuration file:
6899
Match *
69100
```
70101

102+
{% endtab %}
103+
{% endtabs %}
104+
71105
## EC2 tags
72106

73107
EC2 Tags let you label and organize your EC2 instances by creating custom-defined key-value pairs. These tags are commonly used for resource management, cost allocation, and automation. Including them in the Fluent Bit-generated logs is almost essential.
@@ -84,14 +118,33 @@ To use the `tags_enabled true` feature in Fluent Bit, the [instance-metadata-tag
84118

85119
Assume the EC2 instance has many tags, some of which have lengthy values that are irrelevant to the logs you want to collect. Only two tags, `department` and `project`, are valuable for your purpose. The following configuration reflects this requirement:
86120

87-
```python
121+
{% tabs %}
122+
{% tab title="fluent-bit.yaml" %}
123+
124+
```yaml
125+
pipeline:
126+
127+
filters:
128+
- name: aws
129+
match: '*'
130+
tags_enabled: true
131+
tags_include: department,project
132+
```
133+
134+
{% endtab %}
135+
{% tab title="fluent-bit.conf" %}
136+
137+
```text
88138
[FILTER]
89139
Name aws
90140
Match *
91141
tags_enabled true
92142
tags_include department,project
93143
```
94144

145+
{% endtab %}
146+
{% endtabs %}
147+
95148
If you run Fluent Bit logs might look like the following:
96149

97150
```text
@@ -104,16 +157,35 @@ Suppose the EC2 instance has three tags: `Name:fluent-bit-docs-example`, `projec
104157

105158
Here is an example configuration that achieves this:
106159

107-
```python
160+
{% tabs %}
161+
{% tab title="fluent-bit.yaml" %}
162+
163+
```yaml
164+
pipeline:
165+
166+
filters:
167+
- name: aws
168+
match: '*'
169+
tags_enabled: true
170+
tags_exclude: department
171+
```
172+
173+
{% endtab %}
174+
{% tab title="fluent-bit.conf" %}
175+
176+
```text
108177
[FILTER]
109178
Name aws
110179
Match *
111180
tags_enabled true
112181
tags_exclude department
113182
```
114183

184+
{% endtab %}
185+
{% endtabs %}
186+
115187
The resulting logs might look like this:
116188

117-
```shell
189+
```text
118190
{"log"=>"aws is awesome", "az"=>"us-east-1a", "ec2_instance_id"=>"i-0e66fc7f9809d7168", "Name"=>"fluent-bit-docs-example", "project"=>"fluentbit"}
119-
```
191+
```

‎pipeline/filters/ecs-metadata.md

Lines changed: 122 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,9 +35,50 @@ The following template variables can be used for values with the `ADD` option. S
3535

3636
### Configuration file
3737

38+
Below configurations assume a properly configured parsers file and 'storage.path' variable defined in the services
39+
section of the fluent bit configuration (not shown below).
40+
3841
#### Example 1: Attach Task ID and cluster name to container logs
3942

40-
```python
43+
{% tabs %}
44+
{% tab title="fluent-bit.yaml" %}
45+
46+
```yaml
47+
pipeline:
48+
inputs:
49+
- name: tail
50+
tag: ecs.*
51+
path: /var/lib/docker/containers/*/*.log
52+
docker_mode: on
53+
docker_mode_flush: 5
54+
docker_mode_parser: container_firstline
55+
parser: docker
56+
db: /var/fluent-bit/state/flb_container.db
57+
mem_buf_limit: 50MB
58+
skip_long_lines: on
59+
refresh_interval: 10
60+
rotate_wait: 30
61+
storage.type: filesystem
62+
read_from_head: off
63+
64+
filters:
65+
- name: ecs
66+
match: '*'
67+
ecs_tag_prefix: ecs.var.lib.docker.containers.
68+
add:
69+
- ecs_task_id $TaskID
70+
- cluster $ClusterName
71+
72+
outputs:
73+
- name: stdout
74+
match: '*'
75+
format: json_lines
76+
```
77+
78+
{% endtab %}
79+
{% tab title="fluent-bit.conf" %}
80+
81+
```text
4182
[INPUT]
4283
Name tail
4384
Tag ecs.*
@@ -67,6 +108,9 @@ The following template variables can be used for values with the `ADD` option. S
67108
Format json_lines
68109
```
69110

111+
{% endtab %}
112+
{% endtabs %}
113+
70114
The output log should be similar to:
71115

72116
```text
@@ -80,6 +124,42 @@ The output log should be similar to:
80124

81125
#### Example 2: Attach customized resource name to container logs
82126

127+
{% tabs %}
128+
{% tab title="fluent-bit.yaml" %}
129+
130+
```yaml
131+
pipeline:
132+
inputs:
133+
- name: tail
134+
tag: ecs.*
135+
path: /var/lib/docker/containers/*/*.log
136+
docker_mode: on
137+
docker_mode_flush: 5
138+
docker_mode_parser: container_firstline
139+
parser: docker
140+
db: /var/fluent-bit/state/flb_container.db
141+
mem_buf_limit: 50MB
142+
skip_long_lines: on
143+
refresh_interval: 10
144+
rotate_wait: 30
145+
storage.type: filesystem
146+
read_from_head: off
147+
148+
filters:
149+
- name: ecs
150+
match: '*'
151+
ecs_tag_prefix: ecs.var.lib.docker.containers.
152+
add: resource $ClusterName.$TaskDefinitionFamily.$TaskID.$ECSContainerName
153+
154+
outputs:
155+
- name: stdout
156+
match: '*'
157+
format: json_lines
158+
```
159+
160+
{% endtab %}
161+
{% tab title="fluent-bit.conf" %}
162+
83163
```text
84164
[INPUT]
85165
Name tail
@@ -109,6 +189,9 @@ The output log should be similar to:
109189
Format json_lines
110190
```
111191

192+
{% endtab %}
193+
{% endtabs %}
194+
112195
The output log would be similar to:
113196

114197
```text
@@ -124,9 +207,42 @@ The template variables in the value for the `resource` key are separated by dot
124207

125208
#### Example 3: Attach cluster metadata to non-container logs
126209

127-
This examples shows a use case for the `Cluster_Metadata_Only` option- attaching cluster metadata to ECS Agent logs.
210+
This examples shows a use case for the `Cluster_Metadata_Only` option attaching cluster metadata to ECS Agent logs.
211+
212+
{% tabs %}
213+
{% tab title="fluent-bit.yaml" %}
214+
215+
```yaml
216+
pipeline:
217+
inputs:
218+
- name: tail
219+
tag: ecsagent.*
220+
path: /var/log/ecs/*
221+
db: /var/fluent-bit/state/flb_ecs.db
222+
mem_buf_limit: 50MB
223+
skip_long_lines: on
224+
refresh_interval: 10
225+
rotate_wait: 30
226+
storage.type: filesystem
227+
# Collect all logs on instance
228+
read_from_head: on
128229

129-
```python
230+
filters:
231+
- name: ecs
232+
match: '*'
233+
cluster_metadata_only: on
234+
add: cluster $ClusterName
235+
236+
outputs:
237+
- name: stdout
238+
match: '*'
239+
format: json_lines
240+
```
241+
242+
{% endtab %}
243+
{% tab title="fluent-bit.conf" %}
244+
245+
```text
130246
[INPUT]
131247
Name tail
132248
Tag ecsagent.*
@@ -151,3 +267,6 @@ This examples shows a use case for the `Cluster_Metadata_Only` option- attaching
151267
Match *
152268
Format json_lines
153269
```
270+
271+
{% endtab %}
272+
{% endtabs %}

‎pipeline/filters/geoip2-filter.md

Lines changed: 32 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,33 @@ This plugin supports the following configuration parameters:
2222

2323
The following configuration processes the incoming `remote_addr` and appends country information retrieved from the GeoLite2 database.
2424

25-
```python
25+
{% tabs %}
26+
{% tab title="fluent-bit.yaml" %}
27+
28+
```yaml
29+
pipeline:
30+
inputs:
31+
- name: dummy
32+
dummy: {"remote_addr": "8.8.8.8"}
33+
34+
filters:
35+
- name: gioip2
36+
match: '*'
37+
database: GioLite2-City.mmdb
38+
lookup_key: remote_addr
39+
record:
40+
- country remote_addr %{country.names.en}
41+
- isocode remote_addr %{country.iso_code}
42+
43+
outputs:
44+
- name: stdout
45+
match: '*'
46+
```
47+
48+
{% endtab %}
49+
{% tab title="fluent-bit.conf" %}
50+
51+
```text
2652
[INPUT]
2753
Name dummy
2854
Dummy {"remote_addr": "8.8.8.8"}
@@ -40,6 +66,9 @@ The following configuration processes the incoming `remote_addr` and appends cou
4066
Match *
4167
```
4268

69+
{% endtab %}
70+
{% endtabs %}
71+
4372
Each `Record` parameter specifies the following triplet:
4473

4574
- `country`: The field name to be added to records.
@@ -48,6 +77,6 @@ Each `Record` parameter specifies the following triplet:
4877

4978
By running Fluent Bit with this configuration, you will see the following output:
5079

51-
```javascript
80+
```text
5281
{"remote_addr": "8.8.8.8", "country": "United States", "isocode": "US"}
53-
```
82+
```

0 commit comments

Comments
 (0)