Skip to content

Commit 2604a02

Browse files
esmerelesmerel
authored
Merge pull request #1694 from fluent/lynettemiles/sc-136155/update-fluent-bit-docs-pipeline-filters-nightfall
2 parents e2efe28 + ae9d226 commit 2604a02

File tree

1 file changed

+45
-29
lines changed

1 file changed

+45
-29
lines changed

‎pipeline/filters/nightfall.md

Lines changed: 45 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -1,42 +1,31 @@
11
# Nightfall
22

3-
The Nightfall filter scans logs for sensitive data and redacts the sensitive portions. This filter supports scanning for
4-
various sensitive information, ranging from API keys and personally identifiable information(PII) to custom regexes you
5-
define. You can configure what to scan for in the [Nightfall Dashboard](https://app.nightfall.ai).
3+
The _Nightfall_ filter scans logs for sensitive data and redacts any sensitive
4+
portions. This filter supports scanning for various sensitive information, ranging
5+
from API keys and Personally Identifiable Information (PII) to custom regular
6+
expressions you define. You can configure what to scan for in the
7+
[Nightfall Dashboard](https://app.nightfall.ai).
68

7-
> This filter is not enabled by default in 1.9.0 due to a typo. It must be enabled by setting flag ```-DFLB_FILTER_NIGHTFALL=ON``` when building. In 1.9.1 and above this is fixed.
8-
## Configuration Parameters
9+
This filter isn't enabled by default in version 1.9.0 due to a typo. To enable it,
10+
set the flag ```-DFLB_FILTER_NIGHTFALL=ON``` when building. This is fixed for
11+
versions 1.9.1 and later.
12+
13+
## Configuration parameters
914

1015
The plugin supports the following configuration parameters:
1116

1217
| Key | Description | Default |
1318
| :--- | :--- | :--- |
14-
| nightfall\_api_key | The Nightfall API key to scan your logs with, obtainable from the [Nightfall Dashboard](https://app.nightfall.ai) | |
15-
| policy\_id | The Nightfall dev platform policy to scan your logs with, configurable in the [Nightfall Dashboard](https://app.nightfall.ai/developer-platform/policies). | |
16-
| sampling\_rate | The rate controlling how much of your logs you wish to be scanned, must be a float between (0,1]. 1 means all logs will be scanned. Useful for avoiding rate limits in conjunction with Fluent Bit's match rule.| 1 |
17-
| tls.debug | Debug level between 0 (nothing) and 4 (every detail). | 0 |
18-
| tls.verify | When enabled, turns on certificate validation when connecting to the Nightfall API. | true |
19-
| tls.ca_path | Absolute path to root certificates, required if tls.verify is true. | |
20-
21-
### Command Line
19+
| `nightfall_api_key` | The Nightfall API key to scan your logs with, obtainable from the [Nightfall Dashboard](https://app.nightfall.ai) | _none_ |
20+
| `policy_id` | The Nightfall developer platform policy to scan your logs with, configurable in the [Nightfall Dashboard](https://app.nightfall.ai/developer-platform/policies). | _none_ |
21+
| `sampling_rate` | The rate controlling how much of your logs you wish to be scanned. Must be a float between `(0,1]`. `1` means all logs will be scanned. Use this setting to avoid rate limits in conjunction with Fluent Bit match rules.| `1` |
22+
| `tls.debug` | Debug level between `0` (nothing) and `4` (every detail). | `0` |
23+
| `tls.verify` | When enabled, turns on certificate validation when connecting to the Nightfall API. | `true` |
24+
| `tls.ca_path` | Absolute path to root certificates, required if `tls.verify` is true. | _none_ |
2225

23-
```text
24-
$ bin/fluent-bit -c /PATH_TO_CONF_FILE/fluent-bit.conf
25-
26-
[2022/02/09 19:46:22] [ info] [engine] started (pid=53844)
27-
[2022/02/09 19:46:22] [ info] [storage] version=1.1.5, initializing...
28-
[2022/02/09 19:46:22] [ info] [storage] in-memory
29-
[2022/02/09 19:46:22] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
30-
[2022/02/09 19:46:22] [ info] [cmetrics] version=0.2.2
31-
[2022/02/09 19:46:22] [ info] [input:http:http.0] listening on 0.0.0.0:8000
32-
[2022/02/09 19:46:22] [ info] [sp] stream processor started
33-
[2022/02/09 19:46:30] [ info] [filter:nightfall:nightfall.0] Nightfall request http_do=0, HTTP Status: 200
34-
[0] app.log: [1644464790.280412000, {"A"=>"there is nothing sensitive here", "B"=>[{"A"=>"my credit card number is *******************"}, {"A"=>"*********** is my social security."}], "C"=>false, "D"=>"key ********************"}]
35-
[2022/02/09 19:47:25] [ info] [filter:nightfall:nightfall.0] Nightfall request http_do=0, HTTP Status: 200
36-
[0] app.log: [1644464845.675431000, {"A"=>"a very safe string"}]
37-
```
26+
### Configuration file
3827

39-
### Configuration File
28+
The following is an example of a configuration file for the Nightfall filter:
4029

4130
```text
4231
[INPUT]
@@ -56,3 +45,30 @@ $ bin/fluent-bit -c /PATH_TO_CONF_FILE/fluent-bit.conf
5645
Name stdout
5746
```
5847

48+
### Command line
49+
50+
After you configure the filter, you can use the it from the command line by running a
51+
command like:
52+
53+
```shell
54+
bin/fluent-bit -c /PATH_TO_CONF_FILE/fluent-bit.conf
55+
```
56+
57+
Replace _`PATH_TO_CONF_FILE`_ with the path for where your filter configuration file
58+
is located.
59+
60+
Which results in output like:
61+
62+
```text
63+
[2022/02/09 19:46:22] [ info] [engine] started (pid=53844)
64+
[2022/02/09 19:46:22] [ info] [storage] version=1.1.5, initializing...
65+
[2022/02/09 19:46:22] [ info] [storage] in-memory
66+
[2022/02/09 19:46:22] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
67+
[2022/02/09 19:46:22] [ info] [cmetrics] version=0.2.2
68+
[2022/02/09 19:46:22] [ info] [input:http:http.0] listening on 0.0.0.0:8000
69+
[2022/02/09 19:46:22] [ info] [sp] stream processor started
70+
[2022/02/09 19:46:30] [ info] [filter:nightfall:nightfall.0] Nightfall request http_do=0, HTTP Status: 200
71+
[0] app.log: [1644464790.280412000, {"A"=>"there is nothing sensitive here", "B"=>[{"A"=>"my credit card number is *******************"}, {"A"=>"*********** is my social security."}], "C"=>false, "D"=>"key ********************"}]
72+
[2022/02/09 19:47:25] [ info] [filter:nightfall:nightfall.0] Nightfall request http_do=0, HTTP Status: 200
73+
[0] app.log: [1644464845.675431000, {"A"=>"a very safe string"}]
74+
```

0 commit comments

Comments
 (0)