Skip to content

Commit

Permalink
moved to pure go bpf lexer (#113)
Browse files Browse the repository at this point in the history
* move to go-pcap for bpf compile

* updated docs
  • Loading branch information
mosajjal authored Dec 13, 2024
1 parent 3e7f7f6 commit b539808
Show file tree
Hide file tree
Showing 17 changed files with 210 additions and 141 deletions.
2 changes: 1 addition & 1 deletion docs/content/en/docs/Inputs/filters_masks.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ There are a few ways to manipulate incoming packets in various steps of `dnsmons
## BPF
{{< alert >}}Applied at kernel level{{< /alert >}}

BPF is by far the most performant way to filter incoming packets. It's only supported on live capture (`--devName`). It uses the `tcpdump`'s [pcap-filter](https://www.tcpdump.org/manpages/pcap-filter.7.html) language to filter out the packets. There are plans to potentially move away from this method and accept base64-encoded `bpf` bytecode in the future.
BPF is by far the most performant way to filter incoming packets. It's only supported on live capture (`--devName`). It uses the `tcpdump`'s [pcap-filter](https://www.tcpdump.org/manpages/pcap-filter.7.html) language to filter out the packets.

## Sample Ratio
{{< alert >}}Applied at capture level{{< /alert >}}
Expand Down
32 changes: 32 additions & 0 deletions docs/content/en/docs/Outputs/parquet.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
---
title: "Parquet"
linkTitle: "Apache Parquet"
weight: 4
---

Parquet output module is designed to send `dnsmonster` logs to parquet files.


## Configuration Parameters
```ini
[parquet_output]
; What should be written to parquet file. options:
; 0: Disable Output
; 1: Enable Output without any filters
; 2: Enable Output and apply skipdomains logic
; 3: Enable Output and apply allowdomains logic
; 4: Enable Output and apply both skip and allow domains logic
parquetoutputtype = 0

; Path to output folder. Used if parquetoutputtype is not none
parquetoutputpath =

; Number of records to write to parquet file before flushing
parquetflushbatchsize = 10000

; Number of workers to write to parquet file
parquetworkercount = 4

; Size of the write buffer in bytes
parquetwritebuffersize = 256000
```
6 changes: 4 additions & 2 deletions docs/content/en/docs/Outputs/stdout-file-syslog.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@ linkTitle: "Stdout, syslog, or Log File"
weight: 4
---

Stdout, syslog and file are supported outputs for `dnsmonster` out of the box. They are useful specially if you have a SIEM agent reading the files as they come in. Note that `dnsmonster` does not provide support for log rotation and the capacity of the hard drive while writing into a file. You can use a tool like `logrotate` to perform cleanups on the log files. The signalling on log rotation (SIGHUP) has not been tested with `dnsmonster`.
Stdout, syslog and file are supported outputs for `dnsmonster` out of the box. They are useful specially if you have a SIEM agent reading the files as they come in. Note that `dnsmonster` does not provide support for log rotation and the capacity of the hard drive while writing into a file. You can use a tool like `logrotate` to perform cleanups on the log files. The signalling on log rotation (SIGHUP) has not been tested with `dnsmonster`.

The JSON schema used to send the logs can be configured to be compatible with Open Cybersecurity Schema Framework (OCSF) as well.

Currently, Syslog output is only supported on Linux.

Expand All @@ -23,7 +25,7 @@ FileOutputType = 0
; Path to output file. Used if fileOutputType is not none
FileOutputPath =

; Output format for file. options:json,csv, csv_no_header, gotemplate. note that the csv splits the datetime format into multiple fields
; Output format for file. options:json, json-ocsf, csv, csv_no_header, gotemplate. note that the csv splits the datetime format into multiple fields
FileOutputFormat = json

; Go Template to format the output as needed
Expand Down
35 changes: 35 additions & 0 deletions docs/content/en/docs/Outputs/victorialogs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
---
title: "VictoriaLogs"
linkTitle: "VictoriaLogs"
weight: 4
---

VictoriaLogs output module is designed to send `dnsmonster` logs to [victorialogs](https://docs.victoriametrics.com/victorialogs/index.html).


## Configuration Parameters
```ini
[victoria_output]
; Victoria Output Endpoint. example: http://localhost:9428/insert/jsonline?_msg_field=rcode_id&_time_field=time
victoriaoutputendpoint =

; What should be written to Microsoft Victoria. options:
; 0: Disable Output
; 1: Enable Output without any filters
; 2: Enable Output and apply skipdomains logic
; 3: Enable Output and apply allowdomains logic
; 4: Enable Output and apply both skip and allow domains logic
victoriaoutputtype = 0

; Victoria Output Proxy in URI format
victoriaoutputproxy =

; Number of workers
victoriaoutputworkers = 8

; Victoria Batch Size
victoriabatchsize = 100

; Interval between sending results to Victoria if Batch size is not filled. Any value larger than zero takes precedence over Batch Size
victoriabatchdelay = 0s
```
42 changes: 42 additions & 0 deletions docs/content/en/docs/Outputs/zinc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
---
title: "Zinc Search"
linkTitle: "Zinc Search"
weight: 4
---

Zinc Search output module is designed to send `dnsmonster` logs to [zincsearch](https://github.com/zincsearch/zincsearch).


## Configuration Parameters
```ini

[zinc_output]
; What should be written to zinc. options:
; 0: Disable Output
; 1: Enable Output without any filters
; 2: Enable Output and apply skipdomains logic
; 3: Enable Output and apply allowdomains logic
; 4: Enable Output and apply both skip and allow domains logic
zincoutputtype = 0

; index used to save data in Zinc
zincoutputindex = dnsmonster

; zinc endpoint address, example: http://127.0.0.1:9200/api/default/_bulk. Used if zincOutputType is not none
zincoutputendpoint =

; zinc username, example: [email protected]. Used if zincOutputType is not none
zincoutputusername =

; zinc password, example: password. Used if zincOutputType is not none
zincoutputpassword =

; Send data to Zinc in batch sizes
zincbatchsize = 1000

; Interval between sending results to Zinc if Batch size is not filled
zincbatchdelay = 1s

; Zing request timeout
zinctimeout = 10s
```
39 changes: 19 additions & 20 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,16 @@ require (
github.com/arthurkiller/rollingwriter v1.1.3
github.com/deathowl/go-metrics-prometheus v0.0.0-20221009205350-f2a1482ba35b
github.com/golang-collections/collections v0.0.0-20130729185459-604e922904d3
github.com/gopacket/gopacket v1.3.0
github.com/gopacket/gopacket v1.3.1
github.com/hashicorp/go-syslog v1.0.0
github.com/influxdata/influxdb-client-go/v2 v2.14.0
github.com/jackc/pgx/v4 v4.18.3
github.com/parquet-go/parquet-go v0.23.0
github.com/packetcap/go-pcap v0.0.0-20240528124601-8c87ecf5dbc5
github.com/parquet-go/parquet-go v0.24.0
github.com/prometheus/client_golang v1.20.5
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475
github.com/syntaqx/go-metrics-datadog v0.1.3
golang.org/x/sync v0.8.0
golang.org/x/sync v0.10.0
)

require (
Expand All @@ -34,7 +35,7 @@ require (
github.com/fortytw2/leaktest v1.3.0 // indirect
github.com/go-faster/city v1.0.1 // indirect
github.com/go-faster/errors v0.7.1 // indirect
github.com/google/pprof v0.0.0-20241023014458-598669927662 // indirect
github.com/google/pprof v0.0.0-20241210010833-40e02aabc2ad // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/influxdata/line-protocol v0.0.0-20210922203350-b1ad95c89adf // indirect
github.com/jackc/chunkreader/v2 v2.0.1 // indirect
Expand All @@ -46,36 +47,34 @@ require (
github.com/jackc/pgtype v1.14.4 // indirect
github.com/jackc/puddle v1.3.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/klauspost/cpuid/v2 v2.2.8 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/klauspost/cpuid/v2 v2.2.9 // indirect
github.com/lib/pq v1.10.4 // indirect
github.com/mattn/go-runewidth v0.0.16 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/oapi-codegen/runtime v1.1.1 // indirect
github.com/olekukonko/tablewriter v0.0.5 // indirect
github.com/paulmach/orb v0.11.1 // indirect
github.com/pierrec/lz4/v4 v4.1.21 // indirect
github.com/pierrec/lz4/v4 v4.1.22 // indirect
github.com/prometheus/client_model v0.6.1 // indirect
github.com/prometheus/common v0.60.1 // indirect
github.com/prometheus/common v0.61.0 // indirect
github.com/prometheus/procfs v0.15.1 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/robfig/cron v1.2.0 // indirect
github.com/segmentio/asm v1.2.0 // indirect
github.com/segmentio/encoding v0.4.0 // indirect
github.com/shopspring/decimal v1.4.0 // indirect
github.com/stretchr/objx v0.5.2 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
go.opentelemetry.io/otel v1.31.0 // indirect
go.opentelemetry.io/otel/trace v1.31.0 // indirect
golang.org/x/arch v0.11.0 // indirect
golang.org/x/mod v0.21.0 // indirect
golang.org/x/text v0.19.0 // indirect
golang.org/x/tools v0.26.0 // indirect
go.opentelemetry.io/otel v1.33.0 // indirect
go.opentelemetry.io/otel/trace v1.33.0 // indirect
golang.org/x/arch v0.12.0 // indirect
golang.org/x/mod v0.22.0 // indirect
golang.org/x/text v0.21.0 // indirect
golang.org/x/tools v0.28.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)

require (
github.com/bytedance/sonic v1.12.3
github.com/bytedance/sonic v1.12.5
github.com/dnstap/golang-dnstap v0.4.0
github.com/jessevdk/go-flags v1.6.1
github.com/klauspost/compress v1.17.11 // indirect
Expand All @@ -88,8 +87,8 @@ require (
github.com/rogpeppe/fastuuid v1.2.0
github.com/segmentio/kafka-go v0.4.47
github.com/sirupsen/logrus v1.9.3
golang.org/x/crypto v0.28.0 // indirect
golang.org/x/net v0.30.0
golang.org/x/sys v0.26.0 // indirect
google.golang.org/protobuf v1.35.1
golang.org/x/crypto v0.31.0 // indirect
golang.org/x/net v0.32.0
golang.org/x/sys v0.28.0 // indirect
google.golang.org/protobuf v1.35.2
)
Loading

0 comments on commit b539808

Please sign in to comment.