Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
74b7516
Added testing of filebeat version 9.2 in test-integrations-support-fi…
PavelShilin89 Oct 23, 2025
b8f7f0a
Added configuration section for Filebeat 9.2+ with filestream input t…
PavelShilin89 Oct 23, 2025
14e1de0
docs: Auto-translate documentation changes by PavelShilin89
github-actions[bot] Oct 23, 2025
8050f28
Update docs Filebeat.md and test-integrations-support-filebeat-versio…
PavelShilin89 Oct 27, 2025
8bd9df4
docs: Auto-translate documentation changes by PavelShilin89
github-actions[bot] Oct 27, 2025
05a2937
Merge branch 'master' into test/added-test-filebeat-9.2
PavelShilin89 Oct 27, 2025
9d55c08
Unified configuration for Filebeat 9.0+ (9.0, 9.1, 9.2+)
PavelShilin89 Oct 28, 2025
cfb1d80
docs: Auto-translate documentation changes by PavelShilin89
github-actions[bot] Oct 28, 2025
c3f155e
Remove outdated Chinese & Rissuan Filebeat translation
PavelShilin89 Oct 28, 2025
8278899
Revert "Remove outdated Chinese & Rissuan Filebeat translation"
PavelShilin89 Oct 28, 2025
77a21ce
Update test/clt-tests/integrations/test-integrations-support-filebeat…
PavelShilin89 Oct 28, 2025
d23e5c4
For all 9.x versions, used filestream with fingerprint disabled
PavelShilin89 Oct 28, 2025
051bf0e
Merge branch 'test/added-test-filebeat-9.2' of https://github.com/man…
PavelShilin89 Oct 28, 2025
6d1420f
Final fix test-integrations-support-filebeat-versions.rec
PavelShilin89 Oct 28, 2025
dcd7337
Merge branch 'master' into test/added-test-filebeat-9.2
PavelShilin89 Oct 28, 2025
fe1ed8e
Replaced skip_filebeat with skip_standard_test
PavelShilin89 Oct 29, 2025
ae7e17c
Merge branch 'master' into test/added-test-filebeat-9.2
PavelShilin89 Oct 29, 2025
438f269
Update test-integrations-support-filebeat-versions.rec
PavelShilin89 Oct 29, 2025
5e9dbb5
Fix for glibc 2.35+ compatibility (Ubuntu 22.04+)
PavelShilin89 Oct 29, 2025
d325bdc
docs: Auto-translate documentation changes by PavelShilin89
github-actions[bot] Oct 29, 2025
d62436a
Updated output
PavelShilin89 Oct 29, 2025
908ed6b
Merge branch 'test/added-test-filebeat-9.2' of https://github.com/man…
PavelShilin89 Oct 29, 2025
1c310c7
Merge branch 'master' into test/added-test-filebeat-9.2
PavelShilin89 Oct 29, 2025
0b71d48
Updated conflict-handling-verification.rec
PavelShilin89 Oct 28, 2025
c237bdf
Deleted test-integrations-support-filebeat-versions.rec
PavelShilin89 Oct 29, 2025
1499b19
Merge branch 'master' into test/added-test-filebeat-9.2
PavelShilin89 Oct 29, 2025
682e9e6
docs: Auto-translate documentation changes by PavelShilin89
github-actions[bot] Oct 29, 2025
7b8a6d8
Merge branch 'master' into test/added-test-filebeat-9.2
PavelShilin89 Oct 31, 2025
82d024e
Merge branch 'master' into test/added-test-filebeat-9.2
PavelShilin89 Nov 3, 2025
fbf5b30
Update translator
PavelShilin89 Nov 3, 2025
7000ddb
Documentation files for calling the auto-translator have been deleted.
PavelShilin89 Nov 3, 2025
d3c705f
Merge branch 'master' into test/added-test-filebeat-9.2
PavelShilin89 Nov 3, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 36 additions & 8 deletions manual/chinese/Integration/Filebeat.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

[Filebeat](https://www.elastic.co/beats/filebeat) 是一个轻量级的转发器,用于转发和集中日志数据。它作为代理安装后,会监控您指定的日志文件或位置,收集日志事件,并将其转发进行索引,通常是发送到 Elasticsearch 或 Logstash。

现在,Manticore 也支持将 Filebeat 用作处理管道。这允许收集和转换后的数据像发送到 Elasticsearch 一样发送到 Manticore。目前,所有版本至 9.0 都得到全面支持
现在,Manticore 也支持将 Filebeat 用作处理流程。这允许将收集和转换后的数据像发送到 Elasticsearch 一样发送到 Manticore。目前,所有版本直到 9.2 都完全支持

## Filebeat 配置

Expand All @@ -19,7 +19,7 @@ filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/dpkg.log
- /var/log/dpkg.log
close_eof: true
scan_frequency: 1s

Expand All @@ -43,7 +43,7 @@ filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/dpkg.log
- /var/log/dpkg.log
close_eof: true
scan_frequency: 1s

Expand All @@ -59,7 +59,7 @@ setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
```

### Filebeat 8.11 - 8.18 的配置
### Filebeat 8.11 - 8.19 的配置

从 8.11 版本开始,默认启用了输出压缩,因此必须明确设置 `compression_level: 0` 以确保与 Manticore 的兼容性:

Expand All @@ -68,7 +68,7 @@ filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/dpkg.log
- /var/log/dpkg.log
close_eof: true
scan_frequency: 1s

Expand All @@ -84,7 +84,7 @@ setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
```

### Filebeat 9.0 的配置
### Filebeat 9.0 - 9.1 的配置

Filebeat 9.0 引入了重大架构变化,替换了日志输入类型为 filestream。以下是所需的配置:

Expand All @@ -94,7 +94,7 @@ filebeat.inputs:
id: dpkg-log-input
enabled: true
paths:
- /var/log/dpkg.log
- /var/log/dpkg.log
prospector.scanner.check_interval: 1s
close.on_eof: true

Expand All @@ -110,10 +110,38 @@ setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
```

### Filebeat 9.2+ 的配置

从 Filebeat 9.2 开始,指纹功能默认启用,要求文件至少有 1024 字节才能被摄取。对于较小的文件,您需要禁用指纹:

```
filebeat.inputs:
- type: filestream
id: dpkg-log-input
enabled: true
paths:
- /var/log/dpkg.log
prospector.scanner.check_interval: 1s
prospector.scanner.fingerprint.enabled: false
close.on_state_change.inactive: 1s

output.elasticsearch:
hosts: ["http://localhost:9308"]
index: "dpkg_log"
compression_level: 0
allow_older_versions: true

setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
```

**注意:** 设置 `prospector.scanner.fingerprint.enabled: false` 允许 Filebeat 处理任意大小的文件。如果您正在处理较大的文件(>1024 字节),可以省略此选项或调整 `prospector.scanner.fingerprint.length` 和 `prospector.scanner.fingerprint.offset` 以满足您的需求。

## Filebeat 结果

运行此配置的 Filebeat 后,日志数据将被发送到 Manticore 并正确建立索引。以下是 Manticore 创建的表的结果模式以及插入文档的示例:

```
mysql> DESCRIBE dpkg_log;
+------------------+--------+--------------------+
Expand Down
44 changes: 36 additions & 8 deletions manual/english/Integration/Filebeat.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

[Filebeat](https://www.elastic.co/beats/filebeat) is a lightweight shipper for forwarding and centralizing log data. Once installed as an agent, it monitors the log files or locations you specify, collects log events, and forwards them for indexing, usually to Elasticsearch or Logstash.

Now, Manticore also supports the use of Filebeat as processing pipelines. This allows the collected and transformed data to be sent to Manticore just like to Elasticsearch. Currently, All versions to 9.0 are fully supported.
Now, Manticore also supports the use of Filebeat as processing pipelines. This allows the collected and transformed data to be sent to Manticore just like to Elasticsearch. Currently, all versions up to 9.2 are fully supported.

## Filebeat configuration

Expand All @@ -19,7 +19,7 @@ filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/dpkg.log
- /var/log/dpkg.log
close_eof: true
scan_frequency: 1s

Expand All @@ -43,7 +43,7 @@ filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/dpkg.log
- /var/log/dpkg.log
close_eof: true
scan_frequency: 1s

Expand All @@ -59,7 +59,7 @@ setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
```

### Configuration for Filebeat 8.11 - 8.18
### Configuration for Filebeat 8.11 - 8.19

From version 8.11, output compression is enabled by default, so you must explicitly set `compression_level: 0` for compatibility with Manticore:

Expand All @@ -68,7 +68,7 @@ filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/dpkg.log
- /var/log/dpkg.log
close_eof: true
scan_frequency: 1s

Expand All @@ -84,7 +84,7 @@ setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
```

### Configuration for Filebeat 9.0
### Configuration for Filebeat 9.0 - 9.1

Filebeat 9.0 introduces a major architecture change, replacing the log input type with filestream. Here's the required configuration:

Expand All @@ -94,7 +94,7 @@ filebeat.inputs:
id: dpkg-log-input
enabled: true
paths:
- /var/log/dpkg.log
- /var/log/dpkg.log
prospector.scanner.check_interval: 1s
close.on_eof: true

Expand All @@ -110,6 +110,35 @@ setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
```

### Configuration for Filebeat 9.2+

Starting from Filebeat 9.2, the fingerprint feature is enabled by default, which requires files to be at least 1024 bytes to be ingested. For smaller files, you need to disable fingerprinting:

```
filebeat.inputs:
- type: filestream
id: dpkg-log-input
enabled: true
paths:
- /var/log/dpkg.log
prospector.scanner.check_interval: 1s
prospector.scanner.fingerprint.enabled: false
close.on_state_change.inactive: 1s

output.elasticsearch:
hosts: ["http://localhost:9308"]
index: "dpkg_log"
compression_level: 0
allow_older_versions: true

setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
```

**Note:** The `prospector.scanner.fingerprint.enabled: false` setting allows Filebeat to process files of any size. If you're working with larger files (>1024 bytes), you can omit this option or adjust `prospector.scanner.fingerprint.length` and `prospector.scanner.fingerprint.offset` to meet your needs.

## Filebeat results

Once you run Filebeat with this configuration, log data will be sent to Manticore and properly indexed. Here is the resulting schema of the table created by Manticore and an example of the inserted document:
Expand Down Expand Up @@ -142,4 +171,3 @@ host: {"name":"logstash-db848f65f-lnlf9"}
agent: {"ephemeral_id":"587c2ebc-e7e2-4e27-b772-19c611115996","id":"2e3d985b-3610-4b8b-aa3b-2e45804edd2c","name":"logstash-db848f65f-lnlf9","type":"filebeat","version":"7.10.0","hostname":"logstash-db848f65f-lnlf9"}
log: {"offset":80,"file":{"path":"/var/log/dpkg.log"}}
```

44 changes: 36 additions & 8 deletions manual/russian/Integration/Filebeat.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

[Filebeat](https://www.elastic.co/beats/filebeat) — это легкий агент для пересылки и централизованного сбора данных журналов. После установки в виде агента он следит за указанными вами файлами журналов или местоположениями, собирает события из журналов и пересылает их для индексирования, обычно в Elasticsearch или Logstash.

Теперь Manticore также поддерживает использование Filebeat в качестве обработчиков данных. Это позволяет отправлять собранные и преобразованные данные в Manticore так же, как в Elasticsearch. В настоящее время полностью поддерживаются все версии до 9.0.
Теперь Manticore также поддерживает использование Filebeat в качестве конвейеров обработки. Это позволяет отправлять собранные и преобразованные данные в Manticore так же, как и в Elasticsearch. В настоящее время полностью поддерживаются все версии до 9.2.

## Конфигурация Filebeat

Expand All @@ -19,7 +19,7 @@ filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/dpkg.log
- /var/log/dpkg.log
close_eof: true
scan_frequency: 1s

Expand All @@ -43,7 +43,7 @@ filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/dpkg.log
- /var/log/dpkg.log
close_eof: true
scan_frequency: 1s

Expand All @@ -59,7 +59,7 @@ setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
```

### Конфигурация для Filebeat 8.11 - 8.18
### Конфигурация для Filebeat 8.11 - 8.19

Начиная с версии 8.11, сжатие вывода включено по умолчанию, поэтому для совместимости с Manticore нужно явно установить `compression_level: 0`:

Expand All @@ -68,7 +68,7 @@ filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/dpkg.log
- /var/log/dpkg.log
close_eof: true
scan_frequency: 1s

Expand All @@ -84,7 +84,7 @@ setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
```

### Конфигурация для Filebeat 9.0
### Конфигурация для Filebeat 9.0 - 9.1

Filebeat 9.0 вводит значительные изменения в архитектуре, заменяя тип ввода log на filestream. Вот необходимая конфигурация:

Expand All @@ -94,7 +94,7 @@ filebeat.inputs:
id: dpkg-log-input
enabled: true
paths:
- /var/log/dpkg.log
- /var/log/dpkg.log
prospector.scanner.check_interval: 1s
close.on_eof: true

Expand All @@ -110,10 +110,38 @@ setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
```

### Конфигурация для Filebeat 9.2+

Начиная с Filebeat 9.2, функция fingerprint включена по умолчанию, что требует, чтобы файлы имели размер не менее 1024 байт для обработки. Для меньших файлов необходимо отключить fingerprinting:

```
filebeat.inputs:
- type: filestream
id: dpkg-log-input
enabled: true
paths:
- /var/log/dpkg.log
prospector.scanner.check_interval: 1s
prospector.scanner.fingerprint.enabled: false
close.on_state_change.inactive: 1s

output.elasticsearch:
hosts: ["http://localhost:9308"]
index: "dpkg_log"
compression_level: 0
allow_older_versions: true

setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "dpkg_log"
setup.template.pattern: "dpkg_log"
```

**Примечание:** Параметр `prospector.scanner.fingerprint.enabled: false` позволяет Filebeat обрабатывать файлы любого размера. Если вы работаете с файлами большего размера (>1024 байт), вы можете опустить эту опцию или настроить `prospector.scanner.fingerprint.length` и `prospector.scanner.fingerprint.offset` в соответствии с вашими требованиями.

## Результаты работы Filebeat

После запуска Filebeat с этой конфигурацией данные журналов будут отправлены в Manticore и корректно индексированы. Ниже приведена итоговая схема таблицы, созданной Manticore, и пример вставленного документа:

```
mysql> DESCRIBE dpkg_log;
+------------------+--------+--------------------+
Expand Down
Loading