WIP [linux] migrate Linux metrics data streams to TSDB

[AndersonQ]

Proposed commit message

Enable time series data streams (TSDB) for 8 of 11 data streams in the Linux integration: conntrack, entropy, iostat, ksm, memory, pageinfo, raid, and service.

For each data stream:
- Add `elasticsearch.index_mode: "time_series"` to manifest.yml
- Annotate numeric fields with appropriate metric_type (gauge/counter)
- Mark dimension fields to uniquely identify each time series

Common dimensions (all 8 data streams):
- agent.id
- agent.name
- cloud.account.id
- cloud.availability_zone
- cloud.instance.id
- cloud.provider
- cloud.region
- container.id
- host.name

Integration-specific dimensions:
- iostat: linux.iostat.name (disk device)
- raid: system.raid.name (RAID array)
- service: system.service.name (systemd service)

Excluded data streams:
- socket: transient entities with no persistent time series
- users: transient sessions with no numeric metrics
- network_summary: fields use object wildcard mappings that cannot carry metric_type annotations, limiting TSDB benefits

Assisted by Cursor

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Screenshots

[Copilot]

Pull request overview

Migrates several Linux integration metrics data streams to Elasticsearch TSDB / time_series data streams by enabling index_mode: "time_series" and annotating fields with metric_type/dimension so metrics can be stored and queried as time series efficiently.

Changes:

• Enable TSDB (elasticsearch.index_mode: "time_series") for conntrack, entropy, iostat, ksm, memory, pageinfo, raid, and service data streams.
• Mark common identifying fields (e.g., agent/cloud/container/host) as dimension: true and add stream-specific dimensions (e.g., device/service/raid name).
• Annotate numeric metric fields with metric_type (gauge/counter).

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
packages/linux/data_stream/service/manifest.yml	Enables TSDB index mode for the service metrics data stream.
packages/linux/data_stream/service/fields/fields.yml	Adds dimension for service name and metric_type for service resource metrics.
packages/linux/data_stream/service/fields/ecs.yml	Marks host.name as a TSDB dimension for service metrics.
packages/linux/data_stream/service/fields/agent.yml	Adds common TSDB dimensions (agent/cloud/container, etc.) for service metrics.
packages/linux/data_stream/raid/manifest.yml	Enables TSDB index mode for the raid metrics data stream.
packages/linux/data_stream/raid/fields/fields.yml	Marks raid name as a dimension and annotates numeric fields with metric_type.
packages/linux/data_stream/raid/fields/agent.yml	Adds common TSDB dimensions (agent/cloud/container/host.name, etc.) for raid metrics.
packages/linux/data_stream/pageinfo/manifest.yml	Enables TSDB index mode for the pageinfo metrics data stream.
packages/linux/data_stream/pageinfo/fields/fields.yml	Annotates buddyinfo numeric fields with metric_type: gauge for TSDB.
packages/linux/data_stream/pageinfo/fields/agent.yml	Adds common TSDB dimensions (agent/cloud/container/host.name, etc.) for pageinfo metrics.
packages/linux/data_stream/memory/manifest.yml	Enables TSDB index mode for the memory metrics data stream.
packages/linux/data_stream/memory/fields/fields.yml	Adds metric_type annotations across paging/swap/hugepages metrics for TSDB.
packages/linux/data_stream/memory/fields/agent.yml	Adds common TSDB dimensions (agent/cloud/container/host.name, etc.) for memory metrics.
packages/linux/data_stream/ksm/manifest.yml	Enables TSDB index mode for the ksm metrics data stream.
packages/linux/data_stream/ksm/fields/fields.yml	Annotates KSM numeric fields with metric_type for TSDB.
packages/linux/data_stream/ksm/fields/agent.yml	Adds common TSDB dimensions (agent/cloud/container/host.name, etc.) for ksm metrics.
packages/linux/data_stream/iostat/manifest.yml	Enables TSDB index mode for the iostat metrics data stream.
packages/linux/data_stream/iostat/fields/fields.yml	Marks disk device name as a dimension and annotates iostat numeric fields with metric_type.
packages/linux/data_stream/iostat/fields/agent.yml	Adds common TSDB dimensions (agent/cloud/container/host.name, etc.) for iostat metrics.
packages/linux/data_stream/entropy/manifest.yml	Enables TSDB index mode for the entropy metrics data stream.
packages/linux/data_stream/entropy/fields/fields.yml	Annotates entropy numeric fields with metric_type: gauge for TSDB.
packages/linux/data_stream/entropy/fields/agent.yml	Adds common TSDB dimensions (agent/cloud/container/host.name, etc.) for entropy metrics.
packages/linux/data_stream/conntrack/manifest.yml	Enables TSDB index mode for the conntrack metrics data stream.
packages/linux/data_stream/conntrack/fields/fields.yml	Annotates conntrack numeric fields with metric_type for TSDB.
packages/linux/data_stream/conntrack/fields/agent.yml	Adds common TSDB dimensions (agent/cloud/container/host.name, etc.) for conntrack metrics.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

system.service.resources.network.in.packets is a packet count but is still declared with format: bytes, which will cause incorrect formatting/units in Kibana and exported field docs. Remove the bytes format (or switch to a numeric format appropriate for counts).

Suggested change

format: bytes

[github-actions]

Vale Linting Results

Summary: 1 warning, 4 suggestions found

⚠️ Warnings (1)

File	Line	Rule	Message
packages/linux/docs/README.md	306	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'for example' instead of 'e.g'.

💡 Suggestions (4)

File	Line	Rule	Message
packages/linux/docs/README.md	100	Elastic.WordChoice	Consider using 'can, might' instead of 'may', unless the term is in the UI.
packages/linux/docs/README.md	214	Elastic.WordChoice	Consider using 'can, might' instead of 'may', unless the term is in the UI.
packages/linux/docs/README.md	281	Elastic.WordChoice	Consider using 'can, might' instead of 'may', unless the term is in the UI.
packages/linux/docs/README.md	331	Elastic.Wordiness	Consider using 'all' instead of 'all of '.

---

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 15d89c6

Failed CI Steps

• Check integrations linux

History

cc @AndersonQ