Skip to content

Export logs to OTEL #38607

@getvictor

Description

@getvictor

Goal

User story
As a platform engineer or SRE,
I want to see trace_id and span_id in Fleet logs
so that I can correlate logs with traces when debugging issues.

Original requests

Resources

Changes

Product

  • UI changes: No changes
  • CLI (fleetctl) usage changes: No changes
  • YAML changes: No changes
  • REST API changes: No changes
  • Fleet's agent (fleetd) changes: No changes
  • Fleet server configuration changes: Add logging.otel_logs_enabled option (FLEET_LOGGING_OTEL_LOGS_ENABLED env var) to enable OTEL log export
  • Exposed, public API endpoint changes: No changes
  • fleetdm.com changes: No changes
  • GitOps mode UI changes: No changes
  • GitOps generation changes: No changes
  • Activity changes: No changes
  • Permissions changes: No changes
  • Changes to paid features or tiers: Fleet Free
  • My device and fleetdm.com/better changes: No changes
  • Usage statistics: No changes
  • Other reference documentation changes: No changes
  • First draft of test plan added
  • Once shipped, requester has been notified
  • Once shipped, dogfooding issue has been filed

Engineering

  • Test plan is finalized
  • Contributor API changes: No changes
  • Feature guide changes: No changes
  • Database schema migrations: None
  • This is a premium only feature: No

ℹ️ Please read this issue carefully and understand it.

Implementation

POC (not using slog): #38746

  1. Add config option logging.otel_logs_enabled (FLEET_LOGGING_OTEL_LOGS_ENABLED env var) to server/config/config.go

  2. Add OtelLogsEnabled field to logging.Options in server/platform/logging/logging.go

  3. Initialize OTEL LoggerProvider in cmd/fleet/serve.go when OtelLogsEnabled is true:

    • Create OTLP log exporter (uses standard OTEL_EXPORTER_OTLP_ENDPOINT env var)
    • Create LoggerProvider with the exporter
    • Pass to logging package
  4. Create multi-handler in NewSlogLogger when OtelLogsEnabled is true:

    • Primary handler: JSON/Text handler writing to stderr (existing behavior)
    • Secondary handler: otelslog handler sending all logs to OTEL collector
    • All logs go to both destinations
  5. Trace correlation: When a log is sent to the OTEL collector, if a span is active in the context, the otelslog bridge automatically correlates the log with the trace (no extra work needed).

  6. Add logging.otel_logs_enabled server option to contributor docs.

  7. Add contributor docs how to use this.

Dependencies

  • Requires Phase 1 of ADR-0008 (slog foundation) to be completed first

QA

Risk assessment

  • Risk level: Low
  • Risk description: Changes only affect logging output when tracing is explicitly enabled. Log format is preserved for backward compatibility.

Test plan

Not needed since we have not documented OTEL support in our public docs.

Confirmation

  1. Engineer: Added comment to user story confirming successful completion of test plan.
  2. QA: Added comment to user story confirming successful completion of test plan.

Metadata

Metadata

Assignees

Labels

#g-security-complianceSecurity & Compliance product group:releaseReady to write code. Scheduled in a release. See "Making changes" in handbook.storyA user story defining an entire feature~engineering-initiatedEngineering-initiated story, such as a bug, refactor, or contributor experience improvement.

Type

No type

Projects

Status

Done

Status

Done

Status

Done

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions