Observability

Leeway supports distributed tracing using OpenTelemetry to provide visibility into build performance and behavior.

Overview

OpenTelemetry tracing in leeway captures:

Build lifecycle (start to finish)
Individual package builds
Build phase durations (prep, pull, lint, test, build, package)
Cache hit/miss information
GitHub Actions context (when running in CI)
Parent trace context propagation from CI systems

Architecture

Span Hierarchy

Root Span (leeway.build)
├── Package Span 1 (leeway.package)
│   ├── Phase Span (leeway.phase: prep)
│   ├── Phase Span (leeway.phase: pull)
│   ├── Phase Span (leeway.phase: lint)
│   ├── Phase Span (leeway.phase: test)
│   ├── Phase Span (leeway.phase: build)
│   └── Phase Span (leeway.phase: package)
├── Package Span 2 (leeway.package)
│   ├── Phase Span (leeway.phase: prep)
│   └── Phase Span (leeway.phase: build)
└── Package Span N (leeway.package)
    └── ...

Root Span: Created when BuildStarted is called, represents the entire build operation
Package Spans: Created for each package being built, as children of the root span
Phase Spans: Created for each build phase (prep, pull, lint, test, build, package) as children of package spans

Phase spans provide detailed timeline visualization and capture individual phase errors. Only phases with commands are executed and create spans.

Context Propagation

Leeway supports W3C Trace Context propagation, allowing builds to be part of larger distributed traces:

Parent Context: Accepts traceparent and tracestate headers from upstream systems
Root Context: Creates a root span linked to the parent context
Package Context: Each package span is a child of the root span
Phase Context: Each phase span is a child of its package span

Configuration

Environment Variables

OTEL_EXPORTER_OTLP_ENDPOINT: OTLP endpoint URL (e.g., localhost:4318 or api.honeycomb.io:443)
OTEL_EXPORTER_OTLP_INSECURE: Disable TLS for OTLP endpoint (true or false, default: false)
OTEL_EXPORTER_OTLP_HEADERS: HTTP headers for OTLP requests (e.g., x-honeycomb-team=YOUR_API_KEY)
OTEL_EXPORTER_OTLP_TRACES_HEADERS: Trace-specific headers (takes precedence over OTEL_EXPORTER_OTLP_HEADERS)
TRACEPARENT: W3C Trace Context traceparent header (format: 00-{trace-id}-{span-id}-{flags})
TRACESTATE: W3C Trace Context tracestate header (optional)

Note: The OpenTelemetry SDK automatically reads OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_TRACES_HEADERS from the environment. No additional configuration is required.

CLI Flags

--otel-endpoint: OTLP endpoint URL (overrides OTEL_EXPORTER_OTLP_ENDPOINT)
--otel-insecure: Disable TLS for OTLP endpoint (overrides OTEL_EXPORTER_OTLP_INSECURE)
--trace-parent: W3C traceparent header (overrides TRACEPARENT)
--trace-state: W3C tracestate header (overrides TRACESTATE)

Precedence

CLI flags take precedence over environment variables:

CLI flag → Environment variable → Default (disabled)

TLS Configuration

By default, leeway uses secure TLS connections to the OTLP endpoint.

⚠️ Production: Always use secure TLS (default). Never set OTEL_EXPORTER_OTLP_INSECURE=true in production environments.

For local development with tools like Jaeger, you can disable TLS:

# Local development (insecure - for testing only)
export OTEL_EXPORTER_OTLP_INSECURE=true
export OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4318
leeway build :my-package

# Production (secure TLS, default)
export OTEL_EXPORTER_OTLP_ENDPOINT=api.honeycomb.io:443
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=YOUR_API_KEY"
leeway build :my-package

Span Attributes

Root Span Attributes

Attribute	Type	Description	Example
`leeway.version`	string	Leeway version	`"0.7.0"`
`leeway.workspace.root`	string	Workspace root path	`"/workspace"`
`leeway.target.package`	string	Target package being built	`"components/server:app"`
`leeway.target.version`	string	Target package version	`"abc123def"`
`leeway.packages.total`	int	Total packages in build	`42`
`leeway.packages.cached`	int	Packages cached locally	`35`
`leeway.packages.remote`	int	Packages in remote cache	`5`
`leeway.packages.downloaded`	int	Packages downloaded	`3`
`leeway.packages.to_build`	int	Packages to build	`2`

Package Span Attributes

Attribute	Type	Description	Example
`leeway.package.name`	string	Package full name	`"components/server:app"`
`leeway.package.type`	string	Package type	`"go"`, `"yarn"`, `"docker"`, `"generic"`
`leeway.package.version`	string	Package version	`"abc123def"`
`leeway.package.builddir`	string	Build directory	`"/tmp/leeway/build/..."`
`leeway.package.last_phase`	string	Last completed phase	`"build"`
`leeway.package.duration_ms`	int64	Total build duration (ms)	`15234`
`leeway.package.test.coverage_percentage`	int	Test coverage %	`85`
`leeway.package.test.functions_with_test`	int	Functions with tests	`42`
`leeway.package.test.functions_without_test`	int	Functions without tests	`8`

Phase Span Attributes

Phase spans are created for each build phase (prep, pull, lint, test, build, package) that has commands to execute.

Attribute	Type	Description	Example
`leeway.phase.name`	string	Phase name	`"prep"`, `"build"`, `"test"`, etc.

Span Status:

OK: Phase completed successfully
ERROR: Phase failed (error details in span events)

Span Duration: The span's start and end times capture the phase execution duration automatically.

GitHub Actions Attributes

When running in GitHub Actions (GITHUB_ACTIONS=true), the following attributes are added to the root span:

Attribute	Environment Variable	Description
`github.workflow`	`GITHUB_WORKFLOW`	Workflow name
`github.run_id`	`GITHUB_RUN_ID`	Unique run identifier
`github.run_number`	`GITHUB_RUN_NUMBER`	Run number
`github.job`	`GITHUB_JOB`	Job name
`github.actor`	`GITHUB_ACTOR`	User who triggered the workflow
`github.repository`	`GITHUB_REPOSITORY`	Repository name
`github.ref`	`GITHUB_REF`	Git ref
`github.sha`	`GITHUB_SHA`	Commit SHA
`github.server_url`	`GITHUB_SERVER_URL`	GitHub server URL
`github.workflow_ref`	`GITHUB_WORKFLOW_REF`	Workflow reference

Usage Examples

Basic Usage

# Set OTLP endpoint
export OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4318

# Build with tracing enabled
leeway build :my-package

With CLI Flags

leeway build :my-package \
  --otel-endpoint=localhost:4318

With Parent Trace Context

# Propagate trace context from CI system
leeway build :my-package \
  --otel-endpoint=localhost:4318 \
  --trace-parent="00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"

In GitHub Actions

name: Build
on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Build with tracing
        env:
          OTEL_EXPORTER_OTLP_ENDPOINT: ${{ secrets.OTEL_ENDPOINT }}
        run: |
          leeway build :my-package

With Jaeger (Local Development)

# Start Jaeger all-in-one
docker run -d --name jaeger \
  -p 4318:4318 \
  -p 16686:16686 \
  jaegertracing/all-in-one:latest

# Build with tracing (insecure for local development)
export OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4318
export OTEL_EXPORTER_OTLP_INSECURE=true
leeway build :my-package

# View traces at http://localhost:16686

With Honeycomb (Production)

# Configure Honeycomb endpoint with API key
export OTEL_EXPORTER_OTLP_ENDPOINT=api.honeycomb.io:443
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=YOUR_API_KEY"

# Build with tracing (secure by default)
leeway build :my-package

# View traces in Honeycomb UI

In CI/CD with Distributed Tracing

# Propagate trace context from parent CI system
export OTEL_EXPORTER_OTLP_ENDPOINT=api.honeycomb.io:443
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=YOUR_API_KEY"
export TRACEPARENT="00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"

leeway build :my-package

Error Handling

Leeway implements graceful degradation for tracing:

Tracer initialization failures: Logged as warnings, build continues without tracing
Span creation failures: Logged as warnings, build continues
OTLP endpoint unavailable: Spans are buffered and flushed on shutdown (with timeout)
Invalid trace context: Logged as warning, new trace is started

Tracing failures never cause build failures.

Performance Considerations

Overhead: Minimal (<1% in typical builds)
Concurrent builds: Thread-safe with RWMutex protection
Shutdown timeout: 5 seconds to flush pending spans
Batch export: Spans are batched for efficient export

Troubleshooting

No spans appearing in backend

Verify OTLP endpoint is reachable:
```
curl -v http://localhost:4318/v1/traces
```

Check leeway logs for warnings:

leeway build :package 2>&1 | grep -i otel

Verify environment variables:
```
echo $OTEL_EXPORTER_OTLP_ENDPOINT
```

Invalid trace context errors

Validate traceparent format:

Format: 00-{32-hex-trace-id}-{16-hex-span-id}-{2-hex-flags}
Example: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Spans not linked to parent

Ensure both traceparent and tracestate (if present) are provided:

leeway build :package \
  --trace-parent="00-..." \
  --trace-state="..."

Implementation Details

Thread Safety

Single sync.RWMutex protects packageCtxs and packageSpans maps
Safe for concurrent package builds
Read locks for lookups, write locks for modifications

Shutdown

Automatic shutdown with 5-second timeout
Registered as deferred function in getBuildOpts
Ensures all spans are flushed before exit

Testing

Tests use in-memory exporters (tracetest.NewInMemoryExporter()) to verify:

Span creation and hierarchy
Attribute correctness
Concurrent package builds
Parent context propagation
Graceful degradation with nil tracer

Future Enhancements

Potential improvements for future iterations:

Phase-level spans: Create individual spans for each build phase (prep, pull, lint, test, build, package) instead of just attributes
Span events: Add timeline events for build milestones (e.g., cache hit, dependency resolution)
Metrics integration: Export metrics alongside traces (build duration histograms, cache hit rates, concurrent build count)
Sampling configuration: Add configurable sampling strategies for high-volume builds
Additional exporters: Support for Zipkin, Jaeger native protocol, or Prometheus

Navigation Menu

Uh oh!

Observability

Overview

Architecture

Span Hierarchy

Context Propagation

Configuration

Environment Variables

CLI Flags

Precedence

TLS Configuration

Span Attributes

Root Span Attributes

Package Span Attributes

Phase Span Attributes

GitHub Actions Attributes

Usage Examples

Basic Usage

With CLI Flags

With Parent Trace Context

In GitHub Actions

With Jaeger (Local Development)

With Honeycomb (Production)

In CI/CD with Distributed Tracing

Error Handling

Performance Considerations

Troubleshooting

No spans appearing in backend

Invalid trace context errors

Spans not linked to parent

Implementation Details

Thread Safety

Shutdown

Testing

Future Enhancements

Uh oh!

FilesExpand file tree

observability.md

Latest commit

History

observability.md

File metadata and controls

Observability

Overview

Architecture

Span Hierarchy

Context Propagation

Configuration

Environment Variables

CLI Flags

Precedence

TLS Configuration

Span Attributes

Root Span Attributes

Package Span Attributes

Phase Span Attributes

GitHub Actions Attributes

Usage Examples

Basic Usage

With CLI Flags

With Parent Trace Context

In GitHub Actions

With Jaeger (Local Development)

With Honeycomb (Production)

In CI/CD with Distributed Tracing

Error Handling

Performance Considerations

Troubleshooting

No spans appearing in backend

Invalid trace context errors

Spans not linked to parent

Implementation Details

Thread Safety

Shutdown

Testing

Future Enhancements