feat(arcade-serve): optional pluggable telemetry library integration (#862)

sdreyer · claude · web-flow · commit f537771f296c · 2026-06-12T16:29:18.000-07:00
## Summary Adds an opt-in integration with a pluggable telemetry library so deployments that need richer observability (Sentry, request correlation, structured logging) can swap it in without forking `arcade-serve`. When the library isn't installed, behavior is identical to today. - New bridge module `libs/arcade-serve/arcade_serve/fastapi/_arcade_telemetry.py` — guarded `importlib` check, no new declared dependency. - `OTELHandler.instrument_app()` delegates global OTel tracer/meter/logger provider setup to the optional library when available, then still attaches the FastAPI / HTTPX / aiohttp / Requests auto-instrumentors against whatever providers are set. Falls back to the built-in OTLP exporter path otherwise. - `OTELHandler.__init__` accepts `service_name` / `service_version` so the resource identity matches the MCP server's configured name/version instead of the hard-coded `"worker"`. - `create_arcade_mcp()` registers the library's `CorrelationMiddleware` when present, so `X-Request-Id` is propagated through the request lifecycle. - `arcade-serve` patch → minor version bump (3.2.5 → 3.3.0). ## Why `OTELHandler` is currently a hard-coded OTLP exporter setup. Anyone who wants Sentry, request correlation, or non-OTLP destinations either monkey-patches it or runs a fork. This makes that integration a one-line install: drop a wheel into the venv and `OTELHandler` defers to it transparently. Nothing changes for the default install. ## Behavior | Scenario | Provider setup | Auto-instrumentation | Middleware | |---|---|---|---| | Default install (no telemetry lib) | `OTELHandler`'s built-in OTLP exporters (unchanged) | FastAPI / HTTPX / aiohttp / Requests | None added | | Telemetry lib installed | Library's `new_telemetry(...)` | FastAPI / HTTPX / aiohttp / Requests (against global providers) | Library's `CorrelationMiddleware` | | `otel_enable=False` | Nothing initializes | Nothing | None added | The two providers can't coexist (both set global OTel providers — last-writer-wins), so when the optional library is present it fully owns provider setup. ## Test plan - [x] `uv run pytest libs/tests/worker/` — 14 tests pass, including 7 new ones covering both code paths (library absent + simulated-present via `sys.modules` monkeypatch). - [x] `uv run pytest libs/tests/worker/ libs/tests/arcade_mcp_server/` — full sweep: 1429 passing. - [x] `mypy arcade_serve` clean. - [x] `mypy arcade_mcp_server` clean. - [x] `pre-commit` (ruff + ruff-format + guards) clean. ## Notes for reviewers - The bridge module is the **only** file that touches `arcade_telemetry`. Every other call site goes through `is_available()` / `init_providers()` / `correlation_middleware_cls()` / `shutdown()`. - No `optional-dependencies` extra is declared in `pyproject.toml` — the integration is purely runtime-detected. (An extra was attempted but breaks `uv sync` since the target library isn't published to PyPI.) 🤖 Generated with [Claude Code](https://claude.com/claude-code)  --- > [!NOTE] > **Medium Risk** > Changes global OpenTelemetry provider initialization and MCP middleware ordering when OTel and arcade-telemetry are both enabled; default installs are unchanged but observability behavior can shift for deployments that add the optional library. > > **Overview** > Adds a **runtime-detected** bridge to optional `arcade-telemetry` so richer observability (e.g. Sentry, request correlation) can be enabled by installing a wheel, with **no change** when the library is absent. > > **`OTELHandler`** now accepts **`service_name`** / **`service_version`** (MCP passes configured server identity instead of a hard-coded `"worker"`). When `arcade-telemetry` is importable, global tracer/meter/logger setup goes through **`new_telemetry`** (and optional loguru wiring); FastAPI/HTTPX/aiohttp/Requests instrumentors still attach against whatever global providers are set. If the library is missing or opts out (`None` handle), behavior stays on the **built-in OTLP exporter** path; **`shutdown`** routes to the telemetry handle when delegation was used. > > **`create_arcade_mcp`** registers **`CorrelationMiddleware`** from arcade-telemetry when OTel is enabled and the class is available, added **last** so `X-Request-Id` context is set before other middleware runs. > > New **`_arcade_telemetry`** module isolates all `importlib` touches; **`arcade-serve`** bumps **3.2.5 → 3.3.0**. Tests cover delegate vs built-in paths and MCP middleware gating. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit e753a00. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>  --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
diff --git a/libs/arcade-mcp-server/arcade_mcp_server/worker.py b/libs/arcade-mcp-server/arcade_mcp_server/worker.py
@@ -18,6 +18,7 @@
 from arcade_core.discovery import discover_tools
 from arcade_core.toolkit import ToolkitLoadError
 from arcade_serve.fastapi import FastAPIWorker, TaskTrackerMiddleware
+from arcade_serve.fastapi import _arcade_telemetry as _arcade_telemetry_bridge
 from arcade_serve.fastapi.telemetry import OTELHandler
 from fastapi import FastAPI
 from loguru import logger
@@ -159,6 +160,8 @@ def create_arcade_mcp(
     otel_handler = OTELHandler(
         enable=otel_enable,
         log_level=logging.DEBUG if debug else logging.INFO,
+        service_name=mcp_settings.server.name,
+        service_version=mcp_settings.server.version,
     )
 
     @asynccontextmanager
@@ -208,6 +211,16 @@ async def track_tasks_middleware(
 
     app.add_middleware(AddTrailingSlashToPathMiddleware)
 
+    # Register CorrelationMiddleware last so it lands outermost in the
+    # Starlette stack — `add_middleware` order is innermost-first, so the
+    # final call wraps everything else. This ensures every other middleware
+    # (task tracker, trailing-slash, auth) runs with the X-Request-Id
+    # contextvar already populated.
+    if otel_enable:
+        _correlation_mw = _arcade_telemetry_bridge.correlation_middleware_cls()
+        if _correlation_mw is not None:
+            app.add_middleware(_correlation_mw)  # type: ignore[arg-type]
+
     # Add OAuth discovery endpoint if auth is enabled
     if resource_server_validator and resource_server_validator.supports_oauth_discovery():
         canonical_url = getattr(resource_server_validator, "canonical_url", None)
diff --git a/libs/arcade-serve/arcade_serve/fastapi/_arcade_telemetry.py b/libs/arcade-serve/arcade_serve/fastapi/_arcade_telemetry.py
@@ -0,0 +1,93 @@
+"""Optional bridge to the ``arcade-telemetry`` library.
+
+When ``arcade-telemetry`` is installed alongside ``arcade-serve``, the OTel
+tracer/meter/logger providers are set up by ``arcade_telemetry.new_telemetry``
+instead of ``OTELHandler``'s built-in OTLP exporters. ``OTELHandler`` still
+attaches the FastAPI / HTTPX / aiohttp / Requests auto-instrumentors against
+whatever the global providers happen to be, so spans flow through the
+arcade-telemetry pipeline.
+
+When ``arcade-telemetry`` is not installed, every helper here is a no-op and
+``OTELHandler`` runs its original code path.
+"""
+
+from __future__ import annotations
+
+import importlib
+from typing import Any
+
+_TELEMETRY_MODULE = "arcade_telemetry"
+_STARLETTE_MODULE = "arcade_telemetry.starlette"
+
+
+def _try_import(module_name: str) -> Any | None:
+    try:
+        return importlib.import_module(module_name)
+    except ImportError:
+        return None
+
+
+def is_available() -> bool:
+    """Return True iff ``arcade_telemetry`` can be imported."""
+    return _try_import(_TELEMETRY_MODULE) is not None
+
+
+def init_providers(
+    *,
+    service_name: str,
+    environment: str,
+    version: str,
+    log_level: int,
+) -> Any | None:
+    """Initialize global OTel providers via arcade-telemetry, and wire the
+    loguru → OTLP bridge if ``arcade_telemetry.loguru`` is importable.
+
+    Returns the arcade-telemetry ``Telemetry`` handle on success, or ``None``
+    if arcade-telemetry is not installed. Callers should treat ``None`` as "do
+    the OTELHandler in-house setup instead". The return type is intentionally
+    ``Any`` so this module doesn't pull arcade-telemetry types into the
+    public arcade-serve API.
+    """
+    module = _try_import(_TELEMETRY_MODULE)
+    if module is None:
+        return None
+    tel = module.new_telemetry(
+        service_name=service_name,
+        environment=environment,
+        version=version,
+        log_level=log_level,
+    )
+    if tel is None:
+        # arcade-telemetry opted out (e.g. all signals routed to NONE) — skip
+        # the loguru bridge too, otherwise OTELHandler's fallthrough to the
+        # in-house OTLP path would attach loguru AND stdlib handlers.
+        return None
+    loguru_module = _try_import(f"{_TELEMETRY_MODULE}.loguru")
+    if loguru_module is not None:
+        loguru_module.install_loguru_integration(
+            service=service_name,
+            environment=environment,
+            version=version,
+            log_level=log_level,
+        )
+    return tel
+
+
+def correlation_middleware_cls() -> type | None:
+    """Return arcade-telemetry's ASGI CorrelationMiddleware class, or None."""
+    module = _try_import(_STARLETTE_MODULE)
+    if module is None:
+        return None
+    cls = getattr(module, "CorrelationMiddleware", None)
+    if not isinstance(cls, type):
+        return None
+    return cls
+
+
+def shutdown(handle: Any | None) -> None:
+    """Best-effort shutdown of an arcade-telemetry handle."""
+    if handle is None:
+        return
+    shutdown_fn = getattr(handle, "shutdown", None)
+    if callable(shutdown_fn):
+        shutdown_fn()
diff --git a/libs/arcade-serve/arcade_serve/fastapi/telemetry.py b/libs/arcade-serve/arcade_serve/fastapi/telemetry.py
@@ -2,7 +2,7 @@
 import os
 import urllib.parse
 import warnings
-from typing import Literal, Optional
+from typing import Any, Literal, Optional
 
 # requests scans the environment for chardet at import time and emits a
 # RequestsDependencyWarning when chardet>=6 is present (e.g. pulled in by tox).
@@ -34,6 +34,8 @@
 from opentelemetry.semconv._incubating.attributes import deployment_attributes
 from opentelemetry.semconv.attributes import service_attributes
 
+from arcade_serve.fastapi import _arcade_telemetry
+
 EXCLUDED_URLS = "/worker/health"
 EXCLUDED_SPANS: list[Literal["send", "receive"]] = ["send", "receive"]
 
@@ -43,39 +45,75 @@ class ShutdownError(Exception):
 
 
 class OTELHandler:
-    def __init__(self, enable: bool = True, log_level: int = logging.INFO):
+    def __init__(
+        self,
+        enable: bool = True,
+        log_level: int = logging.INFO,
+        *,
+        service_name: str = "worker",
+        service_version: str = "",
+    ):
         self.enable = enable
         self.log_level = log_level
+        self.service_name = service_name
+        self.service_version = service_version
         self._tracer_provider: Optional[TracerProvider] = None
         self._tracer_span_exporter: Optional[OTLPSpanExporter] = None
         self._meter_provider: Optional[MeterProvider] = None
         self._meter_reader: Optional[PeriodicExportingMetricReader] = None
         self._otlp_metric_exporter: Optional[OTLPMetricExporter] = None
         self._logger_provider: Optional[LoggerProvider] = None
         self._log_processor: Optional[BatchLogRecordProcessor] = None
+        self._arcade_telemetry_handle: Optional[Any] = None
         self.environment = os.environ.get("ARCADE_ENVIRONMENT", "local")
 
     def instrument_app(self, app: FastAPI) -> None:
-        if self.enable:
-            logging.info(
-                "🔎 Initializing OpenTelemetry. Use environment variables to configure the connection"
-            )
-            self.resource = Resource(
-                attributes={
-                    service_attributes.SERVICE_NAME: "worker",
-                    deployment_attributes.DEPLOYMENT_ENVIRONMENT_NAME: self.environment,
-                }
-            )
-
-            self._init_tracer()
-            self._init_metrics()
-            self._init_logging(self.log_level)
-            FastAPIInstrumentor().instrument_app(
-                app, excluded_urls=EXCLUDED_URLS, exclude_spans=EXCLUDED_SPANS
+        if not self.enable:
+            return
+
+        if _arcade_telemetry.is_available():
+            logging.info("🔎 Initializing OpenTelemetry via arcade-telemetry")
+            handle = _arcade_telemetry.init_providers(
+                service_name=self.service_name,
+                environment=self.environment,
+                version=self.service_version,
+                log_level=self.log_level,
             )
-            HTTPXClientInstrumentor()._instrument(tracer_provider=self._tracer_provider)
-            AioHttpClientInstrumentor()._instrument(tracer_provider=self._tracer_provider)
-            RequestsInstrumentor()._instrument(tracer_provider=self._tracer_provider)
+            if handle is not None:
+                self._arcade_telemetry_handle = handle
+                FastAPIInstrumentor().instrument_app(
+                    app, excluded_urls=EXCLUDED_URLS, exclude_spans=EXCLUDED_SPANS
+                )
+                # Pass tracer_provider=None so instrumentors pick up the global
+                # provider set by arcade-telemetry.
+                HTTPXClientInstrumentor()._instrument(tracer_provider=None)
+                AioHttpClientInstrumentor()._instrument(tracer_provider=None)
+                RequestsInstrumentor()._instrument(tracer_provider=None)
+                return
+            # init_providers returned None (arcade-telemetry import race or
+            # internal opt-out) — fall through to the in-house OTLP setup so
+            # shutdown() has something to tear down.
+
+        logging.info(
+            "🔎 Initializing OpenTelemetry. Use environment variables to configure the connection"
+        )
+        resource_attrs: dict[str, str] = {
+            service_attributes.SERVICE_NAME: self.service_name,
+            deployment_attributes.DEPLOYMENT_ENVIRONMENT_NAME: self.environment,
+        }
+        if self.service_version:
+            resource_attrs[service_attributes.SERVICE_VERSION] = self.service_version
+        self.resource = Resource(attributes=resource_attrs)
+
+        self._init_tracer()
+        self._init_metrics()
+        self._init_logging(self.log_level)
+        FastAPIInstrumentor().instrument_app(
+            app, excluded_urls=EXCLUDED_URLS, exclude_spans=EXCLUDED_SPANS
+        )
+        HTTPXClientInstrumentor()._instrument(tracer_provider=self._tracer_provider)
+        AioHttpClientInstrumentor()._instrument(tracer_provider=self._tracer_provider)
+        RequestsInstrumentor()._instrument(tracer_provider=self._tracer_provider)
 
     def _init_tracer(self) -> None:
         self._tracer_provider = TracerProvider(resource=self.resource)
@@ -152,6 +190,10 @@ def _shutdown_logging(self) -> None:
         self._logger_provider.shutdown()
 
     def shutdown(self) -> None:
+        if self._arcade_telemetry_handle is not None:
+            _arcade_telemetry.shutdown(self._arcade_telemetry_handle)
+            self._arcade_telemetry_handle = None
+            return
         self._shutdown_tracer()
         self._shutdown_metrics()
         self._shutdown_logging()
diff --git a/libs/tests/worker/test_arcade_telemetry_bridge.py b/libs/tests/worker/test_arcade_telemetry_bridge.py
