Development

[ChrisCoder9000]

Summary by CodeRabbit

Release Notes v2.11.10

• New Features

  • Introduced comprehensive system tracing and monitoring for runtime health, request tracking, and performance metrics.
  • Enhanced OAuth flow to track whether redirect_uri was explicitly provided in authorization requests.

• Chores

  • Updated project version to 2.11.10-dev.
  • Added tracing configuration options to environment settings.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: c8f81eca-be33-4d74-bd4f-ebe0d599fbad

📥 Commits

Reviewing files that changed from the base of the PR and between 06e69ca and d7e9afb.

📒 Files selected for processing (20)

• .env.example
• README.md
• pyproject.toml
• src/core/agents/core/invoke_loop.py
• src/lib/tracing/__init__.py
• src/lib/tracing/events.py
• src/lib/tracing/middleware.py
• src/lib/tracing/runtime.py
• src/lib/tracing/subscribers.py
• src/lib/tracing/tracker.py
• src/lib/tracing/workers.py
• src/services/api/app.py
• src/services/mcp/app.py
• src/services/mcp/main.py
• src/services/mcp/oauth_provider.py
• src/workers/app.py
• tests/test_mcp_oauth_redis_storage.py
• tests/test_tracing_runtime_monitor.py
• tests/test_tracing_sla_tracker.py
• tests/test_tracing_workers_and_agents.py

---

📝 Walkthrough

Walkthrough

This pull request introduces a comprehensive tracing and runtime monitoring system across the codebase. A new src/lib/tracing package is added with event models, HTTP middleware, health probing, process monitoring, and a local queue-backed tracker. The system is integrated into API, MCP, and worker services, agent invoke loops, and Celery task events. Environment variables and version are updated accordingly.

Changes

Cohort / File(s)	Summary
Versioning Updates .env.example, README.md, pyproject.toml	Version bumped from 2.11.9-dev to 2.11.10-dev. .env.example extended with new TRACE_* environment variables for configuring tracker enablement, queue sizing, latency thresholds, and monitoring intervals.
Tracing Core Package src/lib/tracing/__init__.py, src/lib/tracing/events.py, src/lib/tracing/subscribers.py, src/lib/tracing/tracker.py, src/lib/tracing/middleware.py, src/lib/tracing/runtime.py, src/lib/tracing/workers.py	New tracing infrastructure: event type/severity enums and TraceEvent dataclass; abstract TraceSubscriber base class; TraceMiddleware for HTTP/WebSocket instrumentation; LocalTraceQueue and TraceTracker with context-aware span/loop tracking and async subscriber dispatch; RuntimeMonitor with background health checks, process sampling, and unhandled exception hooks; Celery task signal handlers for latency and failure tracking.
Service Integration src/services/api/app.py, src/services/mcp/app.py, src/workers/app.py	Runtime monitoring and request tracing middleware added to FastAPI/Starlette app lifespan and middleware stacks for API, MCP, and worker services.
Agent Loop Instrumentation src/core/agents/core/invoke_loop.py	run_invoke_loop wrapped in tracer span with metadata; new _run_invoke_loop_impl helper tracks iterations and emits expensive-loop events; model invocation failures and parsing errors logged via tracer; attempt counts tracked.
OAuth Enhancement src/services/mcp/main.py, src/services/mcp/oauth_provider.py	OAuth consent flow now propagates redirect_uri_provided_explicitly flag through authorization and code issuance steps, allowing tracking of explicit vs. implicit redirect URI provision.
Test Coverage tests/test_mcp_oauth_redis_storage.py, tests/test_tracing_runtime_monitor.py, tests/test_tracing_sla_tracker.py, tests/test_tracing_workers_and_agents.py	New unit tests verify redirect_uri_provided_explicitly persistence; runtime monitor heartbeat, health checks, resource sampling, and exception handling; SLA tracker error/span/loop events and middleware integration; worker and agent loop tracing event emission.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant TraceMiddleware
    participant Tracer
    participant LocalQueue
    participant Subscriber

    Client->>TraceMiddleware: HTTP Request
    activate TraceMiddleware
    TraceMiddleware->>TraceMiddleware: Create trace_id & tenant_id
    TraceMiddleware->>Tracer: Set context (trace_id, tenant_id)
    
    alt Success (status < 500)
        TraceMiddleware->>TraceMiddleware: Check duration vs slow_request_ms
        alt Duration >= threshold
            TraceMiddleware->>Tracer: emit SLA_BREACH event
        end
    else Error (status >= 500)
        TraceMiddleware->>Tracer: emit ERROR event
    end
    
    Tracer->>LocalQueue: put_nowait(TraceEvent)
    activate LocalQueue
    LocalQueue->>Subscriber: handle(event)
    deactivate LocalQueue
    deactivate TraceMiddleware
    TraceMiddleware->>Client: Response

Loading

sequenceDiagram
    participant Agent
    participant InvokeLoop
    participant Tracer
    participant LocalQueue
    participant RuntimeMonitor
    participant Subscriber

    Agent->>InvokeLoop: run_invoke_loop()
    activate InvokeLoop
    InvokeLoop->>InvokeLoop: Build trace metadata
    InvokeLoop->>Tracer: span("agent.invoke_loop")
    activate Tracer
    InvokeLoop->>InvokeLoop: _run_invoke_loop_impl()
    
    loop Each iteration
        InvokeLoop->>InvokeLoop: Check iteration threshold
        alt Threshold met
            InvokeLoop->>Tracer: expensive_loop event
        end
    end
    
    alt Model invocation fails
        InvokeLoop->>Tracer: exception(attempt context)
    end
    
    deactivate Tracer
    Tracer->>LocalQueue: put_nowait(TraceEvent)
    activate LocalQueue
    LocalQueue->>Subscriber: dispatch_once()
    deactivate LocalQueue
    deactivate InvokeLoop
    
    par Background
        RuntimeMonitor->>RuntimeMonitor: Periodic heartbeat/health checks
        RuntimeMonitor->>Tracer: emit HEARTBEAT/HEALTH_CHECK events
        Tracer->>LocalQueue: put_nowait(TraceEvent)
    end

Loading

sequenceDiagram
    participant CeleryTask
    participant SignalHandler
    participant Tracer
    participant LocalQueue
    participant Subscriber

    CeleryTask->>SignalHandler: task_prerun signal
    activate SignalHandler
    SignalHandler->>SignalHandler: Capture task_id, brain_id
    SignalHandler->>Tracer: Set context (trace_id, tenant_id)
    SignalHandler->>Tracer: emit LATENCY started event
    deactivate SignalHandler
    
    alt Task completes successfully
        CeleryTask->>SignalHandler: task_postrun signal
        activate SignalHandler
        SignalHandler->>Tracer: Compute duration
        SignalHandler->>Tracer: emit SLA_BREACH if threshold met
        deactivate SignalHandler
    else Task fails
        CeleryTask->>SignalHandler: task_failure signal
        activate SignalHandler
        SignalHandler->>Tracer: exception(traceback, duration)
        deactivate SignalHandler
    end
    
    Tracer->>LocalQueue: put_nowait(TraceEvent)
    activate LocalQueue
    LocalQueue->>Subscriber: handle(event)
    deactivate LocalQueue

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• fix(agent base): fixed many iterations and tool calls #27: Modifies src/core/agents/core/invoke_loop.py with tracing integration; directly related to agent loop instrumentation in this PR.
• Development #44: Modifies src/core/agents/core/invoke_loop.py and adds internal helpers; shares core agent instrumentation changes with this PR.
• bump version 2.11.7-dev #58: Updates version and README badge in pyproject.toml and README.md; part of the same versioning release cycle.

Poem

🐰 Hops with glee through traces bright,
Monitoring heartbeats through the night,
Queue events bouncing, left and right,
Health checks hopping with all their might,
A tracer's warren, oh what a sight! ✨

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch development

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 Pylint (4.0.5)

src/lib/tracing/__init__.py

************* Module .pylintrc
.pylintrc:1:0: F0011: error while parsing the configuration: Source contains parsing errors: '.pylintrc'
[line 18]: 'C0103' (config-parse-error)
[
{
"type": "convention",
"module": "src.lib.tracing",
"obj": "",
"line": 1,
"column": 0,
"endLine": null,
"endColumn": null,
"path": "src/lib/tracing/init.py",
"symbol": "missing-module-docstring",
"message": "Missing module docstring",
"message-id": "C0114"
}
]

src/lib/tracing/middleware.py

************* Module .pylintrc
.pylintrc:1:0: F0011: error while parsing the configuration: Source contains parsing errors: '.pylintrc'
[line 18]: 'C0103' (config-parse-error)
[
{
"type": "convention",
"module": "src.lib.tracing.middleware",
"obj": "",
"line": 1,
"column": 0,
"endLine": null,
"endColumn": null,
"path": "src/lib/tracing/middleware.py",
"symbol": "missing-module-docstring",
"message": "Missing module docstring",
"message-id": "C0114"
},
{
"type": "convention",
"module": "src.lib.tracing.middleware",
"obj": "TraceMiddleware",
"line": 11,
"column": 0,
"endLine": 11,
"endColumn": 21,
"path": "src/lib/tracing/middleware.py",
"symbol": "missing-class-docstring",
"message": "Missing class docstring",
"message-id": "C0115"
},
{
"type": "refactor",
"module": "src.lib.tracing.middleware",
"obj": "TraceMiddleware",
"line": 11,
"column": 0,
"endLine": 11,
"endColumn": 21,
"path": "src/lib/tracing/middleware.py",
"symbol": "too-few-public-methods",
"message": "Too few public methods (1/2)",
"message-id": "R0903"
}
]

src/lib/tracing/events.py

************* Module .pylintrc
.pylintrc:1:0: F0011: error while parsing the configuration: Source contains parsing errors: '.pylintrc'
[line 18]: 'C0103' (config-parse-error)
[
{
"type": "convention",
"module": "src.lib.tracing.events",
"obj": "",
"line": 1,
"column": 0,
"endLine": null,
"endColumn": null,
"path": "src/lib/tracing/events.py",
"symbol": "missing-module-docstring",
"message": "Missing module docstring",
"message-id": "C0114"
},
{
"type": "convention",
"module": "src.lib.tracing.events",
"obj": "TraceEventType",
"line": 7,
"column": 0,
"endLine": 7,
"endColumn": 20,
"path": "src/lib/tracing/events.py",
"symbol": "missing-class-docstring",
"message": "Missing class docstring",
"message-id": "C0115"
},
{
"type": "convention",
"module": "src.lib.tracing.events",

... [truncated 645 characters] ...

": "C0115"
},
{
"type": "refactor",
"module": "src.lib.tracing.events",
"obj": "TraceEvent",
"line": 28,
"column": 0,
"endLine": 28,
"endColumn": 16,
"path": "src/lib/tracing/events.py",
"symbol": "too-many-instance-attributes",
"message": "Too many instance attributes (15/7)",
"message-id": "R0902"
},
{
"type": "convention",
"module": "src.lib.tracing.events",
"obj": "TraceEvent.to_dict",
"line": 45,
"column": 4,
"endLine": 45,
"endColumn": 15,
"path": "src/lib/tracing/events.py",
"symbol": "missing-function-docstring",
"message": "Missing function or method docstring",
"message-id": "C0116"
}
]

• 14 others

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}