LangSmith Tracing in Aria Chatbot - Complete Guide

Current Status

✅ LangSmith tracing is working properly. Every chat interaction from Streamlit is being logged to LangSmith dashboard.

Part 1: How LangSmith Works in This App

Data Flow Diagram

User Input (Streamlit Frontend)
        ↓
frontend.py: chatbot_stream_with_tracing()
        ↓
langgraph_backend.py: Opens LangSmith trace context
        ↓
chatbot.stream() executes within trace
        ↓
LLM call (Gemini 2.5) → Tool calls → Response
        ↓
Metrics collected (response time, tools used, message length)
        ↓
Trace finalized & sent to LangSmith API
        ↓
LangSmith Dashboard: Visible as "Chat Turn" trace

Step-by-Step Execution Flow

User sends message in Streamlit frontend
- User types in chat input box
- Frontend collects: user_input, thread_id, CONFIG

Frontend invokes backend via chatbot_stream_with_tracing()

for message_chunk, _ in chatbot_stream_with_tracing(
    {"messages": [HumanMessage(content=user_input)]},
    config=CONFIG,  # Contains thread_id for persistence
    stream_mode="messages",
):
    # Stream yields tokens/chunks to display

Backend creates LangSmith trace span
- Uses @traceable decorator from langsmith library
- Span name: "Chat Turn"
- Tags: ["chat", "stream"]
- Automatically captures this function's inputs and outputs
LangGraph chatbot executes inside trace
- Graph flow: START → chat_node → (tools?) → END
- chat_node: Calls Gemini LLM with system + user messages
- LLM decides if tool calls needed
- Tool executions happen in tool_node
- All steps recorded by LangSmith automatic tracing
Response accumulated during streaming
- Backend collects AI response text
- Tracks which tools were called
- Records response length and metadata
Trace automatically closed & sent to LangSmith
- @traceable decorator handles span lifecycle
- Inputs logged: user_message, thread_id, message_count
- Outputs logged: response_length, tools_used, tool_count
Metrics logged separately
- log_full_chat_interaction() records: response time, tools, message sizes
- Stored as custom metrics in LangSmith
Result visible in LangSmith Dashboard
- Appears under project name: "Advance LangGraph project"
- Grouped by trace (one per chat turn)
- Filterable by thread_id, tool usage, response time

Part 2: What You See in LangSmith Dashboard

Trace Structure Per Chat Turn

Chat Turn (span)
  ├─ Input
  │   ├─ thread_id: "uuid-here"
  │   ├─ user_message: "What is 2+2?"
  │   └─ message_count: 1
  ├─ Internal Execution
  │   ├─ Gemini LLM call → decides to use calculator
  │   ├─ Calculator tool execution → returns result
  │   └─ LLM generates final response
  └─ Output
      ├─ response_length: 247
      ├─ tools_used: ["calculator"]
      └─ tool_count: 1

Key Metrics Captured

Thread ID: Links to conversation history in chatbot DB
User Input Length: Helps analyze message complexity
Response Length: Tracks model verbosity
Tools Used: Which tools the LLM chose to invoke
Tool Count: How many tool calls in one turn
Timestamps: Exact timing of each interaction

Use Cases in LangSmith Dashboard

Monitor Performance
- Average response time per thread
- Pattern: which tools are most used
- LLM routing behavior (when it calls tools vs. answers directly)
Debug Issues
- Trace tool call failures
- See exact prompts sent to LLM
- Identify slow operations
Analyze User Behavior
- Most common questions
- Which documents are accessed via RAG
- User pattern analysis
Cost Analysis
- Track token usage per interaction
- Identify heavy-use features
- Estimate API costs

Part 3: Architecture - Key Components

Backend Configuration (`langgraph_backend.py`)

Environment Setup (lines 1-17):

# Loads before LangChain imports
load_dotenv(override=True)
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_PROJECT"] = "Advance LangGraph project"

LangSmith Client (lines 50-58):

ls_client = Client()  # Auto-uses env vars

Tracing Wrapper (lines 600-650):

@traceable(name="Chat Turn", tags=["chat", "stream"])
def traced_stream():
    # All code here is automatically traced

Connection Test (lines 681-724):

def test_langsmith_connection():
    # Validates API key, project, endpoint
    # Returns True if connection works

Frontend Integration (`frontend.py`)

Configuration for each chat turn (lines 320-325):

CONFIG = {
    "configurable": {"thread_id": thread_key},
    "metadata": {"thread_id": thread_key},
    "run_name": "chat_turn",
}

Stream invocation (lines 330-340):

for message_chunk, _ in chatbot_stream_with_tracing(
    {"messages": [HumanMessage(content=user_input)]},
    config=CONFIG,
    stream_mode="messages",
):
    # Yields chunks for UI display

Status display:

Shows which tool is executing
Displays when tools complete

Part 4: Maintenance & Future Changes Guide

When Updating Backend

If you modify the LLM model or parameters:

Ensure config dict still flows through chatbot.stream(config=config)
The thread_id in config must persist for conversation continuity
LangSmith will auto-capture the change (model name appears in traces)

If you add new tools:

Tools automatically appear in traces as tool_calls
No changes needed to tracing code
Tool execution metrics already logged in log_tool_usage()

If you change the chatbot graph structure:

Tracing still works because it wraps the entire chatbot.stream() call
New nodes automatically traced by LangSmith's built-in graph tracing
Update chat_node() logging if you rename nodes

If you add new LLMs or chains:

Wrap with @traceable decorator:

from langsmith import traceable

@traceable(name="My Feature", tags=["custom"])
def my_chain():
    # Your code here

When Updating Frontend

If you change how messages are sent:

Ensure user_input is captured before calling chatbot_stream_with_tracing()
Keep CONFIG with thread_id intact
Stream mode should remain "messages" for token-level visibility

If you add new UI features that call backend:

Wrap backend calls in the tracing function
Or create new @traceable function for that feature
Always pass thread_id in config for continuity

If you modify the chat interface:

The tracing happens server-side (backend), so UI changes don't affect traces
Streaming display improvements won't break tracing

Extending Tracing to New Features

Example: Add PDF upload tracking

from langsmith import traceable

@traceable(name="PDF Upload", tags=["document", "ingestion"])
def ingest_pdf(file_bytes, thread_id, filename):
    # Existing code...
    # Automatically traced with inputs/outputs

Example: Add custom metric

log_to_langsmith("custom_metric", {
    "thread_id": thread_id,
    "feature": "my_feature",
    "value": some_value
}, tags={"feature": "my_feature"})

Part 5: Environment Configuration

Required Environment Variables

# Option A: Use LANGSMITH_* (Preferred)
LANGSMITH_API_KEY=<your-api-key-from-langsmith.com>
LANGSMITH_TRACING=true
LANGSMITH_PROJECT=Advance LangGraph project
LANGSMITH_ENDPOINT=https://api.smith.langchain.com

# Option B: Use LANGCHAIN_* (Legacy, still works)
LANGCHAIN_API_KEY=<your-api-key>
LANGCHAIN_TRACING_V2=true
LANGCHAIN_PROJECT=Advance LangGraph project
LANGCHAIN_ENDPOINT=https://api.smith.langchain.com

How to Get API Key

Go to https://smith.langchain.com
Sign up or log in
Navigate to Settings → API Keys
Copy your API key
Add to .env file in project root

Verification

Run frontend and check status box:

✅ Green = "LangSmith Tracing Active" → Working
⚠️ Yellow = "LangSmith Not Configured" → Check .env

Part 6: Current Implementation Details

What Fixed the Issue

Before: Relied on automatic tracing via env vars only

❌ Chatbot.stream() wasn't being captured
❌ Only test_connection traces appeared

After: Explicit tracing with @traceable decorator

✅ Stream execution wrapped in traceable function
✅ Inputs/outputs explicitly captured
✅ All metadata logged

Key Files & Lines

Component	File	Lines	Purpose
Env Setup	langgraph_backend.py	1-17	Load & configure LangSmith
LangSmith Client	langgraph_backend.py	50-58	Initialize API connection
Metrics Logging	langgraph_backend.py	60-130	Custom metric functions
Chat Node	langgraph_backend.py	370-420	LLM invocation logging
Tracing Wrapper	langgraph_backend.py	600-650	Main trace span creation
Test Connection	langgraph_backend.py	681-724	Validate LangSmith setup
Frontend Config	frontend.py	320-325	Pass thread_id to backend
Stream Invocation	frontend.py	330-340	Call traced function

Part 7: Troubleshooting & Common Issues

Issue: "LangSmith Connection Error: Client.init() got unexpected keyword argument"

Cause: Trying to pass endpoint parameter to Client()

# ❌ Wrong
Client(api_key=key, endpoint=endpoint)

# ✅ Correct
Client(api_key=key)  # Endpoint from LANGCHAIN_ENDPOINT env var

Issue: No traces appearing in LangSmith

Checklist:

✅ API key in .env file
✅ LANGCHAIN_TRACING_V2=true set
✅ Project name matches in LangSmith
✅ Backend server running (check terminal for print statements)
✅ Frontend calling chatbot_stream_with_tracing() not just chatbot.stream()

Issue: Traces incomplete or truncated

Check:

LangSmith trace finalization: run_tree.end() is called
No exceptions in backend terminal
Response not too large (might hit API limits)

Part 8: Future Development Checklist

When adding new features that should be traced:

Identify if feature calls LLM or tools
Add @traceable decorator with descriptive name
Include relevant tags for filtering
Log inputs (what user requested)
Log outputs (what system provided)
Log metadata (timing, counts, IDs)
Test in LangSmith dashboard
Update this documentation

Summary

LangSmith is now integrated as the observability layer for Aria chatbot:

Real-time monitoring: Every chat turn is traced
Debugging: See exact LLM prompts, tool calls, and responses
Analytics: Track usage patterns, performance, costs
Scalability: Ready for multi-user, production deployment

For questions or updates, refer to this document.

FilesExpand file tree

LANGSMITH_traceing.md

Latest commit

History