Add useAiGateway tool to the Helicone MCP + Claude Agent SDK example for docs

[juliettech13]

Ticket

[ENG-3643] - Claude Agent SDK Integration with AI Gateway

Context

Why are you making this change?

In order to get the Claude Agent SDK working with the Helicone AI Gateway, we had to create a new tool call for our MCP that the Claude Agent SDK can call whenever it wants to make an LLM request.

This PR creates the new MCP tool alongside its documentation and examples, as well as generate the documentation needed for the Claude Agent SDK integration.

Component/Service

What part of Helicone does this affect?

• Web (Frontend)
• Jawn (Backend)
• Worker (Proxy)
• Bifrost (Marketing)
• AI Gateway
• Helicone MCP
• Packages
• Infrastructure/Docker
• Documentation

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update
• Performance improvement
• Refactoring

Testing

• Added/updated unit tests
• Added/updated integration tests
• Tested locally
• Verified in staging environment
• E2E tests pass (if applicable)

Technical Considerations

• Database migrations included (if needed)
• API changes documented
• Breaking changes noted
• Performance impact assessed
• Security implications reviewed

Dependencies

• No external dependencies added
• Dependencies added and documented
• Environment variables added/modified

Deployment Notes

• No special deployment steps required
• Database migrations need to run
• Environment variable changes required
• Coordination with other teams needed

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
helicone	Ready	Preview	Comment	Dec 4, 2025 3:38pm
helicone-bifrost	Ready	Preview	Comment	Dec 4, 2025 3:38pm
helicone-eu	Ready	Preview	Comment	Dec 4, 2025 3:38pm

[claude]

Claude finished @juliettech13's task —— View job

---

PR Code Review Complete

Score: 7/10 - Good implementation with one critical bug that needs immediate fixing

Todo List:

• Read and analyze all changed files
• Check core implementation for bugs and security issues
• Review code quality and best practices
• Evaluate documentation and examples
• Provide specific suggestions with confidence score

---

Critical Issues Found

🚨 Bug: Incorrect Custom Properties Header Format (helicone-client.ts:236)

Problem: The current implementation incorrectly sets custom properties as a JSON string instead of individual headers.

Current Code:

if (options?.customProperties) {
  headers["Helicone-Property-CustomProperties"] = JSON.stringify(options.customProperties);
}

Fix Required:

if (options?.customProperties) {
  Object.entries(options.customProperties).forEach(([key, value]) => {
    headers[`Helicone-Property-${key}`] = String(value);
  });
}

Impact: Custom properties won't work correctly with the Helicone platform, affecting filtering and analytics.

---

Security & Performance Issues

⚠️ API Key Exposure Risk

• Location: Multiple files reference API keys in examples
• Risk: Low (examples only), but ensure production code doesn't log keys
• Recommendation: Add explicit warnings in documentation about not logging API keys

⚠️ No Input Validation

• Location: helicone-client.ts:214-259
• Issue: Missing validation for required fields (model, messages)
• Recommendation: Add input validation before making API calls

💡 Performance: Inefficient Cache Management

• Location: helicone-client.ts:85-87
• Issue: Cache clears all entries when reaching max size instead of using LRU
• Recommendation: Implement proper LRU eviction strategy

---

Code Quality Improvements

✨ Type Safety

• Good: Proper TypeScript interfaces and strong typing
• Suggestion: Consider using branded types for session IDs to prevent mixing different ID types

✨ Error Handling

• Good: Comprehensive error handling with descriptive messages
• Suggestion: Consider adding error codes for programmatic handling

✨ Documentation

• Excellent: Comprehensive documentation with working examples
• Minor: Could add JSDoc comments to exported functions

---

Architecture Review

✅ Strengths

1. Clean separation: Client logic separated from MCP server logic
2. Comprehensive examples: Multiple well-documented use cases
3. Proper async/await: Good async handling throughout
4. Schema validation: Uses Zod for parameter validation in MCP tools

🔧 Suggestions

1. Add request timeout: Network requests could hang indefinitely
2. Add retry logic: For resilience against transient failures
3. Consider rate limiting: To prevent API abuse

---

Specific Inline Suggestions

helicone-client.ts Improvements:

Add input validation:

export async function useAiGateway(
  apiKey: string,
  request: ChatCompletionRequest,
  options?: AIGatewayOptions
): Promise<any> {
  // Add validation
  if (!apiKey?.trim()) {
    throw new Error("API key is required");
  }
  if (!request?.model?.trim()) {
    throw new Error("Model is required");
  }
  if (!Array.isArray(request.messages) || request.messages.length === 0) {
    throw new Error("Messages array is required and cannot be empty");
  }
  // ... rest of function
}

Add timeout and better error handling:

const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 30000); // 30s timeout

try {
  const response = await fetch(`${HELICONE_AI_GATEWAY_BASE}/v1/chat/completions`, {
    method: "POST",
    headers,
    body: JSON.stringify(request),
    signal: controller.signal
  });
  clearTimeout(timeout);
  // ... rest of code
} catch (error) {
  clearTimeout(timeout);
  if (error.name === 'AbortError') {
    throw new Error('Request timeout - AI Gateway did not respond within 30 seconds');
  }
  // ... rest of error handling
}

index.ts Improvements:

Better parameter handling:

// In use_ai_gateway tool, add better parameter validation
const { model, messages, max_tokens, temperature, stream, sessionId, sessionName, userId, customProperties, ...otherParams } = params;

// Validate required fields
if (!model || typeof model !== 'string') {
  return {
    content: [{
      type: "text",
      text: "Error: model parameter is required and must be a string"
    }]
  };
}

if (!Array.isArray(messages) || messages.length === 0) {
  return {
    content: [{
      type: "text", 
      text: "Error: messages parameter is required and must be a non-empty array"
    }]
  };
}

---

Summary

Suggestions Summary:

1. 🚨 CRITICAL: Fix custom properties header format in helicone-client.ts:236
2. ⚠️ IMPORTANT: Add input validation for API calls
3. 💡 NICE-TO-HAVE: Add request timeouts and retry logic
4. 📝 DOCUMENTATION: Add JSDoc comments to exported functions
5. 🔧 PERFORMANCE: Improve cache eviction strategy

Overall Assessment:
This is a solid implementation that adds valuable functionality for Claude Agent SDK integration. The code is well-structured, properly documented, and includes comprehensive examples. However, the critical custom properties bug must be fixed before merging, as it will prevent a core feature from working correctly.

The documentation is excellent and the examples are very helpful for developers getting started. Once the header format is corrected, this will be a great addition to the codebase.

---

[greptile-apps]

Greptile Overview

Greptile Summary

Adds a new use_ai_gateway MCP tool to enable Claude Agent SDK integration with Helicone AI Gateway, allowing agents to make LLM requests with automatic observability tracking.

Key Changes:

• New useAiGateway function in helicone-client.ts for making AI Gateway requests
• New use_ai_gateway MCP tool in index.ts with comprehensive parameter schema
• Complete working example (basic-agent.js) demonstrating the integration
• Comprehensive documentation with integration steps and multiple use cases
• Updated READMEs with clear setup instructions

Issue Found:

• Custom properties header implementation is incorrect - uses JSON stringified value instead of separate headers per property (Helicone-Property-[Name] format)

Confidence Score: 3/5

• This PR has one critical bug that will prevent custom properties from working correctly, but otherwise implements the feature well
• Score reflects the logical error in custom properties header formatting (line 236 in helicone-client.ts) that will cause the feature to not work as documented. The implementation is otherwise solid with good error handling, comprehensive examples, and clear documentation. Once the header format is fixed, this would be safe to merge.
• helicone-mcp/src/lib/helicone-client.ts requires immediate attention to fix the custom properties header format before merging

Important Files Changed

File Analysis

Filename	Score	Overview
helicone-mcp/src/lib/helicone-client.ts	3/5	Added useAiGateway function to make LLM requests through Helicone AI Gateway. Found issue: customProperties header format is incorrect (should be Helicone-Property-[Name] per property, not JSON stringified).
helicone-mcp/src/index.ts	4/5	Added use_ai_gateway MCP tool with proper schema validation and error handling. Implementation looks solid with good parameter descriptions.
examples/helicone-mcp/basic-agent.js	5/5	Well-structured example with comprehensive error handling and helpful developer guidance. Good use of comments and console output for debugging.
docs/gateway/integrations/claude-agent-sdk.mdx	4/5	Comprehensive documentation with clear examples. Follows docs structure well with step-by-step integration guide and working examples.

Sequence Diagram

sequenceDiagram
    participant User as Developer
    participant SDK as Claude Agent SDK
    participant MCP as Helicone MCP Server
    participant Gateway as Helicone AI Gateway
    participant LLM as LLM Provider

    User->>SDK: query() with prompt
    SDK->>MCP: Connect to MCP server
    MCP-->>SDK: Available tools (use_ai_gateway, etc.)
    SDK->>SDK: Analyze prompt & decide to use use_ai_gateway
    SDK->>MCP: Call use_ai_gateway tool
    Note over SDK,MCP: Parameters: model, messages,<br/>sessionId, customProperties, etc.
    MCP->>MCP: Prepare request with<br/>Helicone headers
    MCP->>Gateway: POST /v1/chat/completions
    Note over MCP,Gateway: Headers: Authorization,<br/>Helicone-Session-Id,<br/>Helicone-Property-*
    Gateway->>Gateway: Log request metadata<br/>for observability
    Gateway->>LLM: Forward request to provider
    LLM-->>Gateway: Response
    Gateway->>Gateway: Log response
    Gateway-->>MCP: Chat completion response
    MCP-->>SDK: Tool result
    SDK-->>User: Final result with response

Loading

[greptile-apps]

_{9 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}