Feature request: Support OpenAI /v1/responses (Responses API)

[luhaijun338]

Feature Type

Enhancement to existing feature

Description

OpenAI has introduced the Responses API (POST /v1/responses), a new unified interface that consolidates chat completions, function/tool calling, and built-in tools (web search, file search, code interpreter) with native stateful conversation management (previous_response_id, conversation items).

Currently {REPO} does not expose or proxy /v1/responses, so users cannot access these newer OpenAI capabilities (e.g. o-series reasoning models, built-in tools, structured outputs via text.format) through the proxy. The only workaround is to call OpenAI directly, which bypasses {REPO}'s centralized key management, load balancing, billing, rate limiting, and audit logging — defeating the purpose of running a proxy.

Use Case

OpenAI has introduced the Responses API (POST /v1/responses), a new unified interface that consolidates chat completions, function/tool calling, and built-in tools (web search, file search, code interpreter) with native stateful conversation management (previous_response_id, conversation items).

Currently {REPO} does not expose or proxy /v1/responses, so users cannot access these newer OpenAI capabilities (e.g. o-series reasoning models, built-in tools, structured outputs via text.format) through the proxy. The only workaround is to call OpenAI directly, which bypasses {REPO}'s centralized key management, load balancing, billing, rate limiting, and audit logging — defeating the purpose of running a proxy.

Additional Context

No response

Pre-submission Checklist

• I have searched existing issues to ensure this is not a duplicate

[Mirrobot-Agent]

@luhaijun338 Thanks for the well-structured feature request — proxying OpenAI's /v1/responses (Responses API) alongside the existing chat completions support is a reasonable ask, especially with the o-series models relying on it. I'll investigate how the proxy currently routes endpoints, whether /v1/responses passes through today, and what it would take to support it properly. Report back shortly.

[Mirrobot-Agent]

Initial Analysis Report

Summary: Confirmed — the proxy has no /v1/responses route (a POST today returns FastAPI's default 404); everything currently funnels through chat-completions-shaped machinery, and supporting the Responses API is a real but well-scoped feature with one important design trap (stateful previous_response_id vs. key rotation).

Issue Validation: Confirmed — verified against the codebase; not a duplicate (searched open/closed issues and commit history).

Reproducibility Assessment: Reproducible — POST /v1/responses against a running proxy returns 404; the endpoint list in README.md (API Endpoints table) confirms only /v1/chat/completions, /v1/messages, /v1/messages/count_tokens, and /v1/embeddings exist.

Root Cause / Architecture Analysis:

The gap is structural, not accidental — the entire request pipeline is chat-completions-shaped:

• src/proxy_app/main.py:869-1163 defines the only inference routes (/v1/chat/completions, /v1/messages, /v1/messages/count_tokens, /v1/embeddings). No /v1/responses.
• RotatingClient.acompletion() (src/rotator_library/client/rotating_client.py:373) is the single dispatch point; the executor calls litellm.acompletion(**kwargs) directly (src/rotator_library/client/executor.py:633,860).
• Foreign formats are handled by translation: the Anthropic compat layer (rotating_client.py:905, src/rotator_library/anthropic_compat/) converts Anthropic → OpenAI chat format → acompletion. That's the established pattern a Responses implementation would follow.

Two viable strategies:

1. Translate Responses ↔ Chat Completions (mirror of anthropic_compat/): keeps all credential rotation, quota tracking, retries, and logging working for every provider — but Responses-only features (stateful previous_response_id conversations, built-in web search / file search / code interpreter, text.format structured outputs) have no chat-completions equivalent and would be lost.
2. Pass-through via litellm.aresponses() (litellm has native Responses API support since v1.63; this repo's litellm is unpinned in requirements.txt:12): full fidelity, but requires a new execution path in the executor (currently hardwired to acompletion), and per this repo's own provider metadata (src/rotator_library/litellm_providers.py:729) only a handful of providers (openai, azure, amazon_nova) advertise Responses endpoints — so credential rotation would only apply to those.

Key design consideration — statefulness vs. key rotation: previous_response_id state lives in the OpenAI org/project that created the response. With multiple rotated OpenAI credentials, a follow-up request landing on a different account gets a 404 on the referenced response. The existing sticky-session mechanism (SessionTracker._build_fingerprints, src/rotator_library/session_tracking.py:96-119) fingerprints only explicit session keys and messages — Responses payloads (input, previous_response_id) are invisible to it, so stickiness would silently fail. Adding previous_response_id as a strong fingerprint is essentially mandatory for strategy 2, and worth doing regardless.

Secondary work items for either strategy: usage parsing (Responses returns input_tokens/output_tokens vs. prompt_tokens/completion_tokens), and streaming (Responses SSE uses different event types, e.g. response.output_text.delta, that the existing stream retry/wrapper machinery doesn't know).

Note: o-series reasoning models are already reachable today via /v1/chat/completions (litellm maps reasoning_effort etc.) — the capabilities genuinely blocked are built-in tools, text.format, and stateful conversations.

Suggested Labels: enhancement (new API surface, no existing behavior broken).

Proposed Next Steps:

1. Maintainers pick a strategy — suggested: hybrid, i.e. pass-through via litellm.aresponses() for providers that support it (OpenAI/Azure), since translation can't preserve the API's headline features.
2. Add @app.post("/v1/responses") in src/proxy_app/main.py with the standard verify_api_key dependency.
3. Add an aresponses dispatch path to RotatingClient/executor with rotation support for Responses-capable providers.
4. Add previous_response_id (and conversation id, if used) as strong fingerprints in SessionTracker for sticky-credential routing.
5. Extend usage parsing and the request/response loggers to the Responses payload shape; add SSE event handling for streamed Responses.

Missing Information (if any): Nothing blocking. Two details would help scope the first iteration: (1) do you need this for OpenAI only, or also Azure OpenAI deployments? (2) do you rely on the stateful features (previous_response_id) or mainly on built-in tools / text.format? That decides how urgent the sticky-session work is.

This analysis was generated by an AI assistant.