Skip to content

README.md

Latest commit

 

History

History
208 lines (158 loc) · 6.09 KB

README.md

File metadata and controls

208 lines (158 loc) · 6.09 KB
Raw

OpenAI Responses API Endpoint

Complexity: 🟢 Beginner

This example demonstrates how to configure a NeMo Agent toolkit FastAPI frontend to accept requests in the OpenAI Responses API format.

Overview

The OpenAI Responses API uses a different request format than the Chat Completions API:

Feature Chat Completions API Responses API
Input field messages (array) input (string or array)
System prompt In messages array instructions field
Response object chat.completion response
Streaming events chat.completion.chunk response.created, response.output_text.delta, etc.

This example configures the /v1/responses endpoint to accept the Responses API format while the standard /generate and /chat endpoints continue using Chat Completions format.

⚠️ Important: The Responses API format is provided for pass-through compatibility with managed services that support stateful backends (such as OpenAI and Azure OpenAI). NeMo Agent toolkit workflows do not inherently support stateful backends. Features like previous_response_id will be accepted but ignored.

Prerequisites

  1. Install LangChain integration (required for tool_calling_agent workflow):
uv pip install -e '.[langchain]'
  1. Set up the NVIDIA API key:
export NVIDIA_API_KEY=<YOUR_API_KEY>

Start the Server

nat serve --config_file examples/front_ends/responses_api_endpoint/configs/config.yml

The server will start on port 8088 with the following endpoints:

Endpoint Format Description
/generate NAT default Standard workflow endpoint
/chat Chat Completions OpenAI Chat Completions format
/chat/stream Chat Completions Streaming Chat Completions
/v1/responses Responses API OpenAI Responses API format

Test with curl

Responses API Format (Non-Streaming)

curl -X POST http://localhost:8088/v1/responses \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o-mini",
    "input": "What time is it?"
  }'

Expected Response:

{
  "id": "resp_abc123...",
  "object": "response",
  "status": "completed",
  "model": "gpt-4o-mini",
  "output": [
    {
      "type": "message",
      "role": "assistant",
      "content": [
        {
          "type": "output_text",
          "text": "The current time is..."
        }
      ]
    }
  ]
}

Responses API Format (Streaming)

curl -X POST http://localhost:8088/v1/responses \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o-mini",
    "input": "What time is it?",
    "stream": true
  }'

Expected SSE Events:

event: response.created
data: {"type": "response.created", "response": {"id": "resp_...", "status": "in_progress"}}

event: response.output_item.added
data: {"type": "response.output_item.added", ...}

event: response.output_text.delta
data: {"type": "response.output_text.delta", "delta": "The current"}

event: response.output_text.delta
data: {"type": "response.output_text.delta", "delta": " time is..."}

event: response.done
data: {"type": "response.done", "response": {"status": "completed", ...}}

With System Instructions

curl -X POST http://localhost:8088/v1/responses \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o-mini",
    "input": "What time is it?",
    "instructions": "You are a helpful assistant. Always be concise."
  }'

With Tools

curl -X POST http://localhost:8088/v1/responses \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o-mini",
    "input": "What time is it?",
    "tools": [
      {
        "type": "function",
        "name": "current_datetime",
        "description": "Get the current date and time"
      }
    ]
  }'

Chat Completions Format (Still Works)

The /chat endpoint continues to use the Chat Completions format:

curl -X POST http://localhost:8088/chat \
  -H 'Content-Type: application/json' \
  -d '{
    "messages": [{"role": "user", "content": "What time is it?"}]
  }'

Configuration Options

Using Explicit Format Override

If you want to use the Responses API format on a custom path (not containing "responses"), use the explicit openai_api_v1_format setting:

general:
  front_end:
    _type: fastapi
    workflow:
      openai_api_v1_path: /v1/custom/endpoint
      openai_api_v1_format: responses  # Force Responses API format

Available format options:

  • auto (default): Detects based on path pattern
  • chat_completions: Force Chat Completions API format
  • responses: Force Responses API format

Limitations

  • No Stateful Backend: previous_response_id is accepted but ignored
  • No Built-in Tools: OpenAI built-in tools like code_interpreter are not executed by NAT; use the responses_api_agent workflow type for that functionality
  • Tool Format Conversion: Responses API tool definitions are converted to Chat Completions format internally

Related Examples