[Frontend] DeepSeek-V4 native OpenAI/Anthropic/Responses API + DSML tool parser#1563
Draft
yhl-amd wants to merge 6 commits into
Draft
[Frontend] DeepSeek-V4 native OpenAI/Anthropic/Responses API + DSML tool parser#1563yhl-amd wants to merge 6 commits into
yhl-amd wants to merge 6 commits into
Conversation
Contributor
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
yhl-amd
force-pushed
the
feat/dsv4-pro-openai-anthropic-responses-api
branch
2 times, most recently
from
e884e5c to
aea46ad
Compare
Non-streaming API handlers were not cancelled when the client hung up (Starlette only cancels StreamingResponse, not plain handlers), so the engine kept generating and the sequence's KV blocks leaked until it hit max_tokens. Add a client-disconnect abort path: - engine: EngineCoreMgr.abort_request broadcasts an "abort_request" utility command; _handle_abort_request marks the seq aborted; the scheduler finishes it at the next step via the normal stop path (frees KV, emits a finished RequestOutput). Adds Sequence.aborted. - api_server: _run_nonstream_with_disconnect runs generate_async in a task and polls request.is_disconnected(); on disconnect it cancels the task, whose teardown aborts + pops the request. Wired into /v1/chat/completions and /v1/completions (return HTTP 499 on disconnect). generate_async / generate_async_multimodal / generate_async_fanout / cleanup_streaming_request abort + pop on early exit to avoid leaks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the 0.5s is_disconnected() polling loop in _run_nonstream_with_disconnect with vLLM-style event-driven cancellation: race the generator-collector task against a task that awaits the ASGI http.disconnect event (request.receive()), FIRST_COMPLETED wins. Detection is now immediate (0ms vs up to 500ms) and costs nothing while the client stays connected (no periodic wakeups). The abort path is unchanged: cancelling the collector still propagates into generate_async's finally -> abort_request + io_processor.requests.pop, freeing leaked KV. request.receive() is safe here because FastAPI parses the request body into a pydantic model before the handler runs, so there is no unread body to race against. Verified on DeepSeek-V4-Pro tp8: curl --max-time drop -> immediate "Client disconnected ... aborting request" + abort_request found=True; normal non-stream and streaming requests unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: yihonglie <hyi@amd.com>
…el except Address review of the client-disconnect abort: - Factor the disconnect race into `_race_disconnect(coro, ...)` and make `_run_nonstream_with_disconnect` a thin wrapper that collects the async-generator. This lets the fan-out path (whose `generate_async_fanout` is a coroutine returning a list, not an async generator) reuse the same cancellation machinery. - Wrap the previously-unguarded non-stream branches so an abandoned request is aborted instead of running to max_tokens: multimodal (chat), n>1 fan-out (chat, both multimodal and text), and n>1 fan-out (completions). generate_async_fanout's try/finally aborts every sibling on cancel, so a disconnect frees all n sibling seqs. - Narrow the post-cancel teardown handler from `except BaseException` to `except asyncio.CancelledError` (expected) + `except Exception` (logged), letting KeyboardInterrupt/SystemExit propagate. Verified on DeepSeek-V4-Flash tp4: plain and n=2 fan-out non-stream drops both log "Client disconnected ... aborting" with abort_request found=True for every sibling (2/2 for n=2); normal plain, n=2 fan-out, and streaming unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: yihonglie <hyi@amd.com>
yhl-amd
force-pushed
the
feat/dsv4-pro-openai-anthropic-responses-api
branch
from
aea46ad to
f03ac0e
Compare
Fixes the "Check Code Style with Black" CI check on the disconnect-abort changes (extra blank lines, single-line logger call). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: yihonglie <hyi@amd.com>
…ool parser Serve DeepSeek-V4 (Pro/Flash) so Claude Code (Anthropic /v1/messages) and Codex CLI (OpenAI Responses /v1/responses) talk to ATOM directly, with DSML tool-call parsing shared across /v1/chat/completions, /v1/messages and /v1/responses. - tool_parser: DeepSeek-V4 DSML tool-call format (<|DSML|invoke ...>) with marker-less / self-closing / direct-JSON recovery, schema-driven type coercion and key aliases; streaming + non-streaming (alongside Qwen/GLM/MiniMax). - serving_chat: reasoning-filter + tool-call streaming for /v1/chat/completions. - serving_responses (new) + /v1/responses: OpenAI Responses translation (input/tools<->chat, SSE emitter, reasoning/function_call items). Codex compat: inject a mandatory DSML tool-format instruction and normalize shell tool name/param aliases (exec/shell_exec/... -> registered exec tool; command/script -> cmd) so Codex tool calls execute instead of erroring. - /v1/messages: pass the tool schema to the parser so tool args are typed correctly (fixes "Invalid tool parameters"); streaming UTF-8 incremental detokenization (vLLM-style sliding window) so multi-byte chars (CJK, box-drawing) aren't split into U+FFFD. - reasoning / chat_encoders: DeepSeek-V4 message encoding + reasoning tags. Builds on the abort-on-disconnect path from the previous commit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
yhl-amd
force-pushed
the
feat/dsv4-pro-openai-anthropic-responses-api
branch
from
f03ac0e to
8abadef
Compare
Fixes "Check Code Style with Black" CI on the DSV4 API + DSML tool parser changes (serving_responses.py, tool_parser.py, api_server.py). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: yihonglie <hyi@amd.com>
yhl-amd
force-pushed
the
feat/dsv4-pro-openai-anthropic-responses-api
branch
from
8abadef to
999168e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Serve DeepSeek-V4 (Pro/Flash) so Claude Code (Anthropic
/v1/messages) and Codex CLI (OpenAI Responses/v1/responses) can talk to ATOM directly, with DSML tool-call parsing shared across/v1/chat/completions,/v1/messagesand/v1/responses.Changes
<｜DSML｜invoke ...>) with marker-less / self-closing / direct-JSON recovery, schema-driven type coercion and key aliases; streaming + non-streaming (alongside existing Qwen/GLM/MiniMax)./v1/chat/completions(reasoning_content deltas during think phase, tool_calls deltas)./v1/responsesendpoint: OpenAI Responses translation (input/tools ↔ chat, SSE emitter, reasoning/function_call items). Codex compat: inject a mandatory DSML tool-format instruction; normalize shell tool name/param aliases (exec/shell_exec/… → registered exec tool;command/script→cmd) so Codex tool calls execute instead of erroring.U+FFFD.Test plan
claude-local(Claude Code →/v1/messages): multi-tool agentic session, tools execute, no "Invalid tool parameters", CJK/box-drawing render cleanly.codex-local(Codex →/v1/responses): tool calls execute (no "unsupported call"/"missing field cmd"), multi-step task completes./v1/chat/completionsstreaming: reasoning + tool_calls deltas.🤖 Generated with Claude Code