rocm - rebased on top of the current main branch, nix build, changes to the rocm version of the kernel#290
Closed
alantsev wants to merge 202 commits into
Closed
rocm - rebased on top of the current main branch, nix build, changes to the rocm version of the kernel#290alantsev wants to merge 202 commits into
alantsev wants to merge 202 commits into
Conversation
Implements the Responses API endpoint that Codex CLI (and other modern OpenAI tooling) speaks instead of /v1/chat/completions. The wire format is documented in OpenAI's Responses API; this implementation has been iterated against the Codex CLI binary's SSE parser shape until no remaining schema gaps were found. Request parsing (parse_responses_request, parse_responses_input): - Accepts the typed input array (message, function_call, function_call_output, reasoning, custom_tool_call(_output), local_shell_call(_output), web_search_call(_output), tool_search_call(_output), image_generation_call(_output), compaction, context_compaction). - Maps hosted-tool history to function_call/function_call_output so prior actions survive across turns; rejects unknown item types and non-completed status with 400 to avoid silent context loss. - Strict content-array parsing: only string|null|array of recognized text blocks (input_text/output_text/text/summary_text/ reasoning_text); rejects non-text modalities (input_image/file/ audio) instead of accepting an empty prompt. - Merges adjacent function_call items into the preceding assistant message so text + tool-call turns render as a single assistant block. - Honors reasoning.effort (incl. "minimal"/"none") and gates reasoning summary surface on reasoning.summary opt-in. - Rejects previous_response_id, conversation, and forced tool_choice explicitly (constrained decoding / persisted state not supported). Output (responses_sse_*, responses_final_response): - Emits the full streaming lifecycle: response.created, output_item.added/.done, reasoning_summary_part.added/.done, reasoning_summary_text.delta/.done, content_part.added/.done, output_text.delta/.done, function_call_arguments.delta/.done, response.completed. - Branches the terminal event by finish reason: response.failed for errors and response.incomplete with reason "max_tokens" for length. - Every event carries sequence_number; every output_text part carries annotations:[]; function_call output_item.added ships with an empty arguments string (full args arrive via function_call_arguments.done and output_item.done), and item ids are stable across added/done. - Tracks whether </think> was actually observed so a truncated stream marks the reasoning item incomplete instead of "completed". - Recovers gracefully when the DSML tool parse fails after the model was suppressed at the tool marker: the suppressed tail is flushed as additional output_text deltas so the streamed message matches output_item.done. Tested by 25 rounds of /codex:adversarial-review against the same client this is meant to feed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Broaden the DS4 imatrix prompt dataset with provider-neutral agent/tool traffic, multi-language programming prompts, algorithm recall, Bash scripting, and multilingual translation tasks. Remove duplicate rendered prompts and avoid provider-specific client references in the generated calibration corpus. This improves calibration coverage without claiming to fix a distributed GGUF bug.
Fold the successful CUDA selector/top-k/indexed-attention changes into one clean commit. This excludes rejected experiment commits and the local prefill-slope work log.\n\nMeasured on GB10 with speed-bench/promessi_sposi.txt, 2048-token append chunks: 32K prefill improved from 255.61 tok/s on origin/main to 346.49 tok/s. Full-curve average improved from 316.39 tok/s to 369.76 tok/s. 32K full prompt + 128-token generation prefill improved from 312.87 tok/s to 368.43 tok/s, while generation stayed neutral at 12.49 -> 12.48 tok/s.\n\nCorrectness: make cuda-regression; ./ds4_test --logprob-vectors --tool-call-quality; ./ds4_test --server --metal-kernels.
Build score_official against the CUDA runtime on Linux and select the CUDA backend there, while keeping the existing Metal path on macOS.\n\nCorrectness: make -C gguf-tools quality-score; gguf-tools/quality-testing/score_official ds4flash.gguf /tmp/ds4_quality_smoke/manifest.tsv /tmp/ds4_quality_smoke/scores.tsv 16384.
Replace the default long-context continuation check with a deterministic prose-story retrieval test. The fixture embeds spelled-out person-number assignments in a long rendered prompt, and ds4_test now validates the generated Name=number list instead of brittle sampled prose.
Preserve Responses namespace metadata and tool_search calls while rendering DSML-safe internal tool names. Replay function_call, hosted tool, and tool_search_output items into the shared chat/tool path so Codex and Pi can round-trip tool calls without losing KV-cache prefix reuse. Document the /v1/responses endpoint and add server unit coverage for namespace, tool_search, and replay output shapes.
This reverts commit 2a7a5f3. There was no ack from the user. Don't want to take a fix that is astronautically produced from an unclear error trace.
Project sampled DSML tool calls to Anthropic SSE tool_use blocks while keeping raw DSML as the parser/cache source of truth. Reuse streamed tool ids for final parsed calls so tool_result continuation still matches live state.
Keep normal CUDA context buffers on device allocations, but route very large KV-cache tensors through managed memory so million-token contexts do not starve unified-memory systems during graph/session allocation. The fallback is scoped to the long-lived KV/cache tensors and logs when it is used because it may reduce performance. Tested on 0.180 with: - make cpu - make -B cuda-spark - make cuda-regression - ./ds4_test --server --metal-kernels - ./ds4_test --logprob-vectors --tool-call-quality - ds4-bench ctx-alloc 32768, 250000, and 1000000 - ds4-server --ctx 1000000 startup smoke (cherry picked from commit 0b248a65c07d21f2fc8ff4815bd8b75af26719f9)
Parse Anthropic tool_use blocks by their own type field instead of relying on the enclosing message role being parsed first. Some clients serialize messages as content-before-role, which made full-history tool_result replays look like unknown live-only continuations. Fixes antirez#127.
Return a 400 error with error type "context_exceeded" when prompt tokens exceed
context size. The response includes both n_prompt_tokens and n_ctx fields so
clients can determine exactly why the request failed and how far over the limit
they went.
Error response format:
{
"error": {
"message": "Prompt tokens (N) exceeds context size (M)",
"type": "context_exceeded",
"n_prompt_tokens": N,
"n_ctx": M
}
}
dwarfstar is typoed to drawfstar
fix typo in readme
(cherry picked from commit 0b3efaf86f61421330e90629508adbd6228b4a8b)
Contributor
Author
|
Thanks @fry69 , the PR you mentioned does not have the indexer for rocm platform (unless I misread it) - which is essential for the long context runs. |
Contributor
Author
|
thanks @fry69, let's keep it the way it is. |
# Conflicts: # ds4_cuda.cu
Add shared help text across the CLI, server, agent, bench, and eval tools. Expand distributed-mode guidance, clean up endpoint naming, and use a TTY-only 256-color layout with clearer section titles, option arguments, separators, examples, and explanatory text.
CPU support for Q4_K routed experts (fixes antirez#171)
``` $ ./ds4_test long-context: ds4: CUDA backend initialized on AMD Radeon 8060S Graphics (sm_115) ds4: CUDA registered 80.76 GiB model mapping for device access ds4: CUDA startup model cache prepared 80.76 GiB of tensor spans in 0.000s ds4: cuda backend initialized for graph diagnostics ds4-test: long-context prefill 0/30474 ds4-test: long-context prefill 8192/30474 ds4-test: long-context prefill 16384/30474 ds4-test: long-context prefill 24576/30474 ds4-test: long-context prefill 30474/30474 long-context: OK tool-call-quality: ds4-test: tool-call quality fast path ds4-test: tool-call quality exact path ds4: CUDA backend initialized on AMD Radeon 8060S Graphics (sm_115) ds4: CUDA registered 80.76 GiB model mapping for device access ds4: CUDA startup model cache prepared 80.76 GiB of tensor spans in 0.000s ds4: cuda backend initialized for graph diagnostics tool-call-quality: OK logprob-vectors: ds4: CUDA backend initialized on AMD Radeon 8060S Graphics (sm_115) ds4: CUDA registered 80.76 GiB model mapping for device access ds4: CUDA startup model cache prepared 80.76 GiB of tensor spans in 0.000s ds4: cuda backend initialized for graph diagnostics ds4-test: vector short_italian_fact ds4-test: vector short_code_completion ds4-test: vector short_reasoning_plain ds4-test: vector long_memory_archive skipped (API/official graph mismatch) ds4-test: vector long_code_audit logprob-vectors: OK metal-kernels: ds4: CUDA registered 0.00 GiB model mapping for device access metal-kernels: OK server: server: OK ds4 tests: ok ``` ``` $ ./ds4-eval -m ds4flash.gguf --plain --questions 12 --tokens 2048 --temp 0 --seed 1 ... PASSED got 16 expected 16 (159.8s, 1437 tokens) ds4-eval: 10/12 passed, 2 failed, runtime 00h:27m # state prompt gen total given correct test 1 PASSED 201 1661 1862 B B GPQA Diamond/recNu3MXkvWUzHZr9 2 PASSED 149 370 519 C C SuperGPQA/001b51d76b4d422988f2c11f104a2c6c 3 PASSED 81 623 704 70 70 AIME2025/aime2025-01 4 FAILED 313 2048 2361 A C GPQA Diamond/recoiTJPGUmzAkief 5 PASSED 272 2048 2320 J J SuperGPQA/b7e20eac98764fb0bf30e8366d951daa 6 PASSED 146 1325 1471 468 468 AIME2025/aime2025-16 7 PASSED 156 1303 1459 B B GPQA Diamond/rec4UqStf9WUVif1f 8 PASSED 127 280 407 E E SuperGPQA/4a1d1780a93f4093b6fb7d3c314cbea8 9 FAILED 633 2048 2681 26 588 AIME2025/aime2025-02 10 PASSED 182 1080 1262 B B GPQA Diamond/recgI6tUQ7RLJRWGx 11 PASSED 137 232 369 A A SuperGPQA/6082513c8dba4ec68aa68f1bf5854d09 12 PASSED 165 1437 1602 16 16 AIME2025/aime2025-03 ```
Contributor
Author
after this rebase the build is significantly slower than before the run is quite stable now. the recent eval run (until the first failure) the benchmark the test |
- reved unnecessary permutation in the newly introduced kernel - it did not provided necesssary guaarantees in proper order of sumation anyway - moved it out of the ds3_cuda.cu file - introduced envvar to control it's execution on rocm - reverted the build changes back to use fastmath - to avoid masking the problem (even with fast math disbled there are about 6% chance of failure for the --metal-tensor-equivalence test) the current benchmark run ``` $ ./ds4-bench -m ds4flash.gguf --prompt-file speed-bench/promessi_sposi.txt --ctx-start 2048 --ctx-max 65536 --step-incr 2048 --gen-tokens 128 ds4: CUDA backend initialized on AMD Radeon 8060S Graphics (sm_115) ds4: CUDA registered 80.76 GiB model mapping for device access ds4: CUDA preparing model tensor mappings: 80.24 GiB ds4: CUDA q8 fp16 cache limit reached; using q8 kernels (request=64.00 MiB cached=7.94 GiB limit=8.00 GiB) ds4: CUDA startup model preparation covered 80.76 GiB of tensor spans in 0.454s ds4: cuda backend initialized for graph diagnostics ds4-bench: context buffers 1742.43 MiB (ctx=65665, backend=cuda, prefill_chunk=4096, raw_kv_rows=4352, compressed_kv_rows=16418) ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,kvcache_bytes 2048,2048,66.75,128,11.75,52184460 4096,2048,65.24,128,10.11,80373132 ``` the current version failes the --logprob-vectors test (it generates "```C\nreturn" instead of expected ```c\nreturn) ``` $ ./ds4_test --logprob-vectors logprob-vectors: ds4: CUDA backend initialized on AMD Radeon 8060S Graphics (sm_115) chands4: CUDA registered 80.76 GiB model mapping for device access ds4: CUDA preparing model tensor mappings: 80.24 GiB ds4: CUDA q8 fp16 cache limit reached; using q8 kernels (request=64.00 MiB cached=7.94 GiB limit=8.00 GiB) ds4: CUDA startup model preparation covered 80.76 GiB of tensor spans in 0.315s ds4: cuda backend initialized for graph diagnostics ds4-test: vector short_italian_fact ds4-test: vector short_code_completion ds4-test: vector short_code_completion step 1 selected token mismatch tests/ds4_test.c:808: assertion failed: false ds4-test: vector short_reasoning_plain ds4-test: vector long_memory_archive skipped (API/official graph mismatch) ds4-test: vector long_code_audit logprob-vectors: ERR ds4 tests: 1 failure(s) ``` the evaluation run is the same. the ds3_cuda.cu code changes comparing to the main@upstream branch is minimal
Contributor
Author
|
Simplified the code
the ds3_cuda.cu code changes this way are minimal comparing to the main@upstream the current version with fastmath fails the --logprob-vectors test (always) |
Contributor
Author
|
note: for anyone who decide to use it you will need to restore the "-fnofast-math -fno-finite-math-only -fsigned-zeros -ffp-contract=off" flags back - without these the errors accumulate too fast |
Contributor
Author
|
I think I found the source of the error - you need to use a precise math functions (like metal does) on the critical paths - such as expert selection kernels. I will close this PR and will submit the proper one. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
this PR (next iteration over the #180)
the last change changes the order of reduction (sum) from well-defined (in cuda, from small to large I assume), to not well defined - to mitigate this I disabled fast math options at the rocm build path.
I will reenable these options later after implementing proper version of the optimised kernel.
All tests passed. The evaluation and agent flows feel comparable to the baseline.
I cannot see any obvious problems.
Probably it should not be merged to the upstream until the issues above are resolved (i.e deterministic kernel + minimal changes to the vanilla ds4_cuda.cu code etc).
However, it increases generation throughput from ~8 t/s to ~11 t/s on gfx1151, so I decided to share it with the community.