Skip to content

Commit a542a2b

Browse files
committed
update recipe to use online quant for mtp
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
1 parent 7a39ead commit a542a2b

1 file changed

Lines changed: 1 addition & 3 deletions

File tree

recipes/GLM-5.md

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -184,8 +184,6 @@ Reference numbers on 8×MI355X (TP8, FP8 weights, bf16 KV cache), using the benc
184184
| 8192 | 1024 | 1 | 73 | 669 | 409 | 13.2 |
185185
| 8192 | 1024 | 16 | 645 | 5818 | 418 | 23.3 |
186186
| 8192 | 1024 | 64 | 1210 | 10853 | 483 | 51.3 |
187-
<<<<<<< HEAD
188-
=======
189187

190188
## GLM-5.2 FP8 and MXFP4 Server Recipes
191189

@@ -259,6 +257,6 @@ python -m atom.entrypoints.openai_server \
259257
--no-enable_prefix_caching \
260258
--num-speculative-tokens 3 \
261259
--method mtp \
260+
--online_quant_config '{"layer_quant_config": {"*layers.78*": "ptpc_fp8"}, "exclude_layer": ["*.gate", "*shared_head*", "*embed*"]}' \
262261
-tp $TP 2>&1 | tee server_mtp.log
263262
```
264-
>>>>>>> 7986b0aa (add mtp recipe)

0 commit comments

Comments
 (0)