feat: optimizations for dense models on ROCM/AMD #204
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
DorianZi
changed the title
LLLLKKKK
requested changes
Oct 10, 2025
LLLLKKKK
reviewed
Oct 10, 2025
LLLLKKKK
requested changes
Oct 10, 2025
amd-yilizhao
force-pushed
the
develop/qwen3-rocm-main_more_opt
branch
from
e4c77f2 to
688ee2d
Compare
LLLLKKKK
requested changes
Oct 13, 2025
|
需要增加 smoke 测试 |
DorianZi
force-pushed
the
develop/qwen3-rocm-main_more_opt
branch
4 times, most recently
from
4544c0e to
eea02b6
Compare
|
LLLLKKKK
requested changes
Oct 15, 2025
| } | ||
|
|
||
| #ifdef ENABLE_FP8 | ||
| #if defined(ENABLE_FP8) || defined(USING_ROCM) |
There was a problem hiding this comment.
这里简单处理一下，直接用枚举，就可以不用看到 cuda 或者 rocm 的头文件了。简单一点。
There was a problem hiding this comment.
不要用 __nv_fp8_e4m3 这样的 nv 或者 rocm 类型就可以了。去掉新加的 DeviceTypes.h
There was a problem hiding this comment.
”#if defined(ENABLE_FP8) || defined(USING_ROCM)“ 这个已经去掉了。DeviceTypes.h 这个建议保留，这个文件是想把rocm和cuda的fp8, bf16等类型定义成一个类型暴露出去。而且这个定义不能放在Types.h里面，因为Type.h同时被cc文件和cu文件依赖，在ROCm这边，cc文件和cu文件使用不同的编译器，cc文件使用的gcc编译器没法找到rocm fp8相关的头文件。一个方法是可以让cc和cu都用hipcc编译器，但是这样需要改的地方很多，如果要改，建议这个分支合并后再拉专门分支改。
DorianZi
force-pushed
the
develop/qwen3-rocm-main_more_opt
branch
2 times, most recently
from
90118ef to
6d3697a
Compare
liaocz
force-pushed
the
develop/qwen3-rocm-main_more_opt
branch
3 times, most recently
from
f2d3c6c to
78652a1
Compare
DorianZi
force-pushed
the
develop/qwen3-rocm-main_more_opt
branch
from
78652a1 to
0390b98
Compare
yhalpha
reviewed
Oct 17, 2025
| kv_cache_dtype = "auto" | ||
| key_cache_reshaped = key_cache.permute(0, 1, 3, 2) | ||
| value_cache_reshaped = value_cache.permute(0, 1, 3, 2) | ||
| # key_cache_reshaped = key_cache.permute(0, 1, 3, 2) |
There was a problem hiding this comment.
当前分支相对于之前几个月前提交到main里aiter 接口有变化,
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.