[Performance] Add the support of tensor of pointer in the prefetching and loop pipelining #3634
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was
linked to
issues
Mar 10, 2025
chengjunlu
force-pushed
the
chengjun/tensorptr_prefetch
branch
2 times, most recently
from
ab40c73 to
ddd2c17
Compare
chengjunlu
changed the title
chengjunlu
force-pushed
the
chengjun/tensorptr_prefetch
branch
2 times, most recently
from
2a5b53e to
f6b90ca
Compare
|
Can you please show the performance impact in % instead of the new TFlops? |
etiotto
reviewed
Mar 20, 2025
third_party/intel/include/Dialect/TritonIntelGPU/IR/TritonIntelGPUOps.td
Outdated
Show resolved
Hide resolved
third_party/intel/lib/TritonIntelGPUToLLVM/LoadStoreOpToLLVM.cpp
Outdated
Show resolved
Hide resolved
third_party/intel/lib/TritonIntelGPUToLLVM/LoadStoreOpToLLVM.cpp
Outdated
Show resolved
Hide resolved
third_party/intel/lib/TritonIntelGPUToLLVM/LoadStoreOpToLLVM.cpp
Outdated
Show resolved
Hide resolved
third_party/intel/lib/TritonIntelGPUToLLVM/LoadStoreOpToLLVM.cpp
Outdated
Show resolved
Hide resolved
third_party/intel/lib/TritonIntelGPUToLLVM/LoadStoreOpToLLVM.cpp
Outdated
Show resolved
Hide resolved
third_party/intel/lib/TritonIntelGPUToLLVM/LoadStoreOpToLLVM.cpp
Outdated
Show resolved
Hide resolved
whitneywhtsang
force-pushed
the
chengjun/tensorptr_prefetch
branch
from
f6b90ca to
6334ed5
Compare
This was referenced Apr 24, 2025
whitneywhtsang
added a commit
that referenced
this pull request
Apr 24, 2025
Note: this change split from #3634. Signed-off-by: Whitney Tsang <[email protected]>
whitneywhtsang
added a commit
that referenced
this pull request
Apr 25, 2025
This PR limits prefetch to only dense memory, to avoid polluting the cache. Benchmark CI: https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/14647344471 No performance impact to key benchmarks. Note: this change comes partially from #3634. --------- Signed-off-by: Whitney Tsang <[email protected]>
whitneywhtsang
force-pushed
the
chengjun/tensorptr_prefetch
branch
from
3b523ab to
1f05df2
Compare
whitneywhtsang
added a commit
that referenced
this pull request
Apr 25, 2025
This PR adds a new argument `mask` to the prefetch operation. It is to prepare for handling prefetching of tensor of pointers, as loads from tensor of pointers can be masked. Note: this change comes partially from #3634. --------- Signed-off-by: Whitney Tsang <[email protected]>
whitneywhtsang
force-pushed
the
chengjun/tensorptr_prefetch
branch
2 times, most recently
from
2a0f6b8 to
4bbb4f9
Compare
whitneywhtsang
force-pushed
the
chengjun/tensorptr_prefetch
branch
2 times, most recently
from
75eb507 to
ad37157
Compare
whitneywhtsang
force-pushed
the
chengjun/tensorptr_prefetch
branch
from
24ea0fa to
08e88f3
Compare
… operation. Add a mask operand for boundary check.
Signed-off-by: Whitney Tsang <[email protected]>
Signed-off-by: Whitney Tsang <[email protected]>
Signed-off-by: Tiotto, Ettore <[email protected]>
whitneywhtsang
force-pushed
the
chengjun/tensorptr_prefetch
branch
from
08e88f3 to
e3d441a
Compare
Signed-off-by: Tiotto, Ettore <[email protected]>
Signed-off-by: Tiotto, Ettore <[email protected]>
Signed-off-by: Tiotto, Ettore <[email protected]>
etiotto
added a commit
that referenced
this pull request
May 2, 2025
…r of pointers (#4064) Loads with tensor of pointers operands that have been proven to reference memory in row major (and are contained in a scf.for loop) order are now prefetched using 2D prefetching intrinsics. Note: This PR is derived from PR #3634 --------- Signed-off-by: Tiotto, Ettore <[email protected]>
etiotto
reviewed
May 2, 2025
| @@ -427,6 +427,9 @@ struct PrefetchOpConversion | |||
| // Swap the shape to make it row major and then get the tiling | |||
| // size base on row major shape. | |||
| std::swap(tensorShape[0], tensorShape[1]); | |||
| tensorType = RankedTensorType::get( | |||
There was a problem hiding this comment.
This change will cause GEMM with A transpose to fail.
There was a problem hiding this comment.
yup, just for @chengjunlu to know what changes are not merged from original change.
There was a problem hiding this comment.
Where is the case? I'd like to check the bug.
There was a problem hiding this comment.
Likely is the one reported in https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/14777632147/job/41503097292.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the first implementation of the prefetching the memory referred by tensor of pointers.
There are no degradations and the targeted workload (gemm-tensor-of-ptr) exhibits a performance improvement close to 100% (geomean from 17.1 TFlops to 33 TFlops)