[CIR][X86] Add support for kunpck builtins #168757
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ahmednoursphinx
requested review from
andykaylor,
bcardosolopes,
lanza and
xlauko
as code owners
llvmbot
added
clang
Member
|
@llvm/pr-subscribers-backend-x86 @llvm/pr-subscribers-clang Author: Ahmed Nour (ahmednoursphinx) ChangesPart of #167765 Full diff: https://github.com/llvm/llvm-project/pull/168757.diff 2 Files Affected:
diff --git a/clang/lib/CIR/CodeGen/CIRGenBuiltinX86.cpp b/clang/lib/CIR/CodeGen/CIRGenBuiltinX86.cpp
index ee6900141647f..cb3d50bd7a4ad 100644
--- a/clang/lib/CIR/CodeGen/CIRGenBuiltinX86.cpp
+++ b/clang/lib/CIR/CodeGen/CIRGenBuiltinX86.cpp
@@ -33,6 +33,28 @@ static mlir::Value emitIntrinsicCallOp(CIRGenFunction &cgf, const CallExpr *e,
.getResult();
}
+static mlir::Value getMaskVecValue(CIRGenBuilderTy &builder, mlir::Value mask,
+ unsigned numElems) {
+ auto maskIntType = mlir::cast<cir::IntType>(mask.getType());
+ unsigned maskWidth = maskIntType.getWidth();
+
+ // Create a vector of bool type with maskWidth elements
+ auto maskVecType =
+ cir::VectorType::get(builder.getContext(),
+ cir::BoolType::get(builder.getContext()), maskWidth);
+ mlir::Value maskVec = builder.createBitcast(mask, maskVecType);
+
+ // If we have less than 8 elements, then the starting mask was an i8 and
+ // we need to extract down to the right number of elements.
+ if (numElems < 8) {
+ llvm::SmallVector<int64_t, 4> indices;
+ for (unsigned i = 0; i != numElems; ++i)
+ indices.push_back(i);
+ maskVec = builder.createVecShuffle(mask.getLoc(), maskVec, indices);
+ }
+ return maskVec;
+}
+
// OG has unordered comparison as a form of optimization in addition to
// ordered comparison, while CIR doesn't.
//
@@ -169,6 +191,32 @@ mlir::Value CIRGenFunction::emitX86BuiltinExpr(unsigned builtinID,
case X86::BI__builtin_ia32_vec_set_v16hi:
case X86::BI__builtin_ia32_vec_set_v8si:
case X86::BI__builtin_ia32_vec_set_v4di:
+
+ case X86::BI__builtin_ia32_kunpckdi:
+ case X86::BI__builtin_ia32_kunpcksi:
+ case X86::BI__builtin_ia32_kunpckhi: {
+ auto maskIntType = mlir::cast<cir::IntType>(ops[0].getType());
+ unsigned numElems = maskIntType.getWidth();
+ mlir::Value lhs = getMaskVecValue(builder, ops[0], numElems);
+ mlir::Value rhs = getMaskVecValue(builder, ops[1], numElems);
+ llvm::SmallVector<int64_t, 64> indices;
+ for (unsigned i = 0; i != numElems; ++i)
+ indices.push_back(i);
+
+ // First extract half of each vector. This gives better codegen than
+ // doing it in a single shuffle.
+ mlir::Location loc = getLoc(expr->getExprLoc());
+ lhs = builder.createVecShuffle(
+ loc, lhs, llvm::ArrayRef(indices.data(), numElems / 2));
+ rhs = builder.createVecShuffle(
+ loc, rhs, llvm::ArrayRef(indices.data(), numElems / 2));
+ // Concat the vectors.
+ // NOTE: Operands are swapped to match the intrinsic definition.
+ mlir::Value res = builder.createVecShuffle(
+ loc, rhs, lhs, llvm::ArrayRef(indices.data(), numElems));
+ return builder.createBitcast(res, ops[0].getType());
+ }
+
case X86::BI_mm_setcsr:
case X86::BI__builtin_ia32_ldmxcsr:
case X86::BI_mm_getcsr:
@@ -675,9 +723,6 @@ mlir::Value CIRGenFunction::emitX86BuiltinExpr(unsigned builtinID,
case X86::BI__builtin_ia32_kmovw:
case X86::BI__builtin_ia32_kmovd:
case X86::BI__builtin_ia32_kmovq:
- case X86::BI__builtin_ia32_kunpckdi:
- case X86::BI__builtin_ia32_kunpcksi:
- case X86::BI__builtin_ia32_kunpckhi:
case X86::BI__builtin_ia32_sqrtsh_round_mask:
case X86::BI__builtin_ia32_sqrtsd_round_mask:
case X86::BI__builtin_ia32_sqrtss_round_mask:
diff --git a/clang/test/CIR/CodeGen/X86/avx512-kunpck-builtins.c b/clang/test/CIR/CodeGen/X86/avx512-kunpck-builtins.c
new file mode 100644
index 0000000000000..197f2570769a5
--- /dev/null
+++ b/clang/test/CIR/CodeGen/X86/avx512-kunpck-builtins.c
@@ -0,0 +1,92 @@
+// RUN: %clang_cc1 -x c -flax-vector-conversions=none -ffreestanding %s -triple=x86_64-unknown-linux -target-feature +avx512f -target-feature +avx512bw -fclangir -emit-cir -o %t.cir -Wall -Werror
+// RUN: FileCheck --check-prefix=CIR --input-file=%t.cir %s
+// RUN: %clang_cc1 -x c -flax-vector-conversions=none -ffreestanding %s -triple=x86_64-unknown-linux -target-feature +avx512f -target-feature +avx512bw -fclangir -emit-llvm -o %t.ll -Wall -Werror
+// RUN: FileCheck --check-prefixes=LLVM --input-file=%t.ll %s
+
+// RUN: %clang_cc1 -x c -flax-vector-conversions=none -ffreestanding %s -triple=x86_64-unknown-linux -target-feature +avx512f -target-feature +avx512bw -emit-llvm -o - -Wall -Werror | FileCheck %s -check-prefix=OGCG
+
+// This test exercises the kunpck (mask unpack) builtins for AVX-512.
+
+#include <immintrin.h>
+
+__mmask16 test_mm512_kunpackb(__mmask16 __A, __mmask16 __B) {
+ // CIR-LABEL: test_mm512_kunpackb
+ // LLVM-LABEL: test_mm512_kunpackb
+ // OGCG-LABEL: test_mm512_kunpackb
+ return _mm512_kunpackb(__A, __B);
+ // CIR: [[MASK_A:%.*]] = cir.cast(bitcast, {{%.*}} : !cir.int<u, 16>), !cir.vector<!cir.bool x 16>
+ // CIR: [[EXTRACT_A:%.*]] = cir.vec.shuffle([[MASK_A]], {{.*}}) [#cir.int<0> : !s32i, #cir.int<1> : !s32i, #cir.int<2> : !s32i, #cir.int<3> : !s32i, #cir.int<4> : !s32i, #cir.int<5> : !s32i, #cir.int<6> : !s32i, #cir.int<7> : !s32i]
+ // CIR: [[MASK_B:%.*]] = cir.cast(bitcast, {{%.*}} : !cir.int<u, 16>), !cir.vector<!cir.bool x 16>
+ // CIR: [[EXTRACT_B:%.*]] = cir.vec.shuffle([[MASK_B]], {{.*}}) [#cir.int<0> : !s32i, #cir.int<1> : !s32i, #cir.int<2> : !s32i, #cir.int<3> : !s32i, #cir.int<4> : !s32i, #cir.int<5> : !s32i, #cir.int<6> : !s32i, #cir.int<7> : !s32i]
+ // CIR: [[CONCAT:%.*]] = cir.vec.shuffle([[EXTRACT_B]], [[EXTRACT_A]]) [#cir.int<0> : !s32i, #cir.int<1> : !s32i, #cir.int<2> : !s32i, #cir.int<3> : !s32i, #cir.int<4> : !s32i, #cir.int<5> : !s32i, #cir.int<6> : !s32i, #cir.int<7> : !s32i, #cir.int<8> : !s32i, #cir.int<9> : !s32i, #cir.int<10> : !s32i, #cir.int<11> : !s32i, #cir.int<12> : !s32i, #cir.int<13> : !s32i, #cir.int<14> : !s32i, #cir.int<15> : !s32i]
+ // CIR: {{%.*}} = cir.cast(bitcast, [[CONCAT]] : !cir.vector<!cir.bool x 16>), !cir.int<u, 16>
+
+ // LLVM: [[A_BITCAST:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
+ // LLVM: [[A_SHUFFLE:%.*]] = shufflevector <16 x i1> [[A_BITCAST]], <16 x i1> [[A_BITCAST]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+ // LLVM: [[B_BITCAST:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
+ // LLVM: [[B_SHUFFLE:%.*]] = shufflevector <16 x i1> [[B_BITCAST]], <16 x i1> [[B_BITCAST]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+ // LLVM: [[CONCAT:%.*]] = shufflevector <8 x i1> [[B_SHUFFLE]], <8 x i1> [[A_SHUFFLE]], <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+ // LLVM: bitcast <16 x i1> [[CONCAT]] to i16
+
+ // OGCG: [[A_BITCAST:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
+ // OGCG: [[A_SHUFFLE:%.*]] = shufflevector <16 x i1> [[A_BITCAST]], <16 x i1> [[A_BITCAST]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+ // OGCG: [[B_BITCAST:%.*]] = bitcast i16 %{{.*}} to <16 x i1>
+ // OGCG: [[B_SHUFFLE:%.*]] = shufflevector <16 x i1> [[B_BITCAST]], <16 x i1> [[B_BITCAST]], <8 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+ // OGCG: [[CONCAT:%.*]] = shufflevector <8 x i1> [[B_SHUFFLE]], <8 x i1> [[A_SHUFFLE]], <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+ // OGCG: bitcast <16 x i1> [[CONCAT]] to i16
+}
+
+__mmask32 test_mm512_kunpackw(__mmask32 __A, __mmask32 __B) {
+ // CIR-LABEL: test_mm512_kunpackw
+ // LLVM-LABEL: test_mm512_kunpackw
+ // OGCG-LABEL: test_mm512_kunpackw
+ return _mm512_kunpackw(__A, __B);
+ // CIR: [[MASK_A:%.*]] = cir.cast(bitcast, {{%.*}} : !cir.int<u, 32>), !cir.vector<!cir.bool x 32>
+ // CIR: [[EXTRACT_A:%.*]] = cir.vec.shuffle([[MASK_A]], {{.*}})
+ // CIR: [[MASK_B:%.*]] = cir.cast(bitcast, {{%.*}} : !cir.int<u, 32>), !cir.vector<!cir.bool x 32>
+ // CIR: [[EXTRACT_B:%.*]] = cir.vec.shuffle([[MASK_B]], {{.*}})
+ // CIR: [[CONCAT:%.*]] = cir.vec.shuffle([[EXTRACT_B]], [[EXTRACT_A]])
+ // CIR: {{%.*}} = cir.cast(bitcast, [[CONCAT]] : !cir.vector<!cir.bool x 32>), !cir.int<u, 32>
+
+ // LLVM: [[A_BITCAST:%.*]] = bitcast i32 %{{.*}} to <32 x i1>
+ // LLVM: [[A_SHUFFLE:%.*]] = shufflevector <32 x i1> [[A_BITCAST]], <32 x i1> [[A_BITCAST]], <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+ // LLVM: [[B_BITCAST:%.*]] = bitcast i32 %{{.*}} to <32 x i1>
+ // LLVM: [[B_SHUFFLE:%.*]] = shufflevector <32 x i1> [[B_BITCAST]], <32 x i1> [[B_BITCAST]], <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+ // LLVM: [[CONCAT:%.*]] = shufflevector <16 x i1> [[B_SHUFFLE]], <16 x i1> [[A_SHUFFLE]], <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31>
+ // LLVM: bitcast <32 x i1> [[CONCAT]] to i32
+
+ // OGCG: [[A_BITCAST:%.*]] = bitcast i32 %{{.*}} to <32 x i1>
+ // OGCG: [[A_SHUFFLE:%.*]] = shufflevector <32 x i1> [[A_BITCAST]], <32 x i1> [[A_BITCAST]], <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+ // OGCG: [[B_BITCAST:%.*]] = bitcast i32 %{{.*}} to <32 x i1>
+ // OGCG: [[B_SHUFFLE:%.*]] = shufflevector <32 x i1> [[B_BITCAST]], <32 x i1> [[B_BITCAST]], <16 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15>
+ // OGCG: [[CONCAT:%.*]] = shufflevector <16 x i1> [[B_SHUFFLE]], <16 x i1> [[A_SHUFFLE]], <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31>
+ // OGCG: bitcast <32 x i1> [[CONCAT]] to i32
+}
+
+__mmask64 test_mm512_kunpackd(__mmask64 __A, __mmask64 __B) {
+ // CIR-LABEL: test_mm512_kunpackd
+ // LLVM-LABEL: test_mm512_kunpackd
+ // OGCG-LABEL: test_mm512_kunpackd
+ return _mm512_kunpackd(__A, __B);
+ // CIR: [[MASK_A:%.*]] = cir.cast(bitcast, {{%.*}} : !cir.int<u, 64>), !cir.vector<!cir.bool x 64>
+ // CIR: [[EXTRACT_A:%.*]] = cir.vec.shuffle([[MASK_A]], {{.*}})
+ // CIR: [[MASK_B:%.*]] = cir.cast(bitcast, {{%.*}} : !cir.int<u, 64>), !cir.vector<!cir.bool x 64>
+ // CIR: [[EXTRACT_B:%.*]] = cir.vec.shuffle([[MASK_B]], {{.*}})
+ // CIR: [[CONCAT:%.*]] = cir.vec.shuffle([[EXTRACT_B]], [[EXTRACT_A]])
+ // CIR: {{%.*}} = cir.cast(bitcast, [[CONCAT]] : !cir.vector<!cir.bool x 64>), !cir.int<u, 64>
+
+ // LLVM: [[A_BITCAST:%.*]] = bitcast i64 %{{.*}} to <64 x i1>
+ // LLVM: [[A_SHUFFLE:%.*]] = shufflevector <64 x i1> [[A_BITCAST]], <64 x i1> [[A_BITCAST]], <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31>
+ // LLVM: [[B_BITCAST:%.*]] = bitcast i64 %{{.*}} to <64 x i1>
+ // LLVM: [[B_SHUFFLE:%.*]] = shufflevector <64 x i1> [[B_BITCAST]], <64 x i1> [[B_BITCAST]], <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31>
+ // LLVM: [[CONCAT:%.*]] = shufflevector <32 x i1> [[B_SHUFFLE]], <32 x i1> [[A_SHUFFLE]], <64 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31, i32 32, i32 33, i32 34, i32 35, i32 36, i32 37, i32 38, i32 39, i32 40, i32 41, i32 42, i32 43, i32 44, i32 45, i32 46, i32 47, i32 48, i32 49, i32 50, i32 51, i32 52, i32 53, i32 54, i32 55, i32 56, i32 57, i32 58, i32 59, i32 60, i32 61, i32 62, i32 63>
+ // LLVM: bitcast <64 x i1> [[CONCAT]] to i64
+
+ // OGCG: [[A_BITCAST:%.*]] = bitcast i64 %{{.*}} to <64 x i1>
+ // OGCG: [[A_SHUFFLE:%.*]] = shufflevector <64 x i1> [[A_BITCAST]], <64 x i1> [[A_BITCAST]], <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31>
+ // OGCG: [[B_BITCAST:%.*]] = bitcast i64 %{{.*}} to <64 x i1>
+ // OGCG: [[B_SHUFFLE:%.*]] = shufflevector <64 x i1> [[B_BITCAST]], <64 x i1> [[B_BITCAST]], <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31>
+ // OGCG: [[CONCAT:%.*]] = shufflevector <32 x i1> [[B_SHUFFLE]], <32 x i1> [[A_SHUFFLE]], <64 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31, i32 32, i32 33, i32 34, i32 35, i32 36, i32 37, i32 38, i32 39, i32 40, i32 41, i32 42, i32 43, i32 44, i32 45, i32 46, i32 47, i32 48, i32 49, i32 50, i32 51, i32 52, i32 53, i32 54, i32 55, i32 56, i32 57, i32 58, i32 59, i32 60, i32 61, i32 62, i32 63>
+ // OGCG: bitcast <64 x i1> [[CONCAT]] to i64
+}
+
|
🐧 Linux x64 Test Results
✅ The build succeeded and all tests passed. |
andykaylor
reviewed
Nov 20, 2025
| .getResult(); | ||
| } | ||
|
|
||
| static mlir::Value getMaskVecValue(CIRGenBuilderTy &builder, mlir::Value mask, |
Contributor
There was a problem hiding this comment.
This is function is also being added in #168591
Contributor
Author
There was a problem hiding this comment.
refactored to specifically refer to kunpack operation
Contributor
There was a problem hiding this comment.
I don't think that's what we want. The function will be shared. I was just mentioning it because we'll want the same implementation in both PRs and whichever one is merged second will need to be rebased on the other.
|
✅ With the latest revision this PR passed the C/C++ code formatter. |
RKSimon
reviewed
Nov 20, 2025
Contributor
Oh, sorry. I gave you the wrong path in my comment requesting that you move the tests you originally added. The test should be added to |
Contributor
Author
|
To avoid conflict will work on this again once #168591 is merged |
andykaylor
reviewed
Nov 26, 2025
Contributor
Author
|
Hey @andykaylor @HendrikHuebner I have addressed feedback. Please check again when you have time :) |
andykaylor
approved these changes
Dec 2, 2025
Contributor
andykaylor
left a comment
andykaylor
left a comment
There was a problem hiding this comment.
Looks good to me. Do you need assistance committing this?
Contributor
Author
Yes, please. I’d appreciate your help committing this |
Collaborator
|
LLVM Buildbot has detected a new failure on builder Full details are available at: https://lab.llvm.org/buildbot/#/builders/51/builds/27942 Here is the relevant piece of the build log for the reference |
kcloudy0717
pushed a commit
to kcloudy0717/llvm-project
that referenced
this pull request
Dec 4, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Part of #167765