Cannot build aws-sdk-s3@1.86 under Amazon Linux 2023 using cargo lambda.

[LegoWolf]

Describe the bug

I cannot use cargo lambda to build any Rust project that includes aws-sdk-s3 v1.86, whereas using v1.85 or earlier build fine. The build error only happens when using cargo lambda build not cargo build.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

No build errors.

Current Behavior

Following the reproduction steps produces this output:

    Updating crates.io index
     Locking 9 packages to latest Rust 1.86.0 compatible versions
    Updating aws-sdk-s3 v1.85.0 -> v1.86.0
    Updating aws-smithy-checksums v0.63.1 -> v0.63.2
      Adding crc-fast v1.2.1
      Adding ppv-lite86 v0.2.21
      Adding rand v0.9.1
      Adding rand_chacha v0.9.0
      Adding rand_core v0.9.3
      Adding zerocopy v0.8.25
      Adding zerocopy-derive v0.8.25
   Compiling crc-fast v1.2.1
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:42:18: error: always_inline function '_mm512_loadu_si512' requires target feature 'evex512', but would be inlined into function 'crc32_iscsi_impl' that is compiled without support for 'evex512'
warning: crc-fast@1.2.1:    42 |     __m512i x0 = _mm512_loadu_si512((const void*)buf), y0;
warning: crc-fast@1.2.1:       |                  ^
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:42:18: error: AVX vector return of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:43:18: error: always_inline function '_mm512_loadu_si512' requires target feature 'evex512', but would be inlined into function 'crc32_iscsi_impl' that is compiled without support for 'evex512'
warning: crc-fast@1.2.1:    43 |     __m512i x1 = _mm512_loadu_si512((const void*)(buf + 64)), y1;
warning: crc-fast@1.2.1:       |                  ^
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:43:18: error: AVX vector return of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:44:18: error: always_inline function '_mm512_loadu_si512' requires target feature 'evex512', but would be inlined into function 'crc32_iscsi_impl' that is compiled without support for 'evex512'
warning: crc-fast@1.2.1:    44 |     __m512i x2 = _mm512_loadu_si512((const void*)(buf + 128)), y2;
warning: crc-fast@1.2.1:       |                  ^
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:44:18: error: AVX vector return of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:46:9: error: always_inline function '_mm512_broadcast_i32x4' requires target feature 'evex512', but would be inlined into function 'crc32_iscsi_impl' that is compiled without support for 'evex512'
warning: crc-fast@1.2.1:    46 |     k = _mm512_broadcast_i32x4(_mm_setr_epi32(0xa87ab8a8, 0, 0xab7aff2a, 0));
warning: crc-fast@1.2.1:       |         ^
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:46:9: error: AVX vector return of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:47:27: error: always_inline function '_mm512_castsi128_si512' requires target feature 'evex512', but would be inlined into function 'crc32_iscsi_impl' that is compiled without support for 'evex512'
warning: crc-fast@1.2.1:    47 |     x0 = _mm512_xor_si512(_mm512_castsi128_si512(_mm_cvtsi32_si128(crc0)), x0);
warning: crc-fast@1.2.1:       |                           ^
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:47:27: error: AVX vector return of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:47:10: error: always_inline function '_mm512_xor_si512' requires target feature 'evex512', but would be inlined into function 'crc32_iscsi_impl' that is compiled without support for 'evex512'
warning: crc-fast@1.2.1:    47 |     x0 = _mm512_xor_si512(_mm512_castsi128_si512(_mm_cvtsi32_si128(crc0)), x0);
warning: crc-fast@1.2.1:       |          ^
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:47:10: error: AVX vector argument of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:48:10: error: '__builtin_ia32_pclmulqdq512' needs target feature avx512f,evex512,vpclmulqdq
warning: crc-fast@1.2.1:    48 |     y0 = clmul_lo(x0, k), x0 = clmul_hi(x0, k);
warning: crc-fast@1.2.1:       |          ^
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:20:25: note: expanded from macro 'clmul_lo'
warning: crc-fast@1.2.1:    20 | #define clmul_lo(a, b) (_mm512_clmulepi64_epi128((a), (b), 0))
warning: crc-fast@1.2.1:       |                         ^
warning: crc-fast@1.2.1: /home/ec2-user/bin/zig/lib/include/vpclmulqdqintrin.h:24:13: note: expanded from macro '_mm512_clmulepi64_epi128'
warning: crc-fast@1.2.1:    24 |   ((__m512i)__builtin_ia32_pclmulqdq512((__v8di)(__m512i)(A),  \
warning: crc-fast@1.2.1:       |             ^
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:48:32: error: '__builtin_ia32_pclmulqdq512' needs target feature avx512f,evex512,vpclmulqdq
warning: crc-fast@1.2.1:    48 |     y0 = clmul_lo(x0, k), x0 = clmul_hi(x0, k);
warning: crc-fast@1.2.1:       |                                ^
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:21:25: note: expanded from macro 'clmul_hi'
warning: crc-fast@1.2.1:    21 | #define clmul_hi(a, b) (_mm512_clmulepi64_epi128((a), (b), 17))
warning: crc-fast@1.2.1:       |                         ^
warning: crc-fast@1.2.1: /home/ec2-user/bin/zig/lib/include/vpclmulqdqintrin.h:24:13: note: expanded from macro '_mm512_clmulepi64_epi128'
warning: crc-fast@1.2.1:    24 |   ((__m512i)__builtin_ia32_pclmulqdq512((__v8di)(__m512i)(A),  \
warning: crc-fast@1.2.1:       |             ^
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:49:10: error: '__builtin_ia32_pclmulqdq512' needs target feature avx512f,evex512,vpclmulqdq
warning: crc-fast@1.2.1:    49 |     y1 = clmul_lo(x1, k), x1 = clmul_hi(x1, k);
warning: crc-fast@1.2.1:       |          ^
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:20:25: note: expanded from macro 'clmul_lo'
warning: crc-fast@1.2.1:    20 | #define clmul_lo(a, b) (_mm512_clmulepi64_epi128((a), (b), 0))
warning: crc-fast@1.2.1:       |                         ^
warning: crc-fast@1.2.1: /home/ec2-user/bin/zig/lib/include/vpclmulqdqintrin.h:24:13: note: expanded from macro '_mm512_clmulepi64_epi128'
warning: crc-fast@1.2.1:    24 |   ((__m512i)__builtin_ia32_pclmulqdq512((__v8di)(__m512i)(A),  \
warning: crc-fast@1.2.1:       |             ^
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:49:32: error: '__builtin_ia32_pclmulqdq512' needs target feature avx512f,evex512,vpclmulqdq
warning: crc-fast@1.2.1:    49 |     y1 = clmul_lo(x1, k), x1 = clmul_hi(x1, k);
warning: crc-fast@1.2.1:       |                                ^
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:21:25: note: expanded from macro 'clmul_hi'
warning: crc-fast@1.2.1:    21 | #define clmul_hi(a, b) (_mm512_clmulepi64_epi128((a), (b), 17))
warning: crc-fast@1.2.1:       |                         ^
warning: crc-fast@1.2.1: /home/ec2-user/bin/zig/lib/include/vpclmulqdqintrin.h:24:13: note: expanded from macro '_mm512_clmulepi64_epi128'
warning: crc-fast@1.2.1:    24 |   ((__m512i)__builtin_ia32_pclmulqdq512((__v8di)(__m512i)(A),  \
warning: crc-fast@1.2.1:       |             ^
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:50:10: error: '__builtin_ia32_pclmulqdq512' needs target feature avx512f,evex512,vpclmulqdq
warning: crc-fast@1.2.1:    50 |     y2 = clmul_lo(x2, k), x2 = clmul_hi(x2, k);
warning: crc-fast@1.2.1:       |          ^
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:20:25: note: expanded from macro 'clmul_lo'
warning: crc-fast@1.2.1:    20 | #define clmul_lo(a, b) (_mm512_clmulepi64_epi128((a), (b), 0))
warning: crc-fast@1.2.1:       |                         ^
warning: crc-fast@1.2.1: /home/ec2-user/bin/zig/lib/include/vpclmulqdqintrin.h:24:13: note: expanded from macro '_mm512_clmulepi64_epi128'
warning: crc-fast@1.2.1:    24 |   ((__m512i)__builtin_ia32_pclmulqdq512((__v8di)(__m512i)(A),  \
warning: crc-fast@1.2.1:       |             ^
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:50:32: error: '__builtin_ia32_pclmulqdq512' needs target feature avx512f,evex512,vpclmulqdq
warning: crc-fast@1.2.1:    50 |     y2 = clmul_lo(x2, k), x2 = clmul_hi(x2, k);
warning: crc-fast@1.2.1:       |                                ^
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:21:25: note: expanded from macro 'clmul_hi'
warning: crc-fast@1.2.1:    21 | #define clmul_hi(a, b) (_mm512_clmulepi64_epi128((a), (b), 17))
warning: crc-fast@1.2.1:       |                         ^
warning: crc-fast@1.2.1: /home/ec2-user/bin/zig/lib/include/vpclmulqdqintrin.h:24:13: note: expanded from macro '_mm512_clmulepi64_epi128'
warning: crc-fast@1.2.1:    24 |   ((__m512i)__builtin_ia32_pclmulqdq512((__v8di)(__m512i)(A),  \
warning: crc-fast@1.2.1:       |             ^
warning: crc-fast@1.2.1: include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:51:10: error: '__builtin_ia32_pternlogq512_mask' needs target feature avx512f,evex512
warning: crc-fast@1.2.1:    51 |     x0 = _mm512_ternarylogic_epi64(x0, y0, _mm512_loadu_si512((const void*)(buf + 192)), 0x96);
warning: crc-fast@1.2.1:       |          ^
warning: crc-fast@1.2.1: /home/ec2-user/bin/zig/lib/include/avx512fintrin.h:5971:13: note: expanded from macro '_mm512_ternarylogic_epi64'
warning: crc-fast@1.2.1:  5971 |   ((__m512i)__builtin_ia32_pternlogq512_mask(                                  \
warning: crc-fast@1.2.1:       |             ^
warning: crc-fast@1.2.1: fatal error: too many errors emitted, stopping now [-ferror-limit=]
warning: crc-fast@1.2.1: 20 errors generated.
error: failed to run custom build command for `crc-fast v1.2.1`

Caused by:
  process didn't exit successfully: `/home/ec2-user/aws-s3/target/debug/build/crc-fast-ef0cf203bf559223/build-script-build` (exit status: 1)
  --- stdout
  Building AVX512 VPCLMULQDQ v3x2
  OUT_DIR = Some(/home/ec2-user/aws-s3/target/x86_64-unknown-linux-gnu/debug/build/crc-fast-922b7726669a341b/out)
  OPT_LEVEL = Some(0)
  TARGET = Some(x86_64-unknown-linux-gnu)
  HOST = Some(x86_64-unknown-linux-gnu)
  cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-gnu
  CC_x86_64-unknown-linux-gnu = None
  cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_gnu
  CC_x86_64_unknown_linux_gnu = Some(/home/ec2-user/.cache/cargo-zigbuild/0.20.0/zigcc-x86_64-unknown-linux-gnu-de6.sh)
  RUSTC_WRAPPER = None
  cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some(true)
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=HOST_CFLAGS
  HOST_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_gnu
  CFLAGS_x86_64_unknown_linux_gnu = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-gnu
  CFLAGS_x86_64-unknown-linux-gnu = None
  CARGO_ENCODED_RUSTFLAGS = Some()
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:42:18: error: always_inline function '_mm512_loadu_si512' requires target feature 'evex512', but would be inlined into function 'crc32_iscsi_impl' that is compiled without support for 'evex512'
  cargo:warning=   42 |     __m512i x0 = _mm512_loadu_si512((const void*)buf), y0;
  cargo:warning=      |                  ^
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:42:18: error: AVX vector return of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:43:18: error: always_inline function '_mm512_loadu_si512' requires target feature 'evex512', but would be inlined into function 'crc32_iscsi_impl' that is compiled without support for 'evex512'
  cargo:warning=   43 |     __m512i x1 = _mm512_loadu_si512((const void*)(buf + 64)), y1;
  cargo:warning=      |                  ^
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:43:18: error: AVX vector return of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:44:18: error: always_inline function '_mm512_loadu_si512' requires target feature 'evex512', but would be inlined into function 'crc32_iscsi_impl' that is compiled without support for 'evex512'
  cargo:warning=   44 |     __m512i x2 = _mm512_loadu_si512((const void*)(buf + 128)), y2;
  cargo:warning=      |                  ^
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:44:18: error: AVX vector return of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:46:9: error: always_inline function '_mm512_broadcast_i32x4' requires target feature 'evex512', but would be inlined into function 'crc32_iscsi_impl' that is compiled without support for 'evex512'
  cargo:warning=   46 |     k = _mm512_broadcast_i32x4(_mm_setr_epi32(0xa87ab8a8, 0, 0xab7aff2a, 0));
  cargo:warning=      |         ^
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:46:9: error: AVX vector return of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:47:27: error: always_inline function '_mm512_castsi128_si512' requires target feature 'evex512', but would be inlined into function 'crc32_iscsi_impl' that is compiled without support for 'evex512'
  cargo:warning=   47 |     x0 = _mm512_xor_si512(_mm512_castsi128_si512(_mm_cvtsi32_si128(crc0)), x0);
  cargo:warning=      |                           ^
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:47:27: error: AVX vector return of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:47:10: error: always_inline function '_mm512_xor_si512' requires target feature 'evex512', but would be inlined into function 'crc32_iscsi_impl' that is compiled without support for 'evex512'
  cargo:warning=   47 |     x0 = _mm512_xor_si512(_mm512_castsi128_si512(_mm_cvtsi32_si128(crc0)), x0);
  cargo:warning=      |          ^
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:47:10: error: AVX vector argument of type '__m512i' (vector of 8 'long long' values) without 'evex512' enabled changes the ABI
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:48:10: error: '__builtin_ia32_pclmulqdq512' needs target feature avx512f,evex512,vpclmulqdq
  cargo:warning=   48 |     y0 = clmul_lo(x0, k), x0 = clmul_hi(x0, k);
  cargo:warning=      |          ^
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:20:25: note: expanded from macro 'clmul_lo'
  cargo:warning=   20 | #define clmul_lo(a, b) (_mm512_clmulepi64_epi128((a), (b), 0))
  cargo:warning=      |                         ^
  cargo:warning=/home/ec2-user/bin/zig/lib/include/vpclmulqdqintrin.h:24:13: note: expanded from macro '_mm512_clmulepi64_epi128'
  cargo:warning=   24 |   ((__m512i)__builtin_ia32_pclmulqdq512((__v8di)(__m512i)(A),  \
  cargo:warning=      |             ^
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:48:32: error: '__builtin_ia32_pclmulqdq512' needs target feature avx512f,evex512,vpclmulqdq
  cargo:warning=   48 |     y0 = clmul_lo(x0, k), x0 = clmul_hi(x0, k);
  cargo:warning=      |                                ^
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:21:25: note: expanded from macro 'clmul_hi'
  cargo:warning=   21 | #define clmul_hi(a, b) (_mm512_clmulepi64_epi128((a), (b), 17))
  cargo:warning=      |                         ^
  cargo:warning=/home/ec2-user/bin/zig/lib/include/vpclmulqdqintrin.h:24:13: note: expanded from macro '_mm512_clmulepi64_epi128'
  cargo:warning=   24 |   ((__m512i)__builtin_ia32_pclmulqdq512((__v8di)(__m512i)(A),  \
  cargo:warning=      |             ^
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:49:10: error: '__builtin_ia32_pclmulqdq512' needs target feature avx512f,evex512,vpclmulqdq
  cargo:warning=   49 |     y1 = clmul_lo(x1, k), x1 = clmul_hi(x1, k);
  cargo:warning=      |          ^
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:20:25: note: expanded from macro 'clmul_lo'
  cargo:warning=   20 | #define clmul_lo(a, b) (_mm512_clmulepi64_epi128((a), (b), 0))
  cargo:warning=      |                         ^
  cargo:warning=/home/ec2-user/bin/zig/lib/include/vpclmulqdqintrin.h:24:13: note: expanded from macro '_mm512_clmulepi64_epi128'
  cargo:warning=   24 |   ((__m512i)__builtin_ia32_pclmulqdq512((__v8di)(__m512i)(A),  \
  cargo:warning=      |             ^
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:49:32: error: '__builtin_ia32_pclmulqdq512' needs target feature avx512f,evex512,vpclmulqdq
  cargo:warning=   49 |     y1 = clmul_lo(x1, k), x1 = clmul_hi(x1, k);
  cargo:warning=      |                                ^
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:21:25: note: expanded from macro 'clmul_hi'
  cargo:warning=   21 | #define clmul_hi(a, b) (_mm512_clmulepi64_epi128((a), (b), 17))
  cargo:warning=      |                         ^
  cargo:warning=/home/ec2-user/bin/zig/lib/include/vpclmulqdqintrin.h:24:13: note: expanded from macro '_mm512_clmulepi64_epi128'
  cargo:warning=   24 |   ((__m512i)__builtin_ia32_pclmulqdq512((__v8di)(__m512i)(A),  \
  cargo:warning=      |             ^
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:50:10: error: '__builtin_ia32_pclmulqdq512' needs target feature avx512f,evex512,vpclmulqdq
  cargo:warning=   50 |     y2 = clmul_lo(x2, k), x2 = clmul_hi(x2, k);
  cargo:warning=      |          ^
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:20:25: note: expanded from macro 'clmul_lo'
  cargo:warning=   20 | #define clmul_lo(a, b) (_mm512_clmulepi64_epi128((a), (b), 0))
  cargo:warning=      |                         ^
  cargo:warning=/home/ec2-user/bin/zig/lib/include/vpclmulqdqintrin.h:24:13: note: expanded from macro '_mm512_clmulepi64_epi128'
  cargo:warning=   24 |   ((__m512i)__builtin_ia32_pclmulqdq512((__v8di)(__m512i)(A),  \
  cargo:warning=      |             ^
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:50:32: error: '__builtin_ia32_pclmulqdq512' needs target feature avx512f,evex512,vpclmulqdq
  cargo:warning=   50 |     y2 = clmul_lo(x2, k), x2 = clmul_hi(x2, k);
  cargo:warning=      |                                ^
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:21:25: note: expanded from macro 'clmul_hi'
  cargo:warning=   21 | #define clmul_hi(a, b) (_mm512_clmulepi64_epi128((a), (b), 17))
  cargo:warning=      |                         ^
  cargo:warning=/home/ec2-user/bin/zig/lib/include/vpclmulqdqintrin.h:24:13: note: expanded from macro '_mm512_clmulepi64_epi128'
  cargo:warning=   24 |   ((__m512i)__builtin_ia32_pclmulqdq512((__v8di)(__m512i)(A),  \
  cargo:warning=      |             ^
  cargo:warning=include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c:51:10: error: '__builtin_ia32_pternlogq512_mask' needs target feature avx512f,evex512
  cargo:warning=   51 |     x0 = _mm512_ternarylogic_epi64(x0, y0, _mm512_loadu_si512((const void*)(buf + 192)), 0x96);
  cargo:warning=      |          ^
  cargo:warning=/home/ec2-user/bin/zig/lib/include/avx512fintrin.h:5971:13: note: expanded from macro '_mm512_ternarylogic_epi64'
  cargo:warning= 5971 |   ((__m512i)__builtin_ia32_pternlogq512_mask(                                  \
  cargo:warning=      |             ^
  cargo:warning=fatal error: too many errors emitted, stopping now [-ferror-limit=]
  cargo:warning=20 errors generated.

  --- stderr


  error occurred in cc-rs: command did not execute successfully (status code exit status: 1): LC_ALL="C" "/home/ec2-user/.cache/cargo-zigbuild/0.20.0/zigcc-x86_64-unknown-linux-gnu-de6.sh" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-gdwarf-4" "-fno-omit-frame-pointer" "-m64" "--target=x86_64-unknown-linux-gnu" "-I" "include" "-Wall" "-Wextra" "-msse4.2" "-mpclmul" "-mavx512f" "-mavx512vl" "-mvpclmulqdq" "-o" "/home/ec2-user/aws-s3/target/x86_64-unknown-linux-gnu/debug/build/crc-fast-922b7726669a341b/out/b0926c13c671e7f9-crc32_iscsi_avx512_vpclmulqdq_v3x2.o" "-c" "include/crc32_iscsi_avx512_vpclmulqdq_v3x2.c"

Reproduction Steps

Install:

• Rust toolchain v1.86.0 (latest stable)
• cargo-lambda v1.8.5

Create and build this project:

cargo new aws-s3-test
cd aws-s3-test
cargo add aws-sdk-s3@=1.86.0
cargo lambda build

Possible Solution

The problem occurs when cargo tries to build crc-fast, which is a new dependency of aws-smithy-checksums v0.63.2. I'm not certain whether the problem is in the crate itself or in the build environment. I'm not sure how to isolate the problem further than I have.

Additional Information/Context

I am running Amazon Linux 2023 in Windows using WSL2. uname -a produces this output:

Linux GLYN-PC 5.15.167.4-microsoft-standard-WSL2 #1 SMP Tue Nov 5 00:21:55 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Version

aws-s3 v0.1.0 (/home/ec2-user/aws-s3)
└── aws-sdk-s3 v1.86.0
    ├── aws-credential-types v1.2.3
    │   ├── aws-smithy-async v1.2.5
    │   ├── aws-smithy-runtime-api v1.8.0
    │   │   ├── aws-smithy-async v1.2.5 (*)
    │   │   ├── aws-smithy-types v1.3.1
    │   ├── aws-smithy-types v1.3.1 (*)
    ├── aws-runtime v1.5.7
    │   ├── aws-credential-types v1.2.3 (*)
    │   ├── aws-sigv4 v1.3.2
    │   │   ├── aws-credential-types v1.2.3 (*)
    │   │   ├── aws-smithy-eventstream v0.60.8
    │   │   │   ├── aws-smithy-types v1.3.1 (*)
    │   │   ├── aws-smithy-http v0.62.1
    │   │   │   ├── aws-smithy-eventstream v0.60.8 (*)
    │   │   │   ├── aws-smithy-runtime-api v1.8.0 (*)
    │   │   │   ├── aws-smithy-types v1.3.1 (*)
    │   │   ├── aws-smithy-runtime-api v1.8.0 (*)
    │   │   ├── aws-smithy-types v1.3.1 (*)
    │   ├── aws-smithy-async v1.2.5 (*)
    │   ├── aws-smithy-eventstream v0.60.8 (*)
    │   ├── aws-smithy-http v0.62.1 (*)
    │   ├── aws-smithy-runtime v1.8.3
    │   │   ├── aws-smithy-async v1.2.5 (*)
    │   │   ├── aws-smithy-http v0.62.1 (*)
    │   │   ├── aws-smithy-http-client v1.0.2
    │   │   │   ├── aws-smithy-async v1.2.5 (*)
    │   │   │   ├── aws-smithy-runtime-api v1.8.0 (*)
    │   │   │   ├── aws-smithy-types v1.3.1 (*)
    │   │   │   │   │   ├── aws-lc-rs v1.13.1
    │   │   │   │   │   │   ├── aws-lc-sys v0.29.0
    │   │   │   │   │   │   ├── aws-lc-rs v1.13.1 (*)
    │   │   ├── aws-smithy-observability v0.1.3
    │   │   │   └── aws-smithy-runtime-api v1.8.0 (*)
    │   │   ├── aws-smithy-runtime-api v1.8.0 (*)
    │   │   ├── aws-smithy-types v1.3.1 (*)
    │   ├── aws-smithy-runtime-api v1.8.0 (*)
    │   ├── aws-smithy-types v1.3.1 (*)
    │   ├── aws-types v1.3.7
    │   │   ├── aws-credential-types v1.2.3 (*)
    │   │   ├── aws-smithy-async v1.2.5 (*)
    │   │   ├── aws-smithy-runtime-api v1.8.0 (*)
    │   │   ├── aws-smithy-types v1.3.1 (*)
    ├── aws-sigv4 v1.3.2 (*)
    ├── aws-smithy-async v1.2.5 (*)
    ├── aws-smithy-checksums v0.63.2
    │   ├── aws-smithy-http v0.62.1 (*)
    │   ├── aws-smithy-types v1.3.1 (*)
    ├── aws-smithy-eventstream v0.60.8 (*)
    ├── aws-smithy-http v0.62.1 (*)
    ├── aws-smithy-json v0.61.3
    │   └── aws-smithy-types v1.3.1 (*)
    ├── aws-smithy-runtime v1.8.3 (*)
    ├── aws-smithy-runtime-api v1.8.0 (*)
    ├── aws-smithy-types v1.3.1 (*)
    ├── aws-smithy-xml v0.60.9
    ├── aws-types v1.3.7 (*)

Environment details (OS name and version, etc.)

Amazon Linux 2023, x86_64

Logs

No response

[AndrewLipscomb]

Can confirm this bug and the same errors when doing a cargo-zigbuild based link for a lambda intended for AWS Linux 2023 x86_64 from an Arch Linux build machine - so its not a Windows specific error, its definitely the bumped dependency on https://github.com/awesomized/crc-fast-rust

Edit: this seems more like a cargo lambda bug than an AWS bug - I've raised another bug report cargo-lambda/cargo-lambda#859 linking to this one

For workarounds until this gets figured out - pin your deps like this

# This one wants at least aws-smithy-checksums=>0.63.2
aws-sdk-s3 = { version = "=1.85.0" }
# But we have to limit it to just this version 
aws-smithy-checksums = { version = "=0.63.1" }

Potentially relevant things

OS Arch Linux x86_64
rustc 1.87.0 (17067e9ac 2025-05-09)
zig version 0.13.0
cc (GCC) 15.1.1 20250425

[dcormier]

For what it's worth, I ran into a similar failure on Friday while building in the cargo-lambda container. While digging I noticed that crc-fast was only recently added as a dependency.

[atwam]

We are encountering issues here, probably related to this crate usage as well (still testing).

Our software using 1.83 was fine. It gets killed with a SIGILL (Illegal instruction) when using version 1.85 of the sdk. We don't have yet a core-dump, but the kill happens within the routine that uploads a file to S3, which was working fine last week with 1.83.

The crate is built on the same machine that runs the code, so it's unlikely that the issue arises from compiling on a different architecture, or on a CPU with different features than the one running the code.

Additionally, while running cargo test on a fresh checkout of https://github.com/awesomized/crc-fast-rust is all-green, running it with the same feature flag used by aws-smithy-checksums (cargo test -F optimize_crc32_auto) gets two failed tests (lib::test_medium_lengths, lib::test_large_lengths).

This is on a r6idn.8xlarge machine (Xeon Platinum 8375C)

[GPSnoopy]

The build.rs file in crc-fast looks suspicious:

    // no auto-optimize enabled, return and use the internal Rust implementation
    #[cfg(feature = "optimize_crc32_auto")]
    {
        // for auto, default to the best available implementation based on CPU features

        // in build scripts, the target architecture is only available via an environment variable
        let target_arch = env::var("CARGO_CFG_TARGET_ARCH").unwrap();
        if "x86" == target_arch {
            // this is the only one supported on 32-bit x86 systems
            crate::build_sse_v4s3x3()
        }

        #[cfg(any(target_arch = "x86_64", target_arch = "x86"))]
        if is_x86_feature_detected!("vpclmulqdq")
            && is_x86_feature_detected!("avx512vl")
            && is_x86_feature_detected!("avx512f")
        {
            return build_avx512_vpclmulqdq_v3x2();
        }

        #[cfg(any(target_arch = "x86_64", target_arch = "x86"))]
        if is_x86_feature_detected!("avx512vl")
            && is_x86_feature_detected!("avx512f")
            && is_x86_feature_detected!("pclmulqdq")
        {
            return crate::build_avx512_v4s3x3();
        }

        #[cfg(any(target_arch = "x86_64", target_arch = "x86"))]
        if is_x86_feature_detected!("sse4.2") && is_x86_feature_detected!("pclmulqdq") {
            crate::build_sse_v4s3x3()
        }
    }

The build script does runtime detection of CPU features to do a build, rather than using the compiler target. You cannot trust that the binaries build on one machine will work on another. Given the numbers of download and users of crc-fast, I think that AWS SDK jumping to use this crate might be premature.

[atwam]

I think that's exactly right: we are compiling on a AVX512 machine (r6idn Xeon Platinum 8375C), but the running machine (despite also being x64) is a m6a.2xlarge (AMD EPYC 7R13 Processor) and does not support AVX512.

I can think of two ways to get out of this:

• Either temporarily remove optimize_crc32_auto feature flag so that the default rust implementation is used. Then separately do a PR on crc-fast to bundle various code-paths that get enabled at runtime based on available features.
• ... or revert to the previous version of checksum, slightly slower but using a proven crate.

Also, I can confirm that upon reverting just the aws-smithy-checksums to a previous version, we don't encounter the SIGILL anymore.

[jlizen]

this seems more like a cargo lambda bug than an AWS bug - I've raised another bug report cargo-lambda/cargo-lambda#859 linking to this one

I would consider the cargo lambda breakage as a convenient reproducer rather than a root cause to fix here. Since, other packages are just as likely to enable similar crc-fast build flags if their caller wants to use simd related features. The aws-sdk needs to be resilient to these usage modes rather than playing whack-a-mole with disabling the flags on common sibiling dependencies.

If not in the aws-sdk itself, this should be addressed upstream in crc-fast.

There is a nice guide to reproducing here:
cargo-lambda/cargo-lambda#859 (comment)

[atwam]

Indeed, I just added my comments to this issue because I believe breakage is related. But in our case, this has nothing to do with cargo lambda, and everything to do with the fact that crc-fast does not take into account compilation targets and rather relies on the cpu features of the build machine.

[calavera]

... or revert to the previous version of checksum, slightly slower but using a proven crate.

I would advocate for this path until it has been proven that crc-fast is reliable. I'm sure this change is breaking or will break a lot of production usage.

[onethumb]

Author of crc-fast here...

• Either temporarily remove optimize_crc32_auto feature flag so that the default rust implementation is used. Then separately do a PR on crc-fast to bundle various code-paths that get enabled at runtime based on available features.

I think this is the correct approach in the short-term.

I've done extensive cross-compiling to/from various architectures, but apparently haven't yet covered all the edge cases (and/or don't fully understand the Rust mechanisms to improve this).

Removing the feature flag (there's a reason it's a feature flag in the first place! 😄 ) will use the Rust implementation (which is still extremely fast).

I'm traveling today, but will make a PR to that effect when I find some time if someone doesn't beat me to the punch. 👍

[LegoWolf]

The build.rs file in crc-fast looks suspicious:
The build script does runtime detection of CPU features to do a build, rather than using the compiler target. You cannot trust that the binaries build on one machine will work on another. Given the numbers of download and users of crc-fast, I think that AWS SDK jumping to use this crate might be premature.

Yeah, that was my gut feeling too. I tried re-running my repro steps in Amazon Linux 2023 in a Docker container on my M1 Mac, and the crc-fast compilation failed even more explicitly because of another error to do with architecture selection:

warning: crc-fast@1.2.1: error: unknown CPU: 'armv8.2'
Here's the full build log: cargo-lambda-build-m1-arm.log

This error seems different from the one that I reported in the original bug description, because it is picking a valid architecture for the platform, but somehow it's picking one that either cc-rs or the C compiler don't recognize. Not sure why because it's a valid architecture for gcc and that's what I have installed. Should I log this behaviour as a separate bug?

The takeaway for me is that I agree with the thread that one of the previously suggested fallbacks is best for now until these edge cases can be addressed. I do look forward to seeing the performance impact of crc-fast when that's done! :)