icp: Port AVX2 implementation of aes-gcm from BoringSSL

[lowjoel]

Motivation and Context

Zen 3 CPUs support the VAES and VPCLMULDQ instructions which extend the width of each instruction from 128-bits to 256-bits. BoringSSL has recently implemented this version for AES-GCM and it provides up to a 80% speedup. See google/boringssl@3b6e1be.

Description

I've backported the implementation from BoringSSL, adapting code from google/boringssl@3b6e1be (but picking the tip of master), as well as from google/boringssl@62f9751 which changed the primitive signature from 6 arguments to 7 (by not implicitly relying on the address offset of the ghash structure.)

Adaptations for icp (akin to #9749) as well as to use the RET macro for kernel code are in the third commit.

The fifth to seventh commits combine the use_avx/use_avx2 flags into an enum, allowing toggling of the different implementations that are available. Also, define different values of CAN_USE_GCM_ASM to indicate various levels of compiler support.

How Has This Been Tested?

Compile tested.

I'm now running it on my ZFS-on-root main machine.

@robn has a Wycheproof test set (#17089) that has been merged; these changes pass.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[lowjoel]

Hey @AttilaFueloep I tried to port your changes in #9749 and I can't figure out what you mean by the change with the comment // ICP does not zero key schedule. in the modified assembly sources. I've ported the other two changes, for the round offset in the AES_KEY struct, as well as the OpenSSL vs ICP representation of number of AES rounds. I've also fixed the gcm_init_vpclmulqdq_avx2 method which you mentioned ICP stores H in network order.

[robn]

Strong opening move! I've had a little play with it, here's what I got.

Before the final thing, you should run make checkstyle, and fix the errors it throws out.

I added this patch to let me see what it thinks is happening on my test machines:

diff --git module/zcommon/simd_stat.c module/zcommon/simd_stat.c
index d82a88ca9..da557bbb0 100644
--- module/zcommon/simd_stat.c
+++ module/zcommon/simd_stat.c
@@ -117,6 +117,10 @@ simd_stat_kstat_data(char *buf, size_t size, void *data)
 		    "pclmulqdq", zfs_pclmulqdq_available());
 		off += SIMD_STAT_PRINT(simd_stat_kstat_payload,
 		    "movbe", zfs_movbe_available());
+		off += SIMD_STAT_PRINT(simd_stat_kstat_payload,
+		    "vaes", zfs_vaes_available());
+		off += SIMD_STAT_PRINT(simd_stat_kstat_payload,
+		    "vpclmulqdq", zfs_vpclmulqdq_available());
 
 		off += SIMD_STAT_PRINT(simd_stat_kstat_payload,
 		    "osxsave", boot_cpu_has(X86_FEATURE_OSXSAVE));

With that, my old Intel 2019 junker laptop says:

# grep -E 'vaes|vpclmulqdq' /proc/spl/kstat/zfs/simd
vaes                    0
vpclmulqdq              0

My much nicer Ryzen 5 from last year says:

# grep -E 'vaes|vpclmulqdq' /proc/spl/kstat/zfs/simd
vaes                    1
vpclmulqdq              1

Unfortunately we don't have visibility on the ICP microbenchmarks like we do for checksums and raidz, but we can at least see the options available:

# cat /sys/module/zfs/parameters/icp_gcm_impl
cycle [fastest] avx avx2 generic pclmulqdq

So I'd say it's all wired up right, which is half the fun.

I set it to avx and created a dataset, then unmounted and exported. Then I set it to avx2 and tried to zfs load-key, but that didn't work:

# zfs load-key -a
Enter passphrase for 'tank/enc':
Key load error: Incorrect key provided for 'tank/enc'.

Trying to create the pool and dataset with avx2 selected gets a crash:

# zpool create tank -O encryption=aes-256-gcm -O keyformat=passphrase /home/robn/blk
Enter new passphrase:
Re-enter new passphrase:
Killed

And the kernel has a nice complaint:

[ 2769.438918] BUG: unable to handle page fault for address: ffff9c23c8307e10
[ 2769.439133] #PF: supervisor read access in kernel mode
[ 2769.439290] #PF: error_code(0x0000) - not-present page
[ 2769.439447] PGD 110e01067 P4D 110e01067 PUD 0
[ 2769.439588] Oops: 0000 [#1] PREEMPT SMP NOPTI
[ 2769.439733] CPU: 0 PID: 73114 Comm: zpool Tainted: P           OE      6.1.0-25-amd64 #1  Debian 6.1.106-3
[ 2769.440035] Hardware name: FreeBSD BHYVE/BHYVE, BIOS 14.0 10/17/2021
[ 2769.440236] RIP: 0010:aes_gcm_dec_update_vaes_avx2+0x3e/0x5e0 [zfs]
[ 2769.440596] Code: 0c 24 c4 e2 71 00 c8 c4 42 7d 5a 18 c4 62 25 00 d8 44 8b 91 78 01 00 00 46 8d 14 95 f0 ff ff ff 4e 8d 5c 91 60 c4 62 7d 5a 09 <c4> 42 7d 5a 13 c5 25 fe 1d d5 ed 0b 00 48 83 fa 7f 0f 8
6 31 03 00
[ 2769.441171] RSP: 0018:ffffb1b6c37df358 EFLAGS: 00010086
[ 2769.441339] RAX: ffffffffc089b140 RBX: ffffb1b6c37df43c RCX: ffff9c214b6eae00
[ 2769.441566] RDX: 0000000000000060 RSI: ffff9c208e918500 RDI: ffff9c208e918500
[ 2769.441793] RBP: 0000000000000006 R08: ffffb1b6c37df430 R09: ffff9c20e8eba3c0
[ 2769.442019] R10: 000000009f3073ec R11: ffff9c23c8307e10 R12: ffffb1b6c37df498
[ 2769.442102] R13: ffffffffc089b140 R14: ffffb1b6c37df420 R15: ffffb1b6c37df488
[ 2769.442102] FS:  00007f2d7ead5840(0000) GS:ffff9c23afc00000(0000) knlGS:0000000000000000
[ 2769.442102] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2769.442102] CR2: ffff9c23c8307e10 CR3: 00000001cccaa000 CR4: 00000000003506f0
[ 2769.442102] Call Trace:
[ 2769.442102]  <TASK>
[ 2769.442102]  ? __die_body.cold+0x1a/0x1f
[ 2769.442102]  ? page_fault_oops+0xd2/0x2b0
[ 2769.442102]  ? srso_alias_return_thunk+0x5/0x7f
[ 2769.442102]  ? search_bpf_extables+0x5b/0x80
[ 2769.442102]  ? exc_page_fault+0xca/0x170
[ 2769.442102]  ? asm_exc_page_fault+0x22/0x30
[ 2769.442102]  ? aesni_gcm_decrypt_avx+0x10/0x10 [zfs]
[ 2769.442102]  ? aesni_gcm_decrypt_avx+0x10/0x10 [zfs]
[ 2769.442102]  ? aes_gcm_dec_update_vaes_avx2+0x3e/0x5e0 [zfs]
[ 2769.442102]  ? aesni_gcm_decrypt_avx2+0x2a/0x50 [zfs]
[ 2769.442102]  ? gcm_decrypt_final_avx+0x167/0x460 [zfs]
[ 2769.442102]  ? crypto_update_uio+0xc0/0x120 [zfs]
[ 2769.442102]  ? aesni_gcm_decrypt_avx+0x10/0x10 [zfs]
[ 2769.442102]  ? aes_decrypt_atomic+0x1ca/0x310 [zfs]
[ 2769.442102]  ? crypto_decrypt+0x78/0x1c0 [zfs]
[ 2769.442102]  ? zio_do_crypt_uio+0x2ce/0x400 [zfs]
[ 2769.442102]  ? zio_crypt_key_unwrap+0x254/0x480 [zfs]
[ 2769.442102]  ? dsl_crypto_key_open.constprop.0+0x2d1/0x350 [zfs]
[ 2769.442102]  ? dsl_crypto_key_open.constprop.0+0x2d1/0x350 [zfs]
[ 2769.442102]  ? spa_keystore_dsl_key_hold_dd+0x12e/0x280 [zfs]
[ 2769.442102]  ? __kmalloc_node+0x4c/0x150
[ 2769.442102]  ? spa_keystore_create_mapping+0x79/0x200 [zfs]
[ 2769.442102]  ? dsl_dataset_hold_obj_flags+0x48/0x90 [zfs]
[ 2769.442102]  ? dsl_pool_create+0x1bc/0x490 [zfs]
[ 2769.442102]  ? spa_create+0x83d/0xdb0 [zfs]
[ 2769.442102]  ? zfs_ioc_pool_create+0xab/0x310 [zfs]
[ 2769.442102]  ? zfsdev_ioctl_common+0x6a0/0x7c0 [zfs]
[ 2769.442102]  ? __kmalloc_node+0xbf/0x150
[ 2769.442102]  ? srso_alias_return_thunk+0x5/0x7f
[ 2769.442102]  ? zfsdev_ioctl+0x4f/0xd0 [zfs]
[ 2769.442102]  ? __x64_sys_ioctl+0x90/0xd0
[ 2769.442102]  ? do_syscall_64+0x55/0xb0

That's all I have time for tonight. This is a good start!

[robn]

Oh the other thing I forgot to add, after loading the module it says fastest, and fails as above. So it is selecting avx2 as fastest, which is good! (we need a stat for it, for sure).

[lowjoel]

😅 I can't please spdxcheck and can't figure out how to -- any pointers for the contrib directory? I tried how Attila did it in his PR and it didn't pass either.

[AttilaFueloep]

Regarding the license tag, I'd tag it as OpenSSL-standalone and add the file below that tag in the %override_file_license_tags hash inside scripts/spdxcheck.pl. Legalese is way over my head though, maybe @robn has some insight on this.

Regarding the location I'd prefer feedback from the maintainers since I was only guessing.

[lowjoel]

Oh, that was it - path_license_tags has an exclusion for module/icp. Once @behlendorf can chime in if the licensing is right (and the contrib dir is the right place for the source assembly files) I can fix both at once.

[behlendorf]

@lowjoel putting the original assembly files under contrib/icp/ works for me. As for the licensing that also looks good. You'll just need to add that directory and license to path_license_tags.

[behlendorf]

Merged. Thanks guys for all your work on this.