ACL deconvolution crashes with large weights

[fadara01]

PyTorch reproducer:

python -c "import torch;torch.nn.ConvTranspose1d(2016, 1026, 1024, stride=256)(torch.rand(1, 2016, 224))"

Standalone oneDNN (v3.7.1) with ACL (v25.02) reproducer (with benchdnn):

ONEDNN_VERBOSE=profile,profile_externals ./tests/benchdnn/benchdnn --deconv mb1_ic2016oc1026_ih1oh1kh1sh1dh0ph0_iw224ow58112kw1024sw256dw0pw0

Root cause:

The root cause in ACL is a write to an invalid address on this line

The address we write to is calculated using this ptr_to_element function which calculates the offset in offset_element_in_bytes. The offset in offset_element_in_bytes is int32_t which overflows in this case because of the massive number of parameters in the problem 2016 * 1026 * 1024 (i.e. returning an offset of -2139758464).

A smaller workload like torch.nn.ConvTranspose1d(500, 1026, 1024, stride=256)(torch.rand(1, 500, 224)) doesn't crash.

Suggested fixes:

• use 64 bits for the offset (and the return type) in offset_element_in_bytes
• OR make ACL fail the validation stage if the problem is too big.

Full Stack Trace: See pytorch/pytorch#165654

[fadara01]

CC: @gunes-arm @Dongsung-arm

[fadara01]

After making sure that idx/offsets in both CpuPermuteKernel.cpp and NEReverseKernel.cpp are size_t, we no longer crash, but we get different results compared to benchdnn reference:

onednn_verbose,v1,primitive,create:cache_miss,cpu,reorder,jit:uni,undef,src:f32::blocked:a::f0 dst:f32::blocked:a::f0,,,1026,0.0861816
onednn_verbose,v1,primitive,exec,cpu,reorder,jit:uni,undef,src:f32::blocked:a::f0 dst:f32::blocked:a::f0,,,1026,2.02319
onednn_verbose,v1,primitive,create:cache_miss,cpu,reorder,jit:uni,undef,src:f32::blocked:abcd::f0 dst:f32::blocked:acdb::f0,,,1026x2016x1x1024,0.111084
onednn_verbose,v1,primitive,exec,cpu,reorder,jit:uni,undef,src:f32::blocked:abcd::f0 dst:f32::blocked:acdb::f0,,,1026x2016x1x1024,84.7129
[CHECK_MEM][ERROR]: Memory use is underestimated. Current allocation size: 24.117058 GB; expected size: 18.095520 GB.
onednn_verbose,v1,primitive,create:cache_miss,cpu,reorder,jit:uni,undef,src:f32::blocked:abcd::f0 dst:f32::blocked:acdb::f0,,,1x2016x1x224,0.0930176
onednn_verbose,v1,primitive,exec,cpu,reorder,jit:uni,undef,src:f32::blocked:abcd::f0 dst:f32::blocked:acdb::f0,,,1x2016x1x224,0.248047
onednn_verbose,v1,primitive,exec:external,NEReverseKernel,2218.59
onednn_verbose,v1,primitive,exec:external,CpuPermuteKernel,4798.14
onednn_verbose,v1,primitive,exec:external,CpuGemmAssemblyDispatch/pretranspose_B_array,75.6909
onednn_verbose,v1,primitive,exec:external,CPPUpsampleKernel,1136.1
onednn_verbose,v1,primitive,exec:external,CpuGemmAssemblyWrapperKernel/sve_hybrid_fp32_mla_6x4VL,107209
onednn_verbose,v1,primitive,exec,cpu,deconvolution,acl,forward_training,src:f32:a:blocked:acdb::f0 wei:f32:a:blocked:acdb::f0 bia:f32:a:blocked:a::f0 dst:f32:a:blocked:acdb::f0,,alg:deconvolution_direct,mb1_ic2016oc1026_ih1oh1kh1sh1dh0ph0_iw224ow58112kw1024sw256dw0pw0,132183
onednn_verbose,v1,primitive,create:cache_miss,cpu,reorder,jit:uni,undef,src:f32::blocked:acdb::f0 dst:f32::blocked:abcd::f0,,,1x1026x1x58112,0.0800781
onednn_verbose,v1,primitive,exec,cpu,reorder,jit:uni,undef,src:f32::blocked:acdb::f0 dst:f32::blocked:abcd::f0,,,1x1026x1x58112,2.24707
[   0][DST][0:0:0:0] exp_f32:         647 exp:         647 got:           1 diff:     646 rdiff:0.998454
[   1][DST][0:0:0:1] exp_f32:         -47 exp:         -47 got:           1 diff:      48 rdiff: 1.02128
[   2][DST][0:0:0:2] exp_f32:        -767 exp:        -767 got:           1 diff:     768 rdiff:  1.0013
[   3][DST][0:0:0:3] exp_f32:        -205 exp:        -205 got:           1 diff:     206 rdiff: 1.00488
[   4][DST][0:0:0:4] exp_f32:        1953 exp:        1953 got:           1 diff:    1952 rdiff:0.999488
[   5][DST][0:0:0:5] exp_f32:          50 exp:          50 got:           1 diff:      49 rdiff:    0.98
[   6][DST][0:0:0:6] exp_f32:       -1144 exp:       -1144 got:           1 diff:    1145 rdiff: 1.00087
[   7][DST][0:0:0:7] exp_f32:        2216 exp:        2216 got:           1 diff:    2215 rdiff:0.999549
[   8][DST][0:0:0:8] exp_f32:        -616 exp:        -616 got:           1 diff:     617 rdiff: 1.00162
[   9][DST][0:0:0:9] exp_f32:        -163 exp:        -163 got:           1 diff:     164 rdiff: 1.00613
[COMPARE_STATS][DST]: trh=0 err_max_diff:   21514 err_max_rdiff:    8633 all_max_diff:   21514 all_max_rdiff:    8633
0:FAILED (errors:59610685 total:59622912) (224889 ms) __REPRO: --deconv mb1ic2016ih1iw224oc1026oh1ow58112kh1kw1024sh1sw256ph0pw0
===========================================================
= Failed cases summary (--summary=no-failures to disable) =
===========================================================
0:FAILED (errors:59610685 total:59622912) (224889 ms) __REPRO: --deconv mb1ic2016ih1iw224oc1026oh1ow58112kh1kw1024sh1sw256ph0pw0
============================
tests:1 passed:0 skipped:0 mistrusted:0 unimplemented:0 invalid_arguments:0 failed:1 listed:0
total: 225.93s; create_pd: 0.00s (0%); create_prim: 0.64s (0%); fill: 0.96s (0%); execute: 132.18s (59%); compute_ref: 86.66s (38%); compare: 0.11s (0%);

Not sure if this has to do with this log message indicating that benchdnn didn't allocate enough memory for the probem?

[CHECK_MEM][ERROR]: Memory use is underestimated. Current allocation size: 24.117058 GB; expected size: 18.095520 GB.

[Sqvid]

Not sure if this has to do with this log message indicating that benchdnn didn't allocate enough memory for the probem?

[CHECK_MEM][ERROR]: Memory use is underestimated. Current allocation size: 24.117058 GB; expected size: 18.095520 GB.

This is just a diagnostic and unlikely to be related to this issue.

[morgolock]

Solved with this PR. The fix is present in the latest release of ACL: v52.6.0.

[fadara01]

Thank you, we updated ACL to v52.06 in PyTorch: pytorch/pytorch#165904 which should be picked up by the PyTorch v2.9.1 patch release.