Regression test kernel_regress:skx_avx core dumped with a bus error on FreeBSD on i386

[monwarez]

When attempting to build OpenBLAS 0.3.33 on FreeBSD from within the port on a i386 chroot (jails), the last regression test crashes.

./openblas_utest
TEST 1/125 min:smin_negative [OK]
TEST 2/125 min:dmin_positive [OK]
TEST 3/125 min:smin_zero [OK]
...
...
TEST 124/125 potrf:smoketest_trivial [OK]
TEST 125/125 kernel_regress:skx_avx gmake[1]: *** [Makefile:89: run_test] Bus error (core dumped)
gmake[1]: Leaving directory '/wrkdirs/usr/ports/math/openblas/work/OpenBLAS-0.3.33/utest'

See also the original report here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=293087

[agrajag9]

It seems to have been introduced by commit d9bb8f3

TEST 125/125 kernel_regress:skx_avx gmake[1]: *** [Makefile:86: run_test] Bus error (core dumped)
gmake[1]: Leaving directory '/wrkdirs/usr/ports/math/openblas/work/OpenBLAS-0.3.30/utest'
gmake: *** [Makefile:177: tests] Error 2
*** Error code 1

I was able to get debug symbols to build into the binaries by adding DEBUG_FLAGS to FFLAGS in FreeBSD's ports recipes:

diff --git a/Mk/Features/debug.mk b/Mk/Features/debug.mk
index 6e67a2315223..c9587a82277a 100644
--- a/Mk/Features/debug.mk
+++ b/Mk/Features/debug.mk
@@ -18,6 +18,7 @@ STRIP_CMD=    ${TRUE}
 .  endif
 DEBUG_FLAGS?=  -g
 CFLAGS:=               ${CFLAGS:N-O*:N-fno-strict*} ${DEBUG_FLAGS}
+FFLAGS+=               ${DEBUG_FLAGS}
 .  if defined(INSTALL_TARGET)
 INSTALL_TARGET:=       ${INSTALL_TARGET:S/^install-strip$/install/g}
 .  endif

I could then build the port with make -DWITH_DEBUG build and have both C and Fortran debug symbols.

(gdb) bt
#0  0x00f2531b in dlarf1f (side=..., m=100, n=99, v=..., incv=1, tau=1.1416925237086324, c=..., ldc=100, work=..., _side=4) at dlarf1f.f:156
#1  0x00f2f964 in dgebd2 (m=100, n=100, a=..., lda=100, d=..., e=..., tauq=..., taup=..., work=..., info=0) at dgebd2.f:250
#2  0x00f2f078 in dgebrd (m=100, n=100, a=..., lda=100, d=..., e=..., tauq=..., taup=..., work=..., lwork=6400, info=0) at dgebrd.f:351
#3  0x00f1cdd4 in dgesvd (jobu=..., jobvt=..., m=<optimized out>, n=<optimized out>, a=..., lda=<optimized out>, s=..., u=..., ldu=<optimized out>, vt=..., ldvt=<optimized out>, work=..., lwork=<optimized out>, info=<optimized out>, _jobu=1, _jobvt=1) at dgesvd.f:1982
#4  0x00f1b631 in LAPACKE_dgesvd_work (matrix_layout=101, jobu=65 'A', jobvt=65 'A', m=100, n=100, a=0xf9b708 <m>, lda=100, s=0xf4cec8 <s>, u=0xf4d1e8 <u>, ldu=100, vt=0xf60a68 <vt>, ldvt=100, work=0x21751180, lwork=6700) at lapacke_dgesvd_work.c:113
#5  0x00f1b134 in LAPACKE_dgesvd (matrix_layout=101, jobu=65 'A', jobvt=65 'A', m=100, n=100, a=0xf9b708 <m>, lda=100, s=0xf4cec8 <s>, u=0xf4d1e8 <u>, ldu=100, vt=0xf60a68 <vt>, ldvt=100, superb=0xf87b68 <superb>) at lapacke_dgesvd.c:71
#6  0x004842d9 in __ctest_kernel_regress_skx_avx_run () at test_kernel_regress.c:34
#7  0x00477fc2 in ctest_main (argc=1, argv=0xffffdb68) at ./ctest.h:765
#8  0x004782eb in main (argc=1, argv=0xffffdb68) at utest_main.c:45

disas that subroutine to find the offending instruction:

Dump of assembler code for function dlarf1f_:
   0x00f25300 <+0>:     push   %ebp
   0x00f25301 <+1>:     mov    %esp,%ebp
   0x00f25303 <+3>:     push   %edi
   0x00f25304 <+4>:     push   %esi
   0x00f25305 <+5>:     push   %ebx
   0x00f25306 <+6>:     call   0xf0f9f4 <__x86.get_pc_thunk.bx>
   0x00f2530b <+11>:    add    $0x26a49,%ebx
   0x00f25311 <+17>:    sub    $0x4c,%esp
   0x00f25314 <+20>:    mov    0x24(%ebp),%eax
   0x00f25317 <+23>:    movd   (%eax),%xmm0
=> 0x00f2531b <+27>:    movaps %xmm0,-0x58(%ebp)

What do we have in ebp?

(gdb) info reg
eax            0xffffd918          -9960
ecx            0x21760ca0          561384608
edx            0x217514a0          561321120
ebx            0xf4bd54            16039252
esp            0xffffd554          0xffffd554
ebp            0xffffd5ac          0xffffd5ac
esi            0x1                 1
edi            0xfffffff8          -8
eip            0xf2531b            0xf2531b <dlarf1f+27>
eflags         0x210292            [ AF SF IF RF ID ]
cs             0x33                51
ss             0x3b                59
ds             0x3b                59
es             0x3b                59
fs             0x13                19
gs             0x1b                27
fs_base        0x363340            3552064
gs_base        0x21689140          560501056

ebp - 0x58:

(gdb) x (0xffffd5ac  - 0x58)
0xffffd554:     0x2160b816

That is not 16- or 32-bit aligned, which afaik is required for movaps. However, if we go up the stack to frame 4 we DO see a 16-bit aligned value.

(gdb) up
#4  0x00f1b631 in LAPACKE_dgesvd_work (matrix_layout=101, jobu=65 'A', jobvt=65 'A', m=100, n=100, a=0xf9b708 <m>, lda=100, s=0xf4cec8 <s>, u=0xf4d1e8 <u>, ldu=100, vt=0xf60a68 <vt>, ldvt=100, work=0x21751180, lwork=6700) at lapacke_dgesvd_work.c:113
113             LAPACK_dgesvd( &jobu, &jobvt, &m, &n, a_t, &lda_t, s, u_t, &ldu_t, vt_t,
(gdb) print work
$2 = (double *) 0x21751180

Unfortunately this is where I hit a wall. gdb was reporting that a lot of the variables had been "optimized out" and so we couldn't look deeper into them. But then, when I added DEBUG = 1 to OpenBLAS's Makefile.rule and again ran make -DWITH_DEBUG build all the optimizations went away, but so did the SIGBUS error - the test passed.

[martin-frbg]

Unlikely to be d9bb8f3 (which wouldn't affect DGEMM), probably one of the things recently fixed in the 32bit kernels but I can't check right now.
Is this an AMD cpu by any chance ?

[agrajag9]

For me, yes --- 5950X

I do not know if this is the case for others also experiencing this.

[agrajag9]

If it's not d9bb8f3, it is something else in between v0.30 v0.33

[martin-frbg]

I will try to look into it tonight or early tomorrow. Can you check if the problem persists with current develop branch ?
(Guess DGESVD is probably involved in the comparison branch ... the (LAPACK) commit in question changed some DAXPY/DGESV call sequences so maybe there's some roundabout way this is now hitting a movaps somewhere in ancient 32bit GotoBLAS kernels that could just as well be a movups on any remotely modern hardware

[martin-frbg]

Ofc if it is the compiler generating aligned moves from the Fortran source of LARF1F where the input is already unaligned this won't help. But I'm visiting elderly relatives and hopping around a leaky pond I'm expected to fix...

[martin-frbg]

Took me some time to set up a VM with FreeBSD/i386 (or more to the point, I naively tried to install a 32bit toolchain on a 64bit OS first).
Building the current develop branch, openblas_utest works for me with the default gcc 14.2 of FreeBSD 14.4, and also with 15.2 . I'm currently looking into a TPMV failure in the zblat2 test that goes away at -O1

[martin-frbg]

There's something fishy about this, as the gcc15 keeps stumbling over (as in ctests failing with SIGBUS) issues in common C code that has been in use for years, and the common workaround is to declare a local variable as volatile, or reduce the optimization level.

[clausecker]

I have applied the patches from #5921 to our math/openblas (version 0.3.33) and it does not fix the issue, the same unit test still fails:

It would actually surprise me if volatile had fixed this. volatile does not change the alignment of stack variables after all. I suspect the problem could be ABI related, where some parts of the project are compiled under the assumption that the stack is aligned while others do not (we do not make or guarantee this assumption on i386 FreeBSD). I recall this was recently changed on Linux, so there could be some spillover from ABI confusion.

I'll try passing -mpreferred-stack-boundary=2 to see if it helps.

The backtrace is the same as before with the problem clearly visible:

TEST 125/125 kernel_regress:skx_avx 
Program received signal SIGBUS, Bus error.
Object-specific hardware error.
0x00f482db in dlarf1f_ ()
(gdb) backtrace
#0  0x00f482db in dlarf1f_ ()
#1  0x00f52824 in dgebd2_ ()
#2  0x00f51f38 in dgebrd_ ()
#3  0xffffd72c in ?? ()
#4  0x00f3fd94 in dgesvd_ ()
#5  0x00f3e5f1 in LAPACKE_dgesvd_work ()
#6  0x00f3e0f4 in LAPACKE_dgesvd ()
#7  0x00484089 in __ctest_kernel_regress_skx_avx_run ()
#8  0x00477c74 in ctest_main ()
#9  0x00477f5b in main ()
(gdb) disass
Dump of assembler code for function dlarf1f_:
   0x00f482c0 <+0>:     push   %ebp
   0x00f482c1 <+1>:     mov    %esp,%ebp
   0x00f482c3 <+3>:     push   %edi
   0x00f482c4 <+4>:     push   %esi
   0x00f482c5 <+5>:     push   %ebx
   0x00f482c6 <+6>:     call   0xf32874 <__x86.get_pc_thunk.bx>
   0x00f482cb <+11>:    add    $0x268dd,%ebx
   0x00f482d1 <+17>:    sub    $0x4c,%esp
   0x00f482d4 <+20>:    mov    0x24(%ebp),%eax
   0x00f482d7 <+23>:    movd   (%eax),%xmm0
=> 0x00f482db <+27>:    movaps %xmm0,-0x58(%ebp)
   0x00f482df <+31>:    lea    -0xb6d308(%ebx),%eax
   0x00f482e5 <+37>:    push   $0x1
   0x00f482e7 <+39>:    push   $0x1
   0x00f482e9 <+41>:    push   %eax
   0x00f482ea <+42>:    push   0x8(%ebp)
   0x00f482ed <+45>:    call   0xf32880 <lsame_>
   0x00f482f2 <+50>:    add    $0x10,%esp
   0x00f482f5 <+53>:    movl   $0x0,-0x2c(%ebp)
   0x00f482fc <+60>:    mov    %eax,-0x48(%ebp)
   0x00f482ff <+63>:    mov    0x1c(%ebp),%eax
   0x00f48302 <+66>:    fldl   (%eax)
   0x00f48304 <+68>:    fstl   -0x40(%ebp)
   0x00f48307 <+71>:    fldz
   0x00f48309 <+73>:    fxch   %st(1)
   0x00f4830b <+75>:    fucomip %st(1),%st
   0x00f4830d <+77>:    fstp   %st(0)

[clausecker]

-mpreferred-stack-boundary=2 in FCOMMON_OPTS seems to do the trick. So this is a toolchain misconfiguration, which I'll escalate to the relevant port.
Consider removing patch #5921, it is not required.

[martin-frbg]

Great, thanks for the update. I admit that the patch was mostly a load of Voodoo to work around what I thought I could reproduce in a VM, so as not to delay the 0.3.34 release any further. Of course any self-respecting compiler is now making a lot of unnecessary noise about the volatile nature of the variables not being propagated, so I'm very happy to revert this. Would it make sense to add the Fortran compiler option (guarded by a suitable conditional), or do you expect this to be a short-lived problem with that toolchain ?