Possible regression: cblas_dspmv test fails in v0.3.32

[hmenke]

Observed on Ubuntu 24.04 running on aarch64 as part of another build: https://github.com/flathub-infra/vorarbeiter/actions/runs/23469172794/job/68288026035

 cblas_dspmv  PASSED THE TESTS OF ERROR-EXITS

 ******* FATAL ERROR - COMPUTED RESULT IS LESS THAN HALF ACCURATE *******
           EXPECTED RESULT   COMPUTED RESULT
       1      0.667230          0.667230    
       2      -1.16658          -1.16658    
       3      0.210145          0.210145    
       4      0.543602          0.543602    
       5     -0.675359         -0.675359    
       6       1.33709           1.33709    
       7      0.668852          0.668852    
       8       1.70144           1.70144    
       9     -0.375509         -0.375509    
      10       1.21933           1.21933    
      11      -1.99305          -1.99305    
      12      0.306078          0.306078    
      13       1.00700           1.00700    
      14       1.29749           1.29749    
      15      0.503632          0.503632    
      16      0.624915          0.624915    
      17      -1.05779          -1.05779    
      18       3.71865           3.71865    
      19      -1.45335          -1.45335    
      20       2.15959           2.15959    
      21      -1.31750          -1.31750    
      22     -0.331946         -0.331946    
      23     -0.397639         -0.397639    
      24     -0.898512         -0.898512    
      25     -0.143124         -0.143124    
      26      0.763038          0.763038    
      27     -0.184042         -0.184042    
      28      -1.33639          -1.33639    
      29      -1.81842          -1.81842    
      30     -0.639056         -0.639056    
      31      -1.73327          -1.73327    
      32     -0.603902         -0.603902    
ERROR STOP 

Error termination. Backtrace:
      33     -0.602854          0.273770    
      34     -0.638333         -0.231240    
      35     -0.985313         -0.418380    
      36     -0.386861         -0.189558    
      37      -2.50400          -1.97703    
      38     -0.100883          0.456060    
      39     -0.482794         -0.195581    
      40      0.279730           1.12638    
      41     -0.869296         -0.192472    
      42      0.548802E-01      0.262173    
      43     -0.304101          0.132961    
      44      -1.02007         -0.652936    
      45       1.02721           1.35438    
      46     -0.275883          0.812599E-01
      47      0.605179          0.692592    
      48     -0.218912E-01      0.495092    
      49      0.751895E-01      0.722043    
      50     -0.489880         -0.482387    
      51     -0.759316         -0.522053    
      52      0.139411          0.306744    
      53     -0.798163         -0.145958E-02
      54     -0.303303         -0.175930    
      55     -0.172739          0.713874    
      56       1.07911           1.39629    
      57      0.393113E-01      0.486364    
      58       1.35710           1.63432    
      59     -0.688482          0.118211    
      60      0.847893           1.81443    
      61      0.881102           1.47801    
      62       5.50897           6.43554    
      63     -0.112485          0.844059    
 ******* cblas_dspmv  FAILED ON CALL NUMBER:
   1445: cblas_dspmv (    CblasUpper, 63, 1.0, AP, X, 1, 0.0, Y, 1) .
 ******* cblas_dspmv  FAILED ON CALL NUMBER:
      2: cblas_dspmv (    CblasUpper,  1, 0.0, AP, X, 1, 0.0, Y, 1) .

 ******* FATAL ERROR - TESTS ABANDONED *******
#0  0xff7b4415fc07 in ???
#1  0xff7b44160973 in ???
#2  0xff7b4416214f in ???
#3  0xaaf9666bf013 in ???
#4  0xaaf9666b4fc3 in ???
#5  0xff7b43ee61d3 in ???
#6  0xff7b43ee62b7 in ???
#7  0xaaf9666b50af in _start
	at ../sysdeps/aarch64/start.S:98
#8  0xffffffffffffffff in ???

[martin-frbg]

Very curious as there weren't that many changes in 0.3.32, and if I read your log correctly, the job ran on Neoverse N1 hardware, which is covered by OpenBLAS' CI . Which compiler is your build using, please ?

[hmenke]

The compiler is GCC 14.3.0 (part of freedesktop-sdk 24.08).

[martin-frbg]

Seems to be connected to the DDOT kernel - sometimes it works, sometimes it doesn't, workaround appears to be to compile it at -O1 maximum. This would suggest it it connected to the very recent fix (#5691) for SDOT malfunctions on NEOVERSEV2, where the dot product was apparently optimized away unless the return variable was declared as volatile. Now that very same declaration appears to cause malfunction on other cpus when the same code is used for double precision ... there's something crazily wrong with this.

Perhaps there was something wrong with the much earlier change from #3845 that "removed unnecessary instructions" from that assembly kernel, though the change was proposed by an ARM guy @Mousius

[martin-frbg]

For a quick temporary fix - either change DDOTKERNEL in kernel/arm64/KERNEL.NEOVERSEN1 to "../arm/dot.c" (the generic C implementation), or edit kernel/arm64/dot_kernel_asimd.c to change the single declaration of "volatile RETURN_TYPE dot = 0.0;" into

#ifndef DOUBLE
volatile
#endif
  RETURN_TYPE dot = 0.0;

[Sbozzolo]

Is it possible that some other regression was introduced recently or that the workaround does not fully capture all the failing cases?

Our CI was running with OpenBlas 0.3.29. I skipped updating to 0.3.30 because of build failures (possibly this #5459, and I skipped 0.3.31 because it was not packaged with Spack. I now updated OpenBlas to 0.3.32 and now some regression tests are failing on armv8.4a (numbers do not match reference results). I tried with openblas@develop to include #5710, but the regression tests are still failing.

This is on a c7g AWS instance, with gcc 13 and with llvm 19.

Thank you in advance

[martin-frbg]

The one "suspicious" change I mentioned above was merged in 0.3.22, so is unlikely to be related. Please do not expect me to dig through yor project's CI logs at this time.

[martin-frbg]

@Sbozzolo 0.3.31 had several updates to SGER/DGER, SAXPY/DAXPY, 0.3.30 added GEMV and SYMV kernels , it would help if you could identify BLAS functions involved in the failing tests, or isolate a failing test from your CI.
(For testing, it should be possible to simply comment out (#) the GER and AXPY entries in kernel/arm64/KERNEL.NEOVERSEV1 and rebuild OpenBLAS, and/or to swap in the SGEMVNKERNEL/SGEMVTKERNEL declarations from KERNEL.NEOVERSEN1 - SYMV kernel entries would need to be changed (i.e. borrowed from NEOVERSEN1) in kernel/arm64/KERNEL.ARMV8SVE )

[Sbozzolo]

@martin-frbg, thank you for your suggestions.

I think I tried your suggestions of commenting out/swapping the kernels, but I still found failures. So, I bisected OpenBlas versions and found that 0.3.30 was already leading to failures. I git bisected between 0.3.29 and 0.3.30 and found that f1e628b seems to be the culprit.

I'll try digging further, but I don't know how much I will be able to do.

This was my git bisect script (with Spack and Palace):

# Install directly into the spack openblas prefix
source /data/home/gbozzola/spack/share/spack/setup-env.sh
spack env activate /data/home/gbozzola/prod_palace_neoverse_v1_openblas
INSTALLDIR=$(spack location -i openblas)

echo "=== Building OpenBLAS at $(git rev-parse --short HEAD) ==="
echo "=== Installing to $INSTALLDIR ==="

# Clean and build
make -j$(nproc) TARGET=NEOVERSEV1 clean 2>/dev/null || true
make -j$(nproc) TARGET=NEOVERSEV1 2>&1 | tail -3

if [ ${PIPESTATUS[0]} -ne 0 ]; then
    echo "BUILD FAILED — skipping this commit"
    exit 125  # git bisect skip
fi

make TARGET=NEOVERSEV1 PREFIX=$INSTALLDIR install 2>&1 | tail -3

# Source modules
source /etc/profile.d/modules.sh 2>/dev/null || source /usr/share/modules/init/bash 2>/dev/null || true
module load julia

echo "=== Running Palace dipole test ==="
cd /data/home/gbozzola/palace/test/examples

julia --project runtests.jl --test-cases "antenna/antenna_short_dipole" 2>&1 | tail -20
RESULT=${PIPESTATUS[0]}

if [ $RESULT -eq 0 ]; then
    echo "=== PASS (good commit) ==="
    exit 0
else
    echo "=== FAIL (bad commit) ==="
    exit 1
fi

[martin-frbg]

Great, thanks. I have split out your issue into the new #5735 now. Do I understand correctly that there are further problems beyond the GEMV kernel commit that you flagged ?

[Sbozzolo]

Great, thanks. I have split out your issue into the new #5735 now. Do I understand correctly that there are further problems beyond the GEMV kernel commit that you flagged ?

No, that's the only issue.