thread sanitizer setup

[bigfooted]

Hi all, what is the proper way of using the thread sanitizer? @jblueh

I just do:
./meson.py setup build -Db_sanitize=thread --optimization=0 -Ddebug=true -Denable-pywrapper=false -Denable-autodiff=false -Denable-directdiff=false -Dwith-mpi=disabled -Dwith-omp=true -Denable-cgns=false --prefix=/home/user/Codes/su2_github/su2/

But every su2 case I run gets sanitizer warnings. Or is this normal? I also first had to do some funky thing to get it to run in the first place, because else I get this:
FATAL: ThreadSanitizer: unexpected memory mapping
so I found that I had to do this to get rid of the fatal error:
sudo sysctl vm.mmap_rnd_bits=30
But then I still get a whole bunch of these warnings below for I think every testcase

~/Codes/su2_github/su2/bin/SU2_CFD incomp_cylinder.cfg
#mpirun -np 1 --bind-to numa SU2_CFD -t 4 inv_ONERAM6.cfg
...

Renumbering points (Reverse Cuthill McKee Ordering).
==================
WARNING: ThreadSanitizer: data race (pid=1306170)
  Write of size 8 at 0x72080037e8a0 by main thread:
    #0 CPrimalGrid::SetNode(unsigned short, unsigned long) ../Common/src/geometry/../../include/geometry/primal_grid/CPrimalGrid.hpp:99 (SU2_CFD+0x1401ea8) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #1 CPhysicalGeometry::SetRCM_Ordering(CConfig*) ../Common/src/geometry/CPhysicalGeometry.cpp:4592 (SU2_CFD+0x13be8fc) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #2 CDriver::InitializeGeometryFVM(CConfig*, CGeometry**&) ../SU2_CFD/src/drivers/CDriver.cpp:735 (SU2_CFD+0x1580e4) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #3 CDriver::InitializeGeometry(CConfig*, CGeometry**&, bool) ../SU2_CFD/src/drivers/CDriver.cpp:617 (SU2_CFD+0x157424) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #4 CDriver::CDriver(char*, unsigned short, int, bool) ../SU2_CFD/src/drivers/CDriver.cpp:160 (SU2_CFD+0x150eb7) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #5 CSinglezoneDriver::CSinglezoneDriver(char*, unsigned short, int) ../SU2_CFD/src/drivers/CSinglezoneDriver.cpp:38 (SU2_CFD+0x1f1ff6) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #6 main ../SU2_CFD/src/SU2_CFD.cpp:114 (SU2_CFD+0xf04ca) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)

  Previous read of size 8 at 0x72080037e8a0 by thread T12:
    #0 CPrimalGrid::GetNode(unsigned short) const ../Common/src/grid_movement/../../include/grid_movement/../geometry/primal_grid/CPrimalGrid.hpp:90 (SU2_CFD+0x22bc30) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #1 CPhysicalGeometry::SetPoint_Connectivity() [clone ._omp_fn.0] ../Common/src/geometry/CPhysicalGeometry.cpp:4451 (SU2_CFD+0x1401580) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #2 <null> <null> (libgomp.so.1+0x2279e) (BuildId: fa0c1b446610c4b7aca51e1183050acf5aa79503)

  Location is heap block of size 24 at 0x72080037e8a0 allocated by main thread:
    #0 operator new[](unsigned long) ../../../../src/libsanitizer/tsan/tsan_new_delete.cpp:70 (libtsan.so.2+0xa70fe) (BuildId: 2a13a7710e361d06f7babbea53065ca2be93f738)
    #1 CPrimalGrid::CPrimalGrid(bool, unsigned short, unsigned short) ../Common/src/geometry/primal_grid/CPrimalGrid.cpp:31 (SU2_CFD+0x146e25f) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #2 CPrimalGridWithConnectivity<CTriangleConnectivity>::CPrimalGridWithConnectivity(bool) ../Common/src/geometry/primal_grid/../../../include/geometry/primal_grid/CPrimalGrid.hpp:470 (SU2_CFD+0x147a663) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #3 CTriangle::CTriangle(unsigned long, unsigned long, unsigned long) ../Common/src/geometry/primal_grid/CTriangle.cpp:37 (SU2_CFD+0x14788fd) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #4 CPhysicalGeometry::LoadVolumeElements(CConfig*, CGeometry*) ../Common/src/geometry/CPhysicalGeometry.cpp:2193 (SU2_CFD+0x13ac580) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #5 CPhysicalGeometry::CPhysicalGeometry(CGeometry*, CConfig*) ../Common/src/geometry/CPhysicalGeometry.cpp:238 (SU2_CFD+0x139c4b4) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #6 CDriver::InitializeGeometryFVM(CConfig*, CGeometry**&) ../SU2_CFD/src/drivers/CDriver.cpp:715 (SU2_CFD+0x157e0a) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #7 CDriver::InitializeGeometry(CConfig*, CGeometry**&, bool) ../SU2_CFD/src/drivers/CDriver.cpp:617 (SU2_CFD+0x157424) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #8 CDriver::CDriver(char*, unsigned short, int, bool) ../SU2_CFD/src/drivers/CDriver.cpp:160 (SU2_CFD+0x150eb7) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #9 CSinglezoneDriver::CSinglezoneDriver(char*, unsigned short, int) ../SU2_CFD/src/drivers/CSinglezoneDriver.cpp:38 (SU2_CFD+0x1f1ff6) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #10 main ../SU2_CFD/src/SU2_CFD.cpp:114 (SU2_CFD+0xf04ca) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)

  Thread T12 (tid=1306183, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1022 (libtsan.so.2+0x5ac1a) (BuildId: 2a13a7710e361d06f7babbea53065ca2be93f738)
    #1 <null> <null> (libgomp.so.1+0x22e1e) (BuildId: fa0c1b446610c4b7aca51e1183050acf5aa79503)
    #2 CDriver::InitializeGeometryFVM(CConfig*, CGeometry**&) ../SU2_CFD/src/drivers/CDriver.cpp:730 (SU2_CFD+0x15801d) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #3 CDriver::InitializeGeometry(CConfig*, CGeometry**&, bool) ../SU2_CFD/src/drivers/CDriver.cpp:617 (SU2_CFD+0x157424) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #4 CDriver::CDriver(char*, unsigned short, int, bool) ../SU2_CFD/src/drivers/CDriver.cpp:160 (SU2_CFD+0x150eb7) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #5 CSinglezoneDriver::CSinglezoneDriver(char*, unsigned short, int) ../SU2_CFD/src/drivers/CSinglezoneDriver.cpp:38 (SU2_CFD+0x1f1ff6) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #6 main ../SU2_CFD/src/SU2_CFD.cpp:114 (SU2_CFD+0xf04ca) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)

SUMMARY: ThreadSanitizer: data race ../Common/src/geometry/../../include/geometry/primal_grid/CPrimalGrid.hpp:99 in CPrimalGrid::SetNode(unsigned short, unsigned long)
==================
==================
WARNING: ThreadSanitizer: data race (pid=1306170)
  Write of size 8 at 0x72080037e8a8 by main thread:
    #0 CPrimalGrid::SetNode(unsigned short, unsigned long) ../Common/src/geometry/../../include/geometry/primal_grid/CPrimalGrid.hpp:99 (SU2_CFD+0x1401ea8) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #1 CPhysicalGeometry::SetRCM_Ordering(CConfig*) ../Common/src/geometry/CPhysicalGeometry.cpp:4592 (SU2_CFD+0x13be8fc) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #2 CDriver::InitializeGeometryFVM(CConfig*, CGeometry**&) ../SU2_CFD/src/drivers/CDriver.cpp:735 (SU2_CFD+0x1580e4) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #3 CDriver::InitializeGeometry(CConfig*, CGeometry**&, bool) ../SU2_CFD/src/drivers/CDriver.cpp:617 (SU2_CFD+0x157424) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #4 CDriver::CDriver(char*, unsigned short, int, bool) ../SU2_CFD/src/drivers/CDriver.cpp:160 (SU2_CFD+0x150eb7) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #5 CSinglezoneDriver::CSinglezoneDriver(char*, unsigned short, int) ../SU2_CFD/src/drivers/CSinglezoneDriver.cpp:38 (SU2_CFD+0x1f1ff6) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #6 main ../SU2_CFD/src/SU2_CFD.cpp:114 (SU2_CFD+0xf04ca) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)

  Previous read of size 8 at 0x72080037e8a8 by thread T12:
    #0 CPrimalGrid::GetNode(unsigned short) const ../Common/src/grid_movement/../../include/grid_movement/../geometry/primal_grid/CPrimalGrid.hpp:90 (SU2_CFD+0x22bc30) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #1 CPhysicalGeometry::SetPoint_Connectivity() [clone ._omp_fn.0] ../Common/src/geometry/CPhysicalGeometry.cpp:4457 (SU2_CFD+0x1401799) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #2 <null> <null> (libgomp.so.1+0x2279e) (BuildId: fa0c1b446610c4b7aca51e1183050acf5aa79503)

  Location is heap block of size 24 at 0x72080037e8a0 allocated by main thread:
    #0 operator new[](unsigned long) ../../../../src/libsanitizer/tsan/tsan_new_delete.cpp:70 (libtsan.so.2+0xa70fe) (BuildId: 2a13a7710e361d06f7babbea53065ca2be93f738)
    #1 CPrimalGrid::CPrimalGrid(bool, unsigned short, unsigned short) ../Common/src/geometry/primal_grid/CPrimalGrid.cpp:31 (SU2_CFD+0x146e25f) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #2 CPrimalGridWithConnectivity<CTriangleConnectivity>::CPrimalGridWithConnectivity(bool) ../Common/src/geometry/primal_grid/../../../include/geometry/primal_grid/CPrimalGrid.hpp:470 (SU2_CFD+0x147a663) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #3 CTriangle::CTriangle(unsigned long, unsigned long, unsigned long) ../Common/src/geometry/primal_grid/CTriangle.cpp:37 (SU2_CFD+0x14788fd) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #4 CPhysicalGeometry::LoadVolumeElements(CConfig*, CGeometry*) ../Common/src/geometry/CPhysicalGeometry.cpp:2193 (SU2_CFD+0x13ac580) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #5 CPhysicalGeometry::CPhysicalGeometry(CGeometry*, CConfig*) ../Common/src/geometry/CPhysicalGeometry.cpp:238 (SU2_CFD+0x139c4b4) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #6 CDriver::InitializeGeometryFVM(CConfig*, CGeometry**&) ../SU2_CFD/src/drivers/CDriver.cpp:715 (SU2_CFD+0x157e0a) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #7 CDriver::InitializeGeometry(CConfig*, CGeometry**&, bool) ../SU2_CFD/src/drivers/CDriver.cpp:617 (SU2_CFD+0x157424) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #8 CDriver::CDriver(char*, unsigned short, int, bool) ../SU2_CFD/src/drivers/CDriver.cpp:160 (SU2_CFD+0x150eb7) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #9 CSinglezoneDriver::CSinglezoneDriver(char*, unsigned short, int) ../SU2_CFD/src/drivers/CSinglezoneDriver.cpp:38 (SU2_CFD+0x1f1ff6) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #10 main ../SU2_CFD/src/SU2_CFD.cpp:114 (SU2_CFD+0xf04ca) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)

  Thread T12 (tid=1306183, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1022 (libtsan.so.2+0x5ac1a) (BuildId: 2a13a7710e361d06f7babbea53065ca2be93f738)
    #1 <null> <null> (libgomp.so.1+0x22e1e) (BuildId: fa0c1b446610c4b7aca51e1183050acf5aa79503)
    #2 CDriver::InitializeGeometryFVM(CConfig*, CGeometry**&) ../SU2_CFD/src/drivers/CDriver.cpp:730 (SU2_CFD+0x15801d) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #3 CDriver::InitializeGeometry(CConfig*, CGeometry**&, bool) ../SU2_CFD/src/drivers/CDriver.cpp:617 (SU2_CFD+0x157424) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #4 CDriver::CDriver(char*, unsigned short, int, bool) ../SU2_CFD/src/drivers/CDriver.cpp:160 (SU2_CFD+0x150eb7) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #5 CSinglezoneDriver::CSinglezoneDriver(char*, unsigned short, int) ../SU2_CFD/src/drivers/CSinglezoneDriver.cpp:38 (SU2_CFD+0x1f1ff6) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)
    #6 main ../SU2_CFD/src/SU2_CFD.cpp:114 (SU2_CFD+0xf04ca) (BuildId: 9b9929e68a0b67fd9287c4857ce05dc9130263a8)

SUMMARY: ThreadSanitizer: data race ../Common/src/geometry/../../include/geometry/primal_grid/CPrimalGrid.hpp:99 in CPrimalGrid::SetNode(unsigned short, unsigned long)

[pcarruscag]

You probably need to compile and run with the docker containers. IRC it was necessary to build some system libraries from scratch with the right options to avoid false positives from tsan.

[pcarruscag]

If this is about the multigrid PR, if you point me to a specific case with omp issues I can take a look.

[bigfooted]

I think the real thread sanitizer issue has been fixed with some cleanup of the BEGIN/END_SU2_OMP statements. I also think that what I am seeing are false positives, but was wondering if it can be setup outside of a docker. Working with docker is torture.

[jblueh]

The tsan docker setup uses a modified build of GCC to avoid false positives. You can do the same on your local machine by following the steps from https://github.com/su2code/Docker-Builds/blob/master/build/Dockerfile.tsan

[jblueh]

Regarding the error with unexpected memory mapping, we also had to work around ASLR stuff in the CI pipeline in the same fashion #2242 #2341.

[bigfooted]

Thanks, Johannes!