seg fault when compiling in HPC mirror configuration on later linux version (>= ubuntu 22)

[petersilva]

While building a test case to model hpc-mirroring, I found that when trying to build some code (written in C) the compiler crashes (segmentation fault.) From the messages, I think this is related to link processing added in the latest version.

To Reproduce:

git clone https://github.com/MetPX/sr3-examples

• follow setup instructions.
• go into ~/fakeplace, git clone https://github.com/MetPX/sarrac metpx-sr3c
• make

It will core fault compiling sr_post.c

Isolating it a bit more:

sr3 edit cpost/mirror
  debug on
 change *logLevel info* to *logLevel debug*

${iedir}/run_profile.sh

fractal% LIBCLOCATION="`ldd /bin/sh | awk '/libc\.so\./ { print; }' | cut -d' ' -f3`"
fractal% CFLAGS="-DHAVE_JSONC -DSR_APPNAME=\"sr3\" -DFORCE_LIBC_REGEX=\"${LIBCLOCATION}\" -fPIC -ftest-coverage -std=gnu99 -Wall -g -D_GNU_SOURCE -DINTERCEPT_SYSCALL"

fractal% cc -c ${CFLAGS} sr_post.c
2025-01-20 14:53:08,266 [INFO] logEvents option not implemented, ignored.
2025-01-20 14:53:08,267 [DEBUG] shim 3.24.12rc2 settings: action=foreground, hostname=fractal, config_name=mirror
2025-01-20 14:53:08,267 [DEBUG] 	follow_symlinks=no, realpath: Adjust=-1, Filter=yes, Post=yes
2025-01-20 14:53:08,267 [DEBUG] 	force_polling=off, sleep=0, expire=172800, housekeeping=300, sanity_log_dead=450
2025-01-20 14:53:08,267 [DEBUG] 	nodupe_ttl=0, nodupe_fileAgeMin=0, nodupe_fileAgeMax=2.592e+06,
2025-01-20 14:53:08,267 [DEBUG] 	cache_file=off, cache_basis=path, accept_unmatch=on, messageRateMax=0
2025-01-20 14:53:08,267 [DEBUG] 	loglevel=6, logReject=on, logRotateCount=5, logRotateInterval=86400
2025-01-20 14:53:08,267 [DEBUG] 	fileEvents=006f, directory=(null), queuename=(null), sum=n
2025-01-20 14:53:08,267 [DEBUG] 	statehost=no, v2compatRenameDoublePost=no
2025-01-20 14:53:08,267 [DEBUG] 	message_ttl=0, post_exchange=xs_tfeed_winnow_mirror_primary_to_secondary, post_exchangeSplit=5, post_exchangeSuffix=(null)
2025-01-20 14:53:08,267 [DEBUG] 	source=tfeed, to=localhost, post_baseUrl=file:/, post_baseDir=/, realpath_post_baseDir=/
2025-01-20 14:53:08,267 [DEBUG] 	topicPrefix=v03, post_topicPrefix=v03, pid=-1
2025-01-20 14:53:08,267 [DEBUG] man sr3_cpost(1) for more information
2025-01-20 14:53:08,267 [DEBUG] post_broker: amqp://tfeed:<pw>@localhost:5672
2025-01-20 14:53:08,267 [DEBUG] realpath_adjust -1, /tmp -> /tmp 
2025-01-20 14:53:08,267 [DEBUG] sr_isMatchingPattern, testing string: /tmp/ccABPgyl.s vs mask: accept .*                             next=(nil)
2025-01-20 14:53:08,267 [DEBUG] sr_isMatchingPattern: /tmp/ccABPgyl.s matched mask: accept .*
2025-01-20 14:53:08,269 [DEBUG] 3.24.12rc2 connected to post broker amqp://tfeed@localhost:5672/#xs_tfeed_winnow_mirror_primary_to_secondary
2025-01-20 14:53:08,269 [DEBUG] declaring exchange amqp://tfeed@localhost:5672/#xs_tfeed_winnow_mirror_primary_to_secondary00
2025-01-20 14:53:08,269 [DEBUG] declaring exchange amqp://tfeed@localhost:5672/#xs_tfeed_winnow_mirror_primary_to_secondary01
2025-01-20 14:53:08,269 [DEBUG] declaring exchange amqp://tfeed@localhost:5672/#xs_tfeed_winnow_mirror_primary_to_secondary02
2025-01-20 14:53:08,270 [DEBUG] declaring exchange amqp://tfeed@localhost:5672/#xs_tfeed_winnow_mirror_primary_to_secondary03
2025-01-20 14:53:08,270 [DEBUG] declaring exchange amqp://tfeed@localhost:5672/#xs_tfeed_winnow_mirror_primary_to_secondary04
2025-01-20 14:53:08,270 [DEBUG] realpath_adjust -1, /tmp -> /tmp 
2025-01-20 14:53:08,270 [DEBUG] shim file2message start with: /tmp/ccABPgyl.s sb=0x7ffd460eec60 islnk=0, isdir=0, isreg=1
2025-01-20 14:53:08,270 [DEBUG] sr_post found hlink: (null)
2025-01-20 14:53:08,270 [DEBUG] sr_post regular: 
2025-01-20 14:53:08,271 [DEBUG] v03 pubTime=20250120195308.27033014
2025-01-20 14:53:08,271 [DEBUG] v03 body={ "pubTime" : "20250120T195308.27033014", "baseUrl" : "file:/", "relPath" : "tmp/ccABPgyl.s", "source" : "tfeed", "size" : "0", "atime" : "20250120T195308.26570143", "mtime" : "20250120T195308.26570143", "mode" : "600" }  

2025-01-20 14:53:08,273 [INFO] shim published: { "pubTime":"20250120195308.27033014", "baseUrl":"file:/", "relPath":"tmp/ccABPgyl.s", "topic":"v03.tmp", "mtime":"20250120195308.26570143", "atime":"20250120195308.26570143", "mode":"0600", "size":"0"}
cc1: fatal error: sr_post.c: No such file or directory
compilation terminated.
2025-01-20 14:53:08,278 [DEBUG] realpath_adjust -1, /tmp -> /tmp 
2025-01-20 14:53:08,278 [DEBUG] realpath_adjust -1, /tmp -> /tmp 
2025-01-20 14:53:08,278 [DEBUG] shim file2message start with: /tmp/ccABPgyl.s sb=(nil) islnk=0, isdir=0, isreg=0
2025-01-20 14:53:08,278 [DEBUG] sr_post found hlink: (null)
Segmentation fault (core dumped)
fractal% 

[petersilva]

sigh... the symptom is exactly like #111
It just coredumps doing

   case 'R': 
                ...
                ctx = EVP_MD_CTX_create();
                md = EVP_sha512();
                EVP_DigestInit_ex(ctx, md, NULL);
                ....

[petersilva]

summary so far:

• investigation being done on ubuntu 24.04... these segfaults in the past bugs showed up a bit on 22.04, much more often on 24.04, and not at all on 18.04.
• It dumps core in an SSL library routine, that is initializing a structure... it's seems impossible for application code to cause this...

[petersilva]

oh... and it's not a hardlink.. it's just a plain remove.

[petersilva]

It's actually much easier to reproduce. Start a mirroring session,
then try compiling a non-existent file to an object:

fractal% cc -c aaa.c
2025-01-22 13:19:21,697 [INFO] logEvents option not implemented, ignored.
2025-01-22 13:19:21,697 [DEBUG] shim 3.24.12 settings: action=foreground, hostname=fractal, config_name=mirror
2025-01-22 13:19:21,697 [DEBUG] 	follow_symlinks=no, realpath: Adjust=-1, Filter=yes, Post=yes
2025-01-22 13:19:21,697 [DEBUG] 	force_polling=off, sleep=0, expire=172800, housekeeping=300, sanity_log_dead=450
2025-01-22 13:19:21,697 [DEBUG] 	nodupe_ttl=0, nodupe_fileAgeMin=0, nodupe_fileAgeMax=2.592e+06,
2025-01-22 13:19:21,697 [DEBUG] 	cache_file=off, cache_basis=path, accept_unmatch=on, messageRateMax=0
2025-01-22 13:19:21,697 [DEBUG] 	loglevel=6, logReject=on, logRotateCount=5, logRotateInterval=86400
2025-01-22 13:19:21,697 [DEBUG] 	fileEvents=006f, directory=(null), queuename=(null), sum=n
2025-01-22 13:19:21,697 [DEBUG] 	statehost=no, v2compatRenameDoublePost=no
2025-01-22 13:19:21,697 [DEBUG] 	message_ttl=0, post_exchange=xs_tfeed_winnow_mirror_primary_to_secondary, post_exchangeSplit=5, post_exchangeSuffix=(null)
2025-01-22 13:19:21,697 [DEBUG] 	source=tfeed, to=localhost, post_baseUrl=file:/, post_baseDir=/, realpath_post_baseDir=/
2025-01-22 13:19:21,697 [DEBUG] 	topicPrefix=v03, post_topicPrefix=v03, pid=-1
2025-01-22 13:19:21,697 [DEBUG] man sr3_cpost(1) for more information
2025-01-22 13:19:21,697 [DEBUG] post_broker: amqp://tfeed:<pw>@localhost:5672
2025-01-22 13:19:21,697 [DEBUG] realpath_adjust -1, /tmp -> /tmp 
2025-01-22 13:19:21,697 [DEBUG] sr_isMatchingPattern, testing string: /tmp/cc68xyJd.s vs mask: accept .*                             next=(nil)
2025-01-22 13:19:21,697 [DEBUG] sr_isMatchingPattern: /tmp/cc68xyJd.s matched mask: accept .*
2025-01-22 13:19:21,700 [DEBUG] 3.24.12 connected to post broker amqp://tfeed@localhost:5672/#xs_tfeed_winnow_mirror_primary_to_secondary
2025-01-22 13:19:21,700 [DEBUG] declaring exchange amqp://tfeed@localhost:5672/#xs_tfeed_winnow_mirror_primary_to_secondary00
2025-01-22 13:19:21,700 [DEBUG] declaring exchange amqp://tfeed@localhost:5672/#xs_tfeed_winnow_mirror_primary_to_secondary01
2025-01-22 13:19:21,700 [DEBUG] declaring exchange amqp://tfeed@localhost:5672/#xs_tfeed_winnow_mirror_primary_to_secondary02
2025-01-22 13:19:21,700 [DEBUG] declaring exchange amqp://tfeed@localhost:5672/#xs_tfeed_winnow_mirror_primary_to_secondary03
2025-01-22 13:19:21,700 [DEBUG] declaring exchange amqp://tfeed@localhost:5672/#xs_tfeed_winnow_mirror_primary_to_secondary04
2025-01-22 13:19:21,700 [DEBUG] realpath_adjust -1, /tmp -> /tmp 
2025-01-22 13:19:21,700 [DEBUG] shim file2message start with: /tmp/cc68xyJd.s sb=0x7ffea6c48f00 islnk=0, isdir=0, isreg=1
2025-01-22 13:19:21,700 [DEBUG] sr_post found hlink: (null)
2025-01-22 13:19:21,700 [DEBUG] sr_post regular: 
2025-01-22 13:19:21,702 [DEBUG] v03 pubTime=20250122181921.70053978
2025-01-22 13:19:21,702 [DEBUG] v03 body={ "pubTime" : "20250122T181921.70053978", "baseUrl" : "file:/", "relPath" : "tmp/cc68xyJd.s", "source" : "tfeed", "size" : "0", "atime" : "20250122T181921.69602422", "mtime" : "20250122T181921.69602422", "mode" : "600" }  

2025-01-22 13:19:21,702 [INFO] shim published: { "pubTime":"20250122181921.70053978", "baseUrl":"file:/", "relPath":"tmp/cc68xyJd.s", "topic":"v03.tmp", "mtime":"20250122181921.69602422", "atime":"20250122181921.69602422", "mode":"0600", "size":"0"}
cc1: fatal error: aaa.c: No such file or directory
compilation terminated.
2025-01-22 13:19:21,706 [DEBUG] realpath_adjust -1, /tmp -> /tmp 
2025-01-22 13:19:21,706 [DEBUG] realpath_adjust -1, /tmp -> /tmp 
2025-01-22 13:19:21,706 [DEBUG] shim file2message start with: /tmp/cc68xyJd.s sb=(nil) islnk=0, isdir=0, isreg=0
2025-01-22 13:19:21,706 [DEBUG] sr_post found hlink: (null)
Segmentation fault (core dumped)
fractal%