[Python] pyarrow 24.0.0 regression: co-loading a native extension bundling mimalloc v3 SIGSEGVs in bundled mimalloc at interpreter teardown (macOS arm64, py3.14); 22/23 unaffected

[kkollsga]

Describe the bug, including details regarding any error messages, version, and platform.

Environment: macOS 15 arm64, CPython 3.14.3 (python.org), pyarrow from PyPI.

python3.14 -c "import pyarrow; import kglite; kglite.KnowledgeGraph()"
# pyarrow 24.0.0 → exit 139 (SIGSEGV at teardown); 23.0.1 / 22.0.0 → exit 0

kglite is a PyO3/abi3 Rust extension that statically links mimalloc v3 as its Rust global allocator (a very common Rust-wheel pattern). It links no Arrow code and exports one symbol. Import alone (no pyarrow API use, no kglite↔pyarrow data exchange) arms the crash; ARROW_DEFAULT_MEMORY_POOL=system does not help; os._exit(0) avoids it (teardown-only).

Crash: EXC_BAD_ACCESS inside libarrow's bundled mimalloc (_mi_theap_collect_retired + 124, lldb). Process census via MIMALLOC_VERBOSE: CPython 3.14 vendors mimalloc v3.3.2, the extension carries v3.3.1, libarrow 2400 carries a v2-line copy. Switching the extension's copy to v2 — or downgrading pyarrow to ≤23.0.1 — eliminates the crash; so something introduced in 24.0.0 (bundled-mimalloc bump? teardown/atexit registration change?) makes the pre-existing three-allocator situation fatal.

Impact: any Rust wheel shipping a mimalloc-v3 global allocator (the mimalloc crate defaults to v3 since libmimalloc-sys 0.1.47) can now crash any Python ≥3.13 process it shares with pyarrow 24 — at exit, i.e. flaky-looking CI failures and exit-139s with no traceback.

Cross-filed with microsoft/mimalloc (instance-isolation angle): microsoft/mimalloc#1327

Minimal repro wheels + lldb transcript available on request. A coexistence canary in Arrow's Python CI (import pyarrow alongside a mimalloc-v3 wheel, assert clean exit) would catch this class before release; we've added the mirror-image canary on our side.

Component(s)

Python

[raulcd]

Could you share a gdb stacktrace?

[kkollsga]

I'm on macOS (arm64), so here is the crash data from the macOS crash reporter, which unwinds the full teardown chain:

EXC_BAD_ACCESS (SIGSEGV) KERN_INVALID_ADDRESS at 0x0000000000000017

#0 libarrow.2400.dylib  _mi_theap_collect_retired + 124
#1 libarrow.2400.dylib  mi_theap_collect_ex + 72
#2 libarrow.2400.dylib  mi_process_done + 76
#3 libsystem_c.dylib    __cxa_finalize_ranges + 480
#4 libsystem_c.dylib    exit + 44
#5 libdyld.dylib        dyld4::LibSystemHelpers::exit(int) const + 20
#6 dyld                 dyld4::LibSystemHelpersWrapper::exit(int) const + 172
#7 dyld                 start + 7232

So libarrow's bundled mimalloc crashes in its mi_process_done atexit handler during __cxa_finalize, walking a thread-heap page list that contains a corrupted entry.

Supporting detail from the same macOS crash report:

• Faulting instruction (lldb disasm): ldrb w10, [x21, #0x16] with register x21 = 0x1 → fault address 0x17. The page pointer being walked is literally 0x1.
• vmRegionInfo: "0x17 is not in any region" — a wild near-null pointer, not a mapped-but-protected page.
• Register state at fault: x19 = 0x1195300c0, x20 = 0x1, x21 = 0x1, x24 = 0x9, x25 = 0x4a — consistent with a list walk hitting a sentinel-like value (0x1) where a mi_page_t* was expected. Possibly relevant: with multiple mimalloc instances in-process, one instance's static empty-page sentinel is a foreign address to the other.

Two additional data points that may help localize it:

• Under heavy allocation the same corruption also fires mid-operation rather than at exit (a second crash report shows the fault inside the co-loaded extension during object construction, KERN_PROTECTION_FAILURE on a stack guard page). Teardown is just the most reproducible manifestation.
• Version matrix: the crash requires pyarrow 24.0.0 and a co-loaded extension linking mimalloc v3, on CPython ≥3.13 (which vendors mimalloc v3 itself). pyarrow 23.0.1/22.0.0 are clean with the identical extension, and rebuilding the extension on mimalloc v2 is also clean against 24.0.0 — so whatever changed in 24.0.0's bundled allocator (or its teardown registration) interacts specifically with a second v3 instance in the process.

[raulcd]

cc @pitrou

[pitrou]

PyArrow 25.0.0 will ship with mimalloc 3.3.1, so perhaps the issue will be solved then? (though it would be nice to ensure that this doesn't regress)

You can install a development wheel of PyArrow to try it out from here:
https://anaconda.org/channels/scientific-python-nightly-wheels/packages/pyarrow/overview

[kkollsga]

Nightly (25.0.0.dev257) tested: crash gone. The extension that reliably SIGSEGVs with 24.0.0 is clean across repeated runs, both import orders.

The mechanism is mimalloc ≥ 3.2.6's fixed macOS TLS slots, shared by all co-resident v3 instances with no ownership check (microsoft/mimalloc#1327). 24.0.0's 3.2.7 crashes against a 3.3.1 extension because their heap layouts differ; 25.0.0's 3.3.1 coexists because the layouts happen to match. A pairing where layouts diverge again would re-expose the crash. Bundled ≥ 3.2.6 is the exposure window, which is why 23.0.0 with 3.1.5 was unaffected.

[pitrou]

Based on microsoft/mimalloc#1327 (comment) and the following messages, it seems we could enable MI_TLS_MODEL_LOCAL when building Arrow's mimalloc, at least on Apple.