Upstream tracking#165
Draft
grahamc wants to merge 3188 commits into
Draft
Conversation
fail() is marked noexcept so this does nothing.
Logging functions are often called in contexts where exceptions cannot be handled, such as in completion callbacks or destructors. In particular, TunnelLogger::log() could throw if the daemon client has disconnected, which turned any printError() call in a Callback completion handler into std::terminate() (e.g. via HttpBinaryCacheStore::maybeDisable() when transfers are torn down after a client hangup - Sentry issue DETERMINATE-NIX-4A). So mark log(), logEI(), warn(), startActivity(), stopActivity() and result() as noexcept, and make the implementations handle write failures instead of propagating them. TunnelLogger now drops messages when the client is gone (the connection teardown is handled elsewhere, e.g. via MonitorFdHup); the other loggers already ignored write errors via writeToStderr() or disabled themselves (JSONLogger). Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Make logging functions noexcept
The primOpCalls, functionCalls and attrSelects maps were plain unordered_flat_maps updated from evaluator threads, so running with NIX_COUNT_CALLS=1 crashed when parallel evaluation was enabled. Turn them into boost::concurrent_flat_maps and update them using try_emplace_or_visit(). Also fix a pre-existing bug in printStatistics() where the "attributes" list was built into a copy of the JSON node, so the output always contained "attributes": null. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Make NIX_COUNT_CALLS thread-safe
is-logic-error.hh relied on <cxxabi.h> declaring the Itanium ABI RTTI classes (__cxxabiv1::__si_class_type_info and its __base_type member). Only libstdc++'s <cxxabi.h> does that; libc++abi keeps them in its private, uninstalled private_typeinfo.h, so building with -stdlib=libc++ on Linux failed with error: no type named '__si_class_type_info' in namespace '__cxxabiv1' Since the layout of these classes is fixed by the Itanium C++ ABI (https://itanium-cxx-abi.github.io/cxx-abi/abi.html#rtti) and libc++abi exports their vtables/type infos, declare the parts we need ourselves when not using libstdc++ headers. (macOS was unaffected only because the interposer is Linux-only in meson.build.) Assisted-by: Claude Fable 5 <noreply@anthropic.com>
On macOS, calling std::set_terminate(nullptr) to restore the original terminate handler is ineffective, since std::terminate() calls the handler recorded in the exception. So we end up in an infinite recursion of terminateHandler() calls, which eventually overflows the stack, leading to an unhelpful crash report. So instead, just call onTerminate() to print the stack trace and abort. (On Linux / libstdc++, std::terminate() calls the global terminate handler so we don't have this issue.)
Fix terminateHandler() on macOS
Off-CPU profiling showed that ~2 of 12 worker threads were idle on average during the parallel phase of 'nix search', because work items were only spawned after fully enumerating each attrset: while one thread enumerated legacyPackages.x86_64-linux (~120k attributes), and later each large package subset (pythonPackages, perlPackages, ...), the other workers had nothing new to pick up. Two changes: * Spawn work items incrementally (every 256 attributes) during enumeration instead of in one batch at the end, so idle workers can start on the first attributes while enumeration continues. * Executor::spawn(): Generate queue keys with a thread-local mt19937_64 instead of calling std::random_device per work item, which costs hundreds of cycles per call (RDRAND / /dev/urandom). The key only needs to spread same-priority items around the queue, not be cryptographically random. At 12 eval cores (with GC disabled via GC_INITIAL_HEAP_SIZE=40G), this reduces elapsed time from ~4.6s to ~4.3s, user CPU from ~28s to ~26s, and kernel time from ~5s to ~3.9s for 'nix search nixpkgs --no-eval-cache fizzbuzz'. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Flake lock file updates:
• Updated input 'nix':
'path:../..'
→ 'path:../..'
• Updated input 'nix/nixpkgs':
'https://api.flakehub.com/f/pinned/DeterminateSystems/secure-packages-25.11/0.1.915296%2Brev-03ab51be14d9f1692a35003d8e128e7cefcf6092/019f4296-40f2-7350-859c-547b9fb7e2bf/source.tar.gz' (2026-07-08)
→ 'https://api.flakehub.com/f/pinned/DeterminateSystems/secure-packages-25.11/0.1.915311%2Brev-4ce84e39eb99c2b0aa7f0954446d594adf920b19/019f61fc-d9f0-7682-8cb4-1540d0570ff1/source.tar.gz' (2026-07-14)
…661-99c5-41fc-9737-5111f5bfcb9a Release v3.21.6
nix search: Reduce evaluator worker starvation
Followup to 8fe07ef. This didn't work when Sentry is enabled, because it initializes OpenSSL first. So we now call initLibUtil() before Sentry.
…-attempt Fix disabling OpenSSL's atexit handler
Flake lock file updates:
• Updated input 'nix':
'path:../..'
→ 'path:../..'
• Updated input 'nix/nixpkgs':
'https://api.flakehub.com/f/pinned/DeterminateSystems/secure-packages-25.11/0.1.915311%2Brev-4ce84e39eb99c2b0aa7f0954446d594adf920b19/019f61fc-d9f0-7682-8cb4-1540d0570ff1/source.tar.gz' (2026-07-14)
→ 'https://api.flakehub.com/f/pinned/DeterminateSystems/secure-packages-25.11/0.1.915313%2Brev-6f54b7614d8641bcda103e7426e8371e74d2024d/019f6293-c639-7461-8481-31f6ea891c12/source.tar.gz' (2026-07-14)
…8d8-4cf0-42c3-8f25-8eee063295cf Release v3.21.7
If `git_thread_create` failed in `ll_find_deltas` (e.g. with EAGAIN due to transient resource exhaustion), libgit2 returned an error while the already-created delta-search worker threads were still running. `GitRepoImpl::flush()` would then throw and free the packbuilder, causing the orphaned workers to crash on freed memory (DETERMINATE-NIX-66: SIGSEGV in `git_odb_read` → `git_cache_get_raw` → `pthread_rwlock_rdlock` during `nix flake show`). Patch libgit2 to proceed with however many threads were successfully created, letting the work-stealing loop redistribute the thread-less partitions' work (stolen in their entirety, so every object is still searched for deltas), and to fall back to the single-threaded delta search if no threads could be created at all. Also remove a use-after-free of the `thread_params` array on the (unlikely) mutex lock failure path. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Fix libgit2 ll_find_delta leaking worker threads
SIGQUIT is a user-initiated "quit with core dump" action (e.g. pressing Ctrl-\ at a terminal), not a crash, but crashpad includes it in its set of handled signals, so it got uploaded to Sentry as a fatal error. For example, we received crash reports of nix sitting in the blocking stdin read at the flake nixConfig trust prompt when the user pressed Ctrl-\. Fix this by resetting SIGQUIT to its default disposition in initNix(), which runs after sentry_init() has installed the crashpad handlers. We already did this on macOS; now it's done on all platforms. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
Don't report SIGQUIT to Sentry
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Not intended to be merged directly. This PR is a convenience to show the diff between upstream Nix and Determinate Nix (the
mainbranch).Continuation of #4.