test: make the default suite reproducible in clean environments#68
Merged
Conversation
This was referenced Jul 12, 2026
Warm the dispatch surface once, keep a timed-out background warmup alive explicitly, and release async-write and lifecycle resources safely when startup cancellation escapes the inner run loop. Designed by Marsu — Refined by Codex. Co-Authored-By: Codex <codex@openai.com>
Remove workstation-only and process-history assumptions, keep version checks source-aware, use the standard library for plist parsing, and preserve correctness signals without unstable global monkeypatches or hard timing gates. Designed by Marsu — Refined by Codex. Co-Authored-By: Codex <codex@openai.com>
aa7d150 to
50e18b6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft / stacked PR
Do not merge before #67. This branch is stacked on #67, so the GitHub diff
includes that dependency until it lands. It will then be rebased onto
mainto leave only the baseline-reproducibility changes.
Problem
The default suite contained assumptions tied to a particular checkout, runner
load, or process history: an undeclared XML parser, a local planning artifact,
global
stderrmonkeypatching, hard-coded package versions, cache-keyexpectations that no longer matched the write contract, platform-sensitive
vector normalization, and an assertion comparing asynchronously sampled macOS
memory metrics.
Two concurrency/ranking assertions also over-specified wall-clock behaviour:
approaching 1 GiB of probe-owned allocations and interpreting its own list
reallocations as GIL stalls;
scores differed only by per-candidate age-sampling noise.
Solution
stderr;metadata, with a clear source-only fallback;
target an observable warning;
phys_footprint <= RSScomparison while retaining thedeterministic watchdog decision tests;
network-resolved revision together with its dataset loader;
iai_mcp_nativeextension binaries in the native import test,rather than every unrelated extension installed in
site-packages;Python switching is temporarily disabled and the observer must make progress
inside
py.allow_threads(...)during the native scan;the existing tolerance without assigning meaning to quasi-tie order;
Guarantees and scope
No production safety gate is relaxed. The patch removes machine-specific test
assumptions and keeps the correctness signals: the GIL test still fails against
an intentionally GIL-holding C call, and the profile test still catches any
record-set or per-record score divergence.
Validation
CPU workers; negative control 0/3 progress with an intentional GIL holder;
4 xfailed, 0 failed across 5,375 collected tests; GitHub's single-process
rerun is the final publication gate for this amended tip.
Designed by Marsu — Refined by Codex.
Public CI result
The amended macOS single-process run completed with 5,123 passed, 113 skipped, 107 deselected, and 4 xfailed. Its only three failures are the pre-existing
srcversusIAI-MCPspatial-classification cases fixed by the next stacked PR, #69.The prior daemon pre-warm, GIL-concurrency, and quasi-tie ordering failures are absent from this run. The cumulative #69 tip is fully green at 5,126 passed and also builds the universal2 wheel successfully.