Skip to content

test: make the default suite reproducible in clean environments#68

Merged
CodeAbra merged 2 commits into
CodeAbra:mainfrom
Marsu6996:pr/fix-macos-test-baseline
Jul 13, 2026
Merged

test: make the default suite reproducible in clean environments#68
CodeAbra merged 2 commits into
CodeAbra:mainfrom
Marsu6996:pr/fix-macos-test-baseline

Conversation

@Marsu6996

@Marsu6996 Marsu6996 commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Draft / stacked PR

Do not merge before #67. This branch is stacked on #67, so the GitHub diff
includes that dependency until it lands. It will then be rebased onto main
to leave only the baseline-reproducibility changes.

Problem

The default suite contained assumptions tied to a particular checkout, runner
load, or process history: an undeclared XML parser, a local planning artifact,
global stderr monkeypatching, hard-coded package versions, cache-key
expectations that no longer matched the write contract, platform-sensitive
vector normalization, and an assertion comparing asynchronously sampled macOS
memory metrics.

Two concurrency/ranking assertions also over-specified wall-clock behaviour:

  • the GIL probe appended roughly 23–24 million Python floats while scanning,
    approaching 1 GiB of probe-owned allocations and interpreting its own list
    reallocations as GIL stalls;
  • the empty-profile fallback test required identical ordering for records whose
    scores differed only by per-candidate age-sampling noise.

Solution

  • use the standard library XML parser for launchd plist checks;
  • measure community detection directly instead of requiring a planning file;
  • inject the warning writer instead of mutating global stderr;
  • derive the package version from source metadata or installed distribution
    metadata, with a clear source-only fallback;
  • align cache and normalization tests with their production paths;
  • keep the community performance hard cap while making the historical soft
    target an observable warning;
  • remove the unstable phys_footprint <= RSS comparison while retaining the
    deterministic watchdog decision tests;
  • keep LongMemEval preflight tests hermetic by stubbing the cleaned adapter's
    network-resolved revision together with its dataset loader;
  • count only iai_mcp_native extension binaries in the native import test,
    rather than every unrelated extension installed in site-packages;
  • replace the wall-clock GIL ceiling with a synchronized observer: periodic
    Python switching is temporarily disabled and the observer must make progress
    inside py.allow_threads(...) during the native scan;
  • compare empty/medium fallback membership and score by record ID, preserving
    the existing tolerance without assigning meaning to quasi-tie order;
  • register the existing literature marker.

Guarantees and scope

No production safety gate is relaxed. The patch removes machine-specific test
assumptions and keeps the correctness signals: the GIL test still fails against
an intentionally GIL-holding C call, and the profile test still catches any
record-set or per-record score divergence.

Validation

  • synchronized GIL probe: 20/20 local repetitions; 10/10 under 20 concurrent
    CPU workers; negative control 0/3 progress with an intentional GIL holder;
  • Rust GIL/Send suite: 6/6;
  • empty/medium fallback: 50/50 repeated runs; full knob file 9/9;
  • focused final-stack Python set: 40 passed, 1 skipped;
  • wrapper suite: 48/48, plus TypeScript typecheck;
  • exhaustive default suite on the prior stack tip: 5,159 passed, 212 skipped,
    4 xfailed, 0 failed across 5,375 collected tests; GitHub's single-process
    rerun is the final publication gate for this amended tip.

Designed by Marsu — Refined by Codex.

Public CI result

The amended macOS single-process run completed with 5,123 passed, 113 skipped, 107 deselected, and 4 xfailed. Its only three failures are the pre-existing src versus IAI-MCP spatial-classification cases fixed by the next stacked PR, #69.
The prior daemon pre-warm, GIL-concurrency, and quasi-tie ordering failures are absent from this run. The cumulative #69 tip is fully green at 5,126 passed and also builds the universal2 wheel successfully.

Marsu6996 and others added 2 commits July 13, 2026 01:25
Warm the dispatch surface once, keep a timed-out background warmup alive explicitly, and release async-write and lifecycle resources safely when startup cancellation escapes the inner run loop.

Designed by Marsu — Refined by Codex.

Co-Authored-By: Codex <codex@openai.com>
Remove workstation-only and process-history assumptions, keep version checks source-aware, use the standard library for plist parsing, and preserve correctness signals without unstable global monkeypatches or hard timing gates.

Designed by Marsu — Refined by Codex.

Co-Authored-By: Codex <codex@openai.com>
@Marsu6996 Marsu6996 force-pushed the pr/fix-macos-test-baseline branch from aa7d150 to 50e18b6 Compare July 12, 2026 23:32
@CodeAbra CodeAbra marked this pull request as ready for review July 13, 2026 03:09
@CodeAbra CodeAbra merged commit 3a6126a into CodeAbra:main Jul 13, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants