fix: restore macOS CPU flags (first-boot spin) + settled image + Spotlight off#13
Merged
Conversation
Consumers of tahoe:26 / sequoia:15 hit a ~40min CPU-pegged "completing installation" storm on first boot, because setup provisions the Data volume offline from Recovery and never boots the installed OS — so its one-time first-full-boot finalization (kernelcache/dyld rebuild, LaunchServices, Spotlight full-disk index) ran on the consumer's first boot instead. This also starved sshd (port open, no banner), which is why slim failed "SSH never came up" and images stayed bloated at ~35G. - provision-macos.sh: drop a root-level .metadata_never_index on the Data (and System) volume so mds/diskarbitrationd exclude it at mount time, before any indexing is scheduled — closing the race the RunAtLoad `mdutil -a -i off` loses. Also disable automatic softwareupdate checks offline. The single biggest avoidable first-boot hog. - build-qemu-macos.sh: after Recovery provisioning, reboot into the installed OS and run its finalization ONCE at build time, then clean-shutdown + capture a SETTLED image. Extract absorb_finalization_and_capture (boot -> wait_ssh 300 -> perf recipe -> slim -> shutdown -> capture) shared by setup and slim. - raise the job timeout 180 -> 300 (setup now bakes end-to-end in one job).
dacf35c simplified the macOS -cpu from the flagged Skylake-Client string to "Skylake-Client-v4,vendor=GenuineIntel,kvm=on", silently dropping -hle,-rtm (TSX off), +invtsc and vmware-cpuid-freq=on — and weakened the guard test that existed to prevent exactly this. The regression: a fresh image's first full boot SPINS FOREVER at the "completing installation" progress bar, 4 cores pegged, never reaching the desktop. Root cause is TSC/TSX: - vmware-cpuid-freq=on hands macOS the TSC frequency via CPUID; without it macOS self-calibrates the TSC, which spins under an unstable nested-KVM TSC. - the v4 base enables TSX (hle/rtm); macOS spins on it under nested KVM. The old string disabled it with -hle,-rtm. Testbed A/B/C on AMD Zen5 (booting the same new image, only -cpu differing): Skylake-Client-v4+invtsc+vmware-cpuid-freq still hung (black screen, 400% CPU, overlay frozen); the full flagged Skylake-Client string booted through finalization to the Setup Assistant GUI. The AMD breakthrough was always the LongQT OpenCore EFI, never the CPU simplification, so restoring the flags is safe on both Intel and AMD. Restore the full string in qemu/launch.go and scripts/build-qemu-macos.sh, and restore TestArgsCPU to assert every load-bearing token so it can't be dropped again.
CMGS
changed the title
Contributor
Author
Update: found the real root cause — a CPU-flag regression from dacf35cThe "first boot spins forever at completing-installation" is not just the offline-provision issue — the primary cause is that Testbed A/B/C on AMD Zen5, booting the same new image, only
So the v4 base's TSX is a second trigger on top of the missing TSC-freq hint. The AMD breakthrough was always the LongQT OpenCore EFI, never the CPU simplification — restoring the flags is safe on Intel and AMD. Added commit Validated: gofumpt/build(darwin+linux)/ |
… it) The turnkey images stop at the Setup Assistant, which blocks BOTH sshd (kex reset) and ACPI powerdown — so the SSH-based absorb/slim path could never work (verify's "SSH works" was a false pass: stage_verify never fails on SSH timeout). Rework absorb_finalization_and_capture to be SSH-free: boot, wait for the guest CPU to go idle (finalization done), flush, then hard-stop via monitor quit. Testbed-verified on AMD Zen5: the qcow2 stays consistent across the hard stop (qemu-img check clean) and the settled overlay reboots to the SA in ~2.5min instead of re-running the ~40min finalization — so consumers get a settled image. (The SA itself still isn't auto-skipped on 26/15 — that needs the SA click-through and is tracked separately; this only removes the SSH dependency that made settling impossible.)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Consumers of
tahoe:26/sequoia:15hit a ~40min, 4-core-pegged (400% CPU) "completing installation" storm on the first boot — reproduced on bare-metal Zen5 (testbed run: overlay grew to 18.8G, 145min CPU-time, sshd port open but no banner because it was CPU-starved). Root cause (traced file:line):stage_setupprovisions the Data volume offline from Recovery and never boots the installed OS, so macOS's one-time first-full-boot finalization (kernelcache/dyld rebuild, LaunchServices registration, Spotlight whole-disk index) is deferred to the consumer's first boot. Same root cause behind:slimfailing "SSH never came up" (its wait budget expired mid-finalization), and the bloated ~35G images.Fix (two compounding levers)
.metadata_never_indexon the Data + System volumes inprovision-macos.sh. mds/diskarbitrationd honor it at volume-mount time, before indexing is scheduled — closing the race the existingRunAtLoadmdutil -a -i offloses. Also disable automatic softwareupdate checks offline. (Livemdutilcalls kept as harmless fallback.)stage_setupnow reboots into the installed OS, runs the finalization once in CI, then clean-shuts-down and captures a settled image — so consumers never pay the tax. Extractedabsorb_finalization_and_capture(boot →wait_ssh 300→ perf recipe → slim → clean shutdown → capture), shared bysetupandslim. This also fixes the structural trap that$GHCR_TAGonly got settled if someone remembered to runslimand it happened to succeed.timeout-minutes180 → 300 (setup now does base-pull + provision + full finalize + slim + capture + push in one job).Note: kernelcache/dyld rebuild is irreducible (can't be pre-baked offline), so the finalization still happens once — but now at build time, not on every consumer. Spotlight-off should meaningfully shrink its duration.
Validation
bash -nboth scripts clean; shellcheck 0 findings in changed regions. Thewait_ssh 300(~100min) budget is extrapolated and being re-measured against a live bare-metal finalization on the testbed now — will tune before this is relied upon. Re-run of setup for both versions follows merge.🤖 Generated with Claude Code