Sort cargo ops isolate GRUB drop-in after provider drop-ins

[epompeii]

What

Follow-up to #954, fixing an issue found in field testing of cargo ops isolate:

• Rename the GRUB drop-in from 99-bencher-isolation.cfg to zz-bencher-isolation.cfg. Provider images can ship their own /etc/default/grub.d drop-in (e.g. Hetzner's hetzner.cfg) that overwrites GRUB_CMDLINE_LINUX_DEFAULT instead of appending to it. Drop-ins are sourced in glob order, so 99- sorted before hetzner.cfg and our args were silently clobbered; zz- sorts last.
• Verify the generated /boot/grub/grub.cfg actually contains the isolation args after update-grub and before rebooting. A clobbered cmdline now fails fast with a pointer at drop-in ordering, instead of only surfacing after a wasted reboot cycle.

Why

Without the ordering fix, the command completed update-grub successfully but the rebooted kernel came up without any isolation args, and the failure was only detected by the post-reboot cmdline check.

How

• tasks/runner_ops/src/task/isolate.rs: GRUB_DROP_IN renamed with a comment explaining the ordering constraint; pre-reboot grep check on the generated config
• tasks/runner_ops/CLAUDE.md: document the drop-in name and why it sorts last

[github-actions]

🤖 Claude Code Review

PR: #955
Base: devel
Head: claude/runner-ops-isolate-ordering
Commit: 798252faeef0d44dcd8cf98c21bf1462a2e5cd3d

---

This is a small, focused follow-up PR. I've reviewed the full diff and the surrounding code (isolate.rs, ssh.rs).

Review

Summary

The change does two related things:

1. Renames the GRUB drop-in 99-bencher-isolation.cfg → zz-bencher-isolation.cfg so it sorts after provider drop-ins (e.g. Hetzner's hetzner.cfg) that overwrite rather than append to GRUB_CMDLINE_LINUX_DEFAULT.
2. Adds a post-update-grub verification that greps the generated /boot/grub/grub.cfg for isolcpus={cpus} and bails before rebooting if the args are missing.

Both are sound and directly address a real failure mode (a provider drop-in silently clobbering the isolation args, previously only discovered after a reboot cycle).

Strengths

• Fail-fast before reboot is the right instinct: catching the clobber via grep saves a full reboot + wait_for_reboot (up to 5 min) only to find the args never applied. The error message correctly names the likely cause.
• grep -qF uses a fixed-string match, and cpus is already validated (validate_cpu_list), so there's no injection or regex-metachar concern.
• Uses ssh.check() (bool, non-zero = false) rather than run(), which is the correct primitive here.
• The code comment explains why (glob-order sourcing) rather than what, and avoids emdashes per CLAUDE.md. anyhow is acceptable in a tasks/ crate.
• /boot/grub/grub.cfg is the correct Debian/Ubuntu path, consistent with the existing update-grub usage.

Minor issues

1. Stale 99- drop-in is not cleaned up. If a server ever ran the previous version (which created 99-bencher-isolation.cfg), re-running now writes zz-bencher-isolation.cfg and leaves the old file behind. Both use the $GRUB_CMDLINE_LINUX_DEFAULT isolcpus=... append form, so on a provider that does not overwrite, you'd get duplicated isolcpus=X isolcpus=X (and nohz_full/rcu_nocbs) args in the cmdline. This is low-severity (the subcommand is brand new, and the idempotency check short-circuits already-applied hosts), but a one-line rm -f /etc/default/grub.d/99-bencher-isolation.cfg before writing the new file would make the migration clean. Worth at least a mention if not a fix.

2. No test coverage for the new branch. The added grep-verification path isn't exercised by a unit test. This is understandable since it's an SSH-bound side effect and the pure helpers (is_isolated, benchmark_cpus, validate_cpu_list, isolation_cfg) remain well-tested, but it's a slight deviation from the repo's TDD emphasis. Acceptable given the I/O boundary.

Verdict

Clean, well-scoped, correctly reasoned. No blocking concerns. The only thing I'd consider addressing is the stale 99- file cleanup (issue #1); the rest is polish.

---

_{Model: claude-opus-4-8}

[epompeii]

Took the optional suggestion: the pre-reboot check now uses grep -qF for an explicit fixed-string match.

[github-actions]

Bencher Report

Project	Bencher
Branch	claude/runner-ops-isolate-ordering
Testbed	intel-v1

Click to view all benchmark results

Benchmark	Latency	Benchmark Result microseconds (µs) (Result Δ%)	Upper Boundary microseconds (µs) (Limit %)
Adapter::Json	📈 view plot 🚷 view threshold	4.67 µs (+0.19%) Baseline: 4.66 µs	4.89 µs (95.36%)
Adapter::Magic (JSON)	📈 view plot 🚷 view threshold	4.48 µs (-1.11%) Baseline: 4.53 µs	4.71 µs (95.14%)
Adapter::Magic (Rust)	📈 view plot 🚷 view threshold	26.05 µs (+1.49%) Baseline: 25.66 µs	26.67 µs (97.66%)
Adapter::Rust	📈 view plot 🚷 view threshold	3.49 µs (-0.13%) Baseline: 3.50 µs	3.59 µs (97.34%)
Adapter::RustBench	📈 view plot 🚷 view threshold	3.49 µs (-0.08%) Baseline: 3.50 µs	3.59 µs (97.35%)

🐰 View full continuous benchmarking report in Bencher