Add ModelOpt QAD skill for Slurm workflows

[mxinO]

What does this PR do?

Type of change: new feature

Adds a general Slurm-only QAD skill based on the supported Megatron Bridge
workflow. The skill:

• starts from a measured BF16-to-PTQ benchmark gap and preserves the preceding
  PTQ configuration or recipe;
• gates QAD on exact Megatron Bridge model support and successful Megatron PTQ,
  using its master-rank quantizer summary as a scoped amax sanity check;
• requires model- and hardware-derived TP/PP/CP/EP/ETP topology selection;
• streams and randomly samples only the required
  nvidia/Nemotron-Cascade-2-SFT-Data token budget and uses Megatron sequence
  packing;
• defaults to 32K sequences, LR 1e-5 with cosine decay, a 1000-step cap, and
  GBS 512;
• requires explicit user authorization because QAD is costly, validates two
  batches every 25 steps, saves every 50 steps, and monitors a decreasing
  smoothed loss trend;
• evaluates an early checkpoint around step 150 and continues only when
  benchmark recovery and the loss trend justify more training;
• follows the established common Slurm and remote-execution guidance instead of
  duplicating mutable commands from the Megatron Bridge README.

Also exposes Megatron Bridge save_interval, exit_interval, and
exit_duration_in_mins through examples/megatron_bridge/distill.py, with example-test coverage for checkpoint
and ModelOpt-state preservation at an early exit.

Usage

Use the QAD skill to recover the measured BF16-to-PTQ benchmark gap for
<model> on <Slurm cluster>, preserving the validated PTQ recipe.

Testing

• PYTHONPATH=$PWD pre-commit run --all-files
  • Passed every hook on the rebased branch, including Ruff, Ruff format, mypy,
    YAML/recipe validation, launcher reference validation, Bandit, generated
    arguments, symlink synchronization, and Markdown lint.
• python ~/.codex/skills/.system/skill-creator/scripts/quick_validate.py .agents/skills/qad
  • Skill is valid!

Qwen3-0.6B result-bearing validation:

• Resources: one exclusive node, 8 H100 GPUs

• Container: nvcr.io/nvidia/nemo:26.06

• Quantization: NVFP4, group size 16, embedding excluded

• QAD topology: TP=1, PP=1, CP=4, EP=1, DP=2

• Training validation configuration: sequence length 32768, MBS=1, GBS=8,
  train_iters=1000, LR 1e-5 / minimum LR 1e-6, 50 warmup iterations,
  cosine decay, eval_interval=150, exit_interval=150,
  exit_duration_in_mins=220

• This result-bearing run used the then-current coupled eval/save cadence. The
  final skill now validates two batches every 25 steps and saves every 50; the
  example test covers the independent checkpoint cadence.

• The reduced GBS 8 is intentionally validation-only; the skill retains GBS 512
  as the production default.

• Data: exactly 10,000,000 sampled tokens from four
  nvidia/Nemotron-Cascade-2-SFT-Data configs:

  • math: 2,306,011 tokens / 364 documents
  • science: 1,191,257 tokens / 285 documents
  • chat: 6,142,077 tokens / 1,800 documents
  • instruction following: 360,655 tokens / 411 documents

• Megatron built packed 32K GPT samples from the materialized prefixes; the full
  dataset was not downloaded.

• QAD loss was finite and decreased from 0.2640341 at iteration 10 to
  0.1060580 at iteration 150. Final gradient norm was 0.747, with zero
  skipped and zero NaN iterations. Validation distillation loss was
  0.09715855.

• The iteration-150 checkpoint saved successfully with modelopt_state, and
  both PTQ and QAD-150 exported to unified Hugging Face format.

• Identical full MMLU 0-shot comparison through the Megatron evaluator:

  Model	Accuracy
  BF16	0.39517164
  PTQ	0.32851446
  QAD-150	0.38740921

  QAD-150 recovered 0.05889475 / 0.06665718 = 88.35% of the measured PTQ gap,
  so validation stopped at the early evidence gate rather than continuing
  blindly toward 1000 iterations.

Before your PR is "Ready for review"

• Is this change backward compatible?: ✅
• If you copied code from any other sources or added a new PIP dependency, did
  you follow guidance in CONTRIBUTING.md: N/A
• Did you write any new necessary tests?: ✅
• Did you update Changelog?: N/A — this adds an agent skill and example-only
  lifecycle flags.
• Did you get Claude approval on this PR?: N/A

Additional Information

All seven branch commits are cryptographically signed and include a
Signed-off-by trailer.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 55999bb9-b986-4737-80cf-848202ff5ee6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch mxin/qad-skill

---

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.81%. Comparing base (d984de3) to head (81e57bf).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2010      +/-   ##
==========================================
- Coverage   78.47%   75.81%   -2.67%     
==========================================
  Files         518      518              
  Lines       58574    58574              
==========================================
- Hits        45967    44406    -1561     
- Misses      12607    14168    +1561     

Flag	Coverage Δ
unit	55.02% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.