Several tests/e2e/ training tests fail on main (stale checkpoint assertions + PP/resume subprocess failures)

[amazloumi]

What happened

Running the e2e gate (uv run pytest tests/e2e/ --e2e -v) on a single 4×H200 node, 6 tests fail. Re-running the failing subset on clean main (3ad9d9a) reproduces 5 of them, so these are pre-existing (surfaced while running the e2e gate for an unrelated PR). Two distinct root causes:

(A) Stale checkpoint-log assertions — a test bug. Tests assert the literal log string "Checkpoint saved" (and similar), but the training code now logs "Sync checkpoint saved" / "Checkpoint committed". The checkpoint is written — test_moe_checkpoint_resume's captured output shows Sync checkpoint saved: .../step_10, Checkpoint committed: .../step_10, and Training complete: 10 steps — only the expected substring is outdated. Confirmed for test_moe_checkpoint_resume; very likely the same for test_checkpoint_save_and_resume and test_sigterm_triggers_emergency_checkpoint (the "…was not saved" asserts).

(B) PP / resume subprocess failures. test_pipeline_parallel and test_pp_checkpoint_save_and_resume fail because the launched train.py subprocess exits 1 (assert result.returncode == 0 → assert 1 == 0). The child traceback is not captured (error_file: <N/A>), so the root cause is unknown. A c10d warning is present on this node and may be unrelated: socket … errno: 97 - Address family not supported by protocol (IPv6).

Flaky (not a bug): test_hf_dataset_single_gpu failed once (HF dataset download, returncode 1) but passed on the main re-run → network-dependent.

Reproduction

# clean main (3ad9d9a), single node, 4xH200
uv run pytest tests/e2e/ --e2e -v

# or just the reproducible failures:
uv run pytest \
  tests/e2e/test_training_e2e.py::test_moe_checkpoint_resume \
  tests/e2e/test_training_e2e.py::test_pipeline_parallel \
  --e2e -v

Error / traceback

# (A) stale string -- checkpoint WAS saved:
>   assert "Checkpoint saved" in output, "MoE checkpoint was not saved"
E   AssertionError: MoE checkpoint was not saved
E   assert 'Checkpoint saved' in "... Sync checkpoint saved: .../moe_ckpt/step_10
                                       Checkpoint committed: .../step_10 (step=10)
                                       Training complete: 10 steps ..."

# (B) PP subprocess exit 1, traceback not captured:
E   torch.distributed.elastic.multiprocessing.errors.ChildFailedError: scripts/train.py FAILED
E     rank 0: exitcode 1 ; error_file: <N/A>
[W] socket.cpp: cannot connect to [localhost]:PORT (errno: 97 - Address family not supported by protocol)

Environment

• GPU type and count: 4x NVIDIA H200, single node (SLURM)
• Nodes: 1
• PyTorch version: per .venv (python -c "import torch; print(torch.__version__)")
• KempnerForge commit: 3ad9d9a (main)
• Failing subset on main: 5 failed, 1 passed (the pass = network-flaky test_hf_dataset_single_gpu)

Affected tests

• test_pipeline_parallel — (B)
• test_pp_checkpoint_save_and_resume — (B)
• test_checkpoint_save_and_resume — (A) likely
• test_sigterm_triggers_emergency_checkpoint — (A) likely
• test_moe_checkpoint_resume — (A) confirmed
• test_hf_dataset_single_gpu — network-flaky (passed on main re-run)

Suggested fixes

• (A) Update the e2e assertions to the current log strings ("Sync checkpoint saved" / "Checkpoint committed"), or assert that the checkpoint directory/files exist rather than grepping a log substring.
• (B) Re-run a failing PP test with TORCHELASTIC_ERROR_FILE set to capture the child traceback, then root-cause (and check whether the IPv6/c10d warning is implicated).