Skip to content

Keep CSCS startup channel active#225

Closed
robobryce wants to merge 1 commit into
NVIDIA:event/2026-07-cscs-summer-schoolfrom
robobryce:fix/cscs-startup-channel
Closed

Keep CSCS startup channel active#225
robobryce wants to merge 1 commit into
NVIDIA:event/2026-07-cscs-summer-schoolfrom
robobryce:fix/cscs-startup-channel

Conversation

@robobryce

Copy link
Copy Markdown

Summary

  • print a progress heartbeat every 30 seconds while the Slurm job is running and its web services are starting
  • cancel the submitted allocation if the launcher channel receives HUP, INT, TERM, or PIPE
  • keep cleanup traps active across every post-submission status line
  • report the SSH status and submitted job ID if the workstation loses the launcher connection

This addresses the observed run where workstation output ended after Job 4271851: RUNNING while the unmonitored allocation continued.

Validation

  • bash -n: both changed helpers
  • ShellCheck: both changed helpers
  • git diff --check: clean
  • exact ControlMaster silent-channel probe: remained connected for 245 seconds and exited successfully
  • Daint failure-path test: severed the SSH launcher connection after a dedicated job entered RUNNING; job 4271874 immediately entered COMPLETING and disappeared from squeue within 6 seconds
  • PIPE-path test: forced a closed output pipe; handler exited 141 and invoked allocation cleanup

Test job/state were removed. User job 4271851 was not modified.

@copy-pr-bot

copy-pr-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown
Contributor

👋 Thank you for your contribution!

This pull request is from a forked repository so GitHub Actions will not be able to run CI. A maintainer will review your changes shortly and manually trigger the CI.

@maintainers Please review this PR when you have a chance and follow the instructions in the CONTRIBUTING.md file to trigger the CI.

@robobryce
robobryce marked this pull request as draft July 24, 2026 17:31
@robobryce robobryce closed this Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants