Skip to content

connect: run the GUI on a cluster node, browse it locally (magnetics-connect)#89

Open
matt-pharr wants to merge 2 commits into
developfrom
cybele_support
Open

connect: run the GUI on a cluster node, browse it locally (magnetics-connect)#89
matt-pharr wants to merge 2 commits into
developfrom
cybele_support

Conversation

@matt-pharr

Copy link
Copy Markdown
Collaborator

Why

Users at GA reach the analysis stack by remote-desktopping (NoMachine) into a
virtual workstation on the cybele login cluster, then SSHing to a omega
work node. That breaks the webapp: the server's 127.0.0.1:PORT is on omega,
but the browser is on cybele.

The fix is not an X-forwarded browser — it's the standard SSH tunnel, which
rides the connection you're already making. This adds a launcher that does the
whole dance in one command, abstracted over host/gateway so it works from a
laptop off-site, from a site gateway desktop, or toward any cluster with the
same login-node pattern (flux, feyn, …).

What it does

magnetics-connect <host> opens one authenticated SSH connection that:

  1. asks the remote for a free loopback port and its real node name (submit nodes
    are load-balanced — "omega" resolves to omega-a or omega-b run to run);
  2. starts the server there bound to loopback only, through bash -lc so
    module/profile PATHs apply;
  3. carries a -L port-forward on that same connection;
  4. polls through the tunnel and opens the local browser once the server answers.

Only HTTP crosses the wire: fetches and fits run next to the data, plots stream
back. Ctrl-C stops both ends — the forced remote pty SIGHUPs the server, so
nothing is orphaned on a shared node.

magnetics-connect omega                        # ~/.ssh/config alias
magnetics-connect me@omega.gat.com -J me@cybele.gat.com:2039
magnetics-connect omega --data-dir /cscratch/$USER/magnetics
magnetics-connect flux --remote-cmd 'module load magnetics && magnetics --no-browser --port {port}'

Why a tunnel rather than binding 0.0.0.0

I measured the firewall: cybele → omega on a high TCP port is not blocked, so
a direct bind would "work". It's still the wrong default — it exposes an
unauthenticated API that triggers data fetches and serves shot data to the whole
GA internal network, and it collides when two users pick the same port. A
direct-bind mode should come with a Jupyter-style access token; that's a
follow-up, not this PR.

Standalone by design

src/magnetics/connect.py is stdlib-only and self-contained so it runs on a
gateway node with only system python3 and no magnetics install:

scp src/magnetics/connect.py gateway:
python3 connect.py omega

Verified under Python 3.9 (cybele ships 3.9.25). CLAUDE.md notes the constraint
so it isn't refactored away.

Verification

  • Live, laptop → cybele → omega: tunnel established, readiness poll, remote
    logs streamed back with an [omega] prefix, clean teardown with zero orphaned
    processes on the node. Repeated against a real install of this wheel on omega:
    the bundled GUI and /api/* both serve through the tunnel.
  • Offline: 11 new tests (tests/test_connect_launcher.py) with ssh/subprocess
    faked, covering probe → tunnel → serve command construction, the jump flag,
    {port} substitution, data-dir quoting, probe failure, and the
    never-became-ready path.
  • All CI gates green locally: ruff format/check, pytest (459 passed, 79% coverage),
    ty check.

Follow-ups (not in this PR)

  • Package as an Lmod module on omega so module load magnetics replaces the
    manual install.
  • Optional token-authenticated --host 0.0.0.0 mode for users who'd rather skip
    the tunnel.

🤖 Generated with Claude Code

matt-pharr and others added 2 commits July 10, 2026 03:37
Run the server on a cluster work node, browse it locally — from a laptop
off-site or a NoMachine desktop on the site gateway. One ssh connection
carries the -L tunnel and the server (loopback-only, free port asked from
the remote, real node name reported since submit nodes are load-balanced);
the forced pty SIGHUPs the server on disconnect so nothing is orphaned on
a shared node. Stdlib-only and standalone so it also runs as a bare
python3 connect.py on gateways with no magnetics install.

Verified live laptop -> cybele -> omega-b: tunnel, readiness poll,
streamed remote logs, and clean teardown.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Running the GUI on an omega work node (the NoMachine -> cybele -> omega
flow) sent the remote backend's ssh to omega.gat.com *from* omega. That
node's known_hosts has no entry for it, so ssh stopped at the host-key
confirmation prompt — which a server process has no tty to answer. The
pull sat at 0% forever instead of failing.

on_cluster_host() compares our short hostname against the device's
configured cluster host, gated on already being inside the site domain,
so load-balanced submit nodes (omega-a/-b) and workers (omega25) all
match the configured omega.gat.com. The site gateway does NOT match:
from cybele, ssh to omega is still the correct path. Where it matches,
the pull runs in-process — toksearch when importable, else mdsthin
straight to mdsip, which needs no ssh on-site. MAGNETICS_ON_CLUSTER
forces either answer.

Verified on omega-a: the same --backend remote pull that hung now
reports 'already on the cluster; remote->mdsthin' and fetches.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant