[RFC] In-headset operator feedback as a first-class concern (HUD, haptic grammar, runtime scale control) — seeking design direction

[johnnynunez]

Context

During teleoperation the operator is inside the HMD and physically cannot read the terminal — which is where all of the stack's warnings, harness interventions, and state changes are reported today. Over a week of hardware sessions this produced real operational failures, not just inconvenience:

• Safety-harness interventions (clamps, rejections) happened with zero operator awareness — the arm "stopped following" with no explanation, and the operator's instinct (bigger, faster motions) is exactly wrong.
• A session ran for minutes with the wrong script variant (no harness active); nothing in the headset indicated the difference. Diagnosed afterwards via ps aux.
• Workspace-edge, clutch state, and tracking-loss events were invisible at the moment they mattered.

What we built at the example layer (works, validated in sessions)

1. Haptic feedback grammar — distinct vibration patterns for: workspace edge (leash engaged), clutch engage/disengage, pose-validity loss (PoseGate hold), scale change confirmation.
2. In-headset HUD (Televiz) — harness status, clutch state, joint-limit proximity.
3. Pointing compass + world-anchored aim laser dot (controller aim ray ∩ table plane) — spatial orientation without leaving the headset.
4. Runtime motion-scale control on the thumbstick (×0.25–×2.0, haptic confirmation) — no terminal round-trip to retune.
5. Preflight calibration game ("push forward" check) — doubles as automatic yaw alignment for base_T_anchor.

Ask: design feedback + upstream direction

This is deliberately an RFC-style issue — we'd like the Isaac Teleop team's view on:

• Which of these belong in the library (e.g., a standard operator-feedback channel that retargeter nodes can emit events into, with haptic/HUD sinks) vs. staying example-level?
• Is there an existing/planned event bus in the retargeting graph that feedback primitives should attach to, so every node (rate limiter, bounds, PoseGate) can signal the operator uniformly instead of each integration reinventing it?
• Any constraints from the CloudXR/compositor side on overlay layers we should design against?

Our position from field experience: operator feedback is a safety feature, not cosmetics — most of our "the arm ignores me → push harder" incidents trace back to interventions the operator couldn't perceive. We're happy to upstream any/all of the above once the target architecture is agreed.

---

Part of the field-integration feedback series (SO-101 + reBot DevArm, June 29 – July 6). Priority suggestion: P1 (RFC).