Skip to content

Mori Remote: iOS companion app + cloud relay#20

Draft
vaayne wants to merge 61 commits intomainfrom
explore/mori-remote
Draft

Mori Remote: iOS companion app + cloud relay#20
vaayne wants to merge 61 commits intomainfrom
explore/mori-remote

Conversation

@vaayne
Copy link
Copy Markdown
Owner

@vaayne vaayne commented Mar 25, 2026

Summary

iOS companion app (Mori Remote) that connects to a cloud-hosted WebSocket relay, enabling full interactive tmux terminal sessions from anywhere using libghostty for GPU-accelerated Metal rendering.

  • MoriRemote/ — iOS app (SwiftUI, iOS 17+, ghostty Metal rendering, QR pairing, session list, mode toggle)
  • Sources/MoriRemoteHost/ — Mac relay connector CLI (serve, sessions, qrcode, loopback)
  • Packages/MoriRemoteProtocol/ — Cross-platform protocol package (61 test assertions)
  • mori-relay/ — Go WebSocket relay service (Fly.io ready)
  • vendor/ghostty/ — Fork with Remote termio backend (Zig, fd-pair IO)

Architecture

                        Cloud (Fly.io)
┌──────────┐    WSS    ┌─────────────┐    WSS    ┌──────────┐
│ Mac      │──────────>│ Go Relay    │<──────────│ iOS App  │
│ Connector│ outbound  │ (token pair)│  outbound │ (ghostty)│
└────┬─────┘           └─────────────┘           └──────────┘
     │ pty
     v
  tmux attach -t <session>

Key features

  • QR-code token pairing (no accounts needed)
  • Full interactive terminal via libghostty Metal renderer on iOS
  • Read-only and interactive modes (grouped tmux sessions)
  • Session list with display-friendly names
  • Auto-reconnect with exponential backoff
  • iOS lifecycle: immediate detach on background, fast resume on foreground
  • Localized (English + zh-Hans)

Implementation phases

Phase Status
1. Fork ghostty + universal XCFramework (macOS + iOS)
2. Remote termio backend (Zig) + protocol package
3. Go WebSocket relay service
4. Mac relay connector (MoriRemoteHost)
5A. iOS ghostty rendering + pipe bridge
5B. iOS WebSocket client + reconnect
6. Session list + QR pairing + mode toggle
7. Polish + docs + localization

Test plan

  • macOS build passes (swift build)
  • iOS simulator build passes (xcodebuild)
  • 61 protocol test assertions pass
  • End-to-end: Mac + Fly.io relay + iOS device
  • Real-device suspend/resume testing

See .agents/sessions/2026-03-22-mori-remote/test-plan.md for detailed test scenarios.

vaayne and others added 30 commits March 21, 2026 00:29
Replace the simple pencil icon and +N/-N badges with detailed
starship-style indicators: ↑N ahead, ↓N behind, +N staged,
~N modified, ?N untracked. Data was already parsed by GitStatusInfo
but collapsed into a boolean — now passed through to the model.

Includes backwards-compatible JSON decoding for existing data.
- Show ⊘ indicator when branch has no upstream (not pushed yet)
- Show relative activity time (e.g. "3m ago", "2h ago") in subtitle
- Track lastActiveAt on worktree selection
- Add hasUpstream field to Worktree model with backwards-compat decoding
Enhanced sidebar with starship-style git status
* ✨ feat: add WorkflowStatus enum with 5 cases and display helpers

* ✨ feat: add workflowStatus field to Worktree with backwards-compatible decoding

* ✨ feat: update SidebarMode to workspaces|tasks with backwards-compatible decoding

* ✅ test: add WorkflowStatus/SidebarMode tests and update existing tests for new enum values

* ✨ feat: add TaskWorktreeRowView for task mode sidebar

Shows branch name + project shortName badge + git status indicators + alert
badge. Follows WorktreeRowView patterns with added cross-project context.

* ✨ feat: add WorkflowStatusMenu reusable submenu component

Reusable SwiftUI view showing all 5 WorkflowStatus options with checkmark
on current status. Used in context menus for both sidebar modes.

* ✨ feat: add TaskSidebarView with status grouping and nested windows

Groups worktrees by WorkflowStatus (inProgress → needsReview → todo → done).
Cancelled hidden by default with toggle. Done collapsed by default. Empty
groups hidden. Sorted by lastActiveAt descending within each group. Windows
nest under each worktree. Filters out unavailable worktrees.

* ✨ feat: add SidebarContainerView with Tasks/Workspaces toggle

Segmented control at top conditionally renders TaskSidebarView or
WorktreeSidebarView. Passes through all callbacks including the new
onSetWorkflowStatus.

* ✨ feat: add Set Status context menu and onSetWorkflowStatus callback

Adds onSetWorkflowStatus callback to WorktreeSidebarView and integrates
WorkflowStatusMenu into worktree row context menus in both sidebar views.

* ✨ feat: add workflow status management and auto-transition to WorkspaceManager

- Add setWorkflowStatus(worktreeId:status:) to update AppState + persist (3.1)
- Add setSidebarMode() to persist sidebar toggle state
- Update selectWorktree() to sync selectedProjectId for cross-project selection (3.2)
- Add auto-transition logic: todo → inProgress on first activity in coordinatedPoll() (3.5)
  Signals: modifiedCount > 0, stagedCount > 0, aheadCount > 0, agentState != .none

* ✨ feat: wire SidebarContainerView into HostingControllers with callbacks

- Replace WorktreeSidebarView with SidebarContainerView in SidebarContentView (3.3)
- Add onToggleSidebarMode and onSetWorkflowStatus callbacks through the hosting chain
- Wire callbacks in AppDelegate to WorkspaceManager.setSidebarMode() and setWorkflowStatus() (3.4)

* 📝 docs: update task checklist and handoff for Phase 3 completion

* ✨ feat: add setWorkflowStatus case to IPCCommand enum

* ✨ feat: add handleSetWorkflowStatus in IPCHandler

* ✨ feat: add StatusCmd CLI subcommand for mori status

* ✨ feat: add Set Worktree Status actions to CommandPaletteDataSource

* ✨ feat: handle Set Worktree Status palette actions in AppDelegate

* ✅ test: add IPC tests for setWorkflowStatus command encoding/decoding

* 🐛 fix: correct type inference and property path in palette status actions

* 📝 docs: update tasks and handoff for Phase 4 completion

* 🌐 i18n: add localized strings for task mode sidebar (en + zh-Hans)

Add .localized() calls for computed strings in MoriUI views (WorkflowStatusMenu,
TaskSidebarView, SidebarContainerView, TaskWorktreeRowView) and add corresponding
entries to all 6 Localizable.strings files:
- MoriUI: Tasks/Workspaces toggle, Set Status menu, WorkflowStatus display names,
  Show/Hide Cancelled, relative time strings
- App: Command palette status action strings (Status: %@, Current status, Set to)
- CLI: StatusCmd help text and argument descriptions

* 📝 docs: add Task Mode Sidebar entry to CHANGELOG

Add feature description under [Unreleased] covering workflow status
grouping, sidebar toggle, status management interfaces (context menu,
command palette, CLI), auto-transition, and i18n support.

* 📝 docs: update tasks.md and handoff.md for Phase 5 completion

* 🐛 fix: address review feedback — auto-transition and remove worktree in task mode

- Broaden auto-transition signal to use hasUncommittedChanges (covers
  branches without upstream where aheadCount stays 0)
- Add onRemoveWorktree callback and "Remove Worktree" context menu
  action to TaskSidebarView (was missing from task mode)
- Pass onRemoveWorktree through SidebarContainerView to TaskSidebarView

* 🐛 fix: address PR review — auto-transition guard and selectWindow project sync

- Add autoTransitionedWorktrees set to prevent re-transitioning worktrees
  that the user manually set back to todo (P2 review feedback)
- Mark worktrees as transitioned on manual setWorkflowStatus too, so
  auto-transition respects manual overrides
- Sync selectedProjectId in selectWindow() for cross-project task mode
  window selection (P2 review feedback)
- Show worktree name as primary label with project name as subtitle
- Remove auto-transition from todo → inProgress on activity detection
- Remove auto-sort by lastActiveAt within status groups
* ✨ feat: add network proxy settings to Mori settings UI

Add a Network tab in Settings that lets users configure proxy environment
variables (http_proxy, https_proxy, all_proxy, no_proxy) which are
automatically applied to all tmux sessions managed by Mori via
`tmux set-environment -g`. Both lowercase and uppercase variants are set.

* 🐛 fix: debounce proxy apply and reapply after session creation

- Cancel in-flight proxy apply task before starting a new one to prevent
  interleaved tmux writes from rapid edits
- Add onSessionCreated callback to WorkspaceManager so proxy env vars
  are reapplied after the tmux server starts (fixes fresh-launch case
  where no server exists yet)
- Centralize proxy apply logic in applyProxyToTmux() helper

* 🐛 fix: check cancellation in proxy apply loop and await before templates

- Add Task.isCancelled guard in ProxySettingsApplicator.apply() loop so
  cancelled tasks stop issuing tmux writes immediately
- Make onSessionCreated async so WorkspaceManager awaits proxy setup
  before running TemplateApplicator (ensures proxy env is available for
  template commands)

* ♻️ refactor: redesign network proxy settings UI

- Three proxy modes: System (reads macOS scutil --proxy), Manual, None
- System mode auto-detects HTTP/HTTPS/SOCKS proxy and bypass list from
  macOS system configuration
- Manual mode with "Same as HTTP" toggle for HTTPS proxy
- Explicit Apply button instead of fire-on-keystroke (eliminates race
  conditions entirely)
- Read-only fields in System mode to show detected values
- Renamed allProxy to socksProxy for clarity
- Updated en + zh-Hans localization strings

* 📝 docs: document proxy limitation and add network-proxy.md

- Update settings hint to clearly state proxy changes only affect new
  tabs/panes, existing shells keep their current environment
- Add docs/network-proxy.md with full documentation: modes, env vars,
  the new-panes-only limitation with manual workaround, and system
  proxy detection details
- Updated en + zh-Hans localization
The TmuxThemeApplicator (which sets status bar off, pane border colors,
etc.) was only applied at startup and on theme changes. Sessions created
after startup inherited the default tmux config, leaving the status bar
visible.

Apply theme settings in the onSessionCreated callback so every new
session immediately gets the correct theme, including status=off.
Add detailed implementation plan for Mori Remote — an iOS companion app
that connects to tmux sessions on Mac via a cloud-hosted WebSocket relay,
using libghostty for GPU-accelerated terminal rendering.

- 7-phase plan with 55 tasks, reviewed by internal reviewer and Codex
- Architecture: MoriRemoteHost (Mac) -> Go relay (Fly.io) -> iOS app
- Key components: ghostty Remote termio backend (Zig), MoriRemoteProtocol
  shared package, Go relay service, iOS app with libghostty
- Add ghostty:sync mise task for syncing fork with upstream
- Point .gitmodules to vaayne/ghostty fork
- Build script defaults to universal target (macOS + iOS + iOS Simulator)
- Auto-detects iphoneos SDK, falls back to native if missing
- Add --native flag for explicit macOS-only builds
- Patch only applied for native builds; universal uses upstream as-is
- Verify slice count after build
- MoriRemote/ Xcode project (XcodeGen) targeting iOS 17+
- GhosttyAppContext: singleton managing ghostty_app_t with iOS callbacks
- GhosttySurfaceUIView: UIView + CAMetalLayer hosting ghostty surface
- TerminalView: SwiftUI UIViewRepresentable wrapper
- Builds for iOS Simulator (iPhone 16, iOS 18.2)
- Phase 1 complete: fork + universal build + iOS proof-of-life
- ghostty fork: Remote.zig backend (fd-pair IO, no pty)
- ghostty fork: remote_read_fd/write_fd in surface config
- MoriRemoteProtocol: cross-platform Swift package (macOS + iOS)
  - ControlMessage enum with JSON Codable (handshake, attach, detach, resize, etc.)
  - ConnectionState machine (disconnected -> pairing -> connected -> attached)
  - SessionInfo, Role, SessionMode, ErrorCode types
  - encodeMessage/decodeMessage helpers
- mori-relay/: Go service with nhooyr.io/websocket
- POST /pair: generate one-time pairing tokens (5min expiry, rate limited)
- GET /ws: WebSocket upgrade with token+role or session_id reconnect
- Binary frame relay between paired host/viewer connections
- Session TTL with automatic cleanup
- Dockerfile + fly.toml for Fly.io deployment (nrt region)
- GET /health endpoint
New Swift executable target in root Package.swift for the Mac relay
connector. Uses swift-argument-parser with serve, sessions, and qrcode
subcommands. Depends on MoriRemoteProtocol and MoriTmux packages.
… pipe

RelayConnector actor manages WSS connection to relay as host role.
PTYBridge uses forkpty() to spawn tmux attach-session and provides
read/write access to the pty master. Bidirectional pipe: pty output
streams to WebSocket binary frames, WebSocket input writes to pty.
Handles control messages (attach, detach, resize, mode change, heartbeat).
…ly names

SessionLister uses TmuxCommandRunner + TmuxParser to list local tmux
sessions and converts them to SessionInfo with display-friendly names
via SessionNaming.parse(). Mori-convention sessions show as
"project / branch", non-Mori sessions use raw names.
GroupedSessionManager creates tmux grouped sessions (new-session -t)
for interactive remote mode so iOS gets independent window sizing.
Tracks active sessions, cleans up on viewer disconnect, and runs
periodic GC (60s interval) to remove stale/orphaned grouped sessions.
QRCodeGenerator produces PNG and ASCII terminal representations of QR
codes for iOS pairing. Uses CIQRCodeGenerator filter with medium error
correction. ASCII output uses Unicode half-block characters for compact
terminal display. QRCode CLI subcommand supports --png flag and can
request pairing tokens from the relay /pair endpoint.
SessionIDStore persists relay session IDs to disk for cross-restart
reconnection. Serve command resolves session ID from CLI flag, stored
file, or none. RelayConnector implements exponential backoff with jitter
(1s base, 60s max, 10 attempts) and reuses session ID within TTL.
LoopbackRelay is an in-process WebSocket relay using Network.framework
NWListener that pairs host/viewer connections and relays bytes
bidirectionally. LoopbackHarness runs an end-to-end test: starts relay,
connects RelayConnector as host, verifies session listing and handshake.
Available via `mori-remote-host loopback` CLI subcommand.
- Add MoriRemoteProtocol SPM dependency to project.yml
- Import MoriRemoteProtocol in MoriRemoteApp
- Add scenePhase observer for future WebSocket lifecycle
- Dark color scheme + status bar hidden for terminal UX
- Enhanced error state UI
Create GhosttyRemoteSurfaceView that configures ghostty surface
with remote_read_fd/remote_write_fd for the Remote termio backend.
Accepts a PipeBridge for fd pair management.
Actor-based bridge managing two pipe() fd pairs:
- pipeToGhostty: external data written to ghostty's read_fd
- pipeFromGhostty: ghostty's input read and forwarded via callback
Async read loop with non-blocking I/O. WebSocket integration
deferred to Phase 5B; onInputFromGhostty callback provided as hook.
Rewrite TerminalView to use GhosttyRemoteSurfaceView with PipeBridge.
Safe area: ignores top container + keyboard, black background fills
all edges. Includes canned VT byte test (green text) to verify
Remote backend rendering on simulator.
UIKit accessory view with scrollable key buttons for terminal-specific
keys missing from iOS keyboard: Esc, Ctrl (toggle), Tab, pipe, tilde,
dash, slash, and arrow keys. Ctrl+letter sends control characters.
Blurred background with system material styling.
…hine

- URLSessionWebSocketTask-based actor connecting to relay as viewer
- Binary frames for terminal data, JSON text frames for control messages
- Integrates with PipeBridge: relay data -> ghostty, ghostty input -> relay
- Automatic reconnection with exponential backoff and jitter
- Session ID extraction from handshake capabilities
- TerminalView wires PipeBridge <-> RelayClient bidirectionally
- PipeBridge callback updated to async throws for WebSocket forwarding
…oreground

- scenePhase .background: immediately disconnect relay (no background keep-alive)
- scenePhase .active: reconnect using stored session ID if was connected
- Track wasConnectedBeforeBackground state for clean lifecycle
- RelayClient passed to TerminalView for integrated data flow
vaayne added 27 commits March 25, 2026 20:06
- SessionListView: list of tmux sessions from relay control channel
- SessionRow: display name, window count, attached status indicator
- Pull-to-refresh for session list updates
- "Forget This Device" menu action for device revocation
- Empty state with refresh prompt
- ModeToggleButton: floating capsule overlay on terminal view
- Sends ControlMessage.modeChange to relay -> host changes tmux attach mode
- Visual indicator: eye icon for read-only, keyboard icon for interactive
- ConnectionStatusView: small overlay showing connected/disconnected/reconnecting
- Auto-hides when connected to avoid obscuring terminal
- Animated transitions between states
- TerminalView uses GeometryReader to detect size changes (orientation/layout)
- Estimates cols/rows from pixel size and sends ControlMessage.resize
- Added floating controls: session name, detach button, mode toggle overlay
- Updated to accept mode/detach/resize callbacks from AppViewModel
- AppViewModel: @observable central coordinator for navigation, relay, and state
- Navigation flow: QR scanner -> session list -> terminal (with back navigation)
- forgetDevice(): clears Keychain session ID, disconnects, returns to scanner
- parseRelayURL(): extracts host + token from mori-relay:// URLs
- Updated MoriRemoteApp to use AppViewModel for screen management
- Background/foreground lifecycle integrated with AppViewModel
Add English + zh-Hans Localizable.strings for both targets:
- MoriRemoteHost: CLI help text, session listing, QR code output
- MoriRemote iOS: QR scanner, session list, mode toggle, connection status
- Follow existing .localized() pattern from MoriCLI
- Computed strings use String(localized:), SwiftUI Text literals auto-localize
- README: add Mori Remote section with architecture diagram and quick start
- CHANGELOG: add Mori Remote features under [Unreleased]
- AGENTS.md: add iOS/relay build tasks, iOS 17+ convention, universal build note, new string file locations
New tasks:
- ios:build: build MoriRemote for iOS simulator via xcodebuild
- ios:test: run MoriRemoteProtocol tests
- test:protocol: run MoriRemoteProtocol tests directly
- relay:dev: run Go relay locally
- relay:deploy: deploy relay to Fly.io
Document test procedures for:
- iOS suspend/resume testing on iOS 17.x and 18.x (7 scenarios)
- End-to-end Mac + Fly.io + iOS device testing (9 scenarios)
- RTT measurement approach (timestamp echo + screen recording)
All Phase 7 tasks complete. Final status updated in plan.md.
…ection

Cover token generation/pairing flow, expired token rejection, rate
limiting (checkRate + HTTP 429), session reconnection with session_id,
and cleanup of expired tokens/sessions/rates.
When Ctrl is active and a non-modifier key is pressed, send the
corresponding control character (Ctrl+C = 0x03, etc.) instead of
the unmodified key bytes. Multi-byte keys (arrows, Esc sequences)
are sent unmodified since Ctrl does not apply to them.
- Remove unreachable `defer` after execvp in the child process path
  (execvp replaces process image; _exit on failure skips defer)
- Consolidate waitpid: terminate() only sends signals and closes fd,
  monitorChildExit() handles all reaping via blocking waitpid
Use DispatchSource.makeReadSource on the read fd to efficiently wait
for data instead of polling 100 times/second with Task.sleep. The
dispatch source fires only when data is available, eliminating
unnecessary CPU wake-ups and reducing battery drain.
Replace direct state assignments with transitionState(to:) helper
that validates transitions via ConnectionState.transition(to:),
matching the pattern used by the iOS RelayClient. Invalid transitions
are now logged instead of silently applied.
When handleControlMessage fails to attach a session or change mode,
send a ControlMessage.error back to the viewer with the appropriate
ErrorCode (.sessionNotFound for attach, .internalError for mode
change) so the viewer can display the failure reason.
Task { await ... } may not complete before iOS suspends the app.
Use beginBackgroundTask(expirationHandler:) to request additional
execution time, and make disconnectForBackground() properly async.
Replace fragile string replacement of "https://"/"http://" prefixes
with proper URLComponents parsing to extract host and optional port.
Remove setCancelHandler that captured actor-isolated self across
isolation boundaries, causing a Swift 6 strict concurrency error.
Stop is already called explicitly; the cancel handler was redundant.
Add railway.toml config and update mise relay:deploy task to use Railway CLI.
- Remove embedded ws:// scheme from mori-relay:// URI (was mori-relay://ws://host)
- Now generates correct format: mori-relay://host:port/token
- iOS parser auto-selects ws:// for localhost/LAN, wss:// for public hosts
- Mac app now calls POST /pair to get a relay-generated token
- Was generating UUID locally which the relay rejected
- QR code and pairing URI update async after token is received
- Added requestPairingToken() with ws->http scheme conversion
- Added pairingFailed error case
- Use synchronous HTTP request (semaphore) for POST /pair
- QR code and pairing URI included in the returned model
- Settings view shows QR immediately when toggle is enabled
- Async relay connection starts after QR is displayed
@vaayne vaayne force-pushed the main branch 3 times, most recently from 2575fa1 to 5f5bde4 Compare March 27, 2026 11:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants