Skip to content

[EPIC] LiveCap MVP — local-first live captions & translation overlay #1

Description

@realproject7

Vision

A macOS overlay that captions everything you hear (system audio + mic, language auto-detected, on-device Whisper) and shows a translation under every sentence — powered by the user's own Claude/Codex CLI subscription (Agent SDK credit pool) with a bundled local-model fallback. Live summary, meeting board, reply suggestions, quick translate, and an auto-saved Markdown transcript per session. No cloud backend, no account, no telemetry.

Full spec: docs/PROPOSAL.md (v1.1, in this repo). Design package: design/ (tokens, app icon, menu bar glyph, 2× mockups of every screen — see design/README.md for the asset→section mapping).

Reference paths for agents

  • This repo: docs/PROPOSAL.md, design/
  • Local machine (Claude-direct tickets): ~/Projects/z-docs/PROPOSAL-live-caption-translator.md, ~/Projects/z-design/livecap-design/
  • QuadWork VPS mirror (quadwork tickets; synced by operator when the livecap project page is created): ~/projects/z-docs/PROPOSAL-live-caption-translator.md, ~/projects/z-design/livecap-design/ — use local paths, do not re-download from GitHub.

Decisions (locked)

  • Stack: Tauri 2 + TypeScript (UI) + Rust (audio/STT core). Audio capture + live STT architecture derived from Meetily (MIT); attribution kept.
  • STT is always on-device (whisper.cpp / Parakeet, Metal). Whisper's own translate task is X→English only and is NOT used — translation is a separate engine layer.
  • Translation engine = one interface, three tiers: (1) Claude/Codex CLI via headless stream-json (default when detected; model pinned to Haiku), (2) llama.cpp + Qwen3 4B Instruct local fallback (ships in MVP), (3) BYO API key (post-MVP). No PTY workaround for CLI billing — ever. CLI usage draws from the user's Agent SDK credit pool; the app shows a credit gauge and auto-falls back to local when the pool runs low.
  • CLI integration pattern follows open-design (Apache-2.0): spawn-per-session claude -p --input-format stream-json --output-format stream-json --verbose [--include-partial-messages], prompt via stdin, --session-id/--resume, strip ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN from child env so CLI subscription auth wins (open-design #398 documents the failure mode).
  • Design: "Glass" system (design/system/tokens.css is canonical). Five rules incl. hierarchy-is-brightness and one amber accent. Overlay is excluded from screen capture (macOS sharingType) — this is a launch gate, not a nice-to-have.
  • Session archive: incremental-written Markdown to ~/Documents/LiveCap/ — crash mid-meeting loses nothing.
  • macOS (Apple Silicon) first. EN↔KO launch pair, auto-detection built in. App UI in English.

Agent split

Label Who What Why
agent/claude Local Claude agent on the operator's Mac App shell, audio/STT, UI, CLI-integration verification, E2E Needs real audio devices, real CLIs, a real screen — must run the app on this device
agent/quadwork QuadWork batch (VPS) Engine adapters, stream parser, credit accounting, archive writer, prompt pipeline Pure logic, fixture-driven, headless-testable on Linux CI

QuadWork tickets must keep modules free of Tauri/macOS imports — they are plain TS/Rust packages consumed by the app. Integration into the app is done by the local agent in the consuming ticket.

Engineering policy: no mock/temp/stub code (binding)

  1. BANNED in merged application code: TODO/FIXME/HACK deferring functionality; functions returning hardcoded/sample data; dev-only fake branches; placeholder UI standing in for unimplemented features. CI greps TODO|FIXME|HACK|placeholder|mock outside test dirs; any hit must be justified in the PR description.
  2. Mocks/fixtures live ONLY in test code. Application code never imports them.
  3. If an external dependency is unavailable, the ticket names the substitute reached through the SAME code path as production (e.g. recorded-audio fixture fed through the real capture pipeline; fixture JSONL fed through the real stream parser). Never a parallel fake path.
  4. If a scope can't be completed for real, STOP and comment on the issue — a smaller fully-working scope beats a full-width stubbed one.
  5. Every AC is demonstrated by a test or a reproducible command documented in the PR description.
  6. Security: no credentials/transcripts/model weights in the repo (see .gitignore, SECURITY.md); never log caption content; no network calls outside the engine layer.

Operator gates (deliberately last)

livecap.app domain (registered by operator), Apple Developer signing + notarization, release distribution (#14). Everything before the gates runs locally end-to-end unsigned.

Tickets & order

# Ticket Agent Depends on
#2 Scaffold: Tauri workspace + CI + no-stub gate claude
#3 Phase 0: CLI cost & latency PoC claude — (parallel with #2)
#4 Audio capture + on-device STT core claude #2
#5 Engine interface + Claude CLI adapter quadwork #2, #3 (blocking: stream-json fixtures)
#6 Local LLM fallback engine quadwork #5 (interface)
#7 Credit accounting + auto-fallback quadwork #5
#8 Session archive writer quadwork #2
#9 LLM extras pipeline (summary/board/replies) quadwork #5
#10 Glass window shell claude #2
#11 Caption feed UI + pipeline integration claude #4 #5 #6 #7 #8 #10
#12 Onboarding + Settings claude #5 #6 #7
#13 E2E verification pass claude #11 #12
#14 [GATE] Domain, signing, release gate #13

QuadWork batch 1 (after #2 merges): #5#6/#7/#9 (parallel) + #8 (independent). Claude-direct track runs #3/#4/#10 in parallel with the batch.

Metadata

Metadata

Assignees

No one assigned

    Labels

    epicTop-level epic

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions