Vision
A macOS overlay that captions everything you hear (system audio + mic, language auto-detected, on-device Whisper) and shows a translation under every sentence — powered by the user's own Claude/Codex CLI subscription (Agent SDK credit pool) with a bundled local-model fallback. Live summary, meeting board, reply suggestions, quick translate, and an auto-saved Markdown transcript per session. No cloud backend, no account, no telemetry.
Full spec: docs/PROPOSAL.md (v1.1, in this repo). Design package: design/ (tokens, app icon, menu bar glyph, 2× mockups of every screen — see design/README.md for the asset→section mapping).
Reference paths for agents
- This repo:
docs/PROPOSAL.md, design/
- Local machine (Claude-direct tickets):
~/Projects/z-docs/PROPOSAL-live-caption-translator.md, ~/Projects/z-design/livecap-design/
- QuadWork VPS mirror (quadwork tickets; synced by operator when the livecap project page is created):
~/projects/z-docs/PROPOSAL-live-caption-translator.md, ~/projects/z-design/livecap-design/ — use local paths, do not re-download from GitHub.
Decisions (locked)
- Stack: Tauri 2 + TypeScript (UI) + Rust (audio/STT core). Audio capture + live STT architecture derived from Meetily (MIT); attribution kept.
- STT is always on-device (whisper.cpp / Parakeet, Metal). Whisper's own translate task is X→English only and is NOT used — translation is a separate engine layer.
- Translation engine = one interface, three tiers: (1) Claude/Codex CLI via headless stream-json (default when detected; model pinned to Haiku), (2) llama.cpp + Qwen3 4B Instruct local fallback (ships in MVP), (3) BYO API key (post-MVP). No PTY workaround for CLI billing — ever. CLI usage draws from the user's Agent SDK credit pool; the app shows a credit gauge and auto-falls back to local when the pool runs low.
- CLI integration pattern follows open-design (Apache-2.0): spawn-per-session
claude -p --input-format stream-json --output-format stream-json --verbose [--include-partial-messages], prompt via stdin, --session-id/--resume, strip ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN from child env so CLI subscription auth wins (open-design #398 documents the failure mode).
- Design: "Glass" system (
design/system/tokens.css is canonical). Five rules incl. hierarchy-is-brightness and one amber accent. Overlay is excluded from screen capture (macOS sharingType) — this is a launch gate, not a nice-to-have.
- Session archive: incremental-written Markdown to
~/Documents/LiveCap/ — crash mid-meeting loses nothing.
- macOS (Apple Silicon) first. EN↔KO launch pair, auto-detection built in. App UI in English.
Agent split
| Label |
Who |
What |
Why |
agent/claude |
Local Claude agent on the operator's Mac |
App shell, audio/STT, UI, CLI-integration verification, E2E |
Needs real audio devices, real CLIs, a real screen — must run the app on this device |
agent/quadwork |
QuadWork batch (VPS) |
Engine adapters, stream parser, credit accounting, archive writer, prompt pipeline |
Pure logic, fixture-driven, headless-testable on Linux CI |
QuadWork tickets must keep modules free of Tauri/macOS imports — they are plain TS/Rust packages consumed by the app. Integration into the app is done by the local agent in the consuming ticket.
Engineering policy: no mock/temp/stub code (binding)
- BANNED in merged application code: TODO/FIXME/HACK deferring functionality; functions returning hardcoded/sample data; dev-only fake branches; placeholder UI standing in for unimplemented features. CI greps
TODO|FIXME|HACK|placeholder|mock outside test dirs; any hit must be justified in the PR description.
- Mocks/fixtures live ONLY in test code. Application code never imports them.
- If an external dependency is unavailable, the ticket names the substitute reached through the SAME code path as production (e.g. recorded-audio fixture fed through the real capture pipeline; fixture JSONL fed through the real stream parser). Never a parallel fake path.
- If a scope can't be completed for real, STOP and comment on the issue — a smaller fully-working scope beats a full-width stubbed one.
- Every AC is demonstrated by a test or a reproducible command documented in the PR description.
- Security: no credentials/transcripts/model weights in the repo (see
.gitignore, SECURITY.md); never log caption content; no network calls outside the engine layer.
Operator gates (deliberately last)
livecap.app domain (registered by operator), Apple Developer signing + notarization, release distribution (#14). Everything before the gates runs locally end-to-end unsigned.
Tickets & order
| # |
Ticket |
Agent |
Depends on |
| #2 |
Scaffold: Tauri workspace + CI + no-stub gate |
claude |
— |
| #3 |
Phase 0: CLI cost & latency PoC |
claude |
— (parallel with #2) |
| #4 |
Audio capture + on-device STT core |
claude |
#2 |
| #5 |
Engine interface + Claude CLI adapter |
quadwork |
#2, #3 (blocking: stream-json fixtures) |
| #6 |
Local LLM fallback engine |
quadwork |
#5 (interface) |
| #7 |
Credit accounting + auto-fallback |
quadwork |
#5 |
| #8 |
Session archive writer |
quadwork |
#2 |
| #9 |
LLM extras pipeline (summary/board/replies) |
quadwork |
#5 |
| #10 |
Glass window shell |
claude |
#2 |
| #11 |
Caption feed UI + pipeline integration |
claude |
#4 #5 #6 #7 #8 #10 |
| #12 |
Onboarding + Settings |
claude |
#5 #6 #7 |
| #13 |
E2E verification pass |
claude |
#11 #12 |
| #14 |
[GATE] Domain, signing, release |
gate |
#13 |
QuadWork batch 1 (after #2 merges): #5 → #6/#7/#9 (parallel) + #8 (independent). Claude-direct track runs #3/#4/#10 in parallel with the batch.
Vision
A macOS overlay that captions everything you hear (system audio + mic, language auto-detected, on-device Whisper) and shows a translation under every sentence — powered by the user's own Claude/Codex CLI subscription (Agent SDK credit pool) with a bundled local-model fallback. Live summary, meeting board, reply suggestions, quick translate, and an auto-saved Markdown transcript per session. No cloud backend, no account, no telemetry.
Full spec:
docs/PROPOSAL.md(v1.1, in this repo). Design package:design/(tokens, app icon, menu bar glyph, 2× mockups of every screen — seedesign/README.mdfor the asset→section mapping).Reference paths for agents
docs/PROPOSAL.md,design/~/Projects/z-docs/PROPOSAL-live-caption-translator.md,~/Projects/z-design/livecap-design/~/projects/z-docs/PROPOSAL-live-caption-translator.md,~/projects/z-design/livecap-design/— use local paths, do not re-download from GitHub.Decisions (locked)
claude -p --input-format stream-json --output-format stream-json --verbose [--include-partial-messages], prompt via stdin,--session-id/--resume, stripANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKENfrom child env so CLI subscription auth wins (open-design #398 documents the failure mode).design/system/tokens.cssis canonical). Five rules incl. hierarchy-is-brightness and one amber accent. Overlay is excluded from screen capture (macOSsharingType) — this is a launch gate, not a nice-to-have.~/Documents/LiveCap/— crash mid-meeting loses nothing.Agent split
agent/claudeagent/quadworkQuadWork tickets must keep modules free of Tauri/macOS imports — they are plain TS/Rust packages consumed by the app. Integration into the app is done by the local agent in the consuming ticket.
Engineering policy: no mock/temp/stub code (binding)
TODO|FIXME|HACK|placeholder|mockoutside test dirs; any hit must be justified in the PR description..gitignore,SECURITY.md); never log caption content; no network calls outside the engine layer.Operator gates (deliberately last)
livecap.appdomain (registered by operator), Apple Developer signing + notarization, release distribution (#14). Everything before the gates runs locally end-to-end unsigned.Tickets & order
QuadWork batch 1 (after #2 merges): #5 → #6/#7/#9 (parallel) + #8 (independent). Claude-direct track runs #3/#4/#10 in parallel with the batch.