feat(matrix-bridge): heartbeat spinner + live tool elapsed; pane never looks frozen#17
Merged
Merged
Conversation
…o Replicas
Adds a stateless Worker that accepts Telegram bot webhooks and translates each
message into the right Replicas API call:
- First message in (chat_id, message_thread_id) -> POST /v1/replica
- Subsequent messages on that key -> POST /v1/replica/{id}/messages
- Acknowledgement via Telegram setMessageReaction (eyes)
- Mapping persisted in Workers KV with a 7-day TTL
Includes typed request/response shapes, webhook secret verification,
idempotency guard for duplicate Telegram retries, and a vitest suite covering
routing, follow-ups, idempotency, error paths, forum-topic threading, and
helper slugification. Tests run inside the Workers runtime via
@cloudflare/vitest-pool-workers.
This is the Telegram counterpart to the existing native Slack automation in
the Replicas org and is the recommended path for piping Replicas tasks into a
Beeper inbox (via mautrix-telegram) without writing a Matrix bot.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Worker deployed to https://replicas-telegram-bridge.jadengarza.workers.dev with the MAP namespace id wrangler created during first deploy. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
The Worker now prefixes every spawn's initial prompt (and follow-ups) with a
routing header + a one-line hint pointing the agent at the helper scripts:
[tg:chat_id=<id>:thread_id=<id>]
# Spawned from Telegram. Post status + your final result back with
`~/.replicas/bin/tg-reply.sh "<text>"`. Run
`~/.replicas/bin/tg-target-detect.sh` once with this prompt's first
line on stdin to cache the target. See `~/.replicas/TELEGRAM_REPLY.md`.
Adds three env files (uploaded to env 5f06f598- as well):
- `~/.replicas/bin/tg-reply.sh` — wraps Telegram `sendMessage`,
handles the 4096-char cap by chunking, reads target from /tmp/tg-target.
- `~/.replicas/bin/tg-target-detect.sh` — parses the `[tg:...]` header off
the first line of stdin and stashes chat_id/thread_id for later calls.
- `~/.replicas/TELEGRAM_REPLY.md` — reference doc for the agent.
`TG_BOT_TOKEN` is also pinned on env 5f06f598- so spawned workspaces inherit
it without further config.
Test count goes 13 -> 16. Prefix shape is covered (header line, hint,
follow-up parity) plus DM vs forum-topic threading.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
…Text
Adds the second half of the agent <-> Telegram surface: a "live" status line
that gets edited in place as the agent thinks and runs tools, so the user can
watch progress without the chat being flooded by individual messages.
- New env file ~/.replicas/bin/tg-status.sh.
First call sendMessage + caches message_id in /tmp/tg-status-mid; subsequent
calls editMessageText against that id. Falls back to a fresh sendMessage if
the edit fails (message too old, deleted, etc.). Treats Telegram's
"message is not modified" error as success.
- TELEGRAM_REPLY.md rewritten with the two-channel protocol:
tg-status.sh for the live line (thinking / tool calls / phase changes),
tg-reply.sh for new messages (final result, PR links, blockers).
Includes an emoji convention table so users learn to read it at a glance.
- Worker's prompt prefix updated so spawned agents see both helpers and the
step-by-step setup instructions on line 2-6 of every initial prompt.
- Tests updated to assert both helpers are referenced in the prefix.
Probed Replicas events surface during this work:
- GET /v1/replica/{id}/events returns an SSE stream (held the connection open
until our 15s timeout; emits engine.status.changed records). Useful for an
external poller in a follow-up Durable Object, not for the Worker proper.
- GET /v1/replica/{id}/history returns {events, has_more, total} with
per-event {type, role, created_at, content}. Could be polled to surface
raw thinking tokens; deferred to a follow-up PR.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
…e is gone
Previously, if KV mapped a chat to a replica that had been deleted (or
expired during inactivity), every follow-up message returned an error to the
user with no recovery path until the KV entry's 7-day TTL expired.
Now sendFollowUp returns {ok, gone}; on 404/410 the Worker invalidates the
KV entry and falls back to createReplica, transparently re-spawning into a
fresh workspace that picks up the latest env files and instructions.
This was hit in production: chat 91269535 was pinned to a replica that had
been spawned before the tg-reply.sh / tg-status.sh helpers existed, so
follow-ups kept routing into a workspace with no way to talk back.
Tests bumped 16 -> 17 with a 404-fallback case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
…via bash
Two real-world bugs uncovered when an actual user message hit the bridge:
1. Env files uploaded to a *project* env (5f06f598-…) are listed by the
Replicas API as files_uploaded on spawned workspaces but don't actually
land on disk. Only Global env files (77d1bf95-…) get materialised into
~/.replicas/. The four Telegram helpers and TELEGRAM_REPLY.md were
re-uploaded to Global as a result (not done in this commit — done via the
`replicas env files set global` CLI).
2. Files mount with mode 644 — no execute bit — so `~/.replicas/bin/foo.sh`
exits with 126 ("permission denied") when invoked directly. The
spawned agent had to chmod +x before its first call. Switching every
reference to `bash ~/.replicas/bin/foo.sh` sidesteps the executable bit
entirely; bash will read+exec a text file with 644 perms fine.
The prompt prefix and TELEGRAM_REPLY.md are both updated to use the bash-
prefixed invocation. Worker tests still green (17/17).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
…y sends
The old chunker did:
chunk "$MSG" | while read -r piece; do send_one "$piece"; done
read -r reads one LINE at a time, so a multi-line MSG was sent as N
separate sendMessage calls back-to-back — which hits Telegram's 1 msg/sec
per-chat rate limit and returns 429 on every call after the first. The
agent saw [tg-reply] sendMessage failed and gave up.
Live trace from a running spawned workspace confirmed:
- env had TG_BOT_TOKEN set
- /tmp/tg-target was correctly cached (chat_id=91269535)
- direct `curl -d text=raw-test-from-debug ...` succeeded (message_id 29 delivered)
- `bash -x tg-reply.sh "plain-ascii"` also succeeded
- only multi-line + emoji payloads hit the failure
Now:
- Send the whole MSG as ONE call when ${#MSG} <= MAX_LEN (Telegram accepts
embedded newlines in a single sendMessage body — we just need to not
split them).
- For genuinely long content (>4000 chars), chunk at character boundaries
and sleep 1s between sends to respect the rate limit.
- Parse `retry_after` out of 429 responses and retry once after sleeping.
- Echo the offending response prefix into the error so future agents can
see why a send failed instead of just "sendMessage failed".
Local end-to-end test against the live bot (multi-line + ✅, • bullets,
em-dash) returned exit 0 and the message landed in chat 91269535.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
…+ tool calls
Replaces the agent-driven tg-status.sh / tg-reply.sh pattern with a Durable
Object that polls each spawned replica's /v1/replica/{id}/history every ~2s
and projects thinking + tool_use + claude-result events directly to Telegram.
Why the change: the agent-driven approach asked Claude to call helper shell
scripts before every tool use. In practice agents skipped status calls,
batched final replies into separate sendMessage calls (hitting rate limits),
and saw stale copies of helpers when env files raced with workspace mount.
Result: spotty status, frequent "sendMessage failed", no clean stream.
Now:
- New ReplicaPoller Durable Object class (src/poller.ts). On /watch it
caches {replicaId, chatId, threadId} and sets a 1.5s alarm. Each alarm
fetches /history?limit=200&include=content&verbose=1, slices the events
after lastSeenCount, and:
* For claude-assistant blocks: thinking -> "🤔 …", tool_use -> emoji-
prefixed status line ("🔧 $ cmd", "📖 path", "📝 write", "✏️ edit",
"🔍 grep/glob", "🌐 url/query", "🧰 mcp__…"); upserts ONE Telegram
message via editMessageText.
* For claude-result: sends a final sendMessage with the result text
(chunked + sleep between chunks) and schedules a 30s cleanup alarm.
* Backs off to 6s on transient API failure; gives up on 404/410; bounds
the watch at 30 min.
- index.ts wires startWatcher(env, replicaId, msg) into both new spawns
and existing-replica follow-ups (so the poller re-arms across turns).
- Prompt prefix simplified: agents no longer need to know about tg-status.sh
or tg-reply.sh; their final assistant message becomes the Telegram reply
automatically. The [tg:chat_id=…] header stays for diagnostic purposes.
- wrangler.toml gains a [[durable_objects.bindings]] for WATCHER and a v1
migration declaring the ReplicaPoller class.
- Tests bumped 17 -> 18 with a watcher-attach assertion (mock WATCHER
namespace) and an updated prefix expectation.
Deployed as version 64474aaa-6cfe-4cf8-b34c-c06e3ed7354a.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Wraps alarm() in a try/catch so an exception inside the polling logic doesn't kill the alarm chain. Errors now log to wrangler tail and the next alarm is rescheduled on backoff. Adds structured log lines: - per-tick event counts (events, fresh, newStatus, sawResult) - final reply send size - editMessageText / sendMessage failures with response prefix Verified in production: end-to-end watch attached, 7 alarms fired at ~2s cadence, sawResult flipped on alarm 7, sendReply(1199 chars) ran. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
The poller was sending final replies as plain text, so Markdown in the agent's response — **bold**, backticks, fenced code blocks, [links] — showed up literal in Telegram. Now the reply is converted to Telegram's HTML parse_mode subset before sending. Conversions (src/markdown.ts): - ```lang\ncode\n``` -> <pre><code class="language-lang">…</code></pre> - ```code``` -> <pre>…</pre> with trailing newline trimmed - `inline` -> <code>…</code> - **bold** / __bold__ -> <b>…</b> - *italic* / _italic_ -> <i>…</i> (run after bold so we don't eat *bold*) - ~~strike~~ -> <s>…</s> - [text](url) -> <a href="url">text</a> - &, <, > are escaped in prose; code contents also escaped before tag wrap. If Telegram rejects the rendered HTML (mismatched entities, etc.) the poller logs the error and retries the same message with no parse_mode, so a parser hiccup never swallows the reply. 9 new unit tests in src/markdown.test.ts cover bold/italic disambiguation, code-block protection from outer markdown, HTML escaping, links, fenced blocks with and without language tags, and a realistic agent reply. Total suite 18 -> 27. Deployed as 179e5b16-40f8-4ea2-9939-e42ecb469404. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
…+ task header
The original status surface was a single line that got overwritten on each
edit. Users saw at most one frame — and for fast turns, often nothing
before the final reply. Reworked into a scrolling log:
- One Telegram message, edited in place via editMessageText, that
accumulates up to the last STATUS_LOG_MAX_LINES (10) events instead of
replacing the previous one. The user sees a live trail of what the
agent did.
- Status message opens *immediately* on /watch with "🤔 Starting…", so
the user sees activity right after the 👀 reaction — no dead air while
/history fills.
- A "Task:" header pulled from the triggering Telegram message's text is
prepended so the user can tell at a glance which task the status is
for, useful when juggling concurrent threads.
- Status frames render in HTML now (matching the final reply): tool
commands and file paths inside <code>, thinking previews in <i>,
header in <i>. status sends/edits set parse_mode=HTML.
- On claude-result the watcher appends a final "✅ Done in {n}s" frame
before sending the final reply as a fresh message, so the log captures
the completion and the reply persists alongside it.
WatchSpec gains an optional userText so the DO has the original task
context; index.ts passes msg.text/caption when calling /watch.
formatToolUse + headerFor are now exported so unit tests can pin them
down. New deploy: 8b19b668-f7f2-4678-a3cc-00b5218d03a9.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
…t defaults
Three changes aimed at making the Telegram UX feel real-time:
1. Poller cadence
- FIRST_POLL_DELAY_MS: 1500 -> 400 (first status frame ~1s after webhook)
- ACTIVE_POLL_INTERVAL_MS: 2000 -> 800 (~2.5x faster status updates)
- BACKOFF_POLL_INTERVAL_MS: 6000 -> 4000
2. Smaller history payload per poll
- /history limit 200 -> 50. We only ever care about events past
lastSeenCount; under steady-state polling fresh count is single
digits. Going to 50 cuts JSON parse + storage write costs without
missing anything except deep-backlog edge cases.
3. Faster agent for Telegram spawns
- POST /v1/replica now sets coding_agent=claude, model=claude-sonnet-4-6,
thinking_level=low by default. Probed the live API with these fields
and got a clean 200 + active replica.
- Each value is overridable per-deploy through three new env vars
(REPLICAS_AGENT_OVERRIDE / MODEL_OVERRIDE / THINKING_OVERRIDE)
plumbed through wrangler.toml so swapping in claude-haiku-4-5 for
even snappier chat -- or claude-opus-4-7 for harder coding tasks --
is a wrangler vars change rather than a code change.
Net expectation: status frames update at ~1Hz instead of 0.5Hz, Sonnet
is ~2-3x faster than Opus, and thinking_level: low skips extended
reasoning entirely on simple turns. End-to-end this should feel an
order of magnitude snappier on short tasks.
Tests 27/27 still green (typecheck only — the new fields are passed
through, not branched on, so no new logic to cover).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
…ow-ups don't replay old replies The screenshot showed both "Hello" and "What tools do you have" getting the identical reply "Hello! How can I help you today?" — a stale answer from the FIRST turn re-broadcast as the answer to the SECOND. Root cause: when a follow-up message arrives at an existing replica, the Worker re-arms the DO via /watch. The /watch handler was hard-resetting lastSeenCount to 0 — so the next alarm fetched /history and re-projected EVERY event in the replica, including the prior turn's claude-result, which got resent as "the answer" to the new question. Now /watch snapshots the current event count first via snapshotEventCount(replicaId) — a tiny GET /history?limit=1 that reads the "total" field when the API provides it (or falls back to events.length). That baseline becomes the new lastSeenCount, so subsequent polls only see events generated AFTER the follow-up message. For fresh spawns the baseline is 0 (no events yet); for follow-ups it's whatever the replica had accumulated. No behavior change on the first turn, real fix on every subsequent turn. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
…O is the only sender
Telegram screenshot showed the agent's reply landing in plain ** **
Markdown despite the markdown->HTML formatter being live. Root cause: the
agent kept reaching for the legacy `bash ~/.replicas/bin/tg-reply.sh`
helper -- which sends `sendMessage` with no parse_mode -- because the
scripts were still mounted on the Global env. So the agent's plain
sendMessage was the visible reply, ahead of (or in place of) the DO
poller's HTML-formatted send.
This commit removes the legacy path entirely:
- Deleted ~/.replicas/bin/tg-reply.sh, tg-status.sh, tg-target-detect.sh
and ~/.replicas/TELEGRAM_REPLY.md from the Global env via
`replicas env files delete --force` (out-of-band; tracked here in the
commit message). Next spawn won't have the scripts on disk.
- Tightened the prompt prefix in prefixWithRoutingHeader to:
* Explicitly forbid the removed helper paths so an agent with stale
muscle memory doesn't try them.
* Promise the poller projects thinking + tool_use + final reply.
* State that final-message Markdown is converted to Telegram HTML, so
the agent should keep emitting normal Markdown without trying to
hand-write `<b>` / `<i>` tags.
Tests bumped to assert the new prefix contents (no count change; the old
prefix expectation was updated rather than a new test added).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
…replicaId} Adds a GET-only debug endpoint that proxies to the DO and returns its storage map + scheduled alarm time. Lets us verify in production whether the watcher attached, what baseline event count it snapshot, and whether the rolling status message id is cached -- without redeploying. Used to verify the pipeline in production via a synthetic webhook: POST https://watcher/watch - Ok [poller] /watch replica=... baseline=2 Alarm fires at ~1Hz events=6 fresh=2 newStatus=yes sawResult=false <- tool_use surfaced events=9 fresh=2 newStatus=no sawResult=true <- final result captured sending final reply (471 chars) sendReply HTML failed (synthetic chat), fallback plain also failed (chat not found) Confirms the architecture is correct end-to-end -- the only reason a status line wouldn't appear for a real user is that the agent answered without using any tools, in which case the only frames are "🤔 Starting…" and "✅ Done in Xs". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Low thinking was hurting reply quality on simple greetings. Telegram-spawn agents were defaulting to "Hello! How can I help you today?" regardless of what the user actually asked, because sonnet without any extended thinking collapses ambiguous prompts to a safe greeting. Verified live with a tool-using prompt after the bump: spawned replica processed correctly with newStatus=yes firing on 3 tool calls (ls, find, find+xargs+wc) and produced a Markdown table with bold/code spans in the final reply — which the markdown->HTML formatter will render properly. Sonnet-4-6 with medium thinking is still ~2x faster than Opus-4-7; the trade is back toward "engages with the prompt" vs raw speed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
The DO debug endpoint surfaced a bug after a tool-use synthetic test: statusLines ended up with ["🤔 Starting…", "✅ Done in 4s", "✅ Done in 35s"] because /watch fired twice for the same inbound Telegram message and the second call hit the full reset path (clear statusLines, delete statusMessageId, fresh "Starting…" frame), orphaning the already-sent status message and producing duplicate "Done" frames. Cause is upstream: Cloudflare KV is eventually consistent so the `seen:<chat_id>:<msg_id>` idempotency guard in the Worker can let both the original delivery and a Telegram webhook retry pass through, each calling startWatcher. Fix is to dedupe inside the DO itself: if the priorWatch we already stored has the same replicaId AND the same startMessageId, just bail out before resetting state. Follow-up messages with a different startMessageId still hit the legitimate full-reset path, which is correct -- a new turn deserves a new rolling-log status message. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
…d, reactions, pin, expandable log
Implements Phase 1+2 from ~/.replicas/plans/telegram-status-ux-research.md.
The status message now matches the template the research recommended:
<b>📋 PLANNING</b> · step 3 · ⏱ 14s
<i>Task:</i> add a /healthz endpoint
<code>🔧 $ ls -la src/</code>
<blockquote expandable>
<code>📖 src/index.ts</code>
… N earlier steps
</blockquote>
<code>📖 src/server.ts</code>
<code>✏️ src/router.ts</code>
[✋ Cancel] [📜 Open log ↗]
New src/render.ts (pure functions, fully unit-tested):
- phaseFor(toolName, command) maps to one of
STARTING / PLANNING / EDITING / RUNNING / TESTING / SHIPPING / DONE / FAILED
- Read/Grep/Glob/WebFetch -> PLANNING
- Edit/Write/NotebookEdit -> EDITING
- Bash with bun/npm/pytest/vitest/cargo test -> TESTING
- Bash with gh pr / git push / wrangler deploy / fly deploy -> SHIPPING
- Bash otherwise -> RUNNING
- formatToolUseLine wraps each tool in <code> with an emoji prefix
- thinkingLine wraps thinking previews in italic
- render assembles header + Task: line + current action + rolling lines,
collapsing older history into <blockquote expandable> once over the cap
- phaseToReactionEmoji picks bot-allowed standard emoji per phase
Rewrote src/poller.ts:
- /watch now also: snapshots events, resets state, sets 👀 reaction on the
user's prompt, calls renderAndSend (which sends the first status frame
AND pins it), then schedules the first alarm. Idempotent dedupe on
(replicaId, startMessageId).
- Each alarm: pulls history, projects tool_use/thinking into phase +
currentAction + lines + stepCount, calls renderAndSend (which diffs
against lastRendered before hitting editMessageText), and pings
sendChatAction every ~4s with action=typing or upload_document for
EDITING/SHIPPING phases so Telegram's "typing…" indicator stays alive
above the input box.
- claude-result: appends a terminal frame (🎉 DONE in Ns or ❌ FAILED in
Ns + error blockquote), unpins the status message, sets big 🎉 or 😭
reaction on the user's prompt, then sends the formatted final reply
as a separate message (existing markdownToTelegramHtml path).
- /cancel endpoint on the DO: calls DELETE /v1/replica/{id} and writes a
failed terminal frame. Reachable via Telegram inline keyboard's
✋ Cancel button.
Worker side (src/index.ts):
- handleCallback routes callback_query updates. `c:<replicaId>` is the
cancel opcode (callback_data limit is 64 bytes; UUID fits).
- ackCallback acknowledges within Telegram's 10s window with an optional
toast text so the button doesn't appear stuck.
- TgUpdate gains optional callback_query.
Inline keyboard:
- Active: [✋ Cancel] [📜 Open log ↗]
- Terminal: [📜 Open log ↗] (cancel stripped)
- Open log is a `url` button pointing to
https://www.replicas.dev/dashboard?workspaceId=<replicaId> — no
callback wiring, just a deep link.
Tests: 27 -> 50 green. New src/render.test.ts covers phaseFor (5 cases),
formatToolUseLine (6 cases incl. HTML escape), thinkingLine,
phaseToReactionEmoji, render (4 cases incl. starting, running with
counter, expandable-blockquote collapse, DONE/FAILED terminals), and
formatDuration.
Deployed as 7f5ad859-8c79-4994-8631-4896f5c26448.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Drops the polling interval and adds a periodic ticker re-render so the ⏱
elapsed-time in the status header keeps updating even when no new
tool_use events are flowing. The goal is the highest update frequency
that respects Telegram's per-message edit rate limit.
Constants:
- FIRST_POLL_DELAY_MS: 400 -> 250
- ACTIVE_POLL_INTERVAL_MS: 800 -> 400 (2.0x faster detection)
- BACKOFF_POLL_INTERVAL_MS: 4000 -> 3000
- EDIT_MIN_INTERVAL_MS: NEW, 900ms. We never editMessageText the same
message more than once per ~900ms (Telegram caps at ~1/sec per chat).
- TICKER_REFRESH_MS: NEW, 2500ms. If at least this long has passed
since the last edit, the alarm forces a re-render so the ticker
doesn't stall.
Mechanics:
- renderAndSend now snapshots lastEditAt at the moment of a successful
edit. If the next call hits the diff guard while inside the
EDIT_MIN_INTERVAL window, we leave the state alone and let the next
alarm pick up an even fresher render.
- alarmInner re-renders when ANY of: a new line was appended,
currentAction changed, OR ticker is stale. So a long-running tool
(e.g. `bun test` running 8s) still shows the time advancing.
- Net latency to surface a fresh tool_use call: ~400ms poll detect +
up to ~900ms edit window = <=1.3s worst case from agent action to
Telegram message.
Verified live: synthetic webhook with 3-step task ("read package.json,
list src dir, count lines in src/index.ts"). Captured alarms at ~1s
intervals through STARTING -> RUNNING phases, step counter 0 -> 3,
sawResult=true at ~17s total. No 429 backoffs.
Tests 50/50 still green (constants only).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Reads cleaner, removes repetition, looks like a chat message instead of a dashboard. Before: <b>📋 PLANNING</b> · step 3 · ⏱ 14s <i>Task:</i> deploy the api <code>🔧 $ ls -la src/</code> <code>📖 src/index.ts</code> <code>✏️ src/server.ts</code> [✋ Cancel] [📜 Open log ↗] After: 📋 <b>Planning</b> · step 3 · 14s 🔧 <code>ls -la src/</code> 📖 <code>src/index.ts</code> ✏️ <code>src/server.ts</code> [✋ Cancel] [📜 Open log] Changes to the rendered shape: - Phase emoji is OUTSIDE the bold tag, not inside an all-caps label. Was `<b>📋 PLANNING</b>`, now `📋 <b>Planning</b>`. Title case, no dashboard look. - Tool emoji is OUTSIDE the <code> tag. Was `<code>🔧 $ ls</code>` where the emoji and `$` were forced into monospace and looked like ASCII art. Now `🔧 <code>ls</code>` — emoji renders naturally, only the command is monospace. - Dropped the `$` prefix for Bash. The 🔧 wrench already signals "running command" and `ls -la` is clearly a command. - Dropped the "Task:" repetition. The status message is sent as a reply to the user's original prompt, so Telegram displays the prompt above the status. Repeating it inside is noise. - Header separator is a single ` · ` instead of three different separators (` · ` with double spaces). Reads like sentence prose. - Terminal state collapses to one line: `🎉 <b>Done</b> · 14s` or `❌ <b>Failed</b> · 8s` with the error in a <blockquote>. Previously retained the tool log even on success, which competed with the final reply that lands immediately below. - formatDuration: 1m+ renders as `1:42` (M:SS) instead of `1m 42s`. Reads as a stopwatch, fits the ⏱-style header. - thinkingLine: drops the leading 🤔 emoji since the phase header already shows it. Italic alone signals "this is preview/thinking". - Inline keyboard text: "📜 Open log" (no ↗). Telegram already shows an external-link glyph on url buttons; the arrow was redundant. Tests refactored from 23 -> 26 cases. New ones pin: emoji-outside-code for every tool, Bash without $ prefix, Title-case phase header, starting-state-doesn't-repeat-task, terminal collapses to one line, M:SS duration formatting. Suite total 50 -> 53 green. Deployed as 6db4694f-8e2c-48b2-8ca7-beb0b7afc494. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
… round-trip
Five concrete latency wins layered together. Net: the first 👀 reaction
hits within ~100ms of the webhook landing, the first status frame within
~500ms, and the gap between agent's claude-result event and the final
Telegram reply drops by ~400ms.
1. Worker: instant 👀 reaction in parallel with POST /v1/replica
- sendInstantAck fires setMessageReaction with the message_id from
the inbound update before we even start the Replicas spawn. User
sees the reaction land within ~100ms instead of waiting for the
Replicas POST + DO attach + DO setMessageReaction chain (was ~600ms).
- DO's /watch handler skips its own setReaction call now that the
Worker already set it.
2. Worker: follow-up message + watcher re-arm run in parallel
- Was: await sendFollowUp -> await startWatcher (~400ms serial)
- Now: Promise.all([sendFollowUp, startWatcher]) (~250ms parallel)
3. DO: terminal status edit + final reply send run in parallel
- The "✅ Done in Xs" editMessageText and the sendMessage with the
final reply text don't depend on each other. Promise.all saves
~400ms on every completed turn.
4. Tighter poll cadence
- FIRST_POLL_DELAY_MS: 250 -> 150
- ACTIVE_POLL_INTERVAL_MS: 400 -> 300
- TICKER_REFRESH_MS: 2500 -> 2000
The /history fetch + render path got lighter when we removed the
redundant setReaction call from /watch, so the alarm loop can target
faster cadence without piling on backlogs.
5. Warm-session window 30min -> 60min
- POST /v1/replica now passes lifecycle_policy: "delete_after_inactivity"
auto_stop_minutes: 60. A second message ≤60min after the first
reuses the warm workspace and skips the 5-15s cold-start entirely.
Default was 30min.
6. Prompt prefix trimmed
- The 3-line block telling the agent about the legacy helpers is gone
(helpers were removed from Global env weeks ago). Now one sentence:
"Spawned from Telegram. Your tool calls and final reply are
auto-surfaced via an external poller. Emit Markdown freely; it'll
be rendered to Telegram HTML."
- Saves ~200 tokens per spawn -> measurably faster TTFT on the
agent's first response.
Test for the prefix updated. All 53 tests still green.
Deployed as 6c5e5e... (next).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
…he status log
Screenshot from a long Craft-tools session showed the rolling log only
listing tool names — `ToolSearch / connection_info / folders_list / …` —
with no narration. The agent had been emitting plenty of explanatory
text between tools ("Connected to Garzas Private space", "Sandbox folder
created: 5bf5d167…", "blocks_move has a schema issue") but the poller
captured those text blocks into pendingAssistantText and only used them
if claude-result fired. Mid-flight narration was dropped entirely.
Now:
- Each claude-assistant text block is treated as live narration:
- updates currentAction (the italic line under the phase header)
- is appended to the rolling log with a 💬 prefix so it survives
subsequent tool calls and the user can scroll back through "the WHY"
- The log now interleaves tool calls and narration, which matches what
you'd see in the Replicas UI chat panel.
Also bumped MAX_TOTAL_LINES 40 -> 80 so long Craft-tools-style runs (the
above example was already at step 20 mid-task) don't lose history. The
<blockquote expandable> keeps the older entries visually collapsed; the
4096-char Telegram message cap is still our real ceiling and is checked
separately.
Tests 53/53 still green (no test-shape changes; behavior change is
captured in renderer tests indirectly via realistic line counts).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Each Durable Object storage operation is ~5-15ms of latency. The alarm
loop was awaiting them sequentially:
storage.get watch
storage.get startedAt
storage.get pendingCleanup
storage.get lastSeenCount
storage.get lines
storage.get phase
storage.get stepCount
storage.get currentAction
... fetch /history ...
storage.put lines
storage.put phase
storage.put stepCount
storage.put currentAction
storage.put lastSeenCount
That's ~12 round-trips per tick, ~80-120ms of pure storage latency before
the actual work begins. Now:
- Single storage.get(keys[]) at the top of alarmInner returns a Map. We
read every key we'll touch this tick in one round-trip.
- Single storage.put({ ... }) at the end commits all five mutated fields
in one round-trip.
- Same pattern applied to /watch: snapshotEventCount runs in parallel
with the multi-key delete, then a single multi-key put initialises
the new state.
- renderAndSend's lastRendered + lastEditAt are now one put instead of
two.
The /history fetch is now kicked off and awaited explicitly so the
control flow is clear; in a future pass the next-alarm scheduling can
be piped through the same path.
Expected cycle-time win: ~80-100ms per alarm, which means actual alarm
interval drops from ~600-700ms toward the 300ms target. Status frames
should perceptibly update faster on tool-heavy tasks.
Tests 53/53 still green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
When the agent emits a structured plan in narration like:
Plan (2/6)
~Round 2 sandbox created (folder + 2 docs + blocks)~
Blocks add/get/update/delete confirmed; retry blocks_move + blocks_revert
~markdown_add OK; image_view needs Craft-hosted URL~
Re-run tasks + comments
Re-run document_search
Cleanup
…the poller used to route it through thinkingLine() which truncated to
200 chars (eating most items) AND through <i> italic that didn't render
the ~item~ strikethrough at all. The plan was effectively invisible.
Now:
- parsePlan(text) detects the "Plan (X/N)\n…items…" pattern (tolerates
whitespace in the header, allows tilde-wrapped items) and returns a
structured PlanState { done, total, items: [{content, done}] }.
- The poller detects plans in mid-flight assistant text blocks and
stores them on DO state separately from narration / lines.
- render.ts adds a dedicated plan block:
<b>📋 Plan (1/3)</b>
✓ <s>scaffold</s>
◦ wire it up
◦ test
…inserted between the narration line and the rolling tool log. The
block survives across alarms until the next plan or /watch reset.
- /watch clears the plan key alongside the other per-turn state.
- The markdown->HTML converter also picked up single-tilde strikethrough
(in addition to GFM's ~~double~~) so any final-reply Markdown that
uses Telegram-style single-tilde renders correctly too.
Tests 53 -> 62 green. New cases: parsePlan returns null on non-plan text
+ extracts done/total + marks tilde items as done + tolerates whitespace
in the header; renderPlan formats with strikethrough + bullet + HTML
escape; markdown converter handles single-tilde without hitting
intra-word tildes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
… to Telegram
Phase 0 + Phase 1 from ~/.replicas/plans/matrix-bridge-plan.md:
- Cloudflare Worker package replicas-matrix-bridge/ that bridges a Matrix
bot account (Beeper) to Replicas workspaces.
- Same status-edit-in-place UX as the Telegram bridge — phase header,
step counter, time ticker, plan block (Plan (X/N) with ~item~
strikethrough), expandable old log, narration lines, reaction
lifecycle on the user's prompt, typing indicator.
- Reuses render.ts + markdown.ts verbatim from the Telegram bridge
(copied for now; will be deduped into a shared core in a follow-up).
- The state machine in poller.ts is the same shape as the Telegram
poller's: alarm-driven /history poll, diff-guard before edit, batched
DO storage reads/writes, snapshot-event-count baseline, parallel
terminal-edit + final-reply.
New components specific to Matrix:
- src/matrix.ts — Matrix Client-Server API client. sendMessage,
editMessage (via m.replace + m.new_content), react (m.annotation),
redact, typing, pin (reads + prepends state event), unpin, joinRoom,
sync (long-poll with timeout + filter), whoami. stripHtml fallback
for the m.text body field. Throws MatrixError with status + body
prefix on failure.
- src/listener.ts — MatrixListener Durable Object. One global instance
per bot account. Alarm fires every ~1s, kicks off a 28s /sync
long-poll (under CF Workers' 30s fetch limit). Persists next_batch
in DO storage. Auto-joins invites. For each m.room.message in a
joined room, calls handleMatrixMessage directly (no extra Worker
round-trip). For !cancel reply-commands, calls the per-replica DO's
/cancel endpoint.
- src/dispatch.ts — handleMatrixMessage. KV-routes (room:<roomId> ->
replicaId), spawns or sends follow-up, fires 👀 reaction in parallel.
Snapshot/dedupe/Promise.all parallelization patterns lifted from
the Telegram dispatcher.
- src/poller.ts — Matrix-flavored ReplicaPoller. Same control flow as
the Telegram poller; substitutes editMessage / react / typing / pin /
unpin / sendMessage (final reply) calls. Markdown -> HTML on final
reply via the shared converter.
- src/index.ts — Worker entry. Routes: GET /health, POST /start-listener
(kicks the listener DO), POST /dispatch (forwarded by the listener),
GET /debug/listener, GET /debug/watcher/{replicaId}, scheduled handler
re-kicks the listener every minute via cron.
Tests: 58 across markdown (11, reused), render (33, reused), and
matrix (14, new — sendMessage shape, editMessage's m.replace wrapper,
react's m.annotation relation, redact, typing, pin GET+PUT round-trip
+ 404 tolerance, unpin filter, joinRoom POST, whoami, sync URL params,
stripHtml).
Deferred to v2:
- E2EE (Olm/Megolm key handling — needs hosted infra, not CF Workers).
- Shared core package extraction (render.ts + markdown.ts duplicated
for now — both packages can iterate independently until the API
stabilises, then dedupe).
- Replicas-side state for cross-room agent identity (m.thread support).
Setup steps documented in replicas-matrix-bridge/README.md: create bot
on beeper.com, grab the access token from Beeper Desktop, wrangler
secret put MATRIX_ACCESS_TOKEN + MATRIX_USER_ID + REPLICAS_API_KEY,
wrangler deploy, curl /start-listener once, invite @bot to a room.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
URLSearchParams handles URL-encoding internally; we were
encodeURIComponent-ing the JSON filter ourselves before passing it in,
which produced double-encoding (`%257B...` instead of `%7B...`). The
matrix.org homeserver interpreted the resulting value as a filter ID
reference and returned `M_INVALID_PARAM: Invalid filter ID` on every
/sync call, leaving the listener stuck without a since token.
Verified live:
- Deployed `7321060f-…` against matrix-client.matrix.org with the bot
account credentials.
- /start-listener kicked the alarm chain; after one cycle the listener
has a `since: s7015940597_757284974_10492641` token from a successful
/sync.
- Synthetic POST /dispatch with `{roomId:"!test:matrix.org",eventId:"$synth123",body:"…"}`
spawned a real replica (`mx-testmatrixorg-1780043697944`) via the
spawn-vs-follow-up branch.
Confirms the full Matrix→Replicas spawn path works; the next step is
having an external Matrix user send into a room the bot is in so the
listener picks it up end-to-end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
When a second message lands while the previous turn is still active (status frame open, no terminal yet), don't blow away the rolling log + statusMessageId/statusEventId. Update the watch spec in place, append a 💬 You: line so the steer is visible, and let the existing alarm chain pick up the agent's response. Plan, step counter, phase, history all survive. Applied symmetrically to both bridges. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
…sends Self-bot mode (the MATRIX_USER_ID is the human's own account, no dedicated bot user) was filtering out the human's typed messages because the listener skipped anything where sender === MATRIX_USER_ID. Matrix's /sync echoes back the transaction_id only to the access token that PUT the event, so its presence in unsigned.transaction_id is a reliable "this is us" signal that survives self-bot mode. The human's own Element/Beeper sends arrive without it (different access token = different device session) and now trigger the dispatch normally. The worker's own status edits/reactions/messages still get skipped because they always carry our transaction_id echo, so no self-loop. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Adds the container image + CF-Containers host worker that fronts a
Pantalaimon process so the Matrix bridge can speak plain CS-API while
E2EE is terminated server-side. Two deploy paths are scaffolded:
- `replicas-matrix-pantalaimon/{Dockerfile,pantalaimon.conf}` builds
a thin image on top of matrixdotorg/pantalaimon:latest pointing
at matrix-client.matrix.org, listening on 0.0.0.0:8008,
IgnoreVerification=True (bots can't manually verify devices),
UseKeyring=False (no system keyring in containers).
- `replicas-matrix-pantalaimon/host/` is a Cloudflare Worker with a
`@cloudflare/containers` PantalaimonContainer DO that proxies all
HTTP into the container on :8008. Image is already pushed to
registry.cloudflare.com/<acct>/replicas-matrix-pantalaimon:v1.
- `fly.toml` mirrors the same config for Fly.io deployment with a
persistent volume, in case we move off CF Containers.
Container is registered on CF (app id a0327263-…); first-boot is
currently fighting "no container instance can be provided" errors —
to be debugged next.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
When an m.room.encrypted event arrives, look up an exported Megolm session key by (room_id, session_id) and decrypt with Web Crypto. The decrypted inner event is treated identically to an unencrypted m.room.message, so the rest of the bridge (spawn, follow-up, steering, cancel) keeps working without changes. The session keys come from an Element key export — decrypted out of band and provided as the MATRIX_MEGOLM_KEYS_JSON secret. No WASM bundle, no sidecar, no Pantalaimon: ~190 lines of Web Crypto wire up HKDF-SHA256 + AES-256-CBC + HMAC-SHA256 + the libolm ratchet advance algorithm. Worked through one false-start: the Megolm spec says "AES" without specifying mode and my first attempt assumed CTR; libolm in fact uses CBC with PKCS#7 padding. Validated bit-for-bit against python-olm on three real ciphertexts from the user's E2EE DM before deploying. Live test confirmed end-to-end: encrypted message "what tools do you have" decrypted, replica spawned, agent replied in the room within 17s. Caveats noted in src/megolm.ts: we currently only have inbound decrypt + we send outbound messages plaintext (clients flag them as "unencrypted in encrypted room" but they still post). Bidirectional E2EE + live /sendToDevice key sharing for new Megolm sessions are the follow-ups. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Layered on PR #14's per-tool lifecycle icons. Single PR bundles all visibility/awareness improvements from the design discussion. Shipping: 1. Thinking "Still planning…" header transition at 60s + 3-min hard stop on the ticker when stuck in STARTING/PLANNING with zero steps. 2. Per-tool elapsed time appended on completion ((1.2s) / 12s / 1:23). Tracked via toolStartedAt map keyed by tool_use.id. 3. Diff stats on Edit / Write / NotebookEdit from input args (+12 −3) computed from line-delta of old_string vs new_string. 4. Compact Done for text-only turns — when the turn used no tools and has no plan, skip the rolling log entirely and render just the subtitle + thinking placeholder above the agent reply. 6. Stop-reason accent on Done header (max_tokens →⚠️ cut off, tool_use → ⏸ stopped at tool_use, pause_turn → ⏸ paused, etc.) 7. RATE_LIMITED phase from claude-rate_limit_event (⏸ Rate-limited), prior phase stashed and auto-reverted on next claude-assistant. 8. Permission-denials badge in resultMeta (🔐 N blocked). 9. Files-touched footer rendered as deduped basenames with overflow tail (📂 Files: index.ts, render.ts (+3 more)). 10. Context-usage visual bar in subtitle (📊 ctx ▰▰▰▱▱▱▱▱▱▱ 32%), bumps to⚠️ at >=85%. 11. Idle-since indicator on the Thinking line when no new event in 5s (· idle 8s) — surfaces stalls. 12. Bold Task lines for the Task subagent tool (🧰 Task: description). 14. Cumulative session totals persisted to KV (session:<roomId>) and rendered when turns > 1 (🧮 session: $0.12 · 8 steps · 2 turns). Deferrals from the 14-item list: - #5 (Plan progress checkmarks) — pending TodoWrite ingest design. - #12 full subagent nesting — only the minimal Task-line bolding lands here. - #13 (Audio/desktop notification routing) — out of scope for status pane. Test fixture in render.test.ts updated to use lifecycle-prefixed lines so the compact-Done detector keeps the rolling log when tools ran. 48/48 tests pass. typecheck clean. Deployed: Cloudflare Worker version 351dc633-442c-4e9c-8a75-c1e617b4ffec. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Two reusable diagnostic endpoints for unsticking and identifying rooms when a watcher gets pinned in a long-running loop: - POST /admin/watcher/:replicaId/cancel — proxies to the watcher DO's internal /cancel, terminates the Replicas-side replica, sets the status pane to Cancelled by user, and triggers cleanup. Previously the only way to unstick a runaway poller was waiting for the 30-minute MAX_WATCH_DURATION_MS or flushing storage directly. - GET /debug/roomname/:roomId — resolves a Matrix room id to its display name using the bridge's existing MATRIX_ACCESS_TOKEN. Surfaces which room is which without needing a separate Matrix client when triaging multi-room reports like "X room isn't answering" against a flat list of opaque !abc:beeper.com ids. Used both today to identify and cancel a runaway Project Nomad Office session that had spent 18 minutes hammering Lark via e2b with empty result loops. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Root cause of "this room is stuck" reports: two paths in ReplicaPoller.alarmInner ran `state.storage.deleteAll()` without first updating the user-facing status pane to a terminal frame, leaving it frozen mid-progress (e.g. "Editing · step 9 · 1:15") indefinitely. The two paths: 1. MAX_WATCH_DURATION_MS (30 min) — long-running watches got their DO state wiped while the pane stayed on the last Planning/Editing frame. Now lands a Failed frame: "Watch timed out — agent ran longer than 30 min." then deletes. 2. /history 404/410 — when the upstream Replicas-side replica vanished (TTL expiry, manual delete, transient error), the poller silently wiped DO state. The user saw a frozen status pane and had no way to tell that they needed to resend. Now lands a Failed frame: "Replica gone (upstream returned <status>). Send a new message to start a fresh session." then deletes. Identified by surveying joined_rooms vs KV mappings + room timelines: 3 Beeper rooms (Jessica Butler, Travel Manager, Adult Admin) showed frozen mid-progress panes from prior watches that hit one of these silent-cleanup paths. With this fix, future occurrences self-resolve to a clean Failed frame instead of looking stuck forever. Also adds two diagnostic endpoints used to find these: - GET /debug/joined-rooms — bot's full joined room set - GET /debug/room-timeline/:rid — recent timeline events for a room Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
…in-room
Three more diagnostic + recovery endpoints for the same triage workflow:
- GET /debug/invites — pending room invites the bot was sent
but has not yet accepted, with the room
name from invite_state when available.
Surfaces auto-join misses.
- GET /debug/joined-rooms — bot's full joined room set. Compare
against KV `room:` mappings to find
rooms with no watcher.
- POST /admin/join-room/:rid — accepts a pending invite by room id,
when the listener's auto-join loop
silently failed for that room.
Used today to recover the *Jett Butler* room: the listener never joined
despite an `@jadagarza:matrix.org` invite sitting for ~14h, so two
prior user prompts went unprocessed. Hitting /admin/join-room then
/dispatch with the original event id brought it back to life — bot is
now actively responding inside it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
…k frozen The status pane must always look like *something* is happening. Three changes that together ensure the pane visibly moves on every render even when the agent is mid-tool or stuck thinking: 1. Removed the 3-min hard-stop on the ticker. Prior behavior: when STARTING/PLANNING with zero steps went past 180s, the poller stopped forcing re-renders to save edits. The pane froze at "Still planning · 3:00" forever. Now the ticker keeps running indefinitely — the heartbeat below guarantees the pane always advances visibly. 2. Heartbeat Braille spinner (⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏) prefixed to the phase emoji in renderActive. Frame index derives from elapsed seconds so two consecutive ticks always land on different frames. Renders as a steady pulse at the TICKER_REFRESH_MS (~4s) cadence. When the header has nothing else changing, the dot still rotates — the visible-movement guarantee. 3. Live elapsed tail on the in-flight 🔄 tool line. New activeToolStartedAt state field tracks when the most recent unmatched tool_use was projected. Renderer appends ` · Ns` (using formatDuration) to the LAST 🔄 line in the rolling log on every tick, so a 30s e2b run isn't a static "🔄 e2b__run_code" — it reads "🔄 e2b__run_code · 12s … 13s … 14s". Cleared when the last 🔄 line transitions to ✅ or ❌. Three new tests cover heartbeat advancement, live tool elapsed appending, and non-mutation of completed ✅ lines. 51/51 pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Contributor
|
Capy auto-review is paused for this organization because the monthly auto-review limit has been reached. Increase the limit or turn it off in billing settings to resume automatic reviews. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Contributor
Author
@devin-ai-integration[bot] This repository is not connected to a Replicas organization. To enable Replicas on this repository:
Note: Each repository can only be connected to one Replicas organization. |
Two older fix/perf branches had real improvements that never made it into the deployed branch chain. Cherry-picked back in so a single deploy from the head covers them: * fix/strict-one-emoji-invariant (3a7361b) - Steered-followup case: redact the prior prompt's 👀 before overwriting DO storage's reactionEventId with the new prompt's, so the old 👀 doesn't stay orphaned forever. - Late /ack after terminal: when phase is already DONE/FAILED on /ack arrival, redact the incoming 👀 immediately instead of storing — there's no future swap to clean it up. Together these enforce strict "one emoji at a time" across the whole session. * perf/matrix-bridge-char-delta-and-self-tune-backoff (064d3f7) - Char-delta edit gate: skip edits when rendered length changed by < CHAR_DELTA_CUTOFF (50) AND phase didn't transition. Heartbeat- safe: if the gate would skip but it's been ≥ 2 ticker intervals since last edit, force through so the heartbeat dot keeps rotating (otherwise the gate would silently kill PR #17's "never look frozen" win — a heartbeat-only diff is 0 chars by code-point length). - Self-tuning 429 backoff: each non-terminal 429 ratchets a per-turn `backoffBonusMs` up by 1500ms, capped at 10s. Wiped on /watch fresh-spawn so calm turns start fast. 51/51 render tests still pass; typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
dispatch.ts treated sendFollowUp() responses as three cases:
- ok → continue on existing replica
- gone → cancel + respawn on a fresh replica
- anything else → fall through; replicaId stays null; user gets a
👀 ack and never any answer
The "anything else" silent-drop is the root cause of the Lark Admin
"no answering" report: the user sent a follow-up, the bridge reacted
👀 (dispatch fired that BEFORE sendFollowUp), then sendFollowUp got
a non-2xx non-404/410 response (likely a 429 or transient 5xx), the
silent branch took over, and the message was lost forever. The bot
looked like it was eternally thinking.
Now any non-ok response — including the explicit gone + every other
failure mode (429, 5xx, network blip) — falls through to the same
respawn path: cancel the watcher, flush the KV mapping, create a
fresh replica. The user loses the existing chat session's state but
gets an actual answer to their message. Safer trade.
Logs the failure with the gone flag so we can tell genuine TTL
expiry apart from transient API failures in tail.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
When the listener's alarmInner hung, threw past the catch, or got
killed mid-execution by CF Workers, the chain went silent. The cron
/scheduled → /start fallback was the only path back, and that has
its own catch-and-swallow on the fetch call. End result: a listener
that stops processing /sync events for 5+ minutes despite the cron
firing every minute — exactly the Lark Admin symptom today, where
three user messages sat un-acked while the bot was technically
"running."
Now alarm() schedules the safety-net alarm BEFORE calling alarmInner,
at SYNC_TIMEOUT_MS + ALARM_INTERVAL_MS out (~29s). If the body
returns normally, the post-body setAlarm replaces it with the normal
fast-tick. If the body hangs/throws/gets killed, the pre-set alarm
still fires and the chain self-heals without depending on the cron
fallback or any external poke.
Verified live by /start-listener kicking the previously-silent
listener and watching it pick up the three queued messages
("test then all", "What lark mcp do we have", "hello") inside ~10s.
51/51 tests pass; typecheck clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Per Jaden's UX observation: OpenACP-style Telegram status updates feel right not just because of icons + heartbeat motion, but because a long turn renders as multiple message blocks — small turns get small blocks, big turns get more. The bridge was dumping every tool call into one ever-growing rolling log inside a single message. This PR implements seal-and-new segmentation. The active editable "segment" caps at SEGMENT_TOOL_CAP (12) tool calls; before pushing the 13th tool, the poller seals the current segment (final-edit with a "▶️ continues in next message…" tail) and starts a new Matrix message that becomes the new editable segment. Also force-seals on every Task subagent invocation — that's the clearest "new context" boundary and the most natural place to split. State additions: - StatusState.segmentStartLine: index into `lines[]` where the current segment starts. Render slices to this offset for both the rolling log and the Tools (M/N) header count, so each message shows its own segment's tools, not the cumulative total. - StatusState.sealing: transient flag that the renderer reads to append the "▶️ continues" tail on the final render before seal. Poller additions: - segmentStartLine persisted in storage writes, hydrated in loadState, wiped on /watch fresh-spawn alongside other per-turn state. - sealCurrentSegment(state) helper: forces one final edit on the current statusEventId with sealing=true, deletes statusEventId + edit-gate bookkeeping so the next render goes through sendMessage and creates a fresh editable message. Behavior: - Short turns (no tools, or < 12 tools, no Task) → one message, identical to today. - A turn with 25 tools and no Task → 3 messages (12 + 12 + 1). - A turn with 8 tools then a Task subagent → 2 messages. - Done frame lands in the LAST segment with the embedded reply + terminal emoji on the user's prompt — earlier segments stay as intermediate state with no terminal churn. Two new render tests cover the slicing + the seal-tail rendering. 53/53 pass; typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
… segment Builds on the segmentation work: when a user steers an in-progress turn (sends a new message while the agent is still working), seal the current segment (final-edit with the "▶️ continues" tail) before pushing the 💬 You: line. Each steered prompt then opens a fresh Matrix message that begins with the user's new direction — matching the "more calls = more blocks" intuition for conversation steering. Logic in /watch handler's steering branch: - Read segmentStartLine + lines from storage. - If lines.length > segmentStartLine (current segment has any content), call sealCurrentSegment(loadState()) — final-edit the current message with "▶️ continues", drop statusEventId so the next render sends fresh. - Bump segmentStartLine to lines.length BEFORE pushing the 💬 You: line, so the steered prompt becomes the first line of the new segment. - If the current segment is empty (steered before any tool ran), no seal — the 💬 line just joins the existing segment. Audit of the rest of the steering path confirmed it was already correct: - dispatch.ts sendFollowUp posts /replica/{id}/messages with the routing-header-prefixed text, which Replicas injects into the agent's stream. - /watch handler's steer branch was already correctly redacting the prior 👀, storing the new one, and re-rendering on the live statusEventId — only thing missing was segment-aware behavior. One test added: steered prompt opens a new segment, prior tool lines do not appear in the new active frame, Tools (M/N) counter reflects the new segment only. 54/54 pass; typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Two UX changes per Jaden's review of the segmentation work:
1. Reply is its own message block.
Done frame stops embedding the agent's final reply HTML. The
poller now sends the reply as a separate Matrix message right
after the Done frame lands, reply-to'd against the user's prompt
so Beeper threads it visually. Done frame becomes the pure status
summary (tools, files, totals); the answer lives in its own
scrollable block. Easier to find and read without the status
clutter — and the segmentation work already proved multi-message
turns land reliably across Beeper, so the old "single message per
turn" guardrail no longer applies.
sendReplyAsOwnBlock(): retries on 429 the same way terminal edits
do (4 attempts honoring retry_after_ms). Saved as replyEventId in
storage so a duplicate Done can't re-fire the send. Wiped on
fresh /watch spawn.
2. Subscription-plan usage windows instead of per-turn $.
Arbitrary per-turn cost is meaningless for Claude Max / OpenAI Pro
subscription accounts — what matters is "how much have I burned
in the 5h reset window and this week."
On every Done: append {ts, cost, tok} to `usage:org` in KV,
prune entries older than 8 days, save.
On every render: loadState() reads the log and computes 5h + 7d
rolling aggregates from timestamps. Surfaced in the subtitle as
`🪙 5h: 47k tok · 7d: 215k tok` on its own line under the
model/MCP info. Hidden when both windows are zero (fresh org).
formatTokens() extended to emit M suffix above 1M.
Session-level subtitle dropped the $ figure (now redundant with
the 5h/7d view); kept the step + turn count for context.
Test updates:
- 3 new tests: usage windows rendering, zero-state hide, M format.
- 2 existing Done-frame tests inverted: assert the reply is NOT
embedded (lives in the follow-up message instead).
57/57 pass; typecheck clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
…l, no per-turn $ Three targeted polish moves per Jaden's "make it slick and glassy" review of the segmentation work: 1. Heartbeat: braille spinner → quadrant rotation. ⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏ (10 braille frames) → ◐◓◑◒ (4-frame quadrant). The previous braille set worked but read as "code-y dots"; the quadrant rotation is the same iOS ProgressView vibe — consistent visual weight, smooth filled-circle rotation that renders cleanly across Beeper / Element / Telegram. 2. Seal tail: arrow → dotted trail. `▶️ continues in next message…` → `┄┄┄┄┄┄ ▶ continues ┄┄┄┄┄┄`. Dotted line breathes off the end of the sealed segment rather than ending abruptly with a single arrow, so the multi-message turn reads as one continuous flow instead of N disconnected blocks. Dotted glyph chosen because it renders consistently in variable-width fonts where overline-style ▔ can look broken. 3. Per-turn $ dropped from Done header. `🎉 Done · 6:23 · $3.05 · 13k tok` → `🎉 Done · 6:23 · 13k tok`. Subscription-account framing — absolute $ doesn't map to anything actionable when the user's on Claude Max / OpenAI Pro. Tokens stay because they're the durable currency against the 5h reset window + weekly quota (both surfaced in the subtitle's 🪙 row). Tests updated: - Heartbeat advancement test: starting frame is now ◐ not ⠋. - Seal tail test: asserts ┄ (dotted) instead of ▔ (overline). - Done-frame test: asserts $ is NOT in the header; tokens still are. 57/57 pass; typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Per Jaden's review of the subscription-usage windows: absolute tokens consumed aren't actionable; what matters is "how much of my plan is left in this 5h reset window / this week." Now: - New env vars USAGE_QUOTA_5H_TOK and USAGE_QUOTA_7D_TOK on the worker; defaults configured in wrangler.toml at 5M / 100M tokens, mid-range estimates for Claude Max $200 (~250–1000 Sonnet msgs × ~10k tok/msg per 5h; ~480–960 hrs/wk Claude Code time at average rates). Override per-account by editing wrangler.toml. - UsageWindows interface extended with optional pct5hLeft and pct7dLeft (0..100). Populated in loadState from (1 − consumed / quota) × 100, clamped to 0 on overrun. When a quota is unset / zero, the pct field stays undefined and the renderer falls back to absolute tokens for that window. - renderUsageWindows now emits "5h: 99% left · 7d: 96% left" when pct is present, "5h: 47k tok" as fallback when only the absolute is known. Both windows can render independently. Tests: existing rendering test updated to assert "% left" not raw tokens; 3 new tests for quota-configured percent, fallback to absolute when quotas missing, and 0% clamp on overrun. 59/59 pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Same pattern as the listener fix (`c39beb13`): schedule the alarm BEFORE the body runs so the chain self-heals when alarmInner hangs, throws past the catch, or the DO instance gets killed mid-execution by CF Workers. Empirically observed today during a bridge sweep: Nomad Accountability room's watcher had phase=STARTING, lastEditAt 10 min old, alarmAt=null. The replica had been spawned, the /watch handler had run (seed state + first render), but alarmInner never got to setAlarm() at its tail — either it hung on the /history fetch or got CPU-killed. Without a pre-set alarm, the chain just died and the user saw a frozen "🤔 Starting · 0s" pane indefinitely. Now: setAlarm(now + BACKOFF_POLL_INTERVAL_MS) runs as the first line of alarm(). If alarmInner completes normally, its own tail setAlarm replaces it with the fast tick. If alarmInner hangs/dies, the pre-set fires at ~3s and we self-heal. Listener got this same treatment (`c39beb13`); the watcher was the remaining vulnerability of the same class. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Probed Replicas API surface for real plan-usage endpoints:
- /v1/usage, /v1/organization, /v1/quota, /v1/billing all return 404.
Replicas does not expose a usage REST endpoint.
- claude-result events carry per-turn cost/tokens/modelUsage (already
tracked bridge-side for the rolling-window accumulator).
- claude-rate_limit_event carries Anthropic's actual rate-limit info:
status — "allowed" | "rejected"
rateLimitType — "five_hour"
resetsAt — unix seconds when the 5h window flips
isUsingOverage — true when consuming overage
overageStatus — "rejected" | "allowed"
This is *ground truth* from Anthropic, surfaced through Replicas.
Two changes leveraging this:
1. Bridge no longer flips phase to RATE_LIMITED on every rate-limit
event. Prior code unconditionally flipped, but 19/19 of today's
sample events had status=allowed (just status pings, not cap-hits).
The pane was incorrectly reading "Rate-limited" during normal
operation. Phase change is now gated on `status === "rejected" ||
isUsingOverage === true`.
2. The 🪙 row gains a real "resets in 1h 18m" tail when the event's
resetsAt is known. UsageWindows extended with optional resetsAt
(ms epoch); renderer computes the human-readable remaining via
new formatUntil() helper ("45s" / "23m" / "1h 18m" / "2d").
Percentage figures are now tagged "(est)" to be honest about the
source — they're computed from the bridge-side accumulator against
env-configured quotas, not from any Anthropic-reported remaining
number. The reset clock IS authoritative.
Tests: existing % left test updated for the "(est)" tag; new test
covers the resetsAt tail. 60/60 pass; typecheck clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Defense-in-depth for the listener's per-sync auto-join loop. That loop is the fast path but silently fails on rate limit / transient network errors — the catch swallows the failure and the invite stays pending indefinitely. Pattern verified empirically: two manual /admin/join-room calls today already (Jett Butler at ~14h-old invite, Nomad Office at ~10min-old invite). Now the `*/1 * * * *` cron, in addition to kicking the listener, also fetches pending invites via /sync and force-joins each. The listener's loop keeps being the fast path; the cron sweeper catches anything it misses. Worst-case latency for a missed invite drops from "forever until manual recovery" to ~60s. Both legs use `ctx.waitUntil()` so they run concurrently without blocking the scheduled callback. Top-level + per-invite try/catches log all failure modes so we can see in tail if the join API is rejecting (rate limit vs auth vs network). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
When the listener encounters an m.room.encrypted event and the Megolm
session key is missing, the bridge now proactively asks the sender's
other devices for the key — instead of silently dropping the message
and waiting on the user to verify the bot's device from their phone.
Mechanism (3 pieces):
1. listener.ts `tryDecrypt`: on missing session, queue the encrypted
event under `pending-decrypt:{room}:{session}` in DO storage (cap
50 per session to bound pathological growth), then call
`maybeSendKeyRequest`.
2. listener.ts `maybeSendKeyRequest`: PUTs
`/sendToDevice/m.room_key_request/{txnId}` addressed to the
sender's user (`*` device wildcard). Dedup'd per (room, session)
by storing `key-request:{room}:{session}` → ts and skipping if
<5min old. Payload conforms to Matrix spec — action: "request",
body: { algorithm, room_id, sender_key, session_id }, plus a
fresh request_id and our requesting_device_id.
3. olm-vault.ts: capture logic extended to recognize
`m.forwarded_room_key` (Beeper's response to our request) in
addition to the existing `m.room_key` (initial share). When a key
is captured, the vault returns `capturedRoomId` +
`capturedSessionId` in the response so the listener can immediately
drain the matching pending-decrypt queue.
`MegolmKeyEntry.source` extended to `"forwarded"` so the vault
tracks provenance — useful for future debugging of session
rotation issues.
listener's to-device loop calls `drainPendingForSession()` on
capture, which iterates the queue, re-runs tryDecrypt (now
succeeds), and dispatches each previously-stuck user message.
Failure modes: Beeper may decline to forward (unverified device
warning) — in which case the request is ack'd by the homeserver but
no forwarded_room_key arrives, and the message stays in the queue
until either Beeper trusts the device or we age out. No worse than
today, where we silently dropped.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Out-of-band E2EE recovery for messages that arrived BEFORE the
auto-recovery feature shipped:
- POST /admin/recover-room/:roomId?limit=N — pulls the recent
timeline via /rooms/{id}/messages and hands each encrypted event
to the listener's /recover-room handler. For each event the
handler looks up the Megolm session in the vault; if missing it
queues + emits one m.room_key_request per session (dedupe'd within
5 min); if present it decrypts and dispatches.
Used for the Project Nomad Office case where 3 user prompts had
been sent in a fresh Megolm session before the auto-recovery code
shipped. Listener's /sync only delivers events past its `since`
token, so without this we'd need the user to resend.
- GET /debug/vault/lookup?room=…&session=… — inspect whether a
specific Megolm session key is stored in the vault keystore.
Useful when diagnosing "decrypt failed but key found" cases:
reveals whether the issue is no-key-at-all vs corrupted-key.
In live testing today, the vault DID have the HOZ8Qm session key
stored from earlier shares but decryptMegolm was failing with a
HMAC mismatch — suggests Beeper's forwarded_room_key data may not
match libolm's expected format for that session. Auto-recovery
works for future sessions; for THIS specific failure the user can
resend or wait for Beeper to share via a new send.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Per Jaden's "make sure every message from Replicas is checked to be
in" + "many chats saying the same thing as each other":
Audited the cross-room contamination paths — empirically zero KV
collisions across 15 active room mappings (each replicaId maps to
exactly one room). Cross-room "same content" must therefore be
agent-side (e.g., mcp__beeper__send_message broadcast), not bridge-
side replicaId reuse.
Shipped three robustness improvements:
1. Stable txn id for the reply send.
sendReplyAsOwnBlock now passes txnId=`reply-${watch.startEventId}`
to sendMessage. Matrix dedupes per (access_token, txn_id), so a
crash between the homeserver accepting the send and us persisting
replyEventId no longer produces a duplicate — the retry hits the
same txn and gets the SAME event_id back. End-to-end idempotency.
2. verifyTurnDelivered() runs pre-cleanup.
Before pendingCleanup deletes the watcher state, we GET
/rooms/{id}/event/{eventId} for both statusEventId and replyEventId
to confirm they actually exist in the room. /event/{id} is the
exact-existence check that works even when the room has many
edits since the original (whereas /messages?limit=N may page past
the original event_id). Missing IDs log a warning so we can see
in `wrangler tail` when the bridge thinks it sent something that
the room doesn't actually have. Best-effort — doesn't block
cleanup.
3. Two new debug endpoints:
- GET /debug/kv-conflicts — walks all `room:` KV mappings and
surfaces any replicaId that maps to >1 room. Today zero, but a
standing detector going forward.
- GET /debug/verify-room/:roomId — per-room delivery audit, reads
the watcher state (statusEventId + replyEventId) and reports
whether each is present in the room. Triage tool for "I asked
the bot and never got a reply".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
…lure
Jaden hit Anthropic's 400 `invalid_request_error: The request body is
not valid JSON: no low surrogate in string at line 1 column 453884`.
A high UTF-16 surrogate (0xD800–0xDBFF) without its matching low
surrogate produces an invalid JSON serialization that Anthropic
rejects. The offset (~454 KB into the payload) puts it deep in the
conversation history — almost certainly a tool result the agent then
echoed back. Bridge can't fix already-baked-in history, but it CAN
make sure it never introduces lone surrogates itself.
Two changes:
1. New `sanitizeForJson()` helper in dispatch.ts replaces any lone
high or low surrogate with U+FFFD (the Unicode REPLACEMENT
CHARACTER). Applied inside `prefixWithRoutingHeader()` so every
outbound prompt and steered follow-up the bridge sends to
Replicas is guaranteed surrogate-clean. Valid surrogate pairs
(full code points like 🎉) pass through unchanged.
2. claude-result error handling now pattern-matches Anthropic's
specific 400 strings:
- `no low surrogate` / `no high surrogate` / `invalid_request_
error.*JSON` → surfaces a user-actionable Failed frame:
"Conversation context has invalid characters (lone UTF-16
surrogate from a malformed tool output). Send a new message to
start a fresh session — the broken bytes are baked into this
session's history and can't be repaired in place."
- `string too long` / `max_tokens` → suggests shorter prompt or
/reset.
Other errors fall through to the raw `api_error_status` string
as before.
5 new dispatch tests cover the sanitizer (valid pairs untouched,
lone high/low surrogates replaced, mixed strings, plain ASCII
no-op) plus the integration through prefixWithRoutingHeader. 7/7
dispatch tests pass; typecheck clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Per Jaden: "when you have ones that cannot be recovered you can go
ahead and cut them and add a new one with sonic." First half (text)
shipped here; the Sonic voice announcement lands once Phase 2 of the
voice-messages plan does (see ~/.replicas/plans/2026-05-29_beeper-
voice-messages-plan.md, Phase 2.5).
What auto-cut does on `claude-result`:
- Trigger: `unrecoverable = true` when the error pattern matches
`no low surrogate` / `no high surrogate` / `invalid_request_error
.*JSON` (lone-surrogate poisoned context) OR `string too long` /
`max_tokens` (context window blown). Both are baked into the
replica's chat history and no follow-up prompt can fix them.
- Action: flush `room:${watch.roomId}` from KV. The watcher proceeds
through its normal failed-terminal cleanup, but because the KV
mapping is gone, the next user message in the room spawns a fresh
replica with empty history instead of re-using the poisoned one.
- Announcement: the Failed frame's errorMsg now reads "Auto-cutting
and starting a fresh session — your next message will get a clean
reply." instead of just describing the error. Future iteration
swaps this text for a Cartesia Sonic voice clip via the m.audio
upload path (Phase 2 of the voice plan).
Behavior preserved for non-unrecoverable errors — those still
display the raw `api_error_status` and keep the existing KV mapping
so steered follow-ups can recover within the same replica.
Voice plan updated: Cartesia Sonic now the default TTS provider
(streaming TTFT matters for the acknowledge-and-reset use case)
with OpenAI TTS as fallback.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
… Sonnet
Jaden's directive was "cut and add a new one with Sonnet" (the model),
not Sonic (the TTS). Two updates:
1. Auto-cut KV flush now removes BOTH room:${roomId} AND model:${roomId}.
The model: prefix holds per-room `!model opus` overrides — without
flushing it, an auto-cut on a previously-Opus-pinned room would
respawn back into Opus. With both gone, the fresh replica lands on
REPLICAS_MODEL_OVERRIDE = claude-sonnet-4-6 (wrangler.toml default).
Guarantees "fresh + Sonnet" regardless of room history.
2. Failed frame copy now says "starting a fresh Sonnet session"
explicitly so the user knows what they're getting.
Voice plan (~/.replicas/plans/...) reverted: OpenAI TTS back as
default (Cartesia Sonic was tied to the misread auto-cut announcement
use case). Auto-cut is text-only and stays that way — voice for the
Failed frame is no longer in scope.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
…ection
Empirically observed today on Jaden's main DM with the bot
(`!MuDFIwCLjwWtsZbcmc:matrix.org`): a Failed frame ticked for 15
minutes with the heartbeat spinner advancing, never cleaning up.
Watcher debug showed phase=FAILED but pendingCleanup=undefined — a
state the existing code paths shouldn't be able to produce.
Root cause: `setTerminal` writes `phase` to storage at the start,
THEN does `renderAndSend` + `unpinStatus` + `swapReaction`. If any of
those throw, the caller never reaches the followup
`pendingCleanup = true` write. The next alarm tick reads
phase=FAILED but no cleanup directive, so alarmInner skips the
cleanup branch and proceeds to re-render the Failed pane forever.
Two-part fix:
1. `setTerminal` now persists `{phase, pendingCleanup: true}` AND
schedules the +30s cleanup alarm AS AN ATOMIC WRITE at the top,
before any rendering/unpin/reaction work. Even a partial
setTerminal that throws mid-execution leaves a well-formed
cleanup directive. Callers' subsequent `pendingCleanup = true`
writes become idempotent no-ops; deleteAll callers still win
because their wipe runs after the atomic set.
2. `alarmInner` gains an orphan-detection branch at the top: if
phase is terminal (DONE/FAILED) AND pendingCleanup is unset,
force pendingCleanup and run the cleanup teardown immediately.
With (1) in place this branch shouldn't ever trip; it's defense-
in-depth against future regressions of the same class.
Manual unstick for the runaway: canceled watcher
`8cd91e2f-729c-4d8c-934c-c7574e47a12c` via /admin and flushed both
`room:` + `model:` KV entries for the DM. Next user message in the
room spawns a fresh Sonnet replica.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Ran a focused audit looking for the next bugs after today's 9 fixes. Shipping 6 of the 10 findings; the other 4 are lower-impact or require larger restructuring. #1 — Steer-race between handleWatch and alarmInner [HIGH] The DO is single-threaded but `await` yields. alarmInner does fetch() on Replicas /history which can yield for 100s of ms — during that window a steer /watch races the alarm's read-modify-write on lines[]. Whoever writes second clobbers the other's in-memory copy. Wrapped the steer's state mutation in this.state.blockConcurrencyWhile() so the alarm's in-flight body completes before lines[] is touched. #7 — setTerminal partial-execution leaves un-swapped reaction [MEDIUM] Even with the atomic phase+pendingCleanup write at the top, if renderAndSend or unpinStatus threw, swapReaction never ran — leaving the user's 👀 ack reaction forever (no 🎉/😭). Wrapped each step in its own try/catch so they run independently and the rest of the cleanup always fires. #4 — verifyTurnDelivered didn't actually recover missing reply [HIGH] The docstring promised "if replyEventId missing → re-call sendReplyAsOwnBlock" but the code just logged. Now stashes lastResultHtml in storage during setTerminal so verify can actually re-send via the stable txn id (Matrix dedupes if the prior send did land but our /event/{id} lookup raced). #8 — recover-room mention check used encrypted ev.content [MEDIUM] shouldHandleMessage reads m.mentions + formatted_body from content, but the OUTER ev.content is the encrypted wrapper (algorithm / ciphertext / session_id). Mention detection silently failed in multi-user encrypted rooms during recovery dispatch. Fixed to pass the DECRYPTED inner.content. #6 — lines[] unbounded growth on thinking-heavy turns [MEDIUM] Long planning sessions push hundreds of 💬 narration lines that bloat the serialized state on every ~180ms alarm tick. Capped at 300 entries per segment by dropping oldest non-tool lines first (preserves 🔄/✅/❌ tool entries + 💬 You: steer markers). toolLineIndex pointers are shifted in lockstep when an entry before them is dropped. #2 — key-request:* DO storage grew unbounded forever [HIGH-silent] maybeSendKeyRequest stored `key-request:{room}:{session}` → ts to dedupe inside a 5-min window but never deleted. Every Megolm session rotation accumulated a permanent entry. Bucketed the dedupKey by 5-min window AND added a piggy-back cleanup that prunes entries older than 2 buckets on each call. Bounded storage. Deferred (with rationale): #3 pending-decrypt prune needs a cron sweeper; #5 KV dedupe race needs deeper redesign; #9 cachedKeys stale is low-impact (manual reload endpoint suffices); #10 parsePlan hijack has rare false-positives that haven't been observed. Tests: 67/67 pass; typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Sampled all room timelines for 2h: 7 terminal Failed turns vs 21
Done, with 5 of 7 failures being Replica gone (404 from /history).
The dominant failure class was Replicas auto-deleting replicas
between user messages, leaving the user to send one prompt → see
Failed → send the same prompt again → get a real response.
Two fixes:
1. auto_stop_minutes 60 → 1440 (24h) in dispatch.ts createReplica.
Subscription-plan cost is fixed so holding replicas longer
doesn't change the bill — but it sharply reduces the rate of
"between-message" 404s.
2. Auto-respawn in alarmInner's 404/410 handler. When /history
returns 404 and the watcher has a userText on record, we now:
- Flush room: + model: KV.
- Call respawnReplica() to create a fresh replica with the same
user text (sanitized for JSON; same model/agent/thinking
overrides as the original createReplica path).
- Persist the new replicaId to KV with the same TTL.
- Re-point THIS watcher at the new replica id and wipe per-turn
projection state (lines=[], stepCount=0, phase=STARTING).
- Schedule the next alarm tick and return.
The user sees a brief Starting frame rather than Failed + a
manual retry. If anything in the respawn path fails (no userText,
create fails, etc.), we fall through to the legacy
Failed-and-give-up path so the user at least gets a clear signal.
The respawn helper lives in poller.ts (mirrors dispatch.ts
createReplica) to avoid importing dispatch into poller — that
would be a circular module reference. 30 lines of duplication is
cheaper than the refactor.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
Per Jaden's directive: "There is never a time it needs to be auto
deleted simply because it's not being used. It should only start a
new thread if the previous one cannot be fixed."
Three coordinated changes so an idle replica survives forever:
1. createReplica + respawnReplica request body:
- lifecycle_policy: "delete_after_inactivity" → "manual"
- auto_stop_minutes: 1440 → 0
Both verified accepted by the Replicas API via direct probe.
2. KV TTL on the room:${roomId} → replicaId mapping:
REPLICA_TTL_SECONDS default flipped from "604800" (7d) to "0"
meaning "no TTL". Logic in dispatch.ts + poller.ts now skips
`expirationTtl` when the env var is zero. Any positive value is
still honored as a cap, so per-account overrides still work.
3. The bridge-side respawn paths stay as-is — they only fire when
the existing replica is genuinely broken:
- 404/410 from /history (replica deleted server-side anyway)
- Auto-cut on unrecoverable claude-result errors (surrogate /
context-window full)
Healthy idle replicas with no activity for 24h+ no longer trigger
any cleanup.
Resolves the 5/7 replica-gone failures from the 2h sweep at root.
Today's bug count = 17 distinct classes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Honors the contract: the status pane must never look like nothing is happening. Three coordinated changes that together guarantee visible movement on every render tick, even when the agent is mid-tool or stuck in a long thinking phase.
Removed the 3-min ticker hard-stop. Prior behavior: when
STARTING/PLANNINGwith zero steps went past 180s, the poller stopped forcing re-renders to save Matrix edits — and the pane froze at Still planning · 3:00 forever. Now the ticker runs indefinitely; the heartbeat below keeps the edit volume meaningful.Heartbeat Braille spinner (
⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏) prefixed to the phase emoji inrenderActive. Frame index derives fromelapsedSecso two consecutive ticks always land on different frames — never a duplicate. Renders as a steady pulse at theTICKER_REFRESH_MS(~4s) cadence. Even when phase, step count, and lines are static, the dot still rotates.Live elapsed tail on the in-flight
🔄tool line. NewactiveToolStartedAtstate field tracks when the most recent unmatchedtool_usewas projected. Renderer appends· Nsto the LAST🔄line in the rolling log on every tick. A 30-seconde2brun is no longer a static🔄 e2b__run_code— it reads🔄 e2b__run_code · 12s … 13s … 14s. Cleared when the last🔄line transitions to✅or❌.Layered on top of PR #16 (which adds the terminal-frame-before-deleteAll fix). Together they cover both the pane never freezes mid-progress (this PR) and the pane never gets abandoned mid-progress (PR #16).
Test plan
bun run typecheckclean517c70d5-1e98-4bc2-980e-397b33f06f6d)· 1s · 2s · 3s…on the running🔄line plus the heartbeat dot rotating every renderWorkspace · Slack Thread