feat: April 2026 infrastructure status update#1
Open
itsablabla wants to merge 1 commit into
Open
Conversation
- Replaced outdated Feb 2026 status with comprehensive April 2026 audit - 4 servers tracked: Hostinger H1/H2, Infomaniak IG/ID - 16 H1 services (6 healthy, 10 down) - 4 MCP servers, 4 Railway services, Nextcloud - SSL certificate status, critical issues summary
Contributor
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
replicas-connector Bot
added a commit
that referenced
this pull request
May 29, 2026
Ran a focused audit looking for the next bugs after today's 9 fixes. Shipping 6 of the 10 findings; the other 4 are lower-impact or require larger restructuring. #1 — Steer-race between handleWatch and alarmInner [HIGH] The DO is single-threaded but `await` yields. alarmInner does fetch() on Replicas /history which can yield for 100s of ms — during that window a steer /watch races the alarm's read-modify-write on lines[]. Whoever writes second clobbers the other's in-memory copy. Wrapped the steer's state mutation in this.state.blockConcurrencyWhile() so the alarm's in-flight body completes before lines[] is touched. #7 — setTerminal partial-execution leaves un-swapped reaction [MEDIUM] Even with the atomic phase+pendingCleanup write at the top, if renderAndSend or unpinStatus threw, swapReaction never ran — leaving the user's 👀 ack reaction forever (no 🎉/😭). Wrapped each step in its own try/catch so they run independently and the rest of the cleanup always fires. #4 — verifyTurnDelivered didn't actually recover missing reply [HIGH] The docstring promised "if replyEventId missing → re-call sendReplyAsOwnBlock" but the code just logged. Now stashes lastResultHtml in storage during setTerminal so verify can actually re-send via the stable txn id (Matrix dedupes if the prior send did land but our /event/{id} lookup raced). #8 — recover-room mention check used encrypted ev.content [MEDIUM] shouldHandleMessage reads m.mentions + formatted_body from content, but the OUTER ev.content is the encrypted wrapper (algorithm / ciphertext / session_id). Mention detection silently failed in multi-user encrypted rooms during recovery dispatch. Fixed to pass the DECRYPTED inner.content. #6 — lines[] unbounded growth on thinking-heavy turns [MEDIUM] Long planning sessions push hundreds of 💬 narration lines that bloat the serialized state on every ~180ms alarm tick. Capped at 300 entries per segment by dropping oldest non-tool lines first (preserves 🔄/✅/❌ tool entries + 💬 You: steer markers). toolLineIndex pointers are shifted in lockstep when an entry before them is dropped. #2 — key-request:* DO storage grew unbounded forever [HIGH-silent] maybeSendKeyRequest stored `key-request:{room}:{session}` → ts to dedupe inside a 5-min window but never deleted. Every Megolm session rotation accumulated a permanent entry. Bucketed the dedupKey by 5-min window AND added a piggy-back cleanup that prunes entries older than 2 buckets on each call. Bounded storage. Deferred (with rationale): #3 pending-decrypt prune needs a cron sweeper; #5 KV dedupe race needs deeper redesign; #9 cachedKeys stale is low-impact (manual reload endpoint suffices); #10 parsePlan hijack has rare false-positives that haven't been observed. Tests: 67/67 pass; typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com>
6 tasks
replicas-connector Bot
added a commit
that referenced
this pull request
May 30, 2026
Closes out the deferred-with-rationale items from rounds 1–3 (commits 2313473, PR #18, PR #19). Every item now has a real code fix. **Round 1 (correctness/lifecycle) — 4 items** #1. pending-decrypt:* prune cron — listener alarm now runs a 30-min periodic prune that drops queue entries older than 7 days and deletes empty queues. Bounds long-term DO storage growth from sessions whose forwarders went offline before responding. #2. seen:* KV race redesign — dispatch dedupe was using KV's eventual read-modify-write, letting two concurrent reads both see "absent" and both proceed. New /dedup-claim endpoint on MatrixListener wraps the check + write in blockConcurrencyWhile for genuine put-if-absent semantics. Periodic prune handles cleanup (1h TTL); KV fallback preserved for the case where the listener DO is unreachable. #3. /admin/listener/reload-keys endpoint — clears the in-memory cachedKeys + cachedCurve25519 so the operator can rotate Megolm session keys (MATRIX_MEGOLM_KEYS_JSON) without a full redeploy. #4. parsePlan hijack tightening — only accept Plan(d/t) headers BEFORE any tool calls and before any prior plan has been parsed. Prevents a confused/adversarial agent from overwriting in-progress view by emitting `Plan (5/5)` mid-turn to fake completion. **Round 2 (security/spec/integration) — 4 items** #5. m.forwarded_room_key validation — vault now tags captured Megolm keys as `forwarded` vs `live`; listener checks `key-request:*` prefix to confirm we previously asked for the (room, session). If not, calls new vault `/keystore-delete` to evict the unsolicited key. Closes the "any Olm-paired sender can plant Megolm keys for any room" hole. #6. usage:org KV race — routed through OlmVault singleton DO via new /usage-bump (write, atomic) and /usage-read (read) endpoints. Concurrent Done frames across rooms can no longer drop entries via read-modify-write race. #7. Olm session cap bump — was .slice(-8), now .slice(-32). Verified multi-device senders rotate fast enough that 8 was occasionally dropping legitimate live sessions. Evictions are now logged so the operator can spot pathological churn. #8. /admin/recover-room missing-senderKey warning — surface bridges that strip the outer sender_key field (m.room_key_request routing degrades to `*` when missing, may not land on the originating device). Typecheck clean. 102/102 tests pass. Rolling tally tonight: 19 findings · 19 shipped · 0 deferred. Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
replicas-connector Bot
added a commit
that referenced
this pull request
May 30, 2026
Closes out the deferred-with-rationale items from rounds 1–3 (commits 2313473, PR #18, PR #19). Every item now has a real code fix. **Round 1 (correctness/lifecycle) — 4 items** #1. pending-decrypt:* prune cron — listener alarm now runs a 30-min periodic prune that drops queue entries older than 7 days and deletes empty queues. Bounds long-term DO storage growth from sessions whose forwarders went offline before responding. #2. seen:* KV race redesign — dispatch dedupe was using KV's eventual read-modify-write, letting two concurrent reads both see "absent" and both proceed. New /dedup-claim endpoint on MatrixListener wraps the check + write in blockConcurrencyWhile for genuine put-if-absent semantics. Periodic prune handles cleanup (1h TTL); KV fallback preserved for the case where the listener DO is unreachable. #3. /admin/listener/reload-keys endpoint — clears the in-memory cachedKeys + cachedCurve25519 so the operator can rotate Megolm session keys (MATRIX_MEGOLM_KEYS_JSON) without a full redeploy. #4. parsePlan hijack tightening — only accept Plan(d/t) headers BEFORE any tool calls and before any prior plan has been parsed. Prevents a confused/adversarial agent from overwriting in-progress view by emitting `Plan (5/5)` mid-turn to fake completion. **Round 2 (security/spec/integration) — 4 items** #5. m.forwarded_room_key validation — vault now tags captured Megolm keys as `forwarded` vs `live`; listener checks `key-request:*` prefix to confirm we previously asked for the (room, session). If not, calls new vault `/keystore-delete` to evict the unsolicited key. Closes the "any Olm-paired sender can plant Megolm keys for any room" hole. #6. usage:org KV race — routed through OlmVault singleton DO via new /usage-bump (write, atomic) and /usage-read (read) endpoints. Concurrent Done frames across rooms can no longer drop entries via read-modify-write race. #7. Olm session cap bump — was .slice(-8), now .slice(-32). Verified multi-device senders rotate fast enough that 8 was occasionally dropping legitimate live sessions. Evictions are now logged so the operator can spot pathological churn. #8. /admin/recover-room missing-senderKey warning — surface bridges that strip the outer sender_key field (m.room_key_request routing degrades to `*` when missing, may not land on the originating device). Typecheck clean. 102/102 tests pass. Rolling tally tonight: 19 findings · 19 shipped · 0 deferred. Co-authored-by: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com> Co-authored-by: itsablabla <itsablabla@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
6 tasks
replicas-connector Bot
added a commit
that referenced
this pull request
May 30, 2026
…ention gate, reverse-map backfill (#31) Round-4 audit of the new code from tonight's 12 PRs (#21-#30). Three real bugs found and fixed in this PR. **#1 (HIGH) — R2 archive basename can leak inline env-var values** `archiveBasename("Bash", { command: "OPENAI_API_KEY=sk-xxx node srv.js" })` used the first whitespace-delimited token (`OPENAI_API_KEY=sk-xxx`) as the R2 object's basename, which lands in the *public* r2.dev URL the bridge emits as `📎 View full output`. Any inline env-var assignment in front of a Bash command would publish the value to anyone who reads the chat. Fix: walk tokens skipping `KEY=value` env vars + `nohup`/`time`/`sudo`/ `exec`/`env` prefixes; defensively strip any trailing `=…` even if the regex missed something; cap the final basename at 40 chars. New test exercises the exact leak vector. **#2 (MED) — mention gate skipped over E2EE and voice content** `shouldHandleMessage(roomId, ev)` was reading body/m.mentions from the OUTER ev.content. For E2EE rooms that's only the megolm ciphertext (no body/mentions). For voice messages it's `"Voice message.ogg"` (not the transcript). Result: in E2EE groups the bot stayed silent on explicit mentions; in any group with voice messages the bot couldn't trigger on the "Jada" wake-word even when the transcript started with it. Fix: introduce `mentionContent` alongside `body`/`msgtype`. For plain events it's `ev.content`; for E2EE it's `tryDecrypt`'s inner content; for voice it's a synthetic `{ msgtype: "m.text", body: transcript }`. Then call `shouldHandleMessage(roomId, { content: mentionContent })`. **#3 (MED) — cross-room guard backward-compat hole** PR #29's guard only checks the `replica:{id}` reverse mapping; the 47 existing room mappings that pre-date the guard have no reverse, so the check is a no-op for them. Until each room respawns its replica (could be days), the contamination guard does nothing. Fix: opportunistic backfill on every successful follow-up — if the reverse mapping is absent, write it (this room owns this replica). Race-safe: a colliding peer's NEXT follow-up sees mismatch and respawns. Same TTL as the forward mapping. **Validation** - 173/173 tests pass (+8 new — archive basename security suite) - Typecheck clean - Deployed `43e13059` Co-authored-by: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com> Co-authored-by: itsablabla <itsablabla@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
7 tasks
replicas-connector Bot
added a commit
that referenced
this pull request
May 30, 2026
…sper budget (#32) * fix(matrix-bridge): 3 audit-found bugs — R2 secret leak, E2EE/voice mention gate, reverse-map backfill Round-4 audit of the new code from tonight's 12 PRs (#21-#30). Three real bugs found and fixed in this PR. **#1 (HIGH) — R2 archive basename can leak inline env-var values** `archiveBasename("Bash", { command: "OPENAI_API_KEY=sk-xxx node srv.js" })` used the first whitespace-delimited token (`OPENAI_API_KEY=sk-xxx`) as the R2 object's basename, which lands in the *public* r2.dev URL the bridge emits as `📎 View full output`. Any inline env-var assignment in front of a Bash command would publish the value to anyone who reads the chat. Fix: walk tokens skipping `KEY=value` env vars + `nohup`/`time`/`sudo`/ `exec`/`env` prefixes; defensively strip any trailing `=…` even if the regex missed something; cap the final basename at 40 chars. New test exercises the exact leak vector. **#2 (MED) — mention gate skipped over E2EE and voice content** `shouldHandleMessage(roomId, ev)` was reading body/m.mentions from the OUTER ev.content. For E2EE rooms that's only the megolm ciphertext (no body/mentions). For voice messages it's `"Voice message.ogg"` (not the transcript). Result: in E2EE groups the bot stayed silent on explicit mentions; in any group with voice messages the bot couldn't trigger on the "Jada" wake-word even when the transcript started with it. Fix: introduce `mentionContent` alongside `body`/`msgtype`. For plain events it's `ev.content`; for E2EE it's `tryDecrypt`'s inner content; for voice it's a synthetic `{ msgtype: "m.text", body: transcript }`. Then call `shouldHandleMessage(roomId, { content: mentionContent })`. **#3 (MED) — cross-room guard backward-compat hole** PR #29's guard only checks the `replica:{id}` reverse mapping; the 47 existing room mappings that pre-date the guard have no reverse, so the check is a no-op for them. Until each room respawns its replica (could be days), the contamination guard does nothing. Fix: opportunistic backfill on every successful follow-up — if the reverse mapping is absent, write it (this room owns this replica). Race-safe: a colliding peer's NEXT follow-up sees mismatch and respawns. Same TTL as the forward mapping. **Validation** - 173/173 tests pass (+8 new — archive basename security suite) - Typecheck clean - Deployed `43e13059` Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(bridges): R2 unguessable key + 30-day lifecycle + TG tables + Whisper budget Sweeps the remaining audit notes from the night's work — 3 fixes, both bridges. **#1 R2 archive: unguessable key + 30-day lifecycle** The sha256-derived archive key let an attacker who could *guess* the content (e.g. the bytes of a public /etc/os-release dump) reconstruct the public r2.dev URL without ever seeing it. In E2EE rooms where the URL is the only thing crossing the encryption boundary, that's a real leak. Switched to a 128-bit random prefix (`crypto.getRandomValues(16)` → hex). Loses cross-room dedup, gains genuine unguessability. Also set a 30-day lifecycle rule on the `replicas-output-archive` bucket via the CF R2 API so objects auto-prune. Cost stays bounded as the bridge runs indefinitely. **#2 TG markdown tables → pre-formatted text** Telegram's HTML parse_mode doesn't support `<table>`, but it does support `<pre>`. Added a pre-pass to `markdown.ts` that detects GFM tables (header row + `|---|---|` separator), computes column widths, emits the same data inside a fenced code block with `│` / `─┼` box- drawing separators. The existing fenced-code path picks it up unchanged. 2 new tests. **#3 Whisper cost guardrail** A determined sender could fire 25-minute voice messages back-to-back to burn OpenAI credits at $0.006/min × 25 = $0.15/message. With mirror-mode they'd ALSO trigger TTS replies. Added a per-room rolling 60-min budget tracked in DO storage: when the room has already used > 30 minutes of Whisper time this hour, new voice msgs are skipped (same UX as the no-API-key path). Budget ledger keyed by `voice-budget:{roomId}` with entries pruned to the rolling window on every check. Bounded storage. **Validation** - Matrix: 173/173 ✅ · typecheck clean · deployed `159cbb42` - Telegram: 99/99 ✅ (+2 table tests) · typecheck clean · deployed `a9d1165f` - R2 lifecycle verified live via CF API GET Co-Authored-By: itsablabla <itsablabla@users.noreply.github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com> Co-authored-by: itsablabla <itsablabla@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
replicas-connector Bot
added a commit
that referenced
this pull request
May 30, 2026
#33) Findings from Lark Admin room review: last turn there hit the 30-min timeout while the agent was *actively* making progress through multi- step MCP config cleanups. Session has 317 cumulative steps / 40 turns — this is real heavy ops work, not a runaway loop. Two fixes: **#1 Bump MAX_WATCH_DURATION_MS from 30 → 60 min** (both bridges) Lark Admin's "Watch timed out — agent ran longer than 30 min" was a false-positive cap on legitimate work. 60 min keeps runaway-loop protection while letting heavy ops finish. **#2 Cancel upstream replica + flush KV on timeout** When the watcher gave up at the cap, the upstream Replicas workspace kept processing silently — burning credits indefinitely. The bridge had no visibility into what state the replica ended up in. On timeout we now: - DELETE /v1/replica/{id} so the workspace stops - Flush `room:{roomId}` / `replica:{id}` (or `chat:{id}:thread:{tid}` for TG) KV mappings so the next message in the room spawns fresh - Then setTerminal + deleteAll as before - Failed-frame message updated: "Send a new message to start a fresh session." so the user knows the room is recoverable Matches the behavior we already had on claude-result 404/410 paths. **Validation** - Matrix: 173/173 ✅ · typecheck clean · deployed `1d6a1f2d` - Telegram: 99/99 ✅ · typecheck clean · deployed `944a6aa3` Co-authored-by: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com> Co-authored-by: itsablabla <itsablabla@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks
replicas-connector Bot
added a commit
that referenced
this pull request
May 30, 2026
…#36) Screenshot from Jaden showed a Failed frame fire at exactly 30:06 in a turn where the agent had legitimately done 4 tools + emitted multiple narration blocks, then went into a thinking phase where /history stopped reporting fresh events for 30 min. PR #35's 30-min idle cap was too aggressive. **Two changes (both bridges):** #1 `IDLE_TIMEOUT_MS` 30 min → 6 hours. Long thinking phases + waiting- on-user scenarios fit comfortably; truly abandoned watchers still get reaped instead of polling forever. The dispatch + auto-cleanup paths already handle the orphan case; this just gives them more room. #2 New `SLOW_POLL_THRESHOLD_MS` (10 min) + `SLOW_POLL_INTERVAL_MS` (30s). After 10 min of silence the alarm interval drops from 180ms to 30s — saves CF Worker invocations, KV reads, and Replicas /history calls during agent thinking phases. When events resume, the next tick notices and cadence snaps back to 180ms. Net: heavy ops + long thinking work unchanged; idle phases cost ~99% less; orphan detection still works on a 6-hour horizon. **Validation** - Matrix: 173/173 ✅ · typecheck clean · deployed `9db5e005` - Telegram: 99/99 ✅ · typecheck clean · deployed `1a87a434` Co-authored-by: replicas-connector[bot] <replicas-connector[bot]@users.noreply.github.com> Co-authored-by: itsablabla <itsablabla@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Updated STATUS.md with comprehensive April 2026 infrastructure health data from automated Devin audit. Replaces outdated Feb 2026 status.
Changes:
Review & Testing Checklist for Human
Notes
Link to Devin session: https://app.devin.ai/sessions/dff6dfa6552a4720938aae37389bb707
Requested by: @itsablabla