Conversation
Eng 1014/ci monorepo tests
fix(sdk): ENG-1725 `recallManual` broken after LOW-24
ENG-1409: Add benchmark scripts for Walrus, sidecar and recall latency
Handle delegate key limit in app
…nd-bulk-remember ENG-1406 + ENG-1408: Async remember pipeline and bulk remember
…ization ENG-1405: Optimize recall with LRU blob cache and batched SEAL decrypt
Lifts the practical /api/remember ceiling from ~8 KiB (text-embedding-3-small
context window) to 1 MiB by summarizing the plaintext via gpt-4o-mini before
embedding. The full original text is still SEAL-encrypted and stored on
Walrus — only the embedding input is summarized, so recall returns the
unmodified plaintext.
How it works
------------
text ≤ 8 KiB → embed(text) || encrypt(text) [unchanged]
text > 8 KiB → summarize(text) → embed(summary) || encrypt(text)
Map-reduce summarization handles arbitrarily large input within a single
embedder call:
1. split into ≤ 64 KiB chunks
2. summarize each chunk in parallel (gpt-4o-mini, bounded concurrency)
3. reduce the chunk summaries into one final summary (≤ ~500 words)
4. embed the final summary
Falls back to the direct embed path when OPENAI_API_KEY is unset (mock /
dev mode), so this doesn't introduce a hard dependency on OpenAI.
Boundary handling
-----------------
* MAX_REMEMBER_TEXT_BYTES = 1 MiB enforced inside the remember handler.
* MAX_ANALYZE_TEXT_BYTES = 64 KiB enforced inside the analyze handler
(analyze does a single LLM call with no chunking, so it has a tighter
cap independent of remember).
* Auth middleware caps protected JSON bodies at 2 MiB —
PROTECTED_BODY_LIMIT_BYTES — covering both single 1 MiB remember
payloads and bulk-remember batches.
* Sidecar /seal/encrypt cap raised to 2 MiB to accept the SEAL request
for the full original. The previous global app.use(json({limit:256kb}))
in scripts/sidecar-server.ts was masking per-route overrides on
/seal/decrypt-batch and /walrus/upload — those are now per-route only,
using named JSON_LIMIT_* constants.
Integration with PR #121 (async remember)
------------------------------------------
After the rebase onto dev, /api/remember is async (returns 202 + job_id).
Summarization runs inside spawn_prepare_remember_job and the bulk variant
spawn_prepare_bulk_remember_job, before the embed/encrypt fan-out. The
encrypt fork still uses the original text bytes — only the embedding
input is summarized.
Tests
-----
* services/server/tests/e2e_test.py — adds three parametric size cases:
64 KiB (asserts 200 + summarize log), 512 KiB (asserts 200), and
MAX_REMEMBER_TEXT_BYTES + 1 (asserts 400). Mirrors the Rust constant
to catch drift.
* 135/135 unit tests pass, including new assertions on
MAX_ANALYZE_TEXT_BYTES, PROTECTED_BODY_LIMIT_BYTES, and the bench-
bypass default.
Benchmark harness
-----------------
services/server/scripts/bench-remember-sizes.ts drives 14 hand-curated
fixtures (Wikipedia prose, Project Gutenberg / Journey to the West for
CJK, science-dense prose, structured JSON, mixed markdown+code) at sizes
4 KiB → 1 MiB through the full async lifecycle (POST → poll → recall),
asserting 202 / job done / 400 boundary as appropriate.
Bench results against testnet (RATE_LIMIT_DISABLED=1):
Size Worker Recall Status
4 KiB ~25 s ~3 s done
64 KiB ~22-26 s ~1-3 s done
256 KiB ~36-42 s ~1-3 s done
512 KiB ~42 s ~1 s done
1 MiB ~60 s ~3 s done
1 MiB+1 — — 400 ✅
14/14 fixtures pass end-to-end.
Benchmark-only escape hatch
---------------------------
The async remember pipeline turns one user request into 1 POST + N status
polls + 1 recall, which exceeds the 30-weighted-req/min per-delegate-key
budget on the second fixture. Adds RATE_LIMIT_DISABLED=1 (default off,
asserted in tests, loud tracing::warn at startup, surfaced in /config)
that skips request-rate buckets only — storage quota and auth still
apply. Intended for localhost benchmarks.
Files changed
-------------
services/server/src/routes.rs — summarize_for_embedding + map-reduce,
MAX_ANALYZE_TEXT_BYTES guard,
summarize wired into async/bulk paths
services/server/src/auth.rs — PROTECTED_BODY_LIMIT_BYTES = 2 MiB
services/server/src/main.rs — DefaultBodyLimit + bench-bypass startup warn
services/server/src/rate_limit.rs — bench_bypass_enabled flag + bypass
services/server/src/types.rs — ConfigResponse.rate_limit_disabled
services/server/scripts/sidecar-server.ts
— per-route json() limits with named constants
services/server/scripts/bench-remember-sizes.ts
— async-aware harness with poll loop
services/server/scripts/bench-fixtures.json
— 14 hand-curated realistic fixtures
services/server/tests/e2e_test.py — parametric size cases
Co-authored-by: ducnmm <mauduckiengiang@gmail.com>
hungtranphamminh
approved these changes
May 7, 2026
chore(sdk): bump version to 0.0.3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
staging <- dev