chore(benchmarking): add DuckDB WASM memory POC route#292
Open
ujaval403 wants to merge 1 commit into
Open
Conversation
Adds a /memory-poc route to the benchmarking app to characterize DuckDB-wasm memory: loads the 452MB NYC-taxi parquet via registerFileBuffer, creates a view, runs varied SQL (scan / count / group-by / high-cardinality group-by / order-by / window), and snapshots memory after each. Measures three layers: - duckdb_memory() per-tag (buffer-manager working set — PARQUET_READER, HASH_TABLE, ORDER_BY, WINDOW) - globFiles() — resident registered parquet buffer bytes from the WASM FS - measureUserAgentSpecificMemory() — true per-context total incl. the DuckDB WASM worker (available because the app is cross-origin isolated) Plus reclaim buttons: dropFiles(), reset(), terminate(). Key finding: WebAssembly.Memory only grows, never shrinks. dropFiles()/reset() free bytes inside DuckDB's allocator (duckdb_memory()/globFiles drop to 0) but the WASM linear-memory high-water — and thus the browser tab / OS footprint — stays at peak. Only terminate() (kill the worker) returns the memory to the OS, at the cost of a ~5s WASM recompile on the next query. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a
/memory-pocroute to the benchmarking app to characterize DuckDB-wasm memory behavior — driven by a real question: why does the browser tab hold ~2GB after list queries and only release it on engine shutdown?It loads the 452 MB NYC-taxi parquet via
registerFileBuffer, creates a view, runs varied SQL, and snapshots memory after each step.Measures 3 layers
duckdb_memory()per-tag — buffer-manager working set (PARQUET_READER, HASH_TABLE, ORDER_BY, WINDOW)globFiles()— resident registered parquet buffer bytes, straight from the WASM FSmeasureUserAgentSpecificMemory()— true per-context total including the DuckDB WASM worker (works because the benchmark app is cross-origin isolated via COOP/COEP)Plus reclaim buttons:
dropFiles(),reset(),terminate().Key finding
WebAssembly.Memoryonly grows, never shrinks (noshrink()in the spec).dropFiles()/reset()free the bytes inside DuckDB's allocator —duckdb_memory()andglobFiles()drop to 0, and the space is reusable for the next load — but the WASM buffer stays at its high-water size, so the browser-tab / OS footprint does not drop.terminate()(kill the worker → WASMMemoryArrayBuffer deallocated) returns the memory to the OS — at the cost of a ~5s WASM recompile on the next query.Implication: the idle
terminate()shutdown is the only lever that returns RAM to the browser.dropFiles()/reset()help intra-WASM reuse (cap re-growth on subsequent loads) but cannot reduce tab memory. To keep the peak low, the practical lever isdropFile()per table right after its query so N tables don't stack N×buffer.How to run
Auto-loads the parquet, then use the query + reclaim buttons; watch the snapshot table (DuckDB / globFiles / UA-total / JS-heap) and Activity Monitor.
Notes
meerkat-dbm/group.tsandpackage-lock.jsonin the working tree are unrelated and intentionally excluded from this PR.🤖 Generated with Claude Code