Skip to content

chore(benchmarking): add DuckDB WASM memory POC route#292

Open
ujaval403 wants to merge 1 commit into
mainfrom
chore/duckdb-memory-poc-benchmark
Open

chore(benchmarking): add DuckDB WASM memory POC route#292
ujaval403 wants to merge 1 commit into
mainfrom
chore/duckdb-memory-poc-benchmark

Conversation

@ujaval403

Copy link
Copy Markdown
Contributor

What

Adds a /memory-poc route to the benchmarking app to characterize DuckDB-wasm memory behavior — driven by a real question: why does the browser tab hold ~2GB after list queries and only release it on engine shutdown?

It loads the 452 MB NYC-taxi parquet via registerFileBuffer, creates a view, runs varied SQL, and snapshots memory after each step.

Measures 3 layers

  • duckdb_memory() per-tag — buffer-manager working set (PARQUET_READER, HASH_TABLE, ORDER_BY, WINDOW)
  • globFiles() — resident registered parquet buffer bytes, straight from the WASM FS
  • measureUserAgentSpecificMemory() — true per-context total including the DuckDB WASM worker (works because the benchmark app is cross-origin isolated via COOP/COEP)

Plus reclaim buttons: dropFiles(), reset(), terminate().

Key finding

WebAssembly.Memory only grows, never shrinks (no shrink() in the spec).

  • Loading data + running queries grows the WASM linear memory to a high-water mark (250MB → ~2GB in Activity Monitor).
  • dropFiles() / reset() free the bytes inside DuckDB's allocator — duckdb_memory() and globFiles() drop to 0, and the space is reusable for the next load — but the WASM buffer stays at its high-water size, so the browser-tab / OS footprint does not drop.
  • Only terminate() (kill the worker → WASM Memory ArrayBuffer deallocated) returns the memory to the OS — at the cost of a ~5s WASM recompile on the next query.

Implication: the idle terminate() shutdown is the only lever that returns RAM to the browser. dropFiles()/reset() help intra-WASM reuse (cap re-growth on subsequent loads) but cannot reduce tab memory. To keep the peak low, the practical lever is dropFile() per table right after its query so N tables don't stack N×buffer.

How to run

nx serve benchmarking-app   # → http://localhost:4204/memory-poc

Auto-loads the parquet, then use the query + reclaim buttons; watch the snapshot table (DuckDB / globFiles / UA-total / JS-heap) and Activity Monitor.

Notes

  • POC / diagnostic only — not wired into any product path.
  • meerkat-dbm/group.ts and package-lock.json in the working tree are unrelated and intentionally excluded from this PR.

🤖 Generated with Claude Code

Adds a /memory-poc route to the benchmarking app to characterize DuckDB-wasm
memory: loads the 452MB NYC-taxi parquet via registerFileBuffer, creates a
view, runs varied SQL (scan / count / group-by / high-cardinality group-by /
order-by / window), and snapshots memory after each.

Measures three layers:
- duckdb_memory() per-tag (buffer-manager working set — PARQUET_READER,
  HASH_TABLE, ORDER_BY, WINDOW)
- globFiles() — resident registered parquet buffer bytes from the WASM FS
- measureUserAgentSpecificMemory() — true per-context total incl. the DuckDB
  WASM worker (available because the app is cross-origin isolated)

Plus reclaim buttons: dropFiles(), reset(), terminate().

Key finding: WebAssembly.Memory only grows, never shrinks. dropFiles()/reset()
free bytes inside DuckDB's allocator (duckdb_memory()/globFiles drop to 0) but
the WASM linear-memory high-water — and thus the browser tab / OS footprint —
stays at peak. Only terminate() (kill the worker) returns the memory to the OS,
at the cost of a ~5s WASM recompile on the next query.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant