perf: background write batcher for DuckDB telemetry ingest

[jstojiljkovic]

DuckDB ingest did all storage work inside each request: one connection, one Appender, one WAL commit per HTTP request, with the admission gate capping concurrency at 4. Requests held slots for 1-3s each. At the benchmark cliff, ingest p50 was 1.2-3.2 seconds.

What changed

• Per-table background writers (telemetry/duckdb/writer.go): requests decode, convert, and enqueue onto a bounded per-table queue, then return 200 in milliseconds; a pool of writers per hot table (2xCPU, capped 8) owns persistent Appenders and flushes at 32k rows / 100ms, so one WAL commit is shared by hundreds of requests. A 200 now means "accepted into the queue" (OTel-Collector-style ack, loss window ≤ one flush on crash).
• Reject-before-ack backpressure: a full queue waits up to 2s for space (VM-style), then answers 503 + Retry-After. Acked rows are never silently dropped. Queue depth is exposed in /api/health/deep and traceway.duckdb.write_queue.* metrics.
• Native UUID columns (migration 0002): hot-table ids stored as 16-byte UUIDs instead of VARCHAR(36); converters pass duckdb.UUID zero-alloc.
• Pooled decode scratch: sync.Pool for gzip readers and OTLP body buffers; memoized resource/scope attribute marshals in the logs converter.
• Read-your-writes preserved: the notification event evaluator flushes writers before reading back just-ingested rows; sessions, profiling_stacks, and exception_stack_traces stay synchronous.

Results:

Same hardware (ccx13, 2 vCPU) - datasets committed on this branch:

Metric	spans	metrics	logs
Sustained ingest (items/s)	95,737 -> 151,079 (+58%)	254,242 -> 324,259 (+28%)	75,225 -> 79,839 (+6%)
Ingest p50, best passing bulk step	1,167ms -> 292ms	3,207ms -> 72ms	2,958ms -> 148ms
Read-probe median @ 10M rows	902ms -> 637ms	382ms -> 314ms	85ms -> 55ms

Hardware scaling (same code, ccx23, 4 vCPU), artifacts on runs
30009675827 / 30028543147

Metric	spans	metrics	logs
Sustained ingest (items/s)	285,723 (1.89x)	685,935 (2.12x)	155,870 (1.95x)

Writers, admission, and DuckDB threads all scale with CPU, so capacity scales near-linearly with cores - previously added cores mostly fed per-request overhead. On both tiers, the load generator saturates before the SUT on spans/metrics small-batch traffic (0% SUT errors at the failed loadgen steps).

Zero dropped rows and zero insert failures in every step of every run. Read-probe pass levels are unchanged; the spans/logs 100M read cliffs predate this branch