blog: SQLite vs DuckDB benchmark + ingest admission gate

.github/workflows/benchmark-hardware.yml

@@ -54,6 +54,14 @@ on:
         description: 'Read-probe fill ladder override, comma-separated row counts (default 100000,1000000,10000000,100000000). e.g. 10000000,100000000,1000000000 to probe 1B — raise duration to cover the fill, it is a hard cap on the whole run. Ignored unless scenario=read-probe.'
         required: false
         default: ''
+      phase2_request_rates:
+        description: 'Throughput Phase 2 request-rate ladder override, comma-separated req/sec (default 1,5,25,100,400). Use a finer ramp (e.g. 1,5,10,15,20,25) to pin a cliff the default 5-to-25 jump skips over — needed when the failing step kills the SUT so post-fail bisection cannot run. Ignored unless scenario=throughput.'
+        required: false
+        default: ''
+      read_threshold_ms:
+        description: 'Read-probe pass threshold override in ms (default 5000; probe timeout is threshold + 1s). Raise for diagnostic dispatches (e.g. 60000) to measure how long slow queries actually take instead of truncating at 6s. Leave empty for headline runs so the series stays comparable. Ignored unless scenario=read-probe.'
+        required: false
+        default: ''
       commit_results:
         description: 'Commit results to the dispatched ref (REPLACES the whole scenario results folder with only this dispatch''s entries). Set false for exploratory or partial-matrix runs — per-entry result JSONs are still available as the matrix jobs'' workflow artifacts.'
         required: true
@@ -113,6 +121,8 @@ jobs:
       BENCH_FILL_BATCH_SIZE: ${{ github.event.inputs.fill_batch_size }}
       BENCH_FILL_REQUEST_RATE: ${{ github.event.inputs.fill_request_rate }}
       BENCH_FILL_LEVELS: ${{ github.event.inputs.fill_levels }}
+      BENCH_READ_THRESHOLD_MS: ${{ github.event.inputs.read_threshold_ms }}
+      BENCH_PHASE2_REQUEST_RATES: ${{ github.event.inputs.phase2_request_rates }}
       # Managed-ClickHouse credentials. Used only when modes contains
       # managed-ch; empty for sqlite/pgch entries (harmless — sut-bootstrap
       # only validates them when MODE=managed-ch).

CLAUDE.md

@@ -257,6 +257,10 @@ DUCKDB_MEMORY_LIMIT=                  # e.g. 4GB. Unset = DuckDB auto-tunes (~80
 DUCKDB_THREADS=                       # e.g. 4. Unset = DuckDB auto-tunes (= cores). Cap in constrained/shared environments.
 DUCKDB_CHECKPOINT_THRESHOLD=          # e.g. 256MB. Unset = DuckDB default (16MB). Raise under sustained ingest to reduce WAL checkpoint stalls; costs a larger WAL and longer restart replay.
 
+# Ingest admission gate (all telemetry ingest endpoints: /api/report, /api/profiles/ingest, /api/otel/*)
+INGEST_MAX_CONCURRENT=                # max concurrently processed ingest requests. Unset = 2×CPU cores, min 4. Bounds ingest memory so overload sheds load with 503s instead of the process being OOM-killed (on DuckDB an OOM death is followed by a minutes-long WAL-replay stall on restart).
+INGEST_ADMISSION_WAIT_SECONDS=5       # how long a request may wait for a slot before the 503 + Retry-After; 0 = reject immediately when saturated
+
 # Notifications
 NOTIFICATION_POLL_SECONDS=60          # polled rule evaluation interval; minimum 5, invalid values fall back to 60
 
@@ -797,7 +801,7 @@ Built with `-tags telemetry_duckdb` (`CGO_ENABLED=1` required), this is an alter
 - **Driver:** `github.com/duckdb/duckdb-go/v2` (the official driver; marcboeker/go-duckdb is deprecated). Bundles prebuilt static libs for glibc only — **not musl/Alpine**, so the image uses Debian (`Dockerfile.duckdb`).
 - **Opened in** `backend/app/db/db_telemetry_duckdb.go`: telemetry path is the SQLite path with `.db` swapped for `_telemetry.duckdb`. By default DuckDB auto-tunes to the host; `DUCKDB_MEMORY_LIMIT`/`DUCKDB_THREADS`/`DUCKDB_CHECKPOINT_THRESHOLD` (passed through as DSN config options) let operators cap memory/threads so a memory-capped container doesn't read the host's RAM and OOM-kill the backend, and raise the WAL checkpoint threshold (default 16MB) so sustained Appender ingest isn't stalled by frequent checkpoints. `preserve_insertion_order=false` is always set — telemetry reads all have explicit ORDER BY, and dropping the guarantee lets DuckDB parallelize bulk loads and large scans with less memory. The read pool is bounded (`SetMaxOpenConns(duckDBMaxReadConns)`) since each DuckDB connection can use all threads + its own query memory; Appender writes use their own `DuckDBConnector.Connect()` connections and bypass that cap. Exposes `db.DuckDBConnector` (needed for the Appender).
 - **Writes use the Appender API**, not `INSERT` (`duckdb.NewAppenderFromConn(conn, "", table)` → `AppendRow(...)` → `Close()` flushes). Upserts still go through `ExecContext` with `ON CONFLICT`. The Appender rejects typed `*string` for nullable VARCHAR — use `nullableString()` in `backend/app/repositories/telemetry/duckdb/helpers.go` (returns untyped `nil` or the dereferenced value).
-- **Write-path observability:** a row the Appender rejects is dropped rather than failing the whole frame (the SQLite backend 500s instead), so a poison row cannot wedge the SDK's retry loop. Every drop increments a per-table counter (`db.RecordTelemetryRowDropped`) and fires a rate-limited (1/min per table) `traceway.CaptureException`; Appender flush/connect failures still propagate to the request (500, SDK retries) and increment an insert-failure counter. `GET /api/health/deep` (App auth, all telemetry backends) exposes `telemetryBackend`, `droppedRows` per table, `droppedRowsTotal`, `insertFailures`, and on DuckDB an `engine` object (db/WAL file bytes, `duckdb_memory()` usage, read-pool in-use/wait stats) alongside its existing ClickHouse fields; it 503s only when the configured telemetry backend is ClickHouse and CH is unreachable (the embedded backends answer 200 with `chReachable:false`). The benchmark loadgen polls it before/after every ramp step and fails any step whose drop delta is nonzero; read-probe fills record cumulative `droppedRows` per fill level. When `MONITORING_TRACEWAY_URL` is set, `monitoring.StartTelemetryDBReporter` also emits `traceway.duckdb.*` metrics every 10s: `rows_dropped.delta`, `insert_failures.delta`, `db_size_mb`, `wal_size_mb`, `memory_used_mb`, `read_pool.in_use`, `read_pool.wait_count.delta`, `read_pool.wait_ms.delta`. The hourly retention worker issues a `CHECKPOINT` after its deletes so retention actually reclaims disk (DuckDB otherwise defers reclamation to the WAL checkpoint threshold).
+- **Write-path observability:** a row the Appender rejects is dropped rather than failing the whole frame (the SQLite backend 500s instead), so a poison row cannot wedge the SDK's retry loop. Every drop increments a per-table counter (`db.RecordTelemetryRowDropped`) and fires a rate-limited (1/min per table) `traceway.CaptureException`; Appender flush/connect failures still propagate to the request (500, SDK retries) and increment an insert-failure counter. `GET /api/health/deep` (App auth, all telemetry backends) exposes `telemetryBackend`, `droppedRows` per table, `droppedRowsTotal`, `insertFailures`, `ingestRejected` (requests turned away by the ingest admission gate), and on DuckDB an `engine` object (db/WAL file bytes, `duckdb_memory()` usage, read-pool in-use/wait stats) alongside its existing ClickHouse fields; it 503s only when the configured telemetry backend is ClickHouse and CH is unreachable (the embedded backends answer 200 with `chReachable:false`). The benchmark loadgen polls it before/after every ramp step and fails any step whose drop delta is nonzero; read-probe fills record cumulative `droppedRows` per fill level. When `MONITORING_TRACEWAY_URL` is set, `monitoring.StartTelemetryDBReporter` also emits `traceway.duckdb.*` metrics every 10s: `rows_dropped.delta`, `insert_failures.delta`, `db_size_mb`, `wal_size_mb`, `memory_used_mb`, `read_pool.in_use`, `read_pool.wait_count.delta`, `read_pool.wait_ms.delta`. The hourly retention worker issues a `CHECKPOINT` after its deletes so retention actually reclaims disk (DuckDB otherwise defers reclamation to the WAL checkpoint threshold).
 - **`lit` placeholders:** `db.Driver` stays `lit.SQLite`, which emits `?` — DuckDB accepts these, so no separate driver was needed for reads.
 - **Migrations:** `backend/app/migrations/duckdb_telemetry/` (mirrors `sqlite_telemetry/` table-for-table; integer columns are `BIGINT`, JSON is `VARCHAR`, no secondary indexes since it's columnar).
 - **Dialect gotchas vs SQLite** (the read queries differ): native `quantile_cont(col, p)` for P50/P95/P99 instead of fetch-and-sort; `strftime('%s',col)`→`epoch(col)`; time bucketing via `time_bucket(to_seconds(N), col, TIMESTAMP '1970-01-01')` — the explicit epoch origin is required because DuckDB anchors sub-day buckets at 2000-01-03 by default, which would misalign chart buckets against the SQLite backend's epoch-floored buckets for any interval that doesn't evenly divide a day; `json_extract`→`json_extract_string`; `json_each`→`LATERAL unnest(json_keys(x))`; strict GROUP BY needs `ANY_VALUE`/`arg_max`; `SUM` returns HUGEINT (CAST to BIGINT); `CAST(.. AS REAL)`→`CAST(.. AS DOUBLE)`.

backend/app/controllers/health_deep.go

@@ -35,6 +35,7 @@ type HealthDeepResponse struct {
 	DroppedRows      map[string]uint64        `json:"droppedRows"`
 	DroppedRowsTotal uint64                   `json:"droppedRowsTotal"`
 	InsertFailures   uint64                   `json:"insertFailures"`
+	IngestRejected   uint64                   `json:"ingestRejected"`
 	Engine           *db.TelemetryEngineStats `json:"engine,omitempty"`
 }
 
@@ -49,6 +50,7 @@ func (h healthDeepController) Get(c *gin.Context) {
 	resp.DroppedRows = dropped
 	resp.DroppedRowsTotal = droppedTotal
 	resp.InsertFailures = insertFailures
+	resp.IngestRejected = db.GetIngestRejects()
 	if engine, ok := db.GetTelemetryEngineStats(c.Request.Context()); ok {
 		resp.Engine = &engine
 	}

backend/app/controllers/routes.go

@@ -35,20 +35,20 @@ type Pagination struct {
 
 func RegisterControllers(router *gin.RouterGroup) {
 	router.OPTIONS("/report", middleware.CORSReport)
-	router.POST("/report", middleware.CORSReport, middleware.UseClientAuth, middleware.UseGzip, clientcontrollers.ClientController.Report)
+	router.POST("/report", middleware.CORSReport, middleware.UseClientAuth, middleware.IngestAdmission, middleware.UseGzip, clientcontrollers.ClientController.Report)
 
 	router.OPTIONS("/profiles/ingest", middleware.CORSReport)
-	router.POST("/profiles/ingest", middleware.CORSReport, middleware.UseClientAuth, middleware.UseGzip, clientcontrollers.ProfileIngestController.Ingest)
+	router.POST("/profiles/ingest", middleware.CORSReport, middleware.UseClientAuth, middleware.IngestAdmission, middleware.UseGzip, clientcontrollers.ProfileIngestController.Ingest)
 
 	otelGroup := router.Group("/otel")
 	otelGroup.OPTIONS("/v1/traces", middleware.CORSReport)
 	otelGroup.OPTIONS("/v1/metrics", middleware.CORSReport)
 	otelGroup.OPTIONS("/v1/logs", middleware.CORSReport)
 	otelGroup.OPTIONS("/v1development/profiles", middleware.CORSReport)
-	otelGroup.POST("/v1/traces", middleware.CORSReport, middleware.UseClientAuth, otelcontrollers.OtelController.ExportTraces)
-	otelGroup.POST("/v1/metrics", middleware.CORSReport, middleware.UseClientAuth, otelcontrollers.OtelController.ExportMetrics)
-	otelGroup.POST("/v1/logs", middleware.CORSReport, middleware.UseClientAuth, otelcontrollers.OtelController.ExportLogs)
-	otelGroup.POST("/v1development/profiles", middleware.CORSReport, middleware.UseClientAuth, otelcontrollers.OtelController.ExportProfiles)
+	otelGroup.POST("/v1/traces", middleware.CORSReport, middleware.UseClientAuth, middleware.IngestAdmission, otelcontrollers.OtelController.ExportTraces)
+	otelGroup.POST("/v1/metrics", middleware.CORSReport, middleware.UseClientAuth, middleware.IngestAdmission, otelcontrollers.OtelController.ExportMetrics)
+	otelGroup.POST("/v1/logs", middleware.CORSReport, middleware.UseClientAuth, middleware.IngestAdmission, otelcontrollers.OtelController.ExportLogs)
+	otelGroup.POST("/v1development/profiles", middleware.CORSReport, middleware.UseClientAuth, middleware.IngestAdmission, otelcontrollers.OtelController.ExportProfiles)
 
 	router.GET("/projects", middleware.UseAppAuth, ProjectController.ListProjects)
 	router.POST("/projects", middleware.UseAppAuth, middleware.RequireProjectAccess, ProjectController.CreateProject)

backend/app/db/telemetry_stats.go

@@ -37,8 +37,23 @@ var (
 	telemetryIngestMu       sync.Mutex
 	telemetryDroppedRows    = map[string]uint64{}
 	telemetryInsertFailures uint64
+	telemetryIngestRejects  uint64
 )
 
+// RecordIngestRejected counts requests turned away by the ingest admission
+// gate (503). Rejects are load shedding, not data loss — the client retries.
+func RecordIngestRejected() {
+	telemetryIngestMu.Lock()
+	telemetryIngestRejects++
+	telemetryIngestMu.Unlock()
+}
+
+func GetIngestRejects() uint64 {
+	telemetryIngestMu.Lock()
+	defer telemetryIngestMu.Unlock()
+	return telemetryIngestRejects
+}
+
 func RecordTelemetryRowDropped(table string) {
 	telemetryIngestMu.Lock()
 	telemetryDroppedRows[table]++

backend/app/middleware/ingest_admission.go

@@ -0,0 +1,80 @@
+package middleware
+
+import (
+	"net/http"
+	"os"
+	"runtime"
+	"strconv"
+	"sync"
+	"time"
+
+	"github.com/gin-gonic/gin"
+	"github.com/tracewayapp/traceway/backend/app/db"
+)
+
+// The telemetry ingest path decompresses and decodes every request body it
+// accepts; without a bound, a burst of fat OTLP batches holds decoded copies
+// of all of them in memory at once and the kernel OOM-kills the process
+// (observed on 8 GB hosts under sustained log ingest — and on the DuckDB
+// backend the death is followed by a WAL-replay stall on restart during
+// which nothing answers). The gate caps concurrent ingest processing. The
+// slot is acquired before the body is read, so a waiting request holds only
+// a connection; a request that cannot get a slot within the wait window is
+// rejected with 503 + Retry-After, which SDKs and OTLP collectors retry.
+func newIngestAdmission(capacity int, wait time.Duration) gin.HandlerFunc {
+	slots := make(chan struct{}, capacity)
+	return func(c *gin.Context) {
+		select {
+		case slots <- struct{}{}:
+		default:
+			timer := time.NewTimer(wait)
+			defer timer.Stop()
+			select {
+			case slots <- struct{}{}:
+			case <-timer.C:
+				db.RecordIngestRejected()
+				c.Header("Retry-After", "2")
+				c.AbortWithStatusJSON(http.StatusServiceUnavailable, gin.H{"error": "ingest saturated, retry later"})
+				return
+			case <-c.Request.Context().Done():
+				c.Abort()
+				return
+			}
+		}
+		defer func() { <-slots }()
+		c.Next()
+	}
+}
+
+func ingestCapacityFromEnv() int {
+	n := runtime.NumCPU() * 2
+	if n < 4 {
+		n = 4
+	}
+	if v := os.Getenv("INGEST_MAX_CONCURRENT"); v != "" {
+		if parsed, err := strconv.Atoi(v); err == nil && parsed > 0 {
+			n = parsed
+		}
+	}
+	return n
+}
+
+func ingestWaitFromEnv() time.Duration {
+	if v := os.Getenv("INGEST_ADMISSION_WAIT_SECONDS"); v != "" {
+		if parsed, err := strconv.Atoi(v); err == nil && parsed >= 0 {
+			return time.Duration(parsed) * time.Second
+		}
+	}
+	return 5 * time.Second
+}
+
+// The gate is built on first use, not at package init: godotenv.Load() runs
+// inside run(), after package-level vars initialize, so reading the env here
+// eagerly would silently ignore INGEST_* values set via a .env file.
+var ingestAdmission = sync.OnceValue(func() gin.HandlerFunc {
+	return newIngestAdmission(ingestCapacityFromEnv(), ingestWaitFromEnv())
+})
+
+func IngestAdmission(c *gin.Context) {
+	ingestAdmission()(c)
+}

backend/app/middleware/ingest_admission_test.go

@@ -0,0 +1,93 @@
+package middleware
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/gin-gonic/gin"
+)
+
+func newAdmissionRouter(capacity int, wait time.Duration, release chan struct{}) *gin.Engine {
+	gin.SetMode(gin.TestMode)
+	r := gin.New()
+	r.POST("/ingest", newIngestAdmission(capacity, wait), func(c *gin.Context) {
+		if release != nil {
+			<-release
+		}
+		c.Status(http.StatusOK)
+	})
+	return r
+}
+
+func TestIngestAdmissionPassesUnderCapacity(t *testing.T) {
+	r := newAdmissionRouter(2, 50*time.Millisecond, nil)
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/ingest", nil))
+	if w.Code != http.StatusOK {
+		t.Fatalf("expected 200 under capacity, got %d", w.Code)
+	}
+}
+
+func TestIngestAdmissionRejectsWhenSaturated(t *testing.T) {
+	release := make(chan struct{})
+	r := newAdmissionRouter(1, 50*time.Millisecond, release)
+
+	var wg sync.WaitGroup
+	wg.Add(1)
+	go func() {
+		defer wg.Done()
+		w := httptest.NewRecorder()
+		r.ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/ingest", nil))
+	}()
+
+	// Give the goroutine time to acquire the only slot; its handler then
+	// blocks on the release channel, guaranteeing saturation below.
+	time.Sleep(20 * time.Millisecond)
+
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/ingest", nil))
+	if w.Code != http.StatusServiceUnavailable {
+		t.Fatalf("expected 503 when saturated, got %d", w.Code)
+	}
+	if w.Header().Get("Retry-After") == "" {
+		t.Fatalf("expected Retry-After header on 503")
+	}
+
+	close(release)
+	wg.Wait()
+
+	w2 := httptest.NewRecorder()
+	r.ServeHTTP(w2, httptest.NewRequest(http.MethodPost, "/ingest", nil))
+	if w2.Code != http.StatusOK {
+		t.Fatalf("expected 200 after release, got %d", w2.Code)
+	}
+}
+
+func TestIngestAdmissionWaiterGetsSlotWhenFreed(t *testing.T) {
+	release := make(chan struct{})
+	r := newAdmissionRouter(1, 2*time.Second, release)
+
+	var wg sync.WaitGroup
+	wg.Add(1)
+	go func() {
+		defer wg.Done()
+		w := httptest.NewRecorder()
+		r.ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/ingest", nil))
+	}()
+	time.Sleep(20 * time.Millisecond)
+
+	go func() {
+		time.Sleep(50 * time.Millisecond)
+		close(release)
+	}()
+
+	w := httptest.NewRecorder()
+	r.ServeHTTP(w, httptest.NewRequest(http.MethodPost, "/ingest", nil))
+	if w.Code != http.StatusOK {
+		t.Fatalf("expected waiting request to get the freed slot, got %d", w.Code)
+	}
+	wg.Wait()
+}

benchmarks/README.md

@@ -46,6 +46,21 @@ the `--scenario` flag.
   5 s). Results land in `benchmarks/results-probe/`. Answers "how big
   can the table grow before the dashboard read on this endpoint cliffs?"
 
+  Between the fill and the probes sits a **digestion gate**
+  (`--max-digest-wait`, default 10 m): the loadgen polls `/api/health/deep`
+  until the engine's db+WAL file sizes are stable across two consecutive
+  polls. DuckDB checkpoints its WAL asynchronously after ingest stops, and
+  on a small tier that checkpoint can outlast the settle window — probing
+  through it produces uniform timeouts on sub-millisecond queries
+  (a wedged engine, not query cost; observed as a nondeterministic
+  logs/spans "cliff" on ccx13 before the gate existed). The wait is
+  recorded per fill level as `digestSeconds` (with `digestTimedOut` when
+  the cap expired), so "how long until the box answers again after a
+  burst" is itself a published number. Backends without engine gauges
+  (SQLite, ClickHouse) pass the gate on the first poll, and the probes
+  use a client without the shared 30 s timeout so `--read-threshold-ms`
+  above 29 s actually takes effect.
+
 The two scenarios write to sibling folders so one never overwrites the
 other. Each folder is wiped on each dispatch of its scenario.