diff --git a/AGENTS.md b/AGENTS.md index 1e3ec198..875d0dd0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -59,7 +59,7 @@ The invariant index — what must stay true. Full narrative and rationale live i 10. **Active Sweeper** — purges NATS messages that are both ACKed (written to CH) and older than the gap window; SSE gap-fill uses `DeliverByStartTime`, no in-process ring buffer. 11. **Hasura-style access control: fail-closed (security)** — `policy.IsAdmin` (role == `admin_role`, **exact case-sensitive**, default `"admin"`) is the single admin check, shared by `Evaluate`/`ResolveRole`/`Validate`/the `/v1/admin` gate/`RoleAllowed`. Empty/absent role matches nothing (no `"*"` wildcard); `Validate` rejects empty role keys; a `nil` policy (deleted) denies **everyone incl. admin** via a role — a total lockout for token-based callers, so bootstrap from the policy file, never an implicit admin grant (**exception:** the operator key's `auth.IsOperator` bit passes the `/v1/admin` gate even under a `nil` policy — a deliberate break-glass restore over HTTP, see #7). `default_role` is the one sanctioned roleless exception (`ResolveRole` maps empty → it pre-eval); `default_role == admin_role` is permitted but dev-only and loudly warned (`policy.DefaultRoleGrantsAdmin`). Preserve when touching `internal/policy` (policy twin of #13; see #159). Detail: architecture.md § `policy/`. 12. **Structured queries: column authz fail-closed (security)** — `POST /v1/query?table={table}`: typed AST validated against schema, permission-enforced, timestamp-bucketed for cache, `DefaultMaxRows` (10,000) cap. Every column reference — projection, aggregation args, `filters`, `group_by`, `order_by`, `time_range` — is authorized inside `query.Build` (the single chokepoint that enumerates them all), so no clause can skip the role's `allow_columns`/`deny_columns` check (#223). A `select_all` read by a *column-restricted* role expands to its allowed columns via `policy.AllowedProjection`, never a bare `SELECT *`; *unrestricted*/admin roles keep `SELECT *` (`policy.RestrictsColumns` decides). Omitting `columns` selects nothing (`ErrEmptyProjection` → `200 []`); `["*"]` is the literal column `*` (schema-gated, not a wildcard); a table-granted role with no readable columns fails closed (`ErrNoReadableColumns` → `403`). Structured and live-stream (`stream.filterColumns`) reads share the one per-column decision `policy.IsColumnAllowed`, so column visibility can't drift. Preserve when touching `internal/query` or the structured-query handler. Detail: architecture.md § `query/`. -13. **Named query pipes: fail-closed (security)** — pre-defined SQL templates (Tinybird-style) with param binding + caching; `GET/POST /v1/pipes/{name}` sit outside `RequireAdmin`, so per-pipe `allowed_roles` is the *only* execute-path gate, via `policy.RoleAllowed`: exact allowlist membership (no `"*"`), admin always passes, empty/absent role and empty-string entries authorize nobody, and no `allowed_roles` → admin-only. Preserve and exercise via `testutil.RunRoleMatrix` / `StandardRoleMatrix` (see #159). Detail: architecture.md § `pipes/`. +13. **Named query pipes: fail-closed (security)** — pre-defined SQL templates (Tinybird-style) with param binding + caching; `GET/POST /v1/pipes/{name}` sit outside `RequireAdmin`, so per-pipe `allowed_roles` is the *only* execute-path gate, via `policy.RoleAllowed`: exact allowlist membership (no `"*"`), admin always passes, empty/absent role and empty-string entries authorize nobody, and no `allowed_roles` → admin-only. Preserve and exercise via `testutil.RunRoleMatrix` / `StandardRoleMatrix` (see #159). The pipe's SQL is **not** re-checked against the table policy (no row or column filtering) — `allowed_roles` gates execution, but the author is responsible for scoping the data in the pipe SQL and any views it reads. A pipe is **run as-is** — WaveHouse never parses its SQL and never rejects it (a write/DDL or multi-statement pipe runs on behalf of the caller's role, so the author owns scoping it). Cache invalidation derives the pipe's table dependencies from ClickHouse `EXPLAIN QUERY TREE`: a query it can't analyze (a write/DDL pipe, a missing table, an unreachable server) falls back to the single **database-wide version** namespace that every write bumps, so any write evicts it — never an under-resolution — and a result that can't be reliably version-invalidated (an unfoldable view, an external read like a table function or cross-database table, or a dead-branch-pruned dependency set) is **TTL-capped** (`cache.UnresolvedDepsTTLCap`) rather than trusted to version invalidation. Detail: architecture.md § `pipes/`. 14. **TypeScript SDK** — `@wavehouse/sdk`: zero-dep client, typed query builder, real-time SSE, live queries (incrementable/decomposable/poll aggregation), codegen CLI. The canonical client (see §SDK Sync). 15. **Observability invariants** — stdout always 100% (sampling is OTLP-push-only); WARN+ERROR always export at 100% (a non-configurable floor — don't expose it); gRPC OTel exporters dial lazily so an unreachable collector never blocks startup; the OTel Prometheus exporter uses a **private** `prometheus.Registry`. The OTLP endpoint/TLS/custom-CA/mTLS/headers are delegated to the OpenTelemetry SDK's standard `OTEL_EXPORTER_OTLP_*` env vars — `InitProvider` passes **no** endpoint/header options. Known gap, intentionally not patched in WaveHouse app code: the pinned gRPC logs exporter (`otlploggrpc` v0.19/v0.20) ignores the env TLS-cert vars, so a custom/private CA and mutual TLS apply to traces/metrics but **not** the logs signal (public-CA/system-roots TLS and plaintext still work for logs) — upstream bug open-telemetry/opentelemetry-go#6661. A malformed `OTEL_EXPORTER_OTLP_HEADERS` is logged and skipped by the SDK (fail-soft), not fatal. Preserve when touching the logger/sampler/provider. Detail: architecture.md § `observability/`. 16. **Bearer-token-only CORS posture (security)** — Bearer JWT on every request, no cookies/sessions; `corsMiddleware` deliberately **never** emits `Access-Control-Allow-Credentials` (not needed, and `*` + credentials is a spec violation browsers reject). `cors_allowed_origins` controls who can *read* responses, not cookie scope; CSRF protection is structural. Don't reintroduce cookie auth or `Allow-Credentials` without a design discussion — answers GitHub #29/#30. Code: `internal/api/router.go`. diff --git a/CHANGELOG.md b/CHANGELOG.md index 3db0d718..6fd47b40 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **Named pipes invalidate their cached results on writes to the tables they read, via versioned namespaces and a refresh-time cascade** (`internal/discovery/{discovery,explain}.go`, `internal/api/pipes.go`, `internal/pipes/pipes.go`, `internal/cache/{cache,local,version_manager}.go`, `internal/chsql/nats.go` (moved from `internal/query/ident.go`), `cmd/wavehouse/main.go`, `docs/src/content/docs/{pipes.mdx,architecture.md}`, plus unit + live-ClickHouse tests): closes [#178](https://github.com/Wave-RF/WaveHouse/issues/178). A pipe's table dependencies are resolved by **asking ClickHouse** — `EXPLAIN QUERY TREE` on the bound query the first time it runs (`discovery.ResolveTables`, reading the table list off ClickHouse's own analysis), cached per bound query (parameter combination). WaveHouse never parses pipe SQL itself and never rejects a pipe. `EXPLAIN QUERY TREE` is the pre-optimization tree, so it keeps a table even when the query is metadata-answered (`SELECT count() FROM t` still tracks `t`) or reads a currently-empty table — both of which `system.query_log` and the post-optimization `EXPLAIN PLAN` would drop. It also tracks reads that aren't plain `FROM` clauses: a `joinGet()` of a `Join`-engine table and a `dictGet()` of a dictionary (mapped via `system.dictionaries` to the ClickHouse table backing it). Table-name extraction is robust to `EXPLAIN`-rendering quirks the unit fixtures didn't originally cover: a `FINAL`/`SAMPLE` modifier appends fields after the name (`table_name: db.t, final: 1`) and a string literal can itself contain the text `table_name:` — neither leaks into, nor fabricates, a tracked table — and a `dictGet` against a not-yet-loaded dictionary force-loads it first, since ClickHouse reports an empty `source` for an unloaded dict that would otherwise drop its backing table. A build-tagged differential fuzzer (`internal/discovery/fuzz_deps_test.go`, run with `-tags fuzzdeps` against a local ClickHouse) cross-checks the resolved set for hundreds of nested / `UNION` / parameter-gated queries against a construction-based ground truth. Each resolved table is folded into the cache key as its **own versioned namespace** (a per-name lookup, no per-request graph walk), so a write to it misses the key. Views and materialized views are versioned namespaces too: the schema registry resolves each view's definition (also via EXPLAIN) **once per schema change** into a *cascade* (base table → the views reading it, transitively), pushed into the cache, so a write to a base table also bumps every view over it and evicts a pipe reading the view; a view redefined in place (and the views built on it) is evicted at the next refresh. A materialized view's `TO` target rides the same cascade: `CREATE MATERIALIZED VIEW mv TO target` makes `target` an ordinary base table that only inserts to the MV's source ever populate, so a source write also bumps the target — and chained targets, and the views over them — evicting a pipe that reads the target table directly (the target is parsed off the rendered `create_table_query`, since `system.tables` grew first-class target columns only in ClickHouse 26.6, and propagated along the `dependencies_table` trigger edges — a join or `dictGet` in the MV's definition never fires it). Resolution is **precise per parameter binding, and otherwise over-resolves — never under**: for a parameter-gated `UNION`, ClickHouse folds the bound predicate so a constant-false arm is pruned (`source=web` depends only on `web_events`, not `mobile_events`; resolved and cached per bound query, and emitting the `wavehouse_pipe_dep_tables_pruned_total` metric), while a query ClickHouse can't analyze (a write/DDL pipe, an unreachable server, a missing table) falls back to a single **database-wide version** that every write bumps, so any write evicts it — O(1) per request, and it holds even before the schema registry's first successful refresh. The pruning is conservative — an arm is dropped only when its filter is provably constant-false, so anything unproven stays tracked (over-resolve, never stale). Schema refresh is **atomic** (rebuilt only when content changes) and re-resolves pipes (`PipesHandler.ClearResolvedDeps`). Pipes run the author SQL as-is with no policy row/column filtering (the per-pipe `allowed_roles` gates execution), so every allowed caller shares one cached entry. **Unsupported, but bounded** (run, not blocked): write/ingest pipes fail EXPLAIN and take the database-version fallback (a parameter as a whole table name, `FROM {{tbl}}`, binds to a quoted string and errors at execution and resolution alike, so nothing caches — while a name merely assembled from a parameter, `FROM events_{{year}}`, resolves per binding like any bound query); an external read — a table function (`s3()`/`numbers()`, detected as a `TABLE_FUNCTION` node in the query tree), a cross-database table (direct or a `joinGet`/`dictGet` target), or a dictionary without a local backing table — can never be evicted by a write, so its result is TTL-capped (~10s) instead of cached at full TTL. Single-database (the configured `clickhouse.database`). The version-folded cache key is snapshotted **before** the query executes and reused for its `Set` (`Cache.Key`), so a write landing mid-query orphans the entry rather than re-homing the pre-write result under the post-bump key — neither cached read path (pipes or structured queries) can serve a result older than the write that should have evicted it. Degraded modes fail toward expiry, not staleness: a structured query on an unfoldable view and a dependency set that was dead-branch pruned are TTL-capped (`cache.UnresolvedDepsTTLCap` — the pruned cap is a belt-and-suspenders bound on a hypothetical wrong prune, with an integration test pinning that pruning still fires on the shipped ClickHouse version), and a transient EXPLAIN failure (an unreachable server, a canceled request) is retried on the pipe's next execution instead of memoized as permanent fallback. Schema refreshes are serialized end-to-end so overlapping refreshes can't install an older cascade over a newer one, and a refresh that lands mid-resolution wins over the in-flight result (generation-gated memo insert). The degraded modes are metered: `wavehouse_pipe_dep_resolution_fallback_total` and `wavehouse_pipe_dep_unresolved_total` count degraded *executions* (cache hits included — the share of pipe traffic on the fallback / TTL-capped), and `wavehouse_pipe_dep_memo_evictions_total` counts resolution-memo evictions past its fixed 50k cap (sustained growth = a high-cardinality pipe parameter). - **Non-JWT operator key for full-access admin + break-glass recovery** (`internal/auth/auth.go`, `internal/auth/context.go`, `internal/auth/auth_test.go`, `internal/api/router.go`, `internal/api/router_test.go`, `internal/config/config.go`, `internal/config/config_test.go`, `cmd/wavehouse/main.go`, `config.yaml`, `deployments/compose/standalone.yaml`, `tests/e2e/fixtures/config.yaml`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/access-control.mdx`, `docs/src/content/docs/api.md`, `docs/src/content/docs/deployment.md`, `docs/src/content/docs/development.md`, `docs/src/content/docs/architecture.md`, `docs/src/content/docs/reverse-proxy.mdx`, `AGENTS.md`, `SECURITY.md`): closes #240; partially advances the auth-hardening epic [#228](https://github.com/Wave-RF/WaveHouse/issues/228) and the management/data-plane split [#359](https://github.com/Wave-RF/WaveHouse/issues/359). Adds an optional `auth.operator_key` (`WH_AUTH_OPERATOR_KEY`): a request presenting it — via an `Authorization: Operator ` header (forwarded verbatim by proxies, no collision with Bearer JWTs) or the `X-Operator-Key` alias — is authorized as a full-access platform operator — the whole data plane *and* the `/v1/admin/*` management surface — without minting a JWT and independently of the token verifier, giving the person running the deployment a role-free credential for bootstrap and break-glass. The middleware checks it before the Bearer token with a constant-time comparison (`crypto/subtle`) and stamps two things into the request context: the live `admin_role` (so the policy evaluator's admin bypass grants unrestricted data-plane access while a policy exists) and a platform-operator bit that `RequireAdmin` honors **even when the policy is `nil`/deleted** — the one HTTP path that can restore a wiped policy, which previously required SSH access and a reboot. Empty (the default) disables it, so existing deployments are unchanged; treat it as an admin secret (load from a secret store, serve only over TLS). A successful operator authentication is audit-logged at `INFO`; a request presenting a *non-matching* operator key is logged at `WARN` and counted by a new `wavehouse_auth_operator_key_failures_total` counter (a probing/brute-force signal on the most privileged credential in the system) before falling through to the normal token/default path — the middleware still never rejects. The `Authorization` auth-scheme is matched case-insensitively (RFC 7235) via a shared `authScheme` helper, which also makes the existing `Bearer` JWT scheme case-insensitive (previously it required the canonical `Bearer` casing). Explicitly out of scope, tracked in #359: scoping the operator credential to the management surface only, and capability-scoped admin permissions. - **Missing-dedupe-id observability + optional strict mode** (`internal/api/ingest.go`, `internal/api/ingest_test.go`, `internal/config/config.go`, `cmd/wavehouse/main.go`, `config.yaml`, `deployments/compose/standalone.yaml`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/api.md`, `docs/src/content/docs/architecture.md`): closes #219. With dedupe enabled, a row missing the configured `id_field` can't be deduped — previously it was published with idempotency silently disabled and *no* log or metric, so a producer bug that dropped the id turned off the guarantee for those rows unnoticed. Now every such row is logged at `WARN` and counted by a new `wavehouse_ingest_dedupe_missing_id_total` counter (labeled by `table`), making the loss observable server-side. A new opt-in `dedupe.require_id` (`WH_DEDUPE_REQUIRE_ID`, default `false`) turns that signal into enforcement: a row missing the id is rejected (`400` for a single insert; a per-record failure in a batch) instead of published — a tripwire for producers that must guarantee the id (complements the client-side [#202](https://github.com/Wave-RF/WaveHouse/issues/202)). Default behavior is unchanged. - **"Durability & Storage" operations guide** (`docs/src/content/docs/durability.md` (new), `docs/src/config/sidebar.ts`, `docs/src/content/docs/reverse-proxy.mdx`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/deployment.md`): documents #84. A new Operations page making the embedded-JetStream durability contract explicit before the docs site publishes: a `200` from `POST /v1/ingest` means the event has been `fsync`'d to disk on the node (the server runs with `SyncAlways: true` in `internal/mq/embedded.go`), which makes the storage substrate's `fsync` tail the ingest latency floor. Covers the contract (and how it differs from JetStream's default page-cache-then-periodic-sync mode), why a slow `fsync` tail manifests as `create stream: ... context deadline exceeded` and `503` backpressure, a where-it's-cheap-vs-expensive substrate table (managed cloud block storage and PLP NVMe vs. ZFS-without-SLOG / qcow2-on-`ext4` / spinning disks), an `fio` recipe + verdict bands to measure your own storage (with the macOS `F_FULLFSYNC` honesty caveat), and the symptom checklist. Forward-references the configurable group-commit interval (`mq.sync_interval`, [#139](https://github.com/Wave-RF/WaveHouse/issues/139)) and the planned `wavehouse storage-check` preflight ([#84](https://github.com/Wave-RF/WaveHouse/issues/84)) without claiming either exists yet. Cross-linked from Configuration (Message Queue), Deployment (Persistent Storage), and the Ingest Pipeline's worker-side ack section; no code changes. diff --git a/cmd/wavehouse/main.go b/cmd/wavehouse/main.go index c75eb853..d88f44ac 100644 --- a/cmd/wavehouse/main.go +++ b/cmd/wavehouse/main.go @@ -183,37 +183,25 @@ func run() int { ctx, cancel := context.WithCancel(context.Background()) defer cancel() - // Schema discovery — non-fatal on boot. If the first Refresh fails - // (ClickHouse unreachable, database missing, etc.) we mark the binary - // degraded via bootState (which /livez surfaces as 503 + diagnostic) - // and retry in the background with exponential backoff. The process - // still binds :8080 so operators can `curl /livez` instead of grepping - // a restart-loop log. Once a Refresh succeeds, bootState flips to nil - // and /livez returns 200. The periodic auto-refresh ticker is started - // only after the first successful Refresh (sync or retry) so it never - // races RetryRefresh on Refresh calls or on bootState writes. + // Schema discovery registry. The first Refresh (and the auto-refresh loop) is + // deliberately started much further down, only after every handler it can call + // back into exists — see the "Schema discovery" block below the handler wiring. bootState := api.NewBootState(nil) refreshInterval := time.Duration(cfg.Schema.RefreshInterval) * time.Second registry := discovery.NewSchemaRegistry(chConn, cfg.ClickHouse.Database, refreshInterval, logger) - if err := registry.Refresh(ctx); err != nil { - logger.Warn("schema discovery failed on boot, retrying in background", "error", err) - bootState.Set(fmt.Errorf("schema discovery: %w", err)) - go func() { - retryErr := registry.RetryRefresh(ctx, 2*time.Second, 60*time.Second, func(attemptErr error) { - logger.Warn("schema discovery retry failed", "error", attemptErr) - bootState.Set(fmt.Errorf("schema discovery: %w", attemptErr)) - }) - if retryErr != nil { - // ctx cancelled before success — process is shutting down. - return - } - logger.Info("schema discovery succeeded after retry, /livez now 200") - bootState.Set(nil) - go registry.StartAutoRefresh(ctx) - }() - } else { - go registry.StartAutoRefresh(ctx) + + // TODO: this is where we can switch between ristretto, redis, tiered (both), etc. + l1, err := cache.NewLocal(cfg.Cache.L1MaxCost) + if err != nil { + logger.Error("cache init", "error", err) + return 1 } + defer func() { _ = l1.Close() }() + // Everything downstream takes the Cache INTERFACE, not the concrete L1, so the + // eventual L2/tiered swap happens here and nowhere else. (Named resultCache + // rather than the historical `cache := l1` because a variable named cache would + // shadow the cache package this function still needs.) + var resultCache cache.Cache = l1 // State directories — one configurable root, fixed subdir convention. natsDir := filepath.Join(cfg.DataDir, "nats") @@ -259,16 +247,6 @@ func run() int { } } - // L1 cache only in standalone mode. - l1, err := cache.NewLocal(cfg.Cache.L1MaxCost) - if err != nil { - logger.Error("cache init", "error", err) - return 1 - } - // TODO: eventually this is where we can switch between ristretto, redis, tiered (both), etc - cache := l1 - defer func() { _ = cache.Close() }() - // Policy store (NATS KV + optional file bootstrap). policyStore, err := policy.NewStore(ctx, embeddedMQ.JetStream(), cfg.Policy.FilePath, logger) if err != nil { @@ -301,7 +279,7 @@ func run() int { ingestCleanup, err := ingest.StartIngestWorker( ctx, embeddedMQ.NatsConn(), - cache, + resultCache, cfg.ClickHouse.Addr, cfg.ClickHouse.HTTPPort, // Uses 8123 by default cfg.ClickHouse.HTTPScheme, @@ -385,6 +363,52 @@ func run() int { return 1 } + pipesHandler := api.NewPipesHandler(pipesStore, policyStore, chConn, resultCache, registry, cfg.ClickHouse.QueryTimeout, logger) + + structuredQueryHandler := api.NewStructuredQueryHandler(chConn, resultCache, registry, policyStore, cfg.Cache.TimestampBucketSeconds, cfg.ClickHouse.QueryTimeout, cfg.Query.DefaultMaxRows, logger) + + // Schema discovery — started HERE, after every handler the refresh callback + // reaches into exists, so the callback needs no nil checks and no atomics: the + // handler writes above happen-before the refresh goroutines are created. It is + // non-fatal on boot: if the first Refresh fails (ClickHouse unreachable, + // database missing, etc.) we mark the binary degraded via bootState (which + // /livez surfaces as 503 + diagnostic) and retry in the background with + // exponential backoff; the process still binds :8080 so operators can + // `curl /livez` instead of grepping a restart-loop log. The periodic + // auto-refresh ticker is started only after the first successful Refresh + // (sync or retry) so it never races RetryRefresh on Refresh calls or on + // bootState writes. + registry.SetOnRefresh(func(snap discovery.DependencySnapshot) { + resultCache.SetDependents(snap.Cascade) + if len(snap.ChangedViews) > 0 { + nss := make([]cache.Namespace, len(snap.ChangedViews)) + for i, v := range snap.ChangedViews { + nss[i] = cache.Namespace{Table: v} + } + _, _ = resultCache.Invalidate(ctx, nss) + } + pipesHandler.ClearResolvedDeps() + }) + if err := registry.Refresh(ctx); err != nil { + logger.Warn("schema discovery failed on boot, retrying in background", "error", err) + bootState.Set(fmt.Errorf("schema discovery: %w", err)) + go func() { + retryErr := registry.RetryRefresh(ctx, 2*time.Second, 60*time.Second, func(attemptErr error) { + logger.Warn("schema discovery retry failed", "error", attemptErr) + bootState.Set(fmt.Errorf("schema discovery: %w", attemptErr)) + }) + if retryErr != nil { + // ctx cancelled before success — process is shutting down. + return + } + logger.Info("schema discovery succeeded after retry, /livez now 200") + bootState.Set(nil) + go registry.StartAutoRefresh(ctx) + }() + } else { + go registry.StartAutoRefresh(ctx) + } + deps := api.Dependencies{ Ingest: ingestHandler, Query: queryHandler, @@ -394,8 +418,8 @@ func run() int { Schema: api.NewSchemaHandler(registry), DLQ: dlqHandler, Policy: api.NewPolicyHandler(policyStore), - Pipes: api.NewPipesHandler(pipesStore, policyStore, chConn, cache, cfg.ClickHouse.QueryTimeout, logger), - StructuredQuery: api.NewStructuredQueryHandler(chConn, cache, registry, policyStore, cfg.Cache.TimestampBucketSeconds, cfg.ClickHouse.QueryTimeout, cfg.Query.DefaultMaxRows, logger), + Pipes: pipesHandler, + StructuredQuery: structuredQueryHandler, AuthMW: authMW, PolicyStore: policyStore, Logger: logger, diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index 3302b5af..b86a3fa3 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -477,7 +477,7 @@ The inbound request body is capped at 1 MiB; a body over the cap is rejected wit ### `GET/POST /v1/pipes/{name}` — Execute Named Pipe -Executes a pre-defined named query (pipe) with parameter binding. Parameters can be supplied via query string and/or JSON body. Results are cached in the shared L1 (Ristretto) with singleflight coalescing — same machinery as the structured query endpoint, and again, unlike `/v1/admin/query`. +Executes a pre-defined named query (pipe) with parameter binding. Parameters can be supplied via query string and/or JSON body. Results are cached in the shared L1 (Ristretto) with singleflight coalescing — same machinery as the structured query endpoint, and again, unlike `/v1/admin/query`. Cached results are **invalidated by writes to the tables the pipe reads** — see [Named Pipes](/pipes) for the resolution and invalidation semantics, including the cases that fall back to TTL-only behavior. **Query Parameters:** Any key matching a pipe parameter name. diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 4c6a8958..88f7ac91 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -52,7 +52,7 @@ internal/ ├── api/ HTTP layer (Chi router, handlers, middleware) ├── auth/ JWT/JWKS authentication middleware (HMAC or JWKS, role extraction) ├── cache/ In-process Ristretto cache with singleflight coalescing -├── chsql/ Shared ClickHouse SQL helpers (identifier quoting, bind-safety) +├── chsql/ Shared ClickHouse SQL helpers (identifier quoting, bind-safety, NATS-safe name encoding) ├── config/ YAML + env var configuration loading ├── dedupe/ Optional deduplication (Pebble) ├── discovery/ ClickHouse schema introspection and validation @@ -113,13 +113,14 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ ### `discovery/` — Schema Discovery & Validation -- **discovery.go** — `SchemaRegistry` queries `system.columns` to discover ClickHouse table schemas. Supports periodic auto-refresh, on-demand refresh, and `RetryRefresh` (boot-time exponential backoff loop used by `cmd/wavehouse` so a transiently unreachable ClickHouse doesn't crash-loop the binary). Thread-safe via `sync.RWMutex`. +- **discovery.go** — `SchemaRegistry` queries `system.columns` to discover ClickHouse table schemas. Supports periodic auto-refresh, on-demand refresh, and `RetryRefresh` (boot-time exponential backoff loop used by `cmd/wavehouse` so a transiently unreachable ClickHouse doesn't crash-loop the binary). The registry also owns the cache-invalidation dependency graph: on each content-changed refresh it resolves every view and materialized-view definition (via `ResolveTables`, below) into a base-table→dependents cascade — carrying an MV's trigger edge through to its `TO` target, parsed off the rendered `create_table_query` (`parseMVTarget`) and propagated along `system.tables.dependencies_table` — exposed as `Dependents()` and pushed into the query cache via `SetOnRefresh`; `IsKnown` reports whether a name resolved cleanly into that cascade (the pipe handler's test for a dependency version invalidation can be trusted to watch — see the `pipes/` section for the full invalidation model). Thread-safe via `sync.RWMutex`. +- **explain.go** — `ResolveTables(ctx, conn, database, sql)` asks ClickHouse itself what a query reads: it runs `EXPLAIN QUERY TREE` (deliberately the pre-optimization tree, so it never under-resolves) and parses the table nodes into a `Resolution` — the local table/view names, with `joinGet`/`dictGet` targets tracked too (a dictionary via its backing source table) — plus the two degraded-mode flags the pipe handler TTL-caps on: `Pruned` (dead-branch pruning dropped a parameter-gated `UNION` arm's tables) and `External` (a table function, cross-database table, or non-table dictionary source — a read no table version can watch). - **validation.go** — `Validate(schema, data)` checks incoming JSON against the discovered schema: unknown fields, type compatibility, missing required columns, null handling. - **discovery_test.go** — Unit tests for validation logic. ### `ingest/` — Ingest Pipeline, DLQ & Sweeping -- **worker.go** — `StartIngestWorker` launches an ingest pipeline: a JetStream consumer reads from the `WAVEHOUSE` stream via a durable `buffer-consumer` pull subscription, batches events per table, and performs bulk INSERTs to ClickHouse. The pipeline is **insert-only**. The wire format `EventMessage` carries `{table_name, received_timestamp, data}` and nothing else; the worker accepts any table name now (the table name in the NATS subject is `query.SafeEncodeNATS(rawUnsafeTableName)`), then bulk-INSERTs. The embedded NATS server runs with `DontListen: true` (`internal/mq/embedded.go`), so the only Publishers reachable on the `ingest.>` subjects are in-process Go code — today, only the HTTP `/v1/ingest?table={table}` handler. Non-insert mutations (`DELETE`/`UPDATE`/`TRUNCATE`/…) must go through `POST /v1/admin/query` under the admin role (`policy.admin_role`) — see the Query Path section below; the `/v1/admin/*` `RequireAdmin` middleware enforces the check at the API layer, so a no/invalid-token request (resolved to `default_role`, not admin in a production config) never reaches the proxy. Failed batches are routed to the DLQ (`sendToDLQ`), which republishes the original `EventMessage` envelope to `dlq.{table}` NATS subjects with the failure context in `X-DLQ-*` headers when DLQ is enabled — see [Ingest Pipeline](/ingest-pipeline) for the worker internals. +- **worker.go** — `StartIngestWorker` launches an ingest pipeline: a JetStream consumer reads from the `WAVEHOUSE` stream via a durable `buffer-consumer` pull subscription, batches events per table, and performs bulk INSERTs to ClickHouse. The pipeline is **insert-only**. The wire format `EventMessage` carries `{table_name, received_timestamp, data}` and nothing else; the worker accepts any table name now (the table name in the NATS subject is `chsql.SafeEncodeNATS(rawUnsafeTableName)`), then bulk-INSERTs. The embedded NATS server runs with `DontListen: true` (`internal/mq/embedded.go`), so the only Publishers reachable on the `ingest.>` subjects are in-process Go code — today, only the HTTP `/v1/ingest?table={table}` handler. Non-insert mutations (`DELETE`/`UPDATE`/`TRUNCATE`/…) must go through `POST /v1/admin/query` under the admin role (`policy.admin_role`) — see the Query Path section below; the `/v1/admin/*` `RequireAdmin` middleware enforces the check at the API layer, so a no/invalid-token request (resolved to `default_role`, not admin in a production config) never reaches the proxy. Failed batches are routed to the DLQ (`sendToDLQ`), which republishes the original `EventMessage` envelope to `dlq.{table}` NATS subjects with the failure context in `X-DLQ-*` headers when DLQ is enabled — see [Ingest Pipeline](/ingest-pipeline) for the worker internals. - **types.go** — `EventMessage` struct (TableName, ReceivedTimestamp, Data) and `BufferConsumerName` constant, shared across API handlers and the ingest pipeline. - **sweeper.go** — `Sweeper` implements the Active Sweeper pattern. It runs every minute and purges NATS JetStream messages that are **both** ACKed by the buffer consumer (written to ClickHouse) **and** older than the configurable gap window. @@ -144,7 +145,7 @@ The package's design invariants — stdout always 100%, WARN+ERROR always export ### `pipes/` — Named Query Pipes -- **pipes.go** — `NamedQuery` type with SQL template and parameter definitions, `Store` backed by NATS KV bucket `WAVEHOUSE_PIPES`. Supports `.sql` file directory bootstrap. `BindParams()` resolves `{{param}}` / `{{param:default}}` placeholders by inlining escaped literal values into the SQL (strings single-quote-escaped; arrays rendered as escaped `(…)` `IN`-lists). A non-scalar value with no SQL form (a JSON object, or an empty array) is rejected rather than emitted raw. +- **pipes.go** — `NamedQuery` type with SQL template and parameter definitions, `Store` backed by NATS KV bucket `WAVEHOUSE_PIPES`. Supports `.sql` file directory bootstrap. `BindParams()` resolves `{{param}}` / `{{param:default}}` placeholders by inlining escaped literal values into the SQL (strings single-quote-escaped; arrays rendered as escaped `(…)` `IN`-lists). A non-scalar value with no SQL form (a JSON object, or an empty array) is rejected rather than emitted raw. The Store does **no** SQL analysis and rejects nothing — a pipe's table dependencies are resolved by asking ClickHouse: `api.PipesHandler.resolveDeps` runs `EXPLAIN QUERY TREE` on the bound query (the first time the pipe runs) and reads the table list off ClickHouse's own analysis, cached per bound query (parameter combination). Each resolved table is folded into the cache key as its own versioned namespace; a query ClickHouse can't analyze (a write/DDL pipe, an unreachable server) falls back to the single database-version namespace (`cache.DatabaseNamespace`, bumped by every `Invalidate`) so any write evicts it — never an under-resolution, and O(1) per request. A resolved dependency whose version the cascade can't reliably maintain — an unfoldable view (unparsed definition, or a cross-database/unknown source), per `SchemaRegistry.IsKnown` — is instead TTL-capped (`cache.UnresolvedDepsTTLCap`, applied at the pipe `Set`) rather than trusted to version invalidation; a dependency set that was dead-branch pruned gets the same cap as a belt-and-suspenders bound on a hypothetical wrong prune. `EXPLAIN QUERY TREE` is the pre-optimization tree, so it keeps a table even when the query is metadata-answered (`count()`) and keeps every parameter-gated `UNION` branch, both of which `system.query_log` would drop. Views and materialized views are versioned namespaces too: the schema registry resolves each view's definition (also via `discovery.ResolveTables`/EXPLAIN) once per refresh into a base-table→dependents *cascade* (`SchemaRegistry.Dependents`, pushed into the cache via `SetOnRefresh`), so a write to a base table also bumps every view over it and evicts a pipe reading that view. The cascade also carries a write *through* a materialized view into its `TO` target — an ordinary base table only the MV's trigger ever populates, so a pipe reading the target directly evicts on source writes too: the target is parsed off the rendered `create_table_query` (`discovery.parseMVTarget` — `system.tables` grew first-class target columns only in ClickHouse 26.6, too new to require) and propagated along the `system.tables.dependencies_table` trigger edges (a join or `dictGet` in the MV's definition never fires it), transitively through chained MVs and the views over each target. `PipesHandler.ClearResolvedDeps` re-resolves pipes on each refresh. Four counters make the degraded modes observable: `wavehouse_pipe_dep_resolution_fallback_total` and `wavehouse_pipe_dep_unresolved_total` meter *executions* (cache hits included — the share of pipe traffic on the database-version fallback / TTL-capped), `wavehouse_pipe_dep_tables_pruned_total` meters dead-branch pruning (a rate that goes flat after a ClickHouse upgrade means the rendering coupling stopped matching — failing safe), and `wavehouse_pipe_dep_memo_evictions_total` meters evictions past the resolution memo's fixed cap (`maxPipeDeps`, 50,000 bound queries — sustained growth means a high-cardinality pipe parameter re-paying the EXPLAIN round-trip). An external read — a table function (`s3()`), a cross-database table, a non-local dictionary source — is detected off the query tree (`discovery.Resolution.External`) and TTL-capped the same way, since no table version can watch it; parameterized table names remain unsupported (they fail EXPLAIN and take the database-version fallback). The pipe handler applies no policy row/column filtering to the author SQL (see pipes docs). ### `query/` — Structured Query Engine @@ -153,7 +154,8 @@ The package's design invariants — stdout always 100%, WARN+ERROR always export ### `chsql/` — ClickHouse SQL Helpers -- **chsql.go** — Dependency-free ClickHouse SQL helpers shared by `query/` and `policy/`, kept in their own package to break an import cycle. `QuoteIdent` is the single place every identifier — column, table, alias — becomes SQL text: always backtick-quoted and escaped, so any ClickHouse-legal name (dots, spaces, unicode, keywords) is safe. `BindUnsafe` reports whether a name contains a literal `?`, which would desync clickhouse-go's positional binder; such names are rejected fail-closed rather than silently mis-bound. +- **chsql.go** — Dependency-free ClickHouse SQL helpers shared by `query/` and `policy/`, kept in their own package to break an import cycle. `QuoteIdent` is the single place every identifier — column, table, alias — becomes SQL text: always backtick-quoted and escaped, so any ClickHouse-legal name (dots, spaces, unicode, keywords) is safe. `BindUnsafe` reports whether a name contains a literal `?`, which would desync clickhouse-go's positional binder; such names are rejected fail-closed rather than silently mis-bound. `QuoteString` escapes a string literal (backslash and single-quote doubling) — the one escaper behind pipe parameter inlining. +- **nats.go** — `SafeEncodeNATS`/`SafeDecodeNATS`, the single table-name → NATS-safe-token encoder shared by every side of cache invalidation: the ingest write side (subjects and `Invalidate` namespaces), the pipe/structured-query read side (dependency namespaces folded into cache keys), and the registry's base→dependents cascade. Alphanumerics and `_` pass through; everything else percent-encodes, so any ClickHouse-legal name round-trips and both sides always build identical namespace keys. ## Data Flows diff --git a/docs/src/content/docs/pipes.mdx b/docs/src/content/docs/pipes.mdx index dc9f262c..a6119724 100644 --- a/docs/src/content/docs/pipes.mdx +++ b/docs/src/content/docs/pipes.mdx @@ -119,7 +119,7 @@ Pipe execution is gated by `allowed_roles` — a plain allowlist, evaluated inde To make a pipe public, list the role that `default_role` resolves to. For example, with `default_role: viewer`, a pipe with `"allowed_roles": ["viewer"]` is reachable with no token at all. -This is the *only* authorization check on the execute path — pipes deliberately sit outside the `/v1/admin/*` gate so non-admin callers can run them. The pipe's SQL is not re-checked against the table policy's column or row rules, so **scope the data in the pipe's SQL itself** (or via a claim-templated predicate) rather than assuming policy column masking applies. +This is the *only* authorization check on the execute path — pipes deliberately sit outside the `/v1/admin/*` gate so non-admin callers can run them. The pipe's SQL is **not** re-checked against the table policy's column or row rules — neither the projection nor the rows are filtered by policy — so **scope the data in the pipe's SQL itself** (and in any views it reads) rather than assuming policy column masking or row security applies. Row security can't be applied reliably to opaque SQL that may read arbitrary views (a view over a table without the filter column can't be filtered, and would fail silently), so it is deliberately left to the pipe author. ## Creating and managing pipes @@ -171,6 +171,25 @@ curl -X POST http://localhost:8080/v1/pipes/top_pages \ The response is a JSON array of rows. Results flow through the shared in-process L1 cache (Ristretto) with singleflight coalescing, so concurrent identical calls hit ClickHouse once; an `X-Cache: HIT` or `X-Cache: MISS` header tells you which path served the response. +A cached pipe result is keyed by its SQL and parameter values (every allowed caller shares one entry — pipes apply no per-role filtering), and is **invalidated by writes to the tables it reads**. WaveHouse never parses your SQL to find those tables — it **asks ClickHouse**, running `EXPLAIN QUERY TREE` the first time a pipe runs with a given set of parameter values and reading the table list off ClickHouse's own analysis. That result is cached and reused for the same parameters, so the extra round-trip happens once per parameter combination. Working off the pre-optimization query tree, it captures a table even when the query is answered from metadata (`SELECT count() FROM t` still tracks `t`) or reads a currently-empty table — neither of which a runtime signal like `system.query_log` or the post-optimization `EXPLAIN PLAN` would report. It also tracks reads that aren't plain `FROM` clauses: a `joinGet()` of a `Join`-engine table, and a `dictGet()` of a dictionary (mapped to the ClickHouse table backing it). One side effect worth knowing: resolving a `dictGet()` pipe whose dictionary hasn't been loaded yet (ClickHouse lazy-loads them by default) issues a `SYSTEM RELOAD DICTIONARY`, so the pipe's *first* execution can load the whole dictionary into ClickHouse memory — for a large dictionary that's a real, one-time cost triggered by "just running a pipe". + +Each table EXPLAIN reports carries its **own cache version**, folded into the key — a write to it evicts the result. Views and materialized views are versioned just like base tables. The link between a view and its base is precomputed: the schema registry resolves each view's definition (again via EXPLAIN) **once when the schema is discovered** into a *cascade* — each base table mapped to the views that read it, recursively through chained views — so a write to a base table also bumps every dependent view's version, evicting a pipe reading the view. (For a materialized view, the source table is what ingest bumps and from which ClickHouse maintains the view.) The cascade also carries a write **through** a materialized view into its `TO` target: `CREATE MATERIALIZED VIEW mv TO target` makes `target` an ordinary table that only inserts to the MV's source ever populate, so a source write bumps `target` too — and chained targets, and the views over them — evicting a pipe that reads the target table directly, not just one that reads the MV. A view redefined in place (same sources, new body) is evicted on the next schema refresh, along with the views built on top of it. A view WaveHouse **can't** fold down to base tables (an unparseable definition, or a cross-database source) never enters the cascade, so it can't be evicted on write; a pipe reading one is instead capped to a **short TTL (~10 seconds)** and self-expires — bounded freshness lag rather than staleness. + +Resolution is **precise per parameter binding, and where it can't be precise it over-resolves — never under**. For a parameter-gated `UNION`, ClickHouse folds the bound predicate: with `?source=web`, the `… UNION ALL … WHERE {{source}} = 'mobile'` arm becomes constant-false and reads nothing, so `mobile_events` is dropped from the dependency set — a write to it does **not** evict the `source=web` result (that arm can't read `mobile_events` regardless of its data, so this is precise, not stale). The `source=mobile` result is resolved and cached separately, depending only on `mobile_events`. The pruning is deliberately conservative: it drops an arm only when ClickHouse proves its filter constant-false, so anything it can't prove dead stays tracked — and as a further safety bound, a result whose dependency set *was* pruned also gets the short-TTL cap, so even a hypothetical wrong prune is limited to seconds of staleness, not a full cache lifetime. And if WaveHouse can't get an answer from ClickHouse at all — the query references a table that doesn't exist yet, or ClickHouse is briefly unreachable — the pipe falls back to a **database-wide version that every write bumps**, so any write at all evicts it: again never stale, just coarse until it re-resolves. + +:::caution[Pipes WaveHouse can't precisely track] +Pipes are stored and run **as-is** — WaveHouse never rejects one. But some pipes can't be precisely cache-tracked; WaveHouse bounds the damage (a short TTL, or coarse any-write eviction) instead of blocking them: + +- **Writes / DDL** (`INSERT`, `CREATE`, `DROP`, …). A pipe is meant to be a read. A write pipe runs but doesn't go through the ingest path, and its result takes the database-version fallback (any write evicts it) — not a supported pattern ([#386](https://github.com/Wave-RF/WaveHouse/issues/386) tracks that a mutation pipe's result shouldn't be cached or coalesced at all). +- **Table functions** (`s3()`, `numbers()`, `url()`, …). Their data lives outside ClickHouse's tables, so there's no table to version and a change to the external source can never evict the result. WaveHouse detects the read (a table function is its own node in the query tree) and **caps the cached result at the short TTL (~10 seconds)**, so the staleness window is bounded; the pipe's regular tables are still tracked and evict as usual. The same cap applies to a cross-database table (referenced directly or as a `joinGet`/`dictGet` target) and to a dictionary whose source isn't a local table — [#404](https://github.com/Wave-RF/WaveHouse/issues/404) tracks first-class multi-database support. +- **A parameter as the whole table name** (`FROM {{tbl}}`). Parameter values are inlined as literals, so this binds to a quoted string where an identifier belongs — the query fails at execution just as it does at resolution, and nothing is cached. (A name merely *assembled from* a parameter, like `FROM events_{{year}}`, is fine: resolution runs on the bound SQL, so each binding tracks the table it actually reads.) +- **Refreshable materialized views** (`CREATE MATERIALIZED VIEW … REFRESH EVERY …`). A scheduled refresh rewrites the view's target on ClickHouse's own timer — not on any write WaveHouse observes, so nothing evicts on the tick (the same holds for any data changed directly in ClickHouse, behind WaveHouse's back). A result reading the view or its target can be stale for up to its TTL after a refresh tick; ordinary (insert-triggered) materialized views and their `TO` targets are fully tracked. + +**Watching the degraded modes.** Four counters make all of this observable. `wavehouse_pipe_dep_resolution_fallback_total` counts executions served on the database-version fallback, and `wavehouse_pipe_dep_unresolved_total` counts executions whose result was TTL-capped — both meter *executions* (cache hits included), so each reads as the share of pipe traffic running degraded; a sustained rate means some pipe deserves a look. `wavehouse_pipe_dep_tables_pruned_total` counts dependencies dropped by dead-branch pruning — a rate that goes flat right after a ClickHouse upgrade means pruning stopped matching the new `EXPLAIN` rendering (safe, but the precision is gone). And `wavehouse_pipe_dep_memo_evictions_total` counts resolved-dependency memo entries evicted past the memo's fixed cap (50,000 bound queries) — sustained growth means a high-cardinality parameter is churning the memo, paying the `EXPLAIN` round-trip over and over. + +Pipe dependencies re-resolve on a **schema refresh** (`POST /v1/schema/refresh`, plus the periodic tick), so after you create/edit a pipe or change the schema, a refresh re-pins everything against the current ClickHouse state. +::: + | Status | Body | Cause | | ------ | ---- | ----- | | 404 | `{"error":"pipe not found"}` | No pipe registered under that name | diff --git a/internal/api/boot_chain_test.go b/internal/api/boot_chain_test.go index 4f5b22ad..1871d7bd 100644 --- a/internal/api/boot_chain_test.go +++ b/internal/api/boot_chain_test.go @@ -112,9 +112,8 @@ func TestBoot_Chain_DegradedThenRecovers(t *testing.T) { handler.Liveness(rec, httptest.NewRequestWithContext(context.Background(), http.MethodGet, "/livez", nil)) assert.Equal(t, http.StatusOK, rec.Code) assert.Contains(t, rec.Body.String(), `"status":"ok"`) - - // Sanity: exactly four Query calls — the initial sync Refresh (1) plus - // the retry loop's three attempts (2,3 fail; 4 succeeds). An off-by-one - // in the loop's success short-circuit would show up here. - assert.Equal(t, int32(4), conn.calls.Load(), "expected 4 Query calls total") + // 5 = the three failed attempts (each short-circuits at Refresh's first query, + // system.columns) + the successful 4th (system.columns + system.tables). Pins + // the retry loop against silent extra attempts or a reshaped discovery pass. + assert.Equal(t, int32(5), conn.calls.Load(), "expected 5 Query calls total") } diff --git a/internal/api/dlq.go b/internal/api/dlq.go index 5d57c91b..323e7e37 100644 --- a/internal/api/dlq.go +++ b/internal/api/dlq.go @@ -7,8 +7,8 @@ import ( "net/http" "strings" + "github.com/Wave-RF/WaveHouse/internal/chsql" "github.com/Wave-RF/WaveHouse/internal/mq" - "github.com/Wave-RF/WaveHouse/internal/query" "github.com/nats-io/nats.go/jetstream" ) @@ -37,7 +37,7 @@ func (h *DLQHandler) Stats(w http.ResponseWriter, r *http.Request) { subjectFilter := ">" tableFilter := r.URL.Query().Get("table") if tableFilter != "" { - subjectFilter = "dlq." + query.SafeEncodeNATS(tableFilter) + subjectFilter = "dlq." + chsql.SafeEncodeNATS(tableFilter) } info, err := stream.Info(r.Context(), jetstream.WithSubjectFilter(subjectFilter)) @@ -49,7 +49,7 @@ func (h *DLQHandler) Stats(w http.ResponseWriter, r *http.Request) { tables := make(map[string]uint64) for subject, count := range info.State.Subjects { // TODO: do we need to break out scopes here? - decodedSubject, err := query.SafeDecodeNATS(strings.TrimPrefix(subject, "dlq.")) + decodedSubject, err := chsql.SafeDecodeNATS(strings.TrimPrefix(subject, "dlq.")) if err != nil { continue } diff --git a/internal/api/errors_test.go b/internal/api/errors_test.go index 358a3f34..a7a35123 100644 --- a/internal/api/errors_test.go +++ b/internal/api/errors_test.go @@ -137,7 +137,7 @@ func TestPipesHandler_Execute_DenialLogsAllowedRoles(t *testing.T) { store := pipes.NewMemoryStore( &pipes.NamedQuery{Name: "report", SQL: "SELECT * FROM clicks", AllowedRoles: []string{"analyst", "viewer"}}, ) - h := NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), nil, nil, 0, logger) + h := NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), nil, nil, nil, 0, logger) r := pipesRequest(t, http.MethodPost, "/v1/pipes/report/execute", "report", nil) r = r.WithContext(auth.WithRole(r.Context(), "guest")) diff --git a/internal/api/ingest.go b/internal/api/ingest.go index f283c59a..ace8d696 100644 --- a/internal/api/ingest.go +++ b/internal/api/ingest.go @@ -12,12 +12,12 @@ import ( "time" "github.com/Wave-RF/WaveHouse/internal/auth" + "github.com/Wave-RF/WaveHouse/internal/chsql" "github.com/Wave-RF/WaveHouse/internal/dedupe" "github.com/Wave-RF/WaveHouse/internal/discovery" "github.com/Wave-RF/WaveHouse/internal/ingest" "github.com/Wave-RF/WaveHouse/internal/mq" "github.com/Wave-RF/WaveHouse/internal/policy" - "github.com/Wave-RF/WaveHouse/internal/query" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" @@ -441,9 +441,9 @@ func (h *IngestHandler) processRecord( return false, nil, &requestAbort{Status: http.StatusInternalServerError, Message: "marshal failed"} } - subject := "ingest." + query.SafeEncodeNATS(table) + subject := "ingest." + chsql.SafeEncodeNATS(table) if scope != "" { - subject += "." + query.SafeEncodeNATS(scope) + subject += "." + chsql.SafeEncodeNATS(scope) } h.logger.DebugContext(ctx, "publishing event to NATS", "subject", subject, "table", table, "scope", scope) diff --git a/internal/api/pipe_deps_test.go b/internal/api/pipe_deps_test.go new file mode 100644 index 00000000..cb6a4412 --- /dev/null +++ b/internal/api/pipe_deps_test.go @@ -0,0 +1,389 @@ +package api + +import ( + "context" + "errors" + "fmt" + "net/http" + "net/http/httptest" + "sort" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/ClickHouse/clickhouse-go/v2" + "github.com/ClickHouse/clickhouse-go/v2/lib/driver" + "github.com/Wave-RF/WaveHouse/internal/auth" + "github.com/Wave-RF/WaveHouse/internal/cache" + "github.com/Wave-RF/WaveHouse/internal/chsql" + "github.com/Wave-RF/WaveHouse/internal/discovery" + "github.com/Wave-RF/WaveHouse/internal/pipes" + "github.com/Wave-RF/WaveHouse/internal/policy" + "github.com/Wave-RF/WaveHouse/internal/testutil" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func pipeDepNames(deps []cache.Namespace) []string { + out := make([]string, len(deps)) + for i, d := range deps { + out[i], _ = chsql.SafeDecodeNATS(d.Table) + } + sort.Strings(out) + return out +} + +// resolveDeps asks ClickHouse (EXPLAIN QUERY TREE) which tables a pipe reads — +// mocked here via resolveTablesFn so the unit test needs no server. It covers the +// fold, the per-pipe memo (EXPLAIN runs once), the database-version fallback when +// ClickHouse can't analyze the query, the pruned TTL cap, and ClearResolvedDeps +// re-resolution. +func TestResolveDeps_Explain(t *testing.T) { + ctx := context.Background() + reg := discovery.NewSchemaRegistryForTest( + []*discovery.TableSchema{{Name: "events"}, {Name: "users"}, {Name: "orders"}}, nil) + lc, _ := cache.NewLocal(1 << 20) + defer func() { _ = lc.Close() }() + + calls := 0 + h := &PipesHandler{registry: reg, Cache: lc, logger: testutil.NopLogger(), pipeDeps: map[string]*resolvedDeps{}} + h.resolveTablesFn = func(_ context.Context, sql string) (discovery.Resolution, error) { + calls++ + switch sql { + case "SELECT * FROM events": + return discovery.Resolution{Tables: []string{"events"}}, nil + case "SELECT * FROM events JOIN users ON 1": + return discovery.Resolution{Tables: []string{"events", "users"}}, nil + case "INSERT INTO events VALUES (1)": // a write → EXPLAIN errors + return discovery.Resolution{}, errors.New("syntax error") + case "SELECT * FROM mystery": // resolves to a name the registry doesn't know + return discovery.Resolution{Tables: []string{"mystery_view"}}, nil + case "SELECT * FROM events WHERE 'w'='m'": // a dead-branch-pruned arm + return discovery.Resolution{Tables: []string{"events"}, Pruned: true}, nil + case "SELECT e.id, s3data.v FROM events e JOIN s3('http://x/y') s3data ON 1": // an external read + return discovery.Resolution{Tables: []string{"events"}, External: true}, nil + default: + return discovery.Resolution{}, nil + } + } + + // resolves to the reported tables; all known → not TTL-capped + deps, unresolved := h.resolveDeps(ctx, "SELECT * FROM events") + assert.Equal(t, []string{"events"}, pipeDepNames(deps)) + assert.False(t, unresolved, "a known base table must not cap the TTL") + + // cached: a second call with the same bound query doesn't re-run EXPLAIN + before := calls + _, _ = h.resolveDeps(ctx, "SELECT * FROM events") + assert.Equal(t, before, calls, "resolveTablesFn must be cached per bound query") + + // multi-table + deps, _ = h.resolveDeps(ctx, "SELECT * FROM events JOIN users ON 1") + assert.Equal(t, []string{"events", "users"}, pipeDepNames(deps)) + + // fallback: an un-analyzable query folds the single database-version namespace + // (any write bumps it, so it is version-maintained and NOT TTL-capped) + deps, unresolved = h.resolveDeps(ctx, "INSERT INTO events VALUES (1)") + assert.Equal(t, []cache.Namespace{cache.DatabaseNamespace()}, deps) + assert.False(t, unresolved, "the database-version fallback must not cap the TTL") + + // a resolved dep the registry doesn't know (an unfoldable view) → TTL-capped + deps, unresolved = h.resolveDeps(ctx, "SELECT * FROM mystery") + assert.Equal(t, []string{"mystery_view"}, pipeDepNames(deps)) + assert.True(t, unresolved, "an unknown/unfoldable dep must cap the TTL") + + // a dead-branch-pruned set keeps its tables but caps the TTL + // (belt-and-suspenders against a hypothetical wrong prune) + deps, unresolved = h.resolveDeps(ctx, "SELECT * FROM events WHERE 'w'='m'") + assert.Equal(t, []string{"events"}, pipeDepNames(deps)) + assert.True(t, unresolved, "a pruned dependency set must cap the TTL") + + // an external read (a table function, a cross-database table) keeps its local + // tables but caps the TTL — writes to the external source are invisible + deps, unresolved = h.resolveDeps(ctx, "SELECT e.id, s3data.v FROM events e JOIN s3('http://x/y') s3data ON 1") + assert.Equal(t, []string{"events"}, pipeDepNames(deps)) + assert.True(t, unresolved, "an external read must cap the TTL") + + // ClearResolvedDeps forces re-resolution + callsBefore := calls + h.ClearResolvedDeps() + _, _ = h.resolveDeps(ctx, "SELECT * FROM events") + assert.Greater(t, calls, callsBefore, "ClearResolvedDeps must force re-resolution") + + // no registry → tracking disabled (TTL-only), no deps + hNil := &PipesHandler{Cache: lc, logger: testutil.NopLogger(), pipeDeps: map[string]*resolvedDeps{}} + nilDeps, nilUnresolved := hNil.resolveDeps(ctx, "SELECT 1") + assert.Empty(t, nilDeps) + assert.False(t, nilUnresolved) +} + +// gatedConn signals when its first Query begins and holds every Query until +// released, so a test can land an invalidation while a pipe query is in flight. +// The embedded nil driver.Conn panics on any other method (loud test failure). +type gatedConn struct { + driver.Conn + entered chan struct{} // closed when the first Query begins + release chan struct{} // closed to let queries return + once sync.Once +} + +func (c *gatedConn) Query(_ context.Context, _ string, _ ...any) (driver.Rows, error) { + c.once.Do(func() { close(c.entered) }) + <-c.release + return &chainEmptyRows{}, nil +} + +// End-to-end through Execute: a write that lands while the pipe query is +// executing must not let the pre-write result be served to post-write readers. +// The versioned key is snapshotted before the query runs (Cache.Key contract), +// so the mid-flight bump orphans the Set; re-folding the key at Set time instead +// would cache the pre-write rows under the post-bump key — a stale HIT for the +// full TTL that no write could ever evict. +func TestPipesHandler_Execute_MidFlightWriteIsNotServedStale(t *testing.T) { + t.Parallel() + reg := discovery.NewSchemaRegistryForTest([]*discovery.TableSchema{{Name: "events"}}, nil) + lc, err := cache.NewLocal(1 << 20) + require.NoError(t, err) + defer func() { _ = lc.Close() }() + lc.SetDependents(reg.Dependents()) + + conn := &gatedConn{entered: make(chan struct{}), release: make(chan struct{})} + store := pipes.NewMemoryStore(&pipes.NamedQuery{Name: "recent", SQL: "SELECT * FROM events"}) + h := NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), conn, lc, reg, 5*time.Second, testutil.NopLogger()) + h.resolveTablesFn = func(_ context.Context, _ string) (discovery.Resolution, error) { + return discovery.Resolution{Tables: []string{"events"}}, nil + } + + exec := func() *httptest.ResponseRecorder { + w := httptest.NewRecorder() + r := pipesRequest(t, http.MethodGet, "/v1/pipes/recent/execute", "recent", nil) + r = r.WithContext(auth.WithRole(r.Context(), "admin")) + h.Execute(w, r) + return w + } + + // Request 1 misses and starts executing... + done := make(chan *httptest.ResponseRecorder, 1) + go func() { done <- exec() }() + <-conn.entered + + // ...an ingest write invalidates the table while the query is in flight... + _, err = lc.Invalidate(context.Background(), []cache.Namespace{{Table: "events"}}) + require.NoError(t, err) + close(conn.release) + + w1 := <-done + require.Equal(t, http.StatusOK, w1.Code) + assert.Equal(t, "MISS", w1.Header().Get("X-Cache")) + lc.Wait() // let request 1's (orphaned) Set settle before probing + + // ...so a post-write reader must re-execute, not HIT the pre-write result. + w2 := exec() + require.Equal(t, http.StatusOK, w2.Code) + assert.Equal(t, "MISS", w2.Header().Get("X-Cache"), + "a result computed before a mid-flight write must not be served after it") +} + +// A fallback marker is memoized only when ClickHouse actually ANSWERED that the +// query can't be analyzed (*clickhouse.Exception) — a property of the query +// against the current schema, dropped on refresh via ClearResolvedDeps. A failure +// to ASK — an unreachable server, a canceled request — must not be memoized, or +// one transient blip would pin the pipe to the database-version fallback (evicted +// by every write) until the next schema CHANGE (on a stable schema: never); the +// next execution retries instead. +func TestResolveDeps_FailureMemoization(t *testing.T) { + ctx := context.Background() + reg := discovery.NewSchemaRegistryForTest([]*discovery.TableSchema{{Name: "events"}}, nil) + + // Transient failure: falls back for this execution, retried on the next. + h := &PipesHandler{registry: reg, logger: testutil.NopLogger(), pipeDeps: map[string]*resolvedDeps{}} + calls := 0 + h.resolveTablesFn = func(_ context.Context, _ string) (discovery.Resolution, error) { + calls++ + if calls == 1 { + return discovery.Resolution{}, context.Canceled // transient: the caller's request went away + } + return discovery.Resolution{Tables: []string{"events"}}, nil + } + deps, _ := h.resolveDeps(ctx, "SELECT * FROM events") + assert.Equal(t, []cache.Namespace{cache.DatabaseNamespace()}, deps, "the transient failure still falls back for this execution") + deps, _ = h.resolveDeps(ctx, "SELECT * FROM events") + assert.Equal(t, 2, calls, "a transient failure must not be memoized — the next execution retries") + assert.Equal(t, []string{"events"}, pipeDepNames(deps), "the retry resolves precisely") + _, _ = h.resolveDeps(ctx, "SELECT * FROM events") + assert.Equal(t, 2, calls, "the successful resolution is memoized") + + // ClickHouse-answered failure: memoized like a success. + h = &PipesHandler{registry: reg, logger: testutil.NopLogger(), pipeDeps: map[string]*resolvedDeps{}} + calls = 0 + h.resolveTablesFn = func(_ context.Context, _ string) (discovery.Resolution, error) { + calls++ + return discovery.Resolution{}, fmt.Errorf("explain query tree: %w", &clickhouse.Exception{Code: 62, Message: "Syntax error"}) + } + deps, _ = h.resolveDeps(ctx, "INSERT INTO events VALUES (1)") + assert.Equal(t, []cache.Namespace{cache.DatabaseNamespace()}, deps, "falls back to the database-version namespace") + _, _ = h.resolveDeps(ctx, "INSERT INTO events VALUES (1)") + assert.Equal(t, 1, calls, "a ClickHouse-answered failure is memoized until the schema changes") +} + +// The memo is bounded: at maxPipeDeps entries, inserting a new resolution evicts +// ONE arbitrary existing entry (random replacement) rather than dropping the +// whole map — a high-cardinality parameter can't wipe every working pipe's +// resolution, and the memo can't grow without limit between refreshes. +func TestResolveDeps_MemoCapEvictsOneEntry(t *testing.T) { + ctx := context.Background() + reg := discovery.NewSchemaRegistryForTest([]*discovery.TableSchema{{Name: "events"}}, nil) + h := &PipesHandler{registry: reg, logger: testutil.NopLogger(), pipeDeps: make(map[string]*resolvedDeps, maxPipeDeps)} + h.resolveTablesFn = func(_ context.Context, _ string) (discovery.Resolution, error) { + return discovery.Resolution{Tables: []string{"events"}}, nil + } + + // Fill the memo to its cap (same-package shortcut; going through resolveDeps + // 50k times would just be slower). + h.pipeDepsMu.Lock() + for i := range maxPipeDeps { + h.pipeDeps[fmt.Sprintf("SELECT %d", i)] = &resolvedDeps{tables: []string{"events"}} + } + h.pipeDepsMu.Unlock() + + _, _ = h.resolveDeps(ctx, "SELECT * FROM events") + + h.pipeDepsMu.RLock() + defer h.pipeDepsMu.RUnlock() + assert.Len(t, h.pipeDeps, maxPipeDeps, "one eviction + one insert keeps the memo at its cap") + _, ok := h.pipeDeps["SELECT * FROM events"] + assert.True(t, ok, "the new resolution is memoized after evicting one arbitrary entry") +} + +// slowConn answers every Query with empty rows after a fixed delay, so +// QueryTimeToTTL derives an uncapped TTL well above UnresolvedDepsTTLCap. +type slowConn struct { + driver.Conn + delay time.Duration +} + +func (c *slowConn) Query(_ context.Context, _ string, _ ...any) (driver.Rows, error) { + time.Sleep(c.delay) + return &chainEmptyRows{}, nil +} + +// ttlRecordingCache records the TTL of the last Set so a test can assert the +// cap was applied at the Set site, not just decided by resolveDeps. +type ttlRecordingCache struct { + cache.Cache + lastSetTTL time.Duration +} + +func (c *ttlRecordingCache) Set(ctx context.Context, key string, value []byte, ttl time.Duration) error { + c.lastSetTTL = ttl + return c.Cache.Set(ctx, key, value, ttl) +} + +// A result whose dependency set can't be trusted to version invalidation (here: +// dead-branch pruned) must be STORED with the capped TTL. The cap decision is +// pinned in TestResolveDeps_Explain; this pins its application at the pipe Set. +// The fake query takes 25ms, so the uncapped TTL would be ~25s (QueryTimeToTTL's +// 1000x curve) — the ≤10s assertion genuinely distinguishes capped from not. +func TestPipesHandler_Execute_UnresolvedDepsTTLCapApplied(t *testing.T) { + t.Parallel() + reg := discovery.NewSchemaRegistryForTest([]*discovery.TableSchema{{Name: "events"}}, nil) + lc, err := cache.NewLocal(1 << 20) + require.NoError(t, err) + defer func() { _ = lc.Close() }() + rec := &ttlRecordingCache{Cache: lc} + + store := pipes.NewMemoryStore(&pipes.NamedQuery{Name: "recent", SQL: "SELECT * FROM events"}) + h := NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), &slowConn{delay: 25 * time.Millisecond}, rec, reg, 5*time.Second, testutil.NopLogger()) + h.resolveTablesFn = func(_ context.Context, _ string) (discovery.Resolution, error) { + return discovery.Resolution{Tables: []string{"events"}, Pruned: true}, nil + } + + w := httptest.NewRecorder() + r := pipesRequest(t, http.MethodGet, "/v1/pipes/recent/execute", "recent", nil) + r = r.WithContext(auth.WithRole(r.Context(), "admin")) + h.Execute(w, r) + require.Equal(t, http.StatusOK, w.Code, w.Body.String()) + + assert.Positive(t, rec.lastSetTTL, "the miss must reach Set") + assert.LessOrEqual(t, rec.lastSetTTL, cache.UnresolvedDepsTTLCap, + "a pruned dependency set must cap the stored TTL") +} + +// A ClearResolvedDeps landing while a resolution is in flight must win: the +// in-flight resolution was computed against the pre-refresh schema, so inserting +// it after the clear would silently re-populate the memo with a stale dependency +// set that persists until the NEXT refresh. The generation gate drops it instead, +// and the following execution re-resolves. +func TestResolveDeps_ClearDuringResolveIsNotUndone(t *testing.T) { + ctx := context.Background() + reg := discovery.NewSchemaRegistryForTest([]*discovery.TableSchema{{Name: "events"}}, nil) + h := &PipesHandler{registry: reg, logger: testutil.NopLogger(), pipeDeps: map[string]*resolvedDeps{}} + + calls := 0 + h.resolveTablesFn = func(_ context.Context, _ string) (discovery.Resolution, error) { + calls++ + if calls == 1 { + // The schema refresh fires while this resolution is in flight. + h.ClearResolvedDeps() + } + return discovery.Resolution{Tables: []string{"events"}}, nil + } + + _, _ = h.resolveDeps(ctx, "SELECT * FROM events") + + h.pipeDepsMu.RLock() + _, memoized := h.pipeDeps["SELECT * FROM events"] + h.pipeDepsMu.RUnlock() + assert.False(t, memoized, "a resolution overlapped by ClearResolvedDeps must not be memoized") + + _, _ = h.resolveDeps(ctx, "SELECT * FROM events") + assert.Equal(t, 2, calls, "the next execution re-resolves against the refreshed schema") + h.pipeDepsMu.RLock() + _, memoized = h.pipeDeps["SELECT * FROM events"] + h.pipeDepsMu.RUnlock() + assert.True(t, memoized, "the post-refresh resolution is memoized normally") +} + +// Concurrent cold misses for the same bound query coalesce onto ONE EXPLAIN via +// the resolve singleflight; every waiter shares the outcome. +func TestResolveDeps_ConcurrentColdMissesCoalesce(t *testing.T) { + ctx := context.Background() + reg := discovery.NewSchemaRegistryForTest([]*discovery.TableSchema{{Name: "events"}}, nil) + h := &PipesHandler{registry: reg, logger: testutil.NopLogger(), pipeDeps: map[string]*resolvedDeps{}} + + var calls atomic.Int32 + started := make(chan struct{}) + release := make(chan struct{}) + h.resolveTablesFn = func(_ context.Context, _ string) (discovery.Resolution, error) { + if calls.Add(1) == 1 { + close(started) + } + <-release + return discovery.Resolution{Tables: []string{"events"}}, nil + } + + const waiters = 8 + var wg sync.WaitGroup + results := make([][]cache.Namespace, waiters) + for i := range waiters { + wg.Add(1) + go func() { + defer wg.Done() + results[i], _ = h.resolveDeps(ctx, "SELECT * FROM events") + }() + } + <-started + // The first resolution is now blocked in flight; give the remaining waiters + // time to reach the singleflight and join it. A straggler that misses even + // this window would still be served by the memo after the flight completes, + // so the assertion below stays stable. + time.Sleep(100 * time.Millisecond) + close(release) + wg.Wait() + + assert.Equal(t, int32(1), calls.Load(), "concurrent cold misses must coalesce onto one EXPLAIN") + for i := range waiters { + assert.Equal(t, []string{"events"}, pipeDepNames(results[i])) + } +} diff --git a/internal/api/pipes.go b/internal/api/pipes.go index eac7b853..8a2d9c41 100644 --- a/internal/api/pipes.go +++ b/internal/api/pipes.go @@ -3,19 +3,65 @@ package api import ( "context" "encoding/json" + "errors" "log/slog" "net/http" + "sync" "time" + "github.com/ClickHouse/clickhouse-go/v2" "github.com/ClickHouse/clickhouse-go/v2/lib/driver" "github.com/Wave-RF/WaveHouse/internal/auth" "github.com/Wave-RF/WaveHouse/internal/cache" + "github.com/Wave-RF/WaveHouse/internal/chsql" + "github.com/Wave-RF/WaveHouse/internal/discovery" "github.com/Wave-RF/WaveHouse/internal/pipes" "github.com/Wave-RF/WaveHouse/internal/policy" "github.com/go-chi/chi/v5" + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/metric" "golang.org/x/sync/singleflight" ) +const maxPipeDeps = 50000 + +var ( + pipeFallbackCounter, _ = otel.Meter("wavehouse-pipes").Int64Counter( + "wavehouse_pipe_dep_resolution_fallback_total", + metric.WithDescription("Pipe executions served on the database-version fallback (any write evicts) because ClickHouse couldn't analyze the query (a write/DDL pipe, a missing table, an unreachable server)"), + ) + pipeUnresolvedCounter, _ = otel.Meter("wavehouse-pipes").Int64Counter( + "wavehouse_pipe_dep_unresolved_total", + metric.WithDescription("Pipe executions TTL-capped because the result can't be reliably version-invalidated (an unfoldable view, a cross-database/unknown name, an external read like a table function, or a dead-branch-pruned dependency set)"), + ) + pipeDepsEvictionCounter, _ = otel.Meter("wavehouse-pipes").Int64Counter( + "wavehouse_pipe_dep_memo_evictions_total", + metric.WithDescription("Resolved-dependency memo entries evicted because the memo hit its size cap (maxPipeDeps); sustained growth means a high-cardinality pipe parameter"), + ) +) + +type resolvedDeps struct { + tables []string + // fallback is set when EXPLAIN couldn't be used (a write/DDL pipe, a missing + // table, an unreachable ClickHouse), in which case the pipe folds the single + // DATABASE-version namespace (cache.DatabaseNamespace): any write bumps it, so + // any write evicts the result — never an under-resolution, and O(1) per request + // instead of folding every base table. + fallback bool + // unresolved is set when EXPLAIN succeeded but the result can't be reliably + // version-invalidated, so the Execute path TTL-caps it + // (cache.UnresolvedDepsTTLCap) instead of trusting eviction. Three triggers, + // logged individually at resolve time: a resolved dependency whose version the + // cascade never bumps (an unfoldable view or unknown name, per + // SchemaRegistry.IsKnown), an external read no table version can watch (a + // table function, a cross-database table, a non-local dictionary — + // discovery.Resolution.External), and a dead-branch-pruned dependency set + // (discovery.Resolution.Pruned, a belt-and-suspenders bound on a hypothetical + // wrong prune). Distinct from fallback, whose database-version namespace is + // always maintained. + unresolved bool +} + // PipesHandler handles named query pipe endpoints. type PipesHandler struct { Store *pipes.Store @@ -25,6 +71,21 @@ type PipesHandler struct { sf singleflight.Group maxQueryTimeout time.Duration logger *slog.Logger + registry *discovery.SchemaRegistry + resolveTablesFn func(ctx context.Context, sql string) (discovery.Resolution, error) + // resolveSF coalesces concurrent cold-miss resolutions per bound SQL — one + // EXPLAIN, every waiter shares the outcome. Distinct from sf, which keys on + // the folded cache key and coalesces query EXECUTION. + resolveSF singleflight.Group + // pipeDeps memoizes, per bound query, which cache namespaces gate the pipe — + // the version-key inputs Cache.Key folds. Guarded by pipeDepsMu (RWMutex: the + // steady-state read on every Execute takes only the read lock). pipeDepsGen + // increments on every ClearResolvedDeps so a resolution that RAN against a + // pre-refresh schema is never inserted after the refresh wiped the memo (the + // insert re-checks the generation it started under). + pipeDeps map[string]*resolvedDeps + pipeDepsGen uint64 + pipeDepsMu sync.RWMutex // maxRequestBytes optionally overrides the default inbound request body // cap (maxControlBodyBytes) for the body-decoding paths (Put, Execute). @@ -34,8 +95,23 @@ type PipesHandler struct { maxRequestBytes int64 } -func NewPipesHandler(store *pipes.Store, policyStore *policy.Store, conn driver.Conn, c cache.Cache, queryTimeout time.Duration, logger *slog.Logger) *PipesHandler { - return &PipesHandler{Store: store, PolicyStore: policyStore, CHConn: conn, Cache: c, maxQueryTimeout: queryTimeout, logger: logger} +func NewPipesHandler(store *pipes.Store, policyStore *policy.Store, conn driver.Conn, c cache.Cache, registry *discovery.SchemaRegistry, queryTimeout time.Duration, logger *slog.Logger) *PipesHandler { + h := &PipesHandler{ + Store: store, + PolicyStore: policyStore, + CHConn: conn, + Cache: c, + registry: registry, + maxQueryTimeout: queryTimeout, + logger: logger, + pipeDeps: make(map[string]*resolvedDeps), + } + // Resolve a pipe's tables by asking ClickHouse (EXPLAIN QUERY TREE). Overridable + // in tests; the database comes from the Registry. + h.resolveTablesFn = func(ctx context.Context, sql string) (discovery.Resolution, error) { + return discovery.ResolveTables(ctx, h.CHConn, h.registry.Database(), sql) + } + return h } // List returns all named queries (admin endpoint). @@ -168,14 +244,18 @@ func (h *PipesHandler) Execute(w http.ResponseWriter, r *http.Request) { return } - // Cache. A pipe can read several tables, but the current pipe impl doesn't - // expose its table/scope dependencies, so we pass no deps: the result is keyed - // by sha alone (TTL-only) and the ingest worker cannot version-invalidate it. - // TODO: once pipes expose their tables/scopes, pass them as deps here so writes - // invalidate cached pipe results. cacheKey := queryCacheKey(sql, params) + + deps, depsUnresolved := h.resolveDeps(r.Context(), sql) + + // Snapshot the version-folded key once, before the query runs, and reuse it + // for the Get, the singleflight, and the Set (the Cache.Key contract). Keying + // the singleflight on the folded key (not the bare sha) also keeps a post-bump + // request from coalescing onto a pre-bump flight. + entryKey := cacheKey if h.Cache != nil { - if data, _, err := h.Cache.Get(r.Context(), cacheKey, nil); err == nil && data != nil { + entryKey = h.Cache.Key(cacheKey, deps) + if data, _, err := h.Cache.Get(r.Context(), entryKey); err == nil && data != nil { w.Header().Set("Content-Type", "application/json") w.Header().Set("X-Cache", "HIT") _, _ = w.Write(data) @@ -183,13 +263,11 @@ func (h *PipesHandler) Execute(w http.ResponseWriter, r *http.Request) { } } - // Execute with singleflight. - v, err, _ := h.sf.Do(cacheKey, func() (interface{}, error) { + v, err, _ := h.sf.Do(entryKey, func() (interface{}, error) { queryCtx, cancel := context.WithTimeout(r.Context(), h.maxQueryTimeout) defer cancel() start := time.Now() - rows, err := executeCHQuery(queryCtx, h.CHConn, sql, params) queryDuration := time.Since(start) if err != nil { @@ -203,10 +281,17 @@ func (h *PipesHandler) Execute(w http.ResponseWriter, r *http.Request) { return nil, err } - ttl := cache.QueryTimeToTTL(queryDuration) - if h.Cache != nil { - _ = h.Cache.Set(r.Context(), cacheKey, nil, data, ttl) + ttl := cache.QueryTimeToTTL(queryDuration) + if depsUnresolved { + // The result can't be reliably version-invalidated — an unfoldable + // view, an external read (table function/cross-database), or a + // dead-branch-pruned set (belt-and-suspenders): cap the TTL so it + // self-expires rather than serving stale on a write the folded + // versions never see. + ttl = min(ttl, cache.UnresolvedDepsTTLCap) + } + _ = h.Cache.Set(r.Context(), entryKey, data, ttl) } return data, nil }) @@ -219,3 +304,158 @@ func (h *PipesHandler) Execute(w http.ResponseWriter, r *http.Request) { w.Header().Set("X-Cache", "MISS") _, _ = w.Write(v.([]byte)) } + +// resolveOutcome carries a coalesced resolution out of resolveSF: the resolved +// deps, whether they may be memoized (see resolvePipe), and the memo generation +// the resolution STARTED under (see the insert gate in resolveDeps). +type resolveOutcome struct { + rd *resolvedDeps + memoize bool + gen uint64 +} + +// resolveDeps returns the cache namespaces a pipe's result depends on, by asking +// ClickHouse which tables the bound query reads (EXPLAIN QUERY TREE, via +// resolveTablesFn), memoized per bound query. A table EXPLAIN reports is folded +// as its own versioned namespace; a write to it — or one that reaches it via the +// cascade: a write to a view's base, or to the source of an MV populating it — +// evicts the result. When ClickHouse can't analyze the query (a write/DDL pipe, a +// missing table, an unreachable server) the pipe folds the single database-version +// namespace, which any write bumps: coarse, but never a stale read and O(1) per +// request. +// +// The second return reports whether the result must be TTL-capped because it +// can't be reliably version-invalidated — see resolvedDeps.unresolved for the +// triggers. +// +// TODO: impl scope (per-tenant cache invalidation) — each dep is whole-table for now. +func (h *PipesHandler) resolveDeps(ctx context.Context, boundSQL string) ([]cache.Namespace, bool) { + if h.registry == nil { + return nil, false // dependency tracking disabled (no schema registry): TTL-only + } + + // Steady state is a read — every Execute passes through here, including cache + // hits — so only the read lock is taken; the write lock is for the cold-miss + // insert and ClearResolvedDeps. + h.pipeDepsMu.RLock() + rd, ok := h.pipeDeps[boundSQL] + h.pipeDepsMu.RUnlock() + if !ok { + // Coalesce concurrent cold misses for the same bound query: one EXPLAIN, + // every waiter shares the outcome. The generation is captured INSIDE the + // flight, before the EXPLAIN runs, so it dates the schema state the + // resolution was computed against for every waiter alike. + v, _, _ := h.resolveSF.Do(boundSQL, func() (any, error) { + h.pipeDepsMu.RLock() + gen := h.pipeDepsGen + h.pipeDepsMu.RUnlock() + rd, memoize := h.resolvePipe(ctx, boundSQL) + return resolveOutcome{rd: rd, memoize: memoize, gen: gen}, nil + }) + out := v.(resolveOutcome) + rd = out.rd + if out.memoize { + h.pipeDepsMu.Lock() + // Insert only if no ClearResolvedDeps ran since the resolution started: + // a refresh mid-EXPLAIN means this rd was computed against the + // pre-refresh schema, and inserting it would silently undo the clear — + // leaving a stale dependency set (the version-key inputs) memoized until + // the NEXT refresh. Dropped instead; the next execution re-resolves. + if h.pipeDepsGen == out.gen { + // Bound the memo: a high-cardinality parameter could otherwise grow + // it without limit between schema refreshes. Over the cap, evict ONE + // arbitrary entry (Go map iteration order) — random replacement — + // rather than dropping the whole map and forcing every working pipe + // to re-EXPLAIN. Metered so an operator can spot the cardinality + // problem; the cap bounds the memo's memory, not the per-request + // work (the incoming query's EXPLAIN had to run regardless). + if len(h.pipeDeps) >= maxPipeDeps { + for k := range h.pipeDeps { + delete(h.pipeDeps, k) + break + } + pipeDepsEvictionCounter.Add(ctx, 1) + } + h.pipeDeps[boundSQL] = rd + } + h.pipeDepsMu.Unlock() + } + } + + // fallback and unresolved are mutually exclusive: fallback is set when EXPLAIN + // can't run (resolvePipe returns early), unresolved only when it succeeds. Both + // counters meter EXECUTIONS (including cache hits), not resolution events: the + // operator signal is "what share of pipe traffic is running degraded", which a + // resolve-time-only count would under-report (a memoized fallback resolves once + // but can serve degraded for hours). + if rd.fallback { + pipeFallbackCounter.Add(ctx, 1) + return []cache.Namespace{cache.DatabaseNamespace()}, false + } + if rd.unresolved { + pipeUnresolvedCounter.Add(ctx, 1) + } + deps := make([]cache.Namespace, 0, len(rd.tables)) + for _, name := range rd.tables { + deps = append(deps, cache.Namespace{Table: chsql.SafeEncodeNATS(name)}) + } + return deps, rd.unresolved +} + +// resolvePipe asks ClickHouse for the bound query's tables. An error yields a +// fallback marker so the caller over-resolves rather than risk an under-resolution. +// +// memoize reports whether the result may be kept in the per-bound-query cache. A +// ClickHouse-answered failure (*clickhouse.Exception: a write/DDL pipe, a missing +// table) is a property of the query against the current schema, so it is memoized +// like a success — ClearResolvedDeps drops it when the schema changes. A failure to +// ASK (an unreachable server, the resolve timeout, the caller's request being +// canceled mid-EXPLAIN) is transient, so it is NOT memoized: memoizing it would pin +// the pipe to the database-version fallback — evicted by every write, effectively +// uncacheable — until some future schema CHANGE happens to clear it, which on a +// stable schema is never. The execution still falls back this once (never stale), +// and the next execution simply retries the EXPLAIN. +func (h *PipesHandler) resolvePipe(ctx context.Context, boundSQL string) (rd *resolvedDeps, memoize bool) { + if h.resolveTablesFn == nil { + return &resolvedDeps{fallback: true}, true + } + rctx, cancel := context.WithTimeout(ctx, 5*time.Second) + defer cancel() + res, err := h.resolveTablesFn(rctx, boundSQL) + if err != nil { + h.logger.DebugContext(ctx, "pipe dependency resolution failed; falling back to database-version invalidation", "error", err) + _, answered := errors.AsType[*clickhouse.Exception](err) + return &resolvedDeps{fallback: true}, answered + } + rd = &resolvedDeps{tables: res.Tables, unresolved: res.Pruned || res.External} + if res.Pruned { + h.logger.DebugContext(ctx, "pipe dependency set was dead-branch pruned; TTL-capping result as a safety bound") + } + if res.External { + h.logger.DebugContext(ctx, "pipe reads an external source (table function, cross-database, or non-local dictionary); TTL-capping result") + } + // A resolved dep whose version isn't reliably maintained (an unfoldable view, an + // unknown name) makes the whole result untrustworthy to version invalidation, so + // mark it for a TTL cap. registry is non-nil here (resolveDeps guards it). + for _, t := range res.Tables { + if !h.registry.IsKnown(t) { + rd.unresolved = true + h.logger.DebugContext(ctx, "pipe dependency can't be reliably version-invalidated; TTL-capping result", "table", t) + break + } + } + return rd, true +} + +// ClearResolvedDeps drops every pipe's cached table resolution so each re-resolves +// against the current schema on its next execution. main wires this to a schema +// refresh, so a developer's "edit pipe/schema -> refresh" picks up the change. The +// generation bump makes the clear stick: a resolution already in flight (computed +// against the pre-refresh schema) sees a stale generation at insert time and is +// dropped instead of silently re-populating the memo. +func (h *PipesHandler) ClearResolvedDeps() { + h.pipeDepsMu.Lock() + h.pipeDeps = make(map[string]*resolvedDeps) + h.pipeDepsGen++ + h.pipeDepsMu.Unlock() +} diff --git a/internal/api/pipes_test.go b/internal/api/pipes_test.go index a171b6e0..3455d3d9 100644 --- a/internal/api/pipes_test.go +++ b/internal/api/pipes_test.go @@ -42,7 +42,7 @@ func TestPipesHandler_List(t *testing.T) { &pipes.NamedQuery{Name: "top_pages", SQL: "SELECT page, count(*) FROM clicks GROUP BY page"}, &pipes.NamedQuery{Name: "recent", SQL: "SELECT * FROM clicks ORDER BY ts DESC LIMIT 10"}, ) - h := NewPipesHandler(store, nil, nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, nil, 0, testutil.NopLogger()) w := httptest.NewRecorder() r := httptest.NewRequestWithContext(context.Background(), http.MethodGet, "/v1/pipes", nil) @@ -59,7 +59,7 @@ func TestPipesHandler_Get_Found(t *testing.T) { store := pipes.NewMemoryStore( &pipes.NamedQuery{Name: "top_pages", SQL: "SELECT page FROM clicks"}, ) - h := NewPipesHandler(store, nil, nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, nil, 0, testutil.NopLogger()) w := httptest.NewRecorder() r := pipesRequest(t, http.MethodGet, "/v1/pipes/top_pages", "top_pages", nil) @@ -74,7 +74,7 @@ func TestPipesHandler_Get_Found(t *testing.T) { func TestPipesHandler_Get_NotFound(t *testing.T) { t.Parallel() store := pipes.NewMemoryStore() - h := NewPipesHandler(store, nil, nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, nil, 0, testutil.NopLogger()) w := httptest.NewRecorder() r := pipesRequest(t, http.MethodGet, "/v1/pipes/nope", "nope", nil) @@ -88,7 +88,7 @@ func TestPipesHandler_Get_NotFound(t *testing.T) { func TestPipesHandler_List_Empty(t *testing.T) { t.Parallel() store := pipes.NewMemoryStore() - h := NewPipesHandler(store, nil, nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, nil, 0, testutil.NopLogger()) w := httptest.NewRecorder() r := pipesRequest(t, http.MethodGet, "/v1/pipes", "", nil) @@ -104,7 +104,7 @@ func TestPipesHandler_List_Empty(t *testing.T) { func TestPipesHandler_Execute_NotFound(t *testing.T) { t.Parallel() store := pipes.NewMemoryStore() - h := NewPipesHandler(store, nil, nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, nil, 0, testutil.NopLogger()) w := httptest.NewRecorder() r := pipesRequest(t, http.MethodPost, "/v1/pipes/nope/execute", "nope", nil) @@ -128,7 +128,7 @@ func TestPipesHandler_Execute_RoleAuthorization(t *testing.T) { // A real (non-nil) policy so the default admin role ("admin") is defined // and bypasses the allowlist, per the matrix. With a nil policy nobody is // admin (total lockout) — covered separately in internal/policy tests. - h := NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), nil, nil, nil, 0, testutil.NopLogger()) w := httptest.NewRecorder() r := pipesRequest(t, http.MethodPost, "/v1/pipes/report/execute", "report", nil) @@ -155,7 +155,7 @@ func TestPipesHandler_Execute_RestrictedPipe_EmptyRoleDenied(t *testing.T) { AllowedRoles: []string{"admin"}, }, ) - h := NewPipesHandler(store, nil, nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, nil, 0, testutil.NopLogger()) w := httptest.NewRecorder() // No ContextKeyRole set, which simulates no token or a JWT without the role claim. @@ -176,7 +176,7 @@ func TestPipesHandler_Execute_DefaultRoleGrantsAccess(t *testing.T) { store := pipes.NewMemoryStore( &pipes.NamedQuery{Name: "report", SQL: "SELECT * FROM clicks", AllowedRoles: []string{"viewer"}}, ) - h := NewPipesHandler(store, nil, nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, nil, 0, testutil.NopLogger()) h.PolicyStore = policy.NewMemoryStore(&policy.Policy{DefaultRole: "viewer"}) w := httptest.NewRecorder() @@ -197,7 +197,7 @@ func TestPipesHandler_Execute_DefaultRoleNotInAllowedRolesDenied(t *testing.T) { store := pipes.NewMemoryStore( &pipes.NamedQuery{Name: "admin_report", SQL: "SELECT * FROM clicks", AllowedRoles: []string{"admin"}}, ) - h := NewPipesHandler(store, nil, nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, nil, 0, testutil.NopLogger()) h.PolicyStore = policy.NewMemoryStore(&policy.Policy{DefaultRole: "viewer"}) w := httptest.NewRecorder() @@ -221,7 +221,7 @@ func TestPipesHandler_Execute_MissingParam(t *testing.T) { }, }, ) - h := NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), nil, nil, nil, 0, testutil.NopLogger()) w := httptest.NewRecorder() // No query params or body — missing "page". @@ -246,7 +246,7 @@ func TestPipesHandler_Execute_ParamsFromQuery(t *testing.T) { }, }, ) - h := NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), nil, nil, nil, 0, testutil.NopLogger()) w := httptest.NewRecorder() r := httptest.NewRequestWithContext(context.Background(), http.MethodGet, "/v1/pipes/by_page/execute?page=/home", nil) @@ -275,7 +275,7 @@ func TestPipesHandler_Execute_RequestBodyCap(t *testing.T) { Parameters: []pipes.ParamDef{{Name: "page", Type: "string", Required: true}}, }, ) - h := NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), nil, nil, nil, 0, testutil.NopLogger()) h.maxRequestBytes = 64 w := httptest.NewRecorder() @@ -294,7 +294,7 @@ func TestPipesHandler_Execute_RequestBodyCap(t *testing.T) { func TestPipesHandler_Put_InvalidJSON(t *testing.T) { t.Parallel() store := pipes.NewMemoryStore() - h := NewPipesHandler(store, nil, nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, nil, 0, testutil.NopLogger()) w := httptest.NewRecorder() r := httptest.NewRequestWithContext(context.Background(), http.MethodPut, "/v1/pipes/test", bytes.NewReader([]byte(`{bad}`))) @@ -314,7 +314,7 @@ func TestPipesHandler_Put_InvalidJSON(t *testing.T) { func TestPipesHandler_Put_RequestBodyCap(t *testing.T) { t.Parallel() store := pipes.NewMemoryStore() - h := NewPipesHandler(store, nil, nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, nil, 0, testutil.NopLogger()) h.maxRequestBytes = 64 w := httptest.NewRecorder() @@ -331,7 +331,7 @@ func TestPipesHandler_Put_RequestBodyCap(t *testing.T) { func TestPipesHandler_Put_Success(t *testing.T) { t.Parallel() store := pipes.NewMemoryStore() - h := NewPipesHandler(store, nil, nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, nil, 0, testutil.NopLogger()) w := httptest.NewRecorder() r := pipesRequest(t, http.MethodPut, "/v1/pipes/new_pipe", "new_pipe", map[string]any{ @@ -350,7 +350,7 @@ func TestPipesHandler_Put_Success(t *testing.T) { func TestPipesHandler_Put_MissingSQL(t *testing.T) { t.Parallel() store := pipes.NewMemoryStore() - h := NewPipesHandler(store, nil, nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, nil, 0, testutil.NopLogger()) w := httptest.NewRecorder() r := pipesRequest(t, http.MethodPut, "/v1/pipes/bad", "bad", map[string]any{ @@ -367,7 +367,7 @@ func TestPipesHandler_Delete_Success(t *testing.T) { store := pipes.NewMemoryStore( &pipes.NamedQuery{Name: "to_delete", SQL: "SELECT 1"}, ) - h := NewPipesHandler(store, nil, nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, nil, 0, testutil.NopLogger()) w := httptest.NewRecorder() r := pipesRequest(t, http.MethodDelete, "/v1/pipes/to_delete", "to_delete", nil) @@ -389,7 +389,7 @@ func TestPipesHandler_Execute_PostBodyParams(t *testing.T) { }, }, ) - h := NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), nil, nil, nil, 0, testutil.NopLogger()) w := httptest.NewRecorder() body := map[string]any{"page": "/about"} @@ -412,7 +412,7 @@ func TestPipesHandler_Execute_NoAllowedRoles_NonAdminDenied(t *testing.T) { store := pipes.NewMemoryStore( &pipes.NamedQuery{Name: "open", SQL: "SELECT * FROM clicks"}, // no AllowedRoles ) - h := NewPipesHandler(store, nil, nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, nil, nil, nil, nil, 0, testutil.NopLogger()) w := httptest.NewRecorder() r := pipesRequest(t, http.MethodPost, "/v1/pipes/open/execute", "open", nil) @@ -438,7 +438,7 @@ func TestPipesHandler_Execute_ArrayParamBinds(t *testing.T) { Parameters: []pipes.ParamDef{{Name: "ids", Type: "array", Required: true}}, }, ) - h := NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), nil, nil, nil, 0, testutil.NopLogger()) w := httptest.NewRecorder() body := map[string]any{"ids": []any{"a", "b"}} @@ -459,7 +459,7 @@ func TestPipesHandler_Execute_ObjectParamRejected(t *testing.T) { store := pipes.NewMemoryStore( &pipes.NamedQuery{Name: "by_col", SQL: "SELECT * FROM clicks WHERE col = {{p}}"}, ) - h := NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), nil, nil, nil, 0, testutil.NopLogger()) w := httptest.NewRecorder() body := map[string]any{"p": map[string]any{"k": "v"}} @@ -480,7 +480,7 @@ func TestPipesHandler_Execute_NoAllowedRoles_AdminAllowed(t *testing.T) { store := pipes.NewMemoryStore( &pipes.NamedQuery{Name: "open", SQL: "SELECT * FROM clicks"}, ) - h := NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), nil, nil, 0, testutil.NopLogger()) + h := NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), nil, nil, nil, 0, testutil.NopLogger()) w := httptest.NewRecorder() r := pipesRequest(t, http.MethodPost, "/v1/pipes/open/execute", "open", nil) diff --git a/internal/api/stream.go b/internal/api/stream.go index 44485224..21b5d1df 100644 --- a/internal/api/stream.go +++ b/internal/api/stream.go @@ -7,8 +7,8 @@ import ( "time" "github.com/Wave-RF/WaveHouse/internal/auth" + "github.com/Wave-RF/WaveHouse/internal/chsql" "github.com/Wave-RF/WaveHouse/internal/mq" - "github.com/Wave-RF/WaveHouse/internal/query" "github.com/Wave-RF/WaveHouse/internal/stream" "github.com/nats-io/nats.go/jetstream" ) @@ -49,9 +49,9 @@ func (h *StreamHandler) Handle(w http.ResponseWriter, r *http.Request) { // TODO: impl scope scope := "" - topic := "ingest." + query.SafeEncodeNATS(table) + topic := "ingest." + chsql.SafeEncodeNATS(table) if scope != "" { - topic += "." + query.SafeEncodeNATS(scope) + topic += "." + chsql.SafeEncodeNATS(scope) } w.Header().Set("Content-Type", "text/event-stream") diff --git a/internal/api/structured_query.go b/internal/api/structured_query.go index 3564cf35..b8917d9f 100644 --- a/internal/api/structured_query.go +++ b/internal/api/structured_query.go @@ -12,6 +12,7 @@ import ( "github.com/ClickHouse/clickhouse-go/v2/lib/driver" "github.com/Wave-RF/WaveHouse/internal/auth" "github.com/Wave-RF/WaveHouse/internal/cache" + "github.com/Wave-RF/WaveHouse/internal/chsql" "github.com/Wave-RF/WaveHouse/internal/discovery" "github.com/Wave-RF/WaveHouse/internal/policy" "github.com/Wave-RF/WaveHouse/internal/query" @@ -150,16 +151,21 @@ func (h *StructuredQueryHandler) Handle(w http.ResponseWriter, r *http.Request) // TODO: impl scope scope := "" - safeTableName := query.SafeEncodeNATS(table) + safeTableName := chsql.SafeEncodeNATS(table) // A structured query reads one table, so it depends on a single namespace. // Encode the scope the way the ingest worker does (worker.go handleSuccess) so // the read and invalidation sides build identical namespace keys once scope is // implemented; SafeEncodeNATS("") is "", so this is a no-op while scope is empty. - deps := []cache.Namespace{{Table: safeTableName, Scope: query.SafeEncodeNATS(scope)}} + deps := []cache.Namespace{{Table: safeTableName, Scope: chsql.SafeEncodeNATS(scope)}} - // Try cache. + // Snapshot the version-folded key once, before the query runs, and reuse it + // for the Get, the singleflight, and the Set (the Cache.Key contract). Keying + // the singleflight on the folded key (not the bare sha) also keeps a post-bump + // request from coalescing onto a pre-bump flight. + entryKey := cacheKey if h.Cache != nil { - if data, _, err := h.Cache.Get(r.Context(), cacheKey, deps); err == nil && data != nil { + entryKey = h.Cache.Key(cacheKey, deps) + if data, _, err := h.Cache.Get(r.Context(), entryKey); err == nil && data != nil { w.Header().Set("Content-Type", "application/json") w.Header().Set("X-Cache", "HIT") _, _ = w.Write(data) @@ -168,7 +174,7 @@ func (h *StructuredQueryHandler) Handle(w http.ResponseWriter, r *http.Request) } // Execute with singleflight. - v, err, _ := h.sf.Do(cacheKey, func() (interface{}, error) { + v, err, _ := h.sf.Do(entryKey, func() (interface{}, error) { timeout := h.maxQueryTimeout if perms.MaxExecutionTime > 0 { timeout = min(perms.MaxExecutionTime.Duration(), timeout) @@ -212,9 +218,16 @@ func (h *StructuredQueryHandler) Handle(w http.ResponseWriter, r *http.Request) } ttl := cache.QueryTimeToTTL(queryDuration) + if !h.Registry.IsKnown(table) { + // The name resolved a schema at the top of Handle, but it's an UNFOLDABLE + // view the refresh-time cascade never bumps. Same rule as pipes: cap the + // TTL (cache.UnresolvedDepsTTLCap) so the result self-expires. A base + // table or foldable view is always known, so the happy path is untouched. + ttl = min(ttl, cache.UnresolvedDepsTTLCap) + } if h.Cache != nil { - _ = h.Cache.Set(r.Context(), cacheKey, deps, data, ttl) + _ = h.Cache.Set(r.Context(), entryKey, data, ttl) } return data, nil }) diff --git a/internal/cache/cache.go b/internal/cache/cache.go index 90401d62..94205a51 100644 --- a/internal/cache/cache.go +++ b/internal/cache/cache.go @@ -7,14 +7,29 @@ import ( // Cache provides versioned query-result storage with TTL support. type Cache interface { - // Get retrieves a cached query result and its remaining TTL. sha is the hash of - // the SQL+params; deps are the namespaces the result depends on (one for a - // structured query, several for a pipe). Returns nil, 0, nil on miss. - Get(ctx context.Context, sha string, deps []Namespace) ([]byte, time.Duration, error) + // Key folds sha (the hash of the SQL+params) with every dependency namespace at + // its CURRENT version — a point-in-time snapshot. deps are the namespaces the + // result depends on (one for a structured query, several for a pipe). + // + // A caller MUST compute the key once, BEFORE executing the underlying query, + // and use that same key for both the Get and the Set of one request. Folding + // versions at Set time instead would race Invalidate: a write that lands while + // the query is executing bumps the versions, and the result — computed from + // PRE-write data — would be stored under the POST-bump key and served as fresh + // for its full TTL. With the snapshot key, a mid-flight bump merely orphans the + // Set (stored under a key no future reader computes): wasted work, never a + // stale read. + Key(sha string, deps []Namespace) string - // TODO: TTL should be set based on query execution time - // Set stores a query result keyed by sha + its dependency namespaces. - Set(ctx context.Context, sha string, deps []Namespace, value []byte, ttl time.Duration) error + // Get retrieves a cached query result and its remaining TTL by its folded key + // (see Key). Returns nil, 0, nil on miss. + Get(ctx context.Context, key string) ([]byte, time.Duration, error) + + // Set stores a query result under the folded key snapshotted by Key before the + // query ran. Callers derive ttl from query execution time via QueryTimeToTTL; + // tuning that curve with real-world metrics is the remaining work (see + // QueryTimeToTTL). + Set(ctx context.Context, key string, value []byte, ttl time.Duration) error // TODO: option to prefetch pipes when invalidated? // TODO: AST query builder needs to give us a deterministic key or bypass cache entirely @@ -22,15 +37,40 @@ type Cache interface { // Invalidate bumps the version for each namespace, orphaning every cached query // that depends on it. A namespace with an empty Scope bumps the whole table // (every scope); a non-empty Scope bumps just that scope plus the whole-table - // view. Returns the number of namespaces processed. + // view. A write also fans out to the namespace's dependents (SetDependents — + // the views reading it and the MV TO targets it feeds), and any non-empty batch + // bumps the DATABASE version once (DatabaseNamespace), so a result folding only + // that one namespace is evicted by any write at all. Returns the number of + // namespaces processed. Invalidate(ctx context.Context, namespaces []Namespace) (uint64, error) + // SetDependents installs the base-table -> dependents cascade that Invalidate + // fans out through, so a write to a base table also evicts pipes reading a view + // over it or a table its attached MVs populate. Both sides are NATS-encoded. The + // schema registry pushes a fresh map on every content-changed refresh; replaces + // any prior one. + SetDependents(dependents map[string][]string) + // TODO: for local cache, we can just store the versions in memory, but for distributed/L2 cache, we will need to be able to either have stored procedures/pipelines etc to query them and attach them to a query, or sync them to each edge api server. // Close releases resources. Close() error } +// UnresolvedDepsTTLCap bounds the lifetime of a cached result whose dependencies +// are not reliably version-invalidatable, so it must self-expire quickly instead. +// Applied as min(QueryTimeToTTL, cap) at the Set call sites — and since +// QueryTimeToTTL's minimum is 10s too, the cap effectively PINS such a result's +// TTL at 10s (a ceiling on lifetime, not a floor). It fires for: a pipe with an +// unresolved dependency (an unknown or not-yet-discovered table, an unfoldable +// view), a pipe reading an external source no table version can watch (a table +// function, a cross-database table, a non-local dictionary source), a pipe whose +// dependency set was dead-branch pruned (belt-and-suspenders), and a structured +// query targeting an UNFOLDABLE view (the refresh-time cascade never bumps it; a +// base table or foldable view is unaffected). A fixed in-code backstop, not a +// tuning knob; promote to config only if real data warrants. +const UnresolvedDepsTTLCap = 10 * time.Second + // our tunable function to determine a cache entry's TTL based on how long its queryTime took func QueryTimeToTTL(queryTime time.Duration) time.Duration { // need more real-world data/metrics in order to better refine this, and likely an override option for easy local testing per-deployment diff --git a/internal/cache/local.go b/internal/cache/local.go index 8d6053d9..4fa18042 100644 --- a/internal/cache/local.go +++ b/internal/cache/local.go @@ -28,27 +28,30 @@ func NewLocal(maxCost int64) (*LocalCache, error) { return &LocalCache{cache: cache, versionManager: vm}, nil } -// Get looks up a cached query RESULT by its sha (hash of SQL+params) and the -// namespaces it depends on. Used by BOTH structured queries (which pass one -// Namespace) and pipes (which pass several). Returns nil, 0, nil on miss. -func (l *LocalCache) Get(_ context.Context, sha string, deps []Namespace) ([]byte, time.Duration, error) { - cacheKey := l.versionManager.QueryKey(sha, deps) +// Key folds sha (hash of SQL+params) with each dependency namespace at its +// current version — the point-in-time snapshot both Get and Set of one request +// must share (see the Cache interface contract). Used by BOTH structured +// queries (which pass one Namespace) and pipes (which pass several). +func (l *LocalCache) Key(sha string, deps []Namespace) string { + return l.versionManager.QueryKey(sha, deps) +} - val, found := l.cache.Get(cacheKey) +// Get looks up a cached query RESULT by its folded key. Returns nil, 0, nil on miss. +func (l *LocalCache) Get(_ context.Context, key string) ([]byte, time.Duration, error) { + val, found := l.cache.Get(key) if !found { return nil, 0, nil } - remaining, _ := l.cache.GetTTL(cacheKey) + remaining, _ := l.cache.GetTTL(key) return val, remaining, nil } -// Set stores a query result under the folded key for its dependency namespaces. -// Used by both structured queries and pipes. -func (l *LocalCache) Set(_ context.Context, sha string, deps []Namespace, value []byte, ttl time.Duration) error { - cacheKey := l.versionManager.QueryKey(sha, deps) - - if ok := l.cache.SetWithTTL(cacheKey, value, int64(len(value)), ttl); !ok { - return fmt.Errorf("cache admission rejected for key %q", cacheKey) +// Set stores a query result under the folded key snapshotted before the query +// ran (see Cache.Key); a version bump since the snapshot simply orphans the +// entry — no future Get computes this key. +func (l *LocalCache) Set(_ context.Context, key string, value []byte, ttl time.Duration) error { + if ok := l.cache.SetWithTTL(key, value, int64(len(value)), ttl); !ok { + return fmt.Errorf("cache admission rejected for key %q", key) } return nil } @@ -58,22 +61,52 @@ func (l *LocalCache) Set(_ context.Context, sha string, deps []Namespace, value // scope at once); a non-empty Scope bumps just that scope plus the whole-table // view. Returns the number of namespaces processed. // -// This bumps exactly what it's given. A whole-table bump already subsumes every -// per-scope bump for the same table (the table version is embedded in every -// namespace key), so a caller that knows a whole-table bump is coming should drop -// the now-redundant scope entries itself — the ingest worker does this as it -// builds the batch, where it already loops once and knows it's a single table. +// Each written table also fans out to its dependents (SetDependents) — the views +// reading it and the MV TO targets ClickHouse populates from it: a write to a +// base table whole-table-bumps each, so a pipe that folds a view's or target's +// namespace directly is evicted without the read path ever walking the graph. +// The fan-out is whole-table (the dependent's result depends on the table, not a +// scope) and deduped, so a dependent fed by several written tables bumps once. +// +// This bumps exactly what it's given (plus the fan-out). A whole-table bump already +// subsumes every per-scope bump for the same table (the table version is embedded +// in every namespace key), so a caller that knows a whole-table bump is coming +// should drop the now-redundant scope entries itself — the ingest worker does this +// as it builds the batch, where it already loops once and knows it's a single table. +// +// Any non-empty batch also bumps the DATABASE version once (DatabaseVersionTable), +// continuing the subsumption hierarchy upward: a result folding only the database +// namespace — the dependency-resolution fallback — is evicted by any write, in O(1) +// instead of folding every base table. func (l *LocalCache) Invalidate(_ context.Context, namespaces []Namespace) (uint64, error) { + bumped := make(map[string]struct{}) for _, ns := range namespaces { if ns.Scope == "" { l.versionManager.BumpTable(ns.Table) } else { l.versionManager.BumpNamespace(ns.Table, ns.Scope) } + for _, dep := range l.versionManager.DependentsOf(ns.Table) { + if _, done := bumped[dep]; done { + continue + } + bumped[dep] = struct{}{} + l.versionManager.BumpTable(dep) + } + } + if len(namespaces) > 0 { + l.versionManager.BumpTable(DatabaseVersionTable) } return uint64(len(namespaces)), nil } +// SetDependents installs the base-table -> dependents cascade used by Invalidate +// to fan a base-table write out to the views reading it and the MV targets it +// feeds. The schema registry pushes a fresh map on every content-changed refresh. +func (l *LocalCache) SetDependents(dependents map[string][]string) { + l.versionManager.SetDependents(dependents) +} + // Wait blocks until all buffered writes have been applied. // Exposed for testing; production callers rarely need this. func (l *LocalCache) Wait() { diff --git a/internal/cache/local_test.go b/internal/cache/local_test.go index 47c01fcb..812a76c4 100644 --- a/internal/cache/local_test.go +++ b/internal/cache/local_test.go @@ -15,7 +15,7 @@ func TestLocalCache_GetMiss(t *testing.T) { require.NoError(t, err) defer func() { _ = c.Close() }() - val, ttl, err := c.Get(context.Background(), "missing", []Namespace{{Table: "table"}}) + val, ttl, err := c.Get(context.Background(), c.Key("missing", []Namespace{{Table: "table"}})) assert.NoError(t, err) assert.Nil(t, val) assert.Zero(t, ttl) @@ -28,14 +28,14 @@ func TestLocalCache_SetAndGet(t *testing.T) { defer func() { _ = c.Close() }() ctx := context.Background() - deps := []Namespace{{Table: "table", Scope: "scope"}} - err = c.Set(ctx, "key1", deps, []byte("hello"), 10*time.Second) + key := c.Key("key1", []Namespace{{Table: "table", Scope: "scope"}}) + err = c.Set(ctx, key, []byte("hello"), 10*time.Second) assert.NoError(t, err) // Ristretto uses async admission — wait briefly for it to be admitted. c.Wait() - val, ttl, err := c.Get(ctx, "key1", deps) + val, ttl, err := c.Get(ctx, key) assert.NoError(t, err) assert.Equal(t, []byte("hello"), val) assert.True(t, ttl > 0, "expected positive remaining TTL") @@ -48,16 +48,16 @@ func TestLocalCache_ExpiredKey(t *testing.T) { defer func() { _ = c.Close() }() ctx := context.Background() - deps := []Namespace{{Table: "table"}} + key := c.Key("expires", []Namespace{{Table: "table"}}) // Set with very short TTL. - err = c.Set(ctx, "expires", deps, []byte("data"), 1*time.Millisecond) + err = c.Set(ctx, key, []byte("data"), 1*time.Millisecond) assert.NoError(t, err) // Ensure async admission completes, then wait for expiry. c.Wait() time.Sleep(50 * time.Millisecond) - val, _, err := c.Get(ctx, "expires", deps) + val, _, err := c.Get(ctx, key) assert.NoError(t, err) assert.Nil(t, val, "expected nil for expired key") } @@ -69,13 +69,13 @@ func TestLocalCache_Overwrite(t *testing.T) { defer func() { _ = c.Close() }() ctx := context.Background() - deps := []Namespace{{Table: "table"}} - require.NoError(t, c.Set(ctx, "key", deps, []byte("v1"), 10*time.Second)) + key := c.Key("key", []Namespace{{Table: "table"}}) + require.NoError(t, c.Set(ctx, key, []byte("v1"), 10*time.Second)) c.Wait() - require.NoError(t, c.Set(ctx, "key", deps, []byte("v2"), 10*time.Second)) + require.NoError(t, c.Set(ctx, key, []byte("v2"), 10*time.Second)) c.Wait() - val, _, err := c.Get(ctx, "key", deps) + val, _, err := c.Get(ctx, key) assert.NoError(t, err) assert.Equal(t, []byte("v2"), val) } @@ -87,14 +87,14 @@ func TestLocalCache_ZeroTTL(t *testing.T) { defer func() { _ = c.Close() }() ctx := context.Background() - deps := []Namespace{{Table: "table"}} - err = c.Set(ctx, "notimed", deps, []byte("data"), 0) + key := c.Key("notimed", []Namespace{{Table: "table"}}) + err = c.Set(ctx, key, []byte("data"), 0) assert.NoError(t, err) c.Wait() time.Sleep(10 * time.Millisecond) // arbitrary tiny sleep to see its still here after - val, ttl, err := c.Get(ctx, "notimed", deps) + val, ttl, err := c.Get(ctx, key) assert.NoError(t, err) if val != nil { assert.Equal(t, []byte("data"), val) @@ -112,12 +112,12 @@ func TestLocalCache_Invalidate(t *testing.T) { deps := []Namespace{{Table: "users", Scope: "org_1"}} // Set value - err = c.Set(ctx, "queryHash", deps, []byte("my_data"), 10*time.Second) + err = c.Set(ctx, c.Key("queryHash", deps), []byte("my_data"), 10*time.Second) assert.NoError(t, err) c.Wait() // Ensure readable - val, _, err := c.Get(ctx, "queryHash", deps) + val, _, err := c.Get(ctx, c.Key("queryHash", deps)) assert.NoError(t, err) assert.Equal(t, []byte("my_data"), val) @@ -126,14 +126,118 @@ func TestLocalCache_Invalidate(t *testing.T) { assert.NoError(t, err) assert.Equal(t, uint64(1), count) - // The folded key embeds the namespace version, which was just bumped, so this - // must now miss. - valAfter, ttlAfter, errAfter := c.Get(ctx, "queryHash", deps) + // The folded key embeds the namespace version, which was just bumped, so a + // reader folding the key now must miss. + valAfter, ttlAfter, errAfter := c.Get(ctx, c.Key("queryHash", deps)) assert.NoError(t, errAfter) assert.Nil(t, valAfter) assert.Zero(t, ttlAfter) } +// The Cache.Key contract: the folded key is snapshotted BEFORE the query runs, +// and an Invalidate landing mid-flight (between the snapshot and the Set) must +// orphan the entry — the Set lands under the pre-bump key, which no post-bump +// reader folds. Without the snapshot (folding versions at Set time instead), the +// pre-write result would be stored under the post-bump key and served as fresh +// for its whole TTL: a stale read no write could ever evict. +func TestLocalCache_MidFlightInvalidate_OrphansSet(t *testing.T) { + t.Parallel() + c, err := NewLocal(1 << 20) + require.NoError(t, err) + defer func() { _ = c.Close() }() + + ctx := context.Background() + deps := []Namespace{{Table: "events"}} + + // A read misses and snapshots its key, then starts executing the query... + key := c.Key("sha", deps) + + // ...a write lands and invalidates while the query is in flight... + _, err = c.Invalidate(ctx, deps) + require.NoError(t, err) + + // ...and the (pre-write) result is stored under the snapshot key. + require.NoError(t, c.Set(ctx, key, []byte("pre-write rows"), time.Hour)) + c.Wait() + + // A reader folding the key AFTER the write must not see the pre-write result. + val, _, err := c.Get(ctx, c.Key("sha", deps)) + assert.NoError(t, err) + assert.Nil(t, val, "a result computed before a mid-flight write must not be served after it") +} + +// Any non-empty Invalidate batch also bumps the DATABASE version: an entry +// folding only DatabaseNamespace (the pipe dependency-resolution fallback) is +// evicted by a write to ANY table — including one no registry has discovered — +// mirroring how a whole-table bump subsumes its scopes, one level up. +func TestLocalCache_Invalidate_BumpsDatabaseVersion(t *testing.T) { + t.Parallel() + c, err := NewLocal(1 << 20) + require.NoError(t, err) + defer func() { _ = c.Close() }() + + ctx := context.Background() + deps := []Namespace{DatabaseNamespace()} + + require.NoError(t, c.Set(ctx, c.Key("fallback", deps), []byte("v1"), time.Hour)) + c.Wait() + val, _, err := c.Get(ctx, c.Key("fallback", deps)) + require.NoError(t, err) + require.Equal(t, []byte("v1"), val) + + // A write to an arbitrary, never-registered table must evict it. + _, err = c.Invalidate(ctx, []Namespace{{Table: "some_table"}}) + require.NoError(t, err) + after, _, err := c.Get(ctx, c.Key("fallback", deps)) + assert.NoError(t, err) + assert.Nil(t, after, "any write must evict a database-version-keyed entry") + + // A scoped write evicts it too — the database bump is per batch, not per kind. + require.NoError(t, c.Set(ctx, c.Key("fallback", deps), []byte("v2"), time.Hour)) + c.Wait() + _, err = c.Invalidate(ctx, []Namespace{{Table: "events", Scope: "org_1"}}) + require.NoError(t, err) + after, _, err = c.Get(ctx, c.Key("fallback", deps)) + assert.NoError(t, err) + assert.Nil(t, after, "a scoped write must also evict a database-version-keyed entry") +} + +// SetDependents installs the base-table -> dependent-view cascade: a write to a +// base table must orphan an entry keyed by a VIEW namespace (what a pipe reading +// the view folds), even though the view itself is never written. A write to an +// unrelated table must not. +func TestLocalCache_Invalidate_CascadesToDependentViews(t *testing.T) { + t.Parallel() + c, err := NewLocal(1 << 20) + require.NoError(t, err) + defer func() { _ = c.Close() }() + + ctx := context.Background() + // A pipe reads a view, so its result is keyed by the VIEW's namespace. + viewDeps := []Namespace{{Table: "page_views_mv"}} + c.SetDependents(map[string][]string{"page_views": {"page_views_mv"}}) + + require.NoError(t, c.Set(ctx, c.Key("pipe", viewDeps), []byte("v1"), 10*time.Second)) + c.Wait() + val, _, err := c.Get(ctx, c.Key("pipe", viewDeps)) + require.NoError(t, err) + require.Equal(t, []byte("v1"), val) + + // A write to an unrelated table leaves the view-keyed entry intact. + _, err = c.Invalidate(ctx, []Namespace{{Table: "other"}}) + require.NoError(t, err) + still, _, err := c.Get(ctx, c.Key("pipe", viewDeps)) + require.NoError(t, err) + require.Equal(t, []byte("v1"), still, "an unrelated write must not cascade") + + // A write to the BASE table cascades to the view and orphans the entry. + _, err = c.Invalidate(ctx, []Namespace{{Table: "page_views"}}) + require.NoError(t, err) + after, _, err := c.Get(ctx, c.Key("pipe", viewDeps)) + assert.NoError(t, err) + assert.Nil(t, after, "a base-table write must cascade-invalidate the view-keyed entry") +} + // Invalidate with an empty-scope namespace bumps the whole table, which must // orphan that table's scoped entries too — not just the whole-table view. func TestLocalCache_Invalidate_WholeTable(t *testing.T) { @@ -145,9 +249,9 @@ func TestLocalCache_Invalidate_WholeTable(t *testing.T) { ctx := context.Background() scoped := []Namespace{{Table: "events", Scope: "org_1"}} - require.NoError(t, c.Set(ctx, "q", scoped, []byte("v1"), 10*time.Second)) + require.NoError(t, c.Set(ctx, c.Key("q", scoped), []byte("v1"), 10*time.Second)) c.Wait() - val, _, err := c.Get(ctx, "q", scoped) + val, _, err := c.Get(ctx, c.Key("q", scoped)) require.NoError(t, err) require.Equal(t, []byte("v1"), val) @@ -155,7 +259,7 @@ func TestLocalCache_Invalidate_WholeTable(t *testing.T) { _, err = c.Invalidate(ctx, []Namespace{{Table: "events"}}) require.NoError(t, err) - after, _, err := c.Get(ctx, "q", scoped) + after, _, err := c.Get(ctx, c.Key("q", scoped)) assert.NoError(t, err) assert.Nil(t, after, "whole-table bump must invalidate the scoped entry") } diff --git a/internal/cache/version_manager.go b/internal/cache/version_manager.go index c76b2b3c..7a28e3bc 100644 --- a/internal/cache/version_manager.go +++ b/internal/cache/version_manager.go @@ -2,6 +2,7 @@ package cache import ( "fmt" + "slices" "sort" "strings" "sync" @@ -17,6 +18,7 @@ type VersionManager struct { tableVersions map[string]uint64 // -> table_version namespaceVersions map[string]uint64 //
.. -> namespace_version + dependents map[string][]string conn *nats.Conn } @@ -27,15 +29,60 @@ func NewVersionManager(conn *nats.Conn) *VersionManager { return &VersionManager{ tableVersions: make(map[string]uint64), namespaceVersions: make(map[string]uint64), + dependents: make(map[string][]string), conn: conn, } } +// SetDependents installs the base-table -> dependents cascade (NATS-encoded on +// both sides; the views reading each table plus the MV TO targets it feeds), +// replacing any prior one. Called on each schema refresh. A nil map clears the +// cascade. The slice values are retained as-is; the caller must not mutate them +// afterward (the registry hands over a fresh map each refresh). +func (vm *VersionManager) SetDependents(dependents map[string][]string) { + vm.mu.Lock() + defer vm.mu.Unlock() + if dependents == nil { + dependents = make(map[string][]string) + } + vm.dependents = dependents +} + +// DependentsOf returns the namespaces a write to table must also bump — the views +// reading it and the MV TO targets it feeds. Nil when table feeds nothing. The +// result is a copy, so no caller can reach the manager's internal state — a +// structural guarantee instead of a "don't mutate" contract. It runs per +// invalidation batch (per flush, not per row), so the allocation is negligible. +func (vm *VersionManager) DependentsOf(table string) []string { + vm.mu.RLock() + defer vm.mu.RUnlock() + return slices.Clone(vm.dependents[table]) +} + type Namespace struct { Table string Scope string } +// DatabaseVersionTable is the reserved namespace-table name carrying the +// DATABASE-wide version: Invalidate bumps it once per (non-empty) batch, so a +// result that folds only this namespace is evicted by ANY write. It extends the +// existing subsumption hierarchy one level up — a scoped write also bumps its +// whole-table view (table subsumes scope), and any write also bumps the database +// (database subsumes table) — turning the all-base-tables fallback fold from +// O(tables) per request into O(1) with identical "any write evicts" semantics. +// "*" cannot collide with a real table: every real name is NATS-encoded +// (chsql.SafeEncodeNATS) before it reaches the cache, and the encoding emits +// only [A-Za-z0-9_] and %XX escapes. +const DatabaseVersionTable = "*" + +// DatabaseNamespace returns the namespace a caller folds to depend on the whole +// database — used by the pipe dependency-resolution fallback, whose only safe +// claim is "this result may read anything". +func DatabaseNamespace() Namespace { + return Namespace{Table: DatabaseVersionTable} +} + // namespaceKeyLocked builds the namespace-table key; caller must hold vm.mu. func (vm *VersionManager) namespaceKeyLocked(table, scope string) string { return fmt.Sprintf("%s.%d.%s", table, vm.tableVersions[table], scope) diff --git a/internal/chsql/chsql.go b/internal/chsql/chsql.go index e91ba8b5..f1cce3df 100644 --- a/internal/chsql/chsql.go +++ b/internal/chsql/chsql.go @@ -41,3 +41,17 @@ func QuoteIdent(name string) string { func BindUnsafe(name string) bool { return strings.ContainsRune(name, '?') } + +// stringEscaper escapes a ClickHouse string-literal body: a backslash becomes +// `\\` and a single quote becomes `”` (ClickHouse accepts the doubled quote). A +// single left-to-right pass, so neither replacement re-processes the other's +// output. +var stringEscaper = strings.NewReplacer(`\`, `\\`, `'`, `''`) + +// QuoteString renders a value as a single-quoted ClickHouse string literal, safe +// to inline into SQL text. It is the value twin of QuoteIdent, for the paths that +// must emit a literal rather than bind a positional `?` — notably pipe parameter +// substitution, which inlines values into the SQL string (see pipes.BindParams). +func QuoteString(s string) string { + return "'" + stringEscaper.Replace(s) + "'" +} diff --git a/internal/query/ident.go b/internal/chsql/nats.go similarity index 76% rename from internal/query/ident.go rename to internal/chsql/nats.go index 7fcba6ca..fc2a179e 100644 --- a/internal/query/ident.go +++ b/internal/chsql/nats.go @@ -1,4 +1,4 @@ -package query +package chsql import ( "bytes" @@ -8,6 +8,10 @@ import ( // SafeEncodeNATS converts any ClickHouse table name into a safe, single NATS token. // It preserves alphanumerics and underscores, but percent-encodes everything else. +// It is the single encoder shared by every side of cache invalidation — the ingest +// worker (write side), the pipe/structured-query handlers (read side), and the +// schema registry's dependency cascade — so all three build identical namespace +// keys for the same table. func SafeEncodeNATS(raw string) string { var buf bytes.Buffer for i := 0; i < len(raw); i++ { diff --git a/internal/query/ident_test.go b/internal/chsql/nats_test.go similarity index 99% rename from internal/query/ident_test.go rename to internal/chsql/nats_test.go index 24e6cb34..9c530bbe 100644 --- a/internal/query/ident_test.go +++ b/internal/chsql/nats_test.go @@ -1,4 +1,4 @@ -package query +package chsql import ( "testing" diff --git a/internal/chsql/quote_test.go b/internal/chsql/quote_test.go new file mode 100644 index 00000000..290e3618 --- /dev/null +++ b/internal/chsql/quote_test.go @@ -0,0 +1,17 @@ +package chsql + +import "testing" + +func TestQuoteString(t *testing.T) { + tests := []struct{ in, want string }{ + {"acme", "'acme'"}, + {"a'b", "'a''b'"}, + {"a\\b", "'a\\\\b'"}, + {"org_id = '1'", "'org_id = ''1'''"}, + } + for _, tt := range tests { + if got := QuoteString(tt.in); got != tt.want { + t.Errorf("QuoteString(%q) = %q, want %q", tt.in, got, tt.want) + } + } +} diff --git a/internal/discovery/deps_test.go b/internal/discovery/deps_test.go new file mode 100644 index 00000000..151f576f --- /dev/null +++ b/internal/discovery/deps_test.go @@ -0,0 +1,602 @@ +package discovery + +import ( + "context" + "io" + "log/slog" + "strings" + "testing" + + "github.com/ClickHouse/clickhouse-go/v2/lib/driver" + "github.com/Wave-RF/WaveHouse/internal/chsql" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func tbl(name string, cols ...string) *TableSchema { + ts := &TableSchema{Name: name} + for _, c := range cols { + ts.Columns = append(ts.Columns, Column{Name: c, Type: "UInt64"}) + } + return ts +} + +// IsKnown reports whether a first-level name is safe to fold directly: a base table +// or a view that flattens cleanly to base tables. An unfoldable view (a source it +// can't resolve) and an unknown name are not — the caller TTL-caps those. +func TestIsKnown(t *testing.T) { + tables := []*TableSchema{tbl("base_a", "x"), tbl("base_b", "x"), tbl("v_norm", "x"), tbl("mv1", "x"), tbl("mv2", "x"), tbl("mv_bad", "x")} + viewSources := map[string][]string{ + "v_norm": {"base_a"}, // normal view over a base table + "mv1": {"base_a", "base_b"}, // MV over two sources + "mv2": {"mv1"}, // chained: mv2 -> mv1 -> {base_a, base_b} + "mv_bad": {"ghost"}, // a view over an UNDISCOVERED source: unfoldable + } + sr := NewSchemaRegistryForTest(tables, viewSources) + + known := []string{"base_a", "base_b", "v_norm", "mv1", "mv2"} + for _, n := range known { + assert.True(t, sr.IsKnown(n), "%s should be foldable", n) + } + notKnown := []string{"mv_bad", "ghost", "who_dis"} + for _, n := range notKnown { + assert.False(t, sr.IsKnown(n), "%s must not be foldable", n) + } +} + +// A view we KNOW is a view (engine = View) but whose definition did not parse — +// present in `views`, absent from mvSources — must be reported not-known, never +// mistaken for a base table (which would fold a version nothing maintains). It sits +// in `tables` too (system.columns lists view columns), which is the trap. +func TestIsKnown_UnparsedViewNotFoldable(t *testing.T) { + sr := NewSchemaRegistryForTest([]*TableSchema{tbl("weird_view", "x"), tbl("base", "x")}, map[string][]string{}) + sr.tables["weird_view"].isView = true // a view with no mapped edges (unparsed definition) + + assert.False(t, sr.IsKnown("weird_view"), "an unparsed view must not be foldable") + assert.True(t, sr.IsKnown("base"), "a real base table is foldable") +} + +// computeChangedViews evicts a view redefined in place AND the downstream closure +// of views that transitively read it — staleness no base-table write would catch. +func TestComputeChangedViews(t *testing.T) { + enc := chsql.SafeEncodeNATS + view := func(asSelect string, sources ...string) *tableInfo { + return &tableInfo{isView: true, asSelect: asSelect, sources: sources} + } + base := &tableInfo{schema: &TableSchema{}} + + tests := []struct { + name string + old, new map[string]*tableInfo + want []string // pre-encode names, in sorted order + }{ + { + name: "no change yields nothing", + old: map[string]*tableInfo{"base": base, "v": view("SELECT * FROM base", "base")}, + new: map[string]*tableInfo{"base": base, "v": view("SELECT * FROM base", "base")}, + want: nil, + }, + { + name: "direct redefinition only", + old: map[string]*tableInfo{"base": base, "v": view("SELECT a FROM base", "base")}, + new: map[string]*tableInfo{"base": base, "v": view("SELECT b FROM base", "base")}, + want: []string{"v"}, + }, + { + name: "redefining the middle view also evicts the top (the #1 fix)", + old: map[string]*tableInfo{ + "base": base, + "v_mid": view("SELECT a FROM base", "base"), "v_top": view("SELECT * FROM v_mid", "v_mid"), + }, + new: map[string]*tableInfo{ + "base": base, + "v_mid": view("SELECT b FROM base", "base"), "v_top": view("SELECT * FROM v_mid", "v_mid"), + }, + want: []string{"v_mid", "v_top"}, + }, + { + name: "deep chain fans all the way up", + old: map[string]*tableInfo{ + "base": base, + "v1": view("X", "base"), "v2": view("q2", "v1"), "v3": view("q3", "v2"), + }, + new: map[string]*tableInfo{ + "base": base, + "v1": view("Y", "base"), "v2": view("q2", "v1"), "v3": view("q3", "v2"), + }, + want: []string{"v1", "v2", "v3"}, + }, + { + name: "a view cycle terminates", + old: map[string]*tableInfo{"a": view("A", "b"), "b": view("B", "a")}, + new: map[string]*tableInfo{"a": view("A2", "b"), "b": view("B", "a")}, + want: []string{"a", "b"}, + }, + { + name: "a brand-new downstream reader is still evicted (harmless)", + old: map[string]*tableInfo{"base": base, "v_mid": view("SELECT a FROM base", "base")}, + new: map[string]*tableInfo{ + "base": base, "v_mid": view("SELECT b FROM base", "base"), + "v_new": view("SELECT * FROM v_mid", "v_mid"), + }, + want: []string{"v_mid", "v_new"}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + var want []string + for _, n := range tt.want { + want = append(want, enc(n)) + } + assert.Equal(t, want, computeChangedViews(tt.old, tt.new)) + }) + } +} + +// Dependents is the write-side cascade: base table -> the views a write to it must +// also evict, transitively, both NATS-encoded. It is the precomputed reverse of the +// view->source edges; an unfoldable view contributes no entry. +func TestDependents(t *testing.T) { + enc := chsql.SafeEncodeNATS + tests := []struct { + name string + tables []*TableSchema + viewSources map[string][]string + want map[string][]string // base -> dependent views (pre-encode) + }{ + { + name: "single view over a base", + tables: []*TableSchema{tbl("base", "x"), tbl("v", "x")}, + viewSources: map[string][]string{"v": {"base"}}, + want: map[string][]string{"base": {"v"}}, + }, + { + name: "chained views both cascade off the base", + tables: []*TableSchema{tbl("base", "x"), tbl("mv1", "x"), tbl("mv2", "x")}, + viewSources: map[string][]string{"mv1": {"base"}, "mv2": {"mv1"}}, + want: map[string][]string{"base": {"mv1", "mv2"}}, + }, + { + name: "view over two sources cascades off each", + tables: []*TableSchema{tbl("a", "x"), tbl("b", "x"), tbl("v", "x")}, + viewSources: map[string][]string{"v": {"a", "b"}}, + want: map[string][]string{"a": {"v"}, "b": {"v"}}, + }, + { + name: "unfoldable view yields no cascade", + tables: []*TableSchema{tbl("base", "x"), tbl("v", "x")}, + viewSources: map[string][]string{"v": {"ghost"}}, + want: map[string][]string{}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + sr := NewSchemaRegistryForTest(tt.tables, tt.viewSources) + got := sr.Dependents() + want := make(map[string][]string, len(tt.want)) + for base, views := range tt.want { + ev := make([]string, len(views)) + for i, v := range views { + ev[i] = enc(v) + } + want[enc(base)] = ev + } + require.Len(t, got, len(want)) + for base, views := range want { + assert.ElementsMatch(t, views, got[base], "cascade for %s", base) + } + }) + } +} + +// A materialized view's TO target is an ordinary base table ClickHouse populates +// whenever the MV's source takes an insert — nothing else ever writes it, so the +// write-side cascade must carry a source write through the MV into the target, +// to the foldable views over it, and through chained MVs — or a pipe reading the +// target directly would sit at full TTL with nothing ever bumping its version. +// Trigger edges (attached), not the resolved read set, drive the flow: they stay +// authoritative even for an MV whose definition didn't resolve (defExternal). +func TestDependents_MVTargets(t *testing.T) { + enc := chsql.SafeEncodeNATS + base := func() *tableInfo { return &tableInfo{schema: &TableSchema{}} } + + tests := []struct { + name string + tables map[string]*tableInfo + want map[string][]string // base -> full dependent set (pre-encode) + }{ + { + name: "source write reaches the MV target", + tables: map[string]*tableInfo{ + "src": {schema: &TableSchema{}, attached: []string{"mv"}}, + "tgt": base(), + "mv": {isView: true, sources: []string{"src"}, toTarget: "tgt"}, + }, + want: map[string][]string{"src": {"mv", "tgt"}}, + }, + { + name: "views over the target ride along", + tables: map[string]*tableInfo{ + "src": {schema: &TableSchema{}, attached: []string{"mv"}}, + "tgt": base(), + "mv": {isView: true, sources: []string{"src"}, toTarget: "tgt"}, + "v_t": {isView: true, sources: []string{"tgt"}}, + }, + want: map[string][]string{ + "src": {"mv", "tgt", "v_t"}, + "tgt": {"v_t"}, + }, + }, + { + name: "chained MVs propagate transitively", + tables: map[string]*tableInfo{ + "src": {schema: &TableSchema{}, attached: []string{"mv1"}}, + "tgt1": {schema: &TableSchema{}, attached: []string{"mv2"}}, + "tgt2": base(), + "mv1": {isView: true, sources: []string{"src"}, toTarget: "tgt1"}, + "mv2": {isView: true, sources: []string{"tgt1"}, toTarget: "tgt2"}, + }, + want: map[string][]string{ + "src": {"mv1", "tgt1", "mv2", "tgt2"}, + "tgt1": {"mv2", "tgt2"}, + }, + }, + { + name: "an unresolvable MV definition still feeds its target (trigger edges are authoritative)", + tables: map[string]*tableInfo{ + "src": {schema: &TableSchema{}, attached: []string{"mv"}}, + "tgt": base(), + "mv": {isView: true, sources: []string{"src"}, toTarget: "tgt", defExternal: true}, + }, + // mv itself is unfoldable (its readers TTL-cap), so only the target cascades. + want: map[string][]string{"src": {"tgt"}}, + }, + { + name: "a missing or non-base target contributes no edge", + tables: map[string]*tableInfo{ + "src": {schema: &TableSchema{}, attached: []string{"mv", "mv2"}}, + "mv": {isView: true, sources: []string{"src"}, toTarget: "ghost"}, + "mv2": {isView: true, sources: []string{"src"}, toTarget: "v"}, + "v": {isView: true, sources: []string{"src"}}, + }, + want: map[string][]string{"src": {"mv", "mv2", "v"}}, + }, + { + name: "an MV targeting its own source terminates", + tables: map[string]*tableInfo{ + "src": {schema: &TableSchema{}, attached: []string{"mv"}}, + "mv": {isView: true, sources: []string{"src"}, toTarget: "src"}, + }, + want: map[string][]string{"src": {"mv"}}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := buildDeps(tt.tables) + want := make(map[string][]string, len(tt.want)) + for b, deps := range tt.want { + ed := make([]string, len(deps)) + for i, d := range deps { + ed[i] = enc(d) + } + want[enc(b)] = ed + } + require.Len(t, got, len(want)) + for b, deps := range want { + assert.ElementsMatch(t, deps, got[b], "cascade for %s", b) + } + }) + } +} + +// parseMVTarget reads the TO target off ClickHouse's normalized DDL renderings — +// the exact strings a 26.6 server produces (pinned live by the integration +// MV-target test). It must never scan past the column list / ENGINE clause / +// SELECT body, so definition text (a TTL ... TO DISK, a string literal) can't +// fabricate a target. +func TestParseMVTarget(t *testing.T) { + tests := []struct { + name string + q string + db, table string + ok bool + }{ + { + name: "plain TO target", + q: "CREATE MATERIALIZED VIEW mvtest.mv TO mvtest.tgt (`id` UInt64, `val` String) AS SELECT id, val FROM mvtest.src", + db: "mvtest", table: "tgt", ok: true, + }, + { + name: "quoted target with an embedded dot stays one identifier", + q: "CREATE MATERIALIZED VIEW mvtest.mv TO mvtest.`dot.ted` (`id` UInt64) AS SELECT id FROM mvtest.src", + db: "mvtest", table: "dot.ted", ok: true, + }, + { + name: "quoted target with a space", + q: "CREATE MATERIALIZED VIEW mvtest.mv TO mvtest.`we ird` (`id` UInt64) AS SELECT id FROM mvtest.src", + db: "mvtest", table: "we ird", ok: true, + }, + { + name: "backslash-escaped backtick decodes", + q: "CREATE MATERIALIZED VIEW mvtest.mv TO mvtest.`tick\\`name` (`id` UInt64) AS SELECT id FROM mvtest.src", + db: "mvtest", table: "tick`name", ok: true, + }, + { + name: "refreshable MV: REFRESH clause precedes TO", + q: "CREATE MATERIALIZED VIEW mvtest.mv_refresh REFRESH EVERY 1 HOUR TO mvtest.tgt2 (`id` UInt64, `val` String) DEFINER = default SQL SECURITY DEFINER AS SELECT id, val FROM mvtest.src", + db: "mvtest", table: "tgt2", ok: true, + }, + { + name: "implicit-inner MV (no TO; column list first)", + q: "CREATE MATERIALIZED VIEW mvtest.mv_inner (`id` UInt64) ENGINE = MergeTree ORDER BY id AS SELECT id FROM mvtest.src", + ok: false, + }, + { + name: "TTL ... TO DISK after ENGINE is never a target", + q: "CREATE MATERIALIZED VIEW db.m ENGINE = MergeTree ORDER BY id TTL ts + INTERVAL 1 DAY TO DISK 'cold' AS SELECT id FROM db.src", + ok: false, + }, + { + name: "plain view has no TO clause", + q: "CREATE VIEW test.v (`x` UInt64) AS SELECT x FROM base", + ok: false, + }, + { + name: "a view literally named TO is skipped as a quoted identifier", + q: "CREATE MATERIALIZED VIEW db.`TO` TO db.tgt (`x` UInt64) AS SELECT x FROM db.src", + db: "db", table: "tgt", ok: true, + }, + { + name: "a string literal containing TO does not arm", + q: "CREATE MATERIALIZED VIEW db.m REFRESH EVERY 1 HOUR SETTINGS note = 'go TO disk' TO db.t (`x` UInt64) AS SELECT 1", + db: "db", table: "t", ok: true, + }, + { + name: "unqualified target maps to the configured database", + q: "CREATE MATERIALIZED VIEW mv TO tgt AS SELECT 1", + db: "", table: "tgt", ok: true, + }, + { + name: "unterminated quote is malformed", + q: "CREATE MATERIALIZED VIEW db.mv TO db.`broken AS SELECT 1", + ok: false, + }, + {name: "empty", q: "", ok: false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + db, table, ok := parseMVTarget(tt.q) + assert.Equal(t, tt.ok, ok) + assert.Equal(t, tt.db, db) + assert.Equal(t, tt.table, table) + }) + } +} + +// mkInfos assembles a tableInfo map the way Refresh would, for contentHash tests: +// base tables carry a schema; a name in sources/defs is marked a view. +func mkInfos(tables map[string]*TableSchema, sources map[string][]string, defs map[string]string) map[string]*tableInfo { + m := make(map[string]*tableInfo) + for n, ts := range tables { + m[n] = &tableInfo{schema: ts} + } + mark := func(n string) *tableInfo { + t := m[n] + if t == nil { + t = &tableInfo{} + m[n] = t + } + t.isView = true + return t + } + for n, s := range sources { + mark(n).sources = s + } + for n, d := range defs { + mark(n).asSelect = d + } + return m +} + +func TestContentHash_DeterministicAndSensitive(t *testing.T) { + tables := map[string]*TableSchema{ + "a": tbl("a", "x", "y"), + "b": tbl("b", "z"), + } + mv := map[string][]string{"v": {"a"}} + defs := map[string]string{"v": "SELECT x, y FROM a"} + + base := contentHash(mkInfos(tables, mv, defs)) + + // Determinism: a re-run over equal content (maps built independently) matches. + assert.Equal(t, base, contentHash(mkInfos( + map[string]*TableSchema{"b": tbl("b", "z"), "a": tbl("a", "x", "y")}, + map[string][]string{"v": {"a"}}, + map[string]string{"v": "SELECT x, y FROM a"}, + )), "hash must be stable regardless of map iteration order") + + // Sensitivity: each input changing flips the hash. + assert.NotEqual(t, base, contentHash(mkInfos(map[string]*TableSchema{"a": tbl("a", "x", "y", "extra"), "b": tbl("b", "z")}, mv, defs)), "a new column must change the hash") + assert.NotEqual(t, base, contentHash(mkInfos(tables, map[string][]string{"v": {"a", "b"}}, defs)), "a new view->source edge must change the hash") + assert.NotEqual(t, base, contentHash(mkInfos(tables, mv, map[string]string{"v": "SELECT x, y FROM a WHERE z > 1"})), "a redefined view body (same sources) must change the hash") + + // An MV re-pointed at a different TO target (DROP + CREATE with an identical + // SELECT) changes neither columns, sources, nor asSelect — only this fold. + withTarget := func(target string) uint64 { + infos := mkInfos(tables, mv, defs) + infos["v"].toTarget = target + return contentHash(infos) + } + assert.NotEqual(t, base, withTarget("t1"), "gaining a TO target must change the hash") + assert.NotEqual(t, withTarget("t1"), withTarget("t2"), "a re-pointed TO target must change the hash") +} + +// scriptedConn returns canned rows for the system.columns and system.tables +// queries (distinguished by SQL substring), and can be told to fail either, +// driving Refresh's change-detection and atomic-no-op behavior without ClickHouse. +type scriptedConn struct { + driver.Conn + columnRows [][]any // [table, name, type, default_kind] + tableRows [][]any // [name, engine, as_select, dependencies_table([]string), create_table_query] + explainSources []string // tables ResolveTables (EXPLAIN QUERY TREE) reports for a view + explainCalls int // EXPLAIN queries issued (pass-2 runs) — proves the no-op skip + failTables bool +} + +func (c *scriptedConn) Query(_ context.Context, query string, _ ...any) (driver.Rows, error) { + switch { + case strings.HasPrefix(query, "EXPLAIN"): + // Stand in for ClickHouse resolving a view's definition: emit one + // "table_name: .
" line per configured source. + c.explainCalls++ + rows := make([][]any, len(c.explainSources)) + for i, t := range c.explainSources { + rows[i] = []any{" TABLE id: 1, table_name: test." + t} + } + return &scriptedRows{rows: rows}, nil + case strings.Contains(query, "system.columns"): + return &scriptedRows{rows: c.columnRows}, nil + case strings.Contains(query, "system.tables"): + if c.failTables { + return nil, assert.AnError + } + return &scriptedRows{rows: c.tableRows}, nil + } + return &emptyRows{}, nil +} + +type scriptedRows struct { + driver.Rows + rows [][]any + i int +} + +func (r *scriptedRows) Next() bool { return r.i < len(r.rows) } +func (r *scriptedRows) Close() error { return nil } +func (r *scriptedRows) Err() error { return nil } +func (r *scriptedRows) ColumnTypes() []driver.ColumnType { return nil } +func (r *scriptedRows) Scan(dest ...any) error { + row := r.rows[r.i] + r.i++ + for k := range dest { + switch d := dest[k].(type) { + case *string: + d2, _ := row[k].(string) + *d = d2 + case *[]string: + if row[k] == nil { + *d = nil + } else { + *d = row[k].([]string) + } + } + } + return nil +} + +func newScriptedRegistry(conn *scriptedConn) *SchemaRegistry { + return NewSchemaRegistry(conn, "test", 0, slog.New(slog.NewTextHandler(io.Discard, nil))) +} + +// Refresh notifies onRefresh only when the schema content actually changed, hands +// over the up-to-date cascade, and flags a redefined view (same sources, new body) +// in ChangedViews so the cache can evict it directly. +func TestRefresh_NotifiesOnChange(t *testing.T) { + enc := chsql.SafeEncodeNATS + conn := &scriptedConn{ + columnRows: [][]any{{"base", "x", "UInt64", ""}}, + tableRows: [][]any{{"v", "View", "SELECT x FROM base", nil, "CREATE VIEW test.v (`x` UInt64) AS SELECT x FROM base"}}, + explainSources: []string{"base"}, + } + sr := newScriptedRegistry(conn) + ctx := context.Background() + + var snaps []DependencySnapshot + sr.SetOnRefresh(func(s DependencySnapshot) { snaps = append(snaps, s) }) + + require.NoError(t, sr.Refresh(ctx)) + require.Len(t, snaps, 1, "first refresh must notify") + assert.ElementsMatch(t, []string{enc("v")}, snaps[0].Cascade[enc("base")], "cascade must map base -> v") + assert.Empty(t, snaps[0].ChangedViews, "no views changed on the first refresh") + explainsAfterFirst := conn.explainCalls + require.Equal(t, 1, explainsAfterFirst, "first refresh resolves the one view via EXPLAIN") + + require.NoError(t, sr.Refresh(ctx)) + require.Len(t, snaps, 1, "an identical refresh must NOT notify") + assert.Equal(t, explainsAfterFirst, conn.explainCalls, "an identical refresh must NOT re-run EXPLAIN — pass 2 is skipped on a no-op") + + // Redefine the view body (same source set) — must be detected and reported. + conn.tableRows = [][]any{{"v", "View", "SELECT x FROM base WHERE x > 1", nil, "CREATE VIEW test.v (`x` UInt64) AS SELECT x FROM base WHERE x > 1"}} + require.NoError(t, sr.Refresh(ctx)) + require.Len(t, snaps, 2, "a view-body redefinition must notify") + assert.Greater(t, conn.explainCalls, explainsAfterFirst, "a redefinition changes the cheap signals, so EXPLAIN re-runs") + assert.Equal(t, []string{enc("v")}, snaps[1].ChangedViews, "the redefined view must be flagged for direct eviction") + + // The view still resolves to its base after the redefinition. + assert.True(t, sr.IsKnown("v")) + assert.ElementsMatch(t, []string{enc("v")}, sr.Dependents()[enc("base")]) +} + +// An MV's TO target rides the cascade end-to-end: pass 1 parses the target off +// create_table_query, buildDeps carries a source write into it, and re-pointing +// the MV at a different target (DROP + CREATE, SAME SELECT — only the DDL +// differs) is a content change that re-fires onRefresh with the new cascade — +// the one change only the toTarget hash fold can catch. +func TestRefresh_MVTargetCascade(t *testing.T) { + enc := chsql.SafeEncodeNATS + mvRow := func(target string) []any { + return []any{ + "mv", "MaterializedView", "SELECT x FROM src", nil, + "CREATE MATERIALIZED VIEW test.mv TO test." + target + " (`x` UInt64) AS SELECT x FROM src", + } + } + conn := &scriptedConn{ + columnRows: [][]any{{"src", "x", "UInt64", ""}, {"tgt_a", "x", "UInt64", ""}, {"tgt_b", "x", "UInt64", ""}}, + tableRows: [][]any{ + {"src", "MergeTree", "", []string{"mv"}, "CREATE TABLE test.src (`x` UInt64) ENGINE = MergeTree ORDER BY x"}, + {"tgt_a", "MergeTree", "", nil, "CREATE TABLE test.tgt_a (`x` UInt64) ENGINE = MergeTree ORDER BY x"}, + {"tgt_b", "MergeTree", "", nil, "CREATE TABLE test.tgt_b (`x` UInt64) ENGINE = MergeTree ORDER BY x"}, + mvRow("tgt_a"), + }, + explainSources: []string{"src"}, + } + sr := newScriptedRegistry(conn) + var snaps []DependencySnapshot + sr.SetOnRefresh(func(s DependencySnapshot) { snaps = append(snaps, s) }) + + require.NoError(t, sr.Refresh(context.Background())) + require.Len(t, snaps, 1) + assert.ElementsMatch(t, []string{enc("mv"), enc("tgt_a")}, snaps[0].Cascade[enc("src")], + "a source write must bump the MV and its TO target") + + conn.tableRows[3] = mvRow("tgt_b") + require.NoError(t, sr.Refresh(context.Background())) + require.Len(t, snaps, 2, "a re-pointed TO target must notify") + assert.ElementsMatch(t, []string{enc("mv"), enc("tgt_b")}, snaps[1].Cascade[enc("src")]) + assert.Empty(t, snaps[1].ChangedViews, "the MV body is unchanged — no direct eviction") +} + +func TestRefresh_AtomicNoOpOnViewDiscoveryError(t *testing.T) { + conn := &scriptedConn{ + columnRows: [][]any{{"base", "x", "UInt64", ""}}, + tableRows: [][]any{{"v", "View", "SELECT x FROM base", nil, "CREATE VIEW test.v (`x` UInt64) AS SELECT x FROM base"}}, + explainSources: []string{"base"}, + } + sr := newScriptedRegistry(conn) + ctx := context.Background() + + notifies := 0 + sr.SetOnRefresh(func(DependencySnapshot) { notifies++ }) + + require.NoError(t, sr.Refresh(ctx)) + require.Equal(t, 1, notifies) + + // The tables query now errors; the whole refresh must be a no-op. + conn.failTables = true + require.Error(t, sr.Refresh(ctx)) + assert.Equal(t, 1, notifies, "a failed refresh must not notify") + + // Last-good tree is fully intact: the view still resolves to its base. + assert.True(t, sr.IsKnown("v"), "the previous good tree must survive a transient discovery error") + assert.ElementsMatch(t, []string{chsql.SafeEncodeNATS("v")}, sr.Dependents()[chsql.SafeEncodeNATS("base")]) +} diff --git a/internal/discovery/discovery.go b/internal/discovery/discovery.go index e3b90c5b..390092db 100644 --- a/internal/discovery/discovery.go +++ b/internal/discovery/discovery.go @@ -3,12 +3,16 @@ package discovery import ( "context" "fmt" + "hash/fnv" "io" "log/slog" + "slices" + "strings" "sync" "time" "github.com/ClickHouse/clickhouse-go/v2/lib/driver" + "github.com/Wave-RF/WaveHouse/internal/chsql" "go.opentelemetry.io/otel" ) @@ -40,14 +44,106 @@ func (ts *TableSchema) ColumnNames() []string { return names } -// SchemaRegistry discovers and caches ClickHouse table schemas. +// tableInfo is everything the registry knows about ONE name in the database. Base +// tables and views share this one map (system.columns lists a view's columns too, +// so a view carries a schema as well as its view-only fields). One entry replaces +// what used to be five parallel maps. Rebuilt wholesale on every successful Refresh +// and guarded by SchemaRegistry.mu. +type tableInfo struct { + schema *TableSchema // columns in physical order; drives Get/List/ColumnNames + isView bool // engine is a View family — a view, which ingest never writes + sources []string // a view's immediate source tables (nil for a base table) + asSelect string // a view's SELECT text — diffed across refreshes to catch a redefinition + // attached: the materialized views an INSERT into this table fires + // (system.tables dependencies_table — the write-side trigger edges). Distinct + // from a view's sources: a join/dictGet in an MV's definition is a read that + // never fires it, and these edges stay authoritative even when EXPLAIN can't + // resolve the definition. buildDeps walks them to carry a write through an MV + // into its TO target. + attached []string + // toTarget: a materialized view's same-database TO target table, parsed from + // create_table_query (parseMVTarget). ClickHouse populates the target on every + // insert the view fires on — but the target is an ordinary base table whose + // version no ingest write would otherwise bump, so buildDeps must fan a source + // write out to it or a pipe reading the target directly would serve stale at + // full TTL. Empty for non-MVs, implicit-inner targets (".inner…", which the + // registry never tracks), and cross-database targets (reads of those are + // External and TTL-cap). + toTarget string + foldable bool // derived: a view that flattens cleanly to known base tables + defPruned bool // EXPLAIN dead-branch-pruned this view's definition: its source set may be incomplete, so it is never foldable (readers TTL-cap) + // defExternal: this view's definition reads something no table version can + // watch (a table function, a cross-database table, a non-local dictionary + // source). The cascade can't observe writes to it, so the view is never + // foldable and its readers TTL-cap instead of trusting eviction. + defExternal bool +} + +// SchemaRegistry discovers and caches ClickHouse table schemas, and derives the +// pipe-cache dependency tree from them. type SchemaRegistry struct { conn driver.Conn database string refreshInterval time.Duration logger *slog.Logger - mu sync.RWMutex - tables map[string]*TableSchema + // refreshMu serializes whole Refresh invocations (auto-refresh ticker, boot + // retry, and the manual /v1/schema/refresh trigger can otherwise overlap). mu alone only guards the state swap; without total ordering, an + // older refresh finishing last could install its older cascade over a newer + // one, and computeChangedViews would diff against whichever swap happened to + // land in between. Serializing also stops racing callers from duplicating the + // expensive pass-2 EXPLAIN work — a queued duplicate re-runs pass 1, sees an + // unchanged metaHash, and returns cheaply. Held across onRefresh too, so the + // cascade installs into the cache in refresh order; the callback only touches + // the cache and pipe memo, never Refresh, so it cannot re-enter. + refreshMu sync.Mutex + mu sync.RWMutex + // tables maps every name in the configured database — base tables AND views — + // to its schema, view-ness, sources, definition, and derived foldability (see + // tableInfo). The single source of per-name truth; rebuilt and swapped atomically + // on each successful Refresh. Guarded by mu. + tables map[string]*tableInfo + // cascade maps a base table to the dependent names a write to it must ALSO + // invalidate — the views reading it and the MV TO targets ClickHouse populates + // from it, transitively — both sides NATS-encoded to match the read/write + // sides. It is precomputed from the view->source and trigger->target edges: + // the graph walk happens once at refresh (buildDeps), never per request. + // Pushed into the cache via onRefresh so Cache.Invalidate can fan a base-table + // bump out to its dependents. Guarded by mu. + cascade map[string][]string + // onRefresh, if set, is invoked after every CONTENT-CHANGED refresh with the new + // dependency snapshot, so the cache can install the updated cascade and bump any + // redefined views. Set once via SetOnRefresh. Guarded by mu. + onRefresh func(DependencySnapshot) + // metaHash fingerprints the CHEAP schema signals — columns, view-ness, view + // definition text, and reverse dependencies_table edges — read WITHOUT EXPLAIN. + // Each view's EXPLAIN resolution is a deterministic function of these, so when + // metaHash is unchanged (and the last resolve fully succeeded) Refresh skips the + // per-view EXPLAINs entirely and keeps the last-good tree. Guarded by mu. + metaHash uint64 + // contentHash fingerprints the FULL resolved tree (metaHash's inputs plus the + // EXPLAIN-resolved source edges). onRefresh fires only when it changes, so a + // refresh resolving to the same tree neither re-pushes the cascade nor bumps + // anything. hasRefreshed distinguishes the first refresh from a steady-state one. + // Guarded by mu. + contentHash uint64 + // lastResolveOK is false when the previous refresh's EXPLAIN pass had any failure + // (a broken view, or ClickHouse dropping mid-refresh); it forces the next refresh + // to re-run the EXPLAINs even when metaHash is unchanged, so a transient failure + // self-heals instead of freezing missing edges behind the hash. Guarded by mu. + lastResolveOK bool + hasRefreshed bool +} + +// DependencySnapshot is the per-refresh hand-off from the schema registry to the +// cache. Cascade is the full base-table -> dependents map (NATS-encoded; the views +// reading each table plus the MV TO targets it feeds) to install; ChangedViews are +// the (NATS-encoded) views whose definition changed this refresh and must be +// invalidated directly (a redefinition with the same sources changes results but +// no base-table write would signal it). Both are ready to use as-is — the cache +// neither parses nor re-encodes them. +type DependencySnapshot struct { + Cascade map[string][]string + ChangedViews []string } // NewSchemaRegistry creates a registry that discovers schemas from system.columns. @@ -57,17 +153,278 @@ func NewSchemaRegistry(conn driver.Conn, database string, refreshInterval time.D database: database, refreshInterval: refreshInterval, logger: logger, - tables: make(map[string]*TableSchema), + tables: make(map[string]*tableInfo), } } -// Refresh queries system.columns and rebuilds the in-memory schema cache. +// Refresh re-reads the schema from ClickHouse, recomputes the derived foldable/ +// cascade sets, and atomically swaps the in-memory caches. It runs in two passes so +// the expensive work is gated on change: +// +// - Pass 1 reads the CHEAP signals (system.columns, plus system.tables metadata: +// view-ness, definitions, reverse dependency edges) and fingerprints them +// (metaHash). If nothing moved since the last refresh — and that refresh fully +// resolved — Refresh stops here: no per-view EXPLAINs, no cascade rebuild, no +// onRefresh. This is the steady-state path. +// - Pass 2 runs only on a change: it resolves each view's sources via EXPLAIN (the +// per-view round-trips), rebuilds the cascade, swaps the tree, and — when the +// resolved tree actually differs — notifies onRefresh so the cache installs the +// new cascade and bumps any redefined views. +// +// It is ALL-OR-NOTHING: a read failure aborts with nothing swapped, leaving the +// last-good tree intact. A pass-2 EXPLAIN failure (a broken view, or ClickHouse +// dropping mid-refresh) is recorded (lastResolveOK) so the next refresh re-resolves +// even when the cheap signals are unchanged, rather than freezing missing edges. func (sr *SchemaRegistry) Refresh(ctx context.Context) error { + // Serialize refreshes end-to-end (see refreshMu): concurrent callers queue and + // run in order, and a duplicate that queued behind an identical refresh + // short-circuits on the unchanged metaHash in pass 1. + sr.refreshMu.Lock() + defer sr.refreshMu.Unlock() + tracer := otel.GetTracerProvider().Tracer("wavehouse-discovery") ctx, span := tracer.Start(ctx, "SchemaRegistry.Refresh") defer span.End() - rows, err := sr.conn.Query(ctx, + infos, err := sr.discoverColumns(ctx) + if err != nil { + return err + } + // Pass 1 (cheap, no EXPLAIN): view-ness, definitions, and reverse dependency + // edges — enough to fingerprint the schema and decide whether anything moved. + viewDefs, err := sr.discoverViewMeta(ctx, infos) + if err != nil { + // Atomic: a metadata-read failure aborts the whole refresh with nothing + // swapped, so the previous good tree stays in effect. Retried next tick. + return err + } + + // Each view's source edges are a deterministic function of the definitions and + // column schema just read, so metaHash captures every input to the EXPLAIN step. + // If it's unchanged AND the last resolve fully succeeded, the EXPLAIN results + // would be identical — skip the per-view round-trips and keep the last-good tree. + metaHash := contentHash(infos) + sr.mu.RLock() + skip := sr.hasRefreshed && metaHash == sr.metaHash && sr.lastResolveOK + sr.mu.RUnlock() + if skip { + sr.logger.Info("schema registry refreshed", "tables", len(infos), "changed", false) + return nil + } + + // Pass 2 (expensive): something changed, or a prior EXPLAIN failed — resolve the + // view source edges via EXPLAIN and rebuild the derived sets. + resolveOK := sr.resolveViewSources(ctx, infos, viewDefs) + treeHash := contentHash(infos) // now folds in the EXPLAIN-resolved edges + cascade := buildDeps(infos) // sets foldable on each view; returns the cascade + sr.mu.Lock() + changed := !sr.hasRefreshed || treeHash != sr.contentHash + // changedViews drives direct eviction of views redefined in place (and the + // downstream views built on them) — staleness no base-table write would catch. + var changedViews []string + if sr.hasRefreshed { + changedViews = computeChangedViews(sr.tables, infos) + } + sr.tables = infos + sr.cascade = cascade + sr.metaHash = metaHash + sr.contentHash = treeHash + sr.lastResolveOK = resolveOK + sr.hasRefreshed = true + onRefresh := sr.onRefresh + sr.mu.Unlock() + + if changed && onRefresh != nil { + // Fire outside the lock: the callback reaches into the cache, and must not + // be able to deadlock against a concurrent reader holding sr.mu.RLock. + onRefresh(DependencySnapshot{Cascade: cascade, ChangedViews: changedViews}) + } + sr.logger.Info("schema registry refreshed", "tables", len(infos), "changed", changed) + return nil +} + +// buildDeps walks the view->source and trigger->target graphs ONCE — the only +// place they are walked, done here at refresh so neither read nor write does it +// per call. It marks each view foldable (flattens cleanly to known base tables) +// directly on its tableInfo, and returns the write-side cascade: each base table +// mapped to the foldable views reading it and the MV TO targets an insert into it +// populates (both transitively), NATS-encoded to match the worker (write) and +// handler (read) sides. Only sets the foldable bit on the entries; no lock, no query. +func buildDeps(tables map[string]*tableInfo) map[string][]string { + cascade := make(map[string][]string) + + // flatten returns the base tables `name` transitively reads and whether that + // flatten is COMPLETE (every leaf is a real base table). A view recurses into its + // sources; a base table is itself; a view with no edges (unparsed), an edge-only + // entry with no schema, or an unknown name is incomplete. path is the cycle guard. + var flatten func(name string, path map[string]struct{}) ([]string, bool) + flatten = func(name string, path map[string]struct{}) ([]string, bool) { + if _, cycle := path[name]; cycle { + return nil, false + } + t := tables[name] + if t == nil { + return nil, false // unknown name + } + if t.isView { + if len(t.sources) == 0 { + return nil, false // a view we couldn't map to sources + } + path[name] = struct{}{} + defer delete(path, name) + baseSet := make(map[string]struct{}) + // A defPruned/defExternal definition never flattens completely (see + // tableInfo) — the view stays unfoldable and readers TTL-cap; its known + // bases still feed the cascade walk. + complete := !t.defPruned && !t.defExternal + for _, s := range t.sources { + sb, sok := flatten(s, path) + if !sok { + complete = false + } + for _, b := range sb { + baseSet[b] = struct{}{} + } + } + bases := make([]string, 0, len(baseSet)) + for b := range baseSet { + bases = append(bases, b) + } + return bases, complete + } + if t.schema != nil { + return []string{name}, true // a real base table ingest writes + } + return nil, false // tracked only as an edge target — no schema, not a base table + } + + for name, t := range tables { + if !t.isView { + continue + } + bases, complete := flatten(name, make(map[string]struct{})) + if !complete { + continue // unfoldable: IsKnown reports it not-known so the caller TTL-caps + } + t.foldable = true + ev := chsql.SafeEncodeNATS(name) + for _, b := range bases { + eb := chsql.SafeEncodeNATS(b) + cascade[eb] = append(cascade[eb], ev) + } + } + + // Write flow through materialized-view targets: an insert into a table fires + // its attached MVs, and ClickHouse writes each MV's output into its TO target + // — so a write to the source IS a write to the target. The target is an + // ordinary base table (IsKnown folds it at full TTL) that no ingest write + // would otherwise bump, so without these edges a pipe reading the target + // directly serves stale for its whole TTL. writesTo is the one-step flow; + // the walk below closes it over chains (mv1: a->t1; mv2 attached to t1: ->t2) + // and carries each reached target's own foldable-view cascade along. Trigger + // edges (attached), not the EXPLAIN-resolved read set, drive this: a join or + // dictGet in an MV's definition never fires it, and the trigger edges stay + // authoritative even for a view whose definition didn't resolve. + writesTo := make(map[string][]string) + for name, t := range tables { + for _, m := range t.attached { + mt := tables[m] + if mt == nil || mt.toTarget == "" { + continue + } + if tt := tables[mt.toTarget]; tt == nil || tt.isView || tt.schema == nil { + continue // a target we don't track as a base table (dropped, or exotic) + } + writesTo[name] = append(writesTo[name], mt.toTarget) + } + } + for name, t := range tables { + if t.isView || t.schema == nil || len(writesTo[name]) == 0 { + continue // write events originate at base tables + } + eb := chsql.SafeEncodeNATS(name) + seen := map[string]struct{}{name: {}} // cycle guard (an MV targeting its own source) + queue := slices.Clone(writesTo[name]) + for len(queue) > 0 { + tgt := queue[0] + queue = queue[1:] + if _, dup := seen[tgt]; dup { + continue + } + seen[tgt] = struct{}{} + et := chsql.SafeEncodeNATS(tgt) + cascade[eb] = append(cascade[eb], et) + cascade[eb] = append(cascade[eb], cascade[et]...) // the foldable views over the target + queue = append(queue, writesTo[tgt]...) // chained MVs attached to the target + } + } + + for b := range cascade { + slices.Sort(cascade[b]) + cascade[b] = slices.Compact(cascade[b]) + } + return cascade +} + +// computeChangedViews returns the NATS-encoded views to evict after a refresh: the +// views whose definition (as_select) changed in place since oldTables, PLUS the +// downstream closure of views that transitively read them. A redefined view's +// readers produce stale results too, but their own bodies are unchanged and the +// base-keyed cascade can't reach them — so the redefinition must fan out here. +// Returns nil when nothing changed. `changed` doubles as the visited set, so a +// view cycle terminates. A newly-discovered downstream reader is harmless (nothing +// folds its version yet), so the closure needs no present-before filter. +func computeChangedViews(oldTables, newInfos map[string]*tableInfo) []string { + changed := map[string]struct{}{} + for name, t := range newInfos { + if t.asSelect == "" { + continue // only a view has a definition to compare + } + if old := oldTables[name]; old != nil && old.asSelect != "" && old.asSelect != t.asSelect { + changed[name] = struct{}{} + } + } + if len(changed) == 0 { + return nil + } + + readers := map[string][]string{} // name -> views that DIRECTLY read it + for name, t := range newInfos { + if t.isView { + for _, s := range t.sources { + readers[s] = append(readers[s], name) + } + } + } + queue := make([]string, 0, len(changed)) + for n := range changed { + queue = append(queue, n) + } + for len(queue) > 0 { + n := queue[0] + queue = queue[1:] + for _, r := range readers[n] { + if _, seen := changed[r]; seen { + continue + } + changed[r] = struct{}{} + queue = append(queue, r) + } + } + + out := make([]string, 0, len(changed)) + for name := range changed { + out = append(out, chsql.SafeEncodeNATS(name)) + } + slices.Sort(out) + return out +} + +// discoverColumns reads system.columns and builds the per-name map with schemas +// populated (views included — system.columns lists their columns too). +func (sr *SchemaRegistry) discoverColumns(ctx context.Context) (map[string]*tableInfo, error) { + rows, err := sr.conn.Query( + ctx, `SELECT table, name, type, default_kind FROM system.columns WHERE database = ? @@ -76,43 +433,353 @@ func (sr *SchemaRegistry) Refresh(ctx context.Context) error { sr.database, ) if err != nil { - return fmt.Errorf("query system.columns: %w", err) + return nil, fmt.Errorf("query system.columns: %w", err) } defer func() { _ = rows.Close() }() - tables := make(map[string]*TableSchema) + infos := make(map[string]*tableInfo) for rows.Next() { var tableName, colName, colType, defaultKind string if err := rows.Scan(&tableName, &colName, &colType, &defaultKind); err != nil { - return fmt.Errorf("scan column row: %w", err) + return nil, fmt.Errorf("scan column row: %w", err) } - ts, ok := tables[tableName] + t, ok := infos[tableName] if !ok { - ts = &TableSchema{Name: tableName} - tables[tableName] = ts + t = &tableInfo{schema: &TableSchema{Name: tableName}} + infos[tableName] = t } - col := Column{ + t.schema.Columns = append(t.schema.Columns, Column{ Name: colName, Type: colType, IsNullable: isNullable(colType), HasDefault: defaultKind != "", + }) + } + return infos, rows.Err() +} + +// viewDef pairs a view's name with its SELECT text, carried from pass 1 +// (discoverViewMeta) to pass 2 (resolveViewSources) for EXPLAIN resolution. +type viewDef struct{ name, asSelect string } + +// discoverViewMeta is Refresh's PASS 1: it reads the CHEAP view signals from +// system.tables — view-ness, the SELECT definition, and the reverse +// dependencies_table edges (a source -> its attached MV) — and returns the view +// definitions for pass 2 to resolve. No EXPLAIN runs here, so Refresh can fingerprint +// the schema and skip pass 2 entirely when nothing changed. +func (sr *SchemaRegistry) discoverViewMeta(ctx context.Context, infos map[string]*tableInfo) ([]viewDef, error) { + // get returns the entry for name, creating a schema-less one if the name appeared + // only as an edge target (so a view/edge is never silently dropped; a schema-less + // entry is never treated as a base table — see buildDeps/IsKnown). + get := func(name string) *tableInfo { + t := infos[name] + if t == nil { + t = &tableInfo{} + infos[name] = t + } + return t + } + + var toResolve []viewDef + rows, err := sr.conn.Query(ctx, + "SELECT name, engine, as_select, dependencies_table, create_table_query FROM system.tables WHERE database = ?", + sr.database) + if err != nil { + return nil, fmt.Errorf("query system.tables: %w", err) + } + for rows.Next() { + var name, engine, asSelect, createQuery string + var dependents []string + if err := rows.Scan(&name, &engine, &asSelect, &dependents, &createQuery); err != nil { + _ = rows.Close() + return nil, fmt.Errorf("scan table row: %w", err) + } + t := get(name) + if strings.Contains(engine, "View") { + t.isView = true + // A materialized view's TO target, rendered only in its DDL (system.tables + // grew first-class target_* columns in ClickHouse 26.6 — too new to require; + // an unknown column would fail schema discovery wholesale on ≤26.5). A + // cross-database target is skipped: reads of it are External and TTL-cap. + if db, target, ok := parseMVTarget(createQuery); ok && (db == "" || db == sr.database) { + t.toTarget = target + } + } + if asSelect != "" { + t.asSelect = asSelect + toResolve = append(toResolve, viewDef{name, asSelect}) + } + // Reverse edges: this row is a SOURCE table; each dependent is an attached MV + // an insert here fires. Kept both ways — as the MV's source (the read-side + // tree) and as this table's trigger (the write-side flow into MV targets). + t.attached = dependents + for _, d := range dependents { + get(d).sources = append(get(d).sources, name) + } + } + rerr := rows.Err() + _ = rows.Close() + if rerr != nil { + return nil, rerr + } + return toResolve, nil +} + +// resolveViewSources is Refresh's PASS 2 — the expensive one, run only when pass 1's +// cheap fingerprint changed. It asks ClickHouse (EXPLAIN QUERY TREE, see +// ResolveTables) for each view's source tables, unions them with the reverse edges +// pass 1 recorded, and de-dups. A view whose definition can't be resolved (it reads a +// table function, a cross-database table) simply gets no forward edge — a pipe reading +// it then over-resolves or goes stale per the documented rules, never a silent wrong +// answer. It returns false if ANY EXPLAIN errored (a broken view, or ClickHouse +// dropping mid-refresh) so Refresh re-resolves next tick instead of trusting a partial +// tree behind an unchanged metaHash. +func (sr *SchemaRegistry) resolveViewSources(ctx context.Context, infos map[string]*tableInfo, toResolve []viewDef) bool { + allResolved := true + for _, v := range toResolve { + res, perr := ResolveTables(ctx, sr.conn, sr.database, v.asSelect) + if perr != nil { + sr.logger.Debug("view definition not resolvable via EXPLAIN; relying on dependencies_table", "view", v.name, "error", perr) + allResolved = false + continue + } + if res.Pruned { + // Pruning dropped a table from this view's OWN definition (a constant- + // false arm authored into the view) — never foldable, readers TTL-cap + // (see tableInfo.defPruned). + infos[v.name].defPruned = true + sr.logger.Debug("view definition was dead-branch pruned; treating as unfoldable", "view", v.name) + } + if res.External { + // The view reads something no table version can watch — never foldable, + // readers TTL-cap (see tableInfo.defExternal). + infos[v.name].defExternal = true + sr.logger.Debug("view definition reads an external source; treating as unfoldable", "view", v.name) + } + infos[v.name].sources = append(infos[v.name].sources, res.Tables...) + } + + for _, t := range infos { + if len(t.sources) > 1 { + slices.Sort(t.sources) + t.sources = slices.Compact(t.sources) + } + } + return allResolved +} + +// parseMVTarget extracts a materialized view's TO target from ClickHouse's +// normalized create_table_query rendering: the first standalone TO keyword, +// scanned back-quote- and string-literal-aware so quoted content can never arm +// it, and bounded by the first '(' or AS/ENGINE keyword — the column list, the +// SELECT body, and an implicit-inner engine clause — so nothing inside a +// definition (a TTL … TO DISK, a literal, an alias) can be mistaken for the +// clause. Identifiers decode per ClickHouse's writeBackQuotedString (backslash +// escapes); keywords are matched exactly, as the normalized rendering uppercases +// them. ok=false when there is no TO clause (a plain view, an implicit-inner MV) +// or the rendering is malformed; db is empty for an unqualified target. +// +// This is the version-portable seam: selecting system.tables' target_database/ +// target_table (added in ClickHouse 26.6) would break schema discovery on every +// older server, so the target is read off the canonical DDL instead — swap to +// the columns once the supported floor passes 26.6. Failure posture matches the +// EXPLAIN couplings (explain.go): a rendering this doesn't recognize yields no +// target edge, and the tests/integration MV-target test pins the parse against +// the ClickHouse version we ship. +func parseMVTarget(q string) (db, table string, ok bool) { + i, n := 0, len(q) + // readIdent consumes one identifier at i — back-quoted (escapes resolved) or + // bare — reporting ok=false for anything else (incl. an unterminated quote). + readIdent := func() (string, bool) { + if i < n && q[i] == '`' { + var b strings.Builder + for i++; i < n; i++ { + switch q[i] { + case '\\': + if i+1 < n { + i++ + b.WriteByte(q[i]) + } + case '`': + i++ + return b.String(), true + default: + b.WriteByte(q[i]) + } + } + return "", false + } + start := i + for i < n && isIdentByte(q[i]) { + i++ + } + return q[start:i], i > start + } + + for i < n { + switch c := q[i]; { + case c == '`': // a quoted identifier (e.g. the view's own name) — skip whole + if _, idOK := readIdent(); !idOK { + return "", "", false + } + case c == '\'': // a string literal (defensive; skip whole, honoring escapes) + for i++; i < n; i++ { + if q[i] == '\\' { + i++ + } else if q[i] == '\'' { + i++ + break + } + } + case c == '(': + return "", "", false // the column list — every TO clause precedes it + case isIdentByte(c): + w, _ := readIdent() + switch w { + case "AS", "ENGINE": + return "", "", false // the SELECT body / an implicit-inner engine clause + case "TO": + for i < n && (q[i] == ' ' || q[i] == '\t' || q[i] == '\n' || q[i] == '\r') { + i++ + } + first, firstOK := readIdent() + if !firstOK { + return "", "", false + } + if i < n && q[i] == '.' { + i++ + second, secondOK := readIdent() + if !secondOK { + return "", "", false + } + return first, second, true + } + return "", first, true + } + default: + i++ // whitespace, digits/punctuation in a REFRESH clause, etc. } - ts.Columns = append(ts.Columns, col) } + return "", "", false +} + +// isIdentByte reports whether c can appear in a bare (unquoted) ClickHouse +// identifier or keyword — the token alphabet parseMVTarget scans by. +func isIdentByte(c byte) bool { + return c == '_' || c >= '0' && c <= '9' || c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' +} + +// Database returns the configured ClickHouse database this registry tracks. +func (sr *SchemaRegistry) Database() string { return sr.database } + +// contentHash fingerprints everything dependency resolution derives from the schema: +// per name, its columns, view-ness, source edges, and definition. Names are sorted +// so the hash is stable regardless of map iteration order (an unsorted hash would +// flap and re-fire onRefresh every refresh). Conservative by construction — a false +// "changed" only costs a recompute; a real change can never present as "unchanged" +// because every relevant input is folded in. +func contentHash(infos map[string]*tableInfo) uint64 { + h := fnv.New64a() + write := func(s string) { _, _ = h.Write([]byte(s)); _, _ = h.Write([]byte{0}) } + + names := make([]string, 0, len(infos)) + for n := range infos { + names = append(names, n) + } + slices.Sort(names) + for _, n := range names { + t := infos[n] + write("n") + write(n) + if t.schema != nil { + for _, c := range t.schema.Columns { + write(c.Name) + write(c.Type) + if c.HasDefault { + write("d") + } + } + } + if t.isView { + write("v") + } + // Sort/de-dup a copy before hashing so the fingerprint is order-independent: + // pass 1 leaves reverse edges in system.tables row order, which would + // otherwise flap metaHash between refreshes (pass 2's edges are already sorted). + srcs := slices.Clone(t.sources) + slices.Sort(srcs) + for _, s := range slices.Compact(srcs) { + write("s") + write(s) + } + if t.asSelect != "" { + write("q") + write(t.asSelect) + } + // An MV re-pointed at a different TO target (DROP + CREATE with the same + // SELECT) changes neither columns, sources, nor asSelect — only this. + if t.toTarget != "" { + write("t") + write(t.toTarget) + } + } + return h.Sum64() +} +// IsKnown reports whether name is SAFE to fold directly into a cache key — i.e. +// its version is reliably maintained on writes. That is true for a real base +// table (ingest writes it; an MV TO target is bumped by writes to the MV's +// source via the cascade) and for a foldable view (likewise cascade-bumped), +// but NOT for an unfoldable view (unparsed definition, or a cross-database/ +// unknown source) nor an unknown name: the caller treats those as unresolved and +// TTL-caps the result rather than trust an unmaintained version. Pure in-memory +// lookup over the map built during Refresh — no query. +func (sr *SchemaRegistry) IsKnown(name string) bool { + sr.mu.RLock() + defer sr.mu.RUnlock() + t := sr.tables[name] + if t == nil { + return false // unknown name + } + if t.isView { + return t.foldable // a view: safe only if it flattens cleanly to base tables + } + return t.schema != nil // a real base table ingest writes +} + +// Dependents returns a copy of the current cascade: each (NATS-encoded) base table +// mapped to the (NATS-encoded) dependents a write to it must also invalidate — the +// views reading it and the MV TO targets it feeds. The cache installs this so a +// base-table bump fans out to them. main pushes it through onRefresh; an +// out-of-band caller (e.g. a test wiring its own cache) can pull the current +// value directly. +func (sr *SchemaRegistry) Dependents() map[string][]string { + sr.mu.RLock() + defer sr.mu.RUnlock() + out := make(map[string][]string, len(sr.cascade)) + for k, v := range sr.cascade { + out[k] = append([]string(nil), v...) + } + return out +} + +// SetOnRefresh registers the callback invoked after every content-changed Refresh +// with the new dependency snapshot. Set once at wiring time, before the periodic +// refresh loop starts, so the cache stays in lock-step with the schema. +func (sr *SchemaRegistry) SetOnRefresh(fn func(DependencySnapshot)) { sr.mu.Lock() - sr.tables = tables + sr.onRefresh = fn sr.mu.Unlock() - sr.logger.Info("schema registry refreshed", "tables", len(tables)) - - return nil } // Get returns the schema for a table, or nil if not found. func (sr *SchemaRegistry) Get(name string) *TableSchema { sr.mu.RLock() defer sr.mu.RUnlock() - return sr.tables[name] + if t := sr.tables[name]; t != nil { + return t.schema + } + return nil } // List returns all discovered table schemas. @@ -120,8 +787,10 @@ func (sr *SchemaRegistry) List() []*TableSchema { sr.mu.RLock() defer sr.mu.RUnlock() result := make([]*TableSchema, 0, len(sr.tables)) - for _, ts := range sr.tables { - result = append(result, ts) + for _, t := range sr.tables { + if t.schema != nil { + result = append(result, t.schema) + } } return result } @@ -203,12 +872,32 @@ func isNullable(chType string) bool { // NewSchemaRegistryFromMap creates a SchemaRegistry pre-loaded with the given // table schemas. Intended for testing — no ClickHouse connection is required. func NewSchemaRegistryFromMap(tables []*TableSchema) *SchemaRegistry { - m := make(map[string]*TableSchema, len(tables)) + m := make(map[string]*tableInfo, len(tables)) for _, t := range tables { - m[t.Name] = t + m[t.Name] = &tableInfo{schema: t} } return &SchemaRegistry{ tables: m, logger: slog.New(slog.NewTextHandler(io.Discard, nil)), } } + +// NewSchemaRegistryForTest creates a registry pre-loaded with table schemas AND +// view->source edges, with the foldable/cascade sets derived from them, for testing +// dependency resolution without a ClickHouse connection. Marked as already +// refreshed, mimicking a once-refreshed registry. +func NewSchemaRegistryForTest(tables []*TableSchema, viewSources map[string][]string) *SchemaRegistry { + sr := NewSchemaRegistryFromMap(tables) + for name, srcs := range viewSources { + t := sr.tables[name] + if t == nil { + t = &tableInfo{} + sr.tables[name] = t + } + t.isView = true + t.sources = srcs + } + sr.cascade = buildDeps(sr.tables) + sr.hasRefreshed = true + return sr +} diff --git a/internal/discovery/discovery_test.go b/internal/discovery/discovery_test.go index 67558be5..0ae9c1ae 100644 --- a/internal/discovery/discovery_test.go +++ b/internal/discovery/discovery_test.go @@ -155,7 +155,9 @@ func TestRetryRefresh_SucceedsOnFirstAttempt(t *testing.T) { // regression (the misbehaviour would sleep `initialBackoff` = 1h). assert.Less(t, time.Since(start), 250*time.Millisecond, "should not have slept") assert.Equal(t, int32(0), atomic.LoadInt32(&attempts), "onAttempt should only fire on failure") - assert.Equal(t, int32(1), conn.calls.Load(), "exactly one Query call expected") + // A successful Refresh makes two queries: system.columns (schema) + system.tables + // (the materialized-view -> source dependency map). + assert.Equal(t, int32(2), conn.calls.Load(), "two Query calls on a successful Refresh") } // TestRetryRefresh_RetriesUntilSuccess verifies the loop keeps trying through @@ -173,7 +175,9 @@ func TestRetryRefresh_RetriesUntilSuccess(t *testing.T) { }) require.NoError(t, err) - assert.Equal(t, int32(3), conn.calls.Load(), "two failures + one success") + // Two failed attempts (1 query each — system.columns errors before the + // dependency query) + the succeeding attempt's two queries (columns + tables). + assert.Equal(t, int32(4), conn.calls.Load(), "two failures + one success") require.Len(t, captured, 2) assert.ErrorIs(t, captured[0], errFirst) assert.ErrorIs(t, captured[1], errSecond) diff --git a/internal/discovery/explain.go b/internal/discovery/explain.go new file mode 100644 index 00000000..11ca98ad --- /dev/null +++ b/internal/discovery/explain.go @@ -0,0 +1,448 @@ +package discovery + +import ( + "context" + "fmt" + "slices" + "strings" + + "github.com/ClickHouse/clickhouse-go/v2/lib/driver" + "github.com/Wave-RF/WaveHouse/internal/chsql" + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/metric" +) + +// prunedTablesCounter counts table dependencies dropped by dead-branch pruning +// (a parameter-gated UNION arm whose WHERE folded to a constant false). It lets +// operators measure how often the precision optimization fires. A no-op until a +// meter provider is configured, so it is safe to use unconditionally. +var prunedTablesCounter, _ = otel.Meter("wavehouse-pipes").Int64Counter( + "wavehouse_pipe_dep_tables_pruned_total", + metric.WithDescription("Table dependencies dropped by dead-branch (constant-false) pruning during pipe resolution"), +) + +// Resolution is what ResolveTables reads off ClickHouse's query analysis. +type Resolution struct { + // Tables is the local (configured-database) table/view names the query reads. + Tables []string + // Pruned reports that dead-branch pruning dropped at least one table. The + // caller TTL-caps a pruned result as a belt-and-suspenders bound: pruning is + // fuzzed and fails safe toward keeping, but it is the one precision + // optimization that REMOVES a table from the dependency set, so a hypothetical + // wrong prune would otherwise cache an unwatched result at full TTL. + Pruned bool + // External reports that the query reads something no table version can watch: + // a table function (s3()/numbers()/url()/…), a table in a DIFFERENT database, + // or a dictionary whose source isn't a local table (file/http/executable, or + // cross-database). Such reads contribute nothing to Tables, so writes to them + // can never evict — the caller TTL-caps the result (UnresolvedDepsTTLCap) so + // it self-expires instead of serving stale for a full TTL. + External bool +} + +// ResolveTables asks ClickHouse which tables a query reads, by running +// EXPLAIN QUERY TREE and reading the table nodes off its own analysis. This +// replaces static SQL parsing entirely — ClickHouse resolves the query, we just +// read the answer. +// +// EXPLAIN QUERY TREE is deliberately the PRE-optimization form: it keeps a table +// even when the query is answered from metadata (SELECT count() FROM t), and it +// does NOT drop empty tables or trivial counts the way EXPLAIN PLAN / query_log do. +// So it never UNDER-resolves, the invariant we need. +// +// On top of the raw tree we apply two refinements (parseExplainTables): +// +// - joinGet/dictGet — read-bearing functions whose target is a NAME constant +// rather than a TABLE node — are tracked (the dict via its backing table). Both +// are real reads a write must evict. +// - Dead-branch pruning — a parameter-gated UNION arm whose WHERE folded to a +// constant false (e.g. {{source}}='web' makes the 'mobile' arm 'web'='mobile') +// reads nothing, so its tables are dropped. This is data-INDEPENDENT (pure +// constant folding) and so safe, BUT it makes the resolved set depend on the +// bound parameter values — the caller must therefore cache the result per BOUND +// query, not per template, or a different binding would reuse a wrongly-pruned +// set (see api.PipesHandler.resolveDeps). Pruning fails safe toward KEEPING: any +// unrecognized shape leaves the table tracked (over-resolve), never dropped. +// +// An EXPLAIN error — a write/DDL pipe, a missing table, or an unreachable +// ClickHouse — is returned so the caller can fall back to the database-version +// namespace rather than risk an under-resolution. +func ResolveTables(ctx context.Context, conn driver.Conn, database, sql string) (Resolution, error) { + rows, err := conn.Query(ctx, "EXPLAIN QUERY TREE "+sql) + if err != nil { + return Resolution{}, fmt.Errorf("explain query tree: %w", err) + } + defer func() { _ = rows.Close() }() + + var lines []string + for rows.Next() { + var ln string + if err := rows.Scan(&ln); err != nil { + return Resolution{}, fmt.Errorf("scan explain row: %w", err) + } + lines = append(lines, ln) + } + if err := rows.Err(); err != nil { + return Resolution{}, err + } + + tables, dicts, pruned, external := parseExplainTables(lines, database) + if pruned > 0 { + prunedTablesCounter.Add(ctx, int64(pruned)) + } + // A dictGet target is the dictionary, not a table; map each to its backing + // ClickHouse table (a CLICKHOUSE-sourced dict in the configured db). A + // file/http/executable dict has no local table — an external read, like a + // table function. Only queried when the pipe actually reads a dictionary. + if len(dicts) > 0 { + srcs, dictExternal, err := resolveDictSources(ctx, conn, database, dicts) + if err != nil { + return Resolution{}, err + } + external = external || dictExternal + tables = append(tables, srcs...) + } + slices.Sort(tables) + return Resolution{Tables: slices.Compact(tables), Pruned: pruned > 0, External: external}, nil +} + +// capture tracks whether the previous line armed extraction of the next string +// constant — joinGet/dictGet render their target as the function's first argument, +// a NAME constant, rather than a TABLE node. +const ( + capNone = iota + capJoin // next string constant is a Join-engine TABLE name + capDict // next string constant is a DICTIONARY name +) + +// qscope is one QUERY node in the tree, tracked by indentation. dead is set when +// the node's WHERE folded to a constant false (the arm reads nothing). Scopes are +// retained for the whole parse so a TABLE seen BEFORE its arm's WHERE can still have +// its deadness resolved at the end. +type qscope struct { + indent int + dead bool +} + +// tableOccurrence records one TABLE node plus the QUERY scope stack enclosing it, +// so its deadness (any enclosing arm dead) is evaluated after the full parse. +type tableOccurrence struct { + table string + scopes []int // indices into the scopes slice +} + +// parseExplainTables extracts dependency names from EXPLAIN QUERY TREE output, +// returning (tables, dicts, prunedCount, external): +// +// - tables: configured-database table/view names from TABLE nodes (minus +// dead-branch arms) plus the first (NAME) argument of joinGet/joinGetOrNull. +// - dicts: configured-database dictionary names referenced via a dictGet-family +// function, for the caller to map to their backing tables. +// - prunedCount: distinct tables dropped because every occurrence was in a +// constant-false arm (for the metric). +// - external: the query reads something no local table version can watch — a +// TABLE_FUNCTION node (s3()/numbers()/url()/…), a TABLE node that doesn't +// resolve to a configured-database name (a cross-database table, or an +// unrecognized rendering — conservative: anything we can't attribute counts +// as external, so the caller TTL-caps rather than trusts), or a +// cross-database joinGet/dictGet target. +// +// Structure (verified against ClickHouse 26.6): a UNION arm is a QUERY node; its +// tables sit under JOIN TREE and its filter under a sibling WHERE. ClickHouse folds +// a parameter-gated predicate, so a dead arm's WHERE child is exactly +// "CONSTANT … UInt64_0" — and ONLY the line immediately after WHERE is inspected, so +// a real predicate ("WHERE → FUNCTION greater") or a literal compare +// ("WHERE → FUNCTION equals(x, 0)") is never mistaken for a dead arm. A table is +// dropped only when EVERY occurrence is under a dead arm; appearing in any live arm +// keeps it. Kept pure (no ClickHouse) so the brittle string handling is unit-tested. +func parseExplainTables(lines []string, database string) (tables, dicts []string, prunedCount int, external bool) { + var scopes []qscope + var stack []int // indices into scopes for currently-open QUERY nodes + var occs []tableOccurrence + dictSet := map[string]struct{}{} + joinTables := map[string]struct{}{} // joinGet targets: always kept (safe), never pruned + capture := capNone + afterWhere := false + + for _, raw := range lines { + trimmed := strings.TrimLeft(raw, " ") + if trimmed == "" { + continue + } + indent := len(raw) - len(trimmed) + + // Close any QUERY scopes we have exited (this line is at or shallower than them). + for len(stack) > 0 && scopes[stack[len(stack)-1]].indent >= indent { + stack = stack[:len(stack)-1] + } + + // The single line after a WHERE is its condition root. If that root is the + // folded constant false, this arm (the enclosing QUERY) is dead. Checking only + // this one line is what makes pruning safe: a real predicate or an "x = 0" + // compare renders as a FUNCTION here, not a bare CONSTANT. + // + // CH-VERSION COUPLING: the literal below is ClickHouse 26.6's rendering of a + // false-folded predicate. A future version could fold to a different type + // (Bool_0/UInt8_0) or drop the trailing comma; either way this test stops + // matching and pruning silently STOPS — which fails SAFE (the arm's tables + // stay tracked, over-resolve, never a wrong prune). The regression signal is + // tests/integration's pruning test, which asserts a parameter-gated UNION + // actually prunes against the ClickHouse version we ship, plus the + // wavehouse_pipe_dep_tables_pruned_total metric going flat in production. + if afterWhere { + afterWhere = false + if len(stack) > 0 && strings.Contains(raw, "constant_value: UInt64_0,") { + scopes[stack[len(stack)-1]].dead = true + } + // Fall through: the WHERE condition root may itself be a joinGet/dictGet + // (e.g. WHERE joinGet(...)), which still needs tracking. + } + + switch { + case strings.HasPrefix(trimmed, "QUERY id:"): + scopes = append(scopes, qscope{indent: indent}) + stack = append(stack, len(scopes)-1) + capture = capNone + continue + case trimmed == "WHERE" || strings.HasPrefix(trimmed, "WHERE "): + afterWhere = true + capture = capNone + continue + } + + // A table function (s3()/numbers()/url()/…) is its own node type: a read no + // local table version can watch, so it marks the resolution external. + if strings.HasPrefix(trimmed, "TABLE_FUNCTION ") { + external = true + capture = capNone + continue + } + + // TABLE node — record with its enclosing scope stack (deadness resolved later). + // Gate on the node type ("TABLE ...") so a string-literal value that merely + // contains "table_name:" (a projected label, or a bound parameter value) can't + // be mistaken for a table reference. + if strings.HasPrefix(trimmed, "TABLE ") { + tracked := false + if _, after, found := strings.Cut(raw, "table_name: "); found { + // table_name is NOT always the last field on the line: FINAL appends + // ", final: 1" and SAMPLE ", final: 0, sample_size: …". Keep only the + // qualified name up to the first comma, else the modifier text leaks in. + name, _, _ := strings.Cut(after, ",") + if db, table, ok := strings.Cut(strings.TrimSpace(name), "."); ok && db == database { + occs = append(occs, tableOccurrence{table: table, scopes: append([]int(nil), stack...)}) + tracked = true + } + } + // A TABLE node that doesn't resolve to a configured-database name — a + // cross-database table, or a rendering we don't recognize — is a read we + // can't version-watch: external, conservatively. + if !tracked { + external = true + } + capture = capNone + continue + } + + // joinGet/dictGet: arm capture of the next string constant (the target name). + // The matches are PREFIX matches ("function_name: dictGet" also hits + // dictGetOrDefault, dictGetHierarchy, dictGetChildren, …) and that is + // deliberate: every dictGet*-family function reads the dictionary, so there + // is no dictGet* we want to exclude. dictHas and dictIsIn are the only + // read-bearing dict functions outside that prefix. + switch { + case strings.Contains(raw, "function_name: joinGet"): + capture = capJoin + continue + case strings.Contains(raw, "function_name: dictGet"), + strings.Contains(raw, "function_name: dictHas"), + strings.Contains(raw, "function_name: dictIsIn"): + capture = capDict + continue + } + if capture != capNone { + if val, isStr, found := constantString(raw); found { + if isStr { + if db, name, ok := strings.Cut(val, "."); ok { + if db == database { + addCaptured(capture, name, joinTables, dictSet) + } else { + // A cross-database joinGet/dictGet target: same class of + // unwatchable read as a cross-database TABLE node. + external = true + } + } else { + addCaptured(capture, val, joinTables, dictSet) // bare name -> configured db + } + } + capture = capNone + } + } + } + + // Resolve table deadness: a table is kept if it has at least one occurrence with + // no dead enclosing arm. A table dropped here (all occurrences dead) is counted. + tableSet := map[string]struct{}{} + seen := map[string]struct{}{} + for _, o := range occs { + seen[o.table] = struct{}{} + dead := false + for _, si := range o.scopes { + if scopes[si].dead { + dead = true + break + } + } + if !dead { + tableSet[o.table] = struct{}{} + } + } + for t := range seen { + if _, kept := tableSet[t]; !kept { + prunedCount++ + } + } + for t := range joinTables { + tableSet[t] = struct{}{} + } + return setSlice(tableSet), setSlice(dictSet), prunedCount, external +} + +// addCaptured routes a captured NAME to the table or dict set per the armed kind. +func addCaptured(kind int, name string, tables, dicts map[string]struct{}) { + switch kind { + case capJoin: + tables[name] = struct{}{} + case capDict: + dicts[name] = struct{}{} + } +} + +// constantString parses a "constant_value: " line. found reports the line is a +// constant node; isStr reports the value is a quoted String (the only kind that can +// be a table/dict name) and val is its unquoted content. A non-string constant +// (e.g. "UInt64_1") returns isStr=false so the caller stops capturing. +func constantString(ln string) (val string, isStr, found bool) { + _, after, ok := strings.Cut(ln, "constant_value: ") + if !ok { + return "", false, false + } + after = strings.TrimSpace(after) + if !strings.HasPrefix(after, "'") { + return "", false, true // e.g. UInt64_0 — a non-name constant + } + if i := strings.IndexByte(after[1:], '\''); i >= 0 { + return after[1 : 1+i], true, true + } + return "", false, true // unterminated quote — malformed, ignore +} + +// resolveDictSources maps the named dictionaries to their backing ClickHouse +// tables. system.dictionaries.source renders a CLICKHOUSE source as +// "ClickHouse: .
"; only a source table in the configured database is +// trackable (a write to it evicts). Other source kinds (file/http/executable) and +// cross-database sources yield no local table, like a table function. +// +// A NOT_LOADED dictionary reports an EMPTY source — under ClickHouse's default lazy +// loading a dict stays unloaded until first used, so resolving a dictGet pipe before +// the dict's first load would silently drop its backing table (→ stale reads on +// writes to it). So any referenced-but-unloaded dict is force-loaded once, then +// re-read. Best-effort: a dict that can't load is left untracked rather than failing +// the whole resolution. +// +// external reports that at least one referenced dictionary yielded no trackable +// local table — a file/http/executable or cross-database source, a dict that +// couldn't load, or one missing from system.dictionaries entirely. Writes to such +// a source can never evict, so the caller TTL-caps the result. +func resolveDictSources(ctx context.Context, conn driver.Conn, database string, dicts []string) (out []string, external bool, err error) { + want := make(map[string]struct{}, len(dicts)) + for _, d := range dicts { + want[d] = struct{}{} + } + + readSources := func() (map[string]string, error) { + rows, err := conn.Query(ctx, "SELECT name, source FROM system.dictionaries WHERE database = ?", database) + if err != nil { + return nil, fmt.Errorf("query system.dictionaries: %w", err) + } + defer func() { _ = rows.Close() }() + srcs := make(map[string]string, len(want)) + for rows.Next() { + var name, source string + if err := rows.Scan(&name, &source); err != nil { + return nil, fmt.Errorf("scan dictionary row: %w", err) + } + if _, ok := want[name]; ok { + srcs[name] = source + } + } + return srcs, rows.Err() + } + + sources, err := readSources() + if err != nil { + return nil, false, err + } + + loaded := false + for name, source := range sources { + if source != "" { + continue // already loaded + } + stmt := "SYSTEM RELOAD DICTIONARY " + chsql.QuoteIdent(database) + "." + chsql.QuoteIdent(name) + if err := conn.Exec(ctx, stmt); err == nil { + loaded = true + } + } + if loaded { + if sources, err = readSources(); err != nil { + return nil, false, err + } + } + + for name := range want { + source, found := sources[name] + if !found { + external = true // not in system.dictionaries at all — nothing to watch + continue + } + if table, ok := parseDictSourceTable(source, database); ok { + out = append(out, table) + } else { + external = true // file/http/executable or cross-database source, or still unloaded + } + } + return out, external, nil +} + +// parseDictSourceTable extracts the backing table from a system.dictionaries +// source string, returning ok=false for a non-CLICKHOUSE source or a source in a +// different database (neither is trackable here). +func parseDictSourceTable(source, database string) (string, bool) { + const prefix = "ClickHouse: " + if !strings.HasPrefix(source, prefix) { + return "", false + } + qualified := strings.TrimSpace(strings.TrimPrefix(source, prefix)) + db, table, ok := strings.Cut(qualified, ".") + if !ok { + return qualified, true // bare table name -> configured db + } + if db != database { + return "", false + } + return table, true +} + +// setSlice returns the set's keys as a sorted slice (nil when empty). +func setSlice(set map[string]struct{}) []string { + if len(set) == 0 { + return nil + } + out := make([]string, 0, len(set)) + for s := range set { + out = append(out, s) + } + slices.Sort(out) + return out +} diff --git a/internal/discovery/explain_test.go b/internal/discovery/explain_test.go new file mode 100644 index 00000000..38c1deba --- /dev/null +++ b/internal/discovery/explain_test.go @@ -0,0 +1,343 @@ +package discovery + +import ( + "reflect" + "testing" +) + +// parseExplainTables is the brittle string-handling half of ResolveTables; the +// sample lines below are real EXPLAIN QUERY TREE output shapes (verified against +// ClickHouse 26.6). The ResolveTables round-trip itself is covered by the +// integration test against a live server. +func TestParseExplainTables(t *testing.T) { + tests := []struct { + name string + lines []string + db string + wantTables []string + wantDicts []string + // wantExternal: the query reads something no local table version can watch + // (a TABLE_FUNCTION node, or a TABLE node that doesn't resolve to a + // configured-database name — conservative), so the caller TTL-caps. + wantExternal bool + }{ + { + name: "simple join, both local", + lines: []string{ + " TABLE id: 3, alias: __table1, table_name: default.base", + " TABLE id: 5, alias: __table2, table_name: default.other", + }, + db: "default", + wantTables: []string{"base", "other"}, + }, + { + name: "count() keeps the table (the whole point)", + lines: []string{" TABLE id: 3, alias: __table1, table_name: default.events"}, + db: "default", + wantTables: []string{"events"}, + }, + { + name: "weird name with dots+spaces: split on FIRST dot only", + lines: []string{ + " TABLE id: 3, alias: __table1, table_name: default.weird.name 2024", + }, + db: "default", + wantTables: []string{"weird.name 2024"}, + }, + { + name: "foreign database is dropped and marks the read external", + lines: []string{ + " TABLE id: 3, alias: __table1, table_name: otherdb.events", + " TABLE id: 5, alias: __table2, table_name: default.local", + }, + db: "default", + wantTables: []string{"local"}, + wantExternal: true, + }, + { + name: "dedup repeated table (self-join)", + lines: []string{ + " TABLE id: 3, table_name: default.t", + " TABLE id: 9, table_name: default.t", + }, + db: "default", + wantTables: []string{"t"}, + }, + { + name: "table function: no tracked table, marks the read external", + lines: []string{" QUERY id: 0", " JOIN TREE", " TABLE_FUNCTION numbers"}, + db: "default", + wantExternal: true, + }, + { + name: "table function alongside a local table: table tracked, still external", + lines: []string{ + " TABLE id: 3, alias: __table1, table_name: default.events", + " TABLE_FUNCTION id: 5, alias: __table2, table_function_name: numbers", + }, + db: "default", + wantTables: []string{"events"}, + wantExternal: true, + }, + { + name: "TABLE node without a parseable name is conservatively external", + lines: []string{" TABLE id: 3, alias: __table1"}, + db: "default", + wantExternal: true, + }, + { + name: "no tables (SELECT 1)", + lines: []string{"QUERY id: 0", " PROJECTION COLUMNS", " 1 UInt8"}, + db: "default", + }, + { + name: "joinGet: Join-engine table tracked from the first constant", + lines: []string{ + " FUNCTION id: 4, function_name: joinGet, function_type: ordinary, result_type: String", + " CONSTANT id: 6, constant_value: 'default.jt', constant_value_type: String", + " CONSTANT id: 7, constant_value: 'v', constant_value_type: String", + }, + db: "default", + wantTables: []string{"jt"}, + }, + { + name: "joinGet alongside a FROM table: both tracked", + lines: []string{ + " TABLE id: 3, alias: __table1, table_name: default.users", + " FUNCTION id: 4, function_name: joinGet, function_type: ordinary, result_type: String", + " CONSTANT id: 6, constant_value: 'default.jt', constant_value_type: String", + }, + db: "default", + wantTables: []string{"jt", "users"}, + }, + { + name: "joinGet bare (unqualified) name → configured db", + lines: []string{ + " FUNCTION id: 4, function_name: joinGet, function_type: ordinary, result_type: String", + " CONSTANT id: 6, constant_value: 'jt', constant_value_type: String", + }, + db: "default", + wantTables: []string{"jt"}, + }, + { + name: "joinGet cross-db dropped and marks the read external", + lines: []string{ + " FUNCTION id: 4, function_name: joinGet, function_type: ordinary, result_type: String", + " CONSTANT id: 6, constant_value: 'otherdb.jt', constant_value_type: String", + }, + db: "default", + wantExternal: true, + }, + { + name: "joinGet with a non-string first constant disarms (no false capture)", + lines: []string{ + " FUNCTION id: 4, function_name: joinGet, function_type: ordinary, result_type: String", + " CONSTANT id: 6, constant_value: UInt64_1, constant_value_type: UInt64", + " CONSTANT id: 7, constant_value: 'v', constant_value_type: String", + }, + db: "default", + }, + { + name: "dictGet: dictionary captured; the projection's stray '' before it is ignored", + lines: []string{ + " CONSTANT id: 2, constant_value: '', constant_value_type: String", + " FUNCTION id: 3, function_name: dictGet, function_type: ordinary, result_type: String", + " CONSTANT id: 5, constant_value: 'default.mydict', constant_value_type: String", + " CONSTANT id: 6, constant_value: 'val', constant_value_type: String", + }, + db: "default", + wantDicts: []string{"mydict"}, + }, + { + name: "dictGet cross-db dropped and marks the read external", + lines: []string{ + " FUNCTION id: 3, function_name: dictGet, function_type: ordinary, result_type: String", + " CONSTANT id: 5, constant_value: 'otherdb.mydict', constant_value_type: String", + }, + db: "default", + wantExternal: true, + }, + { + name: "joinGet and dictGet together", + lines: []string{ + " TABLE id: 3, table_name: default.users", + " FUNCTION id: 4, function_name: joinGet, function_type: ordinary, result_type: String", + " CONSTANT id: 6, constant_value: 'default.jt', constant_value_type: String", + " FUNCTION id: 8, function_name: dictGetOrDefault, function_type: ordinary, result_type: String", + " CONSTANT id: 9, constant_value: 'default.mydict', constant_value_type: String", + }, + db: "default", + wantTables: []string{"jt", "users"}, + wantDicts: []string{"mydict"}, + }, + { + name: "FINAL: trailing ', final: 1' must not leak into the table name", + lines: []string{" TABLE id: 3, alias: __table1, table_name: default.rmt, final: 1"}, + db: "default", + wantTables: []string{"rmt"}, + }, + { + name: "SAMPLE: trailing ', final: 0, sample_size: …' stripped", + lines: []string{" TABLE id: 3, table_name: default.smp, final: 0, sample_size: 5 / 10"}, + db: "default", + wantTables: []string{"smp"}, + }, + { + name: "string literal containing 'table_name:' is not mistaken for a table", + lines: []string{ + " CONSTANT id: 2, constant_value: 'table_name: default.injected', constant_value_type: String", + " TABLE id: 3, alias: __table1, table_name: default.real", + }, + db: "default", + wantTables: []string{"real"}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + gotTables, gotDicts, _, gotExternal := parseExplainTables(tt.lines, tt.db) + if !reflect.DeepEqual(gotTables, tt.wantTables) { + t.Errorf("tables = %v, want %v", gotTables, tt.wantTables) + } + if !reflect.DeepEqual(gotDicts, tt.wantDicts) { + t.Errorf("dicts = %v, want %v", gotDicts, tt.wantDicts) + } + if gotExternal != tt.wantExternal { + t.Errorf("external = %v, want %v", gotExternal, tt.wantExternal) + } + }) + } +} + +// TestParseExplainTables_DeadBranchPruning covers the parameter-gated UNION case: +// after binding, a dead arm's WHERE folds to "CONSTANT UInt64_0" and its tables are +// dropped, while a live arm (WHERE → 1, a real predicate, or an "x = 0" compare) is +// kept. The fixtures reproduce real EXPLAIN QUERY TREE indentation (ClickHouse 26.6), +// since pruning is structure-aware. The cardinal safety property: pruning fails +// toward KEEPING — only an exact constant-false WHERE child drops a table. +func TestParseExplainTables_DeadBranchPruning(t *testing.T) { + // arm builds one UNION-arm QUERY: a table under JOIN TREE and a WHERE whose + // folded child is the given constant ("UInt64_1" live, "UInt64_0" dead) or a raw + // node line (e.g. a FUNCTION) for a non-folded predicate. + arm := func(table, whereChild string) []string { + return []string{ + " QUERY id: 6, alias: __table2", + " JOIN TREE", + " TABLE id: 9, alias: __table3, table_name: default." + table, + " WHERE", + " " + whereChild, + } + } + wrap := func(arms ...[]string) []string { + out := []string{ + "QUERY id: 0", + " JOIN TREE", + " UNION id: 3, alias: __table1, is_subquery: 1, union_mode: UNION_ALL", + " LIST id: 5, nodes: 2", + } + for _, a := range arms { + out = append(out, a...) + } + return out + } + const ( + live = "CONSTANT id: 10, constant_value: UInt64_1, constant_value_type: UInt8" + dead = "CONSTANT id: 19, constant_value: UInt64_0, constant_value_type: UInt8" + pred = "FUNCTION id: 4, function_name: greater, function_type: ordinary, result_type: UInt8" + eq0 = "FUNCTION id: 4, function_name: equals, function_type: ordinary, result_type: UInt8" + ) + + tests := []struct { + name string + lines []string + wantTables []string + wantPruned int + }{ + { + name: "source='web': mobile arm dead, pruned", + lines: wrap(arm("web_events", live), arm("mobile_events", dead)), + wantTables: []string{"web_events"}, + wantPruned: 1, + }, + { + name: "both arms live: nothing pruned", + lines: wrap(arm("web_events", live), arm("mobile_events", live)), + wantTables: []string{"mobile_events", "web_events"}, + wantPruned: 0, + }, + { + name: "real predicate (WHERE col>5) is NOT a dead arm", + lines: wrap(arm("web_events", pred), arm("mobile_events", dead)), + wantTables: []string{"web_events"}, + wantPruned: 1, + }, + { + name: "WHERE x=0 (equals function, not a bare constant) is NOT dead", + lines: wrap(arm("web_events", eq0), arm("mobile_events", dead)), + wantTables: []string{"web_events"}, + wantPruned: 1, + }, + { + name: "a table in BOTH a live and a dead arm stays tracked", + lines: wrap( + arm("shared", live), + arm("shared", dead), + arm("mobile_events", dead), + ), + wantTables: []string{"shared"}, + wantPruned: 1, // only mobile_events + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + gotTables, _, gotPruned, _ := parseExplainTables(tt.lines, "default") + if !reflect.DeepEqual(gotTables, tt.wantTables) { + t.Errorf("tables = %v, want %v", gotTables, tt.wantTables) + } + if gotPruned != tt.wantPruned { + t.Errorf("pruned = %d, want %d", gotPruned, tt.wantPruned) + } + }) + } +} + +func TestParseDictSourceTable(t *testing.T) { + tests := []struct { + name, source, db, want string + ok bool + }{ + {"clickhouse qualified, same db", "ClickHouse: default.dsrc", "default", "dsrc", true}, + {"clickhouse bare name", "ClickHouse: dsrc", "default", "dsrc", true}, + {"clickhouse other db dropped", "ClickHouse: otherdb.dsrc", "default", "", false}, + {"file source untracked", "File: /var/lib/x.csv", "default", "", false}, + {"executable source untracked", "Executable: ./gen.sh", "default", "", false}, + {"http source untracked", "HTTP: http://x/y", "default", "", false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, ok := parseDictSourceTable(tt.source, tt.db) + if got != tt.want || ok != tt.ok { + t.Errorf("parseDictSourceTable(%q) = (%q, %v), want (%q, %v)", tt.source, got, ok, tt.want, tt.ok) + } + }) + } +} + +func TestConstantString(t *testing.T) { + tests := []struct { + name, line, wantVal string + wantStr, wantFound bool + }{ + {"quoted string", " CONSTANT id: 6, constant_value: 'default.jt', constant_value_type: String", "default.jt", true, true}, + {"empty string", " CONSTANT id: 2, constant_value: '', constant_value_type: String", "", true, true}, + {"non-string constant", " CONSTANT id: 7, constant_value: UInt64_0, constant_value_type: UInt8", "", false, true}, + {"not a constant line", " TABLE id: 3, table_name: default.users", "", false, false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + val, isStr, found := constantString(tt.line) + if val != tt.wantVal || isStr != tt.wantStr || found != tt.wantFound { + t.Errorf("constantString(%q) = (%q,%v,%v), want (%q,%v,%v)", tt.line, val, isStr, found, tt.wantVal, tt.wantStr, tt.wantFound) + } + }) + } +} diff --git a/internal/discovery/fuzz_deps_test.go b/internal/discovery/fuzz_deps_test.go new file mode 100644 index 00000000..63a8d242 --- /dev/null +++ b/internal/discovery/fuzz_deps_test.go @@ -0,0 +1,357 @@ +//go:build fuzzdeps + +// Differential fuzzer for ResolveTables. Excluded from the normal build by the +// fuzzdeps tag; run on demand against a local ClickHouse: +// +// go test -tags fuzzdeps -run TestFuzzDeps ./internal/discovery/ -v +// +// It generates hundreds of diverse queries (nested/UNION/parameter-gated, plus +// targeted joinGet/dictGet/cross-db/FINAL/SAMPLE/injection cases) with a +// construction-based ground truth — it builds each query, so it knows which tables +// sit in live vs. dead arms — then runs the real ResolveTables and flags +// under-resolution (stale-cache BUG, fails the test), over-resolution (pruning +// miss / parser hallucination), and EXPLAIN errors. Creates and drops its own +// fuzz_deps / fuzz_other databases; skips if no ClickHouse is reachable on :9000. +package discovery + +import ( + "context" + "fmt" + "math/rand" + "sort" + "strings" + "testing" + "time" + + "github.com/ClickHouse/clickhouse-go/v2" + "github.com/ClickHouse/clickhouse-go/v2/lib/driver" +) + +const ( + fdb = "fuzz_deps" + fother = "fuzz_other" +) + +var baseShort = []string{"bt0", "bt1", "bt2", "bt3", "bt4", "bt5", "bt6", "bt7"} + +func full(short string) string { return fdb + "." + short } + +type occ struct { + table string + dead bool +} + +func connectFuzz(t *testing.T) driver.Conn { + open := func(db string) (driver.Conn, error) { + return clickhouse.Open(&clickhouse.Options{ + Addr: []string{"127.0.0.1:9000"}, + Auth: clickhouse.Auth{Database: db, Username: "default", Password: ""}, + DialTimeout: 5 * time.Second, + }) + } + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + boot, err := open("default") + if err != nil { + t.Fatalf("open default: %v", err) + } + if err := boot.Ping(ctx); err != nil { + t.Skipf("clickhouse not reachable: %v", err) + } + _ = boot.Exec(ctx, "CREATE DATABASE IF NOT EXISTS "+fdb) + _ = boot.Exec(ctx, "CREATE DATABASE IF NOT EXISTS "+fother) + _ = boot.Close() + conn, err := open(fdb) + if err != nil { + t.Fatalf("open %s: %v", fdb, err) + } + if err := conn.Ping(ctx); err != nil { + t.Fatalf("ping %s: %v", fdb, err) + } + return conn +} + +func setupSchema(t *testing.T, conn driver.Conn) { + ctx := context.Background() + exec := func(s string) { + if err := conn.Exec(ctx, s); err != nil { + t.Fatalf("setup %q: %v", s, err) + } + } + exec("CREATE DATABASE IF NOT EXISTS " + fdb) + exec("CREATE DATABASE IF NOT EXISTS " + fother) + for _, s := range baseShort { + exec(fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s.%s (x Int32, s String) ENGINE=MergeTree ORDER BY x", fdb, s)) + exec(fmt.Sprintf("INSERT INTO %s.%s SELECT number, toString(number) FROM numbers(5)", fdb, s)) + } + exec(fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s.`weird.name` (x Int32, s String) ENGINE=MergeTree ORDER BY x", fdb)) + exec(fmt.Sprintf("INSERT INTO %s.`weird.name` SELECT number, toString(number) FROM numbers(5)", fdb)) + exec(fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s.jt (k String, v String) ENGINE=Join(ANY, LEFT, k)", fdb)) + exec(fmt.Sprintf("INSERT INTO %s.jt VALUES ('a','1')", fdb)) + exec(fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s.dsrc (id UInt64, val String) ENGINE=MergeTree ORDER BY id", fdb)) + exec(fmt.Sprintf("INSERT INTO %s.dsrc VALUES (1,'one')", fdb)) + exec(fmt.Sprintf("CREATE DICTIONARY IF NOT EXISTS %s.mydict (id UInt64, val String) PRIMARY KEY id SOURCE(CLICKHOUSE(DB '%s' TABLE 'dsrc')) LAYOUT(HASHED()) LIFETIME(0)", fdb, fdb)) + // Deliberately NOT pre-loading mydict, so the dictGet cases exercise + // resolveDictSources' load-before path against a NOT_LOADED dictionary. + exec(fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s.rmt (x Int32, s String) ENGINE=ReplacingMergeTree ORDER BY x", fdb)) + exec(fmt.Sprintf("INSERT INTO %s.rmt SELECT number, toString(number) FROM numbers(5)", fdb)) + exec(fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s.smp (x Int32, h UInt32) ENGINE=MergeTree ORDER BY h SAMPLE BY h", fdb)) + exec(fmt.Sprintf("INSERT INTO %s.smp SELECT number, toUInt32(number) FROM numbers(5)", fdb)) + exec(fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s.ext (x Int32, s String) ENGINE=MergeTree ORDER BY x", fother)) + exec(fmt.Sprintf("INSERT INTO %s.ext SELECT number, toString(number) FROM numbers(5)", fother)) +} + +func teardown(conn driver.Conn) { + ctx := context.Background() + _ = conn.Exec(ctx, "DROP DATABASE IF EXISTS "+fdb) + _ = conn.Exec(ctx, "DROP DATABASE IF EXISTS "+fother) +} + +// genGate returns a WHERE clause and whether it makes the arm dead. Only foldable +// dead gates ('a'='b') are used, since the parser only prunes constant-folded WHEREs. +func genGate(rng *rand.Rand) (string, bool) { + switch rng.Intn(6) { + case 0: + return "", false // no gate (live) + case 1: + return " WHERE 1=1", false // foldable live + case 2: + return " WHERE 'a'='b'", true // foldable DEAD (calibrated → CONSTANT UInt64_0) + case 3: + return " WHERE 'q'='q'", false // foldable live + case 4: + return " WHERE x > 0", false // real predicate (live) + default: + return " WHERE x = 0", false // real predicate that must NOT be read as dead + } +} + +func genSelect(rng *rand.Rand, depth int, ancestorDead bool) (string, []occ) { + if depth <= 0 || rng.Intn(100) < 45 { + short := baseShort[rng.Intn(len(baseShort))] + clause, dead := genGate(rng) + return "SELECT x FROM " + full(short) + clause, []occ{{short, ancestorDead || dead}} + } + clause, dead := genGate(rng) + d := ancestorDead || dead + var inner string + var iocc []occ + if rng.Intn(2) == 0 { + inner, iocc = genUnion(rng, depth-1, d) + } else { + inner, iocc = genSelect(rng, depth-1, d) + } + alias := fmt.Sprintf("a%d", rng.Intn(1_000_000)) + return "SELECT x FROM (" + inner + ") " + alias + clause, iocc +} + +func genUnion(rng *rand.Rand, depth int, ancestorDead bool) (string, []occ) { + k := 2 + rng.Intn(3) + parts := make([]string, 0, k) + var occs []occ + for i := 0; i < k; i++ { + s, o := genSelect(rng, depth-1, ancestorDead) + parts = append(parts, s) + occs = append(occs, o...) + } + return strings.Join(parts, " UNION ALL "), occs +} + +func genTop(rng *rand.Rand) (string, []occ) { + depth := 2 + rng.Intn(5) // 2..6 + if rng.Intn(2) == 0 { + return genUnion(rng, depth, false) + } + return genSelect(rng, depth, false) +} + +func sets(occs []occ) (expected, referenced map[string]bool) { + expected = map[string]bool{} + referenced = map[string]bool{} + for _, o := range occs { + referenced[o.table] = true + if !o.dead { + expected[o.table] = true + } + } + return +} + +type special struct { + cat, sql string + expected []string +} + +func specials() []special { + return []special{ + {"count", "SELECT count() FROM fuzz_deps.bt0", []string{"bt0"}}, + {"join", "SELECT a.x FROM fuzz_deps.bt0 a JOIN fuzz_deps.bt1 b ON a.x=b.x", []string{"bt0", "bt1"}}, + {"self-join", "SELECT a.x FROM fuzz_deps.bt0 a JOIN fuzz_deps.bt0 b ON a.x=b.x", []string{"bt0"}}, + {"join-dead-where", "SELECT a.x FROM fuzz_deps.bt0 a JOIN fuzz_deps.bt1 b ON a.x=b.x WHERE 'a'='b'", []string{}}, + {"in-subquery", "SELECT x FROM fuzz_deps.bt0 WHERE x IN (SELECT x FROM fuzz_deps.bt1)", []string{"bt0", "bt1"}}, + {"in-subquery-dead", "SELECT x FROM fuzz_deps.bt0 WHERE 'a'='b' UNION ALL SELECT x FROM fuzz_deps.bt2 WHERE x IN (SELECT x FROM fuzz_deps.bt3)", []string{"bt2", "bt3"}}, + {"cte", "WITH c AS (SELECT x FROM fuzz_deps.bt2) SELECT x FROM c", []string{"bt2"}}, + {"cte-join", "WITH c AS (SELECT x FROM fuzz_deps.bt2) SELECT c.x FROM c JOIN fuzz_deps.bt3 t ON c.x=t.x", []string{"bt2", "bt3"}}, + {"joinGet", "SELECT joinGet('fuzz_deps.jt','v', toString(x)) AS g FROM fuzz_deps.bt0", []string{"bt0", "jt"}}, + {"joinGet-bare", "SELECT joinGet('jt','v', toString(x)) AS g FROM fuzz_deps.bt0", []string{"bt0", "jt"}}, + {"dictGet", "SELECT dictGet('fuzz_deps.mydict','val', toUInt64(x)) AS d FROM fuzz_deps.bt0", []string{"bt0", "dsrc"}}, + {"dictGetOrDefault", "SELECT dictGetOrDefault('fuzz_deps.mydict','val', toUInt64(x), '') AS d FROM fuzz_deps.bt0", []string{"bt0", "dsrc"}}, + {"dictHas", "SELECT x FROM fuzz_deps.bt0 WHERE dictHas('fuzz_deps.mydict', toUInt64(x))", []string{"bt0", "dsrc"}}, + {"cross-db", "SELECT x FROM fuzz_deps.bt0 UNION ALL SELECT x FROM fuzz_other.ext", []string{"bt0"}}, + {"cross-db-join", "SELECT a.x FROM fuzz_deps.bt0 a JOIN fuzz_other.ext b ON a.x=b.x", []string{"bt0"}}, + {"table-func", "SELECT number AS x FROM numbers(10)", []string{}}, + {"table-func-in", "SELECT x FROM fuzz_deps.bt0 WHERE x IN (SELECT number FROM numbers(5))", []string{"bt0"}}, + {"weird-name", "SELECT x FROM fuzz_deps.`weird.name`", []string{"weird.name"}}, + {"weird-name-dead", "SELECT x FROM fuzz_deps.`weird.name` WHERE 'a'='b'", []string{}}, + {"weird-name-union", "SELECT x FROM fuzz_deps.`weird.name` WHERE 1=1 UNION ALL SELECT x FROM fuzz_deps.bt0 WHERE 'a'='b'", []string{"weird.name"}}, + {"strinj-tablename", "SELECT 'table_name: fuzz_deps.bt5' AS lbl FROM fuzz_deps.bt0 WHERE x>0", []string{"bt0"}}, + {"strinj-deadconst", "SELECT x FROM fuzz_deps.bt0 WHERE s = 'constant_value: UInt64_0,'", []string{"bt0"}}, + {"strinj-where", "SELECT x FROM fuzz_deps.bt0 WHERE s = 'WHERE' OR s = 'QUERY id: 9'", []string{"bt0"}}, + {"groupby-having", "SELECT x, count() c FROM fuzz_deps.bt0 GROUP BY x HAVING c > 0 ORDER BY x LIMIT 3", []string{"bt0"}}, + {"final-prewhere", "SELECT x FROM fuzz_deps.rmt FINAL PREWHERE x > 0 WHERE s != ''", []string{"rmt"}}, + {"final", "SELECT x FROM fuzz_deps.rmt FINAL", []string{"rmt"}}, + {"final-join", "SELECT r.x FROM fuzz_deps.rmt AS r FINAL JOIN fuzz_deps.bt0 AS b ON r.x = b.x", []string{"rmt", "bt0"}}, + {"sample", "SELECT x FROM fuzz_deps.smp SAMPLE 0.5", []string{"smp"}}, + {"final-dead-arm", "SELECT x FROM fuzz_deps.rmt FINAL WHERE 'a'='b' UNION ALL SELECT x FROM fuzz_deps.bt0", []string{"bt0"}}, + {"window", "SELECT sum(x) OVER (PARTITION BY s) AS w FROM fuzz_deps.bt0", []string{"bt0"}}, + {"deep-live", "SELECT x FROM (SELECT x FROM (SELECT x FROM fuzz_deps.bt3 WHERE x>0) z WHERE 1=1) y", []string{"bt3"}}, + {"live-and-dead-same", "SELECT x FROM fuzz_deps.bt4 WHERE 1=1 UNION ALL SELECT x FROM fuzz_deps.bt4 WHERE 'a'='b'", []string{"bt4"}}, + {"all-dead-union", "SELECT x FROM fuzz_deps.bt5 WHERE 'a'='b' UNION ALL SELECT x FROM fuzz_deps.bt6 WHERE 'x'='y'", []string{}}, + {"dead-arm-with-subquery", "SELECT x FROM fuzz_deps.bt0 WHERE 1=1 UNION ALL SELECT x FROM (SELECT x FROM fuzz_deps.bt1) z WHERE 'a'='b'", []string{"bt0"}}, + {"nested-union-in-dead-arm", "SELECT x FROM fuzz_deps.bt0 WHERE 'a'='b' UNION ALL SELECT x FROM fuzz_deps.bt1 WHERE 1=1", []string{"bt1"}}, + } +} + +func toSet(ss []string) map[string]bool { + m := map[string]bool{} + for _, s := range ss { + m[s] = true + } + return m +} + +func keys(m map[string]bool) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, k) + } + sort.Strings(out) + return out +} + +func TestFuzzDeps(t *testing.T) { + conn := connectFuzz(t) + defer conn.Close() + setupSchema(t, conn) + defer teardown(conn) + ctx := context.Background() + + explainDump := func(sql string) string { + rows, err := conn.Query(ctx, "EXPLAIN QUERY TREE "+sql) + if err != nil { + return " " + } + defer func() { _ = rows.Close() }() + var b strings.Builder + for rows.Next() { + var ln string + _ = rows.Scan(&ln) + b.WriteString(" " + ln + "\n") + } + return b.String() + } + + type finding struct { + cat, sql string + expected, got, under, over []string + } + var unders, overs, errs []finding + total := 0 + + check := func(cat, sql string, expected, referenced map[string]bool) { + total++ + res, err := ResolveTables(ctx, conn, fdb, sql) + if err != nil { + errs = append(errs, finding{cat: cat, sql: sql, got: []string{err.Error()}}) + return + } + got := res.Tables + gotSet := toSet(got) + var under, over []string + for tbl := range expected { + if !gotSet[tbl] { + under = append(under, tbl) + } + } + for _, g := range got { + if !expected[g] { + over = append(over, g) + } + } + sort.Strings(under) + sort.Strings(over) + if len(under) > 0 { + unders = append(unders, finding{cat, sql, keys(expected), got, under, over}) + } + if len(over) > 0 { + overs = append(overs, finding{cat, sql, keys(expected), got, under, over}) + } + } + + rng := rand.New(rand.NewSource(0xC0FFEE)) + const N = 500 + for i := 0; i < N; i++ { + sql, occs := genTop(rng) + expected, referenced := sets(occs) + check("random", sql, expected, referenced) + } + for _, sp := range specials() { + check(sp.cat, sp.sql, toSet(sp.expected), nil) + } + injPool := []string{ + "table_name: fuzz_deps.bt5", "table_name: fuzz_deps.bt0", + "function_name: joinGet", "function_name: dictGet", + "constant_value: UInt64_0,", "WHERE", "QUERY id: 0", "JOIN TREE", + } + for i, p := range injPool { + tbl := baseShort[i%len(baseShort)] + sql := fmt.Sprintf("SELECT '%s' AS lbl FROM %s WHERE x > %d", p, full(tbl), i) + check("inject", sql, toSet([]string{tbl}), nil) + } + + fmt.Printf("\n========== FUZZ DEPS REPORT ==========\n") + fmt.Printf("queries run: %d under-resolutions (BUGS): %d over-resolutions: %d explain-errors: %d\n\n", + total, len(unders), len(overs), len(errs)) + + if len(unders) > 0 { + fmt.Printf("---- UNDER-RESOLUTION (stale-cache risk) ----\n") + for _, f := range unders { + fmt.Printf("[%s] MISSED %v\n expected=%v got=%v\n sql: %s\n EXPLAIN:\n%s\n", + f.cat, f.under, f.expected, f.got, f.sql, explainDump(f.sql)) + } + } + + if len(overs) > 0 { + fmt.Printf("---- OVER-RESOLUTION (pruning miss / phantom) ----\n") + shown := 0 + for _, f := range overs { + fmt.Printf("[%s] EXTRA %v expected=%v got=%v\n sql: %s\n", f.cat, f.over, f.expected, f.got, f.sql) + if shown < 6 { + fmt.Printf(" EXPLAIN:\n%s\n", explainDump(f.sql)) + } + shown++ + } + } + + if len(errs) > 0 { + fmt.Printf("---- EXPLAIN ERRORS (would over-resolve to all tables) ----\n") + for _, f := range errs { + fmt.Printf("[%s] %v\n sql: %s\n", f.cat, f.got, f.sql) + } + } + fmt.Printf("========== END REPORT ==========\n\n") + + if len(unders) > 0 { + t.Errorf("%d under-resolution(s) found — see report above", len(unders)) + } +} diff --git a/internal/ingest/worker.go b/internal/ingest/worker.go index e663c2e1..988d485d 100644 --- a/internal/ingest/worker.go +++ b/internal/ingest/worker.go @@ -15,8 +15,8 @@ import ( "time" "github.com/Wave-RF/WaveHouse/internal/cache" + "github.com/Wave-RF/WaveHouse/internal/chsql" "github.com/Wave-RF/WaveHouse/internal/mq" - "github.com/Wave-RF/WaveHouse/internal/query" "github.com/nats-io/nats.go" "github.com/nats-io/nats.go/jetstream" "go.opentelemetry.io/otel/trace" @@ -464,7 +464,7 @@ func (w *IngestWorker) handleSuccess(ctx context.Context, tableName string, msgs // every scope — and there's nothing more to add. Otherwise invalidate each // distinct scope. Doing this here (we already loop the batch once, and know it's // one table) keeps Cache.Invalidate a simple one-pass bump. - encodedTable := query.SafeEncodeNATS(tableName) + encodedTable := chsql.SafeEncodeNATS(tableName) seenScopes := make(map[string]struct{}, len(msgs)) namespaces := make([]cache.Namespace, 0, len(msgs)) @@ -479,7 +479,7 @@ func (w *IngestWorker) handleSuccess(ctx context.Context, tableName string, msgs seenScopes[pm.scope] = struct{}{} namespaces = append(namespaces, cache.Namespace{ Table: encodedTable, - Scope: query.SafeEncodeNATS(pm.scope), + Scope: chsql.SafeEncodeNATS(pm.scope), }) } diff --git a/internal/ingest/worker_test.go b/internal/ingest/worker_test.go index b0642ca1..3e8d61c9 100644 --- a/internal/ingest/worker_test.go +++ b/internal/ingest/worker_test.go @@ -17,8 +17,8 @@ import ( "time" "github.com/Wave-RF/WaveHouse/internal/cache" + "github.com/Wave-RF/WaveHouse/internal/chsql" "github.com/Wave-RF/WaveHouse/internal/mq" - "github.com/Wave-RF/WaveHouse/internal/query" "github.com/Wave-RF/WaveHouse/internal/testutil" "github.com/nats-io/nats.go" "github.com/nats-io/nats.go/jetstream" @@ -79,9 +79,9 @@ func makeEnvelope(t *testing.T, tableName, scope string, data map[string]any) [] // so the subject and envelope can't silently drift from the producer's contract. func newIngestMsg(t *testing.T, table, scope string, data map[string]any) *testutil.MockJetStreamMsg { t.Helper() - subj := "ingest." + query.SafeEncodeNATS(table) + subj := "ingest." + chsql.SafeEncodeNATS(table) if scope != "" { - subj += "." + query.SafeEncodeNATS(scope) + subj += "." + chsql.SafeEncodeNATS(scope) } return &testutil.MockJetStreamMsg{ MsgSubject: subj, diff --git a/internal/pipes/pipes.go b/internal/pipes/pipes.go index 69448bcf..8bd4d793 100644 --- a/internal/pipes/pipes.go +++ b/internal/pipes/pipes.go @@ -13,6 +13,7 @@ import ( "strings" "sync" + "github.com/Wave-RF/WaveHouse/internal/chsql" "github.com/nats-io/nats.go/jetstream" ) @@ -237,9 +238,7 @@ func formatParamValue(v any) (string, error) { if isNumericLiteral(val) { return val, nil } - escaped := strings.ReplaceAll(val, `\`, `\\`) - escaped = strings.ReplaceAll(escaped, `'`, `''`) - return "'" + escaped + "'", nil + return chsql.QuoteString(val), nil case float64: // JSON numbers are float64; if it's a whole number, format as integer. if val == float64(int64(val)) { diff --git a/tests/e2e/sdk/cache.test.ts b/tests/e2e/sdk/cache.test.ts index 06dfec5f..8efa29b7 100644 --- a/tests/e2e/sdk/cache.test.ts +++ b/tests/e2e/sdk/cache.test.ts @@ -1,5 +1,13 @@ import { describe, expect, it } from "vitest"; -import { chQuery, dataClient, makeJWT, testId, WH_URL, waitForCondition } from "./helpers.js"; +import { + adminClient, + chQuery, + dataClient, + makeJWT, + testId, + WH_URL, + waitForCondition, +} from "./helpers.js"; import { suiteTables } from "./tables.js"; describe("Cache", () => { @@ -63,6 +71,85 @@ describe("Cache", () => { expect(res3.headers.get("x-cache")).toEqual("MISS"); }); + it("invalidates a pipe reading a materialized view's TO target on source ingest", async () => { + const admin = adminClient(); + const stamp = Date.now(); + const src = `mv_src_${stamp}`; + const tgt = `mv_tgt_${stamp}`; + const mv = `mv_${stamp}`; + const pipeName = `pipe_mv_target_${stamp}`; + + // Source + TO target are both ordinary tables; ClickHouse populates the + // target through the MV on every insert into the source. Nothing ever + // writes the target directly, so the pipe below stays fresh only if + // WaveHouse carries a source write through the MV into the target's + // cache version (the trigger→target cascade). + await chQuery( + `CREATE TABLE IF NOT EXISTS default.\`${src}\` (event_id String) ENGINE = MergeTree() ORDER BY event_id`, + ); + await chQuery( + `CREATE TABLE IF NOT EXISTS default.\`${tgt}\` (event_id String) ENGINE = MergeTree() ORDER BY event_id`, + ); + await chQuery( + `CREATE MATERIALIZED VIEW IF NOT EXISTS default.\`${mv}\` TO default.\`${tgt}\` AS SELECT event_id FROM default.\`${src}\``, + ); + + // Discover the MV (and its trigger→target edge) before the pipe first runs. + const refreshRes = await admin.schema.refresh(); + expect(refreshRes.error).toBeNull(); + + const setRes = await admin.pipes.set(pipeName, { + sql: `SELECT count() AS c FROM default.\`${tgt}\``, + description: "E2E: MV TO-target invalidation", + }); + expect(setRes.error).toBeNull(); + + // Raw fetch (as in the header test above) so X-Cache is visible. Admin + // bypasses the pipe's allowed_roles allowlist, so no policy edits needed. + const token = makeJWT({ sub: "cache-mv-test", role: "admin", tenant_id: "acme" }); + const exec = () => + fetch(`${WH_URL}/v1/pipes/${pipeName}`, { + headers: { Authorization: `Bearer ${token}` }, + }); + + const res1 = await exec(); + if (!res1.ok) throw new Error(`Pipe failed: ${await res1.text()}`); + expect(res1.headers.get("x-cache")).toEqual("MISS"); + expect(await res1.json()).toEqual([{ c: 0 }]); // target starts empty + + const res2 = await exec(); + expect(res2.headers.get("x-cache")).toBe("HIT"); + + // Ingest into the SOURCE. The worker's flush inserts into ClickHouse + // (firing the MV into the target) and then invalidates the source's + // namespace; the cascade must carry that bump into the target the pipe + // folded. + const eventId = testId(); + const insertRes = await admin.from(src).insert({ event_id: eventId }); + expect(insertRes.error).toBeNull(); + + // The row landing in the TARGET proves the MV fired and the flush ran... + await waitForCondition(async () => { + const r = await chQuery( + `SELECT event_id FROM default.\`${tgt}\` WHERE event_id = '${eventId}'`, + ); + return r.length === 1; + }, 10_000); + + // ...and the eviction follows the flush, so poll to MISS (a HIT poll + // doesn't re-prime anything; the first MISS is the eviction). + await waitForCondition(async () => (await exec()).headers.get("x-cache") === "MISS", 5_000); + + // Whatever we fetch now is post-eviction: the pipe sees the new row. + const res3 = await exec(); + expect(await res3.json()).toEqual([{ c: 1 }]); + + await admin.pipes.delete(pipeName); + await chQuery(`DROP VIEW IF EXISTS default.\`${mv}\``); + await chQuery(`DROP TABLE IF EXISTS default.\`${tgt}\``); + await chQuery(`DROP TABLE IF EXISTS default.\`${src}\``); + }, 30_000); + it("expires cache naturally based on TTL", async () => { const token = makeJWT({ sub: "ttl-test", diff --git a/tests/integration/dlq_test.go b/tests/integration/dlq_test.go index 96bfa697..3c20c698 100644 --- a/tests/integration/dlq_test.go +++ b/tests/integration/dlq_test.go @@ -10,7 +10,7 @@ import ( "testing" "time" - "github.com/Wave-RF/WaveHouse/internal/query" + "github.com/Wave-RF/WaveHouse/internal/chsql" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) @@ -50,7 +50,7 @@ func TestDLQ_PopulatedOnIngestWorkerFailure(t *testing.T) { // A table name that intentionally doesn't exist in ClickHouse. Per-test // suffix keeps tests independent if more DLQ tests get added later. rawTableName := fmt.Sprintf("nonexistent_table_%d", time.Now().UnixNano()) - safeTableName := query.SafeEncodeNATS(rawTableName) + safeTableName := chsql.SafeEncodeNATS(rawTableName) evt := map[string]any{ "table_name": rawTableName, @@ -94,7 +94,7 @@ func TestDLQ_PopulatedOnIngestWorkerFailureWithBadName(t *testing.T) { // Per-test suffix keeps tests independent if more DLQ tests get added later. rawTableName := fmt.Sprintf("no table.!@#&*()_=/_`%d", time.Now().UnixNano()) - safeTableName := query.SafeEncodeNATS(rawTableName) + safeTableName := chsql.SafeEncodeNATS(rawTableName) evt := map[string]any{ "table_name": rawTableName, diff --git a/tests/integration/pipe_cache_test.go b/tests/integration/pipe_cache_test.go new file mode 100644 index 00000000..f7a92d71 --- /dev/null +++ b/tests/integration/pipe_cache_test.go @@ -0,0 +1,322 @@ +//go:build integration + +package tests + +import ( + "context" + "fmt" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "github.com/Wave-RF/WaveHouse/internal/api" + "github.com/Wave-RF/WaveHouse/internal/auth" + "github.com/Wave-RF/WaveHouse/internal/cache" + "github.com/Wave-RF/WaveHouse/internal/chsql" + "github.com/Wave-RF/WaveHouse/internal/discovery" + "github.com/Wave-RF/WaveHouse/internal/pipes" + "github.com/Wave-RF/WaveHouse/internal/policy" + "github.com/Wave-RF/WaveHouse/internal/testutil" + "github.com/go-chi/chi/v5" + "github.com/stretchr/testify/require" +) + +// newPipeHandler wires a one-pipe store and a fresh LocalCache into a PipesHandler +// against the shared ClickHouse. The cache gets the registry's CURRENT base→view +// cascade installed (normally pushed by main on each refresh) — a pipe folds a +// view's own namespace, so a write to the view's base table only evicts it via +// this cascade; tests that create views must Refresh before calling this. +func newPipeHandler(t *testing.T, e *testEnv, name, sql string, roles ...string) (*api.PipesHandler, *cache.LocalCache) { + t.Helper() + localCache, err := cache.NewLocal(1 << 20) + require.NoError(t, err) + t.Cleanup(func() { _ = localCache.Close() }) + localCache.SetDependents(e.registry.Dependents()) + + store := pipes.NewMemoryStore(&pipes.NamedQuery{Name: name, SQL: sql, AllowedRoles: roles}) + h := api.NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), e.chConn, localCache, e.registry, 30*time.Second, testutil.NopLogger()) + return h, localCache +} + +// execPipe drives GET /v1/pipes/{name}/execute as role, requires 200, and waits +// out ristretto's async Set so a following call observes the cached entry. +func execPipe(t *testing.T, h *api.PipesHandler, lc *cache.LocalCache, name, role string) *httptest.ResponseRecorder { + t.Helper() + rctx := chi.NewRouteContext() + rctx.URLParams.Add("name", name) + c := context.WithValue(context.Background(), chi.RouteCtxKey, rctx) + c = auth.WithRole(c, role) + r := httptest.NewRequestWithContext(c, http.MethodGet, "/v1/pipes/"+name+"/execute", nil) + w := httptest.NewRecorder() + h.Execute(w, r) + require.Equal(t, http.StatusOK, w.Code, w.Body.String()) + lc.Wait() + return w +} + +// TestPipeCacheInvalidation drives the full pipe cache path against the shared +// ClickHouse: a pipe reading A NORMAL VIEW, caching, and write-driven invalidation +// via the cascade. The pipe folds the VIEW's own namespace; the registry maps the +// view to its base table at refresh (the view carries no dependencies_table edge, +// so this must come from parsing the view's own as_select), and installs a base-> +// view cascade so a write to the base evicts the view-keyed entry. Pipes run the +// author SQL as-is (no row security), so any allowed role shares one cached result. +func TestPipeCacheInvalidation(t *testing.T) { + e := env(t) + ctx := context.Background() + + // Base table + a normal view over it. The pipe reads the VIEW; resolution + // flattens it through the tree (built from the view's parsed definition) down + // to the base table the registry knows, so a write to the base invalidates. + base := createTable(t, "user_id UInt64, org_id UInt64", "ORDER BY user_id") + require.NoError(t, e.chConn.Exec(ctx, + fmt.Sprintf("INSERT INTO `%s` SELECT number, number%%3 FROM numbers(60)", base))) + + view := base + "_v" + require.NoError(t, e.chConn.Exec(ctx, + fmt.Sprintf("CREATE VIEW `%s` AS SELECT user_id, org_id FROM `%s`", view, base))) + t.Cleanup(func() { _ = e.chConn.Exec(context.Background(), "DROP VIEW IF EXISTS `"+view+"`") }) + + // The view->base edge is discovered at refresh (createTable refreshed before + // the view existed), so refresh again now that the view is created. + require.NoError(t, e.registry.Refresh(ctx)) + + h, localCache := newPipeHandler(t, e, "agg", + fmt.Sprintf("SELECT sum(user_id) AS s FROM `%s`.`%s`", testCHDatabase, view), + "viewer", "editor") + exec := func(role string) *httptest.ResponseRecorder { return execPipe(t, h, localCache, "agg", role) } + + // First call misses (folds the view's own namespace as the dependency), then hits. + w := exec("viewer") + require.Equal(t, "MISS", w.Header().Get("X-Cache")) + require.Equal(t, `[{"s":1770}]`, w.Body.String()) // sum(0..59) + require.Equal(t, "HIT", exec("viewer").Header().Get("X-Cache")) + + // A different allowed role shares the same cached result (pipes aren't per-role). + require.Equal(t, "HIT", exec("editor").Header().Get("X-Cache")) + + // A write to the BASE table cascades to the view (registry base->view edge) and + // invalidates the view-keyed cached result. + _, err := localCache.Invalidate(ctx, []cache.Namespace{{Table: chsql.SafeEncodeNATS(base)}}) + require.NoError(t, err) + require.Equal(t, "MISS", exec("viewer").Header().Get("X-Cache")) +} + +// TestPipeCacheInvalidation_UnionDeadBranchPruned proves the per-binding dead-branch +// pruning: on a parameter-gated UNION, ClickHouse folds the bound predicate so the +// arm that can't match ({{source}}='web' makes the 'mobile' arm 'web'='mobile') +// resolves to a constant-false WHERE. Resolution drops that arm's table, so with +// source='web' the result depends ONLY on `web` — a write to the dead-branch `mobile` +// does NOT evict it, while a write to `web` does. This is precise, not stale: the +// pruned arm reads nothing regardless of `mobile`'s data, so it's a true non-dependency. +func TestPipeCacheInvalidation_UnionDeadBranchPruned(t *testing.T) { + e := env(t) + ctx := context.Background() + + web := createTable(t, "user_id UInt64", "ORDER BY user_id") + mobile := createTable(t, "user_id UInt64", "ORDER BY user_id") + for _, tbl := range []string{web, mobile} { + require.NoError(t, e.chConn.Exec(ctx, + fmt.Sprintf("INSERT INTO `%s` SELECT number FROM numbers(60)", tbl))) + } + + // A parameter-gated UNION; {{source}} defaults to 'web', so the 'mobile' arm's + // WHERE folds to constant-false and `mobile` is pruned from the dependency set. + sql := fmt.Sprintf( + "SELECT sum(user_id) AS s FROM ("+ + "SELECT user_id FROM `%s`.`%s` WHERE {{source:web}} = 'web' "+ + "UNION ALL "+ + "SELECT user_id FROM `%s`.`%s` WHERE {{source:web}} = 'mobile')", + testCHDatabase, web, testCHDatabase, mobile) + + // Pin the ClickHouse-version coupling directly before the cache-level assertions: + // parseExplainTables recognizes a dead UNION arm by ClickHouse's LITERAL rendering + // of a false-folded WHERE ("constant_value: UInt64_0,"). If an upgrade changes + // that rendering, pruning silently stops firing — failing SAFE (the arm's tables + // stay tracked, over-resolve, never a wrong prune) but losing the precision + // optimization and its TTL-cap signal with nothing but a flat + // wavehouse_pipe_dep_tables_pruned_total metric. Assert the prune on the bound + // SQL (what Execute resolves) so a rendering change surfaces in CI with a direct + // diagnostic, not an obscure eviction failure further down. + boundSQL := strings.ReplaceAll(sql, "{{source:web}}", "'web'") + res, err := discovery.ResolveTables(ctx, e.chConn, testCHDatabase, boundSQL) + require.NoError(t, err) + require.Contains(t, res.Tables, web, "the live arm's table stays tracked") + require.NotContains(t, res.Tables, mobile, "the constant-false arm's table must be pruned") + require.True(t, res.Pruned, "ResolveTables must report the prune (it drives the TTL cap)") + + h, localCache := newPipeHandler(t, e, "u", sql, "viewer") + exec := func() *httptest.ResponseRecorder { return execPipe(t, h, localCache, "u", "viewer") } + + require.Equal(t, "MISS", exec().Header().Get("X-Cache")) // deps pruned to [web] + require.Equal(t, `[{"s":1770}]`, exec().Body.String()) // web's sum(0..59), a HIT + require.Equal(t, "HIT", exec().Header().Get("X-Cache")) + + // A write to the pruned dead-branch table `mobile` does NOT evict — it is a true + // non-dependency for source='web' (precise resolution, not over-resolution). + _, err = localCache.Invalidate(ctx, []cache.Namespace{{Table: chsql.SafeEncodeNATS(mobile)}}) + require.NoError(t, err) + require.Equal(t, "HIT", exec().Header().Get("X-Cache")) + + // A write to the live-branch table `web` DOES evict. + _, err = localCache.Invalidate(ctx, []cache.Namespace{{Table: chsql.SafeEncodeNATS(web)}}) + require.NoError(t, err) + require.Equal(t, "MISS", exec().Header().Get("X-Cache")) +} + +// TestPipeCacheInvalidation_MaterializedViewSource proves a pipe reading a +// materialized view invalidates on writes to the MV's SOURCE table. The pipe folds +// the MV's own namespace, which nothing writes directly (ingest writes the source; +// ClickHouse maintains the MV from it), so without the cascade such a pipe would be +// silently stale until its TTL. The registry maps source->MV via +// system.tables.dependencies_table and installs it as a cascade, so a source write +// fans out to evict the MV-keyed cached result. +func TestPipeCacheInvalidation_MaterializedViewSource(t *testing.T) { + e := env(t) + ctx := context.Background() + + // Source table (registry-known, ingestable). The registry refresh happens here, + // BEFORE the MV is created, so the MV is deliberately NOT a registry table — the + // realistic case the source mapping must handle. + src := createTable(t, "id UInt64, v UInt64", "ORDER BY id") + require.NoError(t, e.chConn.Exec(ctx, + fmt.Sprintf("INSERT INTO `%s` SELECT number, number FROM numbers(100)", src))) + + mv := src + "_mv" + require.NoError(t, e.chConn.Exec(ctx, fmt.Sprintf( + "CREATE MATERIALIZED VIEW `%s` ENGINE=AggregatingMergeTree ORDER BY id AS SELECT id, sumState(v) AS s FROM `%s` GROUP BY id", + mv, src))) + t.Cleanup(func() { _ = e.chConn.Exec(context.Background(), "DROP VIEW IF EXISTS `"+mv+"`") }) + + // The materialized-view -> source map is built during schema discovery, so refresh + // AFTER creating the view (createTable refreshed before it existed). + require.NoError(t, e.registry.Refresh(ctx)) + + h, localCache := newPipeHandler(t, e, "mvpipe", + fmt.Sprintf("SELECT sumMerge(s) AS s FROM `%s`.`%s`", testCHDatabase, mv), + "viewer") + exec := func() *httptest.ResponseRecorder { return execPipe(t, h, localCache, "mvpipe", "viewer") } + + require.Equal(t, "MISS", exec().Header().Get("X-Cache")) // folds the MV's namespace; cascade maps source -> MV + require.Equal(t, "HIT", exec().Header().Get("X-Cache")) + + // A write to the SOURCE (what ingest bumps) must evict the MV-reading pipe. + _, err := localCache.Invalidate(ctx, []cache.Namespace{{Table: chsql.SafeEncodeNATS(src)}}) + require.NoError(t, err) + require.Equal(t, "MISS", exec().Header().Get("X-Cache")) +} + +// TestPipeCacheInvalidation_MaterializedViewTarget proves a pipe reading an MV's +// TO TARGET table — the common `CREATE MATERIALIZED VIEW mv TO tgt` pattern, where +// ingest writes the source and ClickHouse populates the target — invalidates on +// writes to the source. The target is an ordinary base table, so the pipe folds +// its namespace at FULL TTL with no TTL-cap safety net: without the +// trigger→target cascade edge nothing would ever bump it, and the pipe would +// serve stale for its whole TTL. The edge is parsed from the MV's rendered +// create_table_query at refresh (discovery.parseMVTarget), so the target's name +// is deliberately dotted (backtick-quoted in the DDL): this pins the parse — and +// its quoted-identifier decode — against the ClickHouse version we ship, the same +// way the pruning test pins the EXPLAIN rendering. +func TestPipeCacheInvalidation_MaterializedViewTarget(t *testing.T) { + e := env(t) + ctx := context.Background() + + src := createTable(t, "id String", "ORDER BY id") + target := createRawTable(t, fmt.Sprintf("it_mv.target_%d", tableCounter.Add(1)), map[string]string{"id": "seed"}) + + mv := src + "_to_mv" + require.NoError(t, e.chConn.Exec(ctx, fmt.Sprintf( + "CREATE MATERIALIZED VIEW `%s` TO %s AS SELECT id FROM `%s`", + mv, chQuoteIdent(target), src))) + t.Cleanup(func() { _ = e.chConn.Exec(context.Background(), "DROP VIEW IF EXISTS `"+mv+"`") }) + + // The trigger→target edge is discovered at refresh; refresh now that the MV + // exists (createRawTable refreshed before it did). + require.NoError(t, e.registry.Refresh(ctx)) + require.Contains(t, e.registry.Dependents()[chsql.SafeEncodeNATS(src)], chsql.SafeEncodeNATS(target), + "the cascade must carry a source write into the MV's TO target") + + h, localCache := newPipeHandler(t, e, "tgtpipe", + fmt.Sprintf("SELECT count() AS c FROM %s.%s", chQuoteIdent(testCHDatabase), chQuoteIdent(target)), + "viewer") + exec := func() *httptest.ResponseRecorder { return execPipe(t, h, localCache, "tgtpipe", "viewer") } + + w := exec() + require.Equal(t, "MISS", w.Header().Get("X-Cache")) // folds the target's own namespace, full TTL + require.Equal(t, `[{"c":1}]`, w.Body.String()) // the seeded row + require.Equal(t, "HIT", exec().Header().Get("X-Cache")) + + // A write to the SOURCE (what ingest bumps; ClickHouse populates the target + // through the MV) must evict the target-reading pipe. + _, err := localCache.Invalidate(ctx, []cache.Namespace{{Table: chsql.SafeEncodeNATS(src)}}) + require.NoError(t, err) + require.Equal(t, "MISS", exec().Header().Get("X-Cache")) +} + +// TestPipeCacheInvalidation_WeirdIdentifier proves the pipe dependency-invalidation +// path operates on a table whose name a safe-identifier allowlist would reject — here +// an embedded-dot name that also LOOKS like a qualified `db.table` reference, the most +// load-bearing weird case for a feature that parses `db.table` out of EXPLAIN. It is +// the pipe-path counterpart to the query builder's identifier_roundtrip_test.go: the +// round-trip under test is that the dependency name EXPLAIN QUERY TREE reports, once +// SafeEncodeNATS-encoded, equals the namespace an ingest write to that table bumps — +// otherwise a write would silently fail to evict. createRawTable is required because +// createTable sanitizes the dots away; dotted-name resolution itself is proven in +// internal/discovery/fuzz_deps_test.go, and this carries it through the full Execute. +// The pipe is a count() on purpose: ClickHouse answers it from metadata (reading no +// data parts), which a runtime signal like system.query_log would drop — the +// pre-optimization EXPLAIN QUERY TREE keeps the table, so the write still evicts. +func TestPipeCacheInvalidation_WeirdIdentifier(t *testing.T) { + e := env(t) + ctx := context.Background() + + // Arbitrary, regex-hostile name: embedded dots so it reads as `a.b.c`. Legal in a + // quoted ClickHouse identifier; createRawTable quotes it correctly and seeds one row. + rawName := fmt.Sprintf("it_weird.dep.name_%d", tableCounter.Add(1)) + table := createRawTable(t, rawName, map[string]string{"id": "row1"}) + + h, localCache := newPipeHandler(t, e, "weird", + fmt.Sprintf("SELECT count() AS c FROM %s.%s", chQuoteIdent(testCHDatabase), chQuoteIdent(table)), + "viewer") + exec := func() *httptest.ResponseRecorder { return execPipe(t, h, localCache, "weird", "viewer") } + + // First call resolves the weird-named table via EXPLAIN and folds its namespace + // into the key; second call hits. + w := exec() + require.Equal(t, "MISS", w.Header().Get("X-Cache")) + require.Equal(t, `[{"c":1}]`, w.Body.String()) + require.Equal(t, "HIT", exec().Header().Get("X-Cache")) + + // A write to the weird-named table, encoded exactly as the ingest worker encodes + // it, must evict the cached result — proving the resolved dependency namespace + // equals SafeEncodeNATS(rawName) despite the dots. + _, err := localCache.Invalidate(ctx, []cache.Namespace{{Table: chsql.SafeEncodeNATS(table)}}) + require.NoError(t, err) + require.Equal(t, "MISS", exec().Header().Get("X-Cache")) +} + +// TestPipeDeps_ExternalReadsAreDetected pins external-read detection against the +// shipped ClickHouse: a table function renders as its own TABLE_FUNCTION node in +// EXPLAIN QUERY TREE, flagged External so the handler TTL-caps the result (a +// write can never evict data that lives outside ClickHouse's tables). Like the +// pruning pin in TestPipeCacheInvalidation_UnionDeadBranchPruned, this surfaces a +// rendering change on a ClickHouse upgrade. +func TestPipeDeps_ExternalReadsAreDetected(t *testing.T) { + e := env(t) + ctx := context.Background() + + local := createTable(t, "x UInt64", "ORDER BY x") + + res, err := discovery.ResolveTables(ctx, e.chConn, testCHDatabase, + fmt.Sprintf("SELECT t.x FROM `%s`.`%s` t JOIN numbers(10) n ON t.x = n.number", testCHDatabase, local)) + require.NoError(t, err) + require.Contains(t, res.Tables, local, "the local table stays tracked") + require.True(t, res.External, "a table function must be flagged external") + + res, err = discovery.ResolveTables(ctx, e.chConn, testCHDatabase, + fmt.Sprintf("SELECT x FROM `%s`.`%s`", testCHDatabase, local)) + require.NoError(t, err) + require.False(t, res.External, "a plain local read must not be flagged external") +} diff --git a/tests/integration/setup_test.go b/tests/integration/setup_test.go index bb3aecdb..5d7ae3a9 100644 --- a/tests/integration/setup_test.go +++ b/tests/integration/setup_test.go @@ -78,9 +78,15 @@ var tableCounter atomic.Uint64 func createTable(t *testing.T, columns, tableOpts string) string { t.Helper() - // Sanitize the test name into a valid CH identifier. - safe := strings.NewReplacer("/", "_", " ", "_", "-", "_").Replace(t.Name()) - name := fmt.Sprintf("it_%s_%d", strings.ToLower(safe), tableCounter.Add(1)) + safe := strings.Map(func(r rune) rune { + switch { + case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9', r == '_': + return r + default: + return '_' + } + }, t.Name()) + name := fmt.Sprintf("it_%s_%d", safe, tableCounter.Add(1)) ctx := context.Background() stmt := fmt.Sprintf(