feat(pipes): resolve table dependencies for cache invalidation

AGENTS.md

@@ -59,7 +59,7 @@ The invariant index — what must stay true. Full narrative and rationale live i
 10. **Active Sweeper** — purges NATS messages that are both ACKed (written to CH) and older than the gap window; SSE gap-fill uses `DeliverByStartTime`, no in-process ring buffer.
 11. **Hasura-style access control: fail-closed (security)** — `policy.IsAdmin` (role == `admin_role`, **exact case-sensitive**, default `"admin"`) is the single admin check, shared by `Evaluate`/`ResolveRole`/`Validate`/the `/v1/admin` gate/`RoleAllowed`. Empty/absent role matches nothing (no `"*"` wildcard); `Validate` rejects empty role keys; a `nil` policy (deleted) denies **everyone incl. admin** via a role — a total lockout for token-based callers, so bootstrap from the policy file, never an implicit admin grant (**exception:** the operator key's `auth.IsOperator` bit passes the `/v1/admin` gate even under a `nil` policy — a deliberate break-glass restore over HTTP, see #7). `default_role` is the one sanctioned roleless exception (`ResolveRole` maps empty → it pre-eval); `default_role == admin_role` is permitted but dev-only and loudly warned (`policy.DefaultRoleGrantsAdmin`). Preserve when touching `internal/policy` (policy twin of #13; see #159). Detail: architecture.md § `policy/`.
 12. **Structured queries: column authz fail-closed (security)** — `POST /v1/query?table={table}`: typed AST validated against schema, permission-enforced, timestamp-bucketed for cache, `DefaultMaxRows` (10,000) cap. Every column reference — projection, aggregation args, `filters`, `group_by`, `order_by`, `time_range` — is authorized inside `query.Build` (the single chokepoint that enumerates them all), so no clause can skip the role's `allow_columns`/`deny_columns` check (#223). A `select_all` read by a *column-restricted* role expands to its allowed columns via `policy.AllowedProjection`, never a bare `SELECT *`; *unrestricted*/admin roles keep `SELECT *` (`policy.RestrictsColumns` decides). Omitting `columns` selects nothing (`ErrEmptyProjection` → `200 []`); `["*"]` is the literal column `*` (schema-gated, not a wildcard); a table-granted role with no readable columns fails closed (`ErrNoReadableColumns` → `403`). Structured and live-stream (`stream.filterColumns`) reads share the one per-column decision `policy.IsColumnAllowed`, so column visibility can't drift. Preserve when touching `internal/query` or the structured-query handler. Detail: architecture.md § `query/`.
-13. **Named query pipes: fail-closed (security)** — pre-defined SQL templates (Tinybird-style) with param binding + caching; `GET/POST /v1/pipes/{name}` sit outside `RequireAdmin`, so per-pipe `allowed_roles` is the *only* execute-path gate, via `policy.RoleAllowed`: exact allowlist membership (no `"*"`), admin always passes, empty/absent role and empty-string entries authorize nobody, and no `allowed_roles` → admin-only. Preserve and exercise via `testutil.RunRoleMatrix` / `StandardRoleMatrix` (see #159). Detail: architecture.md § `pipes/`.
+13. **Named query pipes: fail-closed (security)** — pre-defined SQL templates (Tinybird-style) with param binding + caching; `GET/POST /v1/pipes/{name}` sit outside `RequireAdmin`, so per-pipe `allowed_roles` is the *only* execute-path gate, via `policy.RoleAllowed`: exact allowlist membership (no `"*"`), admin always passes, empty/absent role and empty-string entries authorize nobody, and no `allowed_roles` → admin-only. Preserve and exercise via `testutil.RunRoleMatrix` / `StandardRoleMatrix` (see #159). The pipe's SQL is **not** re-checked against the table policy (no row or column filtering) — `allowed_roles` gates execution, but the author is responsible for scoping the data in the pipe SQL and any views it reads. A pipe is **run as-is** — WaveHouse never parses its SQL and never rejects it (a write/DDL or multi-statement pipe runs on behalf of the caller's role, so the author owns scoping it). Cache invalidation derives the pipe's table dependencies from ClickHouse `EXPLAIN QUERY TREE`: a query it can't analyze (a write/DDL pipe, a missing table, an unreachable server) falls back to the single **database-wide version** namespace that every write bumps, so any write evicts it — never an under-resolution — and a result that can't be reliably version-invalidated (an unfoldable view, an external read like a table function or cross-database table, or a dead-branch-pruned dependency set) is **TTL-capped** (`cache.UnresolvedDepsTTLCap`) rather than trusted to version invalidation. Detail: architecture.md § `pipes/`.
 14. **TypeScript SDK** — `@wavehouse/sdk`: zero-dep client, typed query builder, real-time SSE, live queries (incrementable/decomposable/poll aggregation), codegen CLI. The canonical client (see §SDK Sync).
 15. **Observability invariants** — stdout always 100% (sampling is OTLP-push-only); WARN+ERROR always export at 100% (a non-configurable floor — don't expose it); gRPC OTel exporters dial lazily so an unreachable collector never blocks startup; the OTel Prometheus exporter uses a **private** `prometheus.Registry`. The OTLP endpoint/TLS/custom-CA/mTLS/headers are delegated to the OpenTelemetry SDK's standard `OTEL_EXPORTER_OTLP_*` env vars — `InitProvider` passes **no** endpoint/header options. Known gap, intentionally not patched in WaveHouse app code: the pinned gRPC logs exporter (`otlploggrpc` v0.19/v0.20) ignores the env TLS-cert vars, so a custom/private CA and mutual TLS apply to traces/metrics but **not** the logs signal (public-CA/system-roots TLS and plaintext still work for logs) — upstream bug open-telemetry/opentelemetry-go#6661. A malformed `OTEL_EXPORTER_OTLP_HEADERS` is logged and skipped by the SDK (fail-soft), not fatal. Preserve when touching the logger/sampler/provider. Detail: architecture.md § `observability/`.
 16. **Bearer-token-only CORS posture (security)** — Bearer JWT on every request, no cookies/sessions; `corsMiddleware` deliberately **never** emits `Access-Control-Allow-Credentials` (not needed, and `*` + credentials is a spec violation browsers reject). `cors_allowed_origins` controls who can *read* responses, not cookie scope; CSRF protection is structural. Don't reintroduce cookie auth or `Allow-Credentials` without a design discussion — answers GitHub #29/#30. Code: `internal/api/router.go`.

CHANGELOG.md

@@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **Named pipes invalidate their cached results on writes to the tables they read, via versioned namespaces and a refresh-time cascade** (`internal/discovery/{discovery,explain}.go`, `internal/api/pipes.go`, `internal/pipes/pipes.go`, `internal/cache/{cache,local,version_manager}.go`, `internal/chsql/nats.go` (moved from `internal/query/ident.go`), `cmd/wavehouse/main.go`, `docs/src/content/docs/{pipes.mdx,architecture.md}`, plus unit + live-ClickHouse tests): closes [#178](https://github.com/Wave-RF/WaveHouse/issues/178). A pipe's table dependencies are resolved by **asking ClickHouse** — `EXPLAIN QUERY TREE` on the bound query the first time it runs (`discovery.ResolveTables`, reading the table list off ClickHouse's own analysis), cached per bound query (parameter combination). WaveHouse never parses pipe SQL itself and never rejects a pipe. `EXPLAIN QUERY TREE` is the pre-optimization tree, so it keeps a table even when the query is metadata-answered (`SELECT count() FROM t` still tracks `t`) or reads a currently-empty table — both of which `system.query_log` and the post-optimization `EXPLAIN PLAN` would drop. It also tracks reads that aren't plain `FROM` clauses: a `joinGet()` of a `Join`-engine table and a `dictGet()` of a dictionary (mapped via `system.dictionaries` to the ClickHouse table backing it). Table-name extraction is robust to `EXPLAIN`-rendering quirks the unit fixtures didn't originally cover: a `FINAL`/`SAMPLE` modifier appends fields after the name (`table_name: db.t, final: 1`) and a string literal can itself contain the text `table_name:` — neither leaks into, nor fabricates, a tracked table — and a `dictGet` against a not-yet-loaded dictionary force-loads it first, since ClickHouse reports an empty `source` for an unloaded dict that would otherwise drop its backing table. A build-tagged differential fuzzer (`internal/discovery/fuzz_deps_test.go`, run with `-tags fuzzdeps` against a local ClickHouse) cross-checks the resolved set for hundreds of nested / `UNION` / parameter-gated queries against a construction-based ground truth. Each resolved table is folded into the cache key as its **own versioned namespace** (a per-name lookup, no per-request graph walk), so a write to it misses the key. Views and materialized views are versioned namespaces too: the schema registry resolves each view's definition (also via EXPLAIN) **once per schema change** into a *cascade* (base table → the views reading it, transitively), pushed into the cache, so a write to a base table also bumps every view over it and evicts a pipe reading the view; a view redefined in place (and the views built on it) is evicted at the next refresh. A materialized view's `TO` target rides the same cascade: `CREATE MATERIALIZED VIEW mv TO target` makes `target` an ordinary base table that only inserts to the MV's source ever populate, so a source write also bumps the target — and chained targets, and the views over them — evicting a pipe that reads the target table directly (the target is parsed off the rendered `create_table_query`, since `system.tables` grew first-class target columns only in ClickHouse 26.6, and propagated along the `dependencies_table` trigger edges — a join or `dictGet` in the MV's definition never fires it). Resolution is **precise per parameter binding, and otherwise over-resolves — never under**: for a parameter-gated `UNION`, ClickHouse folds the bound predicate so a constant-false arm is pruned (`source=web` depends only on `web_events`, not `mobile_events`; resolved and cached per bound query, and emitting the `wavehouse_pipe_dep_tables_pruned_total` metric), while a query ClickHouse can't analyze (a write/DDL pipe, an unreachable server, a missing table) falls back to a single **database-wide version** that every write bumps, so any write evicts it — O(1) per request, and it holds even before the schema registry's first successful refresh. The pruning is conservative — an arm is dropped only when its filter is provably constant-false, so anything unproven stays tracked (over-resolve, never stale). Schema refresh is **atomic** (rebuilt only when content changes) and re-resolves pipes (`PipesHandler.ClearResolvedDeps`). Pipes run the author SQL as-is with no policy row/column filtering (the per-pipe `allowed_roles` gates execution), so every allowed caller shares one cached entry. **Unsupported, but bounded** (run, not blocked): write/ingest pipes fail EXPLAIN and take the database-version fallback (a parameter as a whole table name, `FROM {{tbl}}`, binds to a quoted string and errors at execution and resolution alike, so nothing caches — while a name merely assembled from a parameter, `FROM events_{{year}}`, resolves per binding like any bound query); an external read — a table function (`s3()`/`numbers()`, detected as a `TABLE_FUNCTION` node in the query tree), a cross-database table (direct or a `joinGet`/`dictGet` target), or a dictionary without a local backing table — can never be evicted by a write, so its result is TTL-capped (~10s) instead of cached at full TTL. Single-database (the configured `clickhouse.database`). The version-folded cache key is snapshotted **before** the query executes and reused for its `Set` (`Cache.Key`), so a write landing mid-query orphans the entry rather than re-homing the pre-write result under the post-bump key — neither cached read path (pipes or structured queries) can serve a result older than the write that should have evicted it. Degraded modes fail toward expiry, not staleness: a structured query on an unfoldable view and a dependency set that was dead-branch pruned are TTL-capped (`cache.UnresolvedDepsTTLCap` — the pruned cap is a belt-and-suspenders bound on a hypothetical wrong prune, with an integration test pinning that pruning still fires on the shipped ClickHouse version), and a transient EXPLAIN failure (an unreachable server, a canceled request) is retried on the pipe's next execution instead of memoized as permanent fallback. Schema refreshes are serialized end-to-end so overlapping refreshes can't install an older cascade over a newer one, and a refresh that lands mid-resolution wins over the in-flight result (generation-gated memo insert). The degraded modes are metered: `wavehouse_pipe_dep_resolution_fallback_total` and `wavehouse_pipe_dep_unresolved_total` count degraded *executions* (cache hits included — the share of pipe traffic on the fallback / TTL-capped), and `wavehouse_pipe_dep_memo_evictions_total` counts resolution-memo evictions past its fixed 50k cap (sustained growth = a high-cardinality pipe parameter).
 - **Non-JWT operator key for full-access admin + break-glass recovery** (`internal/auth/auth.go`, `internal/auth/context.go`, `internal/auth/auth_test.go`, `internal/api/router.go`, `internal/api/router_test.go`, `internal/config/config.go`, `internal/config/config_test.go`, `cmd/wavehouse/main.go`, `config.yaml`, `deployments/compose/standalone.yaml`, `tests/e2e/fixtures/config.yaml`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/access-control.mdx`, `docs/src/content/docs/api.md`, `docs/src/content/docs/deployment.md`, `docs/src/content/docs/development.md`, `docs/src/content/docs/architecture.md`, `docs/src/content/docs/reverse-proxy.mdx`, `AGENTS.md`, `SECURITY.md`): closes #240; partially advances the auth-hardening epic [#228](https://github.com/Wave-RF/WaveHouse/issues/228) and the management/data-plane split [#359](https://github.com/Wave-RF/WaveHouse/issues/359). Adds an optional `auth.operator_key` (`WH_AUTH_OPERATOR_KEY`): a request presenting it — via an `Authorization: Operator <key>` header (forwarded verbatim by proxies, no collision with Bearer JWTs) or the `X-Operator-Key` alias — is authorized as a full-access platform operator — the whole data plane *and* the `/v1/admin/*` management surface — without minting a JWT and independently of the token verifier, giving the person running the deployment a role-free credential for bootstrap and break-glass. The middleware checks it before the Bearer token with a constant-time comparison (`crypto/subtle`) and stamps two things into the request context: the live `admin_role` (so the policy evaluator's admin bypass grants unrestricted data-plane access while a policy exists) and a platform-operator bit that `RequireAdmin` honors **even when the policy is `nil`/deleted** — the one HTTP path that can restore a wiped policy, which previously required SSH access and a reboot. Empty (the default) disables it, so existing deployments are unchanged; treat it as an admin secret (load from a secret store, serve only over TLS). A successful operator authentication is audit-logged at `INFO`; a request presenting a *non-matching* operator key is logged at `WARN` and counted by a new `wavehouse_auth_operator_key_failures_total` counter (a probing/brute-force signal on the most privileged credential in the system) before falling through to the normal token/default path — the middleware still never rejects. The `Authorization` auth-scheme is matched case-insensitively (RFC 7235) via a shared `authScheme` helper, which also makes the existing `Bearer` JWT scheme case-insensitive (previously it required the canonical `Bearer` casing). Explicitly out of scope, tracked in #359: scoping the operator credential to the management surface only, and capability-scoped admin permissions.
 - **Missing-dedupe-id observability + optional strict mode** (`internal/api/ingest.go`, `internal/api/ingest_test.go`, `internal/config/config.go`, `cmd/wavehouse/main.go`, `config.yaml`, `deployments/compose/standalone.yaml`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/api.md`, `docs/src/content/docs/architecture.md`): closes #219. With dedupe enabled, a row missing the configured `id_field` can't be deduped — previously it was published with idempotency silently disabled and *no* log or metric, so a producer bug that dropped the id turned off the guarantee for those rows unnoticed. Now every such row is logged at `WARN` and counted by a new `wavehouse_ingest_dedupe_missing_id_total` counter (labeled by `table`), making the loss observable server-side. A new opt-in `dedupe.require_id` (`WH_DEDUPE_REQUIRE_ID`, default `false`) turns that signal into enforcement: a row missing the id is rejected (`400` for a single insert; a per-record failure in a batch) instead of published — a tripwire for producers that must guarantee the id (complements the client-side [#202](https://github.com/Wave-RF/WaveHouse/issues/202)). Default behavior is unchanged.
 - **"Durability & Storage" operations guide** (`docs/src/content/docs/durability.md` (new), `docs/src/config/sidebar.ts`, `docs/src/content/docs/reverse-proxy.mdx`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/deployment.md`): documents #84. A new Operations page making the embedded-JetStream durability contract explicit before the docs site publishes: a `200` from `POST /v1/ingest` means the event has been `fsync`'d to disk on the node (the server runs with `SyncAlways: true` in `internal/mq/embedded.go`), which makes the storage substrate's `fsync` tail the ingest latency floor. Covers the contract (and how it differs from JetStream's default page-cache-then-periodic-sync mode), why a slow `fsync` tail manifests as `create stream: ... context deadline exceeded` and `503` backpressure, a where-it's-cheap-vs-expensive substrate table (managed cloud block storage and PLP NVMe vs. ZFS-without-SLOG / qcow2-on-`ext4` / spinning disks), an `fio` recipe + verdict bands to measure your own storage (with the macOS `F_FULLFSYNC` honesty caveat), and the symptom checklist. Forward-references the configurable group-commit interval (`mq.sync_interval`, [#139](https://github.com/Wave-RF/WaveHouse/issues/139)) and the planned `wavehouse storage-check` preflight ([#84](https://github.com/Wave-RF/WaveHouse/issues/84)) without claiming either exists yet. Cross-linked from Configuration (Message Queue), Deployment (Persistent Storage), and the Ingest Pipeline's worker-side ack section; no code changes.