From 0c8272f099533b174ff86085cca6694854b23288 Mon Sep 17 00:00:00 2001 From: taitelee Date: Thu, 9 Jul 2026 11:32:51 -0400 Subject: [PATCH 01/15] fix(ingest): canonicalize DateTime column values to RFC 3339 UTC --- CHANGELOG.md | 2 + docs/src/content/docs/api.md | 7 +- go.mod | 2 +- internal/api/boot_chain_test.go | 18 +++ internal/api/ingest.go | 11 ++ internal/api/ingest_test.go | 101 +++++++++++++++++ internal/discovery/discovery.go | 28 ++++- internal/discovery/discovery_test.go | 18 +++ internal/discovery/timestamp.go | 159 +++++++++++++++++++++++++++ internal/discovery/timestamp_test.go | 133 ++++++++++++++++++++++ internal/discovery/validation.go | 13 ++- internal/ingest/worker.go | 5 + internal/ingest/worker_test.go | 3 + tests/e2e/sdk/streaming.test.ts | 50 +++++++++ 14 files changed, 543 insertions(+), 7 deletions(-) create mode 100644 internal/discovery/timestamp.go create mode 100644 internal/discovery/timestamp_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 9b1ab641..151fa816 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -46,6 +46,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **SSE row `DateTime` columns now arrive in canonical RFC 3339 UTC, matching `/v1/query`, instead of the producer's spelling** (`internal/discovery/timestamp.go` (new), `internal/discovery/{discovery.go,validation.go}`, `internal/api/ingest.go`, `internal/ingest/worker.go`, `docs/src/content/docs/api.md`, plus tests in `internal/discovery/{timestamp_test.go (new),discovery_test.go}`, `internal/api/{ingest_test.go,boot_chain_test.go}`, `tests/e2e/sdk/streaming.test.ts`): closes #372. The stream fans out the pre-insert payload verbatim, so a row `DateTime` reached SSE subscribers in whatever spelling the producer sent — typically the zone-less ClickHouse-native form — while `/v1/query` rendered the stored value as RFC 3339 `Z`; JavaScript's `Date.parse` reads a zone-less string as **local** time, so an SDK `liveQuery` stitching backfill + live landed the two paths hours apart (sign-flipping with the viewer's UTC offset — broke ordering and the "N min ago" label in the Stats live feed). Ingest now **canonicalizes** every `DateTime`/`DateTime64` column value to RFC 3339 UTC (fraction truncated to the column's precision, `time.RFC3339Nano`-trimmed — byte-matching `/v1/query`'s `time.Time` marshaling for UTC columns) after validation and before the NATS publish, so the one payload every consumer shares — SSE subscribers, the ClickHouse insert, the DLQ — carries a single unambiguous spelling. Inputs stay liberal (RFC 3339 with any offset, `YYYY-MM-DD[ T]HH:MM:SS[.fff]`, `YYYY-MM-DD`, Unix seconds as a number or digit-string); a zone-less string is interpreted in the column's declared time zone, else the server's (discovered once per schema refresh via `SELECT timezone()`) — the same rule ClickHouse itself applies, so canonicalization never changes which instant is stored (verified against a live ClickHouse). An unparseable timestamp is now a loud per-record `400` naming the column, instead of passing validation and dying at the async insert into the DLQ. The worker's `INSERT … FORMAT JSONEachRow` sets `date_time_input_format=best_effort`, because ClickHouse's default `basic` parser rejects the canonical form's zone suffix (verified live; `best_effort` is a superset, so pre-upgrade messages still in NATS — zone-less strings, Unix numbers — parse exactly as before). Boundaries: `Date`/`Date32` columns pass through untouched (day precision, no zone ambiguity on this path); events published before the upgrade replay in their original spelling until the sweeper retires them; a column with an explicit non-UTC zone renders as `Z` on the stream but in the column's own offset on `/v1/query` — two zone-explicit spellings of the same instant (byte-identical on UTC columns, the norm). + - **Live demo hero feed renders in `event_ts` order instead of SSE arrival order** (`docs/src/components/LiveDemo.astro`): the landing-page live activity feed prepended each streamed row to the top in the order the SSE stream delivered it, but a producer's webhook burst (a single merge-queue cycle fires ~20 events) arrives in no guaranteed order and the stream relays it in ingest order — so a late or out-of-order delivery landed above newer rows (e.g. a `pushed 12m ago` sitting on top of `reviewed a pull request 9m ago`). `addRow` now keeps the feed sorted by `event_ts` descending — it slots each row in before the first strictly-older sibling rather than blind-prepending — so the live tail matches the already-sorted `gh_activity_recent` backfill. The zone-less-SSE-timestamp normalization the sort relies on (`normTs`) was already in place; equal-second rows keep arrival order (`gh_events.event_ts` is only second-granular for CI/checks, so there's no finer tiebreak), and dedup + the `MAX_ROWS` trim are unchanged. Surfaced in dogfooding on `wavehouse.dev`; the client-side analog of the ingest-order reality the SSE stream can't reorder. - **SSE streams emit a periodic keepalive comment so quiet connections survive proxy and tunnel idle timeouts** (`internal/stream/` (new package: `subscriber.go`, `bucket.go`, `heartbeat.go` + tests), `internal/api/{stream,stream_test}.go`, `internal/config/{config,config_test}.go`, `cmd/wavehouse/main.go`, `config.yaml`, `docs/src/content/docs/{reverse-proxy.mdx,api.md,configuration.mdx,architecture.md,deployment.md}`): closes #226. The stream handler (`internal/stream/metrics.go`, `internal/api/{hub,hub_test}.go` also touched) wrote a single `: connected` comment on open and then sent nothing until an event arrived, so on a quiet table an intermediary's idle timeout reset the connection — Cloudflare's edge (and a Cloudflare Tunnel) dropped quiet streams about every two minutes in dogfooding, and every `curl`/server-side reconnect re-ran NATS gap-fill (browser `EventSource` masked it by auto-reconnecting). A single shared `Heartbeater` goroutine now drives keepalives for every live connection: connections are spread across a ring of buckets and one bucket is pushed a minimal `:` SSE keepalive comment per tick — the writes don't all fire at the same instant, and the per-connection period comes from one timer instead of a `time.Ticker` per connection. The user-facing knob is **`stream.keepalive_interval`** (`WH_STREAM_KEEPALIVE_INTERVAL`, default **30s**) — the longest a quiet stream goes without a write — chosen to clear the common 55–60s idle windows (nginx/ingress-nginx `proxy_read_timeout`, AWS ALB, Heroku) with ~2× margin while still clearing Cloudflare's ~120s edge; `stream.keepalive_buckets` (`WH_STREAM_KEEPALIVE_BUCKETS`, default 3) is an advanced load-spreading knob and the wheel ticks every `keepalive_interval ÷ keepalive_buckets`, so one rotation always spans exactly the interval regardless of bucket count. The wheel lives in a new `internal/stream` package (groundwork for #294, which will move the broadcast hub in alongside it) and exposes a `Bucket` interface (push a byte slice to a set of connections) so the #294 delivery-path throughput work can reuse it to serialize once per (role, table) rather than per subscriber. The keepalive is a standard SSE comment (ignored by `EventSource` and spec-compliant parsers; `curl` just prints it), and a failed keepalive write doubles as a liveness check that ends the handler once a connection has gone away. The reverse-proxy guide gains a per-provider/per-software idle-timeout reference table (with source links and how-to-change notes, plus the absolute-cap exceptions a keepalive can't fix, e.g. Envoy's 15s route timeout), and `-race` tests cover the concurrent connect/disconnect teardown path (the wheel pushing to a connection that is mid-teardown); raising the proxy idle timeout for `/v1/stream` is now optional rather than required. Streams are observed through metrics rather than per-event traces: the handler records `wavehouse_sse_active_streams`, `wavehouse_sse_stream_duration_seconds`, and `wavehouse_sse_frames_sent_total` / `wavehouse_sse_bytes_sent_total` (labeled by `kind`: `keepalive`, `event`, `replay`), and the per-event `SSE.PushEvent` span was dropped — the router already excludes `/v1/stream` from HTTP tracing, and a span per delivered event per subscriber is high-volume, low-value, and existed only to read the hub's `trace_headers` envelope, now collapsed so the hub broadcasts the raw event bytes instead of base64-wrapping them. diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index 3302b5af..617bbe54 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -217,6 +217,8 @@ The body is a **flat JSON object** whose keys must match column names in the tar - Type compatibility: `String`/`DateTime`/`UUID`/`Enum`/`IPv*` accept JSON strings; `Int*`/`Float*`/`Decimal` accept JSON numbers; `Bool` accepts JSON booleans or numbers; `Array` accepts JSON arrays; `Map`/`Tuple` accept JSON objects. - `Nullable()` and `LowCardinality()` wrappers are handled transparently. +**Timestamp canonicalization:** a `DateTime`/`DateTime64` column value may be sent as RFC 3339 (any offset), `YYYY-MM-DD[ T]HH:MM:SS[.fff]` or `YYYY-MM-DD` (zone-less — interpreted in the column's time zone, else the ClickHouse server's, exactly as ClickHouse itself would), or Unix seconds (a JSON number, or the same digits as a string). Whatever the input spelling, WaveHouse rewrites the value to **one canonical wire form — RFC 3339 UTC** (`2026-06-21T04:00:00Z`, fraction truncated to the column's precision) — before publishing, so the stored instant never changes but every downstream consumer (the ClickHouse insert, [SSE subscribers](#get-v1stream--server-sent-events-stream), the DLQ) sees the same spelling `/v1/query` renders. An unparseable timestamp is rejected with a `400` naming the column (previously it surfaced only at the async insert, via the DLQ). `Date`/`Date32` columns are passed through untouched. + **Response (accepted):** ```json @@ -235,6 +237,7 @@ The body is a **flat JSON object** whose keys must match column names in the tar | ------ | ---- | ----- | | 400 | `{"error":"invalid json"}` | Malformed request body | | 400 | `{"error":"unknown column ... for table ..."}` (also: `missing required column ...`, `type mismatch for column ...`, `null value for non-nullable column ...`) | Schema validation failure (unknown fields, type mismatches, missing required columns, null in a non-nullable column). The body is the validator's message verbatim — there is no `validation failed:` prefix. | +| 400 | `{"error":"column \"ts\": unrecognized timestamp ..."}` | A `DateTime`/`DateTime64` column value that parses as none of the accepted timestamp forms (see timestamp canonicalization above). In a batch this is a per-record failure. | | 400 | `{"error":"missing dedupe id field \"event_id\""}` | Only when dedupe is enabled with `dedupe.require_id: true` and the row lacks the configured `id_field`. With `require_id: false` (the default) the row is instead published un-deduped. Either way — reject or publish — the row is logged at `WARN` and counted by `wavehouse_ingest_dedupe_missing_id_total`. In a batch this is a per-record failure, not a whole-request error. | | 401 | `{"error":"invalid token"}` / `{"error":"token expired"}` | A present-but-invalid/expired token was supplied and denied (the gate surfaces the token reason rather than silently falling back to `default_role`) | | 403 | `{"error":"forbidden"}` (empty-role variant: `forbidden: request has no role and no public default_role is configured`) | The resolved role lacks `insert` on the table | @@ -539,6 +542,8 @@ data: {"table_name":"clicks","received_timestamp":"2026-03-24T12:00:01.456Z","da Each SSE connection is bound to a single `?table=`; to consume multiple tables, open one connection per table. +Row `DateTime`/`DateTime64` column values inside `data` arrive in the canonical RFC 3339 UTC form (ingest rewrites them before publishing — see [timestamp canonicalization](#post-v1ingesttabletable--ingest-data)), so a live event and a `/v1/query` read of the same row agree on the instant in zone-explicit form — a zone-less spelling no longer parses as local time in a browser ([#372](https://github.com/Wave-RF/WaveHouse/issues/372)). For a column stored in UTC (the norm) the two renderings are byte-identical; a column declared with a non-UTC zone streams as `Z` while `/v1/query` renders it in the column's own offset — the same instant either way. Events ingested before this behavior shipped replay in whatever spelling they were published with. + **Note:** When access control policies are active, streamed events are filtered per the caller's role — denied columns are removed and tables without select permission are skipped. **CORS:** `/v1/stream` honors the `server.cors_allowed_origins` allowlist like every endpoint, so a browser `EventSource` from an allowed origin connects normally. `Last-Event-ID` is allow-listed in the CORS preflight so fetch-based clients can resume cross-origin. @@ -768,7 +773,7 @@ The message format used on NATS JetStream between ingest and the batch consumer: | ----- | ---- | ----------- | | `table_name` | string | Target ClickHouse table (from URL). | | `received_timestamp` | string | RFC 3339 nano timestamp when WaveHouse received the event. | -| `data` | object | The original flat JSON body. | +| `data` | object | The flat JSON body, with `DateTime`/`DateTime64` column values rewritten to canonical RFC 3339 UTC (see [timestamp canonicalization](#post-v1ingesttabletable--ingest-data)); other values as originally sent. | ### Client-Facing Format (SSE) diff --git a/go.mod b/go.mod index 7a1c7d32..0ea6370e 100644 --- a/go.mod +++ b/go.mod @@ -1,6 +1,6 @@ module github.com/Wave-RF/WaveHouse -go 1.26.4 +go 1.26.5 tool ( github.com/Zxilly/go-size-analyzer/cmd/gsa diff --git a/internal/api/boot_chain_test.go b/internal/api/boot_chain_test.go index 4f5b22ad..b4cb8be7 100644 --- a/internal/api/boot_chain_test.go +++ b/internal/api/boot_chain_test.go @@ -38,6 +38,24 @@ func (c *errsThenSuccessConn) Query(_ context.Context, _ string, _ ...any) (driv return &chainEmptyRows{}, nil } +// QueryRow answers the SELECT timezone() probe Refresh issues before the +// system.columns query (#372); the error sequencing above stays keyed on Query. +func (c *errsThenSuccessConn) QueryRow(context.Context, string, ...any) driver.Row { + return chainTZRow{} +} + +type chainTZRow struct{ driver.Row } + +func (chainTZRow) Scan(dest ...any) error { + if len(dest) == 1 { + if s, ok := dest[0].(*string); ok { + *s = "UTC" + return nil + } + } + return errors.New("unexpected timezone scan") +} + type chainEmptyRows struct{ driver.Rows } func (*chainEmptyRows) Next() bool { return false } diff --git a/internal/api/ingest.go b/internal/api/ingest.go index f283c59a..abc23b31 100644 --- a/internal/api/ingest.go +++ b/internal/api/ingest.go @@ -401,6 +401,17 @@ func (h *IngestHandler) processRecord( } } + // Canonicalize timestamp columns to the RFC 3339 UTC wire form, so the one + // payload published below reaches every consumer — SSE subscribers, the + // ClickHouse insert, the DLQ — in a single unambiguous spelling (#372). After + // the permission checks so check-clause comparisons keep their pre-#372 + // semantics (and claim-injected values are covered); before dedup so a + // rejected record never marks a dedup key. + if err := discovery.CanonicalizeTimestamps(schema, data, h.Registry.ServerTimezone()); err != nil { + h.logger.WarnContext(ctx, "timestamp canonicalization failed", "error", err, "table", table) + return false, &recordReject{Status: http.StatusBadRequest, Message: err.Error()}, nil + } + // Optional deduplication. if h.Dedup != nil && h.IDField != "" { idVal, ok := data[h.IDField] diff --git a/internal/api/ingest_test.go b/internal/api/ingest_test.go index 129aa965..7a4cbaca 100644 --- a/internal/api/ingest_test.go +++ b/internal/api/ingest_test.go @@ -1379,3 +1379,104 @@ func TestIngest_BodyCap_413(t *testing.T) { }) } } + +// tsRegistry returns a registry whose events table carries the timestamp column +// shapes the #372 canonicalization path rewrites. +func tsRegistry() *discovery.SchemaRegistry { + return discovery.NewSchemaRegistryFromMap([]*discovery.TableSchema{ + { + Name: "events", + Columns: []discovery.Column{ + {Name: "name", Type: "String"}, + {Name: "ts", Type: "DateTime('UTC')", HasDefault: true}, + {Name: "ts_ms", Type: "DateTime64(3, 'UTC')", HasDefault: true}, + }, + }, + }) +} + +// publishedData decodes the inner data object of the last published envelope — +// what the stream fans out and the worker inserts. +func publishedData(t *testing.T, pub *testutil.MockPublisher) map[string]any { + t.Helper() + msg := pub.LastMessage() + require.NotNil(t, msg) + var evt struct { + Data map[string]any `json:"data"` + } + require.NoError(t, json.Unmarshal(msg.Data, &evt)) + return evt.Data +} + +// TestIngest_TimestampsCanonicalized is the #372 contract: whatever spelling a +// producer uses, the published payload — the one copy SSE subscribers, the +// ClickHouse insert, and the DLQ all consume — carries RFC 3339 UTC. +func TestIngest_TimestampsCanonicalized(t *testing.T) { + t.Parallel() + pub := &testutil.MockPublisher{} + h := NewIngestHandler(tsRegistry(), pub, testutil.NopLogger()) + + req := ingestRequest(t, "events", map[string]any{ + "name": "e", + "ts": "2026-06-21 04:00:00", // zone-less ClickHouse-native form + "ts_ms": 1782014400.5, // Unix seconds + }) + w := httptest.NewRecorder() + h.Handle(w, req) + + require.Equal(t, http.StatusOK, w.Code) + data := publishedData(t, pub) + assert.Equal(t, "2026-06-21T04:00:00Z", data["ts"]) + assert.Equal(t, "2026-06-21T04:00:00.5Z", data["ts_ms"]) + assert.Equal(t, "e", data["name"], "non-timestamp columns untouched") +} + +// TestIngest_TimestampGarbage_400: an unparseable timestamp is rejected loudly at +// ingest, naming the column — not accepted and left to die at the async insert. +func TestIngest_TimestampGarbage_400(t *testing.T) { + t.Parallel() + pub := &testutil.MockPublisher{} + h := NewIngestHandler(tsRegistry(), pub, testutil.NopLogger()) + + req := ingestRequest(t, "events", map[string]any{"name": "e", "ts": "banana"}) + w := httptest.NewRecorder() + h.Handle(w, req) + + assert.Equal(t, http.StatusBadRequest, w.Code) + assert.Contains(t, w.Body.String(), `column \"ts\"`) + assert.Nil(t, pub.LastMessage(), "nothing published for a rejected record") +} + +// TestIngest_Batch_BadTimestampIsolatedPerRecord: one bad timestamp fails its own +// record; the rest of the batch still publishes (the #195 isolation contract). +func TestIngest_Batch_BadTimestampIsolatedPerRecord(t *testing.T) { + t.Parallel() + pub := &testutil.MockPublisher{} + h := NewIngestHandler(tsRegistry(), pub, testutil.NopLogger()) + + req := ingestRequest(t, "events", []map[string]any{ + {"name": "a", "ts": "2026-06-21T04:00:00Z"}, + {"name": "b", "ts": "banana"}, + {"name": "c", "ts": float64(1782014400)}, + }) + w := httptest.NewRecorder() + h.Handle(w, req) + + require.Equal(t, http.StatusOK, w.Code) + var result struct { + Total int `json:"total"` + Succeeded int `json:"succeeded"` + Failed int `json:"failed"` + Results []struct { + Index int `json:"index"` + Error string `json:"error"` + } `json:"results"` + } + require.NoError(t, json.Unmarshal(w.Body.Bytes(), &result)) + assert.Equal(t, 3, result.Total) + assert.Equal(t, 2, result.Succeeded) + assert.Equal(t, 1, result.Failed) + require.Len(t, result.Results, 3) + assert.Contains(t, result.Results[1].Error, `column "ts"`) + assert.Len(t, pub.Messages, 2, "the two good records published") +} diff --git a/internal/discovery/discovery.go b/internal/discovery/discovery.go index e3b90c5b..23583c1e 100644 --- a/internal/discovery/discovery.go +++ b/internal/discovery/discovery.go @@ -48,6 +48,7 @@ type SchemaRegistry struct { logger *slog.Logger mu sync.RWMutex tables map[string]*TableSchema + serverTZ *time.Location // ClickHouse server default zone; nil until discovered (⇒ UTC) } // NewSchemaRegistry creates a registry that discovers schemas from system.columns. @@ -67,6 +68,18 @@ func (sr *SchemaRegistry) Refresh(ctx context.Context) error { ctx, span := tracer.Start(ctx, "SchemaRegistry.Refresh") defer span.End() + // The server's default time zone is how ClickHouse interprets zone-less + // timestamp strings on columns without an explicit one; ingest canonicalization + // must apply the same rule so it never changes which instant is stored (#372). + var tzName string + if err := sr.conn.QueryRow(ctx, "SELECT timezone()").Scan(&tzName); err != nil { + return fmt.Errorf("query server timezone: %w", err) + } + serverTZ, err := time.LoadLocation(tzName) + if err != nil { + return fmt.Errorf("load server timezone %q: %w", tzName, err) + } + rows, err := sr.conn.Query(ctx, `SELECT table, name, type, default_kind FROM system.columns @@ -102,12 +115,25 @@ func (sr *SchemaRegistry) Refresh(ctx context.Context) error { sr.mu.Lock() sr.tables = tables + sr.serverTZ = serverTZ sr.mu.Unlock() - sr.logger.Info("schema registry refreshed", "tables", len(tables)) + sr.logger.Info("schema registry refreshed", "tables", len(tables), "server_tz", tzName) return nil } +// ServerTimezone returns the ClickHouse server's default time zone, or UTC when it +// has not been discovered yet (a map-built test registry, or before the first +// successful Refresh). +func (sr *SchemaRegistry) ServerTimezone() *time.Location { + sr.mu.RLock() + defer sr.mu.RUnlock() + if sr.serverTZ == nil { + return time.UTC + } + return sr.serverTZ +} + // Get returns the schema for a table, or nil if not found. func (sr *SchemaRegistry) Get(name string) *TableSchema { sr.mu.RLock() diff --git a/internal/discovery/discovery_test.go b/internal/discovery/discovery_test.go index 67558be5..8e8eb9da 100644 --- a/internal/discovery/discovery_test.go +++ b/internal/discovery/discovery_test.go @@ -109,6 +109,24 @@ func (c *fakeConn) Query(_ context.Context, _ string, _ ...any) (driver.Rows, er return &emptyRows{}, nil } +// QueryRow answers the SELECT timezone() probe Refresh issues before the +// system.columns query (#372); errsThenSuccess sequencing stays keyed on Query. +func (c *fakeConn) QueryRow(context.Context, string, ...any) driver.Row { + return tzRow{} +} + +type tzRow struct{ driver.Row } + +func (tzRow) Scan(dest ...any) error { + if len(dest) == 1 { + if s, ok := dest[0].(*string); ok { + *s = "UTC" + return nil + } + } + return errors.New("unexpected timezone scan") +} + type emptyRows struct { driver.Rows } diff --git a/internal/discovery/timestamp.go b/internal/discovery/timestamp.go new file mode 100644 index 00000000..2775c9f8 --- /dev/null +++ b/internal/discovery/timestamp.go @@ -0,0 +1,159 @@ +package discovery + +import ( + "encoding/json" + "fmt" + "math" + "strconv" + "strings" + "time" +) + +// Ingest canonicalizes every DateTime/DateTime64 column value to one wire form — +// RFC 3339 UTC (`2026-06-21T04:00:00Z`, fraction per column precision) — before the +// event is published (#372). The event payload is fanned out verbatim to SSE +// subscribers and inserted into ClickHouse as-is, so without one canonical form the +// same instant reaches stream consumers in whatever spelling the producer chose +// (zone-less ClickHouse-native, Unix seconds, …) while /v1/query renders the stored +// value as RFC 3339 — the two read paths land on different clocks. ClickHouse itself +// discards the input spelling at insert, so canonicalizing costs nothing on the +// query path and makes the streamed form match it. + +// IsTimestampType reports whether chType is a ClickHouse DateTime or DateTime64 +// (unwrapping Nullable/LowCardinality modifiers). Date/Date32 are deliberately not +// included: they are day-precision values with no zone or spelling ambiguity on the +// paths #372 covers, so ingest passes them through untouched. +func IsTimestampType(chType string) bool { + return strings.HasPrefix(unwrapType(chType), "DateTime") +} + +// CanonicalizeTimestamps rewrites every DateTime/DateTime64 column value in data to +// the canonical RFC 3339 UTC form, in place. Zone-less values are interpreted the +// way ClickHouse itself would parse them — in the column's declared time zone, else +// serverTZ (the ClickHouse server default; nil ⇒ UTC) — so canonicalization never +// changes which instant is stored, only how it is spelled. The fraction is truncated +// to the column's precision so the streamed value equals what ClickHouse stores and +// /v1/query returns. Absent and null values are left untouched (defaults and +// Nullable columns). An unparseable value returns an error naming the column, which +// ingest maps to a per-record 400 — the same class of failure previously surfaced +// only at the async insert, via the DLQ. +func CanonicalizeTimestamps(schema *TableSchema, data map[string]any, serverTZ *time.Location) error { + for _, col := range schema.Columns { + if !IsTimestampType(col.Type) { + continue + } + v, ok := data[col.Name] + if !ok || v == nil { + continue + } + precision, loc, err := timestampSpec(col.Type, serverTZ) + if err != nil { + return fmt.Errorf("column %q: %w", col.Name, err) + } + t, err := parseTimestamp(v, loc) + if err != nil { + return fmt.Errorf("column %q: %w", col.Name, err) + } + data[col.Name] = canonicalTimestamp(t, precision) + } + return nil +} + +// timestampSpec extracts a DateTime/DateTime64 type's sub-second precision and time +// zone: `DateTime` / `DateTime('TZ')` / `DateTime64(P)` / `DateTime64(P, 'TZ')`. +// A type without an explicit zone uses serverTZ (nil ⇒ UTC), matching how ClickHouse +// interprets zone-less strings for that column. +func timestampSpec(chType string, serverTZ *time.Location) (precision int, loc *time.Location, err error) { + if serverTZ == nil { + serverTZ = time.UTC + } + t := unwrapType(chType) + + var args []string + if open := strings.IndexByte(t, '('); open != -1 && strings.HasSuffix(t, ")") { + for arg := range strings.SplitSeq(t[open+1:len(t)-1], ",") { + args = append(args, strings.TrimSpace(arg)) + } + t = t[:open] + } + + if t == "DateTime64" { + if len(args) == 0 { + return 0, nil, fmt.Errorf("malformed type %q: DateTime64 requires a precision", chType) + } + precision, err = strconv.Atoi(args[0]) + if err != nil || precision < 0 || precision > 9 { + return 0, nil, fmt.Errorf("malformed type %q: bad precision %q", chType, args[0]) + } + args = args[1:] + } + + if len(args) == 0 { + return precision, serverTZ, nil + } + name := strings.Trim(args[0], "'") + loc, err = time.LoadLocation(name) + if err != nil { + return 0, nil, fmt.Errorf("unknown time zone %q in type %q: %w", name, chType, err) + } + return precision, loc, nil +} + +// parseTimestamp converts one ingested value into a time.Time. Accepted forms are +// the ones ClickHouse itself accepts on this path: RFC 3339 (zone-explicit), +// `YYYY-MM-DD[ T]HH:MM:SS[.fraction]` and `YYYY-MM-DD` (zone-less, interpreted in +// loc), and Unix seconds as a JSON number or a numeric string. Anything else is an +// error. +func parseTimestamp(v any, loc *time.Location) (time.Time, error) { + switch x := v.(type) { + case string: + if t, err := time.Parse(time.RFC3339Nano, x); err == nil { + return t, nil + } + // Zone-less forms; Go's Parse accepts an input fraction after the seconds + // even when the layout carries none. + for _, layout := range []string{"2006-01-02 15:04:05", "2006-01-02T15:04:05", "2006-01-02"} { + if t, err := time.ParseInLocation(layout, x, loc); err == nil { + return t, nil + } + } + // A bare number is Unix seconds — the quoted twin of the JSON-number form. + // ClickHouse's own text parsing read digit-strings this way before #372, so + // the form keeps working (non-finite values are not an instant). + if f, err := strconv.ParseFloat(x, 64); err == nil && !math.IsNaN(f) && !math.IsInf(f, 0) { + return unixToTime(f), nil + } + return time.Time{}, fmt.Errorf( + "unrecognized timestamp %.64q (accepted: RFC 3339, 'YYYY-MM-DD[ T]HH:MM:SS[.fff]', 'YYYY-MM-DD', or Unix seconds)", x) + case float64: + return unixToTime(x), nil + case json.Number: + f, err := x.Float64() + if err != nil { + return time.Time{}, fmt.Errorf("unrecognized timestamp %q: %w", x.String(), err) + } + return unixToTime(f), nil + default: + return time.Time{}, fmt.Errorf("timestamp must be a string or Unix-seconds number, got %T", v) + } +} + +// unixToTime converts fractional Unix seconds to a time.Time (nanoseconds rounded; +// float64 loses sub-microsecond precision near current epochs — producers that need +// exact sub-second values send the string form). +func unixToTime(f float64) time.Time { + sec := math.Floor(f) + return time.Unix(int64(sec), int64(math.Round((f-sec)*1e9))) +} + +// canonicalTimestamp renders t in the canonical wire form: UTC RFC 3339, fraction +// truncated to the column's precision. time.RFC3339Nano trims trailing fractional +// zeros — exactly how /v1/query renders (encoding/json marshals time.Time with the +// same layout), so the two read paths stay byte-identical for UTC columns. +func canonicalTimestamp(t time.Time, precision int) string { + unit := time.Second + for range precision { + unit /= 10 + } + return t.UTC().Truncate(unit).Format(time.RFC3339Nano) +} diff --git a/internal/discovery/timestamp_test.go b/internal/discovery/timestamp_test.go new file mode 100644 index 00000000..2074bed4 --- /dev/null +++ b/internal/discovery/timestamp_test.go @@ -0,0 +1,133 @@ +package discovery + +import ( + "encoding/json" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestIsTimestampType(t *testing.T) { + t.Parallel() + + tests := []struct { + chType string + want bool + }{ + {"DateTime", true}, + {"DateTime('UTC')", true}, + {"DateTime64(3)", true}, + {"DateTime64(6, 'America/New_York')", true}, + {"Nullable(DateTime)", true}, + {"LowCardinality(Nullable(DateTime('UTC')))", true}, + {"Date", false}, // day-precision, no spelling ambiguity — deliberately excluded + {"Date32", false}, // as above + {"String", false}, + {"UInt64", false}, + } + + for _, tt := range tests { + t.Run(tt.chType, func(t *testing.T) { + t.Parallel() + assert.Equal(t, tt.want, IsTimestampType(tt.chType)) + }) + } +} + +// tsSchema builds a one-column schema of the given ClickHouse type, so each case +// exercises exactly one column's canonicalization. +func tsSchema(colType string) *TableSchema { + return &TableSchema{Name: "t", Columns: []Column{{Name: "ts", Type: colType}}} +} + +func TestCanonicalizeTimestamps(t *testing.T) { + t.Parallel() + nyc, err := time.LoadLocation("America/New_York") + require.NoError(t, err) + + tests := []struct { + name string + colType string + serverTZ *time.Location + value any + want any + }{ + {"canonical passes through", "DateTime('UTC')", nil, "2026-06-21T04:00:00Z", "2026-06-21T04:00:00Z"}, + {"offset converts to Z", "DateTime('UTC')", nil, "2026-06-21T06:30:00+02:30", "2026-06-21T04:00:00Z"}, + {"naive space form, column zone", "DateTime('America/New_York')", nil, "2026-06-21 00:00:00", "2026-06-21T04:00:00Z"}, + {"naive T form, column zone", "DateTime('America/New_York')", nil, "2026-06-21T00:00:00", "2026-06-21T04:00:00Z"}, + {"naive form, server zone", "DateTime", nyc, "2026-06-21 00:00:00", "2026-06-21T04:00:00Z"}, + {"naive form, no zone known ⇒ UTC", "DateTime", nil, "2026-06-21 04:00:00", "2026-06-21T04:00:00Z"}, + {"date-only ⇒ midnight in zone", "DateTime('America/New_York')", nil, "2026-06-21", "2026-06-21T04:00:00Z"}, + {"unix seconds number", "DateTime('UTC')", nil, float64(1782014400), "2026-06-21T04:00:00Z"}, + {"unix seconds json.Number", "DateTime('UTC')", nil, json.Number("1782014400"), "2026-06-21T04:00:00Z"}, + {"unix seconds digit-string", "DateTime('UTC')", nil, "1782014400", "2026-06-21T04:00:00Z"}, + {"fractional unix digit-string", "DateTime64(3, 'UTC')", nil, "1782014400.5", "2026-06-21T04:00:00.5Z"}, + {"fractional unix on DateTime64", "DateTime64(3, 'UTC')", nil, 1782014400.5, "2026-06-21T04:00:00.5Z"}, + {"fraction truncated to column precision", "DateTime64(3, 'UTC')", nil, "2026-06-21T04:00:00.123456Z", "2026-06-21T04:00:00.123Z"}, + {"fraction truncated off a second-precision column", "DateTime('UTC')", nil, "2026-06-21 04:00:00.999", "2026-06-21T04:00:00Z"}, + {"trailing fractional zeros trimmed, like /v1/query", "DateTime64(3, 'UTC')", nil, "2026-06-21T04:00:00.120Z", "2026-06-21T04:00:00.12Z"}, + {"Nullable unwraps", "Nullable(DateTime('UTC'))", nil, "2026-06-21 04:00:00", "2026-06-21T04:00:00Z"}, + {"null left for ClickHouse", "Nullable(DateTime)", nil, nil, nil}, + {"String column untouched", "String", nil, "2026-06-21 04:00:00", "2026-06-21 04:00:00"}, + {"Date column untouched (excluded)", "Date", nil, "2026-06-21", "2026-06-21"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + data := map[string]any{"ts": tt.value} + require.NoError(t, CanonicalizeTimestamps(tsSchema(tt.colType), data, tt.serverTZ)) + assert.Equal(t, tt.want, data["ts"]) + }) + } +} + +// TestCanonicalizeTimestamps_AbsentColumn: a column not in the payload (DEFAULT- +// filled by ClickHouse) is left absent, never invented. +func TestCanonicalizeTimestamps_AbsentColumn(t *testing.T) { + t.Parallel() + data := map[string]any{"page": "/home"} + require.NoError(t, CanonicalizeTimestamps( + &TableSchema{Name: "t", Columns: []Column{ + {Name: "ts", Type: "DateTime", HasDefault: true}, + {Name: "page", Type: "String"}, + }}, data, nil)) + assert.Equal(t, map[string]any{"page": "/home"}, data) +} + +func TestCanonicalizeTimestamps_Errors(t *testing.T) { + t.Parallel() + + tests := []struct { + name string + colType string + value any + wantIn string + }{ + {"unrecognized string", "DateTime('UTC')", "banana", `column "ts"`}, + {"non-finite numeric string is not an instant", "DateTime('UTC')", "NaN", "unrecognized timestamp"}, + {"wrong value type", "DateTime('UTC')", true, "must be a string or Unix-seconds number"}, + {"unknown zone in type", "DateTime('Not/AZone')", "2026-06-21 04:00:00", `unknown time zone "Not/AZone"`}, + {"malformed DateTime64 precision", "DateTime64(x)", "2026-06-21 04:00:00", "bad precision"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + t.Parallel() + err := CanonicalizeTimestamps(tsSchema(tt.colType), map[string]any{"ts": tt.value}, nil) + require.Error(t, err) + assert.Contains(t, err.Error(), tt.wantIn) + }) + } +} + +// TestServerTimezone_DefaultsToUTC: a registry that has never refreshed (the +// map-built test form) reports UTC rather than nil. +func TestServerTimezone_DefaultsToUTC(t *testing.T) { + t.Parallel() + reg := NewSchemaRegistryFromMap(nil) + assert.Equal(t, time.UTC, reg.ServerTimezone()) +} diff --git a/internal/discovery/validation.go b/internal/discovery/validation.go index 90af29be..64b41074 100644 --- a/internal/discovery/validation.go +++ b/internal/discovery/validation.go @@ -48,9 +48,9 @@ func Validate(schema *TableSchema, data map[string]any) error { return nil } -// isTypeCompatible checks whether a Go/JSON value can be stored in the given ClickHouse type. -func isTypeCompatible(chType string, val any) bool { - // Robustly unwrap nested modifiers (e.g., LowCardinality(Nullable(String))) +// unwrapType strips Nullable(...) and LowCardinality(...) modifiers (nested in any +// order, e.g. LowCardinality(Nullable(String))) down to the base ClickHouse type. +func unwrapType(chType string) string { for { if strings.HasPrefix(chType, "Nullable(") && strings.HasSuffix(chType, ")") { chType = chType[9 : len(chType)-1] @@ -60,8 +60,13 @@ func isTypeCompatible(chType string, val any) bool { chType = chType[15 : len(chType)-1] continue } - break + return chType } +} + +// isTypeCompatible checks whether a Go/JSON value can be stored in the given ClickHouse type. +func isTypeCompatible(chType string, val any) bool { + chType = unwrapType(chType) switch { // String-compatible types accept Strings, Numbers (coerced), and Bools diff --git a/internal/ingest/worker.go b/internal/ingest/worker.go index e663c2e1..3fec44ef 100644 --- a/internal/ingest/worker.go +++ b/internal/ingest/worker.go @@ -431,6 +431,11 @@ func (w *IngestWorker) insertToClickHouse(ctx context.Context, tableName string, q.Set("database", w.db) q.Set("param_target_table", tableName) q.Set("query", "INSERT INTO {target_table:Identifier} FORMAT JSONEachRow") + // Ingest canonicalizes DateTime values to RFC 3339 UTC (#372), whose zone + // suffix ClickHouse's default 'basic' parser rejects. best_effort is a + // superset: it accepts the canonical form, and pre-#372 messages still in the + // stream (zone-less strings, Unix numbers) parse exactly as before. + q.Set("date_time_input_format", "best_effort") req, err := http.NewRequestWithContext(ctx, "POST", w.chURL+"?"+q.Encode(), &buf) if err != nil { diff --git a/internal/ingest/worker_test.go b/internal/ingest/worker_test.go index b0642ca1..83671a98 100644 --- a/internal/ingest/worker_test.go +++ b/internal/ingest/worker_test.go @@ -323,6 +323,9 @@ func TestInsertToClickHouse_BuildsCorrectRequest(t *testing.T) { assert.Equal(t, "test_db", q.Get("database")) assert.Equal(t, "events", q.Get("param_target_table")) assert.Equal(t, "INSERT INTO {target_table:Identifier} FORMAT JSONEachRow", q.Get("query")) + // #372: canonical RFC 3339 timestamps carry a zone suffix the default + // 'basic' parser rejects, so the insert must opt into best_effort. + assert.Equal(t, "best_effort", q.Get("date_time_input_format")) assert.Equal(t, "application/json", req.Header.Get("Content-Type")) assert.Equal(t, "test_user", req.Header.Get("X-ClickHouse-User")) diff --git a/tests/e2e/sdk/streaming.test.ts b/tests/e2e/sdk/streaming.test.ts index c6b4a5ea..5f406e70 100644 --- a/tests/e2e/sdk/streaming.test.ts +++ b/tests/e2e/sdk/streaming.test.ts @@ -77,6 +77,56 @@ describe("Streaming", () => { } }); + it("row DateTime columns arrive canonicalized, matching /v1/query (#372)", async () => { + // Ingest spells the row timestamp with an offset; the wire form everywhere + // downstream must be canonical RFC 3339 UTC, so the SSE frame and the + // /v1/query rendering of the same stored instant are byte-identical — the + // query/stream clock drift #372 reported. + const whPublic = publicClient(); + const whAuth = dataClient(); + const receivedEvents: any[] = []; + const id = testId(); + + const stream = whPublic.from(T.clicks).stream(); + let unsub: (() => void) | undefined; + try { + unsub = stream.subscribe({ + next: (event) => receivedEvents.push(event), + error: (err) => console.error("SSE error:", err), + }); + await stream.connected(5_000); + + await whAuth.from(T.clicks).insert({ + event_id: id, + page: "/canonical-ts", + user_id: "canon-user", + session_id: "canon-sess", + received_timestamp: "2026-06-21T06:00:00.123+02:00", + }); + + await waitForCondition(() => receivedEvents.some((e) => e.data?.event_id === id), 10_000); + const frame = receivedEvents.find((e) => e.data?.event_id === id); + expect(frame?.data.received_timestamp).toBe("2026-06-21T04:00:00.123Z"); + + // The ClickHouse insert is async behind the stream event — poll the query + // path until the row lands, then compare the two renderings. + let queried: any[] = []; + await waitForCondition(async () => { + const result = await whAuth + .from(T.clicks) + .select("event_id", "received_timestamp") + .where("event_id", "=", id) + .fetch(); + queried = result.data ?? []; + return queried.length === 1; + }, 15_000); + expect(queried[0].received_timestamp).toBe(frame?.data.received_timestamp); + } finally { + if (unsub) unsub(); + stream.close(); + } + }); + it("receives events after insert (authenticated via ?token=)", async () => { const whAuth = dataClient(); const receivedEvents: any[] = []; From e6641791dbc60d3933553289e19b7070cac32d3d Mon Sep 17 00:00:00 2001 From: taitelee Date: Thu, 9 Jul 2026 12:22:01 -0400 Subject: [PATCH 02/15] fix(discovery): precompute timestamp column specs at refresh and embed tzdata --- CHANGELOG.md | 2 +- cmd/wavehouse/main.go | 7 ++ docs/src/content/docs/architecture.md | 6 +- internal/api/ingest.go | 5 +- internal/api/ingest_test.go | 1 + internal/discovery/discovery.go | 41 +++++++---- internal/discovery/timestamp.go | 101 +++++++++++++++++++------- internal/discovery/timestamp_test.go | 71 ++++++++++++++++-- 8 files changed, 179 insertions(+), 55 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 78c5375a..a7a70716 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,7 +50,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed -- **SSE row `DateTime` columns now arrive in canonical RFC 3339 UTC, matching `/v1/query`, instead of the producer's spelling** (`internal/discovery/timestamp.go` (new), `internal/discovery/{discovery.go,validation.go}`, `internal/api/ingest.go`, `internal/ingest/worker.go`, `docs/src/content/docs/api.md`, plus tests in `internal/discovery/{timestamp_test.go (new),discovery_test.go}`, `internal/api/{ingest_test.go,boot_chain_test.go}`, `tests/e2e/sdk/streaming.test.ts`): closes #372. The stream fans out the pre-insert payload verbatim, so a row `DateTime` reached SSE subscribers in whatever spelling the producer sent — typically the zone-less ClickHouse-native form — while `/v1/query` rendered the stored value as RFC 3339 `Z`; JavaScript's `Date.parse` reads a zone-less string as **local** time, so an SDK `liveQuery` stitching backfill + live landed the two paths hours apart (sign-flipping with the viewer's UTC offset — broke ordering and the "N min ago" label in the Stats live feed). Ingest now **canonicalizes** every `DateTime`/`DateTime64` column value to RFC 3339 UTC (fraction truncated to the column's precision, `time.RFC3339Nano`-trimmed — byte-matching `/v1/query`'s `time.Time` marshaling for UTC columns) after validation and before the NATS publish, so the one payload every consumer shares — SSE subscribers, the ClickHouse insert, the DLQ — carries a single unambiguous spelling. Inputs stay liberal (RFC 3339 with any offset, `YYYY-MM-DD[ T]HH:MM:SS[.fff]`, `YYYY-MM-DD`, Unix seconds as a number or digit-string); a zone-less string is interpreted in the column's declared time zone, else the server's (discovered once per schema refresh via `SELECT timezone()`) — the same rule ClickHouse itself applies, so canonicalization never changes which instant is stored (verified against a live ClickHouse). An unparseable timestamp is now a loud per-record `400` naming the column, instead of passing validation and dying at the async insert into the DLQ. The worker's `INSERT … FORMAT JSONEachRow` sets `date_time_input_format=best_effort`, because ClickHouse's default `basic` parser rejects the canonical form's zone suffix (verified live; `best_effort` is a superset, so pre-upgrade messages still in NATS — zone-less strings, Unix numbers — parse exactly as before). Boundaries: `Date`/`Date32` columns pass through untouched (day precision, no zone ambiguity on this path); events published before the upgrade replay in their original spelling until the sweeper retires them; a column with an explicit non-UTC zone renders as `Z` on the stream but in the column's own offset on `/v1/query` — two zone-explicit spellings of the same instant (byte-identical on UTC columns, the norm). +- **SSE row `DateTime` columns now arrive in canonical RFC 3339 UTC, matching `/v1/query`, instead of the producer's spelling** (`internal/discovery/timestamp.go` (new), `internal/discovery/{discovery.go,validation.go}`, `internal/api/ingest.go`, `internal/ingest/worker.go`, `cmd/wavehouse/main.go`, `docs/src/content/docs/{api.md,architecture.md}`, plus tests in `internal/discovery/{timestamp_test.go (new),discovery_test.go}`, `internal/api/{ingest_test.go,boot_chain_test.go}`, `tests/e2e/sdk/streaming.test.ts`): closes #372. The stream fans out the pre-insert payload verbatim, so a row `DateTime` reached SSE subscribers in whatever spelling the producer sent — typically the zone-less ClickHouse-native form — while `/v1/query` rendered the stored value as RFC 3339 `Z`; JavaScript's `Date.parse` reads a zone-less string as **local** time, so an SDK `liveQuery` stitching backfill + live landed the two paths hours apart (sign-flipping with the viewer's UTC offset — broke ordering and the "N min ago" label in the Stats live feed). Ingest now **canonicalizes** every `DateTime`/`DateTime64` column value to RFC 3339 UTC (fraction truncated to the column's precision, `time.RFC3339Nano`-trimmed — byte-matching `/v1/query`'s `time.Time` marshaling for UTC columns) after validation and before the NATS publish, so the one payload every consumer shares — SSE subscribers, the ClickHouse insert, the DLQ — carries a single unambiguous spelling. Inputs stay liberal (RFC 3339 with any offset, `YYYY-MM-DD[ T]HH:MM:SS[.fff]`, `YYYY-MM-DD`, Unix seconds as a number or digit-string); a zone-less string is interpreted in the column's declared time zone, else the server's (discovered once per schema refresh via `SELECT timezone()`) — the same rule ClickHouse itself applies, so canonicalization never changes which instant is stored (verified against a live ClickHouse). Each timestamp column's spec (precision + resolved zone) is precomputed at schema discovery and cached on the column, so the per-record ingest path parses no type strings, loads no zones, and takes no locks; the binary embeds Go's `time/tzdata` (~450 KB) so zone resolution — now load-bearing for schema refresh — doesn't depend on the deploy image shipping a tzdata package, and a zone that still can't resolve fails the refresh loudly rather than silently reinterpreting zone-less values as UTC (#402 review). An unparseable timestamp is now a loud per-record `400` naming the column, instead of passing validation and dying at the async insert into the DLQ. The worker's `INSERT … FORMAT JSONEachRow` sets `date_time_input_format=best_effort`, because ClickHouse's default `basic` parser rejects the canonical form's zone suffix (verified live; `best_effort` is a superset, so pre-upgrade messages still in NATS — zone-less strings, Unix numbers — parse exactly as before). Boundaries: `Date`/`Date32` columns pass through untouched (day precision, no zone ambiguity on this path); events published before the upgrade replay in their original spelling until the sweeper retires them; a column with an explicit non-UTC zone renders as `Z` on the stream but in the column's own offset on `/v1/query` — two zone-explicit spellings of the same instant (byte-identical on UTC columns, the norm). - **Live demo hero feed renders in `event_ts` order instead of SSE arrival order** (`docs/src/components/LiveDemo.astro`): the landing-page live activity feed prepended each streamed row to the top in the order the SSE stream delivered it, but a producer's webhook burst (a single merge-queue cycle fires ~20 events) arrives in no guaranteed order and the stream relays it in ingest order — so a late or out-of-order delivery landed above newer rows (e.g. a `pushed 12m ago` sitting on top of `reviewed a pull request 9m ago`). `addRow` now keeps the feed sorted by `event_ts` descending — it slots each row in before the first strictly-older sibling rather than blind-prepending — so the live tail matches the already-sorted `gh_activity_recent` backfill. The zone-less-SSE-timestamp normalization the sort relies on (`normTs`) was already in place; equal-second rows keep arrival order (`gh_events.event_ts` is only second-granular for CI/checks, so there's no finer tiebreak), and dedup + the `MAX_ROWS` trim are unchanged. Surfaced in dogfooding on `wavehouse.dev`; the client-side analog of the ingest-order reality the SSE stream can't reorder. diff --git a/cmd/wavehouse/main.go b/cmd/wavehouse/main.go index c75eb853..d641b183 100644 --- a/cmd/wavehouse/main.go +++ b/cmd/wavehouse/main.go @@ -14,6 +14,13 @@ import ( "syscall" "time" + // Embed the IANA time zone database (~450 KB). Schema discovery resolves the + // ClickHouse server zone and timestamp-column zones with time.LoadLocation + // (#372), which must not depend on the deploy image shipping a system tzdata + // package — a schema refresh that cannot resolve the server zone fails, and + // stays failed, until tzdata appears. + _ "time/tzdata" + "github.com/ClickHouse/clickhouse-go/v2" "github.com/Wave-RF/WaveHouse/internal/api" "github.com/Wave-RF/WaveHouse/internal/auth" diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 4c6a8958..cb4dd179 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -113,7 +113,8 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ ### `discovery/` — Schema Discovery & Validation -- **discovery.go** — `SchemaRegistry` queries `system.columns` to discover ClickHouse table schemas. Supports periodic auto-refresh, on-demand refresh, and `RetryRefresh` (boot-time exponential backoff loop used by `cmd/wavehouse` so a transiently unreachable ClickHouse doesn't crash-loop the binary). Thread-safe via `sync.RWMutex`. +- **discovery.go** — `SchemaRegistry` queries `system.columns` to discover ClickHouse table schemas. Each refresh also discovers the server's default time zone (`SELECT timezone()`) and bakes every `DateTime`/`DateTime64` column's canonicalization spec (precision + resolved zone) into the cached schema, so the per-record ingest path parses no type strings and loads no zones ([#372](https://github.com/Wave-RF/WaveHouse/issues/372)). Supports periodic auto-refresh, on-demand refresh, and `RetryRefresh` (boot-time exponential backoff loop used by `cmd/wavehouse` so a transiently unreachable ClickHouse doesn't crash-loop the binary). Thread-safe via `sync.RWMutex`. +- **timestamp.go** — `CanonicalizeTimestamps(schema, data)` rewrites every `DateTime`/`DateTime64` value to the canonical RFC 3339 UTC wire form before the event is published (#372): zone-less values are interpreted in the column's declared zone, else the discovered server default — the same rule ClickHouse applies, so the spelling changes but never the instant. An unparseable value is a per-record `400` naming the column. - **validation.go** — `Validate(schema, data)` checks incoming JSON against the discovered schema: unknown fields, type compatibility, missing required columns, null handling. - **discovery_test.go** — Unit tests for validation logic. @@ -167,6 +168,9 @@ Client POST /v1/ingest?table={table} → Validate JSON body against schema (type checks, required columns) → Policy column rules + check clauses (disallowed columns rejected; claim-derived values enforced or injected) + → Canonicalize DateTime/DateTime64 values to RFC 3339 UTC (rewrites the + payload so every consumer shares one spelling; an unparseable value is a + 400 for that record) → Optional deduplication check (configurable ID field; a row missing that field is published un-deduped + logged/counted, or rejected under require_id) → Publish to NATS JetStream (ingest.{table}) diff --git a/internal/api/ingest.go b/internal/api/ingest.go index abc23b31..bb1d7e39 100644 --- a/internal/api/ingest.go +++ b/internal/api/ingest.go @@ -406,8 +406,9 @@ func (h *IngestHandler) processRecord( // ClickHouse insert, the DLQ — in a single unambiguous spelling (#372). After // the permission checks so check-clause comparisons keep their pre-#372 // semantics (and claim-injected values are covered); before dedup so a - // rejected record never marks a dedup key. - if err := discovery.CanonicalizeTimestamps(schema, data, h.Registry.ServerTimezone()); err != nil { + // rejected record never marks a dedup key. Each column's precision and zone + // were resolved at schema discovery, so this is pure computation per record. + if err := discovery.CanonicalizeTimestamps(schema, data); err != nil { h.logger.WarnContext(ctx, "timestamp canonicalization failed", "error", err, "table", table) return false, &recordReject{Status: http.StatusBadRequest, Message: err.Error()}, nil } diff --git a/internal/api/ingest_test.go b/internal/api/ingest_test.go index 7a4cbaca..6201f70c 100644 --- a/internal/api/ingest_test.go +++ b/internal/api/ingest_test.go @@ -1443,6 +1443,7 @@ func TestIngest_TimestampGarbage_400(t *testing.T) { h.Handle(w, req) assert.Equal(t, http.StatusBadRequest, w.Code) + testutil.AssertJSONErrorResponse(t, w) assert.Contains(t, w.Body.String(), `column \"ts\"`) assert.Nil(t, pub.LastMessage(), "nothing published for a rejected record") } diff --git a/internal/discovery/discovery.go b/internal/discovery/discovery.go index 23583c1e..a08ae9f6 100644 --- a/internal/discovery/discovery.go +++ b/internal/discovery/discovery.go @@ -19,6 +19,13 @@ type Column struct { Type string `json:"type"` IsNullable bool `json:"is_nullable"` HasDefault bool `json:"has_default"` + + // tsSpec is a DateTime/DateTime64 column's canonicalization spec, resolved + // once when the schema is built (Refresh / NewSchemaRegistryFromMap) so the + // per-record ingest path parses no type strings and loads no zones. nil for + // non-timestamp columns, hand-built Column literals, and zones Go cannot + // resolve — CanonicalizeTimestamps falls back to per-record resolution. + tsSpec *timestampSpec } // TableSchema holds the discovered schema for one ClickHouse table. @@ -48,7 +55,6 @@ type SchemaRegistry struct { logger *slog.Logger mu sync.RWMutex tables map[string]*TableSchema - serverTZ *time.Location // ClickHouse server default zone; nil until discovered (⇒ UTC) } // NewSchemaRegistry creates a registry that discovers schemas from system.columns. @@ -62,7 +68,10 @@ func NewSchemaRegistry(conn driver.Conn, database string, refreshInterval time.D } } -// Refresh queries system.columns and rebuilds the in-memory schema cache. +// Refresh rebuilds the in-memory schema cache: it discovers the ClickHouse +// server's default time zone (`SELECT timezone()`), queries system.columns, and +// precomputes each timestamp column's canonicalization spec before swapping the +// new cache in. func (sr *SchemaRegistry) Refresh(ctx context.Context) error { tracer := otel.GetTracerProvider().Tracer("wavehouse-discovery") ctx, span := tracer.Start(ctx, "SchemaRegistry.Refresh") @@ -77,6 +86,11 @@ func (sr *SchemaRegistry) Refresh(ctx context.Context) error { } serverTZ, err := time.LoadLocation(tzName) if err != nil { + // Deliberately fails the refresh rather than falling back to UTC: on a + // non-UTC server, a UTC fallback would silently change which instant a + // zone-less string stores. cmd/wavehouse embeds time/tzdata, so this + // resolves even on images without a system zone database; failing here + // means the server's zone is newer than the binary's own tzdata. return fmt.Errorf("load server timezone %q: %w", tzName, err) } @@ -113,27 +127,18 @@ func (sr *SchemaRegistry) Refresh(ctx context.Context) error { ts.Columns = append(ts.Columns, col) } + for _, ts := range tables { + resolveTimestampSpecs(ts, serverTZ, sr.logger) + } + sr.mu.Lock() sr.tables = tables - sr.serverTZ = serverTZ sr.mu.Unlock() sr.logger.Info("schema registry refreshed", "tables", len(tables), "server_tz", tzName) return nil } -// ServerTimezone returns the ClickHouse server's default time zone, or UTC when it -// has not been discovered yet (a map-built test registry, or before the first -// successful Refresh). -func (sr *SchemaRegistry) ServerTimezone() *time.Location { - sr.mu.RLock() - defer sr.mu.RUnlock() - if sr.serverTZ == nil { - return time.UTC - } - return sr.serverTZ -} - // Get returns the schema for a table, or nil if not found. func (sr *SchemaRegistry) Get(name string) *TableSchema { sr.mu.RLock() @@ -228,13 +233,17 @@ func isNullable(chType string) bool { // NewSchemaRegistryFromMap creates a SchemaRegistry pre-loaded with the given // table schemas. Intended for testing — no ClickHouse connection is required. +// Timestamp specs are precomputed like Refresh does (no server to ask ⇒ UTC), so +// handler tests exercise the same precomputed path as production. func NewSchemaRegistryFromMap(tables []*TableSchema) *SchemaRegistry { + logger := slog.New(slog.NewTextHandler(io.Discard, nil)) m := make(map[string]*TableSchema, len(tables)) for _, t := range tables { + resolveTimestampSpecs(t, nil, logger) m[t.Name] = t } return &SchemaRegistry{ tables: m, - logger: slog.New(slog.NewTextHandler(io.Discard, nil)), + logger: logger, } } diff --git a/internal/discovery/timestamp.go b/internal/discovery/timestamp.go index 2775c9f8..cbd57d5e 100644 --- a/internal/discovery/timestamp.go +++ b/internal/discovery/timestamp.go @@ -3,6 +3,7 @@ package discovery import ( "encoding/json" "fmt" + "log/slog" "math" "strconv" "strings" @@ -30,40 +31,84 @@ func IsTimestampType(chType string) bool { // CanonicalizeTimestamps rewrites every DateTime/DateTime64 column value in data to // the canonical RFC 3339 UTC form, in place. Zone-less values are interpreted the // way ClickHouse itself would parse them — in the column's declared time zone, else -// serverTZ (the ClickHouse server default; nil ⇒ UTC) — so canonicalization never -// changes which instant is stored, only how it is spelled. The fraction is truncated -// to the column's precision so the streamed value equals what ClickHouse stores and -// /v1/query returns. Absent and null values are left untouched (defaults and -// Nullable columns). An unparseable value returns an error naming the column, which -// ingest maps to a per-record 400 — the same class of failure previously surfaced -// only at the async insert, via the DLQ. -func CanonicalizeTimestamps(schema *TableSchema, data map[string]any, serverTZ *time.Location) error { +// the server default — so canonicalization never changes which instant is stored, +// only how it is spelled. Each column's precision and zone come from its spec, +// precomputed when the schema was built (Refresh / NewSchemaRegistryFromMap); a +// schema built outside a registry falls back to per-record resolution with UTC as +// the server default. The fraction is truncated to the column's precision so the +// streamed value equals what ClickHouse stores and /v1/query returns. Absent and +// null values are left untouched (defaults and Nullable columns). An unparseable +// value returns an error naming the column, which ingest maps to a per-record 400 — +// the same class of failure previously surfaced only at the async insert, via the +// DLQ. +func CanonicalizeTimestamps(schema *TableSchema, data map[string]any) error { for _, col := range schema.Columns { - if !IsTimestampType(col.Type) { + spec := col.tsSpec + if spec == nil && !IsTimestampType(col.Type) { continue } v, ok := data[col.Name] if !ok || v == nil { continue } - precision, loc, err := timestampSpec(col.Type, serverTZ) - if err != nil { - return fmt.Errorf("column %q: %w", col.Name, err) + if spec == nil { + // No precomputed spec: a hand-built schema, or a zone the schema build + // could not resolve. Resolving here keeps that failure scoped to the + // records that carry the column (a per-record 400), never the schema. + s, err := resolveTimestampSpec(col.Type, nil) + if err != nil { + return fmt.Errorf("column %q: %w", col.Name, err) + } + spec = &s } - t, err := parseTimestamp(v, loc) + t, err := parseTimestamp(v, spec.loc) if err != nil { return fmt.Errorf("column %q: %w", col.Name, err) } - data[col.Name] = canonicalTimestamp(t, precision) + data[col.Name] = canonicalTimestamp(t, spec.precision) } return nil } -// timestampSpec extracts a DateTime/DateTime64 type's sub-second precision and time -// zone: `DateTime` / `DateTime('TZ')` / `DateTime64(P)` / `DateTime64(P, 'TZ')`. -// A type without an explicit zone uses serverTZ (nil ⇒ UTC), matching how ClickHouse -// interprets zone-less strings for that column. -func timestampSpec(chType string, serverTZ *time.Location) (precision int, loc *time.Location, err error) { +// timestampSpec is a timestamp column's precomputed canonicalization inputs: the +// DateTime64 sub-second precision (0 for DateTime) and the zone in which zone-less +// values are interpreted (the column's declared zone, else the ClickHouse server +// default at resolve time). +type timestampSpec struct { + precision int + loc *time.Location +} + +// resolveTimestampSpecs precomputes the timestampSpec of every DateTime/DateTime64 +// column in ts, in place — run once per schema build so the per-record ingest path +// never parses type strings, loads zones, or allocates. A column whose zone Go +// cannot resolve keeps a nil spec and is logged, not fatal: ClickHouse validated +// the name at CREATE TABLE, so a miss means tzdata skew between the ClickHouse +// server and this binary (cmd/wavehouse embeds time/tzdata to keep that window +// minimal). Ingest then resolves that one column per record and rejects its values +// with the same error — per-record 400s on one column, not a failed refresh for +// the whole schema. +func resolveTimestampSpecs(ts *TableSchema, serverTZ *time.Location, logger *slog.Logger) { + for i := range ts.Columns { + col := &ts.Columns[i] + if !IsTimestampType(col.Type) { + continue + } + spec, err := resolveTimestampSpec(col.Type, serverTZ) + if err != nil { + logger.Warn("cannot resolve timestamp column spec; its ingest values will be rejected", + "table", ts.Name, "column", col.Name, "type", col.Type, "error", err) + continue + } + col.tsSpec = &spec + } +} + +// resolveTimestampSpec extracts a DateTime/DateTime64 type's sub-second precision +// and time zone: `DateTime` / `DateTime('TZ')` / `DateTime64(P)` / +// `DateTime64(P, 'TZ')`. A type without an explicit zone uses serverTZ (nil ⇒ +// UTC), matching how ClickHouse interprets zone-less strings for that column. +func resolveTimestampSpec(chType string, serverTZ *time.Location) (timestampSpec, error) { if serverTZ == nil { serverTZ = time.UTC } @@ -77,26 +122,28 @@ func timestampSpec(chType string, serverTZ *time.Location) (precision int, loc * t = t[:open] } + var precision int if t == "DateTime64" { if len(args) == 0 { - return 0, nil, fmt.Errorf("malformed type %q: DateTime64 requires a precision", chType) + return timestampSpec{}, fmt.Errorf("malformed type %q: DateTime64 requires a precision", chType) } - precision, err = strconv.Atoi(args[0]) - if err != nil || precision < 0 || precision > 9 { - return 0, nil, fmt.Errorf("malformed type %q: bad precision %q", chType, args[0]) + p, err := strconv.Atoi(args[0]) + if err != nil || p < 0 || p > 9 { + return timestampSpec{}, fmt.Errorf("malformed type %q: bad precision %q", chType, args[0]) } + precision = p args = args[1:] } if len(args) == 0 { - return precision, serverTZ, nil + return timestampSpec{precision: precision, loc: serverTZ}, nil } name := strings.Trim(args[0], "'") - loc, err = time.LoadLocation(name) + loc, err := time.LoadLocation(name) if err != nil { - return 0, nil, fmt.Errorf("unknown time zone %q in type %q: %w", name, chType, err) + return timestampSpec{}, fmt.Errorf("unknown time zone %q in type %q: %w", name, chType, err) } - return precision, loc, nil + return timestampSpec{precision: precision, loc: loc}, nil } // parseTimestamp converts one ingested value into a time.Time. Accepted forms are diff --git a/internal/discovery/timestamp_test.go b/internal/discovery/timestamp_test.go index 2074bed4..65aa5dc0 100644 --- a/internal/discovery/timestamp_test.go +++ b/internal/discovery/timestamp_test.go @@ -2,6 +2,8 @@ package discovery import ( "encoding/json" + "io" + "log/slog" "testing" "time" @@ -78,13 +80,26 @@ func TestCanonicalizeTimestamps(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { t.Parallel() + // The production path: specs resolved once at schema-build time. + schema := tsSchema(tt.colType) + resolveTimestampSpecs(schema, tt.serverTZ, discardLogger()) data := map[string]any{"ts": tt.value} - require.NoError(t, CanonicalizeTimestamps(tsSchema(tt.colType), data, tt.serverTZ)) + require.NoError(t, CanonicalizeTimestamps(schema, data)) assert.Equal(t, tt.want, data["ts"]) }) } } +// TestCanonicalizeTimestamps_NoPrecomputedSpec: a schema that never went through +// spec resolution (hand-built Column literals, outside a registry) still +// canonicalizes — resolved per record, with UTC as the server default. +func TestCanonicalizeTimestamps_NoPrecomputedSpec(t *testing.T) { + t.Parallel() + data := map[string]any{"ts": "2026-06-21 04:00:00"} + require.NoError(t, CanonicalizeTimestamps(tsSchema("DateTime"), data)) + assert.Equal(t, "2026-06-21T04:00:00Z", data["ts"]) +} + // TestCanonicalizeTimestamps_AbsentColumn: a column not in the payload (DEFAULT- // filled by ClickHouse) is left absent, never invented. func TestCanonicalizeTimestamps_AbsentColumn(t *testing.T) { @@ -94,7 +109,7 @@ func TestCanonicalizeTimestamps_AbsentColumn(t *testing.T) { &TableSchema{Name: "t", Columns: []Column{ {Name: "ts", Type: "DateTime", HasDefault: true}, {Name: "page", Type: "String"}, - }}, data, nil)) + }}, data)) assert.Equal(t, map[string]any{"page": "/home"}, data) } @@ -117,17 +132,57 @@ func TestCanonicalizeTimestamps_Errors(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { t.Parallel() - err := CanonicalizeTimestamps(tsSchema(tt.colType), map[string]any{"ts": tt.value}, nil) + err := CanonicalizeTimestamps(tsSchema(tt.colType), map[string]any{"ts": tt.value}) require.Error(t, err) assert.Contains(t, err.Error(), tt.wantIn) }) } } -// TestServerTimezone_DefaultsToUTC: a registry that has never refreshed (the -// map-built test form) reports UTC rather than nil. -func TestServerTimezone_DefaultsToUTC(t *testing.T) { +// TestResolveTimestampSpecs: schema-build-time spec resolution — timestamp +// columns get a spec (the type's own zone, else the server default), other +// columns don't, and an unresolvable zone degrades to a nil spec (per-record +// rejection at ingest) rather than failing the schema build. +func TestResolveTimestampSpecs(t *testing.T) { t.Parallel() - reg := NewSchemaRegistryFromMap(nil) - assert.Equal(t, time.UTC, reg.ServerTimezone()) + nyc, err := time.LoadLocation("America/New_York") + require.NoError(t, err) + + schema := &TableSchema{Name: "t", Columns: []Column{ + {Name: "plain", Type: "DateTime"}, + {Name: "zoned", Type: "DateTime64(3, 'America/New_York')"}, + {Name: "page", Type: "String"}, + {Name: "broken", Type: "DateTime('Not/AZone')"}, + }} + resolveTimestampSpecs(schema, nyc, discardLogger()) + + require.NotNil(t, schema.Columns[0].tsSpec) + assert.Equal(t, nyc, schema.Columns[0].tsSpec.loc, "zone-less column takes the server zone") + require.NotNil(t, schema.Columns[1].tsSpec) + assert.Equal(t, nyc, schema.Columns[1].tsSpec.loc) + assert.Equal(t, 3, schema.Columns[1].tsSpec.precision) + assert.Nil(t, schema.Columns[2].tsSpec, "non-timestamp column gets no spec") + assert.Nil(t, schema.Columns[3].tsSpec, "unresolvable zone degrades to nil, not a failed build") + + // The degraded column still rejects loudly — per record, naming the column. + err = CanonicalizeTimestamps(schema, map[string]any{"broken": "2026-06-21 04:00:00"}) + require.Error(t, err) + assert.Contains(t, err.Error(), `unknown time zone "Not/AZone"`) +} + +// TestNewSchemaRegistryFromMap_PrecomputesSpecs: the map-built test registry +// resolves specs like Refresh does (UTC server default), so handler tests +// exercise the same precomputed path as production. +func TestNewSchemaRegistryFromMap_PrecomputesSpecs(t *testing.T) { + t.Parallel() + reg := NewSchemaRegistryFromMap([]*TableSchema{tsSchema("DateTime")}) + col := reg.Get("t").Columns[0] + require.NotNil(t, col.tsSpec) + assert.Equal(t, time.UTC, col.tsSpec.loc) +} + +// discardLogger mirrors the registries' test logger: spec-resolution warnings are +// asserted via behavior (nil specs), not log output. +func discardLogger() *slog.Logger { + return slog.New(slog.NewTextHandler(io.Discard, nil)) } From 7607472744241d571baee79e5d1691a9cf53f79b Mon Sep 17 00:00:00 2001 From: taitelee Date: Thu, 9 Jul 2026 13:17:08 -0400 Subject: [PATCH 03/15] docs: correct the non-UTC DateTime rendering claim and index the canonical wire-form invariant Co-Authored-By: Claude Fable 5 --- AGENTS.md | 3 ++- CHANGELOG.md | 2 +- docs/src/content/docs/api.md | 2 +- internal/discovery/timestamp.go | 5 +++-- 4 files changed, 7 insertions(+), 5 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 1e3ec198..7e2006b7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -34,7 +34,7 @@ Fourteen internal packages under `internal/` (plus `internal/testutil/` for shar - **`chsql/`** — dependency-free ClickHouse SQL helpers shared by `query`/`policy` (avoids an import cycle): `QuoteIdent` (backtick-quote every identifier) + `BindUnsafe` (reject names with a literal `?`) - **`config/`** — YAML + env var config loading (cleanenv) - **`dedupe/`** — `Deduplicator` interface → `Embedded` (Pebble) — optional, controlled by `dedupe.enabled` -- **`discovery/`** — `SchemaRegistry` that introspects ClickHouse `system.columns` + `Validate()` for ingest payloads +- **`discovery/`** — `SchemaRegistry` that introspects ClickHouse `system.columns` + `Validate()` for ingest payloads + `CanonicalizeTimestamps()` rewriting `DateTime`/`DateTime64` values to the canonical RFC 3339 UTC wire form pre-publish (Key Design Decision #19) - **`ingest/`** — Ingest worker pipeline (`worker.go`: JetStream input → per-table batch INSERT with DLQ output). The pipeline is **insert-only**. The wire format `EventMessage` (`types.go`) carries `{table_name, scope, received_timestamp, data}` and nothing else; the worker accepts whatever table name the envelope carries (table existence was already checked by the HTTP ingest handler, which `404`s an unknown table before publish; the worker doesn't re-validate), then bulk-INSERTs. In the embedded-NATS deployment (the default), the server runs with `DontListen: true` (`internal/mq/embedded.go`), so the only Publishers reachable on the `ingest.>` subjects are in-process Go code — today, only the HTTP `/v1/ingest?table={table}` handler. Non-insert mutations (`DELETE`/`UPDATE`/`TRUNCATE`/…) must go through `POST /v1/admin/query` under the admin role (the same `RequireAdmin` gate as the rest of `/v1/admin/*`), so non-admin callers never reach the proxy. A request with no token (or an invalid one) resolves to the `default_role`, which in a production config is not the admin role (setting them equal is a loudly-warned dev-only setting), so it can't reach this endpoint. Plus `Sweeper` (Active Sweeper for NATS message lifecycle) + `EventMessage`/`BufferConsumerName` types (`types.go`) - **`mq/`** — `Publisher`/`Subscriber` interfaces → `EmbeddedNATS` + `RemoteNATS` - **`observability/`** — OpenTelemetry pipeline: `InitProvider` wires trace/metric/log providers via OTLP gRPC (each signal independently gated). A top-level `Prometheus` config block drives an optional `/metrics` scrape endpoint that runs independently of OTLP push — standalone (Alloy/Mimir scrape, no collector), alongside OTLP, or off. `NewLogger` produces a slog handler that fans out to stdout AND OTLP (stdout always 100%, OTLP sample-rate-aware). `TraceHandler` injects trace_id/span_id from active spans. `tracer.go` provides W3C trace context propagation over NATS headers. @@ -65,6 +65,7 @@ The invariant index — what must stay true. Full narrative and rationale live i 16. **Bearer-token-only CORS posture (security)** — Bearer JWT on every request, no cookies/sessions; `corsMiddleware` deliberately **never** emits `Access-Control-Allow-Credentials` (not needed, and `*` + credentials is a spec violation browsers reject). `cors_allowed_origins` controls who can *read* responses, not cookie scope; CSRF protection is structural. Don't reintroduce cookie auth or `Allow-Credentials` without a design discussion — answers GitHub #29/#30. Code: `internal/api/router.go`. 17. **Non-fatal boot** — schema-discovery failure on boot is non-fatal: `cmd/wavehouse` records an `api.BootState`, binds `:8080`, serves 503 on `/livez`/`/readyz` with the diagnostic, and retries via `SchemaRegistry.RetryRefresh` (backoff 2s → 60s). Bounds supervisor restart loops. 18. **Health endpoints** — liveness `/livez`, readiness `/readyz` (k8s convention); `/healthz` is a permanent alias of `/livez`; `/health` + `/ready` are deprecated (removal v0.2.0, CHANGELOG #144). `/v1/health` is the SDK's content-free public ping (no ClickHouse check), a `/v1` route so it survives reverse-proxy probe-path filtering. Point k8s at `/livez`/`/readyz`, SDK/online-checks at `/v1/health`, never the deprecated aliases. +19. **Canonical timestamp wire form** — the HTTP ingest handler rewrites every `DateTime`/`DateTime64` value to RFC 3339 UTC (`discovery.CanonicalizeTimestamps`; per-column precision + resolved zone precomputed at schema refresh; `time/tzdata` embedded in `cmd/wavehouse`, an unresolvable zone fails the refresh loudly) after validation + policy checks and **before** the NATS publish, so the one payload every consumer shares — SSE subscribers (the #294 hot path), the ClickHouse insert, the DLQ — carries the same spelling `/v1/query` renders (its `transformRow` also forces UTC): live and query reads of a row can't drift (#372). Zone-less inputs are read in the column's declared zone, else the discovered server default — ClickHouse's own rule, so the spelling changes but never the instant. Don't publish an ingest payload that skipped canonicalization, and don't re-spell timestamps downstream. Preserve when touching `internal/discovery`, the ingest handler, or the SSE fan-out. Detail: architecture.md § `discovery/` + §Ingest Path. ## Code Conventions diff --git a/CHANGELOG.md b/CHANGELOG.md index a7a70716..05178a42 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,7 +50,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed -- **SSE row `DateTime` columns now arrive in canonical RFC 3339 UTC, matching `/v1/query`, instead of the producer's spelling** (`internal/discovery/timestamp.go` (new), `internal/discovery/{discovery.go,validation.go}`, `internal/api/ingest.go`, `internal/ingest/worker.go`, `cmd/wavehouse/main.go`, `docs/src/content/docs/{api.md,architecture.md}`, plus tests in `internal/discovery/{timestamp_test.go (new),discovery_test.go}`, `internal/api/{ingest_test.go,boot_chain_test.go}`, `tests/e2e/sdk/streaming.test.ts`): closes #372. The stream fans out the pre-insert payload verbatim, so a row `DateTime` reached SSE subscribers in whatever spelling the producer sent — typically the zone-less ClickHouse-native form — while `/v1/query` rendered the stored value as RFC 3339 `Z`; JavaScript's `Date.parse` reads a zone-less string as **local** time, so an SDK `liveQuery` stitching backfill + live landed the two paths hours apart (sign-flipping with the viewer's UTC offset — broke ordering and the "N min ago" label in the Stats live feed). Ingest now **canonicalizes** every `DateTime`/`DateTime64` column value to RFC 3339 UTC (fraction truncated to the column's precision, `time.RFC3339Nano`-trimmed — byte-matching `/v1/query`'s `time.Time` marshaling for UTC columns) after validation and before the NATS publish, so the one payload every consumer shares — SSE subscribers, the ClickHouse insert, the DLQ — carries a single unambiguous spelling. Inputs stay liberal (RFC 3339 with any offset, `YYYY-MM-DD[ T]HH:MM:SS[.fff]`, `YYYY-MM-DD`, Unix seconds as a number or digit-string); a zone-less string is interpreted in the column's declared time zone, else the server's (discovered once per schema refresh via `SELECT timezone()`) — the same rule ClickHouse itself applies, so canonicalization never changes which instant is stored (verified against a live ClickHouse). Each timestamp column's spec (precision + resolved zone) is precomputed at schema discovery and cached on the column, so the per-record ingest path parses no type strings, loads no zones, and takes no locks; the binary embeds Go's `time/tzdata` (~450 KB) so zone resolution — now load-bearing for schema refresh — doesn't depend on the deploy image shipping a tzdata package, and a zone that still can't resolve fails the refresh loudly rather than silently reinterpreting zone-less values as UTC (#402 review). An unparseable timestamp is now a loud per-record `400` naming the column, instead of passing validation and dying at the async insert into the DLQ. The worker's `INSERT … FORMAT JSONEachRow` sets `date_time_input_format=best_effort`, because ClickHouse's default `basic` parser rejects the canonical form's zone suffix (verified live; `best_effort` is a superset, so pre-upgrade messages still in NATS — zone-less strings, Unix numbers — parse exactly as before). Boundaries: `Date`/`Date32` columns pass through untouched (day precision, no zone ambiguity on this path); events published before the upgrade replay in their original spelling until the sweeper retires them; a column with an explicit non-UTC zone renders as `Z` on the stream but in the column's own offset on `/v1/query` — two zone-explicit spellings of the same instant (byte-identical on UTC columns, the norm). +- **SSE row `DateTime` columns now arrive in canonical RFC 3339 UTC, matching `/v1/query`, instead of the producer's spelling** (`internal/discovery/timestamp.go` (new), `internal/discovery/{discovery.go,validation.go}`, `internal/api/ingest.go`, `internal/ingest/worker.go`, `cmd/wavehouse/main.go`, `docs/src/content/docs/{api.md,architecture.md}`, plus tests in `internal/discovery/{timestamp_test.go (new),discovery_test.go}`, `internal/api/{ingest_test.go,boot_chain_test.go}`, `tests/e2e/sdk/streaming.test.ts`): closes #372. The stream fans out the pre-insert payload verbatim, so a row `DateTime` reached SSE subscribers in whatever spelling the producer sent — typically the zone-less ClickHouse-native form — while `/v1/query` rendered the stored value as RFC 3339 `Z`; JavaScript's `Date.parse` reads a zone-less string as **local** time, so an SDK `liveQuery` stitching backfill + live landed the two paths hours apart (sign-flipping with the viewer's UTC offset — broke ordering and the "N min ago" label in the Stats live feed). Ingest now **canonicalizes** every `DateTime`/`DateTime64` column value to RFC 3339 UTC (fraction truncated to the column's precision, `time.RFC3339Nano`-trimmed — byte-matching `/v1/query`'s `time.Time` marshaling) after validation and before the NATS publish, so the one payload every consumer shares — SSE subscribers, the ClickHouse insert, the DLQ — carries a single unambiguous spelling. Inputs stay liberal (RFC 3339 with any offset, `YYYY-MM-DD[ T]HH:MM:SS[.fff]`, `YYYY-MM-DD`, Unix seconds as a number or digit-string); a zone-less string is interpreted in the column's declared time zone, else the server's (discovered once per schema refresh via `SELECT timezone()`) — the same rule ClickHouse itself applies, so canonicalization never changes which instant is stored (verified against a live ClickHouse). Each timestamp column's spec (precision + resolved zone) is precomputed at schema discovery and cached on the column, so the per-record ingest path parses no type strings, loads no zones, and takes no locks; the binary embeds Go's `time/tzdata` (~450 KB) so zone resolution — now load-bearing for schema refresh — doesn't depend on the deploy image shipping a tzdata package, and a zone that still can't resolve fails the refresh loudly rather than silently reinterpreting zone-less values as UTC (#402 review). An unparseable timestamp is now a loud per-record `400` naming the column, instead of passing validation and dying at the async insert into the DLQ. The worker's `INSERT … FORMAT JSONEachRow` sets `date_time_input_format=best_effort`, because ClickHouse's default `basic` parser rejects the canonical form's zone suffix (verified live; `best_effort` is a superset, so pre-upgrade messages still in NATS — zone-less strings, Unix numbers — parse exactly as before). Boundaries: `Date`/`Date32` columns pass through untouched (day precision, no zone ambiguity on this path); events published before the upgrade replay in their original spelling until the sweeper retires them; a column with an explicit non-UTC zone renders as `Z` on both paths (`/v1/query` likewise normalizes `DateTime` values to UTC), so the declared zone affects how zone-less inputs are read but never the output spelling — byte-identical regardless of zone. - **Live demo hero feed renders in `event_ts` order instead of SSE arrival order** (`docs/src/components/LiveDemo.astro`): the landing-page live activity feed prepended each streamed row to the top in the order the SSE stream delivered it, but a producer's webhook burst (a single merge-queue cycle fires ~20 events) arrives in no guaranteed order and the stream relays it in ingest order — so a late or out-of-order delivery landed above newer rows (e.g. a `pushed 12m ago` sitting on top of `reviewed a pull request 9m ago`). `addRow` now keeps the feed sorted by `event_ts` descending — it slots each row in before the first strictly-older sibling rather than blind-prepending — so the live tail matches the already-sorted `gh_activity_recent` backfill. The zone-less-SSE-timestamp normalization the sort relies on (`normTs`) was already in place; equal-second rows keep arrival order (`gh_events.event_ts` is only second-granular for CI/checks, so there's no finer tiebreak), and dedup + the `MAX_ROWS` trim are unchanged. Surfaced in dogfooding on `wavehouse.dev`; the client-side analog of the ingest-order reality the SSE stream can't reorder. diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index 617bbe54..987115fd 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -542,7 +542,7 @@ data: {"table_name":"clicks","received_timestamp":"2026-03-24T12:00:01.456Z","da Each SSE connection is bound to a single `?table=`; to consume multiple tables, open one connection per table. -Row `DateTime`/`DateTime64` column values inside `data` arrive in the canonical RFC 3339 UTC form (ingest rewrites them before publishing — see [timestamp canonicalization](#post-v1ingesttabletable--ingest-data)), so a live event and a `/v1/query` read of the same row agree on the instant in zone-explicit form — a zone-less spelling no longer parses as local time in a browser ([#372](https://github.com/Wave-RF/WaveHouse/issues/372)). For a column stored in UTC (the norm) the two renderings are byte-identical; a column declared with a non-UTC zone streams as `Z` while `/v1/query` renders it in the column's own offset — the same instant either way. Events ingested before this behavior shipped replay in whatever spelling they were published with. +Row `DateTime`/`DateTime64` column values inside `data` arrive in the canonical RFC 3339 UTC form (ingest rewrites them before publishing — see [timestamp canonicalization](#post-v1ingesttabletable--ingest-data)), so a live event and a `/v1/query` read of the same row agree on the instant in zone-explicit form — a zone-less spelling no longer parses as local time in a browser ([#372](https://github.com/Wave-RF/WaveHouse/issues/372)). The two renderings are byte-identical regardless of the column's declared time zone — a column declared with a non-UTC zone also streams as `Z`, because `/v1/query` likewise normalizes every `DateTime` value to UTC before rendering. Events ingested before this behavior shipped replay in whatever spelling they were published with. **Note:** When access control policies are active, streamed events are filtered per the caller's role — denied columns are removed and tables without select permission are skipped. diff --git a/internal/discovery/timestamp.go b/internal/discovery/timestamp.go index cbd57d5e..70a10f0b 100644 --- a/internal/discovery/timestamp.go +++ b/internal/discovery/timestamp.go @@ -195,8 +195,9 @@ func unixToTime(f float64) time.Time { // canonicalTimestamp renders t in the canonical wire form: UTC RFC 3339, fraction // truncated to the column's precision. time.RFC3339Nano trims trailing fractional -// zeros — exactly how /v1/query renders (encoding/json marshals time.Time with the -// same layout), so the two read paths stay byte-identical for UTC columns. +// zeros — exactly how /v1/query renders (its transformRow formats every time.Time +// via UTC().Format(time.RFC3339Nano)), so the two read paths stay byte-identical +// whatever zone the column declares. func canonicalTimestamp(t time.Time, precision int) string { unit := time.Second for range precision { From 6aa54ec04e9efd469d15b96c1077e27832fc6391 Mon Sep 17 00:00:00 2001 From: taitelee Date: Thu, 9 Jul 2026 13:52:12 -0400 Subject: [PATCH 04/15] docs: distinguish server vs column zone-resolution failure modes --- AGENTS.md | 2 +- CHANGELOG.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 7e2006b7..25089c11 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -65,7 +65,7 @@ The invariant index — what must stay true. Full narrative and rationale live i 16. **Bearer-token-only CORS posture (security)** — Bearer JWT on every request, no cookies/sessions; `corsMiddleware` deliberately **never** emits `Access-Control-Allow-Credentials` (not needed, and `*` + credentials is a spec violation browsers reject). `cors_allowed_origins` controls who can *read* responses, not cookie scope; CSRF protection is structural. Don't reintroduce cookie auth or `Allow-Credentials` without a design discussion — answers GitHub #29/#30. Code: `internal/api/router.go`. 17. **Non-fatal boot** — schema-discovery failure on boot is non-fatal: `cmd/wavehouse` records an `api.BootState`, binds `:8080`, serves 503 on `/livez`/`/readyz` with the diagnostic, and retries via `SchemaRegistry.RetryRefresh` (backoff 2s → 60s). Bounds supervisor restart loops. 18. **Health endpoints** — liveness `/livez`, readiness `/readyz` (k8s convention); `/healthz` is a permanent alias of `/livez`; `/health` + `/ready` are deprecated (removal v0.2.0, CHANGELOG #144). `/v1/health` is the SDK's content-free public ping (no ClickHouse check), a `/v1` route so it survives reverse-proxy probe-path filtering. Point k8s at `/livez`/`/readyz`, SDK/online-checks at `/v1/health`, never the deprecated aliases. -19. **Canonical timestamp wire form** — the HTTP ingest handler rewrites every `DateTime`/`DateTime64` value to RFC 3339 UTC (`discovery.CanonicalizeTimestamps`; per-column precision + resolved zone precomputed at schema refresh; `time/tzdata` embedded in `cmd/wavehouse`, an unresolvable zone fails the refresh loudly) after validation + policy checks and **before** the NATS publish, so the one payload every consumer shares — SSE subscribers (the #294 hot path), the ClickHouse insert, the DLQ — carries the same spelling `/v1/query` renders (its `transformRow` also forces UTC): live and query reads of a row can't drift (#372). Zone-less inputs are read in the column's declared zone, else the discovered server default — ClickHouse's own rule, so the spelling changes but never the instant. Don't publish an ingest payload that skipped canonicalization, and don't re-spell timestamps downstream. Preserve when touching `internal/discovery`, the ingest handler, or the SSE fan-out. Detail: architecture.md § `discovery/` + §Ingest Path. +19. **Canonical timestamp wire form** — the HTTP ingest handler rewrites every `DateTime`/`DateTime64` value to RFC 3339 UTC (`discovery.CanonicalizeTimestamps`; per-column precision + resolved zone precomputed at schema refresh; `time/tzdata` embedded in `cmd/wavehouse`; an unresolvable server-default zone fails the refresh — deliberately, a silent UTC fallback would move instants — while an unresolvable column-declared zone warns and degrades to per-record `400`s on just that column) after validation + policy checks and **before** the NATS publish, so the one payload every consumer shares — SSE subscribers (the #294 hot path), the ClickHouse insert, the DLQ — carries the same spelling `/v1/query` renders (its `transformRow` also forces UTC): live and query reads of a row can't drift (#372). Zone-less inputs are read in the column's declared zone, else the discovered server default — ClickHouse's own rule, so the spelling changes but never the instant. Don't publish an ingest payload that skipped canonicalization, and don't re-spell timestamps downstream. Preserve when touching `internal/discovery`, the ingest handler, or the SSE fan-out. Detail: architecture.md § `discovery/` + §Ingest Path. ## Code Conventions diff --git a/CHANGELOG.md b/CHANGELOG.md index 05178a42..bbada723 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,7 +50,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed -- **SSE row `DateTime` columns now arrive in canonical RFC 3339 UTC, matching `/v1/query`, instead of the producer's spelling** (`internal/discovery/timestamp.go` (new), `internal/discovery/{discovery.go,validation.go}`, `internal/api/ingest.go`, `internal/ingest/worker.go`, `cmd/wavehouse/main.go`, `docs/src/content/docs/{api.md,architecture.md}`, plus tests in `internal/discovery/{timestamp_test.go (new),discovery_test.go}`, `internal/api/{ingest_test.go,boot_chain_test.go}`, `tests/e2e/sdk/streaming.test.ts`): closes #372. The stream fans out the pre-insert payload verbatim, so a row `DateTime` reached SSE subscribers in whatever spelling the producer sent — typically the zone-less ClickHouse-native form — while `/v1/query` rendered the stored value as RFC 3339 `Z`; JavaScript's `Date.parse` reads a zone-less string as **local** time, so an SDK `liveQuery` stitching backfill + live landed the two paths hours apart (sign-flipping with the viewer's UTC offset — broke ordering and the "N min ago" label in the Stats live feed). Ingest now **canonicalizes** every `DateTime`/`DateTime64` column value to RFC 3339 UTC (fraction truncated to the column's precision, `time.RFC3339Nano`-trimmed — byte-matching `/v1/query`'s `time.Time` marshaling) after validation and before the NATS publish, so the one payload every consumer shares — SSE subscribers, the ClickHouse insert, the DLQ — carries a single unambiguous spelling. Inputs stay liberal (RFC 3339 with any offset, `YYYY-MM-DD[ T]HH:MM:SS[.fff]`, `YYYY-MM-DD`, Unix seconds as a number or digit-string); a zone-less string is interpreted in the column's declared time zone, else the server's (discovered once per schema refresh via `SELECT timezone()`) — the same rule ClickHouse itself applies, so canonicalization never changes which instant is stored (verified against a live ClickHouse). Each timestamp column's spec (precision + resolved zone) is precomputed at schema discovery and cached on the column, so the per-record ingest path parses no type strings, loads no zones, and takes no locks; the binary embeds Go's `time/tzdata` (~450 KB) so zone resolution — now load-bearing for schema refresh — doesn't depend on the deploy image shipping a tzdata package, and a zone that still can't resolve fails the refresh loudly rather than silently reinterpreting zone-less values as UTC (#402 review). An unparseable timestamp is now a loud per-record `400` naming the column, instead of passing validation and dying at the async insert into the DLQ. The worker's `INSERT … FORMAT JSONEachRow` sets `date_time_input_format=best_effort`, because ClickHouse's default `basic` parser rejects the canonical form's zone suffix (verified live; `best_effort` is a superset, so pre-upgrade messages still in NATS — zone-less strings, Unix numbers — parse exactly as before). Boundaries: `Date`/`Date32` columns pass through untouched (day precision, no zone ambiguity on this path); events published before the upgrade replay in their original spelling until the sweeper retires them; a column with an explicit non-UTC zone renders as `Z` on both paths (`/v1/query` likewise normalizes `DateTime` values to UTC), so the declared zone affects how zone-less inputs are read but never the output spelling — byte-identical regardless of zone. +- **SSE row `DateTime` columns now arrive in canonical RFC 3339 UTC, matching `/v1/query`, instead of the producer's spelling** (`internal/discovery/timestamp.go` (new), `internal/discovery/{discovery.go,validation.go}`, `internal/api/ingest.go`, `internal/ingest/worker.go`, `cmd/wavehouse/main.go`, `docs/src/content/docs/{api.md,architecture.md}`, plus tests in `internal/discovery/{timestamp_test.go (new),discovery_test.go}`, `internal/api/{ingest_test.go,boot_chain_test.go}`, `tests/e2e/sdk/streaming.test.ts`): closes #372. The stream fans out the pre-insert payload verbatim, so a row `DateTime` reached SSE subscribers in whatever spelling the producer sent — typically the zone-less ClickHouse-native form — while `/v1/query` rendered the stored value as RFC 3339 `Z`; JavaScript's `Date.parse` reads a zone-less string as **local** time, so an SDK `liveQuery` stitching backfill + live landed the two paths hours apart (sign-flipping with the viewer's UTC offset — broke ordering and the "N min ago" label in the Stats live feed). Ingest now **canonicalizes** every `DateTime`/`DateTime64` column value to RFC 3339 UTC (fraction truncated to the column's precision, `time.RFC3339Nano`-trimmed — byte-matching `/v1/query`'s `time.Time` marshaling) after validation and before the NATS publish, so the one payload every consumer shares — SSE subscribers, the ClickHouse insert, the DLQ — carries a single unambiguous spelling. Inputs stay liberal (RFC 3339 with any offset, `YYYY-MM-DD[ T]HH:MM:SS[.fff]`, `YYYY-MM-DD`, Unix seconds as a number or digit-string); a zone-less string is interpreted in the column's declared time zone, else the server's (discovered once per schema refresh via `SELECT timezone()`) — the same rule ClickHouse itself applies, so canonicalization never changes which instant is stored (verified against a live ClickHouse). Each timestamp column's spec (precision + resolved zone) is precomputed at schema discovery and cached on the column, so the per-record ingest path parses no type strings, loads no zones, and takes no locks; the binary embeds Go's `time/tzdata` (~450 KB) so zone resolution — now load-bearing for schema refresh — doesn't depend on the deploy image shipping a tzdata package, and a server zone that still can't resolve fails the refresh loudly rather than silently reinterpreting zone-less values as UTC, while an unresolvable column-declared zone degrades to per-record `400`s on that column, not a failed refresh (#402 review). An unparseable timestamp is now a loud per-record `400` naming the column, instead of passing validation and dying at the async insert into the DLQ. The worker's `INSERT … FORMAT JSONEachRow` sets `date_time_input_format=best_effort`, because ClickHouse's default `basic` parser rejects the canonical form's zone suffix (verified live; `best_effort` is a superset, so pre-upgrade messages still in NATS — zone-less strings, Unix numbers — parse exactly as before). Boundaries: `Date`/`Date32` columns pass through untouched (day precision, no zone ambiguity on this path); events published before the upgrade replay in their original spelling until the sweeper retires them; a column with an explicit non-UTC zone renders as `Z` on both paths (`/v1/query` likewise normalizes `DateTime` values to UTC), so the declared zone affects how zone-less inputs are read but never the output spelling — byte-identical regardless of zone. - **Live demo hero feed renders in `event_ts` order instead of SSE arrival order** (`docs/src/components/LiveDemo.astro`): the landing-page live activity feed prepended each streamed row to the top in the order the SSE stream delivered it, but a producer's webhook burst (a single merge-queue cycle fires ~20 events) arrives in no guaranteed order and the stream relays it in ingest order — so a late or out-of-order delivery landed above newer rows (e.g. a `pushed 12m ago` sitting on top of `reviewed a pull request 9m ago`). `addRow` now keeps the feed sorted by `event_ts` descending — it slots each row in before the first strictly-older sibling rather than blind-prepending — so the live tail matches the already-sorted `gh_activity_recent` backfill. The zone-less-SSE-timestamp normalization the sort relies on (`normTs`) was already in place; equal-second rows keep arrival order (`gh_events.event_ts` is only second-granular for CI/checks, so there's no finer tiebreak), and dedup + the `MAX_ROWS` trim are unchanged. Surfaced in dogfooding on `wavehouse.dev`; the client-side analog of the ingest-order reality the SSE stream can't reorder. From 8ff9aa8235242430a25cb15e39265313f3054db9 Mon Sep 17 00:00:00 2001 From: taitelee Date: Thu, 9 Jul 2026 14:26:50 -0400 Subject: [PATCH 05/15] fix(ingest): make timestamp canonicalization fail-open and drop the tzdata embed --- AGENTS.md | 2 +- CHANGELOG.md | 2 +- cmd/wavehouse/main.go | 7 -- docs/src/content/docs/api.md | 7 +- docs/src/content/docs/architecture.md | 6 +- internal/api/ingest.go | 18 ++-- internal/api/ingest_test.go | 44 +++++----- internal/discovery/discovery.go | 43 +++++----- internal/discovery/discovery_test.go | 25 +++++- internal/discovery/timestamp.go | 115 ++++++++++---------------- internal/discovery/timestamp_test.go | 62 +++++++------- internal/ingest/worker.go | 7 +- 12 files changed, 162 insertions(+), 176 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 25089c11..52a4676e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -65,7 +65,7 @@ The invariant index — what must stay true. Full narrative and rationale live i 16. **Bearer-token-only CORS posture (security)** — Bearer JWT on every request, no cookies/sessions; `corsMiddleware` deliberately **never** emits `Access-Control-Allow-Credentials` (not needed, and `*` + credentials is a spec violation browsers reject). `cors_allowed_origins` controls who can *read* responses, not cookie scope; CSRF protection is structural. Don't reintroduce cookie auth or `Allow-Credentials` without a design discussion — answers GitHub #29/#30. Code: `internal/api/router.go`. 17. **Non-fatal boot** — schema-discovery failure on boot is non-fatal: `cmd/wavehouse` records an `api.BootState`, binds `:8080`, serves 503 on `/livez`/`/readyz` with the diagnostic, and retries via `SchemaRegistry.RetryRefresh` (backoff 2s → 60s). Bounds supervisor restart loops. 18. **Health endpoints** — liveness `/livez`, readiness `/readyz` (k8s convention); `/healthz` is a permanent alias of `/livez`; `/health` + `/ready` are deprecated (removal v0.2.0, CHANGELOG #144). `/v1/health` is the SDK's content-free public ping (no ClickHouse check), a `/v1` route so it survives reverse-proxy probe-path filtering. Point k8s at `/livez`/`/readyz`, SDK/online-checks at `/v1/health`, never the deprecated aliases. -19. **Canonical timestamp wire form** — the HTTP ingest handler rewrites every `DateTime`/`DateTime64` value to RFC 3339 UTC (`discovery.CanonicalizeTimestamps`; per-column precision + resolved zone precomputed at schema refresh; `time/tzdata` embedded in `cmd/wavehouse`; an unresolvable server-default zone fails the refresh — deliberately, a silent UTC fallback would move instants — while an unresolvable column-declared zone warns and degrades to per-record `400`s on just that column) after validation + policy checks and **before** the NATS publish, so the one payload every consumer shares — SSE subscribers (the #294 hot path), the ClickHouse insert, the DLQ — carries the same spelling `/v1/query` renders (its `transformRow` also forces UTC): live and query reads of a row can't drift (#372). Zone-less inputs are read in the column's declared zone, else the discovered server default — ClickHouse's own rule, so the spelling changes but never the instant. Don't publish an ingest payload that skipped canonicalization, and don't re-spell timestamps downstream. Preserve when touching `internal/discovery`, the ingest handler, or the SSE fan-out. Detail: architecture.md § `discovery/` + §Ingest Path. +19. **Canonical timestamp wire form (fail-open at ingest)** — the HTTP ingest handler rewrites every `DateTime`/`DateTime64` value it can parse to RFC 3339 UTC (`discovery.CanonicalizeTimestamps`; per-column precision + zone precomputed at schema refresh) after validation + policy checks and **before** the NATS publish, so the one payload every consumer shares — SSE subscribers, the ClickHouse insert, the DLQ — carries the same spelling `/v1/query` renders: live and query reads can't drift (#372). Zone-less inputs are read in the column's declared zone, else the discovered server default — ClickHouse's own rule, so the spelling changes but never the instant. Deliberately **fail-open**: an unparseable value or unresolvable zone (no tzdata embedded — never a failed refresh, never a silent UTC reinterpretation, which would move instants) publishes verbatim; ingest must not reject a record over its timestamp spelling — fail-closed enforcement belongs to the stream row-filter (#381). Don't re-spell timestamps downstream. Preserve when touching `internal/discovery`, the ingest handler, or the SSE fan-out. Detail: architecture.md § `discovery/` + §Ingest Path. ## Code Conventions diff --git a/CHANGELOG.md b/CHANGELOG.md index bbada723..c83b69bc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,7 +50,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed -- **SSE row `DateTime` columns now arrive in canonical RFC 3339 UTC, matching `/v1/query`, instead of the producer's spelling** (`internal/discovery/timestamp.go` (new), `internal/discovery/{discovery.go,validation.go}`, `internal/api/ingest.go`, `internal/ingest/worker.go`, `cmd/wavehouse/main.go`, `docs/src/content/docs/{api.md,architecture.md}`, plus tests in `internal/discovery/{timestamp_test.go (new),discovery_test.go}`, `internal/api/{ingest_test.go,boot_chain_test.go}`, `tests/e2e/sdk/streaming.test.ts`): closes #372. The stream fans out the pre-insert payload verbatim, so a row `DateTime` reached SSE subscribers in whatever spelling the producer sent — typically the zone-less ClickHouse-native form — while `/v1/query` rendered the stored value as RFC 3339 `Z`; JavaScript's `Date.parse` reads a zone-less string as **local** time, so an SDK `liveQuery` stitching backfill + live landed the two paths hours apart (sign-flipping with the viewer's UTC offset — broke ordering and the "N min ago" label in the Stats live feed). Ingest now **canonicalizes** every `DateTime`/`DateTime64` column value to RFC 3339 UTC (fraction truncated to the column's precision, `time.RFC3339Nano`-trimmed — byte-matching `/v1/query`'s `time.Time` marshaling) after validation and before the NATS publish, so the one payload every consumer shares — SSE subscribers, the ClickHouse insert, the DLQ — carries a single unambiguous spelling. Inputs stay liberal (RFC 3339 with any offset, `YYYY-MM-DD[ T]HH:MM:SS[.fff]`, `YYYY-MM-DD`, Unix seconds as a number or digit-string); a zone-less string is interpreted in the column's declared time zone, else the server's (discovered once per schema refresh via `SELECT timezone()`) — the same rule ClickHouse itself applies, so canonicalization never changes which instant is stored (verified against a live ClickHouse). Each timestamp column's spec (precision + resolved zone) is precomputed at schema discovery and cached on the column, so the per-record ingest path parses no type strings, loads no zones, and takes no locks; the binary embeds Go's `time/tzdata` (~450 KB) so zone resolution — now load-bearing for schema refresh — doesn't depend on the deploy image shipping a tzdata package, and a server zone that still can't resolve fails the refresh loudly rather than silently reinterpreting zone-less values as UTC, while an unresolvable column-declared zone degrades to per-record `400`s on that column, not a failed refresh (#402 review). An unparseable timestamp is now a loud per-record `400` naming the column, instead of passing validation and dying at the async insert into the DLQ. The worker's `INSERT … FORMAT JSONEachRow` sets `date_time_input_format=best_effort`, because ClickHouse's default `basic` parser rejects the canonical form's zone suffix (verified live; `best_effort` is a superset, so pre-upgrade messages still in NATS — zone-less strings, Unix numbers — parse exactly as before). Boundaries: `Date`/`Date32` columns pass through untouched (day precision, no zone ambiguity on this path); events published before the upgrade replay in their original spelling until the sweeper retires them; a column with an explicit non-UTC zone renders as `Z` on both paths (`/v1/query` likewise normalizes `DateTime` values to UTC), so the declared zone affects how zone-less inputs are read but never the output spelling — byte-identical regardless of zone. +- **SSE row `DateTime` columns now arrive in canonical RFC 3339 UTC, matching `/v1/query`, instead of the producer's spelling** (`internal/discovery/timestamp.go` (new), `internal/discovery/{discovery.go,validation.go}`, `internal/api/ingest.go`, `internal/ingest/worker.go`, `docs/src/content/docs/{api.md,architecture.md}`, plus tests in `internal/discovery/{timestamp_test.go (new),discovery_test.go}`, `internal/api/{ingest_test.go,boot_chain_test.go}`, `tests/e2e/sdk/streaming.test.ts`): closes #372. The stream fans out the pre-insert payload verbatim, so a row `DateTime` reached SSE subscribers in whatever spelling the producer sent — typically the zone-less ClickHouse-native form — while `/v1/query` rendered the stored value as RFC 3339 `Z`; JavaScript's `Date.parse` reads a zone-less string as **local** time, so an SDK `liveQuery` stitching backfill + live landed the two paths hours apart (sign-flipping with the viewer's UTC offset — broke ordering and the "N min ago" label in the Stats live feed). Ingest now **canonicalizes** every `DateTime`/`DateTime64` column value to RFC 3339 UTC (fraction truncated to the column's precision, `time.RFC3339Nano`-trimmed — byte-matching `/v1/query`'s `time.Time` marshaling) after validation and before the NATS publish, so the one payload every consumer shares — SSE subscribers, the ClickHouse insert, the DLQ — carries a single unambiguous spelling. Inputs stay liberal (RFC 3339 with any offset, `YYYY-MM-DD[ T]HH:MM:SS[.fff]`, `YYYY-MM-DD`, Unix seconds as a number or digit-string); a zone-less string is interpreted in the column's declared time zone, else the server's (discovered once per schema refresh via `SELECT timezone()`) — the same rule ClickHouse itself applies, so canonicalization never changes which instant is stored (verified against a live ClickHouse). Each timestamp column's spec (precision + resolved zone) is precomputed at schema discovery and cached on the column, so the per-record ingest path parses no type strings, loads no zones, and takes no locks. Canonicalization is **fail-open** (#402 review): a value outside the accepted forms publishes verbatim — ClickHouse's more liberal `best_effort` parser stays the arbiter of insertability, and a value it too rejects surfaces via the DLQ as before — and an unresolvable zone (no tzdata embedded; the distroless image ships none) warns and passes through, never a failed refresh or a silent UTC reinterpretation that would move instants (`Etc/UTC` is special-cased to UTC); ingest never rejects a record over its timestamp spelling — fail-closed enforcement of the canonical form is the stream row-filter's (#381). The worker's `INSERT … FORMAT JSONEachRow` sets `date_time_input_format=best_effort`, because ClickHouse's default `basic` parser rejects the canonical form's zone suffix (verified live; `best_effort` is a superset, so pre-upgrade messages still in NATS — zone-less strings, Unix numbers — parse exactly as before). Boundaries: `Date`/`Date32` columns pass through untouched (day precision, no zone ambiguity on this path); events published before the upgrade replay in their original spelling until the sweeper retires them; a column with an explicit non-UTC zone renders as `Z` on both paths (`/v1/query` likewise normalizes `DateTime` values to UTC), so the declared zone affects how zone-less inputs are read but never the output spelling — byte-identical regardless of zone. - **Live demo hero feed renders in `event_ts` order instead of SSE arrival order** (`docs/src/components/LiveDemo.astro`): the landing-page live activity feed prepended each streamed row to the top in the order the SSE stream delivered it, but a producer's webhook burst (a single merge-queue cycle fires ~20 events) arrives in no guaranteed order and the stream relays it in ingest order — so a late or out-of-order delivery landed above newer rows (e.g. a `pushed 12m ago` sitting on top of `reviewed a pull request 9m ago`). `addRow` now keeps the feed sorted by `event_ts` descending — it slots each row in before the first strictly-older sibling rather than blind-prepending — so the live tail matches the already-sorted `gh_activity_recent` backfill. The zone-less-SSE-timestamp normalization the sort relies on (`normTs`) was already in place; equal-second rows keep arrival order (`gh_events.event_ts` is only second-granular for CI/checks, so there's no finer tiebreak), and dedup + the `MAX_ROWS` trim are unchanged. Surfaced in dogfooding on `wavehouse.dev`; the client-side analog of the ingest-order reality the SSE stream can't reorder. diff --git a/cmd/wavehouse/main.go b/cmd/wavehouse/main.go index d641b183..c75eb853 100644 --- a/cmd/wavehouse/main.go +++ b/cmd/wavehouse/main.go @@ -14,13 +14,6 @@ import ( "syscall" "time" - // Embed the IANA time zone database (~450 KB). Schema discovery resolves the - // ClickHouse server zone and timestamp-column zones with time.LoadLocation - // (#372), which must not depend on the deploy image shipping a system tzdata - // package — a schema refresh that cannot resolve the server zone fails, and - // stays failed, until tzdata appears. - _ "time/tzdata" - "github.com/ClickHouse/clickhouse-go/v2" "github.com/Wave-RF/WaveHouse/internal/api" "github.com/Wave-RF/WaveHouse/internal/auth" diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index 987115fd..37cb55ad 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -217,7 +217,7 @@ The body is a **flat JSON object** whose keys must match column names in the tar - Type compatibility: `String`/`DateTime`/`UUID`/`Enum`/`IPv*` accept JSON strings; `Int*`/`Float*`/`Decimal` accept JSON numbers; `Bool` accepts JSON booleans or numbers; `Array` accepts JSON arrays; `Map`/`Tuple` accept JSON objects. - `Nullable()` and `LowCardinality()` wrappers are handled transparently. -**Timestamp canonicalization:** a `DateTime`/`DateTime64` column value may be sent as RFC 3339 (any offset), `YYYY-MM-DD[ T]HH:MM:SS[.fff]` or `YYYY-MM-DD` (zone-less — interpreted in the column's time zone, else the ClickHouse server's, exactly as ClickHouse itself would), or Unix seconds (a JSON number, or the same digits as a string). Whatever the input spelling, WaveHouse rewrites the value to **one canonical wire form — RFC 3339 UTC** (`2026-06-21T04:00:00Z`, fraction truncated to the column's precision) — before publishing, so the stored instant never changes but every downstream consumer (the ClickHouse insert, [SSE subscribers](#get-v1stream--server-sent-events-stream), the DLQ) sees the same spelling `/v1/query` renders. An unparseable timestamp is rejected with a `400` naming the column (previously it surfaced only at the async insert, via the DLQ). `Date`/`Date32` columns are passed through untouched. +**Timestamp canonicalization:** a `DateTime`/`DateTime64` column value sent as RFC 3339 (any offset), `YYYY-MM-DD[ T]HH:MM:SS[.fff]` or `YYYY-MM-DD` (zone-less — interpreted in the column's time zone, else the ClickHouse server's, exactly as ClickHouse itself would), or Unix seconds (a JSON number, or the same digits as a string) is rewritten to **one canonical wire form — RFC 3339 UTC** (`2026-06-21T04:00:00Z`, fraction truncated to the column's precision) before publishing, so the stored instant never changes but every consumer (the ClickHouse insert, [SSE subscribers](#get-v1stream--server-sent-events-stream), the DLQ) sees the same spelling `/v1/query` renders. **Fail-open**: a value in none of those forms is published verbatim — ClickHouse's more liberal parser decides insertability, and a value it too rejects surfaces via the DLQ, as before. `Date`/`Date32` columns pass through untouched. **Response (accepted):** @@ -237,7 +237,6 @@ The body is a **flat JSON object** whose keys must match column names in the tar | ------ | ---- | ----- | | 400 | `{"error":"invalid json"}` | Malformed request body | | 400 | `{"error":"unknown column ... for table ..."}` (also: `missing required column ...`, `type mismatch for column ...`, `null value for non-nullable column ...`) | Schema validation failure (unknown fields, type mismatches, missing required columns, null in a non-nullable column). The body is the validator's message verbatim — there is no `validation failed:` prefix. | -| 400 | `{"error":"column \"ts\": unrecognized timestamp ..."}` | A `DateTime`/`DateTime64` column value that parses as none of the accepted timestamp forms (see timestamp canonicalization above). In a batch this is a per-record failure. | | 400 | `{"error":"missing dedupe id field \"event_id\""}` | Only when dedupe is enabled with `dedupe.require_id: true` and the row lacks the configured `id_field`. With `require_id: false` (the default) the row is instead published un-deduped. Either way — reject or publish — the row is logged at `WARN` and counted by `wavehouse_ingest_dedupe_missing_id_total`. In a batch this is a per-record failure, not a whole-request error. | | 401 | `{"error":"invalid token"}` / `{"error":"token expired"}` | A present-but-invalid/expired token was supplied and denied (the gate surfaces the token reason rather than silently falling back to `default_role`) | | 403 | `{"error":"forbidden"}` (empty-role variant: `forbidden: request has no role and no public default_role is configured`) | The resolved role lacks `insert` on the table | @@ -542,7 +541,7 @@ data: {"table_name":"clicks","received_timestamp":"2026-03-24T12:00:01.456Z","da Each SSE connection is bound to a single `?table=`; to consume multiple tables, open one connection per table. -Row `DateTime`/`DateTime64` column values inside `data` arrive in the canonical RFC 3339 UTC form (ingest rewrites them before publishing — see [timestamp canonicalization](#post-v1ingesttabletable--ingest-data)), so a live event and a `/v1/query` read of the same row agree on the instant in zone-explicit form — a zone-less spelling no longer parses as local time in a browser ([#372](https://github.com/Wave-RF/WaveHouse/issues/372)). The two renderings are byte-identical regardless of the column's declared time zone — a column declared with a non-UTC zone also streams as `Z`, because `/v1/query` likewise normalizes every `DateTime` value to UTC before rendering. Events ingested before this behavior shipped replay in whatever spelling they were published with. +Row `DateTime`/`DateTime64` column values inside `data` arrive in the canonical RFC 3339 UTC form (ingest rewrites them before publishing — see [timestamp canonicalization](#post-v1ingesttabletable--ingest-data)), so a live event and a `/v1/query` read of the same row agree on the instant in zone-explicit form — a zone-less spelling no longer parses as local time in a browser ([#372](https://github.com/Wave-RF/WaveHouse/issues/372)). The two renderings are byte-identical regardless of the column's declared time zone — a column declared with a non-UTC zone also streams as `Z`, because `/v1/query` likewise normalizes every `DateTime` value to UTC before rendering. Canonicalization is fail-open at ingest, so a value outside the accepted input forms streams in whatever spelling the producer sent; events ingested before this behavior shipped likewise replay in their original spelling. **Note:** When access control policies are active, streamed events are filtered per the caller's role — denied columns are removed and tables without select permission are skipped. @@ -773,7 +772,7 @@ The message format used on NATS JetStream between ingest and the batch consumer: | ----- | ---- | ----------- | | `table_name` | string | Target ClickHouse table (from URL). | | `received_timestamp` | string | RFC 3339 nano timestamp when WaveHouse received the event. | -| `data` | object | The flat JSON body, with `DateTime`/`DateTime64` column values rewritten to canonical RFC 3339 UTC (see [timestamp canonicalization](#post-v1ingesttabletable--ingest-data)); other values as originally sent. | +| `data` | object | The flat JSON body, with parseable `DateTime`/`DateTime64` column values rewritten to canonical RFC 3339 UTC (see [timestamp canonicalization](#post-v1ingesttabletable--ingest-data)); other values as originally sent. | ### Client-Facing Format (SSE) diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index cb4dd179..df30bf64 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -114,7 +114,7 @@ The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https:/ ### `discovery/` — Schema Discovery & Validation - **discovery.go** — `SchemaRegistry` queries `system.columns` to discover ClickHouse table schemas. Each refresh also discovers the server's default time zone (`SELECT timezone()`) and bakes every `DateTime`/`DateTime64` column's canonicalization spec (precision + resolved zone) into the cached schema, so the per-record ingest path parses no type strings and loads no zones ([#372](https://github.com/Wave-RF/WaveHouse/issues/372)). Supports periodic auto-refresh, on-demand refresh, and `RetryRefresh` (boot-time exponential backoff loop used by `cmd/wavehouse` so a transiently unreachable ClickHouse doesn't crash-loop the binary). Thread-safe via `sync.RWMutex`. -- **timestamp.go** — `CanonicalizeTimestamps(schema, data)` rewrites every `DateTime`/`DateTime64` value to the canonical RFC 3339 UTC wire form before the event is published (#372): zone-less values are interpreted in the column's declared zone, else the discovered server default — the same rule ClickHouse applies, so the spelling changes but never the instant. An unparseable value is a per-record `400` naming the column. +- **timestamp.go** — `CanonicalizeTimestamps(schema, data)` rewrites every `DateTime`/`DateTime64` value it can parse to the canonical RFC 3339 UTC wire form before the event is published (#372): zone-less values are interpreted in the column's declared zone, else the discovered server default — ClickHouse's own rule, so the spelling changes but never the instant. Fail-open: an unparseable value or unresolvable zone passes through verbatim for ClickHouse's own parser to judge; ingest never rejects a record over its timestamp spelling. - **validation.go** — `Validate(schema, data)` checks incoming JSON against the discovered schema: unknown fields, type compatibility, missing required columns, null handling. - **discovery_test.go** — Unit tests for validation logic. @@ -169,8 +169,8 @@ Client POST /v1/ingest?table={table} → Policy column rules + check clauses (disallowed columns rejected; claim-derived values enforced or injected) → Canonicalize DateTime/DateTime64 values to RFC 3339 UTC (rewrites the - payload so every consumer shares one spelling; an unparseable value is a - 400 for that record) + payload so every consumer shares one spelling; fail-open — an unparseable + value passes through verbatim for ClickHouse's own parser to judge) → Optional deduplication check (configurable ID field; a row missing that field is published un-deduped + logged/counted, or rejected under require_id) → Publish to NATS JetStream (ingest.{table}) diff --git a/internal/api/ingest.go b/internal/api/ingest.go index bb1d7e39..a2f3aa93 100644 --- a/internal/api/ingest.go +++ b/internal/api/ingest.go @@ -401,17 +401,13 @@ func (h *IngestHandler) processRecord( } } - // Canonicalize timestamp columns to the RFC 3339 UTC wire form, so the one - // payload published below reaches every consumer — SSE subscribers, the - // ClickHouse insert, the DLQ — in a single unambiguous spelling (#372). After - // the permission checks so check-clause comparisons keep their pre-#372 - // semantics (and claim-injected values are covered); before dedup so a - // rejected record never marks a dedup key. Each column's precision and zone - // were resolved at schema discovery, so this is pure computation per record. - if err := discovery.CanonicalizeTimestamps(schema, data); err != nil { - h.logger.WarnContext(ctx, "timestamp canonicalization failed", "error", err, "table", table) - return false, &recordReject{Status: http.StatusBadRequest, Message: err.Error()}, nil - } + // Canonicalize timestamp columns to RFC 3339 UTC so the one payload published + // below reaches every consumer — SSE, the ClickHouse insert, the DLQ — in the + // spelling /v1/query renders (#372). Fail-open: unparseable values pass through + // for ClickHouse's own parser to judge; fail-closed enforcement is the stream + // row-filter's job (#381). After the permission checks so check-clause + // comparisons keep their pre-#372 semantics. + discovery.CanonicalizeTimestamps(schema, data) // Optional deduplication. if h.Dedup != nil && h.IDField != "" { diff --git a/internal/api/ingest_test.go b/internal/api/ingest_test.go index 6201f70c..61413a31 100644 --- a/internal/api/ingest_test.go +++ b/internal/api/ingest_test.go @@ -1431,9 +1431,9 @@ func TestIngest_TimestampsCanonicalized(t *testing.T) { assert.Equal(t, "e", data["name"], "non-timestamp columns untouched") } -// TestIngest_TimestampGarbage_400: an unparseable timestamp is rejected loudly at -// ingest, naming the column — not accepted and left to die at the async insert. -func TestIngest_TimestampGarbage_400(t *testing.T) { +// TestIngest_TimestampGarbage_PassesThrough: fail-open — an unparseable value +// publishes verbatim; ClickHouse's own parser decides insertability (#372/#381). +func TestIngest_TimestampGarbage_PassesThrough(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} h := NewIngestHandler(tsRegistry(), pub, testutil.NopLogger()) @@ -1442,15 +1442,13 @@ func TestIngest_TimestampGarbage_400(t *testing.T) { w := httptest.NewRecorder() h.Handle(w, req) - assert.Equal(t, http.StatusBadRequest, w.Code) - testutil.AssertJSONErrorResponse(t, w) - assert.Contains(t, w.Body.String(), `column \"ts\"`) - assert.Nil(t, pub.LastMessage(), "nothing published for a rejected record") + require.Equal(t, http.StatusOK, w.Code) + assert.Equal(t, "banana", publishedData(t, pub)["ts"], "unparseable value published verbatim") } -// TestIngest_Batch_BadTimestampIsolatedPerRecord: one bad timestamp fails its own -// record; the rest of the batch still publishes (the #195 isolation contract). -func TestIngest_Batch_BadTimestampIsolatedPerRecord(t *testing.T) { +// TestIngest_Batch_MixedTimestampSpellings: parseable spellings canonicalize, +// the unparseable one passes through — no record fails on its timestamp. +func TestIngest_Batch_MixedTimestampSpellings(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} h := NewIngestHandler(tsRegistry(), pub, testutil.NopLogger()) @@ -1468,16 +1466,24 @@ func TestIngest_Batch_BadTimestampIsolatedPerRecord(t *testing.T) { Total int `json:"total"` Succeeded int `json:"succeeded"` Failed int `json:"failed"` - Results []struct { - Index int `json:"index"` - Error string `json:"error"` - } `json:"results"` } require.NoError(t, json.Unmarshal(w.Body.Bytes(), &result)) assert.Equal(t, 3, result.Total) - assert.Equal(t, 2, result.Succeeded) - assert.Equal(t, 1, result.Failed) - require.Len(t, result.Results, 3) - assert.Contains(t, result.Results[1].Error, `column "ts"`) - assert.Len(t, pub.Messages, 2, "the two good records published") + assert.Equal(t, 3, result.Succeeded) + assert.Equal(t, 0, result.Failed) + require.Len(t, pub.Messages, 3, "every record published") + + var spellings []string + for _, msg := range pub.Messages { + var evt struct { + Data map[string]any `json:"data"` + } + require.NoError(t, json.Unmarshal(msg.Data, &evt)) + spellings = append(spellings, evt.Data["ts"].(string)) + } + assert.Equal(t, []string{ + "2026-06-21T04:00:00Z", // already canonical + "banana", // unparseable — passed through verbatim + "2026-06-21T04:00:00Z", // Unix seconds — canonicalized + }, spellings) } diff --git a/internal/discovery/discovery.go b/internal/discovery/discovery.go index a08ae9f6..8ec9a56c 100644 --- a/internal/discovery/discovery.go +++ b/internal/discovery/discovery.go @@ -21,10 +21,9 @@ type Column struct { HasDefault bool `json:"has_default"` // tsSpec is a DateTime/DateTime64 column's canonicalization spec, resolved - // once when the schema is built (Refresh / NewSchemaRegistryFromMap) so the - // per-record ingest path parses no type strings and loads no zones. nil for - // non-timestamp columns, hand-built Column literals, and zones Go cannot - // resolve — CanonicalizeTimestamps falls back to per-record resolution. + // once at schema build (Refresh / NewSchemaRegistryFromMap). nil for + // non-timestamp columns, hand-built Column literals, and unresolvable zones — + // CanonicalizeTimestamps passes those through untouched (fail-open, #372). tsSpec *timestampSpec } @@ -68,30 +67,31 @@ func NewSchemaRegistry(conn driver.Conn, database string, refreshInterval time.D } } -// Refresh rebuilds the in-memory schema cache: it discovers the ClickHouse -// server's default time zone (`SELECT timezone()`), queries system.columns, and -// precomputes each timestamp column's canonicalization spec before swapping the -// new cache in. +// Refresh rebuilds the in-memory schema cache: it discovers the server's default +// time zone, queries system.columns, and precomputes timestamp column specs. func (sr *SchemaRegistry) Refresh(ctx context.Context) error { tracer := otel.GetTracerProvider().Tracer("wavehouse-discovery") ctx, span := tracer.Start(ctx, "SchemaRegistry.Refresh") defer span.End() - // The server's default time zone is how ClickHouse interprets zone-less - // timestamp strings on columns without an explicit one; ingest canonicalization - // must apply the same rule so it never changes which instant is stored (#372). + // ClickHouse interprets zone-less timestamp strings in the server's default + // zone; canonicalization applies the same rule so the instant never changes (#372). var tzName string if err := sr.conn.QueryRow(ctx, "SELECT timezone()").Scan(&tzName); err != nil { return fmt.Errorf("query server timezone: %w", err) } - serverTZ, err := time.LoadLocation(tzName) - if err != nil { - // Deliberately fails the refresh rather than falling back to UTC: on a - // non-UTC server, a UTC fallback would silently change which instant a - // zone-less string stores. cmd/wavehouse embeds time/tzdata, so this - // resolves even on images without a system zone database; failing here - // means the server's zone is newer than the binary's own tzdata. - return fmt.Errorf("load server timezone %q: %w", tzName, err) + var serverTZ *time.Location + if tzName == "Etc/UTC" { + // Debian-family spelling of UTC; Go can't resolve it without a zone + // database (the distroless image ships none), but "UTC" it always can. + serverTZ = time.UTC + } else if loc, err := time.LoadLocation(tzName); err == nil { + serverTZ = loc + } else { + // Unresolvable — warn, not fatal, and no UTC fallback (that could move + // instants). A nil server zone means zone-less values pass through. + sr.logger.Warn("cannot resolve server timezone; zone-less timestamps will pass through un-canonicalized", + "timezone", tzName, "error", err) } rows, err := sr.conn.Query(ctx, @@ -233,13 +233,12 @@ func isNullable(chType string) bool { // NewSchemaRegistryFromMap creates a SchemaRegistry pre-loaded with the given // table schemas. Intended for testing — no ClickHouse connection is required. -// Timestamp specs are precomputed like Refresh does (no server to ask ⇒ UTC), so -// handler tests exercise the same precomputed path as production. +// Timestamp specs are precomputed like Refresh does (UTC as the server default). func NewSchemaRegistryFromMap(tables []*TableSchema) *SchemaRegistry { logger := slog.New(slog.NewTextHandler(io.Discard, nil)) m := make(map[string]*TableSchema, len(tables)) for _, t := range tables { - resolveTimestampSpecs(t, nil, logger) + resolveTimestampSpecs(t, time.UTC, logger) m[t.Name] = t } return &SchemaRegistry{ diff --git a/internal/discovery/discovery_test.go b/internal/discovery/discovery_test.go index 8e8eb9da..554f6f82 100644 --- a/internal/discovery/discovery_test.go +++ b/internal/discovery/discovery_test.go @@ -99,6 +99,7 @@ type fakeConn struct { driver.Conn errsThenSuccess []error calls atomic.Int32 + tz string // SELECT timezone() answer; "" ⇒ "UTC" } func (c *fakeConn) Query(_ context.Context, _ string, _ ...any) (driver.Rows, error) { @@ -112,15 +113,21 @@ func (c *fakeConn) Query(_ context.Context, _ string, _ ...any) (driver.Rows, er // QueryRow answers the SELECT timezone() probe Refresh issues before the // system.columns query (#372); errsThenSuccess sequencing stays keyed on Query. func (c *fakeConn) QueryRow(context.Context, string, ...any) driver.Row { - return tzRow{} + if c.tz != "" { + return tzRow{tz: c.tz} + } + return tzRow{tz: "UTC"} } -type tzRow struct{ driver.Row } +type tzRow struct { + driver.Row + tz string +} -func (tzRow) Scan(dest ...any) error { +func (r tzRow) Scan(dest ...any) error { if len(dest) == 1 { if s, ok := dest[0].(*string); ok { - *s = "UTC" + *s = r.tz return nil } } @@ -145,6 +152,16 @@ func newFakeRegistry(t *testing.T, errs []error) (*SchemaRegistry, *fakeConn) { return NewSchemaRegistry(conn, "test", time.Hour, logger), conn } +// TestRefresh_UnresolvableServerTimezone_NotFatal: an unresolvable server zone +// degrades to pass-through canonicalization (#372), never a failed refresh. +func TestRefresh_UnresolvableServerTimezone_NotFatal(t *testing.T) { + t.Parallel() + conn := &fakeConn{tz: "Not/AZone"} + logger := slog.New(slog.NewJSONHandler(io.Discard, nil)) + sr := NewSchemaRegistry(conn, "test", time.Hour, logger) + require.NoError(t, sr.Refresh(context.Background())) +} + // TestRetryRefresh_SucceedsOnFirstAttempt is the happy path: Refresh returns // nil immediately and no backoff is observed. func TestRetryRefresh_SucceedsOnFirstAttempt(t *testing.T) { diff --git a/internal/discovery/timestamp.go b/internal/discovery/timestamp.go index 70a10f0b..0e3cbc99 100644 --- a/internal/discovery/timestamp.go +++ b/internal/discovery/timestamp.go @@ -11,83 +11,63 @@ import ( ) // Ingest canonicalizes every DateTime/DateTime64 column value to one wire form — -// RFC 3339 UTC (`2026-06-21T04:00:00Z`, fraction per column precision) — before the -// event is published (#372). The event payload is fanned out verbatim to SSE -// subscribers and inserted into ClickHouse as-is, so without one canonical form the -// same instant reaches stream consumers in whatever spelling the producer chose -// (zone-less ClickHouse-native, Unix seconds, …) while /v1/query renders the stored -// value as RFC 3339 — the two read paths land on different clocks. ClickHouse itself -// discards the input spelling at insert, so canonicalizing costs nothing on the -// query path and makes the streamed form match it. +// RFC 3339 UTC (`2026-06-21T04:00:00Z`, fraction per column precision) — before +// the event is published (#372), so SSE subscribers see the same spelling +// /v1/query renders instead of whatever the producer sent. ClickHouse discards +// the input spelling at insert, so only the streamed form changes. +// +// Canonicalization is fail-open: anything it cannot parse or resolve passes +// through verbatim — ClickHouse's more liberal best_effort parser stays the +// arbiter of what is insertable (failures land in the DLQ, as before #372). +// Fail-closed enforcement of the canonical form is the stream row-filter's (#381). // IsTimestampType reports whether chType is a ClickHouse DateTime or DateTime64 -// (unwrapping Nullable/LowCardinality modifiers). Date/Date32 are deliberately not -// included: they are day-precision values with no zone or spelling ambiguity on the -// paths #372 covers, so ingest passes them through untouched. +// (unwrapping Nullable/LowCardinality). Date/Date32 are excluded — day-precision, +// no zone or spelling ambiguity. func IsTimestampType(chType string) bool { return strings.HasPrefix(unwrapType(chType), "DateTime") } -// CanonicalizeTimestamps rewrites every DateTime/DateTime64 column value in data to -// the canonical RFC 3339 UTC form, in place. Zone-less values are interpreted the -// way ClickHouse itself would parse them — in the column's declared time zone, else -// the server default — so canonicalization never changes which instant is stored, -// only how it is spelled. Each column's precision and zone come from its spec, -// precomputed when the schema was built (Refresh / NewSchemaRegistryFromMap); a -// schema built outside a registry falls back to per-record resolution with UTC as -// the server default. The fraction is truncated to the column's precision so the -// streamed value equals what ClickHouse stores and /v1/query returns. Absent and -// null values are left untouched (defaults and Nullable columns). An unparseable -// value returns an error naming the column, which ingest maps to a per-record 400 — -// the same class of failure previously surfaced only at the async insert, via the -// DLQ. -func CanonicalizeTimestamps(schema *TableSchema, data map[string]any) error { +// CanonicalizeTimestamps rewrites every DateTime/DateTime64 column value in data +// to the canonical RFC 3339 UTC form, in place, best-effort. Zone-less values are +// interpreted as ClickHouse would — in the column's zone, else the server default +// (precomputed on the column's spec at schema build) — so only the spelling ever +// changes, never the stored instant; the fraction is truncated to the column's +// precision to byte-match /v1/query. Everything else passes through verbatim +// (fail-open): absent/null values, unparseable values, columns without a spec, and +// zone-less values when no zone is known (assuming one could move the instant). +func CanonicalizeTimestamps(schema *TableSchema, data map[string]any) { for _, col := range schema.Columns { spec := col.tsSpec - if spec == nil && !IsTimestampType(col.Type) { + if spec == nil { continue } v, ok := data[col.Name] if !ok || v == nil { continue } - if spec == nil { - // No precomputed spec: a hand-built schema, or a zone the schema build - // could not resolve. Resolving here keeps that failure scoped to the - // records that carry the column (a per-record 400), never the schema. - s, err := resolveTimestampSpec(col.Type, nil) - if err != nil { - return fmt.Errorf("column %q: %w", col.Name, err) - } - spec = &s - } t, err := parseTimestamp(v, spec.loc) if err != nil { - return fmt.Errorf("column %q: %w", col.Name, err) + continue } data[col.Name] = canonicalTimestamp(t, spec.precision) } - return nil } // timestampSpec is a timestamp column's precomputed canonicalization inputs: the -// DateTime64 sub-second precision (0 for DateTime) and the zone in which zone-less -// values are interpreted (the column's declared zone, else the ClickHouse server -// default at resolve time). +// DateTime64 sub-second precision (0 for DateTime) and the zone for interpreting +// zone-less values (declared zone, else server default; nil when neither is +// known — then only zone-explicit values canonicalize). type timestampSpec struct { precision int loc *time.Location } // resolveTimestampSpecs precomputes the timestampSpec of every DateTime/DateTime64 -// column in ts, in place — run once per schema build so the per-record ingest path -// never parses type strings, loads zones, or allocates. A column whose zone Go -// cannot resolve keeps a nil spec and is logged, not fatal: ClickHouse validated -// the name at CREATE TABLE, so a miss means tzdata skew between the ClickHouse -// server and this binary (cmd/wavehouse embeds time/tzdata to keep that window -// minimal). Ingest then resolves that one column per record and rejects its values -// with the same error — per-record 400s on one column, not a failed refresh for -// the whole schema. +// column in ts, in place — once per schema build, so the per-record ingest path +// parses no type strings and loads no zones. A column whose zone this binary +// cannot resolve (the distroless image ships no tzdata) keeps a nil spec — +// warned, not fatal: its values pass through un-canonicalized. func resolveTimestampSpecs(ts *TableSchema, serverTZ *time.Location, logger *slog.Logger) { for i := range ts.Columns { col := &ts.Columns[i] @@ -96,7 +76,7 @@ func resolveTimestampSpecs(ts *TableSchema, serverTZ *time.Location, logger *slo } spec, err := resolveTimestampSpec(col.Type, serverTZ) if err != nil { - logger.Warn("cannot resolve timestamp column spec; its ingest values will be rejected", + logger.Warn("cannot resolve timestamp column spec; its ingest values will pass through un-canonicalized", "table", ts.Name, "column", col.Name, "type", col.Type, "error", err) continue } @@ -106,12 +86,9 @@ func resolveTimestampSpecs(ts *TableSchema, serverTZ *time.Location, logger *slo // resolveTimestampSpec extracts a DateTime/DateTime64 type's sub-second precision // and time zone: `DateTime` / `DateTime('TZ')` / `DateTime64(P)` / -// `DateTime64(P, 'TZ')`. A type without an explicit zone uses serverTZ (nil ⇒ -// UTC), matching how ClickHouse interprets zone-less strings for that column. +// `DateTime64(P, 'TZ')`. A type without an explicit zone takes serverTZ (possibly +// nil = unknown) — ClickHouse's own rule for zone-less strings. func resolveTimestampSpec(chType string, serverTZ *time.Location) (timestampSpec, error) { - if serverTZ == nil { - serverTZ = time.UTC - } t := unwrapType(chType) var args []string @@ -146,11 +123,10 @@ func resolveTimestampSpec(chType string, serverTZ *time.Location) (timestampSpec return timestampSpec{precision: precision, loc: loc}, nil } -// parseTimestamp converts one ingested value into a time.Time. Accepted forms are -// the ones ClickHouse itself accepts on this path: RFC 3339 (zone-explicit), -// `YYYY-MM-DD[ T]HH:MM:SS[.fraction]` and `YYYY-MM-DD` (zone-less, interpreted in -// loc), and Unix seconds as a JSON number or a numeric string. Anything else is an -// error. +// parseTimestamp converts one ingested value into a time.Time. Accepted forms: +// RFC 3339, `YYYY-MM-DD[ T]HH:MM:SS[.fraction]` and `YYYY-MM-DD` (zone-less, +// interpreted in loc; skipped when loc is nil), and Unix seconds as a number or +// numeric string. Anything else errors — the caller passes it through. func parseTimestamp(v any, loc *time.Location) (time.Time, error) { switch x := v.(type) { case string: @@ -159,14 +135,15 @@ func parseTimestamp(v any, loc *time.Location) (time.Time, error) { } // Zone-less forms; Go's Parse accepts an input fraction after the seconds // even when the layout carries none. - for _, layout := range []string{"2006-01-02 15:04:05", "2006-01-02T15:04:05", "2006-01-02"} { - if t, err := time.ParseInLocation(layout, x, loc); err == nil { - return t, nil + if loc != nil { + for _, layout := range []string{"2006-01-02 15:04:05", "2006-01-02T15:04:05", "2006-01-02"} { + if t, err := time.ParseInLocation(layout, x, loc); err == nil { + return t, nil + } } } - // A bare number is Unix seconds — the quoted twin of the JSON-number form. - // ClickHouse's own text parsing read digit-strings this way before #372, so - // the form keeps working (non-finite values are not an instant). + // A bare digit-string is Unix seconds, as ClickHouse itself reads it + // (non-finite values are not an instant). if f, err := strconv.ParseFloat(x, 64); err == nil && !math.IsNaN(f) && !math.IsInf(f, 0) { return unixToTime(f), nil } @@ -194,10 +171,8 @@ func unixToTime(f float64) time.Time { } // canonicalTimestamp renders t in the canonical wire form: UTC RFC 3339, fraction -// truncated to the column's precision. time.RFC3339Nano trims trailing fractional -// zeros — exactly how /v1/query renders (its transformRow formats every time.Time -// via UTC().Format(time.RFC3339Nano)), so the two read paths stay byte-identical -// whatever zone the column declares. +// truncated to the column's precision. RFC3339Nano trims trailing zeros exactly +// like /v1/query's transformRow, keeping the two read paths byte-identical. func canonicalTimestamp(t time.Time, precision int) string { unit := time.Second for range precision { diff --git a/internal/discovery/timestamp_test.go b/internal/discovery/timestamp_test.go index 65aa5dc0..dabb5920 100644 --- a/internal/discovery/timestamp_test.go +++ b/internal/discovery/timestamp_test.go @@ -61,7 +61,8 @@ func TestCanonicalizeTimestamps(t *testing.T) { {"naive space form, column zone", "DateTime('America/New_York')", nil, "2026-06-21 00:00:00", "2026-06-21T04:00:00Z"}, {"naive T form, column zone", "DateTime('America/New_York')", nil, "2026-06-21T00:00:00", "2026-06-21T04:00:00Z"}, {"naive form, server zone", "DateTime", nyc, "2026-06-21 00:00:00", "2026-06-21T04:00:00Z"}, - {"naive form, no zone known ⇒ UTC", "DateTime", nil, "2026-06-21 04:00:00", "2026-06-21T04:00:00Z"}, + {"naive form, unknown server zone ⇒ passes through", "DateTime", nil, "2026-06-21 04:00:00", "2026-06-21 04:00:00"}, + {"offset form, unknown server zone still canonicalizes", "DateTime", nil, "2026-06-21T06:00:00+02:00", "2026-06-21T04:00:00Z"}, {"date-only ⇒ midnight in zone", "DateTime('America/New_York')", nil, "2026-06-21", "2026-06-21T04:00:00Z"}, {"unix seconds number", "DateTime('UTC')", nil, float64(1782014400), "2026-06-21T04:00:00Z"}, {"unix seconds json.Number", "DateTime('UTC')", nil, json.Number("1782014400"), "2026-06-21T04:00:00Z"}, @@ -84,65 +85,66 @@ func TestCanonicalizeTimestamps(t *testing.T) { schema := tsSchema(tt.colType) resolveTimestampSpecs(schema, tt.serverTZ, discardLogger()) data := map[string]any{"ts": tt.value} - require.NoError(t, CanonicalizeTimestamps(schema, data)) + CanonicalizeTimestamps(schema, data) assert.Equal(t, tt.want, data["ts"]) }) } } -// TestCanonicalizeTimestamps_NoPrecomputedSpec: a schema that never went through -// spec resolution (hand-built Column literals, outside a registry) still -// canonicalizes — resolved per record, with UTC as the server default. +// TestCanonicalizeTimestamps_NoPrecomputedSpec: a schema that skipped spec +// resolution (hand-built literals) passes through untouched. func TestCanonicalizeTimestamps_NoPrecomputedSpec(t *testing.T) { t.Parallel() data := map[string]any{"ts": "2026-06-21 04:00:00"} - require.NoError(t, CanonicalizeTimestamps(tsSchema("DateTime"), data)) - assert.Equal(t, "2026-06-21T04:00:00Z", data["ts"]) + CanonicalizeTimestamps(tsSchema("DateTime"), data) + assert.Equal(t, "2026-06-21 04:00:00", data["ts"]) } // TestCanonicalizeTimestamps_AbsentColumn: a column not in the payload (DEFAULT- // filled by ClickHouse) is left absent, never invented. func TestCanonicalizeTimestamps_AbsentColumn(t *testing.T) { t.Parallel() + schema := &TableSchema{Name: "t", Columns: []Column{ + {Name: "ts", Type: "DateTime", HasDefault: true}, + {Name: "page", Type: "String"}, + }} + resolveTimestampSpecs(schema, time.UTC, discardLogger()) data := map[string]any{"page": "/home"} - require.NoError(t, CanonicalizeTimestamps( - &TableSchema{Name: "t", Columns: []Column{ - {Name: "ts", Type: "DateTime", HasDefault: true}, - {Name: "page", Type: "String"}, - }}, data)) + CanonicalizeTimestamps(schema, data) assert.Equal(t, map[string]any{"page": "/home"}, data) } -func TestCanonicalizeTimestamps_Errors(t *testing.T) { +// TestCanonicalizeTimestamps_Unparseable_PassThrough: fail-open — unparseable +// values and unresolvable column specs pass through verbatim. +func TestCanonicalizeTimestamps_Unparseable_PassThrough(t *testing.T) { t.Parallel() tests := []struct { name string colType string value any - wantIn string }{ - {"unrecognized string", "DateTime('UTC')", "banana", `column "ts"`}, - {"non-finite numeric string is not an instant", "DateTime('UTC')", "NaN", "unrecognized timestamp"}, - {"wrong value type", "DateTime('UTC')", true, "must be a string or Unix-seconds number"}, - {"unknown zone in type", "DateTime('Not/AZone')", "2026-06-21 04:00:00", `unknown time zone "Not/AZone"`}, - {"malformed DateTime64 precision", "DateTime64(x)", "2026-06-21 04:00:00", "bad precision"}, + {"unrecognized string", "DateTime('UTC')", "banana"}, + {"non-finite numeric string is not an instant", "DateTime('UTC')", "NaN"}, + {"wrong value type", "DateTime('UTC')", true}, + {"unknown zone in type", "DateTime('Not/AZone')", "2026-06-21 04:00:00"}, + {"malformed DateTime64 precision", "DateTime64(x)", "2026-06-21 04:00:00"}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { t.Parallel() - err := CanonicalizeTimestamps(tsSchema(tt.colType), map[string]any{"ts": tt.value}) - require.Error(t, err) - assert.Contains(t, err.Error(), tt.wantIn) + schema := tsSchema(tt.colType) + resolveTimestampSpecs(schema, time.UTC, discardLogger()) + data := map[string]any{"ts": tt.value} + CanonicalizeTimestamps(schema, data) + assert.Equal(t, tt.value, data["ts"]) }) } } -// TestResolveTimestampSpecs: schema-build-time spec resolution — timestamp -// columns get a spec (the type's own zone, else the server default), other -// columns don't, and an unresolvable zone degrades to a nil spec (per-record -// rejection at ingest) rather than failing the schema build. +// TestResolveTimestampSpecs: timestamp columns get a spec (own zone, else server +// default); others don't; an unresolvable zone degrades to nil, not a failure. func TestResolveTimestampSpecs(t *testing.T) { t.Parallel() nyc, err := time.LoadLocation("America/New_York") @@ -164,10 +166,10 @@ func TestResolveTimestampSpecs(t *testing.T) { assert.Nil(t, schema.Columns[2].tsSpec, "non-timestamp column gets no spec") assert.Nil(t, schema.Columns[3].tsSpec, "unresolvable zone degrades to nil, not a failed build") - // The degraded column still rejects loudly — per record, naming the column. - err = CanonicalizeTimestamps(schema, map[string]any{"broken": "2026-06-21 04:00:00"}) - require.Error(t, err) - assert.Contains(t, err.Error(), `unknown time zone "Not/AZone"`) + // The degraded column passes through untouched — fail-open, never a rejection. + data := map[string]any{"broken": "2026-06-21 04:00:00"} + CanonicalizeTimestamps(schema, data) + assert.Equal(t, "2026-06-21 04:00:00", data["broken"]) } // TestNewSchemaRegistryFromMap_PrecomputesSpecs: the map-built test registry diff --git a/internal/ingest/worker.go b/internal/ingest/worker.go index 3fec44ef..24a93f06 100644 --- a/internal/ingest/worker.go +++ b/internal/ingest/worker.go @@ -431,10 +431,9 @@ func (w *IngestWorker) insertToClickHouse(ctx context.Context, tableName string, q.Set("database", w.db) q.Set("param_target_table", tableName) q.Set("query", "INSERT INTO {target_table:Identifier} FORMAT JSONEachRow") - // Ingest canonicalizes DateTime values to RFC 3339 UTC (#372), whose zone - // suffix ClickHouse's default 'basic' parser rejects. best_effort is a - // superset: it accepts the canonical form, and pre-#372 messages still in the - // stream (zone-less strings, Unix numbers) parse exactly as before. + // Canonical RFC 3339 timestamps (#372) carry a zone suffix ClickHouse's + // default 'basic' parser rejects; best_effort is a superset, so non-canonical + // values (pre-#372 messages, fail-open pass-throughs) parse as before. q.Set("date_time_input_format", "best_effort") req, err := http.NewRequestWithContext(ctx, "POST", w.chURL+"?"+q.Encode(), &buf) From 844e7122a15826aa6e27af34c6a9ae36922943cd Mon Sep 17 00:00:00 2001 From: taitelee Date: Thu, 9 Jul 2026 15:39:33 -0400 Subject: [PATCH 06/15] fix(discovery): resolve Etc/UTC column zones without tzdata; document best_effort insert --- docs/src/content/docs/architecture.md | 4 ++++ docs/src/content/docs/ingest-pipeline.md | 7 +++++-- internal/discovery/discovery.go | 6 +----- internal/discovery/timestamp.go | 9 ++++++++- internal/discovery/timestamp_test.go | 4 ++++ 5 files changed, 22 insertions(+), 8 deletions(-) diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index df30bf64..f96d2763 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -181,6 +181,10 @@ Ingest worker pipeline (StartIngestWorker): ← JetStream pull consumer (buffer-consumer) on ingest.> → Parse the event envelope (a malformed envelope is the only poison pill: it's acked-and-dropped) → Batch events per table, bulk INSERT to ClickHouse + (INSERTs set date_time_input_format=best_effort: ClickHouse's default + 'basic' parser rejects the canonical RFC 3339 form's zone suffix; best_effort + is a superset, so pre-canonical and fail-open pass-through spellings parse + as before) → On success: DoubleAck messages → On failure: route to DLQ output (dlq.{table}), then Ack to prevent infinite retry diff --git a/docs/src/content/docs/ingest-pipeline.md b/docs/src/content/docs/ingest-pipeline.md index ac2809ac..367ba93c 100644 --- a/docs/src/content/docs/ingest-pipeline.md +++ b/docs/src/content/docs/ingest-pipeline.md @@ -27,8 +27,11 @@ validates and bulk-`INSERT`s. Non-insert mutations go through a different admin One process consumes a single durable JetStream consumer and fans events out to a goroutine per table. Each table batches independently and POSTs to ClickHouse -over the HTTP interface (`JSONEachRow`). Failed rows go to a dead-letter stream; -a separate sweeper reclaims stream storage. +over the HTTP interface (`JSONEachRow`, with `date_time_input_format=best_effort`: +ClickHouse's default `basic` parser rejects the canonical RFC 3339 timestamps' +zone suffix (#372); `best_effort` is a superset, so pre-canonical spellings still +parse). Failed rows go to a dead-letter stream; a separate sweeper reclaims +stream storage. ```mermaid flowchart LR diff --git a/internal/discovery/discovery.go b/internal/discovery/discovery.go index 8ec9a56c..fa7a0587 100644 --- a/internal/discovery/discovery.go +++ b/internal/discovery/discovery.go @@ -81,11 +81,7 @@ func (sr *SchemaRegistry) Refresh(ctx context.Context) error { return fmt.Errorf("query server timezone: %w", err) } var serverTZ *time.Location - if tzName == "Etc/UTC" { - // Debian-family spelling of UTC; Go can't resolve it without a zone - // database (the distroless image ships none), but "UTC" it always can. - serverTZ = time.UTC - } else if loc, err := time.LoadLocation(tzName); err == nil { + if loc, err := loadLocation(tzName); err == nil { serverTZ = loc } else { // Unresolvable — warn, not fatal, and no UTC fallback (that could move diff --git a/internal/discovery/timestamp.go b/internal/discovery/timestamp.go index 0e3cbc99..59378a93 100644 --- a/internal/discovery/timestamp.go +++ b/internal/discovery/timestamp.go @@ -116,13 +116,20 @@ func resolveTimestampSpec(chType string, serverTZ *time.Location) (timestampSpec return timestampSpec{precision: precision, loc: serverTZ}, nil } name := strings.Trim(args[0], "'") - loc, err := time.LoadLocation(name) + loc, err := loadLocation(name) if err != nil { return timestampSpec{}, fmt.Errorf("unknown time zone %q in type %q: %w", name, chType, err) } return timestampSpec{precision: precision, loc: loc}, nil } +func loadLocation(name string) (*time.Location, error) { + if name == "Etc/UTC" { + return time.UTC, nil + } + return time.LoadLocation(name) +} + // parseTimestamp converts one ingested value into a time.Time. Accepted forms: // RFC 3339, `YYYY-MM-DD[ T]HH:MM:SS[.fraction]` and `YYYY-MM-DD` (zone-less, // interpreted in loc; skipped when loc is nil), and Unix seconds as a number or diff --git a/internal/discovery/timestamp_test.go b/internal/discovery/timestamp_test.go index dabb5920..a4dfca98 100644 --- a/internal/discovery/timestamp_test.go +++ b/internal/discovery/timestamp_test.go @@ -60,6 +60,7 @@ func TestCanonicalizeTimestamps(t *testing.T) { {"offset converts to Z", "DateTime('UTC')", nil, "2026-06-21T06:30:00+02:30", "2026-06-21T04:00:00Z"}, {"naive space form, column zone", "DateTime('America/New_York')", nil, "2026-06-21 00:00:00", "2026-06-21T04:00:00Z"}, {"naive T form, column zone", "DateTime('America/New_York')", nil, "2026-06-21T00:00:00", "2026-06-21T04:00:00Z"}, + {"naive form, Etc/UTC column zone", "DateTime('Etc/UTC')", nil, "2026-06-21 04:00:00", "2026-06-21T04:00:00Z"}, {"naive form, server zone", "DateTime", nyc, "2026-06-21 00:00:00", "2026-06-21T04:00:00Z"}, {"naive form, unknown server zone ⇒ passes through", "DateTime", nil, "2026-06-21 04:00:00", "2026-06-21 04:00:00"}, {"offset form, unknown server zone still canonicalizes", "DateTime", nil, "2026-06-21T06:00:00+02:00", "2026-06-21T04:00:00Z"}, @@ -155,6 +156,7 @@ func TestResolveTimestampSpecs(t *testing.T) { {Name: "zoned", Type: "DateTime64(3, 'America/New_York')"}, {Name: "page", Type: "String"}, {Name: "broken", Type: "DateTime('Not/AZone')"}, + {Name: "etc_utc", Type: "DateTime('Etc/UTC')"}, }} resolveTimestampSpecs(schema, nyc, discardLogger()) @@ -165,6 +167,8 @@ func TestResolveTimestampSpecs(t *testing.T) { assert.Equal(t, 3, schema.Columns[1].tsSpec.precision) assert.Nil(t, schema.Columns[2].tsSpec, "non-timestamp column gets no spec") assert.Nil(t, schema.Columns[3].tsSpec, "unresolvable zone degrades to nil, not a failed build") + require.NotNil(t, schema.Columns[4].tsSpec) + assert.Same(t, time.UTC, schema.Columns[4].tsSpec.loc, "Etc/UTC maps to UTC without a tzdata lookup") // The degraded column passes through untouched — fail-open, never a rejection. data := map[string]any{"broken": "2026-06-21 04:00:00"} From 49837421d374c032323006aa4e2968995b8eed74 Mon Sep 17 00:00:00 2001 From: taitelee Date: Sat, 11 Jul 2026 15:57:11 -0400 Subject: [PATCH 07/15] test: move schema registry test factory to testutil via mock conn + real Refresh --- AGENTS.md | 4 +- CHANGELOG.md | 2 +- internal/api/errors_test.go | 5 +- internal/api/ingest_test.go | 114 +++++++++++++------------- internal/api/router_test.go | 12 +-- internal/api/schema_test.go | 6 +- internal/api/structured_query_test.go | 22 ++--- internal/discovery/discovery.go | 23 +----- internal/discovery/discovery_test.go | 91 ++++++++++++++------ internal/discovery/timestamp_test.go | 13 +-- internal/testutil/testutil.go | 88 +++++++++++++++++++- 11 files changed, 242 insertions(+), 138 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 52a4676e..e133e3ff 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -121,7 +121,7 @@ Tooling notes (the non-obvious bits `make help` won't tell you): - **Table-driven tests**: Use `tests := []struct{ name string; ... }` with `t.Run(tt.name, ...)` for test cases. - **Shared mocks in `internal/testutil/`**: Use `MockPublisher`, `MockCache`, `MockDeduplicator`, `MockSubscriber` instead of creating ad-hoc mocks. See `testutil/mocks.go`. - **JWT helpers**: Use `testutil.MakeJWT(t, claims)` and `testutil.MakeExpiredJWT(t, claims)` for auth tests. See `testutil/jwt.go`. -- **Schema helpers**: Use `testutil.NewTestSchemaRegistry(tables)` or `discovery.NewSchemaRegistryFromMap(tables)` for schema-aware tests. +- **Schema helpers**: Use `testutil.NewTestSchemaRegistry(t, tables)` for schema-aware tests — it builds the registry through the real discovery path (`Refresh` against a mock ClickHouse connection), so timestamp specs are precomputed like production. - **Policy helpers**: Use `policy.NewMemoryStore(p)` for in-memory policy testing without NATS. - **Pipes helpers**: Use `pipes.NewMemoryStore(queries...)` for in-memory pipes testing without NATS. - **Response assertions**: Use `testutil.AssertJSONResponse(t, rec, status, expected)` and `testutil.AssertJSONContains(t, rec, status, substring)`. @@ -379,7 +379,7 @@ Internal-only backend changes (middleware refactors, observability internals, de 1. Create `*_test.go` files in the same package as the code under test. 2. Use table-driven tests with `t.Run(tt.name, ...)` for multiple scenarios. 3. Use shared mocks from `internal/testutil/` (MockPublisher, MockCache, MockDeduplicator, MockSubscriber). -4. Use `testutil.MakeJWT(t, claims)` for auth tests, `discovery.NewSchemaRegistryFromMap(...)` for schema-aware tests, `policy.NewMemoryStore(p)` for policy tests, `pipes.NewMemoryStore(queries...)` for pipes tests. +4. Use `testutil.MakeJWT(t, claims)` for auth tests, `testutil.NewTestSchemaRegistry(t, ...)` for schema-aware tests, `policy.NewMemoryStore(p)` for policy tests, `pipes.NewMemoryStore(queries...)` for pipes tests. 5. Use `testutil.AssertJSONResponse` and `testutil.AssertJSONContains` for HTTP handler assertions. 6. Run `make test` — it gates the unit-test coverage threshold from `.testcoverage.yml`, so a passing run already confirms coverage. 7. Aim for 80%+ coverage on new code. The project-wide CI-enforced minimum is 80% (merged unit + integration + e2e via `.testcoverage.yml`'s `threshold.total`); per-suite minima are unit 80%, integration 20%, e2e 60%, sdk 50%. diff --git a/CHANGELOG.md b/CHANGELOG.md index c83b69bc..8649e664 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -369,7 +369,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Unit tests — security & robustness**: Config validation tests (11 scenarios), coerceValue tests (20 subtests), builder DefaultMaxRows tests, policy Validate/Evaluate edge cases (nil policy, negative MaxRows/MaxExecutionTime, service role, nil claims, multiple templates), ingest policy check clause match/auto-inject/deny-columns/admin bypass, discovery validation (Tuple/Enum16/Decimal/IPv4/IPv6/unknown type, nil data, all-defaults), tiered cache singleflight dedup verification, pipes handler Put/Delete success paths, router wiring verification, middleware default role claim and non-bearer token tests. - **Test infrastructure**: Expanded `internal/testutil/` with shared mocks (`MockPublisher`, `MockCache`, `MockDeduplicator`, `MockSubscriber` in `mocks.go`), JWT helpers (`MakeJWT`, `MakeExpiredJWT` in `jwt.go`), schema helpers (`NewTestSchemaRegistry`), and response assertions (`AssertJSONResponse`, `AssertJSONContains`). - **Policy test helper**: `policy.NewMemoryStore(p)` for in-memory policy testing without NATS. -- **Schema test helper**: `discovery.NewSchemaRegistryFromMap(tables)` factory for schema-aware tests without ClickHouse. +- **Schema test helper**: `testutil.NewTestSchemaRegistry(t, tables)` builds schema-aware test registries through the real discovery path (mock ClickHouse connection) — no ClickHouse required, no test-only constructor in `discovery`. - **Unit tests — critical path**: `middleware_test.go` (12 scenarios: auth disabled, dev mode, JWT validation, role extraction, claim extraction), `policy_test.go` (Evaluate, IsColumnAllowed, IsAggregationAllowed, navigateClaims, resolveTemplate), `config_test.go` (defaults, YAML loading, env overrides, invalid YAML), `ingest_test.go` (valid payload, missing/unknown table, invalid JSON, schema validation, dedup, publish errors, policy enforcement). - **Unit tests — core features**: `hub_test.go`, `health_test.go`, `schema_test.go`, `transform_test.go`, `router_test.go`, `builder_test.go`, `tiered_test.go`, `pipes_test.go`. - **Unit tests — handlers & cache**: `policy_test.go` (handler: Get nil/populated, Put/Validate invalid JSON, Validate valid), `query_test.go` (missing SQL, invalid JSON, policy forbids/allows raw SQL, cache key determinism), `structured_query_test.go` (missing/unknown table, invalid JSON, policy forbidden/column/aggregation checks), `pipes_test.go` (handler: List, Get found/not-found, Execute not-found/role-forbidden/allowed/wildcard/missing-param/query-params, Put invalid JSON), `stream_test.go` (SSE/WS applyStreamPolicy: no-policy passthrough, column filtering, forbidden table, non-event JSON, invalid JSON), `local_test.go` (Get miss, Set and Get, expired key, overwrite, zero TTL). diff --git a/internal/api/errors_test.go b/internal/api/errors_test.go index 358a3f34..bdb2892f 100644 --- a/internal/api/errors_test.go +++ b/internal/api/errors_test.go @@ -11,7 +11,6 @@ import ( "testing" "github.com/Wave-RF/WaveHouse/internal/auth" - "github.com/Wave-RF/WaveHouse/internal/discovery" "github.com/Wave-RF/WaveHouse/internal/pipes" "github.com/Wave-RF/WaveHouse/internal/policy" "github.com/Wave-RF/WaveHouse/internal/stream" @@ -162,7 +161,7 @@ func TestPipesHandler_Execute_DenialLogsAllowedRoles(t *testing.T) { func TestIngest_DenialLogsPolicyGate(t *testing.T) { t.Parallel() logger, buf := warnBufLogger() - h := NewIngestHandler(testRegistry(), &testutil.MockPublisher{}, logger) + h := NewIngestHandler(testRegistry(t), &testutil.MockPublisher{}, logger) h.PolicyStore = policy.NewMemoryStore(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": {Select: map[string]policy.RolePermissions{"viewer": {}}}, // no insert for viewer @@ -190,7 +189,7 @@ func TestIngest_DenialLogsPolicyGate(t *testing.T) { func TestAuthzDenied_LogsChiRoutePattern(t *testing.T) { t.Parallel() logger, buf := warnBufLogger() - reg := discovery.NewSchemaRegistryFromMap(nil) + reg := testutil.NewTestSchemaRegistry(t, nil) router := NewRouter(Dependencies{ Ingest: NewIngestHandler(reg, &testutil.MockPublisher{}, logger), Query: &QueryHandler{}, diff --git a/internal/api/ingest_test.go b/internal/api/ingest_test.go index 61413a31..85bd022f 100644 --- a/internal/api/ingest_test.go +++ b/internal/api/ingest_test.go @@ -20,8 +20,8 @@ import ( "github.com/stretchr/testify/require" ) -func testRegistry() *discovery.SchemaRegistry { - return discovery.NewSchemaRegistryFromMap([]*discovery.TableSchema{ +func testRegistry(t testing.TB) *discovery.SchemaRegistry { + return testutil.NewTestSchemaRegistry(t, []*discovery.TableSchema{ { Name: "clicks", Columns: []discovery.Column{ @@ -46,7 +46,7 @@ func ingestRequest(t *testing.T, table string, body any) *http.Request { func TestIngest_ValidPayload(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) req := ingestRequest(t, "clicks", map[string]any{"page": "/home", "count": 1}) w := httptest.NewRecorder() @@ -96,7 +96,7 @@ func TestIngest_MissingTable(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) req := httptest.NewRequestWithContext( context.Background(), @@ -119,7 +119,7 @@ func TestIngest_MissingTable(t *testing.T) { func TestIngest_UnknownTable(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) req := ingestRequest(t, "nonexistent", map[string]any{"x": 1}) w := httptest.NewRecorder() @@ -133,7 +133,7 @@ func TestIngest_UnknownTable(t *testing.T) { func TestIngest_InvalidJSON(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) r := httptest.NewRequestWithContext(context.Background(), http.MethodPost, "/v1/ingest?table=clicks", bytes.NewReader([]byte("not json"))) @@ -147,7 +147,7 @@ func TestIngest_InvalidJSON(t *testing.T) { func TestIngest_SchemaValidation_UnknownField(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) req := ingestRequest(t, "clicks", map[string]any{"page": "/home", "nonexistent_field": 42}) w := httptest.NewRecorder() @@ -161,7 +161,7 @@ func TestIngest_Dedup_FirstTime(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} dedup := testutil.NewMockDeduplicator() - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.Dedup = dedup h.IDField = "event_id" @@ -177,7 +177,7 @@ func TestIngest_Dedup_Duplicate(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} dedup := testutil.NewMockDeduplicator() - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.Dedup = dedup h.IDField = "event_id" @@ -203,7 +203,7 @@ func TestIngest_Dedup_Duplicate(t *testing.T) { func TestIngest_PublishError_503(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{Err: errors.New("maximum bytes exceeded")} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) req := ingestRequest(t, "clicks", map[string]any{"page": "/home"}) w := httptest.NewRecorder() @@ -217,7 +217,7 @@ func TestIngest_PublishError_503(t *testing.T) { func TestIngest_PublishError_500(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{Err: errors.New("some other error")} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) req := ingestRequest(t, "clicks", map[string]any{"page": "/home"}) w := httptest.NewRecorder() @@ -231,7 +231,7 @@ func TestIngest_PublishError_500(t *testing.T) { func TestIngest_Policy_Forbidden(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.PolicyStore = policy.NewMemoryStore(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { @@ -258,7 +258,7 @@ func TestIngest_Policy_Forbidden(t *testing.T) { func TestIngest_Policy_ColumnDenied(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.PolicyStore = policy.NewMemoryStore(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { @@ -286,7 +286,7 @@ func TestIngest_Policy_ColumnDenied(t *testing.T) { func TestIngest_Policy_CheckClause_Mismatch(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) orgTemplate := "{{ jwt.org_id }}" h.PolicyStore = policy.NewMemoryStore(&policy.Policy{ Tables: map[string]policy.TablePolicy{ @@ -318,7 +318,7 @@ func TestIngest_Policy_CheckClause_Mismatch(t *testing.T) { func TestIngest_Policy_CheckClause_Match(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) orgTemplate := "{{ jwt.org_id }}" h.PolicyStore = policy.NewMemoryStore(&policy.Policy{ Tables: map[string]policy.TablePolicy{ @@ -351,7 +351,7 @@ func TestIngest_Policy_CheckClause_Match(t *testing.T) { func TestIngest_Policy_CheckClause_AutoInject(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) orgTemplate := "{{ jwt.org_id }}" h.PolicyStore = policy.NewMemoryStore(&policy.Policy{ Tables: map[string]policy.TablePolicy{ @@ -403,7 +403,7 @@ func checkInStore() *policy.Store { func TestIngest_Policy_CheckIn_InSet(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.PolicyStore = checkInStore() // org_id is one of the token's allowed orgs — should pass. @@ -422,7 +422,7 @@ func TestIngest_Policy_CheckIn_InSet(t *testing.T) { func TestIngest_Policy_CheckIn_NotInSet(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.PolicyStore = checkInStore() // org_id is NOT one of the token's allowed orgs — forging another tenant's row. @@ -442,7 +442,7 @@ func TestIngest_Policy_CheckIn_NotInSet(t *testing.T) { func TestIngest_Policy_CheckIn_Absent_FailsClosed(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.PolicyStore = checkInStore() // org_id omitted — unlike _eq there's no single value to auto-inject, so the @@ -469,7 +469,7 @@ func TestIngest_Policy_CheckIn_Absent_FailsClosed(t *testing.T) { func TestIngest_Policy_CheckIn_AbsentClaim_FailsClosed(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.PolicyStore = checkInStore() // The `orgs` claim is absent entirely, so the _in set resolves to a typed-nil @@ -493,7 +493,7 @@ func TestIngest_Dedup_MissingIDField(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} dedup := testutil.NewMockDeduplicator() - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.Dedup = dedup h.IDField = "event_id" @@ -512,7 +512,7 @@ func TestIngest_Dedup_MissingIDField(t *testing.T) { func TestIngest_Dedup_RequireID_Rejects(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.Dedup = testutil.NewMockDeduplicator() h.IDField = "event_id" h.RequireID = true @@ -535,7 +535,7 @@ func TestIngest_Dedup_RequireID_Rejects(t *testing.T) { func TestIngest_NDJSON_RequireID_Rejects(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.Dedup = testutil.NewMockDeduplicator() h.IDField = "event_id" h.RequireID = true @@ -563,7 +563,7 @@ func TestIngest_NDJSON_RequireID_Rejects(t *testing.T) { func TestIngest_Policy_DenyColumns(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.PolicyStore = policy.NewMemoryStore(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { @@ -590,7 +590,7 @@ func TestIngest_Policy_DenyColumns(t *testing.T) { func TestIngest_AdminRole_NoPolicy(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.PolicyStore = policy.NewMemoryStore(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": {}, @@ -654,7 +654,7 @@ func resultAt(t *testing.T, resp batchResult, index int) recordResult { func TestIngest_NDJSON_AllValid(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) req := ndjsonRequest(t, "clicks", jsonLine(t, map[string]any{"page": "/a", "count": 1}), @@ -682,7 +682,7 @@ func TestIngest_NDJSON_AllValid(t *testing.T) { func TestIngest_NDJSON_PartialFailure_Validation(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) req := ndjsonRequest(t, "clicks", jsonLine(t, map[string]any{"page": "/a"}), @@ -708,7 +708,7 @@ func TestIngest_NDJSON_PartialFailure_Validation(t *testing.T) { func TestIngest_NDJSON_MalformedLine(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) req := ndjsonRequest(t, "clicks", jsonLine(t, map[string]any{"page": "/a"}), @@ -733,7 +733,7 @@ func TestIngest_NDJSON_MalformedLine(t *testing.T) { func TestIngest_NDJSON_BlankLinesSkipped(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) // Leading, interior, and whitespace-only lines are all skipped; only real // records are counted. @@ -770,7 +770,7 @@ func TestIngest_NDJSON_EmptyBody(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) req := ndjsonRequest(t, "clicks", tt.lines...) w := httptest.NewRecorder() @@ -788,7 +788,7 @@ func TestIngest_NDJSON_Dedup(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} dedup := testutil.NewMockDeduplicator() - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.Dedup = dedup h.IDField = "event_id" @@ -817,7 +817,7 @@ func TestIngest_NDJSON_Backpressure_503(t *testing.T) { // Publisher rejects every publish with the backpressure sentinel; the first // valid record aborts the whole batch with 503 + Retry-After. pub := &testutil.MockPublisher{Err: errors.New("maximum bytes exceeded")} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) req := ndjsonRequest(t, "clicks", jsonLine(t, map[string]any{"page": "/a"}), @@ -834,7 +834,7 @@ func TestIngest_NDJSON_Backpressure_503(t *testing.T) { func TestIngest_NDJSON_PublishError_500(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{Err: errors.New("some other error")} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) req := ndjsonRequest(t, "clicks", jsonLine(t, map[string]any{"page": "/a"})) w := httptest.NewRecorder() @@ -848,7 +848,7 @@ func TestIngest_NDJSON_PublishError_500(t *testing.T) { func TestIngest_NDJSON_Policy_ColumnDenied_PerLine(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.PolicyStore = policy.NewMemoryStore(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { @@ -883,7 +883,7 @@ func TestIngest_NDJSON_Policy_ColumnDenied_PerLine(t *testing.T) { func TestIngest_NDJSON_Policy_TableForbidden(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.PolicyStore = policy.NewMemoryStore(&policy.Policy{ Tables: map[string]policy.TablePolicy{ "clicks": { @@ -910,7 +910,7 @@ func TestIngest_NDJSON_Policy_TableForbidden(t *testing.T) { func TestIngest_NDJSON_Policy_CheckClause_PerLineAndAutoInject(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) orgTemplate := "{{ jwt.org_id }}" h.PolicyStore = policy.NewMemoryStore(&policy.Policy{ Tables: map[string]policy.TablePolicy{ @@ -952,7 +952,7 @@ func TestIngest_NDJSON_Policy_CheckClause_PerLineAndAutoInject(t *testing.T) { func TestIngest_NDJSON_ContentTypeWithCharset(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) req := ndjsonRequest(t, "clicks", jsonLine(t, map[string]any{"page": "/a"})) req.Header.Set("Content-Type", "application/x-ndjson; charset=utf-8") @@ -969,7 +969,7 @@ func TestIngest_NDJSON_ContentTypeWithCharset(t *testing.T) { func TestIngest_NDJSON_ErrorsTruncated(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) const total = maxReportedResults + 50 lines := make([]string, total) @@ -1037,7 +1037,7 @@ func rawIngestRequest(t *testing.T, table, contentType, body string) *http.Reque func TestIngest_JSONArray_AllValid(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) // A bare JSON array with no Content-Type must be accepted as a batch. req := ingestRequest(t, "clicks", []map[string]any{ @@ -1061,7 +1061,7 @@ func TestIngest_JSONArray_AllValid(t *testing.T) { func TestIngest_JSONArray_SingleElement(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) // A one-element array is still a batch (returns the results envelope, not // the single-object {"ok":true}). @@ -1081,7 +1081,7 @@ func TestIngest_JSONArray_SingleElement(t *testing.T) { func TestIngest_JSONArray_WithJSONContentType(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) // Content-Type: application/json must NOT force the single-object path when // the body is an array — the sniffer wins. @@ -1099,7 +1099,7 @@ func TestIngest_JSONArray_WithJSONContentType(t *testing.T) { func TestIngest_JSONArray_PartialValidationFailure(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) req := ingestRequest(t, "clicks", []map[string]any{ {"page": "/a"}, @@ -1124,7 +1124,7 @@ func TestIngest_JSONArray_PartialValidationFailure(t *testing.T) { func TestIngest_JSONArray_ScalarElements(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) // Non-object elements (number, string, nested array) are wrong-typed: the // decoder stays in sync, so each is a per-record error and the objects @@ -1155,7 +1155,7 @@ func TestIngest_JSONArray_ScalarElements(t *testing.T) { func TestIngest_JSONArray_SyntaxError_Fatal(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) // A structural syntax error desyncs the decoder — the whole request fails // (400), unlike a per-element type error. The leading good element may have @@ -1189,7 +1189,7 @@ func TestIngest_JSONArray_Truncated_Fatal(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) req := rawIngestRequest(t, "clicks", "application/json", tt.body) w := httptest.NewRecorder() @@ -1205,7 +1205,7 @@ func TestIngest_JSONArray_Truncated_Fatal(t *testing.T) { func TestIngest_JSONArray_Empty(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) // An explicit empty array is a valid, record-less batch → 200 with no rows. req := rawIngestRequest(t, "clicks", "application/json", `[]`) @@ -1222,7 +1222,7 @@ func TestIngest_JSONArray_Empty(t *testing.T) { func TestIngest_Sniff_ArrayBodyBeatsNDJSONHeader(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) // Body starts with '[' → array path, even though the header says NDJSON. req := rawIngestRequest(t, "clicks", "application/x-ndjson", `[{"page":"/a"},{"page":"/b"}]`) @@ -1239,7 +1239,7 @@ func TestIngest_Sniff_ArrayBodyBeatsNDJSONHeader(t *testing.T) { func TestIngest_SingleObject_PrettyPrinted(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) // A multi-line (pretty-printed) single object must not be mistaken for // NDJSON — it's one record on the single-object path. @@ -1257,7 +1257,7 @@ func TestIngest_SingleObject_PrettyPrinted(t *testing.T) { func TestIngest_Unlabeled_ConcatenatedObjects_FirstOnly(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) // Two concatenated objects with no NDJSON header take the single-object // path and ingest only the first (matching the historical behavior — send @@ -1290,7 +1290,7 @@ func TestIngest_LeadingWhitespace_Sniff(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) req := rawIngestRequest(t, "clicks", "application/json", tt.body) w := httptest.NewRecorder() @@ -1327,7 +1327,7 @@ func TestIngest_EmptyBody(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) req := rawIngestRequest(t, "clicks", tt.contentType, tt.body) w := httptest.NewRecorder() @@ -1366,7 +1366,7 @@ func TestIngest_BodyCap_413(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(testRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(testRegistry(t), pub, testutil.NopLogger()) h.maxRequestBytes = tt.cap // below the body req := rawIngestRequest(t, "clicks", tt.ct, tt.body) @@ -1382,8 +1382,8 @@ func TestIngest_BodyCap_413(t *testing.T) { // tsRegistry returns a registry whose events table carries the timestamp column // shapes the #372 canonicalization path rewrites. -func tsRegistry() *discovery.SchemaRegistry { - return discovery.NewSchemaRegistryFromMap([]*discovery.TableSchema{ +func tsRegistry(t testing.TB) *discovery.SchemaRegistry { + return testutil.NewTestSchemaRegistry(t, []*discovery.TableSchema{ { Name: "events", Columns: []discovery.Column{ @@ -1414,7 +1414,7 @@ func publishedData(t *testing.T, pub *testutil.MockPublisher) map[string]any { func TestIngest_TimestampsCanonicalized(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(tsRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(tsRegistry(t), pub, testutil.NopLogger()) req := ingestRequest(t, "events", map[string]any{ "name": "e", @@ -1436,7 +1436,7 @@ func TestIngest_TimestampsCanonicalized(t *testing.T) { func TestIngest_TimestampGarbage_PassesThrough(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(tsRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(tsRegistry(t), pub, testutil.NopLogger()) req := ingestRequest(t, "events", map[string]any{"name": "e", "ts": "banana"}) w := httptest.NewRecorder() @@ -1451,7 +1451,7 @@ func TestIngest_TimestampGarbage_PassesThrough(t *testing.T) { func TestIngest_Batch_MixedTimestampSpellings(t *testing.T) { t.Parallel() pub := &testutil.MockPublisher{} - h := NewIngestHandler(tsRegistry(), pub, testutil.NopLogger()) + h := NewIngestHandler(tsRegistry(t), pub, testutil.NopLogger()) req := ingestRequest(t, "events", []map[string]any{ {"name": "a", "ts": "2026-06-21T04:00:00Z"}, diff --git a/internal/api/router_test.go b/internal/api/router_test.go index bf9dcaac..e0fe5042 100644 --- a/internal/api/router_test.go +++ b/internal/api/router_test.go @@ -279,7 +279,7 @@ func TestCORSMiddleware_BlockedOriginPreflight(t *testing.T) { func TestNewRouter_RoutesRegistered(t *testing.T) { t.Parallel() - reg := discovery.NewSchemaRegistryFromMap([]*discovery.TableSchema{ + reg := testutil.NewTestSchemaRegistry(t, []*discovery.TableSchema{ {Name: "events", Columns: []discovery.Column{{Name: "id", Type: "String"}}}, }) pub := &testutil.MockPublisher{} @@ -409,7 +409,7 @@ func TestNewRouter_CORSOnStream(t *testing.T) { func TestNewRouter_RawSQLAdminGate(t *testing.T) { t.Parallel() - reg := discovery.NewSchemaRegistryFromMap(nil) + reg := testutil.NewTestSchemaRegistry(t, nil) pub := &testutil.MockPublisher{} hub := stream.NewHub(nil, nil) @@ -470,7 +470,7 @@ func TestNewRouter_RawSQLAdminGate(t *testing.T) { func TestNewRouter_OptionalDepsNil(t *testing.T) { t.Parallel() - reg := discovery.NewSchemaRegistryFromMap(nil) + reg := testutil.NewTestSchemaRegistry(t, nil) pub := &testutil.MockPublisher{} hub := stream.NewHub(nil, nil) @@ -521,7 +521,7 @@ func TestCORSMiddleware_EmptyOrigins_AllowAll(t *testing.T) { func TestNewRouter_NotFoundEmitsJSON(t *testing.T) { t.Parallel() - reg := discovery.NewSchemaRegistryFromMap(nil) + reg := testutil.NewTestSchemaRegistry(t, nil) pub := &testutil.MockPublisher{} hub := stream.NewHub(nil, nil) deps := Dependencies{ @@ -546,7 +546,7 @@ func TestNewRouter_NotFoundEmitsJSON(t *testing.T) { func TestNewRouter_MethodNotAllowedEmitsJSON(t *testing.T) { t.Parallel() - reg := discovery.NewSchemaRegistryFromMap(nil) + reg := testutil.NewTestSchemaRegistry(t, nil) pub := &testutil.MockPublisher{} hub := stream.NewHub(nil, nil) deps := Dependencies{ @@ -643,7 +643,7 @@ func TestJSONRecoverer_PanicAfterPartialWriteDoesNotCorrupt(t *testing.T) { // gate is its entire authorization story. func TestNewRouter_SchemaAdminOnly(t *testing.T) { t.Parallel() - reg := discovery.NewSchemaRegistryFromMap(nil) + reg := testutil.NewTestSchemaRegistry(t, nil) pub := &testutil.MockPublisher{} hub := stream.NewHub(nil, nil) diff --git a/internal/api/schema_test.go b/internal/api/schema_test.go index f7386a2a..dd50be10 100644 --- a/internal/api/schema_test.go +++ b/internal/api/schema_test.go @@ -15,7 +15,7 @@ import ( func TestSchema_List(t *testing.T) { t.Parallel() - reg := discovery.NewSchemaRegistryFromMap([]*discovery.TableSchema{ + reg := testutil.NewTestSchemaRegistry(t, []*discovery.TableSchema{ {Name: "clicks", Columns: []discovery.Column{{Name: "page", Type: "String"}}}, {Name: "users", Columns: []discovery.Column{{Name: "name", Type: "String"}}}, }) @@ -34,7 +34,7 @@ func TestSchema_List(t *testing.T) { func TestSchema_Get_Exists(t *testing.T) { t.Parallel() - reg := discovery.NewSchemaRegistryFromMap([]*discovery.TableSchema{ + reg := testutil.NewTestSchemaRegistry(t, []*discovery.TableSchema{ {Name: "clicks", Columns: []discovery.Column{ {Name: "page", Type: "String"}, {Name: "count", Type: "UInt64"}, @@ -57,7 +57,7 @@ func TestSchema_Get_Exists(t *testing.T) { func TestSchema_Get_NotFound(t *testing.T) { t.Parallel() - reg := discovery.NewSchemaRegistryFromMap([]*discovery.TableSchema{}) + reg := testutil.NewTestSchemaRegistry(t, []*discovery.TableSchema{}) h := NewSchemaHandler(reg) w := httptest.NewRecorder() diff --git a/internal/api/structured_query_test.go b/internal/api/structured_query_test.go index ba13cad3..e18a8c05 100644 --- a/internal/api/structured_query_test.go +++ b/internal/api/structured_query_test.go @@ -29,8 +29,8 @@ func structuredQueryRequest(t *testing.T, table string, sq query.StructuredQuery return httptest.NewRequestWithContext(context.Background(), http.MethodPost, "/v1/query?table="+url.QueryEscape(table), bytes.NewReader(body)) } -func newStructuredQueryHandler() *StructuredQueryHandler { - reg := discovery.NewSchemaRegistryFromMap([]*discovery.TableSchema{ +func newStructuredQueryHandler(t testing.TB) *StructuredQueryHandler { + reg := testutil.NewTestSchemaRegistry(t, []*discovery.TableSchema{ { Name: "clicks", Columns: []discovery.Column{ @@ -76,7 +76,7 @@ func TestStructuredQuery_MissingTable(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() - h := newStructuredQueryHandler() + h := newStructuredQueryHandler(t) body, err := json.Marshal(query.StructuredQuery{Columns: []string{"page"}}) require.NoError(t, err) @@ -100,7 +100,7 @@ func TestStructuredQuery_MissingTable(t *testing.T) { func TestStructuredQuery_UnknownTable(t *testing.T) { t.Parallel() - h := newStructuredQueryHandler() + h := newStructuredQueryHandler(t) r := structuredQueryRequest(t, "nope", query.StructuredQuery{Columns: []string{"x"}}) w := httptest.NewRecorder() h.Handle(w, r) @@ -112,7 +112,7 @@ func TestStructuredQuery_UnknownTable(t *testing.T) { func TestStructuredQuery_InvalidJSON(t *testing.T) { t.Parallel() - h := newStructuredQueryHandler() + h := newStructuredQueryHandler(t) r := httptest.NewRequestWithContext(context.Background(), http.MethodPost, "/v1/query?table=clicks", bytes.NewReader([]byte(`{bad}`))) w := httptest.NewRecorder() h.Handle(w, r) @@ -131,7 +131,7 @@ func TestStructuredQuery_RequestBodyCap(t *testing.T) { t.Parallel() const testCap = 64 - h := newStructuredQueryHandler() + h := newStructuredQueryHandler(t) h.maxRequestBytes = testCap // A valid query whose JSON exceeds the cap — a big `in`-list, the exact @@ -163,7 +163,7 @@ func TestStructuredQuery_PolicyForbidden(t *testing.T) { }, }, } - h := newStructuredQueryHandler() + h := newStructuredQueryHandler(t) h.PolicyStore = policy.NewMemoryStore(p) sq := query.StructuredQuery{Columns: []string{"page"}} @@ -191,7 +191,7 @@ func TestStructuredQuery_ColumnNotAllowed(t *testing.T) { }, }, } - h := newStructuredQueryHandler() + h := newStructuredQueryHandler(t) h.PolicyStore = policy.NewMemoryStore(p) // Request "count" column which is not in AllowColumns. @@ -224,7 +224,7 @@ func TestStructuredQuery_AggregationNotAllowed(t *testing.T) { }, }, } - h := newStructuredQueryHandler() + h := newStructuredQueryHandler(t) h.PolicyStore = policy.NewMemoryStore(p) sq := query.StructuredQuery{ @@ -248,7 +248,7 @@ func TestStructuredQuery_AggregationNotAllowed(t *testing.T) { func TestStructuredQuery_NoPolicyAllowsAll(t *testing.T) { t.Parallel() - h := newStructuredQueryHandler() + h := newStructuredQueryHandler(t) // No PolicyStore — all queries should be allowed (past policy). sq := query.StructuredQuery{Columns: []string{"page"}} r := structuredQueryRequest(t, "clicks", sq) @@ -281,7 +281,7 @@ func (c *sqlCapturingConn) Query(_ context.Context, sql string, _ ...any) (drive // sensitiveSchema has a column (payload, user_id) that restrictive policies hide. func newCapturingHandler(t *testing.T, conn driver.Conn, p *policy.Policy) *StructuredQueryHandler { t.Helper() - reg := discovery.NewSchemaRegistryFromMap([]*discovery.TableSchema{ + reg := testutil.NewTestSchemaRegistry(t, []*discovery.TableSchema{ { Name: "clicks", Columns: []discovery.Column{ diff --git a/internal/discovery/discovery.go b/internal/discovery/discovery.go index fa7a0587..965e8213 100644 --- a/internal/discovery/discovery.go +++ b/internal/discovery/discovery.go @@ -3,7 +3,6 @@ package discovery import ( "context" "fmt" - "io" "log/slog" "sync" "time" @@ -21,9 +20,9 @@ type Column struct { HasDefault bool `json:"has_default"` // tsSpec is a DateTime/DateTime64 column's canonicalization spec, resolved - // once at schema build (Refresh / NewSchemaRegistryFromMap). nil for - // non-timestamp columns, hand-built Column literals, and unresolvable zones — - // CanonicalizeTimestamps passes those through untouched (fail-open, #372). + // once at schema build (Refresh). nil for non-timestamp columns, hand-built + // Column literals, and unresolvable zones — CanonicalizeTimestamps passes + // those through untouched (fail-open, #372). tsSpec *timestampSpec } @@ -226,19 +225,3 @@ func (sr *SchemaRegistry) StartAutoRefresh(ctx context.Context) { func isNullable(chType string) bool { return len(chType) > 9 && chType[:9] == "Nullable(" } - -// NewSchemaRegistryFromMap creates a SchemaRegistry pre-loaded with the given -// table schemas. Intended for testing — no ClickHouse connection is required. -// Timestamp specs are precomputed like Refresh does (UTC as the server default). -func NewSchemaRegistryFromMap(tables []*TableSchema) *SchemaRegistry { - logger := slog.New(slog.NewTextHandler(io.Discard, nil)) - m := make(map[string]*TableSchema, len(tables)) - for _, t := range tables { - resolveTimestampSpecs(t, time.UTC, logger) - m[t.Name] = t - } - return &SchemaRegistry{ - tables: m, - logger: logger, - } -} diff --git a/internal/discovery/discovery_test.go b/internal/discovery/discovery_test.go index 554f6f82..fb145400 100644 --- a/internal/discovery/discovery_test.go +++ b/internal/discovery/discovery_test.go @@ -60,46 +60,50 @@ func TestNewSchemaRegistry_ConstructorDefaults(t *testing.T) { assert.Nil(t, sr.Get("anything")) } -func TestNewSchemaRegistryFromMap_PopulatesAndLookups(t *testing.T) { +// TestRefresh_PopulatesAndLookups: Refresh turns the system.columns scan into +// name-keyed schemas — Get finds them, List returns them all. +func TestRefresh_PopulatesAndLookups(t *testing.T) { t.Parallel() - clicks := &TableSchema{Name: "clicks", Columns: []Column{{Name: "id", Type: "String"}}} - users := &TableSchema{Name: "users", Columns: []Column{{Name: "id", Type: "UInt64"}}} - - sr := NewSchemaRegistryFromMap([]*TableSchema{clicks, users}) - require.NotNil(t, sr) + conn := &fakeConn{columns: [][4]string{ + {"clicks", "id", "String", ""}, + {"clicks", "page", "String", "DEFAULT"}, + {"users", "id", "UInt64", ""}, + }} + sr := NewSchemaRegistry(conn, "test", time.Hour, discardLogger()) + require.NoError(t, sr.Refresh(context.Background())) - assert.Same(t, clicks, sr.Get("clicks")) - assert.Same(t, users, sr.Get("users")) + clicks := sr.Get("clicks") + require.NotNil(t, clicks) + assert.Equal(t, []string{"id", "page"}, clicks.ColumnNames()) + assert.False(t, clicks.Columns[0].HasDefault) + assert.True(t, clicks.Columns[1].HasDefault, "non-empty default_kind ⇒ HasDefault") + require.NotNil(t, sr.Get("users")) assert.Nil(t, sr.Get("missing")) - - got := sr.List() - assert.Len(t, got, 2) - names := map[string]bool{} - for _, s := range got { - names[s.Name] = true - } - assert.True(t, names["clicks"]) - assert.True(t, names["users"]) + assert.Len(t, sr.List(), 2) } -func TestNewSchemaRegistryFromMap_Empty(t *testing.T) { +// TestRefresh_EmptyDatabase: zero system.columns rows yield an empty, usable +// registry — nil lookups, empty List, no error. +func TestRefresh_EmptyDatabase(t *testing.T) { t.Parallel() - sr := NewSchemaRegistryFromMap(nil) - require.NotNil(t, sr) + sr, _ := newFakeRegistry(t, nil) + require.NoError(t, sr.Refresh(context.Background())) assert.Empty(t, sr.List()) assert.Nil(t, sr.Get("x")) } -// fakeConn implements just enough of driver.Conn to drive RetryRefresh. It -// returns successive errors from errsThenSuccess, and once that slice is -// exhausted returns emptyRows (which makes Refresh succeed with zero tables). -// The nil-embedded interface trick handles every other method — none of them +// fakeConn implements just enough of driver.Conn to drive Refresh and +// RetryRefresh. It returns successive errors from errsThenSuccess, and once +// that slice is exhausted serves columns as the system.columns result set +// (emptyRows when nil, which makes Refresh succeed with zero tables). The +// nil-embedded interface trick handles every other method — none of them // are reached by Refresh. type fakeConn struct { driver.Conn errsThenSuccess []error calls atomic.Int32 - tz string // SELECT timezone() answer; "" ⇒ "UTC" + tz string // SELECT timezone() answer; "" ⇒ "UTC" + columns [][4]string // system.columns rows served on success; nil ⇒ none } func (c *fakeConn) Query(_ context.Context, _ string, _ ...any) (driver.Rows, error) { @@ -107,6 +111,9 @@ func (c *fakeConn) Query(_ context.Context, _ string, _ ...any) (driver.Rows, er if int(n) <= len(c.errsThenSuccess) { return nil, c.errsThenSuccess[n-1] } + if c.columns != nil { + return &fakeRows{rows: c.columns}, nil + } return &emptyRows{}, nil } @@ -145,6 +152,37 @@ func (*emptyRows) ColumnTypes() []driver.ColumnType { return nil } +// fakeRows plays a system.columns result set: one [table, name, type, +// default_kind] row per column, in the given order. +type fakeRows struct { + driver.Rows + rows [][4]string + next int +} + +func (r *fakeRows) Next() bool { + r.next++ + return r.next <= len(r.rows) +} + +func (r *fakeRows) Scan(dest ...any) error { + row := r.rows[r.next-1] + if len(dest) != len(row) { + return errors.New("unexpected system.columns scan shape") + } + for i, d := range dest { + s, ok := d.(*string) + if !ok { + return errors.New("system.columns scan dest must be *string") + } + *s = row[i] + } + return nil +} + +func (*fakeRows) Close() error { return nil } +func (*fakeRows) Err() error { return nil } + func newFakeRegistry(t *testing.T, errs []error) (*SchemaRegistry, *fakeConn) { t.Helper() conn := &fakeConn{errsThenSuccess: errs} @@ -354,9 +392,8 @@ func TestClampBackoff(t *testing.T) { // would otherwise panic). func TestStartAutoRefresh_ExitsOnContextCancel(t *testing.T) { t.Parallel() - sr := NewSchemaRegistryFromMap(nil) // Long interval so the ticker never fires before cancel. - sr.refreshInterval = time.Hour + sr := NewSchemaRegistry(nil, "test", time.Hour, discardLogger()) ctx, cancel := context.WithCancel(context.Background()) diff --git a/internal/discovery/timestamp_test.go b/internal/discovery/timestamp_test.go index a4dfca98..e297dcf1 100644 --- a/internal/discovery/timestamp_test.go +++ b/internal/discovery/timestamp_test.go @@ -1,6 +1,7 @@ package discovery import ( + "context" "encoding/json" "io" "log/slog" @@ -176,12 +177,14 @@ func TestResolveTimestampSpecs(t *testing.T) { assert.Equal(t, "2026-06-21 04:00:00", data["broken"]) } -// TestNewSchemaRegistryFromMap_PrecomputesSpecs: the map-built test registry -// resolves specs like Refresh does (UTC server default), so handler tests -// exercise the same precomputed path as production. -func TestNewSchemaRegistryFromMap_PrecomputesSpecs(t *testing.T) { +// TestRefresh_PrecomputesSpecs: schema builds resolve timestamp specs with the +// server zone from SELECT timezone(), so registry consumers — production and +// the testutil mock-conn path alike — exercise the same precomputed path. +func TestRefresh_PrecomputesSpecs(t *testing.T) { t.Parallel() - reg := NewSchemaRegistryFromMap([]*TableSchema{tsSchema("DateTime")}) + conn := &fakeConn{columns: [][4]string{{"t", "ts", "DateTime", ""}}} + reg := NewSchemaRegistry(conn, "test", time.Hour, discardLogger()) + require.NoError(t, reg.Refresh(context.Background())) col := reg.Get("t").Columns[0] require.NotNil(t, col.tsSpec) assert.Equal(t, time.UTC, col.tsSpec.loc) diff --git a/internal/testutil/testutil.go b/internal/testutil/testutil.go index e31a4b07..e5be4e5e 100644 --- a/internal/testutil/testutil.go +++ b/internal/testutil/testutil.go @@ -1,12 +1,16 @@ package testutil import ( + "context" "encoding/json" + "fmt" "io" "log/slog" "net/http/httptest" "testing" + "time" + "github.com/ClickHouse/clickhouse-go/v2/lib/driver" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" @@ -20,12 +24,90 @@ func NopLogger() *slog.Logger { } // NewTestSchemaRegistry creates a SchemaRegistry pre-loaded with the given -// table schemas. Does not require a real ClickHouse connection. -func NewTestSchemaRegistry(tables ...*discovery.TableSchema) *discovery.SchemaRegistry { - reg := discovery.NewSchemaRegistryFromMap(tables) +// table schemas, without a real ClickHouse: a mock connection serves the +// schemas as system.columns rows (UTC as the server zone) and the registry is +// built by the real discovery path — NewSchemaRegistry + Refresh — so +// timestamp column specs are precomputed exactly as in production. +// +// The registry holds schemas rebuilt from those rows (Name, Type, HasDefault; +// IsNullable derived from the type string), not the caller's structs. +func NewTestSchemaRegistry(t testing.TB, tables []*discovery.TableSchema) *discovery.SchemaRegistry { + t.Helper() + reg := discovery.NewSchemaRegistry(&schemaConn{tables: tables}, "test", time.Hour, NopLogger()) + require.NoError(t, reg.Refresh(context.Background())) return reg } +// schemaConn is a mock driver.Conn serving exactly the two queries Refresh +// issues: the SELECT timezone() probe (always "UTC") and the system.columns +// scan (rows synthesized from tables). The nil embedded interface panics on +// any other method — nothing else is part of Refresh's contract. +type schemaConn struct { + driver.Conn + tables []*discovery.TableSchema +} + +func (c *schemaConn) QueryRow(context.Context, string, ...any) driver.Row { + return utcRow{} +} + +func (c *schemaConn) Query(context.Context, string, ...any) (driver.Rows, error) { + r := &columnRows{} + for _, t := range c.tables { + for _, col := range t.Columns { + kind := "" + if col.HasDefault { + kind = "DEFAULT" + } + r.rows = append(r.rows, [4]string{t.Name, col.Name, col.Type, kind}) + } + } + return r, nil +} + +type utcRow struct{ driver.Row } + +func (utcRow) Scan(dest ...any) error { + if len(dest) == 1 { + if s, ok := dest[0].(*string); ok { + *s = "UTC" + return nil + } + } + return fmt.Errorf("unexpected timezone scan into %d dest(s)", len(dest)) +} + +// columnRows plays a system.columns result set: one [table, name, type, +// default_kind] row per column, in the given order. +type columnRows struct { + driver.Rows + rows [][4]string + next int +} + +func (r *columnRows) Next() bool { + r.next++ + return r.next <= len(r.rows) +} + +func (r *columnRows) Scan(dest ...any) error { + row := r.rows[r.next-1] + if len(dest) != len(row) { + return fmt.Errorf("expected %d scan dests, got %d", len(row), len(dest)) + } + for i, d := range dest { + s, ok := d.(*string) + if !ok { + return fmt.Errorf("dest %d: expected *string, got %T", i, d) + } + *s = row[i] + } + return nil +} + +func (*columnRows) Close() error { return nil } +func (*columnRows) Err() error { return nil } + // AssertJSONResponse checks that rec has the expected status code and that // the response body, decoded as JSON, matches expected (compared as Go values). func AssertJSONResponse(t *testing.T, rec *httptest.ResponseRecorder, expectedStatus int, expected any) { From e72574f8d1d7a9561ab9f8a826fa33bd77e9e178 Mon Sep 17 00:00:00 2001 From: taitelee Date: Sat, 11 Jul 2026 16:22:46 -0400 Subject: [PATCH 08/15] fix(ingest): parse digit-strings as Unix seconds only in ClickHouse's 9-10 digit shape; reject ''/Local zones; add differential CH oracle test --- CHANGELOG.md | 2 +- docs/src/content/docs/api.md | 2 +- internal/discovery/timestamp.go | 50 +++++- internal/discovery/timestamp_test.go | 16 ++ .../timestamp_canonicalization_test.go | 146 ++++++++++++++++++ 5 files changed, 207 insertions(+), 9 deletions(-) create mode 100644 tests/integration/timestamp_canonicalization_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 8649e664..4957e77e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,7 +50,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed -- **SSE row `DateTime` columns now arrive in canonical RFC 3339 UTC, matching `/v1/query`, instead of the producer's spelling** (`internal/discovery/timestamp.go` (new), `internal/discovery/{discovery.go,validation.go}`, `internal/api/ingest.go`, `internal/ingest/worker.go`, `docs/src/content/docs/{api.md,architecture.md}`, plus tests in `internal/discovery/{timestamp_test.go (new),discovery_test.go}`, `internal/api/{ingest_test.go,boot_chain_test.go}`, `tests/e2e/sdk/streaming.test.ts`): closes #372. The stream fans out the pre-insert payload verbatim, so a row `DateTime` reached SSE subscribers in whatever spelling the producer sent — typically the zone-less ClickHouse-native form — while `/v1/query` rendered the stored value as RFC 3339 `Z`; JavaScript's `Date.parse` reads a zone-less string as **local** time, so an SDK `liveQuery` stitching backfill + live landed the two paths hours apart (sign-flipping with the viewer's UTC offset — broke ordering and the "N min ago" label in the Stats live feed). Ingest now **canonicalizes** every `DateTime`/`DateTime64` column value to RFC 3339 UTC (fraction truncated to the column's precision, `time.RFC3339Nano`-trimmed — byte-matching `/v1/query`'s `time.Time` marshaling) after validation and before the NATS publish, so the one payload every consumer shares — SSE subscribers, the ClickHouse insert, the DLQ — carries a single unambiguous spelling. Inputs stay liberal (RFC 3339 with any offset, `YYYY-MM-DD[ T]HH:MM:SS[.fff]`, `YYYY-MM-DD`, Unix seconds as a number or digit-string); a zone-less string is interpreted in the column's declared time zone, else the server's (discovered once per schema refresh via `SELECT timezone()`) — the same rule ClickHouse itself applies, so canonicalization never changes which instant is stored (verified against a live ClickHouse). Each timestamp column's spec (precision + resolved zone) is precomputed at schema discovery and cached on the column, so the per-record ingest path parses no type strings, loads no zones, and takes no locks. Canonicalization is **fail-open** (#402 review): a value outside the accepted forms publishes verbatim — ClickHouse's more liberal `best_effort` parser stays the arbiter of insertability, and a value it too rejects surfaces via the DLQ as before — and an unresolvable zone (no tzdata embedded; the distroless image ships none) warns and passes through, never a failed refresh or a silent UTC reinterpretation that would move instants (`Etc/UTC` is special-cased to UTC); ingest never rejects a record over its timestamp spelling — fail-closed enforcement of the canonical form is the stream row-filter's (#381). The worker's `INSERT … FORMAT JSONEachRow` sets `date_time_input_format=best_effort`, because ClickHouse's default `basic` parser rejects the canonical form's zone suffix (verified live; `best_effort` is a superset, so pre-upgrade messages still in NATS — zone-less strings, Unix numbers — parse exactly as before). Boundaries: `Date`/`Date32` columns pass through untouched (day precision, no zone ambiguity on this path); events published before the upgrade replay in their original spelling until the sweeper retires them; a column with an explicit non-UTC zone renders as `Z` on both paths (`/v1/query` likewise normalizes `DateTime` values to UTC), so the declared zone affects how zone-less inputs are read but never the output spelling — byte-identical regardless of zone. +- **SSE row `DateTime` columns now arrive in canonical RFC 3339 UTC, matching `/v1/query`, instead of the producer's spelling** (`internal/discovery/timestamp.go` (new), `internal/discovery/{discovery.go,validation.go}`, `internal/api/ingest.go`, `internal/ingest/worker.go`, `docs/src/content/docs/{api.md,architecture.md}`, plus tests in `internal/discovery/{timestamp_test.go (new),discovery_test.go}`, `internal/api/{ingest_test.go,boot_chain_test.go}`, `tests/e2e/sdk/streaming.test.ts`): closes #372. The stream fans out the pre-insert payload verbatim, so a row `DateTime` reached SSE subscribers in whatever spelling the producer sent — typically the zone-less ClickHouse-native form — while `/v1/query` rendered the stored value as RFC 3339 `Z`; JavaScript's `Date.parse` reads a zone-less string as **local** time, so an SDK `liveQuery` stitching backfill + live landed the two paths hours apart (sign-flipping with the viewer's UTC offset — broke ordering and the "N min ago" label in the Stats live feed). Ingest now **canonicalizes** every `DateTime`/`DateTime64` column value to RFC 3339 UTC (fraction truncated to the column's precision, `time.RFC3339Nano`-trimmed — byte-matching `/v1/query`'s `time.Time` marshaling) after validation and before the NATS publish, so the one payload every consumer shares — SSE subscribers, the ClickHouse insert, the DLQ — carries a single unambiguous spelling. Inputs stay liberal (RFC 3339 with any offset, `YYYY-MM-DD[ T]HH:MM:SS[.fff]`, `YYYY-MM-DD`, Unix seconds as a number or as a string of 9–10 digits — the parser grammar is a provable, instant-identical subset of ClickHouse `best_effort`: other digit lengths are calendar forms there (`YYYYMMDD`, `YYYYMMDDhhmmss`) so they pass through rather than parse as Unix seconds, and Go `LoadLocation`'s `''`/`'Local'` environment quirks are rejected as unresolvable zones — #402 review); a zone-less string is interpreted in the column's declared time zone, else the server's (discovered once per schema refresh via `SELECT timezone()`) — the same rule ClickHouse itself applies, so canonicalization never changes which instant is stored (verified against a live ClickHouse). Each timestamp column's spec (precision + resolved zone) is precomputed at schema discovery and cached on the column, so the per-record ingest path parses no type strings, loads no zones, and takes no locks. Canonicalization is **fail-open** (#402 review): a value outside the accepted forms publishes verbatim — ClickHouse's more liberal `best_effort` parser stays the arbiter of insertability, and a value it too rejects surfaces via the DLQ as before — and an unresolvable zone (no tzdata embedded; the distroless image ships none) warns and passes through, never a failed refresh or a silent UTC reinterpretation that would move instants (`Etc/UTC` is special-cased to UTC); ingest never rejects a record over its timestamp spelling — fail-closed enforcement of the canonical form is the stream row-filter's (#381). The worker's `INSERT … FORMAT JSONEachRow` sets `date_time_input_format=best_effort`, because ClickHouse's default `basic` parser rejects the canonical form's zone suffix (verified live; `best_effort` is a superset, so pre-upgrade messages still in NATS — zone-less strings, Unix numbers — parse exactly as before). Boundaries: `Date`/`Date32` columns pass through untouched (day precision, no zone ambiguity on this path); events published before the upgrade replay in their original spelling until the sweeper retires them; a column with an explicit non-UTC zone renders as `Z` on both paths (`/v1/query` likewise normalizes `DateTime` values to UTC), so the declared zone affects how zone-less inputs are read but never the output spelling — byte-identical regardless of zone. - **Live demo hero feed renders in `event_ts` order instead of SSE arrival order** (`docs/src/components/LiveDemo.astro`): the landing-page live activity feed prepended each streamed row to the top in the order the SSE stream delivered it, but a producer's webhook burst (a single merge-queue cycle fires ~20 events) arrives in no guaranteed order and the stream relays it in ingest order — so a late or out-of-order delivery landed above newer rows (e.g. a `pushed 12m ago` sitting on top of `reviewed a pull request 9m ago`). `addRow` now keeps the feed sorted by `event_ts` descending — it slots each row in before the first strictly-older sibling rather than blind-prepending — so the live tail matches the already-sorted `gh_activity_recent` backfill. The zone-less-SSE-timestamp normalization the sort relies on (`normTs`) was already in place; equal-second rows keep arrival order (`gh_events.event_ts` is only second-granular for CI/checks, so there's no finer tiebreak), and dedup + the `MAX_ROWS` trim are unchanged. Surfaced in dogfooding on `wavehouse.dev`; the client-side analog of the ingest-order reality the SSE stream can't reorder. diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index 37cb55ad..7a6f4292 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -217,7 +217,7 @@ The body is a **flat JSON object** whose keys must match column names in the tar - Type compatibility: `String`/`DateTime`/`UUID`/`Enum`/`IPv*` accept JSON strings; `Int*`/`Float*`/`Decimal` accept JSON numbers; `Bool` accepts JSON booleans or numbers; `Array` accepts JSON arrays; `Map`/`Tuple` accept JSON objects. - `Nullable()` and `LowCardinality()` wrappers are handled transparently. -**Timestamp canonicalization:** a `DateTime`/`DateTime64` column value sent as RFC 3339 (any offset), `YYYY-MM-DD[ T]HH:MM:SS[.fff]` or `YYYY-MM-DD` (zone-less — interpreted in the column's time zone, else the ClickHouse server's, exactly as ClickHouse itself would), or Unix seconds (a JSON number, or the same digits as a string) is rewritten to **one canonical wire form — RFC 3339 UTC** (`2026-06-21T04:00:00Z`, fraction truncated to the column's precision) before publishing, so the stored instant never changes but every consumer (the ClickHouse insert, [SSE subscribers](#get-v1stream--server-sent-events-stream), the DLQ) sees the same spelling `/v1/query` renders. **Fail-open**: a value in none of those forms is published verbatim — ClickHouse's more liberal parser decides insertability, and a value it too rejects surfaces via the DLQ, as before. `Date`/`Date32` columns pass through untouched. +**Timestamp canonicalization:** a `DateTime`/`DateTime64` column value sent as RFC 3339 (any offset), `YYYY-MM-DD[ T]HH:MM:SS[.fff]` or `YYYY-MM-DD` (zone-less — interpreted in the column's time zone, else the ClickHouse server's, exactly as ClickHouse itself would), or Unix seconds (a JSON number, or a string of exactly 9–10 digits — other digit lengths are ClickHouse calendar forms like `YYYYMMDD`, so they pass through untouched) is rewritten to **one canonical wire form — RFC 3339 UTC** (`2026-06-21T04:00:00Z`, fraction truncated to the column's precision) before publishing, so the stored instant never changes but every consumer (the ClickHouse insert, [SSE subscribers](#get-v1stream--server-sent-events-stream), the DLQ) sees the same spelling `/v1/query` renders. **Fail-open**: a value in none of those forms is published verbatim — ClickHouse's more liberal parser decides insertability, and a value it too rejects surfaces via the DLQ, as before. `Date`/`Date32` columns pass through untouched. **Response (accepted):** diff --git a/internal/discovery/timestamp.go b/internal/discovery/timestamp.go index 59378a93..23fe3d31 100644 --- a/internal/discovery/timestamp.go +++ b/internal/discovery/timestamp.go @@ -124,8 +124,15 @@ func resolveTimestampSpec(chType string, serverTZ *time.Location) (timestampSpec } func loadLocation(name string) (*time.Location, error) { - if name == "Etc/UTC" { + switch name { + case "Etc/UTC": return time.UTC, nil + case "", "Local": + // time.LoadLocation reads "" as UTC and "Local" from the process + // environment. Neither is a zone ClickHouse would declare, and an + // environment-dependent zone is the opposite of canonical — treat + // both as unresolvable (nil spec, pass-through) instead of guessing. + return nil, fmt.Errorf("not an IANA zone name: %q", name) } return time.LoadLocation(name) } @@ -133,7 +140,10 @@ func loadLocation(name string) (*time.Location, error) { // parseTimestamp converts one ingested value into a time.Time. Accepted forms: // RFC 3339, `YYYY-MM-DD[ T]HH:MM:SS[.fraction]` and `YYYY-MM-DD` (zone-less, // interpreted in loc; skipped when loc is nil), and Unix seconds as a number or -// numeric string. Anything else errors — the caller passes it through. +// as a string of 9–10 digits. Anything else errors — the caller passes it +// through. The grammar is deliberately a subset of ClickHouse best_effort's, +// instant-identical on that subset: when unsure what ClickHouse would read, +// this parser must fail, never guess. func parseTimestamp(v any, loc *time.Location) (time.Time, error) { switch x := v.(type) { case string: @@ -149,13 +159,19 @@ func parseTimestamp(v any, loc *time.Location) (time.Time, error) { } } } - // A bare digit-string is Unix seconds, as ClickHouse itself reads it - // (non-finite values are not an instant). - if f, err := strconv.ParseFloat(x, 64); err == nil && !math.IsNaN(f) && !math.IsInf(f, 0) { - return unixToTime(f), nil + // A digit-string is Unix seconds only in the one shape ClickHouse's + // best_effort parser reads that way: 9–10 integer digits, optional + // all-digit fraction. Other digit runs are calendar forms there + // (8 ⇒ YYYYMMDD, 14 ⇒ YYYYMMDDhhmmss, 4 ⇒ YYYY), and signs/exponents + // aren't timestamps to it at all — those pass through, so this parser + // can never read a different instant than the insert will store. + if isUnixSecondsString(x) { + if f, err := strconv.ParseFloat(x, 64); err == nil { + return unixToTime(f), nil + } } return time.Time{}, fmt.Errorf( - "unrecognized timestamp %.64q (accepted: RFC 3339, 'YYYY-MM-DD[ T]HH:MM:SS[.fff]', 'YYYY-MM-DD', or Unix seconds)", x) + "unrecognized timestamp %.64q (accepted: RFC 3339, 'YYYY-MM-DD[ T]HH:MM:SS[.fff]', 'YYYY-MM-DD', or 9-10 digit Unix seconds)", x) case float64: return unixToTime(x), nil case json.Number: @@ -169,6 +185,26 @@ func parseTimestamp(v any, loc *time.Location) (time.Time, error) { } } +// isUnixSecondsString reports whether s is Unix seconds in the only string +// shape ClickHouse's best_effort parser reads as such: 9 or 10 ASCII digits, +// optionally followed by a non-empty all-digit fraction. +func isUnixSecondsString(s string) bool { + intPart, frac, hasFrac := strings.Cut(s, ".") + if len(intPart) < 9 || len(intPart) > 10 || !isDigits(intPart) { + return false + } + return !hasFrac || (frac != "" && isDigits(frac)) +} + +func isDigits(s string) bool { + for i := 0; i < len(s); i++ { + if s[i] < '0' || s[i] > '9' { + return false + } + } + return true +} + // unixToTime converts fractional Unix seconds to a time.Time (nanoseconds rounded; // float64 loses sub-microsecond precision near current epochs — producers that need // exact sub-second values send the string form). diff --git a/internal/discovery/timestamp_test.go b/internal/discovery/timestamp_test.go index e297dcf1..f6038605 100644 --- a/internal/discovery/timestamp_test.go +++ b/internal/discovery/timestamp_test.go @@ -69,6 +69,7 @@ func TestCanonicalizeTimestamps(t *testing.T) { {"unix seconds number", "DateTime('UTC')", nil, float64(1782014400), "2026-06-21T04:00:00Z"}, {"unix seconds json.Number", "DateTime('UTC')", nil, json.Number("1782014400"), "2026-06-21T04:00:00Z"}, {"unix seconds digit-string", "DateTime('UTC')", nil, "1782014400", "2026-06-21T04:00:00Z"}, + {"9-digit unix string", "DateTime('UTC')", nil, "999999999", "2001-09-09T01:46:39Z"}, {"fractional unix digit-string", "DateTime64(3, 'UTC')", nil, "1782014400.5", "2026-06-21T04:00:00.5Z"}, {"fractional unix on DateTime64", "DateTime64(3, 'UTC')", nil, 1782014400.5, "2026-06-21T04:00:00.5Z"}, {"fraction truncated to column precision", "DateTime64(3, 'UTC')", nil, "2026-06-21T04:00:00.123456Z", "2026-06-21T04:00:00.123Z"}, @@ -131,6 +132,21 @@ func TestCanonicalizeTimestamps_Unparseable_PassThrough(t *testing.T) { {"wrong value type", "DateTime('UTC')", true}, {"unknown zone in type", "DateTime('Not/AZone')", "2026-06-21 04:00:00"}, {"malformed DateTime64 precision", "DateTime64(x)", "2026-06-21 04:00:00"}, + // Digit-strings outside the 9–10 digit Unix shape mean calendar forms + // (or nothing) to ClickHouse best_effort — parsing them as Unix seconds + // would store a different instant than the insert (PR #402 review). + {"8-digit string is YYYYMMDD to ClickHouse", "DateTime('UTC')", "20260711"}, + {"12-digit string (ClickHouse rejects)", "DateTime('UTC')", "202607111500"}, + {"14-digit string is YYYYMMDDhhmmss to ClickHouse", "DateTime('UTC')", "20260711150000"}, + {"4-digit string is a year to ClickHouse", "DateTime('UTC')", "2026"}, + {"11-digit string", "DateTime('UTC')", "17504784000"}, + {"scientific notation is not a timestamp", "DateTime('UTC')", "1e9"}, + {"negative digit-string", "DateTime('UTC')", "-100"}, + {"empty fraction", "DateTime('UTC')", "1750478400."}, + // "" and "Local" are Go LoadLocation quirks (UTC / process env), not + // zone declarations — strict: unresolvable, pass through. + {"empty zone name in type", "DateTime('')", "2026-06-21 04:00:00"}, + {"Local zone in type", "DateTime('Local')", "2026-06-21 04:00:00"}, } for _, tt := range tests { diff --git a/tests/integration/timestamp_canonicalization_test.go b/tests/integration/timestamp_canonicalization_test.go new file mode 100644 index 00000000..eb878ca4 --- /dev/null +++ b/tests/integration/timestamp_canonicalization_test.go @@ -0,0 +1,146 @@ +//go:build integration + +package tests + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "net/url" + "testing" + "time" + + "github.com/stretchr/testify/require" + + "github.com/Wave-RF/WaveHouse/internal/discovery" +) + +// TestTimestampCanonicalization_DifferentialAgainstClickHouse pins the #372 +// invariant with ClickHouse itself as the oracle (PR #402 review): for every +// corpus input × timestamp column shape, inserting the raw producer value and +// inserting CanonicalizeTimestamps' output must (a) both succeed or both +// fail, and (b) when both succeed, store the same instant. A same-instant +// failure means canonicalization rewrote a value ClickHouse reads differently +// (data corruption); an asymmetric accept means canonicalization changed +// which values are insertable (ClickHouse, not the canonicalizer, is the +// arbiter of insertability). +// +// Inserts go through the same HTTP surface the ingest worker uses — +// JSONEachRow with date_time_input_format=best_effort — so the oracle is the +// production parse, not a lookalike. +func TestTimestampCanonicalization_DifferentialAgainstClickHouse(t *testing.T) { + colTypes := []struct { + name string + ddl string + }{ + {"datetime", "DateTime"}, + {"datetime_nyc", "DateTime('America/New_York')"}, + {"datetime64_3", "DateTime64(3)"}, + {"datetime64_6_tokyo", "DateTime64(6, 'Asia/Tokyo')"}, + } + + // One entry per spelling family. The calendar-shaped digit runs and the + // garbage entries document the pass-through side of the contract: they + // must reach ClickHouse verbatim and get its verdict, never a rewrite. + corpus := []any{ + "2026-06-21T04:00:00Z", // canonical already + "2026-06-21T06:30:00+02:30", // offset form + "2026-06-21 04:00:00", // zone-less space form + "2026-06-21T04:00:00", // zone-less T form + "2026-06-21", // date-only + "2026-06-21 04:00:00.123456", // zone-less with fraction + "2026-06-21T04:00:00.9999Z", // fraction beyond column precision + "1750478400", // 10-digit Unix string + "999999999", // 9-digit Unix string + "1750478400.5", // fractional Unix string + float64(1750478400), // Unix number + "20260711", // 8 digits: YYYYMMDD to ClickHouse + "20260711150000", // 14 digits: YYYYMMDDhhmmss + "202607111500", // 12 digits: ClickHouse rejects + "2026", // 4 digits: a year to ClickHouse + "1e9", // not a timestamp + "-100", // not a timestamp + "banana", // garbage + "2026-11-01 01:30:00", // DST fall-back: ambiguous local time + "2026-03-08 02:30:00", // DST spring-forward: nonexistent local time + } + + for _, ct := range colTypes { + t.Run(ct.name, func(t *testing.T) { + rawTable := createTable(t, "id UInt32, v "+ct.ddl, "ORDER BY id") + canonTable := createTable(t, "id UInt32, v "+ct.ddl, "ORDER BY id") + schema := env(t).registry.Get(rawTable) + require.NotNil(t, schema, "registry must discover the raw table") + + for i, input := range corpus { + id := uint32(i) + rawErr := insertBestEffort(t, rawTable, map[string]any{"id": id, "v": input}) + + canonData := map[string]any{"id": id, "v": input} + discovery.CanonicalizeTimestamps(schema, canonData) + canonErr := insertBestEffort(t, canonTable, canonData) + + if (rawErr == nil) != (canonErr == nil) { + t.Errorf("input %v: asymmetric insertability — raw err=%v, canonicalized (%v) err=%v", + input, rawErr, canonData["v"], canonErr) + continue + } + if rawErr != nil { + continue // both rejected — consistent + } + rawStored := selectInstant(t, rawTable, id) + canonStored := selectInstant(t, canonTable, id) + if !rawStored.Equal(canonStored) { + t.Errorf("input %v: stored instants differ — raw %s vs canonicalized (%v) %s", + input, rawStored.UTC().Format(time.RFC3339Nano), + canonData["v"], canonStored.UTC().Format(time.RFC3339Nano)) + } + } + }) + } +} + +// insertBestEffort inserts one JSONEachRow row over ClickHouse HTTP with +// date_time_input_format=best_effort — the exact settings the ingest worker +// uses (see IngestWorker.insertToClickHouse) — and returns ClickHouse's +// verdict as an error (nil on 2xx). +func insertBestEffort(t *testing.T, table string, row map[string]any) error { + t.Helper() + body, err := json.Marshal(row) + require.NoError(t, err) + + q := url.Values{} + q.Set("database", testCHDatabase) + q.Set("param_target_table", table) + q.Set("query", "INSERT INTO {target_table:Identifier} FORMAT JSONEachRow") + q.Set("date_time_input_format", "best_effort") + + req, err := http.NewRequestWithContext(context.Background(), http.MethodPost, + env(t).chHTTPURL+"?"+q.Encode(), bytes.NewReader(body)) + require.NoError(t, err) + req.Header.Set("Content-Type", "application/json") + req.Header.Set("X-ClickHouse-User", testCHUser) + req.Header.Set("X-ClickHouse-Key", testCHPassword) + + resp, err := http.DefaultClient.Do(req) + require.NoError(t, err) + defer func() { _ = resp.Body.Close() }() + if resp.StatusCode >= 300 { + msg, _ := io.ReadAll(io.LimitReader(resp.Body, 512)) + return fmt.Errorf("HTTP %d: %s", resp.StatusCode, msg) + } + _, _ = io.Copy(io.Discard, resp.Body) + return nil +} + +// selectInstant reads back the single stored timestamp as a time.Time instant. +func selectInstant(t *testing.T, table string, id uint32) time.Time { + t.Helper() + var v time.Time + require.NoError(t, env(t).chConn.QueryRow(context.Background(), + fmt.Sprintf("SELECT v FROM %s WHERE id = ?", table), id).Scan(&v)) + return v +} From c288a1582354b2f25abcfe87d553cc4c7d17f36e Mon Sep 17 00:00:00 2001 From: taitelee Date: Sat, 11 Jul 2026 16:51:44 -0400 Subject: [PATCH 09/15] docs: specify the canonical timestamp form precisely (truncation, zero-trimming, Z-only) --- AGENTS.md | 2 +- docs/src/content/docs/api.md | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index e133e3ff..bd36c32c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -65,7 +65,7 @@ The invariant index — what must stay true. Full narrative and rationale live i 16. **Bearer-token-only CORS posture (security)** — Bearer JWT on every request, no cookies/sessions; `corsMiddleware` deliberately **never** emits `Access-Control-Allow-Credentials` (not needed, and `*` + credentials is a spec violation browsers reject). `cors_allowed_origins` controls who can *read* responses, not cookie scope; CSRF protection is structural. Don't reintroduce cookie auth or `Allow-Credentials` without a design discussion — answers GitHub #29/#30. Code: `internal/api/router.go`. 17. **Non-fatal boot** — schema-discovery failure on boot is non-fatal: `cmd/wavehouse` records an `api.BootState`, binds `:8080`, serves 503 on `/livez`/`/readyz` with the diagnostic, and retries via `SchemaRegistry.RetryRefresh` (backoff 2s → 60s). Bounds supervisor restart loops. 18. **Health endpoints** — liveness `/livez`, readiness `/readyz` (k8s convention); `/healthz` is a permanent alias of `/livez`; `/health` + `/ready` are deprecated (removal v0.2.0, CHANGELOG #144). `/v1/health` is the SDK's content-free public ping (no ClickHouse check), a `/v1` route so it survives reverse-proxy probe-path filtering. Point k8s at `/livez`/`/readyz`, SDK/online-checks at `/v1/health`, never the deprecated aliases. -19. **Canonical timestamp wire form (fail-open at ingest)** — the HTTP ingest handler rewrites every `DateTime`/`DateTime64` value it can parse to RFC 3339 UTC (`discovery.CanonicalizeTimestamps`; per-column precision + zone precomputed at schema refresh) after validation + policy checks and **before** the NATS publish, so the one payload every consumer shares — SSE subscribers, the ClickHouse insert, the DLQ — carries the same spelling `/v1/query` renders: live and query reads can't drift (#372). Zone-less inputs are read in the column's declared zone, else the discovered server default — ClickHouse's own rule, so the spelling changes but never the instant. Deliberately **fail-open**: an unparseable value or unresolvable zone (no tzdata embedded — never a failed refresh, never a silent UTC reinterpretation, which would move instants) publishes verbatim; ingest must not reject a record over its timestamp spelling — fail-closed enforcement belongs to the stream row-filter (#381). Don't re-spell timestamps downstream. Preserve when touching `internal/discovery`, the ingest handler, or the SSE fan-out. Detail: architecture.md § `discovery/` + §Ingest Path. +19. **Canonical timestamp wire form (fail-open at ingest)** — the HTTP ingest handler rewrites every `DateTime`/`DateTime64` value it can parse to RFC 3339 UTC (`discovery.CanonicalizeTimestamps`; per-column precision + zone precomputed at schema refresh) after validation + policy checks and **before** the NATS publish, so the one payload every consumer shares — SSE subscribers, the ClickHouse insert, the DLQ — carries the same spelling `/v1/query` renders: live and query reads can't drift (#372). Zone-less inputs are read in the column's declared zone, else the discovered server default — ClickHouse's own rule, so the spelling changes but never the instant. Deliberately **fail-open**: an unparseable value or unresolvable zone (no tzdata embedded — never a failed refresh, never a silent UTC reinterpretation, which would move instants) publishes verbatim; ingest must not reject a record over its timestamp spelling — fail-closed enforcement belongs to the stream row-filter (#381). Don't re-spell timestamps downstream. Preserve when touching `internal/discovery`, the ingest handler, or the SSE fan-out. Detail: architecture.md § `discovery/` + §Ingest Path; the exact spelling spec (truncation, zero-trimming, `Z`-only) lives in api.md §Timestamp canonicalization — keep it in sync with `canonicalTimestamp`. ## Code Conventions diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index 7a6f4292..02667358 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -219,6 +219,15 @@ The body is a **flat JSON object** whose keys must match column names in the tar **Timestamp canonicalization:** a `DateTime`/`DateTime64` column value sent as RFC 3339 (any offset), `YYYY-MM-DD[ T]HH:MM:SS[.fff]` or `YYYY-MM-DD` (zone-less — interpreted in the column's time zone, else the ClickHouse server's, exactly as ClickHouse itself would), or Unix seconds (a JSON number, or a string of exactly 9–10 digits — other digit lengths are ClickHouse calendar forms like `YYYYMMDD`, so they pass through untouched) is rewritten to **one canonical wire form — RFC 3339 UTC** (`2026-06-21T04:00:00Z`, fraction truncated to the column's precision) before publishing, so the stored instant never changes but every consumer (the ClickHouse insert, [SSE subscribers](#get-v1stream--server-sent-events-stream), the DLQ) sees the same spelling `/v1/query` renders. **Fail-open**: a value in none of those forms is published verbatim — ClickHouse's more liberal parser decides insertability, and a value it too rejects surfaces via the DLQ, as before. `Date`/`Date32` columns pass through untouched. +**The canonical form, precisely.** This is the one strict timestamp spelling in WaveHouse — the same one `/v1/query` renders and the SSE stream carries, and the form the stream row-filter will require for timestamp comparisons once row-level enforcement lands ([#381](https://github.com/Wave-RF/WaveHouse/issues/381)): + +- `YYYY-MM-DDTHH:MM:SSZ`, or `YYYY-MM-DDTHH:MM:SS.FZ` when there is a sub-second part: uppercase `T` separator, uppercase `Z` suffix, always UTC — never a numeric offset — and seconds always present. +- The fraction is **truncated** (never rounded) to the column's precision: a `DateTime` column (whole seconds) never carries a fraction; a `DateTime64(3)` column carries at most three digits. +- Trailing fractional zeros are trimmed and an all-zero fraction is dropped (Go's `time.RFC3339Nano` rendering): `.120` becomes `.12Z`, `.000` becomes plain `Z` — byte-for-byte what `/v1/query` returns for the same stored value. +- A column's declared time zone changes only how zone-less *inputs* are interpreted, never the output: every canonical value ends in `Z`. + +Examples for a `DateTime64(3, 'America/New_York')` column: `"2026-06-21 00:00:00.1239"` (zone-less, read in New York) → `"2026-06-21T04:00:00.123Z"`; `1750478400.5` (Unix seconds) → `"2025-06-21T04:00:00.5Z"`. + **Response (accepted):** ```json From 3b9d79a351b0961ef18b077fcf09e76152997edc Mon Sep 17 00:00:00 2001 From: taitelee Date: Mon, 13 Jul 2026 12:46:08 -0400 Subject: [PATCH 10/15] fix(ingest): mirror ClickHouse per-kind timestamp number/fraction rules --- CHANGELOG.md | 2 +- docs/src/content/docs/api.md | 4 +- docs/src/content/docs/architecture.md | 8 +- docs/src/content/docs/ingest-pipeline.md | 10 +- internal/api/ingest.go | 7 +- internal/api/ingest_test.go | 2 +- internal/discovery/discovery_test.go | 8 +- internal/discovery/timestamp.go | 201 ++++++++++++------ internal/discovery/timestamp_test.go | 36 +++- internal/ingest/worker.go | 3 - internal/ingest/worker_test.go | 5 +- .../timestamp_canonicalization_test.go | 41 ++-- 12 files changed, 218 insertions(+), 109 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4957e77e..c27ea589 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,7 +50,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed -- **SSE row `DateTime` columns now arrive in canonical RFC 3339 UTC, matching `/v1/query`, instead of the producer's spelling** (`internal/discovery/timestamp.go` (new), `internal/discovery/{discovery.go,validation.go}`, `internal/api/ingest.go`, `internal/ingest/worker.go`, `docs/src/content/docs/{api.md,architecture.md}`, plus tests in `internal/discovery/{timestamp_test.go (new),discovery_test.go}`, `internal/api/{ingest_test.go,boot_chain_test.go}`, `tests/e2e/sdk/streaming.test.ts`): closes #372. The stream fans out the pre-insert payload verbatim, so a row `DateTime` reached SSE subscribers in whatever spelling the producer sent — typically the zone-less ClickHouse-native form — while `/v1/query` rendered the stored value as RFC 3339 `Z`; JavaScript's `Date.parse` reads a zone-less string as **local** time, so an SDK `liveQuery` stitching backfill + live landed the two paths hours apart (sign-flipping with the viewer's UTC offset — broke ordering and the "N min ago" label in the Stats live feed). Ingest now **canonicalizes** every `DateTime`/`DateTime64` column value to RFC 3339 UTC (fraction truncated to the column's precision, `time.RFC3339Nano`-trimmed — byte-matching `/v1/query`'s `time.Time` marshaling) after validation and before the NATS publish, so the one payload every consumer shares — SSE subscribers, the ClickHouse insert, the DLQ — carries a single unambiguous spelling. Inputs stay liberal (RFC 3339 with any offset, `YYYY-MM-DD[ T]HH:MM:SS[.fff]`, `YYYY-MM-DD`, Unix seconds as a number or as a string of 9–10 digits — the parser grammar is a provable, instant-identical subset of ClickHouse `best_effort`: other digit lengths are calendar forms there (`YYYYMMDD`, `YYYYMMDDhhmmss`) so they pass through rather than parse as Unix seconds, and Go `LoadLocation`'s `''`/`'Local'` environment quirks are rejected as unresolvable zones — #402 review); a zone-less string is interpreted in the column's declared time zone, else the server's (discovered once per schema refresh via `SELECT timezone()`) — the same rule ClickHouse itself applies, so canonicalization never changes which instant is stored (verified against a live ClickHouse). Each timestamp column's spec (precision + resolved zone) is precomputed at schema discovery and cached on the column, so the per-record ingest path parses no type strings, loads no zones, and takes no locks. Canonicalization is **fail-open** (#402 review): a value outside the accepted forms publishes verbatim — ClickHouse's more liberal `best_effort` parser stays the arbiter of insertability, and a value it too rejects surfaces via the DLQ as before — and an unresolvable zone (no tzdata embedded; the distroless image ships none) warns and passes through, never a failed refresh or a silent UTC reinterpretation that would move instants (`Etc/UTC` is special-cased to UTC); ingest never rejects a record over its timestamp spelling — fail-closed enforcement of the canonical form is the stream row-filter's (#381). The worker's `INSERT … FORMAT JSONEachRow` sets `date_time_input_format=best_effort`, because ClickHouse's default `basic` parser rejects the canonical form's zone suffix (verified live; `best_effort` is a superset, so pre-upgrade messages still in NATS — zone-less strings, Unix numbers — parse exactly as before). Boundaries: `Date`/`Date32` columns pass through untouched (day precision, no zone ambiguity on this path); events published before the upgrade replay in their original spelling until the sweeper retires them; a column with an explicit non-UTC zone renders as `Z` on both paths (`/v1/query` likewise normalizes `DateTime` values to UTC), so the declared zone affects how zone-less inputs are read but never the output spelling — byte-identical regardless of zone. +- **SSE row `DateTime` columns now arrive in canonical RFC 3339 UTC, matching `/v1/query`, instead of the producer's spelling** (`internal/discovery/timestamp.go` (new), `internal/discovery/{discovery.go,validation.go}`, `internal/api/ingest.go`, `internal/ingest/worker.go`, `docs/src/content/docs/{api.md,architecture.md}`, plus tests in `internal/discovery/{timestamp_test.go (new),discovery_test.go}`, `internal/api/{ingest_test.go,boot_chain_test.go}`, `tests/e2e/sdk/streaming.test.ts`): closes #372. The stream fans out the pre-insert payload verbatim, so a row `DateTime` reached SSE subscribers in whatever spelling the producer sent — typically the zone-less ClickHouse-native form — while `/v1/query` rendered the stored value as RFC 3339 `Z`; JavaScript's `Date.parse` reads a zone-less string as **local** time, so an SDK `liveQuery` stitching backfill + live landed the two paths hours apart (sign-flipping with the viewer's UTC offset — broke ordering and the "N min ago" label in the Stats live feed). Ingest now **canonicalizes** every `DateTime`/`DateTime64` column value to RFC 3339 UTC (fraction truncated to the column's precision, `time.RFC3339Nano`-trimmed — byte-matching `/v1/query`'s `time.Time` marshaling) after validation and before the NATS publish, so the one payload every consumer shares — SSE subscribers, the ClickHouse insert, the DLQ — carries a single unambiguous spelling. Inputs stay liberal but are mirrored per column kind, exactly as ClickHouse reads them (#402 review): RFC 3339 with any offset (`.`-fractions only — ClickHouse has no `,` separator), `YYYY-MM-DD[ T]HH:MM:SS[.fff]`, `YYYY-MM-DD`, a Unix-seconds string of 9–10 digits (a fraction after it is honored only for `DateTime64` columns, parsed as an exact decimal and truncated at nine digits — never a `float64` round-trip, which corrupts nanoseconds and can round across the second; other digit lengths are ClickHouse's own forms — calendar `YYYYMMDD`/`YYYYMMDDhhmmss` or its 13/16/19-digit ms/µs/ns epochs — and pass through), and **integer** JSON numbers read the way ClickHouse reads them: Unix seconds for `DateTime`, **ticks at the column's scale** for `DateTime64` (the ms epoch `1750478400500` into a `DateTime64(3)` is a valid 2025 instant; an epoch-*seconds* number there is a 1970 instant — rewriting either as seconds would change what ClickHouse stores; non-integer numbers pass through, ClickHouse rejects them). Values whose instant lies outside the column type's range also pass through — ClickHouse *saturates* out-of-range values spelling-dependently (local time-of-day is kept while the date clamps), so only the producer's own spelling may be the one that saturates. Go `LoadLocation`'s `''`/`'Local'` environment quirks are rejected as unresolvable zones. A zone-less string is interpreted in the column's declared time zone, else the server's (discovered once per schema refresh via `SELECT timezone()`) — the same rule ClickHouse itself applies, so canonicalization never changes which instant is stored (differentially fuzzed against a live ClickHouse — ~35k generated inputs × five column shapes, raw vs canonicalized inserts, zero divergences — with every divergence class found along the way pinned in `tests/integration/timestamp_canonicalization_test.go`). Each timestamp column's spec (precision + resolved zone) is precomputed at schema discovery and cached on the column, so the per-record ingest path parses no type strings, loads no zones, and takes no locks. Canonicalization is **fail-open** (#402 review): a value outside the accepted forms publishes verbatim — ClickHouse's more liberal `best_effort` parser stays the arbiter of insertability, and a value it too rejects surfaces via the DLQ as before — and an unresolvable zone (no tzdata embedded; the distroless image ships none) warns and passes through, never a failed refresh or a silent UTC reinterpretation that would move instants (`Etc/UTC` is special-cased to UTC); ingest never rejects a record over its timestamp spelling — fail-closed enforcement of the canonical form is the stream row-filter's (#381). The worker's `INSERT … FORMAT JSONEachRow` pins `date_time_input_format=best_effort` — the server default since ClickHouse 26.5, and on older servers the `basic` default rejects the canonical form's zone suffix (verified live); a superset of `basic` either way, so pre-upgrade messages still in NATS — zone-less strings, Unix numbers — parse exactly as before. Boundaries: `Date`/`Date32` columns pass through untouched (day precision, no zone ambiguity on this path); events published before the upgrade replay in their original spelling until the sweeper retires them; a column with an explicit non-UTC zone renders as `Z` on both paths (`/v1/query` likewise normalizes `DateTime` values to UTC), so the declared zone affects how zone-less inputs are read but never the output spelling — byte-identical regardless of zone. - **Live demo hero feed renders in `event_ts` order instead of SSE arrival order** (`docs/src/components/LiveDemo.astro`): the landing-page live activity feed prepended each streamed row to the top in the order the SSE stream delivered it, but a producer's webhook burst (a single merge-queue cycle fires ~20 events) arrives in no guaranteed order and the stream relays it in ingest order — so a late or out-of-order delivery landed above newer rows (e.g. a `pushed 12m ago` sitting on top of `reviewed a pull request 9m ago`). `addRow` now keeps the feed sorted by `event_ts` descending — it slots each row in before the first strictly-older sibling rather than blind-prepending — so the live tail matches the already-sorted `gh_activity_recent` backfill. The zone-less-SSE-timestamp normalization the sort relies on (`normTs`) was already in place; equal-second rows keep arrival order (`gh_events.event_ts` is only second-granular for CI/checks, so there's no finer tiebreak), and dedup + the `MAX_ROWS` trim are unchanged. Surfaced in dogfooding on `wavehouse.dev`; the client-side analog of the ingest-order reality the SSE stream can't reorder. diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index 02667358..8807adb7 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -217,7 +217,7 @@ The body is a **flat JSON object** whose keys must match column names in the tar - Type compatibility: `String`/`DateTime`/`UUID`/`Enum`/`IPv*` accept JSON strings; `Int*`/`Float*`/`Decimal` accept JSON numbers; `Bool` accepts JSON booleans or numbers; `Array` accepts JSON arrays; `Map`/`Tuple` accept JSON objects. - `Nullable()` and `LowCardinality()` wrappers are handled transparently. -**Timestamp canonicalization:** a `DateTime`/`DateTime64` column value sent as RFC 3339 (any offset), `YYYY-MM-DD[ T]HH:MM:SS[.fff]` or `YYYY-MM-DD` (zone-less — interpreted in the column's time zone, else the ClickHouse server's, exactly as ClickHouse itself would), or Unix seconds (a JSON number, or a string of exactly 9–10 digits — other digit lengths are ClickHouse calendar forms like `YYYYMMDD`, so they pass through untouched) is rewritten to **one canonical wire form — RFC 3339 UTC** (`2026-06-21T04:00:00Z`, fraction truncated to the column's precision) before publishing, so the stored instant never changes but every consumer (the ClickHouse insert, [SSE subscribers](#get-v1stream--server-sent-events-stream), the DLQ) sees the same spelling `/v1/query` renders. **Fail-open**: a value in none of those forms is published verbatim — ClickHouse's more liberal parser decides insertability, and a value it too rejects surfaces via the DLQ, as before. `Date`/`Date32` columns pass through untouched. +**Timestamp canonicalization:** a `DateTime`/`DateTime64` column value sent as RFC 3339 (any offset; `.`-fractions only — ClickHouse has no `,` separator), `YYYY-MM-DD[ T]HH:MM:SS[.fff]` or `YYYY-MM-DD` (zone-less — interpreted in the column's time zone, else the ClickHouse server's, exactly as ClickHouse itself would), a Unix-seconds string of exactly 9–10 digits (a `.fff` fraction is honored only for `DateTime64` columns, as ClickHouse does; other digit lengths are ClickHouse's own forms — calendar shapes like `YYYYMMDD`, or its 13/16/19-digit ms/µs/ns epochs — so they pass through untouched), or an **integer** JSON number — read the way ClickHouse reads numbers: Unix seconds for `DateTime`, **ticks at the column's scale** for `DateTime64` (`1750478400500` into a `DateTime64(3)` is the millisecond epoch; non-integer numbers pass through, ClickHouse rejects them) — is rewritten to **one canonical wire form — RFC 3339 UTC** (`2026-06-21T04:00:00Z`, fraction truncated to the column's precision) before publishing, so the stored instant never changes but every consumer (the ClickHouse insert, [SSE subscribers](#get-v1stream--server-sent-events-stream), the DLQ) sees the same spelling `/v1/query` renders. **Fail-open**: a value in none of those forms — or whose instant lies outside the column type's range, where ClickHouse *saturates* spelling-dependently instead of erroring — is published verbatim: ClickHouse's more liberal parser decides insertability, and a value it too rejects surfaces via the DLQ, as before. The accepted grammar is differentially tested against a live ClickHouse (raw and canonicalized spellings must insert identically, or both fail). `Date`/`Date32` columns pass through untouched. **The canonical form, precisely.** This is the one strict timestamp spelling in WaveHouse — the same one `/v1/query` renders and the SSE stream carries, and the form the stream row-filter will require for timestamp comparisons once row-level enforcement lands ([#381](https://github.com/Wave-RF/WaveHouse/issues/381)): @@ -226,7 +226,7 @@ The body is a **flat JSON object** whose keys must match column names in the tar - Trailing fractional zeros are trimmed and an all-zero fraction is dropped (Go's `time.RFC3339Nano` rendering): `.120` becomes `.12Z`, `.000` becomes plain `Z` — byte-for-byte what `/v1/query` returns for the same stored value. - A column's declared time zone changes only how zone-less *inputs* are interpreted, never the output: every canonical value ends in `Z`. -Examples for a `DateTime64(3, 'America/New_York')` column: `"2026-06-21 00:00:00.1239"` (zone-less, read in New York) → `"2026-06-21T04:00:00.123Z"`; `1750478400.5` (Unix seconds) → `"2025-06-21T04:00:00.5Z"`. +Examples for a `DateTime64(3, 'America/New_York')` column: `"2026-06-21 00:00:00.1239"` (zone-less, read in New York) → `"2026-06-21T04:00:00.123Z"`; `"1750478400.5"` (Unix-seconds string) → `"2025-06-21T04:00:00.5Z"`; the integer number `1750478400500` (ticks at the column's millisecond scale) → `"2025-06-21T04:00:00.5Z"`. **Response (accepted):** diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index f96d2763..91b85f51 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -181,10 +181,10 @@ Ingest worker pipeline (StartIngestWorker): ← JetStream pull consumer (buffer-consumer) on ingest.> → Parse the event envelope (a malformed envelope is the only poison pill: it's acked-and-dropped) → Batch events per table, bulk INSERT to ClickHouse - (INSERTs set date_time_input_format=best_effort: ClickHouse's default - 'basic' parser rejects the canonical RFC 3339 form's zone suffix; best_effort - is a superset, so pre-canonical and fail-open pass-through spellings parse - as before) + (INSERTs pin date_time_input_format=best_effort — the server default since + ClickHouse 26.5; on older servers the 'basic' default rejects the canonical + RFC 3339 form's zone suffix. A superset of basic, so pre-canonical and + fail-open pass-through spellings parse as before) → On success: DoubleAck messages → On failure: route to DLQ output (dlq.{table}), then Ack to prevent infinite retry diff --git a/docs/src/content/docs/ingest-pipeline.md b/docs/src/content/docs/ingest-pipeline.md index 367ba93c..a95d74f1 100644 --- a/docs/src/content/docs/ingest-pipeline.md +++ b/docs/src/content/docs/ingest-pipeline.md @@ -27,11 +27,11 @@ validates and bulk-`INSERT`s. Non-insert mutations go through a different admin One process consumes a single durable JetStream consumer and fans events out to a goroutine per table. Each table batches independently and POSTs to ClickHouse -over the HTTP interface (`JSONEachRow`, with `date_time_input_format=best_effort`: -ClickHouse's default `basic` parser rejects the canonical RFC 3339 timestamps' -zone suffix (#372); `best_effort` is a superset, so pre-canonical spellings still -parse). Failed rows go to a dead-letter stream; a separate sweeper reclaims -stream storage. +over the HTTP interface (`JSONEachRow`, pinning `date_time_input_format=best_effort` — +the server default since ClickHouse 26.5; on older servers the `basic` default +rejects the canonical RFC 3339 timestamps' zone suffix (#372). A superset of +`basic`, so pre-canonical spellings still parse). Failed rows go to a +dead-letter stream; a separate sweeper reclaims stream storage. ```mermaid flowchart LR diff --git a/internal/api/ingest.go b/internal/api/ingest.go index a2f3aa93..c7da7b08 100644 --- a/internal/api/ingest.go +++ b/internal/api/ingest.go @@ -403,10 +403,9 @@ func (h *IngestHandler) processRecord( // Canonicalize timestamp columns to RFC 3339 UTC so the one payload published // below reaches every consumer — SSE, the ClickHouse insert, the DLQ — in the - // spelling /v1/query renders (#372). Fail-open: unparseable values pass through - // for ClickHouse's own parser to judge; fail-closed enforcement is the stream - // row-filter's job (#381). After the permission checks so check-clause - // comparisons keep their pre-#372 semantics. + // spelling /v1/query renders (#372); fail-open, with fail-closed enforcement + // left to the stream row-filter (#381). After the permission checks so + // check-clause comparisons keep their pre-#372 semantics. discovery.CanonicalizeTimestamps(schema, data) // Optional deduplication. diff --git a/internal/api/ingest_test.go b/internal/api/ingest_test.go index 85bd022f..380e017a 100644 --- a/internal/api/ingest_test.go +++ b/internal/api/ingest_test.go @@ -1419,7 +1419,7 @@ func TestIngest_TimestampsCanonicalized(t *testing.T) { req := ingestRequest(t, "events", map[string]any{ "name": "e", "ts": "2026-06-21 04:00:00", // zone-less ClickHouse-native form - "ts_ms": 1782014400.5, // Unix seconds + "ts_ms": 1782014400500, // integer number = ClickHouse ticks at the column scale (ms here) }) w := httptest.NewRecorder() h.Handle(w, req) diff --git a/internal/discovery/discovery_test.go b/internal/discovery/discovery_test.go index fb145400..1cd36553 100644 --- a/internal/discovery/discovery_test.go +++ b/internal/discovery/discovery_test.go @@ -93,11 +93,9 @@ func TestRefresh_EmptyDatabase(t *testing.T) { } // fakeConn implements just enough of driver.Conn to drive Refresh and -// RetryRefresh. It returns successive errors from errsThenSuccess, and once -// that slice is exhausted serves columns as the system.columns result set -// (emptyRows when nil, which makes Refresh succeed with zero tables). The -// nil-embedded interface trick handles every other method — none of them -// are reached by Refresh. +// RetryRefresh: successive errors from errsThenSuccess, then columns as the +// system.columns result set (emptyRows when nil). The nil-embedded interface +// covers every other method — none are reached by Refresh. type fakeConn struct { driver.Conn errsThenSuccess []error diff --git a/internal/discovery/timestamp.go b/internal/discovery/timestamp.go index 23fe3d31..1a2e85f4 100644 --- a/internal/discovery/timestamp.go +++ b/internal/discovery/timestamp.go @@ -13,13 +13,11 @@ import ( // Ingest canonicalizes every DateTime/DateTime64 column value to one wire form — // RFC 3339 UTC (`2026-06-21T04:00:00Z`, fraction per column precision) — before // the event is published (#372), so SSE subscribers see the same spelling -// /v1/query renders instead of whatever the producer sent. ClickHouse discards -// the input spelling at insert, so only the streamed form changes. -// -// Canonicalization is fail-open: anything it cannot parse or resolve passes -// through verbatim — ClickHouse's more liberal best_effort parser stays the -// arbiter of what is insertable (failures land in the DLQ, as before #372). -// Fail-closed enforcement of the canonical form is the stream row-filter's (#381). +// /v1/query renders; fail-open, with fail-closed enforcement left to the stream +// row-filter (#381). The grammar is a strict subset of what ClickHouse's insert +// reads — mirrored per column kind and differential-tested against a live server +// (tests/integration/timestamp_canonicalization_test.go) — and everything else +// passes through verbatim, so ClickHouse is never second-guessed. // IsTimestampType reports whether chType is a ClickHouse DateTime or DateTime64 // (unwrapping Nullable/LowCardinality). Date/Date32 are excluded — day-precision, @@ -29,13 +27,12 @@ func IsTimestampType(chType string) bool { } // CanonicalizeTimestamps rewrites every DateTime/DateTime64 column value in data -// to the canonical RFC 3339 UTC form, in place, best-effort. Zone-less values are -// interpreted as ClickHouse would — in the column's zone, else the server default -// (precomputed on the column's spec at schema build) — so only the spelling ever -// changes, never the stored instant; the fraction is truncated to the column's -// precision to byte-match /v1/query. Everything else passes through verbatim -// (fail-open): absent/null values, unparseable values, columns without a spec, and -// zone-less values when no zone is known (assuming one could move the instant). +// to the canonical RFC 3339 UTC form, in place: zone-less values are read in the +// column's zone, else the server default — ClickHouse's own rule, so only the +// spelling changes, never the instant — and the fraction truncates to the +// column's precision to byte-match /v1/query. Everything else passes through +// verbatim (fail-open): absent/null values, unparseable values, columns without +// a spec, and instants outside the column kind's range (see rewritable). func CanonicalizeTimestamps(schema *TableSchema, data map[string]any) { for _, col := range schema.Columns { spec := col.tsSpec @@ -46,28 +43,29 @@ func CanonicalizeTimestamps(schema *TableSchema, data map[string]any) { if !ok || v == nil { continue } - t, err := parseTimestamp(v, spec.loc) - if err != nil { + t, err := parseTimestamp(v, spec) + if err != nil || !spec.rewritable(t) { continue } data[col.Name] = canonicalTimestamp(t, spec.precision) } } -// timestampSpec is a timestamp column's precomputed canonicalization inputs: the -// DateTime64 sub-second precision (0 for DateTime) and the zone for interpreting -// zone-less values (declared zone, else server default; nil when neither is -// known — then only zone-explicit values canonicalize). +// timestampSpec is a timestamp column's precomputed canonicalization inputs: +// sub-second precision (0 for DateTime), the column kind (ClickHouse reads +// numbers and Unix-string fractions differently for DateTime64), and the zone +// for zone-less values (nil = unknown ⇒ only zone-explicit values canonicalize). type timestampSpec struct { precision int + isDT64 bool loc *time.Location } // resolveTimestampSpecs precomputes the timestampSpec of every DateTime/DateTime64 -// column in ts, in place — once per schema build, so the per-record ingest path -// parses no type strings and loads no zones. A column whose zone this binary -// cannot resolve (the distroless image ships no tzdata) keeps a nil spec — -// warned, not fatal: its values pass through un-canonicalized. +// column in ts once per schema build, so the per-record ingest path parses no +// type strings and loads no zones. An unresolvable zone (the distroless image +// ships no tzdata) keeps a nil spec — warned, not fatal: those values pass +// through un-canonicalized. func resolveTimestampSpecs(ts *TableSchema, serverTZ *time.Location, logger *slog.Logger) { for i := range ts.Columns { col := &ts.Columns[i] @@ -99,8 +97,9 @@ func resolveTimestampSpec(chType string, serverTZ *time.Location) (timestampSpec t = t[:open] } + isDT64 := t == "DateTime64" var precision int - if t == "DateTime64" { + if isDT64 { if len(args) == 0 { return timestampSpec{}, fmt.Errorf("malformed type %q: DateTime64 requires a precision", chType) } @@ -113,14 +112,14 @@ func resolveTimestampSpec(chType string, serverTZ *time.Location) (timestampSpec } if len(args) == 0 { - return timestampSpec{precision: precision, loc: serverTZ}, nil + return timestampSpec{precision: precision, isDT64: isDT64, loc: serverTZ}, nil } name := strings.Trim(args[0], "'") loc, err := loadLocation(name) if err != nil { return timestampSpec{}, fmt.Errorf("unknown time zone %q in type %q: %w", name, chType, err) } - return timestampSpec{precision: precision, loc: loc}, nil + return timestampSpec{precision: precision, isDT64: isDT64, loc: loc}, nil } func loadLocation(name string) (*time.Location, error) { @@ -129,71 +128,149 @@ func loadLocation(name string) (*time.Location, error) { return time.UTC, nil case "", "Local": // time.LoadLocation reads "" as UTC and "Local" from the process - // environment. Neither is a zone ClickHouse would declare, and an - // environment-dependent zone is the opposite of canonical — treat - // both as unresolvable (nil spec, pass-through) instead of guessing. + // environment — neither is a zone ClickHouse would declare. Treat both + // as unresolvable (nil spec, pass-through) instead of guessing. return nil, fmt.Errorf("not an IANA zone name: %q", name) } return time.LoadLocation(name) } -// parseTimestamp converts one ingested value into a time.Time. Accepted forms: -// RFC 3339, `YYYY-MM-DD[ T]HH:MM:SS[.fraction]` and `YYYY-MM-DD` (zone-less, -// interpreted in loc; skipped when loc is nil), and Unix seconds as a number or -// as a string of 9–10 digits. Anything else errors — the caller passes it -// through. The grammar is deliberately a subset of ClickHouse best_effort's, +// Rewrite bounds. ClickHouse saturates out-of-range instants spelling-dependently +// (the date clamps in the column zone keeping local time-of-day; DateTime64 at +// precision ≥7 even fails the insert past the Int64-nanosecond ceiling), so no +// rewrite out there is safe — the producer's own spelling passes through and +// saturates as it did before #372. One-day margins keep zone arithmetic from +// straddling a bound. +var ( + dtMin = time.Unix(0, 0) + dtMax = time.Unix(4294967295-86400, 0) // UInt32 seconds ceiling, one day of margin + dt64Min = time.Date(1900, 1, 2, 0, 0, 0, 0, time.UTC) + dt64Max = time.Date(2299, 12, 30, 23, 59, 59, 0, time.UTC) + ns64Max = time.Date(2262, 4, 10, 23, 59, 59, 0, time.UTC) // Int64 ns ceiling (precision ≥ 7) +) + +// rewritable reports whether t is safely inside the column kind's range — +// outside it the value passes through, so ClickHouse's saturation applies to +// the producer's spelling, never a rewritten one. +func (s *timestampSpec) rewritable(t time.Time) bool { + lo, hi := dtMin, dtMax + if s.isDT64 { + lo, hi = dt64Min, dt64Max + if s.precision >= 7 { + hi = ns64Max + } + } + return !t.Before(lo) && !t.After(hi) +} + +// parseTimestamp converts one ingested value into a time.Time: RFC 3339 +// ('.'-fraction only), zone-less `YYYY-MM-DD[ T]HH:MM:SS[.fraction]` / +// `YYYY-MM-DD` in the spec's zone (skipped when nil), and Unix seconds per +// parseUnixDigits and unixNumber — anything else errors and the caller passes +// it through. The grammar is deliberately a subset of ClickHouse best_effort's, // instant-identical on that subset: when unsure what ClickHouse would read, // this parser must fail, never guess. -func parseTimestamp(v any, loc *time.Location) (time.Time, error) { +func parseTimestamp(v any, spec *timestampSpec) (time.Time, error) { switch x := v.(type) { case string: + // ClickHouse has no ',' decimal separator (it would fight CSV), while + // Go's RFC3339Nano accepts one per ISO 8601. Reject up front so a + // ",999" fraction can't widen insertability. + if strings.ContainsRune(x, ',') { + return time.Time{}, fmt.Errorf( + "unrecognized timestamp %.64q (',' is not a fraction separator to ClickHouse)", x) + } if t, err := time.Parse(time.RFC3339Nano, x); err == nil { return t, nil } // Zone-less forms; Go's Parse accepts an input fraction after the seconds // even when the layout carries none. - if loc != nil { + if spec.loc != nil { for _, layout := range []string{"2006-01-02 15:04:05", "2006-01-02T15:04:05", "2006-01-02"} { - if t, err := time.ParseInLocation(layout, x, loc); err == nil { + if t, err := time.ParseInLocation(layout, x, spec.loc); err == nil { return t, nil } } } - // A digit-string is Unix seconds only in the one shape ClickHouse's - // best_effort parser reads that way: 9–10 integer digits, optional - // all-digit fraction. Other digit runs are calendar forms there - // (8 ⇒ YYYYMMDD, 14 ⇒ YYYYMMDDhhmmss, 4 ⇒ YYYY), and signs/exponents - // aren't timestamps to it at all — those pass through, so this parser - // can never read a different instant than the insert will store. - if isUnixSecondsString(x) { - if f, err := strconv.ParseFloat(x, 64); err == nil { - return unixToTime(f), nil - } + if t, ok := parseUnixDigits(x, spec.isDT64); ok { + return t, nil } return time.Time{}, fmt.Errorf( "unrecognized timestamp %.64q (accepted: RFC 3339, 'YYYY-MM-DD[ T]HH:MM:SS[.fff]', 'YYYY-MM-DD', or 9-10 digit Unix seconds)", x) case float64: - return unixToTime(x), nil + // Without json.Decoder.UseNumber a producer's integer is exact only + // through 2^53; past that the digits ClickHouse would read are already + // lost. Non-integers pass through — ClickHouse rejects them everywhere. + if x != math.Trunc(x) || math.Abs(x) > 1<<53 { + return time.Time{}, fmt.Errorf("timestamp number %v is not an exact integer", x) + } + return unixNumber(int64(x), spec) case json.Number: - f, err := x.Float64() + s := x.String() + if !isDigits(s) { + // A sign, '.', or exponent: not the integer epoch shape ClickHouse + // reads for DateTime columns — pass through for its verdict. + return time.Time{}, fmt.Errorf("timestamp number %q is not a non-negative integer", s) + } + n, err := strconv.ParseInt(s, 10, 64) if err != nil { - return time.Time{}, fmt.Errorf("unrecognized timestamp %q: %w", x.String(), err) + return time.Time{}, fmt.Errorf("timestamp number %q: %w", s, err) } - return unixToTime(f), nil + return unixNumber(n, spec) default: return time.Time{}, fmt.Errorf("timestamp must be a string or Unix-seconds number, got %T", v) } } -// isUnixSecondsString reports whether s is Unix seconds in the only string -// shape ClickHouse's best_effort parser reads as such: 9 or 10 ASCII digits, -// optionally followed by a non-empty all-digit fraction. -func isUnixSecondsString(s string) bool { +// unixNumber mirrors how ClickHouse reads an integer JSON number: Unix seconds +// for DateTime, ticks at the column's scale for DateTime64 — reading a +// DateTime64 number as seconds would change the stored instant (1750478400 is +// a 1970 instant on a DateTime64(3); the ms epoch 1750478400500 is a valid +// 2025 one). Negative numbers pass through, and out-of-range results are +// caught by rewritable. +func unixNumber(n int64, spec *timestampSpec) (time.Time, error) { + if n < 0 { + return time.Time{}, fmt.Errorf("negative timestamp number %d", n) + } + if !spec.isDT64 { + return time.Unix(n, 0), nil + } + scale := int64(1) + for range spec.precision { + scale *= 10 + } + return time.Unix(n/scale, (n%scale)*(1_000_000_000/scale)), nil +} + +// parseUnixDigits reads Unix seconds in the one string shape ClickHouse +// best_effort does — 9–10 integer digits; other run lengths are its calendar +// forms (8 ⇒ YYYYMMDD, 14 ⇒ YYYYMMDDhhmmss) or its ms/µs/ns epochs (13/16/19) +// and pass through — with a '.' fraction honored only for DateTime64 targets +// (plain DateTime fails the row on it). The fraction is an exact decimal +// truncated at nine digits: ClickHouse truncates, never rounds, and a float64 +// round-trip would corrupt nanoseconds or roll across the second. +func parseUnixDigits(s string, isDT64 bool) (time.Time, bool) { intPart, frac, hasFrac := strings.Cut(s, ".") if len(intPart) < 9 || len(intPart) > 10 || !isDigits(intPart) { - return false + return time.Time{}, false + } + if hasFrac && (!isDT64 || frac == "" || !isDigits(frac)) { + return time.Time{}, false } - return !hasFrac || (frac != "" && isDigits(frac)) + sec, err := strconv.ParseInt(intPart, 10, 64) + if err != nil { + return time.Time{}, false + } + var ns int64 + if hasFrac { + f := frac + if len(f) > 9 { + f = f[:9] + } + f += strings.Repeat("0", 9-len(f)) + ns, _ = strconv.ParseInt(f, 10, 64) // all digits by construction + } + return time.Unix(sec, ns), true } func isDigits(s string) bool { @@ -205,14 +282,6 @@ func isDigits(s string) bool { return true } -// unixToTime converts fractional Unix seconds to a time.Time (nanoseconds rounded; -// float64 loses sub-microsecond precision near current epochs — producers that need -// exact sub-second values send the string form). -func unixToTime(f float64) time.Time { - sec := math.Floor(f) - return time.Unix(int64(sec), int64(math.Round((f-sec)*1e9))) -} - // canonicalTimestamp renders t in the canonical wire form: UTC RFC 3339, fraction // truncated to the column's precision. RFC3339Nano trims trailing zeros exactly // like /v1/query's transformRow, keeping the two read paths byte-identical. diff --git a/internal/discovery/timestamp_test.go b/internal/discovery/timestamp_test.go index f6038605..26f1eef4 100644 --- a/internal/discovery/timestamp_test.go +++ b/internal/discovery/timestamp_test.go @@ -71,7 +71,15 @@ func TestCanonicalizeTimestamps(t *testing.T) { {"unix seconds digit-string", "DateTime('UTC')", nil, "1782014400", "2026-06-21T04:00:00Z"}, {"9-digit unix string", "DateTime('UTC')", nil, "999999999", "2001-09-09T01:46:39Z"}, {"fractional unix digit-string", "DateTime64(3, 'UTC')", nil, "1782014400.5", "2026-06-21T04:00:00.5Z"}, - {"fractional unix on DateTime64", "DateTime64(3, 'UTC')", nil, 1782014400.5, "2026-06-21T04:00:00.5Z"}, + {"nanosecond fraction parsed exactly, not via float64", "DateTime64(9, 'UTC')", nil, "1782014400.123456789", "2026-06-21T04:00:00.123456789Z"}, + {"fraction digits beyond nine truncate, never round", "DateTime64(9, 'UTC')", nil, "1782014400.9999999995", "2026-06-21T04:00:00.999999999Z"}, + // Integer numbers are ClickHouse *ticks* at the column scale on DateTime64 + // — the ms epoch is the natural producer shape there, and an epoch-seconds + // number really is a 1970 instant (what the insert stores either way). + {"epoch-ms number is ticks on DateTime64(3)", "DateTime64(3, 'UTC')", nil, float64(1782014400000), "2026-06-21T04:00:00Z"}, + {"epoch-ms json.Number with sub-second ticks", "DateTime64(3, 'UTC')", nil, json.Number("1782014400123"), "2026-06-21T04:00:00.123Z"}, + {"epoch-seconds number on DateTime64(3) is a 1970 instant", "DateTime64(3, 'UTC')", nil, float64(1782014400), "1970-01-21T15:00:14.4Z"}, + {"number on DateTime64(0) is seconds (scale 1)", "DateTime64(0, 'UTC')", nil, float64(1782014400), "2026-06-21T04:00:00Z"}, {"fraction truncated to column precision", "DateTime64(3, 'UTC')", nil, "2026-06-21T04:00:00.123456Z", "2026-06-21T04:00:00.123Z"}, {"fraction truncated off a second-precision column", "DateTime('UTC')", nil, "2026-06-21 04:00:00.999", "2026-06-21T04:00:00Z"}, {"trailing fractional zeros trimmed, like /v1/query", "DateTime64(3, 'UTC')", nil, "2026-06-21T04:00:00.120Z", "2026-06-21T04:00:00.12Z"}, @@ -140,9 +148,33 @@ func TestCanonicalizeTimestamps_Unparseable_PassThrough(t *testing.T) { {"14-digit string is YYYYMMDDhhmmss to ClickHouse", "DateTime('UTC')", "20260711150000"}, {"4-digit string is a year to ClickHouse", "DateTime('UTC')", "2026"}, {"11-digit string", "DateTime('UTC')", "17504784000"}, + {"13-digit string is ClickHouse's ms epoch, not ours", "DateTime('UTC')", "1752278400000"}, + {"16-digit string is ClickHouse's µs epoch, not ours", "DateTime('UTC')", "1750478400123456"}, {"scientific notation is not a timestamp", "DateTime('UTC')", "1e9"}, {"negative digit-string", "DateTime('UTC')", "-100"}, {"empty fraction", "DateTime('UTC')", "1750478400."}, + // ClickHouse consumes a fraction after a Unix epoch only for DateTime64 + // targets; on plain DateTime the leftover fraction fails the row. + {"fractional unix string on DateTime", "DateTime('UTC')", "1782014400.5"}, + // ClickHouse has no ',' decimal separator (Go's RFC3339Nano accepts one + // per ISO 8601) — rewriting would insert a row ClickHouse rejects raw. + {"comma fraction", "DateTime('UTC')", "2026-06-21T04:00:00,999Z"}, + {"comma fraction on DateTime64", "DateTime64(3, 'UTC')", "2026-06-21T04:00:00,9Z"}, + // ClickHouse rejects non-integer JSON numbers for every DateTime kind. + {"non-integer number", "DateTime64(3, 'UTC')", 1782014400.5}, + {"non-integer number on DateTime", "DateTime('UTC')", 1782014400.5}, + {"negative number", "DateTime('UTC')", float64(-100)}, + {"json.Number with exponent", "DateTime('UTC')", json.Number("1.5e9")}, + // Out of the column kind's range: ClickHouse saturates, and saturation is + // spelling-dependent (local time-of-day is kept while the date clamps), so + // no rewrite is safe — the raw spelling must be the one that saturates. + {"pre-epoch instant on DateTime", "DateTime('UTC')", "1960-01-01T00:00:00Z"}, + {"beyond UInt32 seconds on DateTime", "DateTime('UTC')", "2107-01-01T00:00:00Z"}, + {"number beyond UInt32 seconds on DateTime", "DateTime('UTC')", float64(4294967296)}, + {"beyond 2299 on DateTime64", "DateTime64(3, 'UTC')", "2300-06-30 12:30:00"}, + {"beyond the Int64-ns ceiling on DateTime64(9)", "DateTime64(9, 'UTC')", "2280-01-01T00:00:00Z"}, + // Valid RFC 3339 the subset deliberately omits (Go rejects :60). + {"leap-second spelling", "DateTime('UTC')", "2016-12-31T23:59:60Z"}, // "" and "Local" are Go LoadLocation quirks (UTC / process env), not // zone declarations — strict: unresolvable, pass through. {"empty zone name in type", "DateTime('')", "2026-06-21 04:00:00"}, @@ -179,9 +211,11 @@ func TestResolveTimestampSpecs(t *testing.T) { require.NotNil(t, schema.Columns[0].tsSpec) assert.Equal(t, nyc, schema.Columns[0].tsSpec.loc, "zone-less column takes the server zone") + assert.False(t, schema.Columns[0].tsSpec.isDT64, "DateTime is not kind DateTime64") require.NotNil(t, schema.Columns[1].tsSpec) assert.Equal(t, nyc, schema.Columns[1].tsSpec.loc) assert.Equal(t, 3, schema.Columns[1].tsSpec.precision) + assert.True(t, schema.Columns[1].tsSpec.isDT64, "numbers and unix fractions follow the DateTime64 rules") assert.Nil(t, schema.Columns[2].tsSpec, "non-timestamp column gets no spec") assert.Nil(t, schema.Columns[3].tsSpec, "unresolvable zone degrades to nil, not a failed build") require.NotNil(t, schema.Columns[4].tsSpec) diff --git a/internal/ingest/worker.go b/internal/ingest/worker.go index 24a93f06..5bad2855 100644 --- a/internal/ingest/worker.go +++ b/internal/ingest/worker.go @@ -431,9 +431,6 @@ func (w *IngestWorker) insertToClickHouse(ctx context.Context, tableName string, q.Set("database", w.db) q.Set("param_target_table", tableName) q.Set("query", "INSERT INTO {target_table:Identifier} FORMAT JSONEachRow") - // Canonical RFC 3339 timestamps (#372) carry a zone suffix ClickHouse's - // default 'basic' parser rejects; best_effort is a superset, so non-canonical - // values (pre-#372 messages, fail-open pass-throughs) parse as before. q.Set("date_time_input_format", "best_effort") req, err := http.NewRequestWithContext(ctx, "POST", w.chURL+"?"+q.Encode(), &buf) diff --git a/internal/ingest/worker_test.go b/internal/ingest/worker_test.go index 83671a98..0799525b 100644 --- a/internal/ingest/worker_test.go +++ b/internal/ingest/worker_test.go @@ -323,8 +323,9 @@ func TestInsertToClickHouse_BuildsCorrectRequest(t *testing.T) { assert.Equal(t, "test_db", q.Get("database")) assert.Equal(t, "events", q.Get("param_target_table")) assert.Equal(t, "INSERT INTO {target_table:Identifier} FORMAT JSONEachRow", q.Get("query")) - // #372: canonical RFC 3339 timestamps carry a zone suffix the default - // 'basic' parser rejects, so the insert must opt into best_effort. + // #372: the insert pins best_effort — the server default since + // ClickHouse 26.5; older 'basic' defaults reject the canonical + // form's zone suffix. assert.Equal(t, "best_effort", q.Get("date_time_input_format")) assert.Equal(t, "application/json", req.Header.Get("Content-Type")) diff --git a/tests/integration/timestamp_canonicalization_test.go b/tests/integration/timestamp_canonicalization_test.go index eb878ca4..bd9900e3 100644 --- a/tests/integration/timestamp_canonicalization_test.go +++ b/tests/integration/timestamp_canonicalization_test.go @@ -21,16 +21,11 @@ import ( // TestTimestampCanonicalization_DifferentialAgainstClickHouse pins the #372 // invariant with ClickHouse itself as the oracle (PR #402 review): for every // corpus input × timestamp column shape, inserting the raw producer value and -// inserting CanonicalizeTimestamps' output must (a) both succeed or both -// fail, and (b) when both succeed, store the same instant. A same-instant -// failure means canonicalization rewrote a value ClickHouse reads differently -// (data corruption); an asymmetric accept means canonicalization changed -// which values are insertable (ClickHouse, not the canonicalizer, is the -// arbiter of insertability). -// -// Inserts go through the same HTTP surface the ingest worker uses — -// JSONEachRow with date_time_input_format=best_effort — so the oracle is the -// production parse, not a lookalike. +// inserting CanonicalizeTimestamps' output must both succeed or both fail, and +// when both succeed store the same instant — anything else means the rewrite +// changed what ClickHouse stores or accepts. Inserts go through the worker's +// exact HTTP surface (JSONEachRow, date_time_input_format=best_effort), so the +// oracle is the production parse, not a lookalike. func TestTimestampCanonicalization_DifferentialAgainstClickHouse(t *testing.T) { colTypes := []struct { name string @@ -40,11 +35,18 @@ func TestTimestampCanonicalization_DifferentialAgainstClickHouse(t *testing.T) { {"datetime_nyc", "DateTime('America/New_York')"}, {"datetime64_3", "DateTime64(3)"}, {"datetime64_6_tokyo", "DateTime64(6, 'Asia/Tokyo')"}, + // Precision 9 has its own ceiling (Int64 nanoseconds end 2262-04-11, past + // which an insert fails outright instead of saturating), and precision 0 + // pins the DateTime64-kind rules on a second-granular column. + {"datetime64_9", "DateTime64(9)"}, + {"datetime64_0_utc", "DateTime64(0, 'UTC')"}, } - // One entry per spelling family. The calendar-shaped digit runs and the - // garbage entries document the pass-through side of the contract: they - // must reach ClickHouse verbatim and get its verdict, never a rewrite. + // One entry per spelling family, each found or pinned by differentially + // fuzzing the canonicalizer against a live ClickHouse (PR #402 review). The + // digit runs, number shapes, out-of-range instants, and garbage document the + // pass-through side: they must reach ClickHouse verbatim and get its + // verdict, never a rewrite. corpus := []any{ "2026-06-21T04:00:00Z", // canonical already "2026-06-21T06:30:00+02:30", // offset form @@ -53,19 +55,28 @@ func TestTimestampCanonicalization_DifferentialAgainstClickHouse(t *testing.T) { "2026-06-21", // date-only "2026-06-21 04:00:00.123456", // zone-less with fraction "2026-06-21T04:00:00.9999Z", // fraction beyond column precision + "2026-06-21T04:00:00,999Z", // comma fraction: ISO 8601 yes, ClickHouse no "1750478400", // 10-digit Unix string "999999999", // 9-digit Unix string - "1750478400.5", // fractional Unix string - float64(1750478400), // Unix number + "1750478400.5", // fractional Unix string: DateTime64-only to ClickHouse + "1750478400.123456789", // ns-exact fraction (float64 would corrupt it) + "1750478400.9999999995", // >9 fraction digits: ClickHouse truncates, never rounds + float64(1750478400), // integer number: seconds to DateTime, *ticks* to DateTime64 + float64(1750478400.5), // non-integer number: ClickHouse rejects for every kind + float64(1750478400500), // epoch-ms number: DateTime64(3)'s natural ticks shape "20260711", // 8 digits: YYYYMMDD to ClickHouse "20260711150000", // 14 digits: YYYYMMDDhhmmss "202607111500", // 12 digits: ClickHouse rejects + "1752278400000", // 13 digits: epoch milliseconds to ClickHouse + "1750478400123456", // 16 digits: epoch microseconds to ClickHouse "2026", // 4 digits: a year to ClickHouse "1e9", // not a timestamp "-100", // not a timestamp "banana", // garbage "2026-11-01 01:30:00", // DST fall-back: ambiguous local time "2026-03-08 02:30:00", // DST spring-forward: nonexistent local time + "1960-01-01T00:00:00Z", // pre-range: ClickHouse saturates, spelling-dependently + "2300-06-30 12:30:00", // beyond DateTime64's ceiling, zone-less } for _, ct := range colTypes { From de96ab329679a7c8cc71d81aad4b48f2d6ec20c9 Mon Sep 17 00:00:00 2001 From: taitelee Date: Mon, 13 Jul 2026 13:12:20 -0400 Subject: [PATCH 11/15] test: raise integration timeout to 240s; parallelize oracle subtests --- Makefile | 2 +- tests/integration/timestamp_canonicalization_test.go | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index de1ebea2..f65bfdb6 100644 --- a/Makefile +++ b/Makefile @@ -677,7 +677,7 @@ test-integration: go-mod-download ## Run Go integration tests + render coverage @printf "$(CYAN)==> Running Integration Tests...$(RESET)\n" @rm -rf $(COV_INT)/data && mkdir -p $(COV_INT)/data @GOCOVERDIR="$(CURDIR)/$(COV_INT)/data" go tool gotestsum --format $(GOTESTSUM_FMT) -- \ - -tags="integration $(TAGS)" -timeout 120s -coverpkg=./... -race -count=1 \ + -tags="integration $(TAGS)" -timeout 240s -coverpkg=./... -race -count=1 \ ./tests/integration/... $(ARGS) \ -args -test.gocoverdir="$(CURDIR)/$(COV_INT)/data" @if [ -z "$(COV_DEFER)" ]; then go run ./scripts/cov render integration; fi diff --git a/tests/integration/timestamp_canonicalization_test.go b/tests/integration/timestamp_canonicalization_test.go index bd9900e3..2343a4e0 100644 --- a/tests/integration/timestamp_canonicalization_test.go +++ b/tests/integration/timestamp_canonicalization_test.go @@ -81,6 +81,9 @@ func TestTimestampCanonicalization_DifferentialAgainstClickHouse(t *testing.T) { for _, ct := range colTypes { t.Run(ct.name, func(t *testing.T) { + // Independent tables per shape; parallel keeps the six shapes from + // serializing ~350 single-row inserts against the suite timeout. + t.Parallel() rawTable := createTable(t, "id UInt32, v "+ct.ddl, "ORDER BY id") canonTable := createTable(t, "id UInt32, v "+ct.ddl, "ORDER BY id") schema := env(t).registry.Get(rawTable) From f7b15c0ee4ace2f4f52e2883a548962c033a04eb Mon Sep 17 00:00:00 2001 From: taitelee Date: Mon, 13 Jul 2026 15:02:13 -0400 Subject: [PATCH 12/15] docs: fix tzdata claims, document zone pass-through, trim ingest comment --- CHANGELOG.md | 2 +- docs/src/content/docs/api.md | 2 +- internal/api/ingest.go | 7 ++----- internal/discovery/timestamp.go | 6 +++--- 4 files changed, 7 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c27ea589..f928f129 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,7 +50,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed -- **SSE row `DateTime` columns now arrive in canonical RFC 3339 UTC, matching `/v1/query`, instead of the producer's spelling** (`internal/discovery/timestamp.go` (new), `internal/discovery/{discovery.go,validation.go}`, `internal/api/ingest.go`, `internal/ingest/worker.go`, `docs/src/content/docs/{api.md,architecture.md}`, plus tests in `internal/discovery/{timestamp_test.go (new),discovery_test.go}`, `internal/api/{ingest_test.go,boot_chain_test.go}`, `tests/e2e/sdk/streaming.test.ts`): closes #372. The stream fans out the pre-insert payload verbatim, so a row `DateTime` reached SSE subscribers in whatever spelling the producer sent — typically the zone-less ClickHouse-native form — while `/v1/query` rendered the stored value as RFC 3339 `Z`; JavaScript's `Date.parse` reads a zone-less string as **local** time, so an SDK `liveQuery` stitching backfill + live landed the two paths hours apart (sign-flipping with the viewer's UTC offset — broke ordering and the "N min ago" label in the Stats live feed). Ingest now **canonicalizes** every `DateTime`/`DateTime64` column value to RFC 3339 UTC (fraction truncated to the column's precision, `time.RFC3339Nano`-trimmed — byte-matching `/v1/query`'s `time.Time` marshaling) after validation and before the NATS publish, so the one payload every consumer shares — SSE subscribers, the ClickHouse insert, the DLQ — carries a single unambiguous spelling. Inputs stay liberal but are mirrored per column kind, exactly as ClickHouse reads them (#402 review): RFC 3339 with any offset (`.`-fractions only — ClickHouse has no `,` separator), `YYYY-MM-DD[ T]HH:MM:SS[.fff]`, `YYYY-MM-DD`, a Unix-seconds string of 9–10 digits (a fraction after it is honored only for `DateTime64` columns, parsed as an exact decimal and truncated at nine digits — never a `float64` round-trip, which corrupts nanoseconds and can round across the second; other digit lengths are ClickHouse's own forms — calendar `YYYYMMDD`/`YYYYMMDDhhmmss` or its 13/16/19-digit ms/µs/ns epochs — and pass through), and **integer** JSON numbers read the way ClickHouse reads them: Unix seconds for `DateTime`, **ticks at the column's scale** for `DateTime64` (the ms epoch `1750478400500` into a `DateTime64(3)` is a valid 2025 instant; an epoch-*seconds* number there is a 1970 instant — rewriting either as seconds would change what ClickHouse stores; non-integer numbers pass through, ClickHouse rejects them). Values whose instant lies outside the column type's range also pass through — ClickHouse *saturates* out-of-range values spelling-dependently (local time-of-day is kept while the date clamps), so only the producer's own spelling may be the one that saturates. Go `LoadLocation`'s `''`/`'Local'` environment quirks are rejected as unresolvable zones. A zone-less string is interpreted in the column's declared time zone, else the server's (discovered once per schema refresh via `SELECT timezone()`) — the same rule ClickHouse itself applies, so canonicalization never changes which instant is stored (differentially fuzzed against a live ClickHouse — ~35k generated inputs × five column shapes, raw vs canonicalized inserts, zero divergences — with every divergence class found along the way pinned in `tests/integration/timestamp_canonicalization_test.go`). Each timestamp column's spec (precision + resolved zone) is precomputed at schema discovery and cached on the column, so the per-record ingest path parses no type strings, loads no zones, and takes no locks. Canonicalization is **fail-open** (#402 review): a value outside the accepted forms publishes verbatim — ClickHouse's more liberal `best_effort` parser stays the arbiter of insertability, and a value it too rejects surfaces via the DLQ as before — and an unresolvable zone (no tzdata embedded; the distroless image ships none) warns and passes through, never a failed refresh or a silent UTC reinterpretation that would move instants (`Etc/UTC` is special-cased to UTC); ingest never rejects a record over its timestamp spelling — fail-closed enforcement of the canonical form is the stream row-filter's (#381). The worker's `INSERT … FORMAT JSONEachRow` pins `date_time_input_format=best_effort` — the server default since ClickHouse 26.5, and on older servers the `basic` default rejects the canonical form's zone suffix (verified live); a superset of `basic` either way, so pre-upgrade messages still in NATS — zone-less strings, Unix numbers — parse exactly as before. Boundaries: `Date`/`Date32` columns pass through untouched (day precision, no zone ambiguity on this path); events published before the upgrade replay in their original spelling until the sweeper retires them; a column with an explicit non-UTC zone renders as `Z` on both paths (`/v1/query` likewise normalizes `DateTime` values to UTC), so the declared zone affects how zone-less inputs are read but never the output spelling — byte-identical regardless of zone. +- **SSE row `DateTime` columns now arrive in canonical RFC 3339 UTC, matching `/v1/query`, instead of the producer's spelling** (`internal/discovery/timestamp.go` (new), `internal/discovery/{discovery.go,validation.go}`, `internal/api/ingest.go`, `internal/ingest/worker.go`, `docs/src/content/docs/{api.md,architecture.md}`, plus tests in `internal/discovery/{timestamp_test.go (new),discovery_test.go}`, `internal/api/{ingest_test.go,boot_chain_test.go}`, `tests/e2e/sdk/streaming.test.ts`): closes #372. The stream fans out the pre-insert payload verbatim, so a row `DateTime` reached SSE subscribers in whatever spelling the producer sent — typically the zone-less ClickHouse-native form — while `/v1/query` rendered the stored value as RFC 3339 `Z`; JavaScript's `Date.parse` reads a zone-less string as **local** time, so an SDK `liveQuery` stitching backfill + live landed the two paths hours apart (sign-flipping with the viewer's UTC offset — broke ordering and the "N min ago" label in the Stats live feed). Ingest now **canonicalizes** every `DateTime`/`DateTime64` column value to RFC 3339 UTC (fraction truncated to the column's precision, `time.RFC3339Nano`-trimmed — byte-matching `/v1/query`'s `time.Time` marshaling) after validation and before the NATS publish, so the one payload every consumer shares — SSE subscribers, the ClickHouse insert, the DLQ — carries a single unambiguous spelling. Inputs stay liberal but are mirrored per column kind, exactly as ClickHouse reads them (#402 review): RFC 3339 with any offset (`.`-fractions only — ClickHouse has no `,` separator), `YYYY-MM-DD[ T]HH:MM:SS[.fff]`, `YYYY-MM-DD`, a Unix-seconds string of 9–10 digits (a fraction after it is honored only for `DateTime64` columns, parsed as an exact decimal and truncated at nine digits — never a `float64` round-trip, which corrupts nanoseconds and can round across the second; other digit lengths are ClickHouse's own forms — calendar `YYYYMMDD`/`YYYYMMDDhhmmss` or its 13/16/19-digit ms/µs/ns epochs — and pass through), and **integer** JSON numbers read the way ClickHouse reads them: Unix seconds for `DateTime`, **ticks at the column's scale** for `DateTime64` (the ms epoch `1750478400500` into a `DateTime64(3)` is a valid 2025 instant; an epoch-*seconds* number there is a 1970 instant — rewriting either as seconds would change what ClickHouse stores; non-integer numbers pass through, ClickHouse rejects them). Values whose instant lies outside the column type's range also pass through — ClickHouse *saturates* out-of-range values spelling-dependently (local time-of-day is kept while the date clamps), so only the producer's own spelling may be the one that saturates. Go `LoadLocation`'s `''`/`'Local'` environment quirks are rejected as unresolvable zones. A zone-less string is interpreted in the column's declared time zone, else the server's (discovered once per schema refresh via `SELECT timezone()`) — the same rule ClickHouse itself applies, so canonicalization never changes which instant is stored (differentially fuzzed against a live ClickHouse — ~35k generated inputs × five column shapes, raw vs canonicalized inserts, zero divergences — with every divergence class found along the way pinned in `tests/integration/timestamp_canonicalization_test.go`). Each timestamp column's spec (precision + resolved zone) is precomputed at schema discovery and cached on the column, so the per-record ingest path parses no type strings, loads no zones, and takes no locks. Canonicalization is **fail-open** (#402 review): a value outside the accepted forms publishes verbatim — ClickHouse's more liberal `best_effort` parser stays the arbiter of insertability, and a value it too rejects surfaces via the DLQ as before — and an unresolvable zone (no tzdata embedded — named zones resolve from the runtime's zone database, which the bundled distroless images ship) warns and passes through, never a failed refresh or a silent UTC reinterpretation that would move instants (`Etc/UTC` is special-cased to UTC); ingest never rejects a record over its timestamp spelling — fail-closed enforcement of the canonical form is the stream row-filter's (#381). The worker's `INSERT … FORMAT JSONEachRow` pins `date_time_input_format=best_effort` — the server default since ClickHouse 26.5, and on older servers the `basic` default rejects the canonical form's zone suffix (verified live); a superset of `basic` either way, so pre-upgrade messages still in NATS — zone-less strings, Unix numbers — parse exactly as before. Boundaries: `Date`/`Date32` columns pass through untouched (day precision, no zone ambiguity on this path); events published before the upgrade replay in their original spelling until the sweeper retires them; a column with an explicit non-UTC zone renders as `Z` on both paths (`/v1/query` likewise normalizes `DateTime` values to UTC), so the declared zone affects how zone-less inputs are read but never the output spelling — byte-identical regardless of zone. - **Live demo hero feed renders in `event_ts` order instead of SSE arrival order** (`docs/src/components/LiveDemo.astro`): the landing-page live activity feed prepended each streamed row to the top in the order the SSE stream delivered it, but a producer's webhook burst (a single merge-queue cycle fires ~20 events) arrives in no guaranteed order and the stream relays it in ingest order — so a late or out-of-order delivery landed above newer rows (e.g. a `pushed 12m ago` sitting on top of `reviewed a pull request 9m ago`). `addRow` now keeps the feed sorted by `event_ts` descending — it slots each row in before the first strictly-older sibling rather than blind-prepending — so the live tail matches the already-sorted `gh_activity_recent` backfill. The zone-less-SSE-timestamp normalization the sort relies on (`normTs`) was already in place; equal-second rows keep arrival order (`gh_events.event_ts` is only second-granular for CI/checks, so there's no finer tiebreak), and dedup + the `MAX_ROWS` trim are unchanged. Surfaced in dogfooding on `wavehouse.dev`; the client-side analog of the ingest-order reality the SSE stream can't reorder. diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index 8807adb7..6b15f65a 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -217,7 +217,7 @@ The body is a **flat JSON object** whose keys must match column names in the tar - Type compatibility: `String`/`DateTime`/`UUID`/`Enum`/`IPv*` accept JSON strings; `Int*`/`Float*`/`Decimal` accept JSON numbers; `Bool` accepts JSON booleans or numbers; `Array` accepts JSON arrays; `Map`/`Tuple` accept JSON objects. - `Nullable()` and `LowCardinality()` wrappers are handled transparently. -**Timestamp canonicalization:** a `DateTime`/`DateTime64` column value sent as RFC 3339 (any offset; `.`-fractions only — ClickHouse has no `,` separator), `YYYY-MM-DD[ T]HH:MM:SS[.fff]` or `YYYY-MM-DD` (zone-less — interpreted in the column's time zone, else the ClickHouse server's, exactly as ClickHouse itself would), a Unix-seconds string of exactly 9–10 digits (a `.fff` fraction is honored only for `DateTime64` columns, as ClickHouse does; other digit lengths are ClickHouse's own forms — calendar shapes like `YYYYMMDD`, or its 13/16/19-digit ms/µs/ns epochs — so they pass through untouched), or an **integer** JSON number — read the way ClickHouse reads numbers: Unix seconds for `DateTime`, **ticks at the column's scale** for `DateTime64` (`1750478400500` into a `DateTime64(3)` is the millisecond epoch; non-integer numbers pass through, ClickHouse rejects them) — is rewritten to **one canonical wire form — RFC 3339 UTC** (`2026-06-21T04:00:00Z`, fraction truncated to the column's precision) before publishing, so the stored instant never changes but every consumer (the ClickHouse insert, [SSE subscribers](#get-v1stream--server-sent-events-stream), the DLQ) sees the same spelling `/v1/query` renders. **Fail-open**: a value in none of those forms — or whose instant lies outside the column type's range, where ClickHouse *saturates* spelling-dependently instead of erroring — is published verbatim: ClickHouse's more liberal parser decides insertability, and a value it too rejects surfaces via the DLQ, as before. The accepted grammar is differentially tested against a live ClickHouse (raw and canonicalized spellings must insert identically, or both fail). `Date`/`Date32` columns pass through untouched. +**Timestamp canonicalization:** a `DateTime`/`DateTime64` column value sent as RFC 3339 (any offset; `.`-fractions only — ClickHouse has no `,` separator), `YYYY-MM-DD[ T]HH:MM:SS[.fff]` or `YYYY-MM-DD` (zone-less — interpreted in the column's time zone, else the ClickHouse server's, exactly as ClickHouse itself would), a Unix-seconds string of exactly 9–10 digits (a `.fff` fraction is honored only for `DateTime64` columns, as ClickHouse does; other digit lengths are ClickHouse's own forms — calendar shapes like `YYYYMMDD`, or its 13/16/19-digit ms/µs/ns epochs — so they pass through untouched), or an **integer** JSON number — read the way ClickHouse reads numbers: Unix seconds for `DateTime`, **ticks at the column's scale** for `DateTime64` (`1750478400500` into a `DateTime64(3)` is the millisecond epoch; non-integer numbers pass through, ClickHouse rejects them) — is rewritten to **one canonical wire form — RFC 3339 UTC** (`2026-06-21T04:00:00Z`, fraction truncated to the column's precision) before publishing, so the stored instant never changes but every consumer (the ClickHouse insert, [SSE subscribers](#get-v1stream--server-sent-events-stream), the DLQ) sees the same spelling `/v1/query` renders. **Fail-open**: a value in none of those forms — or whose instant lies outside the column type's range, where ClickHouse *saturates* spelling-dependently instead of erroring — is published verbatim: ClickHouse's more liberal parser decides insertability, and a value it too rejects surfaces via the DLQ, as before. Pass-through likewise applies when a time zone doesn't resolve at runtime — the binary embeds no tzdata, so named zones resolve from the runtime's zone database (the bundled distroless images ship one; a stripped-down custom runtime, or a server zone newer than the image's snapshot, may not resolve): an unresolvable *column* zone skips canonicalization for that column entirely, an unresolvable *server* default skips only zone-less values of columns without a declared zone — warned at schema refresh either way, and never guessed as UTC, which could move the stored instant. The accepted grammar is differentially tested against a live ClickHouse (raw and canonicalized spellings must insert identically, or both fail). `Date`/`Date32` columns pass through untouched. **The canonical form, precisely.** This is the one strict timestamp spelling in WaveHouse — the same one `/v1/query` renders and the SSE stream carries, and the form the stream row-filter will require for timestamp comparisons once row-level enforcement lands ([#381](https://github.com/Wave-RF/WaveHouse/issues/381)): diff --git a/internal/api/ingest.go b/internal/api/ingest.go index c7da7b08..d455b873 100644 --- a/internal/api/ingest.go +++ b/internal/api/ingest.go @@ -401,11 +401,8 @@ func (h *IngestHandler) processRecord( } } - // Canonicalize timestamp columns to RFC 3339 UTC so the one payload published - // below reaches every consumer — SSE, the ClickHouse insert, the DLQ — in the - // spelling /v1/query renders (#372); fail-open, with fail-closed enforcement - // left to the stream row-filter (#381). After the permission checks so - // check-clause comparisons keep their pre-#372 semantics. + // Canonicalize timestamps to RFC 3339 UTC (#372; fail-open — #381's row-filter + // enforces) after the permission checks: check clauses keep pre-#372 semantics. discovery.CanonicalizeTimestamps(schema, data) // Optional deduplication. diff --git a/internal/discovery/timestamp.go b/internal/discovery/timestamp.go index 1a2e85f4..ac596858 100644 --- a/internal/discovery/timestamp.go +++ b/internal/discovery/timestamp.go @@ -63,9 +63,9 @@ type timestampSpec struct { // resolveTimestampSpecs precomputes the timestampSpec of every DateTime/DateTime64 // column in ts once per schema build, so the per-record ingest path parses no -// type strings and loads no zones. An unresolvable zone (the distroless image -// ships no tzdata) keeps a nil spec — warned, not fatal: those values pass -// through un-canonicalized. +// type strings and loads no zones. An unresolvable zone (no embedded tzdata — +// resolution needs the runtime's zone database) keeps a nil spec — warned, not +// fatal: those values pass through un-canonicalized. func resolveTimestampSpecs(ts *TableSchema, serverTZ *time.Location, logger *slog.Logger) { for i := range ts.Columns { col := &ts.Columns[i] From 3f65c5dc8e699aeddd3458607e439cb9942ef7aa Mon Sep 17 00:00:00 2001 From: taitelee Date: Wed, 15 Jul 2026 15:33:18 -0400 Subject: [PATCH 13/15] refactor: trim parseUnixDigits, unexport isTimestampType, share UTC-row stub --- docs/src/content/docs/ingest-pipeline.md | 4 ++-- internal/api/boot_chain_test.go | 15 ++------------- internal/discovery/discovery_test.go | 3 +-- internal/discovery/timestamp.go | 16 ++++++++-------- internal/discovery/timestamp_test.go | 2 +- internal/testutil/testutil.go | 9 ++++++--- 6 files changed, 20 insertions(+), 29 deletions(-) diff --git a/docs/src/content/docs/ingest-pipeline.md b/docs/src/content/docs/ingest-pipeline.md index a95d74f1..a690fa6e 100644 --- a/docs/src/content/docs/ingest-pipeline.md +++ b/docs/src/content/docs/ingest-pipeline.md @@ -29,8 +29,8 @@ One process consumes a single durable JetStream consumer and fans events out to a goroutine per table. Each table batches independently and POSTs to ClickHouse over the HTTP interface (`JSONEachRow`, pinning `date_time_input_format=best_effort` — the server default since ClickHouse 26.5; on older servers the `basic` default -rejects the canonical RFC 3339 timestamps' zone suffix (#372). A superset of -`basic`, so pre-canonical spellings still parse). Failed rows go to a +rejects the canonical RFC 3339 form's zone suffix — a superset of `basic`, so +pre-canonical spellings still parse (#372)). Failed rows go to a dead-letter stream; a separate sweeper reclaims stream storage. ```mermaid diff --git a/internal/api/boot_chain_test.go b/internal/api/boot_chain_test.go index b4cb8be7..917ef70c 100644 --- a/internal/api/boot_chain_test.go +++ b/internal/api/boot_chain_test.go @@ -17,6 +17,7 @@ import ( "github.com/stretchr/testify/require" "github.com/Wave-RF/WaveHouse/internal/discovery" + "github.com/Wave-RF/WaveHouse/internal/testutil" ) // errsThenSuccessConn drives the boot Refresh/RetryRefresh chain in tests. @@ -41,19 +42,7 @@ func (c *errsThenSuccessConn) Query(_ context.Context, _ string, _ ...any) (driv // QueryRow answers the SELECT timezone() probe Refresh issues before the // system.columns query (#372); the error sequencing above stays keyed on Query. func (c *errsThenSuccessConn) QueryRow(context.Context, string, ...any) driver.Row { - return chainTZRow{} -} - -type chainTZRow struct{ driver.Row } - -func (chainTZRow) Scan(dest ...any) error { - if len(dest) == 1 { - if s, ok := dest[0].(*string); ok { - *s = "UTC" - return nil - } - } - return errors.New("unexpected timezone scan") + return testutil.UTCRow{} } type chainEmptyRows struct{ driver.Rows } diff --git a/internal/discovery/discovery_test.go b/internal/discovery/discovery_test.go index 1cd36553..5265edc0 100644 --- a/internal/discovery/discovery_test.go +++ b/internal/discovery/discovery_test.go @@ -193,8 +193,7 @@ func newFakeRegistry(t *testing.T, errs []error) (*SchemaRegistry, *fakeConn) { func TestRefresh_UnresolvableServerTimezone_NotFatal(t *testing.T) { t.Parallel() conn := &fakeConn{tz: "Not/AZone"} - logger := slog.New(slog.NewJSONHandler(io.Discard, nil)) - sr := NewSchemaRegistry(conn, "test", time.Hour, logger) + sr := NewSchemaRegistry(conn, "test", time.Hour, discardLogger()) require.NoError(t, sr.Refresh(context.Background())) } diff --git a/internal/discovery/timestamp.go b/internal/discovery/timestamp.go index ac596858..a91e797c 100644 --- a/internal/discovery/timestamp.go +++ b/internal/discovery/timestamp.go @@ -19,10 +19,10 @@ import ( // (tests/integration/timestamp_canonicalization_test.go) — and everything else // passes through verbatim, so ClickHouse is never second-guessed. -// IsTimestampType reports whether chType is a ClickHouse DateTime or DateTime64 +// isTimestampType reports whether chType is a ClickHouse DateTime or DateTime64 // (unwrapping Nullable/LowCardinality). Date/Date32 are excluded — day-precision, // no zone or spelling ambiguity. -func IsTimestampType(chType string) bool { +func isTimestampType(chType string) bool { return strings.HasPrefix(unwrapType(chType), "DateTime") } @@ -69,7 +69,7 @@ type timestampSpec struct { func resolveTimestampSpecs(ts *TableSchema, serverTZ *time.Location, logger *slog.Logger) { for i := range ts.Columns { col := &ts.Columns[i] - if !IsTimestampType(col.Type) { + if !isTimestampType(col.Type) { continue } spec, err := resolveTimestampSpec(col.Type, serverTZ) @@ -257,18 +257,18 @@ func parseUnixDigits(s string, isDT64 bool) (time.Time, bool) { if hasFrac && (!isDT64 || frac == "" || !isDigits(frac)) { return time.Time{}, false } - sec, err := strconv.ParseInt(intPart, 10, 64) - if err != nil { - return time.Time{}, false - } + sec, _ := strconv.ParseInt(intPart, 10, 64) // 9-10 digits by construction var ns int64 if hasFrac { f := frac if len(f) > 9 { f = f[:9] } - f += strings.Repeat("0", 9-len(f)) ns, _ = strconv.ParseInt(f, 10, 64) // all digits by construction + // Zero-fill the fraction out to nanosecond scale. + for range 9 - len(f) { + ns *= 10 + } } return time.Unix(sec, ns), true } diff --git a/internal/discovery/timestamp_test.go b/internal/discovery/timestamp_test.go index 26f1eef4..9457c70c 100644 --- a/internal/discovery/timestamp_test.go +++ b/internal/discovery/timestamp_test.go @@ -34,7 +34,7 @@ func TestIsTimestampType(t *testing.T) { for _, tt := range tests { t.Run(tt.chType, func(t *testing.T) { t.Parallel() - assert.Equal(t, tt.want, IsTimestampType(tt.chType)) + assert.Equal(t, tt.want, isTimestampType(tt.chType)) }) } } diff --git a/internal/testutil/testutil.go b/internal/testutil/testutil.go index e5be4e5e..d4abab3f 100644 --- a/internal/testutil/testutil.go +++ b/internal/testutil/testutil.go @@ -48,7 +48,7 @@ type schemaConn struct { } func (c *schemaConn) QueryRow(context.Context, string, ...any) driver.Row { - return utcRow{} + return UTCRow{} } func (c *schemaConn) Query(context.Context, string, ...any) (driver.Rows, error) { @@ -65,9 +65,12 @@ func (c *schemaConn) Query(context.Context, string, ...any) (driver.Rows, error) return r, nil } -type utcRow struct{ driver.Row } +// UTCRow is a driver.Row stub answering the SELECT timezone() probe Refresh +// issues with "UTC" — for any mock driver.Conn that must satisfy the +// schema-refresh path. +type UTCRow struct{ driver.Row } -func (utcRow) Scan(dest ...any) error { +func (UTCRow) Scan(dest ...any) error { if len(dest) == 1 { if s, ok := dest[0].(*string); ok { *s = "UTC" From 498c891bd999b12e124351fdc389bd9eb05d7908 Mon Sep 17 00:00:00 2001 From: taitelee Date: Thu, 16 Jul 2026 15:28:48 -0400 Subject: [PATCH 14/15] test: feed the CH oracle production json.Number epochs (>2^53); docs: bullet api.md input forms, six shapes --- CHANGELOG.md | 2 +- docs/src/content/docs/api.md | 9 +++++- .../timestamp_canonicalization_test.go | 31 ++++++++++--------- 3 files changed, 26 insertions(+), 16 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f928f129..ddee78e5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,7 +50,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed -- **SSE row `DateTime` columns now arrive in canonical RFC 3339 UTC, matching `/v1/query`, instead of the producer's spelling** (`internal/discovery/timestamp.go` (new), `internal/discovery/{discovery.go,validation.go}`, `internal/api/ingest.go`, `internal/ingest/worker.go`, `docs/src/content/docs/{api.md,architecture.md}`, plus tests in `internal/discovery/{timestamp_test.go (new),discovery_test.go}`, `internal/api/{ingest_test.go,boot_chain_test.go}`, `tests/e2e/sdk/streaming.test.ts`): closes #372. The stream fans out the pre-insert payload verbatim, so a row `DateTime` reached SSE subscribers in whatever spelling the producer sent — typically the zone-less ClickHouse-native form — while `/v1/query` rendered the stored value as RFC 3339 `Z`; JavaScript's `Date.parse` reads a zone-less string as **local** time, so an SDK `liveQuery` stitching backfill + live landed the two paths hours apart (sign-flipping with the viewer's UTC offset — broke ordering and the "N min ago" label in the Stats live feed). Ingest now **canonicalizes** every `DateTime`/`DateTime64` column value to RFC 3339 UTC (fraction truncated to the column's precision, `time.RFC3339Nano`-trimmed — byte-matching `/v1/query`'s `time.Time` marshaling) after validation and before the NATS publish, so the one payload every consumer shares — SSE subscribers, the ClickHouse insert, the DLQ — carries a single unambiguous spelling. Inputs stay liberal but are mirrored per column kind, exactly as ClickHouse reads them (#402 review): RFC 3339 with any offset (`.`-fractions only — ClickHouse has no `,` separator), `YYYY-MM-DD[ T]HH:MM:SS[.fff]`, `YYYY-MM-DD`, a Unix-seconds string of 9–10 digits (a fraction after it is honored only for `DateTime64` columns, parsed as an exact decimal and truncated at nine digits — never a `float64` round-trip, which corrupts nanoseconds and can round across the second; other digit lengths are ClickHouse's own forms — calendar `YYYYMMDD`/`YYYYMMDDhhmmss` or its 13/16/19-digit ms/µs/ns epochs — and pass through), and **integer** JSON numbers read the way ClickHouse reads them: Unix seconds for `DateTime`, **ticks at the column's scale** for `DateTime64` (the ms epoch `1750478400500` into a `DateTime64(3)` is a valid 2025 instant; an epoch-*seconds* number there is a 1970 instant — rewriting either as seconds would change what ClickHouse stores; non-integer numbers pass through, ClickHouse rejects them). Values whose instant lies outside the column type's range also pass through — ClickHouse *saturates* out-of-range values spelling-dependently (local time-of-day is kept while the date clamps), so only the producer's own spelling may be the one that saturates. Go `LoadLocation`'s `''`/`'Local'` environment quirks are rejected as unresolvable zones. A zone-less string is interpreted in the column's declared time zone, else the server's (discovered once per schema refresh via `SELECT timezone()`) — the same rule ClickHouse itself applies, so canonicalization never changes which instant is stored (differentially fuzzed against a live ClickHouse — ~35k generated inputs × five column shapes, raw vs canonicalized inserts, zero divergences — with every divergence class found along the way pinned in `tests/integration/timestamp_canonicalization_test.go`). Each timestamp column's spec (precision + resolved zone) is precomputed at schema discovery and cached on the column, so the per-record ingest path parses no type strings, loads no zones, and takes no locks. Canonicalization is **fail-open** (#402 review): a value outside the accepted forms publishes verbatim — ClickHouse's more liberal `best_effort` parser stays the arbiter of insertability, and a value it too rejects surfaces via the DLQ as before — and an unresolvable zone (no tzdata embedded — named zones resolve from the runtime's zone database, which the bundled distroless images ship) warns and passes through, never a failed refresh or a silent UTC reinterpretation that would move instants (`Etc/UTC` is special-cased to UTC); ingest never rejects a record over its timestamp spelling — fail-closed enforcement of the canonical form is the stream row-filter's (#381). The worker's `INSERT … FORMAT JSONEachRow` pins `date_time_input_format=best_effort` — the server default since ClickHouse 26.5, and on older servers the `basic` default rejects the canonical form's zone suffix (verified live); a superset of `basic` either way, so pre-upgrade messages still in NATS — zone-less strings, Unix numbers — parse exactly as before. Boundaries: `Date`/`Date32` columns pass through untouched (day precision, no zone ambiguity on this path); events published before the upgrade replay in their original spelling until the sweeper retires them; a column with an explicit non-UTC zone renders as `Z` on both paths (`/v1/query` likewise normalizes `DateTime` values to UTC), so the declared zone affects how zone-less inputs are read but never the output spelling — byte-identical regardless of zone. +- **SSE row `DateTime` columns now arrive in canonical RFC 3339 UTC, matching `/v1/query`, instead of the producer's spelling** (`internal/discovery/timestamp.go` (new), `internal/discovery/{discovery.go,validation.go}`, `internal/api/ingest.go`, `internal/ingest/worker.go`, `docs/src/content/docs/{api.md,architecture.md}`, plus tests in `internal/discovery/{timestamp_test.go (new),discovery_test.go}`, `internal/api/{ingest_test.go,boot_chain_test.go}`, `tests/e2e/sdk/streaming.test.ts`): closes #372. The stream fans out the pre-insert payload verbatim, so a row `DateTime` reached SSE subscribers in whatever spelling the producer sent — typically the zone-less ClickHouse-native form — while `/v1/query` rendered the stored value as RFC 3339 `Z`; JavaScript's `Date.parse` reads a zone-less string as **local** time, so an SDK `liveQuery` stitching backfill + live landed the two paths hours apart (sign-flipping with the viewer's UTC offset — broke ordering and the "N min ago" label in the Stats live feed). Ingest now **canonicalizes** every `DateTime`/`DateTime64` column value to RFC 3339 UTC (fraction truncated to the column's precision, `time.RFC3339Nano`-trimmed — byte-matching `/v1/query`'s `time.Time` marshaling) after validation and before the NATS publish, so the one payload every consumer shares — SSE subscribers, the ClickHouse insert, the DLQ — carries a single unambiguous spelling. Inputs stay liberal but are mirrored per column kind, exactly as ClickHouse reads them (#402 review): RFC 3339 with any offset (`.`-fractions only — ClickHouse has no `,` separator), `YYYY-MM-DD[ T]HH:MM:SS[.fff]`, `YYYY-MM-DD`, a Unix-seconds string of 9–10 digits (a fraction after it is honored only for `DateTime64` columns, parsed as an exact decimal and truncated at nine digits — never a `float64` round-trip, which corrupts nanoseconds and can round across the second; other digit lengths are ClickHouse's own forms — calendar `YYYYMMDD`/`YYYYMMDDhhmmss` or its 13/16/19-digit ms/µs/ns epochs — and pass through), and **integer** JSON numbers read the way ClickHouse reads them: Unix seconds for `DateTime`, **ticks at the column's scale** for `DateTime64` (the ms epoch `1750478400500` into a `DateTime64(3)` is a valid 2025 instant; an epoch-*seconds* number there is a 1970 instant — rewriting either as seconds would change what ClickHouse stores; non-integer numbers pass through, ClickHouse rejects them). Values whose instant lies outside the column type's range also pass through — ClickHouse *saturates* out-of-range values spelling-dependently (local time-of-day is kept while the date clamps), so only the producer's own spelling may be the one that saturates. Go `LoadLocation`'s `''`/`'Local'` environment quirks are rejected as unresolvable zones. A zone-less string is interpreted in the column's declared time zone, else the server's (discovered once per schema refresh via `SELECT timezone()`) — the same rule ClickHouse itself applies, so canonicalization never changes which instant is stored (differentially fuzzed against a live ClickHouse — ~35k generated inputs × six column shapes, raw vs canonicalized inserts, zero divergences — with every divergence class found along the way pinned in `tests/integration/timestamp_canonicalization_test.go`). Each timestamp column's spec (precision + resolved zone) is precomputed at schema discovery and cached on the column, so the per-record ingest path parses no type strings, loads no zones, and takes no locks. Canonicalization is **fail-open** (#402 review): a value outside the accepted forms publishes verbatim — ClickHouse's more liberal `best_effort` parser stays the arbiter of insertability, and a value it too rejects surfaces via the DLQ as before — and an unresolvable zone (no tzdata embedded — named zones resolve from the runtime's zone database, which the bundled distroless images ship) warns and passes through, never a failed refresh or a silent UTC reinterpretation that would move instants (`Etc/UTC` is special-cased to UTC); ingest never rejects a record over its timestamp spelling — fail-closed enforcement of the canonical form is the stream row-filter's (#381). The worker's `INSERT … FORMAT JSONEachRow` pins `date_time_input_format=best_effort` — the server default since ClickHouse 26.5, and on older servers the `basic` default rejects the canonical form's zone suffix (verified live); a superset of `basic` either way, so pre-upgrade messages still in NATS — zone-less strings, Unix numbers — parse exactly as before. Boundaries: `Date`/`Date32` columns pass through untouched (day precision, no zone ambiguity on this path); events published before the upgrade replay in their original spelling until the sweeper retires them; a column with an explicit non-UTC zone renders as `Z` on both paths (`/v1/query` likewise normalizes `DateTime` values to UTC), so the declared zone affects how zone-less inputs are read but never the output spelling — byte-identical regardless of zone. - **Live demo hero feed renders in `event_ts` order instead of SSE arrival order** (`docs/src/components/LiveDemo.astro`): the landing-page live activity feed prepended each streamed row to the top in the order the SSE stream delivered it, but a producer's webhook burst (a single merge-queue cycle fires ~20 events) arrives in no guaranteed order and the stream relays it in ingest order — so a late or out-of-order delivery landed above newer rows (e.g. a `pushed 12m ago` sitting on top of `reviewed a pull request 9m ago`). `addRow` now keeps the feed sorted by `event_ts` descending — it slots each row in before the first strictly-older sibling rather than blind-prepending — so the live tail matches the already-sorted `gh_activity_recent` backfill. The zone-less-SSE-timestamp normalization the sort relies on (`normTs`) was already in place; equal-second rows keep arrival order (`gh_events.event_ts` is only second-granular for CI/checks, so there's no finer tiebreak), and dedup + the `MAX_ROWS` trim are unchanged. Surfaced in dogfooding on `wavehouse.dev`; the client-side analog of the ingest-order reality the SSE stream can't reorder. diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index 6b15f65a..3215ccf5 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -217,7 +217,14 @@ The body is a **flat JSON object** whose keys must match column names in the tar - Type compatibility: `String`/`DateTime`/`UUID`/`Enum`/`IPv*` accept JSON strings; `Int*`/`Float*`/`Decimal` accept JSON numbers; `Bool` accepts JSON booleans or numbers; `Array` accepts JSON arrays; `Map`/`Tuple` accept JSON objects. - `Nullable()` and `LowCardinality()` wrappers are handled transparently. -**Timestamp canonicalization:** a `DateTime`/`DateTime64` column value sent as RFC 3339 (any offset; `.`-fractions only — ClickHouse has no `,` separator), `YYYY-MM-DD[ T]HH:MM:SS[.fff]` or `YYYY-MM-DD` (zone-less — interpreted in the column's time zone, else the ClickHouse server's, exactly as ClickHouse itself would), a Unix-seconds string of exactly 9–10 digits (a `.fff` fraction is honored only for `DateTime64` columns, as ClickHouse does; other digit lengths are ClickHouse's own forms — calendar shapes like `YYYYMMDD`, or its 13/16/19-digit ms/µs/ns epochs — so they pass through untouched), or an **integer** JSON number — read the way ClickHouse reads numbers: Unix seconds for `DateTime`, **ticks at the column's scale** for `DateTime64` (`1750478400500` into a `DateTime64(3)` is the millisecond epoch; non-integer numbers pass through, ClickHouse rejects them) — is rewritten to **one canonical wire form — RFC 3339 UTC** (`2026-06-21T04:00:00Z`, fraction truncated to the column's precision) before publishing, so the stored instant never changes but every consumer (the ClickHouse insert, [SSE subscribers](#get-v1stream--server-sent-events-stream), the DLQ) sees the same spelling `/v1/query` renders. **Fail-open**: a value in none of those forms — or whose instant lies outside the column type's range, where ClickHouse *saturates* spelling-dependently instead of erroring — is published verbatim: ClickHouse's more liberal parser decides insertability, and a value it too rejects surfaces via the DLQ, as before. Pass-through likewise applies when a time zone doesn't resolve at runtime — the binary embeds no tzdata, so named zones resolve from the runtime's zone database (the bundled distroless images ship one; a stripped-down custom runtime, or a server zone newer than the image's snapshot, may not resolve): an unresolvable *column* zone skips canonicalization for that column entirely, an unresolvable *server* default skips only zone-less values of columns without a declared zone — warned at schema refresh either way, and never guessed as UTC, which could move the stored instant. The accepted grammar is differentially tested against a live ClickHouse (raw and canonicalized spellings must insert identically, or both fail). `Date`/`Date32` columns pass through untouched. +**Timestamp canonicalization:** a `DateTime`/`DateTime64` column value sent in any of the forms below is rewritten to **one canonical wire form — RFC 3339 UTC** (`2026-06-21T04:00:00Z`, fraction truncated to the column's precision) before publishing, so the stored instant never changes but every consumer (the ClickHouse insert, [SSE subscribers](#get-v1stream--server-sent-events-stream), the DLQ) sees the same spelling `/v1/query` renders. The accepted input forms: + +- RFC 3339, any offset (`.`-fractions only — ClickHouse has no `,` separator). +- `YYYY-MM-DD[ T]HH:MM:SS[.fff]` or `YYYY-MM-DD`, zone-less — interpreted in the column's time zone, else the ClickHouse server's, exactly as ClickHouse itself would. +- A Unix-seconds string of exactly 9–10 digits (a `.fff` fraction is honored only for `DateTime64` columns, as ClickHouse does). Other digit lengths are ClickHouse's own forms — calendar shapes like `YYYYMMDD`, or its 13/16/19-digit ms/µs/ns epochs — so they pass through untouched. +- An **integer** JSON number — read the way ClickHouse reads numbers: Unix seconds for `DateTime`, **ticks at the column's scale** for `DateTime64` (`1750478400500` into a `DateTime64(3)` is the millisecond epoch). Non-integer numbers pass through; ClickHouse rejects them. + +**Fail-open**: a value in none of those forms — or whose instant lies outside the column type's range, where ClickHouse *saturates* spelling-dependently instead of erroring — is published verbatim: ClickHouse's more liberal parser decides insertability, and a value it too rejects surfaces via the DLQ, as before. Pass-through likewise applies when a time zone doesn't resolve at runtime — the binary embeds no tzdata, so named zones resolve from the runtime's zone database (the bundled distroless images ship one; a stripped-down custom runtime, or a server zone newer than the image's snapshot, may not resolve): an unresolvable *column* zone skips canonicalization for that column entirely, an unresolvable *server* default skips only zone-less values of columns without a declared zone — warned at schema refresh either way, and never guessed as UTC, which could move the stored instant. The accepted grammar is differentially tested against a live ClickHouse (raw and canonicalized spellings must insert identically, or both fail). `Date`/`Date32` columns pass through untouched. **The canonical form, precisely.** This is the one strict timestamp spelling in WaveHouse — the same one `/v1/query` renders and the SSE stream carries, and the form the stream row-filter will require for timestamp comparisons once row-level enforcement lands ([#381](https://github.com/Wave-RF/WaveHouse/issues/381)): diff --git a/tests/integration/timestamp_canonicalization_test.go b/tests/integration/timestamp_canonicalization_test.go index 2343a4e0..282ae6b4 100644 --- a/tests/integration/timestamp_canonicalization_test.go +++ b/tests/integration/timestamp_canonicalization_test.go @@ -64,25 +64,28 @@ func TestTimestampCanonicalization_DifferentialAgainstClickHouse(t *testing.T) { float64(1750478400), // integer number: seconds to DateTime, *ticks* to DateTime64 float64(1750478400.5), // non-integer number: ClickHouse rejects for every kind float64(1750478400500), // epoch-ms number: DateTime64(3)'s natural ticks shape - "20260711", // 8 digits: YYYYMMDD to ClickHouse - "20260711150000", // 14 digits: YYYYMMDDhhmmss - "202607111500", // 12 digits: ClickHouse rejects - "1752278400000", // 13 digits: epoch milliseconds to ClickHouse - "1750478400123456", // 16 digits: epoch microseconds to ClickHouse - "2026", // 4 digits: a year to ClickHouse - "1e9", // not a timestamp - "-100", // not a timestamp - "banana", // garbage - "2026-11-01 01:30:00", // DST fall-back: ambiguous local time - "2026-03-08 02:30:00", // DST spring-forward: nonexistent local time - "1960-01-01T00:00:00Z", // pre-range: ClickHouse saturates, spelling-dependently - "2300-06-30 12:30:00", // beyond DateTime64's ceiling, zone-less + json.Number("1750478400"), // integer seconds as the production-decoded type + json.Number("1750478400.5"), // non-integer json.Number: pass-through, ClickHouse rejects + json.Number("1750478400123456789"), // 19-digit ns epoch > 2^53: rewritten only on DateTime64(9) + "20260711", // 8 digits: YYYYMMDD to ClickHouse + "20260711150000", // 14 digits: YYYYMMDDhhmmss + "202607111500", // 12 digits: ClickHouse rejects + "1752278400000", // 13 digits: epoch milliseconds to ClickHouse + "1750478400123456", // 16 digits: epoch microseconds to ClickHouse + "2026", // 4 digits: a year to ClickHouse + "1e9", // not a timestamp + "-100", // not a timestamp + "banana", // garbage + "2026-11-01 01:30:00", // DST fall-back: ambiguous local time + "2026-03-08 02:30:00", // DST spring-forward: nonexistent local time + "1960-01-01T00:00:00Z", // pre-range: ClickHouse saturates, spelling-dependently + "2300-06-30 12:30:00", // beyond DateTime64's ceiling, zone-less } for _, ct := range colTypes { t.Run(ct.name, func(t *testing.T) { // Independent tables per shape; parallel keeps the six shapes from - // serializing ~350 single-row inserts against the suite timeout. + // serializing ~380 single-row inserts against the suite timeout. t.Parallel() rawTable := createTable(t, "id UInt32, v "+ct.ddl, "ORDER BY id") canonTable := createTable(t, "id UInt32, v "+ct.ddl, "ORDER BY id") From e52c3e302106acac51fd412381d9441ad37d59f7 Mon Sep 17 00:00:00 2001 From: taitelee Date: Thu, 16 Jul 2026 15:44:48 -0400 Subject: [PATCH 15/15] test: feed the CH oracle production json.Number epochs (>2^53); docs: bullet api.md input forms, six shapes --- .../timestamp_canonicalization_test.go | 32 +++++++++---------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/tests/integration/timestamp_canonicalization_test.go b/tests/integration/timestamp_canonicalization_test.go index 282ae6b4..c0463ed7 100644 --- a/tests/integration/timestamp_canonicalization_test.go +++ b/tests/integration/timestamp_canonicalization_test.go @@ -48,22 +48,22 @@ func TestTimestampCanonicalization_DifferentialAgainstClickHouse(t *testing.T) { // pass-through side: they must reach ClickHouse verbatim and get its // verdict, never a rewrite. corpus := []any{ - "2026-06-21T04:00:00Z", // canonical already - "2026-06-21T06:30:00+02:30", // offset form - "2026-06-21 04:00:00", // zone-less space form - "2026-06-21T04:00:00", // zone-less T form - "2026-06-21", // date-only - "2026-06-21 04:00:00.123456", // zone-less with fraction - "2026-06-21T04:00:00.9999Z", // fraction beyond column precision - "2026-06-21T04:00:00,999Z", // comma fraction: ISO 8601 yes, ClickHouse no - "1750478400", // 10-digit Unix string - "999999999", // 9-digit Unix string - "1750478400.5", // fractional Unix string: DateTime64-only to ClickHouse - "1750478400.123456789", // ns-exact fraction (float64 would corrupt it) - "1750478400.9999999995", // >9 fraction digits: ClickHouse truncates, never rounds - float64(1750478400), // integer number: seconds to DateTime, *ticks* to DateTime64 - float64(1750478400.5), // non-integer number: ClickHouse rejects for every kind - float64(1750478400500), // epoch-ms number: DateTime64(3)'s natural ticks shape + "2026-06-21T04:00:00Z", // canonical already + "2026-06-21T06:30:00+02:30", // offset form + "2026-06-21 04:00:00", // zone-less space form + "2026-06-21T04:00:00", // zone-less T form + "2026-06-21", // date-only + "2026-06-21 04:00:00.123456", // zone-less with fraction + "2026-06-21T04:00:00.9999Z", // fraction beyond column precision + "2026-06-21T04:00:00,999Z", // comma fraction: ISO 8601 yes, ClickHouse no + "1750478400", // 10-digit Unix string + "999999999", // 9-digit Unix string + "1750478400.5", // fractional Unix string: DateTime64-only to ClickHouse + "1750478400.123456789", // ns-exact fraction (float64 would corrupt it) + "1750478400.9999999995", // >9 fraction digits: ClickHouse truncates, never rounds + float64(1750478400), // integer number: seconds to DateTime, *ticks* to DateTime64 + float64(1750478400.5), // non-integer number: ClickHouse rejects for every kind + float64(1750478400500), // epoch-ms number: DateTime64(3)'s natural ticks shape json.Number("1750478400"), // integer seconds as the production-decoded type json.Number("1750478400.5"), // non-integer json.Number: pass-through, ClickHouse rejects json.Number("1750478400123456789"), // 19-digit ns epoch > 2^53: rewritten only on DateTime64(9)