Skip to content

bug(sdk): liveQuery dedup boundary is wrong — duplicate and dropped events in the core streaming feature #396

Description

@taitelee

Summary

LiveQuery dedupes buffered live events against the historical snapshot using rows[rows.length - 1].received_timestamp as the boundary. That boundary is wrong in the documented usage, and the column it relies on isn't even in the table.

Detail

clients/ts/src/stream/live-query.ts:63-81:

  • The canonical docs example orders received_timestamp DESC, so the last row is the oldest → boundary is minimal → every buffered event passes and rows already in the snapshot are re-delivered via next().
  • The envelope received_timestamp is never inserted into the table (internal/ingest/worker.go inserts only the inner payload), and .select(...) projections drop it anyway → lastTimestamp is undefined → no dedup at all, despite the "deduplicated automatically" docstring (query-builder.ts:180-188).
  • When the column does exist, envelope timestamps are RFC3339Nano while a second-precision row serializes as ...:05Z; lexicographically "...:05Z" > "...:05.123Z", so buffered events in the same second as the boundary are wrongly discarded — silent loss.
  • On fetch error (:58-61) buffered live events are dropped, never delivered.

Fix direction

Dedup on a stable server-provided key/sequence rather than the envelope timestamp string; use Math.max over the fetched rows' order column (not positional last), and compare with parsed instants, not lexically.

Found in a repo-wide audit; verified by code + docs. Distinct from #372 (data-column zone drift) and #98 (replay-vs-live labeling).

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/dedupeDeduplication (Pebble, ScyllaDB)area/sdkTypeScript SDK (clients/ts/)bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions