Search before asking
Description
Doris currently offers approx_count_distinct / ndv, which is based on HyperLogLog. HLL supports only union — it cannot do clean intersection or difference on the serialized state.
This proposes adding uniq_theta, an approximate distinct-count aggregate backed by the Apache DataSketches Theta
Sketch (ported from ClickHouse's uniqTheta). Theta sketches add capabilities HLL lacks:
- Set operations on the serialized sketch — union / intersect / difference. This enables answering questions like "how many users appeared in both campaign A and B" directly from two
precomputed sketches, without rescanning raw data.
- Precompute-once, query-many — a theta sketch can be materialized as an
agg_state column and repeatedly merged / unioned / intersected across partitions, dimensions, or time
windows (funnels, retention, audience overlap).
- Cross-engine interoperability — feeding raw value bytes to the DataSketches hash (matching ClickHouse's
insertOriginal) keeps the inner sketch payload compatible with other
DataSketches-based engines (ES, ClickHouse, Spark, etc.), a common requirement when sketches are pre-aggregated in Hive and analyzed across systems.
- Documented error bound — ~3.125% relative error at 95% confidence with the default 4096 nominal entries; exact for small cardinalities.
The DataSketches dependency was already introduced by #63143 (which added datasketches_hll_union_agg and the contrib/datasketches-cpp submodule), and that PR explicitly noted theta
sketch as an easy follow-up. This proposal builds directly on that infrastructure.
Use case
Users migrating from ClickHouse expect uniqTheta parity. More generally, teams that pre-aggregate detailed data into Theta sketches in Hive (or another engine) need to load and combine
those sketches in Doris and get cross-engine-consistent cardinality with set operations — which HLL-based approx_count_distinct cannot provide.
Related issues
Are you willing to submit PR?
Code of Conduct
Search before asking
Description
Doris currently offers
approx_count_distinct/ndv, which is based on HyperLogLog. HLL supports only union — it cannot do clean intersection or difference on the serialized state.This proposes adding
uniq_theta, an approximate distinct-count aggregate backed by the Apache DataSketches ThetaSketch (ported from ClickHouse's
uniqTheta). Theta sketches add capabilities HLL lacks:precomputed sketches, without rescanning raw data.
agg_statecolumn and repeatedly merged / unioned / intersected across partitions, dimensions, or timewindows (funnels, retention, audience overlap).
insertOriginal) keeps the inner sketch payload compatible with otherDataSketches-based engines (ES, ClickHouse, Spark, etc.), a common requirement when sketches are pre-aggregated in Hive and analyzed across systems.
The DataSketches dependency was already introduced by #63143 (which added
datasketches_hll_union_aggand thecontrib/datasketches-cppsubmodule), and that PR explicitly noted thetasketch as an easy follow-up. This proposal builds directly on that infrastructure.
Use case
Users migrating from ClickHouse expect uniqTheta parity. More generally, teams that pre-aggregate detailed data into Theta sketches in Hive (or another engine) need to load and combine
those sketches in Doris and get cross-engine-consistent cardinality with set operations — which HLL-based approx_count_distinct cannot provide.
Related issues
Are you willing to submit PR?
Code of Conduct