Use persistent sorted sets for DB indexes

Author: tiensonqin

bench/bench_ocaml.ml

@@ -80,18 +80,6 @@ let bench config name f =
   in
   Printf.printf "%s\t%s\n%!" name (format_ms (median samples))
 
-let one =
-  { cardinality = One
-  ; unique = None
-  ; indexed = false
-  ; is_component = false
-  ; no_history = false
-  ; doc = None
-  ; value_type = None
-  ; tuple_attrs = None
-  ; tuple_types = None
-  }
-
 let indexed =
   { cardinality = One
   ; unique = None

datascript_ocaml.opam

@@ -7,6 +7,7 @@ license: "MIT"
 depends: [
   "ocaml" {>= "5.2.1"}
   "dune" {>= "3.17"}
+  "persistent_sorted_set_ocaml"
   "sqlite3"
   "js_of_ocaml-compiler" {with-test}
   "yojson" {with-test}

docs/design.md

@@ -0,0 +1,217 @@
+# DataScript OCaml Design
+
+This document records the current DB/index design and the parity contract with
+upstream DataScript's Clojure/ClojureScript implementation.
+
+Primary reference:
+
+- `/Users/tiensonqin/Codes/projects/datascript/src/datascript/db.cljc`
+
+The goal is semantic compatibility first. Performance work is acceptable only
+when it preserves DataScript's immutable DB value model, datom ordering, lazy
+public access, and comparator-bound index behavior.
+
+## DB Value Model
+
+DataScript DB values are persistent immutable values. A transaction must keep
+the old DB usable and produce a new DB with updated indexes. In upstream
+DataScript this is handled by persistent sorted set indexes:
+
+- `:eavt`
+- `:aevt`
+- `:avet`
+
+The OCaml port follows that model with:
+
+- `eavt_index : datom Persistent_sorted_set.t`
+- `aevt_index : datom Persistent_sorted_set.t`
+- `avet_index : datom Persistent_sorted_set.t`
+- `vaet_index : datom Persistent_sorted_set.t`
+
+`VAET` is explicit in the OCaml port because the public API exposes it directly
+for reverse-reference style access. Upstream derives reverse-reference behavior
+from indexed ref attrs; the OCaml port keeps a dedicated value-attribute-entity
+index for that public surface.
+
+The DB also keeps `datoms : datom list`. That list is the active fact set used
+by transaction code, serialization, and code paths that need fact-level
+membership independent of index order. It is not a replacement for the sorted
+indexes.
+
+## Index Construction
+
+Bulk DB construction sorts datoms with the same index comparators used by the
+public access paths, then builds a persistent sorted set:
+
+```ocaml
+PSet.of_sorted_array_by (Util.compare_datom index) items
+```
+
+This matches upstream `set/from-sorted-array` in `init-db`.
+
+Empty DB construction creates empty persistent sorted sets with the same
+comparators. Deserialization reconstructs indexes from serialized datoms through
+the normal refresh path; serialized data remains plain schema/datoms/history
+data, not serialized index internals.
+
+## Transaction Updates
+
+For append-only fast paths, the OCaml port updates each persistent sorted set
+with `Persistent_sorted_set.add`. This gives the same structural-sharing model
+as upstream `set/conj`: the old DB keeps the old root, and the new DB points to
+updated roots that share unchanged tree structure.
+
+The port still computes `history_datoms`, `unique_index`, and transaction
+reports separately:
+
+- `history_datoms` are chronological transaction facts, not an active sorted
+  public index.
+- `unique_index` is an auxiliary lookup cache for uniqueness checks, not one of
+  DataScript's ordered datom indexes.
+- transaction reports preserve `db_before`, `db_after`, and `tx_data` values as
+  immutable snapshots.
+
+## Index Order
+
+The OCaml datom comparators mirror upstream:
+
+| Index | Order |
+| --- | --- |
+| `Eavt` | entity, attribute, value, tx |
+| `Aevt` | attribute, entity, value, tx |
+| `Avet` | attribute, value, entity, tx |
+| `Vaet` | value, attribute, entity, tx |
+
+The value comparator is DataScript-aware. Numeric values compare numerically, so
+`Int 1` and `Float 1.0` are comparator-equal for index bounds. Exact AVET
+lookups must therefore return both facts when the only difference is numeric
+representation, matching upstream DataScript.
+
+## Index Access
+
+Upstream `IIndexAccess` uses `set/slice` and `set/rslice` for public index
+operations. The OCaml port follows the same shape:
+
+- `datoms` uses exact prefix slices when arguments form an index prefix.
+- `seek_datoms` uses a lower-bound slice for prefix-compatible bounds.
+- `rseek_datoms` uses a reverse upper-bound slice for prefix-compatible bounds.
+- `index_range` slices `AVET` from an attribute/value lower bound to an
+  attribute/value upper bound.
+
+Non-prefix combinations are still supported because the OCaml public API uses
+named optional arguments. When callers provide a combination that is not an
+index prefix, the implementation falls back to ordered index iteration plus a
+component filter. This preserves compatibility instead of pretending every
+named-argument combination is a native sorted-set range.
+
+Filtered DBs apply the filter predicate after slicing. This matches upstream
+`FilteredDB`, where the filter wraps `-datoms`, `-seek-datoms`,
+`-rseek-datoms`, and `-index-range`.
+
+## Bound Datoms
+
+Upstream bound datoms can contain `nil` components as wildcard-like comparator
+markers. OCaml `datom` fields are not optional, so the port uses synthetic bound
+datoms plus a bound-field mask. The custom slice comparator compares only the
+fields that participate in the requested bound.
+
+This is required for behavior such as:
+
+- `datoms db Aevt ~a`
+- `datoms db Eavt ~e ~a`
+- `datoms db Avet ~a ~v`
+- `seek_datoms db Avet ~a ~v`
+- `rseek_datoms db Vaet ~v ~a`
+- `index_range db attr ?start ?stop`
+
+The actual datoms stored in the persistent sorted sets are always normal
+datoms. Bound masks affect only the temporary comparator used by a slice.
+
+## Public Laziness
+
+Public `datoms` returns `Seq.t`. The implementation must not eagerly
+materialize more than required at API boundaries. The current persistent sorted
+set slice API returns lists, so a prefix slice materializes that bounded slice
+before converting it to `Seq.t`. This is still materially different from
+whole-index materialization, and it preserves the public lazy sequence contract
+for callers. If `Persistent_sorted_set` grows streaming slice cursors, the OCaml
+DB access layer should switch to them.
+
+## Schema and AVET Accessibility
+
+`AVET` contains datoms whose attributes are accessible through value lookup:
+
+- `:db/index true`
+- `:db/unique`
+- ref-valued attributes, matching the port's schema rules
+- tuple attributes that are installed as indexed tuple attrs
+
+The access layer validates `AVET` attribute access and raises the upstream-style
+message:
+
+```text
+Attribute :<attr> should be marked as :db/index true
+```
+
+Schema changes rebuild or update indexes through the same DB refresh/update
+paths, so access reflects the DB value produced by that transaction.
+
+## What Should Not Use PSS
+
+Not every list in the DB is an ordered persistent index.
+
+- `datoms` remains the active fact list for transaction logic and serialization.
+- `history_datoms` remains transaction history, not an active index root.
+- `unique_index` remains a lightweight uniqueness helper. It is not an
+  upstream PSS index and does not provide ordered public access.
+- query rows, pull results, transaction reports, schema data, and storage
+  payloads remain plain OCaml values.
+
+Moving these to PSS would not improve parity with upstream DataScript and would
+make serialization and equality behavior more complex.
+
+## Local Dependency
+
+The repo uses the sibling project:
+
+```text
+persistent-sorted-set-ocaml/lib -> ../../persistent-sorted-set-ocaml/lib
+```
+
+The bridge contains its own `dune-project` so Dune sees
+`persistent_sorted_set_ocaml` as a package while importing only the sibling
+library. It intentionally does not import the sibling tests or benchmarks into
+this workspace.
+
+## Verification Coverage
+
+Important tests covering this design:
+
+- `test_db__test_indexes_use_persistent_sorted_set`
+- `test_db__test_index_lookup_matches_upstream_numeric_comparator_bounds`
+- `test_datoms_returns_lazy_sequence`
+- `test_datoms_slices_before_filtered_predicate`
+- `test_vaet_index_returns_ref_datoms_by_value`
+- `test_incremental_writes_keep_public_datoms_indexes_correct`
+- `test_seek_datoms_scans_forward_from_index_tuple`
+- `test_rseek_datoms_scans_backward_from_index_tuple`
+- `test_seek_datoms_continues_across_avet_attributes`
+- `test_rseek_datoms_continues_across_avet_attributes`
+- tuple AVET lookup and range tests in `test_tuples.ml`
+
+Full `dune runtest` is the verification gate for this design. It includes the
+native test suite, SQLite storage tests when the opam environment resolves
+`sqlite3`, the JS smoke test, and cross-runtime parity checks against the sibling
+DataScript checkout.
+
+## Maintenance Rules
+
+- Keep `eavt/aevt/avet/vaet` backed by `Persistent_sorted_set.t`.
+- Use the index comparator for all index membership, bounds, and slices.
+- Preserve old DB values after every transaction.
+- Apply filtered DB predicates after index slicing.
+- Prefer bounded PSS slices over whole-index iteration whenever arguments form
+  an index prefix.
+- Keep non-prefix named-argument combinations correct, even if they require a
+  filtered ordered scan.
+- Do not serialize PSS internals as part of DB snapshots.

docs/perf.md

@@ -71,30 +71,32 @@ boundary. Added coverage checks that:
 - public `datoms` returns a lazy sequence
 - bounded datoms slicing happens before filtered-db predicate checks
 
-### Bounded Index Iteration
+### Persistent Sorted Indexes
 
-The DB stores sorted array snapshots for EAVT, AEVT, AVET, and VAET when index
-arrays are valid. Datoms lookup can then binary-search a bounded range and expose
-that range as a lazy sequence.
+The DB stores EAVT, AEVT, AVET, and VAET as `Persistent_sorted_set.t` values.
+This matches upstream DataScript's persistent sorted set model better than
+whole-index arrays because transactions must preserve the old immutable DB while
+producing a new DB with updated indexes.
 
-This avoids full-index filtering for common component-constrained accesses such
-as entity or attribute slices.
+Bulk construction sorts datoms once and builds each persistent sorted set from
+the sorted array. Incremental safe-add paths update the persistent sorted sets
+with structural sharing instead of marking whole indexes stale.
 
-Exact prefix slices such as `datoms db Aevt ~a` now compute both start and stop
-bounds up front and then lazily walk the array range by index. This avoids a
-per-datom predicate check inside the hot range.
+Exact prefix reads such as `datoms db Aevt ~a`, lower-bound reads such as
+`seek_datoms db Avet ~a ~v`, reverse reads such as `rseek_datoms`, and
+`index_range` use persistent sorted set `slice`/`rslice` bounds. Non-prefix
+named-argument combinations continue to fall back to ordered filtering for
+compatibility.
 
 ### Incremental Index Refresh
 
-Bulk transaction paths can merge newly added datoms into sorted indexes for
-initial loads. For later safe incremental writes into a non-empty DB, the write
-path updates active datoms and metadata immediately, but marks stored index lists
-and arrays stale instead of maintaining all four sorted indexes on every write.
+Bulk construction uses sorted-array PSS builders. For safe incremental writes,
+the write path updates the active datom list and adds new datoms into each
+relevant persistent sorted set. The old DB keeps its previous set roots, and the
+new DB shares unchanged tree structure with them.
 
-Public `datoms` still returns correct results. If stored indexes are stale, the
-read path builds the requested sorted index from current datoms and applies the
-requested component filters. A regression test covers incremental writes followed
-by public EAVT, AEVT, AVET, and VAET reads.
+A regression test covers incremental writes followed by public EAVT, AEVT,
+AVET, and VAET reads.
 
 ### Query Candidate Narrowing
 
@@ -168,21 +170,31 @@ These constraints must remain true for future performance work:
 ## Remaining Work
 
 The current deterministic benchmark goal is satisfied. Next useful work is to
-validate larger workloads, mixed read/write patterns after stale incremental
-indexes, and public behavior around direct DB record access. DB index fields are
-still present in the public record type, so future API cleanup should either make
-those internals private or clearly document their validity flags.
+validate larger workloads and mixed read/write patterns against the persistent
+sorted set implementation. DB index fields are still present in the public record
+type, so future API cleanup should either make those internals private or keep
+their PSS representation documented.
 
 ## Verification
 
-Latest full test command:
+Latest feasible native test command in the current environment:
 
 ```sh
-dune runtest
+dune runtest test/test_datascript.exe test/test_lru.exe test/test_conn.exe \
+  test/test_core.exe test/test_db.exe test/test_data_readers.exe \
+  test/test_built_ins.exe test/test_issues.exe test/test_entity.exe \
+  test/test_listen.exe test/test_lookup_refs.exe test/test_parser.exe \
+  test/test_parser_find.exe test/test_parser_query.exe \
+  test/test_parser_return_map.exe test/test_parser_rules.exe \
+  test/test_parser_where.exe test/test_pull_api.exe test/test_pull_parser.exe \
+  test/test_query_pull.exe test/test_query_namespace.exe test/test_tuples.exe \
+  test/test_serialize.exe test/test_storage.exe test/test_upsert.exe \
+  test/test_util.exe
 ```
 
-It passed after the current optimization set. The run emitted linker warnings
-about a missing `/opt/homebrew/opt/node@22/lib` search path, but no test failure.
+It passed after the persistent sorted set index change. Full `dune runtest`
+currently depends on Dune/Findlib resolving the `sqlite3` library for the SQLite
+storage tests.
 
 Latest cross-runtime benchmark command: