fix: fulltext() scoring on configured predicates with novelty present, plus fulltext and vector scoring perf

[bplatz]

Problem

Field reports from Solo (Lambda + S3, ledger with ~212k configured rdfs:label values) showed fulltext(?v, "…") returning unbound for every row on an f:fullTextDefaults-configured predicate — indexed or not, lang-tagged or plain — even though the BM25 arena built correctly. A follow-up report measured ~27s wall time per query once scoring worked. While in the area, the inline (flat-rank) vector scoring path was audited for the same per-row inefficiencies.

Root cause (fulltext correctness)

BinaryScanOperator only uses late materialization (Binding::EncodedLit) when the overlay epoch is 0. The moment the ledger has any unindexed commit, every scanned row is eagerly materialized into Binding::Lit — and the Lit arm of eval_fulltext only scored @fulltext-datatype values, returning unbound for configured plain/lang-tagged strings. So inline full-text search worked only while a ledger was exactly caught up to its index and silently went dark one commit later. The existing memory-only reproducers always query at exactly index_t (empty novelty), which is why they kept passing while deployments with separate indexing/query processes kept regressing.

Fix: the Lit arm now mirrors the EncodedLit arm — resolve the arena bucket from the row's lang tag (dict-assigned lang_id; English bucket for untagged/@fulltext values) and score via unified BM25 with the novelty delta. No-arena semantics are unchanged: non-@fulltext strings stay unbound, @fulltext values keep the TF-saturation fallback. ExecutableQuery::uses_fulltext also now covers expression positions outside WHERE (ORDER BY binds, post-aggregation binds, HAVING) so the provider map is attached for those shapes.

Root cause (fulltext performance)

eval_fulltext runs once per bound row and repeated query-invariant work every time:

• Analyzer::for_language clones the language's entire stopword set per call — and was called per row
• the query string was re-tokenized and re-stemmed per row
• effective stats / df / IDF were recomputed per row, with two binary searches over the ~60k-string term dictionary per term (once for df, once for tf)
• materialized Lit rows re-analyzed the document text through the novelty path even when the doc was already in the arena

Fix:

• Analyzer::shared(lang) — process-wide per-language analyzers, built once
• PreparedFulltextQuery — bucket analyzer choice, query stems, arena term ids, IDF weights, and effective corpus stats computed once per (store_id, epoch, to_t, g_id, p_id, lang_id, query), moka-cached with a thread-local one-slot memo in front. store_id in the key pins prepared term ids to the arena instance they index into. Per-row scoring is a doc_bow probe plus one u32 binary search per query term. Scores are unchanged (same formula, same term order).
• Materialized-Lit indexed fast path: resolve the value's string_id (persisted dict, then dict novelty) and score from the precomputed arena BoW; only genuinely unindexed values analyze text

Measured (fulltext)

50k configured labels, file-backed, one unindexed commit to force the materialized-row path (the production shape), release build:

	per query	per row
before	1.45 s	~29 µs
after	61 ms	~1.2 µs

Extrapolated to the field's 211k labels, the ~27s query should drop to ~1–1.5s, dominated by the base scan itself. The next lever, if ever needed, is posting-driven retrieval (WAND top-k pushdown, as the graph-source BM25 already does) — a planner feature, deliberately out of scope here.

Vector (flat-rank) scoring

The inline vector functions (dotProduct, cosineSimilarity, euclideanDistance) had the same class of per-row waste, audited and fixed in two commits:

Zero-copy eval wired to SIMD — both arguments were resolved through ComparableValue, which allocs and copies the full vector per row (including the query vector, which never changes: ~12 KB of memcpy per row on both sides for a 1536-dim embedding). The math used plain iterator closures — cosine made three separate passes — while the runtime-dispatched SIMD kernels in eval::vector_math had no production callers at all. Now: arguments resolve to borrowed &[f64] slices straight from the binding or the expression constant, compute routes through the AVX/SSE2/NEON-dispatched kernels, and cosine_f64/f32 use three dispatched dot passes above the SIMD length threshold instead of one scalar fused pass.

Arc-backed FlakeValue::Vector — the core type held Vec<f64>, so every FlakeValue clone through flake → binding → batch → output (and through transact/novelty merges on vector-heavy ledgers) deep-copied the whole embedding. Now Arc<[f64]>: clones are refcount bumps everywhere. Serde stays a plain number array via a local adapter (wire form identical, covered by a round-trip test); FlakeValue ↔ ComparableValue vector conversions become free Arc clones. 21 files, almost entirely mechanical.

Zero-copy f32 shard reads for indexed vectors — indexed vector arguments (EncodedLit VECTOR_ID) reached eval through decode_value, which collects a fresh Vec<f64> + Arc per row even though the arena already hands out a zero-copy, shard-pinned f32 slice. Eval now borrows the packed f32 data via a new BinaryIndexStore::vector_slice accessor and widens into a reused thread-local scratch — same widened values, same f64 kernels, identical scores. Arena misses (ephemeral novelty handles) fall back to the decode path.

Measured (vector, honest numbers)

50k rows × 256-dim, release, results verified identical against scalar recomputation. Novelty-resident (materialized-row path):

function	baseline	after
dotProduct	163 ms	~137 ms
cosineSimilarity	205 ms	~139 ms
euclideanDistance	167 ms	~134 ms

Indexed (vector arena, late-materialized rows), same query and verification:

	baseline	zero-copy shard reads	+ top-k fast path
dotProduct, warm	27.0 ms	~21 ms	~3.3 ms (8×)

Novelty-resident gains are modest because that path's floor is per-row operator machinery (scan, binding construction, join, sort, format), not scoring or copies — measured directly by a 1024-dim run where the delta stays ~6%. The Arc change's primary value is the O(1)-clone guarantee for embeddings across transact/novelty/query and removing the foot-gun where any binding clone silently paid O(dims).

Flat-rank top-k fast path — the step-change. Detects the canonical similarity shape (single triple over a vector predicate + bind ?score dotProduct(?vec, target) + optional threshold / ORDER BY DESC(?score) / LIMIT, projecting {?s, ?score}) and executes it as a direct scan of the packed f32 shards instead of the operator pipeline: shards pinned once (VectorArenaSnapshot, zero per-row cache probes), rows from the shared overlay-merging cursor (novelty asserts appear, retractions cancel — same lane as the count fast paths), subject-range partitioned onto the global rayon pool with per-partition bounded top-k heaps, deterministic tie-break. Scores widen f32 → f64 through the same SIMD kernel as eval, so results are bit-identical to the pipeline; any uncertainty (non-vector rows, missing arena with novelty, dims mismatch, policy/multi-ledger/time-travel) bails to the generic plan. A dedicated parity test verifies indexed top-k, the overlay lane (novelty assert + retraction via upsert), multi-cardinality subjects, and fallback shapes against scalar ground truth.

Notable finding along the way: the indexed pipeline runs the identical flat-rank query 6.5× faster than novelty-resident (21 ms vs 137 ms per 50k rows) — keeping vector corpora indexed matters more than any eval tuning. Novelty-resident (arena-less) ledgers keep the pipeline path unchanged.

Test coverage

The fulltext suite was memory-only. New persisted-backend tests build in one Fluree instance, drop it, and query from a fresh instance over the same file storage (mirroring the split indexing/query deployment):

• full reindex + fresh-process reload scores untagged and @en probes
• incremental build fetches and extends the persisted arena; old and new values score
• unindexed novelty commit scores and indexed values keep scoring with novelty present (the failing reproducer for the correctness bug)
• language-tagged values build per-language arena buckets (French stemming, cross-bucket isolation) that round-trip the FIR6 root
• fulltext_perf_50k_labels and vector_flatrank_perf_50k — ignored perf smoke harnesses, run manually with --ignored --nocapture; the vector harness verifies hit counts, top-5 ranks, and scores against scalar recomputation of the same generated vectors, across both a novelty-resident and a fully-indexed phase
• vector_topk_fast_path_parity — scalar-ground-truth parity for the top-k fast path across its lanes (indexed base, overlay assert + retraction, multi-cardinality, fallback shapes)
• FlakeValue::Vector serde round-trip test (untagged enum + with adapter → plain number array)

Validation: full-workspace clippy (all features, all targets) clean; query-lib (1168), core (693), binary-index (331), transact (274), grp_query (303), and flatrank (16) suites pass.

[aaj3f]

Good fix!

[aaj3f]

For a detected query with LIMIT but no ORDER BY, the fast path still sorts by score-desc (order_desc || limit.is_some()) and returns the top-k by score, whereas the generic pipeline returns k arbitrary rows (scan/subject order). SPARQL leaves bare-LIMIT order unspecified, so this is spec-legal, but it silently changes which rows come back once this optimization triggers, and it breaks the PR's stated "bit-identical to the pipeline" invariant for that shape. Cleanest fix is to decline the shape in detection so it routes to the pipeline unchanged:

// in detect_vector_topk, after computing order_var and before building the spec:
// A bare LIMIT with no ORDER BY means "arbitrary k rows"; imposing top-k-by-score
// here would diverge from the pipeline. Only serve LIMIT alongside DESC(?score).
if query.limit.is_some() && order_var.is_none() {
    return None;
}

(If serving bare-LIMIT is intentional, please add a comment at line 536 stating the divergence is accepted.)

[bplatz]

Addressed in 0e1eb15

[aaj3f]

scan_partitioned returns Ok(None) for two semantically different outcomes: "too few partitions, fall back to the serial lane" (lines 647/673/684) and "a partition hit a genuine Bail, defer to the pipeline" (line 735). The caller (lines 526–532) maps every None to "try scan_serial", so a genuine bail triggers a full redundant serial re-scan that will hit the same bail row and then defer. Correct result, wasted work on the (rare) bail path. Distinguish the two, e.g.:

// return type -> Result<Option<Result<Vec<(u64,f64)>, Bail>>> or an enum:
//   Ok(None)            => too few partitions, try serial
//   Ok(Some(Err(Bail))) => a partition bailed, defer straight to pipeline
//   Ok(Some(Ok(rows)))  => rows
None => return Ok(Some(Err(Bail))), // was: return Ok(None)

and have the caller short-circuit to Ok(None) (pipeline) on the Bail variant instead of re-scanning serially.

[bplatz]

Addressed in 09b13d8

[aaj3f]

Fold::TopK eagerly pre-allocates BinaryHeap::with_capacity(need + 1) where need = limit.saturating_add(offset) (line 494) is a user literal. The realistic concern (verified): a large-but-legal limit (e.g. LIMIT 10000000 over a few thousand vectors) eagerly allocates one oversized heap per partition (up to MAX_PARTITIONS = 16, plus the merge heap and the serial-lane heap). The need + 1 overflow→debug-panic sub-case is real but pathological — it needs LIMIT ≈ 2^64, and release builds wrap to with_capacity(0) and stay correct. Clamp the pre-allocation to the data:

Some(n) => Fold::TopK {
    need: n,
    // cap the eager reservation; the heap grows if genuinely needed
    heap: BinaryHeap::with_capacity(n.saturating_add(1).min(4096)),
},

(The per-partition need could also be clamped to min(need, partition_row_estimate) in scan_partitioned, but a fixed cap is simplest.)

[bplatz]

Addressed in 0e1eb15

[aaj3f]

vector_topk_fast_path_parity exercises the detection-time bails (projected ?vec, extra hydration triple) but not the runtime Scorer::score → Err(Bail) → pipeline-fallback path — i.e. a non-vector object physically stored on the vector predicate with a bare {?s, ?score} projection (which passes detection, then bails mid-scan). vector_search_mixed_datatypes (line 399) has a non-vector value but projects it, so it never reaches the fast path. Add a case: insert a non-vector literal on the vector predicate, run the canonical {?s ?score} shape, and assert results equal the scalar ground truth (proving the runtime bail correctly defers to the generic plan and emits nothing partial).

[bplatz]

Addressed in 0e1eb15