fluree · bplatz · Jul 5, 2026 · Jul 3, 2026 · Jul 3, 2026 · Jul 3, 2026
diff --git a/.fluree-memory/repo.ttl b/.fluree-memory/repo.ttl
@@ -906,7 +906,7 @@ mem:fact-01kn800ps56as4xe2kycr10yq1 a mem:Fact ;
     mem:createdAt "2026-04-02T21:01:33.861145+00:00"^^xsd:dateTime .
 
 mem:fact-01knjakv2d3t3y7av973g38tm7 a mem:Fact ;
-    mem:content "Incremental indexing has two notable gotchas in `fluree-db-indexer`: 1) when incremental fulltext refresh cannot fetch/decode the prior fulltext arena, `build/incremental.rs` starts from `FulltextArena::new()` and then writes a replacement arena built from novelty only, which can drop historical `fulltext()` scoring data for that `(g_id,p_id)`; 2) incremental schema refresh carries forward the base schema if base-store load fails, which leaves `snapshot.schema` stale for reasoning until a later successful refresh/rebuild. Also, `IndexerConfig.incremental_max_commit_bytes` defaults to `None`, and only the embedded orchestrator populates it from `reindex_max_bytes`." ;
+    mem:content "Incremental indexing gotchas in `fluree-db-indexer`: 1) fulltext prior-arena fetch/decode failure now HARD-FAILS the incremental build (incremental.rs ~1482; recovery contract = build_index_for_ledger falls back to full rebuild) — the old silent start-from-empty fallback that dropped historical fulltext scoring is FIXED; 2) incremental schema refresh carries forward the base schema if base-store load fails, which leaves `snapshot.schema` stale for reasoning until a later successful refresh/rebuild. Also, `IndexerConfig.incremental_max_commit_bytes` defaults to `None`, and only the embedded orchestrator populates it from `reindex_max_bytes`." ;
     mem:tag "fulltext" ;
     mem:tag "incremental-indexing" ;
     mem:tag "indexer" ;
@@ -1662,6 +1662,84 @@ mem:fact-01ktyzzbs4ewxqpgvssvj72tt5 a mem:Fact ;
     mem:createdAt "2026-06-12T22:41:50.884413+00:00"^^xsd:dateTime ;
     mem:rationale "Continuation point for the decimal fix series; the remaining priority-4 cluster (canonical identity) is the prerequisite for issues #1324/#1325." .
 
+mem:decision-01kwphem9r3gvcvj2j5842ddt6 a mem:Decision ;
+    mem:content "Fulltext positioning: f:fullTextDefaults (#config graph) is the RECOMMENDED path — values keep standard xsd:string/rdf:langString datatypes (external RDF consumers see ordinary literals) and language-tagged values get per-language analyzers. The @fulltext datatype REPLACES the stored datatype with Fluree-specific f:fullText, so docs/guidance must not push it as the default; it's a quick-start for siloed databases or properties orthogonal to the core data model. Docs repositioned 2026-07 (fulltext.md, indexing-and-search README, cookbook-search, concepts/datatypes)." ;
+    mem:tag "datatypes" ;
+    mem:tag "docs" ;
+    mem:tag "fulltext" ;
+    mem:tag "positioning" ;
+    mem:tag "rdf" ;
+    mem:scope mem:repo ;
+    mem:artifactRef "docs/concepts/datatypes.md" ;
+    mem:artifactRef "docs/guides/cookbook-search.md" ;
+    mem:artifactRef "docs/indexing-and-search/fulltext.md" ;
+    mem:branch "fix/fulltext-persisted-arena-lookup" ;
+    mem:createdAt "2026-07-04T12:25:30.680597+00:00"^^xsd:dateTime ;
+    mem:rationale "User-stated product intent (bplatz): don't steer standards-focused users to a custom datatype; @fulltext is only for silos/orthogonal properties. Future docs/examples/tutorials should default to the config path." .
+
+mem:fact-01kwn1qw4yhk6fy47xhmc9d7dm a mem:Fact ;
+    mem:content "fulltext() must handle BOTH binding shapes: BinaryScanOperator disables late materialization whenever overlay epoch != 0 (binary_scan.rs), so with ANY novelty every scanned value reaches eval_fulltext as Binding::Lit (p_id + lang in dtc), not EncodedLit. The Lit arm originally scored only @fulltext-datatype values → configured (f:fullTextDefaults) predicates returned unbound for EVERY row the moment one unindexed commit existed (Solo/S3 2026-07-03 bug; memory-mode tests passed because they queried exactly at index_t with empty novelty). Fixed on fix/fulltext-persisted-arena-lookup: Lit arm mirrors EncodedLit arena lookup. Persisted (file-backed, fresh-process) reproducers now in it_query_fulltext.rs." ;
+    mem:tag "binding" ;
+    mem:tag "bm25" ;
+    mem:tag "fulltext" ;
+    mem:tag "materialization" ;
+    mem:tag "novelty" ;
+    mem:tag "query" ;
+    mem:scope mem:repo ;
+    mem:artifactRef "fluree-db-api/tests/it_query_fulltext.rs" ;
+    mem:artifactRef "fluree-db-query/src/binary_scan.rs" ;
+    mem:artifactRef "fluree-db-query/src/eval/fulltext.rs" ;
+    mem:branch "fix/fulltext-persisted-arena-lookup" ;
+    mem:createdAt "2026-07-03T22:31:41.982333+00:00"^^xsd:dateTime ;
+    mem:rationale "Any future change to materialization gates or fulltext eval must keep both arms (EncodedLit and Lit) scoring configured predicates, or the field regression recurs invisibly (memory tests won't catch it)." .
+
+mem:fact-01kwn4gk9gz7vp31r6g9awth8g a mem:Fact ;
+    mem:content "fulltext() per-row eval is hot: PreparedFulltextQuery (eval/fulltext.rs) caches bucket analyzer choice, query stems, arena term_ids, IDFs, and effective stats per (store_id, epoch, to_t, g_id, p_id, lang_id, query) — moka + thread-local one-slot memo; per-row = doc_bow probe + one u32 binary search per term (50k rows: 1.45s→61ms, 24x). store_id in the key pins term_ids to the arena instance. Analyzer::shared(lang) is the per-language static (for_language CLONES the whole stopword set — never call per row). Materialized Lit rows resolve string_id→BoW instead of re-analyzing text. Next lever if still slow: posting-driven retrieval/WAND pushdown (graph-source BM25 already has WAND)." ;
+    mem:tag "bm25" ;
+    mem:tag "cache" ;
+    mem:tag "fulltext" ;
+    mem:tag "performance" ;
+    mem:tag "query" ;
+    mem:scope mem:repo ;
+    mem:artifactRef "fluree-db-api/tests/it_query_fulltext.rs" ;
+    mem:artifactRef "fluree-db-binary-index/src/analyzer.rs" ;
+    mem:artifactRef "fluree-db-query/src/eval/fulltext.rs" ;
+    mem:branch "fix/fulltext-persisted-arena-lookup" ;
+    mem:createdAt "2026-07-03T23:20:09.264477+00:00"^^xsd:dateTime ;
+    mem:rationale "Any future work on fulltext eval or analyzers must preserve the once-per-query invariant; the ignored fulltext_perf_50k_labels test is the regression harness." .
+
+mem:fact-01kwncjr1vh93xbsx3q4213w54 a mem:Fact ;
+    mem:content "Flat vector scoring: eval resolves args to borrowed slices — f64 from Lit/const, and indexed EncodedLit VECTOR_ID via BinaryIndexStore::vector_slice (zero-copy f32 shard slice, widened into thread-local scratch; decode fallback for ephemeral novelty handles) — and routes through vector_math SIMD kernels. FlakeValue::Vector is Arc<[f64]> (O(1) clones). KEY MEASUREMENT (50k x 256): indexed pipeline = 21ms vs novelty-resident = 137ms for the SAME query (6.5x) — keep vector corpora indexed; novelty-path floor is per-row operator machinery, not scoring/copies. Zero-copy shard read saved ~22% on indexed warm (27→21ms). Next step-change = top-k fast path over packed f32 shards (~4-8x more, exact, consistent) or HNSW routing. Harness: vector_flatrank_perf_50k (ignored, scalar-verified, novelty + indexed phases)." ;
+    mem:tag "flatrank" ;
+    mem:tag "performance" ;
+    mem:tag "query" ;
+    mem:tag "simd" ;
+    mem:tag "vector" ;
+    mem:scope mem:repo ;
+    mem:artifactRef "fluree-db-api/tests/it_vector_flatrank.rs" ;
+    mem:artifactRef "fluree-db-query/src/eval/vector.rs" ;
+    mem:artifactRef "fluree-db-query/src/eval/vector_math.rs" ;
+    mem:branch "fix/fulltext-persisted-arena-lookup" ;
+    mem:createdAt "2026-07-04T01:41:08.283260+00:00"^^xsd:dateTime ;
+    mem:rationale "Prevents re-optimizing eval when the flat-rank bottleneck is scan-side vector copying; documents that ComparableValue conversion of vectors is a full copy." .
+
+mem:fact-01kwnn592htcq4hcpjxtj1p69a a mem:Fact ;
+    mem:content "fast_vector_topk.rs: flat-rank top-k fast path — bind ?score dotProduct(?vec, const/1-row-VALUES) [+threshold/orderBy desc/limit, project {?s,?score}] runs as a direct shard scan: VectorArenaSnapshot pins shards once (per-row lookup_vector cache probe was the bottleneck capping gains at 2x), overlay-merging cursor rows, leaf/LEAFLET-refined subject partitions on rayon, ephemeral novelty vectors pre-decoded before the parallel section (ctx not Sync). Scores bit-identical (widen f32→f64, same dot kernel). 50k×256 indexed: 21ms→3.3ms (6.4x). GOTCHA: query VALUES vectors are NOT f32-quantized (only storage is) — ground truth = dot(quantized_row, raw_target)." ;
+    mem:tag "fast-path" ;
+    mem:tag "flatrank" ;
+    mem:tag "performance" ;
+    mem:tag "query" ;
+    mem:tag "topk" ;
+    mem:tag "vector" ;
+    mem:scope mem:repo ;
+    mem:artifactRef "fluree-db-api/tests/it_vector_flatrank.rs" ;
+    mem:artifactRef "fluree-db-binary-index/src/arena/vector.rs" ;
+    mem:artifactRef "fluree-db-query/src/execute/operator_tree.rs" ;
+    mem:artifactRef "fluree-db-query/src/fast_vector_topk.rs" ;
+    mem:branch "fix/fulltext-persisted-arena-lookup" ;
+    mem:createdAt "2026-07-04T04:11:04.145477+00:00"^^xsd:dateTime ;
+    mem:rationale "Extending the fast path (cosine/euclidean, hydration join, larger projections) starts here; snapshot + partition + overlay idioms are the reusable pieces. Parity harness: vector_topk_fast_path_parity." .
+
 mem:fact-01kta2t986nsjneban0hgbwzq7 a mem:Fact ;
     mem:content "RDF term/fact identity in Fluree is (s, p, o, dt, m) where m=FlakeMeta{lang, i} — matching Flake::eq/hash. Any dedup/stale-removal key that omits m collapses distinct facts: langString variants sharing a value (\"animal\"@en vs @fr), and @list members repeating a value at different indices. Issue #1273 was range::remove_stale_flakes (overlay-only/genesis read path) keying on (s,p,o,dt) only; the binary-scan stale-removal (binary_scan.rs FactKeyRef) already included m. Including m also correctly scopes retraction cancellation per variant." ;
     mem:tag "dedup" ;

diff --git a/docs/concepts/datatypes.md b/docs/concepts/datatypes.md
@@ -264,6 +264,14 @@ See [Vector Search](../indexing-and-search/vector-search.md) for complete docume
 
 **@fulltext** (full IRI: `https://ns.flur.ee/db#fullText`, prefix form: `f:fullText`) marks a string value for full-text search indexing. Values annotated with `@fulltext` are automatically analyzed (tokenized, stemmed, stopword-filtered) and indexed into per-predicate fulltext arenas during background index builds. This enables BM25-ranked relevance scoring via the `fulltext()` query function.
 
+Because the annotation *is* the stored datatype, tagged values are
+`f:fullText` literals rather than `xsd:string` — a custom datatype that
+external RDF consumers won't recognize. For standards-conformant data,
+prefer declaring searchable properties via `f:fullTextDefaults` in the
+ledger's `#config` graph, which full-text indexes plain `xsd:string` /
+`rdf:langString` values (with per-language analysis) without changing
+their datatypes.
+
 Without this type annotation, strings are stored as plain `xsd:string` values and support only exact matching and prefix queries -- not relevance-ranked full-text search.
 
 See [Inline Fulltext Search](../indexing-and-search/fulltext.md) for complete documentation.

diff --git a/docs/guides/cookbook-search.md b/docs/guides/cookbook-search.md
@@ -8,6 +8,15 @@ This guide covers practical patterns for both approaches.
 
 ### 1. Insert searchable data
 
+There are two ways to make string values searchable. The examples below use
+the `@fulltext` datatype annotation — the zero-config path, handy for a
+self-contained cookbook. Note that it stores values under the Fluree-specific
+`f:fullText` datatype instead of `xsd:string`; if you're building
+standards-conformant RDF (or want to search values already in your ledger,
+or need non-English analysis), declare the properties once with
+[`f:fullTextDefaults` in the ledger's `#config` graph](../indexing-and-search/fulltext.md#configured-full-text-properties-ffulltextdefaults)
+instead — the queries in this guide work identically with either setup.
+
 Annotate string values with `@fulltext` to make them searchable:
 
 ```bash

diff --git a/docs/indexing-and-search/README.md b/docs/indexing-and-search/README.md
@@ -25,8 +25,8 @@ Manual index rebuilding for recovery and maintenance:
 ### [Inline Fulltext Search](fulltext.md)
 
 Inline BM25-ranked text scoring. Two entry points, same query surface:
-- **`@fulltext` datatype** — per-value annotation (analogous to `@vector`), always English, zero config
-- **`f:fullTextDefaults` config** — declare properties + language once at the ledger level; supports 18 languages with Snowball stemming and per-graph overrides for multilingual setups
+- **`f:fullTextDefaults` config** (recommended) — declare properties + language once in the ledger's `#config` graph; values keep their standard `xsd:string` / `rdf:langString` datatypes, with 18 languages of Snowball stemming and per-graph overrides for multilingual setups
+- **`@fulltext` datatype** — per-value annotation (analogous to `@vector`), always English, zero config; stores values under the custom `f:fullText` datatype, so best suited to siloed databases or properties orthogonal to your core RDF model
 - `fulltext(?var, "query")` scoring function in `bind` expressions (same for both paths)
 - Automatic per-(graph, property, language) fulltext arena construction during background indexing
 - Unified scoring across indexed and novelty documents
@@ -277,8 +277,8 @@ let result = fluree.drop_full_text_index("products-search:main").await?;
 
 ### Vector Search
 
-- **Flat scan (inline functions)**: O(n) brute-force, viable up to ~100K vectors with binary indexing; binary index provides ~6x speedup over novelty-only scans and ~25x for filtered queries
-- **HNSW index**: O(log n) approximate nearest neighbor, recommended for 100K+ vectors or strict latency requirements
+- **Flat scan (inline functions)**: O(n) brute-force, exact and strongly consistent (novelty included). Queries matching the canonical top-k signature (`dotProduct` + threshold/order/limit over one vector predicate) run a parallel shard-scan fast path at ~6M vectors/sec (768-dim) — interactive to ~1M vectors; general-shape queries pay pipeline costs per row and stay interactive to ~100K
+- **HNSW index**: O(log n) approximate nearest neighbor, eventually consistent; recommended for multi-million-vector corpora, strict latency targets at any size, or query shapes the fast path doesn't serve
 - **Space**: ~1.5x embedding size
 - **Updates**: Incremental, O(1) per vector
 - See [Vector Search -- Performance and Scaling](vector-search.md#performance-and-scaling) for benchmark data and guidance on when to adopt HNSW
@@ -449,7 +449,7 @@ The `Bm25MaintenanceWorker` watches for source ledger commits and syncs indexes
 ### 1. Choose Appropriate Index Type
 
 - **Structured queries**: Use core graph indexes
-- **Keyword search (< 500K docs)**: Use inline `@fulltext` for zero-config BM25 scoring
+- **Keyword search (< 500K docs)**: Use inline fulltext (`f:fullTextDefaults` config; or the `@fulltext` datatype for siloed data) for BM25 scoring
 - **Keyword search (1M+ docs)**: Use the BM25 graph source for WAND-optimized top-k retrieval
 - **Semantic similarity**: Use vector search
 - **Hybrid**: Combine multiple indexes
@@ -512,7 +512,7 @@ Limit results for performance:
 ## Related Documentation
 
 - [Background Indexing](background-indexing.md) - Core index details
-- [Inline Fulltext Search](fulltext.md) - `@fulltext` datatype and `fulltext()` scoring
+- [Inline Fulltext Search](fulltext.md) - `f:fullTextDefaults` config, `@fulltext` datatype, and `fulltext()` scoring
 - [BM25](bm25.md) - Dedicated full-text search graph source
 - [Vector Search](vector-search.md) - Similarity search
 - [Graph Sources](../graph-sources/README.md) - Graph source concepts