vector-store: add FTS metrics

[QuerthDP]

Adds Prometheus metrics for full-text search (FTS) indexes, exposed via the existing /metrics endpoint.

New gauges, refreshed on scrape for indexes with pending writes:

• fts_index_size_bytes - in-memory size of a Tantivy index.
• fts_segment_count - number of tantivy segments in a Tantivy index.

Reused existing, index-type-agnostic instruments instead of duplicating them under an fts_ prefix:

• fts_query_latency_seconds -> request_latency_seconds (now also timed around the bm25 query handler)
• fts_query_count_total -> request_latency_seconds_count (comes for free with the histogram)
• fts_index_doc_count -> index_size
• fts_indexing_rate -> index_modified (raw counter; apply rate() in PromQL)
• fts_indexing_lag_seconds -> indexing_lag_seconds (only recorded for CDC-sourced writes, not full-scan indexing)

Deferred to a follow-up: fts_merge_duration_seconds. Tantivy has no completion hook for automatic background merges, so measuring it requires disabling auto-merge and driving/timing merges manually - a bigger, separate
change.

Fixes: VECTOR-629
Follow-up: VECTOR-743

[QuerthDP]

This PR is ready for review. Waiting for #515 and #503 to get in to rebase on them and open.

[Copilot]

Pull request overview

This PR extends the vector-store Prometheus metrics surface by adding FTS-specific index stats (size/segments) and wiring a shared Metrics instance through initialization so /metrics can refresh per-index stats on scrape. It also introduces new CDC reader instrumentation and refactors CDC handling into a dedicated db_cdc module.

Changes:

• Add new Prometheus series for FTS index size (bytes) and segment count, and refresh them during /metrics scrape via a new stats() request to the Tantivy FTS actor.
• Introduce CDC reader metrics (up/errors/restarts/last processed timestamp) and new db_cdc module components (consumer + checkpoint saver).
• Plumb a shared Arc<Metrics> through main/lib/db initialization and time BM25 requests via the existing latency histogram.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
crates/vector-store/tests/integration/db_basic.rs	Updates pattern matching for an expanded Db::GetDbIndex message.
crates/vector-store/src/metrics.rs	Adds new CDC + FTS metric vectors, registration, label cleanup helpers, and export tests.
crates/vector-store/src/main.rs	Creates a shared metrics instance and passes it into DB and HTTP server startup.
crates/vector-store/src/lib.rs	Adds run_with_metrics() / new_metrics() and threads metrics into new_db().
crates/vector-store/src/httproutes.rs	Refreshes FTS metrics on scrape and adds BM25 request timing via request_latency_seconds.
crates/vector-store/src/fts_index/tantivy.rs	Implements stats() to expose Tantivy doc/segment/space usage and adds tests.
crates/vector-store/src/fts_index/actor.rs	Adds FtsStats and a Stats actor message + sender extension method.
crates/vector-store/src/db.rs	Threads Arc<Metrics> into statement/index construction to support CDC/lag metrics.
crates/vector-store/src/db_index.rs	Passes metrics into CDC reader creation for each index.
crates/vector-store/src/db_cdc/mod.rs	Introduces the CDC module and standardizes reader names (wide/fine).
crates/vector-store/src/db_cdc/consumer.rs	Moves CDC row consumption/parsing into a dedicated consumer implementation.
crates/vector-store/src/db_cdc/checkpoint_saver.rs	Adds a checkpoint saver that updates cdc_last_processed_timestamp_seconds.
crates/vector-store/src/db_cdc/actor.rs	Adds CDC reader lifecycle metrics, restart counters, and checkpoint saver wiring.

Comments suppressed due to low confidence (1)

crates/vector-store/src/db_cdc/actor.rs:271

• The CDC reader restart logic claims to preserve the last checkpoint, but stop() sets self.start from the handler task result, and the handler task currently returns cdc_now(). If the reader is behind (non-zero indexing_lag_seconds), restarting from wall-clock time can skip unconsumed CDC entries and cause data loss. The restart timestamp should come from the CDC checkpoint/progress (e.g., the minimum checkpoint timestamp tracked by the checkpoint saver), not from cdc_now().

[QuerthDP]

@ewienik please review

[ewienik]

@knowack1 Do you have any comments?

[QuerthDP]

Changelog:

• moved refreshing index metrics to a separate function
• added fts metrics integration test
• changed metrics description to not expose the Tantivy backend internals
• added get_fts_index

[QuerthDP]

Changelog:

• split get_fts_index addition into separate commit