vector-store: add CDC reader health metrics by QuerthDP · Pull Request #515 · scylladb/vector-store

QuerthDP · 2026-07-10T10:08:04Z

Adds Prometheus metrics for CDC reader health, as a follow-up to a recent
incident where a CDC reader silently stopped processing changes without
any visible signal.

cdc_reader_up - whether each index's CDC reader (wide/fine) is currently running or stopped.
cdc_handler_errors_total - CDC handler errors per index and reader.
cdc_reader_restarts_total - restart-after-backoff attempts per index and reader.

Also fixes two issues found while testing this work:

a pre-existing race in the CDC handler shutdown path that could orphan the driver's per-stream fetch tasks indefinitely instead of stopping them.
a pre-existing leak of indexing_lag_seconds series on index deletion.

Split from: #503 as it kept getting bigger.
This patch will introduce the straightforward metrics that could be supported out of the box.
The latter will include the cdc_last_processed_timestamp_seconds metric and a potential refactor of db_cdc.rs.

Refs: VECTOR-722
Fixes: VECTOR-739

Copilot

Pull request overview

This PR adds Prometheus metrics to surface CDC reader health per index (wide/fine), addressing cases where a CDC reader stops processing without an obvious signal.

Changes:

Add three new exported metrics: cdc_reader_up, cdc_handler_errors_total, and cdc_reader_restarts_total, including unit tests for export and label cleanup behavior.
Wire Arc<Metrics> through DB/index creation into the CDC actor so CDC readers can update these metrics and clean up their series on shutdown.
Update call sites and test match arms to accommodate the new metrics field in Db::GetDbIndex.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
crates/vector-store/tests/integration/db_basic.rs	Updates `Db::GetDbIndex` pattern matching to ignore the newly added message field.
crates/vector-store/src/metrics.rs	Defines/registers new CDC health metrics and adds tests for export/removal semantics.
crates/vector-store/src/engine.rs	Passes `metrics` into `get_db_index` during index creation.
crates/vector-store/src/db.rs	Extends `Db::GetDbIndex` / `DbExt::get_db_index` to carry `Arc<Metrics>` through to index creation.
crates/vector-store/src/db_index.rs	Threads `metrics` into CDC actor creation for each index’s readers.
crates/vector-store/src/db_cdc.rs	Implements CDC reader metric reporting (`up`, handler errors, restarts) and adds targeted tests.

`remove_index_labels` cleared `latency`/`size`/`modified` for a deleted index, but left `indexing_lag_seconds` behind, leaking a high-cardinality series per index over its lifetime.

QuerthDP · 2026-07-14T07:08:14Z

Changelog:

thread Metrics through db::new instead of get_db_index
use KeyspaceName and IndexName
remove unnecessary helper function
change set_reader_up(bool) to set_reader_metric_up() and set_reader_metric_down()
move the CDC reader lifetime commit to vector-store: stop dropping the CDC driver's supervisor future on shutdown #518
use CdcReaderState::new instead of tuple From impl

QuerthDP · 2026-07-14T09:16:17Z

Changelog:

fix cdc_reader_restarts_total metric helper to mention it counts attempts not restarts as requested vector-store: use CDCCheckpointSaver to expose last processed timestamp metric #503 (comment)

Passes the shared metrics registry down through the index-creation call path so CDC reader metrics can be recorded on it, rather than constructing or looking one up separately inside the CDC actor.

Give the wide/fine reader labels names (READER_WIDE, READER_FINE) instead of inline string literals, so every CDC metric can share the same label values.

Reports whether each index's CDC reader (wide/fine) is currently running or stopped, so an incident like a silently dead reader is directly observable. This is the first CDC-reader-scoped metric, so it also introduces a separate cleanup path for CDC reader labels. A reader's actor keeps running asynchronously for a while after its index is deleted, so removing its labels immediately would race with in-flight updates and just recreate the series. Each reader instead removes its own labels once it has fully, finally stopped.

Counts CDC handler errors per index and reader, alongside the existing internal `cdc-handler-errors` counter.

Counts restart after backoff attempts per index and reader, to distinguish a reader that recovers on its own from one stuck restarting.

QuerthDP · 2026-07-14T12:24:00Z

Changelog:

remove run_with_metrics, instead add Metrics as an argument to run
remove leftovers
use KeyspaceName and IndexName in record_* functions

…tamp metric (scylladb#503) Adds last Prometheus metric for CDC reader health (after scylladb#515), as a follow-up to a recent incident where a CDC reader silently stopped processing changes without any visible signal. The added metric is `cdc_last_processed_timestamp_seconds` - a cause-agnostic lag indicator showing how far each CDC reader has consumed its log window (minimum across all tracked streams), reported through a `CDCCheckpointSaver` so it keeps advancing even when the table receives no writes. Resets on generation transitions. Also splits `db_cdc` into its own module, since the actor loop and the CDC consumer were two fairly independent responsibilities sharing one large file, and this metric needed its own submodule for the `checkpoint-saver` wiring. Fixes: VECTOR-722

github-actions Bot added P2 area/vector labels Jul 10, 2026

QuerthDP force-pushed the enhance-metrics-simple branch 10 times, most recently from f711492 to cae49ac Compare July 13, 2026 10:33

QuerthDP requested a review from Copilot July 13, 2026 10:38

Copilot started reviewing on behalf of QuerthDP July 13, 2026 10:39 View session

Copilot AI reviewed Jul 13, 2026

View reviewed changes

Comment thread crates/vector-store/src/db_cdc.rs Outdated

QuerthDP force-pushed the enhance-metrics-simple branch from cae49ac to 78d4fce Compare July 13, 2026 11:03

QuerthDP marked this pull request as ready for review July 13, 2026 11:07

QuerthDP requested a review from ewienik July 13, 2026 11:07

ewienik requested changes Jul 13, 2026

View reviewed changes

vector-store: remove indexing_lag_seconds series on index deletion

82b4b99

`remove_index_labels` cleared `latency`/`size`/`modified` for a deleted index, but left `indexing_lag_seconds` behind, leaking a high-cardinality series per index over its lifetime.

QuerthDP force-pushed the enhance-metrics-simple branch from 78d4fce to 578d5a2 Compare July 14, 2026 07:03

QuerthDP requested a review from ewienik July 14, 2026 07:23

QuerthDP added the ai-assisted label Jul 14, 2026

QuerthDP mentioned this pull request Jul 14, 2026

vector-store: use CDCCheckpointSaver to expose last processed timestamp metric #503

Merged

QuerthDP force-pushed the enhance-metrics-simple branch from 578d5a2 to b7379dd Compare July 14, 2026 09:15

ewienik requested changes Jul 14, 2026

View reviewed changes

Comment thread crates/vector-store/src/lib.rs Outdated

Comment thread crates/vector-store/tests/integration/db_basic.rs Outdated

Comment thread crates/vector-store/src/db.rs Outdated

Comment thread crates/vector-store/src/db_cdc.rs Outdated

QuerthDP added 2 commits July 14, 2026 14:21

vector-store: thread Metrics into the CDC actor path

f4e177d

Passes the shared metrics registry down through the index-creation call path so CDC reader metrics can be recorded on it, rather than constructing or looking one up separately inside the CDC actor.

vector-store: name CDC reader label constants

dc158ee

Give the wide/fine reader labels names (READER_WIDE, READER_FINE) instead of inline string literals, so every CDC metric can share the same label values.

QuerthDP added 3 commits July 14, 2026 14:21

vector-store: add cdc_handler_errors_total metric

a95cc37

Counts CDC handler errors per index and reader, alongside the existing internal `cdc-handler-errors` counter.

vector-store: add cdc_reader_restarts_total metric

1e7a3ad

Counts restart after backoff attempts per index and reader, to distinguish a reader that recovers on its own from one stuck restarting.

QuerthDP force-pushed the enhance-metrics-simple branch from b7379dd to 1e7a3ad Compare July 14, 2026 12:22

QuerthDP requested a review from ewienik July 14, 2026 12:24

ewienik approved these changes Jul 14, 2026

View reviewed changes

ewienik added this pull request to the merge queue Jul 14, 2026

QuerthDP mentioned this pull request Jul 14, 2026

vector-store: add FTS metrics #519

Merged

Merged via the queue into scylladb:master with commit 17f9003 Jul 14, 2026
41 checks passed

QuerthDP deleted the enhance-metrics-simple branch July 16, 2026 07:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

vector-store: add CDC reader health metrics#515

vector-store: add CDC reader health metrics#515
ewienik merged 6 commits into
scylladb:masterfrom
QuerthDP:enhance-metrics-simple

QuerthDP commented Jul 10, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

QuerthDP commented Jul 14, 2026

Uh oh!

QuerthDP commented Jul 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

QuerthDP commented Jul 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

QuerthDP commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

QuerthDP commented Jul 14, 2026

Uh oh!

QuerthDP commented Jul 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

QuerthDP commented Jul 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

QuerthDP commented Jul 10, 2026 •

edited

Loading