vector-store: use CDCCheckpointSaver to expose last processed timestamp metric#503
Merged
Merged
Conversation
QuerthDP
force-pushed
the
enhance-cdc-metrics
branch
from
July 9, 2026 10:02
ec7ec71 to
f3d727a
Compare
QuerthDP
changed the title
QuerthDP
force-pushed
the
enhance-cdc-metrics
branch
4 times, most recently
from
July 9, 2026 10:40
cc47e5b to
5ad6647
Compare
There was a problem hiding this comment.
Pull request overview
Adds Prometheus instrumentation to make CDC reader health and progress observable per index/reader, addressing cases where CDC processing can stall silently.
Changes:
- Introduces CDC reader health/progress metrics (
cdc_reader_up, handler errors, restarts, and consumed-window checkpoint timestamp) and registers them with the existing Prometheus registry. - Plumbs
Metricsinto DB index creation so CDC actors can report per-index/per-reader series. - Adds unit tests validating metric export and label cleanup behavior.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| crates/vector-store/tests/integration/db_basic.rs | Updates test harness pattern matching for the extended Db::GetDbIndex message. |
| crates/vector-store/src/metrics.rs | Adds/registers new CDC metrics and extends label cleanup; adds export/cleanup tests. |
| crates/vector-store/src/engine.rs | Passes metrics into get_db_index so DB/CDC layers can emit per-index metrics. |
| crates/vector-store/src/db.rs | Extends Db::GetDbIndex/DbExt::get_db_index to carry Arc<Metrics> through the DB actor. |
| crates/vector-store/src/db_index.rs | Threads Arc<Metrics> into CDC actor construction (wide/fine). |
| crates/vector-store/src/db_cdc.rs | Implements CDC reader metrics (up/down, restarts, handler errors) and a checkpoint-saver that reports consumed-window progress. |
QuerthDP
added a commit
to QuerthDP/vector-store
that referenced
this pull request
Jul 9, 2026
remove_index_labels cleared latency/size/modified and the CDC metrics for a deleted index, but left indexing_lag_seconds behind, leaking a high-cardinality series per index over its lifetime. Reported by Copilot on PR scylladb#503.
QuerthDP
force-pushed
the
enhance-cdc-metrics
branch
from
July 9, 2026 11:48
5ad6647 to
f4d4255
Compare
QuerthDP
added a commit
to QuerthDP/vector-store
that referenced
this pull request
Jul 9, 2026
remove_index_labels cleared latency/size/modified and the CDC metrics for a deleted index, but left indexing_lag_seconds behind, leaking a high-cardinality series per index over its lifetime. Reported by Copilot on PR scylladb#503.
QuerthDP
force-pushed
the
enhance-cdc-metrics
branch
8 times, most recently
from
July 10, 2026 08:55
a397908 to
eb985a8
Compare
QuerthDP
changed the title
CDCCheckpointSaver to expose last processed timestamp metric
QuerthDP
force-pushed
the
enhance-cdc-metrics
branch
5 times, most recently
from
July 14, 2026 07:53
ce1b75c to
3dd49aa
Compare
QuerthDP
force-pushed
the
enhance-cdc-metrics
branch
2 times, most recently
from
July 14, 2026 09:03
d7a9286 to
83f3caf
Compare
QuerthDP
force-pushed
the
enhance-cdc-metrics
branch
from
July 14, 2026 09:29
83f3caf to
217814e
Compare
Member
Author
|
Changelog:
This PR is ready for review. Waiting for #515 to get in to rebase on that and open. |
ewienik
added a commit
to ewienik/vector-store
that referenced
this pull request
Jul 14, 2026
Adds Prometheus metrics for CDC reader health, as a follow-up to a recent incident where a CDC reader silently stopped processing changes without any visible signal. - `cdc_reader_up` - whether each index's CDC reader (wide/fine) is currently running or stopped. - `cdc_handler_errors_total` - CDC handler errors per index and reader. - `cdc_reader_restarts_total` - restart-after-backoff attempts per index and reader. Also fixes two issues found while testing this work: - a pre-existing race in the CDC handler shutdown path that could orphan the driver's per-stream fetch tasks indefinitely instead of stopping them. - a pre-existing leak of `indexing_lag_seconds` series on index deletion. Split from: scylladb#503 as it kept getting bigger. This patch will introduce the straightforward metrics that could be supported out of the box. The latter will include the `cdc_last_processed_timestamp_seconds` metric and a potential refactor of `db_cdc.rs`. Refs: VECTOR-722 Fixes: VECTOR-739
The actor loop and the CDC consumer are two fairly independent responsibilities that happened to share one large file. Splitting them into their own submodules keeps each one focused and makes room for adding more CDC-related code without the file growing without bound.
Cause-agnostic lag indicator: how far the CDC reader has consumed its log window (minimum across all tracked streams), reported through a `CDCCheckpointSaver` so it keeps advancing even without table writes. Resets on generation transitions.
QuerthDP
force-pushed
the
enhance-cdc-metrics
branch
from
July 14, 2026 14:04
217814e to
be20161
Compare
QuerthDP
marked this pull request as ready for review
July 14, 2026 14:05
ewienik
approved these changes
Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds last Prometheus metric for CDC reader health (after #515), as a follow-up to a recent incident where a CDC reader silently stopped processing changes without any visible signal.
The added metric is
cdc_last_processed_timestamp_seconds- a cause-agnostic lag indicator showing how far each CDC reader has consumed its log window (minimum across all tracked streams), reported through aCDCCheckpointSaverso it keeps advancing even when the table receives no writes. Resets on generation transitions.Also splits
db_cdcinto its own module, since the actor loop and the CDC consumer were two fairly independent responsibilities sharing one large file, and this metric needed its own submodule for thecheckpoint-saverwiring.Fixes: VECTOR-722