[Backport 7.82.x] Add metric lookback DogStatsD retention substrate#53771
Open
lukesteensen wants to merge 1 commit into
Open
[Backport 7.82.x] Add metric lookback DogStatsD retention substrate#53771lukesteensen wants to merge 1 commit into
lukesteensen wants to merge 1 commit into
Conversation
Adds the metric lookback DogStatsD/retention substrate behind disabled-by-default configuration.
This PR introduces:
- shared metric lookback retention rings for scalar points and distribution sketches
- selected DogStatsD admission into lookback retention
- retention-backed lookback sender writer support
- monitor-based range evaluation over retained lookback points
- optional monitor partitioning by selected tag keys
- monitor mode configuration for disabled, dry-run, and enabled behavior
- egress policy/controller for forwarding retained time ranges to intake
- config/schema/template docs for the new substrate
- an Allium spec for the intended configuration → admission → monitor → egress behavior
This intentionally does **not** wire the check scheduler/shadow-check loading stack into retention. That final scheduler integration should land separately after the metric-lookback scheduler/shadow-check stack lands on `main`.
The monitor trigger metric is automatically admitted from DogStatsD when monitor mode is active:
```yaml
metric_lookback:
enabled: true
monitor:
mode: enabled
metric_name: my.trigger.metric
range_epsilon: 0.1
```
No separate DogStatsD enable flag is required.
Additional DogStatsD metrics can be retained with:
```yaml
metric_lookback:
enabled: true
dogstatsd:
metric_names:
- extra.metric.one
- extra.metric.two
```
`dogstatsd.metric_names` controls only additional retained DogStatsD metrics. The monitor metric does not need to be repeated there.
The monitor is controlled by:
```yaml
metric_lookback:
enabled: true
monitor:
mode: disabled
metric_name: ""
range_epsilon: 0
partition_tags: []
```
Valid modes:
```text
disabled
dry_run
enabled
```
```yaml
metric_lookback:
enabled: true
monitor:
mode: disabled
metric_name: ""
range_epsilon: 0
```
Behavior:
- monitor does not run
- monitor-driven egress does not run
- `monitor.metric_name` is not auto-admitted from DogStatsD
```yaml
metric_lookback:
enabled: true
monitor:
mode: dry_run
metric_name: my.trigger.metric
range_epsilon: 0.1
```
Behavior:
- monitor runs
- monitor metric is auto-admitted from DogStatsD
- monitor reads retained points
- monitor state initialization/transitions are logged
- egress starts forwarding
- monitor decisions do not suppress/reopen egress
```yaml
metric_lookback:
enabled: true
monitor:
mode: enabled
metric_name: my.trigger.metric
range_epsilon: 0.1
```
Behavior:
- monitor runs
- monitor metric is auto-admitted from DogStatsD
- monitor reads retained points
- monitor state initialization/transitions are logged
- egress starts suppressed
- breach/unknown opens egress
- healthy windows can suppress egress
The monitor evaluates retained points by metric name from the shared lookback ring.
Supported retained sources in this PR:
- DogStatsD bucketed points
- DogStatsD no-aggregation series
- check/shadow lookback sender points
The monitor classifies completed windows as:
- `unknown` when no partition has enough valid points
- `healthy` when at least one partition has enough valid points and no sufficient partition exceeds `range_epsilon`
- `breach` when any sufficient partition has `max-min > range_epsilon`
Partitioned monitor evaluation can be configured with:
```yaml
metric_lookback:
enabled: true
monitor:
mode: enabled
metric_name: my.trigger.metric
range_epsilon: 0.1
partition_tags:
- az
- instance_type
```
When `partition_tags` is empty, all retained points for the monitor metric are evaluated together.
When `partition_tags` is set, the monitor computes range independently per selected tag-value tuple. This avoids artificial range inflation when the same metric name has different normal baselines across tag populations.
Example:
```text
partition_tags: ["az"]
az:a values: 1.00, 1.01
az:b values: 2.00, 2.01
```
The global range would be `1.01`, but each partition range is only `0.01`, so the monitor remains healthy if `range_epsilon >= 0.01`.
Missing configured tags are retained under a stable missing-value partition rather than dropping the point.
Monitor state initialization and state transitions are logged with partition diagnostics:
```text
partition_tags=[az instance_type]
partition_key="az:us-east-1a,instance_type:m6i.large"
partition_count=12
```
In `mode: enabled`, monitor-backed egress starts suppressed. It opens forwarding when the monitor reports `breach` or `unknown`, and suppresses again after consecutive healthy monitor windows.
Forwarded ranges are clipped by send delay so only sufficiently old retained timestamps are sent. Successfully forwarded non-empty ranges are marked forwarded to avoid duplicates. Empty or failed sends remain retryable.
In `mode: dry_run`, egress starts forwarding and monitor decisions do not change egress state.
New/updated config keys:
```yaml
metric_lookback:
enabled: false
enabled_checks: []
collection_interval: 1s
capacity: 262144
shard_count: 16
dogstatsd:
metric_names: []
monitor:
mode: disabled
metric_name: ""
range_epsilon: 0
partition_tags: []
```
Environment variables:
```text
DD_METRIC_LOOKBACK_ENABLED
DD_METRIC_LOOKBACK_ENABLED_CHECKS
DD_METRIC_LOOKBACK_CAPACITY
DD_METRIC_LOOKBACK_SHARD_COUNT
DD_METRIC_LOOKBACK_DOGSTATSD_METRIC_NAMES
DD_METRIC_LOOKBACK_MONITOR_MODE
DD_METRIC_LOOKBACK_MONITOR_METRIC_NAME
DD_METRIC_LOOKBACK_MONITOR_RANGE_EPSILON
DD_METRIC_LOOKBACK_MONITOR_PARTITION_TAGS
```
This PR intentionally does **not** include:
- check scheduler shadow loading integration
- Python shadow sender routing
- scheduler shadow-loading safety gates
- final scheduler-to-retention glue
- changes to scheduler collection interval behavior
Those belong to the scheduler/shadow-check stack and a later small integration PR. `metric_lookback.collection_interval` may already exist on `main`; this PR does not introduce or wire scheduler collection cadence behavior.
Unit tests:
```bash
dda inv test --targets=./pkg/collector/metriclookback/...
dda inv test --targets=./pkg/aggregator
dda inv test --targets=./comp/aggregator/demultiplexer/impl
dda inv test --targets=./cmd/agent/subcommands/run
dda inv test --targets=./pkg/config/setup
```
Manual local Agent demo:
- Ran a locally built Agent from this branch.
- Configured:
```yaml
metric_lookback:
enabled: true
monitor:
mode: dry_run
metric_name: demo.lookback.trigger
range_epsilon: 0.1
```
- Did **not** configure `metric_lookback.dogstatsd.metric_names`.
- Sent DogStatsD gauge packets for `demo.lookback.trigger`.
- Observed the monitor state log:
```text
metric_lookback monitor state initialized: metric_name="demo.lookback.trigger" state=breach dry_run=true egress_mode=forwarding ... point_count=31 min=1 max=2 range=1 range_epsilon=0.1
```
This confirms the DogStatsD-origin monitor trigger metric is auto-admitted without additional DogStatsD metric configuration.
- Feature is disabled by default.
- Retention rings are allocated lazily after selected metrics are admitted.
- DogStatsD hot-path admission is exact-name filtered.
- Monitor mode has explicit disabled/dry-run/enabled states.
- Monitor partitioning prevents cross-tag population range inflation when configured.
- Egress controller lifecycle is stopped with the demultiplexer.
- Scheduler/shadow-loading files are intentionally excluded from this PR.
Co-authored-by: luke.steensen <luke.steensen@datadoghq.com>
(cherry picked from commit 5c1f2a5)
lukesteensen
added
changelog/no-changelog
Contributor
Go Package Import DifferencesBaseline: 8859577
|
This comment has been minimized.
This comment has been minimized.
evazorro
approved these changes
Jul 16, 2026
Contributor
Files inventory check summaryFile checks results against ancestor 88595771: Results for datadog-agent_7.82.0~rc.3.git.5.5b50b47.pipeline.125140221-1_amd64.deb:No change detected |
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: 8859577 Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | quality_gate_logs | % cpu utilization | +0.85 | [-0.12, +1.82] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_metrics_logs | memory utilization | +0.81 | [+0.55, +1.07] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_idle | memory utilization | +0.42 | [+0.36, +0.48] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_mean_fs_load | memory utilization | -0.18 | [-0.21, -0.14] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle_all_features | memory utilization | -0.18 | [-0.22, -0.14] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_no_fs_load | memory utilization | -0.33 | [-0.43, -0.23] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle | memory utilization | -0.43 | [-0.48, -0.38] | 1 | Logs bounds checks dashboard |
Bounds Checks: ✅ Passed
| perf | experiment | bounds_check_name | replicates_passed | observed_value | links |
|---|---|---|---|---|---|
| ✅ | quality_gate_idle | intake_connections | 10/10 | 3 ≤ 4 | bounds checks dashboard |
| ✅ | quality_gate_idle | memory_usage | 10/10 | 147.47MiB ≤ 154MiB | bounds checks dashboard |
| ✅ | quality_gate_idle | total_bytes_received | 10/10 | 733.05KiB ≤ 819.20KiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | intake_connections | 10/10 | 3 ≤ 4 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | 482.50MiB ≤ 495MiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | total_bytes_received | 10/10 | 1.12MiB ≤ 1.25MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | intake_connections | 10/10 | 3 ≤ 6 | bounds checks dashboard |
| ✅ | quality_gate_logs | memory_usage | 10/10 | 185.89MiB ≤ 195MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_logs | total_bytes_received | 10/10 | 264.17MiB ≤ 292MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | cpu_usage | 10/10 | 367.57 ≤ 2000 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | intake_connections | 10/10 | 3 ≤ 6 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | memory_usage | 10/10 | 415.19MiB ≤ 430MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | total_bytes_received | 10/10 | 0.93GiB ≤ 1.04GiB | bounds checks dashboard |
| ✅ | quality_gate_security_idle | cpu_usage | 10/10 | 27.94 ≤ 40 | bounds checks dashboard |
| ✅ | quality_gate_security_idle | memory_usage | 10/10 | 299.17MiB ≤ 330MiB | bounds checks dashboard |
| ✅ | quality_gate_security_mean_fs_load | cpu_usage | 10/10 | 62.10 ≤ 80 | bounds checks dashboard |
| ✅ | quality_gate_security_mean_fs_load | memory_usage | 10/10 | 279.21MiB ≤ 310MiB | bounds checks dashboard |
| ✅ | quality_gate_security_no_fs_load | cpu_usage | 10/10 | 23.90 ≤ 40 | bounds checks dashboard |
| ✅ | quality_gate_security_no_fs_load | memory_usage | 10/10 | 288.02MiB ≤ 320MiB | bounds checks dashboard |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
Replicate Execution Details
We run multiple replicates for each experiment/variant. However, we allow replicates to be automatically retried if there are any failures, up to 8 times, at which point the replicate is marked dead and we are unable to run analysis for the entire experiment. We call each of these attempts at running replicates a replicate execution. This section lists all replicate executions that failed due to the target crashing or being oom killed.
Note: In the below tables we bucket failures by experiment, variant, and failure type. For each of these buckets we list out the replicate indexes that failed with an annotation signifying how many times said replicate failed with the given failure mode. In the below example the baseline variant of the experiment named experiment_with_failures had two replicates that failed by oom kills. Replicate 0, which failed 8 executions, and replicate 1 which failed 6 executions, all with the same failure mode.
| Experiment | Variant | Replicates | Failure | Logs | Debug Dashboard |
|---|---|---|---|---|---|
| experiment_with_failures | baseline | 0 (x8) 1 (x6) | Oom killed | Debug Dashboard |
The debug dashboard links will take you to a debugging dashboard specifically designed to investigate replicate execution failures.
❌ Retried Profiling Replicate Execution Failures (ddprof)
Note: Profiling replicas may still be executing. See the debug dashboard for up to date status.
| Experiment | Variant | Replicates | Failure | Debug Dashboard |
|---|---|---|---|---|
| quality_gate_idle_all_features | baseline | 10 | Oom killed | Debug Dashboard |
| quality_gate_idle_all_features | comparison | 10 | Oom killed | Debug Dashboard |
| quality_gate_logs | baseline | 10 | Oom killed | Debug Dashboard |
| quality_gate_logs | comparison | 10 | Oom killed | Debug Dashboard |
| quality_gate_metrics_logs | baseline | 10 | Oom killed | Debug Dashboard |
| quality_gate_metrics_logs | comparison | 10 | Oom killed | Debug Dashboard |
| quality_gate_security_idle | baseline | 10 | Crashed (exit code: 134) | Debug Dashboard |
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_security_no_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_no_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport 5c1f2a5 from #53416.
Summary
Adds the metric lookback DogStatsD/retention substrate behind disabled-by-default configuration.
This PR introduces:
This also wires selected Go and Python checks into shadow execution and routes their output exclusively to the shared retention ring.
Behavior
DogStatsD lookback admission
The monitor trigger metric is automatically admitted from DogStatsD when monitor mode is active:
No separate DogStatsD enable flag is required.
Additional DogStatsD metrics can be retained with:
dogstatsd.metric_namescontrols only additional retained DogStatsD metrics. The monitor metric does not need to be repeated there.Monitor mode
The monitor is controlled by:
Valid modes:
mode: disabledBehavior:
monitor.metric_nameis not auto-admitted from DogStatsDmode: dry_runBehavior:
mode: enabledBehavior:
Monitor evaluation
The monitor evaluates retained points by metric name from the shared lookback ring.
Supported retained sources in this PR:
The monitor classifies completed windows as:
unknownwhen no partition has enough valid pointshealthywhen at least one partition has enough valid points and no sufficient partition exceedsrange_epsilonbreachwhen any sufficient partition hasmax-min > range_epsilonMonitor partition tags
Partitioned monitor evaluation can be configured with:
When
partition_tagsis empty, all retained points for the monitor metric are evaluated together.When
partition_tagsis set, the monitor computes range independently per selected tag-value tuple. This avoids artificial range inflation when the same metric name has different normal baselines across tag populations.Example:
The global range would be
1.01, but each partition range is only0.01, so the monitor remains healthy ifrange_epsilon >= 0.01.Missing configured tags are retained under a stable missing-value partition rather than dropping the point.
Monitor state initialization and state transitions are logged with partition diagnostics:
Egress
In
mode: enabled, monitor-backed egress starts suppressed. It opens forwarding when the monitor reportsbreachorunknown, and suppresses again after consecutive healthy monitor windows.Forwarded ranges are clipped by send delay so only sufficiently old retained timestamps are sent. Successfully forwarded non-empty ranges are marked forwarded to avoid duplicates. Empty or failed sends remain retryable.
In
mode: dry_run, egress starts forwarding and monitor decisions do not change egress state.Config
New/updated config keys:
Environment variables:
Shadow-check integration
Eligible Go and Python check instances are loaded as shadow checks according to the global and per-instance metric-lookback policy. Shadow checks run at
metric_lookback.collection_interval, with a minimum interval of one second, and send their output exclusively to the shared retention ring.Validation
Unit tests:
Manual local Agent demo:
Ran a locally built Agent from this branch.
Configured:
Did not configure
metric_lookback.dogstatsd.metric_names.Sent DogStatsD gauge packets for
demo.lookback.trigger.Observed the monitor state log:
This confirms the DogStatsD-origin monitor trigger metric is auto-admitted without additional DogStatsD metric configuration.
Safety
Backport notes
The automatic backport failed due to a conflict in
pkg/aggregator/BUILD.bazel. The conflict was resolved by preserving the release branch's existingdemultiplexer_serverless.goentry while addinglookback.go.Gazelle was regenerated against
7.82.x, retaining the release branch'sgo_testconvention forcomp/metriclookback/impl.Additional backport validation:
dda inv test --targets=./pkg/metriclookback/...,./pkg/collector/metriclookback/...,./pkg/aggregator,./comp/aggregator/demultiplexer/impl,./comp/metriclookback/...,./cmd/agent/subcommands/run,./pkg/config/setup dda inv agent.build --build-exclude=systemd dda inv agent.build --flavor=iot --build-exclude=systemd bazel run //:gazelle -- -mode=diffThe local fakeintake smoke test also passed, verifying timestamped DogStatsD triggering, approximately 1 Hz retained shadow output, identical metric identity, and overlapping normal/shadow timestamps.