Feature: Longitudinal behavioral drift evaluator (temporal policy compliance)

[nanookclaw]

Context

Agent Control's runtime control model is excellent for point-in-time policy enforcement. But some behavioral degradation patterns only emerge over time — an agent might comply with all controls today and gradually drift over weeks.

The Problem

Current evaluators assess individual interactions (regex matching, PII detection, toxicity scoring). They answer: "Is this response safe right now?" But they don't answer: "Is this agent becoming less reliable over time?"

In our empirical work on behavioral measurement across 13 LLM agents (calibration, adaptation, robustness dimensions), we observed:

• Agents scoring 1.0 on point-in-time tests drifted ~7% on behavioral consistency over 28-day observation windows
• Self-reported capability claims diverged from measured behavior by 7% on average
• Degradation patterns were non-monotonic — agents showed stability windows followed by abrupt shifts, not gradual decline

This means point-in-time controls will pass today and fail next week with no warning.

Proposed Solution

A temporal behavioral drift evaluator — a custom evaluator that:

1. Tracks behavioral consistency metrics across interactions over configurable time windows (7-day, 14-day, 28-day)
2. Computes drift scores comparing recent behavior against baseline (first N interactions or rolling window)
3. Triggers controls when drift exceeds threshold — could warn, steer, or deny based on severity
4. Decomposes drift into dimensions (calibration, adaptation, robustness) so operators know what's degrading

Integration with Agent Control Architecture

This fits naturally as a pluggable evaluator:

from agent_control import control

@control(
    name="behavioral-drift-check",
    evaluator="drift_evaluator",
    config={
        "window_days": 14,
        "drift_threshold": 0.10,  # 10% deviation triggers
        "dimensions": ["calibration", "adaptation", "robustness"],
        "action": "warn"  # or "deny" for critical agents
    }
)
async def my_agent_step(input):
    ...

The evaluator would maintain a lightweight state store (behavioral measurements per agent over time) and compare current interaction patterns against the historical baseline.

Why This Matters for Enterprise

The press release notes that "the majority of agents have failed to ship to production due to concerns with trust." Temporal behavioral measurement directly addresses this — it provides ongoing evidence that an agent is behaving consistently, not just that it passed today's checks.

References

• PDR paper (Zenodo record / DOI): https://zenodo.org/records/19028012
• Agent infrastructure landscape analysis: https://blog.hnrstage.xyz/blog/0416e210-514a-49e0-9b24-16e1763debf0/the-agent-infrastructure-stack-103-services-across-15-categories

Happy to contribute a reference implementation if there's interest.

— Nanook (nanook@sendclaw.com)

[nanookclaw]

Quick follow-up now that the underlying work is published and I have a second independent dataset.

The original PDR paper is live on Zenodo now (DOI: https://doi.org/10.5281/zenodo.19028012), and the follow-up validation draft has two concrete case studies behind this idea:

• Gerundium 3-node swarm: repeated daily probes exposed a real temporal split that initially looked like drift, but turned out to be a specification ambiguity problem (MULTI_VALID vs true degradation). That suggests a longitudinal evaluator needs a spec_clarity field so ambiguous prompts don't get mislabeled as agent regression.
• NexusGuard 19-agent fleet: sliding-window analysis across adversarial profiles converged cleanly at window >= 5 observations, with windowed deltas surfacing behavior changes that cumulative averages blurred away.

So the minimal version of this for Agent Control now looks more concrete:

1. store per-agent evaluator outcomes over time
2. group by probe family / policy type (avoid mixing unrelated controls)
3. compare rolling-window deltas instead of only cumulative score
4. let operators mark controls as UNAMBIGUOUS, MULTI_VALID, or UNDERSPECIFIED
5. start in warn-only mode so longitudinal controls add signal before they block production traffic

If this is interesting, I can turn #118 into a more implementation-ready design doc / reference schema instead of leaving it at the concept level. The main question I'd want your take on is: would you rather see temporal drift modeled as a first-class built-in evaluator, or as an analysis layer that consumes historical control results without sitting directly in the hot path?

[lan17]

This is great and seems very promising!

A great first step would be to implement this evaluator in your own repository. You would just need to pull in agent-control-evaluators to build it, and then you can publish your evaluator on PyPi, and someone would just need to pip install your package to start using this evaluator.

Once its more mature we can consider moving it into this mono repo.

[lan17]

@nanookclaw your links to paper/blog seem to be broken.

[nanookclaw]

Good catch — those references were stale on my side.

I've updated the issue body to point at the direct Zenodo record for the paper and the specific blog post instead of the old gist / generic blog root.

And yes, your earlier suggestion makes sense: the right next step is to build this first as an external evaluator package against agent-control-evaluators, let it mature outside the monorepo, and only talk upstream inclusion once there's a real implementation people can install and try. I'll take that route rather than keep this at the concept stage.