Trigger queue delay metric can become negative after system clock adjustments

[viiccwen]

Under which category would you file this issue?

Airflow Core

Apache Airflow version

main

What happened and how to reproduce it?

The triggerer.trigger_queue_delay metric measures the duration between the TriggerRunnerSupervisor queueing a trigger workload and the TriggerRunner creating the trigger. Both timestamps currently use time.time().

If NTP, virtualization, or an administrator adjusts the system clock between those operations, the metric can become negative or be inflated even though the actual queue delay is normal.

Minimal reproduction:

1. Create a trigger workload with queued_at=100.0.
2. Simulate the wall clock moving backward so that time.time() returns 90.0 when the TriggerRunner creates the trigger.
3. Observe that the emitted queue delay is -10000 milliseconds.

This can also be reproduced in the existing test_create_triggers_emits_queue_delay_metric test by setting the workload's queue timestamp to 100.0, mocking time.time() to return 90.0, and expecting the real elapsed delay from a stable clock.

What you think should happen instead?

Queue delay is a duration and should use time.monotonic() at both the producer and consumer ends. System wall-clock djustments should not affect the emitted metric.

Operating System

No response

Deployment

None

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

No response

Official Helm Chart version

Not Applicable

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

The metric was introduced by #67927. A tested fix is ready that changes both timestamp sites to time.monotonic() and adds regression coverage for a backward wall-clock adjustment.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct