Scheduler can lose executor callback events on database commit failure

[viiccwen]

Under which category would you file this issue?

Airflow Core

Apache Airflow version

main

What happened and how to reproduce it?

The Scheduler drains each executor's event buffer before the database transaction that records the events has committed. If that commit raises a transient database error, an executor callback completion event has already been removed from the buffer even though its Callback row remains in QUEUED or RUNNING.

Unlike task instances, executor callbacks have no recovery path that reselects these states. The callback can therefore remain active indefinitely.

To reproduce on main:

1. Create an ExecutorCallback in QUEUED state.
2. Add a matching successful CallbackKey event to an executor's event buffer.
3. Run the Scheduler's executor-event processing while making the event session's commit() raise an OperationalError.
4. Observe that the callback row is still QUEUED, while the original event buffer is empty.

This was reproduced against the production event-processing path with an actual transaction rollback.

What you think should happen instead?

Executor callback completion events should remain available until their state update commits. Transient DB errors should be retried without replaying task-event side effects. If the configured DB retries are exhausted, the callback events should return to their owning executor buffer and be retried by the next Scheduler loop without terminating the Scheduler.

Operating System

No response

Deployment

None

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

No response

Official Helm Chart version

Not Applicable

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct