Description
If Airflow could provide native controls for limiting excessive Airflow UI/API usage per user or session.
In a production Airflow 3.x environment, we observed an incident where multiple users had many Airflow UI tabs open at the same time. Several tabs were repeatedly loading DAG grid, task instance, and log views. This generated repeated API requests and expensive metadata DB queries, including calls such as:
GET /api/v2/dags/~/dagRuns/~/taskInstances
GET /ui/grid/structure/...
GET /ui/grid/ti_summaries/...
Some API requests remained active for a very long time, with one task-instance API request taking approximately 75 minutes. During the incident, the API server became overloaded and the UI became extremely slow. Since Airflow 3.x workers also communicate task execution state through the API server, this also affected task-state transitions and caused task start/status updates to time out.
Use case/motivation
Airflow administrators need a way to protect the API server and metadata DB from excessive UI/API usage by a single user or a small group of users.
Common examples:
A user opens 10-20 Airflow UI tabs.
Multiple tabs refresh the same DAG grid or task-instance views.
Broad task-instance queries run across many DAGs or DAG runs.
Long-running UI/API requests consume API server workers and metadata DB sessions.
Worker execution API traffic is affected because UI/API traffic consumes shared API server capacity.
Today, this must be handled externally through ingress/proxy/database controls. Those controls are useful, but they are not Airflow-user-aware in many deployments and do not provide a clear UI message to the user.
Why This Matters
In Airflow 3.x, the API server is more central to both UI and task execution flows. If expensive UI/API requests overload the API server, task execution communication can also be impacted.
Native user-aware limits would help operators protect production environments without relying only on external load balancers, ingress rules, or database timeouts.
Related issues
No response
Are you willing to submit a PR?
Code of Conduct
Description
If Airflow could provide native controls for limiting excessive Airflow UI/API usage per user or session.
In a production Airflow 3.x environment, we observed an incident where multiple users had many Airflow UI tabs open at the same time. Several tabs were repeatedly loading DAG grid, task instance, and log views. This generated repeated API requests and expensive metadata DB queries, including calls such as:
Some API requests remained active for a very long time, with one task-instance API request taking approximately 75 minutes. During the incident, the API server became overloaded and the UI became extremely slow. Since Airflow 3.x workers also communicate task execution state through the API server, this also affected task-state transitions and caused task start/status updates to time out.
Use case/motivation
Airflow administrators need a way to protect the API server and metadata DB from excessive UI/API usage by a single user or a small group of users.
Common examples:
A user opens 10-20 Airflow UI tabs.
Multiple tabs refresh the same DAG grid or task-instance views.
Broad task-instance queries run across many DAGs or DAG runs.
Long-running UI/API requests consume API server workers and metadata DB sessions.
Worker execution API traffic is affected because UI/API traffic consumes shared API server capacity.
Today, this must be handled externally through ingress/proxy/database controls. Those controls are useful, but they are not Airflow-user-aware in many deployments and do not provide a clear UI message to the user.
Why This Matters
In Airflow 3.x, the API server is more central to both UI and task execution flows. If expensive UI/API requests overload the API server, task execution communication can also be impacted.
Native user-aware limits would help operators protect production environments without relying only on external load balancers, ingress rules, or database timeouts.
Related issues
No response
Are you willing to submit a PR?
Code of Conduct