Slow API server website load times

[trau-sca]

Hi, we've been battling issues for the past few weeks on our api-server pod being very slow to display the website information (up to a minute to load the dags card screen). Sometimes the api server pod even restarts due to the liveness probe. I don't see any errors when it happens either.

We are running on airflow 3.1.0 and the 1.18.0 helm chart in an Azure Kubernetes cluster with an Azure PostgreSQL database.
No pods are going past their resource requests when this issue occurs, and the resources for the database are very low in usage.

I have disabled sqlalchemy pooling, and use pgbouncer in airflow as well as in the PostgreSQL database.
I have the following configuration for airflow:

pgbouncer:
    enabled: true
    maxClientConn: 500
    metadataPoolSize: 50
    resultBackendPoolSize: 10
    extraIni: |
      pool_mode = transaction

On PostgreSQL, I have pgbouncer's pool_mode set to session, and the default_pool_size is 100.

We have 16 DAGs, and have a daily job that runs a db clean for anything older than 90 days.

What can I do to troubleshoot this slow response time?

[potiuk]

I think the best approach would be to inspect your browser and see which requests are taking the long time and report an issue if tehre is a clear "winner".

[trau-sca]

Thanks for the tip, I looked into this and quite a few requests look long, but the biggest one is dags?dag_runs_limit=14&limit=50&offset=0&exclude_stale=true&order_by=-dag_display_name taking 51s, but even dags?is_favorite=true takes 4 seconds.

I then remoted into the dag-processor, did a curl command for the first one, and that also took 51s. I used the private endpoint assigned to the api-server while I was in the dag-processor.

[potiuk]

Could you please report an issue with it @trau-sca - we need to have an issue to track it down.

[rebel-wansoo]

I’m experiencing exactly the same issue and am using Aurora PostgreSQL 17.
My environment is identical. I’m using the same chart as you and running Airflow 3.1.0.

[rcampos87]

We have been experiencing the same since upgrading to airflow 3, we are running with mysql db.

[trau-sca]

I created #56635 for the issue