MWAA Tasks Running on EKS Become Zombie Tasks

[ss1u21]

I'm running multiple DAGs on MWAA (Managed Workflows for Apache Airflow), and these tasks execute on EKS. While the setup works fine most of the time, I occasionally encounter an issue where the logs stop being sent from EKS back to MWAA, or MWAA stops reading them.

As a result, the task gets marked as a zombie once the scheduler_zombie_task_threshold is reached, even though the task is still running in the background on EKS. When the task eventually finishes on EKS, it turns green in Airflow, but it was already marked as failed due to the zombie status.

Steps Taken:

I increased the scheduler_zombie_task_threshold to 20 minutes, which only temporarily mitigates the issue, but doesn't fully resolve it

Additional Information:

Scheduler CPU and memory utilization: both under 50%.

Scheduler logs:

[2024-10-08T11:17:28.212+00000] {{scheduler_job_runner.py:1751}} WARNING - Failing (1) jobs without heartbeat after 2024-10-08 10:57:28.178429+00:00

Detected zombie job: {'full_filepath': '/usr/local/airflow/dags/somefilepath/pipelines/florence_dag.py'}

Marking run <DagRun florence_dag @ 2024-10-08 06:15:00+00:00: scheduled__2024-10-08T06:15:00+00:00, state:running, queued_at: 2024-10-08 10:49:19.121177+00:00. externally triggered: False> failed

Question:

Why does mwaa scheduler is not able to read the logs from eks .Is there a solution for it ?

[enchant3dmango]

I guess it is just a permission issue. Have you solved this?

This docs might help.