Fix max_active_tis_per_dag for deferred task instances

[SakshamSinghal20]

closes: #61700

Description

Deferred tasks were previously excluded from max_active_tis_per_dag and max_active_tis_per_dagrun concurrency checks. This allowed an unlimited number of tasks to enter the DEFERRED state, bypassing configured limits.

This PR enforces these limits for DEFERRED task instances while ensuring that max_active_tasks (which limits worker slot usage) remains unaffected.

Changes:

• airflow/ti_deps/dependencies_states.py: Introduced TASK_CONCURRENCY_EXECUTION_STATES to include DEFERRED, RUNNING, and QUEUED.
• airflow/jobs/scheduler_job_runner.py: Updated ConcurrencyMap to count deferred TIs for task-level limits, but explicitly exclude them from dag_run_active_tasks_map.
• airflow/models/taskinstance.py: Updated get_num_running_task_instances to count deferred TIs for pre-execution dependency checks.

Tests:

Added unit tests covering:

• Deferred tasks blocking new scheduling attempts.
• Mixed state scenarios (Running + Deferred).
• Regression checks ensuring max_active_tasks is not impacted.

---

Was generative AI tooling used to co-author this PR?

• Yes (used for guidance and file locating)

---

• closes: max_active_tis_per_dag not enforced for deferred task instances #61700

[Nataneljpwd]

Looks good, I will check out the tests soon

[SakshamSinghal20]

Just let me know if everything is right or do i need to change anything else?

[SakshamSinghal20]

Hi @jscheffl and @Nataneljpwd! I'm a new contributor and I am very interested in participating in GSoC with this organization this year. I've been reading through the contributing guidelines, but I wanted to ask if there are any specific parts of the codebase you'd recommend I focus on to make a meaningful impact?

[jscheffl]

Hi @jscheffl and @Nataneljpwd! I'm a new contributor and I am very interested in participating in GSoC with this organization this year. I've been reading through the contributing guidelines, but I wanted to ask if there are any specific parts of the codebase you'd recommend I focus on to make a meaningful impact?

I'd take a look for https://github.com/apache/airflow/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22good%20first%20issue%22

[SakshamSinghal20]

@uranusjr I have changed the state to Active. also let me know if i need to do anything else before the PR gets merged.
Thank you.

[jscheffl]

@uranusjr I have changed the state to Active. also let me know if i need to do anything else before the PR gets merged. Thank you.

Discussion in Devlist is still ongoing. I think the outcome of this influenced the implementation strategy of this PR. Please do not merge before settled.

[jscheffl]

"Request changes" mainly to block merge as long as devlist discussion not finalized

[SakshamSinghal20]

"Request changes" mainly to block merge as long as devlist discussion not finalized

Understood.

[jscheffl]

"Request changes" mainly to block merge as long as devlist discussion not finalized

Understood.

Called for LAZY CONSENSUS in devlist in https://lists.apache.org/thread/qvnfj09qlktmbpdpcsgkbdb8pdmfzm77

Would it be OK to prepare the PR towards this so that in 3 days it is ready to be merged?
As this is quite important for me and our setup, let me know if I shall support in making the changes or fixes.

Reset "Request changes" and looking forward for completion!

[jscheffl]

Lazy consensus passed. WOuld it be possible to complete the PR for review by the next 24h?

[potiuk]

@SakshamSinghal20 This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

• ❌ Pre-commit / static checks: Failing: CI image checks / Static checks. Run prek run --from-ref main locally to find and fix issues. See Pre-commit / static checks docs.
• ❌ mypy (type checking): Failing: CI image checks / MyPy checks (mypy-airflow-core). Run prek --stage manual mypy-airflow-core --all-files locally to reproduce. You need breeze ci-image build --python 3.10 for Docker-based mypy. See mypy (type checking) docs.
• ❌ Other failing CI checks: Failing: Postgres tests: core / DB-core:Postgres:14:3.10:API...Serialization, MySQL tests: core / DB-core:MySQL:8.0:3.10:API...Serialization, Sqlite tests: core / DB-core:Sqlite:3.10:API...Serialization, Non-DB tests: core / Non-DB-core::3.10:API...Serialization, Integration and System Tests / Integration core kerberos (+3 more). Run prek run --from-ref main locally to reproduce. See static checks docs.
• ⚠️ Unresolved review comments: This PR has 2 unresolved review threads from maintainers: @jscheffl (MEMBER): 1 unresolved threads; @uranusjr (MEMBER): 1 unresolved threads. Please review and resolve all inline review comments before requesting another review. You can resolve a conversation by clicking 'Resolve conversation' on each thread after addressing the feedback. See pull request guidelines.

Note: Your branch is 1377 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

• The comment informs you what you need to do.
• Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
• There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates.
• Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack.