Airflow Incompatibility with Pylance/Pyright

[amcraig]

Apache Airflow version

3.1.0

If "Other Airflow 2/3 version" selected, which one?

This also occurs in 2.11

What happened?

This drives me nuts, Pylance doesn't seem to be able to pick up correct stubs/typing for Airflow.
Ruff doesn't have issues with the code.

I have ensured VSCode is using the correct PythonPath for my Airflow project.

Common issues include:

• XCom typing between Tasks
• ObjectStoragePath and UPath type differences
• DAG operators >> are flagged as not being supported for the tasks
• Complains about unused values in DAG definitions

Is there anything I can do to alleviate this, or am I faced with just ignoring/turning off my linter for Airflow development?

Here's an example DAG which throws Pylance linting error:

What you think should happen instead?

I'd expect that Airflow should work nicely with Pylance/Pyright.

How to reproduce

Just create a simple DAG like the one in the screenshot provided using ObjectStorage.
An easy linting error is to reproduce is to return a ObjectStoragePath from one task and consume that same ObjectStoragePath in the next task.
The Type of the ObjectStoragePath seems to be lost after passing through XComs

Operating System

MacOS

Versions of Apache Airflow Providers

apache-airflow-providers-google==18.0.0
apache-airflow-providers-standard==1.8.0

Deployment

Virtualenv installation

Deployment details

This is just experienced in local dev

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

Use Mypy instead. This is what we use, check and run in our CI.

There is a Mypy VS Code extension as far as I know that uses MyPy Daemon.

We have MyPy plugins that support those "Airflow" things - this is why it works nicer (despite all the MyPy shortcomings). And last time we checked MyPy is the only one that support plugins that allow us to implement some of the checks you mentioned. We spoke to both PyRefly (Facebook) and Ty (Astral) teams and while PyRefly team is kinda open to add plugins in the near fuiture, Ty will not have plugins for quite long time after it is out of beta.

But If you have other ways, it would be great to contribute it. We are also looking towards ty, pyrefly as replacements for mypy - but lack of plugin support that would allow us to make those things work made it so far not very appealing despite the speed. However maybe things changes - if you would like to invest in Pyrefly or Ty replacement, that would be a good contribution - both of them fully suport Language Server Specification so should nicely fit VSCode.

I believe - unfortunately - pylance/pyright does not support plugins, otherwise we would also welcome someone who really wants to use pyright to contribute plugins to support those things.

[potiuk]

Converted to a dicussion - if more discussion is needed.

[potiuk]

Mypy plugins here: https://github.com/apache/airflow/tree/main/dev/airflow_mypy

[amcraig]

Actually, quick question @potiuk :
What path do I use to include the Mypy Airflow Plugin.
Is it installable via the pip package, or do I have to clone the repo to point to a absolute path?

[potiuk]

I think you shoudl copy it to your project and use it as any other plugin - we do not have it packaged. We even have an open issue to make it easier to use - and it's up for grabs for anyone who will figure it out and contribute a better way. Contributions like that are always most welcome: #38796

[amcraig]

Makes sense.
I was stuck on thinking it was importable.
Thanks for the help

[amcraig]

Thanks for all the info and suggestions @potiuk that's just what I needed to know; really appreciated!
I'll have my team try swapping over to Mypy, and then investigate into Ty as it matures, as we're already in the Astral stack.

[ashb]

Try creating a virtual env with Airflow installed, and the explicitly setting the python interpreter to that venv'd python. That seems to do the trick for me with neovim and basedpyright.