Improve Test OpenAPI client tests #44020
Labels
area:dev-env
CI, pre-commit, pylint and other changes that do not change the behavior of the final code
area:dev-tools
We should move test open API client tests to be run inside breeze.
The Open API client tests are pretty brittle. They run in "runner" directly and they are not using
breeze
image as the base for running them - which means that they need to install airflow in the runner directly. This is a bit problematic in general case - because sometimes changes are introduced in providers that require "main" airflow to use new providers (for example when FAB gets incompatible changes or async IO is tested.Then you need to not only install airflow but also build and install providers - you need to build the providers locally from sources and install some dependencies which might or might not cleanly install on "bare" runner environment
This has been somewhat mitigated by having a list of providers to install - but this is brittle, might change and some people find it "hard" to folllow - because they do not understand why sometimes they need to build and install those providers. Also what does not help is yaml keeping the scripts has some very unobvious problems where indentation might introduce unexpected end of lines etc.
When you attempt to build and install all providers, it's not easy sometimes and it will change over time as well - and when you use
uv
it tries to install and resolve all dependencies (including optional) so while it is way faster thanpip
it also tries to install and build some of the dependencies (like kerberos) that migh need some system-level tools to get installed.Attempt to do so has been done in #44007 and naive "build and install everything" ends up with:
That shows why we have CI image - because the CI image has all the necessary dependencies and base OS to install airflow + all dependencies and we keep it updated as airflow evolves and new dependencies are added.
Therefore it would be best is if the Test Open API client would be run in breeze (there we have pre-installed airflow with editable install for airflow, task_sdk and all providers so we only need to build and install python client).
This would require few things:
Also the whole TestOpenAPI client shoudl likely be moved to "special tests" from "basic tests" - becasause it has to wait for the CI image to be ready and Basic tests are run without waiting for the image.
The text was updated successfully, but these errors were encountered: