-
Notifications
You must be signed in to change notification settings - Fork 15.3k
AIP-82 Save references between assets and triggers #43826
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AIP-82 Save references between assets and triggers #43826
Conversation
947e028
to
598bc69
Compare
@Lee-W @uranusjr When working on it I realized that assets are added in the DB from DAG definition but never removed (or at least I did not see the code). Meaning, as a DAG author if I define an asset in my DAG and then later on remove it, the asset is never removed from the DB. Am I wrong? If not, is it intended? |
Yep, this is by design as of now. To keep the asset history. |
d27cab5
to
682b713
Compare
Alright, thank you. I handled it then. I removed the references from asset and triggers if the asset is no longer used |
@Lee-W Any chance you can review it? You have some experience around assets that could be interesting to have :) |
Sure thing :) Will take a look later today |
682b713
to
c2660ea
Compare
9543a51
to
c4c5c3e
Compare
30c05f1
to
de9386d
Compare
92074e2
to
cc80d52
Compare
cc80d52
to
0f9510d
Compare
Is there a plan to add tests separate ? |
Definitely! It is covered in #42515. However, I did not add unit test for my changes in |
ah okay make sense i didnt see that task. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. I have played around this table updates, and I believe that every DAG collection call verifies the asset relationships and removes any unused ones. Additionally, asset relationships are only valid during the trigger's lifespan.
This is my understand :) ?
This is correct. Just one clarification here. The trigger lifespan here is as long as the trigger is referenced as watcher of one asset. These triggers will basically live way longer than the current triggers today, those which are associated to a task. But as soon as you remove these references, these triggers will be removed |
cool, thank you :) |
Resolves #42510.
This PR adds a new attributes
watchers
to theAsset
class and saves references between assets and triggers in the DB. For example:This PR creates the trigger in the DB if it does not exist and save the reference between
asset
andtrigger
.^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.