-
Notifications
You must be signed in to change notification settings - Fork 16.3k
Prevent OpenLineage failure when BigQuery table metadata is missing #60099
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Prevent OpenLineage failure when BigQuery table metadata is missing #60099
Conversation
|
@KamranImaaz, do you think it might make sense to add a unit test or two to validate this new logic? |
Yes @jroachgolf84 currently working on it. Thanks for pointing it out. |
|
@jroachgolf84 I have added unit test to cover the case when _table is None. Requesting you to check it. |
kaxil
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PR title doesn't confirm to standards -- it doesn't say anything much.
@kaxil I have updated the title. pls check it. |
|
You need to rebase it and fix the errors |
Sure @potiuk |
Hi @potiuk I have gone through the CI Checks error specifically build documentation(docs only). So far what I have understood is that type annotations for execute method(ex: def execute()-> dict | None) is causing error. I'm i correct? |
|
You definitely have checks that can be fixed by installing Also you have to rebase and resolve the conflicts you have - so rebasing (also described in the contribution guide) is something you should start with. |
Yes I have rebased it and resolved the conflicts. Need to check the errors locally. Thanks! |
Fixes #60096
What
Adds a safety condition to OpenLineage extraction to prevent errors when
BigQueryUpdateTableSchemaOperator,BigQueryUpdateTableOperator, orBigQueryUpsertTableOperatordo not return a table resource.Also removes duplicate and unused imports and updates type annotations
to use
dict | Nonewhere applicable.Why
OpenLineage callbacks run after task execution and should not cause
task failures when metadata is unavailable.