Skip to content

Airflow Data Quality Provider Plugin#69413

Open
gopidesupavan wants to merge 19 commits into
apache:mainfrom
gopidesupavan:airflow-dq-provider
Open

Airflow Data Quality Provider Plugin#69413
gopidesupavan wants to merge 19 commits into
apache:mainfrom
gopidesupavan:airflow-dq-provider

Conversation

@gopidesupavan

@gopidesupavan gopidesupavan commented Jul 5, 2026

Copy link
Copy Markdown
Member

Adds a new apache-airflow-providers-dq provider,
DbApiHook-based data quality checks.

Airflow already has SQL check operators, and many users rely on them for data
quality today. This provider does not replace that path; it adds a small
DQRule / RuleSet layer for checks that need stable rule identity, persisted
history, and a connection to Airflow assets. That makes quality results easier
to inspect over time, lets downstream asset consumers gate on recent quality,
and also gives LLM-assisted workflows one schema to generate when proposing
checks from table context. Execution still goes through existing DbApiHook
connections.

Ships:

  • DQRule and RuleSet models for named data quality rules.
  • Built-in SQL checks for common table and column checks, executed through
    common.sql / DbApiHook, plus custom_sql for database-specific or more
    complex checks.
  • DQCheckOperator and the @task.dq_check TaskFlow decorator.
  • A configurable results backend under [dq] results_path for task, run, and
    rule-level history.
  • A read-only API and minimal UI plugin for viewing task/run results and rule
    history.
  • Experimental asset helpers, asset_quality() and require_quality(), that
    attach provider-owned quality metadata to assets without changing Airflow
    core.
  • Documentation and example Dags covering end-to-end usage with and without
    LLM-generated rules.
  • The provider addiitonally ships with dq authoring skills, that LLM's can use to generate rules based on the users need. SKILLS https://github.com/gopidesupavan/airflow/blob/f32940bd261b94238256eaced9150dd51329ce3e/providers/dq/src/airflow/providers/dq/skills/dq-rule-authoring/SKILL.md

This first version is intentionally small. It focuses on a deterministic rule
shape, SQL execution through common.sql, persisted results, and lightweight
visibility in the Airflow UI.

Design decisions:

  • Results are stored through an object-storage/local-file backend instead of
    adding new metadata DB tables in the first provider drop. This keeps the
    provider self-contained, avoids Airflow core migrations, and lets deployments
    choose a durable store such as S3/GCS/local files via [dq] results_path.
    The backend stores keyed JSON records for task runs, task instances, and
    per-rule history so the UI can read common views without scanning unrelated
    runs.
  • Asset support is implemented by extending assets with provider-owned metadata,
    not by changing Airflow core. Static quality configuration is attached to
    Asset.extra["airflow.dq"]; runtime summaries are attached to asset events
    under extra["airflow.dq.result"]. This lets users try asset quality gating
    now, while leaving room to discuss deeper asset integration later if the
    provider gets traction.
  • The first release starts with DbApiHook / SQL execution because Airflow
    already has strong provider coverage through common.sql. File and
    object-store data checks are left for a later iteration.

later iteration:

  • File/object-store based checks, where Airflow reads data from S3/GCS/local
    files or other object stores and runs quality rules directly against that
    data. This PR deliberately starts with the DbApiHook path first.
  • OpenLineage integration for data quality facets.
  • More checks implementations
  • Trigger support for DQ checks
  • DQProfileOperator

Was generative AI tooling used to co-author this PR?
  • Yes

Generated-by: following the guidelines



  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@gopidesupavan

gopidesupavan commented Jul 5, 2026

Copy link
Copy Markdown
Member Author

LLM generated rules and executed via dq

Screenshot 2026-07-05 at 13 29 04 Screenshot 2026-07-05 at 13 29 15

Overall view of the task executed rules and run history

Screenshot 2026-07-05 at 13 34 42 Screenshot 2026-07-05 at 13 34 57

@gopidesupavan
gopidesupavan force-pushed the airflow-dq-provider branch from 0432ee7 to 6acc4da Compare July 5, 2026 22:15
@gopidesupavan
gopidesupavan marked this pull request as ready for review July 6, 2026 15:56
@gopidesupavan
gopidesupavan force-pushed the airflow-dq-provider branch from 5c4aff2 to f32940b Compare July 6, 2026 15:56
@gopidesupavan
gopidesupavan requested review from kaxil and vikramkoka July 6, 2026 15:56
@gopidesupavan
gopidesupavan force-pushed the airflow-dq-provider branch from a8c6dfd to e0f7ff9 Compare July 7, 2026 17:38

@o-nikolas o-nikolas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This first version is intentionally small.

I'm not so sure that it is 😅 This is an absolutely enormous PR, 11.8k lines added across 143 files. This is quite difficult to review. Is it possible to ship this in smaller pieces?

@gopidesupavan

Copy link
Copy Markdown
Member Author

This first version is intentionally small.

I'm not so sure that it is 😅 This is an absolutely enormous PR, 11.8k lines added across 143 files. This is quite difficult to review. Is it possible to ship this in smaller pieces?

hehe it has both UI and backend shipped thats the reason it looks like big.. and some docs 😄

I am happy to ship this into two separate PR's one with UI and another backend provider specific..

@gopidesupavan

gopidesupavan commented Jul 7, 2026

Copy link
Copy Markdown
Member Author

@o-nikolas here is the part 1 #69575 hope its fine 😄 now.. as this is provider starting so it has autogenrated breeze docs and content in documentation ..

@gopidesupavan
gopidesupavan force-pushed the airflow-dq-provider branch 2 times, most recently from 5b30192 to 54ba129 Compare July 15, 2026 12:23
@gopidesupavan gopidesupavan changed the title Airflow Data Quality Provider Airflow Data Quality Provider Plugin Jul 15, 2026
@bbovenzi

Copy link
Copy Markdown
Contributor

Plugin UI is looking good. Would it be helpful to finally add plugins to the asset page to also show Data Quality there?

@gopidesupavan

Copy link
Copy Markdown
Member Author

Plugin UI is looking good. Would it be helpful to finally add plugins to the asset page to also show Data Quality there?

Yeah thought of that it seems currently it doesnt support?, are you referring to the changes in core to support plugins in asset page ?

@bbovenzi

Copy link
Copy Markdown
Contributor

Plugin UI is looking good. Would it be helpful to finally add plugins to the asset page to also show Data Quality there?

Yeah thought of that it seems currently it doesnt support?, are you referring to the changes in core to support plugins in asset page ?

I would open a PR to add support.

@gopidesupavan

Copy link
Copy Markdown
Member Author

Plugin UI is looking good. Would it be helpful to finally add plugins to the asset page to also show Data Quality there?

Yeah thought of that it seems currently it doesnt support?, are you referring to the changes in core to support plugins in asset page ?

I would open a PR to add support.

thanks.. :)

@gopidesupavan
gopidesupavan requested a review from Dev-iL July 16, 2026 20:11

@Dev-iL Dev-iL left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few comments on the docs.

One suggestion: in terms of visualizing the results of a validation, a user might not always be interested in a strict pass/fail, but to be able to visualize "how bad the situation is", so let's say a user specifies a colormap or at least two colors and then then according to the validation score the color is interpolated. In the case of a classical "traffic light" colormap - a color between yellow and green means "pretty good" and between yellow-red is "pretty bad".

The biggest complication in these scenarios is that it's not always a linear mapping - could be inverse, could be logarithmic, etc. Initially it could support only linear interpolation, but later other types of "library" functions and finally make it completely pluggable.

Generating rules with an LLM
==============================

Writing a :class:`~airflow.providers.common.dataquality.rules.RuleSet` by hand for every table doesn't scale.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a middle ground between "by hand" and "via LLM" - deterministic scripts. This also allows users w/o LLM access to use the tools. These can be bundled into the SKILL or placed somewhere in the plugin.

============================

Quality rules can travel with the :class:`~airflow.sdk.Asset` they describe instead of being
scattered across every Dag that checks it, and a downstream consumer Dag can refuse to run when

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a big fan of describing random bad practices that the code replaces. I suggest

Quality rules can travel with the :class:~airflow.sdk.Asset they describe instead of being scattered across every Dag that checks it, and a downstream consumer Dag can refuse to run when


Only one Dag should call ``asset_quality()`` for a given asset ``name``/``uri``. Airflow keeps one
shared record per asset across all Dags, so if more than one Dag attaches (or omits) config for the
same asset, whichever Dag parsed most recently determines what's stored.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this preferred over a loud dag parse error?

the TaskFlow API. ``ruleset`` may be declared as a decorator argument when it exists at
Dag-parse time, or returned by the decorated function as a runtime ruleset. ``table`` or
``asset`` are declared as decorator arguments exactly like the plain operator. The decorated
function is optional plumbing on top: return ``None`` to run the check exactly as declared.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
function is optional plumbing on top: return ``None`` to run the check exactly as declared.
function is optional: return ``None`` to run the check exactly as declared.

Comment on lines +81 to +84
If quality checks are one step inside a larger task, use
:func:`~airflow.providers.common.dataquality.execution.run_quality_checks` directly, then call
:func:`~airflow.providers.common.dataquality.execution.persist_quality_results` to write the same
history records that ``DQCheckOperator`` writes:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping with existing terminology, wouldn't that be known as a "hook"?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants