Airflow Data Quality Provider part 1

[gopidesupavan]

Adds a new apache-airflow-providers-dataquality provider for DbApiHook-based data quality checks.

Airflow already has SQL check operators, and many users rely on them for data quality today. This provider adds a DQRule / RuleSet layer for checks that need stable rule identity, persisted history, and a connection to Airflow assets. That makes quality results easier to analyze over time, lets downstream asset consumers gate on recent quality, and gives LLM-assisted workflows one schema to generate when proposing checks from table context. Execution still goes through existing common.sql / DbApiHook connections.

This PR is the backend/provider slice only. The UI plugin and read-only API are intentionally left for a follow-up PR.

Ships:

• DQRule and RuleSet models for named data quality rules.
• Built-in SQL checks for common table and column checks, executed through common.sql / DbApiHook.
• custom_sql support for database-specific or more complex checks.
• DQCheckOperator and the @task.dq_check TaskFlow decorator.
• A configurable results backend under [dq] results_path for task, run, and rule-level history.
• Experimental asset helpers, asset_quality() and require_quality(), that attach provider-owned quality metadata to assets without changing Airflow core.
• Documentation and example Dags covering end-to-end usage with and without LLM-generated rules.
• A DQ rule-authoring skill that LLM-assisted workflows can use to generate rules from table/schema context:
  https://github.com/gopidesupavan/airflow/blob/f32940bd261b94238256eaced9150dd51329ce3e/providers/dq/src/airflow/providers/dq/skills/dq-rule-authoring/SKILL.md

This first version is intentionally focused on the backend contract: deterministic rule definitions, SQL execution through existing Airflow SQL providers, persisted results, and asset-linked quality summaries.

Design decisions:

• Results are stored through an object-storage/local-file backend instead of adding new metadata DB tables in the first provider drop. This keeps the provider self-contained, avoids Airflow core migrations, and lets deployments choose a durable store such as S3, GCS, or local files via [dq] results_path.
• The backend stores keyed JSON records for task runs, task instances, and per-rule history so later readers, including a future UI/API layer, can access common views without scanning unrelated runs.
• Asset support is implemented with provider-owned metadata, not Airflow core changes. Static quality configuration is attached to Asset.extra["airflow.dq"]; runtime summaries are attached to asset events under extra["airflow.dq.result"].
• The first release starts with DbApiHook / SQL execution because Airflow already has broad database coverage through common.sql. File and object-store data checks are left for a later iteration.

Later iterations:

• Read-only API and minimal Airflow UI plugin for viewing task/run results and rule history.
• File/object-store based checks, where Airflow reads data from S3/GCS/local files or other object stores and runs quality rules directly against that data.
• OpenLineage integration for data quality facets.
• More built-in checks.
• Trigger support for DQ checks.
• DQProfileOperator.

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)
  codex

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[gopidesupavan]

This is part 1 , the UI plugin is part of this big PR #69413, i will ship that separate. but to show how the UI looks like here is some screenshots #69413 (comment) (this is minimal what had done with help of UI not an UI expert anyone can modify or help with this 😄 )