docs: add FinTech data quality framework use case guide

content/main/features/data-quality/fintech-framework.md

@@ -0,0 +1,58 @@
+# FinTech Data Quality Framework
+
+Financial technology data requires a level of consistency, accuracy, and latency control far stricter than standard analytical data stores. A single null value in a settlement column or an unexpected duplicate in a transaction ledger can result in severe financial discrepancies, regulatory non-compliance, or disrupted user operations.
+
+This guide outlines how to implement a robust **Data Quality Framework** for FinTech pipelines using OpenMetadata's native profiling and test suites.
+
+---
+
+## The 3 Critical FinTech Data Failure Modes
+
+### 1. Settlement & Reconciliation Mismatches (Asymmetry)
+* **The Problem**: Transaction data flowing from a payment gateway (e.g., Stripe, Adyen) must reconcile perfectly with internal ledger entries. Discrepancies often arise due to timezone differences, late-arriving settlement records, or multi-currency rounding issues.
+* **Impact**: Inaccurate ledger reporting, incorrect merchant payouts, and balance sheets that do not balance.
+
+### 2. Orphaned Records & Key Violations
+* **The Problem**: High-volume asynchronous transaction pipelines often lead to processing delays, resulting in "orphaned" transactions where the payment event occurs but the associated billing account or user ID does not exist in the source table.
+* **Impact**: Orphaned accounts cause batch analytical pipelines to crash or misrepresent active user revenue metrics.
+
+### 3. Latency & Batch Freshness Violations
+* **The Problem**: In financial operations, data freshness acts as a proxy for pipeline health. If credit scoring features, risk databases, or fraud prevention logs lag behind real-time processing, fraud detection models make decisions based on stale states.
+* **Impact**: Spikes in undetected payment fraud and exposure to chargeback risk.
+
+---
+
+## Implementing the Framework with OpenMetadata
+
+OpenMetadata allows you to operationalize this framework directly on your tables using the **Profiler** and declarative **Test Suites**.
+
+### 1. Guarding Against Settlement Mismatches (Value Range & Sum Assertions)
+To ensure transactional integrity, we run test suites verifying that transactional columns behave within absolute bounds and matching criteria.
+
+* **Column Value Limits**: Ensure transactional amounts are never negative or unexpectedly zero.
+  * *OpenMetadata Test*: `columnValuesToBeBetween` 
+  * *Configuration*: Set `minValue = 0.01` for debit/credit transaction amount fields.
+* **Sum Validations**: Verify that daily ledger updates match expected sum totals.
+  * *OpenMetadata Test*: `columnValuesSumToBeBetween`
+
+### 2. Eliminating Duplicates and Orphans (Integrity Tests)
+Double-submitting a webhook payload should never cause duplicate database entries. We enforce this at the database validation layer using key assertions.
+
+* **Uniqueness Constraints**: Every financial transaction must have a universally unique ID (UUID).
+  * *OpenMetadata Test*: `columnValuesToBeUnique` applied to `transaction_id`.
+* **Null Value Prevention**: Critical foreign keys (like `merchant_id` or `user_id`) must never resolve to null.
+  * *OpenMetadata Test*: `columnValuesToNotBeNull` on `account_id` and `customer_id`.
+
+### 3. Monitoring Processing Latency (Freshness Tests)
+To guarantee real-time updates for downstream fraud modeling, apply table-level freshness tests to transactional timestamps.
+
+* **Timestamp Verification**: Ensure the difference between the transaction event time and the ingest time remains below a critical threshold.
+  * *OpenMetadata Test*: `tableRowCountToBeGreaterThan` (within a moving temporal partition) or customized SQL queries checking `MAX(created_at)` against the current time.
+
+---
+
+## Checklist: Deploying Your FinTech Test Suite
+
+1. **Profile Your Tables**: Run an initial ingestion execution on your payment datasets using the OpenMetadata Profiler to establish baseline means, standard deviations, and null counts.
+2. **Create a Dedicated Test Suite**: In the OpenMetadata UI, navigate to the **Database Service** → **Table** → **Profiler** tab and click **Add Test**.
+3. **Alerting**: Bind the FinTech Test Suite to your alerting channels (Slack, Microsoft Teams, or PagerDuty) so your data engineering on-call rotation is immediately notified when a settlement test fails.

docs.json

@@ -1492,6 +1492,9 @@
                       },
                       "v1.13.x/how-to-guides/data-quality-observability/quality/custom-tests",
                       "v1.13.x/how-to-guides/data-quality-observability/quality/test-library",
+<<<<<<< HEAD
+                      "v1.13.x/how-to-guides/data-quality/fintech-data-quality-framework"
+=======
                       {
                         "group": "Bulk Import, Export, and Edit Test Cases",
                         "pages": [
@@ -1501,6 +1504,7 @@
                           "v1.13.x/how-to-guides/data-quality-observability/quality/import-export-test-cases/edit"
                         ]
                       }
+>>>>>>> origin/main
                     ]
                   },
                   {

v1.13.x/api-reference/main-concepts/metadata-standard.mdx

@@ -43,3 +43,32 @@ Ready to dive in?
 <a href="https://openmetadatastandards.org/" target="_blank" rel="noopener noreferrer" className="link">Visit openmetadatastandards.org</a>
 
 You'll find quick starts, schema references, semantic web resources, and contribution guides on the official site.
+
+## Machine Learning & Data Governance Expansion
+
+The following terms bridge common machine learning and data governance concepts to business-friendly language, with payments-domain examples showing how each one shows up in practice.
+
+#### Model Drift
+* **Technical Definition**: Performance degradation of a model over time caused by changes in real-world data distribution.
+* **Business Translation**: The model's predictions become less accurate as customer habits or external conditions evolve.
+* **Payments Example**: A fraud detection model trained on pre-holiday spending flags normal holiday shopping sprees as fraudulent.
+
+#### Data Lineage
+* **Technical Definition**: The complete lifecycle trajectory of data, tracking its origins, transformations, and final destinations.
+* **Business Translation**: A visible paper trail showing where data came from and how it was modified before reaching reports.
+* **Payments Example**: Tracing a payout error back through intermediate currency conversions to the raw transaction record.
+
+#### Feature Engineering
+* **Technical Definition**: The process of selecting, manipulating, and transforming raw data into useful input variables for machine learning models.
+* **Business Translation**: Converting raw transaction records into meaningful signals that help the model spot patterns.
+* **Payments Example**: Transforming raw transaction timestamps into a "transactions per hour" velocity feature to detect card-testing bots.
+
+#### Label Skew
+* **Technical Definition**: Discrepancy between ground truth labels used during training and actual outcomes in production.
+* **Business Translation**: When real-world results don't match the historical data assumptions used to train the system.
+* **Payments Example**: A chargeback model trained only on settled transactions misses early fraud patterns from pending authorizations.
+
+#### Data Quality SLA (Service Level Agreement)
+* **Technical Definition**: Formal agreements defining measurable thresholds for data accuracy, completeness, latency, and freshness.
+* **Business Translation**: Guaranteed standards ensuring data arrives on time and is reliable enough for business operations.
+* **Payments Example**: A operational rule requiring payment reconciliation feeds to be updated within 15 minutes with 99.9% completeness.