docs: add real-time ML observability and drift metadata definitions - #350
docs: add real-time ML observability and drift metadata definitions#350mitalidaduria wants to merge 11 commits into
Conversation
Expanded the metadata standard documentation to include detailed explanations of machine learning concepts such as model drift, data lineage, feature engineering, label skew, and data quality SLA.
add business-friendly ML glossary with payments domain examples
… and styling per review
| <a href="https://openmetadatastandards.org/" target="_blank" rel="noopener noreferrer" className="link">Visit openmetadatastandards.org</a> | ||
|
|
||
| You'll find quick starts, schema references, semantic web resources, and contribution guides on the official site. | ||
| ## Machine Learning & Data Governance Expansion |
There was a problem hiding this comment.
Neither changed file is registered as the live v1.13 documentation. docs.json navigates to the unchanged v1.13.x/api-reference/main-concepts/metadata-standard page and redirects the modified /v1.13.x/main-concepts/metadata-standard route there, while this content/main/... file has no navigation entry. Consequently the PR publishes none of its new material, and neither changed file implements the promised SHAP/LIME metadata or PSI/recall drift gates. Add the observability guide—with frontmatter and executable monitoring steps—under a current versioned path and register it in docs.json; revert the stale redirect-source edit.
| * **Column Value Limits**: Ensure transactional amounts are never negative or unexpectedly zero. | ||
| * *OpenMetadata Test*: `columnValuesToBeBetween` | ||
| * *Configuration*: Set `minValue = 0.01` for debit/credit transaction amount fields. | ||
| * **Sum Validations**: Verify that daily ledger updates match expected sum totals. |
There was a problem hiding this comment.
columnValuesSumToBeBetween sums one column against configured bounds; it cannot reconcile payment-gateway totals against internal-ledger totals as this section promises. Provide a custom SQL comparison across both sources, or narrow the claim to validating one known expected range.
| ### 2. Eliminating Duplicates and Orphans (Integrity Tests) | ||
| Double-submitting a webhook payload should never cause duplicate database entries. We enforce this at the database validation layer using key assertions. | ||
|
|
||
| * **Uniqueness Constraints**: Every financial transaction must have a universally unique ID (UUID). |
There was a problem hiding this comment.
columnValuesToBeUnique checks repeated non-null values, and columnValuesToNotBeNull only checks presence; together they neither enforce UUID format nor detect a non-null foreign key with no matching parent, despite this section promising both. Add UUID-format validation and a referential-integrity check such as a custom SQL LEFT JOIN, keeping null and uniqueness as separate checks.
| To guarantee real-time updates for downstream fraud modeling, apply table-level freshness tests to transactional timestamps. | ||
|
|
||
| * **Timestamp Verification**: Ensure the difference between the transaction event time and the ingest time remains below a critical threshold. | ||
| * *OpenMetadata Test*: `tableRowCountToBeGreaterThan` (within a moving temporal partition) or customized SQL queries checking `MAX(created_at)` against the current time. |
There was a problem hiding this comment.
tableRowCountToBeGreaterThan is not an available test definition, and a row-count test would not compare transaction event time with ingest time. Use tableDataToBeFresh, tableRowInsertedCountToBeBetween, or provide a concrete custom SQL latency check so readers can implement this test.
| ## Checklist: Deploying Your FinTech Test Suite | ||
|
|
||
| 1. **Profile Your Tables**: Run an initial ingestion execution on your payment datasets using the OpenMetadata Profiler to establish baseline means, standard deviations, and null counts. | ||
| 2. **Create a Dedicated Test Suite**: In the OpenMetadata UI, navigate to the **Database Service** → **Table** → **Profiler** tab and click **Add Test**. |
There was a problem hiding this comment.
This step says it creates a dedicated test suite, but Add Test on a table creates a test case. The documented suite workflow is Quality → Data Quality → By Test Suites → Add Test Suite; table test cases are added from the table’s Data Observability tab. Update the step to distinguish these workflows.
|
|
||
| 1. **Profile Your Tables**: Run an initial ingestion execution on your payment datasets using the OpenMetadata Profiler to establish baseline means, standard deviations, and null counts. | ||
| 2. **Create a Dedicated Test Suite**: In the OpenMetadata UI, navigate to the **Database Service** → **Table** → **Profiler** tab and click **Add Test**. | ||
| 3. **Alerting**: Bind the FinTech Test Suite to your alerting channels (Slack, Microsoft Teams, or PagerDuty) so your data engineering on-call rotation is immediately notified when a settlement test fails. No newline at end of file |
There was a problem hiding this comment.
PagerDuty is not a native OpenMetadata alert destination. Current subscription types support Slack, Microsoft Teams, Google Chat, email, and generic webhooks, so readers cannot bind a suite directly to PagerDuty as written. Remove PagerDuty or document the required webhook or intermediary integration.
478ce1b to
a1a3eb7
Compare
ayush-shah
left a comment
ayush-shah
left a comment
There was a problem hiding this comment.
Defect-first review on head a1a3eb7:
- [P1] Rebuild this PR from its intended scope. The current head contains none of the promised SHAP/LIME metadata, PSI threshold, or recall gate. Fifteen of sixteen changed paths are unrelated AI-review automation, glossary, FinTech, and MSSQL changes. Create a fresh branch from this PR base, apply only the intended observability guide and navigation, validate it, and replace or retarget this PR without rewriting the existing source branch.
Evidence: the complete diff was compared with the title and current navigation. Existing feedback already covers the FinTech defects, so those were not duplicated here.
2 participants
This pull request expands the documentation framework by adding a detailed guide on tracking runtime machine learning observability parameters.
Using production-grade payment anomaly detection environments as a baseline domain, it formally establishes clear definitions and architectural metadata parameters for:
Model Explainability Metadata: Tracking SHAP/LIME key-value logging to expose feature drivers (e.g., transaction velocity window shifts) to downstream data applications.
Ecosystem Performance Degradation Gates: Structuring production alerting thresholds for Population Stability Index (PSI > 0.2) and concept drift (Recall < 88%) monitoring.