Proposal: typed data quality evidence for policy-readable dataset trust signals

[SomeshZanwar]

Summary

PR #2224 added a self-contained example for data-quality-aware agent governance: AGT policy evaluation combined with an external data quality signal as a single decision point.

That example demonstrates the pattern, but the data quality signal is still represented locally inside the example.

This proposal is about the next layer: typed data quality evidence that can make dataset trust signals policy-readable and audit-ready across AGT workflows.

Problem

AGT can evaluate whether an agent is authorized to perform an action.

But in data-heavy workflows, authorization is not enough. The target dataset may be stale, failing validation, below a quality threshold, or owned by a source that needs explicit accountability.

The current example handles this with a simulated DataQualityRegistry. That is useful for demonstrating the pattern, but production systems need a reusable evidence shape that can carry dataset trust state into the governance decision.

The key distinction is:

• AGT policy decides whether the agent/action is allowed.
• Data quality evidence describes whether the target data is trustworthy at the time of access.
• The final governance decision should be able to consider both.

###Proposed direction

Add typed data quality evidence to agent-os, agent-governance, or the appropriate AGT package location.

Possible object shape:

from dataclasses import dataclass
from datetime import datetime

@DataClass(frozen=True)
class DataQualityEvidence:
dataset_id: str
freshness_at: datetime
validation_status: str # "pass" | "warn" | "fail"
quality_score: float | None
quality_profile_id: str | None
dataset_owner_did: str | None
classification: str | None # "public" | "internal" | "regulated" etc.
failed_tests: list[str]
snapshot_hash: str | None

The exact fields are open for discussion. The important part is the pattern: data quality should be represented as typed evidence rather than opaque metadata.

Why this fits AGT

The merged example in PR #2224 already shows the policy pattern:

Check whether the agent is authorized.
Check whether the dataset is trustworthy.
Allow only if both pass.
Record which layer allowed or blocked the action.

Typed data quality evidence would make that pattern reusable beyond the example folder.

It would allow future examples, integrations, or policy checks to consume the same kind of evidence structure instead of each implementation inventing its own local dictionary or registry format.

Relationship to auditability

This object would make the audit trail cleaner.

Instead of recording only:

agent X requested action Y on dataset Z

the audit trail can also include:

dataset Z had freshness_at=T, validation_status=fail, quality_score=0.72, failed_tests=[...]

That makes the governance decision explainable after the fact.

Relationship to CTEF / external receipts

This proposal does not require AGT to implement CTEF semantics.

However, one useful alignment point is snapshot_hash.

A data quality snapshot can be represented in two ways:

• as typed evidence inside AGT for policy evaluation
• as a portable integrity commitment outside AGT for cross-system verification

Same snapshot. Two representations. Different jobs.

The AGT object should stay policy-readable. External receipts or CTEF-style references can carry the portable integrity commitment when needed.

Non-goals

This proposal is not asking AGT to:

• require a specific data quality provider
• require dbt, Great Expectations, or any single validation framework
• define a mandatory protocol field
• implement CTEF semantics
• enforce one universal quality threshold

The goal is a small, reusable evidence shape that production integrations can map into.

Open questions

1. Does this belong in agent-os, agent-governance, or only as an example utility for now?
2. Should this be a dataclass, Pydantic model, or existing AGT evidence type if one already exists?
3. Should validation_status use a closed enum?
4. Should snapshot_hash be included directly, or should integrity references be left to external receipt systems?
5. Should this start as a docs/example artifact before becoming package code?

Suggested first step

If maintainers think this direction is useful, I can start with a small documentation/example PR that introduces the typed evidence shape without changing core behavior.

That would keep the contribution low-risk while giving future data-quality integrations a clearer shape to build on.

[miyannishar]

Hey @SomeshZanwar, good write-up. The schema makes sense and the problem is real.

That said, I think your open question #5 answers itself -- this should start as a docs/example artifact before becoming package code. Data quality is one evidence shape among many (compliance attestations, cost signals, latency SLOs, etc.), and if we bake a typed model into core for each one we'll end up with a grab-bag of domain-specific schemas that don't generalize well.

Your earlier example from PR #2224 is the right pattern. I'd suggest:

Ship a standalone example showing DataQualityEvidence wired into a real policy (dbt or Great Expectations output as input, policy decision as output).
Let it get used by a few people and see what fields actually matter vs what's speculative.
Once the shape stabilizes, we can talk about promoting it to a typed model in the package.
AGT's core should stay domain-agnostic. The value of your proposal is in proving the integration pattern, not in standardizing the schema too early.

Happy to review an example PR if you want to take that route.

[SomeshZanwar]

Thanks, that direction makes sense.

I agree that promoting a domain-specific schema into core too early would make AGT less general. I’ll keep this example-level for now and focus on proving the integration pattern rather than standardizing the model.

I’ll draft a standalone example that shows:

1. a DataQualityEvidence object local to the example,
2. dbt or Great Expectations-style validation output as input,
3. AGT policy evaluation plus data quality evidence as the decision flow,
4. a final allow/block decision with audit-friendly reasoning.

I’ll keep it self-contained, avoid core package changes, and treat the field shape as provisional until there is more usage.

[SomeshZanwar]

Thanks again for the direction here. I opened a follow-up example PR that keeps the evidence shape local to the existing examples/data-quality-aware-governance/ folder and uses dbt run_results.json fixtures as input:

#2278

No core package changes or new dependencies. The goal is to prove the integration pattern first, as suggested.

[ElamOlame31]

One dimension worth incorporating: temporal degradation of trust signals. In AgentGate, a dataset accessed at high velocity within a short window generates a different trust signal than the same dataset accessed normally — even if individual quality signals look identical. The behavioral pattern is part of the evidence.

https://github.com/ElamOlame31/agentgate-public

https://www.tryagentgate.com/