🛡️ Data Contract Validator

Prevent production API breaks by validating data contracts between your data pipelines and API frameworks

🎯 What This Solves

Ever deployed a DBT model change only to break your FastAPI in production? This tool prevents that by validating data contracts between your data pipelines and APIs before deployment.

DBT Models          Contract           FastAPI Models
(What data          Validator          (What APIs
 produces)          ↕️ VALIDATES ↕️      expect)
     ↓                   ↓                   ↓
   Schema              Finds              Schema
 Extraction          Mismatches         Extraction

⚡ Quick Start

Installation

pip install data-contract-validator

30-Second Setup

# 1. Initialize in your project
contract-validator init --interactive

# 2. Test setup
contract-validator test

# 3. Validate contracts
contract-validator validate

# 4. Commit and push - you're protected! 🛡️

Basic Usage

# Validate local DBT project against FastAPI models
contract-validator validate \
  --dbt-project ./my-dbt-project \
  --fastapi-local ./my-api/models.py

# Validate across repositories (microservices)
contract-validator validate \
  --dbt-project . \
  --fastapi-repo "my-org/my-api-repo" \
  --fastapi-path "app/models.py"

🔍 Real Example: Production Validation

Actual output from a production analytics project:

$ contract-validator validate

🔍 Starting contract validation...
📊 Extracting source schemas...
   ✅ Found 14 DBT models (user_analytics_summary: 54 columns)
🎯 Extracting target schemas...  
   ✅ Found 3 FastAPI models
🔍 Validating schema compatibility...

🛡️ Results:
✅ PASSED - 0 critical issues (no production breaks!)
⚠️  42 warnings (type mismatches to review)

Issues caught:
⚠️  user_analytics_summary.age_years: source 'varchar' vs target 'integer'
⚠️  user_analytics_summary.is_verified: source 'varchar' vs target 'boolean'
⚠️  user_analytics_summary.user_created_at: source 'varchar' vs target 'timestamp'

🎉 Your API contracts are protected!

🚨 What It Prevents

Before Data Contract Validation:

-- Analytics team changes DBT model
select
    user_id,
    email,
    -- total_orders,  ❌ REMOVED this column
    revenue
from users

# API team's FastAPI model (unchanged)
class UserAnalytics(BaseModel):
    user_id: str
    email: str
    total_orders: int  # ❌ Still expects this!
    revenue: float

Result: 💥 Production API breaks, angry customers, 2AM debugging

After Data Contract Validation:

$ git push

❌ VALIDATION FAILED
💥 user_analytics.total_orders: FastAPI REQUIRES column but DBT removed it
🔧 Fix: Add 'total_orders' back to DBT model or update FastAPI model

# Push blocked until fixed ✋

Result: 🛡️ Production protected, issues caught in CI/CD

🛠️ Pre-commit Integration

Automatic Setup (Recommended)

# Initialize with pre-commit support
contract-validator init --interactive
contract-validator setup-precommit --install-hooks

# Now every commit validates contracts automatically! 🛡️

Manual Setup

If you prefer manual setup:

Install pre-commit:
```
pip install pre-commit
```

Add to .pre-commit-config.yaml:

repos:
  - repo: https://github.com/OGsiji/data-contract-validator
    rev: v1.0.0
    hooks:
      - id: contract-validation
        name: Validate Data Contracts
        files: '^(.*models.*\.(sql|py)|\.retl-validator\.yml|dbt_project\.yml)$'

Install hooks:
```
pre-commit install
```

How It Works

$ git add models/user_analytics.sql
$ git commit -m "update user analytics model"

# Pre-commit automatically runs:
🔍 Validating Data Contracts...
✅ Contract validation passed
[main abc1234] update user analytics model

On Validation Failure

$ git commit -m "remove important column"

🔍 Validating Data Contracts...
❌ CRITICAL: user_analytics.total_revenue missing
💡 Fix the issue before committing

# Commit blocked until fixed! 🛡️

Skip Validation (Emergency Only)

# Only for emergencies!
git commit -m "emergency fix" --no-verify

Benefits of Pre-commit Integration

✅ Catches issues before they reach CI/CD
✅ Faster feedback loop (seconds, not minutes)
✅ No broken commits in your git history
✅ Team protection - everyone gets validation
✅ Zero configuration after setup

📦 GitHub Actions Integration

Add this to .github/workflows/validate-contracts.yml:

name: 🛡️ Data Contract Validation

on:
  pull_request:
    paths:
      - 'models/**/*.sql'
      - 'dbt_project.yml'
      - '**/*models*.py'

jobs:
  validate-contracts:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-python@v4
      with:
        python-version: '3.9'
    
    - name: Validate contracts
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      run: |
        pip install data-contract-validator
        contract-validator validate

Auto-generated when you run contract-validator init!

🔧 Configuration

Auto-Generated Config (`.retl-validator.yml`)

version: '1.0'
name: 'my-project-contracts'

source:
  dbt:
    project_path: '.'
    auto_compile: true

target:
  fastapi:
    # For GitHub repos
    type: "github"
    repo: "my-org/my-api"
    path: "app/models.py"
    
    # For local files
    # type: "local"
    # path: "../my-api/models.py"

validation:
  fail_on: ['missing_tables', 'missing_required_columns']
  warn_on: ['type_mismatches', 'missing_optional_columns']

Command Line Options

contract-validator validate \
  --dbt-project ./dbt-project \           # DBT project path
  --fastapi-repo "org/repo" \             # GitHub repo
  --fastapi-path "app/models.py" \        # Path to models
  --github-token "$GITHUB_TOKEN" \        # For private repos
  --output json                           # json, terminal, github

🚀 Supported Frameworks

Data Sources ✅

DBT (all adapters: Snowflake, BigQuery, Redshift, etc.)

API Frameworks ✅

FastAPI (Pydantic + SQLModel)

Coming Soon 🔄

Django, Flask-SQLAlchemy
Databricks, Airflow
Request other frameworks

🎯 Output Formats

Terminal (Default)

🛡️ Data Contract Validation Results:
Status: ✅ PASSED
Critical: 0 | Warnings: 5

⚠️  Warnings:
  user_analytics.age: Type mismatch (varchar vs integer)
  user_analytics.country: Type mismatch (integer vs varchar)

🎉 Your API contracts are protected!

JSON (for CI/CD)

{
  "success": true,
  "critical_issues": 0,
  "warnings": 5,
  "issues": [
    {
      "severity": "warning",
      "table": "user_analytics", 
      "column": "age",
      "message": "Type mismatch: source 'varchar' vs target 'integer'",
      "suggested_fix": "Update target to expect 'varchar' or fix source type"
    }
  ]
}

GitHub Actions

::warning::user_analytics.age: Type mismatch detected
✅ Contract validation passed - no critical issues

🏗️ Architecture

Simple Python API

from data_contract_validator import ContractValidator, DBTExtractor, FastAPIExtractor

# Initialize extractors
dbt = DBTExtractor(project_path='./dbt-project')
fastapi = FastAPIExtractor.from_github_repo('my-org/my-api', 'app/models.py')

# Run validation
validator = ContractValidator(source=dbt, target=fastapi)
result = validator.validate()

if not result.success:
    print(f"❌ {len(result.critical_issues)} critical issues found")
    for issue in result.critical_issues:
        print(f"💥 {issue.table}.{issue.column}: {issue.message}")

CLI Interface

# Interactive setup
contract-validator init --interactive

# Test configuration
contract-validator test

# Run validation
contract-validator validate

# Setup pre-commit hooks
contract-validator setup-precommit --install-hooks

# Multiple output formats
contract-validator validate --output json

🔄 Development Workflow

With Pre-commit (Recommended)

# Team workflow with automated validation
git clone your-dbt-project
cd your-dbt-project

# One-time setup for new team members
contract-validator init --interactive
contract-validator setup-precommit --install-hooks

# Protected development workflow:
# 1. Make changes to DBT models
# 2. git add models/my_model.sql
# 3. git commit -m "update model"  # ← Validation runs here automatically
# 4. If validation passes → commit succeeds
# 5. If validation fails → fix issues first
# 6. git push  # ← CI/CD validation as backup

Manual Workflow

# Traditional workflow
# 1. Make changes
# 2. contract-validator validate  # Manual validation
# 3. git commit
# 4. git push

🤝 Contributing

We welcome contributions! This tool is actively used in production.

Development Setup

git clone https://github.com/OGsiji/data-contract-validator
cd data-contract-validator
pip install -e ".[dev]"
pytest

Adding New Extractors

from retl_validator.extractors import BaseExtractor

class MyFrameworkExtractor(BaseExtractor):
    def extract_schemas(self) -> Dict[str, Schema]:
        # Your implementation
        return schemas

Reporting Issues

🐛 Bugs: GitHub Issues
💡 Features: GitHub Discussions

📚 Documentation

Quick Start Guide - Get running in 2 minutes
Configuration Reference - All config options
GitHub Actions Setup - CI/CD integration
Examples - Real-world usage
Pre-commit Integration - Automated validation

🎉 Real-World Usage

This tool is actively preventing production incidents in:

Analytics pipelines with 50+ DBT models
Microservices architectures with multiple APIs
Data engineering teams using Snowflake, BigQuery, Redshift
Cross-repository validation in large organizations

Proven to catch:

✅ Type mismatches (varchar vs integer)
✅ Missing columns (API expects columns DBT doesn't provide)
✅ Schema drift (gradual model changes)
✅ Breaking changes before they reach production

🛡️ Multiple Layers of Protection

Pre-commit hooks: Immediate feedback (fastest)
CI/CD validation: Team protection (backup)
Manual validation: Development testing
Configuration files: Team standards

This creates a comprehensive safety net for your data contracts.

📄 License

MIT License - see LICENSE for details.

🆘 Support

🐛 Issues: GitHub Issues
💬 Discussions: GitHub Discussions
📧 Email: ogunniransiji@gmail.com

⭐ Star the Project

If this tool helps you prevent production incidents, please ⭐ star the repository!

🛡️ Built by data engineers, for data engineers. Stop breaking production with data changes!

🚀 Get Started Now

pip install data-contract-validator
contract-validator init --interactive
contract-validator setup-precommit --install-hooks
# 2 minutes to production protection with automated validation!

Name		Name	Last commit message	Last commit date
Latest commit History 17 Commits
.github/workflows		.github/workflows
data_contract_validator		data_contract_validator
examples		examples
tests		tests
.gitignore		.gitignore
.pre-commit-hooks.yaml		.pre-commit-hooks.yaml
.retl-validator.yml		.retl-validator.yml
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
README.md		README.md
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt

Folders and files

Latest commit

History

Repository files navigation

🛡️ Data Contract Validator

🎯 What This Solves

⚡ Quick Start

Installation

30-Second Setup

Basic Usage

🔍 Real Example: Production Validation

🚨 What It Prevents

Before Data Contract Validation:

After Data Contract Validation:

🛠️ Pre-commit Integration

Automatic Setup (Recommended)

Manual Setup

How It Works

On Validation Failure

Skip Validation (Emergency Only)

Benefits of Pre-commit Integration

📦 GitHub Actions Integration

🔧 Configuration

Auto-Generated Config (.retl-validator.yml)

Command Line Options

🚀 Supported Frameworks

Data Sources ✅

API Frameworks ✅

Coming Soon 🔄

🎯 Output Formats

Terminal (Default)

JSON (for CI/CD)

GitHub Actions

🏗️ Architecture

Simple Python API

CLI Interface

🔄 Development Workflow

With Pre-commit (Recommended)

Manual Workflow

🤝 Contributing

Development Setup

Adding New Extractors

Reporting Issues

📚 Documentation

🎉 Real-World Usage

🛡️ Multiple Layers of Protection

📄 License

🆘 Support

⭐ Star the Project

🚀 Get Started Now

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Auto-Generated Config (`.retl-validator.yml`)

Packages