ai-data-registry

Git-native, PR-driven data platform. Fork, add a data pipeline as a workspace, open a PR. Automated validation, extraction, and catalog federation handle the rest.

How It Works

flowchart LR
    subgraph Contributor
        PR[Open PR with workspace]
    end

    subgraph Validation
        L1[Static analysis]
        L2[Collision check]
        L3[Catalog compat]
        L4[Dry run]
    end

    subgraph Extraction
        GH[GitHub runner]
        HZ[Hetzner ARM]
        HF[HuggingFace GPU]
    end

    subgraph Storage[S3 Storage]
        S3[(Parquet files)]
        CAT[(catalog.duckdb<br/>DuckLake global)]
    end

    PR --> L1 --> L2 --> L3 --> L4
    L4 -->|merge| GH & HZ & HF
    GH & HZ & HF --> S3
    S3 -->|"scan + add_data_files()"| CAT

Loading

Each workspace is an isolated pipeline with its own language, deps, and compute backend. DuckLake federates all outputs into one queryable global catalog via zero-copy file registration.

Getting Started

Two ways to use this project:

Path	Who	How
Contribute a workspace	Data engineers, researchers	Fork, add a workspace, open a PR
Create your own registry	Organizations, platform maintainers	Use as GitHub template, configure infra

Creating your own registry? See docs/template-setup.md for template initialization, S3 configuration, and backend setup.

Contribute a Workspace (Fork)

Prerequisites: Pixi (brew install pixi or curl -fsSL https://pixi.sh/install.sh | bash)

git clone <your-fork-url>
cd ai-data-registry
pixi install

Create a Workspace

/new-workspace my-pipeline python    # Claude Code slash command

Or manually:

mkdir -p workspaces/my-pipeline
cd workspaces/my-pipeline
pixi init . --channel conda-forge --platform osx-arm64 --platform linux-64 --platform win-64
pixi add python

Then add your [tool.registry] config, write your extract script, and open a PR. See workspaces/test-minimal/ for a working example.

Workspace Contract

Every workspace pixi.toml needs:

[tasks]
extract = "python extract.py"                                     # writes to $OUTPUT_DIR/
validate = { cmd = "python validate_local.py", depends-on = ["extract"] }
pipeline = { depends-on = ["extract", "validate"] }               # runner entry point
dry-run = { cmd = "python extract.py", env = { DRY_RUN = "1" } } # PR validation

[tool.registry]
description = "What this pipeline extracts"
schedule = "0 6 * * *"        # cron
timeout = 30                  # minutes
tags = ["topic"]
schema = "my-pipeline"        # S3 prefix + DuckLake schema (must be unique)
table = "data"                # or: tables = ["a", "b"] for multi-table workspaces
mode = "append"               # append | replace | upsert

[tool.registry.runner]
backend = "github"            # github | hetzner | huggingface
flavor = "ubuntu-latest"

[tool.registry.license]
code = "Apache-2.0"           # OSI-approved SPDX
data = "CC-BY-4.0"            # recognized SPDX
data_source = "Source Name"
mixed = false

[tool.registry.checks]
min_rows = 100
max_null_pct = 5
unique_cols = ["id"]

Key rules: Write Parquet to $OUTPUT_DIR/, never to S3 directly. Do not hardcode OUTPUT_DIR in task env. No credentials in code (use $WORKSPACE_SECRET_*).

Compute Backends

Backend	Flavors	Use case
github	ubuntu-latest	Lightweight: API calls, CSV/JSON downloads
hetzner	cax11 cax21 cax31 cax41	Medium: spatial processing, large datasets (ephemeral ARM)
huggingface	cpu-basic cpu-upgrade t4-small t4-medium l4x1 a10g-small a10g-large a10g-largex2 a100-large	GPU: ML inference, embeddings (Docker)

Need something else? Open an issue. Infrastructure is maintainer-managed.

PR Validation

flowchart TD
    PR[PR touches workspaces/] --> L1

    subgraph L1[Layer 1: Static]
        A[Required fields, SPDX licenses, cron, backend/flavor, tasks]
    end

    L1 --> L2

    subgraph L2[Layer 2: Collisions]
        B[schema.table unique across all workspaces]
    end

    L2 --> L3

    subgraph L3[Layer 3: Catalog]
        C[Table existence + schema compatibility in DuckLake]
    end

    L3 --> L4

    subgraph L4[Layer 4: Dry Run]
        D["pixi run --manifest-path workspaces/{name}/pixi.toml dry-run + output validation"]
    end

    L4 -->|pass| OK[ready to merge]
    L4 -->|fail| BLOCK[blocked with details]

Loading

Layers 1-2 and 4 work on fork PRs without secrets. Layer 3 gracefully skips when S3 credentials are unavailable.

After validation, a maintainer can trigger full extraction to a staging prefix:

/run-extract              # auto-detects changed workspaces
/run-extract my-pipeline  # specific workspace

Staging data is auto-cleaned when the PR closes.

Data Flow

flowchart TD
    subgraph Workspace[Workspace Extraction]
        EX["pixi run --manifest-path workspaces/{name}/pixi.toml pipeline"]
        OUT[Parquet in $OUTPUT_DIR/]
        EX --> OUT
    end

    subgraph Upload[Workflow Upload]
        S5[s5cmd to S3]
        OUT --> S5
    end

    subgraph Catalog[DuckLake Federation]
        GC[Global catalog]
        S5 -->|"ducklake_add_data_files()"| GC
    end

    GC -->|query| Q["ATTACH 'ducklake:s3://.../catalog.duckdb' AS registry (READ_ONLY);<br/>SELECT * FROM registry.schema.table"]

Loading

Workspace code has READ-ONLY S3 access. The workflow handles uploads with write credentials.

Project Structure

ai-data-registry/
├── pixi.toml                  # Shared tools (GDAL, DuckDB, gpio, s5cmd, pnpm)
├── pixi.lock                  # Root lock for shared tools only
├── CLAUDE.md                  # AI instructions (shared core + role router)
├── CONTRIBUTING.md            # Contributor guide (workspace creation, contract, PR)
├── MAINTAINING.md             # Maintainer guide (CI/CD, DuckLake, infra)
├── README.md
├── LICENSE
├── .github/
│   ├── registry.config.toml   # Backend + storage config
│   ├── scripts/               # CI scripts (13 Python, uv + PEP 723)
│   └── workflows/             # 11 workflows (validation, extraction, scheduling)
├── workspaces/
│   └── test-minimal/          # Reference implementation
│       ├── pixi.toml          # Full [tool.registry] contract example
│       ├── pixi.lock          # Workspace lock (committed, isolated)
│       ├── extract.py         # Extraction script (writes to $OUTPUT_DIR)
│       └── validate_local.py  # Local validation
├── research/
│   └── architecture.md        # Full platform architecture
├── docs/
│   ├── secrets-setup.md       # Repository secrets reference
│   └── tool-versions.md       # Shared tool versions + deps guide
└── .claude/
    ├── rules/                 # 12 rules (auto-load by file path)
    ├── commands/              # 7 slash commands
    ├── skills/                # 6 skills (duckdb, gdal, geoparquet, etc.)
    └── agents/                # 3 agents (data-explorer, data-quality, pipeline)

Registry Setup (Template Users)

If you created this repo from the GitHub template, see docs/template-setup.md for full instructions. Quick version:

1. Run ./setup.sh (or .\setup.ps1 on Windows). It generates .env for local secrets.
2. Configure storage in .github/registry.config.toml
3. Set GitHub repo secrets per docs/secrets-setup.md
4. Open a PR with a workspace to verify

Shared Tools

All tools run through pixi. Never run directly.

Tool	Command
GDAL >=3.12.3	pixi run gdal ...
DuckDB >=1.5.2	pixi run duckdb ...
gpio >=1.0.0	pixi run gpio ...
s5cmd >=2.3.0	pixi run s5cmd ...
Python >=3.12	pixi run python ...
pnpm >=10.32.1	pixi run pnpm ...

Full versions and dependency guide: docs/tool-versions.md

Claude Code

This repo includes a full AI-assisted development setup in .claude/. Contributors can use Claude Code to scaffold workspaces, debug pipelines, and explore data:

Command	What it does
/new-workspace <name> <lang>	Scaffold workspace with full contract
/inspect-file <path>	Inspect data file (schema, rows, spatial)
/query <SQL>	Run DuckDB query
/add-dep <pkg> [-w ws]	Add dependency
/convert <in> <out>	Convert geospatial formats

Docs

Document	Audience	What it covers
CONTRIBUTING.md	Contributors	Workspace creation, contract, PR flow
MAINTAINING.md	Maintainers	CI/CD, DuckLake, infra, debugging
research/architecture.md	Both	Full platform architecture
docs/template-setup.md	Template users	Create your own registry from template
docs/secrets-setup.md	Maintainers	Repository secrets configuration
docs/tool-versions.md	Both	Shared tool versions and deps guide

License

CC BY 4.0 - Walkthru.Earth - See LICENSE