Framework repository for the opendata-etl project: an AGPLv3 toolkit for Postgres-targeted ETL, derived jobs, read-only APIs, and docs aggregation. Dataset behavior is driven by external definition repositories loaded at deployment time.
- The only repository in the multi-repo workspace that contains executable framework code for this stack.
- Licensed under the GNU Affero General Public License v3.0 (see
LICENSE). SPDX short identifier: AGPL-3.0 (Python sources in this repo use# SPDX-License-Identifier: AGPL-3.0-onlywhere applicable). - Distributed primarily as a Docker image and Python package metadata (
pyproject.toml); deployments consume released artifacts rather than forking the framework to ship city-specific logic.
- Not a definition repository. City- or domain-specific datasets, dbt models, API endpoint specs, and narrative docs live in repos such as
nycdb2. Those repos are separate creative works: shipping YAML/SQL/markdown that the framework reads does not, by itself, make them derivatives of this framework for licensing purposes. They may use their own license at the maintainer's discretion. - Not a place for deployment secrets or environment-specific wiring. Deployment configuration (for example
definitions.yml, environment files, Terraform) lives in directories such asopendata-etl-deployment. - Not a store for protected source credentials. Access to data sources is declared by named references in configuration; values are resolved at runtime from the environment, cloud secret stores, or IAM—never committed to git.
Local and CI environments should use pinned git revisions for each definition repository (commit SHA or immutable tag). For offline or rapid iteration, git URLs may use the file:// scheme to point at a local clone. The loader is pipeline.definitions.load_definitions (see Step 3 in the master plan).
Docker Compose (PostGIS, MinIO, Dagster shell, FastAPI shell): copy .env.example to .env, then from the repo root run docker compose config and docker compose up --build. Detailed workflow, env vars, and health-check notes: docs/local-development.md.
| Path | Role |
|---|---|
pipeline/ |
Definitions loader, validation, minimal dagster_defs shell for Compose. |
api/ |
FastAPI app shell (api/app.py) with GET /healthz. |
schemas/ |
JSON Schema contracts for YAML (repo.yml, datasets, API endpoints, definitions manifest). |
scripts/ |
CLI and validation (scripts/validate_definitions.py). |
docs/ |
MkDocs Material site (mkdocs.yml); run aggregate_docs.py + gen_docs.py before mkdocs build. |
examples/ |
Sample definition repo tree and definitions*.yml manifests. |
docker-compose.yml |
Local runtime: postgres, minio, dagster, api. |
infra/aws/ |
Reference Terraform modules for standard AWS deployments. |
.github/workflows/ |
CI: Python tests, fixture validation, docker compose config (no image pull). |
pip install -e ".[dev]"— local test and validation tools, including pytest anddg.pip install -e ".[docs]"— MkDocs and Material theme (aggregated documentation site).pip install ".[compose]"— Dagster webserver, FastAPI, uvicorn, plus PyYAML/jsonschema (matches the application image install).
CI installs pip install -e ".[dev,docs]" so tests, definition validation, dg check, dbt parsing, and MkDocs all use the same packaged extras.
Production-like use follows a three-layer model: this framework repo, one or more definition repos, and an operator-owned deployment repo. See deployment repositories for the split, image pinning, and Terraform consumption pattern.
The repo includes Docker Compose, the definitions loader, extract/load, dbt and API factories, monitoring, and an MkDocs Material documentation build (mkdocs.yml, scripts/aggregate_docs.py, scripts/gen_docs.py) with optional GitHub Pages deployment.
See CONTRIBUTING.md and CODE_OF_CONDUCT.md.