Skip to content

Home

Jump to bottom
YONG WOOK KIM edited this page Feb 9, 2026 · 6 revisions

This document provides a high-level overview of the OEPS (Opioid Environment Policy Scan) project: what it is, how it works, how to manage it, how data flows, and where updates happen.

What Is OEPS?

OEPS is an open-source data ecosystem that characterizes the multi-dimensional risk environment impacting opioid use and health outcomes across the United States. It provides:

  • Data at multiple spatial scales (state, county, tract, ZCTA)
  • Time periods from 1980 to today
  • Variable themes: Geography, Social, Environment, Economic, Policy, Outcome, Composite

Key URLs

Resource URL
OEPS Explorer (interactive map + download) oeps.healthyregions.org
Technical documentation healthyregions.github.io/oeps
R data package oepsdata.healthyregions.org

Architecture Overview

┌─────────────────────────────────────────────────────────────────────────────┐
│                           SOURCES OF TRUTH                                   │
├─────────────────────────────────────────────────────────────────────────────┤
│  Registry (JSON)          │  Raw Data (CSV)        │  Metadata (Markdown)   │
│  backend/oeps/registry/   │  backend/oeps/data/    │  metadata/             │
│  - variables/*.json       │  - tables/*.csv        │  - Access_MOUDs.md     │
│  - table_sources/*.json   │                        │  - etc.                │
│  - metadata/*.json        │                        │                        │
│  - geodata_sources/*.json │                        │                        │
└─────────────────────────────────────────────────────────────────────────────┘
                                        │
                                        ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                        BACKEND (Flask CLI)                                   │
│  Commands: build-explorer, merge-csv, create-data-package, build-docs, etc.  │
└─────────────────────────────────────────────────────────────────────────────┘
                                        │
                    ┌───────────────────┼───────────────────┐
                    ▼                   ▼                   ▼
┌──────────────────────┐  ┌──────────────────────┐  ┌──────────────────────┐
│  OEPS Explorer       │  │  Data Packages       │  │  BigQuery / oepsData  │
│  (Next.js, Netlify)  │  │  (ZIP downloads)     │  │  (R package, cloud)   │
│  - Map               │  │  DSuite2018,         │  │                       │
│  - Docs page         │  │  DSuite2023, etc.    │  │                       │
│  - Download page     │  │  (S3)                │  │                       │
└──────────────────────┘  └──────────────────────┘  └──────────────────────┘

How It Works

Core Idea

Static content (registry + CSV data) is manipulated by CLI commands to produce exports, transformations, and derived outputs. The registry is the central hub that describes all variables, table sources, metadata, and geodata sources.

Registry Structure

Component Location Purpose
Variables registry/variables/*.json One file per variable (name, title, type, metadata, table_sources)
Table sources registry/table_sources/*.json One per CSV (e.g., county-2023.csv)
Metadata registry/metadata/*.json Links to markdown docs; theme, construct, proxy
Geodata sources registry/geodata_sources/*.json Shapefiles for joins (S3 URLs)

Data Model

  • Variable = one column concept (e.g., TotPop, BupTmDr)
  • Table source = one CSV (e.g., county-2023) at one geography and one year
  • HEROP_ID = join key used across all data
  • Variables can appear in many table sources; table sources link to geodata for spatial joins

Where Data Lives and Gets Updated

1. Raw CSV Data

Location Description
backend/oeps/data/tables/ Main data: {geography}-{year}.csv (e.g., tract-2023.csv)
data/ Travel time matrices, other auxiliary data
data_to_merge/ Staging CSVs awaiting merge into the system

How it's updated: flask merge-csv merges incoming CSVs into existing table source CSVs.

2. Registry (Variables, Table Sources, Metadata)

Location Description
backend/oeps/registry/variables/ Variable definitions
backend/oeps/registry/table_sources/ Table source definitions
backend/oeps/registry/metadata/ Metadata document references
backend/oeps/registry/geodata_sources/ Geodata source definitions

How it's updated:

  • Pages CMS (app.pagescms.org/healthyregions/oeps) for metadata and variables
  • CLI: flask create-table-source, flask merge-csv (updates variable table_sources), flask remove-variable, flask move-variable

3. Metadata Markdown

Location Description
metadata/*.md Human-readable docs for each variable group (e.g., Access_MOUDs.md)

How it's updated: Edited manually or via Pages CMS.

4. Explorer (Map, Docs, Download)

Location Description
explorer/config/ variables.json, sources.json, symbology.json
explorer/meta/ variables.json, csvDownloads.json, metadataVariables.json
explorer/public/csv/ Subset CSVs for the map

How it's updated: Generated by flask build-explorer (and optionally --map-only, --docs-only, --upload-map-data). Do not edit these files directly.

5. Data Packages (Downloadable ZIPs)

Location Description
backend/oeps/data/package_rules/ Rules per package (e.g., DSuite2023)
S3 / download page Final packaged ZIPs (e.g., oeps-DSuite2023)

How it's updated: flask create-data-package --config DSuite2023 --skip-foreign-keys --zip (and optionally --upload).

6. Derived Documentation

Location Description
docs/src/reference/ Registry CSVs, CLI reference, data dictionaries, BigQuery schema

How it's updated: flask build-docs (and flags like --registry-only, --cli-only, etc.).

How to Manage OEPS

Adding New Data

  1. Prepare CSV (CamelCase columns, join column, one year/geography)
  2. Create or update metadata (Pages CMS or JSON)
  3. Create variables in registry (Pages CMS)
  4. Create table source if needed: flask create-table-source -n county-2024 -g counties-2020
  5. Merge: flask merge-csv -s path/to/incoming.csv -t county-2024
  6. Re-provision explorer: flask build-explorer
  7. Optionally: regenerate data package, sync to BigQuery

Key CLI Commands

Command Purpose
flask build-explorer Build map, docs, and download page config
flask build-explorer --map-only --upload-map-data Build map and upload to S3 (production)
flask build-explorer --docs-only Build only docs/download page config
flask merge-csv -s <csv> -t <table_source> Merge CSV into a table source
flask create-data-package --config DSuite2023 --skip-foreign-keys --zip Create a data package
flask create-table-source -n <name> -g <geodata> Create new table source
flask validate-registry Validate registry references
flask build-docs Generate derived docs (CLI ref, registry CSVs, etc.)
flask remove-variable -n <var> -t <table_source> Remove variable from table source

Automation

The Generate Derived Data GitHub Action runs on push to main when backend/oeps/registry/** or backend/oeps/data/** change. It:

  1. Runs flask build-docs (CLI, registry, data dictionaries, BigQuery)
  2. Runs flask build-explorer --upload-map-data (on main only)
  3. Commits generated files back to the repo

Data Flow Summary

From To Command / Process
Registry + CSVs Explorer (map, docs, download) flask build-explorer
Registry + CSVs Data packages (ZIP) flask create-data-package
Registry Derived docs (CSVs, CLI ref) flask build-docs
Incoming CSV Table source CSVs flask merge-csv
Pages CMS Registry JSON Edits via Pages CMS UI

Important Concepts

Map vs. Data Package vs. Registry

  • Registry = source of truth for variables and where they appear
  • Map = uses registry directly; shows all variables with map-ready data
  • Data packages = curated subsets defined by package rules (e.g., DSuite2023 excludes Bk/Wk)

Removing a variable from a data package does not remove it from the registry or map.

Join Key

  • HEROP_ID = primary join key
  • Format includes geography prefix (e.g., US14000US06001400100 for tract)
  • Incoming data can use GEOID, FIPS, or ZCTA5; these are converted to HEROP_ID

Last updated: February 2025

Clone this wiki locally