Automate monthly incremental catalog updates via GitHub Actions

.github/workflows/update.yml

@@ -0,0 +1,160 @@
+name: Monthly incremental update
+
+# Detect new DEM GeoTIFFs on the BC objectstore, build STAC items for them,
+# sync to s3://stac-dem-bc, and commit refreshed data/ caches back to main.
+# Modeled on water-temp-bc's snapshot.yml: monthly cron on a GitHub-hosted
+# runner, AWS auth via OIDC against role_gha_stac_dem_bc (provisioned by
+# NewGraphEnvironment/rtj#184, trust scoped to main).
+#
+# detect step exit contract: 0 = no changes (skip rest, succeed),
+# 1 = changes detected (continue), 2 = error (fail). A deletions-only month
+# exits 1 with no urls_new.txt - item steps are skipped but the cache commit
+# still records the deletions.
+#
+# State model: data/ caches persist only via the end-of-job commit, so a
+# failed run discards its partial state and the next run re-detects cleanly.
+# pgstac registration is a separate manual step on geoserv - see
+# scripts/README.md.
+
+on:
+  schedule:
+    # 3rd of the month, 09:23 UTC - staggered from water-temp-bc (1st) and
+    # off the top of the hour. GitHub may delay scheduled runs under load.
+    - cron: '23 9 3 * *'
+  workflow_dispatch:
+
+permissions:
+  id-token: write   # OIDC token for AWS
+  contents: write   # commit refreshed data/ caches back to main
+
+concurrency:
+  group: stac-update
+  cancel-in-progress: false
+
+env:
+  STAC_OUTPUT_DIR: ${{ github.workspace }}/stac_out
+
+jobs:
+  update:
+    runs-on: ubuntu-latest
+    # Recent growth averages ~7.6k files/month ~ 75-90 min at observed rates.
+    # Oversized batches (>~35k) cannot fit any timeout - see README triage.
+    timeout-minutes: 330
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: r-lib/actions/setup-r@v2
+        with:
+          use-public-rspm: true
+
+      - uses: r-lib/actions/setup-r-dependencies@v2
+        with:
+          extra-packages: any::sessioninfo
+
+      - uses: astral-sh/setup-uv@v5
+
+      - name: Install Python dependencies
+        run: |
+          uv venv --python 3.12 .venv
+          uv pip install --python .venv/bin/python \
+            "pystac[validation]>=1.12.0" "pystac-client>=0.8.0" \
+            "rio-stac>=0.11.0" "rasterio>=1.4.0" rio-cogeo shapely \
+            pandas requests tqdm deepdiff
+          .venv/bin/python -c "import rasterio, rio_stac, pystac, jsonschema; print('imports OK, rasterio', rasterio.__version__)"
+
+      - uses: aws-actions/configure-aws-credentials@v4
+        with:
+          role-to-assume: arn:aws:iam::414155577829:role/role_gha_stac_dem_bc
+          aws-region: us-west-2
+
+      - name: Detect changes
+        id: detect
+        shell: bash
+        run: |
+          set +e
+          Rscript scripts/detect_changes.R
+          code=$?
+          set -e
+          echo "detect exit code: $code"
+          case "$code" in
+            0) echo "changes=false" >> "$GITHUB_OUTPUT" ;;
+            1) echo "changes=true"  >> "$GITHUB_OUTPUT" ;;
+            *) exit "$code" ;;
+          esac
+          if [ -s data/urls_new.txt ]; then
+            echo "new URLs: $(wc -l < data/urls_new.txt)"
+            echo "new_urls=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "no new URLs (no changes, or deletions only)"
+            echo "new_urls=false" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Check source URL access (warn-only)
+        if: steps.detect.outputs.new_urls == 'true'
+        continue-on-error: true
+        run: .venv/bin/python scripts/urls_check_access.py --urls-file data/urls_new.txt
+
+      - name: Fetch current collection from S3
+        if: steps.detect.outputs.new_urls == 'true'
+        run: |
+          mkdir -p "$STAC_OUTPUT_DIR"
+          curl -fsSL https://stac-dem-bc.s3.amazonaws.com/collection.json \
+            -o "$STAC_OUTPUT_DIR/collection.json"
+
+      - name: Create STAC items (incremental)
+        if: steps.detect.outputs.new_urls == 'true'
+        run: .venv/bin/python scripts/item_create.py --incremental
+
+      - name: Count created items (warn on shortfall)
+        id: created
+        if: steps.detect.outputs.new_urls == 'true'
+        run: |
+          NEW=$(wc -l < data/urls_new.txt | tr -d ' ')
+          CREATED=$(find "$STAC_OUTPUT_DIR" -maxdepth 1 -type f -name "*.json" ! -name "collection.json" | wc -l | tr -d ' ')
+          echo "expected $NEW new items, created $CREATED"
+          echo "count=$CREATED" >> "$GITHUB_OUTPUT"
+          if [ "$CREATED" -lt "$NEW" ]; then
+            echo "::warning title=Item shortfall::created $CREATED of $NEW new items - some source URLs failed metadata extraction or were invalid; see run-logs artifact and the triage notes in scripts/README.md"
+          fi
+
+      # created == 0 (an all-invalid batch, precedent: the 90 parenthesized
+      # files arrived as one such delta) must not wedge the month: skip
+      # validate/sync, stay green with the shortfall warning, and let the
+      # cache commit record the batch as attempted.
+      - name: Validate new items (gate)
+        if: steps.detect.outputs.new_urls == 'true' && steps.created.outputs.count != '0'
+        run: .venv/bin/python scripts/item_validate.py --items-dir "$STAC_OUTPUT_DIR" --incremental
+
+      - name: Sync catalog to S3
+        if: steps.detect.outputs.new_urls == 'true' && steps.created.outputs.count != '0'
+        run: bash scripts/s3_sync-ci.sh
+
+      - name: Commit refreshed caches
+        if: steps.detect.outputs.changes == 'true'
+        run: |
+          git config user.name "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          git add -A data/
+          if git diff --cached --quiet; then
+            echo "no cache changes to commit"
+            exit 0
+          fi
+          git commit -m "Monthly incremental update: refresh caches ($(date -u +%Y-%m))"
+          git pull --rebase origin main
+          git push origin HEAD:main
+
+      - name: Upload run logs
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: run-logs
+          overwrite: true   # re-runs of failed jobs otherwise 409 on the existing artifact
+          path: |
+            logs/*.log
+            data/urls_access_checks.csv
+          if-no-files-found: ignore
+
+      - name: Session info
+        if: always()
+        run: Rscript -e 'sessioninfo::session_info()'

CLAUDE.md

@@ -115,10 +115,8 @@ Source URLs → GeoTIFF Validation → Item Creation → JSON Validation → Reg
 - No R dependency for core workflows
 
 ### SRED Tracking
-- Primary: https://github.com/NewGraphEnvironment/sred-2025-2026/issues/8
-- Secondary: https://github.com/NewGraphEnvironment/sred-2025-2026/issues/3
-- Repo issue: https://github.com/NewGraphEnvironment/stac_dem_bc/issues/3
-- Milestone: https://github.com/NewGraphEnvironment/sred-2025-2026/milestone/1
+- Primary: https://github.com/NewGraphEnvironment/sred/issues/8 — PR-body xref form: `Relates to NewGraphEnvironment/sred#8`
+- (The old `sred-2025-2026` repo refs are superseded — SRED tracking lives in the `sred` repo)
 
 ---
 
@@ -220,10 +218,8 @@ WHY: Reprocessing same URLs (e.g., after failures, testing) would create duplica
 ### File Locations
 - **Main repo:** `/Users/airvine/Projects/repo/stac_dem_bc`
 - **Phase 1-2 worktree:** `/Users/airvine/Projects/repo/stac_dem_bc-phase1-2-modernization`
-- **Infrastructure repo:** `/Users/airvine/Projects/repo/awshak` (future migration)
-- **Local STAC output:** `/Users/airvine/Projects/gis/stac_dem_bc/stac/prod/stac_dem_bc`
-- **S3 bucket:** `s3://stac-dem-bc/`
-- **VM path:** `/home/airvine/stac_dem_bc/`
+- **Infrastructure repo:** `/Users/airvine/Projects/repo/rtj` (formerly awshak; provisions the bucket, OIDC role, and geoserv STAC host)
+- **STAC catalog:** `s3://stac-dem-bc/` is the only complete copy. Local builds write to a scratch workspace via the `STAC_OUTPUT_DIR` env override in `scripts/stac_utils.py` (the old `/Users/airvine/Projects/gis/.../stac/prod` dir is empty/historical)
 
 <\!-- BEGIN SOUL CONVENTIONS — DO NOT EDIT BELOW THIS LINE -->
 

DESCRIPTION

@@ -0,0 +1,16 @@
+Package: stac.dem.bc
+Type: Project
+Title: Manifest for stac_dem_bc dependency resolution
+Version: 0.0.0.9000
+Description: Not an R package. This DESCRIPTION exists so that
+    r-lib/actions/setup-r-dependencies can resolve runtime deps for the
+    monthly update GitHub Actions workflow (change detection via ngr).
+    Mirrors the water-temp-bc pattern.
+License: MIT
+Encoding: UTF-8
+Imports:
+    fs,
+    ngr,
+    readr
+Remotes:
+    NewGraphEnvironment/ngr@519c03baa8554b9a82ff995dbe510bfb923a624d

environment.yml

@@ -7,7 +7,7 @@ dependencies:
   - pip
   - pip:
       # Core STAC packages
-      - pystac>=1.12.0
+      - pystac[validation]>=1.12.0  # validation extra = jsonschema, required by item_validate.py
       - pystac-client>=0.8.0
       - rio-stac>=0.11.0
       # Geospatial packages
@@ -17,5 +17,6 @@ dependencies:
       # Data processing
       - pandas
       # Utilities
+      - requests  # HTTP checks (stac_utils, urls_check_access.py)
       - tqdm
       - deepdiff  # JSON/dict comparison for QA and debugging

planning/active/findings.md

@@ -0,0 +1,52 @@
+# Findings — Automate monthly incremental catalog updates via GitHub Actions (#23)
+
+## Issue context
+
+Issue #23 (filed 2026-07-18): catalog five months stale (inventory 2026-02-18, S3 build 2026-02-11, pgstac registration 2026-02-13 with 58,019 items). Phase 3 automation from #5 never re-tracked after PR #9's "Closes #5" auto-closed the umbrella. Adopt the water-temp-bc pattern (monthly GHA cron + OIDC, decision record NewGraphEnvironment/water-temp-bc#17, reference workflow `.github/workflows/snapshot.yml`). Companion infra: rtj#184 (`modules/gha_s3_role` consumer). Registration stays manual-on-geoserv for v1; incremental pypgstac upsert is a named follow-up. Out of scope: full rebuilds (rtj#49), source-file deletions, #16 closure.
+
+## Verified pipeline contracts (2026-07-18 exploration)
+
+- `scripts/detect_changes.R` — self-contained: fetches fresh listing (`ngr::ngr_s3_keys_get`), diffs against `data/urls_list.txt`, writes/deletes `urls_new.txt` + `urls_deleted.txt`, overwrites the cache, sink()-logs to `logs/`. Exit 0 = no changes, 1 = changes; **R errors also exit 1** (ambiguous — fix in Phase 1) and a **deletions-only month exits 1 with no urls_new.txt** (workflow must branch on file presence). `urls_fetch.R` is redundant in CI.
+- `scripts/item_create.py --incremental` — reads `data/urls_new.txt`, needs only `$OUTPUT_DIR/collection.json` locally (10.9 MB from S3; item JSONs not required), appends links with duplicate prevention, saves collection. `get_output_dir()` (`scripts/stac_utils.py:39`) hardcodes `/Users/airvine/...` — used by item_create, collection_create, item_reprocess.
+- `scripts/item_validate.py` — already parameterized (`--items-dir`, `--incremental`) and already exits non-zero when any item invalid (verified `return 0 if invalid == 0 else 1`). Caveat: the count spans history + new, so one bad item blocks the whole batch and re-fails monthly — v1 fail-loud by design; remediation via `urls_invalid_items.txt` + `item_reprocess.py` (document in README triage).
+- `scripts/urls_check_access.py` — **hard-exits 1 on any inaccessible URL** (even from cache). `build_safe.sh` Step 3.5 treats it warn-only; the workflow must too. `data/urls_access_checks.csv` is currently untracked — commit-back uses `git add -A data/` and the CSV gets committed (GeoBC-report purpose from #13).
+- `scripts/s3_sync.R` — laptop-only (`--delete --profile airvine`, hardcoded path). Never run in CI: `--delete` from a stateless runner would wipe the 58k items. **S3 is the only catalog copy** (local prod dir verified empty; CLAUDE.md's "Local STAC output" note is stale).
+- `environment.yml` — pure pip under conda → `uv pip install` works today; #16 not a blocker. Missing `jsonschema` (via `pystac[validation]` — the validate gate depends on it) and `requests`.
+- **~2,098 URLs in the cache have no catalog item** (60,126 URLs vs 58,028 validated items), including the 90 parenthesized files added post-build (commit 0d5ab5c, 2026-02-18 — after the Feb 11 S3 build). Being cached, detect_changes will never re-flag them → one-time reconciliation (`urls_reconcile.py`) trims the cache to item-backed URLs so the catch-up run picks them up naturally.
+- State atomicity on a runner comes free for the linear path (caches persist only via end-of-job commit; failures self-heal) — but a truncated listing would poison the cache silently, hence the <90% plausibility guard.
+
+## Infra (rtj + water-temp-bc)
+
+- `modules/gha_s3_role` (rtj#147): role name derives to `role_gha_stac_dem_bc`; ListBucket + GetBucketLocation on bucket, Get/PutObject on objects, **no DeleteObject by default** — sufficient for no-delete sync and the backstop against the `--delete` wipe scenario. Account OIDC provider already provisioned. ARN pattern: `arn:aws:iam::414155577829:role/role_gha_stac_dem_bc`.
+- Bucket `stac-dem-bc`: public + CORS, **no versioning** (water-temp-bc has it) → optional versioning rider in Phase 2 since collection.json is overwritten in place every run.
+- STAC API = stac-fastapi-pgstac + TiTiler + Caddy on **geoserv** (`images.a11s.one`); DB `stac` holds stac-dem-bc. Registration: `rtj/scripts/geoserv/stac_register-pypgstac.sh` on-host — full delete-and-reload, 46 min for 58k items (S3 download dominates; DB load 10 s).
+- rtj has an in-flight branch (`172-stac-floodplains-bc-bucket`) → Phase 2 branches off rtj main. No branch protection on stac_dem_bc main; `claude.yml` has no push trigger; GITHUB_TOKEN pushes don't retrigger workflows.
+- Public repo → GitHub disables cron after 60 days without repo activity; no-change months produce no commits (document in triage).
+- Side finding (out of scope, flagged to user): `rtj/scripts/geoserv/stac_register-pypgstac.sh:82` still has the parallel-append interleave pattern from the conventions — harmless for KB-scale DEM items, trips on large payloads.
+
+## Phase 1 cold-path rehearsal (2026-07-18)
+
+Stateless-runner path exercised locally with a uv venv (Python 3.12, exact CI dep list — the install itself doubles as the CI-install rehearsal; `import rasterio, rio_stac, pystac, jsonschema` smoke passed, rasterio 1.5.0 manylinux/macos wheels bundle GDAL+PROJ):
+
+- Workspace = temp `STAC_OUTPUT_DIR` holding only the S3-fetched `collection.json` (10.9 MB); synthetic 3-URL `urls_new.txt` from existing catalog URLs.
+- `item_create.py --incremental`: loaded 58,019 links, extracted metadata remotely (~2.3 s/file), wrote 3 item JSONs, added 0 links / skipped 3 duplicates (dedupe ✓), saved collection into the workspace ✓.
+- `item_validate.py --items-dir "$STAC_OUTPUT_DIR" --incremental`: found 3 files, all already in the committed CSV → "Validating 0 new items", exit 0 ✓ (gate semantics for genuinely-new items verified by code review).
+- `s3_sync-ci.sh --dryrun`: 3 item uploads then collection.json last, **zero delete operations** ✓; empty-array expansion works on macOS bash 3.2.
+- `urls_reconcile.py` dry-run: 60,126 cached / 58,019 item-backed / **2,107 never built** (parenthesized "(2)" files lead the list). Item-backed count exactly matches the Feb pgstac registration (58,019).
+- Rehearsal side-effects on tracked `data/` files restored via git checkout (the run's `stac_geotiff_checks.csv` rewrite also shrank it 60,325→60,307 rows — pre-existing item_create cache-merge behavior, not investigated here).
+
+**URL-scheme quirk (load-bearing):** every URL in the cache, the checks CSV, and ngr's live output uses a single-slash scheme (`https:/nrs...`, 0 double-slash of 98,039 fetched). Format is uniform across all three sources, so diffs are sane — but any future ngr "fix" to emit `https://` would make one detection run flag everything new+deleted. The plausibility guard doesn't catch same-size format flips; the massive-new-count log line is the tell.
+
+**Catch-up sizing (changes Phase 4):** live fetch 2026-07-18 returned **98,039 URLs vs 60,126 cached → ~37,900 new in 5 months (+63%)**, echoing the Feb discovery (+35,569). With the 2,107 reconciled, catch-up ≈ 40k items ≈ 6+ h at the documented ~6,450 items/h — beyond the 330-min GHA timeout and non-convergent on retry. The plan's documented oversized-batch fallback (manual local run) is therefore the catch-up path; the dispatch run verifies steady state instead. Steady-state months at the observed growth rate (~7,600 files/month ≈ 75 min) fit the timeout comfortably.
+
+## Code-check (3 fresh-eyes rounds, 2026-07-18)
+
+Round 1: `library(ngr)` sat before the tryCatch — a missing package on a fresh runner would exit 1, which the workflow reads as "changes detected" against the stale tracked urls_new.txt. Fixed by removing library() calls entirely (namespaced calls fail inside tryCatch → exit 2). Round 2: a trailing slash in `STAC_S3_BUCKET` would make the collection.json `cp` write a hidden `/collection.json` key — sync fine, cp exit 0, live collection silently never updates. Fixed with `BUCKET="${BUCKET%/}"`. Round 3 clean; also verified the 8 validation-CSV ids absent from the cache are the 8 known upstream-deleted files, and that the exit-0 path deletes stale tracked urls_new/urls_deleted from the checkout (stateless runner can't reprocess stale change files).
+
+## Code-check Phase 3 (2 fresh-eyes rounds, 2026-07-18)
+
+Round 1: (a) partial item-build failures are silent-permanent — item_create exits 0 on per-item failures, the full fresh list commits, and failed reads get cached as not-a-GeoTIFF (never retried); mitigated with a warn-only shortfall step + README remediation recipe, deeper failure accounting in item_create deferred to a follow-up issue. (b) upload-artifact v4 without `overwrite: true` 409s on re-runs of failed jobs — fixed. (c) README catch-up tense implied already-done — fixed. Round 2 verified all three fixes and found the zero-created corner: an all-invalid batch (precedent: the 90 parenthesized files arrived as one such delta) would hard-fail item_validate ("no item JSONs found" → exit 1) and repeat monthly; fixed by exporting the created count and gating validate/sync on it — the all-invalid month stays green with the warning and the cache commit records the batch as attempted. Also traced and confirmed the README remediation recipe end-to-end (CSV row delete → reconcile --apply → re-detect → rebuild).
+
+## Plan-review disposition (adversarial Plan-agent pass, 2026-07-18)
+
+Absorbed into phases: warn-only access check (B1), deletions-only branch + `git add -A data/` commit semantics (B2, G4), listing plausibility guard (G5), items-before-collection sync order (G6), orphaned-URL reconciliation (G7), rebase-before-push (G8), explicit exit-code capture (G9), cron auto-disable doc (G10), oversized-batch fallback doc (A12 — a naive "process first N" cap would orphan the remainder because the cache updates eagerly), import smoke check + Python ≥3.10 pin (A13, A14), ngr SHA pin in DESCRIPTION Remotes (S18), Phase 4 count math (A21). Dropped as verified-unnecessary: "add exit code to item_validate" (already correct). Catch-up sizing note: at ~6,450 items/hour, a 35k-scale surprise busts the 330-min timeout non-convergently — log `wc -l urls_new.txt` early; fallback is a manual local run.