NewGraphEnvironment · NewGraphEnvironment · Feb 17, 2026 · Feb 17, 2026
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -86,7 +86,7 @@ data/
    - Uses pystac for spec compliance
    - Tracks validation errors for debugging
    - Filters items before PgSTAC registration
-   - Script: `scripts/validate_stac_items.py`
+   - Script: `scripts/item_validate.py`
 
 **Workflow integration:**
 ```

diff --git a/planning/active/findings.md b/planning/active/findings.md
@@ -0,0 +1,36 @@
+# Findings: Issue #7 - .qmd to .py Migration
+
+## Source Analysis
+
+### stac_create_item.qmd (~350 LOC)
+- **R dependency:** Only the conda env activation (reticulate) — not needed in standalone .py
+- **Python chunks:** imports, date extraction, config, pre-validation, parallel item creation
+- **Modes:** test_only, incremental, reprocess_invalid (3 boolean flags)
+- **Workers:** 32 threads (ThreadPoolExecutor)
+- **Dangling R chunk:** `json-clean` at bottom (eval=FALSE, manual delete utility) — don't migrate
+
+### stac_create_collection.qmd (~270 LOC)
+- **R dependency:** `ngr::ngr_s3_keys_get()` for S3 key fetch — REAL dependency, keep in R
+- **Python chunks:** imports, bbox functions, config, temporal extent, collection creation, validation
+- **R→Python bridge:** `test_only` and `test_number_items` passed via `r.test_only`
+- **Unused code:** `bbox_combined()` function defined but commented out (hardcoded bbox used instead)
+
+### Duplicated Functions (3 copies each)
+1. `date_extract_from_path()` — in item.qmd, collection.qmd, item_reprocess.py
+2. `datetime_parse_item()` — in item.qmd, collection.qmd, item_reprocess.py
+3. `check_geotiff_cog()` — only in item.qmd (but could be shared)
+
+### Config Constants (hardcoded in multiple files)
+- `path_local` (dev/prod) — in item.qmd, collection.qmd, item_reprocess.py, catalogue_qa.py
+- `path_s3_stac` — in item.qmd, item_reprocess.py
+- `path_s3` — in item.qmd, item_reprocess.py
+
+## Key Design Decisions
+
+- **fetch_urls.R stays in R:** No Python equivalent for `ngr::ngr_s3_keys_get()`. The R→Python handoff point is `data/urls_list.txt` (a text file), which is a clean interface.
+- **argparse for CLI:** All .py scripts should accept `--test`, `--test-count`, mode flags via CLI args instead of editing source code.
+- **logging module:** Replace `print()` statements with `logging.info()` etc. for proper log capture in cron.
+
+---
+
+**Last updated:** 2026-02-17
diff --git a/planning/active/progress.md b/planning/active/progress.md
@@ -0,0 +1,39 @@
+# Progress: Issue #7 - .qmd to .py Migration
+
+## Session: 2026-02-17
+
+### Completed
+- [x] Created branch `7-migrate-qmd-to-py`
+- [x] Analyzed stac_create_item.qmd (350 LOC, 3 modes, no real R dependency)
+- [x] Analyzed stac_create_collection.qmd (270 LOC, R dependency on ngr for S3 fetch)
+- [x] Identified 3x duplicated utility functions
+- [x] Created planning files (task_plan.md, findings.md, progress.md)
+
+### In Progress
+- [x] Phase 1.1: Created `scripts/stac_utils.py` with shared functions
+  - `date_extract_from_path()`, `datetime_parse_item()`, `check_geotiff_cog()`
+  - `fix_url()`, `url_to_item_id()`, `get_output_dir()`
+  - Path constants: `PATH_S3_STAC`, `PATH_S3_JSON`, `PATH_S3`, `PATH_RESULTS_CSV`, `BBOX_BC`
+- [x] Phase 1.2: Updated `item_reprocess.py` to import from `stac_utils`
+- [x] Phase 1.3: Verified imports and syntax
+
+- [x] Phase 1.4: Renamed all scripts to `noun_verb.py` convention
+- [x] Phase 1.5: Updated all cross-references (0 stale refs in code files)
+- [x] Phase 2.1: Created `scripts/item_create.py` (argparse CLI, logging, imports stac_utils)
+- All 6 .py scripts pass syntax checks
+- Local PROJ conflict blocks `rio_stac` import (homebrew vs conda) — VM-only testing
+
+- [x] Phase 3.1: Created `scripts/urls_fetch.R` (standalone R script for S3 key fetch)
+- [x] Phase 3.2: Created `scripts/collection_create.py` (argparse CLI, logging, imports stac_utils)
+- [x] Phase 4.1: Updated `scripts/build_safe.sh` to use new scripts (no more quarto render)
+  - Added urls_fetch.R step, validation step with item_validate.py
+
+### Next Up
+- [ ] Phase 4.2: Archive .qmd files
+- [ ] Phase 4.3: Update CLAUDE.md
+- [ ] Phase 4.4: Update README.md
+- [ ] Test equivalence on VM
+
+---
+
+**Last updated:** 2026-02-17
diff --git a/planning/active/task_plan.md b/planning/active/task_plan.md
@@ -0,0 +1,146 @@
+# Task Plan: Migrate .qmd to Standalone Scripts (Issue #7)
+
+**Status:** Phase 2 - In Progress
+**Branch:** `7-migrate-qmd-to-py`
+**Started:** 2026-02-17
+**Issue:** https://github.com/NewGraphEnvironment/stac_dem_bc/issues/7
+
+## Goal
+
+Migrate `stac_create_collection.qmd` and `stac_create_item.qmd` to standalone scripts for production automation. Keep .qmd files as archive/reference. Enable clean `python script.py` and `Rscript script.R` execution for Phase 3 VM cron jobs.
+
+## Naming Convention
+
+Scripts use `noun_verb.py` pattern for alphabetical grouping:
+
+| Old Name | New Name |
+|----------|----------|
+| `validate_stac_items.py` | `item_validate.py` |
+| `reprocess_invalid_items.py` | `item_reprocess.py` |
+| `extract_invalid_urls.py` | `item_extract_invalid.py` |
+| `qa_update_catalogue.py` | `catalogue_qa.py` |
+| _(new)_ | `item_create.py` |
+| _(new)_ | `collection_create.py` |
+
+## Current State
+
+**Existing .py scripts (renamed):**
+- `scripts/item_validate.py` (244 LOC)
+- `scripts/item_reprocess.py` (196 LOC) — updated to use stac_utils
+- `scripts/item_extract_invalid.py` (89 LOC)
+- `scripts/catalogue_qa.py` (243 LOC)
+- `scripts/stac_utils.py` (NEW — shared utilities)
+- `scripts/build_safe.sh` (218 LOC)
+
+**Still in .qmd (need migration):**
+- `stac_create_collection.qmd` — R chunk (S3 key fetch via `ngr`) + Python chunks (collection creation)
+- `stac_create_item.qmd` — R chunk (conda env) + Python chunks (validation, item creation)
+
+**R-only scripts (keep as-is):**
+- `scripts/detect_changes.R` — Uses `ngr::ngr_s3_keys_get()`, pure R is correct
+- `scripts/s3_sync.R` — AWS CLI wrapper, fine as R
+- `scripts/functions.R` — `vm_upload_run()` utility
+- `scripts/benchmark_fetch.R` — Dev tool
+- `scripts/footprint_visualize.R` — Exploratory (Issue #2)
+
+## Phases
+
+### Phase 1: Extract shared utilities ✅ COMPLETE
+**Goal:** Create shared Python module to eliminate duplication
+
+- [x] **1.1** Create `scripts/stac_utils.py` with shared functions
+- [x] **1.2** Update `item_reprocess.py` to import from `stac_utils.py`
+- [x] **1.3** Verify imports and syntax
+- [x] **1.4** Rename all scripts to `noun_verb.py` convention
+- [x] **1.5** Update all cross-references (scripts, CLAUDE.md, .qmd)
+
+---
+
+### Phase 2: Migrate stac_create_item.qmd → scripts/item_create.py ⬜ pending
+**Goal:** Standalone Python script for item creation
+
+- [ ] **2.1** Create `scripts/item_create.py` with:
+  - argparse CLI (`--test`, `--test-count N`, `--incremental`, `--reprocess-invalid`)
+  - Python logging module (not print statements)
+  - Import shared utils from `stac_utils.py`
+  - All functionality from stac_create_item.qmd Python chunks
+- [ ] **2.2** Test equivalence: run both .qmd and .py, compare output
+- [ ] **2.3** Update `scripts/build_safe.sh` to call .py instead of `quarto render`
+
+**Verify:** Create 10 test items with .py script, diff against .qmd output
+
+---
+
+### Phase 3: Migrate stac_create_collection.qmd → split R/Python ⬜ pending
+**Goal:** Standalone scripts for collection creation
+
+The collection .qmd has two distinct parts:
+1. **R chunk:** Fetches S3 keys via `ngr::ngr_s3_keys_get()` → `data/urls_list.txt`
+2. **Python chunks:** Creates collection JSON from urls_list.txt
+
+Migration approach:
+- [ ] **3.1** Create `scripts/urls_fetch.R` — Standalone R script for S3 key fetching
+  - Takes `--test` flag, outputs to `data/urls_list.txt`
+  - Replaces the R chunk in collection.qmd
+- [ ] **3.2** Create `scripts/collection_create.py` — Standalone Python script
+  - Reads `data/urls_list.txt` (produced by urls_fetch.R or detect_changes.R)
+  - argparse CLI (`--test`, `--test-count N`)
+  - Temporal extent calculation, spatial extent (hardcoded BC bbox)
+  - Collection creation and validation
+- [ ] **3.3** Test equivalence: compare collection.json from both approaches
+
+**Verify:** `Rscript scripts/urls_fetch.R && python scripts/collection_create.py` produces identical collection.json
+
+---
+
+### Phase 4: Update build_safe.sh and documentation ⬜ pending
+**Goal:** Wire everything together for production
+
+- [ ] **4.1** Update `scripts/build_safe.sh` to use new scripts:
+  - `Rscript scripts/urls_fetch.R` (or `Rscript scripts/detect_changes.R`)
+  - `python scripts/collection_create.py`
+  - `python scripts/item_create.py`
+  - `python scripts/item_validate.py`
+- [ ] **4.2** Archive .qmd files (move to `archive/` or add deprecation header)
+- [ ] **4.3** Update CLAUDE.md with new script paths and workflow
+- [ ] **4.4** Update README.md usage examples
+
+**Verify:** Full `build_safe.sh` run in test mode produces valid catalog
+
+---
+
+## Critical Decisions
+
+| Decision | Choice | Rationale |
+|----------|--------|-----------|
+| Naming convention | `noun_verb.py` | Groups related scripts alphabetically (item_*, collection_*) |
+| Keep R for S3 key fetch | Yes | `ngr::ngr_s3_keys_get()` has no Python equivalent, already works |
+| Shared Python module | `stac_utils.py` | Eliminates 3x duplication of date functions |
+| CLI interface | argparse | Standard, scriptable, supports `--test` flags |
+| Logging | Python `logging` module | Proper log levels, file output, captures in cron |
+| Archive .qmd | Keep in repo (header note) | Reference for literate programming approach |
+
+## Risks
+
+| Risk | Mitigation |
+|------|-----------|
+| Breaking production pipeline | Branch-based development, .qmd still works on main |
+| ngr R dependency hard to replace | Keep R script for S3 fetch, don't force all-Python |
+| Subtle behavior differences | Side-by-side output comparison before merging |
+
+## SRED Tracking
+
+- Primary: NewGraphEnvironment/sred-2025-2026#8
+- Secondary: NewGraphEnvironment/sred-2025-2026#3
+
+---
+
+## Errors Encountered
+
+| Error | Phase | Resolution |
+|-------|-------|------------|
+| PROJ env conflict | 1.3 | Local homebrew/conda conflict — not our bug, works on VM |
+
+---
+
+**Last updated:** 2026-02-17
diff --git a/scripts/build_safe.sh b/scripts/build_safe.sh
@@ -72,9 +72,9 @@ else
 fi
 
 # Check required files
-[[ -f "stac_create_collection.qmd" ]] || error "stac_create_collection.qmd not found"
-[[ -f "stac_create_item.qmd" ]] || error "stac_create_item.qmd not found"
-[[ -f "data/urls_list.txt" ]] || error "data/urls_list.txt not found"
+[[ -f "scripts/collection_create.py" ]] || error "scripts/collection_create.py not found"
+[[ -f "scripts/item_create.py" ]] || error "scripts/item_create.py not found"
+[[ -f "scripts/urls_fetch.R" ]] || error "scripts/urls_fetch.R not found"
 
 # =============================================================================
 # Step 1: Backup Current Production
@@ -102,38 +102,43 @@ log "✓ Build directory ready: $BUILD_DIR"
 log ""
 
 # =============================================================================
-# Step 3: Export Build Path for .qmd Files
+# Step 3: Fetch URLs
 # =============================================================================
 
-log "Step 3: Setting build environment..."
-export STAC_OUTPUT_DIR="$BUILD_DIR"
-log "✓ STAC_OUTPUT_DIR=$STAC_OUTPUT_DIR"
+log "Step 3: Fetching URLs from BC objectstore..."
+mkdir -p "$LOG_DIR"
+URLS_LOG="${LOG_DIR}/${TIMESTAMP}_urls_fetch.log"
+
+if Rscript scripts/urls_fetch.R 2>&1 | tee "$URLS_LOG"; then
+    log "✓ URLs fetched successfully"
+else
+    error "URL fetch failed - check $URLS_LOG"
+fi
 log ""
 
 # =============================================================================
-# Step 4: Run Collection Creation
+# Step 4: Create Collection
 # =============================================================================
 
 log "Step 4: Creating STAC collection..."
-mkdir -p "$LOG_DIR"
 COLLECTION_LOG="${LOG_DIR}/${TIMESTAMP}_collection.log"
 
-if quarto render stac_create_collection.qmd --execute 2>&1 | tee "$COLLECTION_LOG"; then
+if python scripts/collection_create.py 2>&1 | tee "$COLLECTION_LOG"; then
     log "✓ Collection created successfully"
 else
     error "Collection creation failed - check $COLLECTION_LOG"
 fi
 log ""
 
 # =============================================================================
-# Step 5: Run Item Creation
+# Step 5: Create Items
 # =============================================================================
 
-log "Step 5: Creating STAC items (this may take 1-2 hours)..."
+log "Step 5: Creating STAC items (this may take several hours)..."
 ITEMS_LOG="${LOG_DIR}/${TIMESTAMP}_items.log"
 START_TIME=$(date +%s)
 
-if quarto render stac_create_item.qmd --execute 2>&1 | tee "$ITEMS_LOG"; then
+if python scripts/item_create.py 2>&1 | tee "$ITEMS_LOG"; then
     END_TIME=$(date +%s)
     DURATION=$((END_TIME - START_TIME))
     MINUTES=$((DURATION / 60))
@@ -148,20 +153,28 @@ log ""
 # =============================================================================
 
 log "Step 6: Validating build output..."
-BUILD_COUNT=$(count_jsons "$BUILD_DIR")
+BUILD_COUNT=$(count_jsons "$PROD_DIR")
 
 if [[ $BUILD_COUNT -eq 0 ]]; then
-    error "Build validation failed: No JSON files created in $BUILD_DIR"
+    error "Build validation failed: No JSON files created"
 fi
 
-log "✓ Build validation passed: $BUILD_COUNT items created"
+log "✓ File count check passed: $BUILD_COUNT JSON files"
 
 # Check collection.json exists
-if [[ ! -f "$BUILD_DIR/collection.json" ]]; then
+if [[ ! -f "$PROD_DIR/collection.json" ]]; then
     error "Build validation failed: collection.json missing"
 fi
 
 log "✓ collection.json present"
+
+# Run STAC item validation
+VALIDATION_LOG="${LOG_DIR}/${TIMESTAMP}_validation.log"
+if python scripts/item_validate.py 2>&1 | tee "$VALIDATION_LOG"; then
+    log "✓ STAC item validation passed"
+else
+    log "⚠️  Some items failed validation - check $VALIDATION_LOG"
+fi
 log ""
 
 # =============================================================================
@@ -205,7 +218,7 @@ if [[ $CURRENT_COUNT -gt 0 ]]; then
     log "Previous count: $CURRENT_COUNT (${DELTA:+\+}$DELTA)"
     log "Backup location: $BACKUP_DIR"
 fi
-log "Logs: $COLLECTION_LOG, $ITEMS_LOG"
+log "Logs: $URLS_LOG, $COLLECTION_LOG, $ITEMS_LOG, $VALIDATION_LOG"
 log ""
 
 if [[ "$AUTO_PROMOTE" == false ]]; then

diff --git a/scripts/qa_update_catalogue.py → scripts/catalogue_qa.py b/scripts/qa_update_catalogue.py → scripts/catalogue_qa.py
@@ -10,14 +10,14 @@
 5. Logs results to logs/ directory
 
 Usage:
-    python scripts/qa_update_catalogue.py [--sample-percent 1] [--max-items 100]
+    python scripts/catalogue_qa.py [--sample-percent 1] [--max-items 100]
 
 Examples:
     # Check 1% sample (default)
-    python scripts/qa_update_catalogue.py
+    python scripts/catalogue_qa.py
 
     # Check 5% sample, max 200 items
-    python scripts/qa_update_catalogue.py --sample-percent 5 --max-items 200
+    python scripts/catalogue_qa.py --sample-percent 5 --max-items 200
 """
 
 import argparse