Migrate .qmd to standalone scripts for production automation - #15
Merged
Conversation
Phase 1: Shared utilities - Create scripts/stac_utils.py with date extraction, validation, URL helpers, and path constants (eliminates 3x duplication across scripts) Phase 2: Item creation migration - Create scripts/item_create.py replacing stac_create_item.qmd Python chunks - argparse CLI (--test, --incremental, --reprocess-invalid, --workers) - Python logging module for proper log capture in cron Phase 3: Collection creation migration - Create scripts/collection_create.py replacing stac_create_collection.qmd Python chunks - Create scripts/urls_fetch.R replacing stac_create_collection.qmd R chunk - Clean R/Python boundary via data/urls_list.txt file interface Phase 4: Rename scripts to noun_verb convention and wire up - Rename: validate_stac_items.py -> item_validate.py - Rename: reprocess_invalid_items.py -> item_reprocess.py - Rename: extract_invalid_urls.py -> item_extract_invalid.py - Rename: qa_update_catalogue.py -> catalogue_qa.py - Update build_safe.sh to call Python/R scripts instead of quarto render - Add item_validate.py step to build pipeline - Update all cross-references across codebase Closes #14 Refs #7, #6 Relates to NewGraphEnvironment/sred#8, NewGraphEnvironment/sred#3 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrates
stac_create_collection.qmdandstac_create_item.qmdto standalone Python/R scripts for production automation (Phase 3 VM cron jobs). Closes #14, refs #7, #6.scripts/stac_utils.py— eliminates 3x duplicated date/URL functionsitem_create.py,collection_create.py,urls_fetch.Rreplacing .qmd filesnoun_verb.pyconvention (item_validate.py,item_reprocess.py, etc.)build_safe.shto callpython/Rscriptinstead ofquarto renderBefore / After
Test plan
collection_create.py --test— collection created and validateditem_create.py --test --test-count 10— 10 items created in 12sRelates to NewGraphEnvironment/sred-2025-2026#8, NewGraphEnvironment/sred-2025-2026#3
🤖 Generated with Claude Code