Simplify PgSTAC Bootstrap Process

[emmanuelmathot]

Description

This PR streamlines the PgSTAC bootstrap process by leveraging standard tools, simplifying the configuration, and creating a more maintainable structure. It removes unnecessary complexity by utilizing the built-in pypgstac migrate command directly rather than through custom Python scripts.

Key Changes

1. Reorganized File Structure

• Created dedicated directories for better organization:
  • initdb-data/settings/: Contains PgSTAC configuration settings
  • initdb-data/samples/: Contains sample STAC data files that are loaded conditionally

2. Simplified Bootstrap Process

• Replaced custom Python script (pgstac-setup.py) with direct use of the pypgstac migrate command
• Separated the core migration process from sample data loading for better modularity:
  • Core migration job (pgstac-migrate): Always runs to set up schema and settings
  • Sample data job (pgstac-load-samples): Only runs when enabled
• Used standard PostgreSQL environment variables instead of custom DSN approach

3. Improved Configuration Options

• Added clearer configuration settings:
  • New pgstacBootstrap.settings.loadSamples option to control sample data loading
  • Maintained backward compatibility with existing environment variables
• Removed unnecessary KEEP_ALIVE functionality
• Removed unnecessary command and args settings from values.yaml

4. Updated Templates

• Simplified configmap.yaml to only include necessary resources
• Made sample data ConfigMaps conditional based on loadSamples setting
• Ensured file references point to the appropriate directory structure

5. Enhanced Documentation

• Added README.md explaining the new approach, configuration options, and directory structure
• Clearly marked deprecated options to guide users toward the recommended settings

Benefits

• Simplified Maintenance: Fewer custom scripts to maintain
• Better Separation of Concerns: Clear distinction between core setup and sample data
• Improved Configuration: More explicit settings with better defaults
• Standard Tooling: Direct use of standard tools rather than custom wrappers
• Maintained Compatibility: Works with existing configurations while encouraging best practices

Testing

The changes have been tested to ensure:

1. The PgSTAC database schema is properly migrated
2. Settings are correctly applied
3. Sample data loading works when enabled
4. The process remains idempotent (can be run multiple times safely)

Implementation Notes

All changes have been made with backward compatibility in mind. Users can continue to use existing configurations, but are encouraged to adopt the new structure for clarity and maintainability.

[Copilot]

Copilot reviewed 10 out of 11 changed files in this pull request and generated no comments.

Files not reviewed (1)

• helm-chart/eoapi/initdb-data/settings/pgstac-settings.sql: Language not supported

Comments suppressed due to low confidence (2)

helm-chart/eoapi/templates/pgstacboostrap/job.yaml:25

• [nitpick] The file path and metadata refer to 'pgstacboostrap', which appears to be a misspelling. Consider renaming it to 'pgstacbootstrap' for consistency.

name: pgstacbootstrap

helm-chart/eoapi/templates/pgstacboostrap/configmap.yaml:9

• [nitpick] There is an inconsistency in the naming of configmaps (e.g., 'pgstac-settings-config', 'initdb-sql-config', and 'initdb-json-config') that could be confusing. Clarify and standardize these names to better reflect their purpose in the bootstrap process.

name: pgstac-settings-config-{{ $.Release.Name }}

[batpad]

@emmanuelmathot is there a reference for what this is doing exactly / what settings these are, etc? If these are settings, do we need to make them configurable in values? Am guessing this is something just a bit silly to get things to work, but might be nice to have a comment with a link to what these settings are or so?

[emmanuelmathot]

Yes, the idea is to have them in the values. I actually want to have a proper mechanism via pypgstac cli with something like pypgstac load settings that may be used not only in this chart. PR to come in pgstac.

[batpad]

@emmanuelmathot left a small question around the SQL to update the settings, but otherwise this looks good to me.