Set up data with frontend submodule

[DinneK]

module-name: Set up data with frontend submodule

Problem

Currently metrics works as a fully functional frontend application. However, since we will now have a metrics-external and a metrics-internal the frontend for the applications need s to stay in sync.

Solution

Restructure this repository to serve as the external data backend, with the frontend included as a Git submodule. This enables separation of data generation (backend) from presentation (frontend), allowing multiple backends to share the same frontend.

Changes Made

• Removed Frontend Components app/:

  • Deleted app/src/ (CSS, JS source files)
  • Deleted app/site/_includes/ (Liquid includes)
  • Deleted app/site/_layouts/ (Liquid layouts)
  • Deleted app/site/*.liquid files (page templates)
  • Deleted app/.eleventy.js (Eleventy configuration)
  • Deleted app/package.json (frontend dependencies)
  • Deleted app/rollup.config.mjs (frontend build config)
  • Deleted app/postcss.config.js (CSS processing config)

• Add Submodule:

  • Added metrics_frontend as Git submodule at frontend/
  • Created .gitmodules file tracking the submodule
  • Submodule tracks branch: feature/submodule-restructure (will update to main after merge)

• Updated Backend Scripts:

  • Updated paths in scripts/refresh_metrics.py:
    • Changed output from app/site/_data/ → frontend/app/site/_data/
  • Updated paths in scripts/refresh_graphs.py:
    • Changed output from app/site/_graphs/ → frontend/app/site/_graphs/
  • Updated gen_graphs.sh if applicable:
    • Changed output directory references
  • Updated gen_reports.sh if applicable:
    • Changed output directory references

• Updated Configuration

  • Updated .gitignore for new structure
  • Removed old app/ directory from root

Result

Metrics now runs a data generation repository with a frontend submodule.

File structure after changes

metrics-data-external/ 
├── scripts/ 
│ ├── refresh_metrics.py 
│ ├── refresh_graphs.py 
│ └── tests/ 
├── templates/ 
│ ├── org_report_template.md 
│ └── repo_report_template.md 
├── frontend/  (Git submodule) 
│ └── app/ 
│ ├── site/ 
│ │ ├── _data/ (scripts write here) 
│ │ ├── _graphs/ (scripts write here) 
│ │ └── _posts/ (scripts write here) 
│ ├── src/ 
│ ├── .eleventy.js 
│ └── package.json 
├── .gitmodules 
├── .gitignore 
├── gen_reports.sh 
├── gen_graphs.sh 
└── README.md

Test Plan

Verified Data Generation

• ./gen_reports.sh runs successfully
• Data files appear in frontend/app/site/_data/
• ./gen_graphs.sh runs successfully
• Graph images appear in frontend/app/site/_graphs/

Verify Frontend Build

• Submodule clones correctly
• cd frontend/app && npm install completes
• npm start runs development server
• Frontend displays generated data correctly
• npm run build creates production build

Manual Testing Steps

# Fresh clone test
git clone --recursive -b Refactor-for-submodule \
  git@github.com:DSACMS/metrics.git test-clone
cd test-clone

# Verify submodule
`frontend/app`  # Should show frontend structure

# Generate data
./gen_reports.sh  
./gen_graphs.sh  

# Verify data created
frontend/app/site/_data/   # Should show JSON files
frontend/app/site/_graphs/  # Should show graph images

# Build frontend
cd frontend/app
npm install
npm start
# Verify site loads with generated data at localhost:8080

Breaking Changes

Repository structure has fundamentally changed:

• Frontend code moved from app/ to frontend/ (submodule)
• Must clone with --recursive flag or run git submodule update --init
• Existing local clones need to run: git submodule update --init --recursive

Migration for Existing Developers

# In your existing local clone
git pull origin Refactor-for-submodule
git submodule update --init --recursive
cd frontend/app
npm install

Benefits of This Structure

1. Separation of Concerns: Data generation and presentation are decoupled
2. Reusability: Same frontend can be used for internal and external backends
3. Independent Development: Frontend and backend can be updated independently
4. Easier Code Review: Reviewers can focus on backend OR frontend, not both

Related PRs

• metrics_frontend: PR Add env dir to .gitignore #2 (Restructure as frontend-only submodule)

Checklist

• Removed all frontend source code
• Added frontend as submodule
• Updated script paths to write to frontend/app/site/
• Updated .gitignore
• Tested data generation
• Tested frontend build with generated data
• Verify fresh clone works with --recursive

Reviewer Notes

This is a significant architectural change. The functionality remains the same, but the code organization is fundamentally different.

Key Points to Review:

1. All script paths correctly point to frontend/app/site/
2. Submodule is properly configured in .gitmodules
3. Both data generation and frontend build work end-to-end

Testing This PR:

git clone --recursive -b Refactor-for-submodule
cd metrics
python scripts/refresh_metrics.py  # Should create data in frontend/app/site/_data/
cd frontend/app && npm install && npm start  # Should show data in UI

Next Steps

• Create metrics-data-internal repository with same structure
• Updated README with submodule instructions
• Clear documentation for running and updates

[decause-gov]

So much great work being put in here! Cannot wait to see how the new approach pans out!

LGTM +1 🚢

The merge-base changed after approval.

[decause-gov]

LGTM +1 🚢