Skip to content

Reorganize package and deployment structure - #2

Open
allaffa wants to merge 8 commits into
mainfrom
restructure/package-boundaries
Open

Reorganize package and deployment structure#2
allaffa wants to merge 8 commits into
mainfrom
restructure/package-boundaries

Conversation

@allaffa

@allaffa allaffa commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

@allaffa allaffa self-assigned this Aug 17, 2026
Copilot AI lite review requested due to automatic review settings August 17, 2026 20:44

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reorganizes the matsim_agents Python package and repository layout to make architectural boundaries explicit (orchestration vs. backends vs. execution), while preserving backward compatibility via import-path aliases. It also moves machine-specific operational assets into deployments/ and publication / competition artifacts into research/.

Changes:

  • Introduces canonical packages (matsim_agents.orchestration, matsim_agents.backends.*, matsim_agents.execution) and updates imports across the codebase to use them, keeping legacy paths as compatibility aliases.
  • Relocates platform-specific scripts and docs from legacy scripts/... paths into deployments/<platform>/... and updates many references accordingly.
  • Adds new research artifacts (paper materials + Codabench competition bundle, baselines, and example submissions) and adds a regression test to enforce alias behavior.

Reviewed changes

Copilot reviewed 187 out of 244 changed files in this pull request and generated no comments.

Show a summary per file
File Description
tests/test_package_boundaries.py Regression tests for legacy import-path aliases.
tests/integration/test_uma_warmstart.py Updates docstring path to moved Perlmutter job script.
tests/integration/test_uma_vasp_warmstart.py Updates docstring path to moved Perlmutter job script.
tests/integration/test_qe_warmstart.py Updates docstring path to moved Frontier launcher.
src/matsim_agents/tools/warmstart_benchmark_vasp.py Switches imports to canonical backends (VASP + MLIP relaxation).
src/matsim_agents/tools/warmstart_benchmark_qe.py Switches imports to canonical backends (QE + MLIP relaxation).
src/matsim_agents/tools/init.py Re-exports relax_structure from canonical MLIP backend.
src/matsim_agents/state.py Legacy module shim to matsim_agents.orchestration.state.
src/matsim_agents/orchestration/state.py New canonical typed shared workflow state.
src/matsim_agents/orchestration/policies/init.py Introduces orchestration policy package boundary.
src/matsim_agents/orchestration/init.py Exposes orchestration state types at package level.
src/matsim_agents/execution/resources.py Adds scheduler-neutral resource request model.
src/matsim_agents/execution/provenance.py Adds JSONL provenance helper for workflow handoffs.
src/matsim_agents/execution/launchers.py Adds launcher protocol interface.
src/matsim_agents/execution/init.py Exposes execution provenance helper.
src/matsim_agents/discovery/wrapper.py Updates imports to canonical orchestration/backends locations.
src/matsim_agents/discovery/stability.py Updates imports to canonical orchestration state.
src/matsim_agents/cli.py Updates to canonical orchestration graph/state + supervisor entrypoint.
src/matsim_agents/chat.py Updates to canonical LLM provider + MLIP relaxation imports.
src/matsim_agents/backends/mlip/init.py Exposes MLIP relaxation API.
src/matsim_agents/backends/llm/init.py Exposes canonical LLM provider factory.
src/matsim_agents/backends/dft/vasp.py New canonical VASP DFT backend implementation.
src/matsim_agents/backends/dft/init.py Declares DFT backend package.
src/matsim_agents/backends/init.py Declares backends root package.
src/matsim_agents/agents/planner.py Imports state + LLM provider from canonical locations.
src/matsim_agents/agents/executor.py Imports state + MLIP relaxation from canonical locations.
src/matsim_agents/agents/analyst.py Imports state + LLM provider from canonical locations.
src/matsim_agents/active_learning/vasp_runner.py Switches VASP backend import to canonical location.
src/matsim_agents/active_learning/seeds.py Switches LLM provider import to canonical location.
src/matsim_agents/active_learning/dft_backend.py Updates backend factory to canonical QE/VASP backends.
src/matsim_agents/active_learning/config.py Updates docstrings/paths for moved launchers and LLM provider.
src/matsim_agents/active_learning/calculator.py Switches HydraGNN helpers import to canonical MLIP backend.
src/matsim_agents/active_learning/backends/vasp.py Legacy module shim to matsim_agents.backends.dft.vasp.
src/matsim_agents/init.py Exposes MatSimState from canonical orchestration package.
scripts/diagnostics/render_paper_tables.py Updates default paper results paths to research/.
scripts/diagnostics/collect_paper_results.py Updates default paper results paths to research/.
research/README.md Introduces research-only top-level directory documentation.
research/paper/manuscript/tables/warmstart_matrix.tex Adds paper table artifact under research/.
research/paper/manuscript/figures/plot_dft_scaling.py Updates script references/paths for moved assets.
research/paper/manuscript/figures/plot_al_timings.py Updates working-directory reference to research/.
research/paper/manuscript/figures/plot_al_finetune.py Updates working-directory reference to research/.
research/codabench/competition/starting_kit/example_submission/task5.csv Adds example submission artifact.
research/codabench/competition/starting_kit/example_submission/task2.zip Adds example submission artifact (forces zip).
research/codabench/competition/starting_kit/example_submission/task1.csv Adds example submission artifact.
research/codabench/competition/scoring_program/metadata Adds Codabench scoring manifest.
research/codabench/competition/requirements.txt Adds Codabench runtime dependency list.
research/codabench/competition/reference_data/public_ids.txt Adds public split IDs list.
research/codabench/competition/reference_data/private_ids.txt Adds private split IDs list.
research/codabench/competition/pages/terms.md Adds competition terms page.
research/codabench/competition/pages/overview.md Adds competition overview page.
research/codabench/competition/pages/evaluation.md Adds submission format + evaluation details.
research/codabench/competition/pages/data.md Adds data tab content.
research/codabench/competition/fix_h5py_system_conflict_aurora.sh Adds Aurora environment workaround script.
research/codabench/competition/baselines/mace_mp0/model.py Adds MACE baseline implementing competition interface.
research/codabench/competition/baselines/hydragnn/model.py Adds HydraGNN baseline implementing competition interface.
pyproject.toml Adds ruff per-file ignores for new/moved modules.
examples/paper_cases/singlepass.py Updates Perlmutter setup path to deployments/.
examples/paper_cases/README.md Updates Perlmutter job/setup/download script paths to deployments/.
examples/paper_cases/al_zn_formate.yaml Updates Perlmutter launcher paths to deployments/.
examples/paper_cases/al_zn_formate_uma.yaml Updates Perlmutter launcher paths to deployments/.
examples/paper_cases/al_phosphorene.yaml Updates Perlmutter launcher paths to deployments/.
examples/paper_cases/al_phosphorene_qe.yaml Updates Perlmutter launcher paths to deployments/.
examples/paper_cases/al_phosphorene_qe_frontier.yaml Updates Frontier launcher/job paths to deployments/.
examples/paper_cases/al_phosphorene_frontier.yaml Updates Frontier launcher/job paths to deployments/.
examples/paper_cases/al_lifepo4.yaml Updates Perlmutter launcher paths to deployments/.
examples/paper_cases/al_lifepo4_frontier.yaml Updates Frontier launcher paths to deployments/.
examples/paper_cases/al_hea_fcc_scaling.yaml Updates Perlmutter job/launcher paths to deployments/.
examples/paper_cases/al_hea_fcc_scaling_frontier.yaml Updates Frontier job/launcher paths to deployments/.
examples/paper_cases/al_hea_fcc_cantor.yaml Updates Perlmutter launcher paths to deployments/.
examples/paper_cases/al_hea_bcc.yaml Updates Perlmutter launcher paths to deployments/.
examples/paper_cases/al_hea_bcc_frontier.yaml Updates Frontier job/launcher paths to deployments/.
examples/paper_cases/al_cu_bht_2d.yaml Updates Perlmutter launcher paths to deployments/.
examples/active_learning/README.md Updates Frontier setup/launcher paths to deployments/.
examples/active_learning/al_config.prompt.example.yaml Updates Frontier launcher paths to deployments/.
examples/active_learning/al_config.example.yaml Updates Frontier launcher paths to deployments/.
examples/active_learning_uq.py Updates Aurora QE launcher path to deployments/.
docs/workflows/perlmutter-smoke-summary-2026-06-12.txt Updates Perlmutter smoke script path to deployments/.
docs/vllm-aurora.md Updates Aurora vLLM setup paths to deployments/.
docs/vasp-aurora.md Updates Aurora VASP build doc paths to deployments/.
docs/quantum-espresso-frontier.md Updates Frontier QE build/launcher paths to deployments/.
docs/quantum-espresso-aurora.md Updates Aurora QE build/launcher paths to deployments/.
docs/model-download.md Updates Perlmutter UMA download/job paths to deployments/.
docs/model-download-safety.md Updates download/setup paths to deployments/.
docs/architecture.md Adds explicit package architecture + compatibility policy doc.
deployments/README.md Introduces deployments/ policy and platform index.
deployments/perlmutter/smoke-tests/smoke-transformers-perlmutter.sh Updates Perlmutter script references to deployments/.
deployments/perlmutter/smoke-tests/smoke-transformers-multinode-perlmutter.sh Updates Perlmutter script references to deployments/.
deployments/perlmutter/setup/smoke_qe_vasp_perlmutter.sh Updates usage examples to deployments/ path.
deployments/perlmutter/setup/job_perlmutter.sh Adds example Perlmutter Slurm job script.
deployments/perlmutter/setup/build-vasp-gpu-perlmutter.sh Updates internal references to moved setup scripts.
deployments/perlmutter/setup/build-scalapack-perlmutter.sh Updates internal references to moved setup scripts.
deployments/perlmutter/setup/build-qe-gpu-perlmutter.sh Updates internal references to moved setup scripts.
deployments/perlmutter/setup/build-qe-cpu-perlmutter.sh Updates internal references to moved setup scripts.
deployments/perlmutter/setup/build-mace-venv-perlmutter.sh Updates usage header to deployments/ path.
deployments/perlmutter/launchers/submit-finetune-eval-matrix-perlmutter.sh Updates usage header paths to deployments/.
deployments/perlmutter/launchers/run-vasp-gpu-perlmutter.sh Updates module-stack path to deployments/.
deployments/perlmutter/launchers/run-qe-warmstart-benchmark-perlmutter.sh Updates launcher/module-stack paths to deployments/.
deployments/perlmutter/launchers/run-pw-gpu-perlmutter.sh Updates module-stack path to deployments/.
deployments/perlmutter/launchers/launch-test-singlenode-resume-perlmutter.sh Updates smoke-test path to deployments/.
deployments/perlmutter/launchers/launch-test-multinode-perlmutter.sh Updates smoke-test path to deployments/.
deployments/perlmutter/launchers/launch-test-all-models-perlmutter.sh Updates smoke-test path to deployments/.
deployments/perlmutter/launchers/job-finetune-eval-perlmutter.sh Updates module-stack path to deployments/.
deployments/perlmutter/launchers/job-finetune-eval-pack4-perlmutter.sh Updates launcher script path to deployments/.
deployments/perlmutter/launchers/_vasp-step-perlmutter.sh Updates module-stack path to deployments/.
deployments/perlmutter/launchers/_qe-step-perlmutter.sh Updates module-stack path to deployments/.
deployments/perlmutter/launchers/_mace-train-step-perlmutter.sh Updates module-stack path to deployments/.
deployments/perlmutter/jobs/submit-vasp-warmstart-check-matrix.sh Updates referenced job/launcher paths to deployments/.
deployments/perlmutter/jobs/submit-uma-warmstart-check-matrix.sh Updates referenced job/launcher paths to deployments/.
deployments/perlmutter/jobs/submit-uma-vasp-warmstart-check-matrix.sh Updates referenced job/launcher paths to deployments/.
deployments/perlmutter/jobs/submit-qe-warmstart-check-matrix.sh Updates referenced job/launcher paths to deployments/.
deployments/perlmutter/jobs/submit-al-dft-scaling-sweep.sh Updates paper figure paths to research/.
deployments/perlmutter/jobs/submit-active-learning-paper-cases.sh Updates usage header paths to deployments/.
deployments/perlmutter/jobs/job-vllm-smoke-perlmutter.sh Updates usage header paths to deployments/.
deployments/perlmutter/jobs/job-vasp-warmstart-perlmutter.sh Updates module-stack/launcher paths to deployments/.
deployments/perlmutter/jobs/job-uma-warmstart-perlmutter.sh Updates module-stack/launcher/download references to deployments/.
deployments/perlmutter/jobs/job-uma-vasp-warmstart-perlmutter.sh Updates module-stack/launcher/download references to deployments/.
deployments/perlmutter/jobs/job-singlepass-paper-cases-perlmutter.sh Updates setup script path to deployments/.
deployments/perlmutter/jobs/job-single-relaxation-perlmutter.sh Updates module-stack path to deployments/.
deployments/perlmutter/jobs/job-qe-warmstart-perlmutter.sh Updates module-stack/launcher paths to deployments/.
deployments/perlmutter/jobs/job-hypothesis-debate-perlmutter.sh Updates module-stack + driver path to deployments/.
deployments/perlmutter/jobs/job-discovery-vllm-perlmutter.sh Updates module-stack path to deployments/.
deployments/perlmutter/jobs/job-discovery-chat-perlmutter.sh Updates module-stack path to deployments/.
deployments/perlmutter/jobs/job-al-qe-portability-perlmutter.sh Updates module-stack path to deployments/.
deployments/perlmutter/jobs/job-al-dft-scaling-perlmutter.sh Updates module-stack path to deployments/.
deployments/perlmutter/jobs/job-active-learning-uq-perlmutter.sh Updates module-stack/launcher paths to deployments/.
deployments/perlmutter/jobs/job-active-learning-paper-cases-perlmutter.sh Updates module-stack/download/trainer paths to deployments/.
deployments/perlmutter/download/download-uma-perlmutter.sh Updates module-stack/install references to deployments/.
deployments/perlmutter/download/download-open-models-perlmutter.sh Updates module-stack/install references to deployments/.
deployments/perlmutter/download/download-models-perlmutter.sh Updates module-stack/install references to deployments/.
deployments/frontier/utils/_rocr_to_hip.sh Adds ROCR→HIP environment bridge utility.
deployments/frontier/smoke-tests/smoke-vllm-singlenode-frontier.sh Updates module-stack/util paths to deployments/.
deployments/frontier/smoke-tests/smoke-vllm-multinode-frontier.sh Updates module-stack/util paths to deployments/.
deployments/frontier/smoke-tests/smoke-transformers-frontier.sh Updates module-stack path to deployments/.
deployments/frontier/setup/setup_matsim_frontier.sh Adds Frontier quick setup script.
deployments/frontier/setup/README.md Updates Frontier setup README paths to deployments/.
deployments/frontier/setup/prebuild-tvm-ffi-frontier.sh Updates module-stack path to deployments/.
deployments/frontier/setup/makefile.include.frontier-gpu Updates comment reference to new build script path.
deployments/frontier/setup/install-rocm72.sh Updates protected requirements/build script paths to deployments/.
deployments/frontier/setup/install_matsim_frontier.sh Updates vLLM build-script path reference.
deployments/frontier/setup/frontier-module-stack.sh Adds shared Frontier module-stack helper.
deployments/frontier/setup/build-vllm-rocm72.sh Updates module-stack path to deployments/.
deployments/frontier/setup/build-vasp-gpu-frontier.sh Updates usage header to deployments/.
deployments/frontier/setup/build-qe-gpu-frontier.sh Updates usage header to deployments/.
deployments/frontier/setup/build-mace-venv-frontier.sh Updates usage header to deployments/.
deployments/frontier/launchers/run-qe-warmstart-benchmark.sh Updates QE launcher path to deployments/.
deployments/frontier/launchers/run-pw-gpu-frontier.sh Updates header references to deployments/.
deployments/frontier/launchers/run-active-learning-frontier.sh Updates usage header to deployments/.
deployments/frontier/launchers/launch-test-singlenode-resume-frontier.sh Adds resumed single-node vLLM test launcher.
deployments/frontier/launchers/launch-test-all-models-frontier.sh Updates usage header to deployments/.
deployments/frontier/launchers/_qe-step-frontier.sh Updates build-script reference to deployments/.
deployments/frontier/launchers/_hydragnn-train-step-frontier.sh Adds Frontier HydraGNN inner-step trainer launcher.
deployments/frontier/jobs/submit-al-dft-scaling-sweep.sh Updates wrapper references to deployments/ and research/.
deployments/frontier/jobs/job-six-model-benchmark-frontier.sh Updates module-stack reference to deployments/.
deployments/frontier/jobs/job-singlepass-paper-cases-frontier.sh Updates setup path to deployments/.
deployments/frontier/jobs/job-single-relaxation-frontier.sh Updates module-stack reference to deployments/.
deployments/frontier/jobs/job-serve-multinode-frontier.sh Updates tvm-ffi rebuild reference to deployments/.
deployments/frontier/jobs/job-sequential-benchmark-frontier.sh Updates tvm-ffi rebuild reference to deployments/.
deployments/frontier/jobs/job-qe-warmstart-frontier.sh Updates module-stack/launcher references to deployments/.
deployments/frontier/jobs/job-discovery-chat-vllm-frontier.sh Updates module-stack reference to deployments/.
deployments/frontier/jobs/job-discovery-chat-frontier.sh Updates module-stack reference to deployments/.
deployments/frontier/jobs/job-al-dft-scaling-frontier.sh Updates setup path to deployments/.
deployments/frontier/jobs/job-active-learning-uq-frontier.sh Updates module-stack/launcher references to deployments/.
deployments/frontier/jobs/job-active-learning-paper-cases-frontier.sh Updates setup path to deployments/.
deployments/frontier/download/download-open-models-frontier.sh Updates module-stack/install references to deployments/.
deployments/frontier/download/download-models-frontier.sh Updates module-stack/install references to deployments/.
deployments/frontier/docs/README-six-model-benchmark.md Adds Frontier six-model benchmark notes (needs path fixes).
deployments/frontier/docs/README-frontier.md Updates Frontier docs to deployments/ structure.
deployments/frontier/config/all_model_specs.json Adds consolidated local/hosted model spec list.
deployments/aurora/smoke-tests/smoke-vllm-singlenode-aurora.sh Updates entrypoint path to deployments/.
deployments/aurora/smoke-tests/aurora_vllm_entrypoint.py Adds Aurora vLLM subprocess patch entrypoint.
deployments/aurora/setup/setup_matsim_aurora.sh Adds Aurora quick setup script.
deployments/aurora/setup/install-vllm-xpu-aurora.sh Updates printed next-step paths to deployments/.
deployments/aurora/setup/hf_download.py Updates usage path to deployments/.
deployments/aurora/setup/build-vasp-gpu-aurora.sh Updates usage header to deployments/.
deployments/aurora/setup/build-qe-gpu-aurora.sh Updates usage header to deployments/.
deployments/aurora/setup/build-qe-cpu-aurora.sh Updates usage header to deployments/.
deployments/aurora/launchers/run-pw-gpu-aurora.sh Updates usage header to deployments/.
deployments/aurora/jobs/job-step2-perturbation-aurora.sh Updates usage header to deployments/.
deployments/aurora/jobs/job-single-relaxation-aurora.sh Updates usage header to deployments/.
deployments/aurora/jobs/job-serve-multinode-vllm-aurora.sh Updates usage header to deployments/.
deployments/aurora/jobs/job-serve-multinode-aurora.sh Updates usage header to deployments/.
deployments/aurora/jobs/job-qe-warmstart-aurora.sh Updates QE launcher path to deployments/.
deployments/aurora/jobs/job-discovery-chat-vllm-aurora.sh Updates entrypoint path to deployments/.
deployments/aurora/jobs/job-discovery-chat-rhea-aurora.sh Updates entrypoint path to deployments/.
deployments/aurora/jobs/job-discovery-chat-perovskites-aurora.sh Updates entrypoint path to deployments/.
deployments/aurora/jobs/job-discovery-chat-aurora.sh Updates usage header to deployments/.
deployments/aurora/jobs/job-active-learning-uq-aurora.sh Updates QE launcher path to deployments/.
deployments/aurora/download/download-open-models-aurora.sh Updates install references to deployments/.
deployments/aurora/download/download-models-aurora.sh Updates install references to deployments/.
benchmarks/README.md Introduces benchmarks directory guidance.
Suppressed comments (2)

deployments/frontier/smoke-tests/smoke-vllm-singlenode-frontier.sh:41

  • UTILS_DIR is computed as $SCRIPT_DIR/../../utils/frontier, which does not match this repository layout (deployments/frontier/utils). This makes the first check always fail and adds avoidable indirection. Point directly at the correct directory derived from REPO.
    deployments/frontier/smoke-tests/smoke-vllm-multinode-frontier.sh:41
  • Same as the single-node smoke test: $SCRIPT_DIR/../../utils/frontier does not exist in this repo layout, so the fallback always triggers. Use the canonical deployments/frontier/utils path directly.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@allaffa
allaffa force-pushed the restructure/package-boundaries branch from 57f4813 to 14a1c20 Compare August 18, 2026 21:18
- backends/dft/__init__.py: re-export DFTBackend, DFTJobSpec, DFTResult
  (Protocol already existed in active_learning/dft_backend.py)
- backends/mlip/__init__.py: new @runtime_checkable MLIPBackend Protocol
  with name, as_calculator() -> Calculator, relax() -> RelaxationResult
- backends/llm/__init__.py: LLMBackend = BaseChatModel (TYPE_CHECKING alias)
- execution/__init__.py: new @runtime_checkable ExecutionPlatform Protocol
  with name, submit() -> str, available_resources() -> ResourceRequest
- execution/provenance.py: new @runtime_checkable RunStore Protocol +
  concrete JsonlRunStore implementation
- tests/test_package_boundaries.py: 11 tests covering all five interfaces,
  Protocol compliance stubs, and JsonlRunStore round-trip
- README.md project layout: replace flat src/ tree with new
  orchestration/, backends/{llm,mlip,dft}/, and execution/ packages;
  mark state.py / graph.py / llm.py / tools/relaxation.py as compat
  aliases; add vasp_runner.py / orchestration/policies/ / execution/
  launchers & provenance
- README.md programmatic API: update import paths to canonical modules
    tools.relaxation      → backends.mlip.relaxation
    matsim_agents.graph   → orchestration.objective_graph
    matsim_agents.state   → orchestration.state
  Update standalone DFT relaxer table to link backends/dft/ modules
- README.md configuration reference: link to backends/mlip/relaxation.py
- docs/architecture.md: add 'Five stable interfaces' table documenting
  DFTBackend, MLIPBackend, LLMBackend, ExecutionPlatform, RunStore
  matsim_agents.llm             → backends.llm
  matsim_agents.tools.vasp_relax → backends.dft.vasp_relax (×2)
  tools.vasp_relax → backends.dft.vasp_relax
  tools.qe_relax   → backends.dft.qe_relax
@allaffa allaffa added documentation Improvements or additions to documentation enhancement New feature or request labels Aug 18, 2026
The competition bundle is a reusable benchmark suite with documented
inputs (159 structures, 5 tasks) and outputs — a better fit for
benchmarks/ than research/. Update all README.md references.
  cfg.hydragnn.logdir → cfg.mlip.hydragnn.logdir
  data['mlp'] → data['mlip'] (×4 tests)
  regex 'mlp.uma' → 'mlip.uma', 'mlp.hydragnn' → 'mlip.hydragnn'

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 189 out of 250 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants