Heterogeneous graph transformers and heterogeneous gps - #509
Open
allaffa wants to merge 117 commits into
Open
Conversation
- Add per-term EMA normalization so each penalty stays ~1.0 regardless of raw physical magnitude (prevents angle-diff/flow terms dominating) - Add curriculum warmup+ramp scheduling: domain loss is zero for the first warmup_epochs so the model converges on task loss first, then linearly ramped over ramp_epochs to full weight - Fix voltage_output_index default 0->1 (Vm, not Va) - Add --domain_loss_warmup_epochs / --domain_loss_ramp_epochs CLI args - Update batch script: DOMAIN_WARMUP_EPOCHS=3, DOMAIN_RAMP_EPOCHS=3 - Job 4450208 submitted with curriculum schedule
Smoothness (L2 difference of adjacent bus values) is non-zero on the correct OPF solution — adjacent buses naturally have different voltages and angles. It was forcing the model toward a flat profile, causing sharp loss degradation when the curriculum ramped it in at epoch 3. Remaining 3 penalties are all feasibility constraints that are identically zero on any strictly feasible OPF solution: - voltage_bound : vmin <= Vm <= vmax - angle_diff : theta_min <= delta_Va <= theta_max per line - line_flow : |P_ij| = |delta_Va / x_ij| <= rate_a Job 4451003 resubmitted with corrected domain loss.
…comments
- Per-epoch LossBreakdown line in run.log (rank 0 only) showing:
data_driven_mse, physics_penalty_total, curriculum_scale,
and raw_* values for each feasibility constraint term.
Enables direct comparison of task loss vs domain loss contribution
to diagnose whether physics penalties interfere with data-driven learning.
Final epoch flushed explicitly after train_validate_test returns.
- line_flow_slack (default 1e-4): tolerance in DC thermal-limit penalty
absorbing the ~1.3e-5 residual from DC linearisation error on
AC-feasible OPF solutions (verified empirically on pglib_opf_case10000_goc).
- Strategic inline comments explaining purpose of all tolerances and
numerical guards (clamp floors, relu-squared form, EMA floor at 1e-8).
- check_domain_loss_on_gt.py: diagnostic script confirming all penalty
terms are zero on ground-truth solutions.
…ltage_bound_weight: 0.01 -> 0.001 - angle_diff_weight: 0.001 -> 0.0001 - line_flow_weight: 0.001 -> 0.0001 Total domain contribution ~0.001 vs task loss ~0.009 (~11%) rather than ~100% as before. - Switch LossBreakdown from logging.info to print() so it always appears in run.log regardless of HydraGNN logging level.
- Add binary_cross_entropy loss to loss_function_selection() - FT1 feasibility classification: configs, training script, job scripts, infeasible-sample generator (graph-level BCE, case118 → case14 agnostic) - FT3 N-1 contingency regression: configs, training script, job scripts - Shared ft_utils.py: EpochCSVWriter, evaluate_ft1/ft3, save_run_results - --no_pretrained flag on both train scripts for baseline comparison - collect_results.py: aggregate results.json + training_curve.csv files - plot_ft_results.py: bar charts, ROC curves, learning curves, scatter plots - submit_ft1_ft3_jobs.sh: Frontier Slurm orchestration with job dependencies - Pretrained HeteroSAGE_best and HeteroHEAT_best checkpoints
…rain_samples - All 30 FT configs (FT1-FT4, both archs, all regimes): use model-agnostic dataset names (e.g. FT3_contingency_data instead of FT3_contingency_HeteroSAGE_data) - train_opf_ft1_classify.py, train_opf_finetune.py: add --max_train_samples to truncate training split for data-efficiency sweeps - job-frontier-finetune.sh, job-frontier-FT1-*.sh: pass MAX_TRAIN_SAMPLES and NO_PRETRAINED through to training; DATA_MODELNAME now model-agnostic; LOG_NAME encodes arch/regime/scratch/n-samples (no SLURM_JOB_ID) - generate_infeasible_samples.py: replace full-load with reservoir sampling to avoid OOM when generating FT1 feasibility dataset - New scripts: preprocess_ft1_ft3_loginnode.sh, submit_ft1_ft3_debug.sh, submit_ft1_data_efficiency_sweep.sh, submit_ft3_data_efficiency_sweep.sh (72-job FT1 sweep and 80-job FT3 sweep: 4 methods x 2 archs x sample sizes)
…et generation - generate_infeasible_samples.py: move torch.distributed import to module level; fix dist.init_process_group call formatting; add gloo process group init before HDF5Writer.save() to satisfy iterate_tqdm's dist.get_rank() call - train_opf_ft1_classify.py, train_opf_finetune.py: fix line length and blank-line formatting around --max_train_samples Subset truncation block
- train_opf_ft1_classify.py / train_opf_finetune.py: call setup_log()
before save_config() so ./logs/{LOG_NAME}/ exists.
- job-frontier-FT1-single-method.sh / -FT3-: 1-node debug-QOS jobs;
unset AWS-OFI NCCL plugin and force NCCL_P2P_LEVEL=SYS for N<=1
(single-node intra-host xGMI/SHM only).
- FT1 LOG_NAME now keyed on requested N (not TOTAL=2N) to avoid
cross-N directory collisions.
- watch_and_chain_per_n.sh: serial submitter (1 in-flight per debug
QOS limit); skips cells whose results.json already exists; persists
log on Lustre.
- plot_sample_efficiency.py: aggregates results.json across runs into
per-(task, arch, metric) line plots and summary.csv.
- train_opf_finetune.py: FT1/FT3 training entry point with all regime options
- plot_finetune_paper_figures.py: generates all 7 paper figures (fig09-fig14)
- plot_ft1_bce_epoch_curves.py: BCE convergence curves for FT1
- job-frontier-FT{1,3}-single-method.sh: single-run Slurm job scripts
- job-frontier-FT{1,3}-bundled.sh: bundled multi-run job scripts
- submit_ft{1,3}_data_efficiency_sweep.sh: data-efficiency sweep launchers
- submit_ft1_ft3_jobs.sh, submit_ft_{bundled,single_method}.sh: campaign launchers
- submit_ft_debug_{chain,one}.sh: debug submission helpers
- watch_and_chain_{per_n,bundles,debug}.sh: job chaining watchers
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.