Transport improvements from 2025-W05

[khaeru]

#259 ← previous PR | following PR → #300

Carried from the previous PR:

• Use a smaller growth_activity_up value for (t=ICAm_ptrp, n=R12_RCPA). The technology constraint as of 2020 is effective, but this tech/region grows too rapidly to 2025 and beyond. (per discussion with @r-aneeque 2024-12-04.) Next PR.
• Adjust efficiencies of LDV PHEV technologies using duty cycle data. Next PR.
• Ensure growth_activity_up and act-non_ldv.csv values are set for F ROAD technologies with, at least, c=gas inputs. This is to avoid the behaviour (observed by @r-aneeque) that the model quickly switches into these in 2025–2030.
• Rework .transport.Config.constraint to simplify the setting of these constraints. Next PR.

In addition:

• Adjust transport code and tests for genno 1.28.x. This fixes failures of scheduled jobs of the "pytest" workflow on main.
• Drop support for ixmp/message_ix 3.6.0, in line with the upstream version policy.
• Establish SDMX-CSV format for transport outputs for EDITS MCE.
• Improve the dispatch of the "transport" workflow in message_data (see also iiasa/message_data#596). With this, the set of scenarios run can be adjusted by making (temporary or permanent) changes in transport.yaml in this repository. This reduces the need for adjustments on the message_data side.

How to review

• Read the additions to doc/whatsnew.
• Note the CI checks all pass.

PR checklist

• Continuous integration checks all ✅
• Add or expand tests; coverage checks both ✅
• Add, expand, or update documentation.
• Update doc/whatsnew.
• After review approval: rebase to drop the "TEMPORARY" commits.

[codecov]

Codecov Report

Attention: Patch coverage is 97.07865% with 13 lines in your changes missing coverage. Please review.

Project coverage is 77.3%. Comparing base (0475c81) to head (9a38ac2).
Report is 52 commits behind head on main.

Files with missing lines	Patch %	Lines
message_ix_models/model/transport/report.py	91.8%	7 Missing ⚠️
message_ix_models/report/operator.py	0.0%	2 Missing ⚠️
message_ix_models/util/genno.py	90.0%	2 Missing ⚠️
message_ix_models/model/transport/ikarus.py	75.0%	1 Missing ⚠️
message_ix_models/util/sdmx.py	98.6%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##            main    #289      +/-   ##
========================================
+ Coverage   65.9%   77.3%   +11.4%     
========================================
  Files        215     217       +2     
  Lines      16572   16804     +232     
========================================
+ Hits       10922   13006    +2084     
+ Misses      5650    3798    -1852     

Files with missing lines	Coverage Δ
message_ix_models/model/structure.py	100.0% <100.0%> (ø)
message_ix_models/model/transport/base.py	97.9% <100.0%> (+61.3%)	⬆️
message_ix_models/model/transport/build.py	94.9% <100.0%> (+73.9%)	⬆️
message_ix_models/model/transport/config.py	97.3% <100.0%> (+5.8%)	⬆️
message_ix_models/model/transport/demand.py	100.0% <100.0%> (+59.1%)	⬆️
message_ix_models/model/transport/files.py	98.6% <ø> (+10.4%)	⬆️
message_ix_models/model/transport/freight.py	100.0% <100.0%> (+100.0%)	⬆️
message_ix_models/model/transport/key.py	100.0% <100.0%> (ø)
message_ix_models/model/transport/ldv.py	96.7% <100.0%> (+75.8%)	⬆️
message_ix_models/model/transport/non_ldv.py	98.1% <100.0%> (+81.0%)	⬆️
... and 29 more

... and 25 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[glatterf42]

@khaeru I rebased the branch and dropped the TEMPORARY commits, but this broke the v3.6.0 tests. So I guess we will need to restore some of the changes marked as TEMPORARY.

EDIT: None of the 'TEMPORARY' commits contain anything related to v3.6.0, so that shouldn't make a difference. Could it be that we're now running the pytest workflow file from main again, which is still trying to run the v3.6.0 tests, even though this PR dropped support for them?

[khaeru]

Could it be that we're now running the pytest workflow file from main again, which is still trying to run the v3.6.0 tests, even though this PR dropped support for them?

Yes, exactly so.

• The PR contains commits that modify the pytest.yaml workflow file, e.g. "Drop support for ixmp/message-ix version 3.6"
• However, because the default setting is to run that workflow on the 'pull_request_target' event, the version of the workflow file used is the one on main, not the modified one on the branch.
• In order to 'prove' that all jobs will succeed once once the changed pytest.yaml is merged back to main, I add a TEMPORARY commit that turns off the 'pull_request_target' and turns on the 'pull_request' trigger. This results in the workflow file from the branch being used, and we saw that the jobs succeeded.
• In order to avoid disturbing 'pull_request_trigger' config on main, we need to drop that TEMPORARY commit before merging. When we do, the main version of the workflow will run again, and will fail.

This is an unfortunate complication that arises from our setup using 'pull_request_trigger' to allow use of secrets on PRs from forks. It is only necessary when modifying the workflow file itself. Maybe there could be some clever way to circumvent the need to do it this way, but I haven't had time to consider that yet. At minimum I guess I should write down somewhere that this process is the one we must use.

[khaeru]

—so, seeing that all jobs except for those upstream-v3.6.0 jobs pass, I'll merge. Thanks for the review and rebase!