Adding the option to solve concurrently for all regions in MACRO

[daymontas1]

This PR introduces the option to solve for all regions concurrently in MACRO. A parameter (''solve_param'') has been added to ''MESSAGE-MACRO'' to control the solving mechanism for MACRO. Setting ''solve_param'' to 1 (the default value) triggers sequential solving across regions, whereas setting it to 2 enables concurrent solving.

An if statement has also been introduced in ''macro_solve'' that dictates whether sequential or concurrent solving is applied in MACRO, based on the value of the ''solve_param'' parameter defined in ''MESSAGE-MACRO_run''. The default solving is sequential (''solve_param''=1), so if the user doesn't update the solving mechanism, MACRO is solved as it used to be. In terms of efficiency, the concurrent solving is subtly more efficient.

For the concurrent solving mechanism (''solve_param''=2), all nodes are activated from the beginning, thereby enabling simultaneous solving for all regions. This setting allows for factoring in interregional interactions into the optimization process. We want this concurrent optimization option, as we intend to incorporate interregional feedback mechanisms into MACRO, e.g., interregional investments. Given that in the current version of MACRO regions are independent, both solving mechanisms should produce the same results.

How to review

Required: It must be verified that both versions produce the same results (considering there are no interregional interactions in the current version of MACRO).

If the changes are accepted, the model's documentation must be updated to include the concurrent option and instructions on how to trigger it. Additionally, it may be worthwhile to consider making the concurrent solving mechanism for MACRO the default, in case it reduces solving times.

PR checklist

• Continuous integration checks all ✅
• Add or expand tests; coverage checks both ✅
• Add, expand, or update documentation.
• Update release notes.

[CLAassistant]

All committers have signed the CLA.

[codecov]

Codecov Report

Attention: Patch coverage is 91.66667% with 1 line in your changes missing coverage. Please review.

Project coverage is 95.6%. Comparing base (fe2086f) to head (f561a58).
Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
message_ix/models.py	88.8%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main    #808     +/-   ##
=======================================
- Coverage   95.6%   95.6%   -0.1%     
=======================================
  Files         48      48             
  Lines       4394    4404     +10     
=======================================
+ Hits        4204    4213      +9     
- Misses       190     191      +1     

Files with missing lines	Coverage Δ
message_ix/tests/test_macro.py	100.0% <100.0%> (ø)
message_ix/models.py	98.8% <88.8%> (-0.3%)	⬇️

[glatterf42]

Thanks for this PR @daymontas1 :)

Currently, all test failures complain about a GAMS compilation error:

FAILED message_ix/tests/test_macro.py::test_calc_valid_years - ixmp.model.base.ModelError: GAMS errored with return code 2:
    There was a compilation error

For details, see the terminal output above, plus:
Listing   : /Users/runner/work/message_ix/message_ix/message_ix/model/MACRO_run.lst
Log file  : /Users/runner/work/message_ix/message_ix/message_ix/model/MACRO_run.log
Input data: /Users/runner/work/message_ix/message_ix/message_ix/model/data/MsgData_Westeros_Electrified_test_calc_valid_years_macro.gdx
FAILED message_ix/tests/test_macro.py::test_calc_data_missing_ref - ixmp.model.base.ModelError: GAMS errored with return code 2:
    There was a compilation error

For details, see the terminal output above, plus:
Listing   : /Users/runner/work/message_ix/message_ix/message_ix/model/MACRO_run.lst
Log file  : /Users/runner/work/message_ix/message_ix/message_ix/model/MACRO_run.log
Input data: /Users/runner/work/message_ix/message_ix/message_ix/model/data/MsgData_Westeros_Electrified_test_calc_data_missing_ref_macro.gdx
FAILED message_ix/tests/test_macro.py::test_calibrate - ixmp.model.base.ModelError: GAMS errored with return code 2:
    There was a compilation error

For details, see the terminal output above, plus:
Listing   : /Users/runner/work/message_ix/message_ix/message_ix/model/MACRO_run.lst
Log file  : /Users/runner/work/message_ix/message_ix/message_ix/model/MACRO_run.log
Input data: /Users/runner/work/message_ix/message_ix/message_ix/model/data/MsgData_Westeros_Electrified_test_macro_calibration.gdx
FAILED message_ix/tests/test_macro.py::test_calibrate_roundtrip - ixmp.model.base.ModelError: GAMS errored with return code 2:
    There was a compilation error

For details, see the terminal output above, plus:
Listing   : /Users/runner/work/message_ix/message_ix/message_ix/model/MACRO_run.lst
Log file  : /Users/runner/work/message_ix/message_ix/message_ix/model/MACRO_run.log
Input data: /Users/runner/work/message_ix/message_ix/message_ix/model/data/MsgData_Westeros_Electrified_test_calibrate_roundtrip_macro.gdx
FAILED message_ix/tests/test_macro.py::test_sector_map - ixmp.model.base.ModelError: GAMS errored with return code 2:
    There was a compilation error

For details, see the terminal output above, plus:
Listing   : /Users/runner/work/message_ix/message_ix/message_ix/model/MACRO_run.lst
Log file  : /Users/runner/work/message_ix/message_ix/message_ix/model/MACRO_run.log
Input data: /Users/runner/work/message_ix/message_ix/message_ix/model/data/MsgData_Westeros_Electrified_test_sector_map_macro.gdx

If you run these tests on your system, do they work? If not, could you please try troubleshooting what's blocking them from working?

[daymontas1]

@glatterf42 The error arises because the tests cannot read ''solve_param'' when it is defined in ''MESSAGE-MACRO_run.gms''. As a result, an error occurs in ''model_solve'' since the solving mechanism is dictated by ''solve_param''. However, when ''solve_param'' is defined within ''macro_data_load.gms'', the tests are successful. Therefore, ''solve_param'' should be defined in ''macro_data_load.gms'' instead of ''MESSAGE-MACRO_run.gms''.

[glatterf42]

Thanks for fixing the PR, looks good to me now! Please just rebase on top of main (or merge latest main into this branch) to enable merging this back to main :)

[khaeru]

I've updated the PR description to re-add the checklist that appears in the PR template.

If, for a certain PR, one feels that certain items in the checklist are not necessary, use strikeout and put a note to explain why. In this case, however, I do think documentation and an entry in the release notes are needed.

If the branch contains "Merge …" commits, we use squash-and-merge; if it is rebased and these disappear, we can use an ordinary merge.

[khaeru]

@glatterf42 and I discussed that I will take on the work to complete and merge this PR. Things required:

• Tests of running MACRO in the new way.
  • Confirm the old and new methods give the same results.
• Update documentation and release notes.
• Provide a way to control the added setting directly via the Python API. This may involve changing it to be a compile-time or environment variable instead of a parameter; if so it may serve as reference for Add commodity flow of capacity variables #451.

[khaeru]

FYI @glatterf42 — following #909 and #910, this now appears to be working as intended.