Add SCIP Persistent Support #3200

Opt-Mucca · 2024-03-18T14:31:18Z

Fixes # .

None.

Summary/Motivation:

Adds support for SCIP persistent solving. Persistent solving uses the python interface to SCIP directly as opposed to the current standard that involves read and writing a temporary model file.

The new dependency that it introduces is PySCIPOpt

Changes proposed in this PR:

SCIPPersistent
SCIPDirect
Some new tests for the introduced features

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

I agree my contributions are submitted under the BSD license.
I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

Opt-Mucca · 2024-03-18T14:37:24Z

Two open questions left for this pull request:

Is the current standard usage of persistent solving through the APPSI? If so then I should consider adding that.
I avoided tinkering with the GitHub action workflow scripts. I'm not sure if it is appropriate to add a PySCIPOpt dependency there.

For info of how I tested the code:

pip install pyscipopt
python pyomo/solvers/tests/checks/test_SCIP{Persistent/Direct}.py

mrmundt · 2024-03-18T14:48:32Z

@Opt-Mucca - answering your questions:

The current standard is APPSI, yes; however, the future standard is going to be using the persistent interface within pyomo.contrib.solver (the preview of the reworked solver interface system)
If PySCIPOpt is needed in order to run tests, then yes, absolutely, it's appropriate. Happy to help with that if you'd like.

Opt-Mucca · 2024-03-18T15:05:32Z

@mrmundt Thanks for the fast reply!

After having a glance at pyomo.contrib.solver, it seems to overlap pretty heavily with the direct interfaces in pyomo.solvers.plugins. So that should save on work. I will probably avoid writing the APPSI functionality if there's soon to be a newer interface, but can do if there's demand.
Help would be appreciated! I assumed the only changes needed were those that are present in the other PRs Added MAiNGO appsi-interface #3165 and [WIP] Add COPT support for Pyomo #2880 (replacing coptpy / maingopy with pyscipopt). If that's right then I can add those lines.

mrmundt · 2024-03-18T17:05:34Z

@Opt-Mucca - No problem!

Sounds good! There will be inevitable changes for the new interface (e.g., configuration options, results object, solution statuses, etc.), but we hope that they aren't too miserable for porting.
That's about it, yup! Make sure to consider both the pip and conda situations, though, because we test environments built by both types of package managers.

Opt-Mucca · 2024-03-19T10:52:59Z

I have now added:

Some automatic documentation for SCIPPersistent similar to other solver interfaces
The pip instructions to the .github/workflows. (I avoided conda because I'm not familiar with it. I also only added PySCIPOpt via pip when using CPython)

mrmundt

Thanks for submitting this PR! Some comments from a quick pass.

.github/workflows/test_branches.yml

.github/workflows/test_pr_and_main.yml

pyomo/solvers/plugins/solvers/scip_direct.py

mrmundt · 2024-03-19T13:06:27Z

pyomo/solvers/plugins/solvers/scip_direct.py

+            if not flag:
+                raise RuntimeError(
+                    "***The scip_direct solver plugin cannot extract solution suffix="
+                    + suffix
+                )


Design question: is it better to raise an error or to log (or print loudly) that other requests are being ignored?

(Note: I don't have a strong preference either way.)

I would lean slightly towards the print loudly option, but can see the potential danger for users. As a larger preference though, I would like to stick to similar behavior with the other interfaces.

pyomo/solvers/plugins/solvers/scip_persistent.py

mrmundt · 2024-03-19T13:15:54Z

pyomo/solvers/tests/checks/test_SCIPDirect.py

+try:
+    import pyscipopt
+
+    scip_available = True
+except ImportError:
+    scip_available = False


I theorize that @jsiirola may want this logic instead put in the import attempt code: https://github.com/Pyomo/pyomo/blob/main/pyomo/common/dependencies.py#L1037

I will ping @jsiirola again about this thought.

pyomo/solvers/tests/checks/test_SCIPDirect.py

mrmundt · 2024-03-19T13:17:36Z

pyomo/solvers/tests/checks/test_SCIPPersistent.py

+try:
+    import pyscipopt
+
+    scip_available = True
+except ImportError:
+    scip_available = False


I theorize that @jsiirola may want this logic instead put in the import attempt code: https://github.com/Pyomo/pyomo/blob/main/pyomo/common/dependencies.py#L1037

mrmundt · 2024-03-19T13:21:07Z

@Opt-Mucca - Please make sure you're on the most recent version of black and run (in the root): black . -S -C --exclude examples/pyomobook/python-ch/BadIndent.py

Co-authored-by: Miranda Mundt <[email protected]>

Opt-Mucca · 2024-03-19T14:56:45Z

@mrmundt Black should be run and all the issues responded to.

mrmundt · 2024-03-19T15:23:35Z

@Opt-Mucca - Spelling is the new failure. (You are suffering the same issue as all the core devs... The auto-linter and spell checker.)

Opt-Mucca · 2024-03-19T15:42:01Z

@mrmundt TIL there are auto-linters that double as spell checkers. (I'll fix the spelling errors that come up each time the test fails. If it's just the two errors then I'm properly amazed.)

mrmundt · 2024-03-19T15:56:18Z

@Opt-Mucca - If you could have heard the griping in our weekly dev call when I introduced the spell checker...

Opt-Mucca · 2024-03-19T16:35:00Z

@mrmundt I can imagine (would probably be guilty of griping even if makes sense)

For the currently failing runs: I've added a fix that should work, but I will ask some colleagues tomorrow. (I'm not the greatest the warm start code). The fixes raise some warning where a loaded solution has variables of incorrect values (this may stem from non specified variables having a default value of 0)

The current reason of the pipeline failure: SCIP was trying to load an invalid solution before checking for feasibility (I think this is a longstanding python error with the interface and one of the functions)

The added solution: I simply added a check on the feasibility of the solution before adding it to the solution storage.

I will check with colleagues, but I'm not sure on how SCIP handles partial solutions. I am not familiar or don't know if we have any functionality like Gurobi that tries to complete the solution. This also holds for trying to repair a given infeasible solution.

Opt-Mucca · 2024-03-20T09:01:44Z

Support for partial solution loading is now added. The failed tests are now passing locally too.

mrmundt · 2024-04-09T18:52:28Z

@Opt-Mucca - If you need help with your tests, please let us know. We tend to ignore PRs until the tests are passing.

mrmundt · 2024-06-13T14:05:03Z

@Opt-Mucca - Our general thought process is "protect users from themselves." I would recommend converting them but also log / alert the user so they know the conversion is happening.

Opt-Mucca · 2024-06-13T14:37:46Z

@mrmundt I've added a warning now for each time a constraint is added and the RHS / LHS needs to be changed.
I had to use type(x) == y because isinstance(x, float) also passes for np.float (didn't know that). The failed tests are now passing.

Opt-Mucca · 2024-06-14T08:12:29Z

@mrmundt I'm not sure if the failing tests are because of my changes. Can I get a second opinion on that?

Opt-Mucca · 2024-11-20T16:29:02Z

The latest change is pretty minor, but would mean that it's not backwards compatible to pyscipopt versions from years ago (I think compatibility would now be the last 2-3 releases). There was some basic missing functionality for retrieving the original constraints after starting / ending the solving process, and the Pyomo implementation was then giving some incorrect statistics.

pyomo/solvers/plugins/solvers/scip_direct.py

mrmundt · 2025-02-18T20:24:00Z

pyomo/solvers/tests/checks/test_SCIPDirect.py

+try:
+    import pyscipopt
+
+    scip_available = True
+except ImportError:
+    scip_available = False


I will ping @jsiirola again about this thought.

mrmundt

Sorry, I meant to click "Approve" - whoops. I left a few comments but none of them are so substantial that they should hold up this PR. They are mostly stylistic.

codecov · 2025-02-19T15:04:15Z

Codecov Report

Attention: Patch coverage is 76.01547% with 124 lines in your changes missing coverage. Please review.

Project coverage is 88.56%. Comparing base (bb5f993) to head (d0df2fc).
Report is 12 commits behind head on main.

Files with missing lines	Patch %	Lines
pyomo/solvers/plugins/solvers/scip_direct.py	75.93%	109 Missing ⚠️
pyomo/solvers/plugins/solvers/scip_persistent.py	76.56%	15 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3200      +/-   ##
==========================================
- Coverage   88.62%   88.56%   -0.07%     
==========================================
  Files         880      882       +2     
  Lines      100661   101178     +517     
==========================================
+ Hits        89214    89608     +394     
- Misses      11447    11570     +123

Flag	Coverage Δ
linux	`86.14% <76.16%> (-0.06%)`	⬇️
osx	`76.20% <75.19%> (-0.01%)`	⬇️
other	`86.66% <76.16%> (-0.06%)`	⬇️
win	`84.62% <75.58%> (-0.05%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mrmundt · 2025-02-19T15:40:03Z

FYI, the doc build/doc test failures are NOT RELATED. I have #3479 open to resolve it.

Opt-Mucca · 2025-02-19T15:45:09Z

I added a minor change: I no longer check what type the LHS of a constraint is. I added it originally because pyomo supports numpy floats and ints being passed, and I observed some crashes in SCIP's model creation. I looked into this a month or two ago, and concluded that it only happens from np.version() < 2 and for LHS values. So I've just removed the check for the RHS. This should save many users getting spammed with warnings. Associated issues if anyone wants to help: scipopt/PySCIPOpt#937

Add first draft of SCIP persistent solving

e552439

Add SCIPPersistent to docs

6a14f10

Add SCIp to Github action scripts

c107909

mrmundt requested changes Mar 19, 2024

View reviewed changes

Opt-Mucca and others added 9 commits March 19, 2024 14:55

Remove 5.0.0 specific version. Add conda to workflow

e00ded8

Standardise string formatting to fstring

a0b6250

Add parameter link to docstring

d0816eb

Remove redundant second objective sense check

0e11f11

Clean up _post_solve with a helper function for status handling

068ec99

Remove individual skip_test option

63af6d8

Update pyomo/solvers/plugins/solvers/scip_persistent.py

4075d9a

Co-authored-by: Miranda Mundt <[email protected]>

Update from the black command

f55fcc5

Merge remote-tracking branch 'origin/main'

e2b49b2

Fix typos

91eae7b

Replace trySol via more safe checkSol

5ee2007

Adds support for partial solution loading

f6ff092

Add error handling for setting non-linear objective

e7ac980

Opt-Mucca added 2 commits June 13, 2024 16:12

Add transformation for add_cons with non float/int rhs e.g. np.int

30e5e65

Add warning if type is converted. Tidy up logic

9104a92

blnicho and others added 6 commits August 6, 2024 15:29

Merge branch 'main' into main

a0324af

Merge branch 'main' into main

cbcb96d

Merge branch 'main' into main

fecd201

Merge branch 'main' into pr/3200

1a4663d

Merge branch 'main' into main

cadfe5c

Fix num. vars and cons from transformed. Silent warm start fail

f3f2d7c

mrmundt self-requested a review November 26, 2024 20:05

blnicho added 3 commits December 10, 2024 13:01

Merge branch 'main' into main

e4cdc6e

Merge branch 'main' into main

f326493

Merge branch 'main' into main

d0df2fc

mrmundt requested changes Feb 18, 2025

View reviewed changes

mrmundt approved these changes Feb 18, 2025

View reviewed changes

mrmundt added the AT: PRE-TEST INSPECTED label Feb 19, 2025

pyomo-autotest removed the AT: PRE-TEST INSPECTED label Feb 19, 2025

mrmundt mentioned this pull request Feb 19, 2025

Support for upcoming knitro python package #3478

Open

Add minor changes

7b18354

mrmundt added the AT: PRE-TEST INSPECTED label Feb 19, 2025

pyomo-autotest removed the AT: PRE-TEST INSPECTED label Feb 19, 2025

Merge branch 'main' into main

859a1b2

blnicho added the AT: PRE-TEST INSPECTED label Feb 19, 2025

pyomo-autotest removed the AT: PRE-TEST INSPECTED label Feb 19, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add SCIP Persistent Support #3200

Add SCIP Persistent Support #3200

Opt-Mucca commented Mar 18, 2024

Opt-Mucca commented Mar 18, 2024

mrmundt commented Mar 18, 2024

Opt-Mucca commented Mar 18, 2024

mrmundt commented Mar 18, 2024

Opt-Mucca commented Mar 19, 2024

mrmundt left a comment

mrmundt Mar 19, 2024

Opt-Mucca Mar 19, 2024

mrmundt Mar 19, 2024

mrmundt Feb 18, 2025

mrmundt Mar 19, 2024

mrmundt commented Mar 19, 2024

Opt-Mucca commented Mar 19, 2024

mrmundt commented Mar 19, 2024

Opt-Mucca commented Mar 19, 2024

mrmundt commented Mar 19, 2024

Opt-Mucca commented Mar 19, 2024

Opt-Mucca commented Mar 20, 2024

mrmundt commented Apr 9, 2024

mrmundt commented Jun 13, 2024

Opt-Mucca commented Jun 13, 2024 •

edited

Loading

Opt-Mucca commented Jun 14, 2024

Opt-Mucca commented Nov 20, 2024

mrmundt Feb 18, 2025

mrmundt left a comment

codecov bot commented Feb 19, 2025 •

edited

Loading

mrmundt commented Feb 19, 2025

Opt-Mucca commented Feb 19, 2025

Add SCIP Persistent Support #3200

Are you sure you want to change the base?

Add SCIP Persistent Support #3200

Conversation

Opt-Mucca commented Mar 18, 2024

Fixes # .

Summary/Motivation:

Changes proposed in this PR:

Legal Acknowledgement

Opt-Mucca commented Mar 18, 2024

mrmundt commented Mar 18, 2024

Opt-Mucca commented Mar 18, 2024

mrmundt commented Mar 18, 2024

Opt-Mucca commented Mar 19, 2024

mrmundt left a comment

Choose a reason for hiding this comment

mrmundt Mar 19, 2024

Choose a reason for hiding this comment

Opt-Mucca Mar 19, 2024

Choose a reason for hiding this comment

mrmundt Mar 19, 2024

Choose a reason for hiding this comment

mrmundt Feb 18, 2025

Choose a reason for hiding this comment

mrmundt Mar 19, 2024

Choose a reason for hiding this comment

mrmundt commented Mar 19, 2024

Opt-Mucca commented Mar 19, 2024

mrmundt commented Mar 19, 2024

Opt-Mucca commented Mar 19, 2024

mrmundt commented Mar 19, 2024

Opt-Mucca commented Mar 19, 2024

Opt-Mucca commented Mar 20, 2024

mrmundt commented Apr 9, 2024

mrmundt commented Jun 13, 2024

Opt-Mucca commented Jun 13, 2024 • edited Loading

Opt-Mucca commented Jun 14, 2024

Opt-Mucca commented Nov 20, 2024

mrmundt Feb 18, 2025

Choose a reason for hiding this comment

mrmundt left a comment

Choose a reason for hiding this comment

codecov bot commented Feb 19, 2025 • edited Loading

Codecov Report

mrmundt commented Feb 19, 2025

Opt-Mucca commented Feb 19, 2025

Opt-Mucca commented Jun 13, 2024 •

edited

Loading

codecov bot commented Feb 19, 2025 •

edited

Loading