Description
One of the key ideas exploited in Julia's JuMP package is macros. These rewrite the Julia expression tree to optimize the creation of Julia data structures for JuMP constraints and expressions.
The macropy package provides support for similar syntactic macros in Python. Pyomo could leverage this capability to optimize the execution of expression construction.
Specifically, a decorator could be created for rule functions that replace them with functions that are reworked using syntactic macro operations to streamline the execution of expression setup. Consider the following rule:
def rule(model, i)
return 2*model.x[i] == model.y[i]
model.c = Constraint(range(1000000), rule=rule)
In every call to rule, Pyomo creates an expression tree. In so doing, Pyomo incrementally recognizes the monomial term 2*x[i], and it incrementally combines the LHS and RHS. This rule would be more efficiently written by directly creating the expression objects, rather than doing operator overloading.
Other developers have made a similar observation, and specifically they have noted that directly creating Pyomo's linear expression object leads to a significant speed-up in the construction of exprssions. We can automate this activity using syntactic macros.
NOTE:
- Delving into AST expressions is fraught with peril, so this is a high risk/high reward activity
- Support for the macropy package is not great, so this performance optimization idea is probably not something we'd want to commit to being a default behavior in Pyomo
- There are limitations to macropy (and related macro ideas in Python). They override Python's import logic, so these tricks probably couldn't be used by users developing concrete models unless they used rules (and even then there may be some limitations).