Major addon rework

[sstroemer]

Alrighty... this is a first draft of the addon rework that we discussed, to be able to talk a bit about the specifics that we may want to change, before everything is set in stone.

Breaking changes for the outside world

• I've bumped to Julia v1.11, anticipating that we may use that in the Python wrapper to load addons why the new "source" entry (which was weird... REPL is now a dependency)
• We are now able to find addons that are: given in a file, already loaded in IESopt, loaded in Main
• Per default, if possible, each addon is reloaded (=> replacing the module), because otherwise users won't see any changes in their addon code reflected.
• One can disable reloading of addons by passing force_reload = false to generate! (or `run!).
• The warning print regarding module replacement is now properly caught and won't show up in iterative solves, even if reloading is active.
• Component-based addons are gone
• Addons (previous global addons) have slightly changed
• "IESopt.component(...)" is now called "IESopt.get_component(...)"; code like component = component(model, str) was just... weird (and error prone)
• Tags are now properly re-introduced (just add "tags: foo" or "tags: [foo, bar]" to any component)
• Tagged components can be extracted by doing IESopt.get_components(model, tagged="foo"); passing ["foo", "bar"] only returns components that have both tags; types (like "Unit" or "CHP") are automatically set
• "Virtual components" that don't exist in the actual model but were "created" by a user (a component with a type that is some template) can now be interacted with; probably not fully featured, but they are first class CoreComponents now

Showcase

Assuming using IESopt was run.

Working with Virtuals

model = generate!(".dev/files/config.iesopt.yaml")
chp = get_components(model, tagged=["CustomCHP"])[1]

chp
# :: Virtual ::
# ├ name: chp
# └ type: CustomCHP

chp.power
# :: Unit ::
# ├ name: chp.power
# ├ inputs: Dict{Carrier, String}(Carrier("gas", nothing, nothing) => "grid_gas")
# ├ outputs: Dict{Carrier, String}(Carrier("electricity", nothing, nothing) => "grid_electricity", Carrier("co2", nothing, nothing) => "grid_co2")
# └ unit_commitment: off

Modifying a Virtual from within a template

Putting this into the finalize part of a template

finalize: |
  access("self").exp.in_gas = access("power").exp.in_gas + access("heat").exp.in_gas

allows doing

chp.exp.in_gas[1]
# 2.5 chp.power.conversion[1] + 0.5 chp.heat.conversion[1]

optimize!(model)
JuMP.value(chp.exp.in_gas[1])
# 9.375

Within an addon

All of that allows also stuff like

chp.con[:CHP_backpressure] = JuMP.@constraint(model, ...)

where chp is the virtual. All of that can after optimizing be accessed.

[sstroemer]

@daschw (just pinging for after-the-dust-settles, to then go over that together)

[daschw]

Awesome!
Sorry, I just saw this now. I will review this in detail soon, when I find some time.

[sstroemer]

Small update on the internal syntax, after most of it is cleaned up and (at least for the major parts) working:

Templates now properly support dict-valued parameters ...

... which allows configurations like this:

my_component:
  type: InvestableConversionTechnology
  # other stuff ...
  cost:
    invest: 1000.0
    om:
      var: 0.5
      fix:
        rel: 0.05
    lifetime: 25
    rate: 0.07

Getting parameters from within a template ...

... now works as follows (supporting getting/setting and also accessing time series data within the model):

p_min = this.get("p_min", 0)  # automatic support for defaults
p_max = this.get("p_max")

this.set("_cost_om_var_total", cost_om_var)

and since the "wrapper" for these templates (IESopt.Virtual <: IESopt._CoreComponent) works like any other component, dynamically constructing "referrals" between components can make use of the current name of the Virtual

this.set("_ub_single", "100 + $(this.name).invest:value")

Creating first-class math. model items ...

... like expressions, etc., follows the same syntax as everywhere else, allowing the following directly in finalize

this.exp.flow = this.connection_pos.var.flow - this.connection_neg.var.flow

where connection_pos is the directly nested sub-component (not requiring a manual lookup based on its name).

Automatic nested tagging ...

... of Virtuals works as a user might expect it: If Foo is a template that wraps just a single Unit, and Bar wraps Foo to set some (e.g.) sector-specific defaults, then the component

asset:
  type: Bar

will be (internally) flattened all the way down to a Unit. However, tags are created along the way, that allow tracking that asset is actually all of [Unit, Foo, Bar]. Further, if Bar and Foo contain a finalize function that (e.g.) creates convenience results (example: some specialized expressions a user might want to query), they will be called in reverse order: After creating the Unit, Foo.finalize will be called and after that finishes, Bar.finalize can further modify the Virtual.

Note: A warning is in place telling users that this is actually not a good idea, because it may be easy to make mistakes there without fully understanding the internals. So, usage of in-template functions like finalize is recommended to only be used in "actual" templates that contain multiple components.

[sstroemer]

Note: After 7105938 the stack overflow in tests (which only happened in CI) magically vanished... Merging this since it passes everything locally and that change was disabling suppression in test sets. Which I hope indicates some weird bug somewhere else (and not in the actual examples).