Conversation
…SI._update_parameter_values! override to fix merchant sims
There was a problem hiding this comment.
Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: kdayday <kdayday@users.noreply.github.com>
…ions.jl into kd/psy5
Agent-Logs-Url: https://github.com/Sienna-Platform/HybridSystemsSimulations.jl/sessions/ed090d9d-2ba8-422d-be3d-23f23463db3f Co-authored-by: kdayday <12451220+kdayday@users.noreply.github.com>
| 1. **Device formulations** — model a `HybridSystem` as one device inside a standard PSI `ProblemTemplate` (e.g., a UC or ED problem), dispatching the hybrid's internal assets subject to PCC limits. | ||
| 2. **Merchant decision models** — custom `PSI.DecisionModel`s (with their own `build_impl!`) that optimize a price-taker hybrid's energy and ancillary-service bids against day-ahead (DA) and real-time (RT) market prices, including a bilevel/KKT formulation. |
There was a problem hiding this comment.
[JuliaFormatter] reported by reviewdog 🐶
| 1. **Device formulations** — model a `HybridSystem` as one device inside a standard PSI `ProblemTemplate` (e.g., a UC or ED problem), dispatching the hybrid's internal assets subject to PCC limits. | |
| 2. **Merchant decision models** — custom `PSI.DecisionModel`s (with their own `build_impl!`) that optimize a price-taker hybrid's energy and ancillary-service bids against day-ahead (DA) and real-time (RT) market prices, including a bilevel/KKT formulation. | |
| 1. **Device formulations** — model a `HybridSystem` as one device inside a standard PSI `ProblemTemplate` (e.g., a UC or ED problem), dispatching the hybrid's internal assets subject to PCC limits. | |
| 2. **Merchant decision models** — custom `PSI.DecisionModel`s (with their own `build_impl!`) that optimize a price-taker hybrid's energy and ancillary-service bids against day-ahead (DA) and real-time (RT) market prices, including a bilevel/KKT formulation. |
| | Formulation | Description | | ||
| |---|---| | ||
| | `HybridEnergyOnlyDispatch` | Energy-only dispatch of the hybrid's internal assets behind the PCC | |
There was a problem hiding this comment.
[JuliaFormatter] reported by reviewdog 🐶
| | Formulation | Description | | |
| |---|---| | |
| | `HybridEnergyOnlyDispatch` | Energy-only dispatch of the hybrid's internal assets behind the PCC | | |
| | Formulation | Description | | |
| |:---------------------------- |:--------------------------------------------------------------------------------------------------- | | |
| | `HybridEnergyOnlyDispatch` | Energy-only dispatch of the hybrid's internal assets behind the PCC | |
| |---|---| | ||
| | `HybridEnergyOnlyDispatch` | Energy-only dispatch of the hybrid's internal assets behind the PCC | | ||
| | `HybridDispatchWithReserves` | Dispatch with ancillary-service participation, reserve assignment, and reserve coverage constraints | | ||
| | `HybridFixedDA` | Hybrid with fixed day-ahead positions (used downstream of a merchant stage) | |
There was a problem hiding this comment.
[JuliaFormatter] reported by reviewdog 🐶
| | `HybridFixedDA` | Hybrid with fixed day-ahead positions (used downstream of a merchant stage) | | |
| | `HybridFixedDA` | Hybrid with fixed day-ahead positions (used downstream of a merchant stage) | |
| | Decision model | Description | | ||
| |---|---| | ||
| | `MerchantHybridEnergyCase` | DA + RT energy-only bid co-optimization | | ||
| | `MerchantHybridEnergyFixedDA` | RT subproblem with locked DA bids | | ||
| | `MerchantHybridCooptimizerCase` | DA + RT energy and ancillary-service bid co-optimization | | ||
| | `MerchantHybridBilevelCase` | Bilevel formulation (KKT conditions + complementary slackness via SOS1, strong duality) | |
There was a problem hiding this comment.
[JuliaFormatter] reported by reviewdog 🐶
| | Decision model | Description | | |
| |---|---| | |
| | `MerchantHybridEnergyCase` | DA + RT energy-only bid co-optimization | | |
| | `MerchantHybridEnergyFixedDA` | RT subproblem with locked DA bids | | |
| | `MerchantHybridCooptimizerCase` | DA + RT energy and ancillary-service bid co-optimization | | |
| | `MerchantHybridBilevelCase` | Bilevel formulation (KKT conditions + complementary slackness via SOS1, strong duality) | | |
| | Decision model | Description | | |
| |:------------------------------- |:--------------------------------------------------------------------------------------- | | |
| | `MerchantHybridEnergyCase` | DA + RT energy-only bid co-optimization | | |
| | `MerchantHybridEnergyFixedDA` | RT subproblem with locked DA bids | | |
| | `MerchantHybridCooptimizerCase` | DA + RT energy and ancillary-service bid co-optimization | | |
| | `MerchantHybridBilevelCase` | Bilevel formulation (KKT conditions + complementary slackness via SOS1, strong duality) | |
| - **DA bids are hourly slots; RT variables follow the model resolution.** The DA axis is `merchant_da_time_step_range(container, hybrid)` = `1:min(horizon_hours, DA-series length)`; the variable, parameter, constraint, and objective axes must all use it. RT-to-DA index mapping goes through `merchant_rt_to_da_tmap(rt_len, da_len)` — never hand-roll `div`-based maps. | ||
| - **Storage energy quantities are in energy units**, computed as `get_storage_level_limits(storage).{min,max} * get_storage_capacity(storage)`. The same convention applies to initial conditions (`get_initial_storage_capacity_level * capacity`), cycling limits, and `storage_target`. Do not use the bare level fractions. | ||
| - **Market prices are hybrid-attached scalar `SingleTimeSeries`**, keyed by name (defaults `"DA"`/`"RT"`, override via `model.ext["day_ahead_time_series_key"]` / `"real_time_time_series_key"`): | ||
| - Energy: `hybrid_energy_price_time_series_name(key)` → `"HybridSystem__energy_price__<key>"` | ||
| - Ancillary: `hybrid_ancillary_service_price_time_series_name(service, key)` | ||
| - Profiles: `"RenewableDispatch__max_active_power"`, `"PowerLoad__max_active_power"` | ||
| - `Δt_RT` must come from the container/settings resolution (`PSI.get_resolution`), not from `first(PSY.get_time_series_resolutions(sys))` — systems may carry multiple resolutions and `PSI.validate_time_series!` negotiates the selected one into settings. |
There was a problem hiding this comment.
[JuliaFormatter] reported by reviewdog 🐶
| - **DA bids are hourly slots; RT variables follow the model resolution.** The DA axis is `merchant_da_time_step_range(container, hybrid)` = `1:min(horizon_hours, DA-series length)`; the variable, parameter, constraint, and objective axes must all use it. RT-to-DA index mapping goes through `merchant_rt_to_da_tmap(rt_len, da_len)` — never hand-roll `div`-based maps. | |
| - **Storage energy quantities are in energy units**, computed as `get_storage_level_limits(storage).{min,max} * get_storage_capacity(storage)`. The same convention applies to initial conditions (`get_initial_storage_capacity_level * capacity`), cycling limits, and `storage_target`. Do not use the bare level fractions. | |
| - **Market prices are hybrid-attached scalar `SingleTimeSeries`**, keyed by name (defaults `"DA"`/`"RT"`, override via `model.ext["day_ahead_time_series_key"]` / `"real_time_time_series_key"`): | |
| - Energy: `hybrid_energy_price_time_series_name(key)` → `"HybridSystem__energy_price__<key>"` | |
| - Ancillary: `hybrid_ancillary_service_price_time_series_name(service, key)` | |
| - Profiles: `"RenewableDispatch__max_active_power"`, `"PowerLoad__max_active_power"` | |
| - `Δt_RT` must come from the container/settings resolution (`PSI.get_resolution`), not from `first(PSY.get_time_series_resolutions(sys))` — systems may carry multiple resolutions and `PSI.validate_time_series!` negotiates the selected one into settings. | |
| - **DA bids are hourly slots; RT variables follow the model resolution.** The DA axis is `merchant_da_time_step_range(container, hybrid)` = `1:min(horizon_hours, DA-series length)`; the variable, parameter, constraint, and objective axes must all use it. RT-to-DA index mapping goes through `merchant_rt_to_da_tmap(rt_len, da_len)` — never hand-roll `div`-based maps. | |
| - **Storage energy quantities are in energy units**, computed as `get_storage_level_limits(storage).{min,max} * get_storage_capacity(storage)`. The same convention applies to initial conditions (`get_initial_storage_capacity_level * capacity`), cycling limits, and `storage_target`. Do not use the bare level fractions. | |
| - **Market prices are hybrid-attached scalar `SingleTimeSeries`**, keyed by name (defaults `"DA"`/`"RT"`, override via `model.ext["day_ahead_time_series_key"]` / `"real_time_time_series_key"`): | |
| + Energy: `hybrid_energy_price_time_series_name(key)` → `"HybridSystem__energy_price__<key>"` | |
| + Ancillary: `hybrid_ancillary_service_price_time_series_name(service, key)` | |
| + Profiles: `"RenewableDispatch__max_active_power"`, `"PowerLoad__max_active_power"` | |
| - `Δt_RT` must come from the container/settings resolution (`PSI.get_resolution`), not from `first(PSY.get_time_series_resolutions(sys))` — systems may carry multiple resolutions and `PSI.validate_time_series!` negotiates the selected one into settings. |
| - **Custom variables**: market bids (`EnergyDABidOut/In`, `EnergyRTBidOut/In`, `BidReserveVariableOut/In`), internal asset variables (`ThermalPower`, `RenewablePower`, `BatteryCharge/Discharge`, `BatteryStatus`), reserves (`TotalReserve`, slacks), and bilevel dual/complementarity variables (`λ`, `μ`, `γ`, `κ`, `ν` families). | ||
| - **Custom parameters**: `DayAheadEnergyPrice`, `RealTimeEnergyPrice`, `AncillaryServicePrice`, `CyclingCharge/DischargeLimitParameter`. | ||
| - **Feedforwards**: `CyclingChargeLimitFeedforward`, `CyclingDischargeLimitFeedforward` for DA→RT cycling budget coupling. | ||
| - **PSI internal overrides** (sensitive to PSI version changes — re-verify on every PSI bump): | ||
| - `PSI.update_decision_state!` for DA bid and reserve variable state (DA bids span one hour of state rows each; all methods clamp to `max_state_index`). | ||
| - `PSI._update_parameter_values!` for hybrid profile/price updates during simulation (guarded by `PSI.get_component_names(attributes)`). | ||
| - `PSI._constituent_cost_expression(::DayAheadEnergyPrice)` — required by PSI's generic `update_variable_cost!`; without it, merchant simulations fail at the first parameter update (not at build). | ||
| - `PSI.validate_time_series!` for `HybridDecisionProblem` (multi-resolution and interval negotiation). | ||
| - **Catch discipline**: time-series lookups only swallow `ArgumentError` (`e isa ArgumentError || rethrow()`); container probes use `PSI.has_container_key`, never try/catch. |
There was a problem hiding this comment.
[JuliaFormatter] reported by reviewdog 🐶
| - **Custom variables**: market bids (`EnergyDABidOut/In`, `EnergyRTBidOut/In`, `BidReserveVariableOut/In`), internal asset variables (`ThermalPower`, `RenewablePower`, `BatteryCharge/Discharge`, `BatteryStatus`), reserves (`TotalReserve`, slacks), and bilevel dual/complementarity variables (`λ`, `μ`, `γ`, `κ`, `ν` families). | |
| - **Custom parameters**: `DayAheadEnergyPrice`, `RealTimeEnergyPrice`, `AncillaryServicePrice`, `CyclingCharge/DischargeLimitParameter`. | |
| - **Feedforwards**: `CyclingChargeLimitFeedforward`, `CyclingDischargeLimitFeedforward` for DA→RT cycling budget coupling. | |
| - **PSI internal overrides** (sensitive to PSI version changes — re-verify on every PSI bump): | |
| - `PSI.update_decision_state!` for DA bid and reserve variable state (DA bids span one hour of state rows each; all methods clamp to `max_state_index`). | |
| - `PSI._update_parameter_values!` for hybrid profile/price updates during simulation (guarded by `PSI.get_component_names(attributes)`). | |
| - `PSI._constituent_cost_expression(::DayAheadEnergyPrice)` — required by PSI's generic `update_variable_cost!`; without it, merchant simulations fail at the first parameter update (not at build). | |
| - `PSI.validate_time_series!` for `HybridDecisionProblem` (multi-resolution and interval negotiation). | |
| - **Catch discipline**: time-series lookups only swallow `ArgumentError` (`e isa ArgumentError || rethrow()`); container probes use `PSI.has_container_key`, never try/catch. | |
| - **Custom variables**: market bids (`EnergyDABidOut/In`, `EnergyRTBidOut/In`, `BidReserveVariableOut/In`), internal asset variables (`ThermalPower`, `RenewablePower`, `BatteryCharge/Discharge`, `BatteryStatus`), reserves (`TotalReserve`, slacks), and bilevel dual/complementarity variables (`λ`, `μ`, `γ`, `κ`, `ν` families). | |
| - **Custom parameters**: `DayAheadEnergyPrice`, `RealTimeEnergyPrice`, `AncillaryServicePrice`, `CyclingCharge/DischargeLimitParameter`. | |
| - **Feedforwards**: `CyclingChargeLimitFeedforward`, `CyclingDischargeLimitFeedforward` for DA→RT cycling budget coupling. | |
| - **PSI internal overrides** (sensitive to PSI version changes — re-verify on every PSI bump): | |
| + `PSI.update_decision_state!` for DA bid and reserve variable state (DA bids span one hour of state rows each; all methods clamp to `max_state_index`). | |
| + `PSI._update_parameter_values!` for hybrid profile/price updates during simulation (guarded by `PSI.get_component_names(attributes)`). | |
| + `PSI._constituent_cost_expression(::DayAheadEnergyPrice)` — required by PSI's generic `update_variable_cost!`; without it, merchant simulations fail at the first parameter update (not at build). | |
| + `PSI.validate_time_series!` for `HybridDecisionProblem` (multi-resolution and interval negotiation). | |
| - **Catch discipline**: time-series lookups only swallow `ArgumentError` (`e isa ArgumentError || rethrow()`); container probes use `PSI.has_container_key`, never try/catch. |
| - Requires PowerSystems 5.x and PowerSimulations 0.36.2+ (`~0.36.2` compat deliberately excludes PSI 0.36.0/0.36.1, which are broken for PowerModels-translated networks with parallel branches — `get_equivalent_physical_branch_parameters` MethodError against PNM 0.23). | ||
| - Storage cost access must use PSY5 accessors: `get_charge_variable_cost` / `get_discharge_variable_cost` → `get_vom_cost` → `get_proportional_term`. `StorageCost` has no `variable` field. |
There was a problem hiding this comment.
[JuliaFormatter] reported by reviewdog 🐶
| - Requires PowerSystems 5.x and PowerSimulations 0.36.2+ (`~0.36.2` compat deliberately excludes PSI 0.36.0/0.36.1, which are broken for PowerModels-translated networks with parallel branches — `get_equivalent_physical_branch_parameters` MethodError against PNM 0.23). | |
| - Storage cost access must use PSY5 accessors: `get_charge_variable_cost` / `get_discharge_variable_cost` → `get_vom_cost` → `get_proportional_term`. `StorageCost` has no `variable` field. | |
| - Requires PowerSystems 5.x and PowerSimulations 0.36.2+ (`~0.36.2` compat deliberately excludes PSI 0.36.0/0.36.1, which are broken for PowerModels-translated networks with parallel branches — `get_equivalent_physical_branch_parameters` MethodError against PNM 0.23). | |
| - Storage cost access must use PSY5 accessors: `get_charge_variable_cost` / `get_discharge_variable_cost` → `get_vom_cost` → `get_proportional_term`. `StorageCost` has no `variable` field. |
| - Run with `julia --project=test test/runtests.jl` (single testset: append the test file basename, e.g. `test_merchant_sequence`). See Sienna.md for the full test-environment conventions. | ||
| - Test systems come from PowerSystemCaseBuilder (RTS GMLC); market price fixtures are the `test/inputs/chuhsi_*` CSVs (hourly, 5-min, and 300/864-step variants). `attach_hybrid_market_time_series!` in `test/test_utils/function_utils.jl` is the single entry point for attaching them — `use_rt_resolution_for_da` switches between hourly-DA and RT-resolution-DA price setups, and both paths must stay covered. | ||
| - Merchant tests assert hourly DA axes (e.g., 24 DA bids vs 288 RT bids for a 24 h / 5 min model). Always assert `build!`/`solve!`/`execute!` return statuses, not just result shapes. | ||
| - Known modeling caveat: pinning a downstream UC's PCC variables to merchant DA bids via `FixValueFeedforward` is structurally infeasible while merchant DA buy/sell positions can overlap in the same hour (UC's reservation constraint forbids simultaneous in/out). See the note in `test/test_merchant_sequence.jl`. |
There was a problem hiding this comment.
[JuliaFormatter] reported by reviewdog 🐶
| - Run with `julia --project=test test/runtests.jl` (single testset: append the test file basename, e.g. `test_merchant_sequence`). See Sienna.md for the full test-environment conventions. | |
| - Test systems come from PowerSystemCaseBuilder (RTS GMLC); market price fixtures are the `test/inputs/chuhsi_*` CSVs (hourly, 5-min, and 300/864-step variants). `attach_hybrid_market_time_series!` in `test/test_utils/function_utils.jl` is the single entry point for attaching them — `use_rt_resolution_for_da` switches between hourly-DA and RT-resolution-DA price setups, and both paths must stay covered. | |
| - Merchant tests assert hourly DA axes (e.g., 24 DA bids vs 288 RT bids for a 24 h / 5 min model). Always assert `build!`/`solve!`/`execute!` return statuses, not just result shapes. | |
| - Known modeling caveat: pinning a downstream UC's PCC variables to merchant DA bids via `FixValueFeedforward` is structurally infeasible while merchant DA buy/sell positions can overlap in the same hour (UC's reservation constraint forbids simultaneous in/out). See the note in `test/test_merchant_sequence.jl`. | |
| - Run with `julia --project=test test/runtests.jl` (single testset: append the test file basename, e.g. `test_merchant_sequence`). See Sienna.md for the full test-environment conventions. | |
| - Test systems come from PowerSystemCaseBuilder (RTS GMLC); market price fixtures are the `test/inputs/chuhsi_*` CSVs (hourly, 5-min, and 300/864-step variants). `attach_hybrid_market_time_series!` in `test/test_utils/function_utils.jl` is the single entry point for attaching them — `use_rt_resolution_for_da` switches between hourly-DA and RT-resolution-DA price setups, and both paths must stay covered. | |
| - Merchant tests assert hourly DA axes (e.g., 24 DA bids vs 288 RT bids for a 24 h / 5 min model). Always assert `build!`/`solve!`/`execute!` return statuses, not just result shapes. | |
| - Known modeling caveat: pinning a downstream UC's PCC variables to merchant DA bids via `FixValueFeedforward` is structurally infeasible while merchant DA buy/sell positions can overlap in the same hour (UC's reservation constraint forbids simultaneous in/out). See the note in `test/test_merchant_sequence.jl`. |
Depends on Sienna-Platform/PowerSystems.jl#1660
Depends on Sienna-Platform/PowerSystemCaseBuilder.jl#191 for testing
Closes #38
Summary
This branch upgrades HybridSystemsSimulations.jl for current InfrastructureSystems, PowerSystems, and PowerSimulations stacks (PSY 5–oriented work), tightens hybrid / merchant modeling paths, and refreshes documentation (Diataxis-style layout, tutorial scaffolding, Documenter/formatter updates, doc preview cleanup workflow).
Compatibility & API surface
Project.toml/docs/Project.tomland aligns code with newer PSI behavior (e.g. removal ofsystem_to_fileusage where needed for PSI compatibility).StorageDispatchWithReservesand extends validation / initial-conditions handling so models with reserves behave correctly.src/core/formulations.jland relatedconstraints.jl,decision_models.jl,parameters.jl,variables.jl) and wires it through decision models and hybrid device models.PSI._update_parameter_values!override to keep merchant simulations consistent, and a guard that errors if the merchant model is used without aHybridSystem.ComplentarySlacknessCyclingCharge→ComplementarySlacknessCyclingChargeComplentarySlacknessCyclingDischarge→ComplementarySlacknessCyclingDischargeTests
StorageDispatchWithReservesand hybrid templates.HydroPowerSimulationsandStorageSystemsSimulationsfrom the test environment where no longer needed;runtests.jlno longer importsStorageSystemsSimulations.HybridSystem+EnergyReservoirStoragewithStorageDispatchWithReservesinto a singleHybridEnergyOnlyDispatch-style hybrid configuration withenergy_target/ reservation attributes (renamed testset for clarity).test/test_utils/price_generation_utils.jl), and trims dead code from test helpers.Docs & repo hygiene
docs/make.jlchanges plusdocs/make_tutorials.jlfor tutorial generation..github/workflows/doc-preview-cleanup.yml, updatesCONTRIBUTING.md, formatter script tweaks, and internal reference.claude/Sienna.md(from IS).Risk / review focus
system_to_file, old bookkeeping naming, merchant without hybrid)._update_parameter_values!override match your production cases.