Skip to content

Update to psy5 and document package#37

Merged
jd-lara merged 48 commits into
mainfrom
kd/psy5
Jun 12, 2026
Merged

Update to psy5 and document package#37
jd-lara merged 48 commits into
mainfrom
kd/psy5

Conversation

@kdayday

@kdayday kdayday commented Feb 25, 2026

Copy link
Copy Markdown
Collaborator

Depends on Sienna-Platform/PowerSystems.jl#1660
Depends on Sienna-Platform/PowerSystemCaseBuilder.jl#191 for testing
Closes #38


Summary

This branch upgrades HybridSystemsSimulations.jl for current InfrastructureSystems, PowerSystems, and PowerSimulations stacks (PSY 5–oriented work), tightens hybrid / merchant modeling paths, and refreshes documentation (Diataxis-style layout, tutorial scaffolding, Documenter/formatter updates, doc preview cleanup workflow).

Compatibility & API surface

  • Bumps dependency bounds in Project.toml / docs/Project.toml and aligns code with newer PSI behavior (e.g. removal of system_to_file usage where needed for PSI compatibility).
  • Renames Bookkeeping usage to StorageDispatchWithReserves and extends validation / initial-conditions handling so models with reserves behave correctly.
  • Adds / expands core formulation material (src/core/formulations.jl and related constraints.jl, decision_models.jl, parameters.jl, variables.jl) and wires it through decision models and hybrid device models.
  • Merchant path: updated thermal fuel / cost handling, a PSI._update_parameter_values! override to keep merchant simulations consistent, and a guard that errors if the merchant model is used without a HybridSystem.
  • Feedforwards and objective code updated for the new formulation / parameter patterns.
  • Breaking change typo fix: ComplentarySlacknessCyclingChargeComplementarySlacknessCyclingCharge ComplentarySlacknessCyclingDischargeComplementarySlacknessCyclingDischarge

Tests

  • Merchant and hybrid simulation tests updated for PSI/PSY changes; adds coverage around StorageDispatchWithReserves and hybrid templates.
  • Drops HydroPowerSimulations and StorageSystemsSimulations from the test environment where no longer needed; runtests.jl no longer imports StorageSystemsSimulations.
  • Simplifies the hybrid simulation test that previously split HybridSystem + EnergyReservoirStorage with StorageDispatchWithReserves into a single HybridEnergyOnlyDispatch-style hybrid configuration with energy_target / reservation attributes (renamed testset for clarity).
  • Removes obsolete / skipped “x_” test files and redundant price utilities (test/test_utils/price_generation_utils.jl), and trims dead code from test helpers.

Docs & repo hygiene

  • Substantial docs/make.jl changes plus docs/make_tutorials.jl for tutorial generation.
  • Adds .github/workflows/doc-preview-cleanup.yml, updates CONTRIBUTING.md, formatter script tweaks, and internal reference .claude/Sienna.md (from IS).
  • Formulations are documented within type docstrings, rather than separate formulation library

Risk / review focus

  • Breaking changes for callers still on older PSY/PSI: dependency floor and removed APIs (system_to_file, old bookkeeping naming, merchant without hybrid).
  • Merchant simulations: confirm fuel/cost data shapes and the _update_parameter_values! override match your production cases.
  • Docs: preview cleanup workflow and any secrets/permissions for GitHub Actions.

@kdayday kdayday mentioned this pull request Apr 16, 2026
1 task
@kdayday kdayday changed the title [WIP] Update to psy5 [WIP] Update to psy5 and document package Apr 16, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

JuliaFormatter

[JuliaFormatter] reported by reviewdog 🐶

https://github.com/NREL-Sienna/HybridSystemsSimulations.jl/blob/10fae251cacde55d6ccc9cdc6987e40770fc2eef/test/test_utils/function_utils.jl#L126-L126

@kdayday kdayday changed the title [WIP] Update to psy5 and document package Update to psy5 and document package Apr 17, 2026
@kdayday kdayday marked this pull request as ready for review April 17, 2026 16:43
Comment thread test/test_utils/function_utils.jl Outdated
@jd-lara jd-lara merged commit 62a6288 into main Jun 12, 2026
1 of 5 checks passed
@jd-lara jd-lara deleted the kd/psy5 branch June 12, 2026 19:07
Comment thread .claude/claude.md
Comment on lines +11 to +12
1. **Device formulations** — model a `HybridSystem` as one device inside a standard PSI `ProblemTemplate` (e.g., a UC or ED problem), dispatching the hybrid's internal assets subject to PCC limits.
2. **Merchant decision models** — custom `PSI.DecisionModel`s (with their own `build_impl!`) that optimize a price-taker hybrid's energy and ancillary-service bids against day-ahead (DA) and real-time (RT) market prices, including a bilevel/KKT formulation.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
1. **Device formulations** — model a `HybridSystem` as one device inside a standard PSI `ProblemTemplate` (e.g., a UC or ED problem), dispatching the hybrid's internal assets subject to PCC limits.
2. **Merchant decision models** — custom `PSI.DecisionModel`s (with their own `build_impl!`) that optimize a price-taker hybrid's energy and ancillary-service bids against day-ahead (DA) and real-time (RT) market prices, including a bilevel/KKT formulation.
1. **Device formulations** — model a `HybridSystem` as one device inside a standard PSI `ProblemTemplate` (e.g., a UC or ED problem), dispatching the hybrid's internal assets subject to PCC limits.
2. **Merchant decision models** — custom `PSI.DecisionModel`s (with their own `build_impl!`) that optimize a price-taker hybrid's energy and ancillary-service bids against day-ahead (DA) and real-time (RT) market prices, including a bilevel/KKT formulation.

Comment thread .claude/claude.md
Comment on lines +16 to +18
| Formulation | Description |
|---|---|
| `HybridEnergyOnlyDispatch` | Energy-only dispatch of the hybrid's internal assets behind the PCC |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
| Formulation | Description |
|---|---|
| `HybridEnergyOnlyDispatch` | Energy-only dispatch of the hybrid's internal assets behind the PCC |
| Formulation | Description |
|:---------------------------- |:--------------------------------------------------------------------------------------------------- |
| `HybridEnergyOnlyDispatch` | Energy-only dispatch of the hybrid's internal assets behind the PCC |

Comment thread .claude/claude.md
|---|---|
| `HybridEnergyOnlyDispatch` | Energy-only dispatch of the hybrid's internal assets behind the PCC |
| `HybridDispatchWithReserves` | Dispatch with ancillary-service participation, reserve assignment, and reserve coverage constraints |
| `HybridFixedDA` | Hybrid with fixed day-ahead positions (used downstream of a merchant stage) |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
| `HybridFixedDA` | Hybrid with fixed day-ahead positions (used downstream of a merchant stage) |
| `HybridFixedDA` | Hybrid with fixed day-ahead positions (used downstream of a merchant stage) |

Comment thread .claude/claude.md
Comment on lines +28 to +33
| Decision model | Description |
|---|---|
| `MerchantHybridEnergyCase` | DA + RT energy-only bid co-optimization |
| `MerchantHybridEnergyFixedDA` | RT subproblem with locked DA bids |
| `MerchantHybridCooptimizerCase` | DA + RT energy and ancillary-service bid co-optimization |
| `MerchantHybridBilevelCase` | Bilevel formulation (KKT conditions + complementary slackness via SOS1, strong duality) |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
| Decision model | Description |
|---|---|
| `MerchantHybridEnergyCase` | DA + RT energy-only bid co-optimization |
| `MerchantHybridEnergyFixedDA` | RT subproblem with locked DA bids |
| `MerchantHybridCooptimizerCase` | DA + RT energy and ancillary-service bid co-optimization |
| `MerchantHybridBilevelCase` | Bilevel formulation (KKT conditions + complementary slackness via SOS1, strong duality) |
| Decision model | Description |
|:------------------------------- |:--------------------------------------------------------------------------------------- |
| `MerchantHybridEnergyCase` | DA + RT energy-only bid co-optimization |
| `MerchantHybridEnergyFixedDA` | RT subproblem with locked DA bids |
| `MerchantHybridCooptimizerCase` | DA + RT energy and ancillary-service bid co-optimization |
| `MerchantHybridBilevelCase` | Bilevel formulation (KKT conditions + complementary slackness via SOS1, strong duality) |

Comment thread .claude/claude.md
Comment on lines +39 to +45
- **DA bids are hourly slots; RT variables follow the model resolution.** The DA axis is `merchant_da_time_step_range(container, hybrid)` = `1:min(horizon_hours, DA-series length)`; the variable, parameter, constraint, and objective axes must all use it. RT-to-DA index mapping goes through `merchant_rt_to_da_tmap(rt_len, da_len)` — never hand-roll `div`-based maps.
- **Storage energy quantities are in energy units**, computed as `get_storage_level_limits(storage).{min,max} * get_storage_capacity(storage)`. The same convention applies to initial conditions (`get_initial_storage_capacity_level * capacity`), cycling limits, and `storage_target`. Do not use the bare level fractions.
- **Market prices are hybrid-attached scalar `SingleTimeSeries`**, keyed by name (defaults `"DA"`/`"RT"`, override via `model.ext["day_ahead_time_series_key"]` / `"real_time_time_series_key"`):
- Energy: `hybrid_energy_price_time_series_name(key)` → `"HybridSystem__energy_price__<key>"`
- Ancillary: `hybrid_ancillary_service_price_time_series_name(service, key)`
- Profiles: `"RenewableDispatch__max_active_power"`, `"PowerLoad__max_active_power"`
- `Δt_RT` must come from the container/settings resolution (`PSI.get_resolution`), not from `first(PSY.get_time_series_resolutions(sys))` — systems may carry multiple resolutions and `PSI.validate_time_series!` negotiates the selected one into settings.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
- **DA bids are hourly slots; RT variables follow the model resolution.** The DA axis is `merchant_da_time_step_range(container, hybrid)` = `1:min(horizon_hours, DA-series length)`; the variable, parameter, constraint, and objective axes must all use it. RT-to-DA index mapping goes through `merchant_rt_to_da_tmap(rt_len, da_len)` — never hand-roll `div`-based maps.
- **Storage energy quantities are in energy units**, computed as `get_storage_level_limits(storage).{min,max} * get_storage_capacity(storage)`. The same convention applies to initial conditions (`get_initial_storage_capacity_level * capacity`), cycling limits, and `storage_target`. Do not use the bare level fractions.
- **Market prices are hybrid-attached scalar `SingleTimeSeries`**, keyed by name (defaults `"DA"`/`"RT"`, override via `model.ext["day_ahead_time_series_key"]` / `"real_time_time_series_key"`):
- Energy: `hybrid_energy_price_time_series_name(key)``"HybridSystem__energy_price__<key>"`
- Ancillary: `hybrid_ancillary_service_price_time_series_name(service, key)`
- Profiles: `"RenewableDispatch__max_active_power"`, `"PowerLoad__max_active_power"`
- `Δt_RT` must come from the container/settings resolution (`PSI.get_resolution`), not from `first(PSY.get_time_series_resolutions(sys))` — systems may carry multiple resolutions and `PSI.validate_time_series!` negotiates the selected one into settings.
- **DA bids are hourly slots; RT variables follow the model resolution.** The DA axis is `merchant_da_time_step_range(container, hybrid)` = `1:min(horizon_hours, DA-series length)`; the variable, parameter, constraint, and objective axes must all use it. RT-to-DA index mapping goes through `merchant_rt_to_da_tmap(rt_len, da_len)` — never hand-roll `div`-based maps.
- **Storage energy quantities are in energy units**, computed as `get_storage_level_limits(storage).{min,max} * get_storage_capacity(storage)`. The same convention applies to initial conditions (`get_initial_storage_capacity_level * capacity`), cycling limits, and `storage_target`. Do not use the bare level fractions.
- **Market prices are hybrid-attached scalar `SingleTimeSeries`**, keyed by name (defaults `"DA"`/`"RT"`, override via `model.ext["day_ahead_time_series_key"]` / `"real_time_time_series_key"`):
+ Energy: `hybrid_energy_price_time_series_name(key)` → `"HybridSystem__energy_price__<key>"`
+ Ancillary: `hybrid_ancillary_service_price_time_series_name(service, key)`
+ Profiles: `"RenewableDispatch__max_active_power"`, `"PowerLoad__max_active_power"`
- `Δt_RT` must come from the container/settings resolution (`PSI.get_resolution`), not from `first(PSY.get_time_series_resolutions(sys))` — systems may carry multiple resolutions and `PSI.validate_time_series!` negotiates the selected one into settings.

Comment thread .claude/claude.md
Comment on lines +49 to +57
- **Custom variables**: market bids (`EnergyDABidOut/In`, `EnergyRTBidOut/In`, `BidReserveVariableOut/In`), internal asset variables (`ThermalPower`, `RenewablePower`, `BatteryCharge/Discharge`, `BatteryStatus`), reserves (`TotalReserve`, slacks), and bilevel dual/complementarity variables (`λ`, `μ`, `γ`, `κ`, `ν` families).
- **Custom parameters**: `DayAheadEnergyPrice`, `RealTimeEnergyPrice`, `AncillaryServicePrice`, `CyclingCharge/DischargeLimitParameter`.
- **Feedforwards**: `CyclingChargeLimitFeedforward`, `CyclingDischargeLimitFeedforward` for DA→RT cycling budget coupling.
- **PSI internal overrides** (sensitive to PSI version changes — re-verify on every PSI bump):
- `PSI.update_decision_state!` for DA bid and reserve variable state (DA bids span one hour of state rows each; all methods clamp to `max_state_index`).
- `PSI._update_parameter_values!` for hybrid profile/price updates during simulation (guarded by `PSI.get_component_names(attributes)`).
- `PSI._constituent_cost_expression(::DayAheadEnergyPrice)` — required by PSI's generic `update_variable_cost!`; without it, merchant simulations fail at the first parameter update (not at build).
- `PSI.validate_time_series!` for `HybridDecisionProblem` (multi-resolution and interval negotiation).
- **Catch discipline**: time-series lookups only swallow `ArgumentError` (`e isa ArgumentError || rethrow()`); container probes use `PSI.has_container_key`, never try/catch.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
- **Custom variables**: market bids (`EnergyDABidOut/In`, `EnergyRTBidOut/In`, `BidReserveVariableOut/In`), internal asset variables (`ThermalPower`, `RenewablePower`, `BatteryCharge/Discharge`, `BatteryStatus`), reserves (`TotalReserve`, slacks), and bilevel dual/complementarity variables (`λ`, `μ`, `γ`, `κ`, `ν` families).
- **Custom parameters**: `DayAheadEnergyPrice`, `RealTimeEnergyPrice`, `AncillaryServicePrice`, `CyclingCharge/DischargeLimitParameter`.
- **Feedforwards**: `CyclingChargeLimitFeedforward`, `CyclingDischargeLimitFeedforward` for DA→RT cycling budget coupling.
- **PSI internal overrides** (sensitive to PSI version changes — re-verify on every PSI bump):
- `PSI.update_decision_state!` for DA bid and reserve variable state (DA bids span one hour of state rows each; all methods clamp to `max_state_index`).
- `PSI._update_parameter_values!` for hybrid profile/price updates during simulation (guarded by `PSI.get_component_names(attributes)`).
- `PSI._constituent_cost_expression(::DayAheadEnergyPrice)` — required by PSI's generic `update_variable_cost!`; without it, merchant simulations fail at the first parameter update (not at build).
- `PSI.validate_time_series!` for `HybridDecisionProblem` (multi-resolution and interval negotiation).
- **Catch discipline**: time-series lookups only swallow `ArgumentError` (`e isa ArgumentError || rethrow()`); container probes use `PSI.has_container_key`, never try/catch.
- **Custom variables**: market bids (`EnergyDABidOut/In`, `EnergyRTBidOut/In`, `BidReserveVariableOut/In`), internal asset variables (`ThermalPower`, `RenewablePower`, `BatteryCharge/Discharge`, `BatteryStatus`), reserves (`TotalReserve`, slacks), and bilevel dual/complementarity variables (`λ`, `μ`, `γ`, `κ`, `ν` families).
- **Custom parameters**: `DayAheadEnergyPrice`, `RealTimeEnergyPrice`, `AncillaryServicePrice`, `CyclingCharge/DischargeLimitParameter`.
- **Feedforwards**: `CyclingChargeLimitFeedforward`, `CyclingDischargeLimitFeedforward` for DA→RT cycling budget coupling.
- **PSI internal overrides** (sensitive to PSI version changes — re-verify on every PSI bump):
+ `PSI.update_decision_state!` for DA bid and reserve variable state (DA bids span one hour of state rows each; all methods clamp to `max_state_index`).
+ `PSI._update_parameter_values!` for hybrid profile/price updates during simulation (guarded by `PSI.get_component_names(attributes)`).
+ `PSI._constituent_cost_expression(::DayAheadEnergyPrice)` — required by PSI's generic `update_variable_cost!`; without it, merchant simulations fail at the first parameter update (not at build).
+ `PSI.validate_time_series!` for `HybridDecisionProblem` (multi-resolution and interval negotiation).
- **Catch discipline**: time-series lookups only swallow `ArgumentError` (`e isa ArgumentError || rethrow()`); container probes use `PSI.has_container_key`, never try/catch.

Comment thread .claude/claude.md
Comment on lines +82 to +83
- Requires PowerSystems 5.x and PowerSimulations 0.36.2+ (`~0.36.2` compat deliberately excludes PSI 0.36.0/0.36.1, which are broken for PowerModels-translated networks with parallel branches — `get_equivalent_physical_branch_parameters` MethodError against PNM 0.23).
- Storage cost access must use PSY5 accessors: `get_charge_variable_cost` / `get_discharge_variable_cost` → `get_vom_cost` → `get_proportional_term`. `StorageCost` has no `variable` field.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
- Requires PowerSystems 5.x and PowerSimulations 0.36.2+ (`~0.36.2` compat deliberately excludes PSI 0.36.0/0.36.1, which are broken for PowerModels-translated networks with parallel branches — `get_equivalent_physical_branch_parameters` MethodError against PNM 0.23).
- Storage cost access must use PSY5 accessors: `get_charge_variable_cost` / `get_discharge_variable_cost``get_vom_cost``get_proportional_term`. `StorageCost` has no `variable` field.
- Requires PowerSystems 5.x and PowerSimulations 0.36.2+ (`~0.36.2` compat deliberately excludes PSI 0.36.0/0.36.1, which are broken for PowerModels-translated networks with parallel branches — `get_equivalent_physical_branch_parameters` MethodError against PNM 0.23).
- Storage cost access must use PSY5 accessors: `get_charge_variable_cost` / `get_discharge_variable_cost``get_vom_cost``get_proportional_term`. `StorageCost` has no `variable` field.

Comment thread .claude/claude.md
Comment on lines +87 to +90
- Run with `julia --project=test test/runtests.jl` (single testset: append the test file basename, e.g. `test_merchant_sequence`). See Sienna.md for the full test-environment conventions.
- Test systems come from PowerSystemCaseBuilder (RTS GMLC); market price fixtures are the `test/inputs/chuhsi_*` CSVs (hourly, 5-min, and 300/864-step variants). `attach_hybrid_market_time_series!` in `test/test_utils/function_utils.jl` is the single entry point for attaching them — `use_rt_resolution_for_da` switches between hourly-DA and RT-resolution-DA price setups, and both paths must stay covered.
- Merchant tests assert hourly DA axes (e.g., 24 DA bids vs 288 RT bids for a 24 h / 5 min model). Always assert `build!`/`solve!`/`execute!` return statuses, not just result shapes.
- Known modeling caveat: pinning a downstream UC's PCC variables to merchant DA bids via `FixValueFeedforward` is structurally infeasible while merchant DA buy/sell positions can overlap in the same hour (UC's reservation constraint forbids simultaneous in/out). See the note in `test/test_merchant_sequence.jl`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[JuliaFormatter] reported by reviewdog 🐶

Suggested change
- Run with `julia --project=test test/runtests.jl` (single testset: append the test file basename, e.g. `test_merchant_sequence`). See Sienna.md for the full test-environment conventions.
- Test systems come from PowerSystemCaseBuilder (RTS GMLC); market price fixtures are the `test/inputs/chuhsi_*` CSVs (hourly, 5-min, and 300/864-step variants). `attach_hybrid_market_time_series!` in `test/test_utils/function_utils.jl` is the single entry point for attaching them — `use_rt_resolution_for_da` switches between hourly-DA and RT-resolution-DA price setups, and both paths must stay covered.
- Merchant tests assert hourly DA axes (e.g., 24 DA bids vs 288 RT bids for a 24 h / 5 min model). Always assert `build!`/`solve!`/`execute!` return statuses, not just result shapes.
- Known modeling caveat: pinning a downstream UC's PCC variables to merchant DA bids via `FixValueFeedforward` is structurally infeasible while merchant DA buy/sell positions can overlap in the same hour (UC's reservation constraint forbids simultaneous in/out). See the note in `test/test_merchant_sequence.jl`.
- Run with `julia --project=test test/runtests.jl` (single testset: append the test file basename, e.g. `test_merchant_sequence`). See Sienna.md for the full test-environment conventions.
- Test systems come from PowerSystemCaseBuilder (RTS GMLC); market price fixtures are the `test/inputs/chuhsi_*` CSVs (hourly, 5-min, and 300/864-step variants). `attach_hybrid_market_time_series!` in `test/test_utils/function_utils.jl` is the single entry point for attaching them — `use_rt_resolution_for_da` switches between hourly-DA and RT-resolution-DA price setups, and both paths must stay covered.
- Merchant tests assert hourly DA axes (e.g., 24 DA bids vs 288 RT bids for a 24 h / 5 min model). Always assert `build!`/`solve!`/`execute!` return statuses, not just result shapes.
- Known modeling caveat: pinning a downstream UC's PCC variables to merchant DA bids via `FixValueFeedforward` is structurally infeasible while merchant DA buy/sell positions can overlap in the same hour (UC's reservation constraint forbids simultaneous in/out). See the note in `test/test_merchant_sequence.jl`.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update HSS for PSY5 compatibility

4 participants