Update to psy5 and document package

.claude/Sienna.md

@@ -0,0 +1,169 @@
+# Sienna Programming Practices
+
+This document describes general programming practices and conventions that apply across all Sienna packages (PowerSystems.jl, PowerSimulations.jl, PowerFlows.jl, PowerNetworkMatrices.jl, InfrastructureSystems.jl, etc.).
+
+## Performance Requirements
+
+**Priority:** Critical. See the [Julia Performance Tips](https://docs.julialang.org/en/v1/manual/performance-tips/).
+
+### Anti-Patterns to Avoid
+
+#### Type instability
+
+Functions must return consistent concrete types. Check with `@code_warntype`.
+
+  - Bad: `f(x) = x > 0 ? 1 : 1.0`
+  - Good: `f(x) = x > 0 ? 1.0 : 1.0`
+
+#### Abstract field types
+
+Struct fields must have concrete types or be parameterized.
+
+  - Bad: `struct Foo; data::AbstractVector; end`
+  - Good: `struct Foo{T<:AbstractVector}; data::T; end`
+
+#### Untyped containers
+
+  - Bad: `Vector{Any}()`, `Vector{Real}()`
+  - Good: `Vector{Float64}()`, `Vector{Int}()`
+
+#### Non-const globals
+
+  - Bad: `THRESHOLD = 0.5`
+  - Good: `const THRESHOLD = 0.5`
+
+#### Unnecessary allocations
+
+  - Use views instead of copies (`@view`, `@views`)
+  - Pre-allocate arrays instead of `push!` in loops
+  - Use in-place operations (functions ending with `!`)
+
+#### Captured variables
+
+Avoid closures that capture variables causing boxing. Pass variables as function arguments instead.
+
+#### Splatting penalty
+
+Avoid splatting (`...`) in performance-critical code.
+
+#### Abstract return types
+
+Avoid returning `Union` types or abstract types.
+
+### Best Practices
+
+  - Use `@inbounds` when bounds are verified
+  - Use broadcasting (dot syntax) for element-wise operations
+  - Avoid `try-catch` in hot paths
+  - Use function barriers to isolate type instability
+
+> Apply these guidelines with judgment. Not every function is performance-critical. Focus optimization efforts on hot paths and frequently called code.
+
+## Code Conventions
+
+Style guide: [https://sienna-platform.github.io/InfrastructureSystems.jl/stable/style/](https://sienna-platform.github.io/InfrastructureSystems.jl/stable/style/)
+
+Formatter (JuliaFormatter): Use the formatter script provided in each package.
+
+Key rules:
+
+  - Constructors: use `function Foo()` not `Foo() = ...`
+  - Asserts: prefer `InfrastructureSystems.@assert_op` over `@assert`
+  - Globals: `UPPER_CASE` for constants
+  - Exports: all exports in main module file
+  - Comments: complete sentences, describe why not how
+
+## Documentation Practices and Requirements
+
+Framework: [Diataxis](https://diataxis.fr/)
+
+Sienna guide: [https://sienna-platform.github.io/InfrastructureSystems.jl/stable/docs_best_practices/explanation/](https://sienna-platform.github.io/InfrastructureSystems.jl/stable/docs_best_practices/explanation/)
+
+Sienna guide for Diataxis-style tutorials: [https://sienna-platform.github.io/InfrastructureSystems.jl/stable/docs_best_practices/how-to/write_a_tutorial/](https://sienna-platform.github.io/InfrastructureSystems.jl/stable/docs_best_practices/how-to/write_a_tutorial/)
+Format for tutorial scripts: [https://fredrikekre.github.io/Literate.jl/v2/](https://fredrikekre.github.io/Literate.jl/v2/)
+Sienna guide for Diataxis-style how-to's: [https://sienna-platform.github.io/InfrastructureSystems.jl/stable/docs_best_practices/how-to/write_a_how-to/](https://sienna-platform.github.io/InfrastructureSystems.jl/stable/docs_best_practices/how-to/write_a_how-to/)
+Sienna guide for APIs: [https://sienna-platform.github.io/InfrastructureSystems.jl/stable/docs_best_practices/how-to/write_docstrings_org_api/](https://sienna-platform.github.io/InfrastructureSystems.jl/stable/docs_best_practices/how-to/write_docstrings_org_api/)
+
+Docstring requirements:
+
+  - Scope: all elements of public interface (IS is selective about exports)
+  - Include: function signatures and arguments list
+  - Automation: `DocStringExtensions.TYPEDSIGNATURES` (`TYPEDFIELDS` used sparingly in IS)
+  - See also: add links for functions with same name (multiple dispatch)
+
+API docs:
+
+  - Public: typically in `docs/src/api/public.md` using `@autodocs` with `Public=true, Private=false`
+  - Internals: typically in `docs/src/api/internals.md`
+
+## Design Principles
+
+  - Elegance and concision in both interface and implementation
+  - Fail fast with actionable error messages rather than hiding problems
+  - Validate invariants explicitly in subtle cases
+  - Avoid over-adherence to backwards compatibility for internal helpers
+
+## Contribution Workflow
+
+**Note:** The default branch for all Sienna packages is `main`, not `master`.
+
+Branch naming: `feature/description` or `fix/description`
+
+ 1. Create feature branch
+ 2. Follow style guide and run formatter
+ 3. Ensure tests pass
+ 4. Submit pull request
+
+## AI Agent Guidance
+
+**Key priorities:** Read existing patterns first, maintain consistency, use concrete types in hot paths, run formatter, add docstrings to public API, ensure tests pass.
+
+**Critical rules:**
+
+  - Always use `julia --project=<env>` (never bare `julia`)
+  - Never edit auto-generated files directly
+  - Verify type stability with `@code_warntype` for performance-critical code
+  - Consider downstream package impact
+
+## Julia Environment Best Practices
+
+**CRITICAL:** Always use `julia --project=<env>` when running Julia code in Sienna repositories. **NEVER** use bare `julia` or `julia --project` without specifying the environment. Each package typically defines dependencies in `test/Project.toml` for testing.
+
+Common patterns:
+
+```sh
+# Run tests (using test environment)
+julia --project=test test/runtests.jl
+
+# Run specific test
+julia --project=test test/runtests.jl test_file_name
+
+# Run expression
+julia --project=test -e 'using PackageName; ...'
+
+# Instantiate environment
+julia --project=test -e 'using Pkg; Pkg.instantiate()'
+
+# Build docs (using docs environment)
+julia --project=docs docs/make.jl
+```
+
+**Why this matters:** Running without `--project=<env>` will fail because required packages won't be available in the default environment. The test/docs environments contain all necessary dependencies for their respective tasks.
+
+## Troubleshooting
+
+**Type instability**
+
+  - Symptom: Poor performance, many allocations
+  - Diagnosis: `@code_warntype` on suspect function
+  - Solution: See performance anti-patterns above
+
+**Formatter fails**
+
+  - Symptom: Formatter command returns error
+  - Solution: Run the formatter script provided in the package (e.g., `julia -e 'include("scripts/formatter/formatter_code.jl")'`)
+
+**Test failures**
+
+  - Symptom: Tests fail unexpectedly
+  - Solution: `julia --project=test -e 'using Pkg; Pkg.instantiate()'`

.claude/claude.md

@@ -0,0 +1,90 @@
+# HybridSystemsSimulations.jl Repository Guide
+
+> **Development Guidelines:** Always load [Sienna.md](./Sienna.md) development preferences, style conventions, and best practices for projects using Sienna. Before running tests confirm that the [Sienna.md](./Sienna.md) file has been read.
+
+## Overview
+
+HybridSystemsSimulations.jl is an extension of PowerSimulations.jl (PSI) that provides optimization models for `PowerSystems.HybridSystem` devices: a thermal unit, renewable unit, storage, and/or electric load co-located behind a single point of common coupling (PCC). It is part of the Sienna ecosystem.
+
+The package supports two distinct usage modes:
+
+1. **Device formulations** — model a `HybridSystem` as one device inside a standard PSI `ProblemTemplate` (e.g., a UC or ED problem), dispatching the hybrid's internal assets subject to PCC limits.
+2. **Merchant decision models** — custom `PSI.DecisionModel`s (with their own `build_impl!`) that optimize a price-taker hybrid's energy and ancillary-service bids against day-ahead (DA) and real-time (RT) market prices, including a bilevel/KKT formulation.
+
+## Device Formulations
+
+| Formulation | Description |
+|---|---|
+| `HybridEnergyOnlyDispatch` | Energy-only dispatch of the hybrid's internal assets behind the PCC |
+| `HybridDispatchWithReserves` | Dispatch with ancillary-service participation, reserve assignment, and reserve coverage constraints |
+| `HybridFixedDA` | Hybrid with fixed day-ahead positions (used downstream of a merchant stage) |
+
+Optional `DeviceModel` attributes: `"cycling"`, `"energy_target"`, `"regularization"`, `"reservation"`.
+
+## Merchant Decision Models
+
+All subtype `HybridDecisionProblem <: PSI.DecisionProblem` and implement custom `PSI.build_impl!`:
+
+| Decision model | Description |
+|---|---|
+| `MerchantHybridEnergyCase` | DA + RT energy-only bid co-optimization |
+| `MerchantHybridEnergyFixedDA` | RT subproblem with locked DA bids |
+| `MerchantHybridCooptimizerCase` | DA + RT energy and ancillary-service bid co-optimization |
+| `MerchantHybridBilevelCase` | Bilevel formulation (KKT conditions + complementary slackness via SOS1, strong duality) |
+
+A `HybridSystem` must be present in the `System`; the builds error early otherwise.
+
+## Time and Data Conventions (important — easy to get wrong)
+
+- **DA bids are hourly slots; RT variables follow the model resolution.** The DA axis is `merchant_da_time_step_range(container, hybrid)` = `1:min(horizon_hours, DA-series length)`; the variable, parameter, constraint, and objective axes must all use it. RT-to-DA index mapping goes through `merchant_rt_to_da_tmap(rt_len, da_len)` — never hand-roll `div`-based maps.
+- **Storage energy quantities are in energy units**, computed as `get_storage_level_limits(storage).{min,max} * get_storage_capacity(storage)`. The same convention applies to initial conditions (`get_initial_storage_capacity_level * capacity`), cycling limits, and `storage_target`. Do not use the bare level fractions.
+- **Market prices are hybrid-attached scalar `SingleTimeSeries`**, keyed by name (defaults `"DA"`/`"RT"`, override via `model.ext["day_ahead_time_series_key"]` / `"real_time_time_series_key"`):
+  - Energy: `hybrid_energy_price_time_series_name(key)` → `"HybridSystem__energy_price__<key>"`
+  - Ancillary: `hybrid_ancillary_service_price_time_series_name(service, key)`
+  - Profiles: `"RenewableDispatch__max_active_power"`, `"PowerLoad__max_active_power"`
+- `Δt_RT` must come from the container/settings resolution (`PSI.get_resolution`), not from `first(PSY.get_time_series_resolutions(sys))` — systems may carry multiple resolutions and `PSI.validate_time_series!` negotiates the selected one into settings.
+
+## How It Extends PowerSimulations.jl
+
+- **Custom variables**: market bids (`EnergyDABidOut/In`, `EnergyRTBidOut/In`, `BidReserveVariableOut/In`), internal asset variables (`ThermalPower`, `RenewablePower`, `BatteryCharge/Discharge`, `BatteryStatus`), reserves (`TotalReserve`, slacks), and bilevel dual/complementarity variables (`λ`, `μ`, `γ`, `κ`, `ν` families).
+- **Custom parameters**: `DayAheadEnergyPrice`, `RealTimeEnergyPrice`, `AncillaryServicePrice`, `CyclingCharge/DischargeLimitParameter`.
+- **Feedforwards**: `CyclingChargeLimitFeedforward`, `CyclingDischargeLimitFeedforward` for DA→RT cycling budget coupling.
+- **PSI internal overrides** (sensitive to PSI version changes — re-verify on every PSI bump):
+  - `PSI.update_decision_state!` for DA bid and reserve variable state (DA bids span one hour of state rows each; all methods clamp to `max_state_index`).
+  - `PSI._update_parameter_values!` for hybrid profile/price updates during simulation (guarded by `PSI.get_component_names(attributes)`).
+  - `PSI._constituent_cost_expression(::DayAheadEnergyPrice)` — required by PSI's generic `update_variable_cost!`; without it, merchant simulations fail at the first parameter update (not at build).
+  - `PSI.validate_time_series!` for `HybridDecisionProblem` (multi-resolution and interval negotiation).
+- **Catch discipline**: time-series lookups only swallow `ArgumentError` (`e isa ArgumentError || rethrow()`); container probes use `PSI.has_container_key`, never try/catch.
+
+## Source Layout
+
+```
+src/
+  core/                              # Type definitions: decision models + time-series keys
+                                     #   (decision_models.jl), formulations, variables,
+                                     #   aux_variables, constraints, expressions, parameters
+  hybrid_system_decision_models.jl   # Decision-state updates, simulation-stage parameter glue
+  hybrid_system_device_models.jl     # Variable bounds, initial conditions, device-model attributes
+  add_variables.jl                   # Variable constructors (DA axis via merchant_da_time_step_range)
+  add_aux_variables.jl               # Cycling usage aux variables
+  add_parameters.jl                  # Time-series + price parameters, simulation update overrides
+  add_constraints.jl                 # All constraints incl. bilevel KKT/complementary slackness
+  objective_function.jl              # Cost terms (Δt- and system-unit-scaled) and PSY5 cost helpers
+  feedforwards.jl                    # Cycling limit feedforwards
+  decision_models/                   # build_impl! for only_energy, cooptimizer, bilevel cases
+  hybrid_system_constructor.jl       # PSI device constructor entry points
+```
+
+Respect the include order in `src/HybridSystemsSimulations.jl`: `core/` files are included first; new types/constants go there.
+
+## Dependencies and Compatibility
+
+- Requires PowerSystems 5.x and PowerSimulations 0.36.2+ (`~0.36.2` compat deliberately excludes PSI 0.36.0/0.36.1, which are broken for PowerModels-translated networks with parallel branches — `get_equivalent_physical_branch_parameters` MethodError against PNM 0.23).
+- Storage cost access must use PSY5 accessors: `get_charge_variable_cost` / `get_discharge_variable_cost` → `get_vom_cost` → `get_proportional_term`. `StorageCost` has no `variable` field.
+
+## Testing
+
+- Run with `julia --project=test test/runtests.jl` (single testset: append the test file basename, e.g. `test_merchant_sequence`). See Sienna.md for the full test-environment conventions.
+- Test systems come from PowerSystemCaseBuilder (RTS GMLC); market price fixtures are the `test/inputs/chuhsi_*` CSVs (hourly, 5-min, and 300/864-step variants). `attach_hybrid_market_time_series!` in `test/test_utils/function_utils.jl` is the single entry point for attaching them — `use_rt_resolution_for_da` switches between hourly-DA and RT-resolution-DA price setups, and both paths must stay covered.
+- Merchant tests assert hourly DA axes (e.g., 24 DA bids vs 288 RT bids for a 24 h / 5 min model). Always assert `build!`/`solve!`/`execute!` return statuses, not just result shapes.
+- Known modeling caveat: pinning a downstream UC's PCC variables to merchant DA bids via `FixValueFeedforward` is structurally infeasible while merchant DA buy/sell positions can overlap in the same hour (UC's reservation constraint forbids simultaneous in/out). See the note in `test/test_merchant_sequence.jl`.

.github/workflows/doc-preview-cleanup.yml

@@ -0,0 +1,35 @@
+name: Doc Preview Cleanup
+
+on:
+  pull_request:
+    types: [closed]
+
+# Ensure that only one "Doc Preview Cleanup" workflow is force pushing at a time
+concurrency:
+  group: doc-preview-cleanup
+  cancel-in-progress: false
+
+jobs:
+  doc-preview-cleanup:
+    runs-on: ubuntu-latest
+    if: github.event.pull_request.head.repo.fork == false
+    # This workflow pushes to gh-pages; permissions are per-job and independent of docs.yml
+    permissions:
+      contents: write
+    steps:
+      - name: Checkout gh-pages branch
+        uses: actions/checkout@v4
+        with:
+          ref: gh-pages
+      - name: Delete preview and history + push changes
+        run: |
+          if [ -d "${preview_dir}" ]; then
+              git config user.name "Documenter.jl"
+              git config user.email "documenter@juliadocs.github.io"
+              git rm -rf "${preview_dir}"
+              git commit -m "delete preview"
+              git branch gh-pages-new "$(echo "delete history" | git commit-tree "HEAD^{tree}")"
+              git push --force origin gh-pages-new:gh-pages
+          fi
+        env:
+          preview_dir: previews/PR${{ github.event.number }}

.github/workflows/docs.yml

@@ -18,6 +18,16 @@ jobs:
           version: '1'
       - name: Install dependencies
         run: julia --project=docs/ -e 'using Pkg; Pkg.develop(PackageSpec(path=pwd())); Pkg.instantiate()'
+
+      - name: Set DOCUMENTER_CURRENT_VERSION for tutorial download links
+        run: |
+          if [[ "${{ github.event_name }}" == "pull_request" ]]; then
+            echo "DOCUMENTER_CURRENT_VERSION=previews/PR${{ github.event.pull_request.number }}" >> "$GITHUB_ENV"
+          elif [[ "${{ github.ref }}" == refs/tags/* ]]; then
+            echo "DOCUMENTER_CURRENT_VERSION=${GITHUB_REF_NAME}" >> "$GITHUB_ENV"
+          elif [[ "${{ github.ref }}" == "refs/heads/main" ]] || [[ "${{ github.ref }}" =~ ^refs/heads/release- ]]; then
+            echo "DOCUMENTER_CURRENT_VERSION=dev" >> "$GITHUB_ENV"
+          fi
       - name: Build and deploy
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.pre-commit-config.yaml

@@ -0,0 +1,9 @@
+repos:
+  - repo: local
+    hooks:
+      - id: julia-formatter
+        name: Run Julia formatter
+        entry: julia scripts/formatter/formatter_code.jl
+        language: system
+        types: [file]
+        pass_filenames: false

CONTRIBUTING.md

@@ -1,6 +1,7 @@
 # Contributing
 
-Community driven development of this package is encouraged. To maintain code quality standards, please adhere to the following guidlines when contributing:
- - To get started, <a href="https://www.clahub.com/agreements/NREL/SIIP-PACKAGE.jl">sign the Contributor License Agreement</a>.
- - Please do your best to adhere to our [coding style guide](docs/src/developer/style.md).
- - To submit code contributions, [fork](https://help.github.com/articles/fork-a-repo/) the repository, commit your changes, and [submit a pull request](https://help.github.com/articles/creating-a-pull-request-from-a-fork/).
+Community driven development of this package is encouraged. To maintain code quality standards, please adhere to the following guidelines when contributing:
+
+  - To get started, <a href="https://www.clahub.com/agreements/NREL/SIIP-PACKAGE.jl">sign the Contributor License Agreement</a>.
+  - Please do your best to adhere to our [coding style guide](docs/src/developer/style.md).
+  - To submit code contributions, [fork](https://help.github.com/articles/fork-a-repo/) the repository, commit your changes, and [submit a pull request](https://help.github.com/articles/creating-a-pull-request-from-a-fork/).

Project.toml

@@ -13,10 +13,10 @@ PowerSimulations = "e690365d-45e2-57bb-ac84-44ba829e73c4"
 PowerSystems = "bcd98974-b02a-5e2f-9ee0-a103f5c450dd"
 
 [compat]
-DataStructures = "~0.18"
-DocStringExtensions = "~0.8, ~0.9"
-JuMP = "1"
+DataStructures = "~0.18, ^0.19"
+DocStringExtensions = "0.8, 0.9.2"
+JuMP = "^1.28"
 MathOptInterface = "1"
-PowerSimulations = "^0.29"
-PowerSystems = "4"
-julia = "^1.6"
+PowerSimulations = "~0.36.2"
+PowerSystems = "^5.11"
+julia = "^1.10"

docs/Project.toml

@@ -1,9 +1,13 @@
 [deps]
+DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
 DataStructures = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
 Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
+DocumenterInterLinks = "d12716ef-a0f6-4df4-a9f1-a5a34e75c656"
 DocumenterTools = "35a29f4d-8980-5a13-9543-d66fff28ecb8"
 HybridSystemsSimulations = "bed98974-b02a-5e2f-9ee0-a103f5c450dd"
+Literate = "98b081ad-f1c9-55d3-8b20-4c87d4299306"
+PrettyTables = "08abe8d2-0d0c-5749-adfa-8a2ac140af0d"
 
 [compat]
-Documenter = "0.27"
-julia = "^1.6"
+Documenter = "1.0"
+julia = "^1.10"