Jd/misc fixes by jd-lara · Pull Request #73 · Sienna-Platform/PowerAnalytics.jl

jd-lara · 2026-05-19T04:04:57Z

closes a bunch of pending issues to make PowerAnalytics usable with modern PowerSimulations

* Update legacy Sienna org/docs/slack references to Sienna-Platform (#68) * Initial plan * Update legacy org, docs domain, and Slack workspace references Agent-Logs-Url: https://github.com/Sienna-Platform/PowerAnalytics.jl/sessions/f7d7ad47-b1c6-439f-a209-13e78403092c Co-authored-by: jd-lara <16385323+jd-lara@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jd-lara <16385323+jd-lara@users.noreply.github.com> * fix(get_data): match slack vars by bare type name (#46) * Narrow make_fuel_dictionary to fuel-bearing component types Replace PSY.StaticInjection with Union{PSY.StaticLoad, PSY.Generator, PSY.Storage} to exclude Source and other non-fuel injectors that have no mapping entry and would otherwise trigger a KeyError. Fixes PowerGraphics.jl#132. * Map all HydroGen subtypes to Hydropower PowerSystems v5 introduced HydroPumpTurbine (primemover=PS), which previously fell through the YAML resolver to the generic Storage rule. Add a HydroGen catch-all rule before the Any/HY rule so any HydroGen subtype is categorized as Hydropower regardless of prime mover. Also adds a regression test covering HydroPumpTurbine and HydroTurbine. Closes PowerGraphics.jl#132. * Extend natural-gas prime-mover mapping Add coverage for prime-mover codes that PSY v5 systems commonly use: - NG-CT now catches both CT (combustion turbine) and GT (gas turbine) - NG-CC now catches both CC (whole-plant combined cycle) and CA (steam side of a decomposed CC train) Changes are purely additive; existing CT thermals still categorize as NG-CT. Adds a positive regression test exercising each new prime-mover code. * Add regression test for Issue #54 (calc_active_power on ComponentSelector) --------- Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: jd-lara <16385323+jd-lara@users.noreply.github.com> Co-authored-by: PabloBotinGP <Pablo.Botin@nrel.gov>

* Update legacy Sienna org/docs/slack references to Sienna-Platform (#68) * Initial plan * Update legacy org, docs domain, and Slack workspace references Agent-Logs-Url: https://github.com/Sienna-Platform/PowerAnalytics.jl/sessions/f7d7ad47-b1c6-439f-a209-13e78403092c Co-authored-by: jd-lara <16385323+jd-lara@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jd-lara <16385323+jd-lara@users.noreply.github.com> * literate cleanup * add precommit * Initial plan * Fix docs tutorial errors: update PSI compat to 0.34, remove deprecated serialize param Agent-Logs-Url: https://github.com/Sienna-Platform/PowerAnalytics.jl/sessions/16e83192-f92d-44d1-a302-8c822df29bd2 Co-authored-by: kdayday <12451220+kdayday@users.noreply.github.com> * Fix Test-CI: update test/test_data/results_data.jl for PSI 0.34 compat Agent-Logs-Url: https://github.com/Sienna-Platform/PowerAnalytics.jl/sessions/3e4e55ac-6c5b-4991-912a-8c755a926575 Co-authored-by: kdayday <12451220+kdayday@users.noreply.github.com> * Fix test_input.jl: use populate_system=true for system comparison in PSI 0.34 Agent-Logs-Url: https://github.com/Sienna-Platform/PowerAnalytics.jl/sessions/c6708aee-b0ef-44a4-ab8d-024e28362712 Co-authored-by: kdayday <12451220+kdayday@users.noreply.github.com> --------- Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: jd-lara <16385323+jd-lara@users.noreply.github.com> Co-authored-by: kdayday <12451220+kdayday@users.noreply.github.com>

Copilot

Pull request overview

This PR updates PowerAnalytics.jl to be compatible with newer PowerSimulations results storage (notably PSI 0.34), while fixing/locking down several result-categorization and generator-mapping behaviors via expanded regression tests and mapping updates.

Changes:

Update results-loading + test harnesses to work with PSI 0.34+ system storage/population behavior.
Fix and extend categorization logic (storage in/out split, slack surfacing) and generator fuel/prime-mover mapping, with new regression tests.
Improve docs/tutorial generation utilities (download links, notebook post-processing) and add repo tooling/config updates.

Reviewed changes

Copilot reviewed 17 out of 20 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
test/test_result_sorting.jl	Updates expectations and adds regression tests for categorization, slacks, storage split, and generator mapping edge cases.
test/test_input.jl	Adjusts scenario-loading tests to use `populate_system=true` and temporarily disables a failing system-compare check.
test/test_data/results_data.jl	Updates test simulation execution/loading to reflect PSI 0.34+ result/system storage behavior.
test/test_builtin_metrics.jl	Adds regression test for Issue #54 selector/metric dispatch.
src/output_utils.jl	Fixes column-count reporting in error messages.
src/get_data.jl	Refactors filtering component-axis selection; enhances `categorize_data` to split storage and improve slack detection.
src/fuel_results.jl	Refactors `make_fuel_dictionary` to categorize generators/storage only and adds unmapped-generator fallback behavior.
src/definitions.jl	Makes several globals `const` and introduces `UNMAPPED_GENERATOR_CATEGORY`.
src/builtin_metrics.jl	Fixes keyword forwarding when calling `read_system_result`.
Project.toml	Expands compat to allow PowerSimulations 0.34.
docs/src/tutorials/_run_scenarios_RTS_Tutorial.jl	Updates `build!` call signature usage.
docs/make.jl	Formatting/cleanup in docs navigation/config.
docs/make_tutorials.jl	Enhances tutorial artifact generation, download links, notebook postprocessing, and DataFrame display bounding.
deps/generator_mapping.yaml	Updates NG prime-mover mapping, HydroGen catch-all, and OTHER-fuel fallback mapping.
.pre-commit-config.yaml	Adds local formatter hook.
.gitignore	Adds ignores for local tooling/config artifacts.
.github/workflows/docs.yml	Sets `DOCUMENTER_CURRENT_VERSION` to support correct tutorial download links across deploy contexts.
.github/workflows/doc-preview-cleanup.yml	Avoids running preview cleanup on forks.
.claude/Sienna.md	Adds/updates repo guidance for Claude tooling (non-code).
.claude/claude.md	Adds/updates repo guide and conventions (non-code).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    bus_numbers_strings = Vector{String}(undef, length(buses))
+    for bus in buses
+        bus_number = PSY.get_number(bus)
+        bus_numbers_strings[bus_number] = string(bus_number)
+    end
+    return bus_numbers_strings


+    split_categories = Set{String}()
+    for (category, list) in aggregation
+        if any(
+            component_type in split_power_component_types for (component_type, _) in list
+        )
+            push!(split_categories, string(category))
+        end
+    end
+
+    # Non-split components: one column per component under the original category.
+    # Split-power components are skipped here and handled by the In/Out pass below.
    for (category, list) in aggregation
        category_df = DataFrames.DataFrame()
-        for (component_type, variable) in list
-            if haskey(var_types, component_type)
-                category_data = data[var_types[component_type]]
-                colname = typeof(names(category_data)[1]) == String ? "$variable" : variable
+        for (component_type, component_name) in list
+            component_type in split_power_component_types && continue
+            haskey(var_types, component_type) || continue
+            category_data = data[var_types[component_type]]
+            colname =
+                if typeof(names(category_data)[1]) == String
+                    "$component_name"
+                else
+                    component_name
+                end
+            DataFrames.insertcols!(
+                category_df,
+                (colname => category_data[:, colname]);
+                makeunique = true,
+            )
+        end
+        if !isempty(category_df)
+            category_dataframes[string(category)] = category_df
+        end
+    end
+
+    # Split pass: discharging (ActivePowerOutVariable) is generation (+),
+    # charging (ActivePowerInVariable) is load (-).
+    for category in split_categories
+        list = aggregation[category]
+        for (suffix, variable_prefix, sign) in (


    filter_func2 = x -> PSY.get_available(x) && filter_func(x)
-    generators = PSY.get_components(filter_func2, PSY.StaticInjection, sys)
-    gen_categories = Dict()
+    gen_categories = Dict{String, Any}()
+
    for category in unique(values(mapping))
        gen_categories["$category"] = []
    end



+        )
    end
+
    [delete!(gen_categories, "$k") for (k, v) in gen_categories if isempty(v)]


+            markdown_image_pattern.pattern * "|" *
+            standalone_img_pattern.pattern * ")",
+        )
+        if occursin(image_fragment_pattern, text)


codecov · 2026-05-19T04:17:35Z

Codecov Report

❌ Patch coverage is 92.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.79%. Comparing base (db2974a) to head (13310a8).

Files with missing lines	Patch %	Lines
src/get_data.jl	90.00%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #73      +/-   ##
==========================================
+ Coverage   88.04%   88.79%   +0.75%     
==========================================
  Files           7        7              
  Lines         669      723      +54     
==========================================
+ Hits          589      642      +53     
- Misses         80       81       +1

Flag	Coverage Δ
unittests	`88.79% <92.00%> (+0.75%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

m-bossart · 2026-05-19T22:49:26Z

-  - {gentype: Any, primemover: CT, fuel: NATURAL_GAS}
+  - {gentype: Any, primemover: GT, fuel: NATURAL_GAS}
 NG-CC:
  - {gentype: Any, primemover: CC, fuel: NATURAL_GAS}


Add for NG-CC: - {gentype: Any, primemover: CS, fuel: NATURAL_GAS}

m-bossart · 2026-05-19T22:57:38Z

    highs_optimizer = optimizer_with_attributes(HiGHS.Optimizer, "mip_rel_gap" => 0.01)

    template_hydro_st_uc =
        ProblemTemplate(NetworkModel(CopperPlatePowerModel; use_slacks = false))


Can we modify this to include a PTDF network model and DC Power Flow in the loop. There are downstream issues that are hit in PowerAnalytics when trying to process a result that has the power flow auxiliary variables.

jd-lara and others added 12 commits March 26, 2026 00:59

clean up for fuel filter

f0e9710

WIP add storage correctly

6cee849

fixes for storage

b0dd8ab

add claude files

e8ae0e4

fix mappings

2b4e87f

code quality issues

8145cb5

fix the categorization

7bf3128

add support for storage and pump hydro

cfdab17

fix testing and logging

7ceaccd

Merge remote-tracking branch 'origin/main' into jd/misc_fixes

02d69d9

jd-lara requested review from Copilot and daniel-thom May 19, 2026 04:04

jd-lara self-assigned this May 19, 2026

jd-lara mentioned this pull request May 19, 2026

Fix slack variable detection in categorize_data #71

Closed

Copilot started reviewing on behalf of jd-lara May 19, 2026 04:06 View session

Copilot AI reviewed May 19, 2026

View reviewed changes

complete and fix mappings

30166e5

jd-lara requested a review from m-bossart May 19, 2026 15:22

Pr review fixes

13310a8

m-bossart requested changes May 19, 2026

View reviewed changes

jd-lara added 3 commits May 20, 2026 08:16

add missing entry

e5e76f6

review fixes

5f4230f

bump PSI version

a746dda

jd-lara merged commit 5c37545 into main May 20, 2026
1 of 5 checks passed

jd-lara deleted the jd/misc_fixes branch May 20, 2026 18:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jd/misc fixes#73

Jd/misc fixes#73
jd-lara merged 17 commits into
mainfrom
jd/misc_fixes

jd-lara commented May 19, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

codecov Bot commented May 19, 2026 •

edited

Loading

Uh oh!

m-bossart May 19, 2026

Uh oh!

m-bossart May 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

jd-lara commented May 19, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

codecov Bot commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

m-bossart May 19, 2026

Choose a reason for hiding this comment

Uh oh!

m-bossart May 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov Bot commented May 19, 2026 •

edited

Loading