Bc cvar updates by bcakire · Pull Request #107 · ReEDS-Model/ReEDS

bcakire · 2026-05-26T18:30:14Z

Summary

This PR adds CVAR/NCVAR reporting and target checks to the PRAS resource adequacy workflow.

Main idea is:

PRAS can write CVAR/NCVAR-related outputs.
Python stress-period workflow can read those outputs.
CVAR/NCVAR can be checked against thresholds.
For now, CVAR/NCVAR are check-only metrics. They do not add stress periods and do not update PRM.

What changed

`run_pras.jl`

Added:

--cvar_alpha
--write_shortfall_samples_totals

When PRAS.ShortfallSamples() is available, run_pras.jl now writes:

PRAS_{t}i{iteration}-risk_metrics.csv
PRAS_{t}i{iteration}-shortfall_totals_by_sample.h5

risk_metrics.csv includes:

CVAR in MWh
NCVAR in ppm
alpha
estimate
standard error
VaR

shortfall_totals_by_sample.h5 includes total shortfall by sample for USA and regions. This is total over the full PRAS time period, not annual.

`stress_periods.py`

Added CVAR/NCVAR handling with:

get_cvar_alpha()
get_shortfall_totals_by_sample()
_sample_cvar()
get_annual_cvar_stress_metric()
evaluate_cvar_target_check()

CVAR/NCVAR are calculated by hierarchy level.

NCVAR is normalized by total PRAS load and reported in ppm.

CVAR/NCVAR are intentionally kept out of the normal stress-period selection loop for now. They are only used for reporting and target checks.

`ra_calcs.py`

Added logic so write_shortfall_samples_totals is turned on automatically when CVAR/NCVAR checks are requested.

Also keeps it on for PRAS-informed PRM updates, since those need sample-level shortfall totals too.

`cases.csv`

Added/updated these switches:

GSw_PRM_CVARAlpha
GSw_PRM_StressThresholdNCVAR
GSw_PRM_StressThresholdCVAR

The standard metric switch format follows aa/multi_metrics, for example:

GSw_PRM_StressThresholdNEUE = transgrp_1_sum
GSw_PRM_StressThresholdLOLH = transgrp_2.4_sum

<!--
Include comparison reports (e.g., .pptx files generated by compare_cases.py) for cases commensurate with the level of code/data changes:
- No changes to model or data: Pacific test case only
-->

<!--
Include additional illustrative plots describing input data, methods, testing, and/or key differences from the comparison report(s)
-->


## Checklist for author
<!-- Fill out before requesting review -->

### Details to double-check
<!-- Delete or ~~strikethrough~~ if not relevant for this PR -->
- [X] Charge code provided to reviewers
- [ ] Included comparison reports for appropriate test cases
- [ ] Documentation updated if necessary
  <!--
  - model_documentation.md: High-level description of default model behavior
  - user_guide.md: User/developer-facing description of model switches and input files
  - faq.md: Limitations, caveats, and known issues
  - postprocessing_tools.md: User/developer-facing description of scripts in postprocessing folder
  - README.md files: User/developer facing description of individual input folders
  -->
- If input data added/modified:
  - [ ] Dollar year recorded and converted to 2004$ for GAMS
  - [ ] Timeseries are in Central Time
  - [ ] Units are specified
  - [ ] Preprocessing steps have been documented and committed to [ReEDS_Input_Processing](https://github.com/ReEDS-Model/ReEDS_Input_Processing)
  - [ ] New large data files handled with .h5 instead of .csv
  - [ ] If spatially resolved inputs are modified, the following visualizations for each file are included in the PR description (time-averaged if the inputs are time-resolved):
    - [ ] Map of absolute values before
    - [ ] Map of absolute values after
    - [ ] Map of differences: (after - before) or (after / before)
  - If entries are added/removed/changed in the EIA-NEMS unit database:
    - [ ] Changes have been committed to [ReEDS_Input_Processing](https://github.com/ReEDS-Model/ReEDS_Input_Processing)
    - [ ] `hourlize/resource.py` was rerun to regenerate the existing/prescribed VRE capacity data
- [X] Code formatting standardized <!-- Coding conventions: https://reeds-model.github.io/ReEDS/developer_best_practices.html#coding-standards-and-conventions -->
- [X] Reusable functions used where possible instead of copy/pasted code

### General information to guide review
<!-- These do not need to be checked to merge, but if unchecked, a deeper level of review may be required -->
- [X ] Zero impact on results of default case
- [X ] No large data file(s) added/modified
- [X ] No substantive impact on runtime for full-US reference case
- [X] No substantive impact on folder size for full-US reference case
- [X] No change to process flow (runreeds.py, reeds/core/solve/solve.py)
- [ ] No change to code organization
- [X] No change to package requirements (environment.yml or Project.toml)

#### Did you use LLM tools (chatbot or copilot) in the preparation of this PR? If so, describe how

### Tag points of contact here if you would like additional review of the relevant parts of the model
<!-- Model dimensions -->
<!-- - [ ] County resolution: @louisaserpe -->
<!-- - [ ] Demand data: @ahamilton5 -->
<!-- - [ ] Emissions: @atpham88 -->
<!-- - [ ] Financial calculations: @wesleyjcole -->
<!-- - [ ] FINITO: @merveturan or @cavraam -->
<!-- - [ ] Hybrids: @aschleif -->
<!-- - [ ] Monte Carlo: @bsergi -->
<!-- - [ ] Sparse chronology or interday diurnal storage: @Yunzhi-Chen -->
<!-- - [ ] State policies: @wesleyjcole or @aschleif -->
<!-- - [ ] Stress periods, resource adequacy, or ReEDS2PRAS/Julia: @patrickbrown4 -->
<!-- - [ ] Supply curves, hourlize, reeds_to_rev: @bsergi -->
<!-- - [ ] Time resolution: @patrickbrown4 -->
<!-- - [ ] Water cooling: @jcarag or @stuartcohen8 -->

<!-- Pre/Postprocessing -->
<!-- - [ ] Dispatch mode (run_pcm.py): @patrickbrown4 -->
<!-- - [ ] Github actions: @kennedy-mindermann -->
<!-- - [ ] Health impacts: @bsergi -->
<!-- - [ ] Input data structure and processing: @kodiobika -->
<!-- - [ ] Interactive plots (bokeh, vizit): @mmowers -->
<!-- - [ ] Land use: @bsergi -->
<!-- - [ ] R2X: @pesap -->
<!-- - [ ] Retail rates: @wesleyjcole -->
<!-- - [ ] Static plots (single_case_plots.py, compare_cases.py): @patrickbrown4 -->
<!-- - [ ] Tableau: @ahamilton5 -->
<!-- - [ ] Technology value (reValue, valuestreams.py): @mmowers -->

<!-- Technologies -->
<!-- - [ ] Batteries: @wesleyjcole -->
<!-- - [ ] EV managed charging (EVMC): @Max-Vanatta -->
<!-- - [ ] Fossil, CCS, or DAC: @mbrown1 -->
<!-- - [ ] Geothermal: @shashwatsharma24 -->
<!-- - [ ] Hydropower or PSH: @stuartcohen8 -->
<!-- - [ ] Hydrogen: @bsergi -->
<!-- - [ ] Nuclear: @wesleyjcole -->
<!-- - [ ] Solar: @wesleyjcole -->
<!-- - [ ] Transmission: @patrickbrown4 -->
<!-- - [ ] Wind: @mmowers -->

bsergi

Here are some initial thoughts from a quick look. Happy to touch base when you get back to discuss any of these. Let me know when you've taken another pass here and I can review again.

One high-level comment: can you organize the PR summary text a bit more? The technical details section has a lot of information but is a bit difficult to follow. Also, I didn't quite understand your test output GSw_PRM_StressThreshold = country_-1_NCVAR_cvar failed. What does it mean to have a -1 threshold? Ultimately for the PR we'll also want a comparison of default case from cases.csv from both this branch and main to show there weren't any changes.

bsergi · 2026-06-18T14:54:30Z

revert the changes to this file before merging

bsergi · 2026-06-18T14:56:39Z

            required = false
        "--write_shortfall_samples"
-            help = "Write the sample-level shortfall"
+            help = "Write the sample-level shortfall (hourly, large)"


What are the large and small designations? I might suggest just "Write per-sample hourly shortfall by region" for this one and "Write per-sample total shortfall by region" for the one below.

bsergi · 2026-06-18T14:57:02Z

            default = 0
            required = false
+        "--write_shortfall_samples_totals"
+            help = "Write per-sample total shortfall by region (small)"


this is total over the entire PRAS time period and not annual, right?

bsergi · 2026-06-18T14:58:35Z

+        _,_,_,_,energyunit = PRAS.get_params(sys)
+        alpha = Float64(args["cvar_alpha"])
+        cvar_obj = PRAS.CVAR(energyunit, results["short_samples"], alpha)
+        @info(cvar_obj)


I think PRAS is setup so that the above info statements provide context (e.g., @info "$(PRAS.EUE(results["short"]))" results in EUE = 851000±5000 MWh/131400h MWh) being printed). Does this do something similar?

bsergi · 2026-06-18T15:47:19Z

-        end
-        ## Write it
+        sf = results["short_samples"]
+        region_names = sf.regions.names


the previous code filtered using sys.regions.names. I think this was because there are some pseudo regions for DC converter stations that don't have load and thus don't need to be included here, so we may want to continue to drop them (@patrickbrown4 might be able to weigh in on this one).

bsergi · 2026-06-18T16:27:28Z

+    return x.sort_values(ascending=False).iloc[:n_tail].mean()
+
+
+def get_annual_cvar_stress_metric(case, t, stress_metric='NCVAR', iteration=0, alpha=0.95):


this function and the next one get pretty long, so it would be good to add comments throughout to provide some documentation on what they are doing.

bsergi · 2026-06-18T16:31:40Z

+    x = pd.Series(samples).dropna().astype(float)
+    if x.empty:
+        return np.nan
+    if not (0 <= alpha < 1):


Feels like this check might fit better in get_cvar_alpha. An even better approach might be to add a check before the ReEDS run which would avoid having runs get this far and then running into an error (best place for that would probably be here:

ReEDS/runreeds.py

Line 222 in 1a8ab63

def check_compatibility(sw):

)

bsergi · 2026-06-18T16:39:35Z

+
+def get_annual_cvar_stress_metric(case, t, stress_metric='NCVAR', iteration=0, alpha=0.95):
+    stress_metric = stress_metric.upper()
+    if stress_metric not in CVAR_METRICS:


The multi-metrics PR (#99) now drops the metric tag in the switch value; for example, GSw_PRM_StressThresholdNEUE now just has a value of transgrp_1_sum instead of transgrp_1_EUE_sum (see here). I think we could do the same in this PR and then you could drop the CVAR_METRICS and the check and just iterate over whichever switches are activated.

bsergi · 2026-06-18T16:48:21Z

-                                                                   )
+        for criterion in sw[f'GSw_PRM_StressThreshold{metric}'].split('/'):
+            print(f"Evaluating GSw_PRM_StressThreshold {metric} with criterion: {criterion}")
+            stress_criteria = _evaluate_stress_threshold_criterion(stress_criteria, criterion, sw, t, iteration, dfenergy_r, stressperiods_this_iteration,


This line looks like just a formatting change--I'd suggest keeping as the original had it.

bsergi · 2026-06-18T16:53:02Z

+    return pd.concat(_metric, names=['level','metric','region']).rename(stress_metric)
+
+
+def evaluate_cvar_target_check(sw, t, iteration=0, stress_metrics=None):


Much of this function, including the core check of failed = this_test.loc[this_test > threshold], seems to replicate behavior in _evaluate_stress_threshold_criterion. Do you think we could merge the CVAR treatment into that existing function and just have some control statements to turn off adding new stress periods for now when using CVAR metrics? It's possible having a second function makes the most sense here, but I worry a little bit about maintaining two similar functions.

Burcin Cakir Erdener and others added 3 commits May 20, 2026 05:38

add CVaR reporting to run_pras.jl

29b6be0

fix fix

33d2a02

Add CVaR target checks for stress periods

e0ae25e

bcakire marked this pull request as draft May 28, 2026 06:19

merged with aa/multimetrics/06_02_26

9bc6424

patrickbrown4 self-requested a review June 2, 2026 23:50

bsergi reviewed Jun 18, 2026

View reviewed changes

Burcin Cakir Erdener and others added 3 commits June 25, 2026 16:20

Merge aa/multi_metrics and integrate CVAR/NCVAR checks

6aec186

Revert unrelated PR changes

65fd43a

Address CVAR/NCVAR review comments

e41316f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Bc cvar updates#107

Bc cvar updates#107
bcakire wants to merge 7 commits into
ReEDS-Model:aa/multi_metricsfrom
bcakire:bc_cvar_updates

bcakire commented May 26, 2026 •

edited

Loading

Uh oh!

bsergi left a comment •

edited

Loading

Uh oh!

bsergi Jun 18, 2026

Uh oh!

bsergi Jun 18, 2026

Uh oh!

bsergi Jun 18, 2026

Uh oh!

bsergi Jun 18, 2026

Uh oh!

bsergi Jun 18, 2026

Uh oh!

bsergi Jun 18, 2026

Uh oh!

bsergi Jun 18, 2026

Uh oh!

bsergi Jun 18, 2026

Uh oh!

bsergi Jun 18, 2026

Uh oh!

bsergi Jun 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		return x.sort_values(ascending=False).iloc[:n_tail].mean()


		def get_annual_cvar_stress_metric(case, t, stress_metric='NCVAR', iteration=0, alpha=0.95):

		return pd.concat(_metric, names=['level','metric','region']).rename(stress_metric)


		def evaluate_cvar_target_check(sw, t, iteration=0, stress_metrics=None):

Uh oh!

Conversation

bcakire commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changed

run_pras.jl

stress_periods.py

ra_calcs.py

cases.csv

Uh oh!

bsergi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bcakire commented May 26, 2026 •

edited

Loading

`run_pras.jl`

`stress_periods.py`

`ra_calcs.py`

`cases.csv`

bsergi left a comment •

edited

Loading