feat: add sensitivity analysis - Morris screening and Saltelli Sobol indices#20
Merged
Conversation
New crate for parameter sensitivity analysis (Morris/Sobol) over hardware model parameter space. Workspace member registered, module stubs in place, SensitivityError enum covers parameter validation, hardware/engine/monte-carlo propagation, and method-specific checks.
…ping ParameterDef/ParameterKind/ParameterSpace types define the sweepable parameter dimensions. Validated construction rejects empty spaces, duplicates, invalid ranges, and kind-specific bound violations. map_unit_to_physical supports continuous, integer, and uniform odd-integer discrete mappings. validate_against_hw checks factory and routing type compatibility against KNOWN_PARAMS registry. 17 integration tests cover mapping correctness, clamping, and all rejection paths.
Bridge between unit-hypercube sample points and concrete HardwareModel instances. mutate_hw applies a single override, mutate_hw_multi applies all parameters from a ParameterSpace, both validate after mutation.
Add a public wrapper around the private extract_summary function so pirx-sensitivity can extract ReplicaSummary from a single engine run without going through the full Monte Carlo pipeline.
OutputMetric enum maps ReplicaSummary fields to f64 for sensitivity analysis. evaluate_point is the workhorse: unit-hypercube → physical params → HW mutation → engine run → metric extraction, with deterministic seed partitioning across points and single-run / MC modes.
SensitivityConfig with SweepConfig, MorrisConfig, and SobolConfig structs. Parses metric, mc_replicas, base_seed, parameter definitions with kind validation, and optional method-specific sections with serde defaults for confidence and bootstrap_resamples.
…and trajectory generation Add MorrisConfig::validate(), MorrisResult/MorrisParameterResult types, delta_value() step-size function, and generate_trajectories() with adaptive sign selection to guarantee bounds without clamping.
Implement evaluate_trajectories (parallel via rayon), extract_elementary_effects, aggregate_morris (Bessel-corrected σ), and morris_screening entry point. Eight synthetic-function tests verify EE correctness, ranking, sign invariants, and Bessel's correction without requiring engine runs.
Integration tests run full morris_screening through the real engine with pirx-testkit circuits. Property tests (proptest, 30 cases) verify triangle inequality, sigma non-negativity, and evaluation count invariants.
Ignored-by-default test comparing raw engine runs against evaluate_point to verify parameter mapping and HW mutation overhead stays under 1%.
Wire Morris screening into the CLI as `pirx sensitivity morris` with sweep TOML config, human-readable stderr summary (bar chart, category classification, Sobol focus suggestions), and JSON output. Export SensitivityConfig and parse_sensitivity_config from pirx-sensitivity for CLI consumption.
Joe-Kuo (2010) direction numbers for dimensions 2..31, Gray code construction with van der Corput for dimension 1. Returns fallible Result for non-power-of-2 N or out-of-range dimensions.
…uation SobolConfig validation, A/B/AB_i matrix construction from Sobol sequences, and batched parallel point evaluation via rayon for variance-based sensitivity analysis.
Implement variance-based index estimation (S₁, Sₜ) via the Jansen estimator, fix Saltelli matrix construction to use column-split for independent A/B matrices, and add synthetic-function validation tests.
…lidation Paired resampling preserves Saltelli correlation structure. Bootstrap RNG seeded deterministically from base_seed for reproducibility independent of evaluation parallelism. Ishigami test validates full pipeline against analytical indices at N=4096.
Wire up the full Sobol variance-based sensitivity pipeline: public sobol_analysis() orchestrating Saltelli matrices, parallel evaluation, Jansen indices, and bootstrap CIs. Add `pirx sensitivity sobol` CLI subcommand with human-readable summary (ranked parameters, S1/ST with CIs, interaction labels, additivity check). Integration tests validate dominance ordering, determinism, evaluation count, and zero-variance edge case. Property tests cover ST >= S1, index bounds, and CI ordering.
…laky test - Add #![forbid(unsafe_code)] to pirx-sensitivity crate - Remove unnecessary #![allow(dead_code)] from morris module - Add InsufficientSamples error variant for n_samples < 64 (was misreporting as NotPowerOfTwo for valid powers of two) - Fix SobolConfig confidence validation to reject 0.0 and NaN - Widen Sobol property test tolerance for N=128 Jansen estimator noise - Add missing test for negative parameter bound rejection
…I surface Dual-strategy Saltelli matrix construction: column-split for dim≤15 (quasi-independent via Sobol cross-dimension), LHS-based row-split for dim 16–30 (Latin Hypercube provides stratified uniformity with true independence from A matrix, required by Jansen estimator). - Make MAX_DIM pub(crate), add TooManyParameters error variant - Early rejection for dim > 30 in sobol_analysis entry point - Change validate() to take &self (Copy types, no consumption needed) - Add #[must_use] to SobolResult, MorrisResult and ranking methods - Tighten property test tolerances with N=1024 for stable estimates - Add integration test for 16-parameter row-split with determinism check
…clean up Morris grid Restrict sobol_sequence and KNOWN_PARAMS to pub(crate) — these are implementation details not needed by consumers. Add dimension check to map_point() returning DimensionMismatch instead of silently misaligning. Replace float-accumulation loop in Morris trajectory generation with integer-indexed grid construction. Correct Sobol docstring to attribute S₁ estimator to Saltelli (2010) and Sₜ to Jansen (1999).
…ests Test that map_point rejects wrong-length input with DimensionMismatch. Add unit test asserting every KNOWN_PARAMS entry is handled by apply_single — catches drift between the parameter list and mutate logic.
Add pirx-sensitivity entries to CHANGELOG under [Unreleased]: Morris screening, Sobol analysis, parameter space, sweep config, CLI subcommands, and trace_summary public API. Update README to move sensitivity analysis from planned to working, add the crate to the architecture listing, and bump crate count from six to seven.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
New crate
pirx-sensitivityimplementing global sensitivity analysis: parameter space definition, hardware model mutation, Morris elementary effects screening, and Saltelli's algorithm for first-order and total-order Sobol indices with bootstrap confidence intervals.Why
Researchers cannot answer "which hardware parameter should I optimize?" without systematic sensitivity analysis. Aggregate resource estimators give point estimates; Pirx gives temporal profiles — but without quantifying which parameters drive that profile, users are guessing. Morris screening identifies the 3-4 parameters that matter in 30 seconds. Sobol indices then decompose output variance into per-parameter contributions with confidence intervals, revealing both direct effects (S₁) and parameter interactions (Sₜ − S₁).
How
Crate architecture.
pirx-sensitivitydepends onpirx-core,pirx-hw,pirx-ir. Nothing depends on it. The crate wraps the engine — it never touches DES internals, only callsEngine::new/runthroughevaluate_point.Parameter space.
ParameterSpacemaps a unit hypercube [0,1]^P to physical hardware parameters. Three kinds:Continuous(linear interpolation),Integer(nearest),OddInteger(uniform mapping to odd integer set — no rounding bias). 20 known sweepable parameters covering all mutableHardwareModelfields. Mutation viaapply_single(no validation) +mutate_hw_multi(validate once after all overrides).Evaluation.
evaluate_pointmaps unit point → physical params → mutated HW → engine run → metric extraction. Supports single-run and Monte Carlo modes (configurablemc_replicasfor noise reduction on stochastic models). Seed space is partitioned: point N uses seeds[base + N×replicas, base + (N+1)×replicas).Morris screening. Campolongo (2007) OAT trajectories with restricted base points (
[Δ, 1−Δ]per dimension — no boundary clamping artifacts).levelsvalidated as even ≥ 4 for grid alignment. Elementary effects extracted from parallel-evaluated trajectory points. Aggregation: μ, μ* (primary importance), σ (nonlinearity/interaction indicator) with Bessel-corrected sample std.Sobol indices. Joe-Kuo (2010) Sobol sequence generator (~150 lines, direction numbers for 30 dimensions). Saltelli matrix construction: A, B, P×AB_i from 2N quasi-random points. Jansen estimator for S₁ and Sₜ. All N×(P+2) evaluations parallelized via rayon. Allocation-free variance computation. Bootstrap CIs: paired resampling with deterministic RNG (reproducible independent of thread scheduling).
pirx-core change.
extract_summaryexposed aspub fn trace_summaryfor single-run metric extraction without Monte Carlo overhead.CLI.
pirx sensitivity morrisandpirx sensitivity sobolsubcommands. TOML sweep configuration. Human-readable summary to stderr with category annotations.Testing
make cipasses locally (fmt + clippy + test + audit)Checklist
unwrap()/expect()in production codepirx-sensitivitydepends onpirx-core/pirx-hw/pirx-ir, nothing depends on it