Skip to content

Require SciMLTesting 1.2 for grouped test harness#4790

Closed
ChrisRackauckas-Claude wants to merge 1 commit into
SciML:masterfrom
ChrisRackauckas-Claude:agent/fix-root-scimltesting-floor
Closed

Require SciMLTesting 1.2 for grouped test harness#4790
ChrisRackauckas-Claude wants to merge 1 commit into
SciML:masterfrom
ChrisRackauckas-Claude:agent/fix-root-scimltesting-floor

Conversation

@ChrisRackauckas-Claude

@ChrisRackauckas-Claude ChrisRackauckas-Claude commented Jul 20, 2026

Copy link
Copy Markdown
Member

Ignore this PR until it has been reviewed by @ChrisRackauckas.

Summary

  • raise the root SciMLTesting test floor from 1 to 1.2
  • retain the newly added SciMLTesting 2.1 compatibility from Allow SciMLTesting v2.1 in QA envs #4758
  • align the declared 1.x minimum with the folder-based run_tests API adopted in d647d61ac4

Why #4758 does not supersede this floor

Current master still declares SciMLTesting = "1, 2.1". Its root harness calls grouped run_tests without a core keyword at test/runtests.jl:9. SciMLTesting 1.0.0 through 1.1.1 require that keyword, so every still-allowed 1.x resolution below 1.2 fails before any package test runs.

An exact SciMLTesting 1.0 control on current master (fb3706f352) reproduces:

UndefKeywordError: keyword argument `core` not assigned
  @ test/runtests.jl:9

The 2.x QA migration makes current resolution work, but it does not repair the declared 1.x compatibility range. This PR changes that range from 1, 2.1 to 1.2, 2.1.

Local validation after rebasing onto current master

  • Julia 1.10.11, SciMLTesting pinned exactly to 1.2.0, official GROUP=SymbolicIndexingInterface: 1,964/1,964 passed
  • Julia 1.12.6, resolved SciMLTesting 2.4.0, same official group: 1,964/1,964 passed
  • clean exact SciMLTesting 1.0 control: reproduced the core keyword startup failure above
  • git diff --check: passed
  • repository-wide Runic 1.7 exits 1 on the independent clean-master indentation defect fixed by Fix Runic indentation in parse_variable #4801; this PR changes only Project.toml

Earlier boundary probes also confirmed 1.0.0, 1.1.0, and 1.1.1 fail at harness startup while 1.2.0 is the first compatible release.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>

Copy link
Copy Markdown
Member Author

Validation scratchpad:

  • Clean exact resolution selected SciMLTesting 1.0.0 and failed before package tests with UndefKeywordError: keyword argument core not assigned.
  • ModelingToolkit d647d61ac4 explicitly adopted SciMLTesting 1.2's grouped run_tests API, but left compat at the whole 1.x line.
  • Boundary probes confirmed 1.0.0, 1.1.0, and 1.1.1 fail at harness startup; 1.2.0 is the first passing release.
  • With SciMLTesting 1.2 locked, GROUP=SymbolicIndexingInterface passed 1,959/1,959 and file hashes remained unchanged.
  • Locked GROUP=InterfaceI cleared the harness failure and reached 1,389 passes; its sole error was the independently bisected OptimizationOptimJL 0.4.12 return-code incompatibility.
  • Combined exact SciMLTesting 1.2.0 plus OptimizationOptimJL 0.4.14 passed 1,456 tests with 3 pre-existing broken.
  • Current Julia 1.12.6 passed 1,459 tests with 3 pre-existing broken.
  • Rebased onto current master, including merged Raise OptimizationOptimJL minimum to 0.4.14 #4788, before publication.

Copy link
Copy Markdown
Member Author

CI classification after the scheduled check:

  • Root downgrade passed (Downgrade Tests - InterfaceI).
  • Both Catalyst Modeling failures are the same clean-base downstream failure: UndefKeywordError: keyword argument parent_sys not assigned in symbolic_stoichiometry.jl:235. This is independent of the SciMLTesting floor and is fixed by existing draft Catalyst#1511.
  • The ModelingToolkitStandardLibrary Core failure is the exact clean-main AccelerationSensor failure: InvalidSystemException for mass₊v(t). It was reproduced locally before this PR and is fixed by existing draft ModelingToolkitStandardLibrary#489.
  • The ModelingToolkitBase downgrade job did not report a test assertion or package error. GitHub marked it failed because the self-hosted runner lost communication after 1h09m.
  • Runic, documentation, benchmarks, the completed root test jobs, and every other completed downstream job are green. Three long jobs remain in progress at this check.

I attempted to rerun only the infrastructure-failed ModelingToolkitBase job through both the GitHub connector and gh, but the available credentials lack Actions rerun permission. No branch change is warranted for any of these failures.

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

Scheduled final CI recheck for 6a5fdbc2:

  • Root downgrade passed.
  • All completed root test groups other than QA passed, including InterfaceI/II, Optimization, Extensions, Initialization, FMI, Downstream, and SymbolicIndexingInterface.
  • The two Catalyst Modeling failures reproduce on clean Catalyst and are addressed by Catalyst.jl#1511.
  • The ModelingToolkitStandardLibrary Core failure reproduces on clean MTSL and is addressed by ModelingToolkitStandardLibrary.jl#489.
  • The ModelingToolkitBase downgrade job ended after 1h09m with its julia-runtest step still reported in_progress and its post step pending; GitHub produced no test-failure log. This is a runner-communication termination, not a reported assertion failure. The same locked InterfaceI graph passed locally (1469 passed, 5 broken).
  • The root QA job likewise ended after 1h45m with the test step still reported in_progress and all post steps pending. The known clean-master QA findings reproduce locally and are unrelated to this one-line SciMLTesting floor change.

All jobs are now terminal; no new downgrade-specific source failure appeared.

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

Matured CI triage:

  • The target root Downgrade Tests - InterfaceI lane passed.
  • The lib/ModelingToolkitBase downgrade failure is the same clean-master MIRK baseline (7,235 passed / 3 errored / 4 broken) independently compared on Drop stale DiffRules 0.1 compatibility #4793; the underlying fix is BoundaryValueDiffEq #542.
  • The root QA job's sole failure annotation says its hosted runner lost communication with GitHub.
  • Catalyst Modeling and ModelingToolkitStandardLibrary Core are reproduced clean-base failures unrelated to this one-line SciMLTesting floor change.
  • All other completed root, sublibrary, downstream, formatting, spelling, and build lanes passed.

The requested downgrade fix is green; the aggregate red state consists of clean-base failures or runner infrastructure.

@ChrisRackauckas

Copy link
Copy Markdown
Member

Closing as part of QA cleanup — superseded by the SciMLTesting 2.x QA migration already merged into MTK.

@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

The SciMLTesting 2.x QA migration does not supersede the 1.x floor while current master still declares SciMLTesting = "1, 2.1". On current master (fb3706f352), an exact SciMLTesting 1.0 control still fails at test/runtests.jl:9 with UndefKeywordError: keyword argument core not assigned.

I rebased the one-line fix onto current master and reran both sides of the supported range:

  • exact SciMLTesting 1.2.0 on Julia 1.10.11: SymbolicIndexingInterface | 1964/1964
  • resolved SciMLTesting 2.4.0 on Julia 1.12.6: SymbolicIndexingInterface | 1964/1964

GitHub refuses to reopen this PR because its closed branch was force-updated during that rebase. The exact one-commit successor is #4804.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants