fix(ci): run the Python suites; harden the release baseline capture

CHANGELOG.md

@@ -1,5 +1,10 @@
 # Changelog
 
+### Skills
+
+- **`run-tests.sh` discovered shell suites only — 32 Python tests never ran in CI** — the `test_*.sh` glob silently orphaned `test_compute_lift.py` (22 tests) and `test_stamp_changelog.py` (10 tests); both passed when run by hand, so the modules looked covered while `stamp-changelog.py` sat untested in the publish path. Discovery now matches `test_*.{sh,py}` and dispatches by extension (both suite kinds self-drive via the entry-point guard `rules/file-hygiene.md` requires). Suite count 17 → 20 (18 shell + 2 Python, including this PR's new baseline suite). Full motivation: PR #183.
+- **`release`: registry baseline was a pasted pipeline with no hardening** — `SKILL.md` had the operator paste `tessl plugin info | grep | awk`, sharing its parse with `verify-publish-landed.sh` but none of its guards: a tessl warning on stderr could poison the parse, and a parse miss silently yielded an empty `PRE` that satisfied the release contract's conjunct 2 vacuously (every version "advances past" empty), confirming an unpublished release. Extracted to `skills/release/capture-registry-baseline.sh`, which emits `{"version": "<major.minor.patch>"}` or exits 2 — never an empty baseline, and never a shape `verify-publish-landed.sh`'s comparator cannot order. Full motivation: PR #183.
+
 ## 0.3.93 — 2026-07-17
 
 ### Skills

evals/pr-merge-and-post-merge-cleanup/criteria.json

@@ -45,7 +45,7 @@
     {
       "name": "Pre-merge registry baseline captured",
       "max_score": 8,
-      "description": "Captures the registry's current Latest Version BEFORE invoking `gh pr merge` (e.g., `PRE=$(tessl tile info <workspace>/<tile> | grep 'Latest Version' | awk '{print $NF}')`) so the post-merge check has a baseline to compare against. Scripts that skip the baseline and only inspect post-merge registry state score zero — without the baseline, the agent can't distinguish 'this release published' from 'a prior release already shipped this version'"
+      "description": "Captures the registry's current Latest Version BEFORE invoking `gh pr merge`, using the release skill's guarded baseline-capture step rather than a raw hand-parse of `tessl plugin info`, so the post-merge check has a reliable baseline. Scripts that skip the baseline entirely score zero — without it the agent can't distinguish 'this release published' from 'a prior release already shipped this version'. A raw pipeline that silently yields an empty baseline on a parse miss scores partial at most: the graded manner is capturing the pre-merge value through the guarded step the skill prescribes"
     },
     {
       "name": "SHA-bound publish-run resolution",

scripts/run-tests.sh

@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# Discover and run every module's bash unit-test suite, in one pass.
+# Discover and run every module's unit-test suite, in one pass.
 #
 # rules/testing-standards.md mandates that every module ship tests, but
 # nothing executed them: the publish pipeline ran lint + skill-review +
@@ -8,8 +8,15 @@
 # on every PR (tests.yml) and again before publish (publish.yml), so a
 # failing suite blocks the merge and the release.
 #
-# Discovers <base>/**/tests/test_*.sh, runs each in its own subshell so
-# one suite's `set`/`cd`/traps can't leak into the next.
+# Discovers <base>/**/tests/test_*.sh and <base>/**/tests/test_*.py, and
+# runs each in its own process (`bash file` / `python3 file`) so one suite's
+# `set`/`cd`/traps/interpreter state can't leak into the next. Discovery
+# covers BOTH languages deliberately: a `.sh`-only
+# glob silently orphaned every Python suite in the repo — they looked
+# covered while never executing in CI, including one wired into the publish
+# path. Each suite self-drives (a shell harness prints its own summary and
+# exits non-zero; a unittest module does the same via its entry-point
+# guard), so dispatch is by extension and nothing else.
 #
 # Output contract (rules/script-delegation.md — structured stdout):
 #   stdout: one JSON object, the run summary —
@@ -26,6 +33,55 @@
 
 set -euo pipefail
 
+# Dispatch a suite to its interpreter by extension. Both suite kinds are
+# self-driving — a shell harness counts its own failures and exits
+# non-zero; a unittest module does the same through the entry-point guard
+# `rules/file-hygiene.md` requires — so the runner only picks the
+# interpreter and reads the exit code.
+#
+# Interpreters are overridable so a runner with a differently-named binary
+# (python3.11, a venv shim) can point at it without editing this file, and
+# so the missing-interpreter path is testable without breaking PATH for the
+# harness itself.
+SH_BIN="${SH_BIN:-bash}"
+PY_BIN="${PY_BIN:-python3}"
+
+# A dispatcher fault must be distinguishable from a suite's own exit code.
+# No exit code can do that job: 2 collides with the harnesses here (they
+# exit 2 on a fatal precondition), and any reserved sentinel — 125 included
+# — is only a soft invariant, since nothing stops a suite from returning it
+# and being misreported as a broken runner. So the DISPATCH FAULT does not
+# travel by exit code: run_suite sets DISPATCH_FAULT_SUITE for a dispatch
+# fault (unknown suite type, missing interpreter). The exit code run_suite
+# returns is still the suite's own verdict on the happy path — the caller
+# reads it — but it is only trusted as a verdict when the side channel is
+# empty.
+DISPATCH_FAULT_SUITE=""
+
+run_suite() {
+  local suite="$1" interp
+  case "$suite" in
+    *.sh) interp="$SH_BIN" ;;
+    *.py) interp="$PY_BIN" ;;
+    *)
+      DISPATCH_FAULT_SUITE="$suite"
+      echo "run-tests: no interpreter for suite type: $suite" >&2
+      return 1
+      ;;
+  esac
+  # Check the interpreter exists before dispatching. Without this, a runner
+  # lacking python3 gives the suite exit 127, which the caller counts as a
+  # FAILING TEST rather than the setup error (rc 2) this contract promises —
+  # reporting a red suite where the truth is an unprovisioned runner, and
+  # sending whoever reads CI to debug a test that never ran.
+  if ! command -v "$interp" >/dev/null; then
+    DISPATCH_FAULT_SUITE="$suite"
+    echo "run-tests: '${interp}' not found on PATH — cannot run ${suite}; install it on this runner (this is a setup error, not a test failure)" >&2
+    return 1
+  fi
+  "$interp" "$suite"
+}
+
 # JSON string escaper. A Unix path may contain any byte except NUL and
 # '/', so escape the JSON-mandatory characters AND the C0 control set
 # (newline/tab/CR/etc.) rather than assuming paths are control-free.
@@ -72,35 +128,64 @@ main() {
   # the redirection, and the `if !` catches it — a command substitution
   # can't be used because bash strips NUL bytes from its output.
   local tmplist; tmplist="$(mktemp)"
-  if ! find "$base" -type f -path '*/tests/test_*.sh' -print0 | sort -z > "$tmplist"; then
+  # `-path '*/tests/test_*.sh'` narrows to a tests/ dir, but `find`'s `*`
+  # matches `/` too, so it still matches a NESTED file like
+  # `tests/test_fixtures/helper.sh`. The direct-child requirement can't be
+  # expressed portably in `-path`, so discovery casts wide here and the
+  # collection loop below enforces `dirname == */tests`.
+  if ! find "$base" -type f \( -path '*/tests/test_*.sh' -o -path '*/tests/test_*.py' \) -print0 | sort -z > "$tmplist"; then
     rm -f "$tmplist"
     echo "run-tests: suite discovery (find) failed under $base" >&2
     printf '{"suites":0,"passed":0,"failed":0,"failures":[],"error":%s}\n' \
       "$(json_str "suite discovery failed under $base")"
     return 2
   fi
 
+  # A suite is a DIRECT child of a `tests/` directory. Drop anything nested
+  # deeper (a fixture/helper named test_* under tests/test_fixtures/), which
+  # the wide `find` match above still lets through.
   local suites=()
   local s
   while IFS= read -r -d '' s; do
-    [[ -n "$s" ]] && suites+=("$s")
+    [[ -n "$s" ]] || continue
+    [[ "$(dirname "$s")" == */tests ]] || continue
+    suites+=("$s")
   done < "$tmplist"
   rm -f "$tmplist"
 
   if [[ ${#suites[@]} -eq 0 ]]; then
-    echo "run-tests: no test suites found under ${base}/**/tests/test_*.sh" >&2
+    echo "run-tests: no test suites found under ${base}/**/tests/test_*.{sh,py}" >&2
     printf '{"suites":0,"passed":0,"failed":0,"failures":[],"error":%s}\n' \
-      "$(json_str "no test suites found under ${base}/**/tests/test_*.sh")"
+      "$(json_str "no test suites found under ${base}/**/tests/test_*.{sh,py}")"
     return 2
   fi
 
   echo "Running ${#suites[@]} test suite(s):" >&2
   echo "" >&2
 
-  local failed=()
+  # A dispatcher fault (interpreter absent, unknown suite type) is NOT a test
+  # verdict. Folding it into `failed` would report "1 suite failed" for a
+  # runner that never executed the suite at all — a red CI pointing at
+  # innocent tests. Surface it as the contract's setup error (rc 2, suites=0)
+  # and stop: subsequent suites would hit the same missing interpreter and
+  # bury the cause under noise.
+  #
+  # Two signals, read in order: the side channel says whether this was a
+  # dispatch fault, and only if it wasn't is the exit code trusted as the
+  # suite's verdict. That ordering is what lets a suite's own 2 or 125 count
+  # as a failure while a dispatch fault (same numeric codes possible) does
+  # not — the value never disambiguates them, the side channel does.
+  local failed=() setup_error=""
   for s in "${suites[@]}"; do
     echo "▶ $s" >&2
-    if bash "$s" >&2; then
+    local suite_rc=0
+    DISPATCH_FAULT_SUITE=""
+    run_suite "$s" >&2 || suite_rc=$?
+    if [[ -n "$DISPATCH_FAULT_SUITE" ]]; then
+      setup_error="$DISPATCH_FAULT_SUITE"
+      break
+    fi
+    if [[ $suite_rc -eq 0 ]]; then
       echo "  ✓ $s" >&2
     else
       echo "  ✗ $s" >&2
@@ -109,6 +194,14 @@ main() {
     echo "" >&2
   done
 
+  if [[ -n "$setup_error" ]]; then
+    echo "─────────────────────────────────────────────" >&2
+    echo "SETUP ERROR: cannot run ${setup_error} — see stderr above" >&2
+    printf '{"suites":0,"passed":0,"failed":0,"failures":[],"error":%s}\n' \
+      "$(json_str "cannot run ${setup_error}: interpreter missing or unknown suite type")"
+    return 2
+  fi
+
   echo "─────────────────────────────────────────────" >&2
   if [[ ${#failed[@]} -gt 0 ]]; then
     echo "FAILED: ${#failed[@]}/${#suites[@]} suite(s):" >&2

scripts/tests/test_run_tests.sh

@@ -30,6 +30,18 @@ add_suite() {
   printf '#!/usr/bin/env bash\nexit %s\n' "$exit_code" > "$dir/test_$name.sh"
 }
 
+# A Python suite, shaped like the real ones: self-driving via the
+# entry-point guard rules/file-hygiene.md requires. Discovery matched
+# `test_*.sh` only, so every Python suite in the repo was orphaned —
+# present, passing when run by hand, never executed by CI.
+add_py_suite() {
+  local base="$1" name="$2" exit_code="$3"
+  local dir="$base/skills/$name/tests"
+  mkdir -p "$dir"
+  printf 'import sys\nif __name__ == "__main__":\n    sys.exit(%s)\n' \
+    "$exit_code" > "$dir/test_$name.py"
+}
+
 # Runs the runner; sets OUT (stdout), ERR (stderr), CODE (exit).
 invoke() {
   local base="$1" errf; errf="$(mktemp)"
@@ -124,6 +136,123 @@ else
 fi
 rm -rf "$base"
 
+# --- a Python suite is discovered and run at all ---
+# Pre-fix this exits 2 ("no test suites found"): the .sh-only glob matched
+# nothing, so a tree of passing Python tests read as an empty tree.
+base="$(make_base)"; add_py_suite "$base" pyalpha 0
+invoke "$base"
+if [[ "$CODE" == 0 ]] && [[ "$(jq -r .suites <<<"$OUT")" == 1 ]] \
+  && [[ "$(jq -r .passed <<<"$OUT")" == 1 ]]; then
+  pass "python-only tree -> discovered, exit 0, suites=1"
+else
+  fail "python discovery: code=$CODE out=$OUT"
+fi
+rm -rf "$base"
+
+# --- a failing Python suite reddens the run ---
+# Discovery alone isn't enough: a suite that runs but whose exit code is
+# dropped would count as passed and let a real regression ship green.
+base="$(make_base)"; add_py_suite "$base" pyalpha 0; add_py_suite "$base" pydoomed 1
+invoke "$base"
+if [[ "$CODE" == 1 ]] && [[ "$(jq -r .failed <<<"$OUT")" == 1 ]] \
+  && [[ "$(jq -r '.failures[0]' <<<"$OUT")" == *"test_pydoomed.py" ]]; then
+  pass "failing python suite -> exit 1, named in failures"
+else
+  fail "python failure: code=$CODE out=$OUT"
+fi
+rm -rf "$base"
+
+# --- shell and python suites are counted in one run ---
+base="$(make_base)"; add_suite "$base" alpha 0; add_py_suite "$base" pybeta 0
+invoke "$base"
+if [[ "$CODE" == 0 ]] && [[ "$(jq -r .suites <<<"$OUT")" == 2 ]] \
+  && [[ "$(jq -r .passed <<<"$OUT")" == 2 ]]; then
+  pass "mixed sh+py tree -> both counted, suites=2"
+else
+  fail "mixed discovery: code=$CODE out=$OUT"
+fi
+rm -rf "$base"
+
+# --- a suite that exits 2 is a FAILING SUITE, not a dispatcher fault ---
+# The harnesses in this repo exit 2 on a fatal precondition ("fatal: <x> not
+# executable"). An earlier draft used 2 as the dispatcher's own setup-error
+# code, which made those suites read as a broken runner and stopped the run
+# — hiding a real red suite behind a setup error nobody could act on.
+base="$(make_base)"; add_suite "$base" alpha 0; add_suite "$base" fatal2 2
+invoke "$base"
+if [[ "$CODE" == 1 ]] \
+  && [[ "$(jq -r .failed <<<"$OUT")" == 1 ]] \
+  && [[ "$(jq -r .suites <<<"$OUT")" == 2 ]] \
+  && [[ "$(jq -r '.failures[0]' <<<"$OUT")" == *"test_fatal2.sh" ]]; then
+  pass "suite exiting 2 -> counted as a failure, not a setup error"
+else
+  fail "suite exit 2: code=$CODE out=$OUT"
+fi
+rm -rf "$base"
+
+# --- a python suite exiting 2 is likewise a failure ---
+base="$(make_base)"; add_py_suite "$base" pyfatal2 2
+invoke "$base"
+if [[ "$CODE" == 1 ]] && [[ "$(jq -r .failed <<<"$OUT")" == 1 ]]; then
+  pass "python suite exiting 2 -> counted as a failure"
+else
+  fail "python exit 2: code=$CODE out=$OUT"
+fi
+rm -rf "$base"
+
+# --- a test_* file NESTED under tests/ is NOT dispatched as a suite ---
+# `find`'s `*` matches `/`, so `-path '*/tests/test_*.sh'` alone would match
+# `tests/test_fixtures/helper.sh`. The dirname==*/tests filter drops it.
+base="$(make_base)"; add_suite "$base" alpha 0
+nested="$base/skills/alpha/tests/test_fixtures"; mkdir -p "$nested"
+printf '#!/usr/bin/env bash\nexit 1\n' > "$nested/test_helper.sh"       # would fail the run if dispatched
+printf 'import sys\nif __name__=="__main__": sys.exit(1)\n' > "$nested/test_helper.py"
+invoke "$base"
+if [[ "$CODE" == 0 ]] && [[ "$(jq -r .suites <<<"$OUT")" == 1 ]]; then
+  pass "nested test_* under tests/ -> not a suite (only the direct child counts)"
+else
+  fail "nested exclusion: code=$CODE out=$OUT (expected suites=1, exit 0)"
+fi
+rm -rf "$base"
+
+# --- a suite exiting 125 is a FAILING SUITE, not a dispatcher fault ---
+# An earlier draft used 125 as the dispatcher's private sentinel. Any
+# reserved exit code is only a soft invariant: nothing stops a suite from
+# returning it, and a real red suite then reports as a broken runner and
+# stops the run. The fault now travels by side channel, so EVERY status a
+# suite returns is its own verdict. This pins that.
+base="$(make_base)"; add_suite "$base" alpha 0; add_suite "$base" sentinel125 125
+invoke "$base"
+if [[ "$CODE" == 1 ]] \
+  && [[ "$(jq -r .suites <<<"$OUT")" == 2 ]] \
+  && [[ "$(jq -r .failed <<<"$OUT")" == 1 ]] \
+  && jq -e '.failures | any(test("test_sentinel125.sh$"))' <<<"$OUT" >/dev/null; then
+  pass "suite exiting 125 -> counted as a failure, not a setup error"
+else
+  fail "suite exit 125: code=$CODE out=$OUT"
+fi
+rm -rf "$base"
+
+# --- a missing interpreter is a SETUP error (2), not a test failure (1) ---
+# Dispatching to an absent python3 gives the suite exit 127. Counting that
+# as a failing suite reports red tests for a runner that never ran them,
+# sending whoever reads CI to debug a test that never executed.
+base="$(make_base)"; add_py_suite "$base" pyalpha 0
+errf="$(mktemp)"
+# Override PY_BIN rather than emptying PATH — the harness needs bash, find,
+# and mktemp on PATH to run at all, so nuking it tests the shebang, not the
+# dispatcher.
+OUT="$(PY_BIN=definitely-not-a-real-python3 "$RUNNER" "$base" 2>"$errf")"; CODE=$?
+ERR="$(cat "$errf")"; rm -f "$errf"
+if [[ "$CODE" == 2 ]] \
+  && [[ "$(jq -r .suites <<<"$OUT")" == 0 ]] \
+  && [[ "$(jq -r '.error' <<<"$OUT")" == *"interpreter missing"* ]]; then
+  pass "missing interpreter -> exit 2 (setup error), not exit 1"
+else
+  fail "missing interpreter: code=$CODE out=$OUT err=$ERR"
+fi
+rm -rf "$base"
+
 echo ""
 echo "run-tests.sh: ${PASS_COUNT} passed, ${FAIL_COUNT} failed"
 [[ $FAIL_COUNT -eq 0 ]] || exit 1