usbtest: device-side peer for the Linux kernel usbtest battery + DCD fixes across 11 ports

.claude/skills/hil/SKILL.md

@@ -11,9 +11,15 @@ Run TinyUSB HIL tests on real boards. **Run `hostname` first** — it tells you
 |------|--------------|------------------------|
 | `htpc` (dev PC) | `test/hil/local.json` | yes (large pool, `test/hil/tinyusb.json`) |
 | `ci` (the rig) | `test/hil/tinyusb.json` (large pool) | no — can't SSH to htpc, and boards are already local |
+| `hifiphile` (external rig) | `test/hil/hfp.json` | no outbound SSH to htpc/ci; SSH-reachable FROM both |
 
 Default to **local**. Use **remote** only when on `htpc` and the user says `remote`/`ci.lan`. Never attempt remote on `ci`.
 
+The `hifiphile` rig is externally hosted by TinyUSB maintainer hifiphile; its board pool is
+`test/hil/hfp.json` and its HIL runs are triggered by GitHub CI (the `hil-tinyusb (hfp.json)`
+matrix job). **Never run HIL against this rig during development unless the user explicitly
+asks for it.**
+
 ## Board locks — the CI runner keeps running
 
 The `ci` rig also hosts a GitHub Actions runner that flashes boards and runs HIL as part of CI. Hardware access is arbitrated **per board** with kernel flocks in `/tmp/tinyusb-hil-locks/` — do NOT stop the runner service.
@@ -32,7 +38,6 @@ python3 test/hil/board_lock.py release BOARD [BOARD...]
 - Rig-wide operations (uhubctl power cycling, pci-rebind — bus renumbering) affect every board: `board_lock.py hold --all --reason "..."` first.
 - `board_lock.py status` lists holders. Locks auto-release when the holder process dies (kernel flock); `/tmp` clears on reboot.
 - Forcing past a lock: `HIL_NO_BOARD_LOCK=1 python3 test/hil/hil_test.py ...` bypasses the guard without killing the holder. Only with the user's explicit go-ahead — they accept the risk of colliding with whatever holds the board.
-- Caveat until this branch merges to master: CI's checkout of `hil_test.py` does not yet enforce locks — keep dev hardware sessions short and check `gh run list --status in_progress` first.
 
 ## Prerequisites
 

.claude/skills/usb-debug/SKILL.md

@@ -0,0 +1,36 @@
+---
+name: usb-debug
+description: Use when USB enumeration fails or misbehaves and usbmon alone can't explain WHY the host acted — port reset storms, repeated re-enumeration, address errors, xHCI ring/command errors, "device descriptor read error", babble, or when you need the host driver's own reasoning from dmesg on the ci HIL rig.
+---
+
+# usb-debug — host-side kernel dynamic debug for USB
+
+usbmon shows the URBs; kernel **dynamic debug** shows the host driver's
+*reasoning* usbmon can't: port resets and their causes, enumeration retries,
+address (re)assignment, EP halts, xHCI ring/command errors.
+
+Run this skill's `scripts/usb_dyndbg.sh` with `sudo` (abbreviated to
+`usb_dyndbg.sh` in the examples below). It flips the dynamic-debug print flag
+for an allowlisted set of USB host modules only:
+
+```bash
+sudo usb_dyndbg.sh on  usbcore xhci_hcd   # enable +p; pick modules from `lsusb -t` Driver=
+sudo usb_dyndbg.sh status [module]        # list enabled print sites
+sudo usb_dyndbg.sh off usbcore xhci_hcd   # ALWAYS turn off when done — very noisy
+```
+
+Allowlisted modules: `usbcore xhci_hcd xhci_pci xhci_pci_renesas ehci_hcd
+ehci_pci ohci_hcd ohci_pci uhci_hcd dwc2 cdc_acm usb_storage uas`.
+
+## Workflow
+
+1. `sudo usb_dyndbg.sh on usbcore <hcd-module>` — `usbcore` for enumeration/hub
+   logic, plus the controller module (`lsusb -t` shows the driver per bus).
+2. Reproduce (replug / re-enumerate / rerun the failing test) while following
+   `sudo dmesg -w` (or grab `sudo dmesg | tail` afterwards).
+3. `sudo usb_dyndbg.sh off ...` — leaving it on floods the log and skews timing.
+
+Pair with the `usbmon` skill: usbmon for what crossed the bus, dynamic debug for
+why the host reacted. For a wedged device/bus use the `usb-recover` skill.
+
+Requires `CONFIG_DYNAMIC_DEBUG` and mounted debugfs (standard on distro kernels).

.claude/skills/usb-debug/scripts/usb_dyndbg.sh

@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+# usb_dyndbg.sh — toggle kernel dynamic-debug on USB host drivers; run with sudo.
+# Flips +p/-p only on an allowlisted set of USB modules, so it can't reach
+# arbitrary kernel debug or unrelated subsystems.
+#
+# Usage:
+#   sudo usb_dyndbg.sh on  <module>...   # enable  +p  (e.g. usbcore xhci_hcd)
+#   sudo usb_dyndbg.sh off <module>...   # disable -p
+#   sudo usb_dyndbg.sh status [module]   # show enabled sites (or one module's sites)
+set -euo pipefail
+
+CTL=/sys/kernel/debug/dynamic_debug/control
+# Allowlist: USB host-controller + core + common host class drivers.
+ALLOW='usbcore xhci_hcd xhci_pci xhci_pci_renesas ehci_hcd ehci_pci ohci_hcd ohci_pci uhci_hcd dwc2 cdc_acm usb_storage uas'
+
+die() { echo "usb_dyndbg: $*" >&2; exit 1; }
+usage() {
+  echo "usage: usb_dyndbg.sh {on|off} <module>...   modules: $ALLOW" >&2
+  echo "       usb_dyndbg.sh status [module]" >&2
+  exit 2
+}
+allowed() { local m; for m in $ALLOW; do [ "$m" = "$1" ] && return 0; done; return 1; }
+
+[ -e "$CTL" ] || die "dynamic_debug unavailable (need CONFIG_DYNAMIC_DEBUG + debugfs mounted)"
+
+action=${1:-}; shift || true
+case "$action" in
+  on|off)
+    [ "$#" -ge 1 ] || usage
+    flag='+p'; [ "$action" = off ] && flag='-p'
+    for m in "$@"; do allowed "$m" || die "module not allowlisted: $m"; done
+    for m in "$@"; do echo "module $m $flag" > "$CTL"; echo "dynamic debug $action: $m"; done
+    ;;
+  status)
+    m=${1:-}
+    if [ -n "$m" ]; then
+      allowed "$m" || die "module not allowlisted: $m"
+      grep -E "\[$m\]" "$CTL" || echo "(no sites for $m)"
+    else
+      grep -E '=p( |$)' "$CTL" || echo "(no print sites enabled)"
+    fi
+    ;;
+  *)
+    usage
+    ;;
+esac

.claude/skills/usb-recover/SKILL.md

@@ -0,0 +1,93 @@
+---
+name: usb-recover
+description: Use when a USB device or fixture on the ci HIL rig is stuck, hung, not enumerating, or wedged after a failed flash or test, or when processes touching USB (testusb, JLinkExe, uhubctl, libusb tools) start hanging in D state.
+---
+
+# USB Recovery on the HIL Rig
+
+Run this skill's `scripts/usb_recover.sh` with `sudo` (abbreviated to
+`usb_recover.sh` in the examples below). It wraps four sysfs reset actions plus
+a resolver:
+
+```bash
+sudo usb_recover.sh resolve    /dev/ttyACM3    # /dev node -> busport (e.g. 3-4.7); also ttyUSB*, sg*
+sudo usb_recover.sh authorized <busport>       # deauthorize+reauthorize: re-enumerate, no VBUS cut
+sudo usb_recover.sh rebind     <busport>       # usb driver unbind+bind: re-probe
+sudo usb_recover.sh pci-rebind <pciaddr>       # whole HCD controller unbind+bind, e.g. 0000:02:00.0
+sudo usb_recover.sh pci-reset  <pciaddr>       # PCI function-level reset: kills URBs at HW level, no device lock
+sudo usb_recover.sh pci-bind   <pciaddr> [drv] # re-bind a DRIVERLESS controller (auto-tries xHCI drivers)
+```
+
+## Decide first: is anything stuck in D state?
+
+```bash
+ps -eo pid,stat,wchan:30,cmd | awk '$2 ~ /D/'
+```
+
+**If yes** (uninterruptible sleep, typically a usbfs ioctl — e.g. testusb inside
+`usb_sg_wait`): run `pci-reset` and NOTHING ELSE first:
+
+```bash
+sudo usb_recover.sh pci-reset <pciaddr>
+```
+
+FLR kills the URBs at the hardware level without taking the per-device lock;
+the ioctl then returns and the convoy unwinds on its own.
+
+**Not every controller supports FLR.** The Renesas uPD720201 (`0000:01:00.0`)
+has no reset method — `pci-reset` fails with `Inappropriate ioctl for device`
+(ENOTTY). On those, there is no clean software D-state cure — a VM reboot is NOT
+reliable (the MosChip downstream hubs latch up across the PCIe reset and need a
+physical replug); ask the operator for a full PVE host power cycle instead. Do NOT
+fall through to `pci-rebind` (see next).
+
+**`pci-rebind` can strand the controller driverless.** Its unbind succeeds but,
+with a D-state process still holding a URB, the *re-bind* hangs — leaving the
+PCI device with **no driver** (`/sys/bus/pci/devices/<addr>/driver` gone) and the
+whole controller's fixtures offline. A second `pci-rebind` then dies with "no
+driver bound". Recover with `pci-bind <addr>` (re-attaches the xHCI driver);
+if that also hangs because the D-state URB is unkillable, only a full PVE host
+power cycle (operator action) recovers. The Renesas binds via `xhci-pci-renesas` (firmware loader), others via
+`xhci_hcd` — `pci-bind` auto-tries both, or pass the driver explicitly.
+
+**Ordering is critical.** `authorized`/`rebind`/`pci-rebind` all take the
+per-device lock the stuck ioctl holds — they block and join the convoy, and
+soon every libusb tool (uhubctl, JLinkExe) hangs too. Worse, a blocked
+`pci-rebind` grabs the PCI device lock on its way in, which `pci-reset` also
+needs: once a rebind has been attempted and is stuck, even FLR deadlocks and
+**only a full PVE host power cycle recovers**. pci-reset first (if supported), and never
+`pci-rebind` a D-state wedge.
+
+**If no** (device merely dead or silent), escalate gently:
+
+1. `authorized <busport>` — re-enumerates just that device
+2. `rebind <busport>` — re-probe; also worth trying on the parent hub's busport
+3. `pci-rebind <pciaddr>` — last resort: bounces every fixture on that controller
+
+## Finding targets
+
+```bash
+grep -l <SERIAL> /sys/bus/usb/devices/*/serial          # serial -> busport (dir name)
+readlink -f /sys/bus/usb/devices/usb<N>                  # bus N -> its PCI addr in the path
+```
+
+Rig layout: buses 3+4 = `0000:02:00.0` (main fixture tree: J-Links, ST-Links,
+WCH-Links, DUTs); buses 9+12 = `0000:01:00.0`, the only ones with uhubctl port
+power (ganged VBUS: `sudo uhubctl -l 9 -a cycle`). Hubs on buses 1-4 have no
+port power switching — uhubctl reports "No compatible devices" there.
+
+## Common mistakes
+
+- `resolve` takes a **/dev node**, not a busport or serial ("no such device node").
+- `authorized`/`rebind` take a **busport** (`3-4.7`); `pci-rebind`/`pci-reset`
+  take a **PCI addr**.
+- Command produces no output and doesn't return → it is blocked on the device
+  lock: a D-state holder exists; see above.
+- Trying `pci-rebind` on a D-state hang — its re-bind hangs and strands the
+  controller **driverless**; recover with `pci-bind <addr>`, or a PVE host power
+  cycle if the D-state URB is unkillable. Use `pci-reset` (if supported) for D-state, never
+  `pci-rebind`.
+- Running `pci-reset` on a controller without FLR support (Renesas) → ENOTTY;
+  no software recovery — needs a PVE host power cycle.
+- A J-Link reset (`r; go`) does not disconnect a wedged DUT from the host: the
+  DWC2 soft-connect pullup stays up through a core halt, so stuck URBs stay stuck.

.claude/skills/usb-recover/scripts/usb_recover.sh

@@ -0,0 +1,111 @@
+#!/usr/bin/env bash
+# usb_recover.sh — USB recovery helper for the HIL rig; run with sudo. Writes only
+# to the specific sysfs control files below; arg regexes block path traversal.
+#
+# Usage:
+#   sudo usb_recover.sh authorized <busport>   # e.g. 3-2  -> deauthorize+reauthorize (re-enumerate, NO VBUS cut)
+#   sudo usb_recover.sh rebind     <busport>   # e.g. 3-2  -> usb driver unbind+bind (re-probe)
+#   sudo usb_recover.sh pci-rebind <pciaddr>   # e.g. 0000:01:00.0 -> HCD unbind+bind (WHOLE controller)
+#   sudo usb_recover.sh pci-reset  <pciaddr>   # e.g. 0000:01:00.0 -> PCI function-level reset: kills URBs at
+#                                              # HW level WITHOUT the device lock; the only cure when a process
+#                                              # is stuck in D state (usbfs ioctl) and unbind paths would convoy
+#   sudo usb_recover.sh pci-bind   <pciaddr> [driver]  # bind a DRIVERLESS controller (e.g. after a pci-rebind
+#                                              # whose re-bind hung and left it unbound). Auto-tries the xHCI
+#                                              # drivers (xhci-pci-renesas, xhci_hcd) unless one is named.
+#   sudo usb_recover.sh resolve    <devnode>   # e.g. /dev/ttyACM3 -> print its <busport> (no privilege needed)
+set -euo pipefail
+
+USBPATH_RE='^[0-9]+-[0-9]+(\.[0-9]+)*$'
+PCI_RE='^[0-9a-fA-F]{4}:[0-9a-fA-F]{2}:[0-9a-fA-F]{2}\.[0-9]$'
+DRIVER_RE='^[A-Za-z0-9_-]+$'
+
+die() { echo "usb_recover: $*" >&2; exit 1; }
+usage() { grep -E '^#   sudo usb_recover' "$0" >&2; exit 2; }
+
+# Refuse to touch a PCI function that is not a USB controller (class 0x0c03xx), so a stray or
+# mistyped BDF can't unbind/reset an unrelated device (storage, NIC) on a shared HIL host.
+require_usb_controller() {
+  local addr=$1 cls
+  cls=$(cat "/sys/bus/pci/devices/$addr/class" 2>/dev/null) || die "no such pci device: $addr"
+  [[ "$cls" =~ ^0x0c03 ]] || die "$addr is not a USB controller (class $cls); refusing"
+}
+
+# Resolve a /dev node (ttyACMx, ttyUSBx, sgN, ...) up to its USB device busport.
+resolve() {
+  local node=$1 syspath dev
+  [ -e "$node" ] || die "no such device node: $node"
+  syspath=$(udevadm info -q path -n "$node" 2>/dev/null) || die "udevadm failed for $node"
+  dev="/sys$syspath"
+  while [ "$dev" != "/sys" ] && [ -n "$dev" ]; do
+    if [ -e "$dev/busnum" ] && [ -e "$dev/devnum" ] && [ -e "$dev/authorized" ]; then
+      basename "$dev"; return 0
+    fi
+    dev=$(dirname "$dev")
+  done
+  die "could not find parent USB device for $node"
+}
+
+action=${1:-}; target=${2:-}
+[ -n "$action" ] && [ -n "$target" ] || usage
+
+case "$action" in
+  resolve)
+    resolve "$target"
+    ;;
+  authorized)
+    [[ "$target" =~ $USBPATH_RE ]] || die "bad usb path: $target"
+    d="/sys/bus/usb/devices/$target"
+    [ -e "$d/authorized" ] || die "no such usb device: $target"
+    echo 0 > "$d/authorized"; sleep 1; echo 1 > "$d/authorized"
+    echo "re-authorized $target"
+    ;;
+  rebind)
+    [[ "$target" =~ $USBPATH_RE ]] || die "bad usb path: $target"
+    [ -e "/sys/bus/usb/devices/$target" ] || die "no such usb device: $target"
+    echo "$target" > /sys/bus/usb/drivers/usb/unbind; sleep 1
+    echo "$target" > /sys/bus/usb/drivers/usb/bind
+    echo "rebound $target"
+    ;;
+  pci-rebind)
+    [[ "$target" =~ $PCI_RE ]] || die "bad pci addr: $target"
+    require_usb_controller "$target"
+    [ -e "/sys/bus/pci/devices/$target/driver" ] || die "no driver bound to $target"
+    drv=$(basename "$(readlink -f "/sys/bus/pci/devices/$target/driver")")
+    echo "$target" > "/sys/bus/pci/drivers/$drv/unbind"; sleep 1
+    echo "$target" > "/sys/bus/pci/drivers/$drv/bind"
+    echo "rebound pci $target ($drv)"
+    ;;
+  pci-bind)
+    # Re-attach a driver to a controller left DRIVERLESS (e.g. a pci-rebind whose re-bind hung).
+    [[ "$target" =~ $PCI_RE ]] || die "bad pci addr: $target"
+    require_usb_controller "$target"
+    [ -e "/sys/bus/pci/devices/$target" ] || die "no such pci device: $target"
+    [ -e "/sys/bus/pci/devices/$target/driver" ] && die "$target already has a driver bound"
+    drv=${3:-}
+    if [ -n "$drv" ]; then
+      [[ "$drv" =~ $DRIVER_RE ]] || die "bad driver name: $drv"
+      [ -e "/sys/bus/pci/drivers/$drv/bind" ] || die "no such pci driver: $drv"
+      echo "$target" > "/sys/bus/pci/drivers/$drv/bind"
+      echo "bound pci $target ($drv)"
+    else
+      # Auto-try the xHCI drivers (Renesas uPD720201 uses xhci-pci-renesas; others xhci_hcd).
+      for cand in xhci-pci-renesas xhci_hcd; do
+        [ -e "/sys/bus/pci/drivers/$cand/bind" ] || continue
+        if echo "$target" > "/sys/bus/pci/drivers/$cand/bind" 2>/dev/null; then
+          echo "bound pci $target ($cand)"; exit 0
+        fi
+      done
+      die "could not bind $target with a known xHCI driver; pass the driver explicitly"
+    fi
+    ;;
+  pci-reset)
+    [[ "$target" =~ $PCI_RE ]] || die "bad pci addr: $target"
+    require_usb_controller "$target"
+    [ -e "/sys/bus/pci/devices/$target/reset" ] || die "no reset support on $target"
+    echo 1 > "/sys/bus/pci/devices/$target/reset"
+    echo "flr-reset pci $target"
+    ;;
+  *)
+    usage
+    ;;
+esac