Skip to content

CI: run box/mcp/measurement/blufi unit suites (matrix); fix 3 CI-fragile box tests#103

Open
danielrmerskine wants to merge 1 commit into
mainfrom
de/expand-unit-ci
Open

CI: run box/mcp/measurement/blufi unit suites (matrix); fix 3 CI-fragile box tests#103
danielrmerskine wants to merge 1 commit into
mainfrom
de/expand-unit-ci

Conversation

@danielrmerskine

@danielrmerskine danielrmerskine commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

Wire the ~1,190 already-written non-hardware unit tests into CI. Until now unit-tests.yml ran only test/unit/cli/, so the box-side, MCP, measurement, and blufi unit suites — covering debug/JTAG/RTT/gdbserver, device locking, box config/provisioning, SSH, DUT, USB scanner, the MCP server, and more — executed in no CI at all.

Changes

  • .github/workflows/unit-tests.yml: run each suite as its own ubuntu-latest matrix job (cli, box, measurement, blufi, mcp, root). The suites can't share a pytest process — test/unit/measurement/conftest.py stubs the lager package in sys.modules (to avoid heavy box deps), which breaks box/ collection — so each runs separately with --import-mode=importlib -c /dev/null --timeout=60. The job also sets PYTHONPATH = repo-root + box/ so the lager package and the test.* package resolve consistently in-process and in spawned subprocesses — without it, box collection fails on Linux with lager resolving as a namespace package (a Linux-only failure the never-run suite hid; it passed on macOS).
  • test/requirements-unit.txt (new): the pure-Python deps needed to import the box tree headless (numpy, simplejson, pyvisa[-py], flask, pyserial, pyusb, cryptography, fastmcp, pexpect, pygdbmi, flask-socketio). pygdbmi and flask-socketio are import-time deps of lager.debug / lager.http_handlers; without them the box package half-imports and lager resolves as a namespace package on Linux → 7 collection errors (some box tests stub pygdbmi, but only order-dependently, so it was flaky). Hardware SDKs stay mocked by the conftests / importlib-by-path loading, so nothing here pulls a device driver. pymupdf is intentionally excluded (AGPL-3.0 vs this project's Apache-2.0); test_pdf_pages.py skips without it. Deps are unpinned to mirror the box's own (apt/pip) runtime — happy to pin to known-good versions if preferred.
  • 3 CI-fragile box tests fixed (test-only, no behavior change) — all latent because the suite never ran:
    • test_box_authorize.py: the os.path.exists → False mock also defeated shutil.which("ssh-keygen") (same os.path object), so the code raised "ssh-keygen not found." Now also stubs shutil.which.
    • test_monitor_state.py: get_monitor_state reads ocp/ovp_tripped from separate *_is_tripped() methods; the test only mocked the *_value getters, so tripped stayed False. Now also raises on the tripped getters, matching the "falls back to None" intent.
    • test_breakpoint_pause.py: the socket-console test SIGPIPE-killed the run (exit 141) on headless Linux when the client disconnects mid-write. It now ignores SIGPIPE for that test — exactly as the box does in production (box_http_server sets SIG_IGN); the console already handles the resulting BrokenPipeError. (Passed on macOS locally, which ignores SIGPIPE by default — a Linux-only failure the never-run suite was hiding.)

Coverage added

~1,190 tests now gate PRs (verified locally on a clean Python 3.11 env):

suite tests
box 825
mcp 157
root 73 (+1 skipped)
measurement 66
blufi 72 (+4 skipped)
cli (already gated) 504

Testing

  • Ran affected CLI commands manually — n/a (CI + test changes)
  • Ran integration tests against a box — n/a; instead ran the full matrix locally on a clean Python 3.11 env: all suites green (box 825, mcp 157, measurement 66, blufi 72, root 73, cli 504). This PR's own unit-tests run is the on-CI proof.
  • Changes are documentation-only (no testing needed)

Notes / follow-ups

  • test/mcp/integration/ still needs live hardware → stays on the self-hosted bench, not this ubuntu job.
  • Cosmetic: box tests trigger a noisy atexit log from hardware_service._cleanup_device_cache ("I/O operation on closed file") at interpreter shutdown — harmless (jobs pass); worth a small guard in a follow-up.
  • The box still has no dependency manifest; test/requirements-unit.txt is a first cut of one.

Checklist

  • Code follows the project's style guidelines
  • Copyright headers are present on new files
  • CHANGELOG.md updated — n/a (no user-facing change; CI + tests only)
  • Documentation updated (if applicable) — the workflow header documents the per-suite / no-shared-process rationale

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a unit test requirements file (test/requirements-unit.txt) and updates tests in test_box_authorize.py and test_monitor_state.py to mock shutil.which and additional protection-tripped methods. Feedback on these changes suggests pinning the newly added dependencies to specific versions to prevent CI failures and replacing the AGPL-licensed pymupdf with a permissively licensed alternative like pypdf to avoid licensing conflicts with the project's Apache-2.0 license.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread test/requirements-unit.txt Outdated
Comment on lines +22 to +23
# Optional: unskips test/unit/test_pdf_pages.py (PDF text/page extraction).
pymupdf

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

PyMuPDF is licensed under the GNU AGPL v3, which is a strong copyleft license. Since this project is licensed under Apache-2.0, introducing an AGPL v3 dependency (even if optional or for testing) can create licensing compliance issues or copyleft concerns for downstream users and commercial deployments. Consider using a permissively licensed alternative like pypdf (BSD-3-Clause) or pdfminer.six (MIT) for PDF text and page extraction.

# Optional: unskips test/unit/test_pdf_pages.py (PDF text/page extraction).\npypdf

Comment on lines +12 to +21
numpy
simplejson
pyvisa
pyvisa-py
flask
pyserial
pyusb
cryptography
fastmcp
pexpect

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To ensure reproducible test environments and prevent unexpected failures in CI due to breaking changes in upstream packages, it is highly recommended to pin these dependencies to specific versions or version ranges (e.g., numpy>=1.20,<2.0).

@danielrmerskine danielrmerskine changed the title CI: run box/mcp/measurement/blufi unit suites (matrix); fix 2 stale box tests CI: run box/mcp/measurement/blufi unit suites (matrix); fix 3 CI-fragile box tests Jul 8, 2026
@danielrmerskine danielrmerskine force-pushed the de/expand-unit-ci branch 2 times, most recently from 95ed8f1 to bb0271d Compare July 8, 2026 21:51
…ile box tests

unit-tests.yml previously ran only test/unit/cli/ (pip install -e cli/), so
~1,190 already-written non-hardware tests — box-side debug/JTAG/RTT/gdbserver,
device locking, box config/provisioning, SSH, DUT, USB scanner, the MCP server,
measurement, and blufi — ran in no CI at all.

- unit-tests.yml: run each suite as its own ubuntu-latest matrix job
  (cli, box, measurement, blufi, mcp, root). The suites cannot share a pytest
  process (test/unit/measurement/conftest.py stubs `lager` in sys.modules,
  which breaks box-suite collection), so each runs separately with
  --import-mode=importlib -c /dev/null --timeout=60. PYTHONPATH is set to
  repo-root + box/ so the `lager` package and the `test.*` package resolve
  consistently in-process and in spawned subprocesses (without it, box
  collection fails on Linux with `lager` resolving as a namespace package —
  a Linux-only failure the never-run suite was hiding).
- test/requirements-unit.txt (new): the pure-Python deps needed to import the
  box tree headless (numpy, simplejson, pyvisa[-py], flask, pyserial, pyusb,
  cryptography, fastmcp, pexpect, pygdbmi, flask-socketio). pygdbmi and
  flask-socketio are import-time deps of lager.debug / lager.http_handlers;
  without them the box package half-imports and `lager` resolves as a namespace
  on Linux, causing 7 collection errors (some box tests stub pygdbmi, but only
  order-dependently). Hardware SDKs stay mocked. pymupdf is
  intentionally excluded (AGPL-3.0 vs the project's Apache-2.0); test_pdf_pages
  skips accordingly. Deps are unpinned to mirror the box's own runtime.
- Fix 3 CI-fragile box tests (test-only, no behavior change), all latent
  because the suite never ran:
  - test_box_authorize: the os.path.exists=False mock also defeated
    shutil.which("ssh-keygen") (same os.path object), so the code raised
    "ssh-keygen not found". Now also stubs shutil.which.
  - test_monitor_state: get_monitor_state reads ocp/ovp_tripped from separate
    *_is_tripped() methods; the test mocked only the *_value getters, so tripped
    stayed False. Now also raises on the tripped getters, matching the
    "falls back to None" intent.
  - test_breakpoint_pause: the socket-console test SIGPIPE-killed the run
    (exit 141) on headless Linux when the client disconnects mid-write. It now
    ignores SIGPIPE for that test, exactly as the box does in production
    (box_http_server sets SIG_IGN); the console already handles the resulting
    BrokenPipeError.

Adds ~1,190 tests to PR gating.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant