Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/mutation-baseline.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ Per-module (approx. killed / accounted on a clean run):
| `download/naming.py` | ~55–56% | Injection + sanitize properties; CD/header edges survive |
| `job/predict.py` | ~33% | Dry-run heuristics under-specified; many equivalent branches |
| `download/validation.py` | ~30–31% | Message/emit paths + normalize edge strings |
| `slicer/output.py` | ~21% | **Low:** `_finalize_slice` I/O/logging mutates with little unit signal; `_is_valid_sliced_3mf` itself is much better covered |
| `slicer/output.py` | ~21% baseline; **not yet re-measured** since the C.4 hermetic Orca stub landed | **Was low:** `_finalize_slice` I/O/logging mutated with little unit signal. `tests/test_slice_stub_integration.py` (roadmap C.4) now drives its benign-GL / empty / corrupt / missing-output / real-error branches through the real slicer subprocess (line coverage 79.8%→~93%), so a re-run of `mutmut` on this module should score materially higher — update this row after the next mutation run. |

**Honest reading:** the overall score **dropped vs Phase 1** because scope **widened** into harder modules (especially `output._finalize_slice` and `predict`/`validation` emit paths). That is intentional. Do **not** restore a high score by shrinking back to only well-covered files.

Expand All @@ -92,7 +92,7 @@ Enforced by `./scripts/run_mutation_baseline.sh` after `mutmut export-cicd-stats
Categories (not an exhaustive dump of 861 IDs):

1. **Equivalent / cosmetic** — error-message string literals, log format, `getattr` default when tests always set the attribute, `ZipFile(..., "r")` vs default mode.
2. **`_finalize_slice` (output.py)** — subprocess exit interpretation, JSON emit, path display. Deferred: needs hermetic fake-Orca fixtures; not pure safety. Dominates the low output.py score.
2. **`_finalize_slice` (output.py)** — subprocess exit interpretation, JSON emit, path display. **Addressed (C.4):** `tests/fakes/orca_stub` + `tests/test_slice_stub_integration.py` now run these branches against a real fake-slicer subprocess. Residual survivors here should be cosmetic (log strings / path display); re-measure before treating any as "accepted".
3. **DNS cache / hop bookkeeping (netsafety)** — TTL, cache size clear, attribute names on redirect requests. Core `is_global` refuse path is well killed.
4. **URL normalize / Content-Disposition edges (validation/naming)** — ambiguous scheme-less inputs and RFC2231 header tuples; behavior partially covered; full combinatorial matrix deferred.
5. **Dry-run prediction (predict.py)** — Printables/archive/extension branches that return `None` early; many mutants are observationally equivalent under the focused suite.
Expand Down Expand Up @@ -121,4 +121,4 @@ Artifacts (`mutants/`, `.mutmut-cache`, `.hypothesis/`) are gitignored.

- mutmut 3.x needs Python ≥ 3.10 (CI mutation job uses 3.12).
- Hypothesis property tests live in `tests/test_properties_safety.py` and are part of the focused mutmut suite.
- Raising the score further: add hermetic tests for `_finalize_slice` decision branches, or move pure 3mf validation to a tiny module so mutmut does not spend budget on I/O.
- Raising the score further: the hermetic `_finalize_slice` tests now exist (C.4, `tests/test_slice_stub_integration.py`); re-run `mutmut` on `slicer/output.py` and update the per-module row. Optionally still move pure 3mf validation to a tiny module so mutmut does not spend budget on I/O.
6 changes: 3 additions & 3 deletions docs/quality-roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ security is not yet **A+**.
| Correctness / bugs | **A** | dead flags fixed (global `--json` before subcommand); structured errors; purity greps; version single-sourced |
| Typing | **A** | `uvx mypy -p bambu_cli` full package with `check_untyped_defs = true`; no residual excludes |
| Error model | **A** | `sys.exit` only in `cli.py` (errors.py hits are docstrings); domain uses `abort` / `BambuError` |
| Tests | **A−** | **1041** non-live tests collected / **1041** passing (2026-07-29; the latest additions cover the shared `verify_cert_fingerprint` TLS pin checker, incl. malformed/non-ASCII pin fail-closed cases); **84.6%** coverage measured 2026-07-29 on Linux; CI floor **83**; per-module floors not enforced |
| Tests | **A−** | **1065** non-live tests collected / **1064** passing (2026-07-29; latest additions are the C.4 hermetic fake-OrcaSlicer slice tests exercising the real slicer subprocess, and the shared `verify_cert_fingerprint` TLS pin checker incl. malformed/non-ASCII fail-closed cases); **84.9%** coverage measured 2026-07-29 on Linux; CI floor **83**; per-module floors not enforced |
| CI / release | **A−** | single pytest path; purity greps; bandit/audit/mypy blocking; **`--cov-fail-under=83`** (A+ target remains 92) |
| Docs / governance | **A−** | roadmap + backlog + SECURITY + AGENTS aligned (2026-07-24); prior AGENTS mypy-blocklist / backlog ≥98% claims corrected |
| Product polish | **B+** | quality gates in place; still pre-1.0 Beta (version is single-sourced from `pyproject.toml`); coverage ratchet + camera defaults remain for 1.0 A+ |
Expand Down Expand Up @@ -428,7 +428,7 @@ pytest -W error::ResourceWarning --cov=bambu_cli --cov-fail-under=85
| C.1 | **T2** remaining wizard/mDNS cases |
| C.2 | **T3** full download/extract matrix |
| C.3 | **T4** slicer + doctor + print safety |
| C.4 | Hermetic **fake OrcaSlicer** script in `tests/fakes/orca_stub` (exit codes, stdout, profile paths) |
| C.4 | **Done.** Hermetic **fake OrcaSlicer** script in `tests/fakes/orca_stub` (exit codes, stdout, profile paths); `tests/test_slice_stub_integration.py` runs `cmd_slice` end-to-end through the real `_run_orcaslicer`/`_finalize_slice` instead of mocking `subprocess.Popen` |
| C.5 | Coverage total ≥**92%**; module floors per scorecard A+ column for transport/setup/download |

#### Typing
Expand Down Expand Up @@ -459,7 +459,7 @@ bandit + pip-audit blocking
#### DoD

- [ ] Scorecard **A** column green for Tests and Typing (Typing A− OK if strict not yet full-package).
- [ ] Fake Orca used by default in slice unit tests.
- [x] Fake Orca available (`tests/fakes/orca_stub`) and used by the hermetic slice suite (`tests/test_slice_stub_integration.py`); remaining `test_slice_cmd.py` unit tests keep their mocks where they assert argv assembly / profile-resolution branches.
- [ ] `docs/test-backlog.md` reduced to “nice-to-have” only (or empty P1–P5).

#### Score impact
Expand Down
8 changes: 4 additions & 4 deletions docs/test-backlog.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@ Do not treat historical “≥98% coverage” claims as current — see the snap

| Metric | Current (honest) | A+ / 1.0 target |
|--------|------------------|-----------------|
| Non-live tests collected | **1041** collected / **1041** passing (measured 2026-07-29) | ≥550 with zero known flakes ✅ size |
| Line/branch coverage (CI) | **84.32%** Linux / ~**83.9%** Windows measured 2026-07-26; **floor 83** (Windows is the binding leg) | **≥92%** total; optional module floors |
| Non-live tests collected | **1065** collected / **1064** passing (measured 2026-07-29; incl. the C.4 hermetic Orca slice tests and the shared TLS-pin checker tests) | ≥550 with zero known flakes ✅ size |
| Line/branch coverage (CI) | **84.9%** Linux measured 2026-07-29 (C.4 hermetic Orca tests lift `slicer/output.py` 79.8%→92.7%); ~**83.9%** Windows measured 2026-07-26; **floor 83** (Windows is the binding leg) | **≥92%** total; optional module floors |
| Typing | Full package mypy + `check_untyped_defs` | keep; optional full `strict` later |
| Error model | `sys.exit` only in `cli.py` | keep |
| `@mockable` / test-awareness | **0** (CI greps) | keep |
| JSON schemas | **19** files under `docs/schemas/` | every `--json` command + monitor goldens |
| Mutation baseline | Pure safety modules; floor **40%** | optional raise after hermetic Orca stub |
| Mutation baseline | Pure safety modules; floor **40%** | hermetic Orca stub landed (C.4); re-run `mutmut` on `slicer/output.py` to raise its row |
| Live printer | Documented opt-in harness | manual pre-release (optional scheduled lab) |
| Product version | pre-1.0 Beta (single-sourced from `pyproject.toml`) | **v1.0.0** when roadmap §5 is complete |

Expand Down Expand Up @@ -50,7 +50,7 @@ Tracked in [SECURITY.md](../SECURITY.md) known limitations:
|-----|-------|
| Raise CI floor 83 → 85 → 88 → **92** | Residual: mqtt/ftps pin paths, pool recovery, wizard TTY, Orca process |
| Per-module floors (optional) | mqtt / ftps / netsafety / download / camera |
| Hermetic fake Orca binary | Raises mutation kill rate on `slicer/output._finalize_slice`; slice unit tests less mock-heavy |
| ~~Hermetic fake Orca binary~~ | **Done (C.4).** `tests/fakes/orca_stub` + `tests/test_slice_stub_integration.py` run `cmd_slice` end-to-end through the real slicer subprocess (`_run_orcaslicer`/`_finalize_slice`); `slicer/output.py` line coverage 79.8%→~93%. Mutation re-run on that module still pending. |

### P2 — Contracts & agent surface

Expand Down
1 change: 1 addition & 0 deletions tests/fakes/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"""Hermetic test fakes (fake OrcaSlicer, etc.) — see roadmap A.3 / C.4."""
133 changes: 133 additions & 0 deletions tests/fakes/orca_stub/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
"""Hermetic fake OrcaSlicer harness (roadmap C.4).

Provides:

* ``orca_stub.py`` — a standalone fake-slicer script (invoked as the binary).
* ``make_orca_launcher`` — write a cross-platform launcher for that script so
``settings.orca_slicer`` can point at a single directly-executable path that
``subprocess.Popen`` can run on Linux, macOS, and Windows (no shebang/exec-bit
reliance on Windows).
* ``build_profiles_dir`` — materialise a real OrcaSlicer-style ``profiles_dir``
(machine/process/filament JSONs) under a tmp dir so ``cmd_slice`` finds real
files instead of mocking ``os.path.exists``/``os.listdir``.
* ``write_stl`` — write a tiny real ASCII STL model input.
"""

from __future__ import annotations

import json
import os
import stat
import sys

STUB_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "orca_stub.py")


def make_orca_launcher(dest_dir: str, name: str = "orca-slicer") -> str:
"""Write a launcher for ``orca_stub.py`` and return its path.

The launcher is what ``settings.orca_slicer`` points at. ``cmd_slice`` runs
it via ``subprocess.Popen([launcher, ...args])`` and
``_slicer_executable_problem`` checks it exists + is X_OK on POSIX, so the
launcher must be a single directly-executable file on every OS:

* POSIX: a ``sh`` wrapper that ``exec``s ``<python> orca_stub.py "$@"``,
chmod +x. (The X_OK check itself requires this on POSIX.)
* Windows: a ``.cmd`` batch file ``@<python> orca_stub.py %*`` — Popen can
launch a ``.cmd`` directly; there is no exec bit to set.

Both embed the current interpreter (``sys.executable``) so no shebang or
``PATH`` python lookup is needed.
"""
python = sys.executable
if os.name == "nt":
launcher = os.path.join(dest_dir, name + ".cmd")
# %* forwards all args; quote python + script for spaced paths.
content = f'@"{python}" "{STUB_PATH}" %*\r\n'
with open(launcher, "w", encoding="utf-8", newline="") as fh:
fh.write(content)
return launcher

launcher = os.path.join(dest_dir, name)
content = f'#!/bin/sh\nexec "{python}" "{STUB_PATH}" "$@"\n'
with open(launcher, "w", encoding="utf-8") as fh:
fh.write(content)
mode = os.stat(launcher).st_mode
os.chmod(launcher, mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
return launcher


# ---- Minimal but realistic profile JSONs ------------------------------------
# cmd_slice looks for, in ``profiles_dir``:
# machine/<full_model_name> <nozzle> nozzle.json (default P1P 0.4)
# process/<layer> Standard @BBL P1P.json (quality-derived name)
# filament/*<requested>*@base*.json (fuzzy match, default PLA Basic)
# The names below match the default P1P / 0.4 / standard / PLA Basic path.

_MACHINE_JSON = {
"type": "machine",
"name": "Bambu Lab P1P 0.4 nozzle",
"nozzle_diameter": ["0.4"],
"printer_model": "Bambu Lab P1P",
}

_PROCESS_JSON = {
"type": "process",
"name": "0.20mm Standard @BBL P1P",
"layer_height": "0.2",
"compatible_printers": ["Bambu Lab P1P 0.4 nozzle"],
}

_FILAMENT_JSON = {
"type": "filament",
"name": "Bambu PLA Basic @base",
"filament_type": ["PLA"],
}


def build_profiles_dir(root: str) -> str:
"""Create a real profiles_dir tree under *root* and return its path.

Matches the default slice path (P1P, 0.4 nozzle, standard quality, PLA
Basic) so ``cmd_slice`` resolves every profile from real files on disk.
"""
profiles_dir = os.path.join(root, "profiles")
for sub, name, payload in (
("machine", "Bambu Lab P1P 0.4 nozzle.json", _MACHINE_JSON),
("process", "0.20mm Standard @BBL P1P.json", _PROCESS_JSON),
("filament", "Bambu PLA Basic @base.json", _FILAMENT_JSON),
):
d = os.path.join(profiles_dir, sub)
os.makedirs(d, exist_ok=True)
with open(os.path.join(d, name), "w", encoding="utf-8") as fh:
json.dump(payload, fh)
return profiles_dir


# A minimal valid ASCII STL (single degenerate triangle) — enough for the stub
# to receive a real model input path; the stub never actually parses it.
_STL = """solid cube
facet normal 0 0 0
outer loop
vertex 0 0 0
vertex 1 0 0
vertex 0 1 0
endloop
endfacet
endsolid cube
"""


def write_stl(path: str) -> str:
"""Write a tiny real ASCII STL to *path* and return it."""
with open(path, "w", encoding="utf-8") as fh:
fh.write(_STL)
return path


__all__ = [
"STUB_PATH",
"make_orca_launcher",
"build_profiles_dir",
"write_stl",
]
Loading