diff --git a/.githooks/commit-msg b/.githooks/commit-msg old mode 100644 new mode 100755 diff --git a/.githooks/pre-commit b/.githooks/pre-commit old mode 100644 new mode 100755 diff --git a/.githooks/pre-push b/.githooks/pre-push old mode 100644 new mode 100755 index b815b6805..b0a7aadac --- a/.githooks/pre-push +++ b/.githooks/pre-push @@ -40,6 +40,20 @@ set -e REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)" +# Git exports repository-local control variables to hooks. They are valid +# while this hook operates on REPO_ROOT, but the FAST gate runs pytest suites +# that create and mutate their own temporary repositories. In a linked +# worktree, inheriting GIT_INDEX_FILE makes a fixture's `git add` replace the +# real worktree index. Resolve the root first, then clear every variable Git +# itself classifies as repository-local and anchor subsequent Git commands by +# cwd. Author/committer identity and other non-local Git settings survive. +GIT_LOCAL_ENV_VARS="$(git rev-parse --local-env-vars 2>/dev/null || true)" +for var in $GIT_LOCAL_ENV_VARS; do + unset "$var" +done +unset GIT_LOCAL_ENV_VARS +cd "$REPO_ROOT" + PY="${PYTHON:-python}" if ! command -v "$PY" >/dev/null 2>&1; then if command -v py >/dev/null 2>&1; then diff --git a/.github/workflows/roam-ci.yml b/.github/workflows/roam-ci.yml index 7e6a64b4c..9c43d0040 100644 --- a/.github/workflows/roam-ci.yml +++ b/.github/workflows/roam-ci.yml @@ -394,8 +394,12 @@ jobs: fetch-depth: 0 # full history required for pr-risk and git analysis persist-credentials: false + # Bind prerelease self-analysis to this exact local action checkout. + # Explicit source mode has no PyPI fallback, so an unpublished version + # can test itself without weakening downstream exact-version defaults. - uses: ./ # local composite action — tests action.yml itself with: + version: 'source' commands: 'health pr-risk' changed-only: 'true' sarif: 'true' diff --git a/CHANGELOG.md b/CHANGELOG.md index dd26aea53..4b250271a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/). ## [Unreleased] -## [13.10.0] — 2026-07-19 +## [13.10.0] — 2026-07-23 ### Added - **`roam savings` + `roam savings-backfill` — proof-oriented Procedure Foundry telemetry** — prompt compilation and Claude Stop hooks now share a per-turn `episode_id`, preserving compile identity, terminal outcome, verification state, changed-file count, and privacy-preserving diff identity in an append-only local ledger. `savings` materializes the ledger into SQLite and withholds savings claims until known-answer canaries, telemetry integrity, hook-version coverage, compile joins, terminal coverage, and explicit verification-health coverage pass. `savings-backfill` mines local Claude/Codex transcripts into a replaceable historical discovery snapshot: raw prompts, responses, paths, shell values, and tool arguments are excluded, while keyed intent fingerprints plus sanitized shell templates/sequences/n-grams retain repeated workflow structure. Historical frequency can nominate automation experiments but can never satisfy the prospective evidence gate. @@ -23,7 +23,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/). - **`roam at` ranges and whole-symbol reads** — `roam at FILE:START-END` replaces arbitrary `sed` windows, while `roam at FILE:LINE --whole-symbol` expands a search hit to its complete indexed function/class. `--max-lines` bounds output and reports truncation instead of silently dropping source. ### Changed -- **The canonical pre-push gate now has an explicit bounded worker budget** — full-suite pytest execution defaults to at most four workers, accepts a validated `--workers 1..4` override, and schedules by file with `--dist loadfile`. This prevents a large host from translating CPU count into destructive memory and temp-directory contention while retaining deterministic parallel release coverage. +- **The canonical pre-push gate now has an explicit bounded concurrency budget** — full-suite pytest execution defaults to at most four workers, accepts a validated `--workers 1..4` override, schedules by file with `--dist loadfile`, and forces native BLAS/OpenMP runtimes to one thread per worker. This prevents a large host—or poisoned inherited thread settings—from multiplying a bounded worker pool into destructive CPU, memory, and temp-directory contention while retaining deterministic parallel release coverage. - **CI and the reusable action now resolve only reviewed dependency inputs** — every source-checkout workflow environment is created from the exact `uv.lock`, release-only tools install from hash-locked wheelhouses, dependency auditing checks the installed locked environment without re-resolving it, and every third-party action is pinned by full commit SHA. The composite action validates all caller-controlled inputs before they reach shell or JavaScript, maps expressions through bounded environment variables, and requires an explicit opt-in before resolving the moving `latest` release. - **Generated and copy-paste CI templates inherit the same fail-closed policy** — GitHub, GitLab, Azure, Jenkins, CircleCI, and Bitbucket templates pin Roam 13.10.0; GitHub actions and supported runners are immutable, and container-based examples bind verified multi-architecture OCI digests. Required Guard, health, SLSA, and SARIF evidence fails closed; deliberately advisory scans preserve stdout, stderr, exit code, and an explicit state sidecar instead of using blanket `|| true` or `continue-on-error` fallbacks. - **Roam now owns its Claude hook maintenance authority end to end** — canonical v11 hook bodies invoke maintenance-only `verify` and `index` operations with the explicit audited mode override themselves. Compile Code no longer rewrites installed Roam source or hook bodies, so one signed Roam artifact defines the hook behavior on every client. @@ -32,6 +32,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/). - **Generated constitution upgrades are provenance-bound** — fresh `constitution init` files record a semantic digest for the generated mode snapshot. New defaults flow into runtime policy only while that snapshot remains unchanged; customized and legacy policies stay authoritative. `roam constitution check` now exposes migration state, while `roam constitution upgrade` previews per-mode changes and requires both explicit replacement acknowledgement and the preview's expected digest before replacing an unproven policy. ### Fixed +- **Prerelease self-analysis no longer depends on an artifact that cannot exist yet** — the composite action now has an explicit `version: source` mode that installs from its validated action checkout, requires matching non-editable PEP 610 provenance, and never falls back to PyPI. Roam's own PR self-analysis uses this mode, while downstream defaults remain pinned to the exact published release. +- **Interrupted indexing can no longer masquerade as a complete graph** — every index run acquires a held OS lock and publishes the same PID, process-start identity (where available), and random generation in its lock and durably synced `in_progress` marker before the first SQLite mutation. Recovery never unlinks a lock pathname, release only unlocks the exact owning descriptor, and consumers fail closed on unprovable ownership while accepting a completed marker despite stale PID reuse. A proven interrupted generation or stale pre-marker numeric lock is carried by the claim and overrides every direct caller to `force=True, light=False`; watch and Verify refreshes cannot certify a partial DB through an incremental/light run. Genuinely complete legacy DBs and released legacy locks retain normal incremental behavior. `complete` is published only after SQLite checkpoint/fsync and a durable marker rename. This closes the crash/OOM case where a present-but-partial database returned plausible empty results and caused Compile probes to re-derive answers. +- **PR Replay preserves honest verification context on clean checkouts** — every successful replay carries a small deterministic report-generation manifest, so evidence coverage distinguishes “the replay producer ran, but no verification proof exists” from “verification was never considered.” The manifest earns only partial Q7 coverage; it never fabricates `tests_run` or upgrades the packet to verified. - **Python 3.14 no longer strips MCP alias annotations at the concurrency boundary** — Python 3.14 stopped copying a runtime-assigned `__annotations__` mapping through `functools.wraps`, leaving FastMCP/Pydantic with a synthesized signature but no matching type hints and crashing server import with `KeyError: 'symbol'`. The guard now preserves a private copy of realized annotations for sync and async tools; direct Pydantic schema generation and FastMCP 3.4 registration are regression-tested without constraining FastMCP to an older major. - **PR Replay policy assurance is deterministic on fresh checkouts** — every successful replay now records the selected current-detector-set policy as a typed `not_evaluated` decision, explicitly avoiding a fabricated pass/fail result. Constitution, rule, permit, lease, audit-trail, and review decisions remain additive, while Q6 no longer depends on incidental local `.roam/` history. - **MCP receipts and signed run events cannot cross repository roots** — the dispatcher snapshots one canonical invocation root and active run before execution, then binds policy, receipt placement, and HMAC linkage to that immutable context even if CWD or `ROAM_RUN_ID` changes during the call. Foreign, inactive, malformed, hard-linked, symlinked, NTFS-junction, redirected, or concurrently replaced receipt/metadata/event/key paths fail closed. Ledger appends are inode-pinned, and POSIX receipt replacement uses directory handles so a detached-parent race cannot strand evidence outside the verified tree. diff --git a/README.md b/README.md index bfdb52fcd..92c5768ae 100644 --- a/README.md +++ b/README.md @@ -341,7 +341,7 @@ refactors, never lose entries). ## What's New -**v13.10 (2026-07-19) — repeated work becomes measurable procedures, and post-edit verification becomes proof-complete.** Privacy-preserving transcript/shell-template mining can nominate repeated-work interventions without exposing raw prompts or claiming causal savings; `roam savings` promotes only prospectively joined, integrity-checked outcomes. The Claude adapter now binds every edited turn to a strict Verify receipt and blocks unavailable, malformed, incomplete, or failing evidence. Roam owns the canonical hooks end to end—Compile Code no longer rewrites installed source. Full notes: [CHANGELOG.md](CHANGELOG.md). +**v13.10 (2026-07-23) — repeated work becomes measurable procedures, and post-edit verification becomes proof-complete.** Privacy-preserving transcript/shell-template mining can nominate repeated-work interventions without exposing raw prompts or claiming causal savings; `roam savings` promotes only prospectively joined, integrity-checked outcomes. The Claude adapter now binds every edited turn to a strict Verify receipt and blocks unavailable, malformed, incomplete, or failing evidence. Interrupted indexes carry a generation-bound, durably synced lifecycle marker and force a full non-light rebuild before analysis regardless of the direct caller; completion is published only after SQLite checkpoint/fsync, so a crash cannot turn partial graph state into plausible empty answers. Roam owns the canonical hooks end to end—Compile Code no longer rewrites installed source. Full notes: [CHANGELOG.md](CHANGELOG.md).
Earlier release notes — v13.6 → v13.0 diff --git a/action.yml b/action.yml index e0dc2bb47..ab31b9faa 100644 --- a/action.yml +++ b/action.yml @@ -6,7 +6,7 @@ branding: inputs: version: - description: 'Exact roam-code version to install; use latest only with allow-latest=true' + description: 'Exact roam-code version to install; source installs this action checkout; latest requires allow-latest=true' required: false default: '13.10.0' allow-latest: @@ -190,7 +190,11 @@ runs: validate_scalar "version" "${INPUT_VERSION}" 64 version_re='^[0-9]+(\.[0-9]+){1,3}((a|b|rc)[0-9]+)?(\.post[0-9]+)?(\.dev[0-9]+)?$' - if [[ "${INPUT_VERSION}" == "latest" ]]; then + if [[ "${INPUT_VERSION}" == "source" ]]; then + if [[ "${INPUT_ALLOW_LATEST}" != "false" ]]; then + fail_input "version" "source requires allow-latest=false" + fi + elif [[ "${INPUT_VERSION}" == "latest" ]]; then if [[ "${INPUT_ALLOW_LATEST}" != "true" ]]; then fail_input "version" "latest requires allow-latest=true" fi @@ -338,22 +342,69 @@ runs: env: ROAM_VERSION: ${{ steps.validate-inputs.outputs.version }} ALLOW_LATEST: ${{ steps.validate-inputs.outputs.allow-latest }} + ROAM_ACTION_PATH: ${{ steps.validate-inputs.outputs.action-path }} run: | set -euo pipefail - if [[ "${ROAM_VERSION}" == "latest" ]]; then - if [[ "${ALLOW_LATEST}" != "true" ]]; then - echo "::error::Mutable latest install was not explicitly authorized" >&2 - exit 2 - fi - python -m pip install --quiet --disable-pip-version-check roam-code - else - python -m pip install --quiet --disable-pip-version-check "roam-code==${ROAM_VERSION}" - INSTALLED_VERSION=$(python -c 'from importlib.metadata import version; print(version("roam-code"))') - if [[ "${INSTALLED_VERSION}" != "${ROAM_VERSION}" ]]; then - echo "::error::Installed roam-code ${INSTALLED_VERSION}; expected ${ROAM_VERSION}" >&2 - exit 1 - fi - fi + case "${ROAM_VERSION}" in + source) + if [[ "${ALLOW_LATEST}" != "false" ]]; then + echo "::error::Source install must not enable the mutable latest path" >&2 + exit 2 + fi + if [[ ! -f "${ROAM_ACTION_PATH}/pyproject.toml" ]]; then + echo "::error::Source install requires pyproject.toml in the validated action checkout" >&2 + exit 1 + fi + python -m pip install --quiet --disable-pip-version-check --no-cache-dir --force-reinstall "${ROAM_ACTION_PATH}" + python - <<'PY' + import importlib.metadata + import json + import os + from pathlib import Path + from urllib.parse import urlparse + from urllib.request import url2pathname + + expected = Path(os.environ["ROAM_ACTION_PATH"]).resolve(strict=True) + distribution = importlib.metadata.distribution("roam-code") + installed_name = str(distribution.metadata.get("Name", "")).lower().replace("_", "-") + if installed_name != "roam-code": + raise SystemExit(f"installed unexpected distribution name: {installed_name!r}") + raw_direct_url = distribution.read_text("direct_url.json") + if not raw_direct_url: + raise SystemExit("source install produced no PEP 610 direct_url.json") + direct_url = json.loads(raw_direct_url) + parsed = urlparse(direct_url.get("url", "")) + if parsed.scheme != "file" or parsed.netloc not in ("", "localhost"): + raise SystemExit("source install provenance is not a local file URL") + if parsed.params or parsed.query or parsed.fragment: + raise SystemExit("source install provenance contains unexpected URL components") + actual = Path(url2pathname(parsed.path)).resolve(strict=True) + if actual != expected: + raise SystemExit(f"source install provenance mismatch: {actual} != {expected}") + dir_info = direct_url.get("dir_info") + if not isinstance(dir_info, dict): + raise SystemExit("source install provenance has no directory information") + editable = dir_info.get("editable") + if editable is not None and editable is not False: + raise SystemExit("source install unexpectedly used editable mode") + PY + ;; + latest) + if [[ "${ALLOW_LATEST}" != "true" ]]; then + echo "::error::Mutable latest install was not explicitly authorized" >&2 + exit 2 + fi + python -m pip install --quiet --disable-pip-version-check roam-code + ;; + *) + python -m pip install --quiet --disable-pip-version-check "roam-code==${ROAM_VERSION}" + INSTALLED_VERSION=$(python -c 'from importlib.metadata import version; print(version("roam-code"))') + if [[ "${INSTALLED_VERSION}" != "${ROAM_VERSION}" ]]; then + echo "::error::Installed roam-code ${INSTALLED_VERSION}; expected ${ROAM_VERSION}" >&2 + exit 1 + fi + ;; + esac python -m pip check # ---- 6. Build/refresh index --------------------------------------------- diff --git a/docs/ci-integration.md b/docs/ci-integration.md index 06c30faf9..0682f8441 100644 --- a/docs/ci-integration.md +++ b/docs/ci-integration.md @@ -111,7 +111,7 @@ uploads SARIF findings to GitHub Code Scanning, and enforces quality gates. | Input | Default | Description | |-------|---------|-------------| -| `version` | `13.10.0` | Exact roam-code version to install from PyPI. Accepts a closed PEP 440-style release grammar; URLs, VCS references, pip options, whitespace, and local-version suffixes are rejected. | +| `version` | `13.10.0` | Exact roam-code version to install from PyPI. The explicit `source` mode installs from the selected action checkout; `latest` requires `allow-latest: true`. URLs, VCS references, pip options, whitespace, and local-version suffixes are rejected. | | `allow-latest` | `false` | Explicit opt-in required when `version: latest` is requested. This keeps the mutable package path visible in review. | | `commands` | `health` | Space-separated roam commands to run. Each command produces JSON output that feeds into the PR comment and quality gate. | | `changed-only` | `false` | Incremental CI mode. Adapts supported commands to changed files and transitive dependents (when detectable). | @@ -135,6 +135,11 @@ ranges, because a copied downstream action cannot consume this repository's check out roam-code and materialize `uv.lock`, as Roam's own source workflows do. `version: latest` trades that release-level reproducibility for automatic first-party upgrades and therefore requires `allow-latest: 'true'`. +`version: source` is intended for prerelease validation of a selected action +checkout. It installs only from the action's validated `github.action_path`, +requires matching non-editable PEP 610 provenance, and has no PyPI fallback. +It binds first-party Roam code to that checkout, while transitive dependencies +still follow declared package ranges rather than this repository's `uv.lock`. ## Outputs diff --git a/scripts/prepush_check.py b/scripts/prepush_check.py index 55330b5f1..9601074fd 100644 --- a/scripts/prepush_check.py +++ b/scripts/prepush_check.py @@ -146,6 +146,17 @@ def repo_root() -> Path: ) _MAX_PYTEST_WORKERS = 4 +_NATIVE_THREAD_ENV = ( + "BLIS_NUM_THREADS", + "GOTO_NUM_THREADS", + "MKL_NUM_THREADS", + "NUMEXPR_NUM_THREADS", + "OMP_NUM_THREADS", + "OMP_THREAD_LIMIT", + "OPENBLAS_NUM_THREADS", + "VECLIB_MAXIMUM_THREADS", +) +_NATIVE_THREADS_PER_WORKER = "1" _GIB = 1024**3 _MIN_RELEASE_TEMP_FREE_GIB = 4 _MIN_RELEASE_TEMP_FREE_GIB_PER_WORKER = 2 @@ -191,6 +202,47 @@ class GateRunner: def _env(self) -> dict[str, str]: env = os.environ.copy() + # Git invokes hooks with repository-local control variables such as + # GIT_INDEX_FILE. Pytest gates create foreign repositories and run + # `git add` inside them; forwarding an outer linked-worktree index + # redirects those writes into the real repository. Ask Git for its + # authoritative local-variable vocabulary and remove it at the + # subprocess boundary. Keep this defense even though the shell hook + # sanitizes too: prepush_check.py is also invoked directly. + local_vars = { + "GIT_ALTERNATE_OBJECT_DIRECTORIES", + "GIT_COMMON_DIR", + "GIT_CONFIG", + "GIT_CONFIG_PARAMETERS", + "GIT_DIR", + "GIT_GRAFT_FILE", + "GIT_IMPLICIT_WORK_TREE", + "GIT_INDEX_FILE", + "GIT_OBJECT_DIRECTORY", + "GIT_REPLACE_REF_BASE", + "GIT_SHALLOW_FILE", + "GIT_WORK_TREE", + } + try: + proc = subprocess.run( + ["git", "-C", str(self.root), "rev-parse", "--local-env-vars"], + capture_output=True, + text=True, + check=False, + timeout=5, + ) + if proc.returncode == 0: + local_vars.update(name.strip() for name in proc.stdout.splitlines() if name.strip()) + except (FileNotFoundError, subprocess.TimeoutExpired, OSError): + pass + for name in local_vars: + env.pop(name, None) + # Keep --workers as the total concurrency budget. NumPy/SciPy may load + # a native math runtime in each xdist worker; inherited host-sized + # defaults otherwise multiply a bounded worker pool into dozens of + # threads. Override poisoned outer values as well as absent defaults. + for name in _NATIVE_THREAD_ENV: + env[name] = _NATIVE_THREADS_PER_WORKER src = str(self.root / "src") current = env.get("PYTHONPATH") env["PYTHONPATH"] = src if not current else f"{src}{os.pathsep}{current}" @@ -367,6 +419,7 @@ def main(argv: list[str] | None = None) -> int: print(f"[prepush] repo root: {root}") print(f"[prepush] tier: {'RELEASE' if release else 'FULL' if full else 'FAST'}") print(f"[prepush] pytest workers: {args.workers} (loadfile distribution)") + print(f"[prepush] native math threads per worker: {_NATIVE_THREADS_PER_WORKER}") runner = GateRunner(root=root, pytest_workers=args.workers) if release and not runner.run_release_temp_capacity_gate().passed: diff --git a/scripts/seed_hook_body_shas.py b/scripts/seed_hook_body_shas.py index 1a26f5534..ea0eb01d4 100644 --- a/scripts/seed_hook_body_shas.py +++ b/scripts/seed_hook_body_shas.py @@ -46,8 +46,25 @@ def surgered(script: str) -> str: ) +def shipped_variants(body: str, version: str): + """Yield variants that could actually have been deployed at ``version``.""" + yield "pristine", body + legacy_surgery = version == "pre-stamp" or (version.isdigit() and int(version) < 11) + if legacy_surgery: + transformed = surgered(body) + if transformed != body: + yield "surgered", transformed + + def git(*args: str) -> str: - return subprocess.run(["git", "-C", str(REPO), *args], capture_output=True, text=True, check=True).stdout + return subprocess.run( + ["git", "-C", str(REPO), *args], + capture_output=True, + text=True, + encoding="utf-8", + errors="strict", + check=True, + ).stdout def bodies_at(commit: str) -> dict[str, str] | None: @@ -93,9 +110,7 @@ def main() -> int: continue ver = found.pop("_version", "pre-stamp") for kind, body in found.items(): - for variant, text in (("pristine", body), ("surgered", surgered(body))): - if variant == "surgered" and text == body: - continue # surgery is a no-op on this body + for variant, text in shipped_variants(body, ver): sha = hashlib.sha256(text.encode("utf-8")).hexdigest() if sha not in seen: label = f"v{ver}" if ver != "pre-stamp" else "pre-stamp" diff --git a/scripts/verify_release.py b/scripts/verify_release.py index 7633d57e4..39aac6e4e 100644 --- a/scripts/verify_release.py +++ b/scripts/verify_release.py @@ -5,8 +5,8 @@ after a release) can verify the full surface: 1. **PyPI freshness** — the latest published version on pypi.org matches - ``pyproject.toml`` (or is at most one ahead, allowing for the brief - window between bumping pyproject and publishing the tag). + ``pyproject.toml`` (or the repository is exactly one patch or minor + release ahead, allowing for the brief pre-publish window). 2. **Install smoke** — ``pip install roam-code==`` into an ephemeral venv works and ``roam --version`` reports the expected string. @@ -61,8 +61,17 @@ def _pypi_latest() -> str | None: return data.get("info", {}).get("version") +def _final_release_tuple(version: str) -> tuple[int, int, int] | None: + """Normalize final ``major.minor[.patch]`` versions without extra deps.""" + match = re.fullmatch(r"(\d+)\.(\d+)(?:\.(\d+))?", version) + if match is None: + return None + major, minor, patch = match.groups() + return int(major), int(minor), int(patch or 0) + + def check_pypi_freshness() -> bool: - """The PyPI latest version is no more than one minor behind pyproject. + """PyPI matches pyproject or is exactly one patch/minor release behind. Allows pyproject to be one ahead (the brief window between bumping + tagging vs the publish workflow finishing). Anything further is a @@ -73,17 +82,24 @@ def check_pypi_freshness() -> bool: if pypi_v is None: print("FAIL: PyPI unreachable; cannot verify freshness", file=sys.stderr) return False - if repo_v == pypi_v: - print(f"OK: pyproject and PyPI both at {repo_v}") + repo_parts = _final_release_tuple(repo_v) + pypi_parts = _final_release_tuple(pypi_v) + if repo_parts is None or pypi_parts is None: + print( + f"FAIL: cannot compare non-final versions pyproject={repo_v}, PyPI={pypi_v}", + file=sys.stderr, + ) + return False + if repo_parts == pypi_parts: + print(f"OK: pyproject={repo_v} and PyPI={pypi_v} normalize to the same final release") return True - - repo_parts = [int(x) for x in repo_v.split(".") if x.isdigit()] - pypi_parts = [int(x) for x in pypi_v.split(".") if x.isdigit()] - if repo_parts > pypi_parts: + same_line_next_patch = repo_parts[:2] == pypi_parts[:2] and repo_parts[2] == pypi_parts[2] + 1 + next_minor = repo_parts[0] == pypi_parts[0] and repo_parts[1] == pypi_parts[1] + 1 and repo_parts[2] == 0 + if same_line_next_patch or next_minor: print(f"OK: pyproject={repo_v} ahead of PyPI={pypi_v} (pre-publish window)") return True print( - f"FAIL: pyproject={repo_v} BEHIND PyPI={pypi_v} — repo is stale relative to released package", + f"FAIL: pyproject={repo_v} and PyPI={pypi_v} differ by more than one publish step", file=sys.stderr, ) return False diff --git a/src/roam/atomic_io.py b/src/roam/atomic_io.py index 412709bce..e56357be1 100644 --- a/src/roam/atomic_io.py +++ b/src/roam/atomic_io.py @@ -1239,9 +1239,21 @@ def conditional_install_file( os.close(parent_fd) -def atomic_write_text(path: PathLike, content: str, *, encoding: str = "utf-8") -> None: - """Encode *content* and use the shared conditional byte installer.""" - atomic_write_bytes(path, content.encode(encoding)) +def atomic_write_text( + path: PathLike, + content: str, + *, + encoding: str = "utf-8", + durable: bool = False, +) -> None: + """Encode *content* and use the shared conditional byte installer. + + ``durable=True`` fsyncs the completed file and, where the platform + supports it, its parent directory. Text callers that publish lifecycle + or authority state can therefore use the same crash boundary as binary + artifacts without bypassing the conditional installer. + """ + atomic_write_bytes(path, content.encode(encoding), durable=durable) def atomic_write_bytes( diff --git a/src/roam/commands/cmd_compile.py b/src/roam/commands/cmd_compile.py index b0415b584..158422555 100644 --- a/src/roam/commands/cmd_compile.py +++ b/src/roam/commands/cmd_compile.py @@ -269,12 +269,12 @@ def _artifact_for_request(plan, artifact: str, cwd: str) -> tuple[dict, str]: if artifact == "auto": return compile_for_artifact(plan, cwd=cwd) if artifact == "facts": - return plan.to_facts_envelope(), "facts" + return plan.to_facts_envelope(cwd=cwd), "facts" if artifact == "lean": - return plan.to_lean_envelope(), "lean" + return plan.to_lean_envelope(cwd=cwd), "lean" if artifact == "contract": - return plan.to_facts_contract_envelope(), "contract" - return plan.to_envelope(), "full" + return plan.to_facts_contract_envelope(cwd=cwd), "contract" + return plan.to_envelope(cwd=cwd), "full" def _compile_verify_hint(cwd: str) -> str | None: diff --git a/src/roam/commands/cmd_hooks.py b/src/roam/commands/cmd_hooks.py index 552531619..59518d090 100644 --- a/src/roam/commands/cmd_hooks.py +++ b/src/roam/commands/cmd_hooks.py @@ -558,7 +558,10 @@ def status(ctx): # as safely outside the repository. ``os.path.commonpath`` raises ValueError # across drive letters; v11 accidentally collapsed that safe case into an # unavailable evidence root and dropped episode identity. -_HOOK_BODY_VERSION = 12 +# v13 (2026-07-20): bind receipt target counts to ``targets_checked`` rather +# than indexed ``files_checked`` so deleted and non-code episode targets retain +# exact receipt coverage. Deployed v12 bodies heal to the strict v13 contract. +_HOOK_BODY_VERSION = 13 _HOOK_VERSION_MARKER = "# roam-hook-version:" _CLAUDE_UPS_HOOK_FILENAME = "roam-compile-ups.py" @@ -913,7 +916,7 @@ def _start_episode(payload, prompt): _release_lock(lock_path, fd) event = { "schema_version": _EPISODE_SCHEMA, - "hook_version": 12, + "hook_version": 13, "evidence_source": "live_hook", "event_id": "evt_" + hashlib.sha256((episode_id + ":start").encode("utf-8")).hexdigest()[:24], "episode_id": episode_id, @@ -1161,7 +1164,7 @@ def _claude_hook_dir(user_level: bool) -> Path: "llm_smells", "test_hermeticity", "smells", }) _EPISODE_SCHEMA = 1 -_EPISODE_HOOK_VERSION = 12 +_EPISODE_HOOK_VERSION = 13 _EPISODE_LOCK_ATTEMPTS = 20 _EPISODE_LOCK_SLEEP_S = 0.005 _EPISODE_LOCK_STALE_S = 30.0 @@ -1910,6 +1913,7 @@ def _verify_protocol_state(envelope, expected_receipt): verdict = str(summary.get("verdict") or "").strip().upper().split(":", 1)[0].split(None, 1)[0] issue_count = summary.get("violation_count") files_checked = summary.get("files_checked") + targets_checked = summary.get("targets_checked") violations = envelope.get("violations") returncode = envelope.get("_hook_process_returncode") receipt = summary.get("verification_receipt") @@ -1919,6 +1923,8 @@ def _verify_protocol_state(envelope, expected_receipt): return "unavailable" if type(files_checked) is not int or files_checked < 0: return "unavailable" + if type(targets_checked) is not int or targets_checked < 0: + return "unavailable" if not isinstance(violations, list) or len(violations) != issue_count: return "unavailable" if any(not isinstance(item, dict) for item in violations): @@ -1952,7 +1958,7 @@ def _verify_protocol_state(envelope, expected_receipt): or receipt.get("target_file_count") != expected_receipt.get("target_file_count") or receipt.get("scope_stable") is not True or receipt.get("request_match") is not True - or files_checked != expected_receipt.get("target_file_count") + or targets_checked != expected_receipt.get("target_file_count") ): return "unavailable" has_fail_finding = any( @@ -2517,6 +2523,15 @@ def _with_version_stamp(body: str) -> str: # defect that shipped in #77). _KNOWN_HOOK_BODY_SHAS: frozenset[str] = frozenset( { + "25e552061e4737bac27cbd547cade4189c84f207b2f466d410e1d036a1b1f1ca", # ups v12 pristine (2026-07-19 cross-volume evidence) + "4df987f04f024d36867c608ce971bf4a6c18b36b36a1a11babf9f569eb36e9e3", # ups v12 pristine (2026-07-19 evidence hardening) + "d6521a89e559fb875e2d949f2a13e9710aa378edd505b688362c07137bb1e0d1", # stop v12 pristine (2026-07-19 cross-volume evidence) + "d91b18607d6175b4aa90023172f52663a6c8f3d16a714f78615a559d2913e7ef", # ups v2 pristine (2026-07-16 rewritten history) + "f83329cc80eed97f62b388c2a5ee8c8e81118475dc04b04eca6cc2fac28fac1b", # stop pre-stamp pristine (2026-06-10 rewritten history) + "fa249e6a5bd660cd9d6592bab09321552d7c8bc4e01206ffec724ee0d8e85904", # stop pre-stamp pristine (2026-06-11 rewritten history) + "c15f23363571215a5d168d150ba650ca1a6c2cf1480573957bcb9eaadbdfcc3f", # ups pre-stamp pristine (2026-07-11 rewritten history) + "feab198c56d960eb310746712cc170bdbe60a38c78f0508095f1d9ebca9dbdb9", # ups pre-stamp pristine (2026-07-16 rewritten history) + "5c0fde3d99b0d23474d142016487ad0c6e4fb74a978cc113f274fd02a99aef08", # ups pre-stamp pristine (2026-07-16 rewritten history) "6ab4f91dc4c407c2fa82cb4e51bb5fd5268563150598650e4928618834a59a04", # ups v11 pristine (canonical override ownership) "391fd8b9044037b20000fc301415137647b51f7bf60bfcf35a84a05ec94a3bb8", # stop v11 pristine (canonical override ownership) "62e71df1b62b44860b5eccb181022bb5d1dfa1a751d5c0593c787c99d13e59ed", # ups v10 pristine (2026-07-17 receipt-v3 identity binding) diff --git a/src/roam/commands/cmd_pr_replay.py b/src/roam/commands/cmd_pr_replay.py index 1cd0299cf..b3c69c2ce 100644 --- a/src/roam/commands/cmd_pr_replay.py +++ b/src/roam/commands/cmd_pr_replay.py @@ -2050,6 +2050,7 @@ def _collect_change_evidence( from roam.evidence import ( EVIDENCE_SCHEMA_VERSION, ChangeEvidence, # noqa: F401 — re-exported for downstream tests + EvidenceArtifact, EvidenceSubject, collect_change_evidence, ) @@ -2259,6 +2260,43 @@ def _collect_change_evidence( if detector_aliases_changed: packet = dataclasses.replace(packet, findings=tuple(detector_aliased_findings)) + # A PR Replay invocation always produces a report-generation context, + # even when the optional test-impact, run-ledger, and CGA producers are + # unavailable in a clean checkout. Preserve that real producer signal as + # a small manifest artifact. It deliberately does NOT claim that tests + # ran: ``manifest`` is generic context, so evidence_completeness scores + # Q7 as ``partial`` unless a real tests_run row or verification-shaped + # artifact is also present. This keeps "producer ran, no verification + # proof" distinct from "verification was never considered" without + # manufacturing test evidence. + report_manifest = _json.dumps( + { + "producer": "pr-replay", + "git_range": commit_range, + "generated_at": generated_at, + "commit_count": len(commits), + "detector_count": len(by_detector), + "verdict": summary.get("verdict") or "no verdict", + "total_high": int(summary.get("total_high", 0) or 0), + "total_medium": int(summary.get("total_medium", 0) or 0), + }, + sort_keys=True, + separators=(",", ":"), + ) + report_manifest_id = _hashlib.sha256(report_manifest.encode("utf-8")).hexdigest()[:16] + packet = dataclasses.replace( + packet, + artifacts=tuple(packet.artifacts) + + ( + EvidenceArtifact( + artifact_id=f"manifest:pr-replay:{report_manifest_id}", + kind="manifest", + content_inline=report_manifest, + extra={"producer": "pr-replay", "assurance_role": "report_context"}, + ), + ), + ) + # Stable per-(range, generation moment) evidence_id — overrides the # collector-derived one so the value stays human-readable for tickets. packet = dataclasses.replace( diff --git a/src/roam/commands/cmd_service_report.py b/src/roam/commands/cmd_service_report.py index 9fe151087..d5acc0d6f 100644 --- a/src/roam/commands/cmd_service_report.py +++ b/src/roam/commands/cmd_service_report.py @@ -932,7 +932,7 @@ def _start_component_process( _os.set_inheritable(status_read_fd, False) _os.set_inheritable(status_write_fd, False) process_argv = [ - _os.path.realpath(_sys.executable), + _sys.executable, "-c", _LINUX_COMPONENT_WRAPPER, str(status_write_fd), diff --git a/src/roam/commands/resolve.py b/src/roam/commands/resolve.py index ff2bab558..29fac92f0 100644 --- a/src/roam/commands/resolve.py +++ b/src/roam/commands/resolve.py @@ -176,6 +176,76 @@ def index_status() -> dict | None: _MAX_FTS_SUGGESTIONS = 5 +def _existing_index_recovery_state() -> str | None: + """Return ``active`` or ``incomplete`` for an unsafe existing index. + + Older Roam versions did not publish ``index.state``. Preserve those + completed indexes as valid by default, while treating a stale numeric + ``index.lock`` as evidence that an old indexer died mid-run. New indexers + publish a generation-bound owner before mutating SQLite and publish + completion only after the database crosses its durability barrier. + """ + try: + root = find_project_root() + except Exception as exc: # noqa: BLE001 — project discovery is best-effort + from roam.observability import log_swallowed + + log_swallowed("resolve:index_recovery_state:find_project_root", exc) + return None + + roam_dir = root / ".roam" + lock_path = roam_dir / "index.lock" + state_path = roam_dir / "index.state" + + from roam.index.indexer import ( + _decode_index_lock, + _decode_index_state, + _IndexOwner, + _owner_liveness, + _pid_is_running, + _read_control_text, + ) + + try: + state_raw = _read_control_text(state_path) + except (OSError, UnicodeError): + return "incomplete" + state_kind, state_owner = _decode_index_state(state_raw) + + # Completion is authoritative over a leftover lock record. A new writer + # must durably replace this marker before its first database mutation, so + # a stale PID (including a reused live PID) cannot block a completed DB. + if state_kind == "complete": + return None + if state_kind == "in_progress" and isinstance(state_owner, _IndexOwner): + liveness = _owner_liveness(state_owner) + return "incomplete" if liveness is False else "active" + if state_kind == "legacy_in_progress" and isinstance(state_owner, int): + return "active" if _pid_is_running(state_owner) else "incomplete" + if state_kind == "unknown": + return "incomplete" + + try: + lock_raw = _read_control_text(lock_path) + except (OSError, UnicodeError): + # An unreadable lock cannot prove that mutation has stopped. + return "active" + lock_kind, lock_owner = _decode_index_lock(lock_raw) + if lock_kind == "owned" and isinstance(lock_owner, _IndexOwner): + liveness = _owner_liveness(lock_owner) + return "incomplete" if liveness is False else "active" + if lock_kind == "legacy" and isinstance(lock_owner, int): + return "active" if _pid_is_running(lock_owner) else "incomplete" + if lock_kind == "unknown": + return "incomplete" + + # Backward-compatible crash recovery for pre-marker versions: a dead + # numeric lock alongside an existing DB is the durable evidence that the + # writer did not reach its normal cleanup path. ``released`` is the + # documented successful fallback when lock deletion is unavailable. + return None + + def ensure_index(quiet: bool = False, suppress_cold_start_advisory: bool = False) -> None: """Build the index if it doesn't exist yet. @@ -187,17 +257,34 @@ def ensure_index(quiet: bool = False, suppress_cold_start_advisory: bool = False user just asked to create the index, so recommending they run the command they're already running is confusing first-time UX (W1291). """ - if not db_exists(): + index_exists = db_exists() + recovery_state = _existing_index_recovery_state() if index_exists else None + if recovery_state == "active": + raise click.ClickException( + "The roam index is currently being built by another process. Retry after that index run completes." + ) + if not index_exists or recovery_state == "incomplete": if not quiet and not suppress_cold_start_advisory: - click.echo( - "No roam index found. Run `roam init` to create one.\n" - " Tip: If you already ran `roam init`, your current directory may be\n" - " outside the project root. cd into the project root and retry.\n" - " If this looks unexpected, run `roam doctor` to diagnose your install." - ) + if recovery_state == "incomplete": + click.echo("Incomplete roam index detected after an interrupted build. Rebuilding it before analysis.") + else: + click.echo( + "No roam index found. Run `roam init` to create one.\n" + " Tip: If you already ran `roam init`, your current directory may be\n" + " outside the project root. cd into the project root and retry.\n" + " If this looks unexpected, run `roam doctor` to diagnose your install." + ) from roam.index.indexer import Indexer - Indexer().run(quiet=quiet) + indexer = Indexer() + if recovery_state == "incomplete": + completed = indexer.run(force=True, quiet=quiet) + else: + completed = indexer.run(quiet=quiet) + if completed is False: + raise click.ClickException( + "The roam index could not be claimed for a complete build. Retry after the active indexer exits." + ) def require_index() -> None: diff --git a/src/roam/index/indexer.py b/src/roam/index/indexer.py index 87742e44f..cb1e65efa 100644 --- a/src/roam/index/indexer.py +++ b/src/roam/index/indexer.py @@ -4,15 +4,21 @@ import contextlib import importlib +import json import logging import os +import secrets +import sqlite3 +import stat import sys import time from collections.abc import Callable +from dataclasses import dataclass from pathlib import Path log = logging.getLogger(__name__) +from roam.atomic_io import atomic_write_text from roam.db.connection import batched_in, find_project_root, get_db_path, open_db from roam.index.discovery import discover_files @@ -140,12 +146,15 @@ def _log(msg: str): def _pid_is_running(pid: int) -> bool: """Return True when *pid* appears to refer to a live process. - On Windows, probing a process owned by another integrity level can raise - ``PermissionError``. Treat that as "running or inaccessible" so we do not - delete another process's lock by mistake. + Windows must not use ``os.kill(pid, 0)``: signal zero maps to a console + control event there and can interrupt the caller's process group. Native + process-handle inspection is side-effect free. Inaccessible Windows + processes are treated as live so recovery fails closed. """ if pid <= 0: return False + if os.name == "nt": + return _windows_pid_is_running(pid) try: os.kill(pid, 0) except ProcessLookupError: @@ -157,57 +166,536 @@ def _pid_is_running(pid: int) -> bool: return True -def _claim_index_lock(lock_path: Path) -> bool: - """Claim the index lock, tolerating stale locks that cannot be deleted.""" - lock_path.parent.mkdir(exist_ok=True) - if lock_path.exists(): +def _windows_pid_is_running(pid: int) -> bool: + """Side-effect-free Windows liveness probe using a process handle.""" + try: + import ctypes + from ctypes import wintypes + + kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) + kernel32.OpenProcess.argtypes = (wintypes.DWORD, wintypes.BOOL, wintypes.DWORD) + kernel32.OpenProcess.restype = wintypes.HANDLE + kernel32.GetExitCodeProcess.argtypes = (wintypes.HANDLE, ctypes.POINTER(wintypes.DWORD)) + kernel32.GetExitCodeProcess.restype = wintypes.BOOL + kernel32.CloseHandle.argtypes = (wintypes.HANDLE,) + handle = kernel32.OpenProcess(0x1000, False, pid) # PROCESS_QUERY_LIMITED_INFORMATION + if not handle: + error = ctypes.get_last_error() + if error == 87: # ERROR_INVALID_PARAMETER: no such PID + return False + return True try: - raw_pid = lock_path.read_text().strip() - pid = int(raw_pid) - except (ValueError, OSError): - pid = 0 + exit_code = wintypes.DWORD() + if not kernel32.GetExitCodeProcess(handle, ctypes.byref(exit_code)): + return True + return int(exit_code.value) == 259 # STILL_ACTIVE + finally: + kernel32.CloseHandle(handle) + except (AttributeError, OSError, ValueError): + return True - if pid and _pid_is_running(pid): - _log(f"Another indexing process (PID {pid}) is running. Exiting.") - return False - if pid: - _log(f"Removing stale lock file (PID {pid} is not running).") +_INDEX_LIFECYCLE_VERSION = 1 +_MAX_INDEX_CONTROL_BYTES = 4096 + + +@dataclass(frozen=True) +class _IndexOwner: + """Generation-bound identity of one index writer.""" + + pid: int + process_start: str | None + generation: str + + +@dataclass +class _IndexLockClaim: + """A held kernel lock plus the ownership generation published within it.""" + + path: Path + descriptor: int + identity: tuple[int, int] + owner: _IndexOwner + recovery_required: bool + recovery_reason: str | None + locked: bool = True + + +def _process_start_identity(pid: int) -> str | None: + """Return a PID-reuse-resistant process-start identity where available.""" + if pid <= 0: + return None + if os.name == "nt": try: - lock_path.unlink() - except OSError as exc: - _log(f" Could not delete stale lock ({exc}); reusing lock file.") + import ctypes + from ctypes import wintypes + + kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) + kernel32.OpenProcess.argtypes = (wintypes.DWORD, wintypes.BOOL, wintypes.DWORD) + kernel32.OpenProcess.restype = wintypes.HANDLE + kernel32.GetProcessTimes.argtypes = ( + wintypes.HANDLE, + ctypes.POINTER(wintypes.FILETIME), + ctypes.POINTER(wintypes.FILETIME), + ctypes.POINTER(wintypes.FILETIME), + ctypes.POINTER(wintypes.FILETIME), + ) + kernel32.GetProcessTimes.restype = wintypes.BOOL + kernel32.CloseHandle.argtypes = (wintypes.HANDLE,) + handle = kernel32.OpenProcess(0x1000, False, pid) # PROCESS_QUERY_LIMITED_INFORMATION + if not handle: + return None + try: + creation = wintypes.FILETIME() + exit_time = wintypes.FILETIME() + kernel = wintypes.FILETIME() + user = wintypes.FILETIME() + if not kernel32.GetProcessTimes(handle, creation, exit_time, kernel, user): + return None + ticks = (int(creation.dwHighDateTime) << 32) | int(creation.dwLowDateTime) + return f"windows-filetime:{ticks}" + finally: + kernel32.CloseHandle(handle) + except (AttributeError, OSError, ValueError): + return None + proc_stat = Path(f"/proc/{pid}/stat") try: - lock_path.write_text(str(os.getpid())) - except OSError as exc: - _log(f"Could not claim index lock: {exc}") - _log(" If this looks unexpected, run `roam doctor` to diagnose your install.") + raw = proc_stat.read_text(encoding="ascii") + # The command name in field 2 may contain spaces and parentheses. + fields_after_name = raw[raw.rfind(")") + 2 :].split() + start_ticks = fields_after_name[19] # field 22 overall + boot_id = Path("/proc/sys/kernel/random/boot_id").read_text(encoding="ascii").strip() + if not boot_id or not start_ticks.isdigit(): + return None + return f"linux-proc:{boot_id}:{start_ticks}" + except (IndexError, OSError, UnicodeError): + return None + + +def _owner_liveness(owner: _IndexOwner) -> bool | None: + """Return True/False for a proven generation, or None when unprovable.""" + if not _pid_is_running(owner.pid): return False - return True + observed_start = _process_start_identity(owner.pid) + if owner.process_start is None or observed_start is None: + return None + return secrets.compare_digest(owner.process_start, observed_start) + + +def _owner_payload(owner: _IndexOwner) -> dict[str, object]: + return { + "generation": owner.generation, + "pid": owner.pid, + "process_start": owner.process_start, + } + + +def _serialize_lock_owner(owner: _IndexOwner) -> str: + payload = { + "kind": "roam-index-lock", + "owner": _owner_payload(owner), + "version": _INDEX_LIFECYCLE_VERSION, + } + return json.dumps(payload, sort_keys=True, separators=(",", ":")) + "\n" + + +def _serialize_index_state(state: str, owner: _IndexOwner) -> str: + payload: dict[str, object] = { + "generation": owner.generation, + "state": state, + "version": _INDEX_LIFECYCLE_VERSION, + } + if state == "in_progress": + payload["owner"] = _owner_payload(owner) + return json.dumps(payload, sort_keys=True, separators=(",", ":")) + "\n" + + +def _parse_owner(value: object) -> _IndexOwner | None: + if not isinstance(value, dict): + return None + pid = value.get("pid") + process_start = value.get("process_start") + generation = value.get("generation") + if ( + not isinstance(pid, int) + or isinstance(pid, bool) + or pid <= 0 + or (process_start is not None and not isinstance(process_start, str)) + or not isinstance(generation, str) + or len(generation) != 64 + ): + return None + try: + bytes.fromhex(generation) + except ValueError: + return None + return _IndexOwner(pid=pid, process_start=process_start, generation=generation) + + +def _decode_index_state(raw: str | None) -> tuple[str, _IndexOwner | int | None]: + """Decode current and legacy state markers without guessing.""" + if raw is None: + return "missing", None + value = raw.strip() + if value == "complete": + return "complete", None + if value.startswith("in_progress:"): + try: + return "legacy_in_progress", int(value.partition(":")[2]) + except ValueError: + return "unknown", None + try: + payload = json.loads(value) + except (json.JSONDecodeError, UnicodeError): + return "unknown", None + if not isinstance(payload, dict) or payload.get("version") != _INDEX_LIFECYCLE_VERSION: + return "unknown", None + state = payload.get("state") + generation = payload.get("generation") + if state == "complete" and isinstance(generation, str) and len(generation) == 64: + try: + bytes.fromhex(generation) + except ValueError: + return "unknown", None + return "complete", None + if state == "in_progress": + owner = _parse_owner(payload.get("owner")) + if owner is not None and owner.generation == generation: + return "in_progress", owner + return "unknown", None + + +def _decode_index_lock(raw: str | None) -> tuple[str, _IndexOwner | int | None]: + """Decode current and legacy lock records.""" + if raw is None: + return "missing", None + value = raw.strip() + if value == "released": + return "released", None + try: + return "legacy", int(value) + except ValueError: + pass + try: + payload = json.loads(value) + except (json.JSONDecodeError, UnicodeError): + return "unknown", None + if ( + isinstance(payload, dict) + and payload.get("version") == _INDEX_LIFECYCLE_VERSION + and payload.get("kind") == "roam-index-lock" + ): + owner = _parse_owner(payload.get("owner")) + if owner is not None: + return "owned", owner + return "unknown", None + + +def _read_control_text(path: Path) -> str | None: + try: + with path.open("r", encoding="utf-8") as handle: + return handle.read(_MAX_INDEX_CONTROL_BYTES + 1) + except FileNotFoundError: + return None + + +def _lock_descriptor(descriptor: int) -> bool: + """Acquire one non-blocking cross-platform kernel byte-range lock.""" + if os.name != "nt": + import fcntl + + try: + fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB) + return True + except BlockingIOError: + return False + + import ctypes + import msvcrt + from ctypes import wintypes + + class _Overlapped(ctypes.Structure): + _fields_ = [ + ("Internal", ctypes.c_void_p), + ("InternalHigh", ctypes.c_void_p), + ("Offset", wintypes.DWORD), + ("OffsetHigh", wintypes.DWORD), + ("hEvent", wintypes.HANDLE), + ] + + kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) + kernel32.LockFileEx.argtypes = ( + wintypes.HANDLE, + wintypes.DWORD, + wintypes.DWORD, + wintypes.DWORD, + wintypes.DWORD, + ctypes.POINTER(_Overlapped), + ) + kernel32.LockFileEx.restype = wintypes.BOOL + overlapped = _Overlapped() + ok = kernel32.LockFileEx( + wintypes.HANDLE(msvcrt.get_osfhandle(descriptor)), + 0x00000002 | 0x00000001, # EXCLUSIVE_LOCK | FAIL_IMMEDIATELY + 0, + 1, + 0, + ctypes.byref(overlapped), + ) + if ok: + return True + error = ctypes.get_last_error() + if error in {32, 33}: # sharing/lock violation + return False + raise ctypes.WinError(error) + + +def _unlock_descriptor(descriptor: int) -> None: + if os.name != "nt": + import fcntl + + fcntl.flock(descriptor, fcntl.LOCK_UN) + return + + import ctypes + import msvcrt + from ctypes import wintypes + + class _Overlapped(ctypes.Structure): + _fields_ = [ + ("Internal", ctypes.c_void_p), + ("InternalHigh", ctypes.c_void_p), + ("Offset", wintypes.DWORD), + ("OffsetHigh", wintypes.DWORD), + ("hEvent", wintypes.HANDLE), + ] + + kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) + kernel32.UnlockFileEx.argtypes = ( + wintypes.HANDLE, + wintypes.DWORD, + wintypes.DWORD, + wintypes.DWORD, + ctypes.POINTER(_Overlapped), + ) + kernel32.UnlockFileEx.restype = wintypes.BOOL + overlapped = _Overlapped() + if not kernel32.UnlockFileEx( + wintypes.HANDLE(msvcrt.get_osfhandle(descriptor)), + 0, + 1, + 0, + ctypes.byref(overlapped), + ): + raise ctypes.WinError(ctypes.get_last_error()) + + +def _descriptor_path_identity(descriptor: int, path: Path) -> tuple[int, int] | None: + opened = os.fstat(descriptor) + if not stat.S_ISREG(opened.st_mode) or opened.st_nlink != 1: + return None + current = os.stat(path, follow_symlinks=False) + identity = int(opened.st_dev), int(opened.st_ino) + if identity != (int(current.st_dev), int(current.st_ino)): + return None + return identity + + +def _write_locked_owner(descriptor: int, owner: _IndexOwner) -> None: + payload = _serialize_lock_owner(owner).encode("utf-8") + os.lseek(descriptor, 0, os.SEEK_SET) + written = 0 + while written < len(payload): + count = os.write(descriptor, payload[written:]) + if count <= 0: + raise OSError("short write while publishing index lock ownership") + written += count + os.ftruncate(descriptor, len(payload)) + os.fsync(descriptor) + + +def _claim_still_owned(claim: _IndexLockClaim) -> bool: + try: + if _descriptor_path_identity(claim.descriptor, claim.path) != claim.identity: + return False + os.lseek(claim.descriptor, 0, os.SEEK_SET) + raw = os.read(claim.descriptor, _MAX_INDEX_CONTROL_BYTES + 1).decode("utf-8") + except (OSError, UnicodeError): + return False + kind, owner = _decode_index_lock(raw) + return kind == "owned" and owner == claim.owner -def _release_index_lock(lock_path: Path) -> None: - """Release the index lock best-effort. +def _read_locked_control(descriptor: int) -> str: + os.lseek(descriptor, 0, os.SEEK_SET) + return os.read(descriptor, _MAX_INDEX_CONTROL_BYTES + 1).decode("utf-8") - Some Windows/cloud-sync folders allow overwrites but deny deletes. Marking - the lock as released lets the next run overwrite it without failing. + +def _recovery_requirement(state_path: Path, prior_lock_raw: str) -> tuple[bool, str | None] | None: + """Classify a held claim as normal, recovery-required, or blocked. + + ``None`` means ownership cannot be proven safe and the claim must fail + closed. A true first tuple element is carried into :meth:`Indexer.run` so + direct callers cannot accidentally complete an interrupted DB using an + incremental or light refresh. """ try: - lock_path.unlink() + raw = _read_control_text(state_path) + except (OSError, UnicodeError): + return None + state, owner = _decode_index_state(raw) + if state == "complete": + return False, None + if state == "legacy_in_progress" and isinstance(owner, int): + if owner <= 0 or _pid_is_running(owner): + return None + return True, "legacy_interrupted_state" + if state == "in_progress" and isinstance(owner, _IndexOwner): + if _owner_liveness(owner) is not False: + return None + return True, "interrupted_generation" + if state != "missing": + return None + + # Pre-marker compatibility. The descriptor is already kernel-locked, so + # this exact prior content cannot be changed by a cooperating writer while + # it is classified and replaced. Empty content is a freshly created lock; + # ``released`` is the legacy successful-cleanup marker. + if not prior_lock_raw.strip(): + return False, None + lock_kind, lock_owner = _decode_index_lock(prior_lock_raw) + if lock_kind == "released": + return False, None + if lock_kind == "legacy" and isinstance(lock_owner, int): + if lock_owner <= 0 or _pid_is_running(lock_owner): + return None + return True, "legacy_stale_lock" + if lock_kind == "owned" and isinstance(lock_owner, _IndexOwner): + if _owner_liveness(lock_owner) is not False: + return None + return True, "orphaned_lock_generation" + return None + + +def _claim_index_lock(lock_path: Path) -> _IndexLockClaim | None: + """Atomically claim the index through a held kernel lock. + + The lock pathname is never deleted during recovery or release. Stale + ownership is replaced only while this process holds the exact file + descriptor's OS lock, eliminating exists/write and unlink-successor races. + """ + lock_path.parent.mkdir(parents=True, exist_ok=True) + flags = os.O_RDWR | os.O_CREAT | getattr(os, "O_CLOEXEC", 0) + flags |= getattr(os, "O_NOFOLLOW", 0) + try: + descriptor = os.open(lock_path, flags, 0o600) + except OSError as exc: + _log(f"Could not open index lock: {exc}") + return None + locked = False + claim: _IndexLockClaim | None = None + try: + locked = _lock_descriptor(descriptor) + if not locked: + _log("Another indexing process owns the workspace. Exiting.") + return None + identity = _descriptor_path_identity(descriptor, lock_path) + if identity is None: + _log("Could not prove the index lock pathname identity. Exiting.") + return None + try: + prior_lock_raw = _read_locked_control(descriptor) + except (OSError, UnicodeError): + _log("Could not read the held index lock generation. Exiting.") + return None + recovery = _recovery_requirement(lock_path.with_name("index.state"), prior_lock_raw) + if recovery is None: + _log("An index lifecycle owner is live or cannot be proven stale. Exiting.") + return None + recovery_required, recovery_reason = recovery + owner = _IndexOwner( + pid=os.getpid(), + process_start=_process_start_identity(os.getpid()), + generation=secrets.token_hex(32), + ) + _write_locked_owner(descriptor, owner) + if _descriptor_path_identity(descriptor, lock_path) != identity: + _log("The index lock pathname changed during claim. Exiting.") + return None + claim = _IndexLockClaim( + lock_path, + descriptor, + identity, + owner, + recovery_required, + recovery_reason, + ) + except OSError as exc: + _log(f"Could not claim index lock: {exc}") + _log(" If this looks unexpected, run `roam doctor` to diagnose your install.") + finally: + if claim is None: + if locked: + with contextlib.suppress(OSError): + _unlock_descriptor(descriptor) + with contextlib.suppress(OSError): + os.close(descriptor) + return claim + + +def _release_index_lock(claim: _IndexLockClaim) -> None: + """Release only this claim's held descriptor; never mutate the pathname.""" + if not claim.locked: return - except OSError: - # Expected on Windows/cloud-sync folders that deny deletes — the - # write-text fallback below handles it. Not surfaced because the - # fallback fully recovers; only the fallback's own failure is loud. - pass try: - lock_path.write_text("released") + if not _claim_still_owned(claim): + _log("Index lock ownership changed before release; preserving the current pathname.") + _unlock_descriptor(claim.descriptor) except OSError as exc: - # Loud-fallback per CLAUDE.md §"Make fallback chains loud" — if BOTH - # unlink and write fail the lock file is now stale-and-stuck; the - # next index run will treat it as a live lock. Surface the lineage. - log_swallowed(f"index.indexer:release_lock:{lock_path}", exc) + log_swallowed(f"index.indexer:release_lock:{claim.path}", exc) + finally: + claim.locked = False + try: + os.close(claim.descriptor) + except OSError as exc: + log_swallowed(f"index.indexer:close_lock:{claim.path}", exc) + + +def _fsync_directory(path: Path) -> None: + if os.name == "nt": + return + descriptor = os.open(path, os.O_RDONLY | getattr(os, "O_DIRECTORY", 0)) + try: + os.fsync(descriptor) + finally: + os.close(descriptor) + + +def _establish_sqlite_durability(project_root: Path) -> None: + """Checkpoint and fsync the completed index before publishing completion.""" + db_path = get_db_path(project_root) + connection = sqlite3.connect(str(db_path), timeout=30) + try: + connection.execute("PRAGMA busy_timeout=30000") + checkpoint = connection.execute("PRAGMA wal_checkpoint(TRUNCATE)").fetchone() + if checkpoint is not None and int(checkpoint[0]) != 0: + raise sqlite3.OperationalError("index WAL checkpoint remained busy") + connection.commit() + finally: + connection.close() + + sync_flags = (os.O_RDWR if os.name == "nt" else os.O_RDONLY) | getattr(os, "O_CLOEXEC", 0) + for candidate in (db_path, Path(f"{db_path}-wal"), Path(f"{db_path}-journal")): + try: + descriptor = os.open(candidate, sync_flags) + except FileNotFoundError: + continue + try: + os.fsync(descriptor) + finally: + os.close(descriptor) + _fsync_directory(db_path.parent) def _semantic_activation_advice(conn, project_root: Path) -> str | None: @@ -1294,10 +1782,48 @@ def run( # Lock file to prevent concurrent indexing lock_path = self.root / ".roam" / "index.lock" - if not _claim_index_lock(lock_path): + claim = _claim_index_lock(lock_path) + if claim is None: return False + state_path = self.root / ".roam" / "index.state" + effective_force = force or claim.recovery_required + effective_light = light and not claim.recovery_required + if claim.recovery_required: + self._log( + "Interrupted index ownership was recovered; running a full forced rebuild " + f"before completion ({claim.recovery_reason})." + ) try: - self._do_run(force, verbose=verbose, include_excluded=include_excluded, light=light) + # Publish the lifecycle marker before the first database mutation. + # A killed process leaves ``in_progress`` behind, allowing every + # consumer-side ensure_index() call to distinguish a resumable, + # partial SQLite file from a completed index. Atomic replacement + # prevents a torn marker from being mistaken for completion. + atomic_write_text( + state_path, + _serialize_index_state("in_progress", claim.owner), + durable=True, + ) + # Re-prove both sides of the ownership join immediately before + # the first SQLite mutation. A missing/replaced lock pathname or + # a changed marker fails closed without touching the database. + state_kind, state_owner = _decode_index_state(_read_control_text(state_path)) + if not _claim_still_owned(claim) or state_kind != "in_progress" or state_owner != claim.owner: + raise RuntimeError("index ownership changed before database mutation") + self._do_run( + effective_force, + verbose=verbose, + include_excluded=include_excluded, + light=effective_light, + ) + _establish_sqlite_durability(self.root) + if not _claim_still_owned(claim): + raise RuntimeError("index ownership changed before completion publication") + atomic_write_text( + state_path, + _serialize_index_state("complete", claim.owner), + durable=True, + ) return True except KeyboardInterrupt: # graceful Ctrl-C: drop the lock so the user can @@ -1308,7 +1834,7 @@ def run( raise finally: _quiet_mode = False - _release_index_lock(lock_path) + _release_index_lock(claim) def _extract_file_refs( self, diff --git a/src/roam/mcp_server.py b/src/roam/mcp_server.py index 164317682..8e9833455 100644 --- a/src/roam/mcp_server.py +++ b/src/roam/mcp_server.py @@ -6334,7 +6334,13 @@ def _run_roam_subprocess(args: list[str], root: str = ".") -> dict: EXIT_GATE_FAILURE = _EXIT_GATE_FAILURE _success_codes = _SUCCESS_EXIT_CODES - cmd = ["roam", "--json"] + args + # Invoke the package through the interpreter that loaded this MCP + # server. A locked/dev environment can import ``roam`` without + # installing the console-script shim on PATH; using a bare ``roam`` + # executable therefore made every non-local-root tool fail in CI and + # in embedded MCP hosts. ``python -m roam`` is the same installed + # package and remains valid for wheel installs. + cmd = [sys.executable, "-m", "roam", "--json"] + args try: result = subprocess.run( cmd, diff --git a/src/roam/plan/compiler.py b/src/roam/plan/compiler.py index 515d5e604..6a9c2ab45 100644 --- a/src/roam/plan/compiler.py +++ b/src/roam/plan/compiler.py @@ -11249,12 +11249,12 @@ def _effective_forbidden_paths(self) -> list[str]: return list(self.forbidden_paths) return [] - def to_envelope(self) -> dict: + def to_envelope(self, cwd: str | None = None) -> dict: d = asdict(self) # W21: stale-index warning on the FULL envelope. Check both task-extracted # paths AND any likely_files the compiler resolved via search. - named = _extract_file_paths(self.task) + list(self.likely_files or []) - staleness = _named_path_staleness(named, None) + named = _extract_file_paths(self.task, cwd) + list(self.likely_files or []) + staleness = _named_path_staleness(named, cwd) if staleness: d["index_staleness"] = staleness return { @@ -12818,7 +12818,7 @@ def _fallback_envelope_after_probe_degrades( _attach_degraded_probe_signal(plan.to_lean_envelope(cwd=cwd), probe_attempted), "lean", ) - return _attach_degraded_probe_signal(plan.to_envelope(), probe_attempted), "full" + return _attach_degraded_probe_signal(plan.to_envelope(cwd=cwd), probe_attempted), "full" def _emit_result_after_required_compile_side_effects( @@ -12843,6 +12843,11 @@ def _cached_compile_result_if_fresh(plan: "PlanV0", cwd: str | None, started_at: if cached is None: return None cached_env, cached_label = cached + # The cache's dependency fingerprint is an invalidation optimization, + # not the disclosure boundary. Re-evaluate post-index edits on every hit + # so an omitted/legacy dependency row can never make stale coordinates + # look current. + _stamp_index_staleness(cached_env, plan, cwd) # W58 — flag cache hit on the plan so telemetry can record it. object.__setattr__(plan, "_w58_cache_hit", True) _maybe_append_compile_telemetry( @@ -13558,7 +13563,24 @@ def _index_freshness_signals(named_paths: list[str], cwd: str | None) -> tuple[d * newer_files: {"files_newer_than_index": [...]} for paths edited after the index mtime (post-index edits). """ - base = cwd or os.getcwd() + try: + base = cwd or os.getcwd() + except OSError: + if not named_paths: + return None, None + return ( + { + "is_stale": True, + "missing_paths": [], + "index_age_seconds": None, + "working_directory_unavailable": True, + "warning": ( + "named_paths may be unreliable: current working directory is unavailable. " + "Verify with Read/Grep before trusting." + ), + }, + None, + ) index_db = os.path.join(base, ".roam", "index.db") index_mtime = _index_mtime_or_none_for_resilient_diagnostics(index_db) missing, newer_files = _scan_named_paths_for_index_drift(base, named_paths, index_mtime) diff --git a/src/roam/savings.py b/src/roam/savings.py index 8963382b2..4d7640394 100644 --- a/src/roam/savings.py +++ b/src/roam/savings.py @@ -1505,8 +1505,20 @@ def capture_prior_generation() -> None: open_flags = os.O_RDWR | getattr(os, "O_CLOEXEC", 0) | getattr(os, "O_BINARY", 0) open_flags |= getattr(os, "O_NOFOLLOW", 0) descriptor = os.open(database, open_flags) - if file_descriptor_identity(descriptor) != source_generation.identity or not file_descriptor_is_owner_only( - descriptor, database + installed_generation = capture_file_generation(descriptor, max_bytes=MAX_LEDGER_DB_BYTES) + # A successful rename preserves the file object and bytes but can + # legitimately update ctime (POSIX metadata-change time; Windows + # creation time). Bind the published object to every stable + # source-generation field, then use that post-rename generation as + # the baseline for the durability check below. + if ( + installed_generation.identity != source_generation.identity + or installed_generation.size != source_generation.size + or installed_generation.mtime_ns != source_generation.mtime_ns + or installed_generation.nlink != source_generation.nlink + or installed_generation.sha256 != source_generation.sha256 + or file_descriptor_identity(descriptor) != source_generation.identity + or not file_descriptor_is_owner_only(descriptor, database) ): raise SavingsLedgerSafetyError(f"installed savings ledger changed: {database}") os.fsync(descriptor) @@ -1518,7 +1530,7 @@ def capture_prior_generation() -> None: os.close(directory_fd) if capture_file_generation( descriptor, max_bytes=MAX_LEDGER_DB_BYTES - ) != source_generation or not file_descriptor_is_owner_only(descriptor, database): + ) != installed_generation or not file_descriptor_is_owner_only(descriptor, database): raise SavingsLedgerSafetyError(f"installed savings ledger changed after durability sync: {database}") except BaseException: if conn is not None: diff --git a/src/roam/transcript_backfill.py b/src/roam/transcript_backfill.py index 599b4e80c..97f1791ff 100644 --- a/src/roam/transcript_backfill.py +++ b/src/roam/transcript_backfill.py @@ -709,14 +709,19 @@ def _load_or_create_key(root: Path, *, create: bool = True) -> bytes: state = _private_state_directory(root, create=create) if not create and not state.exists(): return os.urandom(32) - with pinned_owner_only_directory(state): - path = state / SALT_NAME - key = _read_private_key(path, transient_retries=20) - if key is not None: - return key - if not create: - return os.urandom(32) - return _create_private_key(path) + try: + with pinned_owner_only_directory(state): + path = state / SALT_NAME + key = _read_private_key(path, transient_retries=20) + if key is not None: + return key + if not create: + return os.urandom(32) + return _create_private_key(path) + except TranscriptBackfillSafetyError: + raise + except (OSError, RuntimeError) as exc: + raise TranscriptBackfillSafetyError(f"cannot safely access savings backfill key state: {state}: {exc}") from exc def _keyed_hex(key: bytes, purpose: str, value: str, length: int = 24) -> str: @@ -2349,22 +2354,34 @@ def validate_destination() -> None: allow_missing=True, ) - with pinned_owner_only_directory(state_dir): - atomic_write_bytes( - output, - payload, - prepare_temp_fd=prepare_private_temp, - before_replace=validate_destination, - durable=True, - create_parents=False, - secure_parent=True, - ) - _private_file_state( - output, - label="derived transcript snapshot", - max_bytes=MAX_SNAPSHOT_BYTES, - allow_missing=False, - ) + try: + with pinned_owner_only_directory(state_dir): + # Validate before entering the atomic writer so a pre-existing + # linked destination is reported through this module's public + # safety contract. Keep the callback as the race-closing check + # immediately before replacement. + validate_destination() + atomic_write_bytes( + output, + payload, + prepare_temp_fd=prepare_private_temp, + before_replace=validate_destination, + durable=True, + create_parents=False, + secure_parent=True, + ) + _private_file_state( + output, + label="derived transcript snapshot", + max_bytes=MAX_SNAPSHOT_BYTES, + allow_missing=False, + ) + except TranscriptBackfillSafetyError: + raise + except (OSError, RuntimeError) as exc: + raise TranscriptBackfillSafetyError( + f"cannot safely write derived transcript snapshot: {output}: {exc}" + ) from exc return { "state": "dry_run" if dry_run else "written", "output": str(output), diff --git a/templates/distribution/landing-page/changelog.html b/templates/distribution/landing-page/changelog.html index 86e3f792f..76685e7c9 100644 --- a/templates/distribution/landing-page/changelog.html +++ b/templates/distribution/landing-page/changelog.html @@ -76,7 +76,7 @@

Changelog

All notable changes to roam-code will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

-

[13.10.0] — 2026-07-19

+

[13.10.0] — 2026-07-23

Added

Changed

Fixed