fix(worker,models): audit hotfixes - manifest crash, VRAM fail-open, real min_ram_mb key

[jaylfc]

Hotfix for the urgent findings from the full code audit of the recently merged work. The deeper redesign parts stay with #1765 / #1766.

What

1. Worker manifest can no longer brick the worker (HIGH, fix(worker): manifest path crashes worker on malformed input + availability depends on backend liveness (audit follow-up to #1762) #1765 item 1): malformed/non-object /etc/taos/worker-models.json degrades to no-manifest with a logged warning; enrichment skips entries without model_id and is wrapped fail-soft; heartbeat failures are logged before being swallowed.
2. VRAM reservation fails open on no-probe hardware (HIGH, fix(models): VRAM reservation fails closed on non-NVIDIA + TOCTOU gate reads dead manifest key (audit follow-up to #1725) #1766 item 1): probe returns None (cannot know) vs (0,0) (known-full); reserve() admits bookkeeping-only on None, matching claim_lease semantics. Previously every pull with a VRAM requirement 503'd on AMD/Apple/Rockchip. Probe moved to asyncio.to_thread (no more event-loop stall under the lock). stats() gains probe_available.
3. TOCTOU gate reads the real key (MEDIUM, fix(models): VRAM reservation fails closed on non-NVIDIA + TOCTOU gate reads dead manifest key (audit follow-up to #1725) #1766 item 2): the rkllama pull gate now takes the max of variant.requires.backends[].min_ram_mb (falling back to the variant-level key), so it is no longer dead code against the entire catalog.
4. backend_services.load_managed_backends logs skipped manifests (silent-drop drift fix).
5. install-rknpu.sh: stale "copy-of-truth" comment corrected (old port 8080 / old models path).

Tests

• Updated the two tests that blessed the crash to assert the degrade behaviour.
• New: non-object manifest degrade, no-probe fail-open (admission + bookkeeping + release), probe_available in stats, and a regression guard that a REAL probe showing insufficient VRAM still denies.
• Local: 155 tests green across worker_manifest / vram_reservation / worker / cluster_worker_protocol / backend_services / routes_system / routes_models; compileall + bash -n clean.

Summary by CodeRabbit

• New Features

  • VRAM checks now “fail open” when hardware probing is unavailable, allowing jobs to proceed while still tracking reservation state.
  • Model download sizing now uses the most demanding backend requirement, improving capacity checks.

• Bug Fixes

  • Malformed or missing worker manifests now fall back to a safe empty state instead of failing.
  • Worker backend detection and heartbeat reporting are more resilient, with clearer warnings when input or local processing is invalid.

• Chores

  • Updated deployment notes and improved warning logs for invalid backend definitions.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[coderabbitai]

📝 Walkthrough

Walkthrough

Changes

Runtime resilience and resource handling

Layer / File(s)	Summary
VRAM probe and reservation semantics tinyagentos/vram_reservation.py, tinyagentos/routes/models.py, tests/test_vram_reservation.py	Unavailable VRAM probes now fail open for reservations, known capacity remains enforced, monitoring reports probe availability, and model estimates use backend requirements.
Manifest validation and worker diagnostics tinyagentos/worker/worker_manifest.py, tinyagentos/worker/agent.py, tinyagentos/cluster/backend_services.py, tests/test_worker_manifest.py	Malformed manifests and invalid backend entries degrade safely, while manifest, backend-loading, enrichment, and heartbeat failures emit warnings.
Installer unit comment alignment scripts/install-rknpu.sh	The rkllama systemd ExecStart comment now reflects port 7833, the unified models root, and shared model preloading.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant WorkerAgent
  participant load_manifest
  participant ProbedBackends
  WorkerAgent->>load_manifest: load and validate manifest
  load_manifest-->>WorkerAgent: manifest or empty fallback
  WorkerAgent->>ProbedBackends: apply compatible model entries
  ProbedBackends-->>WorkerAgent: enriched backend data

Loading

Possibly related issues

• jaylfc/taOS#1766: Covers the same VRAM fail-open behavior and backend-level VRAM estimation changes.
• jaylfc/taOS#1765: Covers the malformed manifest crash path and heartbeat diagnostics.

Possibly related PRs

• jaylfc/taOS#802: Previously changed the installer’s rkllama default port from 8080 to 7833.
• jaylfc/taOS#1725: Added related atomic VRAM reservation logic and tests.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main hotfix areas: manifest handling, VRAM fail-open behavior, and the min_ram_mb routing fix.
Docstring Coverage	✅ Passed	Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/audit-hotfix-vram-manifest

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[gitar-bot]

Important

You are using the Gitar free plan. Upgrade to unlock code review, CI analysis, auto-apply, custom automations, and more.

_Gitar

[kilo-code-bot]

[WARNING]: variant.get("requires") or {} does not guard against requires being a non-dict truthy value

(variant.get("requires") or {}) only falls back to {} when the value is falsy. If requires is present but a list/string (e.g. a malformed catalog variant — exactly the kind of untrusted external input this PR's theme defends against), the truthy value is kept and .get("backends") raises AttributeError, 500-ing the download route. Guard with isinstance instead.

Suggested change

	_backend_reqs = (variant.get("requires") or {}).get("backends") or []
	_requires = variant.get("requires")
	_backend_reqs = ((_requires if isinstance(_requires, dict) else {}) or {}).get("backends") or []

---

Reply with @kilocode-bot fix it to have Kilo Code address this issue.

[kilo-code-bot]

[WARNING]: Fail-open conflates "no nvidia-smi" with "probe errored", over-admitting on real NVIDIA hardware

probe is None now covers both "this host has no VRAM probe" (AMD/Apple/Rockchip) and "nvidia-smi failed transiently". Previously a failed probe returned (0, 0) and the reservation was denied (fail-closed). Now any transient nvidia-smi error on actual NVIDIA GPUs admits the reservation unenforced, so concurrent pulls can over-commit and OOM — the exact failure mode this guard was originally added to prevent (#1706).

Consider letting _probe_vram distinguish "binary absent" (fail open, bookkeeping only) from "probe errored" (fail closed or retry) rather than collapsing both into None.

---

Reply with @kilocode-bot fix it to have Kilo Code address this issue.

[kilo-code-bot]

Code Review Summary

Status: 2 Issues Found | Recommendation: Address before merge

Overview

Severity	Count
CRITICAL	0
WARNING	2
SUGGESTION	0

Issue Details (click to expand)

WARNING

File	Line	Issue
tinyagentos/routes/models.py	395	variant.get("requires") or {} does not guard non-dict truthy requires; malformed variant raises AttributeError and 500s the download route
tinyagentos/vram_reservation.py	93	Fail-open conflates "no probe" with "transient probe error"; transient nvidia-smi failure on real NVIDIA GPUs now over-admits and can OOM

Files Reviewed (8 files)

• scripts/install-rknpu.sh - comment only, 0 issues
• tests/test_vram_reservation.py - 0 issues
• tests/test_worker_manifest.py - 0 issues
• tinyagentos/cluster/backend_services.py - 0 issues
• tinyagentos/routes/models.py - 1 issue
• tinyagentos/vram_reservation.py - 1 issue
• tinyagentos/worker/agent.py - 0 issues
• tinyagentos/worker/worker_manifest.py - 0 issues

Fix these issues in Kilo Cloud

---

_{Reviewed by hy3-20260706:free · Input: 73.6K · Output: 8.5K · Cached: 154K}

[coderabbitai]

🧹 Nitpick comments (1)

tinyagentos/vram_reservation.py (1)

165-191: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

available_vram() and stats() still probe synchronously, blocking the event loop

reserve() correctly moved the probe to asyncio.to_thread, but available_vram() and stats() still call _probe_vram() directly. In tinyagentos/routes/models.py line 405, available_vram() is called from async def download_model on the denial path, causing a 50–200 ms blocking nvidia-smi call that stalls all concurrent coroutines. This is also a redundant probe — reserve() already probed via asyncio.to_thread but discards the result on denial.

Consider making available_vram() async (or caching the last probe result from reserve()) so the denial path doesn't block. Verify that stats() callers aren't async either.

♻️ Suggested fix: make available_vram() async

-    def available_vram(self) -> tuple[int, int]:
-        """Return ``(effective_free_mb, total_mb)`` after subtracting
-        in-flight reservations from the hardware probe.
-
-        When no probe is available, returns ``(0, 0)``; callers on that
-        path should already have been admitted fail-open by :meth:`reserve`
-        (a deny message never reports these zeros as real capacity).
-        """
-        probe = self._probe_vram()
-        if probe is None:
-            return 0, 0
-        free_mb, total_mb = probe
-        effective_free = max(0, free_mb - self._reserved_vram_mb)
-        return effective_free, total_mb
+    async def available_vram(self) -> tuple[int, int]:
+        """Return ``(effective_free_mb, total_mb)`` after subtracting
+        in-flight reservations from the hardware probe.
+
+        When no probe is available, returns ``(0, 0)``; callers on that
+        path should already have been admitted fail-open by :meth:`reserve`
+        (a deny message never reports these zeros as real capacity).
+        """
+        probe = await asyncio.to_thread(self._probe_vram)
+        if probe is None:
+            return 0, 0
+        free_mb, total_mb = probe
+        effective_free = max(0, free_mb - self._reserved_vram_mb)
+        return effective_free, total_mb

And update the caller in tinyagentos/routes/models.py:

-                effective_free, _total = vram_mgr.available_vram()
+                effective_free, _total = await vram_mgr.available_vram()

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tinyagentos/vram_reservation.py` around lines 165 - 191, available_vram() and
stats() still call _probe_vram() synchronously, which can block the event loop
when used from async paths like download_model. Update VRAMReservation so
available_vram() becomes async or reuses a cached probe result from reserve(),
and change the denial-path caller in models route to await/use the non-blocking
path; also review stats() to avoid direct synchronous probing or ensure it is
only used from non-async contexts.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tinyagentos/vram_reservation.py`:
- Around line 165-191: available_vram() and stats() still call _probe_vram()
synchronously, which can block the event loop when used from async paths like
download_model. Update VRAMReservation so available_vram() becomes async or
reuses a cached probe result from reserve(), and change the denial-path caller
in models route to await/use the non-blocking path; also review stats() to avoid
direct synchronous probing or ensure it is only used from non-async contexts.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 6a56d8d4-d207-4eb3-b2c4-891987a8c8da

📥 Commits

Reviewing files that changed from the base of the PR and between 9ac0ff7 and e4c3315.

📒 Files selected for processing (8)

• scripts/install-rknpu.sh
• tests/test_vram_reservation.py
• tests/test_worker_manifest.py
• tinyagentos/cluster/backend_services.py
• tinyagentos/routes/models.py
• tinyagentos/vram_reservation.py
• tinyagentos/worker/agent.py
• tinyagentos/worker/worker_manifest.py