Skip to content

fix(cluster): heartbeat-driven registration refresh for host_lan_ip/url/hardware#2

Open
hognek wants to merge 1 commit into
masterfrom
fix/heartbeat-drift-refresh
Open

fix(cluster): heartbeat-driven registration refresh for host_lan_ip/url/hardware#2
hognek wants to merge 1 commit into
masterfrom
fix/heartbeat-drift-refresh

Conversation

@hognek

@hognek hognek commented Jul 6, 2026

Copy link
Copy Markdown
Owner

Fix: taOS jaylfc#1538 — agent-registration drift (stale IP/status).

Problem: When a worker LXC container comes up on a different IP, the host_lan_ip/url/hardware drifts. Skald routes on these fields — stale values cause hangs or 5xx.

Fix — heartbeat-driven refresh/reconciliation: WorkerAgent.heartbeat() now includes host_lan_ip, url, hardware on every pulse. ClusterManager.heartbeat() updates cached WorkerInfo when these fields are present (backward-compat: None leaves stale values untouched).

Tests: 5 new in TestHeartbeatRegistrationRefresh — all 33 cluster+protocol unit tests pass, no regressions.

Also fixes: storage_cap/storage_used/bytes_deduped were accepted by HeartbeatBody but silently dropped by the handler.


Note

Medium Risk
Changes cluster routing metadata and deploy/proxy paths that depend on worker.url and host_lan_ip; behavior is backward-compatible but incorrect refresh could misroute traffic.

Overview
Fixes jaylfc#1538 by reconciling worker registration metadata on every heartbeat instead of only at initial register, so cluster routing (e.g. Skald) does not keep using stale host_lan_ip, url, or hardware after DHCP moves or LXC restarts.

Workers now send live host_lan_ip (via _detect_lan_ip), url, and hardware on each pulse, alongside existing backend/capacity fields. Controller extends HeartbeatBody and ClusterManager.heartbeat() to apply those fields when present (None leaves cached values for legacy agents). The heartbeat route also forwards LXC storage counters (storage_cap_bytes, storage_used_bytes, bytes_deduped_total) that were accepted on the body but previously dropped. When backends is updated, available_models is flattened from sidecar-enriched backend catalogs into WorkerInfo.

Tests: TestHeartbeatRegistrationRefresh covers IP/URL/hardware refresh, no-op on None, and unknown workers. .gitignore adds venv/.

Reviewed by Cursor Bugbot for commit 3cce536. Bugbot is set up for automated code reviews on this repo. Configure here.

…rl/hardware (jaylfc#1538)

Worker heartbeat now carries host_lan_ip, url, and hardware on every
pulse so the controller can reconcile stale registration data against
container reality. When a worker LXC comes up on a different IP (DHCP
lease change, container restart), the next heartbeat refreshes routing
fields without requiring a full re-registration (which only triggers
on 404 — controller restart).

Changes:
- WorkerAgent.heartbeat(): include host_lan_ip, url, hardware in payload
- HeartbeatBody: add optional host_lan_ip, url, hardware fields
- worker_heartbeat(): pass new fields through to cluster.heartbeat()
- ClusterManager.heartbeat(): update cached WorkerInfo when fields present
- Also fixes a pre-existing gap: storage_cap/storage_used/bytes_deduped
  were accepted by HeartbeatBody but silently dropped by the handler.

Tests: 5 new tests in TestHeartbeatRegistrationRefresh (17/17 pass).
No regressions: 33 cluster+protocol unit tests pass.
@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@hognek, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 10 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ae1921e7-f4cf-432e-a66b-a616b305eab5

📥 Commits

Reviewing files that changed from the base of the PR and between 133e88e and 3cce536.

📒 Files selected for processing (5)
  • .gitignore
  • tests/test_cluster_worker_protocol.py
  • tinyagentos/cluster/manager.py
  • tinyagentos/routes/cluster.py
  • tinyagentos/worker/agent.py
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/heartbeat-drift-refresh

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 3cce536. Configure here.

# registered state and container reality (#1538).
"host_lan_ip": _detect_lan_ip(self.controller_url),
"url": worker_url,
"hardware": asdict(hw),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heartbeat overwrites routable worker URL

High Severity

Each heartbeat now sends a url derived from the first detected backend’s url (typically http://localhost:…). The controller applies that on every pulse, replacing a previously registered LAN or Incus URL. Cluster routing and mesh metadata use worker.url from the controller, so requests can target an address only reachable inside the worker container.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 3cce536. Configure here.

hognek pushed a commit that referenced this pull request Jul 6, 2026
…aylfc#1535) (jaylfc#1537)

* feat(setup): NPU-conditional checklist step for the rkllama backend (jaylfc#1535)

The setup wizard never consumed hardware detection, so on a Rockchip
board nothing in the flow surfaced the NPU backend install; users had to
discover the rkllama entry in the Store themselves (the wizard half of
the gap reported in the #2 thread, tracked as jaylfc#1535).

/api/setup/status now reports npu_present (from the cached hardware
profile) and npu_backend_running (live rkllama probe, off the event
loop), and the checklist appends a sixth step on NPU boards that opens
the Store and completes once the backend answers. Boards without an NPU
see no change. Explicit consent stays: the step is an offer, never an
auto-install.

* fix(setup): fold the review findings on the NPU step

- The checklist no longer hides on core-complete while the NPU step is
  outstanding: an NPU board that finished provider+model would otherwise
  never see the backend install at all (the exact gap jaylfc#1535 closes).
  Dismiss still hides it, and it disappears once the backend runs. Two
  component tests pin both states.
- /api/setup/status docstring documents the two new fields.
- test_npu_absent_by_default pins hardware_profile=None so class test
  order cannot leak the Rockchip profile from the sibling tests.
- The to_thread comment says blocking is the reason (the probe never
  raises).
hognek pushed a commit that referenced this pull request Jul 6, 2026
…pp (static-first v1) (jaylfc#1636)

* feat(web-studio): real AI generation + sandboxed preview + share-as-app (static-first v1)

Turn the three Web Studio dead ends into working features, keeping the
existing structured section-based Site model and WYSIWYG editor intact.

1. Real AI generation. GenerateView now streams the taos-agent
   (streamTaosAgentChat) with a schema-constrained system prompt
   (web-authoring-prompt.ts) asking for a single JSON Site document.
   generate-site.ts tolerant-extracts the JSON (strips code fences /
   prose), JSON.parses it, validates with isValidSite() plus a
   palette/font enum check, and loads it into the editor. On empty
   stream / parse fail / shape mismatch it falls back to the existing
   matchTemplate() path and surfaces an honest parseNotice -- never a
   faked success. Cancel/AbortSignal + progress states included.

2. Sandboxed preview. Saving now also persists the rendered index.html
   (exportSiteHtml) alongside the editable Site JSON (new index_html
   column, idempotent migration). New GET /api/web/sites/{id}/preview
   serves it under a strict CSP (sandbox allow-scripts, opaque origin,
   default-src 'none'). PreviewView renders a sandboxed iframe -- the
   saved backend page when clean, else a srcDoc of the live export --
   both sandbox="allow-scripts" with no allow-same-origin. No
   un-sandboxed React render remains.

3. Share / install as a taOS app. New GET /api/web/sites/{id}/package
   builds a .taosapp via build_package(). New ShareView runs the static
   analyzer on the rendered HTML, blocks on any critical finding, else
   POSTs the package to /api/userspace-apps/install tagged provenance
   ai-generated; Export downloads the same package. Wired into a new
   Share rail entry.

The LXC/container dynamic tier is Phase 2 (left as a code comment in
routes/web.py, no implementation).

* docs: mark Web Studio feature source as reviewed (doc-gate)

The new files under desktop/src/apps/webstudio/ (generate-site,
web-authoring-prompt, ShareView, PreviewView, web-sites-api and their
tests) are additional source modules for the EXISTING Web Studio app,
not a new desktop app or a new API route module needing its own README.
The doc-gate `apps` rule keys off files under desktop/src/apps/ and
cannot tell a new sub-module of an existing app from a whole new app.

Docs-Reviewed: new files are Web Studio feature source (real AI generation, sandboxed preview, share-as-app) for the existing app; no new app/route surface, so no README addition is warranted

* fix(web-studio): fold Kilo review findings

WARNINGS:
- routes/web.py: an explicit JSON null for `content`/`index_html` returned
  None from dict.get and then failed the string type-check as a spurious
  400. New _field_or_default() coerces missing OR null to the fallback
  (empty string on create, existing value on update). Tests added.
- PreviewView: a saved-but-empty index_html (legacy rows) previously
  pointed the iframe at a 404. Now WebStudioApp tracks whether the active
  row has a stored render (activeHasRender) and Preview uses the live URL
  only when present, else a srcDoc of the in-memory export. A site with no
  sections shows an explicit "Nothing to preview yet" empty state instead
  of a near-blank iframe. Tests added.

SUGGESTIONS:
- routes/web.py _WEB_PREVIEW_CSP: verified well-formed (single CSP value,
  `sandbox allow-scripts` as the first semicolon-separated directive, no
  duplicates, opaque origin preserved). `sandbox` is the CSP directive, not
  the iframe attribute -- added a comment saying so; not a bug.
- routes/web.py: the 5MB index_html cap is now mirrored by a client-side
  size guard in saveSite (checks both content and rendered html) so the
  user gets a clear message, not a raw 413.
- web_sites.py _migration_add_index_html: keeps the guarded _post_init +
  manual commit (NOT the MIGRATIONS list) -- db_migrations.py footgun #2
  documents that the runner baseline-stamps pre-existing DBs without running
  SQL, silently skipping retrofit columns on the legacy rows that need them.
  Added a comment; confirmed idempotent on legacy DBs.
- ShareView: install no longer always tags "ai-generated". WebStudioApp
  tracks honest session provenance (ai-generated only for an actually-
  generated site; templates/manual/fallback/reopened -> user-uploaded) and
  passes it down. Tests cover both tags.
hognek added a commit that referenced this pull request Jul 7, 2026
…M reservation

_addmission_check read live nvidia-smi free VRAM with no reservation,
so two concurrent submit_gpu calls could both pass before either model
loaded — the exact concurrent-load crash (NVIDIA Xid 62) the arbiter
exists to prevent.

Add _reserved_vram_mb counter and _pending_reservations dict, guarded
by _reservation_lock.  _reserve_and_check() atomically checks admission
against (probe - _reserved_vram_mb) and reserves on success.  Reservations
are released in _run_gpu_task's finally block and in _evict_task.

_stats() now includes reserved_vram_mb and pending_reservations for
observability.

Fixes: PR jaylfc#1683 review feedback #2
Tests: 168 pass (25 arbiter + 143 cluster/worker)
hognek added a commit that referenced this pull request Jul 7, 2026
…M reservation

_addmission_check read live nvidia-smi free VRAM with no reservation,
so two concurrent submit_gpu calls could both pass before either model
loaded — the exact concurrent-load crash (NVIDIA Xid 62) the arbiter
exists to prevent.

Add _reserved_vram_mb counter and _pending_reservations dict, guarded
by _reservation_lock.  _reserve_and_check() atomically checks admission
against (probe - _reserved_vram_mb) and reserves on success.  Reservations
are released in _run_gpu_task's finally block and in _evict_task.

_stats() now includes reserved_vram_mb and pending_reservations for
observability.

Fixes: PR jaylfc#1683 review feedback #2
Tests: 168 pass (25 arbiter + 143 cluster/worker)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant