Skip to content

feat: solo provisioner daemon core#688

Draft
leninmehedy wants to merge 75 commits into
mainfrom
00499-feat-solo-provisioner-daemon-core
Draft

feat: solo provisioner daemon core#688
leninmehedy wants to merge 75 commits into
mainfrom
00499-feat-solo-provisioner-daemon-core

Conversation

@leninmehedy

@leninmehedy leninmehedy commented Jun 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Description

A long-running, self-healing control plane that sits on every node, watches the cluster for operator intent, and executes upgrades and migration soaks autonomously — without ever taking itself down.

Features

  • It refuses to fail quietly. Bad config? It won't start, and systemd tells you instantly. A monitor crashes? It's caught, backed off, and restarted — the process never dies from a single subsystem fault. You always have a daemon to ask "what's wrong?"
  • It does each thing exactly once. Upgrades are deduplicated three ways, keyed on the cluster's own source of truth. Restart it, re-deliver the event, replay the watch — it won't double-execute. Migration soaks are single-flight and resume after a reboot with elapsed time preserved, because progress is on disk, not in memory.
  • It tells you how to fix it, not just that it broke. Every error in /status carries a machine reason, a human message, a copy-pasteable sudo … resolution, and a "since" timestamp. No journal spelunking for the common cases.
  • Zero new attack surface. Local Unix socket only — no open port. Each component gets its own scoped Kubernetes credentials, so a block-node bug can't touch consensus-node resources. The privileged binary is deliberately tiny: it shares nothing with the CLI.
  • It plugs into the monitoring you already run. Structured reason-tagged logs flow to journald → Grafana Loki through Alloy with zero daemon-side config; metrics go to Prometheus the same way. Every milestone is also written to a tamper-evident, fsync'd JSONL audit trail for postmortems.
  • It's built to grow. Adding a new component or endpoint is a documented recipe that never touches existing routes. Config format changes are forward-safe via sealed, single-step migrations — an old daemon will never misread a new file.

Please review the daemon architecture guide: docs/daemon/daemon-architecture.md

Screenshot 2026-06-12 at 9 20 55 pm

Related Issues

@swirlds-automation

swirlds-automation commented Jun 12, 2026 •
edited
Loading

Copy link
Copy Markdown

✅ Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
✅ Open Source Security 0 0 0 0 0 issues
✅ Licenses 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@leninmehedy leninmehedy force-pushed the 00499-feat-solo-provisioner-daemon-core branch 2 times, most recently from bea81d8 to 51cb661 Compare June 12, 2026 10:42
This was referenced Jun 13, 2026
Closed
Open
leninmehedy and others added 25 commits June 13, 2026 14:35
@leninmehedy
feat(daemon): shared JSONL event writer and strategy-based file pruner (
14d7bd4 
#555) (#610)

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
# Conflicts:
#	CLAUDE.md
@leninmehedy
feat(daemon): implement UpgradeMonitor goroutine for NetworkUpgradeEx…
863b99e 
…ecute CRs (#519) (#609)

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
docs(plans): update #520 soak-watcher plan — correct HIP divergences
d39dcf3 
- Poll interval: 30s → 900s (15 min) per HIP implementation-guide
- Event reasons: corrected to HIP-defined names (SoakStarted, SoakCheck,
  CriterionMet, FleetThresholdReached, DecommissionTriggered, DecommissionCompleted)
- Four concrete soak criteria wired (SoakDuration48h real; 3 stubs with TODO)
- Fleet threshold via flag file /opt/solo/weaver/migration/fleet-threshold-reached
- SoakCheck carries context fields (soak_hours, uploader_backlog_cleared, etc.)
- MigrationMonitorConfig.FleetThresholdPath field added
- NodeID threaded from DaemonConfig into NewMigrationMonitor
- Unit test names updated; UAT steps reference correct event sequence

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
feat: implement soak-watcher for migration soak criteria monitoring
a0ea8eb 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
feat(daemon): add solo-provisioner-daemon.service and CLI service com…
0731057 
…mands (#623)

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
feat(daemon): implement sd_notify READY/STOPPING and K8s RBAC probe (#…
6667f36 
…626)

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
feat(daemon): provision RBAC, kubeconfig and add start/stop commands (#…
87d5e88 
…637)

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix: allow daemon install to create config
8ea266c 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix: add hints for daemon related errors
59ddc83 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
feat: allow provisioner to download or verify a local daemon binary d…
2b0dabb 
…uring installation

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy @claude
feat(daemon): redesign daemon.yaml with components block; component-a…
93aeec5 
…ware install workflow (closes #661)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
feat(daemon): MonitorRunner interface + supervisedMonitor with degrad…
80e5fe9 
…ation alerting (closes #662)

- New internal/daemon/monitor.go: MonitorRunner interface (Run + Name) and
  supervisedMonitor() with capped exponential back-off (5s->5m, x2, resets
  after 60s stable run).
- MonitorDegraded: emits Error log at every supervisedDegradedThreshold (5)
  consecutive crashes without a stable run; fires again at 10, 15, ... so ops
  keeps seeing the alert until recovery. Counter resets after a stable run.
- 6 unit tests: restart-after-crash, no-restart-on-cancel, back-off reset,
  degraded event fired, degraded counter resets after stable run, back-off cap.
- consensus.UpgradeMonitor + MigrationMonitor: Name() methods added.
Wiring into daemon.Run() deferred to S3 (#663).

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
feat(daemon): componentSupervisor absorbs monitor crashes (closes #663)
fe7c463 
Replace the per-monitor errgroup.Go calls with a single componentSupervisor
goroutine that runs each MonitorRunner via supervisedMonitor. A crashing monitor
now restarts independently with capped back-off without cancelling the shared
errgroup context or taking down server.Start. NewFromConfig now respects
ConsensusNodeComponentConfig.Enabled and the per-monitor toggles.

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
feat(daemon): ProbableMonitor, CompositeProbe, probes subpackage (closes
4073543 
 #664)

Introduce a layered probe architecture so each monitor declares its own
startup prerequisites independently:

- probes.Probe: leaf interface defined in probes/ to avoid import cycles
- probes.KubeRBACProbe: verifies K8s RBAC for a group/resource/verb set
- probes.DiskPermissionProbe: checks declared mode bits on a path
- probes.DiskOwnershipProbe: verifies owner user, group, and/or mode bits
- probes.DiskWriteTestProbe: verifies actual process-level write access

- daemon.ComponentProbe: component-boundary interface (CompositeProbe only)
- daemon.CompositeProbe: fans out to leaf probes concurrently; nestable
- daemon.ProbableMonitor: monitors declare prerequisites via RequiredProbe()
- buildComponentProbe: auto-assembles CompositeProbe from enabled monitors;
  disabling a monitor automatically removes its prerequisites

UpgradeMonitor.RequiredProbe() returns a KubeRBACProbe for
networkupgradeexecutes. StatusTracker records per-monitor state
(running/backoff:<dur>/stopped) updated by supervisedMonitor on each
transition. GET /status returns per-component/per-monitor state JSON.

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy @claude
feat(daemon): per-component RBAC and kubeconfig in install workflow (c…
95c705b 
…loses #665)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
feat(daemon): component-scoped CLI flags and install prompts (S6 #666)
e8d86ee 
Rename flat daemon flags to per-component prefixed names:
  --node-id      → --cn-node-id
  --orbit        → --cn-orbit
  --upgrade-dir  → --cn-upgrade-dir

Add new flags:
  --components   comma-separated list of components to enable
                 (choices: "consensus-node", "block-node"); at least one required
  --bn-orbit     block-node orbit namespace (wired in S7 #667)

Interactive install first asks which components to enable via confirm
prompts, then presents per-component input fields. Selecting block-node
surfaces a clear "not yet available" error until S7. To add or remove
components after install, uninstall and reinstall with updated --components.

Closes #666

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy @claude
feat(daemon): block-node component stub (S7 #667)
12e1745 
Add BlockNodeComponentConfig (Enabled, Kubeconfig, Orbit, Monitors.Upgrade)
to DaemonComponents and config_v1.go. Wire a blockNodeUpgradeMonitor stub
in NewFromConfig — logs "not yet implemented" and blocks on ctx. Fix
kubeconfig preflight in Run() to be nil-safe for BN-only configs.
Update DaemonConfig.Validate() to require at least one component (CN or BN).
Lift the "not yet available" prompt guard and wire --bn-orbit through
RunDaemonInstallPrompts and applyFlagOverrides.

Closes #667

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy @claude
docs(daemon): architecture overview and testing guide (19 test cases)
0022ee4 
Add docs/dev/daemon-architecture.md covering goroutine map, component
model, config schema versioning, supervised monitor parameters, HTTP
control plane, install workflow steps, and on-disk file paths.

Add docs/dev/daemon-testing-guide.md with 19 step-by-step test cases
covering install/uninstall, flag overrides, --from-config, HTTP
endpoints, soak idempotency, supervised restart, schema migration,
block-node stub, systemd Restart=always, idempotent re-install, and
TC-19 (startup probe failure: missing/misconfigured upgrade_dir —
operator diagnoses from journalctl and recovers via reinstall).

Includes a CN prerequisite section documenting that the upgrade staging
directory must exist, be owned hedera:hedera with correct permissions,
and that the daemon user must be in the hedera group before install.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
refactor(daemon): extensible HTTP API — ComponentHandler interface + …
aa84c75 
…component-scoped routes

Routes renamed from /migration/consensus/... to /<component>/<monitor>/... pattern.
ServerOptions replaces flat NewServer params; ComponentHandler interface lets each
component register its own route sub-tree without touching NewServer.
ConsensusNodeHandler owns all /consensus_node/ routes.
ConsensusMigrationStatusResponse (GET /consensus_node/migration/status) returns
combined monitor health + soak state. Fixes pre-existing macOS Unix socket path
length failure in server tests.

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
refactor(daemon): sub-package architecture — core, consensus, blocknode
ef2bc2d 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
feat: stub traffic shaper monitor
454899b 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix(daemon): actionable resolution hints on daemon service install fa…
05095fd 
…ilures

- Add IsConfigMalformed() helper to internal/daemon/errors.go
- Attach ErrPropertyResolution to ErrConfigMalformed when an existing
  daemon.yaml fails validation — doctor layer renders Option A/B/C steps
- Attach ErrPropertyResolution to the missing-components error in Case 3
  (no daemon.yaml, non-interactive) with concrete --components examples
- Fix BlockNodeMonitors field: Upgrade → TrafficShaper in install.go

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix(daemon): traverse cause chain in findResolution to surface hints …
3beff3c 
…through wrappers

root.go wraps every cobra error with errorx.IllegalState — previously this
swallowed any ErrPropertyResolution attached to inner errors (e.g. the
configMalformedResolution hints on daemon.config.malformed).

findResolution now walks the full cause chain via errorx.Cast/Cause until
it finds an attached resolution or exhausts the chain.

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix: let version command to output cleanly for daemon
c7fc506 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix(daemon): fix stale --orbit flag in RBAC step resolution hint
556816f 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
leninmehedy added 27 commits June 13, 2026 14:35
@leninmehedy
fix: add SOLO_SOAK_POLL_INTERVAL env variable support to control soak…
87e3248 
… poll interval

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
feat: add consensus migration soak commands structure
9a2ae9b 
Closes #686
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix: daemon config should be readble as it doesn't have secrets
1ec8b96 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix(daemon): make --config flag point to daemon.yaml instead of CLI c…
0ca1fdd 
…onfig

The daemon binary's --config flag was routed through config.Initialize()
which decoded the file as models.Config (log/proxy settings). Operators
passing a daemon.yaml via --config got a confusing "invalid keys:
components, schema_version" error.

--config now sets the daemon config path; initConfig always uses
defaults for log/proxy so the CLI config format is never attempted.

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix(daemon): add status alias for check
ef2f4c4 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix: remove daemon config upon uninstall
92e384e 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
test: update test guide
7b68783 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
feat(daemon): surface upgrade-monitor connectivity errors in /status
eff98fa 
When the upgrade-monitor's K8s watch is stuck in a backoff loop (e.g.
RBAC revoked), /status previously reported "state":"running" — a false-
healthy signal visible only in journalctl.

Changes:
- Add `core.StatusError` (reason, message, resolution, since) used for
  both connectivity failures and disk-prerequisite probe failures
- Add `core.ConnectivityMonitor` interface; UpgradeMonitor implements it
  with an atomic connectivityErr that is set on any list/watch error and
  cleared after a successful listAndSeed (recovery within one cycle)
- `statusSnapshot()` overlays ConnectivityError() onto tracker state so
  a goroutine in a retry loop shows "degraded" instead of "running"
- Add `probes.TaggedProbe` that wraps any Probe and attaches reason +
  resolution as errorx properties (ErrPropertyReason, ErrPropertyResolution);
  UpgradeMonitor.RequiredProbe() wraps each disk probe with TaggedProbe
- `runComponentProbes` builds core.StatusError from errorx properties,
  preserving the first-failure timestamp across retry rounds
- `CheckDaemonComponentPrerequisites` now surfaces both probe errors and
  degraded monitors with reason/resolution/since; daemon service check
  exits non-zero on either class
- Update TC-25 to verify degraded-state output and service check exit code

Closes #687

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix: tidy up code
ddf4287 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
docs: add daemon architecture doc
a00d1c8 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix: convert errors to errorx
1c73e00 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
test: add unit integration tests
6efef68 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
docs: update architecture doc
2b764fc 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
feat(daemon): route daemon-kernel logging through log/slog
ed43f69 
Introduce a log/slog seam in the daemon ahead of the pkg/daemonkit
extraction (#499). Bump automa-saga/logx to v0.5.0 and install its
zerolog-backed slog.Handler via slog.SetDefault in the daemon bootstrap,
so slog records reach the same journald + rotating-file sinks logx
configures. Convert the 8 daemon-kernel logx call sites (5 in
core/monitor.go, 3 in server.go) to slog, preserving the same reason
keys, messages, levels, and field names. CLI/workflows stay on logx.

Closes #691

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix(workflows): use errorx in consensus migration client
c009baf 
Convert the 7 fmt.Errorf sites in consensus_migration_client.go to the
errorx standard (forbidigo lint). Network calls use ExternalError,
request marshal/build use InternalError, response decode uses
IllegalFormat, and remote error statuses use ExternalError.

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix: remove incorrect catalog entry for daemon until daemon is releas…
4e649b0 
…ed to support auto download

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
refactor(daemon): extract supervised-monitor + socket-server kernel i…
498c36b 
…nto pkg/daemonkit

Lift the daemon supervision loop and Unix-socket control plane out of
internal/daemon/core and internal/daemon into a new in-repo public package
pkg/daemonkit, depending only on stdlib, errgroup, and log/slog (no internal/,
cmd/, or k8s.io/). The concrete StatusResponse payload and StatusFn impl stay
in internal/daemon; the server takes an injected StatusFn func() any.

Rewire consensus/blocknode/daemon to import daemonkit; alias probes.Probe to
daemonkit.Probe; delete the now-empty core package and moved files.

Closes #692

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
refactor(daemonkit): move probe framework into pkg/daemonkit, sever p…
41b9f45 
…kg/models coupling

Moves the generic probe framework (disk probes, tagged probe) into
pkg/daemonkit and replaces TaggedProbe's errorx-property coupling with a
kit-native ProbeError{Reason,Resolution,Message,Err} carrying the fields
as plain struct values. The shared models.ErrPropertyReason/Resolution
registry is left untouched so doctor-layer extraction across the CLI and
workflows keeps matching. internal/daemon/probes is trimmed to the
k8s-dependent KubeRBACProbe; runComponentProbes reads ProbeError fields
directly via errors.As.

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
docs(daemon): document daemonkit reusable foundation (#695)
5ffd783 
Add a "daemonkit — reusable daemon foundation" section covering what
pkg/daemonkit provides (SupervisedMonitor/back-off, StatusTracker/
MonitorState/StatusError, the Unix-socket Server, sd_notify, Component
Handler, and the Probe framework incl. CompositeProbe/TaggedProbe/
ProbeError/disk probes), the dependency philosophy (stdlib + errorx +
errgroup only; slog as the logging seam bridged to logx; k8s.io/client-go
deliberately excluded), the pkg/models property-registry boundary and why
TaggedProbe uses ProbeError struct fields, and the planned standalone-
module extraction.

Reconcile the rest of the doc with the post-extraction reality: rewrite
the package-layout tree, update core.* symbol references to daemonkit.*,
correct the TaggedProbe description, distinguish the two import boundaries,
and update the testing key-files and command.

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix: persist soak sync status
4c65387 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix: ensure migration plan path is within the allowed base path
43c63a1 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix: add comments to clarify rationale
1ddd54d 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix: handle parse error on uid and guid
9c9a46b 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix: add comments on sudoers for daemon and associated security postu…
e24932a 
…re notes

Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix: ensure solo provisioner bin is access to root user and group only
f6e8243 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix: stop double soak start event
d0b1793 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy
fix: crash count increment issue
041da2b 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy leninmehedy force-pushed the 00499-feat-solo-provisioner-daemon-core branch from a03d073 to 041da2b Compare June 13, 2026 04:35
@leninmehedy
docs: add design doc
e5d3dc9 
Signed-off-by: Lenin Mehedy <lenin.mehedy@hashgraph.com>
@leninmehedy leninmehedy force-pushed the 00499-feat-solo-provisioner-daemon-core branch from 1803866 to e5d3dc9 Compare June 13, 2026 09:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Epic A38_2 — solo-provisioner-daemon Core

2 participants

@leninmehedy @swirlds-automation