Skip to content

Refactor update config to use automatic retry of failed nodes for cluster fan out.#391

Open
ArgusLi wants to merge 5 commits into
mainfrom
refactor/update-config
Open

Refactor update config to use automatic retry of failed nodes for cluster fan out.#391
ArgusLi wants to merge 5 commits into
mainfrom
refactor/update-config

Conversation

@ArgusLi

@ArgusLi ArgusLi commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Description

Rework cluster fanout end-to-end so:

  1. Node statuses will be classified into attempting, retrying, succeeded, failed, not_attempted.
    1. attempting is for nodes that are actively attempting.
    2. retrying is for nodes whose last attempt failed and are waiting out the backoff delay.
    3. succeeded is for nodes that have succeeded.
    4. failed is for nodes that have failed and have exhausted all retries.
    5. not_attempted is for nodes that have no resgistered metrics process and will not be retried.
  2. The server will create a session for a cluster fanout action. All further actions will supercede an earlier action.
  3. The server will fan out the action to all nodes within the cluster or the sole node in the case for standalone.
  4. The server collects per-node results, with a retry mechanism following Fibonacci backoff, along with a total limit on number of attempts.
  5. Per node statuses are streamed live to the frontend while the run is in progress.
  6. When the session resolves, the per-node results are aggregated into a single reply (sendConfigReply). The reply is keyed by one AggregateReplyId arm (clusterId for a cluster, db-less nodeId for a standalone) and carries:
    1. an outcome: fulfilled when every attempted node succeeded, failed when any attempted node failed, not_attempted when no targeted node had a registered metrics process;
    2. the full nodeResults (one per attempted node) and the notAttemptedNodeIds;
    3. on fulfilled only, the echoed appliedConfig.
      On the wire this is just two message types: updateConfigFulfilled (sent only for fulfilled) and updateConfigFailed (sent for both failed and not_attempted).

ArgusLi added 4 commits July 16, 2026 14:04
Signed-off-by: Argus Li <argus@argusli.dev>
Signed-off-by: Argus Li <argus@argusli.dev>
Signed-off-by: Argus Li <argus@argusli.dev>
Signed-off-by: Argus Li <argus@argusli.dev>
@ArgusLi
ArgusLi requested a review from arseny-kostenko July 20, 2026 17:33
Signed-off-by: Argus Li <argus@argusli.dev>
@ravjotbrar

Copy link
Copy Markdown
Collaborator

How would multiple users sharing the same Valkey deployment be impacted with config aborts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants