Skip to content

Ensure that decomission and dead node scenarios are handled properly #21

Description

@dkropachev

Jira task: https://scylladb.atlassian.net/browse/DRIVER-686
Jira epic: https://scylladb.atlassian.net/browse/DRIVER-38

Copied from Jira epic DRIVER-38:

We need to make sure that dynamodb requests are not failing when node is being decommissioned from the cluster.

Instead failed http request has to be retried on another host.

Idea of advanced handling logic:

  1. Have a success and error metrics for all the nodes
  2. When request is failing client adds to error metric based on what error is it
  3. When requests are failing on certain node put them in the list of broken nodes and exclude from list of active nodes
  4. Once on a while allow request to be routed to the broken node to see if it is back to normal
  5. If such requests succeeds remove node from list of broken node

Migrated from GitHub issue: scylladb/alternator-load-balancing#98

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions