todo:
- multi-raft
- cluster splitting
- smart appendentries intervals for commands (immediate, interval-based batching...)
- speculative batch-writing for speed? (take snapshot, commit speculative, continue or revert to snapshot)
- broadcast replication lag to clients, to allow them to target non-linearizable reads to specific followers
- be consistent with "members" vs "nodes" nomenclature
- adaptive heartbeat intervals
- 4.2.1 server catch up "rounds" hueristic
- flow control (max bytes per message, sliding window, etc)
- don't deserialize entries when fetching from persistence for transmission to followers (also lets us know how big a message is going to be, for congestion control)
features:
- leader leases for fast linearizable reads (pluggable time-source support, batteries included with clockbound)
- linearizability checker with fault injector + visualizer
- cluster visualizer
- 3.10 leadership transfer extension
- PreVote
- CheckQuorum
- rocksdb backend
- fast snapshot transfer via sendfile sysctl
- linearizable follower reads via read-index
- write-optimized mode
telemetry events:
[
[:craft, :quorum, :heartbeat],
[:craft, :quorum, :succeeded],
[:craft, :check_quorum, :succeeded],
[:craft, :check_quorum, :failed],
[:craft, :persistence, :commit_buffer],
[:craft, :persistence, :fetch],
[:craft, :persistence, :fetch_between],
[:craft, :persistence, :fetch_from],
[:craft, :persistence, :reverse_find],
[:craft, :persistence, :rewind],
[:craft, :persistence, :truncate],
[:craft, :heartbeat, :reply, :duplicate],
[:craft, :heartbeat, :reply, :missed_deadline],
[:craft, :heartbeat, :reply, :out_of_order],
[:craft, :heartbeat, :reply, :round_expired],
[:craft, :role, :lonely],
[:craft, :role, :receiving_snapshot],
[:craft, :role, :follower],
[:craft, :role, :candidate],
[:craft, :role, :leader],
[:craft, :message, :sent, :append_entries],
[:craft, :message, :sent, :append_entries_results],
[:craft, :message, :sent, :request_vote],
[:craft, :message, :sent, :request_vote_results],
[:craft, :message, :sent, :install_snapshot],
[:craft, :message, :sent, :install_snapshot_results],
[:craft, :machine, :user, :handle_command],
[:craft, :machine, :user, :handle_commands],
[:craft, :machine, :user, :handle_query],
[:craft, :machine, :user, :snapshot]
]