Skip to content

Avoid tick starvation under sustained inbox load #24

@antondalgren

Description

@antondalgren

Server#tick_loop uses select with a timeout(50.milliseconds) arm to drive @node.tick. Crystal's select only fires the timeout case when no other case is ready — so under sustained inbox/actions load, tick is never called. raft uses tick for heartbeat sends (leader) and election timeouts (followers), so a busy node becomes less reactive the busier it gets.

Not a problem for low-frequency metadata Nodes — select is mostly idle, timeout fires reliably. Becomes a problem for high-throughput data-plane Nodes where every operation translates to an inbox message or propose.

Proposed solution

A dedicated tick-timer fiber that sends on a buffered(1) channel makes tick a first-class select case competing fairly with inbox/actions:

@tick_signal = Channel(Nil).new(1)
spawn { loop { sleep 50.ms; select when @tick_signal.send(nil); else end } }
# tick_loop's select replaces `when timeout(50.ms)` with `when @tick_signal.receive?`

The buffered(1) + select-else pattern coalesces missed ticks rather than backing them up.

The queue example uses the same select+timeout pattern and would benefit similarly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions