Skip to content

fix: accumulating context.WithTimeout cancel funcs in retry loop (goroutine/timer leak)#36

Open
amathxbt wants to merge 1 commit into
asksurf-ai:mainfrom
amathxbt:fix-context-cancel-retry-leak
Open

fix: accumulating context.WithTimeout cancel funcs in retry loop (goroutine/timer leak)#36
amathxbt wants to merge 1 commit into
asksurf-ai:mainfrom
amathxbt:fix-context-cancel-retry-leak

Conversation

@amathxbt

Copy link
Copy Markdown

Problem

In cli/request.go, doRequestWithRetry() defers cancel() inside the retry loop:

for triesLeft > 0 {
    if timeout := viper.GetDuration("rsh-timeout"); timeout > 0 {
        ctx, cancel := context.WithTimeout(req.Context(), timeout)
        defer cancel()  // BUG: accumulates until function returns
        req = req.WithContext(ctx)
    }
    // ...
}

With --rsh-retry=5 and a timeout, 6 cancel funcs accumulate on the defer stack. Each context.WithTimeout spawns an internal timer goroutine that stays alive until its cancel is called. This creates 6 simultaneous timer goroutines instead of 1.

Each req.WithContext(ctx) also chains contexts, so retry N carries N layers of deadline wrappers — an inner timeout cancellation can propagate outward unexpectedly.

Fix

Call cancel() explicitly after client.Do(req) completes (success, error, or retry) so each iteration cleans up its context and timer immediately.

Impact

Affects any invocation with both --rsh-timeout and --rsh-retry > 0 set, which is the recommended configuration for unreliable networks.

In doRequestWithRetry(), each retry iteration that has a timeout configured
creates a new context with context.WithTimeout and defers its cancel.
Because defer runs when the enclosing *function* returns (not when the
loop iteration ends), every retry's cancel accumulates on the defer stack
and is only called when doRequestWithRetry exits.

This means:
1. Contexts from previous retries are not cancelled promptly, leaking their
   goroutine/timer resources for the duration of all remaining retries.
2. On a long retry chain (e.g. rsh-retry=5 with a short timeout) this
   creates 5 live timer goroutines simultaneously instead of 1.

Fix: call cancel() explicitly after the client.Do() call (both success and
error paths) so each iteration cleans up its own context immediately.
@amathxbt

Copy link
Copy Markdown
Author

Hey @akasuv @HappySean2845 @hughzhou-gif @ZhimaoL, wanted to bring this one to your attention. Anyone running with both a timeout and retries set is hitting this, which is probably the majority of production usage. The timer goroutines stacking up is not going to crash anything immediately but it does build up quietly and the context chaining issue can cause some unexpected cancellation behaviour on longer retry sequences. The fix is just one line change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant