Skip to content

refactor: share watch processor lifecycle and use cancellable backoff #605

@abtreece

Description

@abtreece

Problem

The watch processors duplicate substantial lifecycle and watch-loop logic, and their backoff sleeps are not context-aware.

Verified in:

  • pkg/template/processor.go:161-231 for watchProcessor.Process.
  • pkg/template/processor.go:334-409 for batchWatchProcessor.Process.
  • Both paths reload template resources, calculate watched-key metrics, reset internal stop channels, start a stop-monitor goroutine, launch one monitor per template, wait, and restart on reload.
  • monitorPrefix and monitorForBatch both call WatchPrefix, update lastIndex, send errors, and sleep after errors (pkg/template/processor.go:234-304, pkg/template/processor.go:412-445).
  • Error backoff uses plain time.Sleep at pkg/template/processor.go:262 and pkg/template/processor.go:430.

Why this is a problem

The duplicated lifecycle code makes bug fixes easy to apply to one watch mode but miss in the other. The plain time.Sleep calls also delay shutdown or reload by up to WatchErrorBackoff whenever a watch backend is erroring.

The batch path also duplicates pending-flush logic for context cancellation and internal stop (pkg/template/processor.go:492-520 and following), which increases the surface area for subtle lifecycle bugs.

Suggested refactor

Extract shared watch orchestration:

  • A common resource-loading/reload loop.
  • A shared watch monitor helper that accepts an event handler callback.
  • Context-aware backoff using a timer/select on ctx.Done() and internalStop.
  • Shared pending-flush helper for batch mode.

Acceptance criteria

  • Shutdown and reload are not delayed by WatchErrorBackoff.
  • Watch and batch-watch share common monitor/reload plumbing.
  • Tests cover cancellation during watch error backoff.
  • Tests cover reload behavior for both immediate and batch watch modes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions