feat: resilient Watch RPC with reconnect and resourceVersion tracking

[AdilFayyaz]

Tracking issue

Depends on: #7176, #7175, #7166

Why are the changes needed?

K8s watches time out every ~5 minutes by default. The previous implementation closed the client's stream silently on disconnect with no reconnect, making Watch unreliable for long-lived connections.

What changes were proposed in this pull request?

• Replaced the single-shot watch goroutine in AppK8sClient.Watch() with a reconnect loop (watchLoop + drainWatcher) that transparently reopens the K8s watch on unexpected closes or Error events
• Added resourceVersion tracking — extracted from every Added/Modified/Deleted/Bookmark event and passed to the next watch call, ensuring no events are missed or replayed across reconnects
• Added exponential backoff (1s → 2s → 4s → 30s max) between reconnect attempts; backoff resets on any successful event or Bookmark
• K8s Error events are now logged with code/reason/message instead of being silently dropped

How was this patch tested?

• go test ./app/internal/k8s/... -run TestWatch — 6 new tests covering:
  channel close reconnect, Error event reconnect, Bookmark RV propagation to next watch call, exponential backoff timing, ctx cancel stops the goroutine, initial watch error surfaces synchronously
• go test ./app/... — full suite passes with no regressions

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

• main
  • Flyte 2 #6583
    • feat: resilient Watch RPC with reconnect and resourceVersion tracking 👈
      • feat: path based ingress routing for Flyte apps via Traefik + Knative #7197