Track ResourceVersion in TaskAction watcher to prevent duplicate ADDED events

[pingsutw]

Problem

The ActionsClient.doWatch in actions/k8s/client.go does not track or pass a ResourceVersion when re-establishing the watch. When the watch channel closes (server-side timeout, network blip, etc.), watchLoop retries by calling doWatch again, which starts a fresh watch without a ResourceVersion. This causes the API server to replay all existing CRs as ADDED events, resulting in duplicate RecordAction gRPC calls.

This happens even without a server restart — Kubernetes watches have a server-side timeout (typically 5-10 minutes), after which the watch channel closes and reconnects.

Current Mitigation

An OppoBloomFilter was added in #6987 to deduplicate RecordAction calls at the application layer, which prevents the duplicate DB writes. However, the root cause (unnecessary event replays) remains.

Proposed Fix

Track the ResourceVersion from the last seen event and pass it when restarting the watch via ListOptions. On reconnect, the API server will only send events that occurred after that version. Handle 410 Gone (resource version too old / compacted) by resetting and doing a full relist.

References

• actions/k8s/client.go — doWatch and watchLoop methods
• K8s API Concepts: Efficient detection of changes

[machichima]

We set the Kubernetes timeout when starting with single binary (manager/) to 30 sec, that's why we will see watch disconnect and re-establish quite often

flyte/manager/config/config.go

Line 54 in fdceda4

Timeout: "30s",

[majiayu000]

Hi, I've been looking into this and traced the root cause.

Proposed approach: In actions/k8s/client.go: (1) Add a resourceVersion field to ActionsClient or watchLoop state to track the last seen ResourceVersion from watch events. (2) In doWatch, pass the tracked ResourceVersion via metav1.ListOptions when re-establishing the watch. (3) In watchLoop event processing, update the tracked ResourceVersion from each received event's metadata. (4) Handle 410 Gone (StatusReasonGone) errors by resetting resourceVersion to empty string and doing a fresh list+watch to re-sync. Follow standard K8s informer reconnect pattern documented in kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes.

I'll open a draft PR shortly. Happy to adjust based on your preference.

[majiayu000]

Hi, I've been looking into this and traced the root cause.

Proposed approach: In actions/k8s/client.go: (1) Add resourceVersion field to track last seen version from watch events. (2) Pass it via metav1.ListOptions in doWatch on reconnect. (3) Update tracked version from each event's metadata in watchLoop. (4) Handle 410 Gone by resetting resourceVersion to empty string and doing fresh list+watch. Standard K8s informer reconnect pattern. Single-file change with tests.

I'll open a draft PR shortly. Happy to adjust based on your preference.