fix: classify invalid strategy component errors as unrecoverable - #3878
fix: classify invalid strategy component errors as unrecoverable#3878pujitha24 wants to merge 1 commit into
Conversation
Motivation: When a CSV's install strategy contains a permanently malformed component (e.g. a Deployment with an invalid name), the Kubernetes API server rejects the create/update with an "Invalid" (422) error. The deployment installer returned this error unclassified, so install.IsErrorUnrecoverable() treated it as retryable. The CSV state machine in operator.go then cycled InstallReady -> Failed (CSVReasonComponentFailed, retryable) -> Pending (CSVReasonNeedsReinstall) -> InstallReady forever, hot-looping instead of settling in Failed. This was the root cause of the flakiness in the FailForward e2e test disabled by PR operator-framework#3572. The effect is wasted reconciles, CSV phase oscillation, and event/log spam -- the CSV controller itself does not crash or otherwise affect other CSVs. Approach: - Add a classifyInstallError helper in pkg/controller/install/deployment.go that maps apierrors.IsInvalid (in addition to the existing apierrors.IsForbidden) to a StrategyError, and apply it to both the installCertRequirements and installDeployments error paths in Install(). - Add StrategyErrReasonComponentInvalid in pkg/controller/install/errors.go and register it in unrecoverableErrors, so IsErrorUnrecoverable() returns true and the CSV is correctly marked CSVReasonComponentFailedNoRetry, which is terminal (the existing top-of-transitionCSVState guard in operator.go keeps it in Failed rather than reinstalling). Validation: go build ./... go test ./pkg/controller/install/... ./pkg/controller/operators/olm/... Both pass, including a new regression test, TestInstallStrategyDeploymentInstallInvalidDeployment, which asserts that an apierrors.NewInvalid error from the deployment client causes Install() to return StrategyErrReasonComponentInvalid and that IsErrorUnrecoverable() reports it as unrecoverable. Note: the disabled e2e regression test in test/e2e/fail_forward_e2e_test.go (XWhen at line ~324, referencing this issue) requires a live cluster and was intentionally left disabled, as it cannot be validated in this environment. Fixes operator-framework#3573 Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
|
Hi @pujitha24. Thanks for your PR. I'm waiting for a operator-framework member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Pull request overview
This PR fixes a CSV reconciliation hot-loop by classifying Kubernetes API “Invalid” (422) errors encountered during install strategy execution as unrecoverable, allowing the CSV state machine to settle in a terminal Failed state instead of repeatedly retrying.
Changes:
- Add a new strategy error reason (
ComponentInvalid) and mark it unrecoverable viaunrecoverableErrors. - Introduce
classifyInstallErrorin the deployment strategy installer to convertapierrors.IsInvalid(andIsForbidden) intoStrategyErrorvalues. - Add a regression unit test asserting invalid deployment creation/update errors are surfaced as
ComponentInvalidand treated as unrecoverable.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| pkg/controller/install/errors.go | Adds ComponentInvalid and registers it as an unrecoverable strategy error reason. |
| pkg/controller/install/deployment.go | Wraps install strategy errors via classifyInstallError to properly classify Invalid/Forbidden failures. |
| pkg/controller/install/deployment_test.go | Adds a regression test ensuring invalid deployment errors are categorized as unrecoverable ComponentInvalid. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Motivation:
When a CSV's install strategy contains a permanently malformed
component (e.g. a Deployment with an invalid name), the Kubernetes API
server rejects the create/update with an "Invalid" (422) error. The
deployment installer returned this error unclassified, so
install.IsErrorUnrecoverable() treated it as retryable. The CSV state
machine in operator.go then cycled InstallReady -> Failed
(CSVReasonComponentFailed, retryable) -> Pending
(CSVReasonNeedsReinstall) -> InstallReady forever, hot-looping instead
of settling in Failed. This was the root cause of the flakiness in the
FailForward e2e test disabled by PR #3572. The effect is wasted
reconciles, CSV phase oscillation, and event/log spam -- the CSV
controller itself does not crash or otherwise affect other CSVs.
Approach:
pkg/controller/install/deployment.go that maps apierrors.IsInvalid
(in addition to the existing apierrors.IsForbidden) to a
StrategyError, and apply it to both the installCertRequirements and
installDeployments error paths in Install().
pkg/controller/install/errors.go and register it in
unrecoverableErrors, so IsErrorUnrecoverable() returns true and the
CSV is correctly marked CSVReasonComponentFailedNoRetry, which is
terminal (the existing top-of-transitionCSVState guard in operator.go
keeps it in Failed rather than reinstalling).
Validation:
go build ./...
go test ./pkg/controller/install/... ./pkg/controller/operators/olm/...
Both pass, including a new regression test,
TestInstallStrategyDeploymentInstallInvalidDeployment, which asserts
that an apierrors.NewInvalid error from the deployment client causes
Install() to return StrategyErrReasonComponentInvalid and that
IsErrorUnrecoverable() reports it as unrecoverable.
Note: the disabled e2e regression test in
test/e2e/fail_forward_e2e_test.go (XWhen at line ~324, referencing this
issue) requires a live cluster and was intentionally left disabled, as
it cannot be validated in this environment.
Fixes #3573
Signed-off-by: Pujitha Paladugu 10557236+pujitha24@users.noreply.github.com