Skip to content

fix: classify invalid strategy component errors as unrecoverable - #3878

Open
pujitha24 wants to merge 1 commit into
operator-framework:masterfrom
pujitha24:auto/issue-3573
Open

fix: classify invalid strategy component errors as unrecoverable#3878
pujitha24 wants to merge 1 commit into
operator-framework:masterfrom
pujitha24:auto/issue-3573

Conversation

@pujitha24

Copy link
Copy Markdown

Motivation:
When a CSV's install strategy contains a permanently malformed
component (e.g. a Deployment with an invalid name), the Kubernetes API
server rejects the create/update with an "Invalid" (422) error. The
deployment installer returned this error unclassified, so
install.IsErrorUnrecoverable() treated it as retryable. The CSV state
machine in operator.go then cycled InstallReady -> Failed
(CSVReasonComponentFailed, retryable) -> Pending
(CSVReasonNeedsReinstall) -> InstallReady forever, hot-looping instead
of settling in Failed. This was the root cause of the flakiness in the
FailForward e2e test disabled by PR #3572. The effect is wasted
reconciles, CSV phase oscillation, and event/log spam -- the CSV
controller itself does not crash or otherwise affect other CSVs.

Approach:

  • Add a classifyInstallError helper in
    pkg/controller/install/deployment.go that maps apierrors.IsInvalid
    (in addition to the existing apierrors.IsForbidden) to a
    StrategyError, and apply it to both the installCertRequirements and
    installDeployments error paths in Install().
  • Add StrategyErrReasonComponentInvalid in
    pkg/controller/install/errors.go and register it in
    unrecoverableErrors, so IsErrorUnrecoverable() returns true and the
    CSV is correctly marked CSVReasonComponentFailedNoRetry, which is
    terminal (the existing top-of-transitionCSVState guard in operator.go
    keeps it in Failed rather than reinstalling).

Validation:
go build ./...
go test ./pkg/controller/install/... ./pkg/controller/operators/olm/...
Both pass, including a new regression test,
TestInstallStrategyDeploymentInstallInvalidDeployment, which asserts
that an apierrors.NewInvalid error from the deployment client causes
Install() to return StrategyErrReasonComponentInvalid and that
IsErrorUnrecoverable() reports it as unrecoverable.

Note: the disabled e2e regression test in
test/e2e/fail_forward_e2e_test.go (XWhen at line ~324, referencing this
issue) requires a live cluster and was intentionally left disabled, as
it cannot be validated in this environment.

Fixes #3573

Signed-off-by: Pujitha Paladugu 10557236+pujitha24@users.noreply.github.com

Motivation:
When a CSV's install strategy contains a permanently malformed
component (e.g. a Deployment with an invalid name), the Kubernetes API
server rejects the create/update with an "Invalid" (422) error. The
deployment installer returned this error unclassified, so
install.IsErrorUnrecoverable() treated it as retryable. The CSV state
machine in operator.go then cycled InstallReady -> Failed
(CSVReasonComponentFailed, retryable) -> Pending
(CSVReasonNeedsReinstall) -> InstallReady forever, hot-looping instead
of settling in Failed. This was the root cause of the flakiness in the
FailForward e2e test disabled by PR operator-framework#3572. The effect is wasted
reconciles, CSV phase oscillation, and event/log spam -- the CSV
controller itself does not crash or otherwise affect other CSVs.

Approach:
- Add a classifyInstallError helper in
  pkg/controller/install/deployment.go that maps apierrors.IsInvalid
  (in addition to the existing apierrors.IsForbidden) to a
  StrategyError, and apply it to both the installCertRequirements and
  installDeployments error paths in Install().
- Add StrategyErrReasonComponentInvalid in
  pkg/controller/install/errors.go and register it in
  unrecoverableErrors, so IsErrorUnrecoverable() returns true and the
  CSV is correctly marked CSVReasonComponentFailedNoRetry, which is
  terminal (the existing top-of-transitionCSVState guard in operator.go
  keeps it in Failed rather than reinstalling).

Validation:
go build ./...
go test ./pkg/controller/install/... ./pkg/controller/operators/olm/...
Both pass, including a new regression test,
TestInstallStrategyDeploymentInstallInvalidDeployment, which asserts
that an apierrors.NewInvalid error from the deployment client causes
Install() to return StrategyErrReasonComponentInvalid and that
IsErrorUnrecoverable() reports it as unrecoverable.

Note: the disabled e2e regression test in
test/e2e/fail_forward_e2e_test.go (XWhen at line ~324, referencing this
issue) requires a live cluster and was intentionally left disabled, as
it cannot be validated in this environment.

Fixes operator-framework#3573

Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 27, 2026 20:29
@openshift-ci openshift-ci Bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 27, 2026
@openshift-ci

openshift-ci Bot commented Jul 27, 2026

Copy link
Copy Markdown

Hi @pujitha24. Thanks for your PR.

I'm waiting for a operator-framework member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci
openshift-ci Bot requested review from oceanc80 and tmshort July 27, 2026 20:30
@openshift-ci

openshift-ci Bot commented Jul 27, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign perdasilva for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a CSV reconciliation hot-loop by classifying Kubernetes API “Invalid” (422) errors encountered during install strategy execution as unrecoverable, allowing the CSV state machine to settle in a terminal Failed state instead of repeatedly retrying.

Changes:

  • Add a new strategy error reason (ComponentInvalid) and mark it unrecoverable via unrecoverableErrors.
  • Introduce classifyInstallError in the deployment strategy installer to convert apierrors.IsInvalid (and IsForbidden) into StrategyError values.
  • Add a regression unit test asserting invalid deployment creation/update errors are surfaced as ComponentInvalid and treated as unrecoverable.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
pkg/controller/install/errors.go Adds ComponentInvalid and registers it as an unrecoverable strategy error reason.
pkg/controller/install/deployment.go Wraps install strategy errors via classifyInstallError to properly classify Invalid/Forbidden failures.
pkg/controller/install/deployment_test.go Adds a regression test ensuring invalid deployment errors are categorized as unrecoverable ComponentInvalid.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CSV Install/Upgrade hot loop on bad bundle resource

2 participants