Skip to content

Allow node pool major upgrades#4479

Merged
openshift-merge-bot[bot] merged 1 commit intomainfrom
ahitacat/allow-major-upgrades
Apr 17, 2026
Merged

Allow node pool major upgrades#4479
openshift-merge-bot[bot] merged 1 commit intomainfrom
ahitacat/allow-major-upgrades

Conversation

@ahitacat
Copy link
Copy Markdown
Collaborator

@ahitacat ahitacat commented Mar 16, 2026

What

Create a new feature tag AllowMajorUpgrades that will allow node pool major upgrades. This PR only adds support to allow the change. However, cluster .y and .x upgrades are not yet in place, and CS requires more changes to this upgrade be complete.

Why

Adding initial support for node pools major upgrades, to start working on this feature sooner.

Special notes for your reviewer

Refactors and tests in different commits, once it is approved/lgtm I will squash them. Addressing also some related comments from #4720 and #4566

Comment thread internal/admission/admit_nodepool.go Outdated
@ahitacat ahitacat force-pushed the ahitacat/allow-major-upgrades branch from ec4e739 to 6b0d466 Compare March 16, 2026 10:48
Comment thread backend/pkg/controllers/upgradecontrollers/nodepool_version_controller.go Outdated
Comment thread backend/pkg/controllers/upgradecontrollers/nodepool_version_controller.go Outdated
Comment thread internal/admission/admit_nodepool.go Outdated
@ahitacat ahitacat added the ai-assisted AI/LLM tool was used to help create this MR label Mar 16, 2026
Comment thread internal/api/featureflags.go Outdated
machi1990 added a commit to machi1990/ARO-HCP that referenced this pull request Mar 21, 2026
…nodepool creation

While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool.

The PR adds the skew validation by performing the following rules:
- if the version is set
  - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor
  - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install  - validate that when control plane major != nodepool major version then only valid skews are allowed
     - for now the definition of valid skew is:
       if cp on 5.0 then the node pool can be on 4.21 or 4.22
       if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23
       if cp is on 5.2 then the node pool can be only on 4.23
       I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup
- if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor

The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
machi1990 added a commit to machi1990/ARO-HCP that referenced this pull request Mar 21, 2026
…nodepool creation

While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool.

The PR adds the skew validation by performing the following rules:
- if the version is set
  - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor
  - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install  - validate that when control plane major != nodepool major version then only valid skews are allowed
     - for now the definition of valid skew is:
       if cp on 5.0 then the node pool can be on 4.21 or 4.22
       if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23
       if cp is on 5.2 then the node pool can be only on 4.23
       I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup
- if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor

The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
machi1990 added a commit to machi1990/ARO-HCP that referenced this pull request Mar 21, 2026
…nodepool creation

While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool.

The PR adds the skew validation by performing the following rules:
- if the version is set
  - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor
  - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install  - validate that when control plane major != nodepool major version then only valid skews are allowed
     - for now the definition of valid skew is:
       if cp on 5.0 then the node pool can be on 4.21 or 4.22
       if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23
       if cp is on 5.2 then the node pool can be only on 4.23
       I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup
- if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor

The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
machi1990 added a commit to machi1990/ARO-HCP that referenced this pull request Mar 23, 2026
…nodepool creation

While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool.

The PR adds the skew validation by performing the following rules:
- if the version is set
  - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor
  - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install  - validate that when control plane major != nodepool major version then only valid skews are allowed
     - for now the definition of valid skew is:
       if cp on 5.0 then the node pool can be on 4.21 or 4.22
       if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23
       if cp is on 5.2 then the node pool can be only on 4.23
       I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup
- if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor

The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
machi1990 added a commit to machi1990/ARO-HCP that referenced this pull request Mar 23, 2026
…nodepool creation

While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool.

The PR adds the skew validation by performing the following rules:
- if the version is set
  - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor
  - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install  - validate that when control plane major != nodepool major version then only valid skews are allowed
     - for now the definition of valid skew is:
       if cp on 5.0 then the node pool can be on 4.21 or 4.22
       if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23
       if cp is on 5.2 then the node pool can be only on 4.23
       I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup
- if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor

The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
machi1990 added a commit to machi1990/ARO-HCP that referenced this pull request Mar 23, 2026
…nodepool creation

While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool.

The PR adds the skew validation by performing the following rules:
- if the version is set
  - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor
  - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install  - validate that when control plane major != nodepool major version then only valid skews are allowed
     - for now the definition of valid skew is:
       if cp on 5.0 then the node pool can be on 4.21 or 4.22
       if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23
       if cp is on 5.2 then the node pool can be only on 4.23
       I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup
- if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor

The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
cgiradkar pushed a commit to machi1990/ARO-HCP that referenced this pull request Mar 24, 2026
…nodepool creation

While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool.

The PR adds the skew validation by performing the following rules:
- if the version is set
  - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor
  - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install  - validate that when control plane major != nodepool major version then only valid skews are allowed
     - for now the definition of valid skew is:
       if cp on 5.0 then the node pool can be on 4.21 or 4.22
       if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23
       if cp is on 5.2 then the node pool can be only on 4.23
       I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup
- if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor

The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
cgiradkar pushed a commit that referenced this pull request Mar 25, 2026
…nodepool creation

While working on #4505 and further reviewing of #4554 I realised that the frontend was missing skew validation while creating the node pool.

The PR adds the skew validation by performing the following rules:
- if the version is set
  - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor
  - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install  - validate that when control plane major != nodepool major version then only valid skews are allowed
     - for now the definition of valid skew is:
       if cp on 5.0 then the node pool can be on 4.21 or 4.22
       if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23
       if cp is on 5.2 then the node pool can be only on 4.23
       I left a TODO to put this behind an AFEC flag once #4479 is merged and it'll done as a followup
- if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor

The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
cgiradkar pushed a commit to machi1990/ARO-HCP that referenced this pull request Mar 25, 2026
…nodepool creation

While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool.

The PR adds the skew validation by performing the following rules:
- if the version is set
  - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor
  - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install  - validate that when control plane major != nodepool major version then only valid skews are allowed
     - for now the definition of valid skew is:
       if cp on 5.0 then the node pool can be on 4.21 or 4.22
       if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23
       if cp is on 5.2 then the node pool can be only on 4.23
       I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup
- if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor

The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
cgiradkar pushed a commit to machi1990/ARO-HCP that referenced this pull request Mar 25, 2026
…nodepool creation

While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool.

The PR adds the skew validation by performing the following rules:
- if the version is set
  - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor
  - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install  - validate that when control plane major != nodepool major version then only valid skews are allowed
     - for now the definition of valid skew is:
       if cp on 5.0 then the node pool can be on 4.21 or 4.22
       if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23
       if cp is on 5.2 then the node pool can be only on 4.23
       I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup
- if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor

The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
cgiradkar pushed a commit to machi1990/ARO-HCP that referenced this pull request Mar 26, 2026
…nodepool creation

While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool.

The PR adds the skew validation by performing the following rules:
- if the version is set
  - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor
  - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install  - validate that when control plane major != nodepool major version then only valid skews are allowed
     - for now the definition of valid skew is:
       if cp on 5.0 then the node pool can be on 4.21 or 4.22
       if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23
       if cp is on 5.2 then the node pool can be only on 4.23
       I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup
- if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor

The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
cgiradkar pushed a commit to machi1990/ARO-HCP that referenced this pull request Mar 27, 2026
…nodepool creation

While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool.

The PR adds the skew validation by performing the following rules:
- if the version is set
  - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor
  - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install  - validate that when control plane major != nodepool major version then only valid skews are allowed
     - for now the definition of valid skew is:
       if cp on 5.0 then the node pool can be on 4.21 or 4.22
       if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23
       if cp is on 5.2 then the node pool can be only on 4.23
       I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup
- if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor

The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
cgiradkar pushed a commit to machi1990/ARO-HCP that referenced this pull request Mar 30, 2026
…nodepool creation

While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool.

The PR adds the skew validation by performing the following rules:
- if the version is set
  - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor
  - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install  - validate that when control plane major != nodepool major version then only valid skews are allowed
     - for now the definition of valid skew is:
       if cp on 5.0 then the node pool can be on 4.21 or 4.22
       if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23
       if cp is on 5.2 then the node pool can be only on 4.23
       I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup
- if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor

The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
@ahitacat ahitacat force-pushed the ahitacat/allow-major-upgrades branch from 6b0d466 to c5a281c Compare March 31, 2026 11:35
@ahitacat ahitacat force-pushed the ahitacat/allow-major-upgrades branch 2 times, most recently from 7231c62 to d49bc32 Compare March 31, 2026 11:46
cgiradkar pushed a commit to machi1990/ARO-HCP that referenced this pull request Mar 31, 2026
…nodepool creation

While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool.

The PR adds the skew validation by performing the following rules:
- if the version is set
  - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor
  - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install  - validate that when control plane major != nodepool major version then only valid skews are allowed
     - for now the definition of valid skew is:
       if cp on 5.0 then the node pool can be on 4.21 or 4.22
       if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23
       if cp is on 5.2 then the node pool can be only on 4.23
       I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup
- if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor

The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
@ahitacat ahitacat force-pushed the ahitacat/allow-major-upgrades branch from 281317e to c8c060f Compare April 8, 2026 12:06
Comment thread internal/api/constants.go Outdated

// Validate the customer's desired version before setting it
if err := c.validateDesiredNodePoolVersion(ctx, &customerDesiredVersion, existingServiceProviderNodePool, existingServiceProviderCluster, nodePool.Properties.Version.ChannelGroup, clusterKey, clusterUUID); err != nil {
if err := c.validateDesiredNodePoolVersion(ctx, &customerDesiredVersion, existingServiceProviderNodePool, existingServiceProviderCluster, subscription, nodePool.Properties.Version.ChannelGroup, clusterKey, clusterUUID); err != nil {
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

normalization

something like this like we do in the frontend?

operation.Operation{
    Options: validation.AFECsToValidationOptions(subscription.GetRegisteredFeatures()),
}.HasOption(api.FeatureExperimentalReleaseFeatures)

@ahitacat ahitacat force-pushed the ahitacat/allow-major-upgrades branch from 0a9debb to 5ad7780 Compare April 10, 2026 10:19
Comment thread internal/api/constants.go Outdated
@JameelB
Copy link
Copy Markdown
Collaborator

JameelB commented Apr 10, 2026

lgtm, still see comments that's not yet resolved, ptal when you get a chance @geoberle 🙏

Comment thread internal/validation/validators.go Outdated
Comment thread internal/utils/apihelpers/version.go Outdated
Comment thread internal/utils/apihelpers/version.go Outdated
Comment thread internal/admission/admit_nodepool.go Outdated
Comment thread internal/admission/admit_nodepool.go Outdated
Comment thread backend/pkg/controllers/upgradecontrollers/nodepool_version_controller.go Outdated
cgiradkar pushed a commit to machi1990/ARO-HCP that referenced this pull request Apr 13, 2026
…nodepool creation

While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool.

The PR adds the skew validation by performing the following rules:
- if the version is set
  - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor
  - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install  - validate that when control plane major != nodepool major version then only valid skews are allowed
     - for now the definition of valid skew is:
       if cp on 5.0 then the node pool can be on 4.21 or 4.22
       if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23
       if cp is on 5.2 then the node pool can be only on 4.23
       I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup
- if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor

The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
@ahitacat ahitacat force-pushed the ahitacat/allow-major-upgrades branch 2 times, most recently from 8f45f3f to fadc3ec Compare April 15, 2026 09:34
Comment thread internal/api/constants.go Outdated
Comment thread internal/api/constants.go Outdated
Comment thread internal/api/constants.go Outdated
@machi1990
Copy link
Copy Markdown
Collaborator

left minor suggestions, otherwise lgtm

cgiradkar pushed a commit to machi1990/ARO-HCP that referenced this pull request Apr 15, 2026
…nodepool creation

While working on Azure#4505 and further reviewing of Azure#4554 I realised that the frontend was missing skew validation while creating the node pool.

The PR adds the skew validation by performing the following rules:
- if the version is set
  - validate that node pool desired minor version is within the maximum of 2 minors lower skew and not higher than the control plane minor
  - validate that the desired node pool version isn't higher that the lowest active control plane version if this is available: only available post install  - validate that when control plane major != nodepool major version then only valid skews are allowed
     - for now the definition of valid skew is:
       if cp on 5.0 then the node pool can be on 4.21 or 4.22
       if cp is on 5.1 then the node pool can be on 4.21, 4.22 or 4.23
       if cp is on 5.2 then the node pool can be only on 4.23
       I left a TODO to put this behind an AFEC flag once Azure#4479 is merged and it'll done as a followup
- if the lowest control plane version is available and node pool version set, perform the same minor skew as done for the cluster desired version but using the actual cluster version minor

The change also introduces some defaulting logic in case version isn't set: in this case we default to the lowest active control plane version.
@ahitacat ahitacat force-pushed the ahitacat/allow-major-upgrades branch from fadc3ec to 37673ef Compare April 17, 2026 12:14
@openshift-ci
Copy link
Copy Markdown

openshift-ci bot commented Apr 17, 2026

@ahitacat: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/cspr fadc3ec link true /test cspr

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Copy link
Copy Markdown
Collaborator

@machi1990 machi1990 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once the unit test pass the change looks good to me

Use the subscription AFEC flag experimental feature to allow major
upgrades. This is used in both the frontend and the controller that
calculates the desiredVersion.
Refactors the node pool upgrade validations so they can be shared
by frontend and backend.
Refactor apihelpers functions and shared variables.

Signed-off-by: Alba Hita Catala <ahitacat@redhat.com>
@ahitacat ahitacat force-pushed the ahitacat/allow-major-upgrades branch from 37673ef to 6b3f6db Compare April 17, 2026 13:10
@machi1990
Copy link
Copy Markdown
Collaborator

/lgtm

@geoberle
Copy link
Copy Markdown
Collaborator

/approve
/lgtm

@openshift-ci
Copy link
Copy Markdown

openshift-ci bot commented Apr 17, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahitacat, geoberle, machi1990

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot openshift-merge-bot bot merged commit 0d0fd7f into main Apr 17, 2026
23 checks passed
@openshift-merge-bot openshift-merge-bot bot deleted the ahitacat/allow-major-upgrades branch April 17, 2026 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-assisted AI/LLM tool was used to help create this MR approved lgtm

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants