Skip to content

AUTOSCALE-615: include Karpenter node vCPUs in billing metric #8265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-bot merged 1 commit into from
May 1, 2026
Merged

AUTOSCALE-615: include Karpenter node vCPUs in billing metric #8265

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions api/hypershift/v1beta1/hostedcluster_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2205,6 +2205,16 @@ type AutoNodeStatus struct {
// +kubebuilder:validation:Minimum=0
// +optional
NodeClaimCount *int32 `json:"nodeClaimCount,omitempty"`

// vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
// that have registered and reported capacity. This is the sum of CPU capacity
// from each NodeClaim whose corresponding node exists (status.nodeName is set).
// This value is 0 when no Karpenter nodes are provisioned.
// Used by the metrics collector for billing aggregation.
// +kubebuilder:validation:Minimum=0

@JoelSpeed JoelSpeed Apr 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can have no vCPUs? When would you have 0?

Can we set a sensible upper limit for this value?

@maxcao13 maxcao13 Apr 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We would have 0 presumably when there are no Karpenter nodes available to aggregate the vcpus from.

I'm not sure what a sensible limit would be here, isn't it unbounded? This is the total number of vCPUS across all provisioned Karpenter nodes. There's a soft HCP max node limit of 500 I believe, but I'm not sure there would be reason to cap it since instance types will also vary in size.

@JoelSpeed JoelSpeed Apr 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, so you would want 0 to be displayed to the end user when no nodes are provisioned? In which case this is good as is

I'm not sure what a sensible limit would be here, isn't it unbounded? This is the total number of vCPUS across all provisioned Karpenter nodes. There's a soft HCP max node limit of 500 I believe, but I'm not sure there would be reason to cap it since instance types will also vary in size.

Ok so in this case, there are several things to consider.

Could you ever see a cluster having more than 1,000,000 cores for example? (10k nodes with 1k CPU each?)

If yes, how large could it go? Could it be more than 2^32?

Are there any consumers of this API that might not be able to handle supremely large numbers?

@joshbranham joshbranham Apr 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure what the upper limit of vCPU would be, but yeah we have a soft cap of 500 nodes in ROSA HCP. If you can come up with a very high limit then that seems fine. Right now the underlying metric we use for billing is exposed by hypershift-operator as a prom metric. There should be no other consumers of this API that I am aware of, given it's driven purely from a service provider perspective with a need to track billing for NodePools in Hypershift and Karpenter.

@maxcao13 maxcao13 Apr 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Theoretically with the 500 soft cap, and the fact that the largest CPU-focused instance type supported by Karpenter has 192 vCPUs, we could have some arbitrary max limit of 500 * 192 * 2 (margin) = 192000. But I know Joel, you may or may not be looking at some etcd sharding stuff that could increase this node limit 😄, plus instance type maximums can change as well. But even with the case with 10k nodes and 1k cpu each, 1,000,000 cores, it should never happen in the ROSA case and the only consumers of the API should be able to handle that.

If we really need a number here to satisfy openshift API standards, my instinct is to just make the maximum the golang math.MaxInt number. Would that be fine?

EDIT: As per slack, setting it to 1,000,000 as a reasonable upper limit we will never get to.

// +kubebuilder:validation:Maximum=1000000
// +optional
VCPUs *int32 `json:"vcpus,omitempty"`
}

// PlatformStatus contains platform-specific status
Expand Down
5 changes: 5 additions & 0 deletions api/hypershift/v1beta1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 11 additions & 0 deletions ...erated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/AAA_ungated.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6523,6 +6523,17 @@ spec:
format: int32
minimum: 0
type: integer
vcpus:
description: |-
vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
that have registered and reported capacity. This is the sum of CPU capacity
from each NodeClaim whose corresponding node exists (status.nodeName is set).
This value is 0 when no Karpenter nodes are provisioned.
Used by the metrics collector for billing aggregation.
format: int32
maximum: 1000000
minimum: 0
type: integer
type: object
conditions:
description: |-
Expand Down
11 changes: 11 additions & 0 deletions ...egated-crd-manifests/hostedclusters.hypershift.openshift.io/ClusterUpdateAcceptRisks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6506,6 +6506,17 @@ spec:
format: int32
minimum: 0
type: integer
vcpus:
description: |-
vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
that have registered and reported capacity. This is the sum of CPU capacity
from each NodeClaim whose corresponding node exists (status.nodeName is set).
This value is 0 when no Karpenter nodes are provisioned.
Used by the metrics collector for billing aggregation.
format: int32
maximum: 1000000
minimum: 0
type: integer
type: object
conditions:
description: |-
Expand Down
11 changes: 11 additions & 0 deletions ...manifests/hostedclusters.hypershift.openshift.io/ClusterVersionOperatorConfiguration.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6526,6 +6526,17 @@ spec:
format: int32
minimum: 0
type: integer
vcpus:
description: |-
vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
that have registered and reported capacity. This is the sum of CPU capacity
from each NodeClaim whose corresponding node exists (status.nodeName is set).
This value is 0 when no Karpenter nodes are provisioned.
Used by the metrics collector for billing aggregation.
format: int32
maximum: 1000000
minimum: 0
type: integer
type: object
conditions:
description: |-
Expand Down
11 changes: 11 additions & 0 deletions ...rated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/ExternalOIDC.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6838,6 +6838,17 @@ spec:
format: int32
minimum: 0
type: integer
vcpus:
description: |-
vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
that have registered and reported capacity. This is the sum of CPU capacity
from each NodeClaim whose corresponding node exists (status.nodeName is set).
This value is 0 when no Karpenter nodes are provisioned.
Used by the metrics collector for billing aggregation.
format: int32
maximum: 1000000
minimum: 0
type: integer
type: object
conditions:
description: |-
Expand Down
11 changes: 11 additions & 0 deletions ...ests/hostedclusters.hypershift.openshift.io/ExternalOIDCWithUIDAndExtraClaimMappings.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6978,6 +6978,17 @@ spec:
format: int32
minimum: 0
type: integer
vcpus:
description: |-
vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
that have registered and reported capacity. This is the sum of CPU capacity
from each NodeClaim whose corresponding node exists (status.nodeName is set).
This value is 0 when no Karpenter nodes are provisioned.
Used by the metrics collector for billing aggregation.
format: int32
maximum: 1000000
minimum: 0
type: integer
type: object
conditions:
description: |-
Expand Down
11 changes: 11 additions & 0 deletions ...-crd-manifests/hostedclusters.hypershift.openshift.io/ExternalOIDCWithUpstreamParity.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6969,6 +6969,17 @@ spec:
format: int32
minimum: 0
type: integer
vcpus:
description: |-
vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
that have registered and reported capacity. This is the sum of CPU capacity
from each NodeClaim whose corresponding node exists (status.nodeName is set).
This value is 0 when no Karpenter nodes are provisioned.
Used by the metrics collector for billing aggregation.
format: int32
maximum: 1000000
minimum: 0
type: integer
type: object
conditions:
description: |-
Expand Down
11 changes: 11 additions & 0 deletions ...erated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/GCPPlatform.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6952,6 +6952,17 @@ spec:
format: int32
minimum: 0
type: integer
vcpus:
description: |-
vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
that have registered and reported capacity. This is the sum of CPU capacity
from each NodeClaim whose corresponding node exists (status.nodeName is set).
This value is 0 when no Karpenter nodes are provisioned.
Used by the metrics collector for billing aggregation.
format: int32
maximum: 1000000
minimum: 0
type: integer
type: object
conditions:
description: |-
Expand Down
11 changes: 11 additions & 0 deletions ...ated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/HCPEtcdBackup.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6571,6 +6571,17 @@ spec:
format: int32
minimum: 0
type: integer
vcpus:
description: |-
vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
that have registered and reported capacity. This is the sum of CPU capacity
from each NodeClaim whose corresponding node exists (status.nodeName is set).
This value is 0 when no Karpenter nodes are provisioned.
Used by the metrics collector for billing aggregation.
format: int32
maximum: 1000000
minimum: 0
type: integer
type: object
conditions:
description: |-
Expand Down
11 changes: 11 additions & 0 deletions ...fests/hostedclusters.hypershift.openshift.io/HyperShiftOnlyDynamicResourceAllocation.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6528,6 +6528,17 @@ spec:
format: int32
minimum: 0
type: integer
vcpus:
description: |-
vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
that have registered and reported capacity. This is the sum of CPU capacity
from each NodeClaim whose corresponding node exists (status.nodeName is set).
This value is 0 when no Karpenter nodes are provisioned.
Used by the metrics collector for billing aggregation.
format: int32
maximum: 1000000
minimum: 0
type: integer
type: object
conditions:
description: |-
Expand Down
11 changes: 11 additions & 0 deletions ...turegated-crd-manifests/hostedclusters.hypershift.openshift.io/ImageStreamImportMode.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6524,6 +6524,17 @@ spec:
format: int32
minimum: 0
type: integer
vcpus:
description: |-
vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
that have registered and reported capacity. This is the sum of CPU capacity
from each NodeClaim whose corresponding node exists (status.nodeName is set).
This value is 0 when no Karpenter nodes are provisioned.
Used by the metrics collector for billing aggregation.
format: int32
maximum: 1000000
minimum: 0
type: integer
type: object
conditions:
description: |-
Expand Down
11 changes: 11 additions & 0 deletions ...turegated-crd-manifests/hostedclusters.hypershift.openshift.io/KMSEncryptionProvider.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6582,6 +6582,17 @@ spec:
format: int32
minimum: 0
type: integer
vcpus:
description: |-
vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
that have registered and reported capacity. This is the sum of CPU capacity
from each NodeClaim whose corresponding node exists (status.nodeName is set).
This value is 0 when no Karpenter nodes are provisioned.
Used by the metrics collector for billing aggregation.
format: int32
maximum: 1000000
minimum: 0
type: integer
type: object
conditions:
description: |-
Expand Down
11 changes: 11 additions & 0 deletions ...enerated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/OpenStack.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7057,6 +7057,17 @@ spec:
format: int32
minimum: 0
type: integer
vcpus:
description: |-
vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
that have registered and reported capacity. This is the sum of CPU capacity
from each NodeClaim whose corresponding node exists (status.nodeName is set).
This value is 0 when no Karpenter nodes are provisioned.
Used by the metrics collector for billing aggregation.
format: int32
maximum: 1000000
minimum: 0
type: integer
type: object
conditions:
description: |-
Expand Down
11 changes: 11 additions & 0 deletions ...rated.featuregated-crd-manifests/hostedclusters.hypershift.openshift.io/TLSAdherence.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6546,6 +6546,17 @@ spec:
format: int32
minimum: 0
type: integer
vcpus:
description: |-
vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
that have registered and reported capacity. This is the sum of CPU capacity
from each NodeClaim whose corresponding node exists (status.nodeName is set).
This value is 0 when no Karpenter nodes are provisioned.
Used by the metrics collector for billing aggregation.
format: int32
maximum: 1000000
minimum: 0
type: integer
type: object
conditions:
description: |-
Expand Down
11 changes: 11 additions & 0 deletions ...d.featuregated-crd-manifests/hostedcontrolplanes.hypershift.openshift.io/AAA_ungated.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6328,6 +6328,17 @@ spec:
format: int32
minimum: 0
type: integer
vcpus:
description: |-
vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
that have registered and reported capacity. This is the sum of CPU capacity
from each NodeClaim whose corresponding node exists (status.nodeName is set).
This value is 0 when no Karpenter nodes are provisioned.
Used by the metrics collector for billing aggregation.
format: int32
maximum: 1000000
minimum: 0
type: integer
type: object
conditions:
description: |-
Expand Down
11 changes: 11 additions & 0 deletions ...d-crd-manifests/hostedcontrolplanes.hypershift.openshift.io/ClusterUpdateAcceptRisks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6311,6 +6311,17 @@ spec:
format: int32
minimum: 0
type: integer
vcpus:
description: |-
vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
that have registered and reported capacity. This is the sum of CPU capacity
from each NodeClaim whose corresponding node exists (status.nodeName is set).
This value is 0 when no Karpenter nodes are provisioned.
Used by the metrics collector for billing aggregation.
format: int32
maximum: 1000000
minimum: 0
type: integer
type: object
conditions:
description: |-
Expand Down
11 changes: 11 additions & 0 deletions ...ests/hostedcontrolplanes.hypershift.openshift.io/ClusterVersionOperatorConfiguration.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6331,6 +6331,17 @@ spec:
format: int32
minimum: 0
type: integer
vcpus:
description: |-
vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
that have registered and reported capacity. This is the sum of CPU capacity
from each NodeClaim whose corresponding node exists (status.nodeName is set).
This value is 0 when no Karpenter nodes are provisioned.
Used by the metrics collector for billing aggregation.
format: int32
maximum: 1000000
minimum: 0
type: integer
type: object
conditions:
description: |-
Expand Down
11 changes: 11 additions & 0 deletions ....featuregated-crd-manifests/hostedcontrolplanes.hypershift.openshift.io/ExternalOIDC.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6643,6 +6643,17 @@ spec:
format: int32
minimum: 0
type: integer
vcpus:
description: |-
vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
that have registered and reported capacity. This is the sum of CPU capacity
from each NodeClaim whose corresponding node exists (status.nodeName is set).
This value is 0 when no Karpenter nodes are provisioned.
Used by the metrics collector for billing aggregation.
format: int32
maximum: 1000000
minimum: 0
type: integer
type: object
conditions:
description: |-
Expand Down
11 changes: 11 additions & 0 deletions ...hostedcontrolplanes.hypershift.openshift.io/ExternalOIDCWithUIDAndExtraClaimMappings.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6783,6 +6783,17 @@ spec:
format: int32
minimum: 0
type: integer
vcpus:
description: |-
vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
that have registered and reported capacity. This is the sum of CPU capacity
from each NodeClaim whose corresponding node exists (status.nodeName is set).
This value is 0 when no Karpenter nodes are provisioned.
Used by the metrics collector for billing aggregation.
format: int32
maximum: 1000000
minimum: 0
type: integer
type: object
conditions:
description: |-
Expand Down
11 changes: 11 additions & 0 deletions ...manifests/hostedcontrolplanes.hypershift.openshift.io/ExternalOIDCWithUpstreamParity.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6774,6 +6774,17 @@ spec:
format: int32
minimum: 0
type: integer
vcpus:
description: |-
vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
that have registered and reported capacity. This is the sum of CPU capacity
from each NodeClaim whose corresponding node exists (status.nodeName is set).
This value is 0 when no Karpenter nodes are provisioned.
Used by the metrics collector for billing aggregation.
format: int32
maximum: 1000000
minimum: 0
type: integer
type: object
conditions:
description: |-
Expand Down
11 changes: 11 additions & 0 deletions ...d.featuregated-crd-manifests/hostedcontrolplanes.hypershift.openshift.io/GCPPlatform.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6757,6 +6757,17 @@ spec:
format: int32
minimum: 0
type: integer
vcpus:
description: |-
vcpus is the total number of virtual CPUs across all Karpenter-managed nodes
that have registered and reported capacity. This is the sum of CPU capacity
from each NodeClaim whose corresponding node exists (status.nodeName is set).
This value is 0 when no Karpenter nodes are provisioned.
Used by the metrics collector for billing aggregation.
format: int32
maximum: 1000000
minimum: 0
type: integer
type: object
conditions:
description: |-
Expand Down
Loading
Loading