feat(loadbalancers): new selector style annotation to assoc LB with VPC/PN#218
Conversation
… without pnNIC (#1) Signed-off-by: Steffen Karlsson <steffen.karlsson@gmail.com>
* feat(instance): avoid missing Node Refs for Private Network Instances without pnNIC Signed-off-by: Steffen Karlsson <steffen.karlsson@gmail.com> * fix(instances,lb): propagate IPAM errors and harden backend pool updates (scaleway#213) Prevent nodes from being initialized without InternalIPs when IPAM queries fail transiently. By returning errors instead of only logging warnings, the node keeps its uninitialized taint and Kubernetes retries until IPAM is available. Additionally, use internal IPs when the service annotation scw-loadbalancer-pn-ids is set (not just the global PN_ID env var), and refuse to clear existing LB backends when no replacement IPs are found. Co-authored-by: Nicklas Frahm <nfm@corti.ai> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * feat: env to disable tags sync from ccm * fix(examples): update container image for CCM Signed-off-by: Nicklas Frahm <nfm@corti.ai> * Update Scaleway CCM image version Signed-off-by: Andreas Wachs <awa@corti.ai> * Update Scaleway CCM image version to v0.36.0-kommodity.3 Signed-off-by: Andreas Wachs <awa@corti.ai> * WIP Signed-off-by: Andreas Wachs <awa@corti.ai> * Update k8s-scaleway-ccm-latest.yml Signed-off-by: Andreas Wachs <awa@corti.ai> --------- Signed-off-by: Steffen Karlsson <steffen.karlsson@gmail.com> Signed-off-by: Nicklas Frahm <nfm@corti.ai> Signed-off-by: Andreas Wachs <awa@corti.ai> Co-authored-by: Steffen Karlsson <steffen.karlsson@gmail.com> Co-authored-by: Jérémy THERIN <jtherin@users.noreply.github.com> Co-authored-by: Nicklas Frahm <nfm@corti.ai> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Jeremy THERIN <jtherin@scaleway.com>
feat(sync): align with upstream
…ssoc Signed-off-by: Andreas Wachs <awa@corti.ai>
|
I don't think we will accept this pr, vpc and pn names are not unique, duplicate names are allowed. this pr is too error prone |
|
There is a problem though @jtherin. When provisioning a cluster using cluster API, we can't predict the UUID, which means that we would need to manually look up the ID and add it to GitOps. Currently, there is no way to get a deterministic identifier / selector for a VPC. Do you have any alternative suggestion? As far as I know the VPC / PN name combination is unique within a project and the CCM is always linked to a single project AFAIK. Are we missing something? |
Kapsule does not create any PN, you provide a PN to kapsule api, so the issue is more likely on cluster api side
No, you can have many pns with the same name inside the same vpc inside the same project |
|
But this means that we need to pre-create the network with an external orchestration layer, which makes the process rather involved. Would it be an option to hide this behind a feature flag? |
|
Other CCMs, such as Azure allow you to do a similar thing. I feel like having multiple non-unique VPC name / PN names should be a bug not a feature. While not the same, Azure does allow you to associate a pre-created public IP by name using If you can't create a deterministic unique identifier, infrastructure is really hard to manage. |
|
Hey @jtherin. I hope that Nicklas comments could be taken up for debate in the team. Especially the comment on perhaps pairs of PVC name and PN names should perhaps be unique. IF that discussion is not able to materialize or you/the team are in disagreement with us on this, can I propose that we spin this feature into selecting VPC/PN's based on tags on the PNs instead? I see that tags on Scaleway resources are supported and we could add the ability to configure this on Our goal is to simplify our cluster operations and make cluster provisioning with Cluster API more automated and I see this as another solution to the same problem. |
Changes
LoadBalancernamedservice.beta.kubernetes.io/scw-loadbalancer-pn-namesin order to make GitOps easier by using pairs of<VPC-name>/<PN-name>[,<...>/<...>]as opposed to usingservice.beta.kubernetes.io/scw-loadbalancer-pn-idsthat might change over time or adds a second step of configuration when deploying clustersBackwards compatiblity
Since there are already settings for getting the LB associated with a specific PN, we put the priority as such for acting on specific configuration if multiple are present:
pn-ids(annotation) >pn-names(annotation) >PN_IDenv var.Key details
vpc-name/pn-namerequired to avoid ambiguity across VPCs. Comma-separated for multiple networks.getNodeIPsForPrivateNetworks— queries each node's private NICs and resolves IPs through IPAM rather than relying onextractNodesInternalIpswhich may return IPs from unrelated private networks.pn-names(andpn-ids) so private LBs no longer require thePN_IDenv var when annotations are set.This has been tested to work in a self-managed Kubernetes cluster running on Scaleway VMs.