Skip SecretHashes update on ansibleLimit-scoped dataplane deployments by lmiccini · Pull Request #1781 · openstack-k8s-operators/openstack-operator

lmiccini · 2026-01-27T16:35:07Z

For servicesOverride deployments, merge SecretHashes into the existing map without resetting it, since all nodes were touched.
For ansibleLimit-scoped deployments, skip the update entirely as not all nodes received the secrets.

softwarefactory-project-zuul · 2026-01-27T19:55:24Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/56ac80bd0e7547ad88350eb0206886b5

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 18m 47s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 23m 38s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 37m 31s
❌ adoption-standalone-to-crc-ceph-provider FAILURE in 3h 01m 55s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 51m 23s
❌ openstack-operator-docs-preview POST_FAILURE in 2m 32s

softwarefactory-project-zuul · 2026-02-02T23:02:48Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/db62c9cd33b34a538c7eccf243769b6a

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 02m 26s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 20m 56s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 36m 03s
❌ adoption-standalone-to-crc-ceph-provider FAILURE in 1h 46m 57s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 34m 08s
❌ openstack-operator-docs-preview POST_FAILURE in 3m 15s

softwarefactory-project-zuul · 2026-02-07T21:18:03Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/b5d3972863e64857b2da5055f867ef55

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 20m 43s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 21m 41s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 36m 22s
❌ adoption-standalone-to-crc-ceph-provider FAILURE in 2h 05m 30s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 43m 01s
✔️ openstack-operator-docs-preview SUCCESS in 3m 14s

lmiccini · 2026-02-08T06:33:06Z

/retest

lmiccini · 2026-02-08T09:20:21Z

recheck

lmiccini · 2026-02-08T09:55:32Z

/test openstack-operator-build-deploy-kuttl-4-18

lmiccini · 2026-02-12T07:21:29Z

/test functional

lmiccini · 2026-02-13T06:13:20Z

/test openstack-operator-build-deploy-kuttl-4-18

lmiccini · 2026-04-27T10:23:49Z

/test openstack-operator-build-deploy-kuttl-4-18

lmiccini · 2026-04-28T05:49:41Z

/test openstack-operator-build-deploy-kuttl-4-18

rabi · 2026-04-28T08:08:25Z

Why can't infra-operator block deletion of there is mismatch between latest deployment and nodeset secret hashes? I am not sure if we should go this path as it is othrogonal to the original design. Also, users should be discouraged to do secret rotations using ansible_limits. It is there for scenarios where nodes are not reachable for some reason and users are expected to do full deployment across all nodes of the nodeset eventually to keep things in sync.

lmiccini · 2026-04-29T05:40:05Z

Why can't infra-operator block deletion of there is mismatch between latest deployment and nodeset secret hashes? I am not sure if we should go this path as it is othrogonal to the original design. Also, users should be discouraged to do secret rotations using ansible_limits. It is there for scenarios where nodes are not reachable for some reason and users are expected to do full deployment across all nodes of the nodeset eventually to keep things in sync.

Thanks @rabi . One of the challenges with that approach is that the last deployment may not be representative of the overall status of the nodeset. AFAICU we would have to introspect the deployments to check whether ansible_limits was used and/or if only a subset of the services were deployed, and repeat this pattern not just in infra but other operators that need to know if the dataplane is in sync (keystone for appcred, possibly others?).
I thought having a generic "am I up to date" status field would be in line with the loosely coupled approach we have for the dataplane, plus we would be able to use this for credentials rotation (rabbitmq, appcred, etc) and also to facilitate the cleanup after we migrate from mirrored to quorum queues among others. I am sure there are more use cases we could use this for.

rabi · 2026-04-30T06:49:35Z

One of the challenges with that approach is that the last deployment may not be representative of the overall status of the nodeset

I am not sure if we need to solve that by adding per-node tracking or by introspecting all deployments. That would start to break the current NodeSet design and make it overly complex IMO.

For this requirement, I think we can keep it simple:

do not update NodeSet.Status.SecretHashes when ansibleLimit is set, since that deployment did not fully update the NodeSet
with servicesOverride, the last deployment SecretHashes map is only partial, so it should not fully match the NodeSet SecretHashes. This can be false negative but conservative.
block deletion whenever NodeSet.Status.SecretHashes and Deployment.Status.SecretHashes differ

With this approach, any scoped deployment, either via ansibleLimit or servicesOverride, keeps deletion blocked. Only a full unscoped deployment across all nodes brings the NodeSet back in sync.

We can add the ansibleLimit guard where SecretHashes are copied back to the NodeSet in internal/controller/dataplane/openstackdataplanenodeset_controller.go.

lmiccini · 2026-04-30T08:14:05Z

/test functional

centosinfra-prod-github-app · 2026-04-30T08:19:40Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdo/buildset/44b91fa09db54b80829d1eb5022cd8ba

✔️ openstack-k8s-operators-content-provider SUCCESS in 12m 54s
❌ podified-multinode-edpm-deployment-crc RETRY_LIMIT in 30s
❌ cifmw-crc-podified-edpm-baremetal RETRY_LIMIT in 28s
❌ openstack-operator-tempest-multinode RETRY_LIMIT in 30s
❌ openstack-operator-edpm-baremetal-minor-update RETRY_LIMIT in 28s

rabi · 2026-05-04T04:18:28Z

/lgtm

lmiccini · 2026-05-04T13:27:03Z

/test openstack-operator-build-deploy-kuttl-4-18

lmiccini · 2026-05-04T13:27:09Z

recheck

For servicesOverride deployments, merge SecretHashes into the existing map without resetting it, since all nodes were touched. For ansibleLimit-scoped deployments, skip the update entirely as not all nodes received the secrets. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

openshift-ci · 2026-05-05T05:39:57Z

New changes are detected. LGTM label has been removed.

openshift-ci · 2026-05-05T05:40:04Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: lmiccini
Once this PR has been reviewed and has the lgtm label, please ask for approval from rabi. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

centosinfra-prod-github-app · 2026-05-05T05:41:33Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.

openshift-ci · 2026-05-05T06:10:05Z

@lmiccini: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/openstack-operator-build-deploy-kuttl	`1698305`	link	true	`/test openstack-operator-build-deploy-kuttl`
ci/prow/openstack-operator-build-deploy-kuttl-4-18	`b043596`	link	true	`/test openstack-operator-build-deploy-kuttl-4-18`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

lmiccini requested review from fmount and stuggi January 27, 2026 16:35

lmiccini added do-not-merge/work-in-progress do-not-merge/hold labels Jan 27, 2026

openshift-ci Bot removed the do-not-merge/work-in-progress label Jan 27, 2026

openshift-ci Bot requested review from abays and dprince January 27, 2026 16:35

stuggi requested a review from slagle January 28, 2026 08:13

auniyal61 requested changes Jan 30, 2026

View reviewed changes

Comment thread api/core/v1beta1/openstackcontrolplane_webhook.go Outdated

openshift-ci Bot assigned auniyal61 Jan 30, 2026

openshift-merge-robot added the needs-rebase label Jan 30, 2026

lmiccini force-pushed the nodeset_rmqu_finalizer_configmap branch from 04cc55a to 1698305 Compare February 2, 2026 20:59

openshift-merge-robot removed the needs-rebase label Feb 2, 2026

SeanMooney mentioned this pull request Feb 6, 2026

Rabbitmq vhost and user support openstack-k8s-operators/nova-operator#1052

Merged

lmiccini force-pushed the nodeset_rmqu_finalizer_configmap branch 2 times, most recently from 3885c4a to c1fe8f8 Compare February 7, 2026 18:56

lmiccini force-pushed the nodeset_rmqu_finalizer_configmap branch 2 times, most recently from cbfbb7c to f52529a Compare February 8, 2026 15:01

lmiccini removed the do-not-merge/hold label Feb 10, 2026

lmiccini force-pushed the nodeset_rmqu_finalizer_configmap branch from f52529a to 017d2ca Compare February 10, 2026 06:41

lmiccini force-pushed the nodeset_rmqu_finalizer_configmap branch 2 times, most recently from 97bb482 to b1d9350 Compare February 12, 2026 15:46

lmiccini requested a review from slagle April 27, 2026 07:33

lmiccini force-pushed the nodeset_rmqu_finalizer_configmap branch from 187eecc to 9f7cae7 Compare April 27, 2026 07:38

This was referenced Apr 30, 2026

Add application credential finalizer management openstack-k8s-operators/nova-operator#1108

Open

Add application credential finalizer management openstack-k8s-operators/telemetry-operator#881

Open

lmiccini force-pushed the nodeset_rmqu_finalizer_configmap branch from 9f7cae7 to e816aa3 Compare April 30, 2026 07:58

lmiccini changed the title ~~Add secret rotation tracking with composite hash drift detection~~ Skip SecretHashes update on scoped dataplane deployments Apr 30, 2026

lmiccini requested a review from rabi April 30, 2026 08:00

rabi reviewed Apr 30, 2026

View reviewed changes

Comment thread internal/controller/dataplane/openstackdataplanenodeset_controller.go

lmiccini force-pushed the nodeset_rmqu_finalizer_configmap branch from e816aa3 to 13d1c14 Compare April 30, 2026 12:11

lmiccini changed the title ~~Skip SecretHashes update on scoped dataplane deployments~~ Skip SecretHashes update on ansibleLimit-scoped dataplane deployments Apr 30, 2026

lmiccini force-pushed the nodeset_rmqu_finalizer_configmap branch from 13d1c14 to 5761424 Compare April 30, 2026 12:28

openshift-ci Bot assigned rabi May 4, 2026

openshift-ci Bot added the lgtm label May 4, 2026

lmiccini force-pushed the nodeset_rmqu_finalizer_configmap branch from 5761424 to b043596 Compare May 5, 2026 05:39

openshift-ci Bot removed the lgtm label May 5, 2026

Conversation

lmiccini commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

softwarefactory-project-zuul Bot commented Jan 27, 2026

Uh oh!

Uh oh!

softwarefactory-project-zuul Bot commented Feb 2, 2026

Uh oh!

softwarefactory-project-zuul Bot commented Feb 7, 2026

Uh oh!

lmiccini commented Feb 8, 2026

Uh oh!

lmiccini commented Feb 8, 2026

Uh oh!

lmiccini commented Feb 8, 2026

Uh oh!

lmiccini commented Feb 12, 2026

Uh oh!

lmiccini commented Feb 13, 2026

Uh oh!

lmiccini commented Apr 27, 2026

Uh oh!

lmiccini commented Apr 28, 2026

Uh oh!

rabi commented Apr 28, 2026

Uh oh!

lmiccini commented Apr 29, 2026

Uh oh!

rabi commented Apr 30, 2026

Uh oh!

lmiccini commented Apr 30, 2026

Uh oh!

centosinfra-prod-github-app Bot commented Apr 30, 2026

Uh oh!

Uh oh!

rabi commented May 4, 2026

Uh oh!

lmiccini commented May 4, 2026

Uh oh!

lmiccini commented May 4, 2026

Uh oh!

openshift-ci Bot commented May 5, 2026

Uh oh!

openshift-ci Bot commented May 5, 2026

Uh oh!

centosinfra-prod-github-app Bot commented May 5, 2026

Uh oh!

openshift-ci Bot commented May 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

lmiccini commented Jan 27, 2026 •

edited

Loading