Probes configuration and override support by fmount · Pull Request #610 · openstack-k8s-operators/cinder-operator

fmount · 2026-02-18T20:21:51Z

Add ProbeOverrides interface and CreateProbeSet() function from lib-common for unified probe management across Cinder services. Enable probe customization through CRD overrides and remove code duplication. Updates all services (API, Scheduler, Volume, Backup) to use the new pattern with proper scheme handling and consistent defaults.
In addition, webhook validation for probes have been introduced.

Depends-On: openstack-k8s-operators/lib-common#673

softwarefactory-project-zuul · 2026-02-18T20:32:40Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/2813b48c394243ce9a2e8ef372592913

❌ openstack-k8s-operators-content-provider FAILURE in 9m 19s
⚠️ cinder-operator-kuttl SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cinder-operator-tempest SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

softwarefactory-project-zuul · 2026-02-19T09:42:05Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/2896e982a2c54980ac9db89bd06ae45a

❌ openstack-k8s-operators-content-provider FAILURE in 11m 52s
⚠️ cinder-operator-kuttl SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cinder-operator-tempest SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

softwarefactory-project-zuul · 2026-03-11T15:04:44Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/fcdd543dbfb848aab77b2a7d913569b8

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 03m 39s
❌ cinder-operator-kuttl FAILURE in 34m 31s
✔️ cinder-operator-tempest SUCCESS in 1h 44m 57s

softwarefactory-project-zuul · 2026-03-23T13:13:47Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/b2cd3b0a467e469493dfd89037ba89ad

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 01m 18s
❌ cinder-operator-kuttl FAILURE in 36m 02s
✔️ cinder-operator-tempest SUCCESS in 1h 42m 23s

softwarefactory-project-zuul · 2026-04-07T18:00:58Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/28e584974ac74471995b2d48929d570d

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 14m 03s
❌ cinder-operator-kuttl FAILURE in 43m 02s
✔️ cinder-operator-tempest SUCCESS in 1h 47m 53s

fmount · 2026-04-10T07:49:06Z

recheck

fmount · 2026-04-24T12:35:08Z

Here's a quick comparison between what we had before this patch (in terms of defaults) and what we have now (regardless they are used or not):


  ┌───────────┬─────────────────────┬─────────────────┬────────────────┬────────┐
  │   Probe   │        Field        │ Old (hardcoded) │ New (computed) │ Change │
  ├───────────┼─────────────────────┼─────────────────┼────────────────┼────────┤
  │ Liveness  │ TimeoutSeconds      │ 5               │ 5              │ same   │
  ├───────────┼─────────────────────┼─────────────────┼────────────────┼────────┤
  │ Liveness  │ PeriodSeconds       │ 3               │ 20             │ +17    │
  ├───────────┼─────────────────────┼─────────────────┼────────────────┼────────┤
  │ Liveness  │ InitialDelaySeconds │ 3               │ 15             │ +12    │
  ├───────────┼─────────────────────┼─────────────────┼────────────────┼────────┤
  │ Readiness │ TimeoutSeconds      │ --              │ 5              │ new    │
  ├───────────┼─────────────────────┼─────────────────┼────────────────┼────────┤
  │ Readiness │ PeriodSeconds       │ --              │ 20             │ new    │
  ├───────────┼─────────────────────┼─────────────────┼────────────────┼────────┤
  │ Readiness │ InitialDelaySeconds │ --              │ 15             │ new    │
  ├───────────┼─────────────────────┼─────────────────┼────────────────┼────────┤
  │ Startup   │ TimeoutSeconds      │ 5               │ 5              │ same   │
  ├───────────┼─────────────────────┼─────────────────┼────────────────┼────────┤
  │ Startup   │ PeriodSeconds       │ 5               │ 10             │ +5     │
  ├───────────┼─────────────────────┼─────────────────┼────────────────┼────────┤
  │ Startup   │ InitialDelaySeconds │ 5               │ 20             │ +15    │
  ├───────────┼─────────────────────┼─────────────────┼────────────────┼────────┤
  │ Startup   │ FailureThreshold    │ 12              │ 12             │ same   │
  └───────────┴─────────────────────┴─────────────────┴────────────────┴────────┘

TristanCacqueray · 2026-04-29T17:15:33Z

recheck

centosinfra-prod-github-app · 2026-04-29T17:39:04Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdo/buildset/0bfe4cda658848298ae0d1af2cea59e5

✔️ openstack-k8s-operators-content-provider SUCCESS in 21m 04s
❌ cinder-operator-kuttl RETRY_LIMIT in 29s
❌ cinder-operator-tempest RETRY_LIMIT in 30s

centosinfra-prod-github-app · 2026-04-30T08:52:08Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdo/buildset/6da5f313602949f89d590c1c4fb221ba

✔️ openstack-k8s-operators-content-provider SUCCESS in 19m 34s
❌ cinder-operator-kuttl RETRY_LIMIT in 31s
❌ cinder-operator-tempest RETRY_LIMIT in 29s

TristanCacqueray · 2026-04-30T15:16:57Z

recheck

centosinfra-prod-github-app · 2026-04-30T17:29:53Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdo/buildset/63ed3ad49c4e48cb9eacdd55a1ac2873

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 11m 20s
❌ cinder-operator-kuttl FAILURE in 21m 44s
✔️ cinder-operator-tempest SUCCESS in 1h 53m 49s

fmount · 2026-05-05T07:42:19Z

-		InitialDelaySeconds: 5,
+
+	// Note that by default we create probes with the same URIScheme and port
+	apiProbes, err := probes.CreateProbeSet(


@abays this is exactly the same patch we did in manila, can you take a look and see if we can land this as well?

abays · 2026-05-05T09:18:32Z

 		})
 	})
+
+	When("Cinder CR instance is built with custom probes", func() {


Is there any particular reason CinderBackup isn't covered here as well?

The reason I think is that we currently do not test cinderBackup in envTests, and we should! Let me try to add cbak basic coverage for probes, and we can keep any refactoring for follow up patches.

tl;dr we don't test cinder-backup at all with envTest, but I'm going to enable it as part of an additional commit.

Ok I added basic support for CinderBackup. I think it deserves a dedicated PR to go through the interfaces and make sure we are able to cover everything, but here the scope is to try probes override, so we should be good. Thanks @abays, this is a good catch and as a result of that I enabled a missing component in our envTests.

Cool man 👍

Add ProbeOverrides interface and CreateProbeSet() function from lib-common for unified probe management across Cinder services. Enable probe customization through CRD overrides and remove code duplication. Updates all services (API, Scheduler, Volume, Backup) to use the new pattern with proper scheme handling and consistent defaults. In addition, webhook validation for probes have been introduced. Signed-off-by: Francesco Pantano <fpantano@redhat.com>

Replace static probe timeouts with dynamic scaling based on APITimeout parameter. Creates separate probe configurations for API services (HTTP endpoints) and RPC workers (internal services) with appropriate scaling factors. API services use full APITimeout scaling while RPC workers get proportional timeouts, preventing premature pod kills during high load scenarios. Signed-off-by: Francesco Pantano <fpantano@redhat.com>

This patch introduces a document where the design decisions related to the probes settings are described. Signed-off-by: Francesco Pantano <fpantano@redhat.com>

This patch enables cinder-backup in envTests and extend the probes override test to this component. GetDefaultCinderSpec is now extended to return cinderBackup as part of the top-level CR and we can now test overrides for probes. Signed-off-by: Francesco Pantano <fpantano@redhat.com>

abays

/lgtm

openshift-ci · 2026-05-05T10:25:48Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abays, fmount

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [abays,fmount]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

fmount requested a review from stuggi February 18, 2026 20:21

openshift-ci Bot requested a review from eharney February 18, 2026 20:21

openshift-ci Bot added the approved label Feb 18, 2026

fmount changed the title ~~Probe configuration with override support~~ Probes configuration and override support Feb 18, 2026

fmount added the do-not-merge/hold label Feb 18, 2026

fmount force-pushed the probes branch from b8906c9 to 6522840 Compare February 19, 2026 09:29

fmount mentioned this pull request Feb 23, 2026

Introduce configurable interface for probe management openstack-k8s-operators/lib-common#673

Merged

fmount force-pushed the probes branch from 6522840 to 17f9a00 Compare March 11, 2026 12:59

openshift-ci Bot added the needs-rebase label Mar 18, 2026

fmount force-pushed the probes branch from 17f9a00 to 1375d34 Compare March 23, 2026 11:11

openshift-ci Bot removed the needs-rebase label Mar 23, 2026

fmount commented Mar 23, 2026

View reviewed changes

Comment thread internal/cinder/funcs.go

fmount removed the do-not-merge/hold label Mar 23, 2026

fmount force-pushed the probes branch from 1375d34 to 34c0582 Compare March 23, 2026 13:21

fmount force-pushed the probes branch from 34c0582 to 9987d32 Compare April 7, 2026 15:45

fmount force-pushed the probes branch 2 times, most recently from b368ecf to 6d648b3 Compare April 20, 2026 10:05

fmount requested a review from abays April 29, 2026 12:07

fmount force-pushed the probes branch from 63fd761 to d3f12de Compare April 29, 2026 13:13

fmount force-pushed the probes branch from d3f12de to 3ea3b1c Compare April 30, 2026 08:24

fmount force-pushed the probes branch from 3ea3b1c to e64cebd Compare May 4, 2026 15:11

fmount commented May 5, 2026

View reviewed changes

abays reviewed May 5, 2026

View reviewed changes

Comment thread api/v1beta1/cinderbackup_types.go Outdated

abays reviewed May 5, 2026

View reviewed changes

fmount added 3 commits May 5, 2026 12:08

Document healthcheck design decisions

f870a94

This patch introduces a document where the design decisions related to the probes settings are described. Signed-off-by: Francesco Pantano <fpantano@redhat.com>

fmount force-pushed the probes branch from f528e3e to 6fdb15c Compare May 5, 2026 10:08

fmount force-pushed the probes branch from 6fdb15c to 30ff07a Compare May 5, 2026 10:10

fmount requested a review from abays May 5, 2026 10:12

abays approved these changes May 5, 2026

View reviewed changes

openshift-ci Bot assigned abays May 5, 2026

openshift-ci Bot added the lgtm label May 5, 2026

openshift-merge-bot Bot merged commit e0a83a3 into openstack-k8s-operators:main May 5, 2026
6 checks passed

Conversation

fmount commented Feb 18, 2026

Uh oh!

softwarefactory-project-zuul Bot commented Feb 18, 2026

Uh oh!

softwarefactory-project-zuul Bot commented Feb 19, 2026

Uh oh!

softwarefactory-project-zuul Bot commented Mar 11, 2026

Uh oh!

Uh oh!

softwarefactory-project-zuul Bot commented Mar 23, 2026

Uh oh!

softwarefactory-project-zuul Bot commented Apr 7, 2026

Uh oh!

fmount commented Apr 10, 2026

Uh oh!

fmount commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TristanCacqueray commented Apr 29, 2026

Uh oh!

centosinfra-prod-github-app Bot commented Apr 29, 2026

Uh oh!

centosinfra-prod-github-app Bot commented Apr 30, 2026

Uh oh!

TristanCacqueray commented Apr 30, 2026

Uh oh!

centosinfra-prod-github-app Bot commented Apr 30, 2026

Uh oh!

fmount May 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

abays May 5, 2026

Choose a reason for hiding this comment

Uh oh!

fmount May 5, 2026

Choose a reason for hiding this comment

Uh oh!

fmount May 5, 2026

Choose a reason for hiding this comment

Uh oh!

abays May 5, 2026

Choose a reason for hiding this comment

Uh oh!

abays left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci Bot commented May 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fmount commented Apr 24, 2026 •

edited

Loading