Bug Report
What did you do? / Minimal Reproducible Example
We first deployed a five-replica Solr cluster, then modified the CR to scale down to three replicas.
The scale down triggers the solr-operator to perform the ManagedScaleDown operation, invoking the handleManagedCloudScaleDown function. Inside the function, the operator invokes the evictSinglePod function. If the call to the Solr transiently fails (due to transient network issue or transient unavailability of Solr), the operator writes an error message and never retries. Even if the transient fault is resolved later, the operator is stuck and never perform the scale down operation.
To reproduce this bug,
- apply the CR to deploy a 5 replica Solr cluster by changing replicas to 5 in the example: https://github.com/apache/solr-operator/blob/main/example/test_solrcloud.yaml
- Expose faults to the operator (this can be reliably reproduced using the ChaosMesh controller by injecting a network loss between the Solr pods and the Solr operator)
- Change the
replicas: 5 to replicas:3 in the CR to initiate the scale down.
- Observe that the operator reports an error log because it is unable to contact the Solr Pods for status
ERROR Error retrieving cluster status, cannot determine if pod has replicas {"controller": "solrcloud", "controllerGroup": "solr.apache.org", "controllerKind": "SolrCloud", "SolrCloud": {"name":"example","namespace":"default"}, "namespace": "default", "name": "example", "reconcileID": "66d08e1c-a6cd-4176-a792-c706177cc39f", "error": "Get \"http://example-solrcloud-common.default/solr/admin/collections?action=CLUSTERSTATUS&wt=json\": dial tcp 10.96.177.119:80: i/o timeout"}
- Remove the fault and observe that the operator never requeues and reconciles the request
What did you expect to see?
The operator should be able to scale down the cluster after the transient faults go away
What did you see instead?
The operator never requeues the request
Root Cause
The root cause of this bug is at https://github.com/apache/solr-operator/blob/b2c951052828f88e9fc415f54aec189b805117e9/controllers/solr_cluster_ops_util.go#L251C57-L251C71
where the operator does not handle the error returned by evictSinglePod but just directly returns normally.
The fix should be simple to add an error handling branch to requeue the request.
Bug Report
What did you do? / Minimal Reproducible Example
We first deployed a five-replica Solr cluster, then modified the CR to scale down to three replicas.
The scale down triggers the solr-operator to perform the ManagedScaleDown operation, invoking the
handleManagedCloudScaleDownfunction. Inside the function, the operator invokes theevictSinglePodfunction. If the call to the Solr transiently fails (due to transient network issue or transient unavailability of Solr), the operator writes an error message and never retries. Even if the transient fault is resolved later, the operator is stuck and never perform the scale down operation.To reproduce this bug,
replicas: 5toreplicas:3in the CR to initiate the scale down.What did you expect to see?
The operator should be able to scale down the cluster after the transient faults go away
What did you see instead?
The operator never requeues the request
Root Cause
The root cause of this bug is at https://github.com/apache/solr-operator/blob/b2c951052828f88e9fc415f54aec189b805117e9/controllers/solr_cluster_ops_util.go#L251C57-L251C71
where the operator does not handle the error returned by
evictSinglePodbut just directly returns normally.The fix should be simple to add an error handling branch to requeue the request.