fix: PUC-1722 Ironic conductor performance issues

[skrobul]

Overall this PR aims to resolve CPU throttling for Ironic conductors by adjusting existing, overly strict CPU limits as well as horizontally scaling to multiple pods/nodes.

Summary of changes:

• Ironic and dnsmasq are no longer coupled to a single host. dnsmasq stays on a same single node, the Ironic conductor can now live on any non-network node (2 replicas in most clusters).
• the PVC needs to change the mode from ReadWriteOnce(RWO) to ReadWriteMany(RWX) so that it can be accessed by multiple hosts at the same time
• this results in a storage class change and affinity rules change

Unfortunately changing the storage class requires a per-environment migration. Here is rough plan (tested on dev environment)

This partially solves https://rackspace.atlassian.net/browse/PUC-1722, but requires changes in the deploy repo to increase the number of replicas to 2 for each of the environments. These are done in https://github.com/RSS-Engineering/undercloud-deploy/pull/1799

---

RWO → RWX Volume Migration plan

Prerequisites

• pv-migrate kubectl plugin installed
• The ceph-fs-ec storage class will be used, which supports RWX

---

Overview

1. Disable ArgoCD to prevent reconciliation during migration
2. Scale down the ironic-conductor and ironic-dnsmasq StatefulSets
3. Back up existing PVC definitions to /tmp
4. Create temporary RWX PVCs (dnsmasq-dhcp-new, dnsmasq-ironic-new) on ceph-fs-ec
5. Migrate data from the old PVCs into the temporary ones using pv-migrate
6. Delete the old RWO PVCs
7. Recreate the PVCs with their original names on ceph-fs-ec
8. Migrate data from the temporary PVCs into the final ones
9. Delete the temporary PVCs
10. Scale StatefulSets back up and verify
11. Restore ArgoCD access

---

Step 1: Disable ArgoCD

Before making any changes, disable ArgoCD on the cluster to prevent it from reconciling resources during the migration.

k get clusterrolebinding pvceng-argocd-is-cluster-admin -o yaml > /tmp/argocd-binding.yaml
k delete clusterrolebinding pvceng-argocd-is-cluster-admin

---

Step 2: Scale Down StatefulSets

k scale statefulset ironic-conductor --replicas=0
k scale statefulset ironic-dnsmasq --replicas=0

---

Step 3: Back Up Existing PVC Definitions

k get pvc dnsmasq-dhcp -n openstack -o yaml | tee /tmp/backup-dnsmasq-dhcp.yaml
k get pvc dnsmasq-ironic -n openstack -o yaml | tee /tmp/backup-dnsmasq-ironic.yaml

---

Step 4: Create Temporary RWX PVCs

Since two PVCs cannot share the same name in the same namespace, use temporary names during migration.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dnsmasq-dhcp-new
  namespace: openstack
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ceph-fs-ec
  resources:
    requests:
      storage: 16Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dnsmasq-ironic-new
  namespace: openstack
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ceph-fs-ec
  resources:
    requests:
      storage: 16Mi

k apply -f temp-pvcs-rwx.yaml

---

Step 5: Migrate Data with pv-migrate

kubectl pv-migrate migrate \
  --source dnsmasq-dhcp \
  --dest dnsmasq-dhcp-new \
  --source-namespace openstack \
  --dest-namespace openstack

kubectl pv-migrate migrate \
  --source dnsmasq-ironic \
  --dest dnsmasq-ironic-new \
  --source-namespace openstack \
  --dest-namespace openstack

pv-migrate will spin up the rsync job, stream progress logs, and clean up after itself automatically.

---

Step 6: Delete the Old RWO PVCs

k delete pvc dnsmasq-dhcp -n openstack
k delete pvc dnsmasq-ironic -n openstack

---

Step 7: Recreate the PVCs with Original Names as RWX

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dnsmasq-dhcp
  namespace: openstack
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ceph-fs-ec
  resources:
    requests:
      storage: 16Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dnsmasq-ironic
  namespace: openstack
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ceph-fs-ec
  resources:
    requests:
      storage: 16Mi

k apply -f final-pvcs-rwx.yaml

---

Step 8: Migrate Data into the Final PVCs

kubectl pv-migrate migrate \
  --source dnsmasq-dhcp-new \
  --dest dnsmasq-dhcp \
  --source-namespace openstack \
  --dest-namespace openstack

kubectl pv-migrate migrate \
  --source dnsmasq-ironic-new \
  --dest dnsmasq-ironic \
  --source-namespace openstack \
  --dest-namespace openstack

---

Step 9: Delete the Temporary PVCs

k delete pvc dnsmasq-dhcp-new -n openstack
k delete pvc dnsmasq-ironic-new -n openstack

---

Step 10: Scale Back Up and Verify

Your workload manifests require no changes since the PVC names are preserved.

k scale statefulset ironic-conductor --replicas=2
k scale statefulset ironic-dnsmasq --replicas=1
k get pods -n openstack

---

Step 11: Restore ArgoCD Access

k create -f /tmp/argocd-binding.yaml

[skrobul]

Rollback Plan: RWX → RWO

Use this if you need to revert back to the original RWO volumes. This assumes the backup files from Step 3 of the migration are still present in /tmp.

Rollback Overview

1. Disable ArgoCD
2. Scale down StatefulSets
3. Create temporary RWO PVCs from the original backup definitions
4. Migrate data from the current RWX PVCs into the temporary ones
5. Delete the RWX PVCs
6. Recreate the PVCs with original names as RWO
7. Migrate data into the final RWO PVCs
8. Delete the temporary PVCs
9. Scale StatefulSets back up and verify
10. Restore ArgoCD access

---

Rollback Step 1: Disable ArgoCD

k get clusterrolebinding pvceng-argocd-is-cluster-admin -o yaml > /tmp/argocd-binding.yaml
k delete clusterrolebinding pvceng-argocd-is-cluster-admin

---

Rollback Step 2: Scale Down StatefulSets

k scale statefulset ironic-conductor --replicas=0
k scale statefulset ironic-dnsmasq --replicas=0

---

Rollback Step 3: Create Temporary RWO PVCs

No storage class is specified, so Kubernetes will use the cluster default.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dnsmasq-dhcp-old
  namespace: openstack
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 16Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dnsmasq-ironic-old
  namespace: openstack
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 16Mi

k apply -f temp-pvcs-rwo.yaml

---

Rollback Step 4: Migrate Data with pv-migrate

kubectl pv-migrate migrate \
  --source dnsmasq-dhcp \
  --dest dnsmasq-dhcp-old \
  --source-namespace openstack \
  --dest-namespace openstack

kubectl pv-migrate migrate \
  --source dnsmasq-ironic \
  --dest dnsmasq-ironic-old \
  --source-namespace openstack \
  --dest-namespace openstack

---

Rollback Step 5: Delete the RWX PVCs

k delete pvc dnsmasq-dhcp -n openstack
k delete pvc dnsmasq-ironic -n openstack

---

Rollback Step 6: Recreate the PVCs with Original Names as RWO

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dnsmasq-dhcp
  namespace: openstack
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 16Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dnsmasq-ironic
  namespace: openstack
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 16Mi

k apply -f final-pvcs-rwo.yaml

---

Rollback Step 7: Migrate Data into the Final RWO PVCs

kubectl pv-migrate migrate \
  --source dnsmasq-dhcp-old \
  --dest dnsmasq-dhcp \
  --source-namespace openstack \
  --dest-namespace openstack

kubectl pv-migrate migrate \
  --source dnsmasq-ironic-old \
  --dest dnsmasq-ironic \
  --source-namespace openstack \
  --dest-namespace openstack

---

Rollback Step 8: Delete the Temporary PVCs

k delete pvc dnsmasq-dhcp-old -n openstack
k delete pvc dnsmasq-ironic-old -n openstack

---

Rollback Step 9: Scale Back Up and Verify

k scale statefulset ironic-conductor --replicas=1
k scale statefulset ironic-dnsmasq --replicas=1
k get pods -n openstack

---

Rollback Step 10: Restore ArgoCD Access

k create -f /tmp/argocd-binding.yaml

[skrobul]

Once the PVC are migrated, remember to apply ironic_role=conductor label on a non-network node(s) so that we can run multiple replicas.