Skip to content

[WIP] AGENT-1449: Add IRI registry credential rotation support#1

Closed
rwsu wants to merge 2 commits intoAGENT-1449-authfrom
AGENT-1449
Closed

[WIP] AGENT-1449: Add IRI registry credential rotation support#1
rwsu wants to merge 2 commits intoAGENT-1449-authfrom
AGENT-1449

Conversation

@rwsu
Copy link
Owner

@rwsu rwsu commented Mar 13, 2026

- What I did

Implement safe credential rotation for the IRI registry using a desired-vs-current pattern with generation-numbered usernames. The auth secret holds the desired password; the pull secret (read from rendered MachineConfig) holds the deployed password. When they differ, a three-phase rotation is performed:

  1. Deploy dual htpasswd (old + new credentials with different usernames)
  2. Update pull secret after all MCPs finish rolling out
  3. Clean up dual htpasswd to single entry after new pull secret is deployed

This avoids authentication deadlocks during rolling MachineConfig updates because the pull secret always contains the old credentials, which are present in every version of the htpasswd. Mid-rotation password changes are handled by verifying htpasswd hashes with bcrypt.CompareHashAndPassword and regenerating if they don't match.

Key changes:

  • Add MachineConfigPool lister/informer to IRI controller
  • Add reconcileAuthCredentials with three-case rotation logic
  • Add getDeployedIRICredentials (reads from rendered MC, not API)
  • Add areAllPoolsUpdated (checks all pools including workers)
  • Add HtpasswdHasValidEntry, GenerateHtpasswdEntry, GenerateDualHtpasswd,
  • NextIRIUsername, ExtractIRICredentialsFromPullSecret helpers
  • Vendor golang.org/x/crypto/bcrypt for htpasswd hash generation
  • Add credential rotation design doc

- How to verify it

Update the password to trigger the rotation to start:

oc -n openshift-machine-config-operator patch secret internal-release-image-registry-auth \
  --type merge -p '{"data":{"password":"'$(echo -n "new-password" | base64)'"}}'

Verify the /etc/iri-registry/auth/htpasswd has been updated.
Verify iri-registry works for both new and old credentials during rollout.
Verify global pull-secret contains the new credentials after rollout is complete.

- Description for the changelog

Add credential rotation support for the IRI registry. When the auth secret's password field is updated, the controller performs a three-phase rotation: (1) deploys a dual htpasswd with both old and new credentials so all nodes accept both passwords during rollout, (2) updates the global pull secret with the new credentials after all MachineConfigPools are fully updated, and (3) cleans up the dual htpasswd to a single entry once the new credentials are deployed everywhere. This avoids authentication deadlocks caused by api-int load-balancing requests across master nodes that may be at different stages of the rollout.

Also adds e2e tests for registry authentication (401 on unauthenticated requests, 200 with valid credentials) and an end-to-end credential rotation test that exercises all three phases.

Depends on openshift#5765.

rwsu added 2 commits March 13, 2026 16:19
Implement safe credential rotation for the IRI registry using a
desired-vs-current pattern with generation-numbered usernames. The auth
secret holds the desired password; the pull secret (read from rendered
MachineConfig) holds the deployed password. When they differ, a
three-phase rotation is performed:

1. Deploy dual htpasswd (old + new credentials with different usernames)
2. Update pull secret after all MCPs finish rolling out
3. Clean up dual htpasswd to single entry after new pull secret is deployed

This avoids authentication deadlocks during rolling MachineConfig updates
because the pull secret always contains the old credentials, which are
present in every version of the htpasswd. Mid-rotation password changes
are handled by verifying htpasswd hashes with bcrypt.CompareHashAndPassword
and regenerating if they don't match.

Key changes:
- Add MachineConfigPool lister/informer to IRI controller
- Add reconcileAuthCredentials with three-case rotation logic
- Add getDeployedIRICredentials (reads from rendered MC, not API)
- Add areAllPoolsUpdated (checks all pools including workers)
- Add HtpasswdHasValidEntry, GenerateHtpasswdEntry, GenerateDualHtpasswd,
  NextIRIUsername, ExtractIRICredentialsFromPullSecret helpers
- Vendor golang.org/x/crypto/bcrypt for htpasswd hash generation
- Add credential rotation design doc

Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
…ial rotation

Add three new e2e tests:
- TestIRIAuth_UnauthenticatedRequestReturns401: verifies registry rejects
  unauthenticated requests with 401 when auth is enabled
- TestIRIAuth_AuthenticatedRequestSucceeds: verifies registry accepts
  requests with valid Basic Auth credentials
- TestIRIAuth_CredentialRotation: end-to-end test of the three-phase
  credential rotation (dual htpasswd, pull secret update, cleanup)

Assisted-by: Claude Opus 4.6 <noreply@anthropic.com>
@rwsu rwsu closed this Mar 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant