Skip to content

Add Red Hat OpenShift (on Azure and AWS) support#77

Open
michaelryanmcneill wants to merge 4 commits into
jfrog:mainfrom
michaelryanmcneill:feature/add-rosa-support
Open

Add Red Hat OpenShift (on Azure and AWS) support#77
michaelryanmcneill wants to merge 4 commits into
jfrog:mainfrom
michaelryanmcneill:feature/add-rosa-support

Conversation

@michaelryanmcneill

@michaelryanmcneill michaelryanmcneill commented May 26, 2026

Copy link
Copy Markdown

Summary

Adds support for deploying the JFrog Kubelet Credential Provider on Red Hat OpenShift on AWS and Azure. This includes ROSA and ARO.


Overview

OpenShift ships platform kubelet credential providers on worker nodes (ecr-credential-provider on AWS, acr-credential-provider on Azure) under /etc/kubernetes/credential-providers/. RHCOS also mounts /usr read-only, which breaks the existing EKS/AKS install paths.

This PR adds platform: openshift so the chart can:

  1. Stage the JFrog plugin and platform plugin on writable storage (/var/lib/jfrog-credential-provider/bin)
  2. Bind-mount over /usr/libexec/kubelet-image-credential-provider-plugins/
  3. Merge the JFrog provider into the platform YAML (not replace it)
  4. Use per-pod projected service account identity (IRSA on AWS, Entra workload identity on Azure) instead of node-level roles

Tested targets: ROSA (HCP and Classic), ARO.

Not in this PR: OpenShift on Google Cloud (requires additional enhancement to support due to implementation of workload identity in OpenShift on Google Cloud).


User-facing changes

New Helm values

Value Purpose
platform: openshift Enables OpenShift install/merge paths (omit for EKS/AKS/GKE)
openshift.grantPrivilegedSCC RoleBinding to system:openshift:scc:privileged for the injector DaemonSet
openshift.labelNamespacePodSecurity Apply enforce=privileged and scc.podSecurityLabelSync=false on release namespace (default true)
openshift.stagingBinaryDir Writable staging path for plugins (default /var/lib/jfrog-credential-provider/bin)
openshift.targetBinaryDir Node plugin directory / bind-mount target (default /usr/libexec/kubelet-image-credential-provider-plugins)
openshift.targetProviderConfigDir Platform kubelet credential YAML directory (default /etc/kubernetes/credential-providers)
providerConfig[].tokenAttributes.requireServiceAccount Require pulling pod SA annotations (recommended true on OpenShift)

New example values

  • examples/openshift-aws-projected-sa-values.yaml — ROSA / OpenShift on AWS with IRSA
  • examples/openshift-azure-projected-sa-values.yaml — ARO / OpenShift on Azure with workload identity

Documentation

  • OpenShift.md (new) — End-to-end guide: requirements, Helm install, IAM/MI + Artifactory setup, verification, troubleshooting (replaces prior ROSA/ARO split docs)
  • README.md — OpenShift called out in supported distributions
  • AWS.md / AZURE.md / debug.md — Cross-links and OpenShift path notes
  • helm/CHANGELOG.md — Unreleased section documents chart changes

Manual test automation

Removed from this PR at the request of the maintainers. Will be submitting a follow up PR once this PR merges.

  • openshift-tests/openshift-test-plan-aws.sh — Phases 0–8: prerequisites, IAM role + SA, Artifactory IAM mapping, Helm install, node verification, positive/negative pulls, optional cleanup
  • openshift-tests/openshift-test-plan-azure.sh — Same structure for Azure (managed identity, federated credential, Artifactory OIDC)
  • openshift-tests/README-openshift-testing.md — How to run tests and common issues

Helm chart changes

Installer (configmap-setup.yaml)

  • OpenShift path detection via isOpenShiftStaging (AWS or Azure only)
  • Staging + bind-mount for plugin binaries on RHCOS
  • Merge into platform config: ecr-credential-provider.yaml (AWS) or acr-credential-provider.yaml (Azure)
  • cloud_provider env passed to add-provider-config (metadata detection from DaemonSet network often fails on OpenShift)
  • Improved merge error output (capture stderr from add-provider-config)
  • Warn if platform credential config file is missing before merge

Provider config (configmap-provider.yaml, _aws-provider-config.tpl)

  • OpenShift AWS: YAML kubelet config with tokenAttributes (sts.amazonaws.com, requireServiceAccount, eks.amazonaws.com/role-arn, JFrogExchange)
  • OpenShift AWS: omit aws_role_name env when IRSA-only (awsOmitRoleNameFallback)
  • Azure OpenShift: azureEnvYaml helper; omit empty tenant/nodepool env vars for workload identity

OpenShift-specific templates

  • openshift.yaml — Privileged SCC RoleBinding when grantPrivilegedSCC: true
  • openshift-namespace.yaml — Pod Security labels on release namespace (when enabled)

Validations (validations.yaml)

  • platform must be openshift if set
  • OpenShift requires AWS or Azure (gcp alone → fail)
  • OpenShift + AWS requires tokenAttributes.enabled and requireServiceAccount: true

DaemonSet

  • Host mount / paths aligned with OpenShift staging model

Helpers (_helpers.tpl)

New template functions: isOpenShift, isOpenShiftStaging, openshiftPlatformPlugin, openshiftKubeletConfigPath, kubeletConfigYaml (includes OpenShift AWS), useHostMount, addProviderConfigEnvAssignments, azureEnvYaml, awsOmitRoleNameFallback, openshiftLabelNamespacePodSecurity


Binary / Go changes (minimal)

Only three Go files differ from v1.1.2:

File Change
internal/provider/config.go Kubelet health watcher: longer grace period (20s), treat activating/reloading as normal, journalctl hints on rollback (helps OpenShift kubelet restarts after config merge)
internal/utils/utils.go Fix fmt.Errorf format for aws_auth_method validation
internal/utils/merge_platform_test.go New: table-driven tests that ECR/ACR platform args survive YAML merge with JFrog provider

No changes to AWS/Azure/GCP credential exchange handlers in this PR; OpenShift reuses existing web_identity (AWS) and OIDC exchange (Azure) flows with projected SA tokens from the kubelet.


Install notes (reviewers & operators)

  1. OpenShift 4.21+ required for tokenAttributes.
  2. Do not use helm --create-namespace with default openshift.labelNamespacePodSecurity: true — pre-create the namespace and apply Helm ownership labels (documented in OpenShift.md; test scripts do this automatically).
  3. Annotate application ServiceAccounts (eks.amazonaws.com/role-arn + JFrogExchange on AWS; azure.workload.identity/client-id + JFrogExchange on Azure). Never annotate the chart DaemonSet SA.
  4. Artifactory (AWS): IAM role ARN mapping per user (not OIDC provider).
  5. Artifactory (Azure): OIDC provider + identity mapping using cluster issuer; Helm azure_app_client_id is the Artifactory exchange audience (e.g. *@*), not an Entra app ID.

Out of scope / explicitly deferred

  • OpenShift on Google Cloud (platform: openshift + gcp.enabled fails validation)
  • Publishing chart changes to jfrog/jfrog-credential-provider chart repo (install from local ./helm until released)
  • Automatic IAM OIDC provider creation or trust-policy updates in test scripts (phase 1 creates role if missing; operators fix trust manually when needed)

Suggested test plan

Automated

go test ./internal/utils/ -run TestMergeOpenShiftPlatform -v
helm template jfrog-cp ./helm -f examples/openshift-aws-projected-sa-values.yaml --set platform=openshift
helm template jfrog-cp ./helm -f examples/openshift-azure-projected-sa-values.yaml --set platform=openshift

Manual (OpenShift 4.21+)

Removed from this PR at the request of the maintainers. Will be submitting a follow up PR once this PR merges.

AWS (ROSA or self-managed on AWS):

export ARTIFACTORY_URL="<your-instance.jfrog.io>"
export ARTIFACTORY_ADMIN_TOKEN="<your-token>"
export TEST_IMAGE="${ARTIFACTORY_URL}/docker-local/your-image:tag"
bash openshift-tests/openshift-test-plan-aws.sh all --cleanup

Azure (ARO):

export ARTIFACTORY_URL="your-instance.jfrog.io"
export ARTIFACTORY_ADMIN_TOKEN="..."
export RESOURCE_GROUP="..."
export TEST_IMAGE="${ARTIFACTORY_URL}/docker-local/your-image:tag"
bash openshift-tests/openshift-test-plan-azure.sh all --cleanup

Sign-off checklist:

  • DaemonSet rolls out on all workers; privileged SCC + namespace PSA labels present
  • Node: JFrog binary present; merged ecr-credential-provider.yaml or acr-credential-provider.yaml contains tokenAttributes and JFrog provider
  • Pod with annotated SA pulls from Artifactory without imagePullSecrets
  • Pod without WI annotations fails pull when requireServiceAccount: true
  • EKS/AKS/GKE install unchanged with default values (no platform: openshift)

Breaking changes

None for existing EKS/AKS/GKE deployments (platform defaults unset).

New OpenShift deployments must follow OpenShift.md (namespace creation, SCC, projected SA annotations, Artifactory identity setup).


Release versioning (follow-up)

Chart version / appVersion remain 1.1.2 on this branch; helm/CHANGELOG.md tracks OpenShift work under [Unreleased]. Bump chart and binary versions when cutting the next release.

@michaelryanmcneill michaelryanmcneill requested a review from a team May 26, 2026 03:06
@github-actions

github-actions Bot commented May 26, 2026

Copy link
Copy Markdown

All contributors have signed the CLA ✍️ ✅
Posted by the CLA Assistant Lite bot.

@michaelryanmcneill

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

@michaelryanmcneill

Copy link
Copy Markdown
Author

Evidence that demonstrates success from my test cluster:

{"time":"2026-05-26T02:35:54.611515898Z","level":"INFO","msg":"getting envs - awsRoleName :arn:aws:iam::xxx:role/jfrog-pull-app-ns-app-sa","hostname":"ip-10-0-142-223"}
{"time":"2026-05-26T02:35:54.611519249Z","level":"INFO","msg":"Using web_identity aws auth method based on service account annotation","hostname":"ip-10-0-142-223"}
{"time":"2026-05-26T02:35:54.611522318Z","level":"INFO","msg":"running aws assume role auth flow","hostname":"ip-10-0-142-223"}
{"time":"2026-05-26T02:35:54.611524963Z","level":"INFO","msg":"TOKEN_URL :http://169.254.169.254/latest/api/token","hostname":"ip-10-0-142-223"}
{"time":"2026-05-26T02:35:54.612125629Z","level":"INFO","msg":"Using AWS region from aws_region env var: us-east-1","hostname":"ip-10-0-142-223"}
{"time":"2026-05-26T02:35:54.612136458Z","level":"INFO","msg":"running aws web identity auth flow with IRSA","hostname":"ip-10-0-142-223"}
{"time":"2026-05-26T02:35:54.684405243Z","level":"INFO","msg":"Successfully assumed role with web identity","hostname":"ip-10-0-142-223"}
{"time":"2026-05-26T02:35:54.684444131Z","level":"INFO","msg":"Using regional STS endpoint: https://sts.us-east-1.amazonaws.com?Action=GetCallerIdentity&Version=2011-06-15","hostname":"ip-10-0-142-223"}
{"time":"2026-05-26T02:35:54.68461145Z","level":"INFO","msg":"RT token url: https://xxx.jfrog.io/access/api/v1/aws/token?region=us-east-1","hostname":"ip-10-0-142-223"}
{"time":"2026-05-26T02:35:54.684620371Z","level":"INFO","msg":"RT requestBody: {\"expires_in\": 14400}","hostname":"ip-10-0-142-223"}
{"time":"2026-05-26T02:35:54.776700738Z","level":"INFO","msg":"JFrog Username used for pull :xxx@xxx.xxx","hostname":"ip-10-0-142-223"}

@oumkale oumkale added the new feature Automatically generated release notes label May 26, 2026
@oumkale

oumkale commented May 26, 2026

Copy link
Copy Markdown
Member

Hi @michaelryanmcneill,

Thank you for your contribution and for raising the PR for ROSA, we truly appreciate your effort.

We are currently reviewing the PR, and we will get back to you.

Thanks again for your contribution!

Comment thread helm/templates/configmap-setup.yaml Outdated
Comment thread examples/rosa-hcp-irsa-values.yaml Outdated

@michaelryanmcneill michaelryanmcneill left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback! I'll make some changes and then tag you for a second review.

Comment thread examples/rosa-hcp-irsa-values.yaml Outdated
Comment thread helm/templates/configmap-setup.yaml Outdated
Comment thread helm/templates/configmap-setup.yaml Outdated
@michaelryanmcneill

Copy link
Copy Markdown
Author

Temporarily moving this to WIP while I work through adding both Azure and AWS support for OCP. This will ensure it is also extensible for GCP in the future (but our implementation of pod identity in GCP is more complicated and will not be covered by this). Plan to complete this tomorrow.

@michaelryanmcneill michaelryanmcneill marked this pull request as draft May 27, 2026 05:07
Signed-off-by: michaelryanmcneill <michael@michaelryanmcneill.com>
@michaelryanmcneill michaelryanmcneill force-pushed the feature/add-rosa-support branch from 4ac6bd2 to f7b93f2 Compare May 27, 2026 15:10
@michaelryanmcneill michaelryanmcneill changed the title Add Red Hat OpenShift Service on AWS (ROSA) support to Helm chart and docs Add Red Hat OpenShift (on Azure and AWS) support May 27, 2026
@michaelryanmcneill michaelryanmcneill marked this pull request as ready for review May 27, 2026 19:16
@michaelryanmcneill

Copy link
Copy Markdown
Author

@RobinDuhan ready for your review of my updated PR.

@oumkale

oumkale commented May 29, 2026

Copy link
Copy Markdown
Member

Hi @michaelryanmcneill,

Thank you for the Azure addition. We are currently reviewing it internally.

We truly appreciate your contribution and the effort you put into this enhancement!

Comment thread helm/values.yaml
Comment thread OpenShift.md
…ons and updating OpenShift documentation

Signed-off-by: michaelryanmcneill <michael@michaelryanmcneill.com>
@michaelryanmcneill

Copy link
Copy Markdown
Author

All suggestions addressed @oumkale! Ready for final review 👍

Comment thread openshift-tests/README-openshift-testing.md Outdated
Comment thread helm/values.yaml Outdated
Comment thread OpenShift.md Outdated
Signed-off-by: michaelryanmcneill <michael@michaelryanmcneill.com>
Comment thread openshift-tests/openshift-test-plan-aws.sh Outdated
Comment thread OpenShift.md Outdated
Comment thread helm/templates/configmap-setup.yaml
Signed-off-by: michaelryanmcneill <michael@michaelryanmcneill.com>
@michaelryanmcneill

Copy link
Copy Markdown
Author

@oumkale addressed additional comments and updated the PR summary with changes. Please let me know if you see anything else.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new feature Automatically generated release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants