Skip to content

Comments

Kubernetes authentication and RBAC authorization#6

Draft
RaveNoX wants to merge 73 commits intodeckhousefrom
feat/k8s-authz
Draft

Kubernetes authentication and RBAC authorization#6
RaveNoX wants to merge 73 commits intodeckhousefrom
feat/k8s-authz

Conversation

@RaveNoX
Copy link
Collaborator

@RaveNoX RaveNoX commented Feb 9, 2026

Summary

Adds Kubernetes authentication (via TokenReview) and optional Kubernetes RBAC authorization (via SelfSubjectRulesReview).

Users authenticate with Kubernetes bearer tokens; pull/push/delete can be authorized via existing Kubernetes RBAC (Roles/RoleBindings) for a configurable API group and resource.

Changes

Authentication (Kubernetes TokenReview)

  • New config block kubernetes_auth: docker login username is fixed (default "token"), password is a Kubernetes bearer token validated via TokenReview.
  • Optional kubeconfig path; if omitted, in-cluster config is used.
  • Configurable limits (request_timeout, qps, burst) and cache (success_ttl, failure_ttl) for TokenReview.
  • User identity exposed as labels: k8s-username, k8s-uid, k8s-groups, k8s-extra-*, and groups for ACL compatibility.
  • Prometheus metrics: registry_auth_k8s_authn_requests_total, registry_auth_k8s_authn_request_latency_seconds.

Authorization (optional, SelfSubjectRulesReview)

  • Optional authz block under kubernetes_auth enables RBAC-based authorization via SelfSubjectRulesReview.
  • Namespace is taken from the first segment of the repository name; repository path is matched against RBAC resourceNames using recursive globbing (e.g. images/**) via doublestar (godoc).
  • Configurable api_group and resource (e.g. custom resource registries in registry.example.com).
  • Optional namespace_check_verbs: for listed actions (e.g. push, delete), namespace existence is checked before calling SelfSubjectRulesReview.
  • Caching for SelfSubjectRulesReview and namespace existence to reduce API server load.
  • Prometheus metrics: registry_auth_k8s_authz_rules_requests_total, registry_auth_k8s_authz_rules_request_duration_seconds.
  • Evaluation order: ACL is checked first; if it allows, Kubernetes authz is not evaluated (OR logic).

Configuration example

kubernetes_auth:
  # kubeconfig: "/path/to/kubeconfig"  # optional; default: in-cluster
  # username: "token"                   # optional; default: "token"
  limits:
    request_timeout: "10s"
    qps: 5
    burst: 10
  cache:
    success_ttl: "1m"
    failure_ttl: "30s"
  authz:                               # optional; omit for auth-only
    api_group: "registry.example.com"
    resource: "registries"
    namespace_check_verbs: ["push", "delete"]  # optional

RBAC samples

Auth server (TokenReview + SelfSubjectRulesReview)

The auth server's service account needs permission to create TokenReview and SelfSubjectRulesReview, and (if namespace_check_verbs is used) to get/list namespaces:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: registry-auth-server
rules:
- apiGroups: ["authentication.k8s.io"]
  resources: ["tokenreviews"]
  verbs: ["create"]
- apiGroups: ["authorization.k8s.io"]
  resources: ["selfsubjectrulesreviews"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: registry-auth-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: registry-auth-server
subjects:
- kind: ServiceAccount
  name: <auth-server-sa>
  namespace: <auth-server-namespace>

User/application RBAC (registry path access)

When authz is enabled, grant access to repository paths via Role/ClusterRole on the configured resource (e.g. api_group: "registry.example.com", resource: "registries"). resourceNames are repository path patterns; matching uses recursive globbing (e.g. **). Verb mapping: get → pull, list → catalog/list, create → push, delete → delete tag/manifest.

Read-only (pull + list) for one namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: registry-read-my-app
  namespace: my-app
rules:
- apiGroups: ["registry.example.com"]
  resources: ["registries"]
  resourceNames: ["my-app/*", "my-app/**"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: registry-read-my-app
  namespace: my-app
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: registry-read-my-app
subjects:
- kind: ServiceAccount
  name: my-sa
  namespace: my-app

Push and pull (FULL) for a subpath:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: registry-write-my-app-frontend
  namespace: my-app
rules:
- apiGroups: ["registry.example.com"]
  resources: ["registries"]
  resourceNames: ["my-app/frontend/*", "my-app/frontend/**"]
  verbs: ["get", "list", "create", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: registry-write-my-app-frontend
  namespace: my-app
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: registry-write-my-app-frontend
subjects:
- kind: ServiceAccount
  name: ci-sa
  namespace: my-app

(With namespace_check_verbs: ["push", "delete"], push/delete are only allowed if namespace my-app exists.)

Requirements

  • Kubernetes API server must allow TokenReview and (if authz is used) SelfSubjectRulesReview.
  • When authz is enabled, RBAC must grant the auth server's service account permission to create TokenReview and SelfSubjectRulesReview.

Testing

  • auth_server/k8s/config_test.go covers config validation and defaults.
  • Manual/QA: authenticate with a Kubernetes token and (with authz enabled) verify pull/push/delete according to Role/RoleBinding resourceNames and globs.

Backward compatibility

  • Additive only: new config key kubernetes_auth. Existing configs and auth methods are unchanged.

Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: chupakobra6 <igorpheik@gmail.com>
… go.sum

Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: chupakobra6 <igorpheik@gmail.com>
…ective defaults for RPS and Burst parameters

Signed-off-by: chupakobra6 <igorpheik@gmail.com>
…fective defaults for RPS and Burst parameters

Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: chupakobra6 <igorpheik@gmail.com>
…access to "token" only

Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: chupakobra6 <igorpheik@gmail.com>
Signed-off-by: Artur Kraev <artur.kraev@flant.com>
- Implement case-insensitive matching for API groups, resources, and resource paths
- Refactor IsActionAllowed to extract matchRule helper function for better code organization
- Add error handling and logging for pattern matching failures
- Update documentation and examples to use generic resource names (registry.example.com/registries) instead of deckhouse-specific ones
- Add comprehensive test coverage for case-insensitive matching scenarios including uppercase, mixed case, and wildcard patterns
- Simplified timeout configuration in kubernetesAuthz by directly using the configured request timeout.
- Added default request timeout value of 10 seconds in AuthConfig and updated validation logic to set it if not specified or invalid.
- Removed redundant default request timeout constant from const.go to streamline the codebase.

Signed-off-by: Artur Kraev <artur.kraev@flant.com>
- Introduced UIDLabel constant for Kubernetes user UID in const.go.
- Enhanced UserInfo struct to include UID and updated ToLabels and UserInfoFromLabels methods to handle UID.
- Updated documentation to reflect the addition of UID in authentication labels and ACL examples.

Signed-off-by: [Your Name] [Your Email]
Signed-off-by: Artur Kraev <artur.kraev@flant.com>
- Added checks to handle cases where no actions are provided in the authorization request, returning a NoMatch error.
- Updated the logic to return NoMatch if no actions are allowed after processing the request.

Signed-off-by: Artur Kraev <artur.kraev@flant.com>
- Removed unnecessary logging of SSRR rules.
- Enhanced cache handling to differentiate between successful rule retrieval and error cases, ensuring proper error propagation.
- Updated cache addition logic to store errors instead of nil values for better failure tracking.

Signed-off-by: Artur Kraev <artur.kraev@flant.com>
Signed-off-by: Artur Kraev <artur.kraev@flant.com>
- Updated field name from `UserName` to `Username` in the Kubernetes authentication logic and configuration to ensure consistency.
- Adjusted related documentation and examples to reflect the change in the field name.

Signed-off-by: Artur Kraev <artur.kraev@flant.com>
- Updated the field name from `Username` to `UserName` in both the authentication and authorization logic to ensure consistency across the codebase.

Signed-off-by: Artur Kraev <artur.kraev@flant.com>
- Updated the kubernetesAuthz struct to use a RulesFetcher for improved rule retrieval.
- Removed the previous caching logic and integrated a new cached rules fetcher to enhance performance and error handling.
- Simplified the authorization process by directly utilizing the rules fetcher in the Authorize method.

Signed-off-by: Artur Kraev <artur.kraev@flant.com>
- Introduced singleflight to prevent duplicate in-flight requests for the same cache key in the cached rules fetcher.
- Refactored the GetRules method to utilize singleflight for improved performance and error handling.
- Cleaned up code formatting for better readability.

Signed-off-by: Artur Kraev <artur.kraev@flant.com>
- Renamed metrics from k8sAuthzRequestsTotal to k8sAuthzRulesRequestsTotal and k8sAuthzRequestLatencySeconds to k8sAuthzRulesRequestDurationSeconds for clarity.
- Updated the initialization and usage of metrics in the kubernetesAuthz struct to reflect the new naming conventions.

Signed-off-by: Artur Kraev <artur.kraev@flant.com>
- Introduced a new NamespaceCheckVerbs field in AuthzConfig to specify Docker actions that require namespace existence checks before authorization.
- Updated the kubernetesAuthz struct to include a namespaceChecker for validating namespace existence.
- Enhanced the Authorize method to perform namespace checks based on the new configuration, improving authorization accuracy.
- Updated documentation to reflect the new namespace check functionality and its usage.

Signed-off-by: Artur Kraev <artur.kraev@flant.com>
- Added error handling for token authentication to return a specific error message when the token is not authenticated.
- Updated the Authenticate method to return api.WrongPass instead of nil for better clarity on authentication failures.

Signed-off-by: Artur Kraev <artur.kraev@flant.com>
- Introduced a new AuthFailed type to encapsulate authentication errors, providing clearer error messages in HTTP responses.
- Updated the WrongPass variable to use NewAuthFailed for consistency in error handling.
- Modified the Kubernetes authentication logic to return an AuthFailed error instead of a generic error, improving clarity in failure scenarios.
- Enhanced error handling in the AuthServer to log specific authentication failures.

Signed-off-by: Artur Kraev <artur.kraev@flant.com>
Signed-off-by: Artur Kraev <artur.kraev@flant.com>
- Updated the Authenticator interface to clarify special error cases: NoMatch and WrongPass/AuthFailed.
- Improved documentation for authentication failures, specifying HTTP 401 responses and error messages.
- Added notes in examples and documentation to reflect changes in error handling for invalid or unauthenticated tokens.

Signed-off-by: Artur Kraev <artur.kraev@flant.com>
- Included patterns to ignore editor swap files, Helm chart lockfiles, build artifacts, and macOS system files.
- This update helps maintain a cleaner repository by preventing unnecessary files from being tracked.

Signed-off-by: Artur Kraev <artur.kraev@flant.com>
- Changed the default success TTL for caching from 5 minutes to 1 minute in the AuthConfig struct.
- Updated related documentation and examples to reflect the new default value for success_ttl.

Signed-off-by: Artur Kraev <artur.kraev@flant.com>
@RaveNoX RaveNoX self-assigned this Feb 9, 2026
@RaveNoX RaveNoX requested a review from vadimartynov February 9, 2026 17:49
@RaveNoX RaveNoX added the enhancement New feature or request label Feb 9, 2026
- Implemented a warning log in the Authorize method to notify when access is denied due to a non-existent namespace.
- This enhancement improves visibility into authorization failures, aiding in debugging and monitoring.

Signed-off-by: Artur Kraev <artur.kraev@flant.com>
- Added a verbose log statement in the Authorize method to output the retrieved rules for a user and namespace, enhancing visibility into the authorization process.
- This change aids in debugging and monitoring by providing more context during authorization checks.

Signed-off-by: Artur Kraev <artur.kraev@flant.com>
- Updated the IsActionAllowed method to use the original case of the resource path when matching against authorization rules, improving accuracy in rule evaluation.
- This change ensures that the authorization checks are performed without altering the case of the resource path, which may affect rule matching.

Signed-off-by: Artur Kraev <artur.kraev@flant.com>
…mprovements

- Updated the NewRulesFetcher function to accept an optional rulesFilter parameter, allowing for filtering of fetched rules based on API group and resource.
- Refactored the IsActionAllowed method to simplify rule matching logic and improve clarity.
- Introduced a new Validate method for AuthzConfig to ensure proper configuration and error handling.
- Enhanced test coverage for AuthzConfig validation and rule filtering, ensuring robust authorization checks.

Signed-off-by: Artur Kraev <artur.kraev@flant.com>
- Introduced the ApplyDefaults function to set default values for optional fields in AuthConfig, ensuring proper initialization.
- Added a new test, TestApplyDefaults, to verify the correct application of default values and idempotency.
- Updated existing validation tests to incorporate the new applyDefaults parameter, improving clarity and functionality in testing AuthConfig validation.

Signed-off-by: Artur Kraev <artur.kraev@flant.com>
return false, nil, api.NoMatch
}

ctx, cancel := context.WithTimeout(context.Background(), ka.cfg.Limits.RequestTimeout)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RequestTimeout is used twice. Will this cause any errors?:

  • when initializing tokenAuth;
  • when calling the AuthenticateToken method;

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which problem it can be?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just thought about the empty value in RequestTimeout.
It is checked in the method, but not at the call level. But ideally, default values should be set there.

I think we can skip it. There shouldn't be any problems.

- Removed unused time import and the DefaultBackoffInitialDelay constant from const.go, streamlining the code.
- Updated the GetRules method in rules_fetcher.go to utilize the new DefaultRetryBackoff from the webhookauthn package, enhancing the backoff strategy for retries.

Signed-off-by: Artur Kraev <artur.kraev@flant.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants