Serve KMS ClearImageCache on a dedicated authenticated admin listener

[kvinwang]

Problem

The KMS admin RPC ClearImageCache was authenticated differently from every other admin surface in dstack, and in a way that is easy to get wrong:

• It rode on the public KMS RPC service (the same [rpc] listener as GetAppKey/SignCert), gated only by a hand-rolled ensure_admin check inside the handler.
• The credential was passed as a field in the request body (ClearImageCacheRequest.token), compared against [core] admin_token_hash. It did not use the shared dstack-api-auth HTTP authenticator, so — unlike VMM and gateway — Authorization: Bearer / X-Admin-Token headers were ignored.
• The auth was per-RPC and opt-in: any future admin RPC that forgot to call ensure_admin would be unauthenticated, with no framework backstop.

This divergence already caused a documentation error (a sibling PR initially documented the KMS admin token as an Authorization: Bearer header, which the server silently ignored).

Fix

Move ClearImageCache onto a dedicated admin listener that reuses the same shared HTTP authenticator as the gateway, so all three services (VMM, gateway, KMS) authenticate admin/management traffic the same way.

• Proto: split ClearImageCache out of service KMS into a new service KmsAdmin, and drop the token field from ClearImageCacheRequest (the credential now travels in the HTTP header).
• Config: replace [core] admin_token_hash with a gateway-style [core.admin] section, configured identically to the gateway — enabled, address/port, admin_token (plaintext, or the DSTACK_KMS_ADMIN_TOKEN / ADMIN_API_TOKEN env vars), htpasswd_file, insecure_no_auth.
• Auth: new kms/src/admin_auth.rs mirrors the gateway AdminAuthFairing — resolves the token from config or env, adds optional bcrypt htpasswd, and fails closed (enabled with no credential and insecure_no_auth = false refuses to start).
• Service/launch: ClearImageCache moves to a new AdminRpcHandler served on a second Rocket instance bound to [core.admin], behind the HttpAuthFairing. ensure_admin and the in-handler token check are deleted; the cache-clearing logic moves to KmsState.
• Configs/templates: drop the now-dead [core] admin_token_hash from kms.toml, the KMS dstack-app templates, the dstackup KMS template, and the SNP smoke test. The admin API is disabled by default; operators opt in via [core.admin].

No backward-compat shim: the RPC had no known callers.

Verification

• cargo build (full workspace), cargo clippy -p dstack-kms, cargo fmt --all --check — all clean.
• cargo test -p dstack-kms (37) / -p dstack-cli-core (12) pass, including new tests:
  • admin_auth: fail-closed when enabled with no credential (config or env); insecure_no_auth allowed; a configured token builds.
  • config: the default kms.toml parses with [core.admin] disabled and an empty token (fail-closed).
  • main_service::remove_cache_only_deletes_the_named_hex_entry: the relocated cache-clear logic keeps its hex-key guard and all behavior.
• The admin listener's 401/200 enforcement is the shared dstack-api-auth HttpAuthFairing, which already has passing Rocket integration tests (protects_all_methods_and_accepts_compatible_credentials, enabled_with_no_credentials_denies_access).

Not covered: a live HTTP round-trip against a running KMS admin listener. That needs a fully bootstrapped KMS keyset (genesis attestation), which this change does not stand up. The auth enforcement is exercised by the shared fairing's own tests; the KMS-specific wiring is covered by the unit tests above. Happy to add a fixture-based rocket::local integration test if desired.

Coordination

Supersedes the KMS-admin-token documentation in the docs/tooling PR (which described the old request-field mechanism). If that PR lands first, its kms/README.md / deployment.md / security-best-practices.md KMS-admin notes should be reconciled to the header-based mechanism documented here.

[Copilot]

Pull request overview

This PR moves the KMS admin-only ClearImageCache RPC off the public KMS RPC surface and onto a dedicated [core.admin] listener protected by the shared dstack-api-auth HTTP authenticator, aligning KMS with the gateway/VMM admin authentication model.

Changes:

• Split ClearImageCache into a new KmsAdmin proto service and remove the request-body token field (credential now comes from HTTP headers).
• Add a new KMS admin listener configuration ([core.admin]) and wire up a second Rocket instance guarded by an AdminAuthFairing (fail-closed unless explicitly configured).
• Relocate cache-clearing logic into KmsState and update templates/scripts/config snippets to drop the old [core] admin_token_hash.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
dstack/test-scripts/snp-e2e-smoke.sh	Removes the obsolete admin_token_hash field from SNP smoke config snippet.
dstack/kms/src/main.rs	Boots an optional second Rocket instance for admin RPCs on [core.admin], attaching shared HTTP auth.
dstack/kms/src/main_service.rs	Moves cache-clearing logic into KmsState and adds unit tests for cache deletion safety.
dstack/kms/src/config.rs	Replaces the legacy admin_token_hash field with a structured AdminConfig under core.admin.
dstack/kms/src/admin_service.rs	Introduces a dedicated KmsAdmin RPC handler serving ClearImageCache.
dstack/kms/src/admin_auth.rs	Adds KMS adapter fairing that constructs/attaches the shared HTTP authenticator (fail-closed).
dstack/kms/rpc/proto/kms_rpc.proto	Creates service KmsAdmin and removes the token from ClearImageCacheRequest.
dstack/kms/README.md	Documents the new admin listener, header-based auth, and example curl invocation.
dstack/kms/kms.toml	Adds [core.admin] template and removes old [core] admin_token_hash.
dstack/kms/dstack-app/entrypoint.sh	Drops writing the removed [core] admin_token_hash into generated config.
dstack/kms/dstack-app/compose-simple.yaml	Removes the removed admin_token_hash from template config.
dstack/kms/dstack-app/compose-dev.yaml	Removes the removed admin_token_hash from template config.
dstack/crates/dstack-cli-core/src/config.rs	Removes the removed admin_token_hash from embedded config snippet.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.