Merge pull request #791 from Dstack-TEE/hapi-gpu-ocsp-cache

feat: cache NVIDIA GPU attestation collateral

Author: kvinwang

docs/security/cvm-boundaries.md

@@ -74,6 +74,7 @@ This file contains system configuration in JSON format:
 | kms_urls | array of string | List of KMS service URLs |
 | gateway_urls | array of string | List of gateway service URLs |
 | pccs_url | string | URL of the PCCS service (used when dstack components need to verify a remote TD CVM or SGX enclave) |
+| nvidia_attestation_proxy_url | string | Optional persistent OCSP and RIM cache used by NVIDIA local GPU attestation |
 | docker_registry | string | URL of the docker registry |
 | host_api_url | string | VSOCK URL of host API |
 | vm_config | string | JSON string of VM configuration (os_image_hash, cpu_count, memory_size) |
@@ -85,6 +86,7 @@ The hash of this file is not extended to any RTMR because each field has its own
 | kms_urls | URLs themselves aren't security-critical. The trust anchor is the KMS root public key, which is extended as the `key-provider` launch event. On TDX-family platforms this is RTMR3; on AWS NitroTPM this is PCR14. Keys obtained from KMS will either successfully decrypt/encrypt the disk or fail-and-abort. |
 | gateway_urls | URLs aren't security-critical. Trust is established through CA certificates from KMS. App CVM and dstack-gateway CVM verify each other's CA certificates to ensure they're under the same KMS authority. |
 | pccs_url | URL isn't security-critical. Trust is anchored by the root public key pinned in the attestation verification program. |
+| nvidia_attestation_proxy_url | The URL is not a collateral trust anchor. The measured guest verifies NVIDIA signatures and the signed OCSP validity window, and continues to require a fresh GPU evidence nonce. A bad endpoint can withhold collateral and cause a denial of service, but cannot forge a successful attestation or replay an expired `good` response. |
 | docker_registry | Docker daemon verifies image integrity using the pinned image hashes in the docker-compose file. |
 | host_api_url | Used only for reporting or encrypted sealing key transport. An incorrect URL doesn't create security vulnerabilities. |
 | vm_config | Informs the CVM to report virtual hardware info to KMS when requesting keys. KMS uses this info to calculate expected RTMRs and verify image hash. If tampered with, image hash verification would fail and no keys would be distributed. |

docs/tutorials/vmm-configuration.md

@@ -105,6 +105,8 @@ qemu_path = ""
 kms_urls = ["http://127.0.0.1:8081"]
 gateway_urls = ["http://127.0.0.1:8082"]
 pccs_url = "https://pccs.phala.network/sgx/certification/v4"
+# Optional. See "NVIDIA GPU attestation cache" below.
+# nvidia_attestation_proxy_url = "http://10.0.2.2:8090"
 docker_registry = ""
 cid_start = 1000
 cid_pool_size = 1000
@@ -212,6 +214,24 @@ Verify QGS is running:
 systemctl status qgsd
 ```
 
+### Optional: NVIDIA GPU attestation cache
+
+GPU images perform local NVIDIA attestation before app keys are provisioned.
+Without a cache this contacts NVIDIA's OCSP and RIM services during every cold
+boot. A fleet can run the persistent
+[`dstack-nvidia-attest-proxy`](../../dstack/nvidia-attest-proxy/README.md) and pass its
+URL to guests through sys-config:
+
+```toml
+[cvm]
+nvidia_attestation_proxy_url = "http://10.0.2.2:8090"
+```
+
+The example address is the host as seen from QEMU user-mode networking. The
+proxy must be reachable during `dstack-prepare`. It stores only NVIDIA-signed
+collateral and never becomes a signing trust anchor. OCSP entries are not
+served after their signed validity window.
+
 ### Step 7: Create Runtime Directories
 
 ```bash

dstack/Cargo.lock

@@ -59,6 +59,7 @@ members = [
     "serde-duration",
     "dstack-mr",
     "dstack-mr/cli",
+    "nvidia-attest-proxy",
     "verifier",
     "size-parser",
     "crates/dstack-cli-core",

dstack/Cargo.toml

@@ -710,6 +710,10 @@ pub struct SysConfig {
     #[serde(default, alias = "tproxy_urls")]
     pub gateway_urls: Vec<String>,
     pub pccs_url: Option<String>,
+    /// Optional NVIDIA attestation collateral proxy. When present, nvattest
+    /// fetches both OCSP responses and RIM documents through this endpoint.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub nvidia_attestation_proxy_url: Option<String>,
     pub docker_registry: Option<String>,
     pub host_api_url: Option<String>,
     /// MrConfigV3 document string for platform app/config binding.

dstack/dstack-types/src/lib.rs

@@ -29,6 +29,7 @@ sha2.workspace = true
 tokio = { workspace = true, features = ["full"] }
 tracing.workspace = true
 tracing-subscriber.workspace = true
+url.workspace = true
 x25519-dalek.workspace = true
 
 dstack-kms-rpc.workspace = true

dstack/dstack-util/Cargo.toml

@@ -1081,6 +1081,7 @@ mod gpu {
     const ATTESTATION_TIMEOUT: Duration = Duration::from_secs(300);
     const EVENT_VERSION: u32 = 2;
     const POLICY_ENTRYPOINT: &str = "data.policy.nv_match";
+    const TRUST_OUTPOST_POLICY: &str = "/usr/share/nvattest/policies/allow_trust_outpost_ocsp.rego";
     /// Bound Rego evaluation so a runaway application policy cannot hang boot.
     const POLICY_TIMEOUT: Duration = Duration::from_secs(10);
 
@@ -1368,11 +1369,61 @@ mod gpu {
         serde_json::to_vec(&event).context("failed to serialize GPU attestation event")
     }
 
-    /// Run local GPU attestation via nvattest with a fresh nonce and no custom
-    /// relying-party policy. nvattest's built-in appraisal still applies. The
-    /// complete JSON output is preserved under /run and retained with its
-    /// claims for the application policy and versioned RTMR3 summary.
-    pub(super) async fn attest_gpu(expected_devices: u32) -> Result<GpuAttestationResult> {
+    fn normalize_proxy_url(proxy_url: Option<&str>) -> Result<Option<String>> {
+        let Some(proxy_url) = proxy_url.map(str::trim).filter(|url| !url.is_empty()) else {
+            return Ok(None);
+        };
+        let parsed = url::Url::parse(proxy_url).context("invalid NVIDIA attestation proxy URL")?;
+        if !matches!(parsed.scheme(), "http" | "https") || parsed.host_str().is_none() {
+            bail!("NVIDIA attestation proxy must be an absolute HTTP(S) URL");
+        }
+        if parsed.query().is_some()
+            || parsed.fragment().is_some()
+            || !parsed.username().is_empty()
+            || parsed.password().is_some()
+            || parsed.path() != "/"
+        {
+            bail!(
+                "NVIDIA attestation proxy URL must not contain credentials, path, query, or fragment"
+            );
+        }
+        Ok(Some(parsed.as_str().trim_end_matches('/').to_string()))
+    }
+
+    fn nvattest_args(nonce: &str, proxy_url: Option<&str>) -> Result<Vec<String>> {
+        let mut args = vec![
+            "attest".to_string(),
+            "--device".to_string(),
+            "gpu".to_string(),
+            "--verifier".to_string(),
+            "local".to_string(),
+            "--nonce".to_string(),
+            nonce.to_string(),
+            "--format".to_string(),
+            "json".to_string(),
+        ];
+        if let Some(proxy_url) = normalize_proxy_url(proxy_url)? {
+            args.extend([
+                "--ocsp-url".to_string(),
+                format!("{proxy_url}/ocsp"),
+                "--rim-url".to_string(),
+                proxy_url,
+                "--relying-party-policy".to_string(),
+                TRUST_OUTPOST_POLICY.to_string(),
+            ]);
+        }
+        Ok(args)
+    }
+
+    /// Run local GPU attestation via nvattest with a fresh evidence nonce. If
+    /// sys-config selects a collateral proxy, both RIM and OCSP traffic is
+    /// routed through it and NVIDIA's Trust Outpost policy accepts cached OCSP
+    /// responses whose responder nonce no longer matches. The independent GPU
+    /// evidence nonce remains mandatory and is checked below.
+    pub(super) async fn attest_gpu(
+        expected_devices: u32,
+        proxy_url: Option<&str>,
+    ) -> Result<GpuAttestationResult> {
         if !Path::new(NVATTEST).exists() {
             bail!("nvattest is not available in this image");
         }
@@ -1382,22 +1433,14 @@ mod gpu {
             warn!("failed to step system clock: {err:?}");
         }
         let nonce = hex::encode(rand::thread_rng().gen::<[u8; 32]>());
-        let output = run_command(
-            NVATTEST,
-            &[
-                "attest",
-                "--device",
-                "gpu",
-                "--verifier",
-                "local",
-                "--nonce",
-                &nonce,
-                "--format",
-                "json",
-            ],
-            ATTESTATION_TIMEOUT,
-        )
-        .await?;
+        let args = nvattest_args(&nonce, proxy_url)?;
+        if args.iter().any(|arg| arg == "--relying-party-policy")
+            && !Path::new(TRUST_OUTPOST_POLICY).is_file()
+        {
+            bail!("NVIDIA attestation proxy is configured but {TRUST_OUTPOST_POLICY} is missing");
+        }
+        let args = args.iter().map(String::as_str).collect::<Vec<_>>();
+        let output = run_command(NVATTEST, &args, ATTESTATION_TIMEOUT).await?;
         if !output.stderr.is_empty() {
             info!("nvattest: {}", truncated_lossy(&output.stderr, 2048));
         }
@@ -1556,6 +1599,35 @@ mod gpu {
             assert_eq!(nvidia_gpu_count(nvidia).unwrap(), 2);
         }
 
+        #[test]
+        fn proxy_routes_ocsp_and_rim_and_selects_outpost_policy() {
+            let nonce = format!("test-nonce-{}", std::process::id());
+            let args = nvattest_args(&nonce, Some("http://10.0.2.2:8090/")).unwrap();
+            assert!(args
+                .windows(2)
+                .any(|args| args == ["--ocsp-url", "http://10.0.2.2:8090/ocsp"]));
+            assert!(args
+                .windows(2)
+                .any(|args| args == ["--rim-url", "http://10.0.2.2:8090"]));
+            assert!(args
+                .windows(2)
+                .any(|args| args == ["--relying-party-policy", TRUST_OUTPOST_POLICY]));
+
+            let direct = nvattest_args(&nonce, None).unwrap();
+            assert!(!direct.iter().any(|arg| arg == "--ocsp-url"));
+            assert!(!direct.iter().any(|arg| arg == "--relying-party-policy"));
+        }
+
+        #[test]
+        fn proxy_url_validation_is_fail_closed() {
+            let nonce = format!("test-nonce-{}", std::process::id());
+            assert!(nvattest_args(&nonce, Some("file:///tmp/proxy")).is_err());
+            assert!(nvattest_args(&nonce, Some("https://user@example.com")).is_err());
+            assert!(nvattest_args(&nonce, Some("https://example.com?q=1")).is_err());
+            assert!(nvattest_args(&nonce, Some("https://example.com/base")).is_err());
+            assert!(normalize_proxy_url(Some("  ")).unwrap().is_none());
+        }
+
         #[test]
         fn basic_policy_requires_cc_and_rejects_devtools_by_default() {
             let nonce = "44".repeat(32);
@@ -1874,7 +1946,14 @@ impl Stage0<'_> {
         }
         self.vmm.notify_q("boot.progress", "attesting GPU").await;
         info!("verifying GPU TEE attestation");
-        let attestation = gpu::attest_gpu(expected_devices).await?;
+        let attestation = gpu::attest_gpu(
+            expected_devices,
+            self.shared
+                .sys_config
+                .nvidia_attestation_proxy_url
+                .as_deref(),
+        )
+        .await?;
 
         let gpu_state = gpu::query_gpu_state(expected_devices)?;
         attestation

dstack/dstack-util/src/system_setup.rs

@@ -0,0 +1,36 @@
+# SPDX-FileCopyrightText: © 2026 Phala Network <dstack@phala.network>
+#
+# SPDX-License-Identifier: Apache-2.0
+
+[package]
+name = "dstack-nvidia-attest-proxy"
+version.workspace = true
+authors.workspace = true
+edition.workspace = true
+license.workspace = true
+
+[dependencies]
+anyhow.workspace = true
+base64.workspace = true
+bytes.workspace = true
+futures.workspace = true
+chrono.workspace = true
+clap = { workspace = true, features = ["env"] }
+dashmap.workspace = true
+hex = { workspace = true, features = ["alloc"] }
+http.workspace = true
+http-body-util.workspace = true
+humantime.workspace = true
+hyper = { workspace = true, features = ["server"] }
+hyper-util = { workspace = true, features = ["tokio"] }
+reqwest.workspace = true
+serde.workspace = true
+serde_json.workspace = true
+sha2.workspace = true
+tokio = { workspace = true, features = ["full"] }
+tracing.workspace = true
+tracing-subscriber.workspace = true
+url.workspace = true
+
+[dev-dependencies]
+tempfile.workspace = true

dstack/nvidia-attest-proxy/Cargo.toml

@@ -0,0 +1,78 @@
+# NVIDIA GPU Attestation Proxy
+
+`dstack-nvidia-attest-proxy` is a persistent, PCCS-like cache for the two NVIDIA
+services used by local GPU attestation:
+
+- `POST /ocsp` caches DER OCSP responses by the request's certificate IDs. The
+  per-request OCSP nonce is deliberately excluded from the key.
+- `GET /v1/rim/<id>` caches version-addressed NVIDIA RIM documents.
+- `GET /healthz` reports process health.
+- `GET /info` reports fresh/stale entry counts per cache kind.
+
+The proxy never signs or rewrites collateral. `nvattest` in the CVM still
+verifies NVIDIA's certificate chains, OCSP signatures, response validity
+window, RIM signatures, measurements, and the fresh GPU evidence nonce. Cache
+entries are persisted so a warm cache survives both proxy and VM restarts.
+
+## Run
+
+```bash
+cargo build --release -p dstack-nvidia-attest-proxy
+sudo install -m 0755 target/release/dstack-nvidia-attest-proxy /usr/local/bin/
+sudo install -d -m 0750 /var/cache/dstack/nvidia-attest-proxy
+
+sudo /usr/local/bin/dstack-nvidia-attest-proxy \
+  --listen 0.0.0.0:8090 \
+  --cache-dir /var/cache/dstack/nvidia-attest-proxy
+```
+
+All options have environment-variable equivalents; see `--help`. In
+particular, `NV_ATTESTATION_SERVICE_KEY` supplies an optional NVIDIA bearer
+token without placing it on the command line.
+
+Configure the VMM with a URL reachable during early guest boot:
+
+```toml
+[cvm]
+nvidia_attestation_proxy_url = "http://10.0.2.2:8090"
+```
+
+For QEMU user-mode networking, `10.0.2.2` is normally the host address. Other
+networking modes should use the corresponding host or fleet service address.
+
+## Cache behavior
+
+On a cold miss the original request is forwarded to NVIDIA. A successful OCSP
+response is cached no later than its signed `nextUpdate` and no longer than
+`--ocsp-max-ttl` (24 hours by default). Responses without `nextUpdate` use one
+hour from `thisUpdate`, matching the pinned NVIDIA SDK. RIM documents default
+to a 30-day TTL.
+
+Two mechanisms keep entries warm, and they compose: a background sweep (every
+`--refresh-interval`, 10 minutes by default) renews entries that have
+consumed half their lifetime, so a warm cache rides through an NVIDIA outage
+with close to a full validity window instead of only the remainder; entries
+past their usefulness are dropped, not retried. As a synchronous fallback, an
+OCSP response with less than `--ocsp-refresh-before` validity remaining
+(five minutes by default) is refreshed in-line on the next request before it
+is served — concurrent refreshes for the same entry are coalesced. If an
+in-line refresh fails, the proxy keeps serving the old response only until
+its existing signed expiry; it never extends or serves an expired one.
+
+Expired OCSP entries are never served. Therefore a warm cache removes the
+NVIDIA service from the boot path only for the signed validity period; it does
+not turn revocation checking into an indefinite fail-open. RIM documents are
+signed and version-addressed, so an expired RIM entry — unlike OCSP — may
+still be served for up to `--rim-max-stale` (7 days by default) when the
+upstream is unreachable or failing; the guest verifies its signature either
+way. Response headers expose `X-Dstack-Cache: HIT|MISS|REFRESH|STALE`, `Age`,
+and `X-Dstack-Cache-Expires` for operations.
+Each cache kind is capped at 10,000 entries by default; the oldest entry is
+evicted when the limit is reached. Use `--max-cache-entries-per-kind` to tune
+the bound for a deployment.
+
+Because a cached response contains the nonce from the request that populated
+the cache, `nvattest` reports `x-nvidia-cert-ocsp-nonce-matches = false` on a
+hit. When the proxy URL is configured, dstack selects NVIDIA's packaged
+`allow_trust_outpost_ocsp.rego` policy. This relaxes only the OCSP nonce check;
+the GPU attestation report's independent nonce remains required by dstack.