diff --git a/.security-triage.yaml b/.security-triage.yaml new file mode 100644 index 000000000..476f243f4 --- /dev/null +++ b/.security-triage.yaml @@ -0,0 +1,22 @@ +version: "1.0" +project: "nvcf" +last_updated: "2026-07-23T23:43:11Z" + +repository_exposure_classification: + visibility: "Public" + basis: "NVIDIA/nvcf is a public repository on GitHub" + confirmed_by: "FamousDirector" + date: "2026-07-23" + +service_exposure_classification: + tier: "External / Regulated" + confidence: "high" + basis: >- + customer-facing APIs; production GPU workloads; credentials, artifacts, + request data, secrets, and telemetry + confirmed_by: "FamousDirector" + date: "2026-07-23" + +false_positives: [] +accepted_risks: [] +suppressed_cves: [] diff --git a/SECURITY.md b/SECURITY.md index 3447f25fd..1062462b5 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -1,24 +1,249 @@ -## Security +# Security Policy -NVIDIA is dedicated to the security and trust of our software products and services, including all source code repositories managed through our organization. +## Reporting a Vulnerability -If you need to report a security issue, please use the appropriate contact points outlined below. **Please do not report security vulnerabilities through GitHub.** +Do not report security vulnerabilities through a public GitHub issue, pull +request, discussion, or other public channel. -## Reporting Potential Security Vulnerability in an NVIDIA Product +Use one of these private reporting channels: -To report a potential security vulnerability in any NVIDIA product: -- Web: [Security Vulnerability Submission Form](https://www.nvidia.com/object/submit-security-vulnerability.html) -- E-Mail: psirt@nvidia.com - - We encourage you to use the following PGP key for secure email communication: [NVIDIA public PGP Key for communication](https://www.nvidia.com/en-us/security/pgp-key) - - Please include the following information: - - Product/Driver name and version/branch that contains the vulnerability - - Type of vulnerability (code execution, denial of service, buffer overflow, etc.) - - Instructions to reproduce the vulnerability - - Proof-of-concept or exploit code - - Potential impact of the vulnerability, including how an attacker could exploit the vulnerability +- Web: [NVIDIA Vulnerability Disclosure Program](https://www.nvidia.com/en-us/security/) + (preferred) +- Email: [psirt@nvidia.com](mailto:psirt@nvidia.com). Use the + [NVIDIA public PGP key](https://www.nvidia.com/en-us/security/pgp-key) + for encrypted communication. +- GitHub: Use the repository Security tab and select Report a vulnerability. -While NVIDIA currently does not have a bug bounty program, we do offer acknowledgement when an externally reported security issue is addressed under our coordinated vulnerability disclosure policy. Please visit our [Product Security Incident Response Team (PSIRT)](https://www.nvidia.com/en-us/security/psirt-policies/) policies page for more information. +Include: -## NVIDIA Product Security +- The affected NVCF version, release, branch, or commit +- The affected component and vulnerability type +- Reproduction steps and required configuration +- Proof-of-concept code, if available +- The expected and observed behavior +- An impact assessment, including affected security boundaries -For all security-related concerns, please visit NVIDIA's Product Security portal at https://www.nvidia.com/en-us/security +NVIDIA Product Security Incident Response Team (PSIRT) will acknowledge the +report, validate the issue and severity, coordinate a fix, and publish a +security bulletin when appropriate. + +## Security Architecture & Context + +NVIDIA Cloud Functions (NVCF) is a Kubernetes-based platform for deploying, +managing, and invoking GPU workloads. This repository contains service code, +CLIs, shared libraries, deployment charts, migrations, examples, and validation +tooling. + +NVCF has three primary planes: + +- The control plane manages function, task, deployment, secret, and cluster + state. +- The invocation plane authenticates and routes HTTP, streaming, and gRPC + requests to function workloads and applies rate limits. +- The compute plane connects GPU clusters to the platform. NVIDIA Cluster Agent + (NVCA) turns platform requests into Kubernetes resources and manages workload + lifecycle. + +NATS JetStream connects these planes and buffers work. Kubernetes, artifact +registries, object storage, secret stores, telemetry backends, and platform data +stores are external security dependencies. + +Repository Exposure Classification: Public. +Basis: `NVIDIA/nvcf` is a public repository on GitHub. This document uses +public-safe detail. + +Service Exposure Classification: External / Regulated (high confidence). +Basis: NVCF exposes customer-facing APIs, runs production GPU workloads, and +handles credentials, workload artifacts, request and response data, secrets, +and telemetry. + +### Security Boundaries + +1. External callers cross an ingress boundary into the control-plane and + invocation APIs. +2. Invocation services cross an authorization boundary when they validate a + caller and resolve the function versions that caller may invoke. +3. Control-plane and invocation services cross a messaging boundary when they + publish work to NATS JetStream. +4. NVCA crosses a cluster-administration boundary when it creates and manages + pods, Helm releases, custom resources, service accounts, and related + Kubernetes objects. +5. User-supplied function and task containers run inside the compute plane but + are not trusted as platform components. +6. Artifact, secret, registry, object-storage, and telemetry integrations cross + deployment-specific network and credential boundaries. + +## Threat Model + +This threat model incorporates the repository architecture, current service +implementations, and an existing STRIDE assessment. Threats are ordered by +expected impact and exposure. + +1. Authorization and tenant-routing confusion + + The HTTP invocation service protects legacy invocation routes retained only + for backward compatibility, transparent load-balancer routes, and worker + attach routes in + `src/invocation-plane-services/http-invocation/crates/server/src/app.rs`. + The legacy routes are deprecated and must not be used for new integrations. + It delegates function authorization through + `nvcf_api/mod.rs`. The gRPC proxy independently uses bearer tokens, + `function-id`, function-version metadata, request IDs, and connection state + in `src/invocation-plane-services/grpc-proxy/proxy/director.go`. Incorrectly + binding any token, function identifier, cached authorization result, request + ID, or worker connection could let one caller invoke or reconnect to another + tenant's workload. + +2. Privileged workload orchestration + + NVCA consumes platform work messages and manages `ICMSRequest`, + `MiniService`, pod, and Helm workload state in GPU clusters. A forged, + replayed, or incorrectly authorized launch specification could cause + unintended images, charts, service accounts, volumes, or Kubernetes + resources to run with compute-plane privileges. Queue authentication, + namespace isolation, admission policy, and least-privilege RBAC are critical + controls. + +3. Secret and credential disclosure + + Invocation bearer tokens, NATS and OAuth credentials, rate-limiter tokens, + registry credentials, object-storage URLs, and workload secrets pass through + multiple services. Relevant paths include + `src/invocation-plane-services/http-invocation/crates/server/src/middleware/auth.rs`, + `src/invocation-plane-services/grpc-proxy/proxy/credentials/bearer.go`, + `src/control-plane-services/nats-auth-callout`, the OpenBao deployment + assets, and `src/compute-plane-services/image-credential-helper`. Logging, + tracing, error handling, configuration output, or telemetry export that + records these values could expose access to platform or tenant resources. + +4. Request, response, and artifact data exposure or tampering + + The invocation service can buffer or stream request bodies, forward headers + through NATS, issue worker attach tokens, and use object storage for assets + and large responses. + `src/compute-plane-services/worker-init/internal/downloader/downloader.go` + downloads workload artifacts from credential-bearing URLs. A stolen or + misbound attach token or signed URL could expose customer data. A tampered + request, response, model, container, or Helm artifact could alter workload + behavior or execute attacker-controlled code. + +5. Resource exhaustion and rate-limit bypass + + Public HTTP, streaming, gRPC CONNECT, status, and worker attach paths can + hold connections, buffer data, enqueue work, trigger autoscaling, and consume + GPU capacity. Disabled, unavailable, incorrectly scoped, or bypassed rate + limiting could permit denial of service or unexpected cost. Request-size, + connection, queue, timeout, and per-tenant workload limits must remain + effective together. + +6. Telemetry data exposure + + `src/compute-plane-services/byoo-otel-collector` processes workload logs, + metrics, and traces, including log bodies and Kubernetes metadata. + Misconfigured exporters, overly broad metadata collection, unsafe log + content, or weak backend access control could disclose tenant data, + credentials, model details, or operational topology. + +7. Deployment and software supply-chain tampering + + Bazel rules, container image definitions, Helm and Helmfile configuration, + migrations, generated manifests, and release automation determine what runs + in control-plane and compute-plane clusters. A compromised dependency, + registry credential, image tag, chart source, migration, or release workflow + could introduce code or configuration across many NVCF deployments. + +## Critical Security Assumptions + +- The configured identity provider issues valid, short-lived credentials, and + authorization services correctly bind callers to organizations, functions, + function versions, and administrative actions. +- Production ingress terminates TLS with current protocols and certificates, + filters abusive traffic, and exposes only intended service routes. +- NATS JetStream, platform data stores, object storage, registries, secret + stores, and telemetry backends authenticate peers, encrypt traffic, enforce + least privilege, and protect data at rest. +- Kubernetes control planes, nodes, container runtimes, RBAC, admission + controls, network policies, and namespace boundaries isolate platform + components and mutually untrusted workloads. +- User-provided container images and Helm charts are untrusted relative to the + platform. Deployments apply the policies needed to prevent workload access to + host resources, other tenants, and platform credentials. +- Images, charts, binaries, models, and other artifacts come from approved + sources and retain their expected integrity between publication and use. +- Worker attach tokens, signed object-storage URLs, session identifiers, and + cached authorization decisions remain short-lived and bound to the intended + request, caller, function, and deployment. +- Operators enable and correctly scope rate limits, timeouts, quotas, and queue + limits for production traffic. +- Health, metrics, profiling, debugging, and administrative endpoints are + restricted to authorized operator networks and are disabled when unnecessary. +- Applications avoid placing secrets or sensitive request data in logs, traces, + metrics, error messages, or other telemetry. + +## Trust Model + +- NVCF maintainers, signed release automation, and authorized deployment + operators are trusted to publish and configure platform components. +- Authenticated callers are trusted only for the organizations, functions, and + actions granted by the authorization service. +- Function and task images, charts, request bodies, headers, model artifacts, + and workload-generated telemetry are untrusted inputs. +- NVCA and other cluster controllers are privileged components. Their NATS + identities, Kubernetes service accounts, and reconciliation inputs must be + tightly scoped. +- Cloud services, Kubernetes, NATS, registries, secret stores, object storage, + and telemetry systems are conditionally trusted dependencies. NVCF relies on + their configured identity, encryption, isolation, durability, and audit + controls. + +## Deployment Assumptions + +Self-managed operators are responsible for: + +- Configuring TLS, ingress filtering, network policy, and private access for + internal service endpoints +- Supplying least-privilege identities for NATS, Kubernetes, registries, + object storage, data stores, and secret stores +- Enabling authentication, authorization, rate limiting, quotas, and audit + forwarding for production environments +- Pinning and verifying approved image, chart, and dependency versions +- Rotating credentials and limiting access to secret-bearing files and + Kubernetes Secrets +- Restricting workload egress and preventing access to node, metadata-service, + and platform control credentials +- Protecting telemetry pipelines and applying retention, redaction, and access + policies appropriate for customer data + +## Repository Scope Notes + +These notes help distinguish deployable security impact from repository-only +test and documentation content. They do not suppress findings by themselves. + +- Files under `examples/`, test directories, and documentation may contain + placeholder credentials, sample payloads, local endpoints, or intentionally + insecure configurations used to demonstrate a boundary. Confirm a production + path before treating them as deployed secrets or controls. +- Files under `vendor/` are third-party source snapshots. Assess findings + against the version used by a shipped component and the reachable call path. +- Local-development configuration may disable TLS, authentication, or rate + limiting to support isolated testing. Such settings are unsafe if promoted to + a production deployment. +- Security issues in user-provided workload code are normally owned by that + workload. They are NVCF issues when the platform breaks an isolation, + authorization, confidentiality, integrity, or availability boundary. + +## Operational Guidance + +- Use short-lived credentials and least-privilege service accounts. +- Keep authorization, rate limiting, and worker-connection checks fail-safe for + protected operations. +- Pin production artifacts by immutable version or digest and scan dependencies, + images, and charts before release. +- Restrict NATS subjects, Kubernetes namespaces, secret paths, registry scopes, + and object-storage objects to the smallest required audience. +- Redact authorization headers, signed URLs, request bodies, model data, and + secrets from logs and telemetry. +- Monitor authentication failures, denied rate-limit checks, queue growth, + unusual autoscaling, secret access, and privileged Kubernetes changes. +- Re-run the repository security review after material architecture, deployment, + authentication, data-flow, or trust-boundary changes.