ncu_service

Standalone Nsight Compute profiling service for GPU Mode submissions.

The service accepts the same HTTP contract used by popcorn-cli --profile-brev, then runs each requested benchmark shape in a fresh GPU container on the Brev node. It does not depend on KernelBot or KernelBoard.

Requests must include a valid X-Popcorn-Cli-Id header by default. The service validates that header against the KernelBot API before accepting, polling, or serving job artifacts.

Security Model

This is designed so public traffic terminates at the service, while untrusted submission.py code runs in a short-lived container:

• docker run --rm --gpus all
• read-only reference-kernels mount
• read-only submitted submission.py mount
• writable per-job output directory only
• --network none
• --cap-drop ALL
• --security-opt no-new-privileges
• optional read-only container root filesystem
• Landlock wrapper inside the container

Landlock is enabled by default in a CUDA-compatible mode. Docker remains the primary machine-protection boundary: the worker has a read-only root filesystem, no network, no elevated capabilities, read-only inputs, and only per-job output mounts are writable. Landlock is deliberately permissive inside that container because CUDA and Nsight Compute need broad NVIDIA runtime access. Do not mount SSH keys, service source, other users' jobs, or operator secrets into the worker container.

Build Worker Image

Build this on the Brev node:

cd ~/ncu_service
docker build -f docker/worker.Dockerfile -t ncu-service-worker:latest .

The default image installs PyTorch CUDA 13.0 wheels and expects ncu to be available in the image. Adjust docker/worker.Dockerfile if the Brev image needs a different CUDA or Nsight Compute version.

Run Service

Run the API service on the Brev node. Bind to loopback first and reach it with an SSH tunnel:

cd ~/ncu_service
export NCU_SERVICE_REFERENCE_PROBLEMS_DIR="$HOME/reference-kernels/problems"
export NCU_SERVICE_WORKER_IMAGE="ncu-service-worker:latest"
export NCU_SERVICE_JOBS_DIR="$HOME/ncu-service-jobs"
export KERNELBOT_API_URL="https://site--bot--dxfjds728w5v.code.run"
export NCU_SERVICE_REQUIRE_CLI_ID=1
# If the worker image does not include ncu, mount the host install:
export NCU_SERVICE_NCU_HOST_PATH="/opt/nvidia/nsight-compute/2025.1.1"
export NCU_SERVICE_LANDLOCK=1
uvicorn ncu_service.main:app --host 127.0.0.1 --port 8765

NCU Capture Tuning

By default the worker profiles kernels inside the evaluator's custom_kernel NVTX range with --set full. Most problems capture the first 10 matching launches. eigh_py captures the first 64 matching launches because torch.linalg.eigh/cuSOLVER often spends the first launches on tensor clone and solver setup before the hot factorization kernels appear.

The service operator can override the capture window without changing popcorn-cli:

export NCU_SERVICE_NCU_LAUNCH_COUNT=64
export NCU_SERVICE_NCU_LAUNCH_SKIP=10
export NCU_SERVICE_NCU_LAUNCH_SKIP_BEFORE_MATCH=0
export NCU_SERVICE_NCU_KERNEL_NAME_BASE=demangled
export NCU_SERVICE_NCU_KERNEL_NAME='regex:syevd|sytrd|syevj|cusolver'
export NCU_SERVICE_NCU_KERNEL_ID='::regex:^foo:2'

Only these named options are passed into worker containers. User submissions cannot set arbitrary NCU flags through the hosted API.

From your laptop:

ssh -fN -L 127.0.0.1:8765:127.0.0.1:8765 suitable-turquoise-whitefish
curl http://127.0.0.1:8765/health

Then use popcorn-cli:

POPCORN_BREV_PROFILER_URL=http://127.0.0.1:8765 \
  popcorn submit submission.py --leaderboard qr --profile-brev --benchmark-index 0

Omit --benchmark-index to profile every entry in benchmarks:. The service does not embed shapes; it reads them from the configured reference-kernels checkout.

API

POST /profile

Multipart form fields:

• file: submission.py
• leaderboard: for example qr
• benchmark_index: optional integer selecting benchmarks[index]

Response:

{"job_id":"...","status":"queued","queue_position":1}

GET /jobs/{job_id} returns job status, queue position, run summaries, and artifact names.

GET /jobs/{job_id}/artifacts/{artifact_name} downloads a zip containing:

• ncu-details.txt: default human/agent-readable Nsight Compute details
• ncu-details.csv: CSV export of the same details
• profile/profile.ncu-rep: optional Nsight Compute GUI report
• logs and metadata for one benchmark shape

Reference Freshness

Production service-code updates should happen through Northflank, matching the KernelBot service: merge to main, Northflank builds docker/proxy.Dockerfile, the proxy container deploys the code to the already-running GPU host, restarts uvicorn, and reopens the tunnel. It does not start or provision a machine. It also does not update reference-kernels.

See docs/update-service.md for the full update procedure and troubleshooting notes.

scripts/configure_northflank_autodeploy.sh
scripts/update_ncu_service.sh
scripts/update_reference_kernels.sh --branch luc/eigh-profile

The service update path:

• syncs this checkout to ~/ncu_service on suitable-turquoise-whitefish
• installs the service into the existing .venv
• rebuilds ncu-service-worker:latest
• restarts uvicorn ncu_service.main:app on 127.0.0.1:8765
• restarts the existing Northflank deploy/brev-profiler-proxy tunnel

The reference-kernels update path is explicit and manual:

scripts/update_reference_kernels.sh --branch main

Put host-local overrides in ~/ncu-service.env on the GPU host. For example:

NCU_SERVICE_ADMIN_TOKEN=...
NCU_SERVICE_NCU_HOST_PATH=/opt/nvidia/nsight-compute/2025.1.1
NCU_SERVICE_NCU_LAUNCH_COUNT=64

After the Northflank service is configured as git-backed, changes to ncu_service source should go through PRs merged to main. Use scripts/update_ncu_service.sh only for local emergency service deploys. Use scripts/update_reference_kernels.sh --branch <branch> only when explicitly switching the profiler to a different reference-kernels branch.

To manually verify the GPU-host checkout:

git -C "$HOME/reference-kernels" fetch origin main
git -C "$HOME/reference-kernels" status --short
git -C "$HOME/reference-kernels" rev-parse HEAD

Only reset automatically when local evaluator/profile-mode patches have already landed upstream.