Add CUDA GPU coredump support to nvproxy#13644
Open
luiscape wants to merge 1 commit into
Open
Conversation
Adds the five missing `ioctls` required to generate [CUDA GPU coredumps](https://docs.nvidia.com/cuda/cuda-gdb/index.html#gpu-core-dump-support) (`CUDA_ENABLE_COREDUMP_ON_EXCEPTION=1`) under nvproxy, all on the `GT200_DEBUGGER` (`0x83de`) class: - `NV83DE_CTRL_CMD_DEBUG_SET_MODE_MMU_DEBUG` (`0x83de0307`) control command — enables MMU debug mode during exception handling. - `NV83DE_CTRL_CMD_DEBUG_READ_MEMORY` (`0x83de0315`) control command, with its `NV83DE_CTRL_DEBUG_READ_MEMORY_PARAMS` struct — reads a block of memory from the debugged context into an embedded pointer (`buffer`), which requires a custom handler. Note that the driver passes `RMAPI_PARAM_COPY_FLAGS_DISABLE_MAX_SIZE_CHECK` for this control (see `src/nvidia/src/kernel/rmapi/embedded_param_copy.c`), so unlike most embedded pointers the buffer is *not* bounded by `RMAPI_PARAM_COPY_MAX_PARAMS_SIZE`; the handler mirrors this. - `NV83DE_CTRL_CMD_DEBUG_SUSPEND_CONTEXT` (`0x83de0317`) control command — suspends the faulting SM context so its state can be captured. - `NV83DE_CTRL_CMD_DEBUG_RESUME_CONTEXT` (`0x83de0318`) control command — resumes the context after capture. No params. - `NV83DE_CTRL_CMD_DEBUG_SET_MODE_ERRBAR_DEBUG` (`0x83de031f`) control command — enables error-barrier debug mode (Volta+), a prerequisite for capturing SM exception state. All are simple pass-through controls except `DEBUG_READ_MEMORY`. They are gated on `compute,utility`, consistent with the existing `NV83DE` debug controls (`SET_EXCEPTION_MASK`, `READ_ALL_SM_ERROR_STATES`, `CLEAR_ALL_SM_ERROR_STATES`). Coredump generation additionally uses `NV2080_CTRL_CMD_GPU_EXEC_REG_OPS` (already supported), so the feature requires running with `--nvproxy-allowed-driver-capabilities=all,profiling`. Tested on an A10G with a faulting Triton kernel: the generated `.nvcudmp` matches native (runc) byte-for-byte in size, and `cuda-gdb` fully decodes it (`CUDA Exception: Warp Illegal Address`, faulting PC/kernel/warp/lane and backtrace). Without this change, coredump generation silently produces an empty dump (`No exception was found on any device`).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the five missing
ioctlsrequired to generate CUDA GPU coredumps usingCUDA_ENABLE_COREDUMP_ON_EXCEPTION=1, all on theGT200_DEBUGGERclass.All the new ioctls simple pass-through controls except
DEBUG_READ_MEMORY. They are gated oncompute,utility, consistent with the existingNV83DEdebug controls (SET_EXCEPTION_MASK,READ_ALL_SM_ERROR_STATES,CLEAR_ALL_SM_ERROR_STATES).Coredump generation additionally uses
NV2080_CTRL_CMD_GPU_EXEC_REG_OPS(already supported), so the feature requires running with--nvproxy-allowed-driver-capabilities=all,profiling.In order for this to work, I added a
ctrlDebugReadMemoryhandler that translates the embeddedBufferpointer inNV83DE_CTRL_DEBUG_READ_MEMORY_PARAMSby substituting a sentry-allocated buffer of the requestedLengthbefore invoking the host driver and copying the result back out to the application, mirroring the driver's own embedded-pointer copy logic inembedded_param_copy.c(which disables theRMAPI_PARAM_COPY_MAX_PARAMS_SIZEcap for this control).Tested on an A10G with a faulting Triton kernel: the generated
.nvcudmpmatches native (runc) byte-for-byte in size, andcuda-gdbfully decodes it (CUDA Exception: Warp Illegal Address, faulting PC/kernel/warp/lane and backtrace). Without this change, coredump generation silently produces an empty dump (No exception was found on any device).Assisted-by: Claude