tensor_parser_aten: tag planned tensor DataPtr with its real device by shoumikhin · Pull Request #21289 · pytorch/executorch

shoumikhin · 2026-07-23T17:46:20Z

Summary:
In ATen mode, parseTensor built every planned tensor's storage DataPtr with a
hardcoded c10::DeviceType::CPU, and its options with at::CPU(type). A tensor
whose planned buffer lives on an accelerator (e.g. a CUDA-delegate planned
buffer) therefore got a CPU-tagged data pointer, so the runtime treated device
memory as host memory and the delegate rejected it (Method::init fails in
getDeviceFromPtr on a CPU-tagged device pointer).

Fix: read the device from the serialized tensor's extra_tensor_info
(device_type/device_index, defaulting to CPU when absent, matching the portable
parser), then build the tensor with a single
at::from_blob(ptr, sizes, strides, /storage_offset=/0, deleteNothing,
at::TensorOptions().dtype(type).device(device),
/target_device=/device)
so the storage DataPtr, the TensorImpl device (device(), is_cuda()), and the
dispatch key all agree. Passing target_device makes from_blob skip
getDeviceFromPtr, so the same path works for a real device pointer and for a null
runtime-bound pointer without inspecting the pointer. TRT-only / CPU programs are
unchanged (device defaults to CPU).

Also update internal_set_tensor_data (tensor_util_aten.cpp) to preserve the
source tensor's device instead of hardcoding CPU, so sharing a device input's
storage keeps its device tag.

fbcode and xplat copies kept identical.

Differential Revision: D113384858

pytorch-bot · 2026-07-23T17:46:23Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21289

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 27 Pending

As of commit 4ea0725 with merge base 4a26c64 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

meta-codesync · 2026-07-23T17:46:29Z

@shoumikhin has exported this pull request. If you are a Meta employee, you can view the originating Diff in D113384858.

github-actions · 2026-07-23T17:47:24Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Copilot

Pull request overview

This PR fixes incorrect device tagging for planned tensors in ATen-mode deserialization, ensuring that tensors backed by accelerator memory (e.g., CUDA planned buffers) are constructed with consistent device information across the storage DataPtr, TensorImpl device, and dispatch key. It also prevents CUDA backend delegate initialization from clobbering shared-library temp files when identical partitions are loaded multiple times.

Changes:

Read device_type/device_index from extra_tensor_info during ATen tensor parsing and construct tensors via at::from_blob(..., TensorOptions().device(device), /*target_device=*/device) to keep device tagging consistent.
Preserve the source tensor’s device when sharing tensor storage in internal::share_tensor_data.
Ensure unique temp .so paths in the CUDA backend by adding an atomic counter suffix.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
runtime/executor/tensor_parser_aten.cpp	Parse device metadata and rebuild tensors with correct device tagging via `from_blob(..., target_device=...)`.
runtime/core/exec_aten/util/tensor_util_aten.cpp	Preserve device when sharing tensor storage data pointers.
backends/cuda/runtime/cuda_backend.cpp	Avoid temp `.so` path collisions when loading identical CUDA partitions by adding a unique counter.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

shoumikhin · 2026-07-23T20:03:52Z

+    device_type = raw_device_type == executorch_flatbuffer::DeviceType::CUDA
+        ? c10::DeviceType::CUDA
+        : c10::DeviceType::CPU;
+    device_index = static_cast<c10::DeviceIndex>(
+        s_tensor->extra_tensor_info()->device_index());


FlatBuffers byte is int8_t (signed) and the accessor returns int8_t, so a stored value >127 is already negative on the wire rather than produced by the cast. I've added an explicit check that rejects a negative accelerator device_index (and thus the -1 "current device" placeholder) from the untrusted PTE before it is used.

shoumikhin · 2026-07-23T20:03:54Z

  if (s_tensor->shape_dynamism() ==
      executorch_flatbuffer::TensorShapeDynamism::DYNAMIC_UNBOUND) {
-    // Provide fully dynamic tensors with an allocator so they can be resized
-    // within aten kernels.
+    // Fully dynamic tensors get an allocator so aten kernels can resize them.
+    // Device-delegate planned buffers are statically bounded, so a device tensor
+    // never reaches this CPU-tagged path.


The DYNAMIC_UNBOUND path stays CPU-tagged by design: device-delegate planned buffers are statically bounded (not DYNAMIC_UNBOUND), so a device-annotated tensor does not reach here. Hard-rejecting a non-CPU DYNAMIC_UNBOUND tensor is a reasonable additional guard; I'll track it as a follow-up rather than expand this change, since it cannot occur for the delegate path this PR targets.

shoumikhin · 2026-07-23T20:03:55Z

  ET_CHECK_OR_RETURN_ERROR(
      t_src.mutable_data_ptr() != nullptr,
      InvalidArgument,
      "Source tensor should have data_ptr not being nullptr.");
-  // Assign the dataptr as the input tensor dataptr
+  // Preserve the source device; hardcoding CPU would mis-tag a device input's


Agreed this is a pre-existing asymmetry with the portable implementation for empty tensors (nbytes()==0, valid nullptr data). It's outside the device-tagging scope of this change; tracking as a follow-up to allow nullptr when nbytes()==0.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

shoumikhin · 2026-07-23T20:03:57Z

+    // Rebuild so storage DataPtr, TensorImpl device, and dispatch key agree.
+    // target_device makes from_blob skip getDeviceFromPtr, so the same path
+    // works for a real pointer and for a null runtime-bound one.
+    tensor = at::from_blob(
+        data_ptr.get(),


Correct that the null-placeholder path relies on target_device making from_blob skip getDeviceFromPtr. An ATen-mode device-parser unit test (building a flatbuffer tensor with extra_tensor_info.device_type=CUDA and asserting device/dtype, plus the unindexed-CPU default) is planned as a dedicated follow-up wired into OSS CMake + CI, mirroring the portable tensor_parser_device_test.

…ytorch#21289) Summary: In ATen mode, parseTensor built every planned tensor's storage DataPtr with a hardcoded c10::DeviceType::CPU, and its options with at::CPU(type). A tensor whose planned buffer lives on an accelerator (e.g. a CUDA-delegate planned buffer) therefore got a CPU-tagged data pointer, so the runtime treated device memory as host memory and the delegate rejected it (Method::init fails in getDeviceFromPtr on a CPU-tagged device pointer). Fix: read the device from the serialized tensor's extra_tensor_info (device_type/device_index, defaulting to CPU when absent, matching the portable parser), then build the tensor with a single at::from_blob(ptr, sizes, strides, /*storage_offset=*/0, deleteNothing, at::TensorOptions().dtype(type).device(device), /*target_device=*/device) so the storage DataPtr, the TensorImpl device (device(), is_cuda()), and the dispatch key all agree. Passing target_device makes from_blob skip getDeviceFromPtr, so the same path works for a real device pointer and for a null runtime-bound pointer without inspecting the pointer. TRT-only / CPU programs are unchanged (device defaults to CPU). Also update internal_set_tensor_data (tensor_util_aten.cpp) to preserve the source tensor's device instead of hardcoding CPU, so sharing a device input's storage keeps its device tag. fbcode and xplat copies kept identical. Differential Revision: D113384858

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

shoumikhin · 2026-07-23T22:41:16Z

+    filesystem::path so_path = temp_dir /
+        (so_blob_key + to_string(get_process_id()) + "_" +
+         to_string(so_file_counter.fetch_add(1, std::memory_order_relaxed)) +
+         ".so");


Fixed. Since so_blob_key can be an untrusted, variable-length payload key, it's no longer interpolated into the path at all — the filename is now a fixed executorch_cuda_<pid>_<counter>.so prefix under temp_directory_path(), so a key containing / or ../ can no longer influence the write location. The key is still used only to fetch the blob from the named-data map.

…ytorch#21289) Summary: In ATen mode, parseTensor built every planned tensor's storage DataPtr with a hardcoded c10::DeviceType::CPU, and its options with at::CPU(type). A tensor whose planned buffer lives on an accelerator (e.g. a CUDA-delegate planned buffer) therefore got a CPU-tagged data pointer, so the runtime treated device memory as host memory and the delegate rejected it (Method::init fails in getDeviceFromPtr on a CPU-tagged device pointer). Fix: read the device from the serialized tensor's extra_tensor_info (device_type/device_index, defaulting to CPU when absent, matching the portable parser), then build the tensor with a single at::from_blob(ptr, sizes, strides, /*storage_offset=*/0, deleteNothing, at::TensorOptions().dtype(type).device(device), /*target_device=*/device) so the storage DataPtr, the TensorImpl device (device(), is_cuda()), and the dispatch key all agree. Passing target_device makes from_blob skip getDeviceFromPtr, so the same path works for a real device pointer and for a null runtime-bound pointer without inspecting the pointer. TRT-only / CPU programs are unchanged (device defaults to CPU). Also update internal_set_tensor_data (tensor_util_aten.cpp) to preserve the source tensor's device instead of hardcoding CPU, so sharing a device input's storage keeps its device tag. fbcode and xplat copies kept identical. Differential Revision: D113384858

Summary: The CUDA backend writes each compiled AOTInductor library to a temporary file, named from its content key plus the process id, before loading it. Two identical CUDA partitions can share the same content key, so both delegates computed the same path; loading the second library overwrote the first while it was still in use, and symbol lookup could then crash. Append a per-process atomic counter to the temporary file name so every delegate in a process gets a distinct path. The file is still removed when the delegate is destroyed, as before; only the name changes. Differential Revision: D113322459

…ytorch#21289) Summary: In ATen mode, parseTensor built every planned tensor's storage DataPtr with a hardcoded c10::DeviceType::CPU, and its options with at::CPU(type). A tensor whose planned buffer lives on an accelerator (e.g. a CUDA-delegate planned buffer) therefore got a CPU-tagged data pointer, so the runtime treated device memory as host memory and the delegate rejected it (Method::init fails in getDeviceFromPtr on a CPU-tagged device pointer). Fix: read the device from the serialized tensor's extra_tensor_info (device_type/device_index, defaulting to CPU when absent, matching the portable parser), then build the tensor with a single at::from_blob(ptr, sizes, strides, /*storage_offset=*/0, deleteNothing, at::TensorOptions().dtype(type).device(device), /*target_device=*/device) so the storage DataPtr, the TensorImpl device (device(), is_cuda()), and the dispatch key all agree. Passing target_device makes from_blob skip getDeviceFromPtr, so the same path works for a real device pointer and for a null runtime-bound pointer without inspecting the pointer. TRT-only / CPU programs are unchanged (device defaults to CPU). Also update internal_set_tensor_data (tensor_util_aten.cpp) to preserve the source tensor's device instead of hardcoding CPU, so sharing a device input's storage keeps its device tag. fbcode and xplat copies kept identical. Differential Revision: D113384858

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

+    filesystem::path so_path = temp_dir /
+        ("executorch_cuda_" + to_string(get_process_id()) + "_" +
+         to_string(so_file_counter.fetch_add(1, std::memory_order_relaxed)) +
+         ".so");


shoumikhin requested a review from JacobSzwejbka as a code owner July 23, 2026 17:46

Copilot AI review requested due to automatic review settings July 23, 2026 17:46

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 23, 2026

meta-codesync Bot added the meta-exported label Jul 23, 2026

meta-codesync Bot had a problem deploying to cadence July 23, 2026 17:46 Failure

Copilot started reviewing on behalf of shoumikhin July 23, 2026 17:46 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 23, 2026 18:56

shoumikhin force-pushed the export-D113384858 branch from be429c7 to 23658d4 Compare July 23, 2026 18:56

Copilot started reviewing on behalf of shoumikhin July 23, 2026 18:57 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 23, 2026 20:12

shoumikhin force-pushed the export-D113384858 branch from 23658d4 to ec9af03 Compare July 23, 2026 20:12

Copilot started reviewing on behalf of shoumikhin July 23, 2026 20:13 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

shoumikhin added 2 commits July 23, 2026 18:40

Copilot AI review requested due to automatic review settings July 24, 2026 01:40

shoumikhin force-pushed the export-D113384858 branch from ec9af03 to 4ea0725 Compare July 24, 2026 01:40

Copilot started reviewing on behalf of shoumikhin July 24, 2026 01:41 View session

Copilot AI reviewed Jul 24, 2026

View reviewed changes

Comment thread backends/cuda/runtime/cuda_backend.cpp

Comment on lines +341 to +344

filesystem::path so_path = temp_dir /

("executorch_cuda_" + to_string(get_process_id()) + "_" +

to_string(so_file_counter.fetch_add(1, std::memory_order_relaxed)) +

".so");

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

tensor_parser_aten: tag planned tensor DataPtr with its real device#21289

tensor_parser_aten: tag planned tensor DataPtr with its real device#21289
shoumikhin wants to merge 2 commits into
pytorch:mainfrom
shoumikhin:export-D113384858

shoumikhin commented Jul 23, 2026

Uh oh!

pytorch-bot Bot commented Jul 23, 2026 •

edited

Loading

Uh oh!

meta-codesync Bot commented Jul 23, 2026

Uh oh!

github-actions Bot commented Jul 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

shoumikhin Jul 23, 2026

Uh oh!

shoumikhin Jul 23, 2026

Uh oh!

shoumikhin Jul 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

shoumikhin Jul 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

shoumikhin Jul 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

shoumikhin commented Jul 23, 2026

Uh oh!

pytorch-bot Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21289

⏳ No Failures, 27 Pending

Uh oh!

meta-codesync Bot commented Jul 23, 2026

Uh oh!

github-actions Bot commented Jul 23, 2026

This PR needs a release notes: label

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

shoumikhin Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

shoumikhin Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

shoumikhin Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

shoumikhin Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

shoumikhin Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pytorch-bot Bot commented Jul 23, 2026 •

edited

Loading

This PR needs a `release notes:` label