[ET Device Support] CUDA-native Qwen 3.5 MoE inference with device tensor pipeline by Gasoonjia · Pull Request #18788 · pytorch/executorch

Gasoonjia · 2026-04-09T08:22:34Z

Stack from ghstack (oldest at bottom):

-> [ET Device Support] CUDA-native Qwen 3.5 MoE inference with device tensor pipeline #18788
[ET Device Support] Extract shared device test utilities to reduce redundancy #18762
[ET Device Support] Add device tensor helper functions to TensorPtr API #18761
[ET Device Support] Add ExecutorchBackendConfig flags for skipping H2D/D2H copies #18760
[ET Device Support] PropagateDevicePass inserts H2D/D2H copy ops at delegate boundaries #18730
[ET Device Support] Define et_copy runtime h2d and d2h copy ops #18729
[ET Device Support] Define AOT device copy ops registry #18728
[ET Device Support] CudaAllocator: device memory allocator for CUDA backend #18477
[ET Device Support] Module: allocate device memory for planned buffers #18476
[ET Device Support] MemoryManager: add per-buffer device metadata #18475
[ET Device Support] MethodMeta: expose per-buffer device placement API #18474
[ET Device Support] DeviceMemoryBuffer RAII class for device memory lifetime management #18473
[ET Device Support] Emitter reads non_const_buffer_device from graph meta #18472
[ET Device Support] Device-aware memory planning: separate buffers per device type #18375
[ET Device Support] Add NonConstBufferDevice schema for per-buffer device mapping #18330
[ET Device Support] DeviceAllocator interface and DeviceAllocatorRegistry #17535
[ET Device Support] Annotate device attributes of CUDA backend IO tensors cuda device #18080
[ET Device Support] Parse device info from serialized tensor in tensor_parser #18328
[ET Device Support] Propagate device info from TensorSpec into serialized Tensor #18079
[executorch] Propagate device metadata from partitioner result onto TensorSpecs #18078

Integrate the ET device tensor pipeline into the Qwen 3.5 MoE model to
eliminate unnecessary H2D/D2H copies during inference.

Export: Multi-method export (forward + sample) with device memory
planning enabled and method-level H2D/D2H skipping.
Runner: Custom CUDA-native inference loop that keeps logits on GPU
between forward and sample, reuses CUDA tensors across iterations,
and only copies the 8-byte token ID back to CPU for EOS checking.

Differential Revision: D100133933

…nsor pipeline Integrate the ET device tensor pipeline into the Qwen 3.5 MoE model to eliminate unnecessary H2D/D2H copies during inference. - Export: Multi-method export (`forward` + `sample`) with device memory planning enabled and method-level H2D/D2H skipping. - Runner: Custom CUDA-native inference loop that keeps logits on GPU between forward and sample, reuses CUDA tensors across iterations, and only copies the 8-byte token ID back to CPU for EOS checking. Differential Revision: [D100133933](https://our.internmc.facebook.com/intern/diff/D100133933/) [ghstack-poisoned]

pytorch-bot · 2026-04-09T08:22:40Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18788

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 85 New Failures, 1 Cancelled Job

As of commit 5863ffa with merge base b5ae0b9 ():

NEW FAILURES - The following jobs have failed:

Build Presets / apple (pybind) / build (gh)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 2
Build Presets / linux (pybind, linux.2xlarge, executorch-ubuntu-22.04-clang12) / build (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
Build Presets / linux (pybind, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / build (gh)
undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char)'`
Build Presets / windows (pybind) / build (gh)
Process completed with exit code 1.
periodic / test-models-linux (buck2, mv3, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
RuntimeError: Command docker exec -t 247755e23249dd06e8d82b43c88c5d3f3bf28f267f5f4b1ac7c1fd7fdf35e0fb /exec failed with exit code 1
periodic / test-models-linux (cmake, mv3, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
RuntimeError: Command docker exec -t aa8bbb18ef7a3c6a5adb5fa707dc9099592647c572bc62728be79b9e1fdff42b /exec failed with exit code 1
periodic / test-models-linux (cmake, vit, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh)
RuntimeError: Command docker exec -t 0b8f3f7ce42c957b859c8dbeb95a1ba4654b54e4d8b394710ed303fb4570e2ee /exec failed with exit code 1
pull / android / build-android (gh)
Process completed with exit code 2.
pull / test-coreml-bc-macos (macos-m1-stable) / macos-job (gh)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
pull / test-coreml-bc-macos (macos-m2-stable) / macos-job (gh)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
pull / test-llama-runner-linux (bf16, custom, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.arm64.2xlarge, executorch-ubuntu-22.04-gc... / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04... / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu... / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu-22.04-... / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-llama-runner-linux-android / linux-job (gh)
RuntimeError: Command docker exec -t e266b31c6ad732475985bd8368b3b53a8a6a61b90a7773d2c1942e4a6b462979 /exec failed with exit code 2
pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / test-lora-linux / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-lora-multimethod-linux / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-mcu-cortex-m-backend / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / test-models-linux (add_mul, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t c90a19774803811c90ae4e46d3d2b459413e1cdda1baffbd596797b5a735d86d /exec failed with exit code 1
pull / test-models-linux (add, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t dc6f7791ba990e94cc4f434e6830199336abf5cf3f848f04b64488374f33eacf /exec failed with exit code 1
pull / test-models-linux (emformer_join, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
RuntimeError: Command docker exec -t ef92433e8a7a27df7762e592d53ac7d482cec41605d577822ed849ba5363f075 /exec failed with exit code 1
pull / test-models-linux (emformer_transcribe, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 2d8c86969856b5f4c2c8d0a90c3ac5c7703f67392a430d5dc03ede14d6f32d33 /exec failed with exit code 1
pull / test-models-linux (ic3, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 43829d323f6b69a9501369771dc62a9282e1a023592864fbf35dc26173a36191 /exec failed with exit code 1
pull / test-models-linux (ic4, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
RuntimeError: Command docker exec -t 50adfa27199d9b05ee91e6f333c71feb6f751faf1338086dbc77ebda7db22a01 /exec failed with exit code 1
pull / test-models-linux (linear, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t c752c21b07b67afd765eeb3e219bbb671013fa1c07076cd064bc742be68cdba8 /exec failed with exit code 1
pull / test-models-linux (mobilebert, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 11658e47b251b34a154342e70560996dbbc344e21824192077d651bdfd012a57 /exec failed with exit code 1
pull / test-models-linux (mv2, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 7ee7908a5891b879397e4a784949f4b672ee67b8755876b9edaf432bf93b092d /exec failed with exit code 1
pull / test-models-linux (resnet18, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t 04e98c70ef4c61e57cae241dcb0ac1430b37886cde3c26996f75efc776d8e6fc /exec failed with exit code 1
pull / test-models-linux (resnet50, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
RuntimeError: Command docker exec -t f19bcad919d3d2ef2464ec0090d770530e8959f541185b38abcc41395201d58a /exec failed with exit code 1
pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t ce9853e4be6c69d3344fb383ec9f46ae5fa5708b82b8198ec7c679ba70d87d04 /exec failed with exit code 1
pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t a6fc295e5001ec761e2f480a1f37ef323e72a97426765bc67d3f9ad4f6c52c54 /exec failed with exit code 1
pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
RuntimeError: Command docker exec -t f48cc25fda2cc5b7824da6e2bc1e529aa8a7c4b72cf3752cc9a8a33451d086e7 /exec failed with exit code 1
pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t 758e67a3c3498808d1968c76f13942d4a10adaa5655d9b7fe45914f66c658035 /exec failed with exit code 1
pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh)
RuntimeError: Command docker exec -t e1506cdb3e716095f4183c0908c55bcda786ef47035ff6136bd8fc05a34e4d02 /exec failed with exit code 1
pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
RuntimeError: Command docker exec -t a7cec1f264549189bae36eeab90c6ce1de75e43f06c6af38e9da83fe32676fb2 /exec failed with exit code 1
pull / test-moshi-linux / linux-job (gh)
test_mimi
pull / test-multimodal-linux (gemma3-4b) / linux-job (gh)
RuntimeError: Command docker exec -t 77f793bcfb19ee1905fadd9285713fe49d0259cc10eabc5d7336bd0bf7276e20 /exec failed with exit code 1
pull / test-openvino-linux / linux-job (gh)
RuntimeError: Command docker exec -t 143d0223edf53089e0fe9cf651b0a25079c15f6a4d20a594f44c756c091fbd4f /exec failed with exit code 1
pull / test-parakeet-xnnpack-linux / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / test-phi-3-mini-runner-linux / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / test-qnn-wheel-packages-linux (3.10) / linux-job (gh)
RuntimeError: Command docker exec -t 4b42f41c6d3d298d8656ffdde7fcc6cd78d6eb98f8b76dd4ded849fab6ee65e0 /exec failed with exit code 1
pull / test-qnn-wheel-packages-linux (3.11) / linux-job (gh)
RuntimeError: Command docker exec -t f47941328e3cfd5351a3224e7e2dc20c03a8b734881c823dc56f4246308ab590 /exec failed with exit code 1
pull / test-qnn-wheel-packages-linux (3.12) / linux-job (gh)
RuntimeError: Command docker exec -t 922469f8df8c044aaa433c83fb7b58195391cb637106271c6e6ea5eba9d52456 /exec failed with exit code 1
pull / test-qnn-wheel-packages-linux (3.13) / linux-job (gh)
RuntimeError: Command docker exec -t f7368acf5fbeb6932fe46c125ea7ceedf3a76f32de9c57b38b81695b00ae9137 /exec failed with exit code 1
pull / test-samsung-models-linux / linux-job (gh)
test_w2l_fp16
pull / test-sqnr-static-llm-qnn-linux (smollm2_135m) / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / test-static-llama-qnn-linux (stories_110m) / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / test-static-llama-qnn-linux (stories_260k_bc) / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / test-voxtral-realtime-xnnpack-linux / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
pull / test-vulkan-models-linux / linux-job (gh)
RuntimeError: Command docker exec -t 0a33198945c6e5c90411841e601c3878624fcb5fa50baca8daabfc1452ec207d /exec failed with exit code 1
pull / test-vulkan-operators-linux / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
pull / unittest / linux / linux-job (gh)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
pull / unittest / windows / windows-job (gh)
Process completed with exit code 1.
pull / unittest-arm-backend-with-no-deps (test_pytest_models_tosa) / linux-job (gh)
RuntimeError: Command docker exec -t 5b2a9b7285606e362c8295c146fa29475b336b5a35741f678833198a96b543bc /exec failed with exit code 1
pull / unittest-arm-backend-with-no-deps (test_pytest_ops_no_target) / linux-job (gh)
RuntimeError: Command docker exec -t 5636822f605fc3f5a8de994e1f29cd9189be49e4009623ea0f75afd99092d70b /exec failed with exit code 1
pull / unittest-arm-backend-with-no-deps (test_pytest_ops_tosa) / linux-job (gh)
RuntimeError: Command docker exec -t 613315e9bf45bf93b2eca519b5a2b6ae708623d128b37f8c6b06e98fbcd05036 /exec failed with exit code 1
pull / unittest-editable / linux / linux-job (gh)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
pull / unittest-editable / macos / macos-job (gh)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
pull / unittest-editable / windows / windows-job (gh)
Process completed with exit code 1.
pull / unittest-nxp-neutron / linux-job (gh)
RuntimeError: Command docker exec -t 62207ac07aa65fa84a0d982568722f2aa8edaf582c1c6f8c4b90dc5a25150b01 /exec failed with exit code 2
pull / unittest-wasm-bindings (--enable-etdump) / linux-job (gh)
RuntimeError: Command docker exec -t 60c0f48beb8a55665b00294f3e19ba8c777c961ab708b95492b07e1d8c0f19c9 /exec failed with exit code 2
pull / unittest-wasm-bindings / linux-job (gh)
RuntimeError: Command docker exec -t 4cd7ff5e6673adaf1e750c9a394afb9afe0ca63737e61c254d98e988195869a2 /exec failed with exit code 2
Test CUDA Builds / export-model-cuda-artifact (google, gemma-3-4b-it, non-quantized) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
Test CUDA Builds / export-model-cuda-artifact (google, gemma-3-4b-it, quantized-int4-tile-packed) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
Test CUDA Builds / export-model-cuda-artifact (mistralai, Voxtral-Mini-3B-2507, non-quantized) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
Test CUDA Builds / export-model-cuda-artifact (mistralai, Voxtral-Mini-3B-2507, quantized-int4-tile-packed) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
Test CUDA Builds / export-model-cuda-artifact (mistralai, Voxtral-Mini-3B-2507, quantized-int4-weight-only) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
Test CUDA Builds / export-model-cuda-artifact (mistralai, Voxtral-Mini-4B-Realtime-2602, quantized-int4-tile-packed) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
Test CUDA Builds / export-model-cuda-artifact (openai, whisper-large-v3-turbo, quantized-int4-tile-packed) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
Test CUDA Builds / export-model-cuda-artifact (openai, whisper-large-v3-turbo, quantized-int4-weight-only) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
Test CUDA Builds / export-model-cuda-artifact (openai, whisper-small, non-quantized) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
Test CUDA Builds / export-model-cuda-artifact (Qwen, Qwen3-0.6B, non-quantized) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
Test CUDA Builds / export-model-cuda-artifact (Qwen, Qwen3-0.6B, quantized-int4-tile-packed) / linux-job (gh)
AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
Test CUDA Builds / test-models-cuda / linux-job (gh)
undefined reference to executorch::backends::cuda::CudaAllocator::allocate_async(unsigned long, signed char, CUstream_st*)'`
Test CUDA Builds / unittest-cuda / linux-job (gh)
undefined reference to executorch::backends::cuda::CudaAllocator::allocate_async(unsigned long, signed char, CUstream_st*)'`
Test CUDA Windows Export and E2E / export-model-cuda-windows-artifact (facebook, dinov2-small-imagenet1k-1-layer, non-quantized) / linux-job (gh)
RuntimeError: Command docker exec -t 3f1be1f6a464bdb1f6bbec24fcd047f004218f4974af7227c579e0b3bd6d6a09 /exec failed with exit code 1
Test CUDA Windows Export and E2E / export-model-cuda-windows-artifact (nvidia, parakeet-tdt, non-quantized) / linux-job (gh)
RuntimeError: Command docker exec -t d6c92ea0c78184b2be469c46d90e6e7653573ef0ddb68d94119989089263f54c /exec failed with exit code 1
Test CUDA Windows Export and E2E / export-model-cuda-windows-artifact (nvidia, parakeet-tdt, quantized-int4-weight-only) / linux-job (gh)
RuntimeError: Command docker exec -t da292b4c82fea7c31ccfe1a5aca1d11e1fb2c379e6f772388110f2d64c589bee /exec failed with exit code 1
Test QNN Backend / test-qnn / test-backend-linux (qnn, models) / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
Test QNN Backend / test-qnn / test-backend-linux (qnn, operators) / linux-job (gh)
undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`

CANCELLED JOB - The following job was cancelled. Please retry:

pull / unittest / macos / macos-job (gh)
##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

…nsor pipeline Integrate the ET device tensor pipeline into the Qwen 3.5 MoE model to eliminate unnecessary H2D/D2H copies during inference. - Export: Multi-method export (`forward` + `sample`) with device memory planning enabled and method-level H2D/D2H skipping. - Runner: Custom CUDA-native inference loop that keeps logits on GPU between forward and sample, reuses CUDA tensors across iterations, and only copies the 8-byte token ID back to CPU for EOS checking. Differential Revision: [D100133933](https://our.internmc.facebook.com/intern/diff/D100133933/) ghstack-source-id: 364764771 Pull Request resolved: #18788

github-actions · 2026-04-09T08:23:25Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

…h device tensor pipeline" Integrate the ET device tensor pipeline into the Qwen 3.5 MoE model to eliminate unnecessary H2D/D2H copies during inference. - Export: Multi-method export (`forward` + `sample`) with device memory planning enabled and method-level H2D/D2H skipping. - Runner: Custom CUDA-native inference loop that keeps logits on GPU between forward and sample, reuses CUDA tensors across iterations, and only copies the 8-byte token ID back to CPU for EOS checking. Differential Revision: [D100133933](https://our.internmc.facebook.com/intern/diff/D100133933/) [ghstack-poisoned]

…nsor pipeline Pull Request resolved: #18788 Integrate the ET device tensor pipeline into the Qwen 3.5 MoE model to eliminate unnecessary H2D/D2H copies during inference. - Export: Multi-method export (`forward` + `sample`) with device memory planning enabled and method-level H2D/D2H skipping. - Runner: Custom CUDA-native inference loop that keeps logits on GPU between forward and sample, reuses CUDA tensors across iterations, and only copies the 8-byte token ID back to CPU for EOS checking. ghstack-source-id: 364908062 @exported-using-ghexport Differential Revision: [D100133933](https://our.internmc.facebook.com/intern/diff/D100133933/)

Gasoonjia requested review from kirklandsign, larryliu0820 and lucylq as code owners April 9, 2026 08:22

Gasoonjia mentioned this pull request Apr 8, 2026

[ET Device Support] Annotate device attributes of CUDA backend IO tensors cuda device #18080

Open

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 9, 2026

meta-codesync bot added fb-exported meta-exported labels Apr 9, 2026

Gasoonjia mentioned this pull request Apr 9, 2026

add device attribute in ETensor #17161

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ET Device Support] CUDA-native Qwen 3.5 MoE inference with device tensor pipeline#18788

[ET Device Support] CUDA-native Qwen 3.5 MoE inference with device tensor pipeline#18788
Gasoonjia wants to merge 2 commits intogh/gasoonjia/164/basefrom
gh/gasoonjia/164/head

Gasoonjia commented Apr 9, 2026 •

edited

Loading

Uh oh!

pytorch-bot bot commented Apr 9, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Apr 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Gasoonjia commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18788

❌ 85 New Failures, 1 Cancelled Job

Uh oh!

github-actions bot commented Apr 9, 2026

This PR needs a release notes: label

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Gasoonjia commented Apr 9, 2026 •

edited

Loading

pytorch-bot bot commented Apr 9, 2026 •

edited

Loading

This PR needs a `release notes:` label