[CICD] Add T-Head PPU wheel build and integration pipeline - #91
Open
AlexMa616 wants to merge 17 commits into
Open
[CICD] Add T-Head PPU wheel build and integration pipeline#91AlexMa616 wants to merge 17 commits into
AlexMa616 wants to merge 17 commits into
Conversation
Five-piece CI config for the Alibaba T-Head PPU (Jianwu ZW810E) backend. PPU is a CUDA-ABI boxing target (ACCELERATOR=cuda, PPU torch is a local USE_CUDA=1 build), so set_env_ppu.sh builds on the set_env_cuda.sh skeleton (isolated venv + stock +cpu torch) and adds metax-style bundle/core-replace + ascend-style PPU_SDK validation. - configs/ppu.yml + scripts/set_env_ppu.sh: platform manifest + env bootstrap; exports both CUDA-assets kill switches and calls bundle_ppu_libtorch.sh after build_ext - workflows/build-wheel-ppu.yml + integration-test-ppu.yml: entry workflows calling all-tests-common.yml with platform=ppu - workflows/ci.yml: add dev-thead to push.branches Core replacement (ensure_ppu_libtorch_links) fires automatically at import torch_fl time, gated only on lib_ppu/libtorch_cuda.so presence + ACCELERATOR=cuda. Device count asserted > 0 pending CI confirmation.
The PPU image ships FlagGems as source (/workspace/FlagGems has no build/ or lib/, no FlagGemsConfig.cmake, no liboperators.so). The cuda-style discovery in set_env_ppu.sh forced a hard error here. But PPU build_ext runs with FLAGGEMS_KERNEL=OFF (setup.py cuda-branch default), so CMakeLists.txt:502 if(FLAGGEMS_KERNEL) never runs and find_package(FlagGems) is skipped -- the C++ FlagGems dispatch is not linked. FLAGGEMS_PYTHON=ON compiles the Python-path kernels without importing flag_gems at build time. Drop the cuda-style C++ asset discovery + the VENDOR_FLAGGEMS_DIR/LIB references (CMAKE_PREFIX_PATH, LD_LIBRARY_PATH, GITHUB_ENV), mirroring metax which has no such discovery either. Runtime flag_gems import for the FlagGems test step relies on the editable .pth on the container filesystem; step 4 may need follow-up if the isolated venv cannot import flag_gems.
The PPU image's pip.conf points at an internal mirror (aiext-pypi.mirrors.aliyuncs.com) that returns 503 through the pod proxy, so pip install of setuptools/cmake/patchelf/pytest/transformers fails. Add PIP_INDEX_URL (default Tsinghua, reachable through the proxy per curl probe) and pass --index-url to the build-dep and integration-dep pip installs. torch itself still comes from CPU_TORCH_INDEX_URL (download.pytorch.org), unchanged.
The Tsinghua mirror answers 200 to curl but returns 403 to pip for the same wheel URL -- its CDN rejects pip's User-Agent. PyPI proper (pypi.org/simple, files.pythonhosted.org) works through the pod proxy for pip (verified: pip install wheel succeeded). Switch PIP_INDEX_URL default to https://pypi.org/simple.
PPU image ships FlagGems as source only (no built liboperators.so / FlagGemsConfig.cmake under /workspace/FlagGems). setup.py's cuda branch (ACCELERATOR==cuda) does not pass -DFLAGGEMS_KERNEL=OFF -- it assumes a real cuda image has FlagGems C++ installed -- so the generic env pass-through at setup.py:459-468 left FLAGGEMS_KERNEL unset, and CMakeLists.txt:30 defaulted FLAGGEMS_KERNEL=ON, triggering find_package(FlagGems REQUIRED) at CMakeLists.txt:548 -> FATAL_ERROR 'Cannot find FlagGems'. Export FLAGGEMS_KERNEL=0 here so setup.py:459-468 emits -DFLAGGEMS_KERNEL=OFF, skipping CMakeLists.txt:502 if(FLAGGEMS_KERNEL). Mirrors set_env_metax.sh:41 which exports the same for the identical reason. FLAGGEMS_PYTHON stays at its default ON -- it compiles the python-path kernels without importing flag_gems at build time, and the FlagGems runtime test step (step 4) needs them. Also add FLAGGEMS_KERNEL to the GITHUB_ENV export list for cross-step persistence.
PPU core replacement swaps the vendor torch in at import time, so the
active torch reports the vendor version (2.10.0, no +cpu suffix) and
CUDA 13.0, which differs from the stock +cpu CUDA core.
The original asserts were copied from cuda.yml and expected version
2.10.0+cpu, leading to check failures after core replacement.
Manual server verification results:
- torch.__version__ = 2.10.0
- torch.version.cuda = 13.0
- torch.cuda.is_available() = True, device_count = 16
- torch_fl.flagos.is_available() = True, device_count = 16
Changes:
1. Update version assertion to use startswith('2.10.0') and exclude '+cpu'
2. Add torch.version.cuda == '13.0' as the definitive identifier for core replacement
(Stock CPU-only build has torch.version.cuda = None)
K8s + authZ intercepts --privileged on the PPU runner (authorization denied by plugin authZ). Mount AliXPU device nodes explicitly with --device instead: /dev/alixpu, /dev/alixpu_ctl, /dev/alixpu_ppu0-15 (16 compute nodes on the probed pod). GitHub Actions container.options passes these verbatim to docker create, so the list is static (no shell $(ls ...) expansion). shm 512g->4g; drop --cap-add=SYS_PTRACE/--ulimit. PPU-SDK lives on the runner host (not in image); mount read-only via container_volumes so set_env can discover it. ci_image switched to harbor.baai.ac.cn (reg.docker.alibaba-inc.com is Alibaba intranet-only, runner TCP 443 times out). set_env: PPU_SDK now scans candidates (PPU_SDK env -> PPU_HOME env -> /usr/local/PPU-SDK hyphen -> /usr/local/PPU_SDK underscore -> /opt/PPU-SDK) picking the first with CUDA_SDK/lib64/libcudart.so, mirroring set_env_ascend.sh's CANN toolkit scan so layout changes do not break. Drops the redundant libcudart re-check; device-node comment lists all three node classes.
The harbor CI image (= inference-xpu-pytorch) ships no FlagGems at all: find returns nothing site-packages has no flag_gems .pth/finder/dist-info, and import flag_gems raises ModuleNotFoundError. Dev pods get FlagGems via the shared CPFS PV (/workspace/FlagGems source + CPFS-backed site-packages .pth + installed deps); the CI pure container lacks all three. - ppu.yml container_volumes: bind-mount /workspace/FlagGems:ro (runner pod has the CPFS PV; docker create -v passes it through). Verified with --entrypoint /bin/ls -v that the source tree is visible in-container. - set_env_ppu.sh: synthesize _flag_gems_mounted_source.pth in venv site-packages pointing at /workspace/FlagGems/src. Python site adds the path to sys.path at startup so import flag_gems resolves to the mounted source with no editable install (honours the no-editable-acceptance rule). Verified via PYTHONPATH simulation: import resolves to source, walks the chain to sqlalchemy before failing -- .pth mechanism is effective and flag_gems __init__ does not query its own version (no PackageNotFoundError, no dist-info stub needed). - set_env_ppu.sh: venv pip install flag_gems core deps per /workspace/FlagGems/pyproject.toml (packaging>=26.0, PyYAML==6.0.1, sqlalchemy==2.0.48). numpy ships with the stock +cpu torch wheel, not re-pinned to avoid a numpy/torch version clash. triton/triton_kernels stay on copy (image has the vendor build, PyPI generic is wrong).
Aliyun DSW authZ plugin denies --privileged / --ipc=host / --security-opt / --cap-add; --device is allowed (DSW docs PPU example uses it). Multi-card NCCL uses --shm-size shared memory, not SysV IPC, so --ipc=host is not needed.
PPU CUPTI Activity API logs "bufferCompleted called with unknown buffer" and emits no device-side events, so the flagos trace has only ['Trace','cpu_op'] vs the torch-cuda baseline. All 7 tests in test_profiler_parity.py fail on the missing categories. Mirrors the Ascend pattern (same excluded step, same symptom). Re-add once the vendor CUPTI Activity path emits the expected device/runtime categories. No CPU fallback or faked pass.
xmhubj
self-requested a review
August 11, 2026 13:58
lvyufeng
approved these changes
Aug 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a configuration-driven CI pipeline for the T-Head PPU platform using the
existing CUDA-ABI boxing backend.
The pipeline builds and tests a self-contained
torch_flwheel in an isolatedenvironment with standard
torch 2.10.0+cpu. The vendor Torch Python package isused only as the source of the PPU native libraries, which are bundled under
torch_fl/lib_ppu.What's included
.github/configs/ppu.ymlpt-8g-cicd-pytorchrunner and validated PPU CI image.plugin rejects privileged containers and
--ipc=host..github/scripts/set_env_ppu.shPPU_SDKfrom the supported machine layouts.the vendor Torch Python package.
.github/workflows/build-wheel-ppu.yml.github/workflows/integration-test-ppu.ymlTesting
The PPU integration manifest covers:
Static validation completed:
set_env_ppu.shpassesbash -ngit diff --checkpassesKnown limitations
mmandbmmkernels because of a
num_ldmatrixesAPI mismatch, so those two FlagGemsdispatch tests are explicitly excluded. The vendor CUDA-boxing implementations
remain covered.
artifact, avoiding artifact-transfer timeouts on the self-hosted runner.