Skip to content

Commit 447853c

Browse files
lauripiisangopaninakuffo
lauripiisang
and
opaninakuffo
authored
QVAC-21810 feat: self-contained per-platform tetherto-qvac-sdk wheels via GitHub release (#3754)
* feat: self-contained per-platform tetherto-qvac-sdk wheels via GitHub release - add build-sdk-python-fat-wheels.yml: workflow_dispatch (manual, attach to a chosen release tag) + workflow_call build of the fat per-platform wheels, uploaded to the sdk-v<version> release with `gh release upload --clobber` - wire attach-fat-wheels into publish-sdk.yml after publish-release, so a release-* publish attaches wheels automatically - build_wheel.py: prune addon prebuilds to the target platform (npm packages ship every platform, ~4.5 GB) and drive the wheel tag via hatch_build.py; add --plat-name and a lowered macOS floor default - hatch_build.py: stamp py3-none-<plat> when QVAC_WHEEL_PLAT is set; the thin PyPI build stays pure py3-none-any - README: lead with the turnkey `pip install -f <release>` install and document the upgrade / --force-reinstall behavior - test the _bundle resolver tier - ship darwin-arm64, linux-x64, linux-arm64, win32-x64 (darwin-x64 omitted: no in-repo runtime validation for 4 of 11 addons) * chore: prune addon test fixtures and docs from the bundled wheel - drop test/tests/__tests__/testAssets/fixtures/benchmark/docs/coverage/.github from each bundled addon; the npm packages over-publish them (asr-ggml alone ships a 28 MB test audio .raw) - darwin-arm64 wheel 146 MB -> 102 MB; native prebuilds and each addon's own weights/ are kept, so the worker + inference are unaffected * chore: clearer build_wheel CLI flags (--platform, --out-dir) - rename --plat-name -> --platform and --out -> --out-dir; add help text for --sdk-dir / --out-dir / --platform - update build-sdk-python-fat-wheels.yml matrix key + build step to match * fix: scope build_wheel non-runtime-dir prune to each package root - test/docs/fixtures/coverage/.github were dropped at every directory level, reaching into other packages' runtime internals (zod/src/.../tests, tinyld/dist/benchmark) - now dropped only when a direct child of a package root; .git and nested node_modules stay every-level (structural), prebuild host-prune stays scoped to the root-level prebuilds/ dir - validated against the production closure: root-level test/ dirs still pruned (~49 MB, asr-ggml/decoder-audio/...), nested runtime dirs now kept; wheel still 102 MB and the bundled worker still spawns * chore: address zizmor findings in the fat-wheels workflow - set persist-credentials: false on both checkout steps (the git credential is not needed; release uploads use GH_TOKEN) -- fixes artipacked - pass matrix.platform to the build step via env instead of interpolating it into the run block -- fixes template-injection * test: fat-wheel e2e workflow + install-mode-aware worker resolution - add on-pr-sdk-python-wheel-e2e.yml: build the fat wheel per platform on the self-hosted GPU runners, install it into a CLEAN venv, and run the full real-worker suite against the installed bundle -- proving the shipped artifact (incl. Vulkan/HIP .so backends loading from the pruned bundle). Triggers: workflow_dispatch (run on any branch pre-merge) + test-wheel-e2e PR label. Covers the GPU trio (darwin-arm64, linux-x64, win32-x64); linux-arm64 has no arm64 GPU runner and stays covered by the per-addon integration tests - _worker_env: resolve worker/bare from an installed wheel's _bundle when the imported package carries one, else the sibling @qvac/sdk checkout. No env toggle; an editable/source install has no _bundle, so the dev flow is unchanged - unify the real-worker tests onto _worker_env's WORKER_PATH so the whole suite honors the bundle; keep each file's SDK_DIR for the e2e fixtures it reads - neutralize the ambient bundle in the resolver-tier unit tests (autouse fixture) so they stay install-independent - validated locally: dev and installed-wheel modes both 153 passed (fast leg) * fix: address fat-wheel review comments for win32 bare and backfill Resolve bare.exe in Client() production paths, build fat wheels from github.sha so pre-tooling tags don't die on argparse, add zero-config Client() coverage, and align the e2e gating label name. * chore: black-format test_client.py for sdk-python CI --------- Co-authored-by: opaninakuffo <opaninakuffo@outlook.com> Co-authored-by: Opanin Akuffo <46673050+opaninakuffo@users.noreply.github.com>
1 parent 81c0251 commit 447853c

14 files changed

Lines changed: 727 additions & 68 deletions

‎.github/workflows/build-sdk-python-fat-wheels.yml‎

Lines changed: 200 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,200 @@
1+
# Build the self-contained per-platform `tetherto-qvac-sdk` wheels (thin client +
2+
# bundled Bare runtime + built @qvac/sdk worker + pruned native prebuilds) and
3+
# attach them to a GitHub release as assets, for a turnkey
4+
# pip install tetherto-qvac-sdk -f <release expanded_assets URL>
5+
# install that needs no separately-provisioned worker.
6+
#
7+
# These wheels are far too large for PyPI (each ~hundreds of MB of native
8+
# prebuilds); the thin `py3-none-any` wheel + sdist still go to PyPI via
9+
# publish-sdk.yml. Both triggers below share one build:
10+
# - workflow_dispatch: build + attach to a chosen existing release tag (manual
11+
# backfill / re-run / single-platform fix). Builds from `github.sha` (not the
12+
# tag) so pre-tooling tags don't die on argparse; SDK_VERSION must still
13+
# match the target tag.
14+
# - workflow_call: invoked by publish-sdk.yml after a release is created,
15+
# passing the published version.
16+
#
17+
# Only the platforms with genuine in-repo runtime validation ship: darwin-arm64,
18+
# linux-x64, win32-x64 (SDK e2e) and linux-arm64 (per-addon integration tests).
19+
# darwin-x64 is omitted until its uncovered addons gain an Intel-mac inference leg.
20+
name: Build sdk-python fat wheels
21+
22+
on:
23+
workflow_dispatch:
24+
inputs:
25+
tag:
26+
description: "Existing release tag to attach wheels to (e.g. sdk-v0.17.0)"
27+
required: true
28+
type: string
29+
platforms:
30+
description: "Subset to build: 'all' or comma-separated labels (darwin-arm64,linux-x64,linux-arm64,win32-x64)"
31+
required: false
32+
default: "all"
33+
type: string
34+
create_release:
35+
description: "Create a draft release for the tag if none exists"
36+
required: false
37+
default: false
38+
type: boolean
39+
workflow_call:
40+
inputs:
41+
version:
42+
description: "Published version; the release tag is sdk-v<version>"
43+
required: true
44+
type: string
45+
46+
permissions:
47+
contents: read
48+
49+
concurrency:
50+
group: ${{ github.workflow }}-${{ inputs.version || inputs.tag }}
51+
cancel-in-progress: false
52+
53+
jobs:
54+
setup:
55+
name: Resolve tag and matrix
56+
runs-on: ubuntu-latest
57+
permissions:
58+
contents: write # ensure-release (create_release path)
59+
outputs:
60+
tag: ${{ steps.compute.outputs.tag }}
61+
matrix: ${{ steps.compute.outputs.matrix }}
62+
steps:
63+
- name: Compute tag and platform matrix
64+
id: compute
65+
shell: bash
66+
env:
67+
IN_VERSION: ${{ inputs.version }}
68+
IN_TAG: ${{ inputs.tag }}
69+
IN_PLATFORMS: ${{ inputs.platforms }}
70+
run: |
71+
set -euo pipefail
72+
if [ -n "${IN_VERSION:-}" ]; then
73+
tag="sdk-v${IN_VERSION}"
74+
else
75+
tag="${IN_TAG:-}"
76+
fi
77+
if [ -z "$tag" ]; then
78+
echo "::error::no tag resolved (need workflow_call version or dispatch tag)"
79+
exit 1
80+
fi
81+
echo "tag=${tag}" >> "$GITHUB_OUTPUT"
82+
83+
platforms="${IN_PLATFORMS:-all}"
84+
[ -z "$platforms" ] && platforms="all"
85+
all='[
86+
{"label":"darwin-arm64","os":"macos-15","platform":"macosx_11_0_arm64"},
87+
{"label":"linux-x64","os":"ubuntu-22.04","platform":"manylinux_2_35_x86_64"},
88+
{"label":"linux-arm64","os":"ubuntu-22.04-arm","platform":"manylinux_2_35_aarch64"},
89+
{"label":"win32-x64","os":"windows-2025","platform":"win_amd64"}
90+
]'
91+
if [ "$platforms" = "all" ]; then
92+
matrix=$(echo "$all" | jq -c '{include: .}')
93+
else
94+
matrix=$(echo "$all" | jq -c --arg p "$platforms" \
95+
'{include: [ .[] | select(($p | split(",")) | index(.label)) ]}')
96+
fi
97+
echo "matrix=${matrix}" >> "$GITHUB_OUTPUT"
98+
echo "tag=${tag}"
99+
echo "matrix=${matrix}"
100+
101+
- name: Check out workflow revision
102+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # 6.0.2
103+
with:
104+
# Build from this workflow's revision; attach assets to the release tag.
105+
ref: ${{ github.sha }}
106+
fetch-depth: 0
107+
persist-credentials: false
108+
109+
- name: Verify version lockstep with the tag
110+
shell: bash
111+
env:
112+
TAG: ${{ steps.compute.outputs.tag }}
113+
run: |
114+
set -euo pipefail
115+
py_ver=$(sed -n 's/^SDK_VERSION = "\(.*\)"/\1/p' \
116+
packages/sdk-python/src/tetherto/qvac_sdk/_generated/sdk_version.py)
117+
tag_ver="${TAG#sdk-v}"
118+
if [ -z "$py_ver" ]; then
119+
echo "::error::could not parse SDK_VERSION at ${GITHUB_SHA}"
120+
exit 1
121+
fi
122+
if [ "$py_ver" != "$tag_ver" ]; then
123+
echo "::error::tetherto-qvac-sdk ${py_ver} at ${GITHUB_SHA} != tag version ${tag_ver}"
124+
exit 1
125+
fi
126+
echo "OK: tetherto-qvac-sdk ${py_ver} matches ${TAG}"
127+
128+
- name: Ensure release exists
129+
if: ${{ inputs.create_release == true }}
130+
shell: bash
131+
env:
132+
GH_TOKEN: ${{ github.token }}
133+
TAG: ${{ steps.compute.outputs.tag }}
134+
run: |
135+
set -euo pipefail
136+
if ! gh release view "$TAG" --repo "${{ github.repository }}" >/dev/null 2>&1; then
137+
gh release create "$TAG" --repo "${{ github.repository }}" \
138+
--draft --title "$TAG" --notes "Self-contained tetherto-qvac-sdk wheels."
139+
fi
140+
141+
build:
142+
name: fat wheel (${{ matrix.label }})
143+
needs: setup
144+
strategy:
145+
fail-fast: false
146+
matrix: ${{ fromJson(needs.setup.outputs.matrix) }}
147+
runs-on: ${{ matrix.os }}
148+
timeout-minutes: 90
149+
permissions:
150+
contents: write # gh release upload
151+
defaults:
152+
run:
153+
working-directory: packages/sdk-python
154+
steps:
155+
- name: Check out workflow revision
156+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # 6.0.2
157+
with:
158+
ref: ${{ github.sha }}
159+
persist-credentials: false
160+
161+
- uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # 6.2.0
162+
with:
163+
python-version: "3.10"
164+
- uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # 6.3.0
165+
with:
166+
node-version: 22
167+
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # 2.2.0
168+
with:
169+
bun-version: latest
170+
171+
- name: Install build tooling
172+
shell: bash
173+
run: python -m pip install --upgrade pip build
174+
175+
# All-public @qvac deps, no auth: bun install in ../sdk fetches the addon
176+
# packages (with their published native prebuilds) and builds the worker.
177+
- name: Build worker
178+
shell: bash
179+
run: python scripts/build_worker.py
180+
181+
- name: Build fat wheel
182+
shell: bash
183+
env:
184+
WHEEL_PLATFORM: ${{ matrix.platform }}
185+
run: python scripts/build_wheel.py --out-dir dist --platform "$WHEEL_PLATFORM"
186+
187+
- name: Upload wheel to release
188+
shell: bash
189+
env:
190+
GH_TOKEN: ${{ github.token }}
191+
TAG: ${{ needs.setup.outputs.tag }}
192+
run: gh release upload "$TAG" dist/*.whl --repo "${{ github.repository }}" --clobber
193+
194+
- name: Upload wheel as build artifact
195+
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # 7.0.0
196+
with:
197+
name: fat-wheel-${{ matrix.label }}
198+
path: packages/sdk-python/dist/*.whl
199+
if-no-files-found: error
200+
retention-days: 14

‎.github/workflows/on-pr-sdk-python-wheel-e2e.yml‎

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
# QVAC sdk-python fat-wheel e2e - PR (test-wheel-e2e)
2+
#
3+
# Proves the SHIPPED artifact: builds the self-contained fat wheel per platform,
4+
# installs it into a CLEAN venv (no @qvac/sdk checkout on the path, no editable
5+
# src), and runs the full real-worker pytest suite against the installed wheel's
6+
# bundled worker + pruned native prebuilds. This is what a `pip install -f
7+
# <release>` user actually runs, so it catches bundle regressions (missing
8+
# prebuild, lost exec bit, over-aggressive prune, resolver drift) that the
9+
# source-tree e2es (on-pr-sdk-python-e2e*.yml) can't see.
10+
#
11+
# Runs on the self-hosted GPU runners so the GPU inference paths (Vulkan/HIP
12+
# .so backends that ride in the linux/win bundles) are exercised, with the
13+
# shared model cache restored. Covers the three GPU-runner platforms; linux-arm64
14+
# ships a wheel too but has no arm64 GPU runner (its addons are covered by the
15+
# per-addon integration tests).
16+
#
17+
# Plain `pull_request` (public @qvac/* only, no secrets). The `test-wheel-e2e`
18+
# label is the maintainer approval to run PR code on the self-hosted runners;
19+
# `workflow_dispatch` lets a maintainer run it on any branch (e.g. to validate
20+
# this feature branch before merge).
21+
name: QVAC Tests (sdk-python) fat-wheel - PR
22+
23+
on:
24+
pull_request:
25+
# Only `labeled` (fresh approval pinned to the head SHA), not `synchronize`:
26+
# a later push must be re-approved rather than riding a stale label onto an
27+
# unreviewed SHA on the self-hosted runners.
28+
types: [labeled]
29+
paths:
30+
- "packages/sdk-python/**"
31+
- "packages/sdk/**"
32+
workflow_dispatch:
33+
34+
permissions:
35+
contents: read
36+
37+
concurrency:
38+
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
39+
cancel-in-progress: true
40+
41+
jobs:
42+
wheel-e2e:
43+
name: fat-wheel e2e (${{ matrix.label }})
44+
# SHA-bound approval: only a fresh `test-wheel-e2e` (labeled, tied to the head
45+
# SHA at approval time) or a manual dispatch runs the self-hosted GPU suite.
46+
if: >-
47+
github.event_name == 'workflow_dispatch' ||
48+
(github.event.action == 'labeled' && github.event.label.name == 'test-wheel-e2e')
49+
strategy:
50+
fail-fast: false
51+
matrix:
52+
include:
53+
- os: qvac-macos26-arm64-gpu
54+
label: darwin-arm64
55+
platform: macosx_11_0_arm64
56+
- os: qvac-ubuntu2204-x64-gpu
57+
label: linux-x64
58+
platform: manylinux_2_35_x86_64
59+
- os: qvac-win25-x64-gpu
60+
label: win32-x64
61+
platform: win_amd64
62+
runs-on: ${{ matrix.os }}
63+
timeout-minutes: 60
64+
defaults:
65+
run:
66+
shell: bash
67+
working-directory: packages/sdk-python
68+
env:
69+
# Point the worker's model storage at a cacheable, absolute path (same idea
70+
# as the SDK e2e). Applied via BareRpcTransport's __init_config.
71+
QVAC_CACHE_DIR: ${{ github.workspace }}/.qvac-cache
72+
steps:
73+
# Self-hosted runners persist the workspace between runs; wipe it.
74+
- name: Manual Workspace Cleanup
75+
if: runner.environment != 'github-hosted'
76+
working-directory: .
77+
run: rm -rf "$GITHUB_WORKSPACE" && mkdir -p "$GITHUB_WORKSPACE"
78+
79+
# Pin to the exact approved head SHA (empty on workflow_dispatch → default ref).
80+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # 6.0.2
81+
with:
82+
ref: ${{ github.event.pull_request.head.sha }}
83+
persist-credentials: false
84+
85+
# Use the runner's own Python via a venv. actions/setup-python doesn't work
86+
# on these self-hosted GPU runners (its toolcache path isn't writable); the
87+
# runners already provide Python. Fails loudly if < 3.10.
88+
- name: Set up Python (runner's system Python) + build tooling
89+
run: |
90+
set -e
91+
if command -v python3 >/dev/null 2>&1; then PY=python3; else PY=python; fi
92+
echo "System Python: $("$PY" --version) at $(command -v "$PY")"
93+
"$PY" -c 'import sys; assert sys.version_info[:2] >= (3, 10), "need Python >=3.10, got " + sys.version'
94+
"$PY" -m venv .venv
95+
if [ -x ".venv/bin/python" ]; then PYBIN="$(pwd)/.venv/bin/python"; else PYBIN="$(pwd)/.venv/Scripts/python"; fi
96+
echo "PYBIN=$PYBIN" >> "$GITHUB_ENV"
97+
"$PYBIN" -m pip install --upgrade pip build
98+
99+
# node/bun build the worker (they find the pre-cached tool on these runners).
100+
- uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # 6.3.0
101+
with:
102+
node-version: 22
103+
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # 2.2.0
104+
with:
105+
bun-version: latest
106+
107+
# Shared model cache, keyed like the SDK e2e so it invalidates in lockstep.
108+
- name: Restore models cache
109+
id: models-cache
110+
uses: actions/cache/restore@668228422ae6a00e4ad889ee87cd7109ec5666a7 # 5.0.4
111+
with:
112+
path: ${{ github.workspace }}/.qvac-cache
113+
key: qvac-py-models-${{ hashFiles('packages/sdk/models/registry/models.ts') }}
114+
restore-keys: |
115+
qvac-models-${{ hashFiles('packages/sdk/models/registry/models.ts') }}
116+
enableCrossOsArchive: true
117+
118+
- name: Build worker
119+
run: |
120+
"$PYBIN" scripts/build_worker.py
121+
122+
- name: Build fat wheel
123+
env:
124+
WHEEL_PLATFORM: ${{ matrix.platform }}
125+
run: |
126+
"$PYBIN" scripts/build_wheel.py --out-dir dist --platform "$WHEEL_PLATFORM"
127+
128+
# Clean venv, no editable src and no @qvac/sdk checkout on the path: install
129+
# ONLY the built fat wheel (+ test extras). The suite then resolves the
130+
# worker from the wheel's _bundle (see tests/_worker_env.py) -- truly the
131+
# shipped artifact, not the source tree.
132+
- name: Install the built wheel into a clean venv
133+
run: |
134+
set -e
135+
if command -v python3 >/dev/null 2>&1; then PY=python3; else PY=python; fi
136+
"$PY" -m venv .venv-wheel
137+
if [ -x ".venv-wheel/bin/python" ]; then TESTPY="$(pwd)/.venv-wheel/bin/python"; else TESTPY="$(pwd)/.venv-wheel/Scripts/python"; fi
138+
echo "TESTPY=$TESTPY" >> "$GITHUB_ENV"
139+
"$TESTPY" -m pip install --upgrade pip
140+
shopt -s nullglob
141+
wheels=(dist/*.whl)
142+
echo "installing ${wheels[0]}"
143+
"$TESTPY" -m pip install "${wheels[0]}[gen,dev,vla,notebook]"
144+
145+
# Guard: fail loudly if the suite would resolve anything other than the
146+
# installed wheel's bundle (e.g. a stray worker) -- otherwise a green run
147+
# could silently be testing the wrong thing.
148+
- name: Assert the bundle is what's under test
149+
run: |
150+
"$TESTPY" -c 'import sys; sys.path.insert(0, "tests"); import _worker_env as w, tetherto.qvac_sdk as pkg; assert "site-packages" in pkg.__file__, pkg.__file__; assert "_bundle" in w.WORKER_PATH and "_bundle" in w.BARE_BIN, (w.WORKER_PATH, w.BARE_BIN); assert w.WORKER_AVAILABLE; print("OK bundle under test:", w.WORKER_PATH)'
151+
152+
# Full suite (incl. heavy) against the installed wheel. `-o pythonpath=tests`
153+
# overrides pyproject's ["src","tests"] so `tetherto` resolves to the
154+
# installed wheel, not the source tree.
155+
- name: Test the installed wheel (full real-worker suite)
156+
run: |
157+
"$TESTPY" -m pytest tests/ -o pythonpath=tests -v
158+
159+
- name: Save models cache
160+
if: always() && steps.models-cache.outputs.cache-hit != 'true'
161+
uses: actions/cache/save@668228422ae6a00e4ad889ee87cd7109ec5666a7 # 5.0.4
162+
with:
163+
path: ${{ github.workspace }}/.qvac-cache
164+
key: qvac-py-models-${{ hashFiles('packages/sdk/models/registry/models.ts') }}
165+
enableCrossOsArchive: true

‎.github/workflows/publish-sdk.yml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -763,3 +763,16 @@ jobs:
763763
published_version: ${{ needs.publish-npm.outputs.published_version }}
764764
prev_sha: ${{ github.event.before }}
765765
workdir: "packages/sdk"
766+
767+
# Build the self-contained per-platform tetherto-qvac-sdk wheels and attach
768+
# them to the sdk-v<version> release the job above just created, for a turnkey
769+
# `pip install ... -f <release>` with no separate worker. Runs after the
770+
# release exists; the same workflow is dispatchable by hand for backfills.
771+
attach-fat-wheels:
772+
needs: [publish-release, publish-npm]
773+
if: ${{ needs.publish-npm.outputs.published_version != '' }}
774+
permissions:
775+
contents: write
776+
uses: ./.github/workflows/build-sdk-python-fat-wheels.yml
777+
with:
778+
version: ${{ needs.publish-npm.outputs.published_version }}

0 commit comments

Comments
 (0)