Add on-target (QEMU) Xtensa conformance test suite (#55)

Adds `tests/qemu/`, which runs the RFC 8251 decoder-conformance vectors
on an emulated ESP32-S3 under `qemu-system-xtensa`. This exercises the
Xtensa LX7 assembly built with xtensa-gcc, which the host suite
(`tests/conformance/`) never compiles.

**How it works**
- The 12 `.bit` vectors are embedded in the app image and decoded on the
emulated S3 at 1 and 2 channels.
- Raw int16 PCM streams out UART1 (to a host file via `-serial file:`)
one frame at a time, so device RAM stays flat. UART0 carries only
per-decode markers (`@@VEC` / `@@EnD … bytes=N status=ok`).
- On the host, `compare.sh` carves the PCM file by the `@@END` byte
counts and runs `opus_compare` against the references.

A raw binary side-channel (vs base64 over the console) keeps stray logs
out of the stream and runs ~2x faster. Semihosting would be faster
still, but ESP-IDF gates its semihosting VFS on a JTAG debugger that
QEMU doesn't present.

**Build variants**
- `esp32s3` — float decoder (S3 default); exercises `mathops_lx7.c`.
- `esp32s3_fixed` — layers `sdkconfig.fixed.defaults` to exercise the
fixed-point path and `fixed_lx7.h`. Same image, so `run-image.sh` /
`compare.sh` are unchanged.

**CI**
The QEMU job is slow and only matters when the on-target image can
change, so a `changes` job diffs the base and runs it only when
`lib/opus`, `patches/`, `cmake/staging.cmake`, `tests/qemu/`, or the
workflow changed. Wrapper-only changes skip it (the host suite covers
those); a skipped job reports `skipped`, which branch protection counts
as passing. To force a run, add the `ci-run-qemu` label, which re-runs
only the QEMU job. Detection uses plain `git diff` and fails safe if the
base commit is unreachable.

**Running**
```bash
tests/fetch_vectors.sh
tests/qemu/run-qemu.sh                       # float
tests/qemu/run-qemu.sh --env esp32s3_fixed   # fixed-point
```

Author: kahrendt

.github/workflows/ci.yml

@@ -6,6 +6,9 @@ on:
     branches: [main]
   pull_request:
     branches: [main]
+    # `labeled` lets the `ci-run-qemu` label start a run. Every job except changes/cross-qemu is
+    # gated on `github.event.action != 'labeled'`, so labeling re-runs only QEMU, not the whole suite.
+    types: [opened, synchronize, reopened, labeled]
 
 # Least privilege: every job here only reads the repo (checkout, build, test, cache). Jobs that
 # need more should add a narrower job-level permissions block rather than widening this default.
@@ -15,6 +18,8 @@ permissions:
 jobs:
   pre-commit:
     name: Pre-commit checks
+    # Skip on label-only events so labeling runs only the QEMU job (see pull_request types).
+    if: github.event.action != 'labeled'
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
@@ -27,6 +32,7 @@ jobs:
 
   lint:
     name: Static analysis
+    if: github.event.action != 'labeled'
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
@@ -43,6 +49,7 @@ jobs:
 
   build:
     name: Build
+    if: github.event.action != 'labeled'
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
@@ -57,6 +64,7 @@ jobs:
 
   test:
     name: Unit tests (${{ matrix.alloc_mode }})
+    if: github.event.action != 'labeled'
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
@@ -82,6 +90,7 @@ jobs:
 
   conformance:
     name: opus_compare conformance
+    if: github.event.action != 'labeled'
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
@@ -113,8 +122,123 @@ jobs:
       - name: Run conformance tests
         run: ctest --test-dir build -L conformance --output-on-failure
 
+  # Decide whether the on-target build changed. The QEMU run only adds value for Xtensa codegen (the
+  # patches/ asm and fixed-point paths); the src/ wrapper is already covered on host, so wrapper-only
+  # changes skip it. A job skipped via `if:` reports "skipped", which branch protection treats as passing.
+  changes:
+    name: Detect on-target changes
+    runs-on: ubuntu-latest
+    outputs:
+      target: ${{ steps.filter.outputs.target }}
+    steps:
+      # Full history so the diff base (PR base or previous push tip) is reachable.
+      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        with:
+          fetch-depth: 0
+
+      - name: Detect changes to on-target build inputs
+        id: filter
+        env:
+          BASE_SHA: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.event.before }}
+        run: |
+          # Run QEMU only when the on-target image could change. A lib/opus submodule bump appears as
+          # a change to the gitlink path. The src/ and include/ wrapper is omitted on purpose; host
+          # jobs already cover it. Fail safe: run QEMU if the base commit is unreachable (force-push,
+          # first push, shallow clone).
+          if ! git cat-file -e "${BASE_SHA}^{commit}" 2>/dev/null; then
+            echo "Base ${BASE_SHA} unreachable; running QEMU job to be safe."
+            echo "target=true" >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+          changed="$(git diff --name-only "${BASE_SHA}...HEAD")"
+          echo "Changed files:"
+          echo "${changed}"
+          if echo "${changed}" | grep -qE '^(lib/opus$|patches/|cmake/staging\.cmake$|tests/qemu/|\.github/workflows/ci\.yml$)'; then
+            echo "target=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "target=false" >> "$GITHUB_OUTPUT"
+          fi
+
+  cross-qemu:
+    name: Cross (Xtensa QEMU conformance, ${{ matrix.variant }})
+    needs: changes
+    # Skip on wrapper-only changes. Add the `ci-run-qemu` label to force a run.
+    if: >
+      needs.changes.outputs.target == 'true' ||
+      contains(github.event.pull_request.labels.*.name, 'ci-run-qemu')
+    runs-on: ubuntu-latest
+    # esp-idf version matches the PlatformIO framework (esp-idf v5.4.x) so the
+    # xtensa-gcc codegen is the same as the local run. Xtensa emulation is
+    # native-speed on the amd64 runner.
+    container: espressif/idf:v5.4.1
+    # Float is the ESP32-S3 default; fixed layers sdkconfig.fixed.defaults to
+    # exercise the fixed-point decode path and fixed_lx7.h. Separate runners so
+    # the two QEMU runs go in parallel.
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - variant: float
+            sdkconfig_defaults: sdkconfig.defaults
+          - variant: fixed
+            sdkconfig_defaults: sdkconfig.defaults;sdkconfig.fixed.defaults
+    steps:
+      - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        with:
+          submodules: false
+
+      - name: Initialize component submodules
+        run: |
+          git config --global --add safe.directory '*'
+          git submodule update --init --depth 1 lib/opus lib/micro-ogg-demuxer
+
+      # Same static-vector cache as the host conformance job. The vectors must be
+      # present before the firmware is built: embed_vectors.py bakes them into the
+      # app image at configure time.
+      - name: Cache test vectors
+        id: cache-vectors
+        uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        with:
+          path: tests/vectors
+          key: opus-testvectors-${{ hashFiles('tests/fetch_vectors.sh') }}
+
+      - name: Download test vectors
+        if: steps.cache-vectors.outputs.cache-hit != 'true'
+        run: tests/fetch_vectors.sh
+
+      - name: Install QEMU (xtensa)
+        shell: bash
+        run: |
+          . "$IDF_PATH/export.sh"
+          python "$IDF_PATH/tools/idf_tools.py" install qemu-xtensa
+
+      - name: Build conformance firmware (idf.py, esp32s3)
+        shell: bash
+        run: |
+          . "$IDF_PATH/export.sh"
+          idf.py -C tests/qemu \
+            -DSDKCONFIG_DEFAULTS="${{ matrix.sdkconfig_defaults }}" \
+            set-target esp32s3 build
+
+      - name: Decode vectors under QEMU and compare on host
+        shell: bash
+        run: |
+          . "$IDF_PATH/export.sh"
+          QEMU_XTENSA="$(command -v qemu-system-xtensa)"
+          [ -n "$QEMU_XTENSA" ] || { echo "qemu-system-xtensa not found on PATH after install"; exit 1; }
+          export QEMU_XTENSA
+          cc -O2 -o /tmp/opus_compare lib/opus/src/opus_compare.c -lm
+          tests/qemu/run-image.sh \
+            tests/qemu/build/bootloader/bootloader.bin \
+            tests/qemu/build/partition_table/partition-table.bin \
+            tests/qemu/build/qemu_opus_test.bin \
+            /tmp/qemu.log \
+            --timeout 900
+          tests/qemu/compare.sh /tmp/qemu.log /tmp/opus_compare tests/vectors /tmp
+
   platformio:
     name: PlatformIO Build
+    if: github.event.action != 'labeled'
     runs-on: ubuntu-latest
     strategy:
       matrix:

.gitignore

@@ -103,6 +103,20 @@ examples/*/build/
 examples/*/sdkconfig
 examples/*/sdkconfig.old
 
+# PlatformIO build output
+.pio/
+
+# QEMU conformance firmware build artifacts (tests/qemu)
+tests/qemu/build/
+tests/qemu/sdkconfig
+tests/qemu/sdkconfig.old
+# Generated per-env sdkconfig (resolved from sdkconfig.defaults at build time);
+# only sdkconfig.defaults and sdkconfig.fixed.defaults are tracked sources.
+tests/qemu/sdkconfig.esp32s3*
+# Embedded test vectors: generated from tests/vectors/*.bit at configure time
+# into the build dir (main/CMakeLists.txt). A copy in the source tree is stale.
+tests/qemu/vectors_data.c
+
 # Benchmark/test data
 *.tar.gz
 *.tar

CLAUDE.md

@@ -22,6 +22,7 @@ host_examples/           # Host platform tools (opus_to_wav)
 tests/                   # Host CTest suite (unit + opus_compare conformance)
   unit/                  # Wrapper/parser tests
   conformance/           # opus_compare validation against RFC test vectors
+  qemu/                  # On-target (ESP32-S3 QEMU) conformance: validates Xtensa asm
   support/               # Shared test helpers (in-memory Ogg muxing)
   tools/                 # Measurement tools (opt-in)
   fetch_vectors.sh       # Downloads the RFC 8251 test vectors

README.md

@@ -240,6 +240,10 @@ cmake --build tests/build
 ctest --test-dir tests/build --output-on-failure
 ```
 
+An on-target conformance suite in [`tests/qemu/`](tests/qemu/) runs the same RFC 8251 vectors on an
+emulated ESP32-S3 under `qemu-system-xtensa` to exercise the Xtensa LX7 assembly that the host suite
+cannot compile. See [tests/qemu/README.md](tests/qemu/README.md).
+
 ## License
 
 This project uses a dual-license structure:

tests/README.md

@@ -8,6 +8,11 @@ runs two kinds of tests through CTest:
 - **`conformance/`** validates our patched libopus by decoding the official Opus test vectors with
   our decoder and comparing each result against the reference decode with upstream `opus_compare`.
 
+A third suite, **`qemu/`**, runs the same conformance vectors on an emulated ESP32-S3 under
+`qemu-system-xtensa` to validate the Xtensa assembly optimizations that this host suite cannot
+compile. It is a standalone ESP-IDF firmware project, not a CTest target; see
+[`qemu/README.md`](qemu/README.md).
+
 ## Running
 
 ```bash
@@ -65,7 +70,7 @@ The host build is fixed-point and the decode still passes against the float-deri
 because `opus_compare` uses a perceptual threshold, not bit-exactness. The vectors are decoder
 vectors, so they validate the decode path (notably the always-on `celt_stack_alloc` patch). The
 Xtensa DSP optimizations build only for ESP32/ESP32-S3 targets, so they are not exercised by this
-host suite.
+host suite. The [`qemu/`](qemu/README.md) suite covers them on an emulated ESP32-S3.
 
 ## Tools
 

tests/qemu/CMakeLists.txt

@@ -0,0 +1,11 @@
+# Top-level ESP-IDF project for the on-target (QEMU) conformance firmware.
+#
+# Built with idf.py in CI (espressif/idf container) and with PlatformIO locally;
+# both point at main/ as the app component. The micro-opus component is pulled
+# in from the repo root via EXTRA_COMPONENT_DIRS.
+cmake_minimum_required(VERSION 3.16)
+
+list(APPEND EXTRA_COMPONENT_DIRS "${CMAKE_CURRENT_LIST_DIR}/../..")
+
+include($ENV{IDF_PATH}/tools/cmake/project.cmake)
+project(qemu_opus_test)

tests/qemu/README.md

@@ -0,0 +1,89 @@
+# On-target (QEMU) conformance test
+
+Runs the RFC 8251 decoder-conformance vectors on an emulated ESP32-S3 under
+`qemu-system-xtensa` to exercise the Xtensa LX7 assembly. The host suite
+(`tests/conformance/`) builds with host gcc/clang and never compiles the Xtensa
+paths. This firmware is built with xtensa-gcc for `esp32s3`, with
+`CONFIG_OPUS_ENABLE_XTENSA_OPTIMIZATIONS` on, so it covers the asm the host suite
+cannot:
+
+- `mathops_lx7.c`: `loopnez` float-to-int16 conversion (float build)
+- `silk/.../SigProc_FLP_lx7.h`: `round.s` / `float.s` SILK conversions (float build)
+- `fixed_lx7.h`: `mulsh` / `clamps` fixed-point multiplies (fixed build)
+- `pitch_lx7.h`: `dual_inner_prod` on the Xtensa MAC unit (both; the `FIXED_POINT`
+  path only in the fixed build)
+
+There are two build variants, since the ESP32-S3 selects float or fixed at
+compile time and the two paths use different assembly:
+
+- **`esp32s3`** (float): the S3 default (`CONFIG_OPUS_FLOATING_POINT=y`).
+- **`esp32s3_fixed`** (fixed-point): layers `sdkconfig.fixed.defaults` to set
+  `CONFIG_OPUS_FLOATING_POINT=n`. Same target and image, so `run-image.sh` and
+  `compare.sh` are unchanged.
+
+Run both to cover both decode paths. CI runs them as a matrix.
+
+## How it works
+
+`opus_compare` can't run on the device: it inflates both signals to float and
+builds whole-signal spectral buffers (about 30 MB for a 5 MB vector), and this
+build has no PSRAM. So the device decodes and the host compares.
+
+1. On the emulated S3 (`main/qemu_opus_test.cpp`): the 12 `.bit` inputs are baked
+   into the app image (`embed_vectors.py` writes `vectors_data.c` at configure
+   time; the 16 MB flash and a custom `partitions.csv` make room). Each vector is
+   decoded at 1 and 2 channels with `OpusPacketDecoder`, the same framing as
+   `tests/conformance/decode_vectors.cpp`. The raw interleaved int16 PCM goes out
+   UART1 one frame at a time, so RAM stays flat; the console (UART0) carries only
+   per-decode markers (`@@VEC`, `@@END name=... ch=... bytes=... status=ok`).
+2. On the host (`compare.sh`): UART1 is routed to a file (`-serial file:`).
+   `compare.sh` carves it into per-decode `.sw` files by the `@@END` byte counts,
+   then runs a host build of `opus_compare` against the `.dec` references, using
+   the same stereo-or-mono accept logic as the host suite.
+
+## Running locally
+
+Needs PlatformIO, Espressif's `qemu-system-xtensa`, and a C compiler. Install
+QEMU from <https://github.com/espressif/qemu/releases> (xtensa-softmmu) into
+`~/.espressif/tools/qemu-xtensa/`, or point `QEMU_XTENSA` at the binary. The
+build must support `-machine esp32s3` (the Espressif fork, not vanilla Homebrew
+QEMU). It runs natively on macOS, Apple Silicon and Intel; no Docker.
+
+```bash
+tests/fetch_vectors.sh                     # once: vectors must exist before the build
+tests/qemu/run-qemu.sh                     # float build, run under QEMU, compare
+tests/qemu/run-qemu.sh --env esp32s3_fixed # fixed-point build
+tests/qemu/run-qemu.sh --no-build --timeout 900
+```
+
+CI runs the same firmware via `idf.py` in the `espressif/idf` container; see the
+`cross-qemu` job in `.github/workflows/ci.yml`.
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `main/qemu_opus_test.cpp` | Firmware: decode each embedded vector, raw PCM out UART1, markers out UART0 |
+| `main/embed_vectors.py` | Writes `vectors_data.c` from `tests/vectors/*.bit` at configure time |
+| `main/qemu_vectors.h` | Declares the embedded-vector table |
+| `partitions.csv` | 12 MB factory app partition to hold the embedded `.bit` inputs |
+| `sdkconfig.defaults` | esp32s3, no PSRAM, 16 MB flash, perf optimization (float) |
+| `sdkconfig.fixed.defaults` | Fragment layered for the fixed-point variant (`CONFIG_OPUS_FLOATING_POINT=n`) |
+| `run-image.sh` | Assemble the flash image, run QEMU, capture UART0 to the log and UART1 to `<log>.pcm` |
+| `compare.sh` | Carve `<log>.pcm` by the `@@END` byte counts and run `opus_compare` against the references |
+| `run-qemu.sh` | Local driver: PlatformIO build, then `run-image.sh`, then `compare.sh` |
+
+## Transfer channel
+
+The PCM (about 95 MB of raw int16 across the 24 decodes) leaves the chip on its
+own UART (UART1, routed to a file with `-serial file:`); the console and markers
+stay on UART0. A dedicated binary channel keeps stray ESP-IDF log lines out of
+the stream, skips the 4/3 base64 inflation and its encode cost, and runs about
+2x faster than base64 over the console VFS. QEMU models both UARTs and routes
+each `-serial` to its own backend, so this behaves the same in CI and locally.
+
+Semihosting straight to a host file would be faster still (one host trap per
+buffer instead of one MMIO write per byte), but ESP-IDF gates its semihosting VFS
+on an attached JTAG debugger (`esp_cpu_dbgr_is_attached()`), which QEMU does not
+present, so it is unavailable here. The earlier base64-over-UART0 approach also
+worked but moved about 126 MB and was bound by the console UART VFS.