Skip to content

amyboardweb: verify sketch writes to AMYboard, retry on corruption - #1220

Open
bwhitman wants to merge 7 commits into
mainfrom
claude/amyboard-sketch-write-errors-ba04cb
Open

bwhitman wants to merge 7 commits into
mainfrom
claude/amyboard-sketch-write-errors-ba04cb

Conversation

@bwhitman

@bwhitman bwhitman commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

The bug (field reports: Windows)

"Write to your AMYboard" intermittently pops a SyntaxError at a varying line (6, 11, 22 seen in the field), sometimes a UnicodeDecodeError — yet the file on disk later looks fine, and it only takes one user retry to "fix" it.

Root cause

The zT transfer has no integrity check. The failure cascade:

  1. One 188-byte sysex chunk is dropped in transit by the host MIDI stack.
  2. The board's transfer_stored_bytes never reaches the promised size, so transfer_flag stays AMY_TRANSFER_TYPE_FILE forever — there is no board-side timeout.
  3. While that flag is set, amy_parse_message routes every sysex-originated message to parse_transfer_message before command parsing — so the zP environment_transfer_done, liveness probes, and the retried zT header all get base64-decoded into the open file as binary garbage (b64_decode_ex silently stops at the first non-base64 char).
  4. The retry's chunks are valid base64, so the counter finally crosses the threshold mid-retry; the file closes containing original-minus-a-chunk + garbage + partial retry and runs → SyntaxError at wherever the chunk went missing (chunks are ~4–7 sketch lines apart → varying line numbers), or UnicodeDecodeError from the non-UTF-8 garbage.
  5. The user's next clean Write overwrites everything — which is why the file "looks fine" by the time anyone inspects it.

Bench findings (Windows 11 + AMYboard on USB, python-rtmidi via the winmm compat layer)

  • Outbound (host→board) is robust: paced, burst-4, and fully unpaced transfers up to 20KB / 107 chunks all landed byte-perfect (verified by CRC32 computed on the board). The board's 8-slot sysex ring keeps up even with zero pacing.
  • Inbound (board→host) sysex above ~1KB is silently dropped by fixed-size sysex input buffers. The board's zD dump frames (~1030 bytes) are exactly in the kill zone; small frames (ACKs, error reports, CRC replies) always arrive.
  • Chromium (Edge) on the new Windows MIDI Services stack (build 26200, SWD\MMDEVAPI\MIDII_* devices) enumerates zero Web MIDI ports — a separate compat gap worth its own FAQ entry.
  • macOS/CoreMIDI: clean in both directions at every size and pacing tested.

The fix (web-only — works with firmware users already have)

After sending the chunks, verify with a CRC32 computed on the board (zP exec of binascii.crc32 over the written file) reported back in a ~20-byte 'H' sysex frame, compared against the same CRC computed in JS:

  • Mismatch → resend, up to 3 attempts. A best-effort zD read-back locates the first corrupted byte for the [verify] console diagnostic (on hosts that drop the ~1KB zD frames it just times out quietly — the CRC verdict stands regardless).
  • CRC report timeout → the wedged-transfer signature (the request itself was eaten as transfer data). Recovery: zB reboot to clear the stuck transfer, then resend. On Windows Chrome — where WebMIDI can't survive a zB in-document — reuse the existing stash-and-reload flow, which resumes and re-verifies in _upload_sketch_post_bootloader; reloads capped at 2.
  • environment_transfer_done only fires on a verified write; otherwise the user gets a clear alert instead of a corrupt sketch running.

Also adds /miditest.html — a self-contained field diagnostic (paced write of a scratch file, board-CRC verify, zD read-back measurement) with copyable JSON results. Hosted because Firefox only grants Web MIDI via its site-permission add-on on public https origins.

Testing

  • Stubbed-transport tests of the real code paths (synthesized sysex frames through _process_complete_sysex): happy path, corrupted-send (CRC mismatch caught, first-difference diagnostic at the exact corrupted char/line, verified on resend), wedged non-Windows (timeout → zB reboot → verified), wedged Windows Chrome (reload-needed), full Write flow (transfer_done only after verify).
  • JS CRC32 parity-tested against MicroPython/CPython binascii.crc32.
  • Hardware protocol driver (zt_pacing_test.py) run against a real AMYboard from macOS and Windows 11 — results above.

Follow-ups (not in this PR)

Firmware/amy hardening: board-side transfer inactivity timeout (abort + delete partial + error sysex), chunk sequence numbers/CRC in zT, file size/CRC in the X error payload, and smaller zD dump frames (≤1KB incl. framing) so reads survive winmm-buffer-limited hosts.

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown

🔌 AMYboard PR preview

Editor + flasher: https://amyboard-pr-1220.vercel.app/editor/

This preview bundles this PR's firmware — its flasher only flashes this build (not the release). Rebuilt on every push; removed when the PR closes.

The hardware CI has been kicked off and should return within a few minutes, stand by!

bwhitman and others added 7 commits July 27, 2026 09:59
Field reports (Windows Chrome): "Write to your AMYboard" intermittently
produces SyntaxError popups at varying lines (6/11/22) and sometimes
UnicodeDecodeError, while the file later looks fine. Root cause: the zT
transfer has no integrity check. If one 188-byte sysex chunk is dropped,
the board's byte counter never reaches the promised size, the transfer
stays open, and every later sysex command (the zP restart, liveness
probes, a retried zT header) gets base64-decoded INTO the open file as
binary garbage until the counter finally crosses the threshold — then
the corrupted sketch runs.

Fix (web-only, works with existing firmware): after sending the chunks,
read the file straight back with zD and compare byte-for-byte.
- mismatch -> resend (up to 3 attempts), with a [verify] console
  diagnostic pinpointing the first differing char/line
- read-back timeout (the wedged-transfer signature: the zD request
  itself was eaten as transfer data) -> zB reboot to clear the stuck
  transfer, then resend; on Windows Chrome, where WebMIDI can't survive
  a zB in-document, reuse the existing stash-and-reload flow (capped at
  2 reloads), which resumes and re-verifies in
  _upload_sketch_post_bootloader
- only fire environment_transfer_done on a verified write; otherwise
  surface a clear alert instead of running a corrupt sketch

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Self-contained WebMIDI test for field-debugging transfer corruption:
paced zT write of a scratch file (never touches sketch.py), board-side
CRC32 verify over a small sysex frame, then a zD read-back that measures
whether the browser/OS delivers the ~1KB inbound dump frames intact.
Results render as copyable JSON. Needed on a public https origin because
Firefox only offers its Web MIDI site-permission add-on there — local
pages can't use MIDI in Firefox at all.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bench testing on Windows 11 (AMYboard on USB, python-rtmidi through the
winmm compat layer) showed the outbound zT write path is robust — even
107 chunks sent with no ACK pacing land byte-perfect — but INBOUND sysex
frames above ~1KB are silently dropped by fixed-size sysex input
buffers. The zD dump's ~1030-byte frames are exactly in the kill zone,
so a read-back-based verify would falsely report a wedged transfer on
such hosts and trigger pointless recovery reboots.

Switch the verify primitive to a CRC32 computed on the board (zP exec of
binascii.crc32 over the written file) reported back in a ~20-byte 'H'
sysex frame, compared against the same CRC computed in JS. Small frames
survive every MIDI stack we tested. The zD read-back remains as a
best-effort diagnostic to locate the first corrupted byte after a CRC
mismatch; its failure cannot affect the verdict.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
From a live debugging session on a refurb Windows 11 box: leftover
per-device Drivers\midi registry registrations blocked the new Windows
MIDI Services WinMM handoff, making the AMYboard visible to DAWs but
invisible to every browser. Documents the mididiag/midifixreg check and
fix, the Firefox site-permission add-on flow (including its silent
auto-deny when no devices are visible), and corrects the stale claim
that Firefox doesn't support the editor (Web MIDI works; only the
WebSerial flasher is Chrome/Edge-only).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…Chrome

Field-rig testing (Windows 11 + Firefox 153 + AMYboard) showed Firefox
has the same can't-survive-USB-reboot failure as Chrome, with better
camouflage: after zB the stale MIDIOutput keeps reporting
state=connected/connection=open with the pre-reboot port id while every
send goes nowhere, so wait_for_board_ready pings a zombie until timeout.
This broke the site Reset (and would have broken the write-verify wedge
recovery) on Windows Firefox.

Rename _IS_WINDOWS_CHROME to _IS_WINDOWS_BROWSER and gate every
zB+reload flow on Windows regardless of browser. Also drop index.html's
dead local copy of the old flag.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wire-monitor testing on the Windows rig (Firefox 153, parallel native
MIDI client watching the board) proved the post-reload page's sysex
sends never reach the board even though the freshly-opened ports report
state=connected/connection=open: the board's MIDI endpoint keeps the
same id across the zB USB reboot, so a MIDIAccess session opened while
the re-enumeration is still settling binds the dead pre-reboot instance
and never rebinds — raw-port close()/open() cycling doesn't help.

Two-part fix:
1. Lengthen the zB pre-reload wait from 4s to 9s in all three
   stash-and-reload flows, so the reloaded page enumerates after the
   USB re-enumeration has settled and binds the live instance.
2. wait_for_board_ready: on Windows, at 1/3 of the timeout with no
   replies, do a full WebMidi teardown + re-enable (new MIDIAccess) and
   re-run port setup — the only recovery that rebinds the live endpoint
   instance. The raw-port cycle at 1/2 remains as the Chrome fallback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The remaining Windows Firefox failure: Firefox holds MIDI sessions in
process-wide state keyed by the port id, and the AMYboard's id is stable
across the zB USB reboot. A session left open when the device drops
becomes a zombie that blocks every subsequent document in that tab from
opening the re-enumerated port — open() fails with "operation is not
supported by the underlying object", sends silently vanish, and neither
a page reload nor a brand-new MIDIAccess (WebMidi.disable/enable)
escapes it, since the state is per-process. Verified with a parallel
native wire monitor: the page's sysex never reached the board while a
native client pinged it fine.

Fix: _zb_and_release_midi() sends zB, waits 500ms for the sysex to
leave, then WebMidi.disable() — releasing every port while the device
is still alive, so no zombie is recorded and the post-reload document
binds the fresh instance cleanly. Used by all three Windows zB+reload
flows. Chrome doesn't need the release but is unharmed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@bwhitman
bwhitman force-pushed the claude/amyboard-sketch-write-errors-ba04cb branch from eae9cd2 to d79cec4 Compare July 27, 2026 16:59
@github-actions

Copy link
Copy Markdown

🎛️ HW CI (physical bench)

AMYboard (USB-MIDI + AMY zP → audio; built-in tones + AMYboard World sketches acid/house/woodpiano over the SysEx control API): ✅ PASS — flashed this PR’s firmware; all checks matched the references.

Tulip (TULIP4_R11; serial-REPL audio + WiFi screenshot): ✅ PASS — flashed this PR’s firmware; all checks matched the references.

⬇️ Artifacts: recordings · screenshot · serial logs · run logs

Self-hosted bench. Audio spectral-compared to ref/hwci_basic.wav, the AMYboard World sketch refs (ref/{acid_generator,house_generator,woodpiano}.wav) + ref/tulip_basic.wav; Tulip screenshot pixel-compared to ref/tulip_screenshot.png. Both analog outs share one capture card, so the tests run sequentially.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant