sidecar: typed streaming transport + detached exec, and a fail-safe SYSTEM>UPDATE

[colinmcardell]

why

with the current converged binary, updates and system commands such as restart are delegated to the sidecar. update fails completely and restart fails often... requiring manual reboot of the device.

update, restart, and shutdown all need privileged, detached execution. the existing update flow runs tar and update.sh as concurrent fire-and-forget shell calls, racing each other, then shows "done" before either has finished. when an update step stalls, the screen is stuck on its last message forever.

since the sidecar is norns' child process, any command that restarts norns also kills the sidecar and with it the command doing the restarting.

what

this PR reworks the sidecar around a typed, streaming transport with message framing (norns/sidecar_msg), line streaming (norns/sidecar_lines), and supervised execution (norns/sidecar_shell) as pure, separately tested pieces, and sidecar.cpp keeping the NNG glue. additionally, the sidecar gains the ability to launch detached system commands is added, which keep running after norns stops.

the SYSTEM>UPDATE flow is then reworked (using the sidecar transport improvements) into a fail-safe / fail-loud updater that replaces the norns binary and reboots only once the new binary proves launchable, rolling back otherwise. and importantly no longer stalls and fails spectacularly :).

sidecar's protocol can now launch work that outlives norns replacing/restarting itself, so we also got a much more reliable restart mechanism.

highlights

• sidecar - a stuck command can no longer tie up the sidecar (norns/sidecar_shell). commands run with limits. a command that goes quiet or floods output is cut off (each request sets its cap), stderr streams back alongside stdout, and stopping a command asks it to exit before forcing it.
• lua shell - lua's os.execute and io.popen are reimplemented over the sidecar, closing the last fork() opportunities in the JACK-holding converged binary process. os.execute keeps the exact lua contract. io.popen hands back a read-only handle (read/lines/close, see notes below). captured output is delivered through lua's print, so command output still lands in maiden.
• update - the SYSTEM>UPDATE menu guards against a failed or malformed releases.txt reply, which previously left the checking screen spinning forever.
• update - the SYSTEM>UPDATE menu shows the installer's current step on-screen as it runs, with a heartbeat that lets it tell a long quiet step from a stalled install.
• update - the new binary proves it launches using norns --check before the device reboots into it.
• update - the swap stages .new and keeps .old, so a failed check rolls back to the working install.
• update - a failed step writes status (failed:<step> is the format) used to present messaging on-screen, and returns to a clean pre-update state, rather than booting a broken build.
• update - m.releases_url override variable for local update testing (see test running the update locally below).

verification

• tested on hardware end-to-end
  • download -> checksum -> verified swap -> reboot into the new version
  • failure paths (corrupt / checksum / missing-script / bad-binary -> correct failed:<step>, no reboot, on-screen failure)
  • an installer that dies without reporting
  • manual update flow (wget + tar + run ./update.sh)
• new unit test coverage across the changes (c/c++ doctest, lua luaunit, and a bash-based dry-run test for update.sh).
• test-update.sh wired into a new test-update CI job.
• the test-update dry-run suite runs update.sh, aimed at a scratch tree through its NORNS_UPDATE_ROOT / NORNS_UPDATE_SUDO test-time overrides.

the rest of this pr is largely the cleanup that fell out of making this all possible:

• the system actions (shutdown, reset, update) are now a fixed allowlist at the C layer. each action name maps to a fixed command and a fixed transient systemd unit which is launched detached by the sidecar via sudo -n systemd-run. lua only ever supplies the action name, so no new shell execution capability is exposed to scripts.
• tidy: drops the dead passdone copy in lua/core/menu/system.lua (settings and wifi each own their own live copy).

test running the update locally

the SYSTEM>UPDATE menu page can now read its release manifest from an m.releases_url override variable, settable at runtime over the repl. otherwise it defaults to the monome releases.txt on github. with this override, the real download -> verify -> swap -> reboot path can be driven against a local http server for testing:

1. serve a manifest + bundle. releases.txt is lua that the menu load()s. it sets a global releases:

-- example releases.txt
releases = {}
releases.stable = {
  version = "999999", -- greater than the device's ~/version.txt so the menu shows it
  url = "http://<host>:8000/norns999999.tgz",
  sha = "http://<host>:8000/norns999999.sha256",
}
releases.beta = releases.stable -- the menu logic validates both channels

the bundle it points at is a release tarball which is a top dir named for the version, holding update.sh, the built norns/ tree (build/norns/norns + build/maiden-repl/maiden-repl), maiden/, the config/ systemd units, version.txt, and changelog.txt. update/pack.sh <version> tars that dir and writes the matching checksum:

update/pack.sh 999999    # tars ./999999/ -> norns999999.tgz + norns999999.sha256

the quickest valid bundle is a copy of an installed release tree with its version bumped. serve releases.txt, the .tgz, and the .sha256 from one dir over the LAN — python3 -m http.server 8000.

2. point the device at it over the repl (any restart resets the override to upstream):

require("core/menu/update").releases_url = "http://<host>:8000/releases.txt"

note the menu still pings github.com (and wants 400M free on disk) before it reads the override, so the device needs working internet even though the release itself is served from the LAN.

3. open SYSTEM>UPDATE and confirm. the success bundle downloads, verifies, swaps, and reboots into the new version. for the fail-safe paths you don't even need a valid bundle — a corrupt .tgz, a wrong .sha256, or a bundle missing update.sh each surfaces its failed:<step> on screen with no reboot.

notes

• bug: lua io.popen is partially regressed... "w" mode intentionally fails with a loud error, and reads hand back the finished command's whole output rather than streaming it. this should be fixed in a follow-up PR. it requires additional changes to the sidecar to support it, and i'm intentionally not putting that work here, to keep this already massive amount of change as contained as possible at this point. the system cmd and sidecar work from this PR is already an excellent prefactoring for the changes required to get io.popen fully working.
• restart is now dependable as a mechanism, but jackd itself can still hang, across repeated restarts and around ssh sessions. this PR does not fix this issue. two potential causes found while working (ssh'ing and restarting a lot):
  • bug: crone_cleanup() → jack_client_close only runs when the event loop exits on its own. restart delivers SIGTERM, and with no handler installed the process dies before reaching it, so every restart is an unclean JACK client death. the fix should be a SIGTERM handler that routes into the existing teardown path.
  • bug: closing the last ssh session makes systemd-logind purge the shared memory (RemoveIPC=yes), destroying jackd's shm while it runs. the next restart then fails until a power cycle. the fix here is likely a RemoveIPC=no logind addition. this belongs in norns-image for fresh flashes, but it should likely also ship through update.sh, so existing devices would pick it up on their next update without a reflash.
• async norns.system_cmd now has a ceiling upstream does not have. a command that stays silent for more than 4 minutes is killed and reported failed. the timeout bounds silence, not total runtime, so a command that keeps producing output can run as long as it needs. os.execute and io.popen pass no ceiling at all, keeping stdlib duration behavior for scripts that use them (this is on the script author).
• with this PR update.sh has indeed outgrown its bash beginnings at ~500 lines... with a rather silly (but kind of fun to write) hand-rolled test rig as an additional ~500 lines. this is fine, but the bash scripting seems to have reached its limit. it is fully functional and tested work, but further expansion should consider moving to python IMO.
• i'm really excited about this work. looking forward to review(s). thanks.

[colinmcardell]

force pushed lua lint fixes to keep things tidy.

[colinmcardell]

a few of the update screens which now stream each install stage as it runs and, on failure names the step that failed (including whether rollback worked if it had to rollback).

[tehn]

wow!! thank you! i have a couple hectic days but review and testing coming shortly. looking forward to a rollout

[tehn]

likely easier to do string concatenation here ie

screen.text(pre .. " " .. step)

and then use text_center instead of pre-calculating width with text_extents.

there was some concern with text_extents some years ago, i need to refresh my memory on the problem.

[tehn]

curious if --version would be a micro feature that could double as this check

[colinmcardell]

that's a good idea.

[tehn]

reviewed the code, looks good, skimmed the sidecar section.

will be able to test on hardware before the end of the week.

thank you for the thorough notes!