converged 2026 - #1893
Conversation
…ization (#1883) * Add nng to dev/ci container definition to support work on converged branch * single-process rewrite (squashed) fix compilation (sidecar) fix many compiler errors (safe-c, etc), some linker problem remains add the RELEASE cxx flags to norns/wscript, retire crone/wscript * fix jack_client for c++ * change CRLF endings to LF in various source files * matron: embed lua-cjson module as `json` global * ensure edge.sh does release builds (#1552) not sure how often this script is used but figured it should probably default to release builds * remove empty jack_cpu.h file (#1553) assuming the contents of this file were moved to jack_client.h at some point * various build improvements - fix/rename flag to enable gprof profiling - add flag to enable debugging/syms - fix duplicate/conflicting c++ standard flags - add help strings to some configure flags * norns: only fork sidecar performs a single fork for the sidecar process along with: - set the child process name for visibiliy in ps - set the thread name in child process for visibility in gdb - send child a SIGHUP if parent dies - disconnect child from parent stdin the primary motivation for this change is simplicity and to allow easier debugging. the parent process remains the one running crone/matron. * norns: move sidecar to nng * ws-wrapper: replace nanomsg with nng * maiden-repl: replace nanomsg with nng - check for all dependencies in wscript - fix SEGV when ^D is entered - collapse duplicate rx loop implementations - increase const'ness of various functions * Update readme-setup.md change to libnng-dev * audio: connect, disconnect, and inspect audio routing provides the ability to alter audio routing for jack clients either by specific port(s) or by blocks of ports given a routing table. routing tables for system, softcut, and supercollider simplify connection management for those components. modifications to audio routing are tracked and the default routing for all components is restored on script cleanup. the `audio_post_restore_default_routing` hook allows mods to tweak the routing in a consistent fashion * audio: simplify handling of system routing - removes explict handling of reverse io - adds method to temporary disable change tracking * sidecar: serialize command invocation requests - capture buffer termination (issue #1569) - moves async command capture logic from lua to c - ensures sidecar requests from _norns.execute and _norns.system_cmd are not interleaved when called from different threads - adds progressive capture buffer allocation in server up to maximum of 10MB * matron: set thread names for easier debugging giving explicitly created threads names makes it substantially easier to narrow in on a particular thread in gdb * Run clang-format on rebased converged branch * Add libatomic dependency for nng * fix: add converged tape and VU bindings and expanded meters - init vuPoll and tapePoll callbacks in MixerClient to prevent std:bad_function_call on boot. - expand VU meters from 4 to 12 channels across the crone -> matron -> lua bindings. - added visual rendering for the 8 new channels in the mix menu. * refactor: improve crone `Poll` thread lifecycle safety * fix: Initialize mix menu engine, monitor, cut, and tape parameters to prevent UI rendering crashes. * fix: Add tape pause, resume, and loop functions * fix: sidecar - resolves IPC startup race with a sync pipe and implement graceful cleanup * Adds nng git submodule * Update test wscirpt includes and .cc source mapping. * test: remove unnecessary c-linkage from c++ matron tests * test: migrate clock test helpers from c to c++ * matron: update atomic types in internal clock * build: uupdate cmake includes and nng library fallback * Adds command queue concurrency support and increased capacity + unit tests * feat: oracle asynchronous i/o for tape and softcut operations to prevent UI blocking for heavy disk i/o * tidy: moves the git submodules for concurrentqueue and readerwriterqueue from the ./crone/lib path to ./third-party * refactor: replace custom queue in sidecar with BlockingReaderWriterQueue * update: adding additional paths to clang-format.sh * fix: apply clang-format * fix: wscript include of nng * refactor: update Tape to defer `sf_open` to the tapes internal disk thread, removes previously introduced `io_thread` in oracle. * fix: resolves issue with self-kill in system restart * fix: update no longer calls stop on old norns-crone.service * maiden-repl: wip - removes nanomsg in favor of nng maiden-repl application logic still needs nng registration and dialer setup * refactor: replace legacy internal OSC comms with direct C calls and cleanup unused handlers * build: use system libnng-dev and remove third-party/nng submodule * fix: resolve boot instability from JACK auto-start conflict and cleans up debug logs * refactor(maiden-repl): force IPv4, thread-safe rendering, input echo, resize support, rename crone→sc * fix: use systemd Restart=always and self-terminate for reliable restart * fix(ws-wrapper): add retry loop for WebSocket listener binding adds a retry mechanism to gracefully handle momentarily held ports during restarts, preventing crash loops. * Fix `Grid.rotation` bug seen in #1888 (#1890) * When rotated 90/270 degrees led's aren't set appropraitely When a `Grid.rotate()` is called with 1 or 3, the correct quadrant isn't chosen. The issue lies in the function `dev_monome_quad_idx`. This doesn't take into account rotation state, thus when called with something like: ```c dev_monome_quad_idx(md->m, 2, 12) ``` It returns quadrant 2. I think we technically want quadrant 1. We can read the rotation state from the monome device, and use that to correctly calculate the quadrant with a slightly different formula that both works for 128 and 256 grids. * Fix missing variable * We need to pass in the md reference * Possible fix for rows/cols not being reset after a rotation * Don't query grid for rotation status * Use Grid.update_devices * Respond to tehn's comments * Linter * fix(sidecar): unlink stale IPC socket before bind --------- Co-authored-by: Michael Dewberry <712405+Dewb@users.noreply.github.com> Co-authored-by: emb <emb@catfact.net> Co-authored-by: catfact <catfact@users.noreply.github.com> Co-authored-by: Greg Wuller <greg@afofo.com> Co-authored-by: brian crabtree <tehn@monome.org> Co-authored-by: Levi Cole <lckennedy@gmail.com>
|
ok working again on a clean compile from a "stock" norns, intending to merge to main and mark this as 3.0.0 @colinmcardell i am not seeing libnng under third-party, so i added it, but i it seems the wscript doesn't pick it up. in the previous PR you mentioned it was set up to use either system installed nng or the third-party, but it seems the second option is failing for me still. perhaps i missed a step? symptom is: even if compiled on a norns that has the converged branch working (likely due to system install of nng) the "stock" norns fails as ws-wrapper is looking for a shared lib and not finding it |
|
side issue is that CI is failing because i don't think it knows about the new json lib |
|
Oh nice. Progress! Regarding libnng, we'll have to add it as a system dependency installed as a package through the update.sh. I backed out of the submodule approach because it wasn't working properly. |
|
sudo apt-get update && sudo apt-get -y install libnng1 libnng-dev This should work. |
|
thanks for the reminder, I lost track of that. no problem!
…-------- Original Message --------
On Tuesday, 05/26/26 at 16:56 Colin McArdell ***@***.***> wrote:
colinmcardell left a comment [(monome/norns#1893)](#1893 (comment))
sudo apt-get update && sudo apt-get -y install libnng1 libnng-dev
This should work.
—
Reply to this email directly, [view it on GitHub](#1893?email_source=notifications&email_token=AAB4I4DQIBPJNAM7YYK5NHT44YAJFA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINJUHA3TOMZQGIY2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-4548773021), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAB4I4F2POETR7JBHO6YIHT44YAJFAVCNFSM6AAAAACZNZAUZSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DKNBYG43TGMBSGE).
Triage notifications on the go with GitHub Mobile for [iOS](https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675) or [Android](https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub).
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
|
https://gist.github.com/colinmcardell/3b8982327c009c2c74f7e2a0dff22cab This has all the things 🙂 |
I can whip up a patch for the lua linter warnings. |
|
ok, the beta installer works! it required a couple extra steps as the apt sources are so old! the update script is pretty patchwork at this point. thinking going forward this might be a cusp where update happens to 3.0.0 and then future updates assume that the system has already been "fixed" to this point. i'm thinking this should be good to merge to main, but i'd suggest we do a beta trial for a week or so for any needed hotfixes prior to listing the update on the stable channel. also still pretty excited about 6 vu's on the mixer screen, wow |
Ah, easy enough: https://gist.github.com/colinmcardell/c559a740a41374f32a8c3e83553ad9c5 |
fix pushed! |
|
ok! we have a release available through SYSTEM > BETA now tested on hardware. |
|
Hello. I don't know if this is helpful, but on DIY norns shield running latest release, this beta update
Is there something I could help testing? |
|
@xmacex thanks for testing! we've done a lot of testing already but there's so much surface area that any additional "normal use" reports would be much appreciated. |
|
Hmm did cranes start running not continuously but by seconds, or did it always do so? I am looking at both the timestamp numbers on the top left, and the flight animation and I think the numbers were running continuously with one decimal and the birds were flapping faster... or am I imagining it? |
|
can't comment on that script directly, i have no idea |
|
Thinking aloud: something something do the softcut phase polls run at the speed set/requested by the Lua library, (in cranes, |
|
thank you all so much for the continuous effort! this is a huge update and the vu meters are such a joy to look at. |
|
thank you for flagging this. i'll get some test cases together and confirm. @colinmcardell are there any known limitations in the current implementation? |
|
hey all - new to posting feedback here, but I'm on the newest beta and enjoying it a lot - noticed that there seems to be an alternative font buried in the new OS? i'm getting a whole new system font when loading certain scripts, but it shows up inconsistently. it will load / reload the stock norns font when entering and exiting the PARAMS page, whether it's via my scrolling to the right via E1 or pressing F4 on my keypad. by "load / unload", what i mean is that the new font will appear for the first time while accessing PARAMS, carry across the other norns pages (tape, mixer etc), and will only reload to the stock font when i go to the currently running script screen (equivalent of F5 input), then going to the home screen (equivalent to F3 input). scrolling to the home screen from Params will keep the "new" font loaded across all screens. my question is.....is there now a way to load custom fonts onto norns?? is there a way to have finer-grain control over this? i love the look of the new font, but would love to toggle between the two consistently. thank you all for your lovely work on this platform - the new VU meters are so snappy and great. what an incredible upgrade! |
|
i believe you've discovered a bug-- there was no new intended font selection feature. do you have specific steps (and scripts?) to replicate this situation that we can try to reproduce? thanks for testing! |
|
I first encountered this bug using Seeker II - had Norns and Grid connected to Crow via USB, sending note information to JF. went to PARAMS and was met with this: 
I was able to revert to stock Norns font using F5 to go back to the script screen, then F3 to go to the Home Screen directly. When accessing Params directly via F4, this font reappears. |
|
i'm actually not sure which code would've changed in the converged branch that would've impacted font/menu/screen at all? i will attempt to replicate. |
|
I found, when using the hardware Norns inputs over my keypad, that scrolling across from Params via E1 would have the font carry across all screens, not just Params: entering into the script screen, exiting back to home via K1 brought back the stock Norns font. |
Yes happening on mlre. If I rise an octave ie. double the playback rate, also the UI update seems to run at double time. |
|
Apologies for the delay. Think I found a few issues. The phase poll issue is a missing entry in the new param dispatch map. The truncated waveforms seem to be because of a bug in the render callback. I'm still looking into the font issue... it does seem like a pretty odd one. I will try to open a PR this weekend after some testing on hardware. |
|
Well done spotting the cause. Don't hesitate to let me know @colinmcardell if I can help test something. Meta: is this the good way for us to give feedback on beta norns, here in a thread, or should we do it otherwise e.g. by rising issues, or some way which would be best for devs/maintainers? |
|
I've opened this PR #1896 which should address the softcut param dispatch map and render callback issues. |
|
fantastic, on my list for tomorrow and can post up a new beta. thanks tons! |
|
I've been digging deeper into the menu font issue... and generally bouncing back and forth reading through the pre<->post converged seams and changes (it touches a lot 🙂). I think there are a few things worth addressing. If it's easy enough to make a new beta and then later follow up with another, merging #1896 now and then following up with some additional fixes would be great. I will try to get some additional fixes in shortly.
I'm open to feedback on priority and thoughts in general of these issues. I've got them listed the way that I'm ranking them at the moment. In the meantime, let's keep testing things with normal usage, and let me know if you experience or notice any other issues. |
|
@colinmcardell thank you for looking into this! I'm building the new beta now. @ngwese might have some insight around the sidecar issue, as he authored that aspect (though, it's been awhile I admit) |
|
@colinmcardell I've just discovered a new bug related to But, thanks tons for the softcut fixes! There's a new beta posted up but now not a smooth way to update from the existing beta (it should work fine from stable). For those people on current beta wanting to upgrade, you'll need to do it manually: ssh/screen into norns via cli then: |
|
Installed norns260616 and the softcut polls and wave rendering are working as expected. thank you so much! please let me know if there is anything in specific I could test or help in any other way. @colinmcardell regarding the vu meters I have a minor suggestion. would it be possible to scale the monitor vu with the monitor level? otherwise it simply mirrors the input vu. |
This makes sense. Ha. Ok. There is a pretty natural boundary presenting itself between the norns application itself (now a single binary) and the system level stuff (a new binary hanging out to do system/environment related work). Honestly, the restart command is similar and has some pretty flaky stuff to get it to "work". It kind of all wants a service like the sidecar to delegate to. I need to think on this for a bit. 🤔
No problem! Excited things continue to progress. |
|
I have a repro and fix for the menu font issue. Will open a PR soon. |
|
I also did the dance above and installed the new beta. I did get this from the Restarted, and softcut scripts I tried earlier namely Cranes, nmRain and softcut study 9 query work fine and update in usual speed rather than what seemed like 1Hz earlier (although the flapping animation in Cranes looks kind of weird). |
I'm actually going to back out on this for the moment. I'm able to reproduce the issue. And I can patch this and that... but it's deeper than simply setting the font when the menu is displayed. It's an issue of screen "ownership", and not a regression as far as I can tell. A test given the same conditions with Seeker II on the latest non-converged release might be informative? Also, not a race condition because there is no threading, nothing to race. It's that a script can run a metro that sets the font (or any screen/Cairo surface param/state) and the menu display will be affected. The fix would be a refactoring to decouple how the screen and the surface is drawn to. The script should get its own surface. The menu pages, screensaver, and system overlays (e.g. battery warning), should likely get a surface to draw to as well. Transformations if imbalanced or not managed correctly, carry over to the menu as well (rotating the screen in the script, results in a rotated menu). Overall a rather complex thread to pull on at the moment, and I would rather table it for now and focus on landing the work that has changed. Deprioritizing the "menu font issue" for now. |
I can look into it. You are basically suggesting pre/post fader metering capabilities? |
you're correct about this. as you state, there are various ways in which a script can mess up the menu by calling screen functions outside of the suggested/required i apologize that this was likely not a regression and an unnecessary sidequest. i agree that it should be shelved. |
|
I'm working through solving the sidecar command/update/restart issue(s). The update portion is launch gating and I'm working through what might make that the most robust. There is a gap across the stack (Lua, C, script execution method) and I need to thread it all through. Asynchronous/streaming sidecar communication seems to be another pretty clear win to include with the v3 release. I'm bundling the work together. I will spend some time this weekend and have tested code to PR with solutions and fixes. 🤞 |
thanks for looking into it. what I meant was to have the monitor vu display the post fader value like all the other vu meters. currently it's mirroring the input signal. |
|
A couple of updates:
I think that's all the things. Will continue to update. Just wanted to make sure to keep the comms open. |
|
I can take a look at the menu screen context issue this week. Probably not a new issue, but perhaps the converged architecture makes losing the race condition more likely. |
Awesome. Sharing where I was at in the investigation. The mega version of the fix seems to be to build A possible lighter version might be to set all of the things that can be changed on the context (font, anti-aliasing, and transformations/rotations), within the function that toggles the menu into view, along with some method to reenforce in a script that calling screen methods outside redraw will produce unexpected results. The transform/rotate part requires exposing a C function to reset the context to identity. I'm not certain how we might reenforce the screen methods to keep them inside redraw, which is what lead me to the mega version. |
|
Closing the loop on the menu font issue: I think for a few different reasons it would be good to do as @colinmcardell describes and completely separate the menu context/buffer from the script context/buffer. Doing so would allow screen.peek to work correctly on the first frame after a menu toggle, would allow feedback-based effects to continue uninterrupted, and would allow things like ndi-mod or a SDL output to display the menu and script graphics on separate surfaces. But that's not necessary here. There is no race or bug in the menu system font selection, it's all fine. The observed behavior comes because the seeker_ii specifically overwrites the core menu's drawing routine. 
|
wow, thanks claude! i agree that while a split screen context would be rad, it's a bit outside the immediate scope. even further outside the scope, years ago i strongly considered implementing the entire menu/management system in C, just to define a clearer separation between scripts and the infrastructure. the introduction of the mod system made this pretty unrealistic without breakage. (sorry for the distraction, this is all a historical aside) in other news i'm available for further testing! |
Great sleuthing. All makes sense and lines up. Happy to contribute, once the dust settles with this line of work. I am getting close to a PR. I have been chipping away each evening. The surface area of the system commands/sidecar is spread across a good amount of norns and so I've had to go through and patch a few things here and there. The PR will be a bit hefty in the sidecar area... all great discoveries and improvements throughout. Stay tuned.
❤️ the distraction is great! This converged work is definitely creating momentum in a direction that lends itself nicely to this idea of pushing whatever core Lua code down a layer... also building C interfaces that separate away the specifics of the underlying Linux system. |
|
I am wrapping up the work on sidecar/system commands, and the update path. Curious for some feedback as I'm working through an issue with The reference leak I'll fix. The real question is the callback contract... keep the behavior where the callback doesn't run on empty output (compatible, but the chained callbacks break with no error), or have it always fire once with an empty string (predictable and consistent with every other async callback, but a visible change to the scripting contract). Update: I answered my own question. Pre-converged |
|
awesome, this is indeed the behavior that would preserve backwards compatibility. thank you! |
|
I have some availability this week for testing--- anything I can help with? Looking forward to putting out another beta build. |
|
Yes. Thanks for the nudge. I have been making progress. I feel good about a PR to test this week. Fair warning: it's a good amount of change to the sidecar and the update.sh. I've had to touch other system level functionality to gracefully delegate them through sidecar and get it out of the way of the converged binary which now holds the JACK thread and must not fork. It will need testing on the update process end to end, and testing to confirm that all of the system level function that has been touched is continuing to work correctly. If it's look and feeling alright, we can put together a beta release which will require a manual update. Then we can follow up with the mods fix and the VU meter tweak as another beta release that can test the update process. |
6 participants
No description provided.