folk2 (parallel Folk evaluator)

[osnr]

This is not ready to merge yet -- it isn't 100% on par with folk1 in functionality, it has some random instability and leaks (and blinking), and I might want to make a few more breaking changes.

But it works well enough that it's probably time for it to get more outside testing (calibration works, tabletop editor works, printing works).

I'll be editing this description over time to document the changes; this is a first draft.

---

Major changes:

• Multithreaded evaluator: multiple threads (usually 1 active thread roughly = 1 CPU) can execute Folk/Tcl code at the same time. The database is now implemented in pure C with a mix of locks and atomic-retry and reference counting, so it's thread-safe and any thread can write/read it. When a statement is added, triggered When bodies (and destructors) are scheduled out to any available thread (CPU) using a work-stealing scheduler.
  • 
• No convergence & no 'ticks': there is no consistent world-state; for example, the renderer just displays whatever statements are in-database when it checks. This is probably the most controversial design choice in folk2. I'm curious if it'll hold up, but it seems necessary to me to hold off the system from getting arbitrarily slow as new programs get added. Thoughts/arguments welcome.
  • An unfortunate number of sort-of-arbitrary time-to-live keep 10ms, keep 8ms, etc times have been added to various places in the system. I want to think about ways to get rid of most/all of these (can we have receivers say they want to use either the current statement or the previous matching one? can we retain previous versions of statements by default?), or at least to have some more principled discipline of where they should go and why they should have that length
• Draws happen to per-page canvases (drawable-onto GPU textures) instead of to the screen
  • This allows for big reductions in statement traffic and draw calls, since many draws don't change at all from frame to frame with respect to the page
• Regions are gone; you'll be drawing to a per-page texture and probably using page-local meter coordinates (should fix Refactor all user-facing units to be in real-world (e.g. cm/m) units #202)
• Statements::findMatches (which was sort of a hacky escape valve) is being promoted to Query! and ForEach!, which are more first-class
• All imperative commands (Hold, Assert, Retract) now have ! suffix
  • If you have a setup.folk, you need to change any Asserts in it to be Assert!s (ideally, make one an alias/wrapper/shim for the other)
  • Hold is being replaced with Hold! -- you can give it either a code block or a Claim/Wish to hold, and it takes -on and -key options
• Global variables and procs will not work, because blocks of code may run in completely different CPUs/threads/interpreters. All sharing of code/data has to happen through statements (or, as an escape hatch, through unsafe C).
  • A new version of fn, a new C module system, and a new Tcl module system (to be documented) make it reasonably easy to share code.
• C is slightly different: invoke with [C] instead of [c create], and $cc compile returns a library object that you then call methods on instead of dumping the compiled functions into global namespace. (then this library object can be safely shared over statements and invoked from any thread)
• collected matches have been renamed to collected results (because Match is a specific datatype internally that I don't want to confuse)
• Notify: and Subscribe: have been added for point events (keypresses, print requests)

---

Known issues:

• camera slice works but is quite buggy. I think it's pushing the texture loading/unloading and the destructor implementation pretty hard. You'll get bugs where it causes other pages to blink in and out, where the camera slices get really laggy when projected, where it leaks MBs of memory and causes RAM exhaustion, and where the system crashes because the GPU-side texture storage is exhausted (presumably because we leaked camera slice textures?)
  • (update: exhaustion seems to be fixed in e897364)
  • (update: camera slice blinking and other-page blinking mostly fixed)
    • camera slices still laggier/choppier than I'd like!
  • also probably implies animation is buggy/broken
  • https://youtu.be/R6ZYPGkWd8w
• There are some random other memory leaks to dig into (not much worse than folk1, but still).
• Freeze/lock-in bug where page outlines sometimes lock in as ghosts when you leave the system on for a while (you'll see the stale detection/quad statements stuck in the database with a relatively lower statement id, also)
• (also see README.md)
• lots of functionality yet to port: high-level draw operations, points-at/other geometric operations, halo/title/other text operations, audio, terminal, web editor can't outline out of the box
• reset exposure setting post-calibration? it's probably underexposed in most cases since the user is setting it for a fully-illuminated projection area / sometimes system doesn't work post-calibration
  • calibrate also can leave the system in a weird state when canceled or completed

---

The design of folk2 is overall probably more complicated than folk1, but not enormously so: the core is now fixed C modules that are compiled ahead-of-time, since we barely touched them anyway, so boot times are faster and the whole thing is a single ./folk binary like a 'normal' PL runtime, instead of leaning on tclsh8.6.

It runs in one process that you can kill instead of a mess of forked processes.

There is one multithreaded Folk system instead of multiple federated ones on the machine, so you don't need to worry about Start process and statement sharing.

folk2 uses the Jim Tcl interpreter instead of Tcl 8.6, and it vendors Jim Tcl, so we can take more control over it and drop the external dependency.

---

Nice new things:

• It's noticeably much, much faster. Judging by calibration latency metric, this is about twice as fast in end-to-end latency as folk1, but that might be an underestimate.
  • Calibration feels much faster, much more 'live'
  • The editor also feels much faster, more like a usable/real editor
  • Calibration should be faster but also more accurate: partly because it's faster, partly because it now refines the individual homographies while calibrating (and uses better algorithm to compute the homographies) (fixes Apriltag corrupts while sliding the calibration board #183)
• Calibration comes with camera image preview (including AprilTag detect results) of each pose. (Fixes /calibrate: Tag detection feedback overlay on camera preview #197.)
• You can do blocking operations (like exec curl or sleep) in the middle of programs, or hit Folk web endpoints like / or /camera-frame from another computer, or write an infinite loop, without blocking the rest of the system. It should be harder for 'naive' programming to take down Folk, just like in any other preemptive OS.
  • The implementation of many Folk subsystems becomes a lot clearer: camera, AprilTag detector, GPU, etc are all just infinite loops that block on the device and then do some work when ready, no forking a process out and setting up communications channels, it's all just statements :-)
  • The process subsystem (Start process, On process) has been removed; you can do bare When { ... } instead of Start process { ... } if you want to not block the current execution context
• Because of the canvas design (you draw on canvas with real-world coordinates), text is now universally measured in meters and will scale properly to stay the same real-world size/shape as you lift/skew a page, because it's being set in real-world coordinates.
• Jim Tcl comes with some nice bonuses: $(...) shorthand for expressions, x(y) shorthand for dicts, better stack traces, some OO stuff, some lexical scope stuff.
• More powerful/C-compatible destructor system: statements can have a destructor which is guaranteed to be pinned until all downstream statements/When bodies are gone, so you can use the destructor to free memory or other unsafe operations like that and be guaranteed not to have a use-after-free (nice for heap-allocated camera images, GPU textures, other C objects)
• (serially) modifier for When that makes only one of that When run at a time (for stuff like camera frames where you don't want to blow up the CPU with too much work and it's OK to miss one)
• (keep) modifier for Claim/Hold that keeps it around for a few milliseconds past its expiration to reduce blinking and keep downstream statements alive while you wait for the successor statement to come in (in case the downstream statement would be some really expensive operation that you don't want to wastefully/unnecessarily redo)
• settle parameter for Collect (which is used to stabilize graphics -- don't draw until the display list has been stable for a few milliseconds)
• A whole Tracy profiler module has been added, tracy, which lets you low-overhead record zones, allocations, stack traces, events. Incredibly useful for performance monitoring, evaluator monitoring, blink debugging
  • Address Sanitizer support has also been added, although haven't tested recently
  • (these should also propagate into dynamically compiled C)

---

• Need to check if parallel event handling helps with Editor events pause or pile up when multiple people are typing #127 or Ctrl-S broken sometimes, just types 's' #163.
• Need to check on Animation program slows down framerate #126.
• Closes Main thread performance regression from #103 #107 and Terrible bug with collected matches sticking around #105 because those systems are rewritten.
• Closes Event statements #76 by adding Notify: and Subscribe:

---

Next steps:

• Test it on your system
• I want to test @smj-edison 's branch that should allow much more intense object sharing and save us a lot of serialize/deserialize time (I think most of the work the system does in folk2 right now is literally just making and unmaking strings every time a When block runs, and these objects are immutable, so we should be able to just share the Tcl data...)
• Cleanup unused cache.c and other stuff
• Port missing functionality and programs (animation, sound), fix web editor so it makes quads, port live-usb, update documentation
• More breaking changes: collected matches -> collected results, maybe nicer Hold syntax, maybe keep messing with the GPU/canvas/quad interfaces to make them cleaner, maybe rename On unmatch
• Do a more extensive writeup

[smj-edison]

Event statements would be amazing to ensure delivery (I've been thinking that asserts are for describing the universe as it currently is, so old state is no longer useful, but a message must be processed even if it is old). It's not necessary to get the current system working though.

[smj-edison]

Is this still being used?

[osnr]

nope! it might be a good idea to bring back, though -- it was just more broken / causing more problems than I thought and performance is still pretty good without it. it's sort of fulfilling the same role as your branch