Some questions regarding suitability for embedded systems.

[c2akula]

Context

@slightknack I run a R&D studio based in the UK providing services for projects and clients involving Robotics, Automation, IoT and ML. Many of the projects that my company provides services for are hard realtime, ie., if a switch isn't toggled at the right moment, people could get hurt kind of realtime. We use C++ for everything embedded, Python and more recently Go for gui/application/user facing code. I'm aware that most people in the industry would suggest that I stick to C/C++/Rust because of performance, close to metal, control, safety etc. While these are nevertheless important, in our experience, these aren't even in the top 5 pain points. Instead it is lack of joy while programming, lack of support from the compilers, too much friction from our tools, requiring too much assistance from the developer etc., creature comfort features that obscure runtime behavior predictability etc. Now the immediate reply I can foresee someone reply is that we're probably doing something wrong or if we use X or do Y then we can workaround these issues etc. There have been innumerable moments where I have thought about giving up, and just changing careers. But I love robotics too much to be able to ever do that. In any case, I say all of this to point out that my frustrations have been making me feel that I might have to write a programming language and develop tools for what my studio needs that best serves our clients. In the process of research, I found Passerine and was struck by its simplicity. In fact, if I had to pick my favorite feature about it, I'd choose the macro system. It's just beautiful, and something I have never seen before. I'm sure other programming languages might have done it, but in my experience, I've never seen something this beautiful.

Preface

The aesthetic of a language is an important thing that many tend to brush aside as subjective and superficial. I'd counter that everyone would agree on certain qualities/characteristics of beautiful code. One of the reasons we picked Go was that it was small, in that we could hold the entire language and its specification in our head while working on solving problems. Having to think about which tool to use and micromanaging its details while implementing a solution is a lot of wasted cognitive energy which is better spent on the problem. Unfortunately, as much as we'd wish, we can't use Go on our embedded platforms with a few hundred kilobytes of memory, and we have no control or influence on the runtime behavior of programs written in Go or many times even in C++ (without going low-level, at which point we might as well use C). This is perhaps the most important aspect of the kind of work that we do - predictability of runtime characteristics of a program. I want to make it clear that these are some things I have been thinking about and wanted to ask you. In no way is this a criticism or a request of sorts to influence the design of your language. I would like to understand what and how you would approach an issue like the one I'm going to communicate if my company were to hypothetically use Passerine in a hard realtime context.

Issue

It's been my observation that from a hard realtime standpoint, the most critical aspect of programs is the repeatability, responsiveness and worst-case execution times of critical sections. Most of the development energy is spent optimizing for these aspects so that runtime behavior analysis is possible and quantifiable before the program is even executed. The reason why C is still used is, primarily, not because it is low-level and provides direct access to the metal, but that it is transparent. Let me explain. Features, whether provided at compiletime or runtime, encapsulate some behavior because the code that demonstrates the effects of a feature needs to run at some point. Any code that's not in the text of the program contributes uncertainty about its runtime characteristics. This is why such systems take long and are difficult to develop because they need to be thoroughly tested and verified against a set of requirements of runtime behavior. In addition, specific patterns of language usage or idioms have been developed and passed down as tradition because it's been observed that they don't introduce unnecessary variability. One extreme example when using C are programs where all variables are global, and functions take no parameters, return nothing and don't even use local variables. I have seen programs like this controlling actuation systems in entertainment rides. Whatever or however one may feel about this, it cannot be argued with that this code has been justified by the developers for some XYZ reason, and that it works as per the specifications. I am reminded of Dijkstra's "GOTO Considered Harmful" paper where he essentially says that our static text must reflect dynamic runtime behavior. Given that Passerine, and so does Rust and many new languages use compiletime to reason about the text to enable certain optimizations whether for safety or performance, how can the text help reason about the runtime behavior in a predicatable sense without ever executing it? Alternatively, how can we look at programs written in Passerine with its interesting memory management strategy or its clever compiletime analysis and provide an analysis of its runtime behavior that matches or guarantees this behavior once deployed? Is it possible? If no, can it done in a hypothetical Passerine, and how would that look like? I ask this because, Passerine is bloody beautiful (pardon my French), and I'm enamored by it, to say the least. I'd love to know your thoughts and pick your brain.

Sincerely,
CA

[dumblob]

I can't speak for the Passerine community, but I find you insight very interesting (I myself actually have a somewhat similar experience with the "dissatisfaction with the state of programming").

Anyway I'll play a devil's advocate here and say: look at few other languages like these:

1. XL - it has quite long history (began in early 1990) and seems to provide everything you seem to desire - but of course, the syntax might feel slightly unusual as it's a "conceptual language" (one describes & uses pure programming concepts of all kinds) and not a "constrained" language - because of that, it provides IMHO even cleaner and probably more efficient "macro" system (in the context of XL called "concept system").
   Btw. it provides compilation to C (unlike Passerine) and thus binary as well as compilation to bytecode and corresponding VM (like Passerine).
2. Lobster - it's a full-featured compiled language with full control flow analysis in compile time and thus is able to provide memory management determinism - but again, the syntax is a bit unusual (it feels close to Python)
3. Nim - it has the third most advanced macro system of a practical language I know of (first is XL, second Passerine) and offers a hard-realtime memory management etc. - but yet again, it's syntax is a bit unusual (some say it's similar to Python, but I wouldn't say so - it differs too much to my liking and is actually quite free, but most user don't know about it and follow the recommended style)
4. V - this is IMHO an attempt to "correct Go" and thus provides predictable memory management, Go syntax, modern features - the downside is it's immaturity as it stands (it didn't reach 1.0 yet)

Just my 2 cents.

[slightknack]

I appreciate these large and thought-provoking questions! I'd like to apologize for not answering sooner. I have been working on a response: I'd prefer to answer you fairly and comprehensively — essay to essay — so please excuse the time it's taking to formulate a response that fully addresses the nuances of what you've written. Thanks!

(I'll try to finish by tomorrow 🤞)
(Update: my computer restarted and I lost my draft)
(Update: 1.5k words and counting)

[c2akula]

I appreciate these large and thought-provoking questions! I'd like to apologize for not answering sooner. I have been working on a response: I'd prefer to answer you fairly and comprehensively — essay to essay — so please excuse the time it's taking to formulate a response that fully addresses the nuances of what you've written. Thanks!

(I'll try to finish by tomorrow 🤞)

Please, take your time. I look forward to it. :)

[slightknack]

I'll try to answer what you wrote to the best of my ability, but if I'm missing something or you'd like me to elaborate on a particular point, please let me know. I had written more, but after I updated my computer midway through writing, I had to revert to an earlier draft that I'd saved. I'm going to publish this as it stands instead of fretting over replicating exactly what I had. Please take my argument as a whole, rather than fixating on any individual part, however rough it may be.

But first, a disclaimer: building safe systems within hard real-time constraints is hard, and it's something I don't feel qualified to give an expert opinion on. Although I've worked on systems with such constraints, I don't know whether I'd trust myself to do so when human lives are at stake. With this in mind, take the following related to such systems with a grain of salt.

---

I think that writing programming languages that are fun and easy to use should be a priority of every language designer. With that said, it's surprising how many pain points exist in modern languages — as you've pointed out:

[Pain points we've encountered are] lack of joy while programming [as well as] too much friction from our tools.

I'm fairly certain that there are two barriers that need to be surpassed before a language becomes a joy to write on a daily basis. The first barrier is that of developing tools — whether they be editors, compilers, or otherwise — that encapsulate the abstract models used to manipulate programs. The second barrier is adoption and community-building: the social tools that fix our code when existing tooling can not.

Good tools — not necessarily great tools — make programming fun and enjoyable. The only issue is that to build tools, we need people familiar with that which they're building tools for, in other words, we need adoption. To drive adoption, however, we need the language itself to be enjoyable and friction-less, hence, we need tooling. There's a bit of a strange loop going on here. I guess this is why, despite all the technical innovation we've seen within the space of language design, the methods of writing and running code have largely remained the same: a text editor and a compiler.

New languages are in ecosystemic debt, and have to split their time between catching up with other languages, and developing the core language itself. This isn't to say that languages with cool tooling innovations don't exist: smalltalk and unison come to mind. However, languages that make improvements over traditional tooling have to make that new tooling a core feature of that language's identity.

There are two major solutions I see that can help break this loop: either we design a language that is fun to work in, and hope an ecosystem forms around it; or we design a solid ecosystem and hope developers find the tools we've built to be enjoyable. The latter method ensures at least some form of ecosystem will arise, but such a method relies on such a large up-front capital/time investment it's impractical for most. With Passerine, I've taken the former approach: I've tried to design a language I'd want to use and would enjoy working in, so that tools can be built around it to leverage its unique feature set. Time will tell.

In my opinion, workarounds are a glaring indicator that a feature needs to be reworked. Of course, there are usually more pressing issues at hand that get in the way of fixing a workaround, so it's not so cut-and-dry all the time. I guess if you're ever stuck looking at an old library and wondering how to improve it or write a better version of it, workarounds are a great place to figure out, and perhaps correct, conceptual errors in the model. It's important to point out that workarounds are different from hacks. Hacks take advantage of a system to reduce the amount of code it takes to do something; workarounds do the opposite. (Enough about workarounds!)

Anyway, let's talk about the first obstacle in this marathon: running Passerine on an embedded system:

Unfortunately, as much as we'd wish, we can't use Go on our embedded platforms with a few hundred kilobytes of memory

Passerine currently has two implementations: there's Passerine-Rust, which resides in this repository, and Passerine-D, which is written in D, by @ShawSumma. Passerine-Rust is interpreted, and I doubt it has less of a runtime-overhead than Go.

Passerine-D, on the other hand, is fully compiled, and targets a number of backends, including D, Wasm, and minivm (a small vISA that is ~1k lines of portable C), among others. The other day, Shaw demonstrated compiling and running some code on an Arduino (via either the D or Wasm backend, I forget). This leads me to believe that with some work, other embedded systems can be supported as well. Since Shaw has shown that Passerine can be compiled in a fairly intuitive manner, I'm currently working on expanding Passerine-Rust to support that functionality as well.

On to the next point:

Specific patterns of language usage or idioms have been developed and passed down as tradition because it's been observed that they don't introduce unnecessary variability.

Idioms and tradition are not a bad thing: they help keep style consistent and were usually introduced with good reason. However, it's easy to forget that before GOTO Considered Harmful, even patterns as simple as higher-level control-flow constructs weren't that mainstream. I can't help but think what other constructs exist that seem strictly necessary now, yet will be viewed as rudimentary at best years down the line.

To prevent old idioms from squashing innovation, I think transparency is key. This is in part why I've opted to add a simple-yet-powerful macro system to Passerine. It should be as easy to use (and understand) new patterns as it is to do so with old ones. With the addition of fine-grained control over effects, it should be possible to introduce new patterns that compose well with others.

I'm not sure what you call a feature that arises from the rules of a simple system, rather than being implemented, but I can give a few concrete examples. One such feature are update rules, first hypothesized by @IFcoltransG in #36. I won't go into too much depth there: in short, with the introduction of token-based macros, it's possible to implement a macro that updates some data in a fine grained manner, without the boilerplate of repeatedly destructuring and restructuring data.

Introducing new patterns requires some form of compile-time transformation: this may cause unnecessary variability. So the question becomes: in languages that are powerful enough to introduce new patterns, how do we ensure that patterns remain transparent so that they can be used in a way that doesn’t introduce additional variability?

‘Variability,’ as you’ve used the word, really refers to the ‘hidden’ code introduced when a pattern is used. I think you’ve summarized this nicely:

Any code that's not in the text of the program contributes uncertainty about its runtime characteristics.

This includes code in libraries, sure, but it also includes code generated by macros or code introduced by the compiler. So with respect to compile time reasoning:

[Passerine, Rust, and many other] new languages use compile-time to reason about the text to enable certain optimizations [...] how can the text help reason about the runtime behavior in a predictable sense without ever executing it?

There are two ways that I can see that we can ensure that introduced code does not affect the runtime effects of a system: the first way is to have a simple language with transparent semantics; the second way is to have fine-grained control over the effects of a system.

If you’ll allow me to discuss Rust for a second, it’s a great language for the latter point: Rust’s ownership model gives the programmer really tight control over memory, while still ensuring that that power is used properly (e.g. preventing double-frees). This makes it really easy to reason about the allocation of objects at any given point in time: just follow the lifetimes. A let x: u64 = ...; is exactly that, a 64-bit unsigned integer stored on the stack.

Rust, however, doesn’t do that good of a job on the former point: because it’s such a complex language, it can be hard to know what exact machine code some higher-level constructs, like closures or higher-order functions, will map to. Performance-wise, this isn’t much of a problem: the combination of zero-cost abstraction and insane optimization ensures that whatever the generated machine code is, it will be fairly minimal — it’s just opaque to the programmer in which way this minimality will be achieved.

Passerine is a simple language with transparent semantics: you could probably fit the core spec on a couple of pages. Macros, for instance, are ‘just’ hygienic templates — once you wrap your head around the core concept, it’s fairly transparent as to how any given macro works. The same can be said for lambdas and pattern matching, and so on and so forth.

Unlike Rust, however, Passerine is not a systems programming language. For this reason, it doesn’t give you fine-grained controls over a systems resources out of the box. What it does have (or rather will have), however, are algebraic effects and clear boundaries to manage the runtime characteristics of a program. What do I mean by this?

Algebraic effects are a generalized way to reason about the side effects of a program. They take a minute to wrap your head around, but once you do you’ll realize the implications are massive: since control flow and allocation can be described as side effects, we have the ability to determine how they affect state at each point in the program. With this in place, we can ensure that functions are, say, total, meaning they have no side effects and are guaranteed to terminate; we also know exactly where errors can be raised, and effects open up the possibility to recover from errors in a predictable manner.

Effects are, of course, a work in progress. They require a non-trivial reworking of the current compiler pipeline before we can start implementing them. (You can see my progress on this in the big-refactor branch.) Needless to say, I’m hoping that Passerine reaches the point where it’s not only concise to express programs, but how programs run.

I guess the consequence can be stated as follows. All languages introduce ‘text not in the program’ while compiling (this is how compilers work, haha). However, if a language has a small set of straightforward semantics, one of them being fine-grained control over the runtime, we can constrain the ‘hidden text’ to uphold certain invariants (or fail to compile). With this in place, such languages can leverage metaprogramming to introduce new idioms that preserve transparency while still being composable.

While we’re on the topic of transparency, we should probably discuss C. You mention:

C is still used is, primarily, not because it is low-level and provides direct access to the metal, but that it is transparent.

In C, it’s very transparent as to what text is included in a program: after all, #include is literally copy-and-paste. Although this is transparent in the simplest sense, the lack of namespacing and general indifference of the C preprocessor makes it difficult to reason about how #included code interacts. Of course, C is a mature language, and I understand that much tooling exists to ensure such conflicts are minimized. There’s a lot of nuance here, and it’s really not my place to make these sweeping generalizations about the transparency of a system.

Likewise, Rust has many people — much smarter than I am, who I look up to quite a bit — working on it 24/7. However, accidental complexity is introduced as a system grows: passerine is ~4k loc, whereas rustc is ~360k loc (not including llvm). You can make something quite complex in 4k loc, but never as complex as something several orders of magnitude larger with the same information density.

Now that we’ve discussed how systems can be conceptually transparent

The most critical aspect of programs is the repeatability, responsiveness and worst-case execution times of critical sections.

I agree with you, but I’d like to go out on a limb and state that this is the case even outside of embedded contexts. With that said:

[...] how can we look at programs written in Passerine [and determine their] runtime behavior [in a way that ensures the same] behavior once deployed? Is it possible? If not, [what would that look like in] hypothetical Passerine?

At this point, I’ve stated that Passerine is transparent, but haven’t shown why it is that way. In this next section I’d like to go over some more subtle design restrictions that lend themselves to maintaining transparency.

The biggest elephant in the room is probably the way data is constructed, transformed, and managed. What restrictions are needed so that data can be managed without the need of a garbage collector or complex compile-time systems?

In most modern languages, types can have cycles. Because C allows for arbitrary pointer manipulation, it’s trivial to construct cycles. Higher-level languages like Python allow the construction of cycles via circular references. This is not a bad thing: cycles make it easy to turn any graph into a data-type — and as just about every problem in computer science is a graph traversal problem, this intuitive construction lends itself very well to solving these problems. Allowing any graph, though — especially ones with cycles — makes it harder for the programmer and compiler to reason about how data is referenced, hence the need for manually memory management or garbage collectors.

In Passerine, however, data-types must be (co-)inductive or recursive. What this means is that, rather than allowing data to be an arbitrary graph, Passerine restricts all data to being a tree. At first, this might seem like an issue: how do we construct cycles and graphs in Passerine? The answer, of course, is another level of indirection.

Any directed graph can be represented as a tree with references to earlier nodes. While this may seem limiting, in combination Passerine’s powerful constructs for destructuring data and abstracting transformations, cyclic data-types can be implemented in a first-class manner. This shouldn’t come as a surprise. In most programming languages, building graphs through cycles of objects is generally frowned upon — instead, abstractions like adjacency lists or matrices are used. In this sense, we don’t rely on Passerine’s runtime to manage this graph of objects for us. For this reason, the implementation of commonly-used graphs can be optimized to represent the data they hold in a safe and compact manner.

Closures are also data; for this reason we must take care to ensure that no cycles can be constructed in closures. In practice, this means that the values captured by escaping closures must be immutable. This gives rise to local mutability in Passerine, meaning variables can only be mutated in the scope in which they’re defined. This may sound restricting at first, but in practice it means that it’s very easy to keep track of how and when data is transformed. I’m not going to go into detail here (we’re fast approaching 3k words) but if you’d like to know more, let me know!

In short, Passerine's memory management strategy isn't complex: rather, we restrict the language to remove the complexity traditionally associated with memory management. More generally, by carefully choosing restrictions on core language features — such as hygiene in macros or inductivity in types, — we can simplify the conceptual overhead of reasoning about such systems while still providing similar levels of expressive power. It’s all about finding that balance.

---

In no way is this a criticism or a request of sorts to influence the design of your language.

On the contrary, please do criticize Passerine! I can think of about a dozen rough corners in the language specification alone that needed to be smoothed over. Fresh pairs of eyes ensure that I stay on track — many have commented on the 'elegance' of Passerine — please hold me to that standard. It's a bit nerve-wracking, really. I feel like I've stumbled upon a diamond in the rough, but I'm too paralyzed to dig it out, so to speak, for fear of it falling apart in my hands. I'm sure any developer is familiar with this dilemma; the last thing to do is to sit on it.

I've never seen something this beautiful. [...] I'd love to know your thoughts and pick your brain.

Thank you, I hope what I’ve written answers the questions you’ve raised :)

You've raised some incredible points, and I'd like to chat with you some more if possible. If you're open sometime, I'm on Discord at slightknack#4221 or you can email me at hello@slightknack.dev so we can schedule something.

Cheers,
Isaac

(Holy cow I wrote way more than I thought I would)

[dumblob]

Thanks @slightknack for the peak into the future. It actually raised a bit my interest in Passerine 😉 and I found your blog post https://slightknack.dev/blog/next-steps/ (any plans to finish it?).

Some of the choices you mentioned seem rather limiting in practical apps (judging based on my experience with many different languages), but I'm an optimist and will instead follow-up on what you wrote above and list some resources to take inspiration from:

1. Kind - a proof language which is actually a readable practical language (not kidding!) - and btw. their community is very welcoming

2. with the new macro construct in Passerine, feel free to take some inspiration from what XL does to achieve practicality a it provides the greatest syntactical as well as semantical flexibility I've ever seen in a practical (non-research, non-toy) language (XL can model any effect handling, any error handling, any type, nearly anything... - and yes, there is the problem of undecidability but is being tackled with some careful choices)

3. there'll always be a set of parameters an app will require which Passerine (or any other higher-than-ASM language) will not cover by default (incl. parallelism & concurrency AOT type checking incl. memory management etc.).

   So what about not enforcing a specific use case by the code itself (can't return a malloc()ked memory to enclosing scope ...) and instead force the end-user to choose (e.g. by providing a compile-time argument --hard-real-time which would turn on all possible checks to ensure timing guarantees & max space guarantees etc.) with the additional icing on the cake by providing code syntax for the same functionality (e.g. #constraint memory_not_garbage_collected 1 or with some better syntax like [mem_not_gced]) useful for cases when the implemented algorithm in the given scope would make sense only with the constrained applied (e.g. because it'd handle memory management itself)?

   In other words - don't try to be smarter (by guessing the future instead giving users the final choice) than necessary 😉.

[c2akula]

@slightknack Thank you for the detailed reply.

Passerine-D, on the other hand, is fully compiled, and targets a number of backends, including D, Wasm, and minivm (a small vISA that is ~1k lines of portable C), among others. The other day, Shaw demonstrated compiling and running some code on an Arduino (via either the D or Wasm backend, I forget).

I'll be following this closely. It definitely sounds exciting at first glance.

the first way is to have a simple language with transparent semantics; the second way is to have fine-grained control over the effects of a system.

You've summarized here, succinctly, what I wanted to express with respect to where most languages suitable for systems programming fall short. The language alone, as you mention elsewhere, is part of a bigger piece where in the evolution of a language is driven partly by the hardware that it runs on, tying to the adoption issue - which direction a language goes appears to me to be determined by the ecosystem it targets or creates. In the case of Go, it created an ecosystem or rather found itself in one where distributed computing and web-development thrive.

I think transparency is key. This is in part why I've opted to add a simple-yet-powerful macro system to Passerine. It should be as easy to use (and understand) new patterns as it is to do so with old ones. With the addition of fine-grained control over effects, it should be possible to introduce new patterns that compose well with others.

My sense is that the control is going to be a challenge?

it’s such a complex language, it can be hard to know what exact machine code some higher-level constructs, like closures or higher-order functions, will map to. Performance-wise, this isn’t much of a problem: the combination of zero-cost abstraction and insane optimization ensures that whatever the generated machine code is, it will be fairly minimal — it’s just opaque to the programmer in which way this minimality will be achieved.

It is exactly against this situation that I find Passerine's simplicity intriguing.

The same can be said for lambdas and pattern matching, and so on and so forth.

I wonder, from the standpoint of implementation, what a lambda or pattern matching construct in Passerine is? In C, the closest thing to a lambda I know is a function pointer, C++ does it as a struct with an overloaded call operator, and Go's approach makes it feel like a function pointer but then moves the captured variables to the heap etc. In other words, what do these constructs translate to? Would all lambdas in all Passerine translate the identically or is it context dependent? If the latter, what controls are offered to the user to choose which option or make explicit?

Unlike Rust, however, Passerine is not a systems programming language. For this reason, it doesn’t give you fine-grained controls over a systems resources out of the box.

Given that Passerine has the ability to define custom syntax through syntax macros and the like, do you think it might be possible for a user to design specific syntactic constructs that can optionally be lowered by the VM to the underlying hardware for direct access (kind of like plugging in a cartridge) when the program goes through such sections (imagine sending bytes over I2C/CAN). What I'm saying here is definitely specific to an embedded context. I do, however, wonder if a Smalltalk like managed environment where a user could specify the particulars of the underlying hardware (how many SPI/I2C/UART/Timers does it have? how to initialize them?) is possible. I suppose I'm thinking of an abstract machine that the environment operates on but whose details are filled in by the user. For instance, for years we have used platforms from a variety of manufacturers (ST, NXP, TI, AD etc.) using multiple architectures (PIC, ARM, AVR etc.), and at some point we were just tired. Recently we've been making a push towards RISC-V by making our own custom platform specific to our studio's needs and experience that best suits our clients.

What it does have (or rather will have), however, are algebraic effects and clear boundaries to manage the runtime characteristics of a program. [...] control flow and allocation can be described as side effects, we have the ability to determine how they affect state at each point in the program. With this in place, we can ensure that functions are, say, total, meaning they have no side effects and are guaranteed to terminate; we also know exactly where errors can be raised, and effects open up the possibility to recover from errors in a predictable manner

At a glance it does seem interesting, but I wonder what implications it would have on runtime to provide the mentioned features? Alternatively, if possible at compile time and somehow embedded into the type system, how it would the aesthetic and complexity of the language? One of the things that appeals to me about Passerine, is the delicate dance between ALGOL and ML, and I wonder which way would this cause the ergonomics to swing, or simply its minimalism like in Go's "do less to enable more"?

On the note of error recovery, we pretty much see errors as inevitable values and matter-of-fact at the level we operate at. I wonder, tying in the abstract machine idea, if debugging could be improved? Often, the quick but dirty way we attempt debugging is to dump over UART or the like. On the other hand, we use expensive proprietary hardware debuggers that use their own proprietary protocols, some requiring setup or their own IDEs to use.

Needless to say, I’m hoping that Passerine reaches the point where it’s not only concise to express programs, but how programs run.

Will be looking forward to this with utmost attention.

Closures are also data; for this reason we must take care to ensure that no cycles can be constructed in closures. In practice, this means that the values captured by escaping closures must be immutable. This gives rise to local mutability in Passerine, meaning variables can only be mutated in the scope in which they’re defined. This may sound restricting at first, but in practice it means that it’s very easy to keep track of how and when data is transformed. I’m not going to go into detail here (we’re fast approaching 3k words) but if you’d like to know more, let me know!

I'd definitely love to know more. Please go into as much detail as you can. If I'm understanding what you say correctly, escaping closures don't need to move captured variables to the heap because if this was allowed, the closure would need to hold references or pointers to the captured variables since they could be mutated within the closure, and the parent that launched the closure could depend on it to mutate them. This would in turn bring the issues related to lifetimes, ownership and memory management in general. To avoid all this, if a closure is returned by some parent and this closure captures some values declared inside the parent, you move these values into the closure, as if they were declared inside the closure. Does this also mean no allocations? Because if I have a function that returned a closure like in the example below,

fn init_serial_stream(baud: u32) -> fn ([u8]) {
    // do some initialization of some default serial peripheral
    ser = hw_init()

    return [ser] (b: [u8]) {
        b.each ser.read()
    }
}

The idea in the above example is that init_serial_stream initializes some hardware peripheral and returns a closure that the user can repeatedly use as if they have access to an object that represents this peripheral to read len(b) bytes into b. In C/C++, such functionality would usually be encapsulated inside a struct which would be instantiated as a global variable and provided to the user for use, with appropriate protections against concurrent access from interrupts. Since this would be a global, there is only one allocation into the .data section of memory and that's it. If I were to assume that Passerine doesn't have global variables then moving ser into the closure would require a heap allocation once so that the closure when invoked repeatedly can refer to the object to get the bytes from (which will mutate the state of ser non-locally and any errors, if any, will happen at the peripheral level causing so how does algebraic effects help here?). How would Passerine handle an example as the one above, assuming that we can make it possible?

I will take you up on the email sometime :)

Kind Regards,
CA

[slightknack]

Just as a heads up, I will be out of town for the next week or so on vacation. I'll reply when I get back. :)

[slightknack]

I need to respond to both @dumblob and @c2akula's comments on this issue, sorry for not getting back to it sooner.