AL Design Doc: Effects on the Edge
So far AL has mainly focused on its basic relational/object semantics within a transaction. I feel that AL’s intra-transactional semantics are now relatively well-defined and am happy with that design and, to some extent, its implementation. However, with every operation taking place within a transaction, the guarantees are quite strict, with no defined behaviour for OS-level operations. How do we open sockets? How do we open, read files, and close them? These are inherently unsafe, irrevocable operations. They are not really suited for a transaction. I feel that it is essential to solve this problem in order to write meaningful applications in AL.
At the current time, every OS-level operation essentially takes up its own command, and a ‘scheduler’ has been statically extended to execute upon this extended primitive command set, with the ‘scheduler’ watching for such commands over the table. For example, in the case of async sends, these commands simply trigger a new transaction that does a send. However, this design flies in the face of the original ethos of AL: Firstly and most importantly, a small fixed number of commands should be able to express a huge open-polymorphic surface. Secondly, this is slow, every write causes a scheduler call. Thirdly, it does not aggregate information.
I have identified two or three useful solutions that we can make use of to manage this problem, and will elaborate on them here and suggest how they might fit into AL.
The first realistic solution I came to involved the possibility of making reconciliation logic pluggable. Instead of having explicit commands for things like send_async, connect to a socket, etc, one solution would be to provide a point at which to extend the VM to understand how to aggregate and reconcile changes in certain objects. For example, if one wanted to have a method on object A called in a transaction whenever object B was changed, one would simply change a slot on object A such as :watch to the id of object B, and the scheduler would then be able to register object A as a watcher. If object B changes, then the transaction would then package up all of those changes at once, and asynchronously call the method on object A.
Whilst this solution works well for its actual intended job of keeping note of how to reconcile, and reconciling changes, it is awkward for external activity expressions. For example, having to change the slot does work to retain provenance over behaviour, but it is awkward as a user interaction. Having to change a slot is overkill for things such as asynchronous sends too. Furthermore, it’s not clear how to express such things as delays, repeats, receives. However I still believe this method has a lot of utility and can be used.
The second possible solution I came to was essentially the solution of Gemstone/S: to simply accept that OS-level operations are unsafe. Whilst I dislike this solution due to it ruining transactionality, it most certainly would be the fastest. However, it may also mean blocking certain transaction threads on OS-level operations, which is a bad pattern. However, we may come back to this solution later, as it is possible that we would add a fast ‘unsafe’ mode to AL in the future (like Rust does).
The third solution I came to is the solution of Urbit: To make use of transactional effect semantics (Iris Example | Urbit Docs). This is similar to solution 1, but grander in scope. Firstly, it requires explicit effect calls, rather than simply making objects and changing slots. Secondly, workflows are composed around transactions, allowing for the explicit composition of transactions, as opposed to having such behaviours be incidental.
I suspect that the correct direction for AL is to retain the possibility for some hybrid of all three of these solutions, but the third solution is the most flexible in my eyes. Both solutions 1 and 3 rely on the notion of a transactional outbox, but use it in slightly different ways. It may be that solutions 1 and 3 coalesce into one design at some point. Solution 3 provides a good framework for thinking about the composition of transactions, and reconciliation is nice for describing persistent relationships between committed state and external behaviour. Solution 2 could provide an escape hatch if the current approach becomes imperformant.
In conclusion, we should seek to generally retain a semantics that keeps transactions fully safe and that pushes effects to the edge.
P.S. Note that this is not related explicitly to natives or jetting. Natives are mostly runnable inside a transaction provided they are safe, and the same is true of jets.