I propose a microVM-based architecture as the execution environment and message passing system between processes/services/engines in a distributed operating system.
MicroVMs
MicroVMs are language-agnostic single-application VMs
that may either run a unikernel, a.k.a. library operating system,
or a stripped down (Linux-based) system with a single application.
Main advantage of this approach is cross-language composability,
secure isolation between components, reduced Trusted Computing Base (TCB),
and straightforward integration of various already existing components,
such as (P2P) transport protocols, local data stores, local secure key stores,
and it also serves as the message passing mechanism between local services and applications.
The message passing interface between services running in microVMs is based on IPv6/UDP,
where each microVM has a cryptographically generated IPv6 address.
A suitable application layer protocol would be CoAP, which is a simple protocol
that supports both send-only and request-response patterns, as well as multicast.
Other protocols should work as well, including custom application-specific protocols,
as long as all communicating parties support it.
A local router running in a dedicated microVM is responsible for routing messages
between local microVMs and various transport protocols (each running in their own microVM) responsible for host-to-host communication, by tunneling IPv6 packets inside the transport protocols.
Local services may communicate with each other directly via a local bridge interface using both unicast and multicast.
For direct local communication any protocol may be used, however for remote connections UDP is preferred as it allows supporting delay-tolerant networks with high-latency transports.
Message queues / mailboxes may be used by services for both unicast and multicast messaging,
and may run either in the same or separate microVMs as the services they serve.
A Decentralized Name System is used for address resolution based on signed advertisements,
which is mapped locally to a plain DNS service that microVMs can use.
MicroVMs sign a record that contains the host’s public key that hosts the microVM,
while hosts sign a record that contains their transport addresses.
Elixir
Implementing this in Elixir would work as follows.
Elixir can run in a unikernel via e.g. Unikraft, NanoVMs, or Rump, or in a stripped-down Linux-based environment via e.g. microvm.nix or munix
Each engine listens on an UDP port of the IPv6 address that belongs to its microVM for unicast messages, and on IPv6 addresses of pub/sub topics for multicast messages.
CoAP is used as the application layer protocol,
with a serialized and potentially encrypted payload that contains the engine message.
Incoming messages are deserialized then passed on to the appropriate message handler.
Outgoing messages are serialized, then sent out to the destination IPv6 address and port.
Local engines in the same microVM/BEAM instance may use native GenServer message passing without serialization.
There’s no more router and transport engines in Elixir,
those are implemented in external microVMs.
The layered protocols and deserialization by engines
ensure there’s no more global data structure needed that lists all possible message types,
each engine needs to care only about its own message types, according to the end-to-end principle.
Applications
Applications also run in microVMs, and may have a graphical user interface via Wayland forwarding.
Alternatively, browser-based or native apps may be used with connections to a local microVM, which is the more feasible approach for now.
There are a work-in-progress efforts to bring microVMs to desktop/mobile, by SpectrumOS, microvm.nix, and munix. The Android Virtualization Framework supports Wayland GUIs via the Terminal app.
Next steps
I’m going to continue working on protocol designs and specifications for this microVM-based message passing and execution system, which serves as the foundation for building higher layers of distributed operating systems.
An upcoming paper will provide details on the design of the system.
See also a summary / overview at WaxOS.net