# \[Proposal\] Anoma Network Architecture **URL:** \ **Category:** ART\ **Created:** [February 2, 2024, 5:20pm UTC](https://forum.anoma.net/t/proposal-anoma-network-architecture/465 "2024-02-02T17:20:14Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![tg-x](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.anoma.net/tg-x/32/11_2.png) [@tg-x](https://forum.anoma.net/u/tg-x)\ **Post date:** [February 2, 2024, 5:20pm UTC](https://forum.anoma.net/t/proposal-anoma-network-architecture/465/1 "2024-02-02T17:20:14Z")
# Network Architecture - ART Report ## Introduction - motivation / angle of approach / models - models to architecture & protocols ## Requirements Design requirements for: - messaging patterns - message delivery semantics - Network architecture & protocols - Software architecture ## Related work - mention other p2p libraries/frameworks with similar scope (e.g. libp2p, gnunet) ## Network Architecture - describe overall network architecture - identities for addressing & message authentication - unicast/multicast messaging - sovereign domains - describe each network protocol - messages types & protocol logic - how different protocols interact with / rely on each other ## Software Architecture - describe engines & message passing, motivated by the actor model - unicast/multicast/anycast comm patterns - message preferences - privacy/security/reliability/ordering - describe each engine of the networking machine - what network protocols they implement - message flows between engines - protocol testing & evaluation - launch many nodes on the same machine - multiple nodes (set of actors) directly connected - network transport for simulation: - add delay + message loss ## Future work - open problems/questions - improvements necessary - future research directions - future ART reports ---
**Author:** ![cwgoes](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.anoma.net/cwgoes/32/22_2.png) [@cwgoes](https://forum.anoma.net/u/cwgoes)\ **Post date:** [February 22, 2024, 2:46pm UTC](https://forum.anoma.net/t/proposal-anoma-network-architecture/465/4 "2024-02-22T14:46:32Z")
_Sketch of high-level mathematical model_ - We assume an opaque, cryptographic, compositional identity system (as defined [here](https://specs.anoma.net/v1/architecture-1/abstractions/identity.html), although rather than citing the specs we should include these definitions directly in the report). - We assume a dynamic underlying physical network topology which is constantly changing - nodes (identified by cryptographic identity) are moving between networks, switching IP addresses, even switching physical models completely (e.g. Bluetooth, LoRa). - We assume that - at any network time slice - each node has a specific subscription preference function, which can be expressed as a predicate `Payload -> Boolean`, which specifies which sorts of message payloads they would like to receive. This predicate is also expected to be constantly changing (although not too quickly). - We want to provide a generalized primitive `Send` - `Send` takes: - an opaque payload (`[]byte`) - a target external identity (which may be decomposable) - routing preferences and constraints (e.g. cost vs latency vs bandwidth) - information flow preferences and constraints (exact form TBD, see [here](https://forum.anoma.net/t/simple-information-flow-control-for-partial-transactions/360) for inspiration) - The implementation of `Send` should (across this distributed network) - deliver the payload to all nodes who: - are part of the target external identity, and - are interested in the payload (determined by their preference predicate) - … while optimizing for the specified routing preferences as best as possible, - … not violating any of the routing constraints, - … respecting the specified information flow preferences as best as possible, - … and not violating any of the information flow constraints. - In general, the metadata gossiped around the network is designed to facilitate this implementation of `Send`, both from a completeness perspective (the message payloads reaching the right destinations) and from an optimization perspective (the routing and information flow preferences being optimized for as best as possible). - In general, routing is a Bayesian inference problem, which can be split into two parts: 1. Given known information (observed network traffic in the past), infer the probability distribution of underlying network topology states. 2. Given the probability distribution of underlying network topology states and the routing preferences/constraints, route the message in a way which optimizes for the preferences and does not violate any of the constraints. ---
**Author:** ![tg-x](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.anoma.net/tg-x/32/11_2.png) [@tg-x](https://forum.anoma.net/u/tg-x)\ **Post date:** [February 23, 2024, 12:50pm UTC](https://forum.anoma.net/t/proposal-anoma-network-architecture/465/5 "2024-02-23T12:50:50Z")
In general good direction/starting points for defining the underlying model, see some of my notes below. > each node has a specific subscription preference function To be precise: each engine instance > We assume that - at any network time slice - each node has a specific subscription preference Again, each engine instance. This assumption seems to be specific to pub/sub, which takes topic subscriptions, and can possibly be further extended with content-based filters, such as tags or other specific attributes of received events For unicast messages the assumption is that any valid message according to the protocol an engine implements is accepted, however engines may specify a _routing scope_ where an incoming/outgoing message may go to, such as local to the node, a topic, or a domain. Constraining certain messages to the local node only is an important security consideration that constrains access to e.g. local storage, config, etc engines. > `Send` takes: Here we need to separate two kinds of preferences: - next-hop node selection by a routing protocol (done by an engine implementing a routing protocol, e.g. onion routed relay, multicast routing for topic-based pub/sub, anycast routing for domain requests) - transport selection when sending a message to a specific node (done by the Transport engine) These preferences & constraints are by default only influence local choice, however a subset of these may be forwarded along with a message by a multi-hop routing protocol (e.g. message expiry) This way we can talk about multi-hop routing as recursive Send calls by different nodes until it reaches the destination. > The implementation of `Send` should (across this distributed network) Here I would point out that `Send` is implemented in a layered way: a high-level Send (provided by the Router) is responsible for sending direct unicast messages between nodes via Transport, and other types of messages (relay, multicast, anycast) the Router hands off to the appropriate protocol implemented by another engine, which have their own `Send` implementation that selects a next-hop node > metadata gossiped around We refer to these as authenticated node/topic/domain _advertisements_, and they can be delivered via various protocols not just _gossip_, i.e. direct messages, pubsub, or an actual p2p gossip protocol, etc. Indeed, these facilitate routing by associating node/transport addresses to identities, which the routing and transport protocols use for next-hop node/transport selection. Next to `Send `, we should also specify a `Deliver` primitive, which delivers a message to a local engine instance once it has been received. This takes into considerations delivery semantics specified in the message by the sender: - unordered - delivered immediately - FIFO order - delivered once previous messages for the same sender-destination pair has been delivered - implemented using sequence numbers - causal order - delivered once all causal dependencies of the message has been delivered - implemented using causal barriers, i.e. listing direct deps of a message in the message header In case of FIFO or Causal order, when a missing message is detected, these should be requested from the network by the receiver, i.e. the routing protocol that is delivering the message, e.g. the Router for unicast messages, or PubSub/Relay/DomainRouting/etc for other protocols. Related recent discussions around the Ordering Machine: - the Ordering Machine assumes reliable delivery of messages, and it needs causal delivery semantics which it implements on its own - however, this is a generic useful primitive not just by the OM that the Network Machine should be able to provide, and once it does, OM can rely on those directly - this should be eventually specified in the specs under [properties](https://specs.anoma.net/main/architecture-1/properties.html#reliable-delivery) Related work to transport selection algorithms and p2p software & network architecture: > [Automatic Transport Selection and Resource Allocation for Resilient Communication in Decentralised Networks](https://git.gnunet.org/bibliography.git/plain/docs/paper_short.pdf) > We begin by detailing the overall requirements for an algorithm for transport selection and resource allocation, and then compare three different solutions using (1) a heuristic, (2) linear optimisation, and (3) machine learning > [The GNUnet System](https://grothoff.org/christian/habil.pdf) > Describes network & software architecture, and P2P protocols > Chapter 3.1 Transport underlay abstraction: summary of the paper above ---
**Author:** ![tg-x](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.anoma.net/tg-x/32/11_2.png) [@tg-x](https://forum.anoma.net/u/tg-x)\ **Post date:** [February 27, 2024, 3:05pm UTC](https://forum.anoma.net/t/proposal-anoma-network-architecture/465/6 "2024-02-27T15:05:18Z")
Related to the system model, verification considerations: [https://martin.kleppmann.com/2022/10/12/verifying-distributed-systems-isabelle.html](https://martin.kleppmann.com/2022/10/12/verifying-distributed-systems-isabelle.html) ---
**Author:** ![cwgoes](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.anoma.net/cwgoes/32/22_2.png) [@cwgoes](https://forum.anoma.net/u/cwgoes)\ **Post date:** [May 2, 2024, 6:39am UTC](https://forum.anoma.net/t/proposal-anoma-network-architecture/465/7 "2024-05-02T06:39:42Z")
> [@tg-x](#): > > To be precise: each engine instance 👍 > [@tg-x](#): > > Again, each engine instance. 👍 > [@tg-x](#): > > This assumption seems to be specific to pub/sub, which takes topic subscriptions, and can possibly be further extended with content-based filters, such as tags or other specific attributes of received events > > For unicast messages the assumption is that any valid message according to the protocol an engine implements is accepted, however engines may specify a _routing scope_ where an incoming/outgoing message may go to, such as local to the node, a topic, or a domain. > Constraining certain messages to the local node only is an important security consideration that constrains access to e.g. local storage, config, etc engines. I think in the mathematical model this can be unified - “any valid message according to the protocol an engine implements” is still a subscription preference, just a simple one. Additionally, even for unicast messages, receiving engines may have restrictions on which other nodes they will accept incoming messages from. > [@tg-x](#): > > Here we need to separate two kinds of preferences: > > - next-hop node selection by a routing protocol (done by an engine implementing a routing protocol, e.g. onion routed relay, multicast routing for topic-based pub/sub, anycast routing for domain requests) > - transport selection when sending a message to a specific node (done by the Transport engine) > > These preferences & constraints are by default only influence local choice, > however a subset of these may be forwarded along with a message by a multi-hop routing protocol (e.g. message expiry) > > This way we can talk about multi-hop routing as recursive Send calls by different nodes until it reaches the destination. 👍