# Quick proposal write-up for distributed indexing **URL:** \ **Category:** Specs V2 Review\ **Created:** [December 19, 2024, 12:08am UTC](https://forum.anoma.net/t/quick-proposal-write-up-for-distributed-indexing/978 "2024-12-19T00:08:46Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![cwgoes](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.anoma.net/cwgoes/32/22_2.png) [@cwgoes](https://forum.anoma.net/u/cwgoes)\ **Post date:** [December 19, 2024, 12:08am UTC](https://forum.anoma.net/t/quick-proposal-write-up-for-distributed-indexing/978/1 "2024-12-19T00:08:46Z")
Per my understanding, the following product requirement [in v0.2](https://forum.anoma.net/t/v0-2-product-requirements/895): > [@On another topic](https://forum.anoma.net/t/895/1): > > Basic distributed indexing with custom projection functions > > - Nodes can subscribe to indices with custom projection functions (computed over state) > - Deltas will be broadcast to them when the state changes > - Basic DSL or analytical tools to scale this efficiently is not yet satisfied by the v0.2 specs. Before making a PR, I thought it would be a good idea to put a quick proposal here and sync between relevant parties (paging @jonathan @AHart @tg-x @mariari @ray). To deconstruct the product requirement a bit more, for v0.2 I think we want specifically: 1. An indexer “process” (engine instance) that can be run separately from the executor node (could be on the executor node, but could also just be another node subscribing to all the transaction results / blocks) 2. The ability to write projection functions in Juvix (compiled to Anockma) which can be sent to this indexer process and subscribed to - then associated with a particular pub/sub channel to which the indexer engine instance will broadcast updates to when the result of computing the projection function changes. For v0.2 I think we can defer actual distribution of the _computation_ involved in indexing, optimization of such (e.g. avoiding recomputation, more advanced “database indexing” techniques), proofs of indexing correctness, and any incentive modeling whatsoever. We should, however, have some notion of gas limits to ensure that the indexer engine instance can’t just be trivially DoS’ed. From the application user perspective, this kind of indexing is intended to provide something that feels like “state sync”, in the sense that a UI (e.g. web application) could subscribe to state updates, and display the latest state in near realtime as soon as any transaction changes it, where most of the involved computation is offloaded to servers elsewhere (for v0.2, a single indexer engine instance, with more complex topologies supported in subsequent versions). In accordance with these specifics, I propose: - an indexer engine as part of the ordering machine (where it seems to fit best) - messages to subscribe to new projection functions (submitted as Anockma functions), and unsubscribe from previously-subscribed ones - indexer engine behaviour in line with the above descriptions How does that sound? Does this fit well with engineering’s indexer work thus far, or would it be a different direction? Also, is there a nice way to broadcast “data structure deltas” (roughly a la [JSON Patch](https://jsonpatch.com/), but perhaps in a more efficient binary format) for Anockma terms, or would we need to create such a mini-protocol? ---
**Author:** ![ArtemG](https://avatars.discourse-cdn.com/v4/letter/a/45deac/32.png) [@ArtemG](https://forum.anoma.net/u/ArtemG)\ **Post date:** [December 19, 2024, 1:01pm UTC](https://forum.anoma.net/t/quick-proposal-write-up-for-distributed-indexing/978/2 "2024-12-19T13:01:20Z")
> [@cwgoes](#): > > executor node Could you by chance clarify what you mean by the executor node? Is this the executor engine? Or something more general? ---
**Author:** ![graphomath](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.anoma.net/graphomath/32/9_2.png) [@graphomath](https://forum.anoma.net/u/graphomath)\ **Post date:** [December 20, 2024, 3:43pm UTC](https://forum.anoma.net/t/quick-proposal-write-up-for-distributed-indexing/978/3 "2024-12-20T15:43:44Z")
I would guess it is something like the node hosting execution engine processes, which takes care of post-ordering execution and RSM state updates. But yes, it would be nice to have this confirmed @cwgoes. FWIW [notes on execution engine](https://forum.anoma.net/uploads/short-url/8OeXg86QozJRFTl09tSpp3vGkQp.pdf) ---
**Author:** ![cwgoes](https://dub1.discourse-cdn.com/flex013/user_avatar/forum.anoma.net/cwgoes/32/22_2.png) [@cwgoes](https://forum.anoma.net/u/cwgoes)\ **Post date:** [January 1, 2025, 9:23pm UTC](https://forum.anoma.net/t/quick-proposal-write-up-for-distributed-indexing/978/4 "2025-01-01T21:23:18Z")
> [@ArtemG](#): > > Could you by chance clarify what you mean by the executor node? Is this the executor engine? Or something more general? I mean whichever node is specifically ordering and executing transactions and producing blocks. > [@graphomath](#): > > I would guess it is something like the node hosting execution engine processes, which takes care of post-ordering execution and RSM state updates. Yes.