Data node v2 spec

[davidsiska-vega]

No description provided.

[gordsport]

Need to add the FE team requirements: https://www.notion.so/vegaprotocol/Core-API-design-ideas-data-node-Requirements-e8d00fbd0c96425687bfa0bc00baf8eb#915a0966aec742eba64aaa0125816c8e

[gordsport]

@campbellssource - please can you review this and add the FE requirements to this as its part of Oregon Trail

[gordsport]

@pscott31 and @Vegaklaus - to discuss the db choice and how this can work with hashing and update spec requirements if needed

[pscott31]

Is it part of this spec to think about how the data is to be retrieved retrieved as well? Perhaps that's a separate thing to think about

[pscott31]

There are a few things that we have talked about at various points which aren't in here; such as proof of storage, bootstrapping another node without playing the whole chain etc.. Again maybe they are topics for another day or another spec. Something else to consider is what behaviour to expect when there is restore from a checkpoint (delete everything?) or a snapshot (delete everything with a block height > the snapshot and continue?)

[Vegaklaus]

@pscott31 and @Vegaklaus - to discuss the db choice and how this can work with hashing and update spec requirements if needed

Results: The database will be designed so it can (deterministically) map a random number into a table, and then (also deterministically) be able to go sequentially through the contents of this table (e.g., by timestamp) starting at a point determined by another random number.
The database can also (again deterministically) convert records into an appropriate serialisation that can be fed into a hash function.

@pscott31 does that sum it up correctly ?

[pscott31]

Yeah, that's about it Klaus :)

[Vegaklaus]

Added two questions for clarification.

[davidsiska-vega]

The data node v2 must support the following workflow:

1. a vega network is running
2. a core node joins (as non-validator) by restoring state from a snaphot
3. a new data node can be spun up that obtains full history from other data nodes (get addresses from the non-validator node and then download) and finally start receiving events from "its own" non validator node.

[campbellssource]

@gordsport forgive this PR into a PR #886
I thing this spec is too general to add all of the FE requirements to.
As mentioned, we need to figure out a process for specifying each FE requirement on data node. At the moment we have this list in notion under the new from front end column. Either the Q2 objectives get agreed and we write a mini-spec for each Data node augmentation or we just have data node capacity to work in real time with Front end team (as well as other data node users)

[davidsiska-vega]

If we get this working vegaprotocol/quant#22 then there may be a way of using the quant libraries we have (or with small extension) on client side.

[gordsport]

Hi @jeremyletang @pscott31
I have renamed this file and created a ticked to review this spec with the aim to add acceptance criteria:

• 0011-NP-DANO-data-node.md needs acceptance criteria adding #999

If you are OK with the naming please can you give it a review and merge?

[gordsport]

Agreed in protocol design meeting this can be merged