Retrieve milestones by index as well as by solidity

[karimodm]

Description

New nodes sync the tangle by recursively requesting transactions they don't know about: during the syncing process milestones are solidified in the same way, once all the milestones between the genesis (snapshot) and latest known milestone are solid the node is to be considered solid. In other words the node is solidified tips to genesis.

If a node lags considerably behind in terms of solid milestones it may take a really long time (or forever) for the tips to genesis approach to retrieve all the necessary milestones to achieve solidity.

I propose a code change in which we specifically insert into the requester queue the milestones transactions, starting from the first unsolid one. This can be easily achieved by querying peers using findTransactions with a trytes representation of the milestone index as tag. The returned hash of the only signed transactions corresponding to coordinator address will be added to the transaction requester queue.

Motivation

• Generally speaking, nodes will be able to sync faster.
• Nodes stuck before a Coo local snapshot will be able to recover by requesting the specific milestones they missed.

Requirements

• Parallel thread to feed transaction requester queue.
• Incrementally retrieve unsolid milestones by index.
• transactions returned by findTransactions should match coordinator address and should bring a valid signature.

Am I planning to do it myself with a PR?

Yes

[GalRogozinski]

This can be done as an IXI

[alexsporn]

Why are you thinking in doing this at the "API"-level instead of the gossip-level? Not every node has the API publicly available.

[muXxer]

Why don't we change the gossip protocol and add this functionality?

For example we could create a special frame, which sends a "tip request" to our neighbors (req hash = 9's), but instead of containing a valid TX, we could insert other data in the frame. What about the latest milestone hash of our own node, and the solid milestone hash of our node. Now the neighbor knows which milestones in between we are missing. Then he could send the hashes of the milestones again in a special frame (containing a lot of milestone hashes). Now we can start to request these hashes from all neighbors.

Maybe this special frame could contain a "command" identifier in the beginning. This way we can add more functionality to the gossip protocol later.

Neighbors with an old version would just drop the frames, because the TX hash is invalid.

[GalRogozinski]

Will be solved as part of #1791
Closing this since this is outdated