Revert writing to local dag during proposal

[liamsi]

Summary

Previously, we wanted to store on ipld (and provide to the underlying dht) as early as possible so that validators could do DAS. Hence, for instance the proposer calls PutBlock as early as possible. While it still could make sense to call putting and providing earlier, I would recommend treating the ipld store as a replacement for the tendermint block store instead. This means removing the calls to the ipld/write methods during proposal.

Details

It is not (consensus) critical to store the block ipfs earlier. Hence, we can simply store the block (during consensus) during finalizeCommit. We can remove the the earlier storing to ipfs for proposals now. Everything else is already part of separate issues/PRs.
Once below action items are tackled we have half the story of a "full node based" storage node (see #435). Although I agree that we should write an ADR describing them in more detail still. This issue is mostly about getting rid of storing to ipfs during proposal only. Otherwise proposers will store the block twice (once during proposal and once when finalizing the commit; at least currently).

Action Items

Update: We decided to move the IPLD/IPFS logic into a separate repository. Hence, this issue boils down to remove calling PutBlock during decideProposal.

• separate storing from "providing" (@Wondertan already did the changes necessary in Preliminary introduction of RowSet with supporting changes #427; these should be broken out of Preliminary introduction of RowSet with supporting changes #427 and merged separately)
• store the block data on IPFS only (this will be done once store: change to save DA and headers #218 is merged; see also Store: only store header + DA header #182)
• remove storing to IPFS in defaultDecideProposal
• make sure existing DAS light client still works

References

Saving the block data to IPFS was also already done in: #374
Storing the block data exclusively in the ipld store will done in: #218
Related discussion: https://github.com/celestiaorg/lazyledger-core/pull/374/files#r645077665

[Wondertan]

@liamsi, During the proposal stage, nodes must verify the integrity of a block received from the proposer. To verify that they need to erasure code the block and build IPLD/NMT trees of it. If we want to decouple saving from the proposal stage, we need to do verification twice: during proposal and saving, where in the first case, we do not pass the visitor option. IMO, we should avoid double verification, though I agree that saving in finalizing makes more sense.

[liamsi]

Yeah, good point. I did not mention the fact that we should still compute the EDS to be able to compute the data root correctly (sth that is currently done via storing). We need to think of ways to make this efficient. I think the erasure coding is the expensive part. Computing the eds could be "cached" (e.g. in a private field in the block) to only happen once. Computing the tree twice is probably very cheap; storing or even network IO is the expensive part.

(Note that if we switch block gossiping to using shares or even rows like in #434 we would be computing the tree or subtrees for verification as well and the latter quite often; not only during saving).

[Wondertan]

Let's introduce caching. I’ll try caching ExtendedDataSqaure in on block in #427