7935: Add E2StoreReader #8329
7935: Add E2StoreReader #8329
Conversation
Signed-off-by: Matilda Clerke <[email protected]>
Signed-off-by: Matilda Clerke <[email protected]>
Signed-off-by: Matilda Clerke <[email protected]>
Signed-off-by: Matilda Clerke <[email protected]>
Signed-off-by: Matilda Clerke <[email protected]>
| * @param beaconState The beacon state | ||
| * @param slot The slot number | ||
| */ | ||
| public record E2BeaconState(byte[] beaconState, int slot) {} |
There was a problem hiding this comment.
We don't need the beacon types as we won't be writing or reading the beacon data from anywhere
There was a problem hiding this comment.
I was initially writing a generic e2 file reader. Are we unlikely to use era files for populating the chain beyond the merge? I realise we can populate it through the p2p network, or potentially from portal, but it shouldn't be difficult to support this too.
| * @param signedBeaconBlock The signed beacon block | ||
| * @param slot The slot number | ||
| */ | ||
| public record E2SignedBeaconBlock(byte[] signedBeaconBlock, int slot) {} |
There was a problem hiding this comment.
Same here, we don't need this type
| * @param startingSlot The first slot number indexed by this slot index | ||
| * @param indexes The indexes of the slots indexed by this slot index | ||
| */ | ||
| public record E2SlotIndex(long startingSlot, List<Long> indexes) {} |
| void handleBeaconState(E2BeaconState beaconState); | ||
|
|
||
| /** | ||
| * Handles the supplied E2SlotIndex | ||
| * | ||
| * @param slotIndex the E2SlotIndex | ||
| */ | ||
| void handleSlotIndex(E2SlotIndex slotIndex); | ||
|
|
||
| /** | ||
| * Handles the supplied E2SignedBeaconBlock | ||
| * | ||
| * @param signedBeaconBlock the E2SignedBeaconBlock | ||
| */ | ||
| void handleSignedBeaconBlock(E2SignedBeaconBlock signedBeaconBlock); |
There was a problem hiding this comment.
can remove the beacon handlers, they won't be in the era1 files we use
| COMPRESSED_SIGNED_BEACON_BLOCK(new byte[] {0x01, 0x00}), | ||
| /** A snappy compressed, SSZ encoded, beacon state */ | ||
| COMPRESSED_BEACON_STATE(new byte[] {0x02, 0x00}), |
There was a problem hiding this comment.
Also can remove these as we don't need to parse era files
| * @param header The execution block header | ||
| * @param slot The slot number | ||
| */ | ||
| public record E2ExecutionBlockHeader(byte[] header, int slot) {} |
| * @param receipts The execution block's transaction receipts | ||
| * @param slot The slot number | ||
| */ | ||
| public record E2ExecutionBlockReceipts(byte[] receipts, int slot) {} |
| import org.xerial.snappy.SnappyFramedInputStream; | ||
|
|
||
| /** Reads E2 store file such a .era and .era1 */ | ||
| public class E2StoreReader { |
There was a problem hiding this comment.
Think it would be nicer to have this in the e2 package as it's not general io it's specific to era
| case COMPRESSED_BEACON_STATE -> { | ||
| byte[] compressedBeaconStateArray = inputStream.readNBytes(length); | ||
| try (SnappyFramedInputStream decompressionStream = | ||
| snappyFactory.createFramedInputStream(compressedBeaconStateArray)) { | ||
| // TODO: decode with SSZ | ||
| listener.handleBeaconState( | ||
| new E2BeaconState(decompressionStream.readAllBytes(), slot++)); | ||
| } | ||
| } | ||
| case COMPRESSED_SIGNED_BEACON_BLOCK -> { | ||
| byte[] compressedSignedBeaconBlockArray = inputStream.readNBytes(length); | ||
| try (SnappyFramedInputStream decompressionStream = | ||
| snappyFactory.createFramedInputStream(compressedSignedBeaconBlockArray)) { | ||
| // TODO: decode with SSZ | ||
| listener.handleSignedBeaconBlock( | ||
| new E2SignedBeaconBlock(decompressionStream.readAllBytes(), slot++)); | ||
| } | ||
| } |
| } | ||
| case COMPRESSED_EXECUTION_BLOCK_HEADER -> { | ||
| byte[] compressedExecutionBlockHeader = inputStream.readNBytes(length); | ||
| try (SnappyFramedInputStream decompressionStream = |
There was a problem hiding this comment.
Since we have already read the bytes it would be simpler to use SnappyCompressor.decompress to decompress snappy instead of creating the snappy input stream
There was a problem hiding this comment.
Unfortunately, the use of snappy framing means this won't work afaik. When I tried it on the beacon state initially, it failed to decompress.
PR description
Adds an Es2StoreReader for reading e2 files such as era and era1 files
Fixed Issue(s)
#7935