geth full archive snapshots might not be working #351
Comments
|
### This was shared from a community member : this is what i just tested with 2 servers (dedicated, 7tb nvme, amd 5950x , 128gb ram) Both download stopped at the same time. server 1 : (wget with long timeout flag) base-mainnet-full-1731839014.tar.zst 6%[====> ] 119.68G 19.5MB/s in 71m 19s 2024-11-19 11:59:24 (28.6 MB/s) - Connection closed at byte 128505195208. Retrying. server 2: aria with longer timeout flag + -x4 -s4 Download Progress Summary as of Tue Nov 19 12:59:18 2024[#0b986e 417GiB/1,717GiB(24%) CN:1 DL:78MiB ETA:4h43m55s]
|
|
Hey y'all, have you tried again after the failure, we hosted our snapshots on cloudflare, and this does seem like either a cloudflare issue or network issue. Also FYI, there's some cloudflare maintenance going on that seems relevant: https://www.cloudflarestatus.com/ |
same question. I try more than 10 times always 403 error 11/20 09:31:59 [ERROR] CUID#7 - Download aborted. URI=https://mainnet-full-snapshots.base.org/base-mainnet-full-1731839014.tar.zst 11/20 09:31:59 [NOTICE] Download GID#8a9a7f7227700a58 not complete: /home/snap1/base-mainnet-full-1731839014.tar.zst Download Results: Status Legend: aria2 will resume download if the transfer is restarted. |
|
Trying 2 downloads to see if I could reproduce:
|
I download latest 11/21 02:53:43 [ERROR] CUID#7 - Download aborted. URI=https://mainnet-full-snapshots.base.org/base-mainnet-full-1731925706.tar.zst 11/21 02:53:43 [NOTICE] Download GID#a702387759a7b2b3 not complete: /home/snap1/base-mainnet-full-1731925706.tar.zst Download Results: Status Legend: aria2 will resume download if the transfer is restarted. |
How long before your download was aborted? I was able to replicate this myself, digging into this |
|
submitted ticket to cloudflare (where we host the snapshts) for triage, will update here once we hear back |
|
I can't download with wget, I can download for a while and then have to download again. I download it takes about 10% to 30% and then the connection is lost and it asks to re-download. This happens constantly and I can't complete the download.
|
|
I am experiencing the same issues. I believe the problem can be addressed by enabling the server to support the Accept-Ranges header. This would allow downloads to resume from where the connection was interrupted instead of restarting from the beginning. |
|
Update: The download just failed again. I confirm that the issue persists, and I have been unable to download the snapshot for several days. I tried around 10 times, and none of the downloads exceeded 500GB before the connection dropped. In my opinion, connection interruptions are not the core issue—network reliability is inherently variable. The key is to ensure that downloads can resume from where they left off. Utilities like
If the server can ensure these features are correctly implemented, it would allow users to resume downloads seamlessly, even after interruptions, resolving much of the frustration around this issue. |
|
Is there anywhere else that stores the snapshot? |
|
Hey everybody, just wanna pop in here to say that we're actively investigating this issue along with cloudflare, and will update here asap when we have a solution. In the meantime, you can follow the incident at https://status.base.org/ cc: @usoftvn @Mohammed-Alanazisa @bab3nk0v @liusDong @azeonmarket |
|
Update that cloudflare team has seen range request headers being dropped, they're investigating further |
|
it did work for me but unstable .. was doing with base-mainnet-full-1731147805.tar.zst |
|
This issue is taking a while to resolve. While investigating, is it possible to push the snapshot to another server? |
|
It's been two weeks and I can't run the node because I can't download the snapshot. Does anyone have a quick solution while waiting for the project to resolve this issue? |
|
Hey y'all, we've worked with cloudflare team and believe this issue is fixed now. Please try again and let us know if you are still experiencing the issue. TLDR of the root cause is that there's this cache feature for cloudflare R2 that tries to cache the underlying object when serving the download requests, and since our snapshots are too big (in terabytes), the cache feature is not actually designed for that (it does not issue range request for large files), therefore the download breaks a lot. Now that we've disabled the caching feature, it should work now. cc: @usoftvn @Mohammed-Alanazisa @bab3nk0v @liusDong @azeonmarket @jun0tpyrc |
The snapshot download has completed. It looks like the problem is solved. Thanks. |
|
Closing this now as this should've been fixed by various reports |
|
The download is indeed working now (thank you), but it seems the snapshots (at least for Reth archive), are ~2.5 weeks behind, and syncing is slow enough that it may not catch up to head. |
@jotto it seems like our latest reth snapshot is just two days old: |
|
The timestamps on the URLs look 1-2 days old, but when you download, untar, and start Reth from the snapshot, it's somewhere around ~22,222,400 (latest block is 23,275,735). I haven't tried yesterday's snapshot 1733199514 (Tuesday, December 3, 2024 4:18:34 AM) but have tried the 2 previous days:
are you able to check if the reth node producing these snapshots is synced to head? |
and others
The Geth full archive snapshots not working for downloading snapshot to be able to run a full archive Geth Node. https://mainnet-archive-snapshots.base.org/$(curl https://mainnet-archive-snapshots.base.org/latest)
If true of malfunctions/not working properly, changes on https://docs.base.org/tutorials/run-a-base-node/ on this ligne from the table :
Mainnet | Geth | Archive | wget https://mainnet-archive-snapshots.base.org/$(curl https://mainnet-archive-snapshots.base.org/latest) -- | -- | -- | --would need to be implemented to allow node operators to deploy easily.
The text was updated successfully, but these errors were encountered: