The stellar-horizon program cannot download stellar-history.json

[airstring]

What version are you using?

$ bin/stellar-horizon version
2.28.3-fa5404bc79ba72ec9311e063e1a27cd5712941be
go1.20.14

What is the problem?

The configuration file stellar-horizon uses is: https://github.com/stellar/quickstart/blob/master/pubnet/horizon/etc/stellar-captive-core.cfg

An error was reported after stellar-horizon started:
Feb 22 23:09:02 ip-172-24-103-x stellar-horizon[2860207]: time="2024-02-22T23:09:02.819+08:00" level=info msg="History: Downloading history archive state: .well-known/stellar-history.json" pid=2860207 service=ingest subservice=stellar-core
Feb 22 23:09:03 ip-172-24-103-x stellar-horizon[2860207]: time="2024-02-22T23:09:03.540+08:00" level=info msg="History: Downloading history archive state: .well-known/stellar-history.json" pid=2860207 service=ingest subservice=stellar-core
Feb 22 23:09:04 ip-172-24-103-x stellar-horizon[2860207]: time="2024-02-22T23:09:04.228+08:00" level=warning msg="Process: process 2860300 exited 22: curl -sf https://stellar-archive-1-lobstr.s3.amazonaws.com/.well-known/stellar-history.json -o buckets/tmp/history-c6bddb5037dc9f5d/1e56388d76334b42-stellar-history.json" pid=2860207 service=ingest subservice=stellar-core
Feb 22 23:09:04 ip-172-24-103-x stellar-horizon[2860207]: time="2024-02-22T23:09:04.228+08:00" level=error msg="History: Could not download file: archive lobstr_1_europe maybe missing file .well-known/stellar-history.json" pid=2860207 service=ingest subservice=stellar-core
Feb 22 23:09:04 ip-172-24-103-x stellar-horizon[2860207]: time="2024-02-22T23:09:04.229+08:00" level=error msg="History: Could not download .well-known/stellar-history.json file: corrupt archive lobstr_1_europe" pid=2860207 service=ingest subservice=stellar-core
Feb 22 23:09:04 ip-172-24-103-x stellar-horizon[2860207]: time="2024-02-22T23:09:04.229+08:00" level=error msg="History: Failed to obtain archive information: lobstr_1_europe" pid=2860207 service=ingest subservice=stellar-core
...
Feb 23 01:08:22 ip-172-24-103-x stellar-horizon[2900699]: time="2024-02-23T01:08:22.988+08:00" level=info msg="Ledger: Got consensus: [seq=50487327, prev=000000, txs=0, ops=0, sv: [ SIGNED@lobstr_5_australia txH: 5f7eeb, ct: 1708621697, upgrades: [ ] ]]" pid=2900699 service=ingest subservice=stellar-core
Feb 23 01:08:22 ip-172-24-103-x stellar-horizon[2900699]: time="2024-02-23T01:08:22.988+08:00" level=info msg="Ledger: Close of ledger 50487327 buffered. mSyncingLedgers has 32 ledgers" pid=2900699 service=ingest subservice=stellar-core
Feb 23 01:08:22 ip-172-24-103-x stellar-horizon[2900699]: time="2024-02-23T01:08:22.997+08:00" level=info msg="History: Catching up to ledger 50487295: downloading ledger files 175/467 (37%)" pid=2900699 service=ingest subservice=stellar-core
Feb 23 01:08:23 ip-172-24-103-x stellar-horizon[2900699]: time="2024-02-23T01:08:23.009+08:00" level=info msg="History: Catching up to ledger 50487295: downloading ledger files 176/467 (37%)" pid=2900699 service=ingest subservice=stellar-core
Feb 23 01:08:23 ip-172-24-103-x stellar-horizon[2900699]: time="2024-02-23T01:08:23.020+08:00" level=info msg="History: Catching up to ledger 50487295: downloading ledger files 177/467 (37%)" pid=2900699 service=ingest subservice=stellar-core
Feb 23 01:08:23 ip-172-24-103-x stellar-horizon[2900699]: time="2024-02-23T01:08:23.044+08:00" level=info msg="History: Catching up to ledger 50487295: downloading ledger files 178/467 (38%)" pid=2900699 service=ingest subservice=stellar-core
Feb 23 01:08:23 ip-172-24-103-x stellar-horizon[2900699]: time="2024-02-23T01:08:23.044+08:00" level=info msg="History: Catching up to ledger 50487295: downloading ledger files 179/467 (38%)" pid=2900699 service=ingest subservice=stellar-core
Feb 23 01:08:23 ip-172-24-103-x stellar-horizon[2900699]: time="2024-02-23T01:08:23.078+08:00" level=warning msg="Process: process 2978053 exited 22: curl -sf https://stellar-archive-5-lobstr.s3.amazonaws.com/ledger/03/02/12/ledger-030212bf.xdr.gz -o buckets/tmp/catchup-960e7fc35e95f441/ledger/03/02/12/ledger-030212bf.xdr.gz.tmp" pid=2900699 service=ingest subservice=stellar-core
Feb 23 01:08:23 ip-172-24-103-x stellar-horizon[2900699]: time="2024-02-23T01:08:23.078+08:00" level=error msg="History: Could not download file: archive lobstr_5_australia maybe missing file ledger/03/02/12/ledger-030212bf.xdr.gz" pid=2900699 service=ingest subservice=stellar-core
Feb 23 01:08:23 ip-172-24-103-x stellar-horizon[2900699]: time="2024-02-23T01:08:23.135+08:00" level=warning msg="Process: process 2978060 exited 22: curl -sf https://stellar-archive-3-lobstr.s3.amazonaws.com/ledger/03/02/0a/ledger-03020aff.xdr.gz -o buckets/tmp/catchup-960e7fc35e95f441/ledger/03/02/0a/ledger-03020aff.xdr.gz.tmp" pid=2900699 service=ingest subservice=stellar-core
Feb 23 01:08:23 ip-172-24-103-x stellar-horizon[2900699]: time="2024-02-23T01:08:23.135+08:00" level=error msg="History: Could not download file: archive lobstr_3_north_america maybe missing file ledger/03/02/0a/ledger-03020aff.xdr.gz" pid=2900699 service=ingest subservice=stellar-core
Feb 23 01:08:23 ip-172-24-103-x stellar-horizon[2900699]: time="2024-02-23T01:08:23.183+08:00" level=warning msg="Process: process 2978115 exited 22: curl -sf https://stellar-archive-4-lobstr.s3.amazonaws.com/ledger/03/02/17/ledger-0302173f.xdr.gz -o buckets/tmp/catchup-960e7fc35e95f441/ledger/03/02/17/ledger-0302173f.xdr.gz.tmp" pid=2900699 service=ingest subservice=stellar-core
Feb 23 01:08:23 ip-172-24-103-x stellar-horizon[2900699]: time="2024-02-23T01:08:23.183+08:00" level=error msg="History: Could not download file: archive lobstr_4_asia maybe missing file ledger/03/02/17/ledger-0302173f.xdr.gz" pid=2900699 service=ingest subservice=stellar-core
Feb 23 01:08:23 ip-172-24-103-x stellar-horizon[2900699]: time="2024-02-23T01:08:23.222+08:00" level=info msg="History: Catching up to ledger 50487295: downloading ledger files 180/467 (38%)" pid=2900699 service=ingest subservice=stellar-core

[mollykarcher]

@airstring this looks like an issue with the history archives you're trying to use, which based on the logs looked to be owned/maintained by Lobstr (or Ultrastellar). You'll need to contact that provider or configure yourself to use more reliable history archives.