/exp/services/ledgerexporter: resumable export, check data storage for optimal starting point

[sreuland]

PR Checklist

PR Structure

• This PR has reasonably narrow scope (if not, break it down into smaller PRs).
• This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
• This PR's title starts with name of package that is most changed in the PR, ex.
  services/friendbot, or all or doc if the changes are broad or impact many
  packages.

Thoroughness

• This PR adds tests for the most critical parts of the new functionality or fixes.
• I've updated any docs (developer docs, .md
  files, etc... affected by this change). Take a look in the docs folder for a given service,
  like this one.

Release planning

• I've updated the relevant CHANGELOG (here for Horizon) if
  needed with deprecations, added features, breaking changes, and DB schema changes.
• I've decided if this PR requires a new major/minor version according to
  semver, or if it's mainly a patch change. The PR is targeted at the next
  release branch if it's not a patch change.

What

• Added ResumableManager, app will call this during init to apply a best effort attempt to check data storage layer and increase the effective start value for captive core ingestion to the boundary aligned starting ledger of the nearest absent object key of data store relative to requested start.
• Enhanced the DataStoreConfig toml model to be more generic key/value params to allow for holding different data store configuration values going forward, the app checks datastore_config.type to know which datastore instance to create, passing the datastore_config.params to it:

  [datastore_config]
  type = "GCS"
  
  [datastore_config.params]
  destination_url = "gcs://your-bucket-name"
  

Why

Efficiency of exporter, avoiding exporting files to data store that may already be present.

Closes: https://stellarorg.atlassian.net/browse/HUBBLE-272

Known limitations

[sreuland]

I need to run some live integration tests, building ledgerexporter and running it against GCS bucket.

[sreuland]

I need to run some live integration tests, building ledgerexporter and running it against GCS bucket.

I ran ledgerexporter on testnet against a real gcs bucket doing a sequence of commands that requested ranges overlapping, to confirm the expected resumable starts were calculated, using ledgers_per_file=1:

ledgerexporter --config-file config.toml --start 842267 --end 844267
... started core at 842267, uploaded all 2000

ledgerexporter --config-file config.toml --start 842267 --end 844267
... log output stating full range existed on data store already, no core or upload attempted

ledgerexporter --config-file config.toml --start 842267 --end 844277
... log output stating start was adjusted to 844268, uploaded next 10

 ledgerexporter --config-file config.toml --start 842267 --end 844287
... log output stating start was adjusted to 844278, uploaded next 10

 ledgerexporter --config-file config.toml --start 842267 --end 0
... log output stating start was adjusted to 844288, and continued on unbounded upload from there

[urvisavla]

lgtm!

[tamirms]

looks good! I left a few minor suggestions, feel free to apply them as you see fit