feat: path-based locations

[mmalenic]

Closes #281
Depends on #284

Features

• Add a simple path-based way to writing location lists that includes file://, s3:// or http(s)://.
  *The first part of the location determines the storage, and everything in the path after determines the prefix which matches on ids. For example, s3://bucket/prefix says to find files in bucket when the request also contains the prefix - /reads/prefix/.
  • Locations can be specified as either a single location, or a list of locations with [].
  • The regex-style of specifying locations is still available. Config that is considered "advanced" is separated into a module, which is reflected in the docs.
• Paths are joined with Pathbuf rather than string formating, so config values like file://data/path and file://data/path/ are equivalent.

Docs

• Re-worked docs to include simplified locations first, with "advanced" config included later.

Recent fixes

• Checked all examples and fixed any errors.

[brainstorm]

s3-storage shouldn't be required, IMHO?

[mmalenic]

Yeah that's fair. It's required because the secrets manager has all the AWS dependencies, which s3-storage enables.

Originally, only S3 was required for any AWS-specific things, but now that's changed, maybe this feature should be called aws rather than s3-storage.

[mmalenic]

It might be good to rename/simplify url-storage too? E.g.

s3-storage -> aws
url-storage -> url

[mmalenic]

I'll make this a separate PR.

[brainstorm]

Agreed on the renaming, but shouldn't we have a file:// for the secrets management part? I'm thinking about local testing and/or devving, right?

[mmalenic]

There is kind = "File" for local keys:

backend.keys.kind = "File"
backend.keys.private = "data/c4gh/keys/bob.sec" # pragma: allowlist secret
backend.keys.public = "data/c4gh/keys/alice.pub"

And kind = "SecretsManager" for secrets manager keys:

backend.keys.kind = "SecretsManager"
backend.keys.private = "htsget/test_c4gh_private_key" # pragma: allowlist secret
backend.keys.public = "htsget/test_c4gh_public_key"

Is that what you mean? Or some sort of local SecretsManager option?

[brainstorm]

No, I'm only concerned about being able to run c4gh infra locally, without requiring aws deps... that "Run with cargo run..." line change seems to imply that s3-storage is absolutely required? From what you just said it seems it's not, perhaps being explicit about both options (only via experimental and experimental+aws feature flags) should be explicitly stated?

[mmalenic]

Yes, you're right, it's not required. It's only required if you remove the comments in that file that use kind = "SecretsManager".

I'll make this clearer on the next PR. Maybe it should just be --all-features in all the examples to avoid any confusion?

[brainstorm]

Maybe it should just be --all-features in all the examples to avoid any confusion?

🤔 ... that'd exercise our error management logic and corresponding error messages, i.e: running with file:// paths and aws-related c4gh secrets management. It might make more sense this way if errors are clear, yeah!