rustic v0.7.0 and iDrive S3 backend - how to configure

[kapitainsky]

Checking example config - https://github.com/rustic-rs/rustic/blob/main/config/s3.toml

# rustic config file to use s3 storage
# Note that this internally uses opendal S3 service, see https://opendal.apache.org/docs/rust/opendal/services/struct.S3.html
# where endpoint, bucket and root are extracted from the repository URL.
[repository]
repository = "s3:https://s3.amazonaws.com/bucket/my/repopath"
password = "mypassword"

# Other options can be given here - note that opendal also support reading config from env files or AWS config dirs, see the opendal S3 docu
[repository.options]
region = "us-east-1"

And I am not not sure at all how to use it... Where I put access and secret keys? I found - https://opendal.apache.org/docs/services/s3 - but reading this site does not make me any closer to how to configure rustic to use S3...

I think that for such popular backend like S3 it would make sense to provide full example. Something every idiot like myself can simply reuse:)

[aawsome]

You can use

[repository.options]
access_key_id = "..."
secret_access_key = "..."

...

in your config. This is what I used to successfully test S3 backend (but I didn't test against AWS).

AFAIR opendal also reads the commonly used AWS config dirs and env variables like ~/.aws/credentials, so this might not be needed.

Anyway, if you get a working example please don't hesitate to propose additions to the config files provided!

[aawsome]

You might also ask in the opendal github project. If you use repsoitory = "opendal:s3", you have to specify all s3 options under [repository.options], just like specified in https://opendal.apache.org/docs/rust/opendal/services/struct.S3.html#configuration

[l-austenfeld]

I have the same problem using Minio.

[INFO] using config ./rustic.toml
[ERROR] service=s3 operation=stat path=config -> Unexpected (persistent) at stat, context: { uri: http://127.0.0.1:9000//rustic/config, response: Parts { status: 400, version: HTTP/1.1, headers: {"accept-ranges": "bytes", "content-length": "253", "content-type": "application/xml", "server": "MinIO", "vary": "Origin", "date": "Sun, 04 Feb 2024 15:35:50 GMT"} }, service: s3, path: config }
error: error listing the repo config file

However if I use repsoitory = "opendal:s3" like this

[repository]
repository = "opendal:s3"
password = "***"

[repository.options]
root = "/"
bucket = "rustic"
endpoint = "http://127.0.0.1:9000"
region = "auto"
access_key_id = "admin"
secret_access_key = "***"

it works:

[INFO] using config ./rustic.toml
[WARN] service=s3 operation=stat path=config -> NotFound (persistent) at stat, context: { uri: http://127.0.0.1:9000/rustic/config, response: Parts { status: 404, version: HTTP/1.1, headers: {"accept-ranges": "bytes", "content-length": "0", "server": "MinIO", "strict-transport-security": "max-age=31536000; includeSubDomains", "vary": "Origin", "vary": "Accept-Encoding", "x-amz-id-2": "dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8", "x-amz-request-id": "17B0B315AFE6988E", "x-content-type-options": "nosniff", "x-minio-error-code": "NoSuchKey", "x-minio-error-desc": "\"The specified key does not exist.\"", "x-xss-protection": "1; mode=block", "date": "Sun, 04 Feb 2024 15:39:24 GMT"} }, service: s3, path: config }
[WARN] service=s3 operation=stat path=config -> NotFound (persistent) at stat, context: { uri: http://127.0.0.1:9000/rustic/config, response: Parts { status: 404, version: HTTP/1.1, headers: {"accept-ranges": "bytes", "content-length": "0", "server": "MinIO", "strict-transport-security": "max-age=31536000; includeSubDomains", "vary": "Origin", "vary": "Accept-Encoding", "x-amz-id-2": "dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8", "x-amz-request-id": "17B0B315AFF099D9", "x-content-type-options": "nosniff", "x-minio-error-code": "NoSuchKey", "x-minio-error-desc": "\"The specified key does not exist.\"", "x-xss-protection": "1; mode=block", "date": "Sun, 04 Feb 2024 15:39:24 GMT"} }, service: s3, path: config }
[INFO] key 8e7c14f2 successfully added.
[INFO] repository 9c1532d0 successfully created.
[INFO] using cache at /home/lennart/.cache/rustic/9c1532d06537cbb19ab02ead47ed6632b4d27c5641a18e411f05093a3e952af1

[kapitainsky]

AFAIK iDrive S3 is using Minio. So maybe related. Going to try your way.

[kapitainsky]

Ok this is what works for iDrive:

[repository]
repository = "opendal:s3"
password = "pass"

[repository.options]
root = "/"
bucket = "test-dal"
endpoint = "https://p7v1.ldn.idrivee2-40.com"
region = "gb-ldn"
access_key_id = "xxx"
secret_access_key = "xxx"

For repository = "s3:... despite trying everything what I could think of does not work.

So for now it has to be repository = "opendal:s3" way

[simonsan]

Can you PR that under config/services/iDrive.toml please?

[kapitainsky]

np

[kapitainsky]

As this works:

[repository]
repository = "opendal:s3"
password = "pass"

[repository.options]
root = "/"
bucket = "test-dal"
endpoint = "https://p7v1.ldn.idrivee2-40.com"
region = "gb-ldn"
access_key_id = "xxx"
secret_access_key = "xxx"

and this does not:

[repository]
repository = "s3:https://p7v1.ldn.idrivee2-40.com/test-dal"
password = "pass"

[repository.options]
access_key_id = "xxx"
secret_access_key = "xxx"
region = "gb-ldn"

It looks like s3 backend has some problem collecting all data from toml file. Both examples above contain all the same information.

[l-austenfeld]

It looks like rustic internally sets endpoint=http://127.0.0.1:9000/ (with a trailing slash). This causes a request url like this http://127.0.0.1:9000//rustic/config.
If I set endpoint to that value manually I get the same error.
It seems like Minio does not like the //.

[simonsan]

Tracking issue: #1045

[aawsome]

Thanks a lot for investigating. I think we'll add a note in the config file short term and fix the parsing when using s3: syntax.
Meanwhile, please use the opendal:s3 syntax until this is fixed!

[kapitainsky]

What is the purpose of s3 backend? It only adds complexity IMO. Any benefits?

[aawsome]

Mainly convenience. Also may make switching from restic easier. But we also already discussed that and will proclaim the opendal variant as the preferred one.