Switch to using MinIO

[isedwards]

Currently we're tied to AWS S3.

Investigate switching to MinIO to become cloud provider agnostic.

[faysal-ishtiaq]

I have written two simple scripts to test minio's compatibility. And it seems that minio works quite well.

boto3_list_buckets.py

import boto3

# Retrieve the list of existing buckets
s3 = boto3.client(
    's3', 
    aws_access_key_id="ACCESS_KEY",
    aws_secret_access_key="SECRET_KEY"
)
response = s3.list_buckets()

# Output the bucket names
print('Existing buckets:')
for bucket in response['Buckets']:
    print(f'  {bucket["Name"]}')

Outputs:

$ python boto3_list_buckets.py 
Existing buckets:
  climsoft-paper-archive
  demo.opencdms.org

minio_list_buckets.py

import minio

# Retrieve the list of existing buckets
s3 = minio.Minio(
    's3.amazonaws.com', 
    access_key="ACCESS_KEY",
    secret_key="SECRET_KEY"
)

response = s3.list_buckets()

# Output the bucket names
print('Existing buckets:')
for bucket in response:
    print(f'  {bucket.name}')

Outputs:

$ python minio_list_buckets.py 
Existing buckets:
  climsoft-paper-archive
  demo.opencdms.org