Cloudflare R2 bucket not supported

[shinebayar-g]

Report

Hi. I was trying to set up PerconaXtraDBCluster with Cloudflare R2 bucket. They are one of the S3 compatible providers, but lack some APIs. Because of this, both xb-* and pitr pods are not working.

More about the problem

❯ k get pods                
NAME                           READY   STATUS             RESTARTS         AGE
pxc-db-haproxy-0               2/2     Running            0                4h13m
pxc-db-haproxy-1               2/2     Running            0                4h11m
pxc-db-haproxy-2               2/2     Running            0                4h10m
pxc-db-pitr-65b4c5d9d7-fftpf   0/1     CrashLoopBackOff   15 (3m54s ago)   56m
pxc-db-pxc-0                   3/3     Running            0                4h13m
pxc-db-pxc-1                   3/3     Running            0                4h11m
pxc-db-pxc-2                   3/3     Running            0                4h9m
xb-backup1-5785n               0/1     Error              0                22m
xb-backup1-fgsmb               0/1     Error              0                27m
xb-backup1-j7hzr               0/1     Error              0                89s
xb-backup1-qf8m9               0/1     Error              0                26m
xb-backup1-s497p               0/1     Error              0                28m
xb-backup1-v9t67               0/1     Error              0                24m
xb-backup1-vc8zd               0/1     Error              0                18m
xb-backup1-vmmqs               0/1     Error              0                12m

backup pod logs

❯ k logs -f xb-backup1-vmmqs
Defaulted container "xtrabackup" out of: xtrabackup, backup-init (init)
...
2025-01-10 02:03:16.307  INFO: [SST script] 250110 02:03:15 xbcloud: [0] successfully uploaded chunk: pxc-db-2025-01-10-01:46:13-full/xtrabackup_tablespaces.00000000000000000001, size: 36

2025-01-10 02:03:16.307  INFO: [SST script] 250110 02:03:15 xbcloud: Uploading md5

2025-01-10 02:03:16.307  INFO: [SST script] 250110 02:03:16 xbcloud: Upload completed.

2025-01-10 02:03:16.307  INFO: [SST script] ++ handle_sigterm

2025-01-10 02:03:16.307  INFO: [SST script] ++ (( 1 == 0 ))

2025-01-10 02:03:16.308  INFO: [SST script] + trap '' 15

2025-01-10 02:03:16.308  INFO: [SST script] + mc -C /tmp/mc stat dest/pxc-backup/pxc-db-2025-01-10-01:46:13-full.md5

2025-01-10 02:03:16.551  INFO: [SST script] mc: <ERROR> Unable to stat `dest/pxc-backup/pxc-db-2025-01-10-01:46:13-full.md5`. 501 Not Implemented.

2025-01-10 02:03:16.555  INFO: SST script ended by termination
2025-01-10 02:03:16.555  INFO: Closing send monitor...
2025-01-10 02:03:16.555  INFO: Closed send monitor.
2025-01-10 02:03:16.555  INFO: gcomm: terminating thread
2025-01-10 02:03:16.555  INFO: gcomm: joining thread
2025-01-10 02:03:16.555  INFO: gcomm: closing backend
...
2025-01-10 02:03:16.560  INFO: recv_thread() joined.
2025-01-10 02:03:16.560  INFO: Closing send queue.
2025-01-10 02:03:16.560  INFO: Closing receive queue.
2025-01-10 02:03:16.560  INFO: Exiting main loop
2025-01-10 02:03:16.560  INFO: Garbd returns 1
+ EXID_CODE=1
+ '[' -f /tmp/backup-is-completed ']'
+ log ERROR 'Backup was finished unsuccessful'
+ exit 1
Terminating processProcess completed with error: /usr/bin/run_backup.sh: 1 (Operation not permitted)2025-01-10 02:03:16 [ERROR] Backup was finished unsuccessful

pitr pod logs

❯ k logs -f pxc-db-pitr-65b4c5d9d7-fftpf                
Defaulted container "pitr" out of: pitr, pitr-init (init)
2025/01/10 02:15:56 run binlog collector
2025/01/10 02:15:58 Reading binlogs from pxc with hostname= pxc-db-pxc-0.pxc-db-pxc.pxc-db.svc.cluster.local
2025/01/10 02:15:58 Starting to process binlog with name binlog.000002
2025/01/10 02:15:58 ERROR: collect binlog files: manage binlog: put binlog.000002 object: put object binlog_1736460206_dc3e5ea63dad01f12979e81c74d24912: Header 'x-amz-checksum-algorithm' with value 'CRC32C' not implemented

Quick google search for Cloudflare R2 Header 'x-amz-checksum-algorithm' with value 'CRC32C' not implemented suggests that Cloudflare R2 and other providers don't support this API.

However, this error is specifically thrown with minio-go client which is used in this project. Example forum

I'm sure other S3 compatible providers like DigitalOcean and Google Cloud Storage etc suffer from the same problem. Would you consider swapping out minio client with official AWS S3 SDK or something similar? This would increase the adoption and compatibility of the operator.

Steps to reproduce

1. Create Cloudflare R2 bucket
2. Configure PerconaXtraDBCluster with

storages:
  cloudflare-r2:
    s3:
      bucket: pxc-backup
      credentialsSecret: cloudflare-r2-secret
      endpointUrl: https://<ID>.r2.cloudflarestorage.com
      region: auto
    type: s3

Versions

1. Kubernetes 1.31.4
2. Operator 1.16.1
3. Database percona-xtradb-cluster:8.0.39-30.1

Anything else?

No response

[hors]

Hi @shinebayar-g, we have the same thoughts about it, and that is why we are going to use aws cli in the next release PXCO PR

[hors]

@shinebayar-g I have merged PR and now you can test main branch for your tests. Please do not use it in production (before the official release).

[shinebayar-g]

percona/percona-docker#1048