[BUG]pg-basebackup restore failed on huaweiCloud

[ahjing99]

➜ ~ kbcli version
Kubernetes: v1.25.4-r10-25.2.34.1
KubeBlocks: 0.7.0-beta.1
kbcli: 0.7.0-beta.1

1. Install kb

kbcli kubeblocks install --set provider=huaweiCloud

2. create backuprepo

➜  ~ k get backuprepo
NAME                    STATUS   STORAGEPROVIDER   DEFAULT   AGE
backuprepo-kbcli-test   Ready    s3                true      25m

3. create cluster

kbcli cluster create pg --cluster-definition postgresql --set cpu=100m,memory=100Mi

4. Backup and restore

      `kbcli cluster backup pg --method pg-basebackup --namespace default `

Backup backup-default-pg-20231012155152 created successfully, you can view the progress:
	kbcli cluster list-backups --name=backup-default-pg-20231012155152 -n default
check backup status

      `kbcli cluster list-backups pg --namespace default `

NAME                               NAMESPACE   SOURCE-CLUSTER   METHOD          STATUS    TOTAL-SIZE   DURATION   CREATE-TIME                  COMPLETION-TIME   EXPIRATION
backup-default-pg-20231012155152   default     pg               pg-basebackup   Running                           Oct 12,2023 15:51 UTC+0800                     Oct 19,2023 15:51 UTC+0800
backup_status:pg-pg-basebackup-Running
^@check backup status done
backup_status:backup-default-pg-20231012155152   default     pg               pg-basebackup   Completed   69M          14s        Oct 12,2023 15:51 UTC+0800   Oct 12,2023 15:52 UTC+0800   Oct 19,2023 15:51 UTC+0800
cluster restore backup

      `kbcli cluster describe-backup backup-default-pg-20231012155152 --namespace default `

Name: backup-default-pg-20231012155152	Cluster: pg	Namespace: default

Spec:
  Method:             pg-basebackup
  Policy Name:        pg-postgresql-backup-policy

Status:
  Phase:              Completed
  Total Size:         69M
  ActionSet Name:     postgres-basebackup
  PVC Name:           pvc-backuprepo-kbcli-test-lqhj48
  Duration:           14s
  Expiration Time:    Oct 19,2023 15:51 UTC+0800
  Start Time:         Oct 12,2023 15:51 UTC+0800
  Completion Time:    Oct 12,2023 15:52 UTC+0800
  Path:               /default/pg-5300afa3-7baf-4767-9f62-44301f69dbdb/postgresql/backup-default-pg-20231012155152
  Time Range Start:   Oct 12,2023 08:00 UTC+0800
  Time Range End:     Oct 12,2023 15:51 UTC+0800

Warning Events: <none>

      `kbcli cluster restore pg-backup --backup backup-default-pg-20231012155152 --namespace default `

Cluster pg-backup created

5. Restore failed

➜  ~ k get pod | grep pg
pg-postgresql-0                                                   8/8     Running   0          12m
pg-postgresql-1                                                   8/8     Running   0          12m
pg-postgresql-2                                                   8/8     Running   0          12m
restore-preparedata-9ca27a98-backup-default-pg-2023101215525rt6   0/1     Error     0          10m
restore-preparedata-9ca27a98-backup-default-pg-2023101215544bgt   0/1     Error     0          9m33s
restore-preparedata-9ca27a98-backup-default-pg-202310121558448j   0/1     Error     0          10m
restore-preparedata-9ca27a98-backup-default-pg-20231012155dcdwr   0/1     Error     0          10m

➜  ~ k logs restore-preparedata-9ca27a98-backup-default-pg-2023101215544bgt
bash: line 2: cd: /backup-default-pg-20231012155152/default/pg-5300afa3-7baf-4767-9f62-44301f69dbdb/postgresql/backup-default-pg-20231012155152: No such file or directory

➜  ~ k get cluster | grep pg
pg             postgresql           postgresql-14.8.0   WipeOut              Running   42m
pg-backup      postgresql           postgresql-14.8.0   WipeOut                        11m

➜  ~ k describe cluster pg-backup
Name:         pg-backup
Namespace:    default
Labels:       clusterdefinition.kubeblocks.io/name=postgresql
              clusterversion.kubeblocks.io/name=postgresql-14.8.0
Annotations:  kubeblocks.io/restore-from-backup:
                {"postgresql":{"name":"backup-default-pg-20231012155152","namespace":"default","managementPolicy":"Parallel"}}
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2023-10-12T07:52:15Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:  1
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubeblocks.io/restore-from-backup:
      f:spec:
        .:
        f:affinity:
          .:
          f:podAntiAffinity:
          f:tenancy:
        f:backup:
          .:
          f:enabled:
          f:method:
          f:pitrEnabled:
          f:retentionPeriod:
        f:clusterDefinitionRef:
        f:clusterVersionRef:
        f:componentSpecs:
          .:
          k:{"name":"postgresql"}:
            .:
            f:classDefRef:
              .:
              f:class:
            f:componentDefRef:
            f:enabledLogs:
              .:
              v:"running":
            f:monitor:
            f:name:
            f:noCreatePDB:
            f:replicas:
            f:resources:
              .:
              f:limits:
                .:
                f:cpu:
                f:memory:
              f:requests:
                .:
                f:cpu:
                f:memory:
            f:serviceAccountName:
            f:switchPolicy:
              .:
              f:type:
            f:volumeClaimTemplates:
        f:monitor:
        f:resources:
          .:
          f:cpu:
          f:memory:
        f:storage:
          .:
          f:size:
        f:terminationPolicy:
    Manager:      kbcli
    Operation:    Update
    Time:         2023-10-12T07:52:15Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"cluster.kubeblocks.io/finalizer":
        f:labels:
          .:
          f:clusterdefinition.kubeblocks.io/name:
          f:clusterversion.kubeblocks.io/name:
    Manager:      manager
    Operation:    Update
    Time:         2023-10-12T07:52:15Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
    Manager:         manager
    Operation:       Update
    Subresource:     status
    Time:            2023-10-12T07:54:21Z
  Resource Version:  52095
  UID:               d3d54637-a3e4-4894-985f-b61156bb050b
Spec:
  Affinity:
    Pod Anti Affinity:  Preferred
    Tenancy:            SharedNode
  Backup:
    Enabled:               false
    Method:
    Pitr Enabled:          false
    Retention Period:      7d
  Cluster Definition Ref:  postgresql
  Cluster Version Ref:     postgresql-14.8.0
  Component Specs:
    Class Def Ref:
      Class:
    Component Def Ref:  postgresql
    Enabled Logs:
      running
    Monitor:        true
    Name:           postgresql
    No Create PDB:  false
    Replicas:       3
    Resources:
      Limits:
        Cpu:     200m
        Memory:  644245094400m
      Requests:
        Cpu:               200m
        Memory:            644245094400m
    Service Account Name:  kb-pg
    Switch Policy:
      Type:  Noop
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  23Gi
  Monitor:
  Resources:
    Cpu:     0
    Memory:  0
  Storage:
    Size:              0
  Termination Policy:  WipeOut
Status:
  Conditions:
    Last Transition Time:  2023-10-12T07:52:15Z
    Message:               The operator has started the provisioning of Cluster: pg-backup
    Observed Generation:   1
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2023-10-12T07:54:21Z
    Message:               restore "pg-backup-postgresql-d3d54637-preparedata" is Failed, you can describe it and re-restore the cluster.
    Reason:                RestoreFailed
    Status:                False
    Type:                  ApplyResources
Events:
  Type     Reason                 Age                    From                Message
  ----     ------                 ----                   ----                -------
  Normal   NeedWaiting            11m (x3 over 11m)      cluster-controller  waiting for restore "pg-backup-postgresql-d3d54637-preparedata" successfully
  Normal   PreCheckSucceed        11m                    cluster-controller  The operator has started the provisioning of Cluster: pg-backup
  Normal   ApplyResourcesSucceed  11m                    cluster-controller  Successfully applied for resources
  Warning  RestoreFailed          4m8s (x18 over 9m36s)  cluster-controller  restore "pg-backup-postgresql-d3d54637-preparedata" is Failed, you can describe it and re-restore the cluster.

➜ ~ k logs kubeblocks-7984446cd-dxf7f -n kb-system>kblog.txt
Defaulted container "manager" out of: manager, tools (init), datascript (init)
kblog.txt