Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Solve the problem of backup failure when using pg_basebackup. #1072

Closed

Conversation

taylor840326
Copy link

@taylor840326 taylor840326 commented Oct 4, 2024

Describe the bug
Backup failure when using pg_basebackup for PostgreSQL 16.4.0

To Reproduce
Steps to reproduce the behavior:

  1. kbcli version is 0.9.1
$ kbcli version
Kubernetes: v1.30.2
KubeBlocks: 0.9.1
kbcli: 0.9.1

$ kbcli addon list
NAME                           VERSION        PROVIDER    STATUS     AUTO-INSTALL
...
postgresql                     0.9.0          community   Enabled    true
...
  1. Create a postgresql 16.4.0 cluster
cat <<EOF | kubectl apply -f -
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
  name: rds-1
  namespace: rds-postgresql
spec:
  clusterDefinitionRef: postgresql
  clusterVersionRef: postgresql-16.4.0
  terminationPolicy: Delete
  affinity:
    podAntiAffinity: Preferred
    topologyKeys:
    - kubernetes.io/hostname
  tolerations:
    - key: kb-data
      operator: Equal
      value: 'true'
      effect: NoSchedule
  componentSpecs:
  - name: postgresql
    componentDefRef: postgresql
    enabledLogs:
    - running
    disableExporter: true
    replicas: 2
    resources:
      limits:
        cpu: '1'
        memory: 2Gi
      requests:
        cpu: '1'
        memory: 2Gi
    volumeClaimTemplates:
    - name: data
      spec:
        accessModes:
        - ReadWriteOnce
        storageClassName: hostpath
        resources:
          requests:
            storage: 20Gi
EOF
  1. Backup database
$ kbcli cluster backup rds-1 --method pg-basebackup
Backup backup-rds-postgresql-rds-1-20241004213223 created successfully, you can view the progress:
	kbcli cluster list-backups --name=backup-rds-postgresql-rds-1-20241004213223 -n rds-postgresql
  1. See error
$ kbcli cluster list-backups
NAME                                             NAMESPACE        SOURCE-CLUSTER      METHOD          STATUS                        TOTAL-SIZE   DURATION   CREATE-TIME                  COMPLETION-TIME              EXPIRATION
...
backup-rds-postgresql-rds-1-20241004213223       rds-postgresql   rds-1               pg-basebackup   Failed                                                Oct 04,2024 21:32 UTC+0800

$ kbcli cluster describe-backup backup-rds-postgresql-rds-1-20241004213223
Name: backup-rds-postgresql-rds-1-20241004213223	Cluster: rds-1	Namespace: rds-postgresql

Spec:
  Method:             pg-basebackup
  Policy Name:        rds-1-postgresql-backup-policy

Status:
  Phase:              Failed
  ActionSet Name:     postgres-basebackup
  Repository:         my-repo
  Start Time:         Oct 04,2024 21:32 UTC+0800
  Path:               /rds-postgresql/rds-1-c94db9a7-62ba-415d-b387-91f4d2a44531/postgresql/backup-rds-postgresql-rds-1-20241004213223

Warning Events:
TIME                         TYPE      REASON            OBJECT                                              MESSAGE
Oct 04,2024 21:32 UTC+0800   Warning   ReconcileFailed   Backup/backup-rds-postgresql-rds-1-20241004213223   Reconciling failed, error: there are failed actions, you can obtain the more informations in the....

$ k describe backup backup-rds-postgresql-rds-1-20241004213223
....
Status:
  Actions:
    Action Type:           Job
    Completion Timestamp:  2024-10-04T13:32:58Z
    Failure Reason:        Log Collector Output: Password:
pg_basebackup: error: incompatible server version 16.4 (Ubuntu 16.4-1.pgdg22.04+1)
failed with exit code 1

    Name:  dp-backup-0
    Object Ref:
      API Version:       batch/v1
      Kind:              Job
      Name:              dp-backup-0-backup-rds-postgresql-rds-1-20241004213223-a4e44859
      Namespace:         rds-postgresql

@taylor840326
Copy link
Author

taylor840326 commented Oct 4, 2024

The following method can be used to temporarily solve this problem.

$ kubectl edit as postgres-basebackup    
change image TAG from  14.8.0-pgvector-v0.6.1  to  16.4.0-pgvector-v0.7.4

$ kubectl edit as postgresql-pitr
change image TAG from  14.8.0-pgvector-v0.6.1  to  16.4.0-pgvector-v0.7.4

@wangyelei
Copy link
Contributor

wangyelei commented Oct 8, 2024

@taylor840326 you can add a env mapping to the backupPolicyTemplate. and merge it to release-0.9 branch.
clusterVersion will be removed in 1.0.
image

@taylor840326
Copy link
Author

@taylor840326 you can add a env mapping to the backupPolicyTemplate. and merge it to release-0.9 branch. clusterVersion will be removed in 1.0. image

🆗

@wangyelei wangyelei closed this Oct 10, 2024
@wangyelei
Copy link
Contributor

fixed at #1080

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants