Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pgbackrest exec fails intermittently #925

Open
dcaputo-harmoni opened this issue Oct 10, 2024 · 0 comments · May be fixed by #926
Open

Pgbackrest exec fails intermittently #925

dcaputo-harmoni opened this issue Oct 10, 2024 · 0 comments · May be fixed by #926
Labels
bug Something isn't working

Comments

@dcaputo-harmoni
Copy link

Report

Exec commands failing due to lack of retry, similar to an issue encountered with the mysql xtradb operator. Looks like the same code in clientcmd.go is present here. Submitting a PR shortly with the same fix.

More about the problem

: exec: : error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-postgres/pods/pg-db-postgres-jms4-0/exec?command=pgbackrest&command=info&command=--output%3Djson&command=--repo%3D1&container=database&stderr=true&stdout=true\": unexpected EOF", "errorVerbose": "error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-postgres/pods/pg-db-postgres-jms4-0/exec?command=pgbackrest&command=info&command=--output%3Djson&command=--repo%3D1&container=database&stderr=true&stdout=true\": unexpected EOF\nexec: \ngithub.com/percona/percona-postgresql-operator/percona/pgbackrest.GetInfo\n\t/go/src/github.com/percona/percona-postgresql-operator/percona/pgbackrest/pgbackrest.go:61\ngithub.com/percona/percona-postgresql-operator/percona/controller/pgcluster.(*PGClusterReconciler).cleanupOutdatedBackups\n\t/go/src/github.com/percona/percona-postgresql-operator/percona/controller/pgcluster/backup.go:60\ngithub.com/percona/percona-postgresql-operator/percona/controller/pgcluster.(*PGClusterReconciler).reconcileBackups\n\t/go/src/github.com/percona/percona-postgresql-operator/percona/controller/pgcluster/backup.go:29\ngithub.com/percona/percona-postgresql-operator/percona/controller/pgcluster.(*PGClusterReconciler).Reconcile\n\t/go/src/github.com/percona/percona-postgresql-operator/percona/controller/pgcluster/controller.go:226\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695\nget pgBackRest info\ngithub.com/percona/percona-postgresql-operator/percona/controller/pgcluster.(*PGClusterReconciler).cleanupOutdatedBackups\n\t/go/src/github.com/percona/percona-postgresql-operator/percona/controller/pgcluster/backup.go:66\ngithub.com/percona/percona-postgresql-operator/percona/controller/pgcluster.(*PGClusterReconciler).reconcileBackups\n\t/go/src/github.com/percona/percona-postgresql-operator/percona/controller/pgcluster/backup.go:29\ngithub.com/percona/percona-postgresql-operator/percona/controller/pgcluster.(*PGClusterReconciler).Reconcile\n\t/go/src/github.com/percona/percona-postgresql-operator/percona/controller/pgcluster/controller.go:226\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
github.com/percona/percona-postgresql-operator/percona/controller/pgcluster.(*PGClusterReconciler).Reconcile
    /go/src/github.com/percona/percona-postgresql-operator/percona/controller/pgcluster/controller.go:226
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222

Steps to reproduce

Run cluster with pgbackup enabled, observe intermittent errors (in my case on AKS)

Versions

  1. Kubernetes 1.30.4
  2. Operator 2.4.0
  3. Database 16

Anything else?

No response
@dcaputo-harmoni dcaputo-harmoni added the bug Something isn't working label Oct 10, 2024
@dcaputo-harmoni dcaputo-harmoni linked a pull request Oct 10, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
1 participant
@dcaputo-harmoni