Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] official postgresql cluster multiple replicas create failed #5591

Closed
JashBook opened this issue Oct 23, 2023 · 0 comments · Fixed by #5607
Closed

[BUG] official postgresql cluster multiple replicas create failed #5591

JashBook opened this issue Oct 23, 2023 · 0 comments · Fixed by #5607
Assignees
@kizuna-lek
Labels
bug ci kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Milestone
Release 0.7.0

Comments

@JashBook
Copy link
Collaborator

JashBook commented Oct 23, 2023 •
edited
Loading

Describe the bug
official postgresql cluster multiple replicas create failed.
executing checkRole error {"error": "failed to connect to host=localhost user=postgres database=postgres: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)"}

kbcli version
Kubernetes: v1.26.7-gke.500
KubeBlocks: 0.7.0-beta.9
kbcli: 0.7.0-beta.9

To Reproduce
Steps to reproduce the behavior:

  1. create cluster
kbcli cluster create  offpg-dzrgpx             --termination-policy=Delete --cluster-definition=official-postgresql --cluster-version=official-postgresql-14.7 --set cpu=100m,memory=0.5Gi,replicas=2,storage=3Gi
  1. See error
kubectl get cluster 
NAME             CLUSTER-DEFINITION    VERSION                     TERMINATION-POLICY   STATUS     AGE
offpg-dzrgpx     official-postgresql   official-postgresql-14.7    Delete               Updating   10m

kubectl get pod
NAME                                            READY   STATUS             RESTARTS         AGE
offpg-dzrgpx-postgresql-0                       2/2     Running            0                11m
offpg-dzrgpx-postgresql-1                       1/2     CrashLoopBackOff   4 (31s ago)      11m

logs error pod

kubectl logs offpg-dzrgpx-postgresql-1
Defaulted container "postgresql" out of: postgresql, kb-checkrole
offpg-dzrgpx-postgresql-0.offpg-dzrgpx-postgresql-headless:5432 - accepting connections
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 1/2F000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1332"
pg_basebackup: error: could not create directory "/var/lib/postgresql/data/pg_wal": Permission denied
pg_basebackup: removing contents of data directory "/var/lib/postgresql/data"
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 1/30000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1336"
pg_basebackup: error: could not create directory "/var/lib/postgresql/data/pg_wal": Permission denied
pg_basebackup: removing contents of data directory "/var/lib/postgresql/data"
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 1/31000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1341"
pg_basebackup: error: could not create directory "/var/lib/postgresql/data/pg_wal": Permission denied
pg_basebackup: removing contents of data directory "/var/lib/postgresql/data"
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 1/32000060 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1345"
pg_basebackup: error: could not create directory "/var/lib/postgresql/data/pg_wal": Permission denied
pg_basebackup: removing contents of data directory "/var/lib/postgresql/data"
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 1/33000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1349"
pg_basebackup: error: could not create directory "/var/lib/postgresql/data/pg_wal": Permission denied
pg_basebackup: removing contents of data directory "/var/lib/postgresql/data"
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 1/34000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1353"
pg_basebackup: error: could not create directory "/var/lib/postgresql/data/pg_wal": Permission denied
pg_basebackup: removing contents of data directory "/var/lib/postgresql/data"
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 1/35000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1357"
pg_basebackup: error: could not create directory "/var/lib/postgresql/data/pg_wal": Permission denied
pg_basebackup: removing contents of data directory "/var/lib/postgresql/data"
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 1/36000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1361"
pg_basebackup: error: could not create directory "/var/lib/postgresql/data/pg_wal": Permission denied
pg_basebackup: removing contents of data directory "/var/lib/postgresql/data"
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 1/37000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1365"
pg_basebackup: error: could not create directory "/var/lib/postgresql/data/pg_wal": Permission denied
pg_basebackup: removing contents of data directory "/var/lib/postgresql/data"
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 1/38000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1369"
pg_basebackup: error: could not create directory "/var/lib/postgresql/data/pg_wal": Permission denied
pg_basebackup: removing contents of data directory "/var/lib/postgresql/data"
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 1/39000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1373"
pg_basebackup: error: could not create directory "/var/lib/postgresql/data/pg_wal": Permission denied
pg_basebackup: removing contents of data directory "/var/lib/postgresql/data"
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 1/3A000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1378"
pg_basebackup: error: could not create directory "/var/lib/postgresql/data/pg_wal": Permission denied
pg_basebackup: removing contents of data directory "/var/lib/postgresql/data"
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 1/3B000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1382"
pg_basebackup: error: could not create directory "/var/lib/postgresql/data/pg_wal": Permission denied
pg_basebackup: removing contents of data directory "/var/lib/postgresql/data"
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 1/3C000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1386"
pg_basebackup: error: could not create directory "/var/lib/postgresql/data/pg_wal": Permission denied
pg_basebackup: removing contents of data directory "/var/lib/postgresql/data"
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 1/3D000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1390"
pg_basebackup: error: could not create directory "/var/lib/postgresql/data/pg_wal": Permission denied
pg_basebackup: removing contents of data directory "/var/lib/postgresql/data"
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 1/3E000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1394"
pg_basebackup: error: could not create directory "/var/lib/postgresql/data/pg_wal": Permission denied
pg_basebackup: removing contents of data directory "/var/lib/postgresql/data"
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 1/3F000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1398"
pg_basebackup: error: could not create directory "/var/lib/postgresql/data/pg_wal": Permission denied
pg_basebackup: removing contents of data directory "/var/lib/postgresql/data"
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 1/40000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_1402"
pg_basebackup: error: could not create directory "/var/lib/postgresql/data/pg_wal": Permission denied
pg_basebackup: removing contents of data directory "/var/lib/postgresql/data"

check-role pod

kubectl logs offpg-dzrgpx-postgresql-1 kb-checkrole

{"level":"error","ts":1698053091.417659,"caller":"binding/operation_volume_protection.go:182","msg":"unmarshal volume protection spec error","raw spec":"","error":"unexpected end of JSON input","stacktrace":"github.com/apecloud/kubeblocks/lorry/binding.(*operationVolumeProtection).initVolumes\n\t/src/lorry/binding/operation_volume_protection.go:182\ngithub.com/apecloud/kubeblocks/lorry/binding.init.1\n\t/src/lorry/binding/operation_volume_protection.go:103\nruntime.doInit1\n\t/usr/local/go/src/runtime/proc.go:6740\nruntime.doInit\n\t/usr/local/go/src/runtime/proc.go:6707\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:249"}
{"level":"info","ts":1698053091.417746,"caller":"binding/operation_volume_protection.go:104","msg":"init volumes to monitor failed","error":"unexpected end of JSON input"}
{"level":"info","ts":1698053091.4177907,"caller":"binding/operation_volume_protection.go:106","msg":"succeed to init volume protection, pod: offpg-dzrgpx-postgresql-1, spec: {\"highWatermark\":\"0\",\"volumes\":[]}"}
2023-10-23T09:24:51Z	INFO	Postgres	Initializing Postgres binding
2023-10-23T09:24:55Z	INFO	grpc	role probe request	{"type": "postgresql"}
2023-10-23T09:24:55Z	ERROR	Postgres	query sql:select pg_is_in_recovery(); failed	{"error": "failed to connect to `host=localhost user=postgres database=postgres`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)"}
github.com/apecloud/kubeblocks/lorry/component/postgres.(*Manager).QueryWithHost
	/src/lorry/component/postgres/query.go:49
github.com/apecloud/kubeblocks/lorry/component/postgres/officalpostgres.(*Manager).GetMemberRoleWithHost
	/src/lorry/component/postgres/officalpostgres/manager.go:199
github.com/apecloud/kubeblocks/lorry/binding/postgres.(*PostgresOperations).GetRole
	/src/lorry/binding/postgres/postgres.go:150
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).CheckRoleOps
	/src/lorry/binding/base.go:229
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).Invoke
	/src/lorry/binding/base.go:194
github.com/apecloud/kubeblocks/lorry/middleware/probe.route
	/src/lorry/middleware/probe/router.go:170
github.com/apecloud/kubeblocks/lorry/middleware/grpc.NewGRPCServer.GetGrpcRouter.func1
	/src/lorry/middleware/probe/router.go:176
github.com/apecloud/kubeblocks/lorry/middleware/grpc.(*GRPCServer).Check
	/src/lorry/middleware/grpc/gprc_server.go:47
google.golang.org/grpc/health/grpc_health_v1._Health_Check_Handler
	/go/pkg/mod/google.golang.org/[email protected]/health/grpc_health_v1/health_grpc.pb.go:165
google.golang.org/grpc.(*Server).processUnaryRPC
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1336
google.golang.org/grpc.(*Server).handleStream
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/go/pkg/mod/google.golang.org/[email protected]/server.go:965
2023-10-23T09:24:55Z	ERROR	Postgres	get member role failed	{"error": "failed to connect to `host=localhost user=postgres database=postgres`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)"}
github.com/apecloud/kubeblocks/lorry/component/postgres/officalpostgres.(*Manager).GetMemberRoleWithHost
	/src/lorry/component/postgres/officalpostgres/manager.go:201
github.com/apecloud/kubeblocks/lorry/binding/postgres.(*PostgresOperations).GetRole
	/src/lorry/binding/postgres/postgres.go:150
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).CheckRoleOps
	/src/lorry/binding/base.go:229
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).Invoke
	/src/lorry/binding/base.go:194
github.com/apecloud/kubeblocks/lorry/middleware/probe.route
	/src/lorry/middleware/probe/router.go:170
github.com/apecloud/kubeblocks/lorry/middleware/grpc.NewGRPCServer.GetGrpcRouter.func1
	/src/lorry/middleware/probe/router.go:176
github.com/apecloud/kubeblocks/lorry/middleware/grpc.(*GRPCServer).Check
	/src/lorry/middleware/grpc/gprc_server.go:47
google.golang.org/grpc/health/grpc_health_v1._Health_Check_Handler
	/go/pkg/mod/google.golang.org/[email protected]/health/grpc_health_v1/health_grpc.pb.go:165
google.golang.org/grpc.(*Server).processUnaryRPC
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1336
google.golang.org/grpc.(*Server).handleStream
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/go/pkg/mod/google.golang.org/[email protected]/server.go:965
2023-10-23T09:24:55Z	ERROR	Postgres	executing checkRole error	{"error": "failed to connect to `host=localhost user=postgres database=postgres`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)"}
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).CheckRoleOps
	/src/lorry/binding/base.go:231
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).Invoke
	/src/lorry/binding/base.go:194
github.com/apecloud/kubeblocks/lorry/middleware/probe.route
	/src/lorry/middleware/probe/router.go:170
github.com/apecloud/kubeblocks/lorry/middleware/grpc.NewGRPCServer.GetGrpcRouter.func1
	/src/lorry/middleware/probe/router.go:176
github.com/apecloud/kubeblocks/lorry/middleware/grpc.(*GRPCServer).Check
	/src/lorry/middleware/grpc/gprc_server.go:47
google.golang.org/grpc/health/grpc_health_v1._Health_Check_Handler
	/go/pkg/mod/google.golang.org/[email protected]/health/grpc_health_v1/health_grpc.pb.go:165
google.golang.org/grpc.(*Server).processUnaryRPC
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1336
google.golang.org/grpc.(*Server).handleStream
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/go/pkg/mod/google.golang.org/[email protected]/server.go:965
2023-10-23T09:24:55Z	INFO	Postgres	role checks failed continuously	{"times": 0}
2023-10-23T09:24:55Z	INFO	Postgres	send event: map[event:Failed message:failed to connect to `host=localhost user=postgres database=postgres`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address) operation:checkRole originalRole:]
2023-10-23T09:24:55Z	INFO	Postgres	operation called	{"operation": "checkRole", "result": {"event":"Failed","message":"failed to connect to `host=localhost user=postgres database=postgres`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)","operation":"checkRole","originalRole":""}}
2023-10-23T09:24:55Z	INFO	grpc	No event detected	{"meta": "{\"duration\":\"20.479237ms\",\"end-time\":\"2023-10-23T09:24:55.808644728Z\",\"operation\":\"checkRole\",\"start-time\":\"2023-10-23T09:24:55.788165486Z\"}", "data": "{\"event\":\"Failed\",\"message\":\"failed to connect to `host=localhost user=postgres database=postgres`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)\",\"operation\":\"checkRole\",\"originalRole\":\"\"}"}
2023-10-23T09:24:56Z	INFO	grpc	role probe request	{"type": "postgresql"}
2023-10-23T09:24:56Z	ERROR	Postgres	query sql:select pg_is_in_recovery(); failed	{"error": "failed to connect to `host=localhost user=postgres database=postgres`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)"}
github.com/apecloud/kubeblocks/lorry/component/postgres.(*Manager).QueryWithHost
	/src/lorry/component/postgres/query.go:49
github.com/apecloud/kubeblocks/lorry/component/postgres/officalpostgres.(*Manager).GetMemberRoleWithHost
	/src/lorry/component/postgres/officalpostgres/manager.go:199
github.com/apecloud/kubeblocks/lorry/binding/postgres.(*PostgresOperations).GetRole
	/src/lorry/binding/postgres/postgres.go:150
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).CheckRoleOps
	/src/lorry/binding/base.go:229
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).Invoke
	/src/lorry/binding/base.go:194
github.com/apecloud/kubeblocks/lorry/middleware/probe.route
	/src/lorry/middleware/probe/router.go:170
github.com/apecloud/kubeblocks/lorry/middleware/grpc.NewGRPCServer.GetGrpcRouter.func1
	/src/lorry/middleware/probe/router.go:176
github.com/apecloud/kubeblocks/lorry/middleware/grpc.(*GRPCServer).Check
	/src/lorry/middleware/grpc/gprc_server.go:47
google.golang.org/grpc/health/grpc_health_v1._Health_Check_Handler
	/go/pkg/mod/google.golang.org/[email protected]/health/grpc_health_v1/health_grpc.pb.go:165
google.golang.org/grpc.(*Server).processUnaryRPC
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1336
google.golang.org/grpc.(*Server).handleStream
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/go/pkg/mod/google.golang.org/[email protected]/server.go:965
2023-10-23T09:24:56Z	ERROR	Postgres	get member role failed	{"error": "failed to connect to `host=localhost user=postgres database=postgres`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)"}
github.com/apecloud/kubeblocks/lorry/component/postgres/officalpostgres.(*Manager).GetMemberRoleWithHost
	/src/lorry/component/postgres/officalpostgres/manager.go:201
github.com/apecloud/kubeblocks/lorry/binding/postgres.(*PostgresOperations).GetRole
	/src/lorry/binding/postgres/postgres.go:150
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).CheckRoleOps
	/src/lorry/binding/base.go:229
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).Invoke
	/src/lorry/binding/base.go:194
github.com/apecloud/kubeblocks/lorry/middleware/probe.route
	/src/lorry/middleware/probe/router.go:170
github.com/apecloud/kubeblocks/lorry/middleware/grpc.NewGRPCServer.GetGrpcRouter.func1
	/src/lorry/middleware/probe/router.go:176
github.com/apecloud/kubeblocks/lorry/middleware/grpc.(*GRPCServer).Check
	/src/lorry/middleware/grpc/gprc_server.go:47
google.golang.org/grpc/health/grpc_health_v1._Health_Check_Handler
	/go/pkg/mod/google.golang.org/[email protected]/health/grpc_health_v1/health_grpc.pb.go:165
google.golang.org/grpc.(*Server).processUnaryRPC
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1336
google.golang.org/grpc.(*Server).handleStream
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/go/pkg/mod/google.golang.org/[email protected]/server.go:965
2023-10-23T09:24:56Z	ERROR	Postgres	executing checkRole error	{"error": "failed to connect to `host=localhost user=postgres database=postgres`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)"}
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).CheckRoleOps
	/src/lorry/binding/base.go:231
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).Invoke
	/src/lorry/binding/base.go:194
github.com/apecloud/kubeblocks/lorry/middleware/probe.route
	/src/lorry/middleware/probe/router.go:170
github.com/apecloud/kubeblocks/lorry/middleware/grpc.NewGRPCServer.GetGrpcRouter.func1
	/src/lorry/middleware/probe/router.go:176
github.com/apecloud/kubeblocks/lorry/middleware/grpc.(*GRPCServer).Check
	/src/lorry/middleware/grpc/gprc_server.go:47
google.golang.org/grpc/health/grpc_health_v1._Health_Check_Handler
	/go/pkg/mod/google.golang.org/[email protected]/health/grpc_health_v1/health_grpc.pb.go:165
google.golang.org/grpc.(*Server).processUnaryRPC
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1336
google.golang.org/grpc.(*Server).handleStream
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/go/pkg/mod/google.golang.org/[email protected]/server.go:965
2023-10-23T09:24:56Z	INFO	Postgres	operation called	{"operation": "checkRole", "result": {"event":"Failed","message":"failed to connect to `host=localhost user=postgres database=postgres`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)","operation":"checkRole","originalRole":""}}
2023-10-23T09:24:56Z	INFO	grpc	No event detected	{"meta": "{\"duration\":\"1.944428ms\",\"end-time\":\"2023-10-23T09:24:56.484817168Z\",\"operation\":\"checkRole\",\"start-time\":\"2023-10-23T09:24:56.482872748Z\"}", "data": "{\"event\":\"Failed\",\"message\":\"failed to connect to `host=localhost user=postgres database=postgres`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)\",\"operation\":\"checkRole\",\"originalRole\":\"\"}"}
2023-10-23T09:24:57Z	INFO	grpc	role probe request	{"type": "postgresql"}
2023-10-23T09:24:57Z	ERROR	Postgres	query sql:select pg_is_in_recovery(); failed	{"error": "failed to connect to `host=localhost user=postgres database=postgres`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)"}
github.com/apecloud/kubeblocks/lorry/component/postgres.(*Manager).QueryWithHost
	/src/lorry/component/postgres/query.go:49
github.com/apecloud/kubeblocks/lorry/component/postgres/officalpostgres.(*Manager).GetMemberRoleWithHost
	/src/lorry/component/postgres/officalpostgres/manager.go:199
github.com/apecloud/kubeblocks/lorry/binding/postgres.(*PostgresOperations).GetRole
	/src/lorry/binding/postgres/postgres.go:150
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).CheckRoleOps
	/src/lorry/binding/base.go:229
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).Invoke
	/src/lorry/binding/base.go:194
github.com/apecloud/kubeblocks/lorry/middleware/probe.route
	/src/lorry/middleware/probe/router.go:170
github.com/apecloud/kubeblocks/lorry/middleware/grpc.NewGRPCServer.GetGrpcRouter.func1
	/src/lorry/middleware/probe/router.go:176
github.com/apecloud/kubeblocks/lorry/middleware/grpc.(*GRPCServer).Check
	/src/lorry/middleware/grpc/gprc_server.go:47
google.golang.org/grpc/health/grpc_health_v1._Health_Check_Handler
	/go/pkg/mod/google.golang.org/[email protected]/health/grpc_health_v1/health_grpc.pb.go:165
google.golang.org/grpc.(*Server).processUnaryRPC
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1336
google.golang.org/grpc.(*Server).handleStream
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/go/pkg/mod/google.golang.org/[email protected]/server.go:965
2023-10-23T09:24:57Z	ERROR	Postgres	get member role failed	{"error": "failed to connect to `host=localhost user=postgres database=postgres`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)"}
github.com/apecloud/kubeblocks/lorry/component/postgres/officalpostgres.(*Manager).GetMemberRoleWithHost
	/src/lorry/component/postgres/officalpostgres/manager.go:201
github.com/apecloud/kubeblocks/lorry/binding/postgres.(*PostgresOperations).GetRole
	/src/lorry/binding/postgres/postgres.go:150
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).CheckRoleOps
	/src/lorry/binding/base.go:229
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).Invoke
	/src/lorry/binding/base.go:194
github.com/apecloud/kubeblocks/lorry/middleware/probe.route
	/src/lorry/middleware/probe/router.go:170
github.com/apecloud/kubeblocks/lorry/middleware/grpc.NewGRPCServer.GetGrpcRouter.func1
	/src/lorry/middleware/probe/router.go:176
github.com/apecloud/kubeblocks/lorry/middleware/grpc.(*GRPCServer).Check
	/src/lorry/middleware/grpc/gprc_server.go:47
google.golang.org/grpc/health/grpc_health_v1._Health_Check_Handler
	/go/pkg/mod/google.golang.org/[email protected]/health/grpc_health_v1/health_grpc.pb.go:165
google.golang.org/grpc.(*Server).processUnaryRPC
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1336
google.golang.org/grpc.(*Server).handleStream
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/go/pkg/mod/google.golang.org/[email protected]/server.go:965
2023-10-23T09:24:57Z	ERROR	Postgres	executing checkRole error	{"error": "failed to connect to `host=localhost user=postgres database=postgres`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)"}
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).CheckRoleOps
	/src/lorry/binding/base.go:231
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).Invoke
	/src/lorry/binding/base.go:194
github.com/apecloud/kubeblocks/lorry/middleware/probe.route
	/src/lorry/middleware/probe/router.go:170
github.com/apecloud/kubeblocks/lorry/middleware/grpc.NewGRPCServer.GetGrpcRouter.func1
	/src/lorry/middleware/probe/router.go:176
github.com/apecloud/kubeblocks/lorry/middleware/grpc.(*GRPCServer).Check
	/src/lorry/middleware/grpc/gprc_server.go:47
google.golang.org/grpc/health/grpc_health_v1._Health_Check_Handler
	/go/pkg/mod/google.golang.org/[email protected]/health/grpc_health_v1/health_grpc.pb.go:165
google.golang.org/grpc.(*Server).processUnaryRPC
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1336
google.golang.org/grpc.(*Server).handleStream
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/go/pkg/mod/google.golang.org/[email protected]/server.go:965
2023-10-23T09:24:57Z	INFO	Postgres	operation called	{"operation": "checkRole", "result": {"event":"Failed","message":"failed to connect to `host=localhost user=postgres database=postgres`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)","operation":"checkRole","originalRole":""}}
2023-10-23T09:24:57Z	INFO	grpc	No event detected	{"meta": "{\"duration\":\"800.916µs\",\"end-time\":\"2023-10-23T09:24:57.482192184Z\",\"operation\":\"checkRole\",\"start-time\":\"2023-10-23T09:24:57.481391276Z\"}", "data": "{\"event\":\"Failed\",\"message\":\"failed to connect to `host=localhost user=postgres database=postgres`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)\",\"operation\":\"checkRole\",\"originalRole\":\"\"}"}
2023-10-23T09:24:58Z	INFO	grpc	role probe request	{"type": "postgresql"}
2023-10-23T09:24:58Z	ERROR	Postgres	query sql:select pg_is_in_recovery(); failed	{"error": "failed to connect to `host=localhost user=postgres database=postgres`: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)"}
github.com/apecloud/kubeblocks/lorry/component/postgres.(*Manager).QueryWithHost
	/src/lorry/component/postgres/query.go:49
github.com/apecloud/kubeblocks/lorry/component/postgres/officalpostgres.(*Manager).GetMemberRoleWithHost
	/src/lorry/component/postgres/officalpostgres/manager.go:199
github.com/apecloud/kubeblocks/lorry/binding/postgres.(*PostgresOperations).GetRole
	/src/lorry/binding/postgres/postgres.go:150
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).CheckRoleOps
	/src/lorry/binding/base.go:229
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).Invoke
	/src/lorry/binding/base.go:194
github.com/apecloud/kubeblocks/lorry/middleware/probe.route
	/src/lorry/middleware/probe/router.go:170
github.com/apecloud/kubeblocks/lorry/middleware/grpc.NewGRPCServer.GetGrpcRouter.func1
	/src/lorry/middleware/probe/router.go:176
github.com/apecloud/kubeblocks/lorry/middleware/grpc.(*GRPCServer).Check
	/src/lorry/middleware/grpc/gprc_server.go:47
google.golang.org/grpc/health/grpc_health_v1._Health_Check_Handler
	/go/pkg/mod/google.golang.org/[email protected]/health/grpc_health_v1/health_grpc.pb.go:165
google.golang.org/grpc.(*Server).processUnaryRPC
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1336
google.golang.org/grpc.(*Server).handleStream
	/go/pkg/mod/google.golang.org/[email protected]/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/go/pkg/mod/google.golang.org/[email protected]/server.go:965
...

Expected behavior
official postgresql cluster multiple replicas create success.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
@JashBook JashBook added kind/bug Something isn't working severity/major Great chance user will encounter the same problem labels Oct 23, 2023
@JashBook JashBook added this to the Release 0.7.0 milestone Oct 23, 2023
@kizuna-lek kizuna-lek linked a pull request Oct 25, 2023 that will close this issue
fix: remove gosu for pg basebackup #5607
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kizuna-lek kizuna-lek

Labels
bug ci kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Projects
None yet
Milestone
Release 0.7.0
Development

Successfully merging a pull request may close this issue.

fix: remove gosu for pg basebackup apecloud/kubeblocks
3 participants
@kizuna-lek @JashBook @apecloud-bot