Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]pg 12.15.0 cluster is ConditionsError when upgrade pgvector for failed to get config template cm object #5352

Closed
ahjing99 opened this issue Oct 9, 2023 · 1 comment
Closed

[BUG]pg 12.15.0 cluster is ConditionsError when upgrade pgvector for failed to get config template cm object #5352

ahjing99 opened this issue Oct 9, 2023 · 1 comment
Assignees
@Y-Rookie @JashBook
Labels
bug kind/bug Something isn't working
Milestone
Release 0.7.0

Comments

@ahjing99
Copy link
Collaborator

ahjing99 commented Oct 9, 2023

  1. Install kb 0.6.2, create cluster
➜  ~ cat pg125.yaml
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
  labels:
    clusterdefinition.kubeblocks.io/name: postgresql
    clusterversion.kubeblocks.io/name: postgresql-12.15.0
  generateName: postgres-
  namespace: default
spec:
  affinity:
    nodeLabels: {}
    podAntiAffinity: Preferred
    tenancy: SharedNode
    topologyKeys: []
  clusterDefinitionRef: postgresql
  clusterVersionRef: postgresql-12.15.0
  componentSpecs:
  - componentDefRef: postgresql
    monitor: true
    name: postgresql
    replicas: 1
    resources:
      limits:
        cpu: 1000m
        memory: 1024Mi
      requests:
        cpu: 100m
        memory: 102Mi
    serviceAccountName: dbname
    switchPolicy:
      type: Noop
    volumeClaimTemplates:
    - name: data
      spec:
        storageClassName: standard-rwo
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 3Gi
  terminationPolicy: WipeOut
  tolerations: []

2.hscale/stop/start cluster

   `kbcli cluster list-instances postgres-skpzl --namespace default `

^@NAME                          NAMESPACE   CLUSTER          COMPONENT    STATUS    ROLE        ACCESSMODE   AZ              CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)           STORAGE    NODE                                                CREATED-TIME
postgres-skpzl-postgresql-0   default     postgres-skpzl   postgresql   Running   primary     <none>       us-central1-c   200m / 200m          644245094400m / 644245094400m   data:5Gi   gke-yjtest-default-pool-c51609d3-jp0w/10.128.0.45   Oct 09,2023 14:49 UTC+0800
postgres-skpzl-postgresql-1   default     postgres-skpzl   postgresql   Running   secondary   <none>       us-central1-c   200m / 200m          644245094400m / 644245094400m   data:5Gi   gke-yjtest-default-pool-c51609d3-jx9l/10.128.0.44   Oct 09,2023 14:49 UTC+0800
postgres-skpzl-postgresql-2   default     postgres-skpzl   postgresql   Running   secondary   <none>       us-central1-c   200m / 200m          644245094400m / 644245094400m   data:5Gi   gke-yjtest-default-pool-c51609d3-ss98/10.128.0.46   Oct 09,2023 14:49 UTC+0800
postgres-skpzl-postgresql-3   default     postgres-skpzl   postgresql   Running   secondary   <none>       us-central1-c   200m / 200m          644245094400m / 644245094400m   data:5Gi   gke-yjtest-default-pool-c51609d3-jp0w/10.128.0.45   Oct 09,2023 14:49 UTC+0800
  1. Create cv
create postgresql cluster version postgresql-12.15.0-pgvector-v0.5.0
apiVersion: apps.kubeblocks.io/v1alpha1
kind: ClusterVersion
metadata:
  annotations:
    kubeblocks.io/is-default-cluster-version: "false"
  labels:
    app.kubernetes.io/instance: kb-addon-postgresql
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: postgresql
    app.kubernetes.io/version: 14.7.2
    clusterdefinition.kubeblocks.io/name: postgresql
    config.kubeblocks.io/constraints-postgresql12-cc: postgresql12-cc
    config.kubeblocks.io/tpl-agamotto-configuration: postgresql-agamotto-configuration
    config.kubeblocks.io/tpl-postgresql-configuration: postgresql12-configuration
    config.kubeblocks.io/tpl-postgresql-custom-metrics: postgresql12-custom-metrics
  name: postgresql-12.15.0-pgvector-v0.5.0
spec:
  clusterDefinitionRef: postgresql
  componentVersions:
  - componentDefRef: postgresql
    configSpecs:
    - constraintRef: postgresql12-cc
      defaultMode: 511
      keys:
      - postgresql.conf
      name: postgresql-configuration
      namespace: default
      templateRef: postgresql12-configuration
      volumeName: postgresql-config
    - defaultMode: 511
      name: postgresql-custom-metrics
      namespace: default
      templateRef: postgresql12-custom-metrics
      volumeName: postgresql-custom-metrics
    - defaultMode: 511
      name: agamotto-configuration
      namespace: default
      templateRef: postgresql-agamotto-configuration
      volumeName: agamotto-configuration
    switchoverSpec:
      cmdExecutorConfig:
        image: registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:12.15.0-pgvector-v0.5.0
    systemAccountSpec:
      cmdExecutorConfig:
        image: registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:12.15.0-pgvector-v0.5.0
    versionsContext:
      containers:
      - image: registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:12.15.0-pgvector-v0.5.0
        name: postgresql
        resources: {}
      - image: registry.cn-hangzhou.aliyuncs.com/apecloud/pgbouncer:1.19.0
        name: pgbouncer
        resources: {}
      initContainers:
      - image: registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:12.15.0-pgvector-v0.5.0
        name: pg-init-container
        resources: {}

      `kubectl apply -f postgresql-12.15.0-pgvector-v0.5.0.yaml`

    `kbcli cluster upgrade postgres-skpzl --auto-approve                 --cluster-version=postgresql-12.15.0-pgvector-v0.5.0 --namespace default `

OpsRequest postgres-skpzl-upgrade-47wbn created successfully, you can view the progress:
	kbcli cluster describe-ops postgres-skpzl-upgrade-47wbn -n default
  1. cv status is null and pg cluster is ConditionsError, seems the namespace values has not been rendered in cv yaml
➜  ~ k get cv postgresql-12.15.0-pgvector-v0.5.0 -o yaml
apiVersion: apps.kubeblocks.io/v1alpha1
kind: ClusterVersion
metadata:
  annotations:
    kubeblocks.io/is-default-cluster-version: "false"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"ClusterVersion","metadata":{"annotations":{"kubeblocks.io/is-default-cluster-version":"false"},"labels":{"app.kubernetes.io/instance":"kb-addon-postgresql","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"postgresql","app.kubernetes.io/version":"14.7.2","clusterdefinition.kubeblocks.io/name":"postgresql","config.kubeblocks.io/constraints-postgresql12-cc":"postgresql12-cc","config.kubeblocks.io/tpl-agamotto-configuration":"postgresql-agamotto-configuration","config.kubeblocks.io/tpl-postgresql-configuration":"postgresql12-configuration","config.kubeblocks.io/tpl-postgresql-custom-metrics":"postgresql12-custom-metrics"},"name":"postgresql-12.15.0-pgvector-v0.5.0"},"spec":{"clusterDefinitionRef":"postgresql","componentVersions":[{"componentDefRef":"postgresql","configSpecs":[{"constraintRef":"postgresql12-cc","defaultMode":511,"keys":["postgresql.conf"],"name":"postgresql-configuration","namespace":"default","templateRef":"postgresql12-configuration","volumeName":"postgresql-config"},{"defaultMode":511,"name":"postgresql-custom-metrics","namespace":"default","templateRef":"postgresql12-custom-metrics","volumeName":"postgresql-custom-metrics"},{"defaultMode":511,"name":"agamotto-configuration","namespace":"default","templateRef":"postgresql-agamotto-configuration","volumeName":"agamotto-configuration"}],"switchoverSpec":{"cmdExecutorConfig":{"image":"registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:12.15.0-pgvector-v0.5.0"}},"systemAccountSpec":{"cmdExecutorConfig":{"image":"registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:12.15.0-pgvector-v0.5.0"}},"versionsContext":{"containers":[{"image":"registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:12.15.0-pgvector-v0.5.0","name":"postgresql","resources":{}},{"image":"registry.cn-hangzhou.aliyuncs.com/apecloud/pgbouncer:1.19.0","name":"pgbouncer","resources":{}}],"initContainers":[{"image":"registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:12.15.0-pgvector-v0.5.0","name":"pg-init-container","resources":{}}]}}]}}
  creationTimestamp: "2023-10-09T06:50:52Z"
  finalizers:
  - clusterversion.kubeblocks.io/finalizer
  generation: 1
  labels:
    app.kubernetes.io/instance: kb-addon-postgresql
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: postgresql
    app.kubernetes.io/version: 14.7.2
    clusterdefinition.kubeblocks.io/name: postgresql
    config.kubeblocks.io/constraints-postgresql12-cc: postgresql12-cc
    config.kubeblocks.io/tpl-agamotto-configuration: postgresql-agamotto-configuration
    config.kubeblocks.io/tpl-postgresql-configuration: postgresql12-configuration
    config.kubeblocks.io/tpl-postgresql-custom-metrics: postgresql12-custom-metrics
  name: postgresql-12.15.0-pgvector-v0.5.0
  resourceVersion: "210824"
  uid: 0ae48060-4ca8-4293-9dbe-be72ed71d757
spec:
  clusterDefinitionRef: postgresql
  componentVersions:
  - componentDefRef: postgresql
    configSpecs:
    - constraintRef: postgresql12-cc
      defaultMode: 511
      keys:
      - postgresql.conf
      name: postgresql-configuration
      namespace: default
      templateRef: postgresql12-configuration
      volumeName: postgresql-config
    - defaultMode: 511
      name: postgresql-custom-metrics
      namespace: default
      templateRef: postgresql12-custom-metrics
      volumeName: postgresql-custom-metrics
    - defaultMode: 511
      name: agamotto-configuration
      namespace: default
      templateRef: postgresql-agamotto-configuration
      volumeName: agamotto-configuration
    switchoverSpec:
      cmdExecutorConfig:
        image: registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:12.15.0-pgvector-v0.5.0
    systemAccountSpec:
      cmdExecutorConfig:
        image: registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:12.15.0-pgvector-v0.5.0
    versionsContext:
      containers:
      - image: registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:12.15.0-pgvector-v0.5.0
        name: postgresql
        resources: {}
      - image: registry.cn-hangzhou.aliyuncs.com/apecloud/pgbouncer:1.19.0
        name: pgbouncer
        resources: {}
      initContainers:
      - image: registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:12.15.0-pgvector-v0.5.0
        name: pg-init-container
        resources: {}

➜  ~ kbcli cluster describe postgres-skpzl
Name: postgres-skpzl	 Created Time: Oct 09,2023 14:11 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION                              STATUS            TERMINATION-POLICY
default     postgresql           postgresql-12.15.0-pgvector-v0.5.0   ConditionsError   WipeOut

Endpoints:
COMPONENT    MODE        INTERNAL                                                   EXTERNAL
postgresql   ReadWrite   postgres-skpzl-postgresql.default.svc.cluster.local:5432   <none>
                         postgres-skpzl-postgresql.default.svc.cluster.local:6432

Topology:
COMPONENT    INSTANCE                      ROLE        STATUS    AZ              NODE                                                CREATED-TIME
postgresql   postgres-skpzl-postgresql-0   primary     Running   us-central1-c   gke-yjtest-default-pool-c51609d3-jp0w/10.128.0.45   Oct 09,2023 14:49 UTC+0800
postgresql   postgres-skpzl-postgresql-1   secondary   Running   us-central1-c   gke-yjtest-default-pool-c51609d3-jx9l/10.128.0.44   Oct 09,2023 14:49 UTC+0800
postgresql   postgres-skpzl-postgresql-2   secondary   Running   us-central1-c   gke-yjtest-default-pool-c51609d3-ss98/10.128.0.46   Oct 09,2023 14:49 UTC+0800
postgresql   postgres-skpzl-postgresql-3   secondary   Running   us-central1-c   gke-yjtest-default-pool-c51609d3-jp0w/10.128.0.45   Oct 09,2023 14:49 UTC+0800

Resources Allocation:
COMPONENT    DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)           STORAGE-SIZE   STORAGE-CLASS
postgresql   false       200m / 200m          644245094400m / 644245094400m   data:5Gi       standard-rwo

Images:
COMPONENT    TYPE         IMAGE
postgresql   postgresql   registry.cn-hangzhou.aliyuncs.com/apecloud/spilo:12.15.0-pgvector-v0.5.0

Data Protection:
AUTO-BACKUP   BACKUP-SCHEDULE   TYPE     BACKUP-TTL   LAST-SCHEDULE   RECOVERABLE-TIME
Disabled      <none>            <none>   7d           <none>          <none>

Show cluster events: kbcli cluster list-events -n default postgres-skpzl

kb logs


2023-10-09T07:24:01.431Z	ERROR	failed to validate config template!	{"controller": "clusterversion", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ClusterVersion", "ClusterVersion": {"name":"postgresql-12.15.0-pgvector-v0.5.0"}, "namespace": "", "name": "postgresql-12.15.0-pgvector-v0.5.0", "reconcileID": "63431e9b-7f56-47ca-8322-6f64bced8ad1", "clusterDefinition": "/postgresql-12.15.0-pgvector-v0.5.0", "templateName": "postgresql-configuration", "configMapName": "postgresql12-configuration", "error": "configmaps \"postgresql12-configuration\" not found"}
github.com/apecloud/kubeblocks/controllers/apps/configuration.validateConfigTemplate
	/src/controllers/apps/configuration/config_util.go:348
github.com/apecloud/kubeblocks/controllers/apps/configuration.checkConfigTemplate.func1
	/src/controllers/apps/configuration/config_util.go:298
github.com/apecloud/kubeblocks/controllers/apps/configuration.handleConfigTemplate
	/src/controllers/apps/configuration/config_util.go:256
github.com/apecloud/kubeblocks/controllers/apps/configuration.checkConfigTemplate
	/src/controllers/apps/configuration/config_util.go:300
github.com/apecloud/kubeblocks/controllers/apps/configuration.ReconcileConfigSpecsForReferencedCR[...]
	/src/controllers/apps/configuration/config_util.go:105
github.com/apecloud/kubeblocks/controllers/apps.(*ClusterVersionReconciler).Reconcile
	/src/controllers/apps/clusterversion_controller.go:134
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235

➜ ~ k logs kubeblocks-9b454c778-s48mb -n kb-system >kblog.txt
Defaulted container "manager" out of: manager, tools (init), datascript (init)
kblog.txt
@ahjing99 ahjing99 added the kind/bug Something isn't working label Oct 9, 2023
@JashBook
Copy link
Collaborator

JashBook commented Oct 9, 2023

fix via https://github.com/apecloud/testinfra/pull/381

@JashBook JashBook closed this as completed Oct 9, 2023
@github-actions github-actions bot added this to the Release 0.7.0 milestone Oct 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Y-Rookie Y-Rookie

@JashBook JashBook

Labels
bug kind/bug Something isn't working
Projects
None yet
Milestone
Release 0.7.0
Development

No branches or pull requests
4 participants
@Y-Rookie @ahjing99 @JashBook @apecloud-bot