[BUG]cluster created with cmpd restart not work #7027

ahjing99 · 2024-04-11T10:09:11Z

➜ ~ kbcli version
Kubernetes: v1.27.8-gke.1067004
KubeBlocks: 0.9.0-beta.5
kbcli: 0.9.0-beta.1

Create cluster
kubectl apply -f cluster.yaml

 apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
  name: acmysql-cluster
  namespace: default
spec:
  terminationPolicy: Delete
  componentSpecs:
    - name: mysql
      componentDef: apecloud-mysql
      replicas: 1
      resources:
        limits:
          cpu: "0.5"
          memory: "0.5Gi"
        requests:
          cpu: "0.5"
          memory: "0.5Gi"
      volumeClaimTemplates:
        - name: data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi

ops scceed soon, but the pods does not restart

 ➜  ~ k get pod
NAME                          READY   STATUS    RESTARTS   AGE
acmysql-cluster-mysql-0       4/4     Running   0          6m58s

➜  ~ k describe ops acmysql-restart
Name:         acmysql-restart
Namespace:    default
Labels:       app.kubernetes.io/instance=acmysql-cluster
              ops.kubeblocks.io/ops-type=Restart
Annotations:  <none>
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         OpsRequest
Metadata:
  Creation Timestamp:  2024-04-11T10:02:05Z
  Finalizers:
    opsrequest.kubeblocks.io/finalizer
  Generation:  1
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:clusterRef:
        f:restart:
          .:
          k:{"componentName":"mysql"}:
            .:
            f:componentName:
        f:ttlSecondsBeforeAbort:
        f:type:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2024-04-11T10:02:05Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"opsrequest.kubeblocks.io/finalizer":
        f:labels:
          .:
          f:app.kubernetes.io/instance:
          f:ops.kubeblocks.io/ops-type:
        f:ownerReferences:
          .:
          k:{"uid":"9ac449c0-dd1a-489c-ba74-e0b359f08f6c"}:
    Manager:      manager
    Operation:    Update
    Time:         2024-04-11T10:02:05Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterGeneration:
        f:completionTimestamp:
        f:components:
          .:
          f:mysql:
            .:
            f:phase:
            f:progressDetails:
        f:conditions:
          .:
          k:{"type":"Restarting"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
          k:{"type":"Succeed"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
          k:{"type":"Validated"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
          k:{"type":"WaitForProgressing"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
        f:phase:
        f:progress:
        f:startTimestamp:
    Manager:      manager
    Operation:    Update
    Subresource:  status
    Time:         2024-04-11T10:02:07Z
  Owner References:
    API Version:     apps.kubeblocks.io/v1alpha1
    Kind:            Cluster
    Name:            acmysql-cluster
    UID:             9ac449c0-dd1a-489c-ba74-e0b359f08f6c
  Resource Version:  2789172
  UID:               fcb2c760-9298-4bb9-b8b8-c57b38f248b6
Spec:
  Cluster Ref:  acmysql-cluster
  Restart:
    Component Name:          mysql
  Ttl Seconds Before Abort:  0
  Type:                      Restart
Status:
  Cluster Generation:    3
  Completion Timestamp:  2024-04-11T10:02:07Z
  Components:
    Mysql:
      Phase:  Running
      Progress Details:
        End Time:    2024-04-11T10:02:06Z
        Message:     Successfully restart: Pod/acmysql-cluster-mysql-0 in Component: mysql
        Object Key:  Pod/acmysql-cluster-mysql-0
        Status:      Succeed
  Conditions:
    Last Transition Time:  2024-04-11T10:02:05Z
    Message:               wait for the controller to process the OpsRequest: acmysql-restart in Cluster: acmysql-cluster
    Reason:                WaitForProgressing
    Status:                True
    Type:                  WaitForProgressing
    Last Transition Time:  2024-04-11T10:02:05Z
    Message:               OpsRequest: acmysql-restart is validated
    Reason:                ValidateOpsRequestPassed
    Status:                True
    Type:                  Validated
    Last Transition Time:  2024-04-11T10:02:05Z
    Message:               Start to restart database in Cluster: acmysql-cluster
    Reason:                RestartStarted
    Status:                True
    Type:                  Restarting
    Last Transition Time:  2024-04-11T10:02:07Z
    Message:               Successfully processed the OpsRequest: acmysql-restart in Cluster: acmysql-cluster
    Reason:                OpsRequestProcessedSuccessfully
    Status:                True
    Type:                  Succeed
  Phase:                   Succeed
  Progress:                1/1
  Start Timestamp:         2024-04-11T10:02:05Z
Events:
  Type     Reason                           Age                  From                    Message
  ----     ------                           ----                 ----                    -------
  Normal   WaitForProgressing               6m4s                 ops-request-controller  wait for the controller to process the OpsRequest: acmysql-restart in Cluster: acmysql-cluster
  Normal   ValidateOpsRequestPassed         6m4s (x2 over 6m4s)  ops-request-controller  OpsRequest: acmysql-restart is validated
  Normal   RestartStarted                   6m4s (x2 over 6m4s)  ops-request-controller  Start to restart database in Cluster: acmysql-cluster
  Warning  ReconcileStatusFailed            6m3s                 ops-request-controller  Failed to reconcile the status of OpsRequest: Operation cannot be fulfilled on clusters.apps.kubeblocks.io "acmysql-cluster": the object has been modified; please apply your changes to the latest version and try again
  Normal   Succeed                          6m2s (x2 over 6m3s)  ops-request-controller  Successfully restart: Pod/acmysql-cluster-mysql-0 in Component: mysql
  Normal   OpsRequestProcessedSuccessfully  6m2s (x2 over 6m3s)  ops-request-controller  Successfully processed the OpsRequest: acmysql-restart in Cluster: acmysql-cluster

kb logs:
2024-04-11T10:02:48.046Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"acmysql-cluster","namespace":"default"}, "namespace": "default", "name": "acmysql-cluster", "reconcileID": "20c3b3f8-0c20-4120-a2ba-1007daa8266f", "error": "ClusterDefinition.apps.kubeblocks.io \"\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
2024-04-11T10:03:29.013Z	INFO	ClusterDefinition.apps.kubeblocks.io "" not found	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"acmysql-cluster","namespace":"default"}, "namespace": "default", "name": "acmysql-cluster", "reconcileID": "909d3f31-cf89-44a3-98aa-c4107c7d006c", "cluster": {"name":"acmysql-cluster","namespace":"default"}}
2024-04-11T10:03:29.013Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"acmysql-cluster","namespace":"default"}, "namespace": "default", "name": "acmysql-cluster", "reconcileID": "909d3f31-cf89-44a3-98aa-c4107c7d006c", "error": "ClusterDefinition.apps.kubeblocks.io \"\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
W0411 10:03:37.087412       1 warnings.go:70] Due to the lack of practical use cases, this API is deprecated from KB 0.9.0.
2024-04-11T10:03:42.787Z	INFO	illegal version string "5.0"	{"controller": "componentdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentDefinition", "ComponentDefinition": {"name":"mongodb"}, "namespace": "", "name": "mongodb", "reconcileID": "109d0302-0c76-410a-b1f6-c5811e466045", "component": {"name":"mongodb"}}
2024-04-11T10:03:42.787Z	ERROR	Reconciler error	{"controller": "componentdefinition", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "ComponentDefinition", "ComponentDefinition": {"name":"mongodb"}, "namespace": "", "name": "mongodb", "reconcileID": "109d0302-0c76-410a-b1f6-c5811e466045", "error": "illegal version string \"5.0\""}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
➜  ~

The text was updated successfully, but these errors were encountered:

ahjing99 added kind/bug Something isn't working severity/major Great chance user will encounter the same problem labels Apr 11, 2024

ahjing99 added this to the Release 0.9.0 milestone Apr 11, 2024

ahjing99 assigned free6om Apr 11, 2024

free6om mentioned this issue Apr 15, 2024

fix: restart not working #7049

Merged

free6om closed this as completed in #7049 Apr 15, 2024

This was referenced Apr 15, 2024

[BUG]Pulsar config brokerShutdownTimeoutMs not working #7012

Closed

[BUG]ob distributed cluster configure not work #7023

Closed

sophon-zt mentioned this issue May 15, 2024

[BUG] oceanbase cluster status is always Updating after Reconfiguring #6989

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG]cluster created with cmpd restart not work #7027

[BUG]cluster created with cmpd restart not work #7027

ahjing99 commented Apr 11, 2024

[BUG]cluster created with cmpd restart not work #7027

[BUG]cluster created with cmpd restart not work #7027

Comments

ahjing99 commented Apr 11, 2024