Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] oceanbase cluster status is always Updating after Reconfiguring #6989

Closed
JashBook opened this issue Apr 9, 2024 · 2 comments
Closed

[BUG] oceanbase cluster status is always Updating after Reconfiguring #6989

JashBook opened this issue Apr 9, 2024 · 2 comments
Assignees
@sophon-zt
Labels
kind/bug Something isn't working severity/major Great chance user will encounter the same problem Stale
Milestone
Release 0.9.0

Comments

@JashBook
Copy link
Collaborator

JashBook commented Apr 9, 2024

Describe the bug
oceanbase ce and ent have the same problem

To Reproduce
Steps to reproduce the behavior:

  1. create cluster
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
  name: obce-rbklwy
  namespace: default
  annotations:
    "kubeblocks.io/extra-env": '{"TENANT_NAME":"tenant1","ZONE_COUNT":"1","OB_CLUSTERS_COUNT":"1","TENANT_CPU":"2","TENANT_MEMORY":"2G","TENANT_DISK":"5G"}'
spec:
  terminationPolicy: Halt
  componentSpecs:
    - name: ob-ce
      componentDefRef: ob-ce
      componentDef: ob-ce
      replicas: 1
      resources:
        requests:
          cpu: 3000m
          memory: 8Gi
        limits:
          cpu: 3000m
          memory: 8Gi
      volumeClaimTemplates:
        - name: data-file
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: "50Gi"
        - name: data-log
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: "50Gi"
        - name: log
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: "10Gi"
        - name: workdir
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: "100Mi"
  1. config
kbcli cluster configure obce-rbklwy --auto-approve --set "system_memory=2G" --components ob-ce --config-spec ob-ce-config
  1. See error
➜  ~ kubectl get cluster obce-rbklwy
NAME          CLUSTER-DEFINITION   VERSION   TERMINATION-POLICY   STATUS     AGE
obce-rbklwy                                  Halt                 Updating   33m

➜  ~ kubectl get pod,ops -l app.kubernetes.io/instance=obce-rbklwy                           
NAME                      READY   STATUS    RESTARTS   AGE
pod/obce-rbklwy-ob-ce-0   2/2     Running   0          33m

NAME                                                            TYPE            CLUSTER       STATUS    PROGRESS   AGE
opsrequest.apps.kubeblocks.io/obce-rbklwy-reconfiguring-w496p   Reconfiguring   obce-rbklwy   Running   -/-        26m

describe cluster

kubectl describe cluster obce-rbklwy
Name:         obce-rbklwy
Namespace:    default
Labels:       app.kubernetes.io/instance=obce-rbklwy
Annotations:  kubeblocks.io/extra-env:
                {"TENANT_NAME":"tenant1","ZONE_COUNT":"1","OB_CLUSTERS_COUNT":"1","TENANT_CPU":"2","TENANT_MEMORY":"2G","TENANT_DISK":"5G"}
              kubeblocks.io/ops-request: [{"name":"obce-rbklwy-reconfiguring-w496p","type":"Reconfiguring"}]
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2024-04-09T02:31:27Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:  3
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubeblocks.io/extra-env:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:terminationPolicy:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2024-04-09T02:31:27Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .:
          f:app.kubernetes.io/instance:
    Manager:      kbcli
    Operation:    Update
    Time:         2024-04-09T02:32:35Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:kubeblocks.io/ops-request:
        f:finalizers:
          .:
          v:"cluster.kubeblocks.io/finalizer":
      f:spec:
        f:componentSpecs:
        f:monitor:
        f:resources:
          .:
          f:cpu:
          f:memory:
        f:storage:
          .:
          f:size:
    Manager:      manager
    Operation:    Update
    Time:         2024-04-09T02:38:46Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:components:
          .:
          f:ob-ce:
            .:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
        f:conditions:
        f:observedGeneration:
        f:phase:
    Manager:         manager
    Operation:       Update
    Subresource:     status
    Time:            2024-04-09T02:38:46Z
  Resource Version:  317658074
  UID:               edf5ae0b-d8fe-402e-b7dd-949edc9d7772
Spec:
  Component Specs:
    Component Def:      ob-ce
    Component Def Ref:  ob-ce
    Monitor:            false
    Name:               ob-ce
    Replicas:           1
    Resources:
      Limits:
        Cpu:     3
        Memory:  8Gi
      Requests:
        Cpu:          3
        Memory:       8Gi
    Service Version:  4.2.0
    Volume Claim Templates:
      Name:  data-file
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  50Gi
      Name:           data-log
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  50Gi
      Name:           log
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  10Gi
      Name:           workdir
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  100Mi
  Monitor:
  Resources:
    Cpu:     0
    Memory:  0
  Storage:
    Size:              0
  Termination Policy:  Halt
Status:
  Components:
    Ob - Ce:
      Phase:            Updating
      Pods Ready:       false
      Pods Ready Time:  2024-04-09T02:32:03Z
  Conditions:
    Last Transition Time:  2024-04-09T02:31:27Z
    Message:               The operator has started the provisioning of Cluster: obce-rbklwy
    Observed Generation:   3
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2024-04-09T02:31:27Z
    Message:               Successfully applied for resources
    Observed Generation:   3
    Reason:                ApplyResourcesSucceed
    Status:                True
    Type:                  ApplyResources
    Last Transition Time:  2024-04-09T02:38:46Z
    Message:               pods are not ready in Components: [ob-ce], refer to related component message in Cluster.status.components
    Reason:                ReplicasNotReady
    Status:                False
    Type:                  ReplicasReady
    Last Transition Time:  2024-04-09T02:38:46Z
    Message:               pods are unavailable in Components: [ob-ce], refer to related component message in Cluster.status.components
    Reason:                ComponentsNotReady
    Status:                False
    Type:                  Ready
  Observed Generation:     3
  Phase:                   Updating
Events:
  Type     Reason                    Age                 From                       Message
  ----     ------                    ----                ----                       -------
  Normal   ComponentPhaseTransition  34m                 cluster-controller         component is Creating
  Normal   Running                   33m                 cluster-controller         Cluster: obce-rbklwy is ready, current phase is Running
  Normal   ComponentPhaseTransition  33m                 cluster-controller         component is Running
  Normal   AllReplicasReady          33m                 cluster-controller         all pods of components are ready, waiting for the probe detection successful
  Normal   ClusterReady              33m                 cluster-controller         Cluster: obce-rbklwy is ready, current phase is Running
  Normal   ApplyResourcesSucceed     27m (x3 over 34m)   cluster-controller         Successfully applied for resources
  Normal   PreCheckSucceed           27m (x3 over 34m)   cluster-controller         The operator has started the provisioning of Cluster: obce-rbklwy
  Warning  NotFound                  27m (x20 over 33m)  system-account-controller  ClusterDefinition.apps.kubeblocks.io "" not found
  Normal   ComponentPhaseTransition  27m                 cluster-controller         component is Updating
  Warning  ReplicasNotReady          27m                 cluster-controller         pods are not ready in Components: [ob-ce], refer to related component message in Cluster.status.components
  Warning  ComponentsNotReady        27m                 cluster-controller         pods are unavailable in Components: [ob-ce], refer to related component message in Cluster.status.components

kubectl describe opsrequest.apps.kubeblocks.io/obce-rbklwy-reconfiguring-w496p
Name:         obce-rbklwy-reconfiguring-w496p
Namespace:    default
Labels:       app.kubernetes.io/instance=obce-rbklwy
              app.kubernetes.io/managed-by=kubeblocks
              ops.kubeblocks.io/ops-type=Reconfiguring
              system_memory=2G
Annotations:  oceanbase.conf: {"system_memory":"1G"}
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         OpsRequest
Metadata:
  Creation Timestamp:  2024-04-09T02:38:46Z
  Finalizers:
    opsrequest.kubeblocks.io/finalizer
  Generate Name:  obce-rbklwy-reconfiguring-
  Generation:     2
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:generateName:
        f:labels:
          .:
          f:app.kubernetes.io/instance:
          f:app.kubernetes.io/managed-by:
      f:spec:
        .:
        f:clusterRef:
        f:reconfigure:
          .:
          f:componentName:
          f:configurations:
            .:
            k:{"name":"ob-ce-config"}:
              .:
              f:keys:
                .:
                k:{"key":"oceanbase.conf"}:
                  .:
                  f:key:
                  f:parameters:
              f:name:
        f:ttlSecondsBeforeAbort:
        f:type:
    Manager:      kbcli
    Operation:    Update
    Time:         2024-04-09T02:38:46Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:oceanbase.conf:
        f:finalizers:
          .:
          v:"opsrequest.kubeblocks.io/finalizer":
        f:labels:
          f:ops.kubeblocks.io/ops-type:
          f:system_memory:
        f:ownerReferences:
          .:
          k:{"uid":"edf5ae0b-d8fe-402e-b7dd-949edc9d7772"}:
    Manager:      manager
    Operation:    Update
    Time:         2024-04-09T02:38:46Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterGeneration:
        f:conditions:
          .:
          k:{"type":"Reconfigure"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
          k:{"type":"Validated"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
          k:{"type":"WaitForProgressing"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
        f:phase:
        f:progress:
        f:reconfiguringStatus:
          .:
          f:conditions:
            .:
            k:{"type":"Pending"}:
              .:
              f:lastTransitionTime:
              f:message:
              f:reason:
              f:status:
              f:type:
            k:{"type":"ReconfigurePersisted"}:
              .:
              f:lastTransitionTime:
              f:message:
              f:reason:
              f:status:
              f:type:
          f:configurationStatus:
            .:
            k:{"name":"ob-ce-config"}:
              .:
              f:expectedCount:
              f:name:
              f:status:
              f:succeedCount:
              f:updatedParameters:
                .:
                f:updatedKeys:
                  .:
                  f:oceanbase.conf:
        f:startTimestamp:
    Manager:      manager
    Operation:    Update
    Subresource:  status
    Time:         2024-04-09T02:38:46Z
  Owner References:
    API Version:     apps.kubeblocks.io/v1alpha1
    Kind:            Cluster
    Name:            obce-rbklwy
    UID:             edf5ae0b-d8fe-402e-b7dd-949edc9d7772
  Resource Version:  317658069
  UID:               04219001-c71c-40fb-9849-36e591061692
Spec:
  Cluster Ref:  obce-rbklwy
  Reconfigure:
    Component Name:  ob-ce
    Configurations:
      Keys:
        Key:  oceanbase.conf
        Parameters:
          Key:               system_memory
          Value:             2G
      Name:                  ob-ce-config
  Ttl Seconds Before Abort:  0
  Type:                      Reconfiguring
Status:
  Cluster Generation:  3
  Conditions:
    Last Transition Time:  2024-04-09T02:38:46Z
    Message:               wait for the controller to process the OpsRequest: obce-rbklwy-reconfiguring-w496p in Cluster: obce-rbklwy
    Reason:                WaitForProgressing
    Status:                True
    Type:                  WaitForProgressing
    Last Transition Time:  2024-04-09T02:38:46Z
    Message:               OpsRequest: obce-rbklwy-reconfiguring-w496p is validated
    Reason:                ValidateOpsRequestPassed
    Status:                True
    Type:                  Validated
    Last Transition Time:  2024-04-09T02:38:46Z
    Message:               Start to reconfigure in Cluster: obce-rbklwy, Component: ob-ce
    Reason:                ReconfigureStarted
    Status:                True
    Type:                  Reconfigure
  Phase:                   Running
  Progress:                -/-
  Reconfiguring Status:
    Conditions:
      Last Transition Time:  2024-04-09T02:38:46Z
      Message:               Reconfiguring in Cluster: obce-rbklwy, Component: ob-ce, ConfigSpec: ob-ce-config, info: updated: map[oceanbase.conf:{"system_memory":"2G"}], added: map[], deleted:map[]
      Reason:                ReconfigurePersisted
      Status:                True
      Type:                  ReconfigurePersisted
      Last Transition Time:  2024-04-09T02:38:46Z
      Message:               Reconfiguring in Cluster: obce-rbklwy, Component: ob-ce, ConfigSpec: ob-ce-config
      Reason:                Pending
      Status:                True
      Type:                  Pending
    Configuration Status:
      Expected Count:  -1
      Name:            ob-ce-config
      Status:          ReconfigurePersisted
      Succeed Count:   0
      Updated Parameters:
        Updated Keys:
          oceanbase.conf:  {"system_memory":"2G"}
  Start Timestamp:         2024-04-09T02:38:46Z
Events:
  Type    Reason                    Age   From                    Message
  ----    ------                    ----  ----                    -------
  Normal  WaitForProgressing        27m   ops-request-controller  wait for the controller to process the OpsRequest: obce-rbklwy-reconfiguring-w496p in Cluster: obce-rbklwy
  Normal  ValidateOpsRequestPassed  27m   ops-request-controller  OpsRequest: obce-rbklwy-reconfiguring-w496p is validated
  Normal  ReconfigureStarted        27m   ops-request-controller  Start to reconfigure in Cluster: obce-rbklwy, Component: ob-ce
  Normal  ReconfigurePersisted      27m   ops-request-controller  the reconfiguring operation of component[ob-ce] in cluster[obce-rbklwy] merged successfully

describe cm

kubectl describe cm obce-rbklwy-ob-ce-ob-ce-config 
Name:         obce-rbklwy-ob-ce-ob-ce-config
Namespace:    default
Labels:       app.kubernetes.io/component=
              app.kubernetes.io/instance=obce-rbklwy
              app.kubernetes.io/managed-by=kubeblocks
              app.kubernetes.io/name=
              apps.kubeblocks.io/component-name=ob-ce
              config.kubeblocks.io/config-constraints-name=ob-ce-parameters-cc
              config.kubeblocks.io/config-hash=7b699fc4c5
              config.kubeblocks.io/config-spec=ob-ce-config
              config.kubeblocks.io/config-template-name=ob-ce-config
              config.kubeblocks.io/config-type=instance
              config.kubeblocks.io/last-applied-reconfigure-phase=created
              config.kubeblocks.io/template-name=ob-ce-config
Annotations:  config.kubeblocks.io/config-applied-version:
                {"name":"ob-ce-config","payload":null,"configSpec":{"name":"ob-ce-config","templateRef":"ob-ce-config","namespace":"kb-system","volumeName...
              config.kubeblocks.io/config-template-version: 
              config.kubeblocks.io/configuration-revision: 2
              config.kubeblocks.io/disable-reconfigure: false
              config.kubeblocks.io/last-applied-configuration:
                {"oceanbase.conf":"\ncpu_count=3\nmemory_limit=8G\nsystem_memory=1G\n__min_full_resource_pool_memory=1073741824\ndatafile_size=43G\nlog_di...
              config.kubeblocks.io/reconfigure-source: manager
              config.kubeblocks.io/revision-reconcile-phase-1:
                {"phase":"Finished","revision":"1","policy":"","execResult":"","succeedCount":0,"expectedCount":-1,"retry":false,"failed":false,"message":...
              config.kubeblocks.io/update-config-hash: b49bb4c8d

Data
====
oceanbase.conf:
----
cpu_count = 3
memory_limit = 8G
system_memory = 2G
__min_full_resource_pool_memory = 1073741824
datafile_size = 43G
log_disk_size = 43G
net_thread_count = 2
stack_size = 512K
cache_wash_threshold = 1G
schema_history_expire_time = 1d
enable_separate_sys_clog = false
enable_merge_by_turn = false
enable_syslog_recycle = true
enable_syslog_wf = false
max_syslog_file_count = 4
mysql_port = 2881
rpc_port = 2882


BinaryData
====

Events:  <none>

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]
kbcli version
Kubernetes: v1.26.14-gke.1006000
KubeBlocks: 0.9.0-beta.4
kbcli: 0.9.0-beta.1

Additional context
Add any other context about the problem here.
@JashBook JashBook added kind/bug Something isn't working severity/major Great chance user will encounter the same problem labels Apr 9, 2024
@JashBook JashBook added this to the Release 0.9.0 milestone Apr 9, 2024
@github-actions GitHub Actions
Copy link

This issue has been marked as stale because it has been open for 30 days with no activity

@github-actions github-actions bot added the Stale label May 13, 2024
@sophon-zt
Copy link
Contributor

related pr: #7027

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sophon-zt sophon-zt

Labels
kind/bug Something isn't working severity/major Great chance user will encounter the same problem Stale
Projects
None yet
Milestone
Release 0.9.0
Development

No branches or pull requests
2 participants
@JashBook @sophon-zt