Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]Mysql smartengine cluster hscale 4->3 not work for unexpected end of JSON input #5560

Closed
ahjing99 opened this issue Oct 20, 2023 · 1 comment
Closed

[BUG]Mysql smartengine cluster hscale 4->3 not work for unexpected end of JSON input #5560

ahjing99 opened this issue Oct 20, 2023 · 1 comment
Assignees
@derecknowayback
Labels
bug kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Milestone
Release 0.7.0

Comments

@ahjing99
Copy link
Collaborator

➜ ~ kbcli version
Kubernetes: v1.27.3-gke.100
KubeBlocks: 0.7.0-beta.7
kbcli: 0.7.0-beta.7

  1. Create cluster
- name: mysql
  componentDefRef: mysql
  replicas: 3
  resources:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      cpu: 500m
      memory: 1Gi
  volumeClaimTemplates:
    - name: data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 20Gi

      `kbcli cluster create  smarte-awtarr             --termination-policy=DoNotTerminate             --monitoring-interval=0 --enable-all-logs=false --cluster-definition=apecloud-mysql --cluster-version=ac-mysql-8.0.30 --set-file=test_create_smarte-awtarr.yaml  --namespace default `

Cluster smarte-awtarr created

 `kbcli cluster configure smarte-awtarr --auto-approve                 --set loose_smartengine=ON,binlog_format=ROW,default_storage_engine=smartengine --component mysql --config-spec mysql-consensusset-config  --namespace default `

Will updated configure file meta:
  ConfigSpec: mysql-consensusset-config	  ConfigFile: my.cnf	ComponentName: mysql	ClusterName: smarte-awtarr
OpsRequest smarte-awtarr-reconfiguring-pjmql created successfully, you can view the progress:
	kbcli cluster describe-ops smarte-awtarr-reconfiguring-pjmql -n default
  1. Hscale out
      `kbcli cluster hscale smarte-awtarr --auto-approve --components mysql --replicas 4 --namespace default `

OpsRequest smarte-awtarr-horizontalscaling-v8c74 created successfully, you can view the progress:
	kbcli cluster describe-ops smarte-awtarr-horizontalscaling-v8c74 -n default


      `kbcli cluster list-instances smarte-awtarr --namespace default `

NAME                    NAMESPACE   CLUSTER         COMPONENT   STATUS    ROLE       ACCESSMODE   AZ              CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE     NODE                                                  CREATED-TIME
smarte-awtarr-mysql-0   default     smarte-awtarr   mysql       Running   follower   <none>       us-central1-c   500m / 500m          1Gi / 1Gi               data:20Gi   gke-yjtest-default-pool-ff74a9b4-40mr/10.128.15.213   Oct 20,2023 11:43 UTC+0800
smarte-awtarr-mysql-1   default     smarte-awtarr   mysql       Running   leader     <none>       us-central1-c   500m / 500m          1Gi / 1Gi               data:20Gi   gke-yjtest-default-pool-ff74a9b4-dh81/10.128.15.214   Oct 20,2023 11:44 UTC+0800
smarte-awtarr-mysql-2   default     smarte-awtarr   mysql       Running   follower   <none>       us-central1-c   500m / 500m          1Gi / 1Gi               data:20Gi   gke-yjtest-default-pool-ff74a9b4-w1bp/10.128.15.211   Oct 20,2023 11:45 UTC+0800
smarte-awtarr-mysql-3   default     smarte-awtarr   mysql       Running   follower   <none>       us-central1-c   500m / 500m          1Gi / 1Gi               data:20Gi   gke-yjtest-default-pool-ff74a9b4-dh81/10.128.15.214   Oct 20,2023 11:49 UTC+0800
  1. Hscale in not work
   `kbcli cluster hscale smarte-awtarr --auto-approve --components mysql --replicas 3 --namespace default `

OpsRequest smarte-awtarr-horizontalscaling-xm66n created successfully, you can view the progress:
	kbcli cluster describe-ops smarte-awtarr-horizontalscaling-xm66n -n default

➜  ~ k describe cluster smarte-awtarr
Name:         smarte-awtarr
Namespace:    default
Labels:       app.kubernetes.io/instance=smarte-awtarr
              clusterdefinition.kubeblocks.io/name=apecloud-mysql
              clusterversion.kubeblocks.io/name=ac-mysql-8.0.30
Annotations:  kubeblocks.io/ops-request: [{"name":"smarte-awtarr-horizontalscaling-xm66n","type":"HorizontalScaling"}]
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2023-10-20T03:40:18Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:  3
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          f:app.kubernetes.io/instance:
      f:spec:
        .:
        f:affinity:
          .:
          f:podAntiAffinity:
          f:tenancy:
        f:clusterDefinitionRef:
        f:clusterVersionRef:
        f:componentSpecs:
          .:
          k:{"name":"mysql"}:
            .:
            f:componentDefRef:
            f:enabledLogs:
              .:
              v:"error":
              v:"general":
              v:"slow":
            f:monitor:
            f:name:
            f:noCreatePDB:
            f:resources:
              .:
              f:limits:
                .:
                f:cpu:
                f:memory:
              f:requests:
                .:
                f:cpu:
                f:memory:
            f:serviceAccountName:
            f:volumeClaimTemplates:
        f:monitor:
        f:resources:
          .:
          f:cpu:
          f:memory:
        f:storage:
          .:
          f:size:
        f:terminationPolicy:
    Manager:      kbcli
    Operation:    Update
    Time:         2023-10-20T03:48:11Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubeblocks.io/ops-request:
        f:finalizers:
          .:
          v:"cluster.kubeblocks.io/finalizer":
        f:labels:
          .:
          f:clusterdefinition.kubeblocks.io/name:
          f:clusterversion.kubeblocks.io/name:
      f:spec:
        f:componentSpecs:
          k:{"name":"mysql"}:
            f:replicas:
    Manager:      manager
    Operation:    Update
    Time:         2023-10-20T03:51:36Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterDefGeneration:
        f:components:
          .:
          f:mysql:
            .:
            f:consensusSetStatus:
              .:
              f:followers:
              f:leader:
                .:
                f:accessMode:
                f:name:
                f:pod:
            f:membersStatus:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
        f:conditions:
        f:observedGeneration:
        f:phase:
    Manager:         manager
    Operation:       Update
    Subresource:     status
    Time:            2023-10-20T03:51:36Z
  Resource Version:  3837828
  UID:               cc708f18-256e-4bef-a419-2397158b19bb
Spec:
  Affinity:
    Pod Anti Affinity:     Preferred
    Tenancy:               SharedNode
  Cluster Definition Ref:  apecloud-mysql
  Cluster Version Ref:     ac-mysql-8.0.30
  Component Specs:
    Component Def Ref:  mysql
    Enabled Logs:
      error
      general
      slow
    Monitor:        false
    Name:           mysql
    No Create PDB:  false
    Replicas:       3
    Resources:
      Limits:
        Cpu:     500m
        Memory:  1Gi
      Requests:
        Cpu:               500m
        Memory:            1Gi
    Service Account Name:  kb-smarte-awtarr
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  20Gi
  Monitor:
  Resources:
    Cpu:     0
    Memory:  0
  Storage:
    Size:              0
  Termination Policy:  DoNotTerminate
Status:
  Cluster Def Generation:  2
  Components:
    Mysql:
      Consensus Set Status:
        Followers:
          Access Mode:  Readonly
          Name:         follower
          Pod:          smarte-awtarr-mysql-3
          Access Mode:  Readonly
          Name:         follower
          Pod:          smarte-awtarr-mysql-2
          Access Mode:  Readonly
          Name:         follower
          Pod:          smarte-awtarr-mysql-0
        Leader:
          Access Mode:  ReadWrite
          Name:         leader
          Pod:          smarte-awtarr-mysql-1
      Members Status:
        Pod Name:  smarte-awtarr-mysql-1
        Role:
          Access Mode:  ReadWrite
          Can Vote:     true
          Is Leader:    true
          Name:         leader
        Pod Name:       smarte-awtarr-mysql-3
        Role:
          Access Mode:  Readonly
          Can Vote:     true
          Is Leader:    false
          Name:         follower
        Pod Name:       smarte-awtarr-mysql-2
        Role:
          Access Mode:  Readonly
          Can Vote:     true
          Is Leader:    false
          Name:         follower
        Pod Name:       smarte-awtarr-mysql-0
        Role:
          Access Mode:  Readonly
          Can Vote:     true
          Is Leader:    false
          Name:         follower
      Phase:            Running
      Pods Ready:       true
      Pods Ready Time:  2023-10-20T03:50:14Z
  Conditions:
    Last Transition Time:  2023-10-20T03:40:18Z
    Message:               The operator has started the provisioning of Cluster: smarte-awtarr
    Observed Generation:   3
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2023-10-20T03:51:36Z
    Message:               unexpected end of JSON input
    Reason:                ApplyResourcesFailed
    Status:                False
    Type:                  ApplyResources
    Last Transition Time:  2023-10-20T03:50:14Z
    Message:               all pods of components are ready, waiting for the probe detection successful
    Reason:                AllReplicasReady
    Status:                True
    Type:                  ReplicasReady
    Last Transition Time:  2023-10-20T03:50:14Z
    Message:               Cluster: smarte-awtarr is ready, current phase is Running
    Reason:                ClusterReady
    Status:                True
    Type:                  Ready
  Observed Generation:     2
  Phase:                   Running
Events:
  Type     Reason                    Age                    From                       Message
  ----     ------                    ----                   ----                       -------
  Normal   ComponentPhaseTransition  20m                    cluster-controller         Create a new component
  Normal   SysAcctCreate             19m                    system-account-controller  Created accounts for cluster: smarte-awtarr, component: mysql, accounts: kbdataprotection
  Normal   SysAcctCreate             19m                    system-account-controller  Created accounts for cluster: smarte-awtarr, component: mysql, accounts: kbadmin
  Normal   SysAcctCreate             19m                    system-account-controller  Created accounts for cluster: smarte-awtarr, component: mysql, accounts: kbmonitoring
  Normal   SysAcctCreate             19m                    system-account-controller  Created accounts for cluster: smarte-awtarr, component: mysql, accounts: kbprobe
  Normal   SysAcctCreate             19m                    system-account-controller  Created accounts for cluster: smarte-awtarr, component: mysql, accounts: kbreplicator
  Warning  ApplyResourcesFailed      16m                    cluster-controller         Operation cannot be fulfilled on pods "smarte-awtarr-mysql-1": the object has been modified; please apply your changes to the latest version and try again
  Normal   ClusterReady              15m (x2 over 19m)      cluster-controller         Cluster: smarte-awtarr is ready, current phase is Running
  Normal   Running                   15m (x2 over 19m)      cluster-controller         Cluster: smarte-awtarr is ready, current phase is Running
  Normal   ComponentPhaseTransition  15m (x2 over 19m)      cluster-controller         component is Running
  Normal   AllReplicasReady          15m (x2 over 19m)      cluster-controller         all pods of components are ready, waiting for the probe detection successful
  Normal   PreCheckSucceed           12m (x2 over 20m)      cluster-controller         The operator has started the provisioning of Cluster: smarte-awtarr
  Normal   ComponentPhaseTransition  12m (x2 over 17m)      cluster-controller         component is Updating
  Warning  ReplicasNotReady          12m (x2 over 17m)      cluster-controller         pods are not ready in Components: [mysql], refer to related component message in Cluster.status.components
  Warning  ComponentsNotReady        12m (x2 over 17m)      cluster-controller         pods are unavailable in Components: [mysql], refer to related component message in Cluster.status.components
  Normal   ApplyResourcesSucceed     12m (x3 over 20m)      cluster-controller         Successfully applied for resources
  Normal   HorizontalScale           11m (x5 over 12m)      cluster-controller         start horizontal scale component mysql of cluster smarte-awtarr from 3 to 4
  Warning  ApplyResourcesFailed      3m27s (x18 over 9m1s)  cluster-controller         unexpected end of JSON input


➜  ~ kbcli cluster describe smarte-awtarr
Name: smarte-awtarr	 Created Time: Oct 20,2023 11:40 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION           STATUS            TERMINATION-POLICY
default     apecloud-mysql       ac-mysql-8.0.30   ConditionsError   DoNotTerminate

Endpoints:
COMPONENT   MODE        INTERNAL                                             EXTERNAL
mysql       ReadWrite   smarte-awtarr-mysql.default.svc.cluster.local:3306   <none>

Topology:
COMPONENT   INSTANCE                ROLE       STATUS    AZ              NODE                                                  CREATED-TIME
mysql       smarte-awtarr-mysql-0   follower   Running   us-central1-c   gke-yjtest-default-pool-ff74a9b4-40mr/10.128.15.213   Oct 20,2023 11:43 UTC+0800
mysql       smarte-awtarr-mysql-1   leader     Running   us-central1-c   gke-yjtest-default-pool-ff74a9b4-dh81/10.128.15.214   Oct 20,2023 11:44 UTC+0800
mysql       smarte-awtarr-mysql-2   follower   Running   us-central1-c   gke-yjtest-default-pool-ff74a9b4-w1bp/10.128.15.211   Oct 20,2023 11:45 UTC+0800
mysql       smarte-awtarr-mysql-3   follower   Running   us-central1-c   gke-yjtest-default-pool-ff74a9b4-dh81/10.128.15.214   Oct 20,2023 11:49 UTC+0800

Resources Allocation:
COMPONENT   DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
mysql       false       500m / 500m          1Gi / 1Gi               data:20Gi      kb-default-sc

Images:
COMPONENT   TYPE    IMAGE
mysql       mysql   infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta2.20230830.g4ca1eb8.13

Show cluster events: kbcli cluster list-events -n default smarte-awtarr

➜  ~    kbcli cluster describe-ops smarte-awtarr-horizontalscaling-xm66n -n default
Spec:
  Name: smarte-awtarr-horizontalscaling-xm66n	NameSpace: default	Cluster: smarte-awtarr	Type: HorizontalScaling

Command:
  kbcli cluster hscale smarte-awtarr --components=mysql --replicas=3 --namespace=default

Last Configuration:
COMPONENT   REPLICAS
mysql              4

Status:
  Start Time:         Oct 20,2023 11:51 UTC+0800
  Duration:           10m
  Status:             Running
  Progress:           -/-

Conditions:
LAST-TRANSITION-TIME         TYPE                REASON                         STATUS   MESSAGE
Oct 20,2023 11:51 UTC+0800   Progressing         OpsRequestProgressingStarted   True     Start to process the OpsRequest: smarte-awtarr-horizontalscaling-xm66n in Cluster: smarte-awtarr
Oct 20,2023 11:51 UTC+0800   Validated           ValidateOpsRequestPassed       True     OpsRequest: smarte-awtarr-horizontalscaling-xm66n is validated
Oct 20,2023 11:51 UTC+0800   HorizontalScaling   HorizontalScalingStarted       True     Start to horizontal scale replicas in Cluster: smarte-awtarr

Warning Events: <none>
@ahjing99 ahjing99 added kind/bug Something isn't working severity/major Great chance user will encounter the same problem labels Oct 20, 2023
@ahjing99 ahjing99 added this to the Release 0.7.0 milestone Oct 20, 2023
@derecknowayback
Copy link
Contributor

Should be fixed with 3915df9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@derecknowayback derecknowayback

Labels
bug kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Projects
None yet
Milestone
Release 0.7.0
Development

No branches or pull requests
3 participants
@derecknowayback @ahjing99 @apecloud-bot