Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]Mysql Hscale not work on huaweiCloud #5408

Closed
ahjing99 opened this issue Oct 12, 2023 · 2 comments
Closed

[BUG]Mysql Hscale not work on huaweiCloud #5408

ahjing99 opened this issue Oct 12, 2023 · 2 comments
Assignees
@ldming
Labels
bug kind/bug Something isn't working
Milestone
Release 0.7.0

Comments

@ahjing99
Copy link
Collaborator

➜ ~ kbcli version
Kubernetes: v1.25.4-r10-25.2.34.1
KubeBlocks: 0.7.0-beta.1
kbcli: 0.7.0-beta.1

  1. Install kb
kbcli kubeblocks install --set provider=huaweiCloud
  1. create backuprepo
➜  ~ k get backuprepo
NAME                    STATUS   STORAGEPROVIDER   DEFAULT   AGE
backuprepo-kbcli-test   Ready    s3                true      25m
  1. Create cluster

      `kbcli cluster create  mysql-zkcunx             --termination-policy=WipeOut             --monitoring-interval=0 --enable-all-logs=false --cluster-definition=apecloud-mysql --cluster-version=ac-mysql-8.0.30 --set cpu=100m,memory=0.5Gi,replicas=1,storage=1Gi  --namespace default `

Cluster mysql-zkcunx created

      `kbcli cluster list-instances mysql-zkcunx --namespace default `

NAME                   NAMESPACE   CLUSTER        COMPONENT   STATUS    ROLE     ACCESSMODE   AZ           CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE    NODE                        CREATED-TIME
mysql-zkcunx-mysql-0   default     mysql-zkcunx   mysql       Running   leader   <none>       cn-east-3a   100m / 100m          512Mi / 512Mi           data:1Gi   192.168.0.33/192.168.0.33   Oct 12,2023 15:22 UTC+0800
  1. Hscale
 `kbcli cluster hscale mysql-zkcunx --auto-approve --components mysql --replicas 2 --namespace default `

OpsRequest mysql-zkcunx-horizontalscaling-tsvsc created successfully, you can view the progress:
	kbcli cluster describe-ops mysql-zkcunx-horizontalscaling-tsvsc -n default

➜  ~ kbcli cluster describe-ops mysql-zkcunx-horizontalscaling-tsvsc -n default
Spec:
  Name: mysql-zkcunx-horizontalscaling-tsvsc	NameSpace: default	Cluster: mysql-zkcunx	Type: HorizontalScaling

Command:
  kbcli cluster hscale mysql-zkcunx --components=mysql --replicas=2 --namespace=default

Last Configuration:
COMPONENT   REPLICAS
mysql              1

Status:
  Start Time:         Oct 12,2023 15:37 UTC+0800
  Duration:           11m
  Status:             Running
  Progress:           -/-

Conditions:
LAST-TRANSITION-TIME         TYPE                REASON                         STATUS   MESSAGE
Oct 12,2023 15:37 UTC+0800   Progressing         OpsRequestProgressingStarted   True     Start to process the OpsRequest: mysql-zkcunx-horizontalscaling-tsvsc in Cluster: mysql-zkcunx
Oct 12,2023 15:37 UTC+0800   Validated           ValidateOpsRequestPassed       True     OpsRequest: mysql-zkcunx-horizontalscaling-tsvsc is validated
Oct 12,2023 15:37 UTC+0800   HorizontalScaling   HorizontalScalingStarted       True     Start to horizontal scale replicas in Cluster: mysql-zkcunx

Warning Events: <none>

➜  ~ k describe cluster mysql-zkcunx
Name:         mysql-zkcunx
Namespace:    default
Labels:       app.kubernetes.io/instance=mysql-zkcunx
              clusterdefinition.kubeblocks.io/name=apecloud-mysql
              clusterversion.kubeblocks.io/name=ac-mysql-8.0.30
Annotations:  kubeblocks.io/ops-request: [{"name":"mysql-zkcunx-horizontalscaling-tsvsc","type":"HorizontalScaling"}]
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2023-10-12T07:22:00Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:  7
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          f:app.kubernetes.io/instance:
      f:spec:
        .:
        f:affinity:
          .:
          f:podAntiAffinity:
          f:tenancy:
        f:backup:
          .:
          f:enabled:
          f:method:
          f:pitrEnabled:
          f:retentionPeriod:
        f:clusterDefinitionRef:
        f:clusterVersionRef:
        f:componentSpecs:
          .:
          k:{"name":"mysql"}:
            .:
            f:componentDefRef:
            f:enabledLogs:
              .:
              v:"error":
              v:"general":
              v:"slow":
            f:monitor:
            f:name:
            f:noCreatePDB:
            f:resources:
              .:
              f:limits:
                .:
                f:cpu:
                f:memory:
              f:requests:
                .:
                f:cpu:
                f:memory:
            f:serviceAccountName:
            f:volumeClaimTemplates:
        f:terminationPolicy:
    Manager:      kbcli
    Operation:    Update
    Time:         2023-10-12T07:36:22Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubeblocks.io/ops-request:
        f:finalizers:
          .:
          v:"cluster.kubeblocks.io/finalizer":
        f:labels:
          .:
          f:clusterdefinition.kubeblocks.io/name:
          f:clusterversion.kubeblocks.io/name:
      f:spec:
        f:componentSpecs:
          k:{"name":"mysql"}:
            f:replicas:
        f:monitor:
        f:resources:
          .:
          f:cpu:
          f:memory:
        f:storage:
          .:
          f:size:
    Manager:      manager
    Operation:    Update
    Time:         2023-10-12T07:37:16Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterDefGeneration:
        f:components:
          .:
          f:mysql:
            .:
            f:consensusSetStatus:
              .:
              f:leader:
                .:
                f:accessMode:
                f:name:
                f:pod:
            f:membersStatus:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
        f:conditions:
        f:observedGeneration:
        f:phase:
    Manager:         manager
    Operation:       Update
    Subresource:     status
    Time:            2023-10-12T07:37:16Z
  Resource Version:  43067
  UID:               22f96029-7f8b-4b60-9c08-d6e96ee52c87
Spec:
  Affinity:
    Pod Anti Affinity:  Preferred
    Tenancy:            SharedNode
  Backup:
    Enabled:               false
    Method:
    Pitr Enabled:          false
    Retention Period:      7d
  Cluster Definition Ref:  apecloud-mysql
  Cluster Version Ref:     ac-mysql-8.0.30
  Component Specs:
    Component Def Ref:  mysql
    Enabled Logs:
      error
      general
      slow
    Monitor:        true
    Name:           mysql
    No Create PDB:  false
    Replicas:       2
    Resources:
      Limits:
        Cpu:     100m
        Memory:  512Mi
      Requests:
        Cpu:               100m
        Memory:            512Mi
    Service Account Name:  kb-mysql-zkcunx
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  1Gi
  Monitor:
  Resources:
    Cpu:     0
    Memory:  0
  Storage:
    Size:              0
  Termination Policy:  WipeOut
Status:
  Cluster Def Generation:  2
  Components:
    Mysql:
      Consensus Set Status:
        Leader:
          Access Mode:  ReadWrite
          Name:         leader
          Pod:          mysql-zkcunx-mysql-0
      Members Status:
        Pod Name:  mysql-zkcunx-mysql-0
        Role:
          Access Mode:  ReadWrite
          Can Vote:     true
          Is Leader:    true
          Name:         leader
      Phase:            Running
      Pods Ready:       true
      Pods Ready Time:  2023-10-12T07:37:07Z
  Conditions:
    Last Transition Time:  2023-10-12T07:22:00Z
    Message:               The operator has started the provisioning of Cluster: mysql-zkcunx
    Observed Generation:   7
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2023-10-12T07:37:16Z
    Message:               failed to get API group resources: unable to retrieve the complete list of server APIs: snapshot.storage.k8s.io/v1: the server could not find the requested resource
    Reason:                ApplyResourcesFailed
    Status:                False
    Type:                  ApplyResources
    Last Transition Time:  2023-10-12T07:37:07Z
    Message:               all pods of components are ready, waiting for the probe detection successful
    Reason:                AllReplicasReady
    Status:                True
    Type:                  ReplicasReady
    Last Transition Time:  2023-10-12T07:37:07Z
    Message:               Cluster: mysql-zkcunx is ready, current phase is Running
    Reason:                ClusterReady
    Status:                True
    Type:                  Ready
  Observed Generation:     6
  Phase:                   Running
Events:
  Type     Reason                    Age                 From                       Message
  ----     ------                    ----                ----                       -------
  Normal   ComponentPhaseTransition  27m                 cluster-controller         Create a new component
  Normal   SysAcctCreate             25m                 system-account-controller  Created accounts for cluster: mysql-zkcunx, component: mysql, accounts: kbdataprotection
  Normal   SysAcctCreate             25m                 system-account-controller  Created accounts for cluster: mysql-zkcunx, component: mysql, accounts: kbadmin
  Normal   SysAcctCreate             25m                 system-account-controller  Created accounts for cluster: mysql-zkcunx, component: mysql, accounts: kbreplicator
  Normal   SysAcctCreate             25m                 system-account-controller  Created accounts for cluster: mysql-zkcunx, component: mysql, accounts: kbprobe
  Normal   SysAcctCreate             25m                 system-account-controller  Created accounts for cluster: mysql-zkcunx, component: mysql, accounts: kbmonitoring
  Normal   ComponentPhaseTransition  25m                 cluster-controller         component is Stopping
  Normal   HorizontalScale           25m                 cluster-controller         start horizontal scale component mysql of cluster mysql-zkcunx from 1 to 0
  Normal   Stopped                   25m                 cluster-controller         Cluster: mysql-zkcunx stopped successfully.
  Normal   ComponentPhaseTransition  25m                 cluster-controller         component is Stopped
  Normal   HorizontalScale           24m                 cluster-controller         start horizontal scale component mysql of cluster mysql-zkcunx from 0 to 1
  Warning  ComponentsNotReady        24m                 cluster-controller         pods are unavailable in Components: [mysql], refer to related component message in Cluster.status.components
  Normal   ApplyResourcesSucceed     24m (x4 over 27m)   cluster-controller         Successfully applied for resources
  Normal   PreCheckSucceed           24m (x4 over 27m)   cluster-controller         The operator has started the provisioning of Cluster: mysql-zkcunx
  Normal   Running                   24m (x2 over 26m)   cluster-controller         Cluster: mysql-zkcunx is ready, current phase is Running
  Normal   ClusterReady              24m (x2 over 26m)   cluster-controller         Cluster: mysql-zkcunx is ready, current phase is Running
  Normal   AllReplicasReady          24m (x2 over 26m)   cluster-controller         all pods of components are ready, waiting for the probe detection successful
  Normal   ComponentPhaseTransition  16m (x3 over 24m)   cluster-controller         component is Updating
  Warning  ReplicasNotReady          16m (x3 over 24m)   cluster-controller         pods are not ready in Components: [mysql], refer to related component message in Cluster.status.components
  Normal   ComponentPhaseTransition  12m (x6 over 26m)   cluster-controller         component is Running
  Warning  ApplyResourcesFailed      56s (x19 over 11m)  cluster-controller         failed to get API group resources: unable to retrieve the complete list of server APIs: snapshot.storage.k8s.io/v1: the server could not find the requested resource

➜  ~  kubectl  get crd|grep -i snapshot
[volumesnapshotclasses.snapshot.storage.k8s.io](http://volumesnapshotclasses.snapshot.storage.k8s.io/)     2023-10-12T06:24:08Z
[volumesnapshotcontents.snapshot.storage.k8s.io](http://volumesnapshotcontents.snapshot.storage.k8s.io/)    2023-10-12T06:24:08Z
[volumesnapshots.snapshot.storage.k8s.io](http://volumesnapshots.snapshot.storage.k8s.io/)           2023-10-12T06:24:08Z

➜ ~ k logs kubeblocks-7984446cd-dxf7f -n kb-system > kblog.txt
Defaulted container "manager" out of: manager, tools (init), datascript (init)
kblog.txt
@ahjing99 ahjing99 added the kind/bug Something isn't working label Oct 12, 2023
@ahjing99 ahjing99 added this to the Release 0.7.0 milestone Oct 12, 2023
@ldming
Copy link
Collaborator

ldming commented Oct 12, 2023 •
edited
Loading

HuaweiCloud install the v1beta1 snapshot.storage.k8s.io, KubeBlocks code should use the volume snapshot client that is compatible with VolumeSnapshot v1 and v1beta1.

yud9YoyBAv

Should already been resolved by d4c4799

@wangyelei
Copy link
Contributor

fixed at #5391

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ldming ldming

Labels
bug kind/bug Something isn't working
Projects
None yet
Milestone
Release 0.7.0
Development

No branches or pull requests
4 participants
@ldming @wangyelei @ahjing99 @apecloud-bot