[BUG]Hscale out failed for Failed to set default snapshot class

[ahjing99]

➜ ~ kbcli version
Kubernetes: v1.27.3-gke.100
KubeBlocks: 0.7.0-beta.3
kbcli: 0.7.0-beta.3

1. Create cluster

      `kbcli cluster create  mysql  scale-cqsuvd             --termination-policy=Delete             --cpu=0.5             --memory=1             --storage=20                      --mode='raftGroup'                     --replicas=3                     --storage=20                     --availability-policy=none                     --proxy-enabled=true  --namespace default `

Info: --version is not specified, ac-mysql-8.0.30 is applied by default.
Cluster scale-cqsuvd created

2. Hscale

 `kbcli cluster hscale scale-cqsuvd --auto-approve --components mysql --replicas 4 --namespace default `

OpsRequest scale-cqsuvd-horizontalscaling-5flrg created successfully, you can view the progress:
	kbcli cluster describe-ops scale-cqsuvd-horizontalscaling-5flrg -n default

3. Hscale failed

➜  ~ kbcli cluster describe-ops scale-cqsuvd-horizontalscaling-5flrg -n default
Spec:
  Name: scale-cqsuvd-horizontalscaling-5flrg	NameSpace: default	Cluster: scale-cqsuvd	Type: HorizontalScaling

Command:
  kbcli cluster hscale scale-cqsuvd --components=mysql --replicas=4 --namespace=default

Last Configuration:
COMPONENT   REPLICAS
mysql              3

Status:
  Start Time:         Oct 13,2023 10:29 UTC+0800
  Duration:           12m
  Status:             Running
  Progress:           -/-

Conditions:
LAST-TRANSITION-TIME         TYPE                REASON                         STATUS   MESSAGE
Oct 13,2023 10:29 UTC+0800   Progressing         OpsRequestProgressingStarted   True     Start to process the OpsRequest: scale-cqsuvd-horizontalscaling-5flrg in Cluster: scale-cqsuvd
Oct 13,2023 10:29 UTC+0800   Validated           ValidateOpsRequestPassed       True     OpsRequest: scale-cqsuvd-horizontalscaling-5flrg is validated
Oct 13,2023 10:29 UTC+0800   HorizontalScaling   HorizontalScalingStarted       True     Start to horizontal scale replicas in Cluster: scale-cqsuvd

Warning Events: <none>

➜  ~ k describe cluster scale-cqsuvd
Name:         scale-cqsuvd
Namespace:    default
Labels:       app.kubernetes.io/instance=scale-cqsuvd
              app.kubernetes.io/version=8.0.30
              clusterdefinition.kubeblocks.io/name=apecloud-mysql
              clusterversion.kubeblocks.io/name=ac-mysql-8.0.30
              helm.sh/chart=apecloud-mysql-cluster-0.7.0-alpha.0
Annotations:  kubeblocks.io/extra-env: {"KB_PROXY_ENABLED":"on"}
              kubeblocks.io/ops-request: [{"name":"scale-cqsuvd-horizontalscaling-5flrg","type":"HorizontalScaling"}]
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2023-10-13T02:23:01Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:  5
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubeblocks.io/extra-env:
        f:labels:
          .:
          f:app.kubernetes.io/instance:
          f:app.kubernetes.io/version:
          f:helm.sh/chart:
      f:spec:
        .:
        f:affinity:
          .:
          f:podAntiAffinity:
          f:tenancy:
        f:clusterDefinitionRef:
        f:clusterVersionRef:
        f:componentSpecs:
          .:
          k:{"name":"mysql"}:
            .:
            f:componentDefRef:
            f:enabledLogs:
              .:
              v:"error":
              v:"slow":
            f:monitor:
            f:name:
            f:noCreatePDB:
            f:resources:
              .:
              f:limits:
                .:
                f:memory:
              f:requests:
                .:
                f:memory:
            f:volumeClaimTemplates:
          k:{"name":"vtcontroller"}:
            .:
            f:componentDefRef:
            f:enabledLogs:
              .:
              v:"error":
              v:"info":
              v:"warning":
            f:monitor:
            f:name:
            f:noCreatePDB:
            f:replicas:
            f:resources:
              .:
              f:limits:
                .:
                f:cpu:
                f:memory:
            f:volumeClaimTemplates:
          k:{"name":"vtgate"}:
            .:
            f:componentDefRef:
            f:enabledLogs:
              .:
              v:"error":
              v:"info":
              v:"queryLog":
              v:"warning":
            f:monitor:
            f:name:
            f:noCreatePDB:
            f:replicas:
            f:resources:
              .:
              f:limits:
              f:requests:
        f:terminationPolicy:
    Manager:      kbcli
    Operation:    Update
    Time:         2023-10-13T02:24:26Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:kubeblocks.io/ops-request:
        f:finalizers:
          .:
          v:"cluster.kubeblocks.io/finalizer":
        f:labels:
          f:clusterdefinition.kubeblocks.io/name:
          f:clusterversion.kubeblocks.io/name:
      f:spec:
        f:componentSpecs:
          k:{"name":"mysql"}:
            f:replicas:
            f:resources:
              f:limits:
                f:cpu:
              f:requests:
                f:cpu:
          k:{"name":"vtgate"}:
            f:classDefRef:
              .:
              f:class:
            f:resources:
              f:limits:
                f:cpu:
                f:memory:
              f:requests:
                f:cpu:
                f:memory:
        f:monitor:
        f:resources:
          .:
          f:cpu:
          f:memory:
        f:storage:
          .:
          f:size:
    Manager:      manager
    Operation:    Update
    Time:         2023-10-13T02:29:06Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterDefGeneration:
        f:components:
          .:
          f:mysql:
            .:
            f:consensusSetStatus:
              .:
              f:followers:
              f:leader:
                .:
                f:accessMode:
                f:name:
                f:pod:
            f:membersStatus:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
          f:vtcontroller:
            .:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
          f:vtgate:
            .:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
        f:conditions:
        f:observedGeneration:
        f:phase:
    Manager:         manager
    Operation:       Update
    Subresource:     status
    Time:            2023-10-13T02:29:08Z
  Resource Version:  3606683
  UID:               5f682d35-8b83-4c37-aa37-343aedfd923b
Spec:
  Affinity:
    Pod Anti Affinity:     Required
    Tenancy:               SharedNode
  Cluster Definition Ref:  apecloud-mysql
  Cluster Version Ref:     ac-mysql-8.0.30
  Component Specs:
    Component Def Ref:  mysql
    Enabled Logs:
      slow
      error
    Monitor:        false
    Name:           mysql
    No Create PDB:  false
    Replicas:       4
    Resources:
      Limits:
        Cpu:     500m
        Memory:  1Gi
      Requests:
        Cpu:     500m
        Memory:  1Gi
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:    20Gi
    Component Def Ref:  vtcontroller
    Enabled Logs:
      error
      warning
      info
    Monitor:        false
    Name:           vtcontroller
    No Create PDB:  false
    Replicas:       1
    Resources:
      Limits:
        Cpu:     500m
        Memory:  128Mi
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  20Gi
    Class Def Ref:
      Class:
    Component Def Ref:  vtgate
    Enabled Logs:
      error
      warning
      info
      queryLog
    Monitor:        false
    Name:           vtgate
    No Create PDB:  false
    Replicas:       1
    Resources:
      Limits:
        Cpu:     600m
        Memory:  1536Mi
      Requests:
        Cpu:     600m
        Memory:  1536Mi
  Monitor:
  Resources:
    Cpu:     0
    Memory:  0
  Storage:
    Size:              0
  Termination Policy:  Delete
Status:
  Cluster Def Generation:  2
  Components:
    Mysql:
      Consensus Set Status:
        Followers:
          Access Mode:  Readonly
          Name:         follower
          Pod:          scale-cqsuvd-mysql-2
          Access Mode:  Readonly
          Name:         follower
          Pod:          scale-cqsuvd-mysql-1
        Leader:
          Access Mode:  ReadWrite
          Name:         leader
          Pod:          scale-cqsuvd-mysql-0
      Members Status:
        Pod Name:  scale-cqsuvd-mysql-0
        Role:
          Access Mode:  ReadWrite
          Can Vote:     true
          Is Leader:    true
          Name:         leader
        Pod Name:       scale-cqsuvd-mysql-2
        Role:
          Access Mode:  Readonly
          Can Vote:     true
          Is Leader:    false
          Name:         follower
        Pod Name:       scale-cqsuvd-mysql-1
        Role:
          Access Mode:  Readonly
          Can Vote:     true
          Is Leader:    false
          Name:         follower
      Phase:            Running
      Pods Ready:       true
      Pods Ready Time:  2023-10-13T02:28:33Z
    Vtcontroller:
      Phase:            Running
      Pods Ready:       true
      Pods Ready Time:  2023-10-13T02:23:22Z
    Vtgate:
      Phase:            Running
      Pods Ready:       true
      Pods Ready Time:  2023-10-13T02:25:19Z
  Conditions:
    Last Transition Time:  2023-10-13T02:23:01Z
    Message:               The operator has started the provisioning of Cluster: scale-cqsuvd
    Observed Generation:   5
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2023-10-13T02:29:08Z
    Message:               backup for horizontalScaling failed: Failed to set default snapshot class with error cannot find default snapshot class
    Reason:                BackupFailed
    Status:                False
    Type:                  ApplyResources
    Last Transition Time:  2023-10-13T02:28:33Z
    Message:               all pods of components are ready, waiting for the probe detection successful
    Reason:                AllReplicasReady
    Status:                True
    Type:                  ReplicasReady
    Last Transition Time:  2023-10-13T02:28:33Z
    Message:               Cluster: scale-cqsuvd is ready, current phase is Running
    Reason:                ClusterReady
    Status:                True
    Type:                  Ready
  Observed Generation:     5
  Phase:                   Running
Events:
  Type     Reason                    Age                  From                       Message
  ----     ------                    ----                 ----                       -------
  Warning  ApplyResourcesFailed      19m                  cluster-controller         Operation cannot be fulfilled on pods "scale-cqsuvd-vtcontroller-0": the object has been modified; please apply your changes to the latest version and try again
  Warning  ApplyResourcesFailed      19m                  cluster-controller         Operation cannot be fulfilled on pods "scale-cqsuvd-mysql-1": the object has been modified; please apply your changes to the latest version and try again
  Normal   ComponentPhaseTransition  19m (x9 over 19m)    cluster-controller         Create a new component
  Normal   ApplyResourcesSucceed     19m (x3 over 19m)    cluster-controller         Successfully applied for resources
  Warning  ReplicasNotReady          18m                  cluster-controller         pods are not ready in Components: [mysql], refer to related component message in Cluster.status.components
  Warning  ComponentsNotReady        18m                  cluster-controller         pods are unavailable in Components: [mysql], refer to related component message in Cluster.status.components
  Normal   Running                   18m                  cluster-controller         Cluster: scale-cqsuvd is ready, current phase is Running
  Normal   ClusterReady              18m                  cluster-controller         Cluster: scale-cqsuvd is ready, current phase is Running
  Normal   AllReplicasReady          18m                  cluster-controller         all pods of components are ready, waiting for the probe detection successful
  Normal   SysAcctCreate             18m                  system-account-controller  Created accounts for cluster: scale-cqsuvd, component: mysql, accounts: kbdataprotection
  Normal   SysAcctCreate             18m                  system-account-controller  Created accounts for cluster: scale-cqsuvd, component: mysql, accounts: kbadmin
  Normal   SysAcctCreate             18m                  system-account-controller  Created accounts for cluster: scale-cqsuvd, component: mysql, accounts: kbmonitoring
  Normal   SysAcctCreate             18m                  system-account-controller  Created accounts for cluster: scale-cqsuvd, component: mysql, accounts: kbprobe
  Normal   SysAcctCreate             18m                  system-account-controller  Created accounts for cluster: scale-cqsuvd, component: mysql, accounts: kbreplicator
  Normal   PreCheckSucceed           17m (x3 over 19m)    cluster-controller         The operator has started the provisioning of Cluster: scale-cqsuvd
  Normal   ComponentPhaseTransition  13m (x5 over 18m)    cluster-controller         component is Running
  Warning  BackupFailed              118s (x19 over 13m)  cluster-controller         backup for horizontalScaling failed: Failed to set default snapshot class with error cannot find default snapshot class

logs in kb controller:
2023-10-13T02:31:59.818Z	INFO	backup for horizontalScaling failed: Failed to set default snapshot class with error cannot find default snapshot class	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"scale-cqsuvd","namespace":"default"}, "namespace": "default", "name": "scale-cqsuvd", "reconcileID": "a5e80bcc-90ee-4261-8ca5-141b9160a8cb", "cluster": {"name":"scale-cqsuvd","namespace":"default"}}
2023-10-13T02:31:59.818Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"scale-cqsuvd","namespace":"default"}, "namespace": "default", "name": "scale-cqsuvd", "reconcileID": "a5e80bcc-90ee-4261-8ca5-141b9160a8cb", "error": "backup for horizontalScaling failed: Failed to set default snapshot class with error cannot find default snapshot class"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226

➜ ~ k logs kubeblocks-68cf47bc7-6njf8 -n kb-system >kblog.txt
Defaulted container "manager" out of: manager, tools (init), datascript (init)

kblog.txt