[BUG]Mysql-scale cluster is always updating after configure vtgate

[ahjing99]

1. Create cluster and configure

      `kbcli cluster create  mysql  scale-vwczxp             --termination-policy=DoNotTerminate             --cpu=0.5             --memory=1             --storage=20                      --mode='raftGroup'                     --replicas=3                     --storage=20                     --availability-policy=none                     --proxy-enabled=true  --namespace default `

Info: --version is not specified, ac-mysql-8.0.30 is applied by default.
Cluster scale-vwczxp created

  `kbcli cluster configure scale-vwczxp --auto-approve                 --set healthcheck_timeout=4s,read_write_splitting_policy=random --component vtgate --config-spec vtgate-config  --namespace default `

Will updated configure file meta:
  ConfigSpec: vtgate-config	  ConfigFile: vtgate.cnf	ComponentName: vtgate	ClusterName: scale-vwczxp
OpsRequest scale-vwczxp-reconfiguring-l65tv created successfully, you can view the progress:
	kbcli cluster describe-ops scale-vwczxp-reconfiguring-l65tv -n default

➜  ~ kbcli cluster describe-ops scale-vwczxp-reconfiguring-l65tv -n default
Spec:
  Name: scale-vwczxp-reconfiguring-l65tv	NameSpace: default	Cluster: scale-vwczxp	Type: Reconfiguring

Command:
  kbcli cluster configure scale-vwczxp --components=vtgate --config-spec=vtgate-config --config-file=vtgate.cnf --set healthcheck_timeout=4s --set read_write_splitting_policy=random --namespace=default

Status:
  Start Time:         Oct 13,2023 16:05 UTC+0800
  Completion Time:    Oct 13,2023 16:05 UTC+0800
  Duration:           1s
  Status:             Succeed
  Progress:           -/-

Conditions:
LAST-TRANSITION-TIME         TYPE                REASON                            STATUS   MESSAGE
Oct 13,2023 16:05 UTC+0800   Progressing         OpsRequestProgressingStarted      True     Start to process the OpsRequest: scale-vwczxp-reconfiguring-l65tv in Cluster: scale-vwczxp
Oct 13,2023 16:05 UTC+0800   Validated           ValidateOpsRequestPassed          True     OpsRequest: scale-vwczxp-reconfiguring-l65tv is validated
Oct 13,2023 16:05 UTC+0800   Reconfigure         ReconfigureStarted                True     Start to reconfigure in Cluster: scale-vwczxp, Component: vtgate
Oct 13,2023 16:05 UTC+0800   ReconfigureMerged   ReconfigureMerged                 True     Reconfiguring in Cluster: scale-vwczxp, Component: vtgate, ConfigSpec: vtgate-config, info: updated: map[vtgate.cnf:{"vtgate":{"healthcheck_timeout":"4s","read_write_splitting_policy":"random"}}], added: map[], deleted:map[]
Oct 13,2023 16:05 UTC+0800   Succeed             OpsRequestProcessedSuccessfully   True     Successfully processed the OpsRequest: scale-vwczxp-reconfiguring-l65tv in Cluster: scale-vwczxp

Warning Events: <none>


➜  ~ kbcli cluster describe-config scale-vwczxp --components=vtgate
component: vtgate

ConfigSpecs Meta:
CONFIG-SPEC-NAME   FILE         ENABLED   TEMPLATE                 CONSTRAINT                              RENDERED                            COMPONENT   CLUSTER
vtgate-config      vtgate.cnf   true      vtgate-config-template   mysql-scale-vtgate-config-constraints   scale-vwczxp-vtgate-vtgate-config   vtgate      scale-vwczxp

History modifications:
OPS-NAME                           CLUSTER        COMPONENT   CONFIG-SPEC-NAME   FILE         STATUS    POLICY   PROGRESS   CREATED-TIME                 VALID-UPDATED
scale-vwczxp-reconfiguring-l65tv   scale-vwczxp   vtgate      vtgate-config      vtgate.cnf   Succeed            -/-        Oct 13,2023 16:05 UTC+0800   {"vtgate.cnf":"{\"vtgate\":{\"healthcheck_timeout\":\"4s\",\"read_write_splitting_policy\":\"random\"}}"}

➜  ~ k get cm scale-vwczxp-vtgate-vtgate-config -o yaml
apiVersion: v1
data:
  vtgate.cnf: |
    [vtgate]
    buffer_max_failover_duration=60s
    buffer_min_time_between_failovers=60s
    buffer_size=10000
    buffer_window=30s
    enable_buffer=false
    enable_logs=true
    enable_query_log=true
    gateway_initial_tablet_timeout=30s
    grpc_keepalive_time=10s
    grpc_keepalive_timeout=10s
    healthcheck_timeout=4s
    mysql_auth_server_impl=none
    mysql_auth_server_static_file=
    mysql_server_require_secure_transport=false
    mysql_server_ssl_cert=
    mysql_server_ssl_key=
    read_after_write_consistency=EVENTUAL
    read_after_write_timeout=30.0
    read_write_splitting_policy=random
    read_write_splitting_ratio=100
    srv_topo_timeout=1s
    tablet_refresh_interval=1m
kind: ConfigMap
metadata:
  annotations:
    config.kubeblocks.io/config-applied-version: '{"name":"vtgate-config","configSpec":{"name":"vtgate-config","templateRef":"vtgate-config-template","namespace":"kb-system","volumeName":"mysql-scale-config","constraintRef":"mysql-scale-vtgate-config-constraints"},"importTemplateRef":null,"configFileParams":{"vtgate.cnf":{"content":null,"parameters":{"healthcheck_timeout":"4s","read_write_splitting_policy":"random"}}}}'
    config.kubeblocks.io/config-template-version: ""
    config.kubeblocks.io/configuration-revision: "2"
    config.kubeblocks.io/disable-reconfigure: "false"
    config.kubeblocks.io/last-applied-configuration: '{"vtgate.cnf":"[vtgate]\ngateway_initial_tablet_timeout=30s\nhealthcheck_timeout=2s\nsrv_topo_timeout=1s\ngrpc_keepalive_time=10s\ngrpc_keepalive_timeout=10s\ntablet_refresh_interval=1m\nread_write_splitting_policy=disable\nread_write_splitting_ratio=100\nread_after_write_consistency=EVENTUAL\nread_after_write_timeout=30.0\nenable_buffer=false\nbuffer_size=10000\nbuffer_window=30s\nbuffer_max_failover_duration=60s\nbuffer_min_time_between_failovers=60s\nmysql_auth_server_impl=none\nmysql_server_require_secure_transport=false\nmysql_auth_server_static_file=\nmysql_server_ssl_key=\nmysql_server_ssl_cert=\n\n\nenable_logs=true\nenable_query_log=true\n"}'
    config.kubeblocks.io/reconfigure-source: manager
    config.kubeblocks.io/revision-reconcile-phase-1: Finished
    config.kubeblocks.io/revision-reconcile-phase-2: Upgrading
    config.kubeblocks.io/update-config-hash: 5f479b6fc9
  creationTimestamp: "2023-10-13T07:52:11Z"
  labels:
    app.kubernetes.io/component: vtgate
    app.kubernetes.io/instance: scale-vwczxp
    app.kubernetes.io/managed-by: kubeblocks
    app.kubernetes.io/name: apecloud-mysql
    apps.kubeblocks.io/component-name: vtgate
    config.kubeblocks.io/config-constraints-name: mysql-scale-vtgate-config-constraints
    config.kubeblocks.io/config-hash: 67bc97b484
    config.kubeblocks.io/config-spec: vtgate-config
    config.kubeblocks.io/config-template-name: vtgate-config-template
    config.kubeblocks.io/config-type: instance
    config.kubeblocks.io/last-applied-reconfigure-phase: created
    config.kubeblocks.io/template-name: vtgate-config-template
  name: scale-vwczxp-vtgate-vtgate-config
  namespace: default
  ownerReferences:
  - apiVersion: apps.kubeblocks.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Configuration
    name: scale-vwczxp-vtgate
    uid: 81865f46-9d96-467f-8d90-caebefac5b00
  resourceVersion: "3842135"
  uid: 9ef5da1f-0d4d-4191-acf7-66cf6bae54de

the cluster status is always updating

➜  ~ k describe cluster scale-vwczxp
Name:         scale-vwczxp
Namespace:    default
Labels:       app.kubernetes.io/instance=scale-vwczxp
              app.kubernetes.io/version=8.0.30
              clusterdefinition.kubeblocks.io/name=apecloud-mysql
              clusterversion.kubeblocks.io/name=ac-mysql-8.0.30
              helm.sh/chart=apecloud-mysql-cluster-0.7.0-alpha.0
Annotations:  kubeblocks.io/extra-env: {"KB_PROXY_ENABLED":"on"}
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2023-10-13T07:52:10Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:  4
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubeblocks.io/extra-env:
        f:labels:
          .:
          f:app.kubernetes.io/instance:
          f:app.kubernetes.io/version:
          f:helm.sh/chart:
      f:spec:
        .:
        f:affinity:
          .:
          f:podAntiAffinity:
          f:tenancy:
        f:clusterDefinitionRef:
        f:clusterVersionRef:
        f:componentSpecs:
          .:
          k:{"name":"mysql"}:
            .:
            f:componentDefRef:
            f:enabledLogs:
              .:
              v:"error":
              v:"slow":
            f:monitor:
            f:name:
            f:noCreatePDB:
            f:resources:
              .:
              f:limits:
                .:
                f:memory:
              f:requests:
                .:
                f:memory:
            f:volumeClaimTemplates:
          k:{"name":"vtcontroller"}:
            .:
            f:componentDefRef:
            f:enabledLogs:
              .:
              v:"error":
              v:"info":
              v:"warning":
            f:monitor:
            f:name:
            f:noCreatePDB:
            f:replicas:
            f:resources:
              .:
              f:limits:
                .:
                f:cpu:
                f:memory:
            f:volumeClaimTemplates:
          k:{"name":"vtgate"}:
            .:
            f:componentDefRef:
            f:enabledLogs:
              .:
              v:"error":
              v:"info":
              v:"queryLog":
              v:"warning":
            f:monitor:
            f:name:
            f:noCreatePDB:
            f:replicas:
            f:resources:
              .:
              f:limits:
              f:requests:
        f:terminationPolicy:
    Manager:      kbcli
    Operation:    Update
    Time:         2023-10-13T07:54:50Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"cluster.kubeblocks.io/finalizer":
        f:labels:
          f:clusterdefinition.kubeblocks.io/name:
          f:clusterversion.kubeblocks.io/name:
      f:spec:
        f:componentSpecs:
          k:{"name":"mysql"}:
            f:replicas:
            f:resources:
              f:limits:
                f:cpu:
              f:requests:
                f:cpu:
          k:{"name":"vtgate"}:
            f:resources:
              f:limits:
                f:cpu:
                f:memory:
              f:requests:
                f:cpu:
        f:monitor:
        f:resources:
          .:
          f:cpu:
          f:memory:
        f:storage:
          .:
          f:size:
    Manager:      manager
    Operation:    Update
    Time:         2023-10-13T08:05:43Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterDefGeneration:
        f:components:
          .:
          f:mysql:
            .:
            f:consensusSetStatus:
              .:
              f:followers:
              f:leader:
                .:
                f:accessMode:
                f:name:
                f:pod:
            f:membersStatus:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
          f:vtcontroller:
            .:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
          f:vtgate:
            .:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
        f:conditions:
        f:observedGeneration:
        f:phase:
    Manager:         manager
    Operation:       Update
    Subresource:     status
    Time:            2023-10-13T08:05:43Z
  Resource Version:  3842140
  UID:               e526b66a-11ec-424d-a8ad-a8e75e6b0add
Spec:
  Affinity:
    Pod Anti Affinity:     Required
    Tenancy:               SharedNode
  Cluster Definition Ref:  apecloud-mysql
  Cluster Version Ref:     ac-mysql-8.0.30
  Component Specs:
    Component Def Ref:  mysql
    Enabled Logs:
      slow
      error
    Monitor:        false
    Name:           mysql
    No Create PDB:  false
    Replicas:       5
    Resources:
      Limits:
        Cpu:     500m
        Memory:  1Gi
      Requests:
        Cpu:     500m
        Memory:  1Gi
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:    20Gi
    Component Def Ref:  vtcontroller
    Enabled Logs:
      error
      warning
      info
    Monitor:        false
    Name:           vtcontroller
    No Create PDB:  false
    Replicas:       1
    Resources:
      Limits:
        Cpu:     500m
        Memory:  128Mi
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:    20Gi
    Component Def Ref:  vtgate
    Enabled Logs:
      error
      warning
      info
      queryLog
    Monitor:        false
    Name:           vtgate
    No Create PDB:  false
    Replicas:       1
    Resources:
      Limits:
        Cpu:     500m
        Memory:  0
      Requests:
        Cpu:  500m
  Monitor:
  Resources:
    Cpu:     0
    Memory:  0
  Storage:
    Size:              0
  Termination Policy:  DoNotTerminate
Status:
  Cluster Def Generation:  2
  Components:
    Mysql:
      Consensus Set Status:
        Followers:
          Access Mode:  Readonly
          Name:         follower
          Pod:          scale-vwczxp-mysql-4
          Access Mode:  Readonly
          Name:         follower
          Pod:          scale-vwczxp-mysql-3
          Access Mode:  Readonly
          Name:         follower
          Pod:          scale-vwczxp-mysql-2
          Access Mode:  Readonly
          Name:         follower
          Pod:          scale-vwczxp-mysql-1
        Leader:
          Access Mode:  ReadWrite
          Name:         leader
          Pod:          scale-vwczxp-mysql-0
      Members Status:
        Pod Name:  scale-vwczxp-mysql-0
        Role:
          Access Mode:  ReadWrite
          Can Vote:     true
          Is Leader:    true
          Name:         leader
        Pod Name:       scale-vwczxp-mysql-4
        Role:
          Access Mode:  Readonly
          Can Vote:     true
          Is Leader:    false
          Name:         follower
        Pod Name:       scale-vwczxp-mysql-3
        Role:
          Access Mode:  Readonly
          Can Vote:     true
          Is Leader:    false
          Name:         follower
        Pod Name:       scale-vwczxp-mysql-2
        Role:
          Access Mode:  Readonly
          Can Vote:     true
          Is Leader:    false
          Name:         follower
        Pod Name:       scale-vwczxp-mysql-1
        Role:
          Access Mode:  Readonly
          Can Vote:     true
          Is Leader:    false
          Name:         follower
      Phase:            Running
      Pods Ready:       true
      Pods Ready Time:  2023-10-13T08:04:01Z
    Vtcontroller:
      Phase:            Running
      Pods Ready:       true
      Pods Ready Time:  2023-10-13T07:52:23Z
    Vtgate:
      Phase:            Updating
      Pods Ready:       false
      Pods Ready Time:  2023-10-13T07:52:28Z
  Conditions:
    Last Transition Time:  2023-10-13T07:52:10Z
    Message:               The operator has started the provisioning of Cluster: scale-vwczxp
    Observed Generation:   4
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2023-10-13T08:00:54Z
    Message:               Successfully applied for resources
    Observed Generation:   4
    Reason:                ApplyResourcesSucceed
    Status:                True
    Type:                  ApplyResources
    Last Transition Time:  2023-10-13T08:05:43Z
    Message:               pods are not ready in Components: [vtgate], refer to related component message in Cluster.status.components
    Reason:                ReplicasNotReady
    Status:                False
    Type:                  ReplicasReady
    Last Transition Time:  2023-10-13T08:05:43Z
    Message:               pods are unavailable in Components: [vtgate], refer to related component message in Cluster.status.components
    Reason:                ComponentsNotReady
    Status:                False
    Type:                  Ready
  Observed Generation:     4
  Phase:                   Updating
Events:
  Type     Reason                    Age                From                       Message
  ----     ------                    ----               ----                       -------
  Normal   ComponentPhaseTransition  40m (x3 over 40m)  cluster-controller         Create a new component
  Warning  ReplicasNotReady          40m                cluster-controller         pods are not ready in Components: [mysql vtgate], refer to related component message in Cluster.status.components
  Warning  ComponentsNotReady        40m                cluster-controller         pods are unavailable in Components: [mysql vtgate], refer to related component message in Cluster.status.components
  Warning  ComponentsNotReady        40m                cluster-controller         pods are unavailable in Components: [mysql], refer to related component message in Cluster.status.components
  Normal   ClusterReady              40m                cluster-controller         Cluster: scale-vwczxp is ready, current phase is Running
  Normal   AllReplicasReady          40m                cluster-controller         all pods of components are ready, waiting for the probe detection successful
  Normal   Running                   40m                cluster-controller         Cluster: scale-vwczxp is ready, current phase is Running
  Normal   SysAcctCreate             40m                system-account-controller  Created accounts for cluster: scale-vwczxp, component: mysql, accounts: kbdataprotection
  Normal   SysAcctCreate             40m                system-account-controller  Created accounts for cluster: scale-vwczxp, component: mysql, accounts: kbprobe
  Normal   SysAcctCreate             40m                system-account-controller  Created accounts for cluster: scale-vwczxp, component: mysql, accounts: kbmonitoring
  Normal   SysAcctCreate             40m                system-account-controller  Created accounts for cluster: scale-vwczxp, component: mysql, accounts: kbreplicator
  Normal   SysAcctCreate             40m                system-account-controller  Created accounts for cluster: scale-vwczxp, component: mysql, accounts: kbadmin
  Normal   PreCheckSucceed           37m (x4 over 40m)  cluster-controller         The operator has started the provisioning of Cluster: scale-vwczxp
  Normal   ApplyResourcesSucceed     37m (x4 over 40m)  cluster-controller         Successfully applied for resources
  Normal   HorizontalScale           37m (x2 over 37m)  cluster-controller         start horizontal scale component mysql of cluster scale-vwczxp from 3 to 5
  Warning  ReplicasNotReady          37m (x2 over 40m)  cluster-controller         pods are not ready in Components: [mysql], refer to related component message in Cluster.status.components
  Normal   ComponentPhaseTransition  34m (x2 over 37m)  cluster-controller         component is Updating
  Normal   ComponentPhaseTransition  30m (x5 over 40m)  cluster-controller         component is Running