[BUG] Executing ops VolumeExpansion twice causes the cluster status to always be Updating, which affects other ops operations

[linghan-hub]

Executing ops VolumeExpansion twice causes the cluster status to always be Updating, which affects other ops operations，a layer of interception should be added

kbcli version
Kubernetes: v1.25.12-eks-2d98532
KubeBlocks: 0.6.3-beta.0
kbcli: 0.6.3-beta.0

1. create mysql cluster

---
# Source: apecloud-mysql-cluster/templates/rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kb-mysql-cluster
  namespace: default
  labels:
    helm.sh/chart: apecloud-mysql-cluster-0.7.0-alpha.0
    app.kubernetes.io/version: "8.0.30"
    app.kubernetes.io/instance: mysql-cluster
---
# Source: apecloud-mysql-cluster/templates/rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kb-mysql-cluster
  labels:
    helm.sh/chart: apecloud-mysql-cluster-0.7.0-alpha.0
    app.kubernetes.io/version: "8.0.30"
    app.kubernetes.io/instance: mysql-cluster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubeblocks-volume-protection-pod-role
subjects:
  - kind: ServiceAccount
    name: kb-mysql-cluster
    namespace: default
---
# Source: apecloud-mysql-cluster/templates/rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kb-mysql-cluster
  labels:
    helm.sh/chart: apecloud-mysql-cluster-0.7.0-alpha.0
    app.kubernetes.io/version: "8.0.30"
    app.kubernetes.io/instance: mysql-cluster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubeblocks-cluster-pod-role
subjects:
  - kind: ServiceAccount
    name: kb-mysql-cluster
    namespace: default
---
# Source: apecloud-mysql-cluster/templates/cluster.yaml
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
  name: mysql-cluster
  namespace: default
  labels: 
    helm.sh/chart: apecloud-mysql-cluster-0.7.0-alpha.0
    app.kubernetes.io/version: "8.0.30"
    app.kubernetes.io/instance: mysql-cluster
spec:
  clusterVersionRef: ac-mysql-8.0.30
  terminationPolicy: Delete  
  affinity:
    podAntiAffinity: Required
    topologyKeys:
      - kubernetes.io/hostname
    tenancy: SharedNode
  clusterDefinitionRef: apecloud-mysql # ref clusterdefinition.name
  componentSpecs:
    - name: mysql
      componentDefRef: mysql # ref clusterdefinition componentDefs.name      
      monitor: false
      replicas: 1
      enabledLogs:
        - slow
        - error
      serviceAccountName: kb-mysql-cluster      
      resources:
        limits:
          cpu: "0.5"
          memory: "0.5Gi"
        requests:
          cpu: "0.5"
          memory: "0.5Gi"      
      volumeClaimTemplates:
        - name: data # ref clusterDefinition components.containers.volumeMounts.name
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi      
      services:

2. create ops
   First time

apiVersion: apps.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  generateName: mysql-cluster-vexpand-
spec:
  clusterRef: mysql-cluster
  type: VolumeExpansion
  volumeExpansion:
    - componentName: mysql
      volumeClaimTemplates:
        - name: data
          storage: "21Gi"

Second time

apiVersion: apps.kubeblocks.io/v1alpha1
kind: OpsRequest
metadata:
  generateName: mysql-cluster-vexpand-
spec:
  clusterRef: mysql-cluster
  type: VolumeExpansion
  volumeExpansion:
    - componentName: mysql
      volumeClaimTemplates:
        - name: data
          storage: "30Gi"

3. see resources

k get cluster
NAME            CLUSTER-DEFINITION   VERSION           TERMINATION-POLICY   STATUS     AGE
mysql-cluster   apecloud-mysql       ac-mysql-8.0.30   Delete               Updating   5m54s

k get ops
NAME                          TYPE              CLUSTER         STATUS    PROGRESS   AGE
mysql-cluster-vexpand-c9q7c   VolumeExpansion   mysql-cluster   Succeed   1/1        2m28s
mysql-cluster-vexpand-d8vk8   VolumeExpansion   mysql-cluster   Failed    1/1        86s
mysql-cluster-vexpand-slh7g   VolumeExpansion   mysql-cluster   Failed    1/1        15s

4. see logs

k describe ops mysql-cluster-vexpand-d8vk8
Name:         mysql-cluster-vexpand-d8vk8
Namespace:    default
Labels:       app.kubernetes.io/instance=mysql-cluster
              ops.kubeblocks.io/ops-type=VolumeExpansion
Annotations:  kubeblocks.io/reconcile: 2023-10-08T11:21:50.328413725Z
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         OpsRequest
Metadata:
  Creation Timestamp:  2023-10-08T11:21:08Z
  Finalizers:
    opsrequest.kubeblocks.io/finalizer
  Generate Name:  mysql-cluster-vexpand-
  Generation:     1
  Owner References:
    API Version:     apps.kubeblocks.io/v1alpha1
    Kind:            Cluster
    Name:            mysql-cluster
    UID:             b05f5548-6bb5-48ad-97bb-6fbd78a87826
  Resource Version:  4914875
  UID:               97c05778-bf2a-4216-bd81-8b25e7853182
Spec:
  Cluster Ref:               mysql-cluster
  Ttl Seconds Before Abort:  0
  Type:                      VolumeExpansion
  Volume Expansion:
    Component Name:  mysql
    Volume Claim Templates:
      Name:     data
      Storage:  30Gi
Status:
  Cluster Generation:    3
  Completion Timestamp:  2023-10-08T11:21:52Z
  Components:
    Mysql:
      Progress Details:
        End Time:    2023-10-08T11:21:52Z
        Group:       data
        Message:     resize volume "pvc-553ef757-2591-4c45-81dd-29cd416dc7ba" by resizer "ebs.csi.aws.com" failed: rpc error: code = Internal desc = Could not resize volume "vol-03cc82b292e85fe5f": rpc error: code = Internal desc = Could not modify volume "vol-03cc82b292e85fe5f": volume "vol-03cc82b292e85fe5f" in OPTIMIZING state, cannot currently modify
        Object Key:  PVC/data-mysql-cluster-mysql-0
        Status:      Failed
  Conditions:
    Last Transition Time:  2023-10-08T11:21:08Z
    Message:               Start to process the OpsRequest: mysql-cluster-vexpand-d8vk8 in Cluster: mysql-cluster
    Reason:                OpsRequestProgressingStarted
    Status:                True
    Type:                  Progressing
    Last Transition Time:  2023-10-08T11:21:08Z
    Message:               OpsRequest: mysql-cluster-vexpand-d8vk8 is validated
    Reason:                ValidateOpsRequestPassed
    Status:                True
    Type:                  Validated
    Last Transition Time:  2023-10-08T11:21:08Z
    Message:               Start to expand the volumes in Cluster: mysql-cluster
    Reason:                VolumeExpansionStarted
    Status:                True
    Type:                  VolumeExpanding
    Last Transition Time:  2023-10-08T11:21:52Z
    Message:               Failed to process OpsRequest: mysql-cluster-vexpand-d8vk8 in cluster: mysql-cluster
    Reason:                OpsRequestFailed
    Status:                False
    Type:                  Failed
  Last Configuration:
    Components:
      Mysql:
        Volume Claim Templates:
          Name:     data
          Storage:  21Gi
  Phase:            Failed
  Progress:         1/1
  Start Timestamp:  2023-10-08T11:21:08Z
Events:
  Type     Reason                        Age   From                    Message
  ----     ------                        ----  ----                    -------
  Normal   OpsRequestProgressingStarted  103s  ops-request-controller  Start to process the OpsRequest: mysql-cluster-vexpand-d8vk8 in Cluster: mysql-cluster
  Normal   ValidateOpsRequestPassed      103s  ops-request-controller  OpsRequest: mysql-cluster-vexpand-d8vk8 is validated
  Normal   VolumeExpansionStarted        103s  ops-request-controller  Start to expand the volumes in Cluster: mysql-cluster
  Warning  Failed                        59s   event-controller        resize volume "pvc-553ef757-2591-4c45-81dd-29cd416dc7ba" by resizer "ebs.csi.aws.com" failed: rpc error: code = Internal desc = Could not resize volume "vol-03cc82b292e85fe5f": rpc error: code = Internal desc = Could not modify volume "vol-03cc82b292e85fe5f": volume "vol-03cc82b292e85fe5f" in OPTIMIZING state, cannot currently modify
  Warning  VolumeResizeFailed            59s   event-controller        resize volume "pvc-553ef757-2591-4c45-81dd-29cd416dc7ba" by resizer "ebs.csi.aws.com" failed: rpc error: code = Internal desc = Could not resize volume "vol-03cc82b292e85fe5f": rpc error: code = Internal desc = Could not modify volume "vol-03cc82b292e85fe5f": volume "vol-03cc82b292e85fe5f" in OPTIMIZING state, cannot currently modify
  Warning  OpsRequestFailed              59s   ops-request-controller  Failed to process OpsRequest: mysql-cluster-vexpand-d8vk8 in cluster: mysql-cluster