Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] stop ops status is always Running #7150

Closed
JashBook opened this issue Apr 24, 2024 · 0 comments · Fixed by #7165
Closed

[BUG] stop ops status is always Running #7150

JashBook opened this issue Apr 24, 2024 · 0 comments · Fixed by #7165
Assignees
@wangyelei
Labels
kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Milestone
Release 0.9.0

Comments

@JashBook
Copy link
Collaborator

JashBook commented Apr 24, 2024 •
edited
Loading

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. create cluster
kbcli cluster create  mysql-mtvtkq --termination-policy=Delete --monitoring-interval=0 --cluster-definition=apecloud-mysql --enable-all-logs=false --cluster-version=ac-mysql-8.0.30 --set cpu=100m,memory=0.5Gi,replicas=3,storage=1Gi
  1. stop cluster
kbcli cluster stop mysql-cluster --auto-approve
  1. See error
 kubectl get cluster 
NAME            CLUSTER-DEFINITION   VERSION           TERMINATION-POLICY   STATUS    AGE
mysql-cluster   apecloud-mysql       ac-mysql-8.0.30   WipeOut              Stopped   120m
➜  ~ 
➜  ~ kubectl get pod -l app.kubernetes.io/instance=mysql-cluster 
No resources found in default namespace.
➜  ~ 
➜  ~ kubectl get ops
NAME                       TYPE   CLUSTER         STATUS    PROGRESS   AGE
mysql-cluster-stop-c8xqz   Stop   mysql-cluster   Running   3/3        6m14s

describe cluster

kubectl describe cluster mysql-cluster 
Name:         mysql-cluster
Namespace:    default
Labels:       clusterdefinition.kubeblocks.io/name=apecloud-mysql
              clusterversion.kubeblocks.io/name=ac-mysql-8.0.30
Annotations:  kubeblocks.io/ops-request: [{"name":"mysql-cluster-stop-c8xqz","type":"Stop"}]
              kubeblocks.io/snapshot-for-start: {"mysql":3}
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2024-04-24T04:09:23Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:  2
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:affinity:
          .:
          f:podAntiAffinity:
          f:tenancy:
        f:clusterDefinitionRef:
        f:clusterVersionRef:
        f:resources:
          .:
          f:cpu:
          f:memory:
        f:storage:
          .:
          f:size:
        f:terminationPolicy:
    Manager:      kbcli
    Operation:    Update
    Time:         2024-04-24T04:09:23Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubeblocks.io/ops-request:
          f:kubeblocks.io/snapshot-for-start:
        f:finalizers:
          .:
          v:"cluster.kubeblocks.io/finalizer":
        f:labels:
          .:
          f:clusterdefinition.kubeblocks.io/name:
          f:clusterversion.kubeblocks.io/name:
      f:spec:
        f:componentSpecs:
    Manager:      manager
    Operation:    Update
    Time:         2024-04-24T06:03:38Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterDefGeneration:
        f:components:
          .:
          f:mysql:
            .:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
        f:conditions:
        f:observedGeneration:
        f:phase:
    Manager:         manager
    Operation:       Update
    Subresource:     status
    Time:            2024-04-24T06:04:11Z
  Resource Version:  334575853
  UID:               cbf004ee-fd29-43ac-a1e4-c64eb27c0dca
Spec:
  Affinity:
    Pod Anti Affinity:     Preferred
    Tenancy:               SharedNode
  Cluster Definition Ref:  apecloud-mysql
  Cluster Version Ref:     ac-mysql-8.0.30
  Component Specs:
    Component Def Ref:  mysql
    Enabled Logs:
      auditlog
      error
      general
      slow
    Name:      mysql
    Replicas:  0
    Resources:
      Limits:
        Cpu:     100m
        Memory:  512Mi
      Requests:
        Cpu:               100m
        Memory:            512Mi
    Service Account Name:  kb-mysql-cluster
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  1Gi
  Resources:
    Cpu:     0
    Memory:  0
  Storage:
    Size:              0
  Termination Policy:  WipeOut
Status:
  Cluster Def Generation:  2
  Components:
    Mysql:
      Phase:            Stopped
      Pods Ready:       true
      Pods Ready Time:  2024-04-24T04:11:39Z
  Conditions:
    Last Transition Time:  2024-04-24T04:09:23Z
    Message:               The operator has started the provisioning of Cluster: mysql-cluster
    Observed Generation:   2
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2024-04-24T04:09:23Z
    Message:               Successfully applied for resources
    Observed Generation:   2
    Reason:                ApplyResourcesSucceed
    Status:                True
    Type:                  ApplyResources
    Last Transition Time:  2024-04-24T04:11:39Z
    Message:               all pods of components are ready, waiting for the probe detection successful
    Reason:                AllReplicasReady
    Status:                True
    Type:                  ReplicasReady
    Last Transition Time:  2024-04-24T04:11:39Z
    Message:               Cluster: mysql-cluster is ready, current phase is Running
    Reason:                ClusterReady
    Status:                True
    Type:                  Ready
  Observed Generation:     2
  Phase:                   Stopped
Events:
  Type    Reason                    Age                   From                  Message
  ----    ------                    ----                  ----                  -------
  Normal  PreCheckSucceed           6m57s (x2 over 121m)  cluster-controller    The operator has started the provisioning of Cluster: mysql-cluster
  Normal  ApplyResourcesSucceed     6m57s (x2 over 121m)  cluster-controller    Successfully applied for resources
  Normal  HorizontalScale           6m57s                 component-controller  start horizontal scale component mysql of cluster mysql-cluster from 3 to 0
  Normal  ComponentPhaseTransition  6m56s                 cluster-controller    component is Stopping
  Normal  ComponentPhaseTransition  6m24s                 cluster-controller    component is Stopped
  Normal  Stopped                   6m24s                 cluster-controller    Cluster: mysql-cluster stopped successfully.

describe ops

kubectl describe ops mysql-cluster-stop-c8xqz 
Name:         mysql-cluster-stop-c8xqz
Namespace:    default
Labels:       app.kubernetes.io/instance=mysql-cluster
              app.kubernetes.io/managed-by=kubeblocks
              ops.kubeblocks.io/ops-type=Stop
Annotations:  <none>
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         OpsRequest
Metadata:
  Creation Timestamp:  2024-04-24T06:03:38Z
  Finalizers:
    opsrequest.kubeblocks.io/finalizer
  Generate Name:  mysql-cluster-stop-
  Generation:     2
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:generateName:
        f:labels:
          .:
          f:app.kubernetes.io/instance:
          f:app.kubernetes.io/managed-by:
      f:spec:
        .:
        f:clusterRef:
        f:ttlSecondsBeforeAbort:
        f:type:
    Manager:      kbcli
    Operation:    Update
    Time:         2024-04-24T06:03:38Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"opsrequest.kubeblocks.io/finalizer":
        f:labels:
          f:ops.kubeblocks.io/ops-type:
        f:ownerReferences:
          .:
          k:{"uid":"cbf004ee-fd29-43ac-a1e4-c64eb27c0dca"}:
    Manager:      manager
    Operation:    Update
    Time:         2024-04-24T06:03:38Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterGeneration:
        f:components:
          .:
          f:mysql:
            .:
            f:phase:
            f:progressDetails:
        f:conditions:
          .:
          k:{"type":"Stopping"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
          k:{"type":"Validated"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
          k:{"type":"WaitForProgressing"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
        f:lastConfiguration:
          .:
          f:components:
            .:
            f:mysql:
              .:
              f:replicas:
        f:phase:
        f:progress:
        f:startTimestamp:
    Manager:      manager
    Operation:    Update
    Subresource:  status
    Time:         2024-04-24T06:04:11Z
  Owner References:
    API Version:     apps.kubeblocks.io/v1alpha1
    Kind:            Cluster
    Name:            mysql-cluster
    UID:             cbf004ee-fd29-43ac-a1e4-c64eb27c0dca
  Resource Version:  334575854
  UID:               b2a98c09-5207-4aed-9f71-0611507259ca
Spec:
  Cluster Ref:               mysql-cluster
  Ttl Seconds Before Abort:  0
  Type:                      Stop
Status:
  Cluster Generation:  2
  Components:
    Mysql:
      Phase:  Stopped
      Progress Details:
        End Time:    2024-04-24T06:04:11Z
        Group:       mysql
        Message:     Successfully delete pod: Pod/mysql-cluster-mysql-0 in Component: mysql
        Object Key:  Pod/mysql-cluster-mysql-0
        Status:      Succeed
        End Time:    2024-04-24T06:04:11Z
        Group:       mysql
        Message:     Successfully delete pod: Pod/mysql-cluster-mysql-1 in Component: mysql
        Object Key:  Pod/mysql-cluster-mysql-1
        Status:      Succeed
        End Time:    2024-04-24T06:04:11Z
        Group:       mysql
        Message:     Successfully delete pod: Pod/mysql-cluster-mysql-2 in Component: mysql
        Object Key:  Pod/mysql-cluster-mysql-2
        Status:      Succeed
  Conditions:
    Last Transition Time:  2024-04-24T06:03:38Z
    Message:               wait for the controller to process the OpsRequest: mysql-cluster-stop-c8xqz in Cluster: mysql-cluster
    Reason:                WaitForProgressing
    Status:                True
    Type:                  WaitForProgressing
    Last Transition Time:  2024-04-24T06:03:38Z
    Message:               OpsRequest: mysql-cluster-stop-c8xqz is validated
    Reason:                ValidateOpsRequestPassed
    Status:                True
    Type:                  Validated
    Last Transition Time:  2024-04-24T06:03:38Z
    Message:               Start to stop the Cluster: mysql-cluster
    Reason:                StopStarted
    Status:                True
    Type:                  Stopping
  Last Configuration:
    Components:
      Mysql:
        Replicas:   3
  Phase:            Running
  Progress:         3/3
  Start Timestamp:  2024-04-24T06:03:38Z
Events:
  Type    Reason                    Age    From                    Message
  ----    ------                    ----   ----                    -------
  Normal  WaitForProgressing        6m30s  ops-request-controller  wait for the controller to process the OpsRequest: mysql-cluster-stop-c8xqz in Cluster: mysql-cluster
  Normal  ValidateOpsRequestPassed  6m30s  ops-request-controller  OpsRequest: mysql-cluster-stop-c8xqz is validated
  Normal  StopStarted               6m30s  ops-request-controller  Start to stop the Cluster: mysql-cluster
  Normal  Succeed                   5m57s  ops-request-controller  Successfully delete pod: Pod/mysql-cluster-mysql-0 in Component: mysql
  Normal  Succeed                   5m57s  ops-request-controller  Successfully delete pod: Pod/mysql-cluster-mysql-1 in Component: mysql
  Normal  Succeed                   5m57s  ops-request-controller  Successfully delete pod: Pod/mysql-cluster-mysql-2 in Component: mysql

logs kubeblocks

2024-04-24T06:04:10.366Z	ERROR	Reconciler error	{"controller": "instanceset", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "InstanceSet", "InstanceSet": {"name":"mysql-cluster-mysql","namespace":"default"}, "namespace": "default", "name": "mysql-cluster-mysql", "reconcileID": "c7826d79-016a-4898-8098-198857332a13", "error": "Operation cannot be fulfilled on instancesets.workloads.kubeblocks.io \"mysql-cluster-mysql\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
2024-04-24T06:04:10.431Z	INFO	component status conditions, isInstanceSetRunning: false, isAllConfigSynced: true, hasRunningVolumeExpansion: false, hasFailure: false,  isInCreatingPhase: false, isComponentAvailable: false	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"mysql-cluster-mysql","namespace":"default"}, "namespace": "default", "name": "mysql-cluster-mysql", "reconcileID": "fe8f097a-9f1a-4bec-8c83-ba052dddc1d4", "component": {"name":"mysql-cluster-mysql","namespace":"default"}}
2024-04-24T06:04:10.920Z	ERROR	delete *v1.Pod error: mysql-cluster-mysql-2	{"controller": "instanceset", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "InstanceSet", "InstanceSet": {"name":"mysql-cluster-mysql","namespace":"default"}, "namespace": "default", "name": "mysql-cluster-mysql", "reconcileID": "c21ff1d2-96af-4cc3-b4da-288df28e1414", "InstanceSet": {"name":"mysql-cluster-mysql","namespace":"default"}, "error": "Operation cannot be fulfilled on pods \"mysql-cluster-mysql-2\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/apecloud/kubeblocks/pkg/controller/kubebuilderx.(*PlanBuilder).deleteObject
	/src/pkg/controller/kubebuilderx/plan_builder.go:276
github.com/apecloud/kubeblocks/pkg/controller/kubebuilderx.(*PlanBuilder).defaultWalkFunc
	/src/pkg/controller/kubebuilderx/plan_builder.go:239
github.com/apecloud/kubeblocks/pkg/controller/kubebuilderx.(*Plan).Execute
	/src/pkg/controller/kubebuilderx/plan_builder.go:213
github.com/apecloud/kubeblocks/pkg/controller/kubebuilderx.(*controller).Commit
	/src/pkg/controller/kubebuilderx/controller.go:111
github.com/apecloud/kubeblocks/controllers/workloads.(*InstanceSetReconciler).Reconcile
	/src/controllers/workloads/instanceset_controller.go:92
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
2024-04-24T06:04:10.920Z	ERROR	Reconciler error	{"controller": "instanceset", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "InstanceSet", "InstanceSet": {"name":"mysql-cluster-mysql","namespace":"default"}, "namespace": "default", "name": "mysql-cluster-mysql", "reconcileID": "c21ff1d2-96af-4cc3-b4da-288df28e1414", "error": "Operation cannot be fulfilled on pods \"mysql-cluster-mysql-2\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
2024-04-24T06:04:10.995Z	INFO	component status conditions, isInstanceSetRunning: false, isAllConfigSynced: true, hasRunningVolumeExpansion: false, hasFailure: false,  isInCreatingPhase: false, isComponentAvailable: false	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"mysql-cluster-mysql","namespace":"default"}, "namespace": "default", "name": "mysql-cluster-mysql", "reconcileID": "db950370-4b59-42c0-8354-e40090f56436", "component": {"name":"mysql-cluster-mysql","namespace":"default"}}
2024-04-24T06:04:11.019Z	INFO	cluster component phase transition: Stopping -> Stopped (component is Stopped)	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"mysql-cluster","namespace":"default"}, "namespace": "default", "name": "mysql-cluster", "reconcileID": "4645eaf6-33c2-4cbd-8b22-22b1c6d497a7", "cluster": {"name":"mysql-cluster","namespace":"default"}}
2024-04-24T06:04:11.059Z	ERROR	Reconciler error	{"controller": "instanceset", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "InstanceSet", "InstanceSet": {"name":"mysql-cluster-mysql","namespace":"default"}, "namespace": "default", "name": "mysql-cluster-mysql", "reconcileID": "162ccadf-4baa-4e28-a7d3-2cd4dbd9a125", "error": "Operation cannot be fulfilled on instancesets.workloads.kubeblocks.io \"mysql-cluster-mysql\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
2024-04-24T06:04:11.197Z	INFO	component status conditions, isInstanceSetRunning: false, isAllConfigSynced: true, hasRunningVolumeExpansion: false, hasFailure: false,  isInCreatingPhase: false, isComponentAvailable: false	{"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"mysql-cluster-mysql","namespace":"default"}, "namespace": "default", "name": "mysql-cluster-mysql", "reconcileID": "4538ac33-e90a-44f5-9c67-3500938d3585", "component": {"name":"mysql-cluster-mysql","namespace":"default"}}
W0424 06:06:24.087895       1 warnings.go:70] Due to the lack of practical use cases, this API is deprecated from KB 0.9.0.
W0424 06:06:53.541697       1 warnings.go:70] The ClusterVersion CRD has been deprecated since 0.9.0
W0424 06:10:44.580198       1 warnings.go:70] Due to the lack of practical use cases, this API is deprecated from KB 0.9.0.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]
kbcli version 
Kubernetes: v1.26.14-gke.1044000
KubeBlocks: 0.9.0-beta.13
kbcli: 0.9.0-beta.3

Additional context
Add any other context about the problem here.
@JashBook JashBook added the kind/bug Something isn't working label Apr 24, 2024
@JashBook JashBook added this to the Release 0.9.0 milestone Apr 24, 2024
@ahjing99 ahjing99 added the severity/major Great chance user will encounter the same problem label Apr 24, 2024
@wangyelei wangyelei linked a pull request Apr 25, 2024 that will close this issue
chore: support to abort last running opsRequest with same kind when using 'force' flag #7165
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wangyelei wangyelei

Labels
kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Projects
None yet
Milestone
Release 0.9.0
3 participants
@wangyelei @ahjing99 @JashBook