Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]Mongo cluster is Running but all role are none #5426

Closed
ahjing99 opened this issue Oct 13, 2023 · 2 comments · Fixed by #5430
Closed

[BUG]Mongo cluster is Running but all role are none #5426

ahjing99 opened this issue Oct 13, 2023 · 2 comments · Fixed by #5430
Assignees
@derecknowayback
Labels
bug kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Milestone
Release 0.7.0

Comments

@ahjing99
Copy link
Collaborator

➜ ~ kbcli version
Kubernetes: v1.27.3-gke.100
KubeBlocks: 0.7.0-beta.3
kbcli: 0.7.0-beta.3

➜  ~ kbcli version
Kubernetes: v1.27.3-gke.100
KubeBlocks: 0.7.0-beta.3
kbcli: 0.7.0-beta.3

➜  ~ kbcli cluster describe mongo-huyvrr
Name: mongo-huyvrr	 Created Time: Oct 13,2023 09:46 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION       STATUS    TERMINATION-POLICY
default     mongodb              mongodb-5.0   Running   WipeOut

Endpoints:
COMPONENT   MODE        INTERNAL                                               EXTERNAL
mongodb     ReadWrite   mongo-huyvrr-mongodb.default.svc.cluster.local:27017   <none>

Topology:
COMPONENT   INSTANCE                 ROLE     STATUS    AZ              NODE                                                CREATED-TIME
mongodb     mongo-huyvrr-mongodb-0   <none>   Running   us-central1-c   gke-yjtest-default-pool-c51609d3-ss98/10.128.0.46   Oct 13,2023 09:46 UTC+0800
mongodb     mongo-huyvrr-mongodb-1   <none>   Running   us-central1-c   gke-yjtest-default-pool-c51609d3-jx9l/10.128.0.44   Oct 13,2023 09:46 UTC+0800
mongodb     mongo-huyvrr-mongodb-2   <none>   Running   us-central1-c   gke-yjtest-default-pool-c51609d3-jp0w/10.128.0.45   Oct 13,2023 09:46 UTC+0800

Resources Allocation:
COMPONENT   DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
mongodb     false       100m / 100m          512Mi / 512Mi           data:3Gi       kb-default-sc

Images:
COMPONENT   TYPE      IMAGE
mongodb     mongodb   registry.cn-hangzhou.aliyuncs.com/apecloud/mongo:5.0.14

Show cluster events: kbcli cluster list-events -n default mongo-huyvrr

➜  ~ kbcli cluster connect mongo-huyvrr
error: failed to find the instance to connect, please check cluster status

➜  ~ k describe pod mongo-huyvrr-mongodb-0
Name:         mongo-huyvrr-mongodb-0
Namespace:    default
Priority:     0
Node:         gke-yjtest-default-pool-c51609d3-ss98/10.128.0.46
Start Time:   Fri, 13 Oct 2023 09:46:30 +0800
Labels:       app.kubernetes.io/component=mongodb
              app.kubernetes.io/instance=mongo-huyvrr
              app.kubernetes.io/managed-by=kubeblocks
              app.kubernetes.io/name=mongodb
              app.kubernetes.io/version=mongodb-5.0
              apps.kubeblocks.io/component-name=mongodb
              apps.kubeblocks.io/workload-type=Consensus
              controller-revision-hash=mongo-huyvrr-mongodb-5d6cb78b9c
              statefulset.kubernetes.io/pod-name=mongo-huyvrr-mongodb-0
Annotations:  apps.kubeblocks.io/component-replicas: 3
Status:       Running
IP:           10.104.2.191
IPs:
  IP:           10.104.2.191
Controlled By:  StatefulSet/mongo-huyvrr-mongodb
Containers:
  mongodb:
    Container ID:  containerd://365b5ea6c1291bb77543700128a5abefa1565b4bfb7fd018a6c77f0d66525095
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/mongo:5.0.14
    Image ID:      registry.cn-hangzhou.aliyuncs.com/apecloud/mongo@sha256:6cf671de7c85493d6dde9aafef383570161809336d6c41f431ac3a692155ece9
    Port:          27017/TCP
    Host Port:     0/TCP
    Command:
      /scripts/replicaset-setup.sh
    State:          Running
      Started:      Fri, 13 Oct 2023 09:46:37 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  512Mi
    Requests:
      cpu:     100m
      memory:  512Mi
    Environment Variables from:
      mongo-huyvrr-mongodb-env      ConfigMap  Optional: false
      mongo-huyvrr-mongodb-rsm-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               mongo-huyvrr-mongodb-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           mongo-huyvrr
      KB_COMP_NAME:              mongodb
      KB_CLUSTER_COMP_NAME:      mongo-huyvrr-mongodb
      KB_CLUSTER_UID_POSTFIX_8:  3475a282
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      MONGODB_ROOT_USER:         <set to the key 'username' in secret 'mongo-huyvrr-conn-credential'>  Optional: false
      MONGODB_ROOT_PASSWORD:     <set to the key 'password' in secret 'mongo-huyvrr-conn-credential'>  Optional: false
    Mounts:
      /data/mongodb from data (rw)
      /etc/mongodb/keyfile from mongodb-config (rw,path="keyfile")
      /etc/mongodb/mongodb.conf from mongodb-config (rw,path="mongodb.conf")
      /scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-v8jsz (ro)
  metrics:
    Container ID:  containerd://577c6d8cff548e26671d7c6a652cab5afaea23371fcb0217c6a8d935cc6cf172
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1
    Image ID:      registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto@sha256:cbab349b90490807a8d5039bf01bc7e37334f20c98c7dd75bc7fc4cf9e5b10ee
    Port:          9216/TCP
    Host Port:     0/TCP
    Command:
      /bin/agamotto
      --config=/opt/conf/metrics-config.yaml
    State:          Running
      Started:      Fri, 13 Oct 2023 09:46:37 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:     0
      memory:  0
    Environment Variables from:
      mongo-huyvrr-mongodb-env      ConfigMap  Optional: false
      mongo-huyvrr-mongodb-rsm-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:               mongo-huyvrr-mongodb-0 (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:              default (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           mongo-huyvrr
      KB_COMP_NAME:              mongodb
      KB_CLUSTER_COMP_NAME:      mongo-huyvrr-mongodb
      KB_CLUSTER_UID_POSTFIX_8:  3475a282
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      MONGODB_ROOT_USER:         <set to the key 'username' in secret 'mongo-huyvrr-conn-credential'>  Optional: false
      MONGODB_ROOT_PASSWORD:     <set to the key 'password' in secret 'mongo-huyvrr-conn-credential'>  Optional: false
    Mounts:
      /opt/conf from mongodb-metrics-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-v8jsz (ro)
  kb-checkrole:
    Container ID:  containerd://364b2b7de52fa7597aea04d252b5a20598daf13dee0826d2ae94f7ad9aece52c
    Image:         registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.7.0-beta.3
    Image ID:      registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools@sha256:b30e6017777e4e136cd10f52a918831c83ee62916d2aa60ad1d9c2c995b904f7
    Ports:         3501/TCP, 50001/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      lorry
      --port
      3501
      --grpcport
      50001
    State:          Running
      Started:      Fri, 13 Oct 2023 09:46:37 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     0
      memory:  0
    Requests:
      cpu:      0
      memory:   0
    Readiness:  http-get http://:3501/v1.0/bindings/mongodb%3Foperation=checkRole&workloadType=Consensus delay=0s timeout=2s period=2s #success=1 #failure=3
    Startup:    tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      mongo-huyvrr-mongodb-env      ConfigMap  Optional: false
      mongo-huyvrr-mongodb-rsm-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                mongo-huyvrr-mongodb-0 (v1:metadata.name)
      KB_POD_UID:                  (v1:metadata.uid)
      KB_NAMESPACE:               default (v1:metadata.namespace)
      KB_SA_NAME:                  (v1:spec.serviceAccountName)
      KB_NODENAME:                 (v1:spec.nodeName)
      KB_HOST_IP:                  (v1:status.hostIP)
      KB_POD_IP:                   (v1:status.podIP)
      KB_POD_IPS:                  (v1:status.podIPs)
      KB_HOSTIP:                   (v1:status.hostIP)
      KB_PODIP:                    (v1:status.podIP)
      KB_PODIPS:                   (v1:status.podIPs)
      KB_CLUSTER_NAME:            mongo-huyvrr
      KB_COMP_NAME:               mongodb
      KB_CLUSTER_COMP_NAME:       mongo-huyvrr-mongodb
      KB_CLUSTER_UID_POSTFIX_8:   3475a282
      KB_POD_FQDN:                $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      KB_SERVICE_USER:            <set to the key 'username' in secret 'mongo-huyvrr-conn-credential'>  Optional: false
      KB_SERVICE_PASSWORD:        <set to the key 'password' in secret 'mongo-huyvrr-conn-credential'>  Optional: false
      KB_SERVICE_PORT:            27017
      KB_DATA_PATH:               /data/mongodb
      KB_SERVICE_CHARACTER_TYPE:  mongodb
      KB_WORKLOAD_TYPE:           Consensus
      KB_SERVICE_USER:            <set to the key 'username' in secret 'mongo-huyvrr-conn-credential'>  Optional: false
      KB_SERVICE_PASSWORD:        <set to the key 'password' in secret 'mongo-huyvrr-conn-credential'>  Optional: false
    Mounts:
      /data/mongodb from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-v8jsz (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-mongo-huyvrr-mongodb-0
    ReadOnly:   false
  mongodb-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mongo-huyvrr-mongodb-mongodb-config
    Optional:  false
  mongodb-metrics-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mongo-huyvrr-mongodb-mongodb-metrics-config-new
    Optional:  false
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mongo-huyvrr-mongodb-mongodb-scripts
    Optional:  false
  kube-api-access-v8jsz:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 kb-data=true:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason                  Age    From                     Message
  ----    ------                  ----   ----                     -------
  Normal  Scheduled               5m56s  default-scheduler        Successfully assigned default/mongo-huyvrr-mongodb-0 to gke-yjtest-default-pool-c51609d3-ss98
  Normal  SuccessfulAttachVolume  5m51s  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-a1ab3e36-2f6c-4d25-bc60-d35482f92f9b"
  Normal  Pulled                  5m50s  kubelet                  Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/mongo:5.0.14" already present on machine
  Normal  Created                 5m50s  kubelet                  Created container mongodb
  Normal  Started                 5m49s  kubelet                  Started container mongodb
  Normal  Pulled                  5m49s  kubelet                  Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1" already present on machine
  Normal  Created                 5m49s  kubelet                  Created container metrics
  Normal  Started                 5m49s  kubelet                  Started container metrics
  Normal  Pulled                  5m49s  kubelet                  Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.7.0-beta.3" already present on machine
  Normal  Created                 5m49s  kubelet                  Created container kb-checkrole
  Normal  Started                 5m49s  kubelet                  Started container kb-checkrole


k logs mongo-huyvrr-mongodb-0 -c kb-checkrole
...
2023-10-13T01:52:51Z	ERROR	Mongo	DB is not ready	{"error": "server selection error: context deadline exceeded, current topology: { Type: Single, Servers: [{ Addr: localhost:27017, Type: Unknown }, ] }"}
github.com/apecloud/kubeblocks/lorry/component/mongodb.(*Manager).IsDBStartupReady
	/src/lorry/component/mongodb/manager.go:273
github.com/apecloud/kubeblocks/lorry/binding/mongodb.(*MongoDBOperations).Init.(*BaseOperations).RegisterOperationOnDBReady.StartupCheckWraper.func2
	/src/lorry/binding/utils.go:333
github.com/apecloud/kubeblocks/lorry/binding.(*BaseOperations).Invoke
	/src/lorry/binding/base.go:194
github.com/apecloud/kubeblocks/lorry/middleware/probe.route
	/src/lorry/middleware/probe/router.go:160
main.main.GetRouter.func3
	/src/lorry/middleware/probe/router.go:127
main.main.SetMiddleware.func4
	/src/lorry/middleware/probe/checks_middleware.go:105
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2136
net/http.(*ServeMux).ServeHTTP
	/usr/local/go/src/net/http/server.go:2514
net/http.serverHandler.ServeHTTP
	/usr/local/go/src/net/http/server.go:2938
net/http.(*conn).serve
	/usr/local/go/src/net/http/server.go:2009

➜ ~ k logs mongo-huyvrr-mongodb-0 -c kb-checkrole >checkrole.txt

checkrole.txt
@ahjing99 ahjing99 added kind/bug Something isn't working severity/major Great chance user will encounter the same problem labels Oct 13, 2023
@ahjing99 ahjing99 added this to the Release 0.7.0 milestone Oct 13, 2023
@xuriwuyun
Copy link
Contributor

MongoDB initialization fails due to changes introduced in PR #5353

@derecknowayback
Copy link
Contributor

WIP

@derecknowayback derecknowayback linked a pull request Oct 13, 2023 that will close this issue
fix: fix grpc.serve block bug #5430
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@derecknowayback derecknowayback

Labels
bug kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Projects
None yet
Milestone
Release 0.7.0
Development

Successfully merging a pull request may close this issue.

fix: fix grpc.serve block bug apecloud/kubeblocks
4 participants
@xuriwuyun @derecknowayback @ahjing99 @apecloud-bot