Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] pulsar cluster broker crash Failed to start Pulsar service: Unable to read VM NIC speed #8750

Closed
JashBook opened this issue Jan 6, 2025 · 1 comment
Closed

[BUG] pulsar cluster broker crash Failed to start Pulsar service: Unable to read VM NIC speed #8750

JashBook opened this issue Jan 6, 2025 · 1 comment
Assignees
@Y-Rookie
Labels
kind/bug Something isn't working
Milestone
Release 0.9.3

Comments

@JashBook
Copy link
Collaborator

JashBook commented Jan 6, 2025

Describe the bug

kbcli version
Kubernetes: v1.28.3-vke.17
KubeBlocks: 1.0.0-beta.21
kbcli: 1.0.0-beta.8

To Reproduce
Steps to reproduce the behavior:

  1. create cluster
apiVersion: apps.kubeblocks.io/v1
kind: Cluster
metadata:
  name: pulsar-cluster
  namespace: default
  labels: 
    helm.sh/chart: pulsar-cluster-1.0.0-alpha.0
    app.kubernetes.io/version: "3.0.2"
    app.kubernetes.io/instance: pulsar-cluster
  annotations:
    resource.kubeblocks.io/ignore-constraint: "true"
spec:
  terminationPolicy: Delete
  services:
    - name: broker-bootstrap
      serviceName: broker-bootstrap
      componentSelector: broker
      spec:
        type: ClusterIP
        ports:
          - name: pulsar
            port: 6650
            targetPort: 6650
          - name: http
            port: 80
            targetPort: 8080
          - name: kafka-client
            port: 9092
            targetPort: 9092
    - name: zookeeper
      serviceName: zookeeper
      componentSelector: zookeeper
      spec:
        type: ClusterIP
        ports:
          - name: client
            port: 2181
            targetPort: 2181
  componentSpecs:
    - name: proxy
      componentDef: pulsar-proxy
      serviceVersion: 3.0.2
      
      
      replicas: 3
      resources:
        limits:
          cpu: 
          memory: "512Mi"
        requests:
          cpu: "200m"
          memory: "512Mi"
    - name: bookies-recovery
      componentDef: pulsar-bookies-recovery
      serviceVersion: 3.0.2
      
      
      replicas: 1
      resources:
        limits:
          cpu: 
          memory: "512Mi"
        requests:
          cpu: "200m"
          memory: "512Mi"
    - name: broker
      componentDef: pulsar-broker
      serviceVersion: 3.0.2
      env:
        - name: KB_PULSAR_BROKER_NODEPORT
          value: "false"
      
      
      replicas: 1
      resources:
        limits:
          cpu: 
          memory: "512Mi"
        requests:
          cpu: "200m"
          memory: "512Mi"
    - name: bookies
      componentDef: pulsar-bookkeeper
      serviceVersion: 3.0.2
      
      
      replicas: 4
      resources:
        limits:
          cpu: 
          memory: "512Mi"
        requests:
          cpu: "200m"
          memory: "512Mi"
      volumeClaimTemplates:
        - name: ledgers
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
        - name: journal
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
    - name: zookeeper
      componentDef: pulsar-zookeeper
      serviceVersion: 3.0.2
      replicas: 1
      resources:
        limits:
          cpu: 
          memory: "512Mi"
        requests:
          cpu: "100m"
          memory: "512Mi"
      volumeClaimTemplates:
        - name: data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 20Gi
  1. Click on '....'
  2. Scroll down to '....'
  3. See error
➜  ~ kubectl get cluster  pulsar-cluster
NAME             CLUSTER-DEFINITION   TERMINATION-POLICY   STATUS     AGE
pulsar-cluster                        Delete               Abnormal   2m56s
➜  ~ 
➜  ~ 
➜  ~ kubectl get pod -l app.kubernetes.io/instance=pulsar-cluster 
NAME                                READY   STATUS             RESTARTS      AGE
pulsar-cluster-bookies-0            2/2     Running            0             3m1s
pulsar-cluster-bookies-1            2/2     Running            0             3m1s
pulsar-cluster-bookies-2            2/2     Running            0             3m1s
pulsar-cluster-bookies-3            2/2     Running            0             3m1s
pulsar-cluster-bookies-recovery-0   1/1     Running            0             3m5s
pulsar-cluster-broker-0             1/2     CrashLoopBackOff   4 (35s ago)   3m2s
pulsar-cluster-proxy-0              1/1     Running            0             3m4s
pulsar-cluster-proxy-1              1/1     Running            0             3m4s
pulsar-cluster-proxy-2              1/1     Running            0             3m4s
pulsar-cluster-zookeeper-0          1/1     Running            0             3m4s

logs pod

kubectl logs pulsar-cluster-broker-0
Defaulted container "broker" out of: broker, config-manager, init-broker-cluster (init), init-sysctl (init), init-pulsar-tools (init), init-pulsar-client-config (init)
[conf/client.conf] Adding config: chunkingEnabled=false
[conf/client.conf] Adding config: compressionType=NONE
[conf/broker.conf] Updating config acknowledgmentAtBatchIndexLevelEnabled=true
[conf/broker.conf] Updating config backlogQuotaDefaultLimitGB=-1
[conf/broker.conf] Updating config backlogQuotaDefaultRetentionPolicy=producer_request_hold
[conf/broker.conf] Updating config bookkeeperClientReorderReadSequenceEnabled=true
[conf/broker.conf] Updating config bookkeeperDiskWeightBasedPlacementEnabled=true
[conf/broker.conf] Updating config bookkeeperEnableStickyReads=true
[conf/broker.conf] Updating config defaultNamespaceBundleSplitAlgorithm=topic_count_equally_divide
[conf/broker.conf] Updating config defaultNumPartitions=1
[conf/broker.conf] Updating config defaultNumberOfNamespaceBundles=4
[conf/broker.conf] Updating config delayedDeliveryEnabled=true
[conf/broker.conf] Updating config dispatcherMaxReadBatchSize=1000
[conf/broker.conf] Updating config enableReplicatedSubscriptions=false
[conf/broker.conf] Updating config managedLedgerNumSchedulerThreads=2
[conf/broker.conf] Adding config: managedLedgerNumWorkerThreads=2
[conf/broker.conf] Updating config maxConcurrentLookupRequest=1000
[conf/broker.conf] Updating config maxConcurrentTopicLoadRequest=1000
[conf/broker.conf] Updating config maxConsumersPerSubscription=5000
[conf/broker.conf] Updating config maxConsumersPerTopic=50000
[conf/broker.conf] Updating config maxNumPartitionsPerPartitionedTopic=2048
[conf/broker.conf] Updating config maxProducersPerTopic=10000
[conf/broker.conf] Updating config maxUnackedMessagesPerConsumer=10000
[conf/broker.conf] Updating config maxUnackedMessagesPerSubscription=50000
[conf/broker.conf] Updating config preciseDispatcherFlowControl=true
[conf/broker.conf] Updating config subscriptionKeySharedUseConsistentHashing=true
[conf/broker.conf] Updating config transactionCoordinatorEnabled=true
[conf/broker.conf] Updating config systemTopicEnabled=true
[conf/broker.conf] Updating config statusFilePath=/pulsar/status
[conf/broker.conf] Updating config subscriptionKeySharedEnable=true
[conf/broker.conf] Adding config: brokerInterceptors=
[conf/broker.conf] Adding config: messagingProtocols=kafka
[conf/broker.conf] Updating config allowAutoTopicCreationType=partitioned
[conf/broker.conf] Adding config: brokerEntryMetadataInterceptors=org.apache.pulsar.common.intercept.AppendIndexMetadataInterceptor
[conf/broker.conf] Updating config brokerDeleteInactiveTopicsEnabled=false
[conf/broker.conf] Adding config: kafkaTransactionCoordinatorEnabled=true
[conf/broker.conf] Updating config brokerDeduplicationEnabled=true
[conf/broker.conf] Applying config advertisedListeners = cluster:pulsar://pulsar-cluster-broker-0.pulsar-cluster-broker-headless.default.svc.cluster.local:6650
[conf/broker.conf] Applying config allowAutoTopicCreationType = partitioned
[conf/broker.conf] Applying config internalListenerName = cluster
[conf/broker.conf] Applying config clusterName = default-pulsar-cluster-broker
[conf/broker.conf] Applying config configurationStoreServers = pulsar-cluster-zookeeper.default.svc:2181
[conf/broker.conf] Applying config zookeeperServers = pulsar-cluster-zookeeper.default.svc:2181
[conf/broker.conf] Updating config advertisedListeners = cluster:pulsar://pulsar-cluster-broker-0.pulsar-cluster-broker-headless.default.svc.cluster.local:6650
[conf/broker.conf] Updating config allowAutoTopicCreationType = partitioned
[conf/broker.conf] Updating config internalListenerName = cluster
[conf/broker.conf] Adding config kafkaAdvertisedListeners = CLIENT://pulsar-cluster-broker-0.pulsar-cluster-broker-headless.default.svc.cluster.local:9092
[conf/broker.conf] Adding config kafkaListeners = CLIENT://0.0.0.0:9092
[conf/broker.conf] Adding config kafkaProtocolMap = CLIENT:PLAINTEXT
[conf/client.conf] Applying config brokerServiceUrl = pulsar://pulsar-cluster-broker-bootstrap.default.svc.cluster.local:6650
[conf/client.conf] Applying config webServiceUrl = http://pulsar-cluster-broker-bootstrap.default.svc.cluster.local:80
[conf/client.conf] Adding config advertisedListeners = cluster:pulsar://pulsar-cluster-broker-0.pulsar-cluster-broker-headless.default.svc.cluster.local:6650
[conf/client.conf] Adding config allowAutoTopicCreationType = partitioned
[conf/client.conf] Adding config internalListenerName = cluster
[conf/client.conf] Adding config kafkaAdvertisedListeners = CLIENT://pulsar-cluster-broker-0.pulsar-cluster-broker-headless.default.svc.cluster.local:9092
[conf/client.conf] Adding config kafkaListeners = CLIENT://0.0.0.0:9092
[conf/client.conf] Adding config kafkaProtocolMap = CLIENT:PLAINTEXT
VM settings:
    Max. Heap Size (Estimated): 154.00M
    Using VM: OpenJDK 64-Bit Server VM

2025-01-06T08:50:33,421+0000 [main] INFO  org.apache.pulsar.broker.PulsarService - Starting Pulsar Broker service; version: '3.0.2'
2025-01-06T08:50:33,423+0000 [main] INFO  org.apache.pulsar.broker.PulsarService - Git Revision 12c92fed7847965e3bc3769a91c866b2f0ec2e44(dirty)
2025-01-06T08:50:33,424+0000 [main] INFO  org.apache.pulsar.broker.PulsarService - Git Branch branch-3.0
2025-01-06T08:50:33,424+0000 [main] INFO  org.apache.pulsar.broker.PulsarService - Built by fengyubiao <[email protected]> on fengyubiaodeMacBook-Pro.local at 2023-11-17T15:50:55+0800
2025-01-06T08:50:33,432+0000 [main] ERROR org.apache.pulsar.broker.PulsarService - Failed to start Pulsar service: Unable to read VM NIC speed. You must set [loadBalancerOverrideBrokerNicSpeedGbps] to override it when load balancer is enabled.
java.lang.IllegalStateException: Unable to read VM NIC speed. You must set [loadBalancerOverrideBrokerNicSpeedGbps] to override it when load balancer is enabled.
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:736) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:276) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:356) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
2025-01-06T08:50:33,437+0000 [main] ERROR org.apache.pulsar.PulsarBrokerStarter - Failed to start pulsar service.
org.apache.pulsar.broker.PulsarServerException: java.lang.IllegalStateException: Unable to read VM NIC speed. You must set [loadBalancerOverrideBrokerNicSpeedGbps] to override it when load balancer is enabled.
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:931) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:276) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:356) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
Caused by: java.lang.IllegalStateException: Unable to read VM NIC speed. You must set [loadBalancerOverrideBrokerNicSpeedGbps] to override it when load balancer is enabled.
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:736) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]
	... 2 more
2025-01-06T08:50:33,438+0000 [main] WARN  org.apache.pulsar.common.util.ShutdownUtil - Triggering immediate shutdown of current process with status 1
java.lang.Exception: Stacktrace for immediate shutdown
	at org.apache.pulsar.common.util.ShutdownUtil.triggerImmediateForcefulShutdown(ShutdownUtil.java:55) ~[org.apache.pulsar-pulsar-common-3.0.2.jar:3.0.2]
	at org.apache.pulsar.common.util.ShutdownUtil.triggerImmediateForcefulShutdown(ShutdownUtil.java:50) ~[org.apache.pulsar-pulsar-common-3.0.2.jar:3.0.2]
	at org.apache.pulsar.common.util.ShutdownUtil.triggerImmediateForcefulShutdown(ShutdownUtil.java:82) ~[org.apache.pulsar-pulsar-common-3.0.2.jar:3.0.2]
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:359) ~[org.apache.pulsar-pulsar-broker-3.0.2.jar:3.0.2]

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
@JashBook JashBook added the kind/bug Something isn't working label Jan 6, 2025
@JashBook JashBook added this to the Release 1.0.0 milestone Jan 6, 2025
@caiq1nyu
Copy link
Collaborator

caiq1nyu commented Jan 6, 2025

Pulsar currently does not support the vNIC environment and requires iteration support

@github-actions github-actions bot modified the milestones: Release 1.0.0, Release 0.9.3 Jan 14, 2025
@shanshanying shanshanying closed this as not planned Won't fix, can't repro, duplicate, stale Jan 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Y-Rookie Y-Rookie

Labels
kind/bug Something isn't working
Projects
None yet
Milestone
Release 0.9.3
Development

No branches or pull requests
4 participants
@caiq1nyu @Y-Rookie @JashBook @shanshanying