[BUG] risingwave cluster creation failed in kubeblocks e2e testing

[dengkailu]

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Create risingwave cluster
   `apiVersion: apps.kubeblocks.io/v1alpha1
   kind: Cluster
   metadata:
   name: risingwave-cluster
   namespace: default
   labels:
   helm.sh/chart: risingwave-cluster-0.1.0
   app.kubernetes.io/name: risingwave-cluster
   app.kubernetes.io/instance: risingwave-cluster
   app.kubernetes.io/version: "v1.0.0"
   app.kubernetes.io/managed-by: Helm
   annotations:
   "kubeblocks.io/extra-env": "{"RW_STATE_STORE":"hummock+s3://REPLACE-WITH-YOUR-BUCKET","AWS_REGION":"REPLACE-WITH-YOUR-REGION","AWS_ACCESS_KEY_ID":"REPLACE-WITH-YOUR-AK","AWS_SECRET_ACCESS_KEY":"REPLACE-WITH-YOUR-SK","RW_DATA_DIRECTORY":"risingwave","RW_S3_ENDPOINT":"https://s3.REPLACE-WITH-YOUR-REGION.amazonaws.com.cn\",\"RW_ETCD_ENDPOINTS\":\"REPLACE-WITH-YOUR-ETCD-ENDPOINT:2379\",\"RW_ETCD_AUTH\":\"false\"}"
   spec:
   clusterDefinitionRef: risingwave
   clusterVersionRef: risingwave-v1.0.0
   terminationPolicy: Delete
   affinity:
   topologyKeys:
   • kubernetes.io/hostname
     componentSpecs:

• componentDefRef: frontend
  name: frontend
  replicas: 1
  serviceAccountName:
  resources:
  limits:
  cpu: "1"
  memory: "1Gi"
  requests:
  cpu: "500m"
  memory: "500Mi"
• componentDefRef: meta
  name: meta
  replicas: 1
  serviceAccountName:
  resources:
  limits:
  cpu: "1"
  memory: "1Gi"
  requests:
  cpu: "500m"
  memory: "500Mi"
• componentDefRef: compute
  name: compute
  replicas: 1
  serviceAccountName:
  resources:
  limits:
  cpu: "1"
  memory: "1Gi"
  requests:
  cpu: "500m"
  memory: "500Mi"
• componentDefRef: compactor
  name: compactor
  replicas: 1
  serviceAccountName:
  resources:
  limits:
  cpu: "1"
  memory: "1Gi"
  requests:
  cpu: "500m"
  memory: "500Mi"
• componentDefRef: connector
  name: connector
  replicas: 1
  serviceAccountName:
  resources:
  limits:
  cpu: "1"
  memory: "1Gi"
  requests:
  cpu: "500m"
  memory: "500Mi"`

2. see error
   k get pods NAME READY STATUS RESTARTS AGE risingwave-cluster-compactor-0 0/1 CrashLoopBackOff 15 (3m55s ago) 44m risingwave-cluster-compute-0 0/1 CrashLoopBackOff 15 (3m36s ago) 44m risingwave-cluster-connector-0 1/1 Running 0 44m risingwave-cluster-frontend-0 0/1 CrashLoopBackOff 15 (3m36s ago) 44m risingwave-cluster-meta-0 0/1 CrashLoopBackOff 15 (3m35s ago) 44m

3. Risingwave cluster meta-0 failed to start "client risingwave-cluster-meta-0.risingwave-cluster-meta-headless.default.svc:5690"
   k logs risingwave-cluster-meta-0 |head -50 launching meta`
   2024-04-14T13:09:02.372455557Z INFO risingwave_meta: Starting meta node
   at src/meta/src/lib.rs:210

2024-04-14T13:09:02.372497757Z INFO risingwave_meta: > options: MetaNodeOpts { vpc_id: None, security_group_id: None, listen_addr: "0.0.0.0:5690", advertise_addr: "risingwave-cluster-meta-0.risingwave-cluster-meta-headless.default.svc:5690", dashboard_host: Some("0.0.0.0:5691"), prometheus_host: Some("0.0.0.0:1250"), etcd_endpoints: "REPLACE-WITH-YOUR-ETCD-ENDPOINT:2379", etcd_auth: false, etcd_username: "", etcd_password: "", dashboard_ui_path: Some("/risingwave/ui"), prometheus_endpoint: None, connector_rpc_endpoint: Some("risingwave-cluster-connector:50051"), privatelink_endpoint_default_tags: None, config_path: "/risingwave/config/risingwave.toml", override_opts: OverrideConfigOpts { backend: Some(Etcd), barrier_interval_ms: None, sstable_size_mb: None, block_size_kb: None, bloom_false_positive: None, state_store: Some("hummock+s3://REPLACE-WITH-YOUR-BUCKET"), data_directory: Some("risingwave"), do_not_config_object_storage_lifecycle: None, backup_storage_url: None, backup_storage_directory: None, object_store_streaming_read_timeout_ms: None, object_store_streaming_upload_timeout_ms: None, object_store_upload_timeout_ms: None, object_store_read_timeout_ms: None } }
at src/meta/src/lib.rs:211

2024-04-14T13:09:02.372632459Z INFO risingwave_meta: > config: RwConfig { server: ServerConfig { heartbeat_interval_ms: 1000, connection_pool_size: 16, metrics_level: 0, telemetry_enabled: true, unrecognized: {} }, meta: MetaConfig { min_sst_retention_time_sec: 604800, collect_gc_watermark_spin_interval_sec: 5, periodic_compaction_interval_sec: 60, vacuum_interval_sec: 30, hummock_version_checkpoint_interval_sec: 30, min_delta_log_num_for_hummock_version_checkpoint: 10, max_heartbeat_interval_secs: 300, disable_recovery: false, meta_leader_lease_secs: 30, dangerous_max_idle_secs: None, default_parallelism: Full, enable_compaction_deterministic: false, enable_committed_sst_sanity_check: false, node_num_monitor_interval_sec: 10, backend: Etcd, periodic_space_reclaim_compaction_interval_sec: 3600, periodic_ttl_reclaim_compaction_interval_sec: 1800, periodic_split_compact_group_interval_sec: 180, max_compactor_task_multiplier: 2, move_table_size_limit: 4294967296, split_group_size_limit: 68719476736, unrecognized: {}, do_not_config_object_storage_lifecycle: false, partition_vnode_count: 64, table_write_throughput_threshold: 134217728, min_table_split_write_throughput: 33554432, compaction_task_max_heartbeat_interval_secs: 60 }, batch: BatchConfig { worker_threads_num: None, developer: BatchDeveloperConfig { connector_message_buffer_size: 16, output_channel_size: 64, chunk_size: 1024 }, distributed_query_limit: None, enable_barrier_read: true, unrecognized: {} }, streaming: StreamingConfig { in_flight_barrier_nums: 10000, actor_runtime_worker_threads_num: None, async_stack_trace: ReleaseVerbose, developer: StreamingDeveloperConfig { connector_message_buffer_size: 16, unsafe_extreme_cache_size: 10, chunk_size: 256, exchange_initial_permits: 2048, exchange_batched_permits: 256, exchange_concurrent_barriers: 1, dml_channel_initial_permits: 32768 }, unique_user_stream_errors: 10, unrecognized: {} }, storage: StorageConfig { share_buffers_sync_parallelism: 1, share_buffer_compaction_worker_threads_number: 4, shared_buffer_capacity_mb: None, shared_buffer_flush_ratio: 0.8, imm_merge_threshold: 4, write_conflict_detection_enabled: false, block_cache_capacity_mb: None, high_priority_ratio_in_percent: None, meta_cache_capacity_mb: None, disable_remote_compactor: false, share_buffer_upload_concurrency: 8, compactor_memory_limit_mb: None, sstable_id_remote_fetch_number: 10, file_cache: FileCacheConfig { dir: "", capacity_mb: 1024, total_buffer_capacity_mb: None, cache_file_fallocate_unit_mb: 512, cache_meta_fallocate_unit_mb: 16, cache_file_max_write_size_mb: 4, unrecognized: {} }, min_sst_size_for_streaming_upload: 33554432, max_sub_compaction: 4, max_concurrent_compaction_task_number: 16, max_preload_wait_time_mill: 10, object_store_streaming_read_timeout_ms: 600000, object_store_streaming_upload_timeout_ms: 600000, object_store_upload_timeout_ms: 3600000, object_store_read_timeout_ms: 3600000, unrecognized: {} }, unrecognized: {} }
at src/meta/src/lib.rs:213

2024-04-14T13:09:02.372675659Z INFO risingwave_meta: > version: 1.0.0 (c320675ef628c0c8d6bab7d60b90141d9c41adf2)
at src/meta/src/lib.rs:214

2024-04-14T13:09:02.372692359Z INFO risingwave_meta: Meta server listening at 0.0.0.0:5690
at src/meta/src/lib.rs:249

2024-04-14T13:09:02.37364457Z INFO risingwave_meta::rpc::election_client: client risingwave-cluster-meta-0.risingwave-cluster-meta-headless.default.svc:5690 start election
at src/meta/src/rpc/election_client.rs:77

2024-04-14T13:09:02.384957996Z ERROR risingwave_meta::rpc::server: election error happened, Election failed: grpc request error: status: Unavailable, message: "error trying to connect: dns error: failed to lookup address information: Name or service not known", details: [], metadata: MetadataMap { headers: {} }
at src/meta/src/rpc/server.rs:202

2024-04-14T13:09:02.384990096Z INFO risingwave_meta::rpc::election_client: client risingwave-cluster-meta-0.risingwave-cluster-meta-headless.default.svc:5690 start election
at src/meta/src/rpc/election_client.rs:77

2024-04-14T13:09:02.390079452Z ERROR risingwave_meta::rpc::server: election error happened, Election failed: grpc request error: status: Unavailable, message: "error trying to connect: dns error: failed to lookup address information: Name or service not known", details: [], metadata: MetadataMap { headers: {} }
at src/meta/src/rpc/server.rs:202

2024-04-14T13:09:02.390102353Z INFO risingwave_meta::rpc::election_client: client risingwave-cluster-meta-0.risingwave-cluster-meta-headless.default.svc:5690 start election
at src/meta/src/rpc/election_client.rs:77

2024-04-14T13:09:02.395931518Z ERROR risingwave_meta::rpc::server: election error happened, Election failed: grpc request error: status: Unavailable, message: "error trying to connect: dns error: failed to lookup address information: Name or service not known", details: [], metadata: MetadataMap { headers: {} }
at src/meta/src/rpc/server.rs:202

2024-04-14T13:09:02.395950418Z INFO risingwave_meta::rpc::election_client: client risingwave-cluster-meta-0.risingwave-cluster-meta-headless.default.svc:5690 start election
at src/meta/src/rpc/election_client.rs:77

2024-04-14T13:09:02.400380767Z ERROR risingwave_meta::rpc::server: election error happened, Election failed: grpc request error: status: Unavailable, message: "error trying to connect: dns error: failed to lookup address information: Name or service not known", details: [], metadata: MetadataMap { headers: {} }
at src/meta/src/rpc/server.rs:202

2024-04-14T13:09:02.400396667Z INFO risingwave_meta::rpc::election_client: client risingwave-cluster-meta-0.risingwave-cluster-meta-headless.default.svc:5690 start election
at src/meta/src/rpc/election_client.rs:77

2024-04-14T13:09:02.405642025Z ERROR risingwave_meta::rpc::server: election error happened, Election failed: grpc request error: status: Unavailable, message: "error trying to connect: dns error: failed to lookup address information: Name or service not known", details: [], metadata: MetadataMap { headers: {} }
at src/meta/src/rpc/server.rs:202

2024-04-14T13:09:02.405657526Z INFO risingwave_meta::rpc::election_client: client risingwave-cluster-meta-0.risingwave-cluster-meta-headless.default.svc:5690 start election
at src/meta/src/rpc/election_client.rs:77

2024-04-14T13:09:02.410404978Z ERROR risingwave_meta::rpc::server: election error happened, Election failed: grpc request error: status: Unavailable, message: "error trying to connect: dns error: failed to lookup address information: Name or service not known", details: [], metadata: MetadataMap { headers: {} }`

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):
kb version Kubernetes: v1.27.9 KubeBlocks: 0.8.2 kbcli: 0.8.2

Additional context
Add any other context about the problem here.