[BUG]opensearch monitoring access reported a 404 error

[0120208]

Describe the bug
Opensearch cluster file after open the monitor, start the service visit http://localhost:9600/metrics error 404

To Reproduce
Steps to reproduce the behavior:

1. deploy kb opensearch

kbcli addon enable opensearch  
kubectl edit cd opensearch

The modified CD file is as follows：

apiVersion: apps.kubeblocks.io/v1alpha1
kind: ClusterDefinition
metadata:
  annotations:
    meta.helm.sh/release-name: kb-addon-opensearch
    meta.helm.sh/release-namespace: kb-system
  creationTimestamp: "2023-11-21T09:50:31Z"
  finalizers:
  - clusterdefinition.kubeblocks.io/finalizer
  generation: 6
  labels:
    app.kubernetes.io/instance: kb-addon-opensearch
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: opensearch
    app.kubernetes.io/version: 2.7.0
    config.kubeblocks.io/tpl-opensearch-config-template: opensearch-config-template
    helm.sh/chart: opensearch-0.7.0
  name: opensearch
  resourceVersion: "78596258"
  uid: f9d5e52f-696a-4232-9b9c-dc2a3e154b78
spec:
  componentDefs:
  - characterType: opensearch
    configSpecs:
    - name: opensearch-config-template
      namespace: kb-system
      templateRef: opensearch-config-template
      volumeName: opensearch-config
    monitor:
      builtIn: false
      exporterConfig:
        scrapePath: /metrics
        scrapePort: 9600
    name: opensearch
    podSpec:
      containers:
      - env:
        - name: node.name
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: cluster.initial_master_nodes
          value: $(KB_CLUSTER_NAME)-$(KB_COMP_NAME)-0
        - name: discovery.seed_hosts
          value: $(KB_CLUSTER_NAME)-$(KB_COMP_NAME)-headless
        - name: cluster.name
          value: $(KB_CLUSTER_NAME)
        - name: network.host
          value: 0.0.0.0
        - name: OPENSEARCH_JAVA_OPTS
          value: -Xmx512M -Xms512M
        - name: node.roles
          value: master,ingest,data,remote_cluster_client
        imagePullPolicy: IfNotPresent
        name: opensearch
        ports:
        - containerPort: 9200
          name: http
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        - containerPort: 9600
          name: metrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          periodSeconds: 5
          tcpSocket:
            port: 9200
          timeoutSeconds: 3
        resources: {}
        startupProbe:
          failureThreshold: 30
          initialDelaySeconds: 5
          periodSeconds: 10
          tcpSocket:
            port: 9200
          timeoutSeconds: 3
        volumeMounts:
        - mountPath: /usr/share/opensearch/data
          name: data
        - mountPath: /usr/share/opensearch/config/opensearch.yaml
          name: opensearch-config
          subPath: opensearch.yaml
      - command:
        - sh
        - -c
        - |
          #!/usr/bin/env bash
          set -eo pipefail

          http () {
              local path="${1}"
              if [ -n "${USERNAME}" ] && [ -n "${PASSWORD}" ]; then
                BASIC_AUTH="-u ${USERNAME}:${PASSWORD}"
              else
                BASIC_AUTH=''
              fi
              curl -XGET -s -k --fail ${BASIC_AUTH} https://$(KB_CLUSTER_NAME)-$(KB_COMP_NAME)-headless:9200:${path}
          }

          cleanup () {
            while true ; do
              local master="$(http "/_cat/master?h=node" || echo "")"
              if [[ $master == "$(KB_CLUSTER_NAME)-$(KB_COMP_NAME)"* && $master != "${NODE_NAME}" ]]; then
                echo "This node is not master."
                break
              fi
              echo "This node is still master, waiting gracefully for it to step down"
              sleep 1
            done

            exit 0
          }

          trap cleanup SIGTERM

          sleep infinity &
          wait $!
        imagePullPolicy: IfNotPresent
        name: opensearch-master-graceful-termination-handler
        resources: {}
      initContainers:
      - args:
        - chown -R 1000:1000 /usr/share/opensearch/data
        command:
        - sh
        - -c
        imagePullPolicy: IfNotPresent
        name: fsgroup-volume
        resources: {}
        securityContext:
          runAsUser: 0
        volumeMounts:
        - mountPath: /usr/share/opensearch/data
          name: data
      - command:
        - sh
        - -c
        - |
          set -xe
          DESIRED="262144"
          CURRENT=$(sysctl -n vm.max_map_count)
          if [ "$DESIRED" -gt "$CURRENT" ]; then
            sysctl -w vm.max_map_count=$DESIRED
          fi
        imagePullPolicy: IfNotPresent
        name: sysctl
        resources: {}
        securityContext:
          privileged: true
          runAsUser: 0
    service:
      ports:
      - name: http
        port: 9200
        protocol: TCP
        targetPort: http
      - name: transport
        port: 9300
        protocol: TCP
        targetPort: transport
      - name: metric
        port: 9600
        protocol: TCP
        targetPort: metric
    volumeTypes:
    - name: data
      type: data
    workloadType: Stateful
  - characterType: opensearch-dashboard
    name: dashboard
    podSpec:
      containers:
      - command:
        - sh
        - -c
        - |
          #!/usr/bin/bash
          set -e
          bash opensearch-dashboards-docker-entrypoint.sh opensearch-dashboards
        env:
        - name: OPENSEARCH_HOSTS
          valueFrom:
            secretKeyRef:
              key: endpoints
              name: $(CONN_CREDENTIAL_SECRET_NAME)
              optional: false
        - name: SERVER_HOST
          value: 0.0.0.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 10
          initialDelaySeconds: 10
          periodSeconds: 20
          successThreshold: 1
          tcpSocket:
            port: 5601
          timeoutSeconds: 5
        name: dashboard
        ports:
        - containerPort: 5601
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 10
          initialDelaySeconds: 10
          periodSeconds: 20
          successThreshold: 1
          tcpSocket:
            port: 5601
          timeoutSeconds: 5
        resources: {}
        startupProbe:
          failureThreshold: 20
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 5601
          timeoutSeconds: 5
    service:
      ports:
      - name: http
        port: 5601
        protocol: TCP
        targetPort: http
    workloadType: Stateless
  connectionCredential:
    endpoint: $(SVC_FQDN):$(SVC_PORT_http)
    endpoints: https://$(SVC_FQDN):$(SVC_PORT_http)
    host: $(SVC_FQDN)
    password: $(RANDOM_PASSWD)
    port: $(SVC_PORT_http)
    username: root
  type: opensearch

2. Set componentSpecs.name: opensearch monitor: true on the opensearch cluster
   The modified cluster file is as follows

apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
  annotations:
    kubeblocks.io/reconcile: "2023-12-04T08:02:36.272322616Z"
    meta.helm.sh/release-name: kb-opensearch-ekpnxjdonxv
    meta.helm.sh/release-namespace: foo
  creationTimestamp: "2023-12-04T02:55:25Z"
  finalizers:
  - cluster.kubeblocks.io/finalizer
  generation: 1
  labels:
    app.kubernetes.io/instance: kb-opensearch-ekpnxjdonxv
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kb-opensearch
    app.kubernetes.io/version: 2.7.0
    clusterdefinition.kubeblocks.io/name: opensearch
    clusterversion.kubeblocks.io/name: opensearch-2.7.0
    helm.sh/chart: kb-opensearch-2.7.0
  name: kb-opensearch-ekpnxjdonxv
  namespace: foo
  resourceVersion: "78884881"
  uid: 45542a52-04ee-4d00-b8e3-0ff6ddedd547
spec:
  affinity:
    podAntiAffinity: Preferred
    tenancy: SharedNode
    topologyKeys:
    - kubernetes.io/hostname
  clusterDefinitionRef: opensearch
  clusterVersionRef: opensearch-2.7.0
  componentSpecs:
  - componentDefRef: opensearch
    monitor: true
    name: opensearch
    noCreatePDB: false
    replicas: 3
    resources:
      limits:
        cpu: 1000m
        memory: 2Gi
      requests:
        cpu: 1000m
        memory: 2Gi
    serviceAccountName: kb-opensearch-ekpnxjdonxv-kb-os-sa
    volumeClaimTemplates:
    - name: data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: rook-ceph-block
  - componentDefRef: dashboard
    monitor: false
    name: dashboard
    noCreatePDB: false
    replicas: 1
    resources:
      limits:
        cpu: 1000m
        memory: 2Gi
      requests:
        cpu: 1000m
        memory: 2Gi
    serviceAccountName: kb-opensearch-ekpnxjdonxv-kb-os-sa
  terminationPolicy: Delete

3.deploy opensearch cluster yaml file：
deploy opensearch

 helm install kb-os kb-os.2.7.0.gz

4. exec pod and access metrics Service curl -XGET -k -u 'admin:admin' http://localhost:9600/metrics ,then see error

   404 Not Found

   No context found for request

kubectl exec -it clusterName -- /bin/bash
curl -XGET -k -u 'admin:admin' http://localhost:9600/metrics 

5. i dry to aceess opensearch metric API curl -XGET -k -u 'admin:admin' 'http://localhost:9600/_plugins/_performanceanalyzer/metrics?metrics=Latency,CPU_Utilization&agg=avg,max&dim=ShardID&nodes=all' ,and have normal access

curl -XGET -k -u 'admin:admin' 'http://localhost:9600/_plugins/_performanceanalyzer/metrics?metrics=Latency,CPU_Utilization&agg=avg,max&dim=ShardID&nodes=all'

Expected behavior
I hope to visit kubeblocks integrated opensearch monitoring service is normal, namely through http://localhost:9600/metrics to get the opensearch related index

Screenshots
If applicable, add screenshots to help explain your problem.
acees the url:

acees the metric api:

Desktop (please complete the following information):

• OS:centos x86
• Kubernetes: v1.27.6
• KubeBlocks: 0.7.0
• kbcli: 0.7.0

Additional context

[nayutah]

@sophon-zt It seems that opensearch does not have a metric uri "http://localhost:9600/metrics"

[sophon-zt]

Here is the architecture of the kb scrape engine metrics:

opensearch metric plugin: https://github.com/aparo/opensearch-prometheus-exporter

In addition, prometheus may not be able to scrape directly, and account information needs to be configured. so a metrics sidecar is needed to scrape opensearch metrics. You can refer to pulsar addons:
https://github.com/apecloud/kubeblocks/blob/release-0.7/deploy/pulsar/templates/clusterdefinition.yaml#L215
https://github.com/apecloud/kubeblocks/blob/release-0.7/deploy/pulsar/templates/agamotto-configmap.yaml

[sophon-zt]

"_plugins/_performanceanalyzer/metrics" is ok, means that there is already a metrics plugin. You can try to add a prometheus scrape configs or add a sidecar.

[sophon-zt]

Thank you, we will solve the problem of metrics in the next minor version of 0.7

[sophon-zt]

If authentication is not required, you can set cd.spec.componentDefs[*].monitor.exporterConfig.scrapePath to the url path provided by the metrics plugin, metrics sidecar is not needed, prometheus can collect metrics directly.

cd.spec.componentDefs[*].monitor.exporterConfig.scrapePath: "/_plugins/_performanceanalyzer/metrics?metrics=Latency,CPU_Utilization&agg=avg,max&dim=ShardID&nodes=all"

[github-actions]

This issue has been marked as stale because it has been open for 30 days with no activity