Obsrv Deployment Issues Using Helm

[syedsalman3753]

Document followed: https://obsrv.sunbird.org/use/obsrv-2.0-installation-guide/obsrv-deployment-using-helm

During the deployment of the Kafka Helm chart from the above document, an error was encountered: no matches for kind 'ServiceMonitor' in version 'monitoring.coreos.com/v1'.
I kindly request your assistance in providing a resolution for this matter.

~/deployment/obsrv/obsrv-automation/terraform/modules/helm$ helm upgrade --install --atomic kafka kafka/kafka-helm-chart -n kafka --create-namespace --debug
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/techno-467/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/techno-467/.kube/config
history.go:56: [debug] getting history for release kafka
Release "kafka" does not exist. Installing it now.
install.go:200: [debug] Original chart version: ""
install.go:217: [debug] CHART PATH: /home/techno-467/deployment/obsrv/obsrv-automation/terraform/modules/helm/kafka/kafka-helm-chart
Error: unable to build kubernetes objects from release manifest: [resource mapping not found for name: "kafka-jmx-metrics" namespace: "kafka" from "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
ensure CRDs are installed first, resource mapping not found for name: "kafka-metrics" namespace: "kafka" from "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
ensure CRDs are installed first]
helm.go:84: [debug] [resource mapping not found for name: "kafka-jmx-metrics" namespace: "kafka" from "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
ensure CRDs are installed first, resource mapping not found for name: "kafka-metrics" namespace: "kafka" from "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
ensure CRDs are installed first]
unable to build kubernetes objects from release manifest
helm.sh/helm/v3/pkg/action.(*Install).RunWithContext
	helm.sh/helm/v3/pkg/action/install.go:307
main.runInstall
	helm.sh/helm/v3/cmd/helm/install.go:287
main.newUpgradeCmd.func2
	helm.sh/helm/v3/cmd/helm/upgrade.go:130
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/[email protected]/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/[email protected]/command.go:1044
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/[email protected]/command.go:968
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
	runtime/proc.go:250
runtime.goexit
	runtime/asm_amd64.s:1598

Additionally, I have identified a minor discrepancy in the documentation, which I would like to bring to your attention:

• The velero repository URL requires an update: https://vmware-tanzu.github.io/helm-charts (please note that a typo error is present, as the character 's' is missing).
• After adding the helm repository, kindly execute the command helm repo update to ensure it is up to date.

[manjudr]

@syedsalman3753 - Please take the latest changes from the main branch and execute the monitoring stack command

helm upgrade --install --atomic monitoring monitoring/kube-prometheus-stack -n monitoring -f monitoring/values.yaml --create-namespace --debug 

[syedsalman3753]

@manjudr

From the documentation, I could see the helm repo name is set to prometheus instead of monitoring as shown in the below diagram.

Additionally, I have identified a minor discrepancy in the documentation, which I would like to bring to your attention:

• The velero repository URL requires an update: https://vmware-tanzu.github.io/helm-charts (please note that a typo error is present, as the character 's' is missing).
• After adding the helm repository, kindly execute the command helm repo update to ensure it is up to date.

Requesting to update the documentation for the above issues

[manjudr]

Thanks for identifying it. We will revisit and correct the setup documentation.

[syedsalman3753]

Hi @manjudr

I have discovered several additional issues that require attention:

• The storage class for the druid_cluster Helm chart has been hard-coded to the gp2 storage class link. To ensure better flexibility and compatibility, it is recommended to update the Helm charts to utilize the default storage class.

• As an interim solution, we have undertaken manual updates to proceed with the installation process. However, it is essential to bring to your attention that several pods are presently experiencing issues within the druid_cluster deployment, which are elaborated below:

  • The druid-raw-coordinators pod is encountering a password authentication error.
  • Both druid-raw-indexers-0 and druid-raw-overlords pods are failing due to the absence of S3 details. These details were not provided in the documentation; hence, we request you to update the necessary information accordingly.
  • We would like to inquire if there is a possibility of utilizing a minio URL in lieu of an S3 bucket.
    

[manjudr]

Syed -

1. Did we provision the Postgresql before provisioning the Apache druid and are you able to login into the Postgres?
2. The integration of the cloud storage dependency, Minio, with Obsrv, has not been verified yet. However, this feature is planned for upcoming releases.

[syedsalman3753]

1. Yes I'm able to access via the postgres user but not via the druid_raw user
   
   

2. Noted. will create a bucket for the same

can you please help us to resolve issue 1?

[Prafulrakhade]

Hi @manjudr

Now we are able to login with druid_raw user, that postgresql issue is resolved.
But "druid-raw-indexers-0" and "druid-raw-overlords" these two pods are crashloopbackoff, I'm attaching logs here can you please check and help us to bring up these pods.

[Prafulrakhade]

@manjudr we have resolved druid_cluster issue, The reason for the failure was AWS region was not provided for the bucket.

[Prafulrakhade]

@manjudr We have attempted to deploy the dataset-api Helm chart, but unfortunately, it is encountering an error. We kindly request your assistance in resolving this issue

[anandp504]

@Prafulrakhade Where are you trying to install the dataset api? Is it a Kubernetes cluster in your data center? Does it have ways to expose the dataset api service as a LoadBalancer? You need to have MetalLB or Traefik to expose the services using a LoadBalancer IP which can be accessed outside the Kubernetes cluster. Can you give some more details on the error?

[syedsalman3753]

• Where are you trying to install the dataset API?

  • We are trying to deploy dataset-API to a Rancher k8s cluster with Istio working as an ingress controller.

• Is it a Kubernetes cluster in your data center?

  • It can be considered as a k8s cluster in our data centers, the only difference is that we are using the compute resources (VMs) from the E2E networks ( cloud provider ) to create the Rancher k8s cluster.

• Does it have ways to expose the dataset api service as a LoadBalancer?

  • We don't have LoadBalancer for this k8s cluster, we are exposing the services via istio gateways as a nodeport service. We have a separate nginx server that will work as a load balancer via the round-robin method and is pointing to these istio gateways.

• You need to have MetalLB or Traefik to expose the services using a LoadBalancer IP which can be accessed outside the Kubernetes cluster.

  • As mentioned in the last answer, we have a way to expose the service via the istio gateway and nginx proxy.

• Can you give some more details on the error?

  • Dataset-API pods are deleting automatically, and in cli could see the error service does not have LoadBalancer IP address.
  • Also, could see the service_account_annotations, CLOUD_STORAGE* are hardcoded in values.yaml.
    

[anandp504]

@syedsalman3753 You need to change the type of services for dataset-api and superset to type: NodePort and use the Istio Ingress to access these services outside the cluster. In cloud based environments, we access these services using LoadBalancer services which provides external IP to access them outside the cluster.

[Prafulrakhade]

Hi @anandp504

1. dataset-api :- We have successfully deployed the dataset-api within our cluster and subsequently modified the service type to NodePort, enabling external access to this service beyond the confines of the cluster.

2. superset :- The documentation for Sunbird Onsrv
is currently incomplete, as it lacks guidance on deploying the Superset Helm chart. Kindly consider updating this section to ensure comprehensive coverage of the Superset deployment process.

3. flink :- We have successfully deployed the Flink Service Account. However, for the Flink Merged Pipeline Job and Flink Master Data Processor Job, the values.yaml file within the Flink Helm chart mandates the provisioning of specific values such as S3 credentials, bucket configuration, object names, and related parameters. Unfortunately, these specific requirements have not been outlined in the existing documentation. Therefore, we kindly request an updated version of the documentation to include comprehensive guidance on the configuration and deployment of Flink, addressing the aforementioned aspects. Your prompt attention to this matter would greatly assist in our seamless Flink deployment efforts.

[anandp504]

@Prafulrakhade We will update the installation document with instructions for installing Superset. Druid, Secor and Flink are the three components which relies on distributed storage such as AWS S3 or MinIO etc. Please update the S3 related configuration as mentioned below if you are planning to use MinIO as your distributed storage.

Flink: Please update the S3 related configuration if you are using MinIO as your distributed storage.

s3_access_key: ""
s3_secret_key: ""
s3_endpoint: "http://172.20.126.232:9000"
base.url = "s3://"

Secor:

aws.access.key=""
aws.secret.key=""
aws.region=""
aws.endpoint=""
secor.s3.bucket=""

Druid:

druid.storage.bucket=druid-test-bucket
druid.storage.baseKey=druid/segments
druid.s3.accessKey=""
druid.s3.secretKey=""
druid.s3.endpoint.signingRegion=us-east-2
druid.s3.endpoint.url=http://172.20.126.232:9000/
For MinIO
druid.indexer.logs.directory=