Trying to Upgrade knowlg on AWS from 5.7 to 6.1.0 facing issues

[AbhilashKD]

Hi,
We are trying to upgrade the knowlg on AWS. Earlier we successfully deployed the knowlg 5.2 on AWS using helm charts but while upgrading the knowlg to 6.1.0 we are facing issues in Flink jobs and neo4j logstash the more details as follows :

for upgradation we are referring to a repo : https://github.com/Sunbird-Knowlg/knowledge-platform with branch release-6.1.0

Changes made :

1. Changed the cloud platform from Azure to AWS
2. Changed the Azure blob storage with AWS s3
3. Added variables for AWS to access the s3 and other resources
4. Added variable for AWS region as well

after doing changes in the helm charts of the Knowlg-automation in above mentioned repo. trying to deploy the knowlg on AWS
So except flink job for task manager and job manager and logstash under neo4j deployment facing issues.
The issues and snapshots are attached below :
I have did the changes in helm charts and given the deployment so below are the errors iam getting :
in flink jobs :
`Flink Prod Job manager :

Type     Reason       Age                  From     Message                                                                                                                                           Warning  FailedMount  22s (x746 over 25h)  kubelet  MountVolume.SetUp failed for volume "flink-config-volume" : configmap references non-existent config key: flink-conf

`Flink Prod Task manager :

Type     Reason       Age                  From     Message                                                                                                                                 Warning  FailedMount  97s (x746 over 25h)  kubelet  MountVolume.SetUp failed for volume "flink-config-volume" : configmap references non-existent config key: flink-conf  
[12:59](https://tekditech.slack.com/archives/C06FED4TPA8/p1733470179450029?thread_ts=1733462521.731339&cid=C06FED4TPA8)

and in Neo4j logstash facing error as :

[2024-12-06T07:25:10,532][WARN ][org.apache.kafka.clients.ClientUtils] Couldn't resolve server kafka-headless.sunbird.svc.cluster.local:9092 from bootstrap.servers as DNS resolution failed for kafka-he │
│ [2024-12-06T07:25:10,534][INFO ][org.apache.kafka.clients.producer.KafkaProducer] [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 0 ms.                                    │
│ [2024-12-06T07:25:10,541][ERROR][logstash.outputs.kafka   ] Unable to create Kafka producer from given configuration {:kafka_error_message=>org.apache.kafka.common.KafkaException: Failed to construct k │
│ [2024-12-06T07:25:10,562][ERROR][logstash.pipeline        ] Error registering plugin {:pipeline_id=>"main", :plugin=>"#<LogStash::OutputDelegator:0x5c200f0b>", :error=>"Failed to construct kafka produc │
│ [2024-12-06T07:25:10,568][ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>org.apache.kafka.common.KafkaException: Failed to construct kafka producer,  │
│ [2024-12-06T07:25:10,589][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineActi │
│ [2024-12-06T07:25:11,022][INFO ][logstash.agent           ] Successfully start
[sunbird-inquirybb-cassandra-migration-job-psb7r-1734004908027614895.log](https://github.com/user-attachments/files/18111101/sunbird-inquirybb-cassandra-migration-job-psb7r-1734004908027614895.log)
ed Logstash API endpoint {:port=>9600}                                                                                      │
[sunbird-neo4j-568cbb8685-ttkfg-1734004775773841496.log](https://github.com/user-attachments/files/18111076/sunbird-neo4j-568cbb8685-ttkfg-1734004775773841496.log)

│ [2024-12-06T07:25:15,775][INFO ][logstash.runner          ] Logstash shut down.                                                                                                                           │
│ [2024-12-06T07:25:15,821][ERROR][org.logstash.Logst
[sunbird-neo4j-568cbb8685-ttkfg-1734004775773841496.log](https://github.com/user-
![All-services](https://github.com/user-attachments/assets/b4595e95-e626-4d19-b9fd-43cadb781b8c)
![inquiry-error](https://github.com/user-attachments/assets/8cc66ecc-7110-4f08-bcd0-4996621c9dc0)
attachments/files/18111084/sunbird-neo4j-568cbb8685-ttkfg-1734004775773841496.log)
[sunbird-neo4j-568cbb8685-ttkfg-1734004775773841496.log](https://github.com/user-attachments/files/18111092/sunbird-neo4j-568cbb8685-ttkfg-1734004775773841496.log)
ash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

@pallakartheekreddy @gandham-santhosh @aimansharief @rhwarrier @coolbung @snehal09004 @rajnishdargan


pods-all-1732167530473705762.csv
sunbird-inquirybb-cassandra-migration-job-psb7r-1734004908027614895.log
sunbird-neo4j-568cbb8685-ttkfg-1734004775773841496.log

[aimansharief]

@AbhilashKD The Flink error appears to be caused by an issue in the ConfigMap, while the errors in the Logstash and Cassandra jobs seem to be related to connection problems with their respective databases.

[aimansharief]

We can connect tomorrow to resolve the issue

[AbhilashKD]

Yes sure

…

On Thu, 12 Dec, 2024, 23:28 Aiman Sharief, ***@***.***> wrote: We can connect tomorrow to resolve the issue — Reply to this email directly, view it on GitHub <#4023 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A5SL2YNL7YQ73RO6RE63EB32FHFGBAVCNFSM6AAAAABTPUXNUKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNJUG42TQNI> . You are receiving this because you were mentioned.Message ID: <project-sunbird/sunbird-devops/repo-discussions/4023/comments/11547585@ github.com>

[aimansharief]

can you send an invite to connect at 2'45

[AbhilashKD]

@aimansharief
Shared invite with you

[AbhilashKD]

@aimansharief
Waiting for you on call

[aimansharief]

@AbhilashKD Sorry, previous call got extended . I'm joining now

[aimansharief]

Please find the link to the docker file here:
https://github.com/Sunbird-Knowlg/knowledge-platform/blob/release-6.1.0/build/content-service/Dockerfile

Steps to build and push docker image to your repo

docker build -f build/content-service/Dockerfile -t <repository>/content-service:6.0.1 .
docker tag <repository>/content-service:6.0.1
docker push <repository>/content-service:6.0.1

Also follow the below steps before building docker image
https://github.com/Sunbird-Knowlg/knowledge-platform/tree/release-6.1.0?tab=readme-ov-file#running-content-service

[aimansharief]

Once you fix the kafka issue, this is where you configure kafka-logstash connection url.
https://github.com/Sunbird-Knowlg/knowlg-automation/blob/c481a8dbca2a7a038132c1d1947b5476ac62f7ad/terraform/modules/helm/neo4j/neo4j-helm-chart/values.yaml#L18

Please update it to fix the existing error.

[AbhilashKD]

@aimansharief
Can you share me the images directly so that i can take a pull and check all the configs by deploying it to my EKS cluster.
As the building images its a long process also if we catch in an issue then it will again take more time to debug it or you also need to assist us while debugging and troubleshooting.
So if you share images directly with 6.1.0 then it will be helpful for us.
@coolbung @gandham-santhosh @rhwarrier @pallakartheekreddy @snehal0904

[aimansharief]

@AbhilashKD Please find the list of images for knowlg-6.1.0 :

sunbirded.azurecr.io/content-service:release-6.1.0_RC5_f6fb77d_893
sunbirded.azurecr.io/taxonomy-service:release-6.1.0_RC4_2272e93_47
sunbirded.azurecr.io/search-service:release-6.1.0_RC4_2272e93_40
sunbirded.azurecr.io/knowledge-platform-jobs:release-6.1.0_RC1_5470dbe_87
sunbirded.azurecr.io/sunbird-dial-service:release-6.1.0_RC2_213843a_30

Let me know if you need any other images.