Error in Kubernetes pod logs Deploy/KnowledgePlatform/FlinkJobs

[Munishdev97]

@santhosh-tg @gandham-santhosh

For Deploy/KnowledgePlatform/FlinkJobs We are facing the below errors in pod logs.
Please let us know if any different branch tag we have to use for oci.

java.util.ServiceConfigurationError: org.apache.flink.core.fs.FileSystemFactory: Provider org.apache.flink.fs.s3hadoop.S3FileSystemFactory could not be instantiated
        at java.util.ServiceLoader.fail(Unknown Source) ~[?:?]
        at java.util.ServiceLoader$ProviderImpl.newInstance(Unknown Source) ~[?:?]
        at java.util.ServiceLoader$ProviderImpl.get(Unknown Source) ~[?:?]
        at java.util.ServiceLoader$3.next(Unknown Source) ~[?:?]
        at org.apache.flink.core.fs.FileSystem.addAllFactoriesToList(FileSystem.java:1077) [flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.core.fs.FileSystem.loadFileSystemFactories(FileSystem.java:1058) [flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.core.fs.FileSystem.initialize(FileSystem.java:338) [flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.configureFileSystems(ClusterEntrypoint.java:225) [flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:182) [flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:600) [flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:85) [flink-dist_2.12-1.13.5.jar:1.13.5]
Caused by: java.lang.NoSuchMethodError: 'void org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.<init>(java.lang.String, org.apache.flink.fs.s3hadoop.common.HadoopConfigLoader)'
        at org.apache.flink.fs.s3hadoop.S3FileSystemFactory.<init>(S3FileSystemFactory.java:50) ~[flink-s3-fs-hadoop-1.13.5.jar:1.13.5]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
        ... 10 more
2023-12-06 12:29:14,910 ERROR org.apache.flink.core.fs.FileSystem                          [] - Failed to load a file system via services
java.util.ServiceConfigurationError: org.apache.flink.core.fs.FileSystemFactory: Provider org.apache.flink.fs.s3hadoop.S3AFileSystemFactory could not be instantiated
        at java.util.ServiceLoader.fail(Unknown Source) ~[?:?]
        at java.util.ServiceLoader$ProviderImpl.newInstance(Unknown Source) ~[?:?]
        at java.util.ServiceLoader$ProviderImpl.get(Unknown Source) ~[?:?]
        at java.util.ServiceLoader$3.next(Unknown Source) ~[?:?]
        at org.apache.flink.core.fs.FileSystem.addAllFactoriesToList(FileSystem.java:1077) [flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.core.fs.FileSystem.loadFileSystemFactories(FileSystem.java:1058) [flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.core.fs.FileSystem.initialize(FileSystem.java:338) [flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.configureFileSystems(ClusterEntrypoint.java:225) [flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:182) [flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:600) [flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:85) [flink-dist_2.12-1.13.5.jar:1.13.5]
Caused by: java.lang.NoSuchMethodError: 'void org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.<init>(java.lang.String, org.apache.flink.fs.s3hadoop.common.HadoopConfigLoader)'
        at org.apache.flink.fs.s3hadoop.S3FileSystemFactory.<init>(S3FileSystemFactory.java:50) ~[flink-s3-fs-hadoop-1.13.5.jar:1.13.5]
        at org.apache.flink.fs.s3hadoop.S3AFileSystemFactory.<init>(S3AFileSystemFactory.java:22) ~[flink-s3-fs-hadoop-1.13.5.jar:1.13.5]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Unknown Source) ~[?:?]
        ... 10 more

[santhosh-tg]

@Munishdev97 Can you try with this tag https://github.com/Sunbird-Knowlg/sunbird-learning-platform/tree/release-5.5.0_RC1

[Munishdev97]

@santhosh-tg, Even after deploying with the above tag, the error still remains same.

what should be the tag for build pipeline (for Flinkjobs, Inquiryflinkjobs) ?
Because in deploy, it is using some dependencies that are missing in the pom.xml mentioned in the current build release-5.5.0_RC2 and release-5.7.0_RC1 for flink and inquiryflinkjobs respectively

The following ansible variables are missing in the above mentioned tag. We have provided the empty strings for now to test, please share the values for these variables.

oci_media_prefix_input: ""
oci_media_dist_channel_id: ""
oci_media_work_flow_id: ""
oci_media_stream_config_id: ""
oci_media_gateway_domain: ""
cloud_storage_proxy_host: ""

[santhosh-tg]

Sharing the build parameters for reference

[Munishdev97]

Hi @santhosh-tg

For inquiryFlinkJobs, we have different config for build with parameters, other parameters are missing. please share the config.xml to be used for this
For Flinkjobs, it is same as you have provided.

[santhosh-tg]

@Munishdev97 Can you try with this ?
https://raw.githubusercontent.com/santhosh-tg/sunbird-devops/8cad6b2c131eb4666181f6c902db8e62af5ce6eb/deploy/jenkins/jobs/Build/jobs/KnowledgePlatform/jobs/InquiryFlinkJob/config.xml

[ddevadat]

those errors can be ignored if its creating the checkpoint bucket objects

[Munishdev97]

@ddevadat @santhosh-tg

After the java exception errors, the akka RPC service is getting stopped in the job manager pods

2023-12-11 07:45:19,732 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Shutting StandaloneApplicationClusterEntryPoint down with application status UNKNOWN. Diagnostics Cluster entrypoint has been closed externally..
2023-12-11 07:45:19,732 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shutting down rest endpoint.
2023-12-11 07:45:19,735 INFO  org.apache.flink.runtime.blob.BlobServer                     [] - Stopped BLOB server at 0.0.0.0:6124
2023-12-11 07:45:19,746 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Removing cache directory /tmp/flink-web-b9344517-f6c8-4384-ba80-774ef2177504/flink-web-ui
2023-12-11 07:45:19,747 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - http://async-questionset-publish-jobmanager:8081/ lost leadership
2023-12-11 07:45:19,747 INFO  org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shut down complete.
2023-12-11 07:45:19,747 INFO  org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent [] - Closing components.
2023-12-11 07:45:19,747 INFO  org.apache.flink.runtime.dispatcher.runner.DefaultDispatcherRunner [] - DefaultDispatcherRunner was revoked the leadership with leader id 00000000-0000-0000-0000-000000000000. Stopping the DispatcherLeaderProcess.
2023-12-11 07:45:19,748 INFO  org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - Stopping SessionDispatcherLeaderProcess.
2023-12-11 07:45:19,748 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Stopping dispatcher akka.tcp://flink@async-questionset-publish-jobmanager:6123/user/rpc/dispatcher_1.
2023-12-11 07:45:19,748 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Stopping all currently running jobs of dispatcher akka.tcp://flink@async-questionset-publish-jobmanager:6123/user/rpc/dispatcher_1.
2023-12-11 07:45:19,748 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Closing the slot manager.
2023-12-11 07:45:19,748 INFO  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Suspending the slot manager.
2023-12-11 07:45:19,749 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Stopped dispatcher akka.tcp://flink@async-questionset-publish-jobmanager:6123/user/rpc/dispatcher_1.
2023-12-11 07:45:19,755 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService             [] - Stopping Akka RPC service.
2023-12-11 07:45:19,759 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService             [] - Stopping Akka RPC service.
2023-12-11 07:45:19,806 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService             [] - Stopped Akka RPC service.

and therefore task manager pods are not running

INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://flink@async-questionset-publish-jobmanager:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://flink@async-questionset-publish-jobmanager:6123/user/rpc/resourcemanager_*.

what are the dependencies to be checked for these pods.

[mohitga]

@ddevadat @santhosh-tg
folks can you help @Munishdev97 here

[ddevadat]

can you share the build and deployment repo and branch information?
Also is the job manager in question is questionset-publish?

[vj4a]

@ddevadat , FYI

Build/KnowledgePlatform/flinkJobs
Repo: https://github.com/project-sunbird/knowledge-platform-jobs.git
Branch: release-5.5.0_RC2

Deploy/KnowledgePlatform/flinkJobs
Repo: https://github.com/project-sunbird/sunbird-learning-platform.git
Branch: release-5.4.1_RC1

Build/KnowledgePlatform/inquiryFlinkJobs
Repo: https://github.com/Sunbird-inQuiry/data-pipeline.git
Branch: release-5.7.0_RC3

Deploy/KnowledgePlatform/inquiryFlinkJobs
Repo: https://github.com/Sunbird-inQuiry/data-pipeline.git
Branch: release-5.7.0_RC3

The job manager pod error is for all the pods under flinkjobs and inquiryflinkjobs including the questionset-publish.

Also the checkpoint objects were created in the bucket as you had mentioned before for flinkjobs
How to validate if the files were created as expected.

cc: @Munishdev97

[mohitga]

@gandham-santhosh @ddevadat please have a look

[ddevadat]

lets try to narrow down this.

Looking at the build and deployment tags(release-5.4.1_RC1 ) used for KP Flinkjobs, I dont think it should be creating any checkpoints stores in the object storage, because the deployment job doesnt have the s3 blocks in it. Please confirm if this is the case. One way to check will be to inspect the configmap for KP flinkjobs.

look for base.url value. Ideally it should be something like s3:// , but i am guessing in this case it will be something like wasbs
You need to deploy the right build and deploy tags for kp flinkjobs.

Over to inquiry flinkjobs.
These tags does contain the oci specifc blocks. But i am not sure on the error that you are getting. My advise would be to triage one failed job in inquiry flinkjob ( async-questionset-publish). Most likely if you solve this the other jobs in the inquiry flinkjob set should also resolve by itself.

One starting point is to check the config map of async-questionset-publish and see if everything is set correctly like base.url, kafka, redis connectivity etc

Other thing you can try is to set the value of flink debug in the private inventory and deploy the inquiry flink job to see some more detailed error message

flink_jobs_console_log_level: DEBUG

[vj4a]

@ddevadat @santhosh-tg ,

Inquiry flinkjobs is working now. Previously, the error was due to kafka connectivity and it was fixed in https://github.com/orgs/Sunbird-Ed/discussions/662.

Please share the repo and branch details for build and deploy of KP, Lern and Datapipeline flinkjobs.

[mohitga]

@ddevadat @santhosh-tg @gandham-santhosh
folks share the branch details @vj4a

[ddevadat]

@mohitga i am assuming you are following release 6 documentation for build and deploy tags.

@santhosh-tg the branch tags for build/kp/flinkjobs and deploy/kp/flinkjobs doesnt have the oci specific changes. Could you please advise which branch to take

[santhosh-tg]

@vj4a
KP:
build: release-5.5.0_RC2 (tag)
deploy: release-5.5.0 (branch)

Lern:
build and deploy: release-5.3.1_RC2 (tag)

@sowmya-dixit please help with the tags /branch for DP

[vj4a]

@santhosh-tg , KP and Lern are working fine.

@sowmya-dixit, Please share tags for DP.

[sowmya-dixit]

@vj4a
Obsrv:
Build: release-5.2.0_RC3
Deploy: release-5.2.0_RC3

[vj4a]

@santhosh-tg,

For KP flinkjobs, metrics-data-transformer-jobmanager pod fails with below error. Please share if any dependencies to be added for build.
Full log: metrics-data-transformer-jobmanager_log.txt

Caused by: org.apache.flink.client.program.ProgramInvocationException: The program's entry point class 'org.sunbird.job.metricstransformer.task.MetricsDataTransformerStreamTask' was not found in the jar file.

@sowmya-dixit - For DP flinkjobs, can you share the SSLmode to be used for postgres.

[sowmya-dixit]

@vj4a Please use sslMode: false

[santhosh-tg]

@santhosh-tg,

For KP flinkjobs, metrics-data-transformer-jobmanager pod fails with below error. Please share if any dependencies to be added for build. Full log: metrics-data-transformer-jobmanager_log.txt

Caused by: org.apache.flink.client.program.ProgramInvocationException: The program's entry point class 'org.sunbird.job.metricstransformer.task.MetricsDataTransformerStreamTask' was not found in the jar file.

Hi @vinukumar-vs @pallakartheekreddy Any inputs here ?

[mohitga]

@vinukumar-vs @pallakartheekreddy folks is this thread unblocked?

[maheshkumargangula]

@santhosh-tg - The build used here don't have the metrics-data-transformer related classes. Please check you build and ensure the metrics-data-transformer is included in your build.

[akulavasu]

@santhosh-tg , is there any update regarding build tag for metrics-data-transformer.

[santhosh-tg]

@sowmya-dixit @maheshkumargangula Please share the build tag that needs to used here.

[mohitga]

@sowmya-dixit @maheshkumargangula @vinukumar-vs who can share the build tags required?
this thread already delayed

[maheshkumargangula]

@santhosh-tg @akulavasu - As part of the building blocks separation, the dependency for metrics-data-transformer is removed from Knowlg repository.
https://github.com/Sunbird-Knowlg/knowledge-platform-jobs/blob/release-5.5.0/pom.xml#L41

Please fork this repository and use release-5.5.0 branch to uncomment and build the code to enable this job.
Cc: @mohitga @vinukumar-vs

[akulavasu]

@maheshkumargangula ,

The same error is being shown in the pod now.
On uncommenting the module in pom.xml, the jar is being built but not getting added to the docker build.
Is there another place we need to add dependency for metrics-data-transformer.

Please find the full log here.
full-log.txt

metrics-data-transformer jar being built.

[INFO] Reactor Summary:
[INFO] 
[INFO] Knowledge Platform Jobs 1.0 ........................ SUCCESS [  0.274 s]
[INFO] jobs-core 1.0.0 .................................... SUCCESS [ 18.636 s]
[INFO] post-publish-processor 1.0.0 ....................... SUCCESS [ 19.317 s]
[INFO] publish-pipeline 1.0 ............................... SUCCESS [  0.013 s]
[INFO] publish-core 1.0.0 ................................. SUCCESS [ 18.055 s]
[INFO] content-publish 1.0.0 .............................. SUCCESS [ 35.765 s]
[INFO] live-node-publisher 1.0.0 .......................... SUCCESS [ 39.755 s]
[INFO] search-indexer 1.0.0 ............................... SUCCESS [ 27.650 s]
[INFO] auto-creator-v2 1.0.0 .............................. SUCCESS [ 29.905 s]
[INFO] content-auto-creator 1.0.0 ......................... SUCCESS [ 28.048 s]
[INFO] audit-history-indexer 1.0.0 ........................ SUCCESS [ 21.557 s]
[INFO] audit-event-generator 1.0.0 ........................ SUCCESS [ 23.669 s]
[INFO] metrics-data-transformer 1.0.0 ..................... SUCCESS [ 22.060 s]
[INFO] qrcode-image-generator 1.0.0 ....................... SUCCESS [ 23.305 s]
[INFO] dialcode-context-updater 1.0.0 ..................... SUCCESS [ 23.170 s]
[INFO] cassandra-data-migration 1.0.0 ..................... SUCCESS [ 20.766 s]
[INFO] live-video-stream-generator 1.0.0 .................. SUCCESS [ 30.477 s]
[INFO] csp-migrator 1.0.0 ................................. SUCCESS [ 28.734 s]
[INFO] asset-enrichment 1.0.0 ............................. SUCCESS [01:38 min]
[INFO] video-stream-generator 1.0.0 ....................... SUCCESS [ 32.876 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  09:02 min
[INFO] Finished at: 2024-01-11T13:41:31+05:30
[INFO] ------------------------------------------------------------------------

metrics-data-transformer jar missing in docker build stage.

[INFO] Step 7/13 : RUN tar -xvf /tmp/jobs-distribution-1.0.tar.gz -C $FLINK_HOME/lib/
[INFO] 
[INFO]  ---> Running in 6de3d41c5166
[INFO] flink-streaming-scala_2.12-1.13.6.jar
[INFO] post-publish-processor-1.0.0.jar
[INFO] video-stream-generator-1.0.0.jar
[INFO] content-publish-1.0.0.jar
[INFO] search-indexer-1.0.0.jar
[INFO] asset-enrichment-1.0.0.jar
[INFO] audit-history-indexer-1.0.0.jar
[INFO] auto-creator-v2-1.0.0.jar
[INFO] content-auto-creator-1.0.0.jar
[INFO] qrcode-image-generator-1.0.0.jar
[INFO] audit-event-generator-1.0.0.jar
[INFO] dialcode-context-updater-1.0.0.jar
[INFO] csp-migrator-1.0.0.jar
[INFO] live-node-publisher-1.0.0.jar
[INFO] live-video-stream-generator-1.0.0.jar
[INFO] cassandra-data-migration-1.0.0.jar
[INFO] scala-library-2.12.11.jar
[INFO] Removing intermediate container 6de3d41c5166

[maheshkumargangula]

@akulavasu - Yes. Please uncomment in the below pom.xml file for creating the docker image with this dependency.
https://github.com/Sunbird-Knowlg/knowledge-platform-jobs/blob/master/jobs-distribution/pom.xml#L107-L112

[akulavasu]

It is working now as expected.
Thanks @maheshkumargangula

[mohitga]

@akulavasu is this issue resolved?
Can you mark this issue answered

[akulavasu]

@mohitga , This is resolved. Marking as closed.