java.lang.ClassNotFoundException: org.apache.spark.SparkConf - Works with IDE but not when running the simulation

[SafrasIqbal]

I am trying to use Spark data processing , I tested the code in the Eclipse IDE, all dependencies added and working. However, when try to use with the simulation its throws an error cannot find the class.

I try to add them separately to the MOSAIC lib folder /Application folder in the scenario nothing works.

Please help me figure out what am I missing here. Thank you.

[SafrasIqbal]

Hello @kschrab and @schwepmo , Please help me with some direction/workaround for the above issue. greatly appreciate. Thank you

[kschrab]

If you start your simulation in the IDE, you need to add your application project to the rti/mosaic-starter/pom.xml, similar to this:

        <dependency>
            <groupId>org.eclipse.mosaic.app</groupId>
            <artifactId>your-application-name</artifactId>
            <version>${mosaic.version}</version>
            <scope>runtime</scope>
        </dependency>

Otherwise, if you start your simulation from the installation bundle, you need to copy all required jar files to the lib/third-party directory.

[SafrasIqbal]

Hi @kschrab ,

Thank you for the response. I tried both the methods still it did not work for me. When add the dependency in the rti/mosaic-starter/pom.xml with version as <version>${mosaic.version}</version> it fails when installing. error as below.

Downloading from central: https://repo.maven.apache.org/maven2/org/apache/spark/spark-core_2.12/22.0-SNAPSHOT/maven-metadata.xml Downloading from central: https://repo.maven.apache.org/maven2/org/apache/spark/spark-core_2.12/22.0-SNAPSHOT/spark-core_2.12-22.0-SNAPSHOT.pom [WARNING] The POM for org.apache.spark:spark-core_2.12:jar:22.0-SNAPSHOT is missing, no dependency information available Downloading from central: https://repo.maven.apache.org/maven2/org/apache/spark/spark-core_2.12/22.0-SNAPSHOT/spark-core_2.12-22.0-SNAPSHOT.jar [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 7.898 s [INFO] Finished at: 2023-05-02T14:25:34+01:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project mosaic-starter: Could not resolve dependencies for project org.eclipse.mosaic:mosaic-starter:jar:22.0-SNAPSHOT: Could not find artifact org.apache.spark:spark-core_2.12:jar:22.0-SNAPSHOT in central (https://repo.maven.apache.org/maven2/) -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException

However, when I append the corresponding version its works but the original issue is not resolved. e.g. version replaced with <version>3.3.0</version> . I also tried placing the jar file in lib/third-party directory but no luck.

Thank you for the support. please let me know if any alternatives to resolve this issue.