windows-instructions.md

Building Mobius

Prerequisites

• Windows Server 2012 or above; or, 64-bit Windows 7 or above.
• Developer Command Prompt for Visual Studio 2013 or above, which comes with .NET Framework 4.5 or above. Note: Visual Studio 2015 Community Edition is FREE.
• 64-bit JDK 7u85 or above; or, 64-bit JDK 8u60 or above. OpenJDK for Windows can be downloaded from http://www.azul.com/downloads/zulu/zulu-windows/; Oracle JDK8 for Windows is available at Oracle website.

The following environment variables should be set properly in the Developer Command Prompt for Visual Studio:

• JAVA_HOME

Note: Mobius includes a C++ component (RIOSock.dll) for leveraging the socket optimization available in Windows. If your build environment does not have VC++ Build Toolset installed, RIOSock project will be skipped during the build. This is an optional component and Mobius will be fully functional even without this component (socket optimization will be unavaiable though). You need to enable VC++ project type in Visual Studio, or install Visual C++ Build Tools, if you want to build C++ components. Mobius releases include RIOSock.dll.

Instructions

• In the Developer Command Prompt for Visual Studio where JAVA_HOME is set properly, navigate to Mobius\build directory:

   Build.cmd  
  

• Optional:

  • Under Mobius\scala directory, run the following command to clean spark-clr*.jar built above:

     mvn clean
    

  • Under Mobius\csharp directory, run the following command to clean the .NET binaries built above:

     Clean.cmd  
    

Build.cmd downloads necessary build tools; after the build is done, it prepares the folowing directories under Mobius\build\runtime:

• bin ( Microsoft.Spark.CSharp.Adapter.dll, CSharpWorker.exe)
• data ( Mobius\csharp\Samples\Microsoft.Spark.CSharp\data\* )
• dependencies (jar files Mobius functionality depends on)
• lib ( spark-clr*.jar )
• repl (components required for REPL functionality in Mobius)
• samples ( The contents of Mobius\csharp\Samples\Microsoft.Spark.CSharp\bin\Release\*, including Microsoft.Spark.CSharp.Adapter.dll, CSharpWorker.exe, SparkCLRSamples.exe, SparkCLRSamples.exe.Config etc. )
• scripts ( sparkclr-submit.cmd, sparkclr-submit.sh, sparclr-shell.cmd )

Running Unit Tests

• In Visual Studio: Install NUnit3 Test Adapter. Run the tests through "Test" -> "Run" -> "All Tests"

• Install NUnit Runner 3.0 or above using NuGet (see https://www.nuget.org/packages/NUnit.Runners/). In Developer Command Prompt for VS, set NUNITCONSOLE to the path to nunit console, and navigate to Mobius\csharp and run the following command:

  Test.cmd
  

Running Samples

Samples demonstrate comprehesive usage of Mobius API and also serve as functional tests for the API. Following are the options to run samples:

• Local mode
• Standalone cluster
• YARN cluster
• Local mode dev environment (using artifacts built in the local Git repo)

The prerequisites for running Mobius samples are same as the ones for running any other Mobius applications. Refer to instructions for details on that. Local mode dev environment makes it easier to run samples in dev environment by downloading Spark.

Running in Local mode

sparkclr-submit.cmd --verbose --jars c:\MobiusRelease\dependencies\spark-csv_2.10-1.3.0.jar,c:\MobiusRelease\dependencies\commons-csV-1.1.jar --exe SparkCLRSamples.exe  c:\MobiusRelease\samples sparkclr.sampledata.loc c:\MobiusRelease\samples\data

Running in Standalone mode

sparkclr-submit.cmd --verbose --master spark://host:port --jars <hdfs path to spark-csv_2.10-1.3.0.jar,commons-csv-1.1.jar> --exe SparkCLRSamples.exe  %SPARKCLR_HOME%\samples sparkclr.sampledata.loc hdfs://path/to/mobius/sampledata

• When option --deploy-mode is specified with cluster, option --remote-sparkclr-jar is required and needs to be specified with a valid file path of spark-clr*.jar on HDFS.

Running in YARN mode

sparkclr-submit.cmd --verbose --master yarn-cluster --jars <hdfs path to spark-csv_2.10-1.3.0.jar,commons-csv-1.1.jar> --exe SparkCLRSamples.exe %SPARKCLR_HOME%\samples sparkclr.sampledata.loc hdfs://path/to/mobius/sampledata

Running in local mode dev environment

In the Developer Command Prompt for Visual Studio where JAVA_HOME is set properly, navigate to Mobius\build directory:

RunSamples.cmd  

It is required to run Build.cmd prior to running RunSamples.cmd.

RunSamples.cmd downloads the version of Apache Spark referenced in the current branch, sets up SPARK_HOME environment variable, points SPARKCLR_HOME to Mobius\build\runtime directory created by Build.cmd, and invokes sparkclr-submit.cmd, with spark.local.dir set to Mobius\build\runtime\Temp.

A few more RunSamples.cmd examples:

• To display all options supported by RunSamples.cmd:

  RunSamples.cmd  --help
  

• To run PiSample only:

  RunSamples.cmd  --torun pi*
  

• To run PiSample in verbose mode, with all logs displayed at console:

  RunSamples.cmd  --torun pi* --verbose