Running Mobius Spark – The system cannot find the path specified error

Mobius is a project to enable using C# to develop against Apache Spark. The project page on GIT has the directions to set up and use Mobius. However, when I tried those, I ran into “The system cannot find the path specified” error. Essentially, the error is because the various directory and file paths referred to in the scripts did not exist on my machine.

The entry point for running a Mobius project is the “sparkclr-submit.cmd” found in the “..\runtime\scripts” folder.  This script gets the various PATH variables and in turn calls the “spark-submit.cmd” command found in the bin folder of the Spark installation directory.

Obviously, you need to have Spark installed. Then, ensure that you have set up all the PATH variables referred to in the “sparkclr-submit.cmd”; these inlude the following variables among others

  1. SPARK_HOME
  2. JAVA_HOME
  3. HADOOP_HOME
  4. SPARKCLR_HOME

Next, ensure that the various paths implied in the script are valid on your machine. For example, the sparkclr-submit.cmd file uses the JAVA_HOME variable to call and submit to the java program. And the command looks something like this:

“%JAVA_HOME%\bin\java” -cp “%LAUNCH_CLASSPATH%”

On my machine though, the path “%JAVA_HOME%\bin\java” did not exist and the actual path on my machine was “%JAVA_HOME%\java” (notice that “bin” is not present). This could have been because of something I did when I first installed java.

Likewise, ensure that all the other paths implied in the sparkclr-submit.cmd script are valid. If you still keep getting the “cannot find the path specified” error, you may have to hunt through the various paths implied in the child scripts that the sparkclr-submit.cmd calls.

Advertisements