ClassNotFoundException running GiraphRunner on a modified SimpleShortestPathsVertex

OK, after looking at the hadoop scripts along with Hadoop and Giraph source, I think I figured it out. The big hint came from Using the libjars option with Hadoop along with this line from the output:

WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

The cause appears to be that GiraphRunner uses its own ConfigurationUtils.parseArgs() to get the org.apache.commons.cli.CommandLine instead of using the recommended org.apache.hadoop.util.GenericOptionsParser.getCommandLine(), which honors the 'libjars' option. This led me to fall back on Hadoop's generic classpath-handling tools: CLASSPATH and/or HADOOP_CLASSPATH. Here's what worked:

• Set HADOOP_CLASSPATH to include your application jar and the gigraph core jar, using a colon delimiter.
• Pass -libjars using that same classpath but with a comma delimiter.

For example, on my machine:

$ export GIRAPH_HOME=/share/apps/giraph$ export HADOOP_CLASSPATH=/home/<me>/kdl_hadoop_play.jar:$GIRAPH_HOME/giraph-ex.jar:$HADOOP_CLASSPATH$ export LIBJARS=/home/<me>/kdl_hadoop_play.jar,$GIRAPH_HOME/giraph-core.jar$ hadoop fs -rm -R goutput/shortestpathsC2$ hadoop jar $GIRAPH_HOME/giraph-ex.jar org.apache.giraph.GiraphRunner \-Dgiraph.zkList=<myhost>:2181 \-libjars ${LIBJARS} \KdlSimpleShortestPathsVertex \-vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat \-vip /user/cornell/ginput/tiny_graph.txt \-of org.apache.giraph.io.formats.IdWithValueTextOutputFormat \-op /user/cornell/goutput/shortestpathsC2 \-ca SimpleShortestPathsVertex.source=2 \-w 1...$ hadoop fs -cat goutput/shortestpathsC2/p*

Which gives the expected output and results.

More generally, it would be helpful if the Giraph team changed the code to use the (apparently) more standard parser.

Hope that helps!

I don't know why this isn't working but there is a quick-and-dirty way to fix this. Try putting your code in giraph-examples/src/main/java/org/apache/giraph/examples/ directory (where SimpleShortestPath is located). And then build giraph-examples jar by running mvn -DskipTests --projects giraph-examples --also-make package. Then simply run your program as you did for SimpleShortestPath replacing SimpleShortestPath by your file name. I hope that helps.