NullPointerException in MapReduce Sorting Program NullPointerException in MapReduce Sorting Program hadoop hadoop

NullPointerException in MapReduce Sorting Program


Actually, there is a problem with MySortComparator2 constructor. The code should looks like

protected MySortComparator2() {      super(Text.class, true);}

where the first parameter is your key class and the second parameter's value ensures WritableComparator is instantiated in a way that WritableComparator.compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) can invoke MySortComparator2.compare(WritableComparable a, WritableComparable b)


You need to implement/override this method, too:

public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {    // per your desired no-sort logic    return 0;}

I think that your comparator is being constructed in such a way that the variables mentioned in the super implementation are null (and this is the method that's being called in support of the sort - not the method you wrote above). That's why you're getting the null pointer exception. By overriding the method with an implementation that doesn't use the variables, you can avoid the exception.


As Chris Gerken said You need to override this method while extending WritableComparator or implement RawComparator instead of WritableComparator.

public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) {    return 0;}

and as you said you wanted to see no sorting to be done but if you return 0 that means every time MapReduce tries to sort/compare it sees every key as the same thing so, you will receive only one key, value pair which will be the first key in the map task which gets finished first and the value with number of words in the input file. Hope you understand what I am saying. If your input is something like this

why are rockets cylindrical

your reduce output will be

why  4

since it assumes everything as the same key. I hope this helps.