Hadoop Streaming - Unable to find file error

Looking at the example on the HadoopStreaming wiki page, it seems that you should change

-mapper scripts/mapper.py -file scripts/mapper.py 

to

-mapper mapper.py -file scripts/mapper.py 

since "shipped files go to the working directory". You might also need to specify the python interpreter directly:

-mapper /path/to/python mapper.py -file scripts/mapper.py 

Your problem most likely is that python executable does not exist on the slaves (where TaskTracker is running). Java will give the same error message.

Install it everywhere where it's used. Un your file you can use shebang as you probably already do:

#!/usr/bin/python -Orestofthecode

Make sure that the path after the shebang is the same where python is installed on the TaskTrackers.

One other sneaky thing can cause this. If your line-endings on the script are DOS-style, then your first line (the "shebang line") may look like this to the naked eye:

#!/usr/bin/python...my code here...

but its bytes look like this to the kernel when it tries to execute your script:

% od -a myScript.py0000000   #   !   /   u   s   r   /   b   i   n   /   p   y   t   h   o0000020   n  cr  nl  cr  nl   .   .   .   m   y  sp   c   o   d   e  sp0000040   h   e   r   e   .   .   .  cr  nl

It's looking for an executable called "/usr/bin/python\r", which it can't find, so it dies with "No such file or directory".

This bit me today, again, so I had to write it down somewhere on SO.