How can I iterately process all files under one directory using mrjob
Well, finally I find that I can specify a directory as the input path and Hadoop will process all files in that directory.
Further in my case, I have sub-directories containing the input files. Hadoop will not transverse directory recursively and will raise error by default. A common trick is to use wildcard glob like
python count.py hdfs://master-host/directory/*/*.txt > result