Python read file as stream from HDFS
You want xreadlines, it reads lines from a file without loading the whole file into memory.
Edit:
Now I see your question, you just need to get the stdout pipe from your Popen object:
cat = subprocess.Popen(["hadoop", "fs", "-cat", "/path/to/myfile"], stdout=subprocess.PIPE)for line in cat.stdout: print line
If you want to avoid adding external dependencies at any cost, Keith's answer is the way to go. Pydoop, on the other hand, could make your life much easier:
import pydoop.hdfs as hdfswith hdfs.open('/user/myuser/filename') as f: for line in f: do_something(line)Regarding your concerns, Pydoop is actively developed and has been used in production for years at CRS4, mostly for computational biology applications.
Simone
In the last two years, there has been a lot of motion on Hadoop-Streaming. This is pretty fast according to Cloudera: http://blog.cloudera.com/blog/2013/01/a-guide-to-python-frameworks-for-hadoop/ I've had good success with it.