Lazy Method for Reading Big File in Python?
To write a lazy function, just use
def read_in_chunks(file_object, chunk_size=1024): """Lazy function (generator) to read a file piece by piece. Default chunk size: 1k.""" while True: data = file_object.read(chunk_size) if not data: break yield datawith open('really_big_file.dat') as f: for piece in read_in_chunks(f): process_data(piece)
Another option would be to use
iter and a helper function:
f = open('really_big_file.dat')def read1k(): return f.read(1024)for piece in iter(read1k, ''): process_data(piece)
If the file is line-based, the file object is already a lazy generator of lines:
for line in open('really_big_file.dat'): process_data(line)
If your computer, OS and python are 64-bit, then you can use the mmap module to map the contents of the file into memory and access it with indices and slices. Here an example from the documentation:
import mmapwith open("hello.txt", "r+") as f: # memory-map the file, size 0 means whole file map = mmap.mmap(f.fileno(), 0) # read content via standard file methods print map.readline() # prints "Hello Python!" # read content via slice notation print map[:5] # prints "Hello" # update content using slice notation; # note that new content must have same size map[6:] = " world!\n" # ... and read again using standard file methods map.seek(0) print map.readline() # prints "Hello world!" # close the map map.close()
If either your computer, OS or python are 32-bit, then mmap-ing large files can reserve large parts of your address space and starve your program of memory.
file.readlines() takes in an optional size argument which approximates the number of lines read in the lines returned.
bigfile = open('bigfilename','r')tmp_lines = bigfile.readlines(BUF_SIZE)while tmp_lines: process([line for line in tmp_lines]) tmp_lines = bigfile.readlines(BUF_SIZE)