How do I split a huge text file in python
linux has a split command
split -l 100000 file.txt
would split into files of equal 100,000 line size
Check out os.stat() for file size and file.readlines([sizehint]). Those two functions should be all you need for the reading part, and hopefully you know how to do the writing :)
As an alternative method, using the logging library:
>>> import logging.handlers>>> log = logging.getLogger()>>> fh = logging.handlers.RotatingFileHandler("D://filename.txt", maxBytes=2**20*100, backupCount=100) # 100 MB each, up to a maximum of 100 files>>> log.addHandler(fh)>>> log.setLevel(logging.INFO)>>> f = open("D://biglog.txt")>>> while True:... log.info(f.readline().strip())Your files will appear as follows:
filename.txt (end of file)
filename.txt.1
filename.txt.2
...
filename.txt.10 (start of file)
This is a quick and easy way to make a huge log file match your RotatingFileHandler implementation.