decompressing huge json gzip file causing memory error in python decompressing huge json gzip file causing memory error in python json json

decompressing huge json gzip file causing memory error in python


Create a decompression object using z=zlib.decompressobj(), and then do z.decompress(some_compressed_data, max), which will return no more than max bytes of uncompressed data. You then call again with z.decompress(z.unconsumed_tail, max) until the rest of some_compressed_data is consumed, and then feed it more compressed data.

You will need to then be able to process the resulting uncompressed data a chunk at a time.