decompressing huge json gzip file causing memory error in python
Create a decompression object using z=zlib.decompressobj()
, and then do z.decompress(some_compressed_data, max)
, which will return no more than max
bytes of uncompressed data. You then call again with z.decompress(z.unconsumed_tail, max)
until the rest of some_compressed_data
is consumed, and then feed it more compressed data.
You will need to then be able to process the resulting uncompressed data a chunk at a time.