Python - mechanism to identify compressed file type and uncompress

This page has a list of "magic" file signatures. Grab the ones you need and put them in a dict like below. Then we need a function that matches the dict keys with the start of the file. I've written a suggestion, though it can be optimized by preprocessing the magic_dict into e.g. one giant compiled regexp.

magic_dict = {    "\x1f\x8b\x08": "gz",    "\x42\x5a\x68": "bz2",    "\x50\x4b\x03\x04": "zip"    }max_len = max(len(x) for x in magic_dict)def file_type(filename):    with open(filename) as f:        file_start = f.read(max_len)    for magic, filetype in magic_dict.items():        if file_start.startswith(magic):            return filetype    return "no match"

This solution should be cross-plattform and is of course not dependent on file name extension, but it may give false positives for files with random content that just happen to start with some specific magic bytes.

Based on lazyr's answer and my comment, here is what I mean:

class CompressedFile (object):    magic = None    file_type = None    mime_type = None    proper_extension = None    def __init__(self, f):        # f is an open file or file like object        self.f = f        self.accessor = self.open()    @classmethod    def is_magic(self, data):        return data.startswith(self.magic)    def open(self):        return Noneimport zipfileclass ZIPFile (CompressedFile):    magic = '\x50\x4b\x03\x04'    file_type = 'zip'    mime_type = 'compressed/zip'    def open(self):        return zipfile.ZipFile(self.f)import bz2class BZ2File (CompressedFile):    magic = '\x42\x5a\x68'    file_type = 'bz2'    mime_type = 'compressed/bz2'    def open(self):        return bz2.BZ2File(self.f)import gzipclass GZFile (CompressedFile):    magic = '\x1f\x8b\x08'    file_type = 'gz'    mime_type = 'compressed/gz'    def open(self):        return gzip.GzipFile(self.f)# factory function to create a suitable instance for accessing filesdef get_compressed_file(filename):    with file(filename, 'rb') as f:        start_of_file = f.read(1024)        f.seek(0)        for cls in (ZIPFile, BZ2File, GZFile):            if cls.is_magic(start_of_file):                return cls(f)        return Nonefilename='test.zip'cf = get_compressed_file(filename)if cf is not None:    print filename, 'is a', cf.mime_type, 'file'    print cf.accessor

Can now access the compressed data using cf.accessor. All the modules provide similar methods like 'read()', 'write()', etc. to do this.

This is a complex question that depends on a number of factors: the most important being how portable your solution needs to be.

The basics behind finding the file type given a file is to find an identifying header in the file, usually something called a "magic sequence" or signature header, which identifies that a file is of a certain type. Its name or extension is usually not used if it can be avoided. For some files, Python has this built in. For example, to deal with .tar files, you can use the tarfile module, which has a convenient is_tarfile method. There is a similar module named zipfile. These modules will also let you extract files in pure Python.

For example:

f = file('myfile','r')if zipfile.is_zipfile(f):    zip = zipfile.ZipFile(f)    zip.extractall('/dest/dir')elif tarfile.is_tarfile(f):    ...

If your solution is Linux or OSX only, there is also the file command which will do a lot of the work for you. You can also use the built-in tools to uncompress the files. If you are just doing a simple script, this method is simpler and will give you better performance.