Convert UTF-8 with BOM to UTF-8 with no BOM in Python

Simply use the "utf-8-sig" codec:

fp = open("file.txt")s = fp.read()u = s.decode("utf-8-sig")

That gives you a unicode string without the BOM. You can then use

s = u.encode("utf-8")

to get a normal UTF-8 encoded string back in s. If your files are big, then you should avoid reading them all into memory. The BOM is simply three bytes at the beginning of the file, so you can use this code to strip them out of the file:

import os, sys, codecsBUFSIZE = 4096BOMLEN = len(codecs.BOM_UTF8)path = sys.argv[1]with open(path, "r+b") as fp:    chunk = fp.read(BUFSIZE)    if chunk.startswith(codecs.BOM_UTF8):        i = 0        chunk = chunk[BOMLEN:]        while chunk:            fp.seek(i)            fp.write(chunk)            i += len(chunk)            fp.seek(BOMLEN, os.SEEK_CUR)            chunk = fp.read(BUFSIZE)        fp.seek(-BOMLEN, os.SEEK_CUR)        fp.truncate()

It opens the file, reads a chunk, and writes it out to the file 3 bytes earlier than where it read it. The file is rewritten in-place. As easier solution is to write the shorter file to a new file like newtover's answer. That would be simpler, but use twice the disk space for a short period.

As for guessing the encoding, then you can just loop through the encoding from most to least specific:

def decode(s):    for encoding in "utf-8-sig", "utf-16":        try:            return s.decode(encoding)        except UnicodeDecodeError:            continue    return s.decode("latin-1") # will always work

An UTF-16 encoded file wont decode as UTF-8, so we try with UTF-8 first. If that fails, then we try with UTF-16. Finally, we use Latin-1 — this will always work since all 256 bytes are legal values in Latin-1. You may want to return None instead in this case since it's really a fallback and your code might want to handle this more carefully (if it can).

In Python 3 it's quite easy: read the file and rewrite it with utf-8 encoding:

s = open(bom_file, mode='r', encoding='utf-8-sig').read()open(bom_file, mode='w', encoding='utf-8').write(s)

import codecsimport shutilimport syss = sys.stdin.read(3)if s != codecs.BOM_UTF8:    sys.stdout.write(s)shutil.copyfileobj(sys.stdin, sys.stdout)