Write to UTF-8 file in Python
I believe the problem is that codecs.BOM_UTF8
is a byte string, not a Unicode string. I suspect the file handler is trying to guess what you really mean based on "I'm meant to be writing Unicode as UTF-8-encoded text, but you've given me a byte string!"
Try writing the Unicode string for the byte order mark (i.e. Unicode U+FEFF) directly, so that the file just encodes that as UTF-8:
import codecsfile = codecs.open("lol", "w", "utf-8")file.write(u'\ufeff')file.close()
(That seems to give the right answer - a file with bytes EF BB BF.)
EDIT: S. Lott's suggestion of using "utf-8-sig" as the encoding is a better one than explicitly writing the BOM yourself, but I'll leave this answer here as it explains what was going wrong before.
Read the following: http://docs.python.org/library/codecs.html#module-encodings.utf_8_sig
Do this
with codecs.open("test_output", "w", "utf-8-sig") as temp: temp.write("hi mom\n") temp.write(u"This has ♭")
The resulting file is UTF-8 with the expected BOM.
@S-Lott gives the right procedure, but expanding on the Unicode issues, the Python interpreter can provide more insights.
Jon Skeet is right (unusual) about the codecs
module - it contains byte strings:
>>> import codecs>>> codecs.BOM'\xff\xfe'>>> codecs.BOM_UTF8'\xef\xbb\xbf'>>>
Picking another nit, the BOM
has a standard Unicode name, and it can be entered as:
>>> bom= u"\N{ZERO WIDTH NO-BREAK SPACE}">>> bomu'\ufeff'
It is also accessible via unicodedata
:
>>> import unicodedata>>> unicodedata.lookup('ZERO WIDTH NO-BREAK SPACE')u'\ufeff'>>>