Any gotchas using unicode_literals in Python 2.6? Any gotchas using unicode_literals in Python 2.6? python python

Any gotchas using unicode_literals in Python 2.6?


The main source of problems I've had working with unicode strings is when you mix utf-8 encoded strings with unicode ones.

For example, consider the following scripts.

two.py

# encoding: utf-8name = 'helló wörld from two'

one.py

# encoding: utf-8from __future__ import unicode_literalsimport twoname = 'helló wörld from one'print name + two.name

The output of running python one.py is:

Traceback (most recent call last):  File "one.py", line 5, in <module>    print name + two.nameUnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 4: ordinal not in range(128)

In this example, two.name is an utf-8 encoded string (not unicode) since it did not import unicode_literals, and one.name is an unicode string. When you mix both, python tries to decode the encoded string (assuming it's ascii) and convert it to unicode and fails. It would work if you did print name + two.name.decode('utf-8').

The same thing can happen if you encode a string and try to mix them later.For example, this works:

# encoding: utf-8html = '<html><body>helló wörld</body></html>'if isinstance(html, unicode):    html = html.encode('utf-8')print 'DEBUG: %s' % html

Output:

DEBUG: <html><body>helló wörld</body></html>

But after adding the import unicode_literals it does NOT:

# encoding: utf-8from __future__ import unicode_literalshtml = '<html><body>helló wörld</body></html>'if isinstance(html, unicode):    html = html.encode('utf-8')print 'DEBUG: %s' % html

Output:

Traceback (most recent call last):  File "test.py", line 6, in <module>    print 'DEBUG: %s' % htmlUnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 16: ordinal not in range(128)

It fails because 'DEBUG: %s' is an unicode string and therefore python tries to decode html. A couple of ways to fix the print are either doing print str('DEBUG: %s') % html or print 'DEBUG: %s' % html.decode('utf-8').

I hope this helps you understand the potential gotchas when using unicode strings.


Also in 2.6 (before python 2.6.5 RC1+) unicode literals doesn't play nice with keyword arguments (issue4978):

The following code for example works without unicode_literals, but fails with TypeError: keywords must be string if unicode_literals is used.

  >>> def foo(a=None): pass  ...  >>> foo(**{'a':1})  Traceback (most recent call last):    File "<stdin>", line 1, in <module>      TypeError: foo() keywords must be strings


I did find that if you add the unicode_literals directive you should also add something like:

 # -*- coding: utf-8

to the first or second line your .py file. Otherwise lines such as:

 foo = "barré"

result in an an error such as:

SyntaxError: Non-ASCII character '\xc3' in file mumble.py on line 198, but no encoding declared; see http://www.python.org/peps/pep-0263.html  for details