Python UTF-8 conversion problem Python UTF-8 conversion problem django django

Python UTF-8 conversion problem


Try to put the unicode signature u before your string, e.g. u'YOUR_ALFA_CHAR' and revise your database encoding, because Django always supports UTF-8 .


What you seem to have is the individual bytes of a UTF-8 encoded string interpreted as unicode codepoints. You can "decode" your string out of this strange form with:

p.name = ''.join(chr(ord(x)) for x in p.name)

or perhaps

p.name = ''.join(chr(ord(x)) for x in p.name).decode('utf8')

One way to get your strings "encoded" into this form is

''.join(unichr(ord(x)) for x in '\xce\xb1')

although I have a feeling your strings actually got in this state by different components of your system disagreeing on the encoding in use.

You will probably have to fix the source of your bad "encoding" rather than just fixing the data currently in your database. And the code above might be okay to convert your bad data once, but I would advise you don't insert this code into your Django app.


The problem is that p.name was not correctly stored and/or read in from the database.

Unicode small alpha is U+03B1 and p.name should have printed as u'\x03b1' or if you were using a Unicode capable terminal the actual alpha symbol itself may have been printed in quotes. Note the difference between u'\xce\xb1' and u'\xceb1'. The former is a two character string and the latter in a single character string. I have no idea how the '03' byte of the UTF-8 got translated into 'CE'.