How to fetch a non-ascii url with urlopen?
Strictly speaking URIs can't contain non-ASCII characters; what you have there is an IRI.
To convert an IRI to a plain ASCII URI:
non-ASCII characters in the hostname part of the address have to be encoded using the Punycode-based IDNA algorithm;
non-ASCII characters in the path, and most of the other parts of the address have to be encoded using UTF-8 and %-encoding, as per Ignacio's answer.
So:
import re, urlparsedef urlEncodeNonAscii(b): return re.sub('[\x80-\xFF]', lambda c: '%%%02x' % ord(c.group(0)), b)def iriToUri(iri): parts= urlparse.urlparse(iri) return urlparse.urlunparse( part.encode('idna') if parti==1 else urlEncodeNonAscii(part.encode('utf-8')) for parti, part in enumerate(parts) )>>> iriToUri(u'http://www.a\u0131b.com/a\u0131b')'http://www.xn--ab-hpa.com/a%c4%b1b'
(Technically this still isn't quite good enough in the general case because urlparse
doesn't split away any user:pass@
prefix or :port
suffix on the hostname. Only the hostname part should be IDNA encoded. It's easier to encode using normal urllib.quote
and .encode('idna')
at the time you're constructing a URL than to have to pull an IRI apart.)
In python3, use the urllib.parse.quote
function on the non-ascii string:
>>> from urllib.request import urlopen >>> from urllib.parse import quote >>> chinese_wikipedia = 'http://zh.wikipedia.org/wiki/Wikipedia:' + quote('首页')>>> urlopen(chinese_wikipedia)
Python 3 has libraries to handle this situation. Useurllib.parse.urlsplit
to split the URL into its components, andurllib.parse.quote
to properly quote/escape the unicode charactersand urllib.parse.urlunsplit
to join it back together.
>>> import urllib.parse>>> url = 'http://example.com/unicodè'>>> url = urllib.parse.urlsplit(url)>>> url = list(url)>>> url[2] = urllib.parse.quote(url[2])>>> url = urllib.parse.urlunsplit(url)>>> print(url)http://example.com/unicod%C3%A8