How to handle IncompleteRead: in python
The link you included in your question is simply a wrapper that executes urllib's read() function, which catches any incomplete read exceptions for you. If you don't want to implement this entire patch, you could always just throw in a try/catch loop where you read your links. For example:
try: page = urllib2.urlopen(urls).read()except httplib.IncompleteRead, e: page = e.partial
for python3
try: page = request.urlopen(urls).read()except (http.client.IncompleteRead) as e: page = e.partial
Note this answer is Python 2 only (it was published in 2013)
I find out in my case : send HTTP/1.0 request , adding this , fix the problem.
import httplibhttplib.HTTPConnection._http_vsn = 10httplib.HTTPConnection._http_vsn_str = 'HTTP/1.0'
after I do the request :
req = urllib2.Request(url, post, headers)filedescriptor = urllib2.urlopen(req)img = filedescriptor.read()
after I back to http 1.1 with (for connections that support 1.1) :
httplib.HTTPConnection._http_vsn = 11httplib.HTTPConnection._http_vsn_str = 'HTTP/1.1'
the trick is use http 1.0 instead the default http/1.1http 1.1 could handle chunks but for some reason webserver don't , so we do the request in http 1.0
for Python3, it will tell you
ModuleNotFoundError: No module named 'httplib'
then try to use http.client Module it will solve the problem
import http.client as httphttp.HTTPConnection._http_vsn = 10http.HTTPConnection._http_vsn_str = 'HTTP/1.0'
What worked for me is catching IncompleteRead as an exception and harvesting the data you managed to read in each iteration by putting this into a loop like below: (Note, I am using Python 3.4.1 and the urllib library has changed between 2.7 and 3.4)
try: requestObj = urllib.request.urlopen(url, data) responseJSON="" while True: try: responseJSONpart = requestObj.read() except http.client.IncompleteRead as icread: responseJSON = responseJSON + icread.partial.decode('utf-8') continue else: responseJSON = responseJSON + responseJSONpart.decode('utf-8') break return json.loads(responseJSON)except Exception as RESTex: print("Exception occurred making REST call: " + RESTex.__str__())