Python urllib2 Progress Hook

Here's a fully working example that builds on Anurag's approach of chunking in a response. My version allows you to set the the chunk size, and attach an arbitrary reporting function:

import urllib2, sysdef chunk_report(bytes_so_far, chunk_size, total_size):   percent = float(bytes_so_far) / total_size   percent = round(percent*100, 2)   sys.stdout.write("Downloaded %d of %d bytes (%0.2f%%)\r" %        (bytes_so_far, total_size, percent))   if bytes_so_far >= total_size:      sys.stdout.write('\n')def chunk_read(response, chunk_size=8192, report_hook=None):   total_size = response.info().getheader('Content-Length').strip()   total_size = int(total_size)   bytes_so_far = 0   while 1:      chunk = response.read(chunk_size)      bytes_so_far += len(chunk)      if not chunk:         break      if report_hook:         report_hook(bytes_so_far, chunk_size, total_size)   return bytes_so_farif __name__ == '__main__':   response = urllib2.urlopen('http://www.ebay.com');   chunk_read(response, report_hook=chunk_report)

Why not just read data in chunks and do whatever you want to do in between, e.g. run in a thread, hook into a UI, etc etc

import urllib2urlfile = urllib2.urlopen("http://www.google.com")data_list = []chunk = 4096while 1:    data = urlfile.read(chunk)    if not data:        print "done."        break    data_list.append(data)    print "Read %s bytes"%len(data)

output:

Read 4096 bytesRead 3113 bytesdone.

urlgrabber has built-in support for progress notification.