How can I speed up fetching pages with urllib2 in python?

EDIT: I'm expanding the answer to include a more polished example. I have found a lot hostility and misinformation in this post regarding threading v.s. async I/O. Therefore I also adding more argument to refute certain invalid claim. I hope this will help people to choose the right tool for the right job.

This is a dup to a question 3 days ago.

Python urllib2.open is slow, need a better way to read several urls - Stack Overflow Python urllib2.urlopen() is slow, need a better way to read several urls

I'm polishing the code to show how to fetch multiple webpage in parallel using threads.

import timeimport threadingimport Queue# utility - spawn a thread to execute target for each argsdef run_parallel_in_threads(target, args_list):    result = Queue.Queue()    # wrapper to collect return value in a Queue    def task_wrapper(*args):        result.put(target(*args))    threads = [threading.Thread(target=task_wrapper, args=args) for args in args_list]    for t in threads:        t.start()    for t in threads:        t.join()    return resultdef dummy_task(n):    for i in xrange(n):        time.sleep(0.1)    return n# below is the application codeurls = [    ('http://www.google.com/',),    ('http://www.lycos.com/',),    ('http://www.bing.com/',),    ('http://www.altavista.com/',),    ('http://achewood.com/',),]def fetch(url):    return urllib2.urlopen(url).read()run_parallel_in_threads(fetch, urls)

As you can see, the application specific code has only 3 lines, which can be collapsed into 1 line if you are aggressive. I don't think anyone can justify their claim that this is complex and unmaintainable.

Unfortunately most other threading code posted here has some flaws. Many of them do active polling to wait for the code to finish. join() is a better way to synchronize the code. I think this code has improved upon all the threading examples so far.

keep-alive connection

WoLpH's suggestion about using keep-alive connection could be very useful if all you URLs are pointing to the same server.

twisted

Aaron Gallagher is a fans of twisted framework and he is hostile any people who suggest thread. Unfortunately a lot of his claims are misinformation. For example he said "-1 for suggesting threads. This is IO-bound; threads are useless here." This contrary to evidence as both Nick T and I have demonstrated speed gain from the using thread. In fact I/O bound application has the most to gain from using Python's thread (v.s. no gain in CPU bound application). Aaron's misguided criticism on thread shows he is rather confused about parallel programming in general.

Right tool for the right job

I'm well aware of the issues pertain to parallel programming using threads, python, async I/O and so on. Each tool has their pros and cons. For each situation there is an appropriate tool. I'm not against twisted (though I have not deployed one myself). But I don't believe we can flat out say that thread is BAD and twisted is GOOD in all situations.

For example, if the OP's requirement is to fetch 10,000 website in parallel, async I/O will be prefereable. Threading won't be appropriable (unless maybe with stackless Python).

Aaron's opposition to threads are mostly generalizations. He fail to recognize that this is a trivial parallelization task. Each task is independent and do not share resources. So most of his attack do not apply.

Given my code has no external dependency, I'll call it right tool for the right job.

Performance

I think most people would agree that performance of this task is largely depend on the networking code and the external server, where the performance of platform code should have negligible effect. However Aaron's benchmark show an 50% speed gain over the threaded code. I think it is necessary to response to this apparent speed gain.

In Nick's code, there is an obvious flaw that caused the inefficiency. But how do you explain the 233ms speed gain over my code? I think even twisted fans will refrain from jumping into conclusion to attribute this to the efficiency of twisted. There are, after all, a huge amount of variable outside of the system code, like the remote server's performance, network, caching, and difference implementation between urllib2 and twisted web client and so on.

Just to make sure Python's threading will not incur a huge amount of inefficiency, I do a quick benchmark to spawn 5 threads and then 500 threads. I am quite comfortable to say the overhead of spawning 5 thread is negligible and cannot explain the 233ms speed difference.

In [274]: %time run_parallel_in_threads(dummy_task, [(0,)]*5)CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 sWall time: 0.00 sOut[275]: <Queue.Queue instance at 0x038B2878>In [276]: %time run_parallel_in_threads(dummy_task, [(0,)]*500)CPU times: user 0.16 s, sys: 0.00 s, total: 0.16 sWall time: 0.16 sIn [278]: %time run_parallel_in_threads(dummy_task, [(10,)]*500)CPU times: user 1.13 s, sys: 0.00 s, total: 1.13 sWall time: 1.13 s       <<<<<<<< This means 0.13s of overhead

Further testing on my parallel fetching shows a huge variability in the response time in 17 runs. (Unfortunately I don't have twisted to verify Aaron's code).

0.75 s0.38 s0.59 s0.38 s0.62 s1.50 s0.49 s0.36 s0.95 s0.43 s0.61 s0.81 s0.46 s1.21 s2.87 s1.04 s1.72 s

My testing does not support Aaron's conclusion that threading is consistently slower than async I/O by a measurable margin. Given the number of variables involved, I have to say this is not a valid test to measure the systematic performance difference between async I/O and threading.

Use twisted! It makes this kind of thing absurdly easy compared to, say, using threads.

from twisted.internet import defer, reactorfrom twisted.web.client import getPageimport timedef processPage(page, url):    # do somewthing here.    return url, len(page)def printResults(result):    for success, value in result:        if success:            print 'Success:', value        else:            print 'Failure:', value.getErrorMessage()def printDelta(_, start):    delta = time.time() - start    print 'ran in %0.3fs' % (delta,)    return deltaurls = [    'http://www.google.com/',    'http://www.lycos.com/',    'http://www.bing.com/',    'http://www.altavista.com/',    'http://achewood.com/',]def fetchURLs():    callbacks = []    for url in urls:        d = getPage(url)        d.addCallback(processPage, url)        callbacks.append(d)    callbacks = defer.DeferredList(callbacks)    callbacks.addCallback(printResults)    return callbacks@defer.inlineCallbacksdef main():    times = []    for x in xrange(5):        d = fetchURLs()        d.addCallback(printDelta, time.time())        times.append((yield d))    print 'avg time: %0.3fs' % (sum(times) / len(times),)reactor.callWhenRunning(main)reactor.run()

This code also performs better than any of the other solutions posted (edited after I closed some things that were using a lot of bandwidth):

Success: ('http://www.google.com/', 8135)Success: ('http://www.lycos.com/', 29996)Success: ('http://www.bing.com/', 28611)Success: ('http://www.altavista.com/', 8378)Success: ('http://achewood.com/', 15043)ran in 0.518sSuccess: ('http://www.google.com/', 8135)Success: ('http://www.lycos.com/', 30349)Success: ('http://www.bing.com/', 28611)Success: ('http://www.altavista.com/', 8378)Success: ('http://achewood.com/', 15043)ran in 0.461sSuccess: ('http://www.google.com/', 8135)Success: ('http://www.lycos.com/', 30033)Success: ('http://www.bing.com/', 28611)Success: ('http://www.altavista.com/', 8378)Success: ('http://achewood.com/', 15043)ran in 0.435sSuccess: ('http://www.google.com/', 8117)Success: ('http://www.lycos.com/', 30349)Success: ('http://www.bing.com/', 28611)Success: ('http://www.altavista.com/', 8378)Success: ('http://achewood.com/', 15043)ran in 0.449sSuccess: ('http://www.google.com/', 8135)Success: ('http://www.lycos.com/', 30349)Success: ('http://www.bing.com/', 28611)Success: ('http://www.altavista.com/', 8378)Success: ('http://achewood.com/', 15043)ran in 0.547savg time: 0.482s

And using Nick T's code, rigged up to also give the average of five and show the output better:

Starting threaded reads:...took 1.921520 seconds ([8117, 30070, 15043, 8386, 28611])Starting threaded reads:...took 1.779461 seconds ([8135, 15043, 8386, 30349, 28611])Starting threaded reads:...took 1.756968 seconds ([8135, 8386, 15043, 30349, 28611])Starting threaded reads:...took 1.762956 seconds ([8386, 8135, 15043, 29996, 28611])Starting threaded reads:...took 1.654377 seconds ([8117, 30349, 15043, 8386, 28611])avg time: 1.775sStarting sequential reads:...took 1.389803 seconds ([8135, 30147, 28611, 8386, 15043])Starting sequential reads:...took 1.457451 seconds ([8135, 30051, 28611, 8386, 15043])Starting sequential reads:...took 1.432214 seconds ([8135, 29996, 28611, 8386, 15043])Starting sequential reads:...took 1.447866 seconds ([8117, 30028, 28611, 8386, 15043])Starting sequential reads:...took 1.468946 seconds ([8153, 30051, 28611, 8386, 15043])avg time: 1.439s

And using Wai Yip Tung's code:

Fetched 8117 from http://www.google.com/Fetched 28611 from http://www.bing.com/Fetched 8386 from http://www.altavista.com/Fetched 30051 from http://www.lycos.com/Fetched 15043 from http://achewood.com/done in 0.704sFetched 8117 from http://www.google.com/Fetched 28611 from http://www.bing.com/Fetched 8386 from http://www.altavista.com/Fetched 30114 from http://www.lycos.com/Fetched 15043 from http://achewood.com/done in 0.845sFetched 8153 from http://www.google.com/Fetched 28611 from http://www.bing.com/Fetched 8386 from http://www.altavista.com/Fetched 30070 from http://www.lycos.com/Fetched 15043 from http://achewood.com/done in 0.689sFetched 8117 from http://www.google.com/Fetched 28611 from http://www.bing.com/Fetched 8386 from http://www.altavista.com/Fetched 30114 from http://www.lycos.com/Fetched 15043 from http://achewood.com/done in 0.647sFetched 8135 from http://www.google.com/Fetched 28611 from http://www.bing.com/Fetched 8386 from http://www.altavista.com/Fetched 30349 from http://www.lycos.com/Fetched 15043 from http://achewood.com/done in 0.693savg time: 0.715s

I've gotta say, I do like that the sequential fetches performed better for me.

Here is an example using python Threads. The other threaded examples here launch a thread per url, which is not very friendly behaviour if it causes too many hits for the server to handle (for example it is common for spiders to have many urls on the same host)

from threading import Threadfrom urllib2 import urlopenfrom time import time, sleepWORKERS=1urls = ['http://docs.python.org/library/threading.html',        'http://docs.python.org/library/thread.html',        'http://docs.python.org/library/multiprocessing.html',        'http://docs.python.org/howto/urllib2.html']*10results = []class Worker(Thread):    def run(self):        while urls:            url = urls.pop()            results.append((url, urlopen(url).read()))start = time()threads = [Worker() for i in range(WORKERS)]any(t.start() for t in threads)while len(results)<40:    sleep(0.1)print time()-start

Note: The times given here are for 40 urls and will depend a lot on the speed of your internet connection and the latency to the server. Being in Australia, my ping is > 300ms

With WORKERS=1 it took 86 seconds to run
With WORKERS=4 it took 23 seconds to run
with WORKERS=10 it took 10 seconds to run

so having 10 threads downloading is 8.6 times as fast as a single thread.

Here is an upgraded version that uses a Queue. There are at least a couple of advantages.
1. The urls are requested in the order that they appear in the list
2. Can use q.join() to detect when the requests have all completed
3. The results are kept in the same order as the url list

from threading import Threadfrom urllib2 import urlopenfrom time import time, sleepfrom Queue import QueueWORKERS=10urls = ['http://docs.python.org/library/threading.html',        'http://docs.python.org/library/thread.html',        'http://docs.python.org/library/multiprocessing.html',        'http://docs.python.org/howto/urllib2.html']*10results = [None]*len(urls)def worker():    while True:        i, url = q.get()        # print "requesting ", i, url       # if you want to see what's going on        results[i]=urlopen(url).read()        q.task_done()start = time()q = Queue()for i in range(WORKERS):    t=Thread(target=worker)    t.daemon = True    t.start()for i,url in enumerate(urls):    q.put((i,url))q.join()print time()-start