Mysterious exceptions when making many concurrent requests from urllib.request to HTTPServer

You're using the default listen() backlog value, which is probably the cause of a lot of those errors. This is not the number of simultaneous clients with connection already established, but the number of clients waiting on the listen queue before the connection is established. Change your server class to:

class FancyHTTPServer(ThreadingMixIn, HTTPServer):    def server_activate(self):        self.socket.listen(128)

128 is a reasonable limit. You might want to check socket.SOMAXCONN or your OS somaxconn if you want to increase it further. If you still have random errors under heavy load, you should check your ulimit settings and increase if needed.

I did that with your example and I got over 1000 threads running fine, so I think that should solve your problem.

Update

If it improved but it's still crashing with 200 simultaneous clients, then I'm pretty sure your main problem was the backlog size. Be aware that your problem is not the number of concurrent clients, but the number of concurrent connection requests. A brief explanation on what that means, without going too deep into TCP internals.

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)s.bind((HOST, PORT))s.listen(BACKLOG)while running:    conn, addr = s.accept()    do_something(conn, addr)

In this example, the socket is now accepting connections on the given port, and the s.accept() call will block until a client connects. You can have many clients trying to connect simultaneously, and depending on your application you might not be able to call s.accept() and dispatch the client connection as fast as the clients are trying to connect. Pending clients are queued, and the max size of that queue is determined by the BACKLOG value. If the queue is full, clients will fail with a Connection Refused error.

Threading doesn't help, because what the ThreadingMixIn class does is to execute the do_something(conn, addr) call in a separate thread, so the server can return to the mainloop and the s.accept() call.

You can try increasing the backlog further, but there will be a point where that won't help because if the queue grows too large some clients will timeout before the server performs the s.accept() call.

So, as I said above, your problem is the number of simultaneous connection attempts, not the number of simultaneous clients. Maybe 128 is enough for your real application, but you're getting an error on your test because you're trying to connect with all 200 threads at once and flooding the queue.

Don't worry about ulimit unless you get a Too many open files error, but if you want to increase the backlog beyond 128, do some research on socket.SOMAXCONN. This is a good start: https://utcc.utoronto.ca/~cks/space/blog/python/AvoidSOMAXCONN

I'd say that your issue is related to some IO blocking since I've successfully executed your code on NodeJs. I also noticed that both the server and the client have trouble to work individually.

But it is possible to increase the number of requests with a few modifications:

• Define the number of concurrent connections:

  http.server.HTTPServer.request_queue_size = 500

• Run the server in a different process:

  server = multiprocessing.Process(target=RunHTTPServer) server.start()

• Use a connection pool on the client side to execute the requests

• Use a thread pool on the server side to handle the requests

• Allow the reuse of the connection on the client side by setting the schema and by using the "keep-alive" header

With all these modifications, I managed to run the code with 500 threads without any issue. So if you want to give it a try, here is the complete code:

import randomfrom time import sleep, clockfrom http.server import BaseHTTPRequestHandler, HTTPServerfrom multiprocessing import Processfrom multiprocessing.pool import ThreadPoolfrom socketserver import ThreadingMixInfrom concurrent.futures import ThreadPoolExecutorfrom urllib3 import HTTPConnectionPoolfrom urllib.error import HTTPErrorclass HTTPServerThreaded(HTTPServer):    request_queue_size = 500    allow_reuse_address = True    def serve_forever(self):        executor = ThreadPoolExecutor(max_workers=self.request_queue_size)        while True:          try:              request, client_address = self.get_request()              executor.submit(ThreadingMixIn.process_request_thread, self, request, client_address)          except OSError:              break        self.server_close()class MyRequestHandler(BaseHTTPRequestHandler):    default_request_version = 'HTTP/1.1'    def do_GET(self):        sleep(random.uniform(0, 1) / 100.0)        data = b"abcdef"        self.send_response(200)        self.send_header("Content-type", 'text/html')        self.send_header("Content-length", len(data))        self.end_headers()        self.wfile.write(data)    def log_request(self, code=None, size=None):        passdef RunHTTPServer():    server = HTTPServerThreaded(('127.0.0.1', 5674), MyRequestHandler)    server.serve_forever()client_headers = {     'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)',    'Content-Type': 'text/plain',    'Connection': 'keep-alive'}client_pool = Nonedef request_is_ok(number):    response = client_pool.request('GET', "/test" + str(number), headers=client_headers)    return response.status == 200 and response.data == b"abcdef"if __name__ == '__main__':    # start the server in another process    server = Process(target=RunHTTPServer)    server.start()    # start a connection pool for the clients    client_pool = HTTPConnectionPool('127.0.0.1', 5674)    # execute the requests    with ThreadPool(500) as thread_pool:        start = clock()        for i in range(5):            numbers = [random.randint(0, 99999) for j in range(20000)]            for j, result in enumerate(thread_pool.imap(request_is_ok, numbers)):                if j % 1000 == 0:                    print(i, j, result)        end = clock()        print("execution time: %s" % (end-start,))

Update 1:

Increasing the request_queue_size just gives you more space to store the requests that can't be executed at the time so they can be executed later.So the longer the queue, the higher the dispersion for the response time, which is I believe the opposite of your goal here.As for ThreadingMixIn, it's not ideal since it creates and destroy a thread for every request and it's expensive. A better choice to reduce the waiting queue is to use a pool of reusable threads to handle the requests.

The reason for running the server in another process is to take advantage of another CPU to reduce the execution time.

For the client side using a HTTPConnectionPool was the only way I found to keep a constant flow of requests since I had some weird behaviour with urlopen while analysing the connections.

The norm is to only use as many threads as cores, hence the 8 thread requirement (including virtual cores). The threading model is the easiest to get working, but it's really a rubbish way of doing it. A better way to handle multiple connections is to use an asynchronous approach. It's more difficult though.

With your threading method you could start by investigating whether the process stays open after you exit the program. This would mean that your threads aren't closing, and will obviously cause issues.

Try this...

class FancyHTTPServer(ThreadingMixIn, HTTPServer):    daemon_threads = True

That will ensure that your threads close properly. It may well happen automatically in the thread pool but it's probably worth trying anyway.