Is the Session object from Python's Requests library thread safe?
After reviewing the source of requests.session
, I'm going to say the session object might be thread-safe, depending on the implementation of CookieJar being used.
Session.prepare_request
reads from self.cookies
, and Session.send
calls extract_cookies_to_jar(self.cookies, ...)
, and that calls jar.extract_cookies(...)
(jar
being self.cookies
in this case).
The source for Python 2.7's cookielib
acquires a lock (threading.RLock
) while it updates the jar, so it appears to be thread-safe. On the other hand, the documentation for cookielib
says nothing about thread-safety, so maybe this feature should not be depended on?
UPDATE
If your threads are mutating any attributes of the session object such as headers
, proxies
, stream
, etc. or calling the mount
method or using the session with the with
statement, etc. then it is not thread-safe.
https://github.com/psf/requests/issues/1871 implies that Session is not thread-safe, and that at least one maintainer recommends one Session per thread.
I just opened https://github.com/psf/requests/issues/2766 to clarify the documentation.
I also faced the same question and went to the source code to find a suitable solution for me.In my opinion Session class generally has various problems.
- It initializes the default HTTPAdapter in the constructor and leaks it if you mount another one to 'http' or 'https'.
- HTTPAdapter implementation maintains the connection pool, I think it is not something to create on each Session object instantiation.
- Session closes HTTPAdapter, thus you can't reuse the connection pool between different Session instances.
- Session class doesn't seem to be thread safe according to various discussions.
- HTTPAdapter internally uses the urlib3.PoolManager. And I didn't find any obvious problem related to the thread safety in the source code, so I would rather trust the documentation, which says that urlib3 is thread safe.
As the conclusion from the above list I didn't find anything better than overriding Session class
class HttpSession(Session): def __init__(self, adapter: HTTPAdapter): self.headers = default_headers() self.auth = None self.proxies = {} self.hooks = default_hooks() self.params = {} self.stream = False self.verify = True self.cert = None self.max_redirects = DEFAULT_REDIRECT_LIMIT self.trust_env = True self.cookies = cookiejar_from_dict({}) self.adapters = OrderedDict() self.mount('https://', adapter) self.mount('http://', adapter) def close(self) -> None: pass
And creating the connection factory like:
class HttpSessionFactory: def __init__(self, pool_max_size: int = DEFAULT_CONNECTION_POOL_MAX_SIZE, retry: Retry = DEFAULT_RETRY_POLICY): self.__http_adapter = HTTPAdapter(pool_maxsize=pool_max_size, max_retries=retry) def session(self) -> Session: return HttpSession(self.__http_adapter) def close(self): self.__http_adapter.close()
Finally, somewhere in the code I can write:
with self.__session_factory.session() as session: response = session.get(request_url)
And all my session instances will reuse the same connection pool.And somewhere at the end when the application stops I can close the HttpSessionFactory.Hope this will help somebody.