How to get response SSL certificate from requests in python?
requests
deliberately wraps up low-level stuff like this. Normally, the only thing you want to do is to verify that the certs are valid. To do that, just pass verify=True
. If you want to use a non-standard cacert bundle, you can pass that too. For example:
resp = requests.get('https://example.com', verify=True, cert=['/path/to/my/ca.crt'])
Also, requests
is primarily a set of wrappers around other libraries, mostly urllib3
and the stdlib's http.client
(or, for 2.x, httplib
) and ssl
.
Sometimes, the answer is just to get at the lower-level objects (e.g., resp.raw
is the urllib3.response.HTTPResponse
), but in many cases that's impossible.
And this is one of those cases. The only objects that ever see the certs are an http.client.HTTPSConnection
(or a urllib3.connectionpool.VerifiedHTTPSConnection
, but that's just a subclass of the former) and an ssl.SSLSocket
, and neither of those exist anymore by the time the request returns. (As the name connectionpool
implies, the HTTPSConnection
object is stored in a pool, and may be reused as soon as it's done; the SSLSocket
is a member of the HTTPSConnection
.)
So, you need to patch things so you can copy the data up the chain. It may be as simple as this:
HTTPResponse = requests.packages.urllib3.response.HTTPResponseorig_HTTPResponse__init__ = HTTPResponse.__init__def new_HTTPResponse__init__(self, *args, **kwargs): orig_HTTPResponse__init__(self, *args, **kwargs) try: self.peercert = self._connection.sock.getpeercert() except AttributeError: passHTTPResponse.__init__ = new_HTTPResponse__init__HTTPAdapter = requests.adapters.HTTPAdapterorig_HTTPAdapter_build_response = HTTPAdapter.build_responsedef new_HTTPAdapter_build_response(self, request, resp): response = orig_HTTPAdapter_build_response(self, request, resp) try: response.peercert = resp.peercert except AttributeError: pass return responseHTTPAdapter.build_response = new_HTTPAdapter_build_response
That's untested, so no guarantees; you may need to patch more than that.
Also, subclassing and overriding would probably be cleaner than monkeypatching (especially since HTTPAdapter
was designed to be subclassed).
Or, even better, forking urllib3
and requests
, modifying your fork, and (if you think this is legitimately useful) submitting pull requests upstream.
Anyway, now, from your code, you can do this:
resp.peercert
This will give you a dict with 'subject'
and 'subjectAltName'
keys, as returned by pyopenssl.WrappedSocket.getpeercert
. If you instead want more information about the cert, try Christophe Vandeplas's variant of this answer that lets you get an OpenSSL.crypto.X509
object. If you want to get the entire peer certificate chain, see GoldenStake's answer.
Of course you may also want to pass along all the information necessary to verify the cert, but that's even easier, because it already passes through the top level.
To start, abarnert's answer is very complete. While chasing the proposed connection-close
issue of Kalkran I actually discovered that the peercert
didn't contain detailed information about the SSL Certificate.
I dug deeper in the connection and socket info and extracted the self.sock.connection.get_peer_certificate()
function which contains great functions like:
get_subject()
for CNget_notAfter()
andget_notBefore()
for expiration datesget_serial_number()
andget_signature_algorithm()
for crypto related technical details- ...
Note that these are only available if you have pyopenssl
installed on your system. Under the hood, urllib3
uses pyopenssl
if it's available and the standard library's ssl
module otherwise. The self.sock.connection
attribute shown below only exists if self.sock
is a urllib3.contrib.pyopenssl.WrappedSocket
, not if it's a ssl.SSLSocket
. You can install pyopenssl
with pip install pyopenssl
.
Once that's done, the code becomes:
import requestsHTTPResponse = requests.packages.urllib3.response.HTTPResponseorig_HTTPResponse__init__ = HTTPResponse.__init__def new_HTTPResponse__init__(self, *args, **kwargs): orig_HTTPResponse__init__(self, *args, **kwargs) try: self.peer_certificate = self._connection.peer_certificate except AttributeError: passHTTPResponse.__init__ = new_HTTPResponse__init__HTTPAdapter = requests.adapters.HTTPAdapterorig_HTTPAdapter_build_response = HTTPAdapter.build_responsedef new_HTTPAdapter_build_response(self, request, resp): response = orig_HTTPAdapter_build_response(self, request, resp) try: response.peer_certificate = resp.peer_certificate except AttributeError: pass return responseHTTPAdapter.build_response = new_HTTPAdapter_build_responseHTTPSConnection = requests.packages.urllib3.connection.HTTPSConnectionorig_HTTPSConnection_connect = HTTPSConnection.connectdef new_HTTPSConnection_connect(self): orig_HTTPSConnection_connect(self) try: self.peer_certificate = self.sock.connection.get_peer_certificate() except AttributeError: passHTTPSConnection.connect = new_HTTPSConnection_connect
You will be able to access the result easily:
r = requests.get('https://yourdomain.tld', timeout=0.1)print('Expires on: {}'.format(r.peer_certificate.get_notAfter()))print(dir(r.peer_certificate))
If, like me, you want to ignore SSL Certificate warnings just add the following in the top of the file and do not SSL verify:
from requests.packages.urllib3.exceptions import InsecureRequestWarningrequests.packages.urllib3.disable_warnings(InsecureRequestWarning)r = requests.get('https://yourdomain.tld', timeout=0.1, verify=False)print(dir(r.peer_certificate))
Thanks for everyone's awesome answers.
It helped me over engineer an answer to this question:
How to add a custom CA Root certificate to the CA Store used by Python in Windows?
UPDATE 2019-02-12
Please take a look at Cert Human: SSL Certificates for Humans for an impressive rewrite of my https://github.com/neozenith/get-ca-py project by lifehackjim.
I have archived the original repository now.
Stand alone snippet
#! /usr/bin/env python# -*- coding: utf-8 -*-"""Get Certificates from a request and dump them."""import argparseimport sysimport requestsfrom requests.packages.urllib3.exceptions import InsecureRequestWarningrequests.packages.urllib3.disable_warnings(InsecureRequestWarning)"""Inspired by the answers from this Stackoverflow question:https://stackoverflow.com/questions/16903528/how-to-get-response-ssl-certificate-from-requests-in-pythonWhat follows is a series of patching the low level libraries in requests.""""""https://stackoverflow.com/a/47931103/622276"""sock_requests = requests.packages.urllib3.contrib.pyopenssl.WrappedSocketdef new_getpeercertchain(self, *args, **kwargs): x509 = self.connection.get_peer_cert_chain() return x509sock_requests.getpeercertchain = new_getpeercertchain"""https://stackoverflow.com/a/16904808/622276"""HTTPResponse = requests.packages.urllib3.response.HTTPResponseorig_HTTPResponse__init__ = HTTPResponse.__init__def new_HTTPResponse__init__(self, *args, **kwargs): orig_HTTPResponse__init__(self, *args, **kwargs) try: self.peercertchain = self._connection.sock.getpeercertchain() except AttributeError: passHTTPResponse.__init__ = new_HTTPResponse__init__HTTPAdapter = requests.adapters.HTTPAdapterorig_HTTPAdapter_build_response = HTTPAdapter.build_responsedef new_HTTPAdapter_build_response(self, request, resp): response = orig_HTTPAdapter_build_response(self, request, resp) try: response.peercertchain = resp.peercertchain except AttributeError: pass return responseHTTPAdapter.build_response = new_HTTPAdapter_build_response"""Attempt to wrap in a somewhat usable CLI"""def cli(args): parser = argparse.ArgumentParser(description="Request any URL and dump the certificate chain") parser.add_argument("url", metavar="URL", type=str, nargs=1, help="Valid https URL to be handled by requests") verify_parser = parser.add_mutually_exclusive_group(required=False) verify_parser.add_argument("--verify", dest="verify", action="store_true", help="Explicitly set SSL verification") verify_parser.add_argument( "--no-verify", dest="verify", action="store_false", help="Explicitly disable SSL verification" ) parser.set_defaults(verify=True) return vars(parser.parse_args(args))def dump_pem(cert, outfile="ca-chain.crt"): """Use the CN to dump certificate to PEM format""" PyOpenSSL = requests.packages.urllib3.contrib.pyopenssl pem_data = PyOpenSSL.OpenSSL.crypto.dump_certificate(PyOpenSSL.OpenSSL.crypto.FILETYPE_PEM, cert) issuer = cert.get_issuer().get_components() print(pem_data.decode("utf-8")) with open(outfile, "a") as output: for part in issuer: output.write(part[0].decode("utf-8")) output.write("=") output.write(part[1].decode("utf-8")) output.write(",\t") output.write("\n") output.write(pem_data.decode("utf-8"))if __name__ == "__main__": cli_args = cli(sys.argv[1:]) url = cli_args["url"][0] req = requests.get(url, verify=cli_args["verify"]) for cert in req.peercertchain: dump_pem(cert)