Use Signed Cookies

When I use CloudFront with many private URLs, I prefer to use Signed Cookies when all the restrictions are met. This does not speed up the generation of signed cookies but it reduces the number of signing requests to be one per user until they expire.

Tuning RSA Signature Generation

I can imagine you may have requirements which render signed cookies as an invalid option. In that case I tried to speed up the signing by comparing the RSA module used with boto and cryptography. Two additional alternative options are m2crypto and pycrypto but for this example I will use cryptography.

In order to test performance of signing URLs with different modules I reduced the method _sign_string to remove any logic except the signing of a string then created a new Distribution class. Then I took the private key and example URL from boto tests to test with.

The results show that cryptography is quicker but still requires close to 1ms per signing request. These results are skewed higher by iPython's use of scoped variables in timing.

timeit -n10000 rsa_distribution.create_signed_url(url, message, expire_time)10000 loops, best of 3: 6.01 ms per looptimeit -n10000 cryptography_distribution.create_signed_url(url, message, expire_time)10000 loops, best of 3: 644 µs per loop

The full script:

from cryptography.hazmat.primitives.asymmetric import paddingfrom cryptography.hazmat.primitives import serializationfrom cryptography.hazmat.backends import default_backendfrom cryptography.hazmat.primitives import hashesimport rsafrom boto.cloudfront.distribution import Distributionfrom textwrap import dedent# The private key provided in the Boto testspk_key = dedent("""    -----BEGIN RSA PRIVATE KEY-----    MIICXQIBAAKBgQDA7ki9gI/lRygIoOjV1yymgx6FYFlzJ+z1ATMaLo57nL57AavW    hb68HYY8EA0GJU9xQdMVaHBogF3eiCWYXSUZCWM/+M5+ZcdQraRRScucmn6g4EvY    2K4W2pxbqH8vmUikPxir41EeBPLjMOzKvbzzQy9e/zzIQVREKSp/7y1mywIDAQAB    AoGABc7mp7XYHynuPZxChjWNJZIq+A73gm0ASDv6At7F8Vi9r0xUlQe/v0AQS3yc    N8QlyR4XMbzMLYk3yjxFDXo4ZKQtOGzLGteCU2srANiLv26/imXA8FVidZftTAtL    viWQZBVPTeYIA69ATUYPEq0a5u5wjGyUOij9OWyuy01mbPkCQQDluYoNpPOekQ0Z    WrPgJ5rxc8f6zG37ZVoDBiexqtVShIF5W3xYuWhW5kYb0hliYfkq15cS7t9m95h3    1QJf/xI/AkEA1v9l/WN1a1N3rOK4VGoCokx7kR2SyTMSbZgF9IWJNOugR/WZw7HT    njipO3c9dy1Ms9pUKwUF46d7049ck8HwdQJARgrSKuLWXMyBH+/l1Dx/I4tXuAJI    rlPyo+VmiOc7b5NzHptkSHEPfR9s1OK0VqjknclqCJ3Ig86OMEtEFBzjZQJBAKYz    470hcPkaGk7tKYAgP48FvxRsnzeooptURW5E+M+PQ2W9iDPPOX9739+Xi02hGEWF    B0IGbQoTRFdE4VVcPK0CQQCeS84lODlC0Y2BZv2JxW3Osv/WkUQ4dslfAQl1T303    7uwwr7XTroMv8dIFQIPreoPhRKmd/SbJzbiKfS/4QDhU    -----END RSA PRIVATE KEY-----""")# Initializing keys in a global contextcryptography_private_key = serialization.load_pem_private_key(    pk_key,    password=None,    backend=default_backend())# Instantiate a signer object using PKCS 1v 15, this is not recommended but required for Amazondef sign_with_cryptography(message):    signer = cryptography_private_key.signer(        padding.PKCS1v15(),        hashes.SHA1())    signer.update(message)    return signer.finalize()# Initializing the key in a global contextrsa_private_key = rsa.PrivateKey.load_pkcs1(pk_key)def sign_with_rsa(message):    signature = rsa.sign(str(message), rsa_private_key, 'SHA-1')    return signature# All this information comes from the Boto tests.url = "http://d604721fxaaqy9.cloudfront.net/horizon.jpg?large=yes&license=yes"expected_url = "http://d604721fxaaqy9.cloudfront.net/horizon.jpg?large=yes&license=yes&Expires=1258237200&Signature=Nql641NHEUkUaXQHZINK1FZ~SYeUSoBJMxjdgqrzIdzV2gyEXPDNv0pYdWJkflDKJ3xIu7lbwRpSkG98NBlgPi4ZJpRRnVX4kXAJK6tdNx6FucDB7OVqzcxkxHsGFd8VCG1BkC-Afh9~lOCMIYHIaiOB6~5jt9w2EOwi6sIIqrg_&Key-Pair-Id=PK123456789754"message = "PK123456789754"expire_time = 1258237200class CryptographyDistribution(Distribution):    def _sign_string(            self,            message,            private_key_file=None,            private_key_string=None):        return sign_with_cryptography(message)class RSADistribution(Distribution):    def _sign_string(            self,            message,            private_key_file=None,            private_key_string=None):        return sign_with_rsa(message)cryptography_distribution = CryptographyDistribution()rsa_distribution = RSADistribution()cryptography_url = cryptography_distribution.create_signed_url(    url,    message,    expire_time)rsa_url = rsa_distribution.create_signed_url(    url,    message,    expire_time)assert cryptography_url == rsa_url == expected_url, "URLs do not match"

Conclusion

Although the cryptography module performs better in this test, I recommend trying to find a way to utilize signed cookies but I hope this information is useful.

Briefly

Consider whether you can (in addition to using python-cryptography, per @erik-e) use a shorter key length (and probably change keys more frequently), given the particulars of your use case. While I can sign with the 2048-bit key AWS generated in ~1550µs, it only takes ~307µs at 1028 bits, ~184µs at 768 bits, and ~113µs at 512 bits.

Explanation

After looking into this for a bit, I'm going to go in another direction and build off of the (already great) answer @erik-e gave. I should mention before I get into it that I don't know how acceptable this idea is; I am just reporting on the performance impact it has (see the end of the post for a question I asked on the security SE seeking input on this).

I was collecting timings on signing with cryptography as @erik-e suggests, and because of the still large performance gulf between it and our existing signing method for S3, I decided to profile the code to see if it looked like there might be anything obvious chewing up time:

>>> cProfile.runctx('[sign_url_cloudfront2("...") for x in range(0,100)]', globals(), locals(), sort="time")         9403 function calls in 0.218 seconds   Ordered by: internal time   ncalls  tottime  percall  cumtime  percall filename:lineno(function)      200    0.161    0.001    0.161    0.001 {_Cryptography_cffi_a269d620xd5c405b7.EVP_PKEY_sign}      100    0.006    0.000    0.186    0.002 rsa.py:214(_finalize_pkey_ctx)     1200    0.004    0.000    0.008    0.000 {isinstance}      400    0.004    0.000    0.007    0.000 api.py:212(new)      100    0.003    0.000    0.218    0.002 views.py:888(sign_url_cloudfront2)      300    0.002    0.000    0.004    0.000 abc.py:128(__instancecheck__)      100    0.002    0.000    0.008    0.000 hashes.py:53(finalize)      200    0.002    0.000    0.005    0.000 gc_weakref.py:10(build)      100    0.002    0.000    0.007    0.000 hashes.py:15(__init__)      100    0.002    0.000    0.018    0.000 rsa.py:151(__init__)      100    0.002    0.000    0.014    0.000 hashes.py:68(__init__)      200    0.002    0.000    0.003    0.000 gc_weakref.py:14(remove)      200    0.002    0.000    0.003    0.000 api.py:239(cast)      100    0.002    0.000    0.190    0.002 rsa.py:207(finalize)      200    0.001    0.000    0.007    0.000 api.py:325(gc)      500    0.001    0.000    0.001    0.000 {getattr}      400    0.001    0.000    0.001    0.000 {_cffi_backend.newp}      400    0.001    0.000    0.001    0.000 api.py:150(_typeof)      200    0.001    0.000    0.002    0.000 api.py:266(buffer)      200    0.001    0.000    0.001    0.000 utils.py:18(<lambda>)      300    0.001    0.000    0.001    0.000 _weakrefset.py:68(__contains__)      200    0.001    0.000    0.001    0.000 {_cffi_backend.buffer}      100    0.001    0.000    0.002    0.000 hashes.py:49(update)      100    0.001    0.000    0.010    0.000 hashes.py:102(finalize)      100    0.001    0.000    0.003    0.000 hashes.py:88(update)      200    0.001    0.000    0.001    0.000 {method 'encode' of 'str' objects}      100    0.001    0.000    0.019    0.000 rsa.py:528(signer)      300    0.001    0.000    0.001    0.000 {len}      100    0.001    0.000    0.001    0.000 base64.py:42(b64encode)      100    0.001    0.000    0.008    0.000 backend.py:148(create_hash_ctx)      200    0.001    0.000    0.001    0.000 {_cffi_backend.cast}      200    0.001    0.000    0.001    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_get_digestbyname}      100    0.001    0.000    0.001    0.000 {method 'format' of 'str' objects}      100    0.001    0.000    0.003    0.000 rsa.py:204(update)      200    0.000    0.000    0.000    0.000 {method 'pop' of 'dict' objects}      100    0.000    0.000    0.000    0.000 {binascii.b2a_base64}      200    0.000    0.000    0.000    0.000 {_cffi_backend.typeof}      100    0.000    0.000    0.000    0.000 {time.time}      100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_DigestUpdate}        1    0.000    0.000    0.218    0.218 <string>:1(<module>)      100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_DigestInit_ex}      100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_PKEY_CTX_new}      100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_PKEY_CTX_free}      100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_DigestFinal_ex}      100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_MD_CTX_create}      100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_PKEY_CTX_set_rsa_padding}      100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_PKEY_size}      100    0.000    0.000    0.000    0.000 {method 'translate' of 'str' objects}      100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_MD_CTX_cleanup}      100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_PKEY_CTX_set_signature_md}      100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_PKEY_sign_init}      100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_MD_CTX_destroy}        1    0.000    0.000    0.000    0.000 {range}        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

While there might be some small savings lurking inside signer, the vast majority of the time is spent inside of the finalize() call, and almost all of that time is spent inside the actual sign call to openssl. While this was a little disappointing, it was a clear indicator that I should look to the actual signing process for savings.

I was just using the 2048-bit key CloudFront generated for us, so I decided to see what impact a smaller key would have on performance. I re-ran the profile using the shorter key:

>>> cProfile.runctx('[sign_url_cloudfront2("...") for x in range(0,100)]', globals(), locals(), sort="time")        9203 function calls in 0.063 seconds  Ordered by: internal time  ncalls  tottime  percall  cumtime  percall filename:lineno(function)     100    0.008    0.000    0.008    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_PKEY_sign}     400    0.005    0.000    0.008    0.000 api.py:212(new)     100    0.004    0.000    0.033    0.000 rsa.py:214(_finalize_pkey_ctx)    1200    0.004    0.000    0.008    0.000 {isinstance}     100    0.003    0.000    0.063    0.001 views.py:897(sign_url_cloudfront2)     300    0.002    0.000    0.004    0.000 abc.py:128(__instancecheck__)     100    0.002    0.000    0.008    0.000 hashes.py:53(finalize)     200    0.002    0.000    0.005    0.000 gc_weakref.py:10(build)     100    0.002    0.000    0.007    0.000 hashes.py:15(__init__)     100    0.002    0.000    0.014    0.000 hashes.py:68(__init__)     100    0.002    0.000    0.018    0.000 rsa.py:151(__init__)     200    0.002    0.000    0.003    0.000 gc_weakref.py:14(remove)     100    0.001    0.000    0.036    0.000 rsa.py:207(finalize)     200    0.001    0.000    0.003    0.000 api.py:239(cast)     200    0.001    0.000    0.006    0.000 api.py:325(gc)     500    0.001    0.000    0.001    0.000 {getattr}     200    0.001    0.000    0.002    0.000 api.py:266(buffer)     400    0.001    0.000    0.001    0.000 {_cffi_backend.newp}     400    0.001    0.000    0.001    0.000 api.py:150(_typeof)     100    0.001    0.000    0.010    0.000 hashes.py:102(finalize)     200    0.001    0.000    0.002    0.000 utils.py:18(<lambda>)     300    0.001    0.000    0.001    0.000 _weakrefset.py:68(__contains__)     100    0.001    0.000    0.002    0.000 hashes.py:88(update)     100    0.001    0.000    0.001    0.000 hashes.py:49(update)     200    0.001    0.000    0.001    0.000 {method 'encode' of 'str' objects}     200    0.001    0.000    0.001    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_get_digestbyname}     100    0.001    0.000    0.001    0.000 base64.py:42(b64encode)     100    0.001    0.000    0.008    0.000 backend.py:148(create_hash_ctx)     100    0.001    0.000    0.019    0.000 rsa.py:520(signer)     200    0.001    0.000    0.001    0.000 {_cffi_backend.buffer}     200    0.001    0.000    0.001    0.000 {method 'pop' of 'dict' objects}     200    0.001    0.000    0.001    0.000 {_cffi_backend.cast}     100    0.001    0.000    0.001    0.000 {method 'format' of 'str' objects}     100    0.001    0.000    0.001    0.000 {time.time}     100    0.001    0.000    0.003    0.000 rsa.py:204(update)     200    0.000    0.000    0.000    0.000 {len}     200    0.000    0.000    0.000    0.000 {_cffi_backend.typeof}     100    0.000    0.000    0.000    0.000 {binascii.b2a_base64}     100    0.000    0.000    0.000    0.000 {method 'translate' of 'str' objects}       1    0.000    0.000    0.063    0.063 <string>:1(<module>)     100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_DigestUpdate}     100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_PKEY_CTX_new}     100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_DigestInit_ex}     100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_MD_CTX_destroy}     100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_DigestFinal_ex}     100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_MD_CTX_create}     100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_PKEY_sign_init}     100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_PKEY_size}     100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_MD_CTX_cleanup}     100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_PKEY_CTX_free}     100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_PKEY_CTX_set_signature_md}     100    0.000    0.000    0.000    0.000 {_Cryptography_cffi_a269d620xd5c405b7.EVP_PKEY_CTX_set_rsa_padding}       1    0.000    0.000    0.000    0.000 {range}       1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

As mentioned in my comment on erik-e's answer, the runtime I saw for our full signing method using the 2048-bit key with the cryptography module was ~1550µs. Repeating this same test with the 512-bit key brings the runtime down to about ~113µs (a stone's-throw from the ~30µs of our S3 signing method).

This result seems meaningful, but it hinges on how acceptable it is to use a shorter key for your purpose. I was able to find a comment from March on a Mozilla issue report suggesting a 512-bit key could be broken for $75 in 8 hours on EC2.