How to Mock an HTTP request in a unit testing scenario in Python
Starting a web server for unit testing is definitely not a good practice. Unit tests should be simple and isolated, which means that they should avoid performing IO operations for example.
If what you want to write are really unit tests then you should craft your own test inputs and also look into mock objects. Python being a dynamic language, mocking and monkey pathing are easy and powerful tools for writing unit test. In particular, have a look at the excellent Mock module.
Simple unit test
So, if we have a look at your CssTests example, you are trying to test that css.getCssUriList is able to extract all the CSS stylesheet referenced in a piece of HTML you give it. What you are doing in this particular unit test is not testing that you can send a request and get a response from a website, right? You simply want to make sure that given some HTML, your function returns the correct list of CSS URLs. So, in this test, you clearly do not need to talk to a real HTTP server.
I would do something like the following:
import unittestclass CssListTestCase(unittest.TestCase): def setUp(self): self.css = core.Css() def test_css_list_should_return_css_url_list_from_html(self): # Setup your test sample_html = """ <html> <head> <title>Some web page</title> <link rel='stylesheet' type='text/css' media='screen' href='http://example.com/styles/full_url_style.css' /> <link rel='stylesheet' type='text/css' media='screen' href='/styles/relative_url_style.css' /> </head> <body><div>This is a div</div></body> </html> """ base_url = "http://example.com/" # Exercise your System Under Test (SUT) css_urls = self.css.get_css_uri_list(sample_html, base_url) # Verify the output expected_urls = [ "http://example.com/styles/full_url_style.css", "http://example.com/styles/relative_url_style.css" ] self.assertListEqual(expected_urls, css_urls) Mocking with Dependency Injection
Now, something less obvious would be unit testing the getContent() method of your core.HttpRequests class. I suppose you are using an HTTP library and not making your own requests on top of TCP sockets.
To keep your tests at the unit level, you don't want to send anything over the wire. What you can do to avoid that, is having tests that ensure that you make use of your HTTP library correctly. This is about testing not the behaviour of your code but rather the way it interacts with the other objects around it.
One way to do so would be to make the dependency on that library explicit: we can add a parameter to the HttpRequests.__init__ to pass it an instance of library's HTTP client. Say I use an HTTP library that provides a HttpClient object on which we can call get(). You could do something like:
class HttpRequests(object): def __init__(self, http_client): self.http_client = http_client def get_content(self, url): # You could imagine doing more complicated stuff here, like checking the # response code, or wrapping your library exceptions or whatever return self.http_client.get(url)We have made the dependency explicit and the requirement now needs to be met by the caller of HttpRequests: this is called Dependency Injection (DI).
DI is very useful for two things:
- it avoids surprises where your code is secretely relying on some object to exist somewhere
- it allows test to be written that inject different sort of objects depending on the goal of that test
Here, we can use a mock object that we will give to core.HttpRequests and that it will use, unknowingly, as if it were the real library. After that, we can test that the interaction was conducted as expected.
import coreclass HttpRequestsTestCase(unittest.TestCase): def test_get_content_should_use_get_properly(self): # Setup url = "http://example.com" # We create an object that is not a real HttpClient but that will have # the same interface (see the `spec` argument). This mock object will # also have some nice methods and attributes to help us test how it was used. mock_http_client = Mock(spec=somehttplib.HttpClient) # Exercise http_requests = core.HttpRequests(mock_http_client) content = http_requests.get_content(url) # Here, the `http_client` attribute of `http_requests` is the mock object we # have passed it, so the method that is called is `mock.get()`, and the call # stops in the mock framework, without a real HTTP request being sent. # Verify # We expect our get_content method to have called our http library. # Let's check! mock_http_client.get.assert_called_with(url) # We can find out what our mock object has returned when get() was # called on it expected_content = mock_http_client.get.return_value # Since our get_content returns the same result without modification, # we should have received it self.assertEqual(content, expected_content)We have now tested that our get_content method interacts correctly with our HTTP library. We have defined the boundaries of our HttpRequests object and tested them, and this is as far as we should go at the unit test level. The request is now in the hand of that library and it is certainly not the role of our unit test suite to test that the library works as expected.
Monkey patching
Now imagine that we decide to use the great requests library. Its API being more procedural, it does not present an object we can grab to make HTTP requests from. Instead, we would import the module and call its get method.
Our HttpRequests class in core.py would then look somethings like the following:
import requestsclass HttpRequests(object): # No more DI in __init__ def get_content(self, url): # We simply delegate the HTTP work to the `requests` module return requests.get(url)No more DI, so now, we are left wondering:
- How am I to prevent network interaction from happening?
- How am I to test that I use the
requestsmodule properly?
This is where you can use another fantastic, yet controversial, mechanism that dynamic languages offer: monkey patching. We will replace, at runtime, the requests module with an object we craft and can use in our test.
Our unit test will then look something like:
import coreclass HttpRequestsTestCase(unittest.TestCase): def setUp(self): # We create a mock to replace the `requests` module self.mock_requests = Mock() # We keep a reference to the current, real, module self.old_requests = core.requests # We replace the module with our mock core.requests = self.mock_requests def tearDown(self): # It is very important that each unit test be isolated, so we need # to be good citizen and clean up after ourselves. This means that # we need to put back the correct `requests` module where it was core.requests = self.old_requests def test_get_content_should_use_get_properly(self): # Setup url = "http://example.com" # Exercise http_client = core.HttpRequests() content = http_client.get_content(url) # Verify # We expect our get_content method to have called our http library. # Let's check! self.mock_requests.get.assert_called_with(url) # We can find out what our mock object has returned when get() was # called on it expected_content = self.mock_requests.get.return_value # Since our get_content returns the same result without modification, # we should have received self.assertEqual(content, expected_content)To make this process less verbose, the mock module has a patch decorator that looks after the scaffolding. We then only need to write:
import coreclass HttpRequestsTestCase(unittest.TestCase): @patch("core.requests") def test_get_content_should_use_get_properly(self, mock_requests): # Notice the extra param in the test. This is the instance of `Mock` that the # decorator has substituted for us and it is populated automatically. ... # The param is now the object we need to make our assertions against expected_content = mock_requests.get.return_valueConclusion
It is very important to keep unit test small, simple, fast, and self-contained. A unit test that relies on another server to be running is simply not a unit test. To help with that, DI is a great practice, and mock objects a great tool.
At first, it is not easy to get your head around the concept of mock and how to use them though. Like every power tool, they can also explode in your hands and for example make you believe you have tested something when in reality you have not. Making sure that the behaviour and input/output of mock objects reflects the reality is paramount.
P.S.
Given that we have never interacted with a real HTTP server at the unit test level, it is important to write Integration Tests that will make sure our application is able to talk to the sort of servers it will deal with in real life. We could do this with a fully fledged server set up specially for Integration Testing, or write a contrived one.