Browser performance tests through selenium

There is a possibility to get closer to what browser-perf is doing by collecting the chrome performance logs and analyzing them.

To get performance logs, turn on performance logs by tweaking loggingPrefs desired capability:

from selenium import webdriverfrom selenium.webdriver.common.desired_capabilities import DesiredCapabilitiescaps = DesiredCapabilities.CHROMEcaps['loggingPrefs'] = {'performance': 'ALL'}driver = webdriver.Chrome(desired_capabilities=caps)driver.get('https://stackoverflow.com')logs = [json.loads(log['message'])['message'] for log in driver.get_log('performance')]with open('devtools.json', 'wb') as f:    json.dump(logs, f)driver.close()

At this point, devtools.json file would contain a bunch of trace records:

[  {    "params": {      "timestamp": 1419571233.19293,      "frameId": "16639.1",      "requestId": "16639.1",      "loaderId": "16639.2",      "type": "Document",      "response": {        "mimeType": "text/plain",        "status": 200,        "fromServiceWorker": false,        "encodedDataLength": -1,        "headers": {          "Access-Control-Allow-Origin": "*",          "Content-Type": "text/plain;charset=US-ASCII"        },        "url": "data:,",        "statusText": "OK",        "connectionId": 0,        "connectionReused": false,        "fromDiskCache": false      }    },    "method": "Network.responseReceived"  },  {    "params": {      "timestamp": 1419571233.19294,      "encodedDataLength": 0,      "requestId": "16639.1"    },    "method": "Network.loadingFinished"  },  ..]

Now, the question is, what to do with it.

One option that was initially suggested during the Google Test Automation Conference is to submit the logs to webpagetest.org. There is an example in java available here, but, at the moment, I had no luck implementing it in Python.

In theory, the UI report generated by webpagetest.org would look like this:

They also provide the metrics in JSON/XML and other formats that can be further analyzed.

This is really something, thanks to Vivek Singh for the pointing comment.

browser-perf also uses the logging functionality to pick up the tracing logs, and analyzes the data.

It is possible to do performance regression testing with Selenium. However as you might have already noted. The core essence of Selenium is that it mimics user behavior. This means that Selenium will only perform the action (e.g. clicking on the button) if the user is able to perform the same action. Also taking into account certain code, workarounds (i.e. hard waits, various checks and custom code), required to even be able to run the Selenium script. This means that the "definition" of performance testing using Selenium will be slightly different compared to traditional performance testing.

What you will want to do is have a timer (start/stop) for each action Selenium is performing. For example: Clicking on a button and log this to a file for later use.

Using Selenium you can create a performance baseline and from there on onwards compare each consecutive result with the baseline. This will give you statistics that you can then use for further analysis.

Selenium nor Webdriver (Selenium 2.0) come with this feature out of the box. So some custom coding needs to happen for this to work.

Performance testing through Selenium is not advised because it is not optimised for the job. It has been listed as one of the worst practices by the Selenium team:

It may seem ideal to performance test in the context of the user but a suite of WebDriver tests are subjected to many points of external and internal fragility which are beyond your control; for example browser startup speed, speed of HTTP servers, response of third party servers that host JavaScript or CSS, and the instrumentation penalty of the WebDriver implementation itself. Variation at these points will cause variation in your results. It is difficult to separate the difference between the performance of your website and the performance of external resources, and it is also hard to tell what the performance penalty is for using WebDriver in the browser, especially if you are injecting scripts.