Need to dump entire DOM tree with element id from selenium server

The Problem

Ok, so there may be cases where you need to perform some substantial processing of a page on the client (Python) side rather than on the server (browser) side. For instance, if you have some sort of machine learning system already written in Python and it needs to analyze the whole page before performing actions on them, then although it is possible to do it with a bunch of find_element calls, this gets very expensive because each call is a round-trip between the client and the server. And rewriting it to work in the browser may be too expensive.

Why Selenium's Identifiers wont' do it

However, I do not see an efficient way to get a serialization of the DOM together with Selenium's own identifiers. Selenium creates these identifiers on an as-needed basis, when you call find_element or when DOM nodes are returned from an execute_script call (or passed to the callback that execute_async_script gives to the script). But if you call find_element to get identifiers for each element, then you are back to square one. I could imagine decorating the DOM in the browser with the required information but there is no public API to request some sort of pre-assignment of WebElement ids. As a matter of fact, these identifiers are designed to be opaque so even if a solution managed somehow to get the required information, I'd be concerned about cross-browser viability and ongoing support.

A Solution

There is however a way to get an addressing system that would work on both sides: XPath. The idea is to parse the DOM serialization into a tree on the client side and then get the XPath of the nodes you are interested in and use this to get the corresponding WebElement. So if you'd have to perform dozens of client-server roundtrips to determine which single element you need to perform a click on, you'd be able so reduce this to an initial query of the page source plus a single find_element call with the XPath you need.

Here is a super simple proof of concept. It fetches the main input field of the Google front page.

from StringIO import StringIOfrom selenium import webdriverimport lxml.etree## Make sure that your chromedriver is in your PATH, and use the following line...#driver = webdriver.Chrome()## ... or, you can put the path inside the call like this:# driver = webdriver.Chrome("/path/to/chromedriver")#parser = lxml.etree.HTMLParser()driver.get("http://google.com")# We get this element only for the sake of illustration, for the tests later.input_from_find = driver.find_element_by_id("gbqfq")input_from_find.send_keys("foo")html = driver.execute_script("return document.documentElement.outerHTML")tree = lxml.etree.parse(StringIO(html), parser)# Find our element in the tree.field = tree.find("//*[@id='gbqfq']")# Get the XPath that will uniquely select it.path = tree.getpath(field)# Use the XPath to get the element from the browser.input_from_xpath = driver.find_element_by_xpath(path)print "Equal?", input_from_xpath == input_from_find# In JavaScript we would not call ``getAttribute`` but Selenium treats# a query on the ``value`` attribute as special, so this works.print "Value:", input_from_xpath.get_attribute("value")driver.quit()

Notes:

1. The code above does not use driver.page_source because Selenium's documentation states that there is no guarantee as to the freshness of what it returns. It could be the state of the current DOM or the state of the DOM when the page was first loaded.

2. This solution suffers from the exact same problems that find_element suffers from regarding dynamic contents. If the DOM changes while the analysis is occurring, then you are working on a stale representation of the DOM.

3. If you have to generate JavaScript events while performing the analysis, and these events change the DOM, then you'd need fetch the DOM again. (This is similar to the previous point but a solution that uses find_element calls could conceivably avoid the problem I'm talking about in this point by ordering the sequence of calls carefully.)

4. lxml's tree could possibly differ structurally from the DOM tree in such a way that the XPath obtained from lxml does not address the corresponding element in the DOM. What lxml processes is the cleaned up serialized view that the browser has of the HTML passed to it. Therefore, so long as the code is written to prevent the problems I've mentioned in point 2 and 3, I do not see this as a likely scenario, but it is not impossible.

Try:

find_elements_by_xpath("//*")

That should match all elements in the document.

UPDATE (to match question refinements):

Use javascript and return the DOM as a string:

execute_script("return document.documentElement.outerHTML")

See my other answer for the issues regarding any attempts at getting Selenium's identifiers.

Again, the problem is to reduce a bunch of find_element calls so as to avoid the round-trips associated with them.

A different method from my other answer is to use execute_script to perform the search on the browser and then return all the elements needed. For instance, this code would require three round-trips but can be reduced to just one round-trip:

el, parent, text = driver.execute_script("""var el = document.querySelector(arguments[0]);return [el, el.parentNode, el.textContent];""", selector)

This returns an element, the element's parent and the element's textual contents on the basis of whatever CSS selector I wish to pass. In a case where the page has jQuery loaded, I could use jQuery to perform the search. And the logic can get as complicated as needed.

This method takes care of the vast majority of cases where reducing round-trips is desirable but it does not take care of a scenario like the one I've given in illustration in my other answer.