Parse HTML and Read HTML Table with Selenium Python

Ok, after some hefty digging around I feel like I came to good solution--matching that of R. If you are looking at the HTML provided in the link above, Dog Breeds, and you have the web driver running for that link you can run the following code:

tbl = driver.find_element_by_xpath("//html/body/main/article/section[2]/div/article/table").get_attribute('outerHTML')

df = pd.read_html(tbl)

Then you are looking a pretty nice dataframe after only a couple lines of code:

In [145]: dfOut[145]:[ 0 1 2 3 0 BREED 2015 2014 2013.0 1 Retrievers (Labrador) 1 1 1.0 2 German Shepherd Dogs 2 2 2.0 3 Retrievers (Golden) 3 3 3.0 4 Bulldogs 4 4 5.0 5 Beagles 5 5 4.0

I feel like this is much easier than working through the tags, creating a dict, and looping through each row of data as the blog suggests. It might not be the most correct way of doing things, I'm new to Python, but it gets the job done quickly. I hope this helps out some fellow web-scrapers.

tbl = driver.find_element_by_xpath("//html/body/main/article/section[2]/div/article/table").get_attribute('outerHTML')df  = pd.read_html(tbl)

it Worked pretty well.

First, read Selenium with Python, you will get basic idea of how Selenium work with Python.

Than, if you want to locate element in Python, there are tow ways:

1. Use Selenium API, you can refer Locating Elements
2. Use BeautifulSoup, there is nice Document you can readBeautifulSoupDocumentation