Get HTML table body in Python using Selenium Get HTML table body in Python using Selenium selenium selenium

Get HTML table body in Python using Selenium


What is happening here is that the table loads through JS after the page loads. You have to wait until the table loads. To do that, you'll have to use any one of the Waits specified here.

I'll recommend using Explicit Wait. You can do this:

First, you'll need to add the following imports.

from selenium.webdriver.support.ui import WebDriverWaitfrom selenium.webdriver.support import expected_conditions as ECfrom selenium.webdriver.common.by import Byfrom selenium.common.exceptions import TimeoutException

Then change

main_table = driver.find_elements_by_tag_name('table')outer_table = main_table[3].find_element_by_tag_name('table')print outer_table.get_attribute('innerHTML') 

to

try:    WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, 'gvContractors')))except TimeoutException:    pass  # Handle the exception heretable = driver.find_element_by_id('gvContractors').get_attribute('innerHTML')print(table)

It'll give you the required output. I'm not posting the output here since it is too large, but you can verify it by doing this

print('Company/Address' in table)

which prints True

Note:
Instead of finding the tables one by one using _by_tag_name you can directly use _by_id to find the table you want. (Here the table has id="gvContractors")