Phantom JS driver is unable to locate elements sometimes

So... it looks like any element that uses Selenium WebDriver Wait is actually using a polling scheme. Remember that a Selenium explicit wait is code you define to wait for a certain condition to occur before proceeding further in the code. WebDriverWait by default calls the ExpectedCondition every 500 milliseconds until it returns successfully. (reference.) What that means is every 500 mSecs, Selenium tests the wait condition. If true, proceed. If not yet true, wait for another poll frequency cycle, then try again.

And from my testing, my belief is that a poll test and not-ready-yet so call it a fail definitely generates an errors in my ghostdriver.log

[ERROR - 2016-08-14T08:50:12.896Z] WebElementLocator - _handleLocateCommand - Element(s) NOT Found: GAVE UP. Search Stop Time: 1471164612878

I'm working on a project that scrapes a complex single page AJAX site. Because the site re-uses common div elements, I have to make a whole lot of calls (clear, paste value, click, wait, go to a different table on page, copy data of interest, repeat...) to obtain the data that I need. And because the site is sort of slow, I apply wait elements via:

driver.wait.until(EC.presence_of_element_located((By.XPATH, "//*[@id='special_content_id']//td[contains(.,'" + info.unitId + "')]")))

It turns out that polling frequency is set in the class selenium.webdriver.support.wait.WebDriverWait(driver, timeout, poll_frequency=0.5, ignored_exceptions=None)

Reference here.

I was curious about the behavior. When my poll_frequency was set to 0.5 seconds, I received 98 WebElementLocator errors. When I switched that to 7.5 seconds, driver.wait = WebDriverWait(driver, 60, 7.5), I received only 36 errors. Smaller poll times generate more errors.

What I really find odd is that I wouldn't expect a normal polling behavior (Selenium) to set an log error anywhere (PhantomJS). I do wish there was a different log entry for a polling, element-not-yet-ready...

I have noticed those elements are not being located in test cases using find_element methods by phantomJS driver, that can be accessed by javascript using JavascriptExecutor. Use java script to access the element and for the action on an element.I hope that will solve your problem.

It is possible that your selenium test tries to find element before it becomes loaded. After first unsuccessful attempt you should try again and again and go on until you either find the element or timeout exceeded.

Look at splinter https://splinter.readthedocs.org/en/latest/index.htmlIt works with selenium and implements logic of waiting for elements.