Scrapy with selenium, webdriver failing to instantiate

The reason for this behavior is how the PhantomJS driver's Service class is implemented.

There is a __del__ method defined that calls self.stop() method:

def __del__(self):    # subprocess.Popen doesn't send signal on __del__;    # we have to try to stop the launched process.    self.stop()

And, self.stop() is assuming the service instance is still alive trying to access it's attributes:

def stop(self):    """    Cleans up the process    """    if self._log:        self._log.close()        self._log = None    #If its dead dont worry    if self.process is None:        return    ...

The same exact problem is perfectly described in this thread:

• Python attributeError on __del__

What you should do is to silently ignore AttributeError occurring while quitting the driver instance:

try:    driver.quit()except AttributeError:    pass

The problem was introduced by this revision. Which means that downgrading to 2.40.0 would also help.

I had that problem because phantomjs was not available from script (was not in path).You can check it by running phantomjs in console.

Selenium version 2.44.0 on pypi needs the following patch in Service.__init__ of selenium.webdriver.common.phantomjs.service

self.process = None

I was thinking of submitting a patch but this already exists in the most recent version on google code.