httplib.BadStatusLine: ''

Based on Python Doc, httplib.BadStatusLine raised if a server responds with a HTTP status code that we don’t understand.You can try to pass this exception. You should not close your driver if you are going to call more than one url.

Try this:

def parse(self, response):    try:        print response.status        print '???????????????????????????????????'        if response.status == 200:            self.driver.implicitly_wait(5)            self.driver.get(response.url)            print response.url            print '!!!!!!!!!!!!!!!!!!!!'            # DO STUFF    except httplib.BadStatusLine:        pass

I made a decorator to do what the top answer does, so as to make the code easily reusable. Here it is:

import httpdef pass_bad_status_line_exc(wrapped_function):    """    Silently pass this exception `http.client.BadStatusLine` decorator    """    def _wrapper(*args, **kwargs):        try:            result = wrapped_function(*args, **kwargs)        except http.client.BadStatusLine:            return        return result    return _wrapper

I hit this error because I defined a selenium.webdriver instance (named driver), called driver.quit() on it, then tried to call driver.get(url) on the quit driver. The solution is to not call driver.quit().