python selenium does not scale with firefox - Message: Can't load the profile. Profile Dir
Selenium is not failing while running the loop, it is failing at webdriver.Firefox(). The error is recognized when the connection times out at a certain time after line 231 when the script still cant recognize an open firefox application.
It is just a bug in selenium when running with a headless browser. To get a working version of selenium, run
sudo pip install -U selenium This should be a similar issue: Selenium: FirefoxProfile exception Can't load the profile