Multi-threading in selenium python Multi-threading in selenium python selenium selenium

Multi-threading in selenium python


My understanding is that Selenium drivers are not thread-safe. In the WebDriver spec, the Thread Safety section is empty...which I take to mean they have not addressed the topic at all. https://www.w3.org/TR/2012/WD-webdriver-20120710/#thread-safety

So while you could share the driver reference with multiple threads and make calls to the driver from multiple threads, there is no guarantee that the driver will be able to handle multiple asynchronous calls correctly.

Instead, you must either synchronize calls from multiple threads to ensure one is completed before the next starts, or you should have just one thread making Selenium API calls...potentially handling commands from a queue that is filled by multiple other threads.

Also, see Can Selenium use multi threading in one browser?


I you are using the script to automatically submit forms (simply said doing GET and POST requests), I would recommend you to look at requests. You can easily capture Post requests from your Browser (Network tab in Developer Pane on both Firefox and Chrome), and submit them. Something like:

session = requests.session()response = session.get('https://stackoverflow.com/')soup = BeautifulSoup(response.text)

and even POST data like:

postdata = {'username':'John','password':password}response=session.post('example.com',data=postdata,allow_redirects=True)

It can be easily threaded, Multiple times faster than using selenium, the only problem is there is no JavaScript or Form support, so you need to do it the old fashioned way.

EDIT:Also take a look at ThreadPoolExecutor