Scrapy - Disable Selenium after first request
You could modify your middleware so that it only uses Selenium when the request comes with a render_js meta key.
Something like this:
class SeleniumMiddleware(object): ... def process_request(self, request, spider): if not request.meta.get('render_js'): # disable js rendering in a per-request basis return # render with selenium ...This works because when a downloader middleware's process_request returns None, the request will proceed to the next middlewares in the chain, eventually hitting Scrapy's downloader.
More info here: https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#scrapy.downloadermiddlewares.DownloaderMiddleware.process_request