Web scraping: Automating button click [closed]
First of all, yes - you can use PhantomJS ghostdriver with python. It is built-in to python-selenium:
pip install seleniumDemo:
>>> from selenium import webdriver>>> driver = webdriver.PhantomJS()>>> driver.get('https://stackoverflow.com/questions/27813251')>>> driver.titleu'javascript - Web scraping: Automating button click - Stack Overflow'There are also several other threads that provide examples of "scrapy+selenium" spiders:
Also there is a scrapy-webdriver module that can probably help with it too.
Using scrapy with selenium would give you a huge overhead and slow things down drammatically even with a headless PhantomJS browser.
There is a huge chance you can mimic that "show more offers" button click by simulating the underlying request going to get the data you need. Use browser developer tools to explore what kind of request is fired and use scrapy.http.Request for simulation inside the spider.