How Can I Scrape Twitter Now That They Require JavaScript? How Can I Scrape Twitter Now That They Require JavaScript? curl curl

How Can I Scrape Twitter Now That They Require JavaScript?


You can not "Enable" JavaScript on curl. It is not a browser, it only does HTTP requests. Have you considered using the Twitter API?

You can also intercept XHRs on twitter using your browser's development tools and work your way through them to figure out what HTTP request you need to make in order to get the data you want.

Another solution is to use an scriptable "headless" browser. check out CapsperJS. Simply put it is a fully functional browser that does not show any UI and you can control it via JS.


There are many free endpoints available that can help solve this, rather than having to scrape the webpage. If you're looking for specific Tweets, try the new v2. Search API: https://developer.twitter.com/en/docs/twitter-api/tweets/search/introduction

You just need to have an approved developer account.


You need to render page with javascript, to get final html. You can use phantomjs (headless browser) for that purpose. Here is php plugin: http://jonnnnyw.github.io/php-phantomjs/