Scrape web pages in real time with Node.js

Node.io seems to take the cake :-)

All aforementioned solutions presume running the scraper locally. This means you will be severely limited in performance (due to running them in sequence or in a limited set of threads). A better approach, imho, is to rely on an existing, albeit commercial, scraping grid.

Here is an example:

var bobik = new Bobik("YOUR_AUTH_TOKEN");bobik.scrape({  urls: ['amazon.com', 'zynga.com', 'http://finance.google.com/', 'http://shopping.yahoo.com'],  queries:  ["//th", "//img/@src", "return document.title", "return $('script').length", "#logo", ".logo"]}, function (scraped_data) {  if (!scraped_data) {    console.log("Data is unavailable");    return;  }  var scraped_urls = Object.keys(scraped_data);  for (var url in scraped_urls)    console.log("Results from " + url + ": " + scraped_data[scraped_urls[url]]);});

Here, scraping is performed remotely and a callback is issued to your code only when results are ready (there is also an option to collect results as they become available).

You can download Bobik client proxy SDK at https://github.com/emirkin/bobik_javascript_sdk

I've been doing research myself, and https://npmjs.org/package/wscraper boasts itself as a

a web scraper agent based on cheerio.js a fast, flexible, and lean implementation of core jQuery; built on top of request.js; inspired by http-agent.js

Very low usage (according to npmjs.org) but worth a look for any interested parties.