Selenium Webdriver vs Mechanize
These are completely different tools that somewhat "cross" in the web-scraping, web automation, automated data extraction scope.
mechanize
is a mature and widely-used tool for programmatic web-browsing with a lot of built-in features, like cookie handing, browser history, form submissions. The key thing to understand here is that mechanize.Browser
is not a real browser, it cannot execute and understand javascript, it cannot send asynchronous requests often needed to form a web page.
This is where selenium
comes into play - it is a browser automation tool which is also widely used in web-scraping. selenium
usually becomes a "fall-back" tool - when someone cannot web-scrape a site with mechanize
or RoboBrowser
or MechanicalSoup
(note - another alternatives) because of, for instance, it's javascript "heaviness", the choice is usually selenium
. With selenium
you can also go headless, automating PhantomJS
browser, or having a virtual display. As a commonly mentioned drawback, performance is often mentioned - with selenium
you are working with a target site as a real user in a web browser, which is loading additional files needed to form a page, making XHR requests, rendering etc.
And this itself does not mean you should use selenium
everywhere - choose the tool wisely, choose it because it fits the problem better, not because you are more familiar with an instrument.
Also note that you should, first, consider using an API (if provided by the target website) instead of going down to web-scraping. And, if it comes to it, be a good web-scraping citizen: