Mechanize

Mechanize is ruby library which imititates the behaviour of a web browser. You can click links, fill out forms und submit them. It even has a history and remebers cookies. It seems your problem could be easily solved with the help of mechanize.

The following example is taken from http://docs.seattlerb.org/mechanize/EXAMPLES_rdoc.html:

require 'rubygems'require 'mechanize'a = Mechanize.newa.get('http://rubyforge.org/') do |page|  # Click the login link  login_page = a.click(page.link_with(:text => /Log In/))  # Submit the login form  my_page = login_page.form_with(:action => '/account/login.php') do |f|    f.form_loginname  = ARGV[0]    f.form_pw         = ARGV[1]  end.click_button  my_page.links.each do |link|    text = link.text.strip    next unless text.length > 0    puts text  endend

You can try use wget to fetch the page. You can analyse login process with this app www.portswigger.net/proxy/.

For what it's worth, you could check out Webrat. It is meant to be used a tool for automated acceptance tests, but I think you could use it to simulate filling out the login fields, then click through links by their names, and grab the needed HTML as a string. Haven't tried doing anything like it, tho.