Submitting to a web form using python
If you want to pass q
as a parameter in the URL using requests
, use the params
argument, not data
(see Passing Parameters In URLs):
r = requests.get('http://stackoverflow.com', params=data)
This will request https://stackoverflow.com/?q=%5Bpython%5D , which isn't what you are looking for.
You really want to POST
to a form. Try this:
r = requests.post('https://stackoverflow.com/search', data=data)
This is essentially the same as GET
-ting https://stackoverflow.com/questions/tagged/python , but I think you'll get the idea from this.
import urllibimport urllib2url = 'http://www.someserver.com/cgi-bin/register.cgi'values = {'name' : 'Michael Foord', 'location' : 'Northampton', 'language' : 'Python' }data = urllib.urlencode(values)req = urllib2.Request(url, data)response = urllib2.urlopen(req) the_page = response.read()
This makes a POST request with the data specified in the values. we need urllib to encode the url and then urllib2 to send a request.
Mechanize library from python is also great allowing you to even submit forms. You can use the following code to create a browser object and create requests.
import mechanize,rebr = mechanize.Browser()br.set_handle_robots(False) # ignore robotsbr.set_handle_refresh(False) # can sometimes hang without thisbr.addheaders = [('User-agent', 'Firefox')] br.open( "http://google.com" )br.select_form( 'f' )br.form[ 'q' ] = 'foo'br.submit()resp = Nonefor link in br.links(): siteMatch = re.compile( 'www.foofighters.com' ).search( link.url ) if siteMatch: resp = br.follow_link( link ) breakcontent = resp.get_data()print content