Replace Selenium with Requests in dynamic form Replace Selenium with Requests in dynamic form selenium selenium

Replace Selenium with Requests in dynamic form


The short answer to your question is yes, you can use the requests library to make the post requests. For an example you can easily open the inspector on your browser and copy a request using the following site:

https://curl.trillworks.com/

Then you can feed the response.text into BeautifulSoup to parse out the tables you want.

When I do this with the site in your example I get the following:

import requestscookies = {    'ASPSESSIONIDCQTTBCRB': 'BFDPGLCCEJMKPFKGJJFHKHFC',}headers = {    'Connection': 'keep-alive',    'Pragma': 'no-cache',    'Cache-Control': 'no-cache',    'Origin': 'http://dgasatel.mop.cl',    'Upgrade-Insecure-Requests': '1',    'Content-Type': 'application/x-www-form-urlencoded',    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36',    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',    'Referer': 'http://dgasatel.mop.cl/filtro_paramxestac_new.asp',    'Accept-Encoding': 'gzip, deflate',    'Accept-Language': 'en-US,en;q=0.9',}data = {  'estacion1': '-1',  'estacion2': '-1',  'estacion3': '-1',  'accion': 'refresca',  'tipo': 'ANO',  'fecha_fin': '11/12/2018',  'hora_fin': '0',  'period': '1d',  'fecha_ini': '11/12/2018',  'fecha_finP': '11/12/2018',  'UserID': 'nobody',  'EsDL1': '0',  'EsDL2': '0',  'EsDL3': '0'}response = requests.post(    'http://dgasatel.mop.cl/filtro_paramxestac_new.asp',    headers=headers, cookies=cookies, data=data)

For cleaning up the data, I recommend you map the data points you are wanting onto a dictionary or into a csv with loops.

for table in data:    if table.tail(1) and table.tail(1).iloc:        print(table.tail(1).iloc[0][2])