How to get the raw JSON response of a HTTP request from `driver.page_source` in Selenium webdriver Firefox
use the "view-source:" parameter in your url
Simple Mode:
example:
url = 'view-source:https://httpbin.org/headers'driver.get(url)content = driver.page_sourceprint(content)output:
'<html><head><meta name="viewport" content="width=device-width"><title>https://httpbin.org/headers</title><link rel="stylesheet" type="text/css" href="resource://content-accessible/viewsource.css"></head><body id="viewsource" class="highlight" style="-moz-tab-size: 4"><pre>{\n "headers": {\n "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", \n "Accept-Encoding": "gzip, deflate, br", \n "Accept-Language": "en-US,en;q=0.5", \n "Host": "httpbin.org", \n "Upgrade-Insecure-Requests": "1", \n "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0"\n }\n}\n</pre></body></html>'Best Mode: (for JSON)
example:
url = 'view-source:https://httpbin.org/headers'driver.get(url)content = driver.page_sourcecontent = driver.find_element_by_tag_name('pre').textparsed_json = json.loads(content)print(parsed_json)output:
{'headers': {'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'en-US,en;q=0.5', 'Host': 'httpbin.org', 'Upgrade-Insecure-Requests': '1', 'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0'}}
In addition to the raw JSON response, driver.page_source also contains the HTML to "pretty print" the response in the browser. You'll get the same result, if you use the Firefox DOM and Style Inspector to view the source of the JSON response in the browser.
To get the raw JSON response you can navigate the HTML elements as usual:
print(driver.find_element_by_xpath("//div[@id='json']").text)
This article helped me to solve the issue with firefox: https://blog.francium.tech/firefox-selenium-disable-json-formatting-cfaf466fd20f
I've added this preference to my driver factory:
from selenium.webdriver.firefox.options import Options as FirefoxOptions @staticmethod def get_firefox_options(headless): options = FirefoxOptions() options.set_preference('devtools.jsonview.enabled', False) if headless: options.headless = True return options