BS4 Scraping Hidden Content
Based on your url page, the <div id="salData" class="projectionsView">
is re-write by the javascript, but urllib.request
will get the whole response before running your callback, it means that the javascript generated content will be not in the response. Hence the div will be empty:
<div id="salData" class="projectionsView"> <!-- Fill in with Salary Data --></div>
you better try with selenium and splash will work for this kind of dynamic website.
BTW, after you get the right response, you select div
by id
, it will be more specific:
salarybox = page_soup.find("div",{"id":"salData"})