BS4 Scraping Hidden Content BS4 Scraping Hidden Content selenium selenium

BS4 Scraping Hidden Content


Based on your url page, the <div id="salData" class="projectionsView">is re-write by the javascript, but urllib.request will get the whole response before running your callback, it means that the javascript generated content will be not in the response. Hence the div will be empty:

<div id="salData" class="projectionsView">    <!-- Fill in with Salary Data --></div>

you better try with selenium and splash will work for this kind of dynamic website.

BTW, after you get the right response, you select div by id, it will be more specific:

salarybox = page_soup.find("div",{"id":"salData"})