Python Scraping JavaScript using Selenium and Beautiful Soup
There are some mistakes in your code that are fixed below. However, the class "postText" must exist elsewhere, since it is not defined in the original source code.My revised version of your code was tested and is working on multiple websites.
from selenium import webdriver from selenium.common.exceptions import NoSuchElementException from selenium.webdriver.common.keys import Keys from bs4 import BeautifulSoupbrowser = webdriver.Firefox() browser.get('http://techcrunch.com/2012/05/15/facebook-lightbox/') html_source = browser.page_source browser.quit()soup = BeautifulSoup(html_source,'html.parser') #class "postText" is not defined in the source codecomments = soup.findAll('div',{'class':'postText'}) print comments