Selenium Python Unable to scroll down, while fetching google reviews
Use Javascript to scroll to the last review, this will trigger additional review load.
last_review = self.driver.find_element_by_css_selector('div.gws-localreviews__google-review:last-of-type')self.driver.execute_script('arguments[0].scrollIntoView(true);', last_review)EDIT:
The following example is working correctly for me on Firefox and Chrome, you can reuse the extract google reviews function for your needs
import timefrom selenium import webdriverfrom selenium.common.exceptions import NoSuchElementExceptionfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.support import expected_conditions as ECfrom selenium.webdriver.support.wait import WebDriverWaitdef extract_google_reviews(driver, query): driver.get('https://www.google.com/?hl=en') driver.find_element_by_name('q').send_keys(query) WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.NAME, 'btnK'))).click() reviews_header = driver.find_element_by_css_selector('div.kp-header') reviews_link = reviews_header.find_element_by_partial_link_text('Google reviews') number_of_reviews = int(reviews_link.text.split()[0]) reviews_link.click() all_reviews = WebDriverWait(driver, 3).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'div.gws-localreviews__google-review'))) while len(all_reviews) < number_of_reviews: driver.execute_script('arguments[0].scrollIntoView(true);', all_reviews[-1]) WebDriverWait(driver, 5, 0.25).until_not(EC.presence_of_element_located((By.CSS_SELECTOR, 'div[class$="activityIndicator"]'))) all_reviews = driver.find_elements_by_css_selector('div.gws-localreviews__google-review') reviews = [] for review in all_reviews: try: full_text_element = review.find_element_by_css_selector('span.review-full-text') except NoSuchElementException: full_text_element = review.find_element_by_css_selector('span[class^="r-"]') reviews.append(full_text_element.get_attribute('textContent')) return reviewsif __name__ == '__main__': try: driver = webdriver.Firefox() reviews = extract_google_reviews(driver, 'STANLEY BRIDGE CYCLES AND SPORTS LIMITED') finally: driver.quit() print(reviews)
lenOfPage = driver.execute_script('window.scrollTo(0, [hard code the height])')
For me I would hardcorde the height if I am using this automated test for this same page over and over again.
Or you can have it to continuously loop to scroll down the page until the element is found if any.
Alternatively, you can also get all of the reviews without the browser automation.
The only thing you need is the data_fid, which you can find in the page source of a place you searched for.
In this case that's: 0x48762038283b0bc3:0xc373b8d4227d0090
After that, you just have to make a request to: https://www.google.com/async/reviewDialog?hl=en&async=feature_id:0x48762038283b0bc3:0xc373b8d4227d0090,sort_by:,next_page_token:,associated_topic:,_fmt:pc
There you will find all the reviews data, as well as the next_page_token, so you can query the next 10 reviews.
In this case next_page_token is: EgIICg
So, the request URL for the next 10 reviews would be: https://www.google.com/async/reviewDialog?hl=en&async=feature_id:0x48762038283b0bc3:0xc373b8d4227d0090,sort_by:,next_page_token:EgIICg,associated_topic:,_fmt:pc
You could also use a third party solution like SerpApi. It's a paid API with a free trial. We handle proxies, solve captchas, and parse all rich structured data for you.
Example python code (available in other libraries also):
from serpapi import GoogleSearchparams = { "api_key": "secret_api_key", "engine": "google_maps_reviews", "hl": "en", "data_id": "0x48762038283b0bc3:0xc373b8d4227d0090",}search = GoogleSearch(params)results = search.get_dict()Example JSON output:
"place_info": { "title": "Stanley Bridge Cycles & Sports Ltd", "address": "Newnham Parade, 11 College Rd, Cheshunt, Waltham Cross, United Kingdom", "rating": 5, "reviews": 53},"reviews": [ { "user": { "name": "Armilson Correia", "link": "https://www.google.com/maps/contrib/102797076683495103766?hl=en-US&sa=X&ved=2ahUKEwja2tvQj-DxAhUHMVkFHcJuD_MQvvQBegQIARAh", "thumbnail": "https://lh3.googleusercontent.com/a-/AOh14GgCCH69E_qgfu3pa1xbTsyvH9ORn8PEonb5FcubKg=s40-c-c0x00000000-cc-rp-mo-ba3-br100", "local_guide": true, "reviews": 48, "photos": 9 }, "rating": 5, "date": "2 days ago", "snippet": "In my opinion The best bike shop In radios of 60 miles Very professional and excellent customer service My bike come out from there riding like a New ,no Words just perfect" }, { "user": { "name": "John Janes", "link": "https://www.google.com/maps/contrib/104286744244406721398?hl=en-US&sa=X&ved=2ahUKEwja2tvQj-DxAhUHMVkFHcJuD_MQvvQBegQIARAt", "thumbnail": "https://lh3.googleusercontent.com/a/AATXAJzRZRQx74RYqpNQArE0ER-d24iQ-3kAwK64-46u=s40-c-c0x00000000-cc-rp-mo-br100", "reviews": 2, "photos": 1 }, "rating": 5, "date": "a year ago", "snippet": "The guys recently built my new bike and the advice on components to use was invaluable. Even the wheels were built from scratch. A knowledgeable efficient team with great attention to detail. I wouldn't go anywhere else .", "likes": 1, "images": [ "https://lh5.googleusercontent.com/p/AF1QipMc5u1rIZ88w-cfeAeF2s6bSndHMhLw8YC_BllS=w100-h100-p-n-k-no" ] }, { "user": { "name": "James Wainwright", "link": "https://www.google.com/maps/contrib/116302076794615919905?hl=en-US&sa=X&ved=2ahUKEwja2tvQj-DxAhUHMVkFHcJuD_MQvvQBegQIARA6", "thumbnail": "https://lh3.googleusercontent.com/a/AATXAJwx8OTba1pQ9lrzxy7LU5BnrJYWu90METBaK68F=s40-c-c0x00000000-cc-rp-mo-br100", "reviews": 36, "photos": 7 }, "rating": 5, "date": "a month ago", "snippet": "Want to thank the guys for giving my bike the full service it needed .Its now like new again and I didn't realise how much had worn out.Recomend to anyone in the cheshunt area." }, ...]Check out the documentation for more details.
Disclaimer: I work at SerpApi.