Scraping a dynamic ecommerce page with infinite scroll Scraping a dynamic ecommerce page with infinite scroll r r

Scraping a dynamic ecommerce page with infinite scroll


As @nrussell suggested, you can use RSelenium to programatically scroll down the page before getting the source code.

You could for example do:

library(RSelenium)library(rvest)#start RSeleniumcheckForServer()startServer()remDr <- remoteDriver()remDr$open()#navigate to your pageremDr$navigate("http://www.linio.com.co/tecnologia/celulares-telefonia-gps/")#scroll down 5 times, waiting for the page to load at each timefor(i in 1:5){      remDr$executeScript(paste("scroll(0,",i*10000,");"))Sys.sleep(3)    }#get the page htmlpage_source<-remDr$getPageSource()#parse ithtml(page_source[[1]]) %>% html_nodes(".product-itm-price-new") %>%  html_text()


library(rvest)url<-"https://www.linio.com.co/c/celulares-y-tablets?page=1"page<-html_session(url)html_nodes(page,css=".price-secondary") %>% html_text()

Loop through the website https://www.linio.com.co/c/celulares-y-tablets?page=2 and 3 and so on and it will be easy for you to scrape the data

EDIT dated 07/05/2019

The website elements changed. Hence new code

library(rvest)url<-"https://www.linio.com.co/c/celulares-y-tablets?page=1"page<-html_session(url)html_nodes(page,css=".price-main") %>% html_text()