Scraping a dynamic ecommerce page with infinite scroll
As @nrussell suggested, you can use RSelenium
to programatically scroll down the page before getting the source code.
You could for example do:
library(RSelenium)library(rvest)#start RSeleniumcheckForServer()startServer()remDr <- remoteDriver()remDr$open()#navigate to your pageremDr$navigate("http://www.linio.com.co/tecnologia/celulares-telefonia-gps/")#scroll down 5 times, waiting for the page to load at each timefor(i in 1:5){ remDr$executeScript(paste("scroll(0,",i*10000,");"))Sys.sleep(3) }#get the page htmlpage_source<-remDr$getPageSource()#parse ithtml(page_source[[1]]) %>% html_nodes(".product-itm-price-new") %>% html_text()
library(rvest)url<-"https://www.linio.com.co/c/celulares-y-tablets?page=1"page<-html_session(url)html_nodes(page,css=".price-secondary") %>% html_text()
Loop through the website https://www.linio.com.co/c/celulares-y-tablets?page=2
and 3 and so on and it will be easy for you to scrape the data
EDIT dated 07/05/2019
The website elements changed. Hence new code
library(rvest)url<-"https://www.linio.com.co/c/celulares-y-tablets?page=1"page<-html_session(url)html_nodes(page,css=".price-main") %>% html_text()