RSelenium: Scraping a dynamically loaded page that loads slowly

The website provides an api which should be the first port of call. Failing this you can access individual pages using for example:

http://www.codewars.com/kata?page=21

If you want to scroll to the bottom of the page until there is no more content with RSelenium you can use the "Loading..." element it has a class=js-infinite-marker. While we still have this element on the page we attempt to scroll down to it every second (with some error catching for any issues). If the element is not present we assume all content is loaded:

library(RSelenium)rD <- rsDriver(port = 4444L, browser = 'chrome')remDr <- rD$client # You dont need to use the open method remDr$navigate('http://www.codewars.com/kata')chk <- FALSEwhile(!chk){  webElem <- remDr$findElements("css", ".js-infinite-marker")  if(length(webElem) > 0L){    tryCatch(      remDr$executeScript("elem = arguments[0];                       elem.scrollIntoView();                         return true;", list(webElem[[1]])),       error = function(e){}    )    Sys.sleep(1L)  }else{    chk <- TRUE  }}