How to load all entries in an infinite scroll at once to parse the HTML in python

This you won't be able to do with requests and BeautifulSoup as the page that you want to extract the information from loads the rest of the entries through JS when you scroll down. You can do this using selenium which opens a real browser and you can pass page down key press events programmatically. Watch this video to see the action. http://www.youtube.com/watch?v=g54xYVMojos

http://www.tidbitsofprogramming.com/2014/02/crawling-website-that-loads-content.html

Below is the script that extracts all the 100 post titles using selenium.

import timefrom selenium import webdriverfrom selenium.webdriver.common.keys import Keysbrowser = webdriver.Chrome()browser.get("https://medium.com/top-100/december-2013")time.sleep(1)elem = browser.find_element_by_tag_name("body")no_of_pagedowns = 20while no_of_pagedowns:    elem.send_keys(Keys.PAGE_DOWN)    time.sleep(0.2)    no_of_pagedowns-=1post_elems = browser.find_elements_by_class_name("post-item-title")for post in post_elems:    print post.text

Output:

When Your Mother Says She’s FatWhen “Life Hacking” Is Really White PrivilegeAs tendências culturais dos anos 2000 adiantadas pelo É o Tchan na década de 90Coming Out as BiracialComo ganhar discussões com seus parentes de direita neste NatalHow to save local bookstores in two easy stepsWelcome to DinovemberHow to Piss Off Your BaristaThe boy whose brain could unlock autismCrossFit’s Dirty Little SecretWelcome to MediumHere’s How the Military Wasted Your Money in 2013Why I Wear Nail PolishThe day of High School I’ll never forget7 Reasons Buffalonians Shouldn’t Hate SnowDear Guy Who Just Made My Burrito:Is the Mona Lisa Priceless?Please stop live tweeting people’s private conversationsYour Friends and RapistsEight things you can live withoutThe Value of Content40 Ways To Make Life Simple AgainManila-Beijing-Washington:Things I Wish Someone Had Told Me When I Was Learning How to CodeDear Ticketmaster,Steve Jobs Danced To My Song11 Things I Wish I Knew When I Started My BusinessBullish: Benevolent Sexism and “That Guy” Who Makes Everything AwkwardAdvice to a College Music StudentSilver Gyninen joutui sotaanImagining the Post-Antibiotics FutureWhich side are you on?Put it away, junior. Casual PredationThe sad little iPhone commercialHow Node.js is Going to Replace JavaScriptWhy you should have your heart broken into a million little pieces. How to Write Emails Like a CEODesigning Products That ScaleHow radioactive poison became the assassin’s weapon of choiceWhy do people hate CrossFit?We (Still) Need Feminism10 Advanced Hearthstone Arena TipsLet It Full-BleedWhat Medium Is ForHow a Small Force of Finnish Ski Troops Fought Off a Massive Soviet ArmyAn Introvert’s Guide to Better PresentationsMandela The TerroristWhy You Should have a Messy DeskWhy I’m Not a TEDx SpeakerFonts have feelings tooYou Don’t Want Your Thanksgiving to Go Like ThisWhat I’ve Learned in My First Month as a VCWhy Quantity Should be Your PriorityMy Airbnb storyI Wanna Date You Like An AnimalThe GIF Guide to Getting PaidHow We Discovered the Underground Chinese App MarketFirst Images of a Heart Injected with Liquid Metal Beyonce Broke the Music Business“View mode” approach to responsive web designSometimes You Will Forget Your Mom Has CancerDarkness Ray Beams Invisibility From A DistanceWhy Work As We Know It May Be ImmoralStaying Ahead of the CurveThe Geekiest Game Ever Made Has Been Released In Germany The Dirty Secret Behind the Salesforce $1M HackathonI’m a really good impostorMathematical Model of Zombie Epidemics Reveals Two Types of Living-Dead InfectionsThe Heartbreak Kid200 ThingsI’m Not Racist But—Duel of the Superbattleships23 and YouThe Seattle NOI’m a vaccine refuser. There, I said it. The Year We Broke EverythingHow to make a DIY home alarm system with a raspberry pi and a webcamStrike While the App is HotHow to Fall In (and Out) of Love:Why did Google make an ad for promoting “Search” in India where it has over 97% market share?A Holiday Message From JesusRevealed: The Soviet Union’s $1 Billion ‘Psychotronic’ Arms Race with the USPostmortem of a Venture-backed StartupThe 1.x Crore MythThe “Getting Shit Done” Sleep Cycle Is the F-35 Joint Strike Fighter the New F-4?Can the F-35 Win a Dogfight?Responsive PhotosetsFightball: Millennials vs BoomersThe iconicity of “peaceful resistance”How We Make ChocolateFive Ships of the Chinese Navy You Really Ought to Know AboutGlassholes and Black Rock CityBad News for U.S. Warplane Pilots: Russia’s New Dogfighting Missile Can’t MissHow Antisec Died10 ways you’ll probably f**k up your startupUPDATED: Finding the unjustly homeless, and teaching them to code.Technology hasn’t Changed Us.What I’ve learned from fatherhood 

You can try with this :

from selenium import webdriverfrom bs4 import BeautifulSoupfrom selenium.webdriver.support.ui import WebDriverWaitpause = 10driver = webdriver.PhantomJS(executable_path='phantomjs.exe')driver.get("your_url")#This code will scroll down to the endwhile True:     try:        # Action scroll down        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")     break except:      pass

Alternatively, you could try the answer given by Andrew Che in this question. This is a 3-year-old question, and the page is not there anymore, but if you try something similar with a scrolling results page (e.g. I searched for 'top'), you could find out the endpoint you can call to get the different pages in the results list, which in this case was 'https://medium.com/search/posts?q=top&page=1'.