How can I search for a HTML element using Python? Python Web Bot Error

To get all header text Induce WebDriverWait() and wait for visibility_of_all_elements_located() and following css selector.

driver.get("https://www.theverge.com/")headerelements=WebDriverWait(driver,20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"div.c-entry-box--compact__body>h2>a")))for head in headerelements:    print(head.text)

Console Output:

BRYDGE’S LATEST KEYBOARDS TURN A SURFACE PRO OR GO INTO A STANDARD LAPTOPIkea gives its 2021 catalog an Animal Crossing-themed makeover in TaiwanSchool nurses are on the frontlines of the pandemicAN INNOCENT TYPO LED TO A GIANT 212-STORY OBELISK IN MICROSOFT FLIGHT SIMULATORThe epic campaign to win Elon Musk’s Tesla factory with memesNASA is going to try to hunt down a leak on the International Space Station this weekendWhat’s the best student laptop? We asked studentsGoodbye to Patriot Act, a comedy show that was a different kind of angryHow to pick the right headphones for kidsSwipe left, Elon stans: that Tesla dating app is a joke, for nowLeaked Google Pixel 5 renders show dual rear camera and fingerprint sensorMinecraft Education is perfectly suited for this surreal back-to-school momentWhat we listen to while working from homeSamsung’s Galaxy S20 is receiving Note 20 features with new One UI updateFacebook’s old web design will disappear in SeptemberApple reportedly using cheaper iPhone battery parts to offset 5G costTHE VERGE’S BACK TO SCHOOL SPECIALEpic to host a #FreeFortnite tournament with anti-Apple prizesAfter inking a deal with Netflix, Trump impersonator Sarah Cooper is also getting a TV showMagic Leap’s lost work The Last Light gets a surprise release after its developers were laid offAndroid 11 phones will summon Android Auto wirelessly, no need to pull out your deviceHOW FORTNITE’S EPIC BATTLE WITH APPLE COULD RESHAPE THE ANTITRUST FIGHTAdobe accidentally deleted people’s photos in latest Lightroom updateMajor news publishers ask Apple what can get them an App Store deal like Amazon’sTesla is working on a sensor that can detect a child left behind in a hot carFertility app Premom reportedly shared customer data with Chinese companiesMark Zuckerberg testified before the FTC as part of its Facebook antitrust probeHow to get Microsoft’s xCloud and stream Xbox games on your phone right nowWhere to sit on the school bus just got a lot more complicatedFormer Uber security chief charged with paying hush money to cover up 2016 hackGoogle confirms Android 11 will limit third-party camera apps because of location spying fearsUber and Lyft shutdown in California averted as judge grants emergency stayNetflix is re-creating iconic Stranger Things sets in LA, and you can drive your car through themGoogle’s Pixel Buds are now available in more colors nearly four months after launchAirbnb puts global ban on house parties to support social distancing guidelinesHOUSES ARE INFLUENCERS NOW, AND THIS ONE BURNED TO THE GROUNDLyft will suspend its ride-hailing service in CaliforniaReddit reports 18 percent reduction in hateful content after banning nearly 7,000 subredditsA mail-in COVID-19 test company switched to FedEx because of USPS delaysSteve Bannon charged with fraud over crowdfunded border wallRazer gets into the ergonomic game with its new $99.99 Pro Click wireless mouseSAMSUNG GALAXY NOTE 20 ULTRA REVIEW: BIG PHONE, SMALL UPDATESGoogle’s Pixel Buds get new transcribe mode, attention alerts, and sharing detectionControl’s publisher explains why it won’t offer a free next-gen upgradeSpaceX still pressing ahead with its Air Force lawsuit, despite winning coveted Air Force contractWe're building great things, and we need your talent.DoorDash launches grocery delivery to compete with Amazon and Instacart

For your script there is problemarticles = main.find_element_by_class_name("c-entry-box--compact__title")

find_element_by_class_name() will return single webelement. To get list of elements you need to use find_elements_by_class_name()

Therefor it should be

articles = main.find_elements_by_class_name("c-entry-box--compact__title")

However I would suggest use my approach which is very linear.