Extract text between two different tags beautiful soup

All the paragraphs that you want are located inside the <div class="td-post-content"> tag along with the paragraphs for the author information. But, the required <p> tags are direct children of this <div> tag, while the other not required <p> tags are not direct children (they are nested inside other div tags).

So, you can use recursive=False to access those tags only.

Code:

import requestsfrom bs4 import BeautifulSoupheaders = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36'}r = requests.get('https://www.the-blockchain.com/2018/06/29/mcafee-labs-report-6x-increase-in-crypto-mining-malware-incidents-in-q1-2018/', headers=headers)soup = BeautifulSoup(r.text, 'lxml')container = soup.find('div', class_='td-post-content')for para in container.find_all('p', recursive=False):    print(para.text)

Output:

Cybersecurity giant McAfee released its McAfee Labs Threat Report: June 2018 on Wednesday, outlining the growth and trends of new malware and cyber threats in Q1 2018. According to the report, coin mining malware saw a 623 percent growth in the first quarter of 2018, infecting 2.9 million machines in that period. McAfee Labs counted 313 publicly disclosed security incidents in the first three months of 2018, a 41 percent increase over the previous quarter. In particular, incidents in the healthcare sector rose 57 percent, with a significant portion involving Bitcoin-based ransomware that healthcare institutions were often compelled to pay.Chief Scientist at McAfee Raj Samani said, “There were new revelations this quarter concerning complex nation-state cyber-attack campaigns targeting users and enterprise systems worldwide. Bad actors demonstrated a remarkable level of technical agility and innovation in tools and tactics. Criminals continued to adopt cryptocurrency mining to easily monetize their criminal activity.”Sizeable criminal organizations are responsible for many of the attacks in recent months. In January, malware dubbed Golden Dragon attacked organizations putting together the Pyeongchang Winter Olympics in South Korea, using a malicious word attachment to install a script that would encrypt and send stolen data to an attacker’s command center. The Lazarus cybercrime ring launched a highly sophisticated Bitcoin phishing campaign called HaoBao that targeted global financial organizations, sending an email attachment that would scan for Bitcoin activity, credentials and mining data.Chief Technology Officer at McAfee Steve Grobman said, “Cybercriminals will gravitate to criminal activity that maximizes their profit. In recent quarters we have seen a shift to ransomware from data-theft,  as ransomware is a more efficient crime. With the rise in value of cryptocurrencies, the market forces are driving criminals to crypto-jacking and the theft of cryptocurrency. Cybercrime is a business, and market forces will continue to shape where adversaries focus their efforts.”

you need to use selenium, because i try to do it with requests and it don't work because data is load with javascript and follow by bs4

import requests, bs4from selenium import webdriverdriver = webdriver.Chrome('/usr/local/bin/chromedriver') website = "https://www.the-blockchain.com/2018/06/29/mcafee-labs-report-6x-increase-in-crypto-mining-malware-incidents-in-q1-2018/"driver.get(website) html = driver.page_sourcesoup = bs4.BeautifulSoup(html, "html.parser")elements = soup.select('#wpautbox_latest-post > ul > li')for elem in elements:    print(elem.text)

Output

McAfee Labs Report 6x Increase in Crypto Mining Malware Incidents in Q1 2018 - June 29, 2018Facebook Updates Policy To Allow Vetted Crypto Businesses to Advertise, ICOs Still Banned - June 27, 2018Following in Vitalik’s Footsteps? Polkadot’s Habermeier Awarded Thiel Fellowship - June 26, 2018And many other article titles

If you want to kick out About the author along with the stuffs not within the paragraph, you can do it by printing the content of span tag under p tag within class td-post-content. To make it concise, I'm using selector in this case. Try out the below approach as well.

import requestsfrom bs4 import BeautifulSoupurl = 'https://www.the-blockchain.com/2018/06/29/mcafee-labs-report-6x-increase-in-crypto-mining-malware-incidents-in-q1-2018/'res = requests.get(url,headers={"User-Agent":"defined"})soup = BeautifulSoup(res.text, 'lxml')paragraph = [p.text for p in soup.select('.td-post-content p span')]print(paragraph)