Data scraping with Beautifulsoup from HTML

I attempted to scrape data from HTML. First, I imported BeautifulSoup, then sent a request to get the content of the website as follow:

from bs4 import BeautifulSoup
soup = BeautifulSoup(webpage_response.content)
print (soup)

At first glance I thought i got it, but then realized that it said “Request unsuccessful” . Does it mean that I did not get the data?

Did you import requests library?

You have to create a webpage_response object and use requests.get(whatever the site is goes here)
Then something like webpage = webpage_response.content
When you create the soup object, in addition to using the webpage you have to also use html.parser to parse the data.

2 Likes

Yes, I did. Thanks for your reply.

I figured out the error. The data contained JavaScript. So, it cannot be processed by BeautifulSoup.

1 Like