FAQ: Web Scraping with Beautiful Soup - Reading Text

This community-built FAQ covers the “Reading Text” exercise from the lesson “Web Scraping with Beautiful Soup”.

Paths and Courses
This exercise can be found in the following Codecademy content:

FAQs on the exercise Reading Text

There are currently no frequently asked questions associated with this exercise – that’s where you come in! You can contribute to this section by offering your own questions, answers, or clarifications on this exercise. Ask or answer a question by clicking reply (reply) below.

If you’ve had an “aha” moment about the concepts, formatting, syntax, or anything else with this exercise, consider sharing those insights! Teaching others and answering their questions is one of the best ways to learn and stay sharp.

Join the Discussion. Help a fellow learner on their journey.

Ask or answer a question about this exercise by clicking reply (reply) below!

Agree with a comment or answer? Like (like) to up-vote the contribution!

Need broader help or resources? Head here.

Looking for motivation to keep learning? Join our wider discussions.

Learn more about how to use this guide.

Found a bug? Report it!

Have a question about your account or billing? Reach out to our customer support team!

None of the above? Find out where to ask other questions here!

I want to ask, what’s wrong in my variant of solve:

import requests
from bs4 import BeautifulSoup

prefix = "https://s3.amazonaws.com/codecademy-content/courses/beautifulsoup/"
webpage_response = requests.get('https://s3.amazonaws.com/codecademy-content/courses/beautifulsoup/shellter.html')

webpage = webpage_response.content
soup = BeautifulSoup(webpage, "html.parser")

turtle_links = soup.find_all("a")
links = []
# go through all of the a tags and get the links associated with them:
for a in turtle_links:
    links.append(prefix + a["href"])

# Define turtle_data:
turtle_data = {}

# follow each link:
for link in links:
    webpage = requests.get(link)
    turtle = BeautifulSoup(webpage.content, "html.parser")
    turtle_name = turtle.select(".name")[0].get_text()
    turtle_data[turtle_name] = turtle.find("ul").get_text("|").split('|')

It’s give the same result as a variant from site

{'Aesop': ['\n', 'AGE: 7 Years Old', '\n', 'WEIGHT: 6 lbs', '\n', 'SEX: Female', '\n', 'BREED: African Aquatic Sideneck Turtle', '\n', 'SOURCE: found in Lake Erie', '\n'], 'Caesar': ['\n', 'AGE: 2 Years Old', '\n', 'WEIGHT: 4 lbs', '\n', 'SEX: Male', '\n', 'BREED: Greek Tortoise', '\n', 'SOURCE: hatched in house', '\n'], 'Sulla': ['\n', 'AGE: 1 Year Old', '\n', 'WEIGHT: 1 lb', '\n', 'SEX: Male', '\n', 'BREED: African Aquatic Sideneck Turtle', '\n', 'SOURCE: found in Lake Erie', '\n'], 'Spyro': ['\n', 'AGE: 6 Years Old', '\n', 'WEIGHT: 3 lbs', '\n', 'SEX: Female', '\n', 'BREED: Greek Tortoise', '\n', 'SOURCE: hatched in house', '\n'], 'Zelda': ['\n', 'AGE: 3 Years Old', '\n', 'WEIGHT: 2 lbs', '\n', 'SEX: Female', '\n', 'BREED: Eastern Box Turtle', '\n', 'SOURCE: surrendered by owner', '\n'], 'Bandicoot': ['\n', 'AGE: 2 Years Old', '\n', 'WEIGHT: 2 lbs', '\n', 'SEX: Male', '\n', 'BREED: African Aquatic Sideneck Turtle', '\n', 'SOURCE: hatched in house', '\n'], 'Hal': ['\n', 'AGE: 1 Year Old', '\n', 'WEIGHT: 1.5 lbs', '\n', 'SEX: Female', '\n', 'BREED: Eastern Box Turtle', '\n', 'SOURCE: surrendered by owner', '\n'], 'Mock': ['\n', 'AGE: 10 Years Old', '\n', 'WEIGHT: 10 lbs', '\n', 'SEX: Male', '\n', 'BREED: Greek Tortoise', '\n', 'SOURCE: surrendered by owner', '\n'], 'Sparrow': ['\n', 'AGE: 1.5 Years Old', '\n', 'WEIGHT: 4.5 lbs', '\n', 'SEX: Female', '\n', 'BREED: African Aquatic Sideneck Turtle', '\n', 'SOURCE: found in Lake Erie', '\n']}
1 Like

Is there something wrong when the site check codes? I copied the solution to the reset excersise, I still could not pass question 2.

7 Likes

Same here. Am I missing something ? I’ve also tried turtle.ul and turtle.find_all(“ul”)

3 Likes

Have the same like a previous guys.

Also have this problem. I was stuck on question two, but it gives syntax error even if you paste in the given solution.

Given solution does not work for part 2.

This is the first lesson that I completely feel lost and had to look at the solution. I don’t understand why .split was in the solution but not in the lesson. I have been doing the data science track, am I severely missing HTML knowledge to complete these?

7 Likes

I agree. Seems web scraping course could have been much more comprehensive. This seem like an overview instead of teaching one how to do it. Maybe in the grand scheme of things of what data scientists actually do it is not that important? I’m not sure.

3 Likes

Hello, same problem here. Have you succesfuly been to question 3?

I think there might be something off with the code checker since copy/pasting the solution to the editor does not solve the problem. I recommend just unlocking exercise by looking at the solution.

Anyway the final list still has line breaks “\n” in it. To remove them I simply used list comprehension like this:

#follow each link:
for link in links:
webpage = requests.get(link)
turtle = BeautifulSoup(webpage.content, “html.parser”)
turtle_name = turtle.select(".name")[0].get_text()

stats = turtle.find(“ul”)
stats_text = stats.get_text("|")
turtle_data[turtle_name] = [info for info in stats_text.split("|") if info != “\n”]
print(turtle_data[turtle_name])

2 Likes

there is some problem in this exercise. the perfect code would not even pass. Better get the solution from the codecademy and move on.

1 Like

This material assumes a basic familiarity with HTML, which if I did not incidentally have I would feel even more lost. Also the course has not gone into too much detail on working with dictionaries.

On the whole there isn’t so much continuity between this chapter and what came before it which at least in my case breaks the learning flow and that is a bit of a shame, since BeautifulSoup clearly is a powerful tool. The course creators should work it better into the course to foster the same level of confidence as with the Pandas, Matplotlib and stats sections, which are quite good.

2 Likes

turtle_links = soup.find_all(“a”)

links =

#go through all of the a tags and get the links associated with them"

for a in turtle_links:

links.append(prefix+a[“href”])

From the above code, I wonder why do we need to .append(prefix + a[‘href’]) instead of .append(a)?

I confused. Could anyone help me out? thank you!

a is an Anchor Tag through which we paste link in HTML. Then we also need to specify a name for it. This command is concatenating all those names with prefix and then appending it.

Try printing out the links list and see it.

Another way I cleaned the data was to use the existing line break as entry separators (instead of adding ‘|’ as a separator).
The last piece was to use list comprehension to filter out the two empty strings per turtle info (which correspond to the ‘/n’).

for link in links:
  webpage = requests.get(link)
  turtle = BeautifulSoup(webpage.content, "html.parser")
  turtle_name = turtle.select(".name")[0].get_text()
  turtle_info = turtle.find('ul').get_text()
  turtle_data[turtle_name] = [info for info in turtle_info.split('\n') if info]

This gives you a clean list for each turtle key in the dictionary:

{
'Aesop': ['AGE: 7 Years Old', 'WEIGHT: 6 lbs', 'SEX: Female', 'BREED: African Aquatic Sideneck Turtle', 'SOURCE: found in Lake Erie'], 
'Caesar': ['AGE: 2 Years Old', 'WEIGHT: 4 lbs', 'SEX: Male', 'BREED: Greek Tortoise', 'SOURCE: hatched in house'],
...
}

Hello. I want to ask about a line of code from the solution:

turtle_name = turtle.select(“.name”)[0].get_text()

Why do we need to get the index [0]? I try to remove it and the code just doesn’t work.

I would like to add to this. Seeing the [0] implies to me that we’re using the indices of the turtle variable. I added a print("Turtle Name: " + turtle_name) below this to see each name printed out. Okay that makes sense.

Now I tried adding:
turtle_age = turtle.select(".age")[2].get_text()
print("Turtle Age: " + str(turtle_age))

I would normally assume that this would grab the third index of the same data, or at least it is in the dataframe we create at the end. In the dataframe itself, the names don’t appear under the ‘0’ column, yet the 0 still seems to be pointing to the first index. I’ve tried it with other indices with and without get_text and I don’t see how to do it.

Yea I agree this whole section was pretty terrible and confusing.

Yea they should have really added an explanation there because it is not intuitive at all. Pretty terrible section honestly.