I cannot seem to identify the correct elements of a website in order to scrape it

Get Help General Programming
1

Hello!

Im completely new to coding and I want to create a code that scrapes the titles of all publications of a website.
However I cannot seem to identify the correct elements of the titles of said website (https://www.bfs.admin.ch/bfs/de/home/statistiken/kataloge-datenbanken/publikationen.html)
I have used chatgtp to help me but even with that it doesn’t work.
Probably I just don’t understand this stuff at all and the solution is very simple but I just can’t figure it out!
Please help!

Here is my code

from bs4 import BeautifulSoup
import requests
import csv

page_to_scrape = requests.get (“https://www.bfs.admin.ch/bfs/de/home/statistiken/kataloge-datenbanken/publikationen.html”)
soup = BeautifulSoup(page_to_scrape.text, “html.parser”)
titles = [a_tag.text for a_tag in soup.find_all(“a”, class_=“publication-list__link”)]

data =

for title in titles:
item = {}
item[“text”] = title
data.append(item)

file = open (“scraped_Publikationen.csv”, “w”)
writer = csv.writer (file)
writer.writerow ([item[‘text’]])

file.close()

2

Did you happen to check out the CC course “Learn Web scraping with Beautiful Soup” ? Or, a Python course? Or, maybe YouTube prior to writing this code?

There is some knowledge to have before one can do this. I’m not saying that one needs to be Jedi Master in Python, but an understanding of lists, dictionaries, loops, etc will help here. Essentially, an understanding of Python & its libraries. It will also help greatly if you understand the elements of a webpage and the document object model (DOM) and how to traverse them & access elements.
ChatGPT isn’t a learning tool. Having a ML model tell you how to do something isn’t learning core concepts and applying them.

1 Like
3

I had been watching Youtube videos on this specific topic, tutorials I mean.

but yeah I had been trying to figure it out through yt resources and then when it didn’t work I tried Chat gtp, maybe it would be to help ig. Ur right though I guess I will have to learn it the traditional way.

Thank you very much for the quick reply :smile:, its greatly appreciated. I will check out that CC course and if it doesn’t help ill look up a beginners python course.

1 Like