Accessing Google webscraped data

This may seem like a siily question - but is there any way to legally and programmatically access (even for fee) and use Google’s webscraped data for commercial purposes? I find Google’s explanations of their offerings maddening. They seem geared toward offering better search of a website to users coming to a website, or using their ad platform to make money, or using their ad platform to promote a business, or placing better in their search results, or monitoring mentions of a particular URL, or storage or fast processing, etc (among others).

I just want the same kind of data that their public search homepage or RSS Alerts serve up, but in a form that’s easier to work from and at scale. A use case might be something like asking Google once a day over 20 years to tell me what URLs in the past 24 hours were updated and have content in English including the words ‘perovskite’ and ‘mine’ but not ‘solar.’

But I want to do that for ultimately tens or hundreds of thousands of different words and phrases. Let me be clear that I do not want to store or transfer or serve up all the content on all those pages (I am not trying to compete with Google’s homepage, as if I could); I simply want to identify that a term of interest appeared on a given URL at a certain time in a certain context (e.g., the sentence containing each term). I could get pretty close with RSS Alerts if there wasn’t a cap on how many one user/user account could have. I would also like to be able to add different parameters to each search over time (e.g., to exclude content that contains specific proper names like ‘Jimmy Perovskite’ containing one of the search terms).

Any direction greatly, greatly appreciated.

Apologies if this is the wrong forum; feel free to point me elsewhere.

So you want access to the data that Google collects from private users’ searches? I think only the US government can request that. :thinking:

I think people in general want to stop Google from tracking their searches. So, why would they want a stranger to have access to the information without their knowledge? This opens up a whole host of privacy concerns (IMO).

If you have your own website you can set up Site Search to see how users interact with your site. If you use Google Analytics tho, you have to let the users be aware of it and how you use the data you collect. (There’s documentation on that)

Or, in another direction, you could look into something like IFTTT and set up an applet to track hashtags on Twitter or something like that.

Thanks for your response. Let me clarify that I am only interested in the content that are published on the web, not information about what people search for. Additionally, I am not really interested in social media content.

Do you mean more like this?

Thanks very much for pointing me to that. It seems to prove that there are at least some companies who build and maintain tools to scrape Google SERP data and charge others to use the tool. I am guessing that means that none of Google’s APIs offer the same functionality, even for a fee. I find it odd that Google offers APIs to work with map data and shopping data but not more unstructured data like text on webpages. JSON Custom Search API seems designed to feed display of content on websites and to limit the range of websites searched. I can’t make sense of what Google Programmable Search Engine offers.