This may seem like a siily question - but is there any way to legally and programmatically access (even for fee) and use Google’s webscraped data for commercial purposes? I find Google’s explanations of their offerings maddening. They seem geared toward offering better search of a website to users coming to a website, or using their ad platform to make money, or using their ad platform to promote a business, or placing better in their search results, or monitoring mentions of a particular URL, or storage or fast processing, etc (among others).
I just want the same kind of data that their public search homepage or RSS Alerts serve up, but in a form that’s easier to work from and at scale. A use case might be something like asking Google once a day over 20 years to tell me what URLs in the past 24 hours were updated and have content in English including the words ‘perovskite’ and ‘mine’ but not ‘solar.’
But I want to do that for ultimately tens or hundreds of thousands of different words and phrases. Let me be clear that I do not want to store or transfer or serve up all the content on all those pages (I am not trying to compete with Google’s homepage, as if I could); I simply want to identify that a term of interest appeared on a given URL at a certain time in a certain context (e.g., the sentence containing each term). I could get pretty close with RSS Alerts if there wasn’t a cap on how many one user/user account could have. I would also like to be able to add different parameters to each search over time (e.g., to exclude content that contains specific proper names like ‘Jimmy Perovskite’ containing one of the search terms).
Any direction greatly, greatly appreciated.
Apologies if this is the wrong forum; feel free to point me elsewhere.