Comments
You must log in or register to comment.
robots.txt is a suggestion, and using it to block malicious actors is pointless.
Unfortunately, yeah. AI industry has proven that it’s willing to beg for forgiveness rather than asking for permission.
Scrapers are not new. They’ve been doing this shit for years, the result of the scraping is just more transparent to the general internet public now.
They’ve never asked permission.
