The world’s top two AI startups are ignoring requests by media publishers to stop scraping their web content for free model training data, Business Insider has learned.
OpenAI and Anthropic have been found to be either ignoring or circumventing an established web rule, called robots.txt, that prevents automated scraping of websites.
TollBit, a startup aiming to broker paid licensing deals between publishers and AI companies, found several AI companies are acting in this way and informed certain large publishers in a Friday letter, which was reported earlier by Reuters. The letter did not include the names of any of the AI companies accused of skirting the rule.
OpenAI and Anthropic have stated publicly that they respect robots.txt and blocks to their specific web crawlers, GPTBot and ClaudeBot.
However, according to TollBit’s findings, such blocks are not being respected, as claimed. AI companies, including OpenAI and Anthropic, are simply choosing to “bypass” robots.txt in order to retrieve or scrape all of the content from a given website or page.
A spokeswoman for OpenAI declined to comment beyond pointing BI to a corporate blogpost from May, in which the company says it takes web crawler permissions “into account each time we train a new model.” A spokesperson for Anthropic did not respond to emails seeking comment.
Robots.txt is a single bit of code that’s been used since the late 1990s as a way for websites to tell bot crawlers they don’t want their data scraped and collected. It was widely accepted as one of the unofficial rules supporting the web.
If you put it online, you gave it away. If someone reads it and then uses that information to answer questions for someone else without giving credit to the author, that’s called a conversation. As long as no copyrights are being abused, there is no problem and this is just corporations upset with what they think is piracy, pandering to people who are still on the fence about AI.
Except we shouldn’t be giving corporations same rights as individuals. Doing so leads to corporate feudalism.
So it’s okay for me to pirate something but not a corporation?
OpenAI has a clause that one cannot train their own AI on OpenAI chatbots
If it was all a giant open source project I’m sure many would be more accommodating to your argument.
So if it’s open source, it’s okay?
You got it backwards. According to OpenAI and Microsoft you have to respect their copyright but they can ignore yours.
Also no you can’t pirate but they can.
Any questions?
Id say yes to that statement, but for reasons that dont have to do with AI as I dont really view AI training as piracy.