PolicyNews
OpenAI to honor shared robots.txt AI training signal
OpenAI will honor a common robots.txt signal that lets websites tell multiple AI providers at once whether their content may be used for model training.
By: Mikola Máté Patrik · · 2 min read
Mikola Máté Patrik
What was announced?
OpenAI announced on Friday that it will honor a common signal in robots.txt (OpenAI). It lets website operators state whether their content may be used to train foundational generative AI models.
The point is that one signal reaches every participating provider at once. Until now, operators had to identify individual crawlers and maintain separate rules for each provider.
When does it apply?
OpenAI aims to implement the signal in the coming months and will share details about its implementation when it does. The new signal will not replace existing robots.txt crawler controls, which continue to apply.
Where does it come from?
The signal builds on the work of the Internet Engineering Task Force's (IETF) AI Preferences working group. OpenAI announced it together with other participants in the working group and is encouraging other AI providers to honor it too.
Why it matters
Site operators may soon handle with one rule what used to take a separate entry per crawler. The scheme is voluntary, though: it only covers participating providers, and the exact syntax and rollout details are still to come.
Produced with the help of AI, reviewed by an editor. How we make our articles
Reviewed and edited by the Neural Daily desk. About
