Robots.txt

Robots.txt news and guides from Robotica Guide: how crawler rules, AI training opt-outs and new policy tokens work at Google, Microsoft, OpenAI and Anthropic, with dates and limits checked against official docs.