--- title: Google and OpenAI back one robots.txt rule for AI training canonical: https://thebeat.dev/issues/2026-W44/ published: 2026-11-01 updated: 2026-11-04 covers_from: 2026-10-26 period: weekly covers: 2026-W44 week: 2026-W44 tags: - docs - distribution - pricing collection: issues site: https://thebeat.dev/ license: https://creativecommons.org/licenses/by/4.0/ --- # Google and OpenAI back one robots.txt rule for AI training > Google and OpenAI say they'll support one new robots.txt rule for AI training. A site can then give one answer for its docs and another for the rest. We checked 25 devtools docs sites: none block OpenAI or Google, and 14 already say yes in a format no lab has promised to read. ## What happened on Friday On Friday, [Google](https://developers.google.com/search/blog/2026/10/ai-training-control) and [OpenAI](https://openai.com/index/helping-websites-express-ai-training-preferences/) said they'll support a new rule in robots.txt. It lets a website tell AI labs whether they may train their models on its pages. Someone at Anthropic wrote the rule, and [Microsoft says](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/) Bing will follow it too. So that's Google, OpenAI and Microsoft behind one rule. But it's still [only a proposal](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/), and it may never become an official internet standard. Why should a devtools company care? Because training is how a model learns your API. A model that learned your API in training can write code against it before a developer searches for anything. Until now, you had to give each lab its own answer. From next year, one line covers every lab that supports the rule. There's a twist, too: your hosting provider may already be writing part of this file for you. We'll get to that below. ### How the rule works First, a quick refresher. robots.txt is a small text file at the root of a website. It tells bots which pages they may fetch, but each bot chooses whether to follow it. The file is made of groups. Each group starts with a bot's name and then lists the pages that bot may or may not fetch. The new rule's trick is a made-up bot name: `__usage_ai-training`. Whatever you write under that name is your answer to every AI lab that supports the rule. "Disallow" means don't train on these pages, and "Allow" means you may. Since it works like any other group, you can [give one answer for your docs and another for the rest of your site](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/). The proposal's own example blocks training on a whole site except its press kit: ``` User-Agent: __usage_ai-training Disallow: / Allow: /press-kit/ ``` Two details are easy to miss: - **Your catch-all rules don't count.** `User-Agent: *` is the group whose rules apply to every bot. The new rule [ignores that group](https://www.ietf.org/archive/id/draft-kelley-aipref-training-preference-signal-00.txt). So to answer on training, you need the new group. - **It's about use, not access.** The rule tells a lab what it may do with your pages. Bots can still fetch them. To block a bot, you still name that bot. ### When it starts Here's when each lab says it will start reading the rule: - **Anthropic:** it has started building support, but says that will take time. The proposal's author said so in a [post on an IETF mailing list](https://mailarchive.ietf.org/arch/msg/ai-control/mddjatOxSR0h6GmLRBWlc839c7w/), the only thing Anthropic has said about it so far. - **Microsoft:** [next February](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/). - **Google:** [early next year](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers). - **OpenAI:** [in the coming months](https://openai.com/index/helping-websites-express-ai-training-preferences/), with no date. The old opt-outs keep working, before and after. If you block GPTBot at OpenAI or Google-Extended at Google today, that still counts. [OpenAI](https://openai.com/index/helping-websites-express-ai-training-preferences/) and [Google](https://developers.google.com/search/blog/2026/10/ai-training-control) both say so. ### What saying no covers AI labs use your pages in three different ways: - **Training:** your pages help build the model itself. - **Grounding:** a model reads your page while it writes an answer for someone. - **Search:** the lab indexes your pages and shows them as results. The new rule only covers training. At OpenAI, you can say no to training and stay in ChatGPT search. It has [a separate bot for each job](https://platform.openai.com/docs/bots): GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User when someone asks ChatGPT to open a page. OpenAI says each setting works on its own. Google is where it gets tricky. Its AI answers in Search are called AI Overviews and AI Mode. They follow the [normal Googlebot rules](https://developers.google.com/search/docs/appearance/ai-features). So you stay in them after a no to training. Gemini is another story. It's Google's chatbot, and it only uses sites that allow Google-Extended for grounding. That's the same name Google already uses for training. And Google [will read a "no" on the new rule](https://developers.google.com/search/blog/2026/10/ai-training-control) as a "no" to Google-Extended, unless you set Google-Extended yourself. Put those two together. If you tell Google no to training, you also drop out of Gemini's answers. That's our reading: you only see it when you put Google's two pages side by side. Microsoft [keeps training and grounding apart](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/) too, and it's frank about what you get either way: > Allowing training can help improve future generative models. Neither choice guarantees inclusion, traffic, or payment. In other words: say yes, and you may help the next model. Microsoft promises you nothing back. ### We checked 25 devtools docs sites So what are devtools companies doing about this today? We opened the robots.txt files of 25 devtools docs sites, [Stripe](https://docs.stripe.com/robots.txt) and [Docker](https://docs.docker.com/robots.txt) among them. We picked the 25 before looking at any of them. None of them block GPTBot or Google-Extended, the names OpenAI and Google use for training. Nine don't mention AI at all. But more than half already say yes to training. Fourteen of the 25 have this line: `Content-Signal: ai-train=yes`. It's a format Cloudflare [created last year](https://blog.cloudflare.com/content-signals-policy/), with separate switches for training, search and grounding. Cloudflare warned back then that AI labs might ignore it. And this week, the labs' announcements only mention the new rule. So these 14 sites said "yes, train on us" in a format no lab has promised to read. We could only trace one of the 14 to a written reason. Sentry added its line [in May](https://github.com/getsentry/sentry-docs/pull/17796). The person who made the change explained why: > it feels like are [sic] direction so far has been to optimize the docs for LLM usage, so training seemed like the right call. One site went the other way: Twilio. Its file says [`ai-train=no, search=yes, ai-input=yes`](https://www.twilio.com/robots.txt). That's yes to search, yes to grounding and no to training. We couldn't find anywhere that Twilio says why. And one file contradicts itself. Render's robots.txt has a comment about reserving its rights against AI training, above a list of the big AI crawlers. Then the very next line says [`ai-train=yes`](https://render.com/robots.txt). ### Your host may be writing this for you Now, the twist. Cloudflare can write a customer's robots.txt for them. Last year, its default for those customers was [`ai-train=no`](https://blog.cloudflare.com/content-signals-policy/). This year, it [started writing the file](https://blog.cloudflare.com/bot-preference-sync) from the settings in its dashboard. That means a change at Cloudflare can change what your file says. In September, Cloudflare switched on new rules for AI bots. The next day, HasData [counted more than a hundred sites](https://hasdata.com/blog/ai-crawler-block-index) whose robots.txt had lost that default "no", along with their GPTBot block. Most of the sites HasData then tested still blocked GPTBot in some other way. HasData is the only source for this count, and it sells a scraping API. Vercel shipped its own control the same week, but it sorts bots differently. Its firewall now [blocks "AI crawlers" and "AI assistants" separately](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules). But Vercel's "crawlers" include bots that collect pages for indexing as well as for training. So if you block them to stop training, you also block the bots that build search indexes. ### Our take Our take: decide this yourself, page by page, before your host decides for you. For most devtools companies, that means letting AI labs train on your docs. A model that learned your API in training can write code against it before a developer searches for anything. The best argument against this is that training goes out of date: a model keeps writing code for last year's version of your API. Upstash built a product around that problem. [Context7](https://github.com/upstash/context7) feeds current library docs to coding agents. Its README describes the symptom: > Code examples are outdated and based on year-old training data Of course, Upstash sells the fix. If you agree with Upstash, what matters is grounding: the model reads your current docs while it answers. The new rule doesn't cover grounding. Twilio's file takes exactly that position: no to training, yes to grounding. But you may not have to choose. The rule works page by page. So you could let labs train on your current docs and block the pages for old versions. That's our idea, not something a lab has proposed. It only works if labs read the rule page by page. Google hasn't said whether it will. The company to watch is Cloudflare. It managed robots.txt for [more than 3.8 million domains](https://blog.cloudflare.com/content-signals-policy/) last year. If it starts writing the new rule for them, millions of sites will say yes or no to AI training by default. ## GitHub puts Copilot in the Enterprise seat, but charges for chat, review and agents On Wednesday, GitHub [added Copilot to GitHub Enterprise](https://github.blog/changelog/2026-10-28-github-enterprise-now-includes-access-to-github-copilot), its top plan for companies. Every licensed user gets it. The plan still costs [$21 per user per month](https://github.com/pricing). For now, this covers companies that buy through a GitHub or Microsoft account team. Companies that pay by card get it in the coming weeks. So, free Copilot? Not quite. The catch is in one sentence of the changelog: > There is no included usage allowance to track or top up. In other words, the seat comes with no usage at all. Every chat, code review and cloud agent task is billed from the first request, out of a budget the company sets. (The cloud agent is Copilot working on tasks in the background.) GitHub says [an empty budget never removes access](https://github.blog/changelog/2026-10-28-github-enterprise-now-includes-access-to-github-copilot). In practice, autocomplete keeps working while chat, code review and the cloud agent stop. So is it cheaper than before? Let's do the math. Take an Enterprise developer who also had a Copilot Business seat. Until now, the company paid twice: - **The Enterprise seat:** $21. - **A Copilot Business seat:** [$19](https://docs.github.com/copilot/get-started/whats-changing-with-github-enterprise-and-copilot). It came with 1,900 [credits](https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing) (GitHub's billing unit) at a cent each. So the second seat was really prepaid usage. Now the company pays the $21 seat, plus a cent for every credit the developer uses. A developer who uses the same 1,900 credits costs $21 + $19 = $40. That's exactly what they cost before! Only the developers who use Copilot less will cost less. The math is ours, not GitHub's. We've seen this move before, at Atlassian. It put its Rovo AI assistant into its paid Jira and Confluence cloud plans, and sold Rovo as coming ["at no extra cost"](https://www.atlassian.com/blog/blog/announcements/team25-europe-rovo-everywhere). Each plan gets a [small allowance of credits](https://support.atlassian.com/organization-administration/docs/usage-limits-in-atlassian-intelligence/). Past that, it's a cent a credit, the same price as GitHub. So there's one difference: Atlassian includes a little usage, and GitHub includes none. To be fair to GitHub, the change does help some people. A light user now gets Copilot for the price of the seat alone. And autocomplete is free and keeps working when the budget is gone. Our take: GitHub chose an honest word, and that's worth copying. Its [changelog](https://github.blog/changelog/2026-10-28-github-enterprise-now-includes-access-to-github-copilot) title and its [pricing page](https://github.com/pricing) both say the seat includes "access" to Copilot. Atlassian said "at no extra cost". "Access" is the accurate word for a seat that comes with no usage. GitHub's changelog also says plainly what stops when the money runs out. If your AI feature comes with a seat but no usage, do the same: call it "access", and say what stops on the same page. ## Google gives Jules users 13 days before it goes read-only Jules is Google's coding agent that works in the background. You give it a task on a GitHub repository, it works on a machine in Google's cloud, and it comes back with a pull request. On Thursday, a [changelog entry](https://jules.google/docs/changelog/) titled "Jules Turndown" said users can finish their sessions until 11 November. After that, Jules goes read-only, and it shuts down at the end of the year. That leaves users 13 days to wrap up their work. The export is thorough. It covers settings, environment variables, scheduled tasks, memories and the history of every session. And your code already lives in GitHub anyway. What's missing is where to go next. [The notice](https://jules.google/docs/changelog/) links to Jules's command-line tool rather than a migration guide. It also doesn't say what carries over to Antigravity, Google's newer app for coding with agents. That's odd, because the same notice says Jules helped shape Antigravity. And that changelog entry is the whole announcement. There's a bigger difference, too. Jules ran its jobs on Google's machines in the cloud. Antigravity [can't do that yet](https://www.testingcatalog.com/cloud-execution-is-coming-to-antigravity-soon/), according to TestingCatalog. It's a site that digs through unreleased software, and it found a switch for cloud jobs in one test build. That's the only report so far, and Google hasn't given a date. Your subscription is safe, though. Jules came with the [Google AI Pro and Ultra plans](https://jules.google/docs/usage-limits/), and those plans continue. Still, 13 days is the shortest notice we've seen this year. For comparison, GitHub gave users of Spark (its app builder) [27 days](https://github.blog/changelog/2026-08-04-upcoming-deprecation-of-github-spark-on-github-com/) in the summer. After 11 November, Jules users lose their cloud agent, and Antigravity has no date for one. ## Briefly **PostHog published its onboarding emails, flaws included.** It asked three lifecycle marketers to critique its onboarding flow, about a hundred emails. The [post](https://posthog.com/blog/lifecycle-marketing) admits most of those emails came from taste, not experiments. It ends with a list of changes. One example is measuring emails against activation with holdouts: groups of users who deliberately get no email. **Supabase sponsored a benchmark that shows it losing.** [DBArena](https://dbarena.com) is a benchmark site that [Supabase sponsored](https://supabase.com/blog/introducing-dbarena). It scores hosted Postgres by the work you get per dollar of list price. At one mid-sized instance, Amazon RDS does more than twice the work per dollar. Supabase wins at the smallest size, and its beta engine wins at the largest. A vendor's benchmark is worth reading when it shows the vendor losing. **OpenRouter writes for Claude Code users at the moment they get stuck.** OpenRouter resells models from many labs behind one API key, Claude included. It wrote two posts named after [a Claude error](https://openrouter.ai/blog/tutorials/claude-529-overloaded) and [a Claude Code usage limit](https://openrouter.ai/blog/tutorials/claude-code-usage-limit). The second shows how to keep coding past the limit by running Claude Code through an OpenRouter key. That's content aimed at Anthropic's own customers, right when Anthropic stops serving them. ## Sources - [Google Search Central - Supporting a shared AI training control for websites](https://developers.google.com/search/blog/2026/10/ai-training-control) - [Google - Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) - [OpenAI - Making it easier for websites to express AI training preferences](https://openai.com/index/helping-websites-express-ai-training-preferences/) - [Bing Blog - Your content, your choice: clearer preferences for training and grounding](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/) - [IETF - draft-kelley-aipref-training-preference-signal-00](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/) - [IETF - draft-kelley-aipref-training-preference-signal-00 (text)](https://www.ietf.org/archive/id/draft-kelley-aipref-training-preference-signal-00.txt) - [IETF ai-control list - New draft: draft-kelley-aipref-training-preference-signal-00](https://mailarchive.ietf.org/arch/msg/ai-control/mddjatOxSR0h6GmLRBWlc839c7w/) - [OpenAI - Overview of OpenAI crawlers](https://platform.openai.com/docs/bots) - [Google Search Central - AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) - [Cloudflare - Giving users choice with Cloudflare's new Content Signals Policy](https://blog.cloudflare.com/content-signals-policy/) - [Cloudflare - Bot Preference Sync](https://blog.cloudflare.com/bot-preference-sync) - [HasData - AI Crawler Block Index](https://hasdata.com/blog/ai-crawler-block-index) - [Vercel changelog - Block AI crawlers and assistants independently with managed rules](https://vercel.com/changelog/block-ai-crawlers-and-assistants-independently-with-managed-rules) - [getsentry/sentry-docs PR #17796](https://github.com/getsentry/sentry-docs/pull/17796) - [Twilio - robots.txt](https://www.twilio.com/robots.txt) - [Render - robots.txt](https://render.com/robots.txt) - [Upstash - Context7 README](https://github.com/upstash/context7) - [GitHub changelog - GitHub Enterprise now includes access to GitHub Copilot](https://github.blog/changelog/2026-10-28-github-enterprise-now-includes-access-to-github-copilot) - [GitHub Docs - What's changing with GitHub Enterprise and Copilot](https://docs.github.com/copilot/get-started/whats-changing-with-github-enterprise-and-copilot) - [GitHub Docs - Models and pricing for GitHub Copilot](https://docs.github.com/en/copilot/reference/copilot-billing/models-and-pricing) - [GitHub - Pricing](https://github.com/pricing) - [Atlassian Support - How Rovo credits work](https://support.atlassian.com/organization-administration/docs/usage-limits-in-atlassian-intelligence/) - [Atlassian - Meet Rovo everywhere](https://www.atlassian.com/blog/blog/announcements/team25-europe-rovo-everywhere) - [Jules - Changelog](https://jules.google/docs/changelog/) - [Jules docs - Usage limits](https://jules.google/docs/usage-limits/) - [TestingCatalog - Cloud execution is coming to Antigravity soon](https://www.testingcatalog.com/cloud-execution-is-coming-to-antigravity-soon/) - [GitHub changelog - Upcoming deprecation of GitHub Spark on github.com](https://github.blog/changelog/2026-08-04-upcoming-deprecation-of-github-spark-on-github-com/) - [PostHog - Lifecycle marketers roasted our lifecycle marketing](https://posthog.com/blog/lifecycle-marketing) - [Supabase - Introducing DBArena](https://supabase.com/blog/introducing-dbarena) - [DBArena](https://dbarena.com) - [OpenRouter - Claude 529 Overloaded: What It Means and How to Fix It](https://openrouter.ai/blog/tutorials/claude-529-overloaded) - [OpenRouter - Hit Your Claude Code Usage Limit? How to Keep Coding](https://openrouter.ai/blog/tutorials/claude-code-usage-limit) ## Related - [Last week - Vercel paid $242,000 of its $1 million sandbox bounty](https://thebeat.dev/issues/2026-W43/) - [Cursor's new $30 plan leaves out Claude](https://thebeat.dev/issues/2026-W42/) - [MCP allows a year to migrate, GitHub Models gives six weeks](https://thebeat.dev/issues/2026-W31/) --- Part of [The Beat - developer marketing, on the record](https://thebeat.dev/). Content license: CC BY 4.0 - quote it, link the canonical page.