Google and OpenAI back one robots.txt rule for AI training

What happened on Friday

On Friday, Google (opens in a new tab) and OpenAI (opens in a new tab) said they’ll support a new rule in robots.txt. It lets a website tell AI labs whether they may train their models on its pages.

Someone at Anthropic wrote the rule, and Microsoft says (opens in a new tab) Bing will follow it too. So that’s Google, OpenAI and Microsoft behind one rule. But it’s still only a proposal (opens in a new tab), and it may never become an official internet standard.

Why should a devtools company care? Because training is how a model learns your API. A model that learned your API in training can write code against it before a developer searches for anything. Until now, you had to give each lab its own answer. From next year, one line covers every lab that supports the rule.

There’s a twist, too: your hosting provider may already be writing part of this file for you. We’ll get to that below.

How the rule works

First, a quick refresher. robots.txt is a small text file at the root of a website. It tells bots which pages they may fetch, but each bot chooses whether to follow it. The file is made of groups. Each group starts with a bot’s name and then lists the pages that bot may or may not fetch.

The new rule’s trick is a made-up bot name: __usage_ai-training. Whatever you write under that name is your answer to every AI lab that supports the rule. “Disallow” means don’t train on these pages, and “Allow” means you may.

Since it works like any other group, you can give one answer for your docs and another for the rest of your site (opens in a new tab). The proposal’s own example blocks training on a whole site except its press kit:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Two details are easy to miss:

Your catch-all rules don’t count. User-Agent: * is the group whose rules apply to every bot. The new rule ignores that group (opens in a new tab). So to answer on training, you need the new group.
It’s about use, not access. The rule tells a lab what it may do with your pages. Bots can still fetch them. To block a bot, you still name that bot.

When it starts

Here’s when each lab says it will start reading the rule:

Anthropic: it has started building support, but says that will take time. The proposal’s author said so in a post on an IETF mailing list (opens in a new tab), the only thing Anthropic has said about it so far.
Microsoft: next February (opens in a new tab).
Google: early next year (opens in a new tab).
OpenAI: in the coming months (opens in a new tab), with no date.

The old opt-outs keep working, before and after. If you block GPTBot at OpenAI or Google-Extended at Google today, that still counts. OpenAI (opens in a new tab) and Google (opens in a new tab) both say so.

What saying no covers

AI labs use your pages in three different ways:

Training: your pages help build the model itself.
Grounding: a model reads your page while it writes an answer for someone.
Search: the lab indexes your pages and shows them as results.

The new rule only covers training. At OpenAI, you can say no to training and stay in ChatGPT search. It has a separate bot for each job (opens in a new tab): GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User when someone asks ChatGPT to open a page. OpenAI says each setting works on its own.

Google is where it gets tricky. Its AI answers in Search are called AI Overviews and AI Mode. They follow the normal Googlebot rules (opens in a new tab). So you stay in them after a no to training.

Gemini is another story. It’s Google’s chatbot, and it only uses sites that allow Google-Extended for grounding. That’s the same name Google already uses for training. And Google will read a “no” on the new rule (opens in a new tab) as a “no” to Google-Extended, unless you set Google-Extended yourself.

Put those two together. If you tell Google no to training, you also drop out of Gemini’s answers. That’s our reading: you only see it when you put Google’s two pages side by side.

Microsoft keeps training and grounding apart (opens in a new tab) too, and it’s frank about what you get either way:

Allowing training can help improve future generative models. Neither choice guarantees inclusion, traffic, or payment.

In other words: say yes, and you may help the next model. Microsoft promises you nothing back.

We checked 25 devtools docs sites

So what are devtools companies doing about this today? We opened the robots.txt files of 25 devtools docs sites, Stripe (opens in a new tab) and Docker (opens in a new tab) among them. We picked the 25 before looking at any of them.

None of them block GPTBot or Google-Extended, the names OpenAI and Google use for training. Nine don’t mention AI at all.

But more than half already say yes to training. Fourteen of the 25 have this line: Content-Signal: ai-train=yes. It’s a format Cloudflare created last year (opens in a new tab), with separate switches for training, search and grounding. Cloudflare warned back then that AI labs might ignore it. And this week, the labs’ announcements only mention the new rule. So these 14 sites said “yes, train on us” in a format no lab has promised to read.

We could only trace one of the 14 to a written reason. Sentry added its line in May (opens in a new tab). The person who made the change explained why:

it feels like are [sic] direction so far has been to optimize the docs for LLM usage, so training seemed like the right call.

One site went the other way: Twilio. Its file says ai-train=no, search=yes, ai-input=yes (opens in a new tab). That’s yes to search, yes to grounding and no to training. We couldn’t find anywhere that Twilio says why.

And one file contradicts itself. Render’s robots.txt has a comment about reserving its rights against AI training, above a list of the big AI crawlers. Then the very next line says ai-train=yes (opens in a new tab).

Your host may be writing this for you

Now, the twist. Cloudflare can write a customer’s robots.txt for them. Last year, its default for those customers was ai-train=no (opens in a new tab). This year, it started writing the file (opens in a new tab) from the settings in its dashboard.

That means a change at Cloudflare can change what your file says. In September, Cloudflare switched on new rules for AI bots. The next day, HasData counted more than a hundred sites (opens in a new tab) whose robots.txt had lost that default “no”, along with their GPTBot block. Most of the sites HasData then tested still blocked GPTBot in some other way. HasData is the only source for this count, and it sells a scraping API.

Vercel shipped its own control the same week, but it sorts bots differently. Its firewall now blocks “AI crawlers” and “AI assistants” separately (opens in a new tab). But Vercel’s “crawlers” include bots that collect pages for indexing as well as for training. So if you block them to stop training, you also block the bots that build search indexes.

Our take

Our take: decide this yourself, page by page, before your host decides for you. For most devtools companies, that means letting AI labs train on your docs. A model that learned your API in training can write code against it before a developer searches for anything.

The best argument against this is that training goes out of date: a model keeps writing code for last year’s version of your API. Upstash built a product around that problem. Context7 (opens in a new tab) feeds current library docs to coding agents. Its README describes the symptom:

Code examples are outdated and based on year-old training data

Of course, Upstash sells the fix.

If you agree with Upstash, what matters is grounding: the model reads your current docs while it answers. The new rule doesn’t cover grounding. Twilio’s file takes exactly that position: no to training, yes to grounding.

But you may not have to choose. The rule works page by page. So you could let labs train on your current docs and block the pages for old versions. That’s our idea, not something a lab has proposed. It only works if labs read the rule page by page. Google hasn’t said whether it will.

The company to watch is Cloudflare. It managed robots.txt for more than 3.8 million domains (opens in a new tab) last year. If it starts writing the new rule for them, millions of sites will say yes or no to AI training by default.

GitHub puts Copilot in the Enterprise seat, but charges for chat, review and agents

On Wednesday, GitHub added Copilot to GitHub Enterprise (opens in a new tab), its top plan for companies. Every licensed user gets it. The plan still costs $21 per user per month (opens in a new tab). For now, this covers companies that buy through a GitHub or Microsoft account team. Companies that pay by card get it in the coming weeks.

So, free Copilot? Not quite. The catch is in one sentence of the changelog:

There is no included usage allowance to track or top up.

In other words, the seat comes with no usage at all. Every chat, code review and cloud agent task is billed from the first request, out of a budget the company sets. (The cloud agent is Copilot working on tasks in the background.) GitHub says an empty budget never removes access (opens in a new tab). In practice, autocomplete keeps working while chat, code review and the cloud agent stop.

So is it cheaper than before? Let’s do the math. Take an Enterprise developer who also had a Copilot Business seat. Until now, the company paid twice:

The Enterprise seat: $21.
A Copilot Business seat: $19 (opens in a new tab). It came with 1,900 credits (opens in a new tab) (GitHub’s billing unit) at a cent each. So the second seat was really prepaid usage.

Now the company pays the $21 seat, plus a cent for every credit the developer uses. A developer who uses the same 1,900 credits costs $21 + $19 = $40. That’s exactly what they cost before! Only the developers who use Copilot less will cost less. The math is ours, not GitHub’s.

We’ve seen this move before, at Atlassian. It put its Rovo AI assistant into its paid Jira and Confluence cloud plans, and sold Rovo as coming “at no extra cost” (opens in a new tab). Each plan gets a small allowance of credits (opens in a new tab). Past that, it’s a cent a credit, the same price as GitHub. So there’s one difference: Atlassian includes a little usage, and GitHub includes none.

To be fair to GitHub, the change does help some people. A light user now gets Copilot for the price of the seat alone. And autocomplete is free and keeps working when the budget is gone.

Our take: GitHub chose an honest word, and that’s worth copying. Its changelog (opens in a new tab) title and its pricing page (opens in a new tab) both say the seat includes “access” to Copilot. Atlassian said “at no extra cost”. “Access” is the accurate word for a seat that comes with no usage. GitHub’s changelog also says plainly what stops when the money runs out. If your AI feature comes with a seat but no usage, do the same: call it “access”, and say what stops on the same page.

Google gives Jules users 13 days before it goes read-only

Jules is Google’s coding agent that works in the background. You give it a task on a GitHub repository, it works on a machine in Google’s cloud, and it comes back with a pull request.

On Thursday, a changelog entry (opens in a new tab) titled “Jules Turndown” said users can finish their sessions until 11 November. After that, Jules goes read-only, and it shuts down at the end of the year. That leaves users 13 days to wrap up their work.

The export is thorough. It covers settings, environment variables, scheduled tasks, memories and the history of every session. And your code already lives in GitHub anyway.

What’s missing is where to go next. The notice (opens in a new tab) links to Jules’s command-line tool rather than a migration guide. It also doesn’t say what carries over to Antigravity, Google’s newer app for coding with agents. That’s odd, because the same notice says Jules helped shape Antigravity. And that changelog entry is the whole announcement.

There’s a bigger difference, too. Jules ran its jobs on Google’s machines in the cloud. Antigravity can’t do that yet (opens in a new tab), according to TestingCatalog. It’s a site that digs through unreleased software, and it found a switch for cloud jobs in one test build. That’s the only report so far, and Google hasn’t given a date.

Your subscription is safe, though. Jules came with the Google AI Pro and Ultra plans (opens in a new tab), and those plans continue. Still, 13 days is the shortest notice we’ve seen this year. For comparison, GitHub gave users of Spark (its app builder) 27 days (opens in a new tab) in the summer. After 11 November, Jules users lose their cloud agent, and Antigravity has no date for one.

Briefly

PostHog published its onboarding emails, flaws included. It asked three lifecycle marketers to critique its onboarding flow, about a hundred emails. The post (opens in a new tab) admits most of those emails came from taste, not experiments. It ends with a list of changes. One example is measuring emails against activation with holdouts: groups of users who deliberately get no email.

Supabase sponsored a benchmark that shows it losing. DBArena (opens in a new tab) is a benchmark site that Supabase sponsored (opens in a new tab). It scores hosted Postgres by the work you get per dollar of list price. At one mid-sized instance, Amazon RDS does more than twice the work per dollar. Supabase wins at the smallest size, and its beta engine wins at the largest. A vendor’s benchmark is worth reading when it shows the vendor losing.

OpenRouter writes for Claude Code users at the moment they get stuck. OpenRouter resells models from many labs behind one API key, Claude included. It wrote two posts named after a Claude error (opens in a new tab) and a Claude Code usage limit (opens in a new tab). The second shows how to keep coding past the limit by running Claude Code through an OpenRouter key. That’s content aimed at Anthropic’s own customers, right when Anthropic stops serving them.

Everything else this week

Launch · 4

Release · 6

News · 5

Campaign · 3

Sources, check them yourself

Every load-bearing claim above links to one of these. A single-sourced claim says so in the text.