A growing share of developers meet your product as a paraphrase. A coding assistant summarizes your docs inside the editor; an answer engine compares you against two competitors in a chat window; an agent reads your API reference to decide whether it can do the job. The discipline that has formed around this goes by several names - AEO (answer engine optimization), GEO (generative engine optimization) - and it arrives with a venture-funded tooling category and a large pile of ritual attached. Sorting the real work from the ritual is most of the job.

The one-line position: machine-mediated discovery is a real channel with real conversion numbers, but almost everything sold under the AEO label is either premature or free - the durable work is making your docs and API surfaces machine-legible, which is work you should want anyway.

Three layers, three different bets

Everything in this space sorts into three layers with very different risk profiles. Conflating them is what makes the category feel like snake oil.

  • The reading layer - real, cheap, do it now. Models already mediate first impressions. The work is unglamorous: serve docs as static HTML rather than client-rendered JavaScript; make one section answer one question, so a chunk retrieved alone still makes sense; put real numbers - pricing, rate limits, latency - on ungated pages where a model can quote them instead of guessing; publish OpenAPI for the reference; and check that robots.txt isn’t blocking retrieval bots while you deliberate about training bots, which are a separate population and a separate decision. The accidental version is common: a September 2026 audit of the Tranco top 5,000 (one AI-operated author, dataset public, not reproduced) found 238 of the 535 sites blocking GPTBot also block OAI-SearchBot - the crawler that produces ChatGPT’s citations - while welcoming every other search engine, which reads as a catch-all rule nobody decided on. The file and the actual policy also drift apart on their own - enough that Cloudflare shipped Bot Preference Sync (August 2026) to auto-write dashboard-set search/agent/training rules into the published robots.txt; if you manage the file by hand, diff it against what you actually enforce as part of the recurring check below. Cloudflare then made the split itself a setting (2026-09-15): “Disallow AI Training” keeps a site indexable by AI search and answer engines while blocking training crawlers, backed by an “Accountable” designation for operators that run the two crawlers apart and promise that opting out of training does not touch search - Apple, Google, Microsoft, Amazon, Anthropic, Meta and OpenAI qualify at launch. It is available on every plan and off by default, so the decision the accidental block was making for you is now one checkbox you have to go and set. The training half is getting one name for every lab. On 2026-10-30 Google, OpenAI and Microsoft each said they will honour a shared robots.txt group, User-Agent: __usage_ai-training, from an Internet-Draft written at Anthropic, which says it has started implementing it: Disallow states a preference against training, Allow one for it, and ordinary path rules apply, so docs and the rest of a site can answer differently. Microsoft starts 2027-02-01, Google says early 2027 and OpenAI gives no date; the per-lab names (GPTBot, Google-Extended) keep working alongside it. Two details decide whether your file says what you think. A User-Agent: * group does not count as an answer. And at Google a “no” maps to Google-Extended, which also gates grounding in Gemini Apps, so there a training opt-out costs a citation surface too, while AI Overviews follow Googlebot. Cloudflare’s Content-Signal: ai-train= line, which 14 of 25 devtools docs sites the Beat read on 2026-11-01 already carry, is not the token any lab has promised to read.
  • The selection layer - early, design for it if you sell an API. Nobody can show you conversion data on agent-initiated purchases yet, but the pattern is legible: machine-readable everything, quote-then-execute pricing so an agent can’t be surprise-billed, idempotency keys so a retry can’t double-buy, scoped tokens so an agent can search without holding purchase permission, and consent records of what was agreed and when. GoDaddy’s developer platform (July 2026) is the most copyable artifact the thread has produced. Every element doubles as good API design for humans, which is why it’s worth adopting before the data arrives. The first measurement of what a selecting agent actually reads arrived in October 2026, from retail rather than devtools: Profound watched Meta’s Muse shopping agent and found that through the open browser it visited about three domains and returned four products from two merchants, while through Meta’s product catalog it returned about sixty products from thirty sites (50 prompts, one category, a sample its authors call “small, exploratory”). The agent reads the feed it was integrated with far more than it browses the web, so where your product has a catalog or a price list, the lookup step belongs in data - a feed, a read-only tool - before it belongs in prose. The flow itself, and the directories and marketplaces an agent finds you through, are agent rails’ subject; this section stays on the reader.
  • The measurement layer - forming, mostly don’t buy yet. Enterprises will pay to know how AI describes them; Profound raised $96M at a $1B valuation in February 2026 with 700+ enterprise customers, then $180M at $1.8B seven months later (2026-09-15), citing more than 1,000 enterprise clients and revenue tripled in six months - the company’s own figures. At Fortune-500 scale, across hundreds of SKUs and ten AI surfaces, a dashboard may earn its keep. For a typical devtool company the math is different, and the free check gets you most of the signal.

What the numbers actually support

The honest read requires holding two sets of evidence at once.

The channel is real. Vercel reported ChatGPT referring 10% of new signups in April 2025, up from under 1% six months earlier - first-party attribution on its own funnel. The mechanism was plumbing, not magic: docs rendered as static HTML and structured so a model could retrieve a clean answer to a task-shaped question. The conversion premium shows up in every dataset that measures it, though never at the same number - Adobe’s retail read (reported via eMarketer) had AI-referred traffic converting 42% better than non-AI traffic by March 2026, a full reversal from a year earlier when it converted worse, and Visibility Labs’ analysis of 94 ecommerce sites had ChatGPT traffic converting 31% above non-branded organic in 2025 (1.81% vs 1.39%) - so treat any single rate as that vendor’s panel, not the channel’s truth. The direction, though, is consistent: a visitor arriving from an AI answer arrives pre-qualified - the model already matched your tool to their task and answered their first three objections.

The traffic exchange is lopsided, and that’s fine for you. Cloudflare’s crawl-to-refer ratios (July 2025) ran to roughly 38,000:1 for Anthropic’s crawlers, ~1,100:1 for OpenAI and ~195:1 for Perplexity, with around 80% of AI crawler activity being model training rather than answering a live question. Publishers who sell pageviews have a real grievance here. You don’t sell pageviews: if a model ingests your docs ten thousand times to produce one well-timed “use X for this, here’s the code” in a developer’s editor, you got the good end of that trade.

The ranking metaphor doesn’t hold. Citations churn - by the measure of the vendor selling the fix, up to 90% of sources cited in AI answers shift over time, and different models draw on largely distinct source sets. The split runs deeper than model families: Profound’s August 2026 observational study (24,135 responses; vendor-collected) found Claude and Claude Code behave as two distinct answer engines - Claude Code invoked web search in only 13% of responses against over 93% for Claude, and when it visited a site it went to documentation and pricing pages nearly three-quarters of the time, where Claude’s visits skewed to robots.txt, sitemaps and homepages. The reader inside the editor mostly isn’t searching; it is fetching your pages directly. There is no stable position to hold, which is precisely why per-model rank tracking is the weakest thing to buy first. Nor is a citation share one number: Profound’s September 2026 persona study (71,147 responses across ChatGPT, Claude and Gemini; vendor-collected) found that identical questions asked by different personas shared about one in four brand mentions and one in five cited domains, and higher-income personas drew more of both - so a tool reporting one score per query is averaging over people, and the free check below should be run as the buyer you actually sell to. The evidence layer itself is no steadier than the rankings: when Google added Gemini 3.8 Flash to AI Mode in September 2026, citations and links stopped appearing on top-of-funnel queries for the subscribers who had picked that model - reported 2026-09-03, confirmed by Google’s head of Search product as “not working as intended”, links back the next morning. Not platform-wide, and over in about a day, but an outage of the very signal presence dashboards measure, caused by a routine model swap.

The churn now has a half-life. Profound followed 883,000 pages (1.19 million page-engine lifecycles) across ChatGPT, Perplexity, Claude, Gemini, Copilot, AI Overviews and AI Mode from September 2025 to September 2026 and found the median cited page falls to half its peak citation share in 11 days; 78% do so within two weeks, and the pages that persist have a median half-life of 27 to 29 days (published 2026-09-30; Profound sells the tracker the study was run on, so treat the decimals as one vendor’s instrument and the shape as the finding). Two consequences for the work. A citation is a flow rather than a stock, so the pages that carry yours need a refresh cadence - the study’s own advice is to publish “with enough consistency to be chosen again” - and the cross-engine correlation of 0.03 to 0.09 (0.12 to 0.17 among Google’s three surfaces) means presence on one engine says almost nothing about another, while a page cited by five or more of the seven was 5.5 times as likely to still hold half its peak after eight weeks. Run the free check below across every engine your buyers use, and protect the pages that show up on several.

The citation can also be bought. A measurement study posted 2026-10-08 (ten AI-search engines, 17,211 citations over 6,356 domains; a preprint that names NDSS 2027 as its venue) found each engine draws 20.5% to 70.8% of its citations from twenty domains - Reddit alone 572 citations across four engines, YouTube 497 across three - and that of the 22 publishing platforms behind them, 19 put only low or medium friction on creating an account, 18 review a post in minutes or not visibly at all, and 15 clear both bars at once. A concept the authors invented, posted on nine such platforms with Medium and YouTube among them, was cited by eight of the ten engines within seven days, and a $14 order to a content vendor produced 13 posts in 28 minutes and a citation on ByteDance’s Doubao within the hour (the paid demonstration ran on Chinese platforms; the Western engines were reached through Medium and YouTube for free). Two consequences for the free check below. Record the domain that carries each citation, not only the engine: a citation from your own docs or a filing is an asset, one from a post anyone could have written is a rental. And when a competitor appears suddenly in an answer, or a confident wrong claim about you does, consider that it was planted before assuming the model drifted.

llms.txt: ship it, expect nothing

This one is settled enough to state plainly, because the two authorities finally agree from opposite directions.

Ahrefs checked all 137,210 domains in its Web Analytics that received traffic in May 2026. About 28% had shipped a valid llms.txt - remarkable adoption for an unofficial spec - but 97% of those files received zero requests. Of requests that did arrive, 96% were bots, and the largest single category was SEO audit tools (21.7%), the industry checking its own homework; AI retrieval bots were 1.1%. AI bots made essentially no requests for files that don’t exist, meaning they aren’t probing for it.

Google then made the split official. Search Central’s AI-features guidance states that you “don’t need to create new machine readable files, AI text files, or markup to appear in these features” (the page’s wording as of 2026-09-23; an earlier revision added that Google Search itself doesn’t use them) - so the file is not a ranking or visibility input. Meanwhile Chrome’s Lighthouse added an Agentic Browsing audit category that checks for llms.txt and WebMCP as agent-readiness signals - explicitly experimental: Chrome’s own docs say testing it requires Chrome 150 or later (the WebMCP audits additionally need an origin trial), and it deliberately carries no 0-100 score, only a ratio of checks passed, because while the standards settle the aim is “to gather data and provide actionable signals rather than a definitive ranking”.

That is retrieval versus ranking, made explicit by the one company that owns both a search index and a browser: nothing for the index, something still unmeasured for browser-borne agents. So ship it - an afternoon’s cheap insurance, and its absence reads badly to the audit tools your prospects run - but expect nothing from it, and never let a checkbox file substitute for the docs work underneath. The file is the index; the content is the product.

“Expect nothing” has one measured refinement, and it says when the file is read rather than whether. Gauge, which runs coding agents against real product tasks and logs every fetch, found agents opened a site’s llms.txt in 36.3% of build tasks and 0.5% of vendor-selection tasks (its 2026-09-22 writing guide; Gauge sells agent analytics and the page states no session count, window or model list, so treat the split as one vendor’s instrument rather than a market rate). After opening it, the same page says, agents “usually continue into the linked docs”. The reading is that the file does not win the pick - a selection task is answered from search results and model memory - and does onboard an agent already committed to the tool. Write it accordingly: a routing index grouped by what a build task needs (get started, authentication, reference and examples, troubleshooting), every link to a complete and authoritative page, major features present because an omission reads as a missing capability, and no marketing pages in it at all. The pitch belongs where the selection happens, on the pages the answer engines cite. Gauge’s crawler-side read a week later (2026-09-29) says the same thing from the other direction: across more than 500 million bot events on the sites it instruments, roughly 600,000 requests carrying ChatGPT-User’s user agent asked for llms.txt exactly zero times - so the crawlers that produce citations are not reading the file either, and the only reader it has is the agent already building on you.

And audit it like code, because an agent executes what it reads. In August 2026 researcher Alon Hertz scanned 6,214 corporate domains and found more than 237 install commands in their llms.txt files pointing at package names nobody had registered - names an attacker can claim. He registered a handful with a harmless beacon and got a callback from inside a Fortune 500 in under four minutes; in Clerk’s docs, an unknown party had already registered a referenced package name and loaded it with live malware. Every install command in an llms.txt, an AGENTS.md or an agent-facing quickstart must resolve to a package you own, and the check belongs in CI, not in a quarterly review.

And hold the evidence for it - yours or a vendor’s - to a referral standard. In August 2026 SEO Mark Williams-Cook published cats.txt, a satirical standard for declaring office cats, and collected every proof point the industry cites for llms.txt: AI bots crawled it, Google indexed it, ChatGPT endorsed it and an AI Overview cited its fictional data - then ChatGPT reversed itself once public discussion called it a joke, showing the endorsement was echoed consensus, not evaluation. Crawled, indexed and cited are observations that occur whether or not a tactic works. The only proof that survives is an attributed number from your own funnel, which is what the measurement section below is for.

Measuring it without buying anything

The free check gets most of the signal and costs twenty minutes a month:

  • Open three assistants and ask the task-shaped questions your buyers actually ask - “how do I add auth to a Next.js app”, not “best auth provider”. Record whether you appear, what’s said about your pricing and limits, and whether it is true. Accuracy matters more than presence: a confident wrong answer about your free tier costs you more than absence does. And when an answer is wrong, trace it to the page it cites before assuming the model hallucinated: Profound’s August 2026 analysis of 158,000+ brand claims found the most common source of an inaccurate claim was the brand’s own site - 54% of brands had a wrong claim citing their own content, ahead of earned media - with pricing the concentration point (12% of evaluated claims, 24% of the inaccuracies). The fix is usually deleting or correcting a stale page you forgot you owned, not filing a bug against the model.
  • Add an AI-referral segment in analytics and a free-text “how did you hear about us” field. Vercel’s 10% figure came from exactly that kind of first-party attribution, not from a platform - and randomize any survey’s answer order: Val Town found whichever option listed first got picked more, before it could trust its own July 2026 reading that most known-source new Pro signups arrived via AI. And know that the segment rests on a parameter the citing engine controls: for about a week from 2026-10-08 most ChatGPT citation links lost the utm_source=chatgpt.com tag they had carried since June. Profound’s crawl put the share of answers whose links carried it as low as about one in twenty at the worst hour on 10-14, and LLM Pulse saw ChatGPT sessions across its monitored sites run about 46% below the same days of the prior two weeks over 9-11 October (both sell the tracker; neither publishes a sample). Where the visits went is one analyst’s read rather than either vendor’s: an Adobe director found direct traffic rising by about 26,000 visits a day across some 500 pages (9-13 October against the prior week, sampled), the links opening in a new tab with noreferrer, while LLM Pulse found the rise in direct too small to spot in its own site-level data. OpenAI called it a bug, to that Adobe director rather than in public, and the tag was back near its prior level by 10-16. Annotate the dates in analytics, keep the free-text field as the count the tag cannot move, and treat any attributed AI-referral rate as one the engine can switch off without telling you.
  • Turn on Search Console’s generative AI performance reports, rolled out to every site worldwide as of 2026-08-31 after a phased start in June: impressions from AI Overviews, AI Mode and AI-powered Discover by page, country and device, with a matching opt-out that removes a site from those surfaces without touching standard ranking. It reports impressions, not clicks, so treat it as a free presence meter - weather, not attribution.
  • Watch your own logs for retrieval bots. Which crawlers fetch which pages is a fact you own, and it tells you more than a rank tracker. The population is unstable enough that this check has to recur: Meta’s own indexer went from roughly 2% to 37.8% of observed AI-crawler requests in under a month (Promptwatch, August 2026) - the single heaviest AI crawler in those logs, arriving almost from nowhere - so a robots policy or allowlist tuned to last year’s bot names quietly goes stale.
  • Buy tooling when the free check stops scaling - hundreds of pages, many surfaces, someone accountable for the number - not before.

What earns citations, and what repels

The work that gets you quoted is mostly the work that serves human readers, which is the tell that it’s durable:

  • Specific, quotable facts. Concrete numbers, named limits and dated benchmarks are what a model can lift into an answer. Marketing adjectives are unquotable by construction.
  • One question per section. Retrieval returns chunks, not pages. A section that only makes sense after reading the three above it is invisible to the reader that matters.
  • Verifiable claims over positioning language. When the evaluator is a machine comparing three tools, published, checkable numbers beat superlatives.
  • Loud deprecation. Machine readers make this a survival trait: an agent that receives a clean, versioned deprecation notice reroutes; an agent that hits a silent 404 tells its developer your product is broken.
  • Gating and client-side rendering repel. Content behind a form or assembled by JavaScript is content the model cannot cite. This is the oldest lesson in the guide arriving through a new door.

The honest caveat

This section describes a channel that is moving faster than the evidence about it. The lab result everyone quotes - visibility gains “up to 40%” from adding quotable statistics and citations - comes from an academic benchmark (Aggarwal et al., KDD 2024), not from the field, and the paper itself warns the effect varies widely by domain. Treat any vendor number about your own visibility as a hypothesis you test against your funnel. The reading-layer work is the part that pays regardless of how the rest of it resolves, because it is indistinguishable from doing your docs properly.