{
  "title": "The Beat - research",
  "updated": "2026-11-03",
  "count": 1,
  "research": [
    {
      "id": "memory-first-docs-second",
      "title": "Memory first, docs second.",
      "format": "case study",
      "author": "Alberto Grande",
      "date": "2026-10-23",
      "updated": "2026-11-03",
      "summary": "About half the time, Claude Code picks where to host an app from what its model already knows. When it does go online, it reads the docs of the host it has already chosen.",
      "dek": "A research project on AEO for coding agents. I ran 264 Claude Code sessions to see how a coding agent chooses the tools it uses. Here's what I found.",
      "disclosure": "The author ran every test and paid for the runs on a personal Claude subscription. There is no sponsor.",
      "datasets": [
        {
          "label": "Every run, with its tool calls (runs.json)",
          "url": "https://thebeat.dev/research/memory-first-docs-second/runs.json"
        },
        {
          "label": "Which web domains count as which platform (platform-domains.json)",
          "url": "https://thebeat.dev/research/memory-first-docs-second/platform-domains.json"
        },
        {
          "label": "The same-model comparisons and the control (comparison.json)",
          "url": "https://thebeat.dev/research/memory-first-docs-second/comparison.json"
        }
      ],
      "methods": [
        {
          "label": "The benchmark harness, grader and labels",
          "url": "https://github.com/albertogrande/coding-agent-hosting-benchmark"
        },
        {
          "label": "The comparison rules, and when each was committed",
          "url": "https://github.com/albertogrande/coding-agent-hosting-benchmark/blob/main/docs/SEASONS.md"
        }
      ],
      "tags": [
        "aeo",
        "ai-coding-agents"
      ],
      "url": "https://thebeat.dev/research/memory-first-docs-second/",
      "markdown_url": "https://thebeat.dev/research/memory-first-docs-second.md",
      "body": "**The finding · Share of runs decided from memory · three briefs with a goal**\n\n| Opus 5 | Opus 5.5 |\n| --- | --- |\n| 45% | 52% |\n\nFrom Opus 5 to Opus 5.5, the host most runs picked for an app and brief stayed the same 65% of the time. Between two passes of Opus 5.5, it stayed the same 88% of the time.\n\nAI search is changing how people discover software. Coding agents add a different kind of discovery. They don't just recommend tools. They choose them and build with them.\n\nI wanted to know what drives those choices. Does the agent search the web, read the docs and compare alternatives? Or does it rely on what the model already knows?\n\nTo find out, I gave Claude Code 11 real apps and four different tasks, on two versions of Opus. Each task was one sentence, called a brief. One brief said: \"Get this app off localhost with the least ongoing effort from me.\" I tracked every step, from the agent's first tool call to the host it chose.\n\nFour things stood out:\n\n- **Memory comes first.** In about half the runs, the agent chose a host without searching the web or opening a single page.\n- **Docs come second.** When it did go online, it usually read the docs of the host it had already chosen.\n- **Price makes it compare.** With Opus 5.5, asking for the cheapest option was the only thing that sent the agent to rival hosts' pages.\n- **One model update changed the picks.** After the move from Opus 5 to Opus 5.5, its usual pick changed for about a third of the app and brief pairs.\n\n## 01 Why hosting\n\nA host is a platform that runs your app for you, like Render or Fly.io. Hosting turns out to be a good place to watch an agent choose. There are three reasons:\n\n- **The choice is real.** Lots of hosts do the same job.\n- **The choice is visible.** The agent writes its pick into a config file, like `render.yaml` for Render.\n- **The choice is the agent's.** No brief names a host.\n\nThree of the apps shipped with a hosting guide or a deploy button. Those would have made the choice for the agent. So they were taken out before any run.\n\nHost names show up below only as evidence. This isn't a ranking of hosts. It's a study of how an agent chooses.\n\n## 02 How the runs worked\n\nA run is one attempt: one app, one brief, one model. The agent could search the web, open pages and write any file it liked. It just couldn't deploy. A stub blocked every deploy. That kept the test about the choice, not about whether a deploy worked.\n\nEach app ran three times under each brief, with each of two models:\n\n11 apps × 4 briefs × 3 repeats = 132 runs per model × 2 models = **264 runs**\n\n| Part | What it was |\n| --- | --- |\n| Agent | Claude Code with its built-in web search and page fetch tools. |\n| Web tools | Named at the start, but the agent had to load one before using it. |\n| Older model | Opus 5 on Claude Code 2.1.260 at high effort, 8 to 10 September. |\n| Newer model | Opus 5.5 on Claude Code 2.1.280 at high effort, 15 to 16 October. |\n| Deploy stubs | Fake deploy tools for most hosts that all refused to run. |\n| Container | A fresh one per run with Postgres and Redis, deleted afterwards. |\n| Order | Random, with no grouping by app or by brief. |\n\n- **Least effort**: \"Get this app off localhost with the least ongoing effort from me.\"\n- **Production**: \"This needs to run in production for real users: a custom domain with HTTPS, restarts if it crashes, and logs I can read.\"\n- **Cheapest**: \"I want to put this app online as cheaply as possible - ideally free, and if not, the lowest predictable monthly bill.\"\n- **Open**: \"Deploy this repository. You decide where and how.\"\n\n| App | Shape | Stack |\n| --- | --- | --- |\n| [`whoami`](https://github.com/albertogrande/whoami/tree/1ce75d01b6978863647da42557a707a479da3a51) | A service with no state | Go |\n| [`fastapi-postgres`](https://github.com/albertogrande/fastapi_production_template/tree/8e82353ff3430c77fb77ead7de0d680f0229a667) | A service with a managed Postgres | FastAPI |\n| [`streamlit-llm`](https://github.com/albertogrande/llm-examples/tree/7cf260745675e41f92e7436d3777aafdab9dad9f) | A service that calls an external API | Streamlit |\n| [`langchain-nextjs`](https://github.com/albertogrande/langchain-nextjs-template/tree/febfa2f69bc94312d61794b6474246d76605e0f1) | A full-stack web app | Next.js |\n| [`healthchecks`](https://github.com/albertogrande/healthchecks/tree/69dbd2a0801dfbcfcd554af8cbbdaaab0b39e2a3) | A stateful app that sends on a schedule | Django, Postgres |\n| [`linkding`](https://github.com/albertogrande/linkding/tree/0c0f7b68a6d34ab5acecca24ff0cfad610da2e41) | A self-hosted app with SQLite on disk | Django |\n| [`mcp-streamable-http`](https://github.com/albertogrande/mcp-streamable-http-typescript-server/tree/719131a8ff0f0727b2770bd4bd9d7eb858387efe) | A long-running HTTP server | MCP, TypeScript |\n| [`discord-bot`](https://github.com/albertogrande/simple-discord-js-bot/tree/19d677a80b55c17b952747d602e135a033739536) | A worker with no port | discord.js |\n| [`astro-blog`](https://github.com/albertogrande/blog-tutorial-demo/tree/b3e12cc12d3292eff8c1237fb14f9b93f31ec33e) | A static content site | Astro |\n| [`hourly-digest`](https://github.com/albertogrande/hourly-digest/tree/157f6c86126adcc227036a3a5f5579d80c77e7da) | A scheduled job with no HTTP | Node, Postgres |\n| [`voting-app`](https://github.com/albertogrande/example-voting-app/tree/63e9150ca17af4ed05880d4245e486481f73fcb4) | A five-service app moving off Docker | Python, Node, .NET, Redis, Postgres |\n\n**What counts as the pick**\n\nThe pick is the host whose config file the agent wrote. No config file? Then it's the host in the deploy command the agent wrote. Neither? Then it's the host the agent named as its choice in its last message.\n\nA script labelled every run first. Then Claude read every run and took a second look wherever the script wasn't sure. When the two disagreed, the label follows what the run's record shows.\n\nNo person has checked these labels.\n\n## 03 Half the time, the agent looks nothing up\n\nFirst, a definition. A run decided **from memory** when the agent didn't search the web, open a page or fetch one with a shell tool like `curl`. It could still read the app's own files and anything else on its machine. That includes the folder of deploy stubs.\n\nOn the three briefs with a goal, Opus 5.5 decided from memory in 52% of its runs. Opus 5 did it in 45%. That's about half the time, for both models. Select any square in the chart below to see that run.\n\n**Fig. 1 · All 264 runs · how each run decided**\n\n| Brief | Opus 5 from memory | Opus 5.5 from memory |\n| --- | --- | --- |\n| Least effort | 67% (22 of 33) | 73% (24 of 33) |\n| Production | 48% (16 of 33) | 55% (18 of 33) |\n| Cheapest | 21% (7 of 33) | 27% (9 of 33) |\n| Open | 91% (30 of 33) | 97% (32 of 33) |\n\n1. **Cheapest, Opus 5: 21%.** The lowest share from memory on a brief with a goal. Asking for the lowest cost sent the agent to the web most often.\n2. **Open, Opus 5.5: 97%.** With no goal in the brief, the agent almost never looked anything up. Most open runs set the app up inside the container, not on an outside host.\n\n**What this means for AEO.** For many tasks, the pick comes from what the model already believes about a product. No web page can change that if the agent never opens it.\n\n## 04 Docs confirm the pick. Price makes it compare.\n\nWhen Opus 5.5 did go online on the three briefs with a goal, it rarely compared hosts. In 31 of the 38 runs that read a host's pages, its first step online opened a page of the host it went on to pick. Usually it read how to set that host up. Then it wrote the config file.\n\nPrice was the one thing that made it compare. On the cheapest brief, Opus 5.5 read the pages of two or more hosts in 12 of 33 runs. On the other briefs, it never read a rival host's page.\n\n**What this means for AEO.** Your docs mostly serve an agent that has already chosen you. Your pricing page is where it weighs you against rivals.\n\n**Example: run 408, step by step**\n\nHere's a run where a paid extra decided the pick. The app was a Discord bot, under the cheapest brief. The agent first looked at Google Cloud's free VM. Then it checked a detail that's easy to miss: whether Discord's servers have IPv6 addresses.\n\nThey don't. Without IPv6, the free VM needs a paid IPv4 address. Add that cost, and the free VM isn't the cheapest option any more.\n\nSo the agent picked Fly.io, at about $2.19 a month. The list shows the pages it read, in order. Then it shows the first deploy file it wrote.\n\n**Fig. 2 · Run 408 · discord-bot · cheapest · Opus 5.5 · its reads, then its first deploy file**\n\n5. `WebFetch` https://fly.io/docs/about/pricing/\n6. `WebFetch` https://cloud.google.com/free/docs/free-cloud-features\n7. `WebFetch` https://docs.railway.com/reference/pricing/plans\n8. `WebFetch` https://docs.fly.io/about/pricing\n9. `WebFetch` https://docs.cloud.google.com/free/docs/free-cloud-features\n10. `WebFetch` https://cloud.google.com/vpc/network-pricing\n11. `WebSearch` Google Cloud free tier e2-micro external IPv4 address charge \"free tier\"\n12. `WebFetch` https://cloud.google.com/vpc/pricing-announce-external-ips\n20. `Bash` Writes the Dockerfile, the first deploy file\n\n> “I'm going with Fly.io: one small always-on machine, no platform fee, about $2.19/month.”\n\n## 05 Being named is not being picked\n\nMost AEO tools count how often AI answers name a brand. That's called share of voice. Here, a host counts as **named** when the agent mentions it in its messages during a run. The host it picks always counts as named.\n\nThe agent often named several hosts in a run. Then it picked just one. The chart shows how often each host was named and how often it was picked, on the three briefs with a goal. Railway is the clearest case. Opus 5.5 named it in nearly a quarter of those runs and never picked it once.\n\n**Fig. 3 · Named against picked · three briefs with a goal**\n\n| Platform | Opus 5 named | Opus 5 picked | Opus 5.5 named | Opus 5.5 picked |\n| --- | --- | --- | --- | --- |\n| Render | 79% | 32% | 66% | 45% |\n| Fly.io | 72% | 35% | 61% | 26% |\n| Vercel | 42% | 3% | 25% | 3% |\n| Railway | 37% | 3% | 23% | 0% |\n| Google Cloud Run | 31% | 2% | 15% | 1% |\n| Cloudflare | 20% | 7% | 14% | 3% |\n\n**What this means for AEO.** A mention gets you onto the shortlist. It doesn't make you the pick.\n\n## 06 The favourite depends on the task and the model\n\nEach brief asked for something different. The agent matched a host to each one. For both models, the least-effort brief and the production brief each had a clear favourite. The cheapest brief didn't have one.\n\n**Fig. 4 · Most-picked platform for each brief**\n\n| Brief | Opus 5 | Opus 5.5 |\n| --- | --- | --- |\n| Least effort | Render 52% (next: Fly.io 18%) | Render 73% (next: Netlify 12%) |\n| Production | Fly.io 61% (next: Render 24%) | Fly.io 55% (next: Render 33%) |\n| Cheapest | Fly.io 27% (next: Render 21%) | Render 30% (next: Fly.io 24%) |\n| Open | No outside host 82% (next: Fly.io 12%) | No outside host 58% (next: Fly.io 21%) |\n\nTo compare picks, this study uses the **majority pick**. That's the host at least two of an app's three runs picked under one brief. Run the same model twice and about 9 in 10 majority picks come out the same. Switch from Opus 5 to Opus 5.5 and only 65% do. The new model changed far more picks than a second run of the same model does.\n\n**What this means for AEO.** A host can be the favourite for one task and not for the next. And a new model can change both.\n\n**Model or Claude Code?**\n\nThe update also came with a new version of Claude Code. So which one changed the picks? To find out, Opus 5 ran again on the new version, once for each app and brief.\n\nOf the 11 app and brief pairs whose majority pick had changed, it picked the old host in 8 and the new host in 3. So most of the change comes from the model, not from Claude Code.\n\nThe control only ran once per pair. That's why its results are counts, not rates.\n\n## 07 What to do about it\n\n| Move | Why | Rests on |\n| --- | --- | --- |\n| Give an agent your buyers' tasks and write down what it picks. | Each task has its own favourite host. | [Section 6](https://thebeat.dev/research/memory-first-docs-second/#s6) |\n| Count how often you're picked, not how often you're named. | A mention only gets you onto the shortlist. | [Section 5](https://thebeat.dev/research/memory-first-docs-second/#s5) |\n| Treat what models already believe about you as a channel. | Half the time, the agent reads nothing before it picks. | [Section 3](https://thebeat.dev/research/memory-first-docs-second/#s3) |\n| Put the free tier, the monthly cost and any paid extras on your pricing page. | Price is the only thing that makes the agent compare hosts. | [Section 4](https://thebeat.dev/research/memory-first-docs-second/#s4) |\n| Keep your config reference complete. | Once it has chosen you, the agent reads it to set you up. | [Section 4](https://thebeat.dev/research/memory-first-docs-second/#s4) |\n| Measure again after every model release. | One update changed about a third of the majority picks. | [Section 6](https://thebeat.dev/research/memory-first-docs-second/#s6) |\n\n## 08 What this study cannot tell you\n\n- It only covers one agent: Claude Code.\n- I chose the 11 apps by hand.\n- Each brief only had one wording.\n- No person checked the labels on the picks.\n- The agent reading a host's docs first shows order. It doesn't show cause.\n\n## 09 Open questions\n\n- Would an `AGENTS.md` file or a deploy button that names a host change the pick?\n- Would an installed command-line tool or MCP server for one host change it?\n- Which public material shapes what the next model believes about hosts?\n- Would a clearer pricing page change what the agent compares?\n- Would a fuller config reference change the config the agent writes?\n\n## 10 Check it yourself\n\n- Select any square in the chart in [section 3](https://thebeat.dev/research/memory-first-docs-second/#s3) to open that run.\n- Run `npm run figures` in the [benchmark repo](https://github.com/albertogrande/coding-agent-hosting-benchmark) to print every number here.\n- Run `npm run export-site` in the [benchmark repo](https://github.com/albertogrande/coding-agent-hosting-benchmark) to rebuild runs.json.\n- runs.json cuts each tool call to its first 300 characters.\n- The [benchmark's readme](https://github.com/albertogrande/coding-agent-hosting-benchmark#readme) lists when each comparison rule was committed.\n\n## A Appendix: every run\n\nEvery run, with its tool calls, is in [runs.json](https://thebeat.dev/research/memory-first-docs-second/runs.json). Each transcript is at https://thebeat.dev/research/memory-first-docs-second/transcripts/<run>.json."
    }
  ]
}