A field note, with the sample size stated

The infrastructure companies are building the agent web in public right now. Cloudflare has shipped the metering layer. Google is working on its own. Within a year, every hosting platform will offer a dashboard that tells you which AI agents read your site.

I built ours in May, put the public interface up in June, and have been logging every read since August. This is not a prediction post. It is what twenty-two days of actual agent traffic looked like, and the one thing it taught me that I think those dashboards are going to get wrong.

Short version: six of the seven AI agents that read our machine-readable files never said they were AI agents. They arrived wearing ordinary browser strings. Nothing in the request identified them. Only the network did.

01 What is being built

The Web Is Growing a Second Set of Doors

The old bargain was simple. You let search engines crawl your site, they sent you a human, and you monetised that human's attention with an ad, an email capture, or a sale.

An AI agent breaks the second half of that trade. It reads your page, extracts the answer, hands it to the person, and the person never arrives. The content still created the value. The site lost the visit.

The publisher grievance about that is real but it is the small version of the story. The larger one is that agents will consume the web the way software consumes infrastructure — requesting resources, calling tools, comparing options, retrieving data — and infrastructure gets metered, not advertised against. A human would never pay a fraction of a cent to read a recipe. A machine does not care.

Cloudflare has already shipped most of that idea. AI Crawl Control gives site owners visibility into which AI crawlers are hitting them and the ability to allow or block each one. Pay Per Crawl lets a site charge for access. And the Monetization Gateway generalises it past crawling entirely — charge any caller for any resource behind Cloudflare, from an API response to an MCP tool call, settled over x402, the open protocol built on the long-dormant HTTP 402 "Payment Required" status code.

Strip away the plumbing and the shape is clear. The human web monetised attention. The agent web meters useful resources, and the request itself becomes the transaction.

Which means the first question is no longer "how do I rank?" It is "what do I have that an agent needs badly enough, often enough, to come back for?"

There is a stack forming under that: raw messy web, cleaned into structured data, exposed through a door an agent can actually use, with rules about what is free and what is paid, and analytics telling you which resources are worth anything. Cloudflare is selling the last two layers. Google will sell its own. The first two are still yours to build, and almost nobody has.

02 What we shipped, and when

We Did Not Wait for the Platforms

I am dating this precisely, because in a field moving this fast the dates are the argument:

1

May 2026 — the machine-readable layer

Our site started generating a structured markdown edition of itself at build time — every page, restated for a reader that wants the substance rather than the layout. 152 documents today. This is the "clean fuel" layer, and it is the one you cannot buy.

2

7 June 2026 — the public agent interface

A discovery manifest and a public page describing what an agent can do here: read our services, check availability, request a quote, book a call. Programmatically, without a form. It has been live and publicly verifiable since that date at /agent-access.

3

10 August 2026 — measurement

Serving those agent-facing files through our own gateway instead of the CDN, so every read is recorded rather than silently cached. Responses are deliberately uncacheable: a cached response is an unlogged read, and the measurement was the entire point.

4

16 August 2026 — identifying agents by network

Six days after switching on measurement, the measurement told us it was wrong. That is the rest of this post.

03 The finding

The Agents That Matter Most Do Not Announce Themselves

Every AI crawler dashboard I am aware of identifies agents the same way: by the user-agent string in the request. GPTBot, ClaudeBot, PerplexityBot, CCBot. It works, it is cheap, and it catches everything that is willing to be caught.

The trouble is that agent traffic is not one thing. It is three, and they are worth wildly different amounts to you:

Training
A corpus crawler taking your text to train on. You may or may not want this. It is not a customer.
Search
A vendor building its index so it can cite you later. Useful, indirect, delayed.
User
A real person, right now, asked a model a question and the model came to your site to answer it. This is the whole reason to build any of this. It is the agent-era equivalent of a click, and it is the one that converts.

Now the problem. On 25 August, six reads arrived from ordinary-looking browsers. Nothing in the user-agent string suggested anything but a person with a laptop. Our own classifier recorded all six as not an AI agent. A dashboard would have shown nothing.

Looking up who actually owned the addresses told a completely different story: 

66.249.93.205  ->  google-proxy-66-249-93-205.google.com
74.125.208.33  ->  google-proxy-74-125-208-33.google.com
66.102.9.2     ->  google-proxy-66-102-9-2.google.com

google-proxy is not googlebot. Googlebot crawls the web to build an index, and it says so in every request. A Google proxy address is Google fetching a page on a specific person's behalf — a model retrieving your page because someone asked it something.

The highest-value category of agent traffic is precisely the category that does not declare itself.

That is not an accident and it is not deception. A crawler building an index has every reason to identify itself — it wants to be allowed. A model fetching one page for one user is behaving like a browser because, functionally, it is one.

So we added a second identification method: when a request declares nothing, look up who owns the address, then confirm that answer forwards before believing it. Two rules keep it honest. A reverse lookup alone proves nothing, because whoever controls an address controls what it claims — so it only counts if the forward lookup agrees. And generic cloud hostnames are deliberately not matched, because those identify rented server capacity, not an AI vendor, and counting them would inflate the exact number the whole system exists to measure.

Every record keeps the two grades of evidence apart: self-declared, or inferred from the network. Anything published from that table can say which it is. This paragraph is an example of it doing its job.

04 The numbers, in full

Twenty-Two Days, One Site, Thirty-Nine Reads

Here is everything, including the parts that undercut the headline. This is a pilot on a single site, not a fleet study, and I would rather say so than let you assume otherwise: 

window            14 Aug - 4 Sep 2026   (22 days)
scope             1 site, 152 published documents
total reads       39        from 26 unique addresses
identified as AI  7
  by user-agent   1         CCBot (training)
  by network      6         Google proxies (user)
everything else   32        SEO crawlers, audit tools, browsers
paths requested   llms.txt: 39          everything else: 0

Three honest caveats, because a number without its limits is just a claim:

  • Six reads is not six events. All six Google proxy reads landed inside a four-minute window. That is realistically one person asking one question, not six independent visits.
  • The unclassified 32 are not hidden agents. They are SEO vendor crawlers, site audit bots and browsers — including our own audit tool. The classifier is not missing a silent majority; the traffic is genuinely small.
  • Training ingestion is invisible to everyone. No method here or anywhere else observes a model that learned from your site months ago. Nobody can measure that. Be suspicious of anyone selling it.

And the finding I did not expect, which is the most immediately useful thing in this post: every single request was for llms.txt. Not one agent, in twenty-two days, went deeper into the structured tree behind it. If you are deciding what to build for agents, build the single index file first and be very slow about building anything more elaborate. The ambitious part is not being read yet.

05 What we have not built

The Meter Is the Part We Left Alone

We built the machine-readable layer, the public interface and the measurement. We have not built payment, and that is the layer Cloudflare just shipped and Google is working on.

That is the right order, and it is worth saying plainly, because the temptation runs the other way. Metering is worthless until you know what is being consumed, by whom, and how often. A price on a resource nobody has measured is a guess with a decimal point on it. Twenty-two days of reads told us that one file is doing all the work and the rest is theatre — and that is a thing worth knowing before attaching a price to any of it.

The platforms will happily sell you the meter. They cannot sell you the thing worth metering. Whatever a business knows that is genuinely hard to assemble — pricing, availability, local conditions, the answer to the question a customer actually asks — that is the resource, and it has to be built by whoever holds it.

The meter is a commodity. The thing being metered is not.

The bottom line

If your AI analytics identify agents by what they call themselves, they will miss the only visitors who came because a customer asked about you.

One site, 22 days, 39 reads — and six of the seven agents in them arrived anonymous.

See Our Agent Interface Talk to Araptus

Sources

All traffic figures are first-party, read directly from our own read log for the window stated, and are reported with their sample size rather than extrapolated. Dates for our own work are taken from commit history. Descriptions of Cloudflare's products are drawn from Cloudflare's own announcements and documentation.