AI crawlers read your site once, without JavaScript

Every request to this site passes through a middleware function before any page renders. That's where we watch AI crawlers — because it's the only place they're visible.

Here is the mechanical part, and it changes how you should build pages:

AI crawlers don't execute JavaScript. Analytics scripts never see them. A tracking pixel never fires. The single-page app that hydrates your headline into the DOM after load presents them a shell. What they read is the HTML your server sends on the first response, once, and then they leave.

So the measurement has to live server-side. We built crawlwatch — a first-party beacon vendored into the middleware of every property we operate, this one included. It classifies the user agent (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest), and forwards the hit — path, agent, referrer — to a collector. No cookies, no fingerprinting, nothing a visitor would need to consent to: it counts robots.

What that instrument teaches, before any numbers:

• If a sentence matters to an answer engine, it has to be in the server-rendered HTML. Not behind a click, not injected after load.
• The pages crawlers hit are not the pages you'd guess. Middleware-level logging shows which routes each bot actually requests — that list, not your sitemap, is your real AI surface.
• robots.txt is the one lever they reliably respect. Everything else — llms.txt included — is speculation, and we label it that way.

We keep saying "measured" instead of quoting figures because this journal only prints numbers with a stored source behind them. When the crawlwatch dashboards for this property have a month of history, the follow-up post carries the counts.

Want the same read on your own site? Start with the audit.