Bot analytics

Answer metrics tell you what AI systems say. Bot analytics tells you whether they ever fetched your pages in the first place. The data comes from your own infrastructure, so it is available immediately and independently of any collection cycle.

How it works

You generate an ingest key and configure your platform to forward request metadata to the ingest endpoint. Incoming requests are matched against a registry of known crawlers and appear in the Bots tab.

Only request metadata is sent: what was requested, when, and how your server responded. No page content and no request bodies.

Setting it up

1. Open the Bots tab and generate an ingest key.
2. Copy it immediately — it is shown once and cannot be retrieved later. Generating a new key invalidates the old one.
3. Pick your platform and apply the setup it shows.

The app provides ready configurations for Cloudflare Workers, Vercel (log drain or middleware), WordPress, nginx and AWS, plus a generic option for anything else. Pick your platform and copy what it shows — the connection details come with it.

Choosing an integration point

Edge or CDN (Cloudflare, Vercel) — the most complete picture, because it sees requests that never reach your origin. Recommended where available.

Web server (nginx) or application (WordPress) — sees what reaches the origin. Fine, as long as you know that anything served from cache at the edge will be missing.

Custom — for anything else. Batch requests where you can rather than sending one call per hit.

Practical notes

• Send everything, filter later. Excluding paths at source permanently removes data; the app can filter afterwards.
• Do not block the response. Forward asynchronously so a slow ingest never delays a visitor. The supplied snippets already do this.
• Cover every hostname you want measured, including www and non-www if both serve content.
• Treat the key like a password. Rotate it if it may have leaked, then update every integration point.

Verifying

After deploying, visit your site and check the Bots tab. Your own visit will not appear — only identified crawlers are counted — but crawler traffic should show within a short time.

If nothing arrives:

1. Confirm the integration is deployed and running on the right hostname.
2. Confirm the key is exactly the one that was generated, in the right header.
3. Check for errors from the forwarding code in your platform's logs.

Related

• Bots tab — reading the data
• Robots.txt audit — whether crawlers are allowed in
• Technical checks — what they find when they arrive