Robots.txt audit

robots.txt decides which automated clients may fetch your pages. AI crawlers are newer than most robots files, so it is common to find a site quietly blocking them through a rule written years ago for a different purpose — or through a blanket rule that never anticipated them.

This audit reads your robots.txt and reports, crawler by crawler, whether it can reach your content.

What it reports

Per-crawler access. For each known AI and search crawler: allowed, blocked, or not addressed. "Not addressed" means the file has no rule naming it, so it falls through to your default rules.

The rules that produced each verdict. Which directive applies to which crawler, so a surprising verdict can be traced to the line that caused it.

Recommendations. Concrete changes to your file, with the text to add or remove.

If your site has no robots.txt, the audit says so. That is permissive — everything is allowed — but it also means you have no control, and it is worth adding one deliberately.

Reading it

Blocked AI crawlers are the finding to act on. A blocked crawler cannot cite you under any circumstances, no matter how good your content is. This makes the audit the correct first stop whenever Visibility is near zero for reasons nothing else explains.

Check your defaults. A broad Disallow under a catch-all rule blocks crawlers you never considered. This is the single most common cause of unintentional blocking.

Blocking can be intentional. Some organisations deliberately keep AI crawlers out for licensing or policy reasons. That is a legitimate choice — but it should be a decision, and its consequence is that AI visibility work will not produce results.

Blocking assets matters too. Rules that block resources needed to render a page can leave a crawler with a partial view even when the page itself is allowed.

After changing your robots.txt

1. Deploy the change and re-run the audit to confirm the verdicts flipped.
2. Watch the Bots tab — crawler visits are the evidence that access actually changed.
3. Expect a lag before answers move. Access must lead to crawling, crawling to indexing, indexing to citation.

Limits

• robots.txt is a request, not enforcement. It tells well-behaved crawlers what to do; it is not a security control. Anything that must not be public needs real access control.
• It answers access, not usefulness. A permitted crawler still has to find a page worth reading — see Technical checks and Schema audit.

Related

• Bots tab — which crawlers actually arrive
• Report tab — where the audits live