The 403 problem: why AI crawlers cannot read a third of B2B sites

Key takeaways
  • Check robots.txt and your CDN's bot rules separately; they fail independently.
  • Pricing and integrations pages are the ones most often blocked.
  • Unblocking is usually a one-hour fix worth several visibility points.

Where the blocks come from

Three places: robots.txt disallow rules that were copied from a template, CDN bot-management defaults that treat AI crawlers as scrapers, and login walls on pages that should be public.

How to check

Run a site audit per URL and read the response code per bot. A 200 for Googlebot and a 403 for GPTBot on the same page is the signature of a CDN rule.

What to change

Allow the AI crawlers you want to be cited by, keep the ones you do not, and re-run the audit. Then watch the affected prompts for two weeks.

See what AI says about your brand
Six engines, a grade and the first three fixes. Free for 14 days.
Start for free