What 815 Websites Get Wrong for AI Crawlers (2026 Data)

As of 16 October 2026, Faro's AI crawler readiness scans of 815 websites found that 9.1% hide their main content behind JavaScript, 12.9% of 743 testable sites return a soft 404, and 97.2% do not link an llms.txt file from the homepage. Only 2.5% block OAI-SearchBot. Below: every check, the industry splits, and what to fix first.

How many websites hide their content behind JavaScript?

As of 16 October 2026, 74 of the 815 sites Faro checked (9.1%) sent fewer than 30 words of homepage body text in the HTML their server returned. Everything else appeared only after JavaScript ran. A crawler that reads raw HTML and skips scripts gets something close to a blank page from those sites.

Many of the 74 failing sites sent no readable body text at all in their first response; the passing sites ranged from 30 words to several thousand.

Search engines have handled this for years with a rendering step. Google's own documentation says server-side or pre-rendering is still a good idea because it makes a site faster for users and crawlers, and because not all bots can run JavaScript. An AI fetcher that reads raw HTML and stops there sits in that second group. If you want the mechanics of what they do and do not execute, our breakdown of whether AI crawlers can run JavaScript covers it in detail.

The scan does not record why a site renders client-side, and the fix depends on the stack. The test is the same everywhere: fetch the page with curl, or turn JavaScript off, and read what is left.

Which industries hide the most content behind JavaScript?

Only 6 industries in the Registry had 30 or more sites with a definite result on this check, so every industry figure below rests on a small sample and should be read as directional.

• Retail: 8 of 31 sites (25.8%) failed, the highest rate measured. Directional, sample of 31.
• Disaster restoration: 5 of 31 sites (16.1%). Directional, sample of 31.
• News and media: 2 of 30 sites (6.7%). Directional, sample of 30.
• Marketing agencies: 1 of 30 sites (3.3%), and medspas 1 of 31 (3.2%). Directional.
• Food and recipes: 0 of 38 sites failed. Directional, sample of 38.

If you run an online store, the AI visibility guide for ecommerce covers the storefront-specific checks.

How common are soft 404s on real websites?

As of 16 October 2026, 96 of 743 testable sites (12.9%) returned 200 OK for a URL that should not exist. Faro requested a made-up path on each site and read the status code. Another 72 sites blocked or timed out on that request, so they were excluded rather than counted as passes or failures.

In a browser the page looks fine, but the status line tells every client the URL is real. Google treats the same pattern as an error worth flagging. Its crawling documentation says that if a 2xx page's content suggests an error, an empty page or an error message, Search Console will show a soft 404 error. The correct answer for a missing resource is the 404 Not Found status, which MDN defines as the server indicating it cannot find the requested resource.

Some failing sites served a 200 on the made-up path directly; others redirected it to the homepage or an error page that answered 200. The causes and framework-level fixes are in our guide to soft 404s and why they confuse AI agents.

Only 1 industry, food and recipes, had 30 or more definite results on this check: 1 of 36 sites (2.8%) failed, a directional figure on a sample of 36.

Do sites with an llms.txt file actually link to it?

Mostly not. As of 16 October 2026, 792 of 815 sites (97.2%) had no link to an llms.txt file from the homepage. Most of those had no file at all, but the gap holds even for sites that made one: of 255 sites with a confirmed llms.txt, 232 (91.0%) never pointed to it.

The llms.txt proposal describes a markdown file placed on a website to help agents use it, offering brief background, guidance and links to more detailed files. Faro's broader file check, which has run longer and across more of the Registry, found that 71.5% of 2,345 sites with a definite result had no llms.txt at all.

A file at the root that nothing references depends on an agent deciding to look for it. A link in the homepage head, a footer link, or a Link response header puts it in the path of an agent that is already reading the page. Any one of the 3 passes the check.

An llms.txt file is cheap to create and link, and it does not replace readable HTML. Our guide on how to create an llms.txt file walks through the format, and the llms.txt generator builds a first draft from your site.

How many sites block AI crawlers in robots.txt?

Fewer than you might expect. As of 16 October 2026, 20 of 815 sites (2.5%) blocked OAI-SearchBot, the crawler behind ChatGPT search. Across a larger set of 2,454 sites, 13.4% failed the GPTBot check and 12.6% failed the ClaudeBot check.

OAI-SearchBot is a separate user agent from GPTBot, and the scans found it blocked far less often. A site can block one and allow the other; robots.txt treats each user agent on its own terms.

The GPTBot and ClaudeBot checks fail a site in 2 ways: a robots.txt rule that disallows the bot, or an edge network that refuses or challenges a request identifying as the bot while a normal request loads. For a line-by-line look at each search and user-triggered crawler, see our robots.txt guide to OAI-SearchBot, Claude-SearchBot and Claude-User, and paste your own file into the robots.txt checker to see what each bot is allowed to fetch.

Which AI crawler readiness checks fail most often?

By failure rate, the most common problem is not access at all. As of 16 October 2026, 75.0% of 2,040 sites with a definite result failed the answer-first check: the opening text of the homepage or pricing page did not say plainly what the business does, or the pricing page opened without a number.

The check reads the first few hundred characters of visible text. A tagline and a navigation list can fill that space before a single sentence explains the product.

The XML sitemap check failed on 21.7% of 2,420 sites. Faro looks for a sitemap at /sitemap.xml, at /sitemap_index.xml, or referenced from robots.txt, and fails the site only if none of the 3 turns up.

Every check, side by side

"Sites checked" counts only sites with a definite pass or fail. The 4 newer checks ran from 30 September 2026, so they cover fewer sites.

Check	What it tests	Sites checked	Share failing
llms.txt linked	The homepage points to an llms.txt file through a link or a Link header	815	97.2%
Answer first	The homepage and pricing page state what you do, and a price, in their opening text	2,040	75.0%
llms.txt exists	A file is served at /llms.txt	2,345	71.5%
XML sitemap	A sitemap is found at a standard path or in robots.txt	2,420	21.7%
GPTBot allowed	GPTBot is not blocked by robots.txt or at the network edge	2,454	13.4%
Soft 404	A made-up URL returns a real 404 status	743	12.9%
ClaudeBot allowed	ClaudeBot is not blocked by robots.txt or at the network edge	2,454	12.6%
Content in raw HTML	The homepage HTML carries at least 30 words of body text before JavaScript runs	815	9.1%
OAI-SearchBot allowed	OAI-SearchBot is not blocked in robots.txt	815	2.5%

Source: Faro Registry scans, 28 August to 16 October 2026 (newer checks from 30 September 2026).

How we collected this data

Every figure comes from sites in the Faro Registry, scanned by Faro's public scanner. The longer-running checks cover scans from 28 August to 16 October 2026. The 4 newer checks (content in raw HTML, soft 404, llms.txt linked, OAI-SearchBot allowed) went live on 30 September 2026 and cover scans from then to 16 October 2026. Each site counts once, using its most recent scan.

This is not a random sample of the web. The Registry holds the sites that have been scanned with Faro's scanner, so it reflects whoever chose to scan, not the web at large. Treat the rates as a description of this set, not an estimate for every website.

The pass rules, in plain words:

• Content in raw HTML: fails if the homepage HTML the server sends holds fewer than 30 words of body text.
• Soft 404: fails if a made-up URL ends on a 200 status, directly or after a redirect.
• llms.txt linked: passes if the homepage links to /llms.txt in the head, in the page body, or through a Link header.
• OAI-SearchBot, GPTBot, ClaudeBot: fails if robots.txt disallows the bot; the GPTBot and ClaudeBot checks also fail if the network edge refuses a request identifying as that bot.
• llms.txt exists and XML sitemap: pass if the file is found at its standard location (or, for the sitemap, referenced from robots.txt).
• Answer first: fails if the opening visible text does not say what the business does, or the pricing page opens without a price.

When a firewall blocked the scanner or a request timed out, the result was marked uncertain and left out of both the numerator and the denominator. Industry figures appear only where 30 or more sites had a definite result, and every figure under 50 sites carries its sample size.

What to fix first

Order the work by how badly a failure stops an agent, then by how often it shows up.

1. Put your content in the HTML. It failed on 9.1% of sites, but when it fails, a non-rendering crawler sees nothing. Render public pages on the server or at build time.
2. Remove accidental crawler blocks. GPTBot and ClaudeBot failed on 13.4% and 12.6% of sites. Check robots.txt and your edge network's bot settings, and decide per crawler on purpose.
3. Front-load the answer. At 75.0%, this is the most common failure, and the fix is a rewrite of 2 opening paragraphs.
4. Return a real 404. 12.9% of testable sites got this wrong. A routing change usually fixes it.
5. Publish a sitemap. 21.7% of sites had none Faro could find.
6. Link your llms.txt. If you have the file, one line in the head closes the 91.0% gap. It is the lowest-stakes item here and the quickest.

The fastest way to know which of these apply to you is to scan your site with the AI visibility checker.

FAQ

Can AI crawlers read my website?

Fetch your homepage with curl or with JavaScript turned off. If your main copy is there, crawlers that skip scripts can read it. In Faro's scans as of 16 October 2026, 9.1% of 815 sites failed that test.

What is a soft 404?

A soft 404 is a missing page that answers 200 OK instead of 404 Not Found, often by showing a friendly error or redirecting to the homepage. Faro found it on 12.9% of 743 testable sites as of 16 October 2026.

Should I block GPTBot or OAI-SearchBot?

That is a business decision, and robots.txt lets you set each crawler separately. OAI-SearchBot is the crawler behind ChatGPT search, while GPTBot is OpenAI's training crawler. In Faro's scans, 2.5% of 815 sites blocked OAI-SearchBot.

Is this data a random sample of the web?

No. It covers sites in the Faro Registry, scanned by Faro's public scanner between 28 August and 16 October 2026. Uncertain results were excluded, and every industry figure on fewer than 50 sites is labeled directional.

In short

The most common AI crawler readiness failures in the Faro Registry are not dramatic blocks. They are buried answers, missing sitemaps and unlinked llms.txt files, plus a minority of sites that hide content behind JavaScript or tell agents a missing page is real. The highest-stakes fix is making sure your content is in the HTML at all.

Run the free AI visibility checker and see which of these 9 checks your site passes.

Cite this data: "What 815 Websites Get Wrong for AI Crawlers (2026 Data)", Faro, October 2026, https://byfaro.ai/blog/ai-crawler-readiness-data-2026.