Why AI Can't See My Content, and How to Fix It

If AI cannot see your content, the cause is almost always one of four things: a crawler is blocked, the text only appears after JavaScript runs, access is gated, or the page is too unclear to be retrieved. You can diagnose all four in under an hour by requesting your page the way a bot does and reading what comes back.

This page covers the AI side of invisibility. For pages Google ignores, see why search engines ignore your content.

What are the main reasons AI cannot see a page?

Cause	Symptom	How to confirm	Fix
Bots blocked in robots.txt	A named AI bot is disallowed	Read /robots.txt; run the AI crawlability checker	Allow the bots you want
Blocked outside robots.txt	Bots get 403 or a challenge page	Server or CDN logs, curl test	Adjust firewall or bot-management rules
Text needs JavaScript	Raw HTML is nearly empty	curl the URL, view source	Server-render or pre-render key text
Gated content	Login, cookie or geo wall	Load the page in a private window	Expose the core answer publicly
Not indexed	Page missing from Google	Search Console	Fix indexing, see sibling article
Unclear content	Visible but never cited	Prompt tests	Rewrite for answer-first, add evidence

Are you blocking AI bots without knowing it?

Probably check this first. Robots.txt is only one layer. CDNs, WAFs and plugins can block bots independently, and some do it by default. Cloudflare, for example, changed its default to block AI crawlers unless they pay as of July 2025, and your hosting stack may have similar settings.

To test, request a page with an AI bot’s user agent:

curl -I -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://example.com/your-page

A 200 means that layer lets the user agent through. A 403, 429 or a challenge page means something is blocking. Note this tests only the user agent string, since real bots also come from published IP ranges that some firewalls check.

Which bots to allow, and what each does, is covered in how to optimize for AI crawlers.

Is your content hidden behind JavaScript?

Many AI crawlers fetch raw HTML and do not run scripts. Vercel’s December 2024 analysis found that none of the major AI crawlers it measured rendered JavaScript, with Gemini (using Google’s infrastructure) and Applebot as exceptions. That was a point-in-time study, so verify with the bots that matter to you.

Test it:

curl -s https://example.com/your-page | grep -i "a phrase from your key paragraph"

If nothing returns, the text is not in the initial HTML. Server-side rendering, static generation or pre-rendering fixes it. Also check for answers that live only in tabs, accordions loaded on click, images, or PDFs without a text layer.

Is the content gated or fragmented?

• Login and paywalls. Bots usually cannot pass them. Expose a public summary that contains the actual answer.
• Cookie and consent walls that replace the page with a banner on first load.
• Geo redirects that send bots to a thin regional page.
• Fragment URLs or infinite scroll where the content lives behind interactions.

Is the page too unclear to retrieve?

Access can be perfect and the page still loses. Signs:

• The topic is split across several thin pages, none of which answers the question completely.
• The opening is a story, not an answer.
• The brand or product name varies from page to page.
• No dates, no sources, no author.

Fix with the answer-first method and consistent entity names (see entity SEO).

Why might AI see you but still not cite you?

Different failure, different fix. Engines can reach a page and choose others because competitors have clearer passages, fresher facts, or more third-party mentions. Look at which domains get cited for your target prompts and compare their format and depth with yours.

GeoGen’s citation data shows which domains and URLs the engines cite for each tracked prompt, so you can see who is winning those slots rather than guessing.

What is a 30-minute fix plan?

1. Minutes 0 to 5. Open /robots.txt. Confirm the AI bots you want are not disallowed, including the wildcard User-agent: * group.
2. 5 to 10. Run the AI crawlability checker.
3. 10 to 15. curl your three most important pages with a bot user agent. Note any non-200 responses.
4. 15 to 20. curl the same pages and search the raw HTML for key text.
5. 20 to 25. Check Search Console’s page indexing report for those URLs.
6. 25 to 30. Test five prompts in two AI engines and note whether you are mentioned or cited.

Log what you find, fix technical blockers first, and recheck. The fuller scoring method is in the GEO content audit checklist.