Technical basics: let AI read your site

Check 1

Which AI crawlers should you allow in robots.txt?

Allow the search crawlers: OAI-SearchBot for ChatGPT search, PerplexityBot for Perplexity, plus Googlebot and Bingbot. Blocking training crawlers like GPTBot is a separate choice.

robots.txt is a small text file at the root of your site that tells crawlers which pages they may fetch (Google Search Central, Introduction to robots.txt). AI companies use several crawlers, and they do not do the same job:

“Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”

A safe starting point for most sites looks like this (adapt the Disallow lines to your private pages):

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Disallow: /admin/

Sitemap: https://www.yoursite.com/sitemap.xml

Sources: OpenAI, Overview of OpenAI crawlers, Perplexity, Perplexity crawlers, Google Search Central, Google's common crawlers (Google-Extended). Check your current file in seconds with the free AI crawler checker.

Watch out. Some security and CDN settings block AI bots for you, without touching robots.txt. If a checker says a bot is blocked and your robots.txt allows it, look at your firewall or CDN bot settings.

Check 2

Is your content in the HTML, or only in JavaScript?

AI crawlers read the HTML your server sends. If your text only appears after JavaScript runs in a browser, most of them see an empty page.

“The results consistently show that none of the major AI crawlers currently render JavaScript.”

In that same analysis of its network, GPTBot made 569 million fetches in a month and Claude 370 million, about 20% of Googlebot's 4.5 billion (Vercel, The rise of the AI crawler (December 17, 2024)). These bots matter, and they read raw HTML.

1. Open one of your key pages, right-click and choose View page source (not Inspect).
2. Search the source for a sentence from the middle of the page.
3. If you cannot find it, your content is added by JavaScript. Ask your developer for server-side rendering or static pages. Or run the free AI crawler URL inspector, which shows what GPTBot gets.

Check 3

Are your pages indexed by Google and Bing?

AI Overviews and AI Mode only show pages that Google has indexed, and Bing runs the other big search index. Being indexed is the entry ticket, not a bonus.

Google states that a page must be indexed and eligible to be shown with a snippet to appear as a supporting link in AI Overviews or AI Mode (Google Search Central, AI features and your website). Submit a sitemap, fix pages marked as excluded, and make sure no page you want quoted carries a noindex tag.

Connect both consoles to your site, then to Howseen: lesson 4.2 for Google Search Console and lesson 4.3 for Bing Webmaster Tools.

Check 4

What structured data should you add?

Add Organization on your home page, FAQPage where you have a FAQ, and Article on your guides. Structured data states facts in a format machines read without guessing.

“Structured data is a standardized format for providing information about a page and classifying the page content.”

Google uses structured data to understand the content of a page and the people and companies it mentions (Google Search Central, Introduction to structured data). For GEO, the useful part is clarity: your name, what you sell, your questions and answers, written in a standard format. Generate the code for free with the schema generator.

Check 5

Should you add an llms.txt file?

Yes, as cheap hygiene, not as a ranking lever. llms.txt is a proposal for a plain file that points AI agents to your most useful pages.

The llms.txt idea is described on llmstxt.org, The /llms.txt file as a proposal to help agents use a website. It takes ten minutes, it does no harm, and it forces you to list your key pages. Do not expect it alone to change your visibility. Create one with the free llms.txt generator.

Check 6

Which page basics still count?

One clear title, one meta description, one H1, a canonical tag, a contact page and fast responses. They help both Google and AI read your page correctly.

• A title that says what the page is about, and a meta description that says what you do and for whom.
• Exactly one H1 near the top, in plain words.
• A canonical tag that points to the preferred address of the page.
• Visible trust pages: contact, refund or return policy, privacy. They show people and machines that a real, reachable business stands behind the site.
• A fast server response, so crawlers do not give up on large pages.

In Howseen

How does Howseen check this for you?

The Technical Audit page checks your home page for these blockers, gives each issue a priority and the exact fix, and feeds the Technical lever of your GEO score.

1. Open Technical Audit in the left menu, under Optimize.
2. Read the Health score and the list of Issues detected. Each issue has a priority, the points you can gain, and a How to fix: line.
3. Fix the high priority issues first: noindex, content that needs JavaScript, AI crawlers blocked in robots.txt.
4. Click Re-run after your changes to confirm the blockers are gone.

The audit is included on every plan, Starter included.