Can ChatGPT read your store? Check your robots.txt in five minutes

If an AI assistant can’t read your site, it can only describe your brand from whatever it picked up elsewhere. The first thing to check is your robots.txt, the small file that tells crawlers what they may visit. It takes five minutes.

Step 1: open your robots.txt

Visit yourstore.com/robots.txt in a browser. You’ll see a list of User-agent lines followed by Allow or Disallow rules.

Step 2: look for AI crawlers

These are the names to search for. Each AI company publishes its crawlers; check their documentation for the latest list.

• GPTBot and OAI-SearchBot (OpenAI). GPTBot collects content for training; OAI-SearchBot is used for search results inside ChatGPT.
• ClaudeBot (Anthropic).
• Google-Extended (Google). This controls whether your content is used for Gemini. It does not affect your ranking in Google Search.
• PerplexityBot (Perplexity).
• Bytespider (ByteDance).

A block looks like this:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Disallow: / under a crawler’s name means it may not read any page on your site.

Step 3: decide what you want

Blocking AI crawlers is a legitimate choice for some businesses, for example to keep content out of training data. But if you want AI assistants to recommend your products, the crawlers they use to read the web need access to your product and collection pages.

If you decide to allow them, the rules look like this:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: PerplexityBot
Allow: /

Changing robots.txt on Shopify

Shopify generates robots.txt automatically. To change it, go to Online Store › Themes › Edit code, add a new template called robots.txt.liquid, and edit the rules there. Shopify’s help centre documents the template; keep its default rules and only add or remove what you need.

Also check any SEO or security apps you have installed. Some add their own crawler rules.

Step 4: check the rest of the basics

• llms.txt: an emerging convention where a site lists its most important pages for AI. It is not required, but it’s cheap to add.
• Schema: product, FAQ, review and organisation schema help machines understand what you sell.

On every plan, including Free, Signal AI checks your robots.txt for GPTBot, ClaudeBot, Google-Extended, PerplexityBot and Bytespider, looks for llms.txt and tests your schema, then flags anything that stops an AI from reading your store.