AI-powered search engines need two things from your website: permission to access your pages and a clear guide to what matters most. The first is handled by your robots.txt file. The second is handled by a relatively new file called “llms.txt” that sits in your site’s root directory alongside your sitemap and robots.txt.<\/p>\n\n\n\n
Most local businesses have never heard of llms.txt or the ai crawlers that read it. Yet these crawlers, including GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot, are already visiting millions of websites every day. GPTBot remains one of the most-blocked AI crawlers across the web, often because default CMS or CDN settings block it without anyone on the team knowing.<\/p>\n\n\n\n
This guide walks local businesses through three connected tasks: configuring robots.txt for AI crawlers, deploying an llms.txt file, and verifying that AI engines can actually read your site. Each step includes concrete examples, code, and testing instructions.<\/p>\n\n\n\n
LLMs.txt is a simple Markdown file that sits at yourdomain.com\/llms.txt and provides AI models with a structured summary of your website’s most important content. It was proposed by Jeremy Howard in 2024 and updated to version 2 in August 2026. Unlike robots.txt, which controls access, llms.txt controls priority by telling AI models which pages to read first.<\/p>\n\n\n\n
Think of llms.txt as a reading list for AI. Your sitemap lists every URL on your site. Your robots.txt specifies which URLs crawlers can access. Your llms.txt tells AI models which URLs actually matter and provides a one-line description of each.<\/p>\n\n\n\n
The file uses standard Markdown formatting. A typical llms.txt file includes your site title, a brief description, and a list of your highest-value pages with descriptive labels. Here is an example for a local law firm:<\/p>\n\n\n\n
# Davis & Clark Family Law<\/p>\n\n\n\n
> Family law firm serving Austin, TX. Divorce, custody, mediation, and prenuptial agreements.<\/p>\n\n\n\n
## Main Pages<\/p>\n\n\n\n
– [Home](https:\/\/www.davisclarklaw.com): Family law services in Austin, TX<\/p>\n\n\n\n
– [Divorce Services](https:\/\/www.davisclarklaw.com\/divorce): Contested and uncontested divorce representation<\/p>\n\n\n\n
– [Child Custody](https:\/\/www.davisclarklaw.com\/custody): Custody agreements, modifications, and enforcement<\/p>\n\n\n\n
– [Mediation](https:\/\/www.davisclarklaw.com\/mediation): Court-ordered and voluntary mediation services<\/p>\n\n\n\n
– [About](https:\/\/www.davisclarklaw.com\/about): Attorney bios, bar admissions, and case history<\/p>\n\n\n\n
## Blog<\/p>\n\n\n\n
– [How Long Does a Divorce Take in Texas?](https:\/\/www.davisclarklaw.com\/blog\/divorce-timeline-texas): Average timelines and factors that delay proceedings<\/p>\n\n\n\n
– [Child Custody Laws in Texas: 2026 Update](https:\/\/www.davisclarklaw.com\/blog\/texas-custody-laws-2026): Current statutes, presumptions, and recent changes<\/p>\n\n\n\n
llms-full.txt concatenates the complete Markdown content of your most important pages into a single file, separated by dividers. It allows AI models to ingest your entire knowledge base in one pass without crawling each URL individually. Small sites can rely on llms.txt alone; larger sites benefit from both files.<\/p>\n\n\n\n
The llms.txt file acts as a front desk directory. It points AI models to the right pages. The llms-full.txt file acts as a complete briefing packet. It contains the actual content so the model does not need to visit each page separately.<\/p>\n\n\n\n
This distinction matters because AI models consume significant processing power when crawling individual pages. Providing a single, clean text file reduces friction and increases the chance that your full content is ingested accurately. However, llms-full.txt can become very large for sites with hundreds of pages, so limit it to your top 20 to 30 most important URLs.<\/p>\n\n\n\n
Place llms-full.txt at yourdomain.com\/llms-full.txt. Use the same Markdown formatting as llms.txt but include the full text of each page below its URL. Separate each page’s content with a line of three dashes (—).<\/p>\n\n\n\n
“AI crawlers” are automated bots that scan websites on behalf of AI platforms. The major ones include GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot (Perplexity). Each crawler has a different purpose: some collect training data, while others fetch live content for real-time search answers.<\/p>\n\n\n\n
Not all AI crawlers do the same job. OpenAI alone runs four distinct crawlers, and understanding the difference matters for local businesses making access decisions.<\/p>\n\n\n\n
GPTBot<\/strong> is OpenAI’s training data crawler. It collects content that may be used to train future AI models. Blocking GPTBot prevents your content from being used as training data. However, blocking GPTBot does not remove your site from ChatGPT search results, because ChatGPT search uses a different crawler.<\/p>\n\n\n\n OAI-SearchBot<\/strong> is the crawler ChatGPT uses for real-time web search. When a ChatGPT user asks a question and the model searches the web, OAI-SearchBot fetches the pages. Blocking this crawler removes your site from ChatGPT’s live search results.<\/p>\n\n\n\n ChatGPT-User<\/strong> handles on-demand fetches when a user directly pastes or references a URL in a ChatGPT conversation. Blocking it prevents ChatGPT from reading a URL even when a user specifically shares it.<\/p>\n\n\n\n ClaudeBot<\/strong> is Anthropic’s crawler. It collects content for training Claude’s models. Anthropic has stated that ClaudeBot respects robots.txt directives.<\/p>\n\n\n\n PerplexityBot<\/strong> crawls content for Perplexity’s AI search engine. Allowing PerplexityBot gives your content a chance to be cited in Perplexity answers.<\/p>\n\n\n\n A significant share of top websites currently block GPTBot through their robots.txt files, many without realizing the trade-off. For local businesses that want AI visibility, blocking these crawlers means forfeiting the chance to be cited in AI-generated answers entirely.<\/p>\n\n\n\n Your robots.txt file at yourdomain.com\/robots.txt controls which crawlers can access your site. For AI visibility, explicitly allow GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot on your public content while blocking sensitive paths like admin panels and checkout flows.<\/p>\n\n\n\n Many local businesses inherit robots.txt files from their CMS or hosting provider that unintentionally block ai crawlers. Some CDN providers, including Cloudflare, have been blocking AI crawlers by default for new domains since 2025. If you have never checked your robots.txt, there is a real possibility that AI engines cannot read your site right now.<\/p>\n\n\n\n Here is a robots.txt example optimized for AI visibility:<\/p>\n\n\n\n # Allow all standard search crawlers<\/p>\n\n\n\n User-agent: *<\/p>\n\n\n\n Allow: \/<\/p>\n\n\n\n # Explicitly allow AI crawlers<\/p>\n\n\n\n User-agent: GPTBot<\/p>\n\n\n\n Allow: \/<\/p>\n\n\n\n User-agent: OAI-SearchBot<\/p>\n\n\n\n Allow: \/<\/p>\n\n\n\n User-agent: ChatGPT-User<\/p>\n\n\n\n Allow: \/<\/p>\n\n\n\n User-agent: ClaudeBot<\/p>\n\n\n\n Allow: \/<\/p>\n\n\n\n User-agent: anthropic-ai<\/p>\n\n\n\n Allow: \/<\/p>\n\n\n\n User-agent: PerplexityBot<\/p>\n\n\n\n Allow: \/<\/p>\n\n\n\n User-agent: Google-Extended<\/p>\n\n\n\n Allow: \/<\/p>\n\n\n\n # Block sensitive paths for all crawlers<\/p>\n\n\n\n User-agent: *<\/p>\n\n\n\n Disallow: \/admin\/<\/p>\n\n\n\n Disallow: \/checkout\/<\/p>\n\n\n\n Disallow: \/my-account\/<\/p>\n\n\n\n The explicit Allow directives matter. Some default robots.txt templates include broad Disallow patterns that catch AI crawlers by accident. Adding explicit Allow entries for each named AI crawler ensures there is no ambiguity.<\/p>\n\n\n\n After updating your robots.txt, fetch the file in your browser at yourdomain.com\/robots.txt and visually confirm every directive. Then monitor your server logs for two weeks and look for hits from GPTBot, ClaudeBot, and PerplexityBot to verify that the crawlers are actually visiting.<\/p>\n\n\n\n If your local business wants to appear in AI-generated search answers across ChatGPT, Gemini, and Perplexity, allow AI crawlers on your public content. Block them only on pages containing proprietary data, user accounts, or transactional flows. Selective access gives you visibility without exposing sensitive content.<\/p>\n\n\n\n The decision is not all or nothing. You can allow AI crawlers on your blog, service pages, and about page while blocking them from internal tools, customer portals, and checkout flows.<\/p>\n\n\n\n Allow AI crawlers if:<\/strong><\/p>\n\n\n\n Block AI crawlers if:<\/strong><\/p>\n\n\n\n For most local businesses, the visibility benefit of allowing AI crawlers far outweighs the downside. AI search traffic already converts at significantly higher rates than traditional organic traffic. Blocking crawlers removes your business from that channel entirely.<\/p>\n\n\n\n Deploying llms.txt takes under 30 minutes for most websites. Create the file in a text editor, upload it to your site’s root directory, verify it returns a 200 HTTP response, and validate the format with a free checker. The steps below cover each platform.<\/p>\n\n\n\n Step 1: Write the file.<\/strong> Open a text editor and create a file named llms.txt. Add your site title as a Markdown H1 heading. Add a one-line description in a blockquote. List your 10 to 20 most important pages with descriptive labels.<\/p>\n\n\n\n Step 2: Upload the file.<\/strong> Place llms.txt in the same root directory as your robots.txt and sitemap.xml. On WordPress, use the Website LLMs.txt plugin, which has over 10,000 installs. On Webflow, upload through the built-in settings. On Shopify, use the Files section in your admin panel.<\/p>\n\n\n\n Step 3: Verify the URL.<\/strong> Open yourdomain.com\/llms.txt in your browser. It should display the Markdown text of your file with a 200 HTTP status code. If it returns a 404 or redirects to your homepage, the file is not properly deployed.<\/p>\n\n\n\n Step 4: Validate the format.<\/strong> Run your file through a free validator at llmstxt.io or firecrawl.dev. These tools check your Markdown structure and flag formatting errors that could prevent AI models from parsing the file correctly.<\/p>\n\n\n\n Step 5: Monitor crawler activity.<\/strong> Check your server logs for requests to your llms.txt URL over the following two weeks. You should see hits from AI crawlers, confirming that the file is being discovered and read.<\/p>\n\n\n\n Measure the impact of your llms.txt and AI crawler access by tracking three things: server log hits on your llms.txt URL, AI visibility across ChatGPT, Gemini, and Perplexity prompts, and citation accuracy when your business is mentioned in AI-generated answers.<\/p>\n\n\n\n Start with server logs. After deploying llms.txt, you should see HTTP requests from AI crawler user agents within one to two weeks. If you see no hits, check that your CDN or hosting provider is not blocking the requests at the network level.<\/p>\n\n\n\n Next, run monthly AI visibility audits. Pick 20 to 50 prompts that your customers would type into ChatGPT or Perplexity, like “best [service] in [city]” or “how to choose a [service provider] near me.” Record whether your business is mentioned, cited with a link, or absent from each answer.<\/p>\n\n\n\n No major AI platform has officially confirmed that llms.txt influences content selection. However, early adopters report correlations in citation behavior for sites that publish the file alongside clean schema and open crawler access. The low effort and zero risk of deployment make it a worthwhile addition to any AI search strategy.<\/p>\n\n\n\n AI search visibility depends on two technical foundations that most local businesses overlook: crawler access and content indexing. Your robots.txt file determines whether AI engines can read your pages. Your llms.txt file determines which pages they prioritize. Together with proper schema markup, these files form the complete technical stack for AI search readiness.<\/p>\n\n\n\n Deploying llms.txt takes under 30 minutes. Auditing your robots.txt for AI crawler directives takes even less. These are not complex development projects. They are simple configuration tasks with outsized impact on whether your business appears in AI-generated answers across ChatGPT, Gemini, Perplexity, and Google AI Overviews.<\/p>\n\n\n\n If you want a team to handle the full stack, from llms.txt deployment to monthly AI visibility tracking, Online Ranking AI<\/a> includes llms.txt, schema validation, and prompt-level tracking in every AI SEO program<\/a>. Run a free AI audit<\/a> to find out which AI crawlers can and cannot read your site right now.<\/p>\n\n\n\n Robots.txt controls whether AI crawlers can access your site. LLMs.txt tells AI models which pages are most important and what each one covers. Robots.txt governs permission. LLMs.txt governs priority. Both files sit in your site’s root directory.<\/p>\n\n\n\n OpenAI has not officially confirmed that ChatGPT uses llms.txt in content selection. However, the file follows a community standard that is gaining adoption. Early adopters report improved citation behavior, and the zero-risk deployment makes it worth implementing.<\/p>\n\n\n\n Blocking GPTBot only prevents your content from being used as training data. ChatGPT search uses OAI-SearchBot, a separate crawler. To remove your site from ChatGPT search results, you would need to block OAI-SearchBot specifically in your robots.txt.<\/p>\n\n\n\n List 10 to 20 of your highest-value pages for a small local business site. Include your homepage, main service pages, about page, and top-performing blog posts. For larger sites, use llms-full.txt to provide the complete content of each listed page.<\/p>\n\n\n\n No. LLMs.txt is a plain text file written in Markdown. You can create it in any text editor and upload it to your site’s root directory. WordPress users can install the Website LLMs.txt plugin. Most CMS platforms support direct file uploads to the root directory.<\/p>\n","protected":false},"excerpt":{"rendered":" AI-powered search engines need two things from your website: permission to access your pages and a clear guide to what matters most. The first is handled by your robots.txt file. The second is handled by a relatively new file called “llms.txt” that sits in your site’s root directory alongside your sitemap and robots.txt. Most local […]<\/p>\n","protected":false},"author":2,"featured_media":3025,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[40,44],"class_list":["post-3024","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-seo","tag-ai-seo","tag-llms"],"yoast_head":"\nHow to Configure Robots.txt for AI Crawlers<\/h2>\n\n\n\n
Should You Block or Allow AI Crawlers? A Decision Framework<\/h2>\n\n\n\n
\n
\n
How to Deploy LLMs.txt: Step by Step<\/h2>\n\n\n\n
How to Measure Whether LLMs.txt and AI Crawler Access Are Working<\/h2>\n\n\n\n
Conclusion<\/h2>\n\n\n\n
Frequently Asked Questions<\/h2>\n\n\n\n
What is the difference between llms.txt and robots.txt?<\/h3>\n\n\n\n
Does ChatGPT actually read llms.txt files?<\/h3>\n\n\n\n
Will blocking GPTBot remove my site from ChatGPT search?<\/h3>\n\n\n\n
How many URLs should I list in my llms.txt file?<\/h3>\n\n\n\n
Do I need a developer to deploy llms.txt?<\/h3>\n\n\n\n