llms.txt vs robots.txt: what each AI crawler setting actually controls
These files are often discussed together. They answer different questions: who may fetch a URL, and how an agent can find useful information once it arrives.
If you are trying to make a website AI-agent ready, the first useful distinction is simple: robots.txt is a crawl preference, while llms.txt is a proposed documentation convention. Neither one is a login system, a guarantee of citations, or proof that an agent can complete a form.
This matters when a page is missing from an AI answer or an automated visitor stops at your pricing page. You need to know whether the problem is access, discovery, content, or interaction. The AI-agent readiness checklist covers the complete journey; this article focuses on the files people most often configure incorrectly.
llms.txt vs robots.txt: the useful comparison
| Resource | Primary job | Does it grant access? | Does it prove AI visibility? |
|---|---|---|---|
robots.txt | Tell cooperating crawlers which paths they may fetch. | No. It is not authentication. | No. It expresses a preference; the crawler still decides what to do. |
llms.txt | Offer concise background, guidance, and links for agents. | No. The file is public documentation. | No. It is a proposal and does not guarantee retrieval or citation. |
| Sitemap | Advertise canonical URLs and optional freshness metadata. | No. | No. It helps discovery for systems that use it. |
noindex | Tell search systems not to index a page when they can crawl it. | No. | No. It controls indexing, not task completion. |
| Authentication and authorization | Enforce who may read or change data. | Yes, when correctly implemented server-side. | No. Access is a prerequisite, not a visibility promise. |
The practical sequence is: protect private data, make the pages you want discovered reachable, publish clear facts, and test the journey. Adding a file cannot repair a blocked CDN, an empty JavaScript shell, or an unlabeled form field.
What robots.txt controls
A robots file lives at the origin root, for example https://example.com/robots.txt. A cooperating crawler can read its groups and decide whether to fetch a matching path. Google describes robots.txt as a way to manage crawler access; it also warns that the file is not a way to keep a URL secret or prevent access by a determined client. Google’s robots.txt introduction.
Keep the policy narrow and review it like code. A harmless-looking rule can block your pricing page, a stylesheet needed for rendering, or an entire locale. Test the exact URL, including redirects, and check your server or CDN logs for the requests you expect.
Here is an illustrative policy for a public site that wants ChatGPT search access, does not want training crawler access to a private path, and gives other cooperating crawlers the normal site-wide preference:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /private/
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlThis example is a starting point, not a universal policy. Replace the host, review every sensitive path, and check the current documentation for each provider you care about. Do not put credentials or confidential URLs in this file. Use server-side access controls for those.
Specificity matters. A named crawler group is not the same as a wildcard group, and different crawlers may interpret policies independently. Validate the published file with a parser and then verify the real response from your edge infrastructure.
Which AI crawler is which?
“The AI bot” is not one audience. OpenAI documents separate user-agent tokens for separate purposes:
- OAI-SearchBot is used to surface websites in ChatGPT search. OpenAI says opting out means a site will not be shown in ChatGPT search answers, while it can still appear as a navigational link.
- GPTBot is used for content that may be used to improve OpenAI’s foundation models. Allowing it is a different decision from allowing search.
- ChatGPT-User is associated with certain user-initiated visits. It is not the automatic search crawler, and robots.txt rules may not apply to those requests.
Read the current OpenAI crawler reference before changing an allow or disallow rule. It also publishes IP ranges and notes that robots changes can take time to propagate.
Google’s Google-Extended token is another example of a control with a specific purpose. Google says it manages some training and grounding uses, does not have a separate HTTP user-agent string, and does not affect inclusion or ranking in Google Search. Google-Extended documentation.
Do not infer a provider’s behavior from an unfamiliar request alone. Confirm the user-agent documentation, published IP range, request result, and the business purpose you are trying to support.
What llms.txt is for
llms.txt is a proposal for an agent-readable Markdown resource. The current v2 proposal describes short background information, guidance, and links to detailed Markdown pages. It can live at a site root or within a path, so a documentation area could have its own /docs/llms.txt. The llms.txt proposal.
That makes it closer to a well-organized index or reading guide than to a firewall rule. It can tell a documentation agent which pages explain authentication, rate limits, or a product’s current limits. It cannot force an agent to fetch a page, change a search ranking, or authorize a write operation.
If you publish one, keep it maintained:
- State what the site or section is, in plain language.
- Link to the few pages an agent needs for the common task.
- Label drafts, deprecated versions, pricing conditions, and access requirements honestly.
- Remove links that return errors or point at superseded instructions.
- Keep linked HTML pages useful for people; do not make the Markdown file the only source of important facts.
Version 2 also discusses alternate Markdown representations and link relations such as rel="describedby". Treat those as part of the proposal’s evolving ecosystem, not as a browser or search requirement. A plain, accurate HTML page remains the foundation for both humans and automated readers.
Do not confuse the other controls
A sitemap advertises URLs
A sitemap gives search systems a list of canonical URLs and can carry metadata such as modification dates. It does not override robots rules, create permissions, or guarantee indexing. Link the sitemap from robots.txt when appropriate and submit it through the search tools you use.
Noindex controls indexing
A noindex directive is meant for search indexing. A crawler must be able to fetch the page to see a directive in the HTML or response headers, so blocking the same path in robots.txt can prevent a search system from observing your preference. Google’s noindex guidance explains this relationship.
Authentication protects data
Use sessions, tokens, authorization checks, and rate limits to protect account data and consequential actions. Never rely on robots.txt, a hidden link, an obscure URL, or an llms.txt omission as a security boundary. If an agent should see public pricing but not customer-specific pricing, enforce that distinction on the server and test it with separate accounts.
HTML and accessibility carry meaning
A crawler can be allowed through every policy and still receive no useful answer. Put important facts in text, use stable links, label controls, and expose clear states. The demo-form testing guide shows how to verify the interaction layer after your discovery files look healthy.
A safe implementation workflow
Make one change at a time and record the evidence. This avoids the common situation where a team edits robots.txt, adds llms.txt, changes a CDN rule, and cannot tell which change altered access.
- Choose a public task. Write the question an agent should answer and the URL where the answer lives.
- Check the edge. Request robots.txt, the sitemap, and the target page through the production hostname. Note status codes, redirects, content type, and challenge pages.
- Review crawler policy by purpose. Decide separately about search, training, and user-initiated access. Document the decision and owner.
- Publish accurate page content. Put the answer in ordinary HTML. Add structured data only when it matches visible content.
- Add llms.txt if it helps your workflow. Link to maintained, high-value pages and check it like documentation.
- Test the full journey. Confirm that the agent can move from discovery to a permitted action. A public scan cannot prove this browser step.
The free Adapt My Page scan checks public discovery signals and returns evidence-based findings. The API documentation describes the same passive scope for automation; it does not execute JavaScript, sign in, or complete a task.
Troubleshoot a page that an AI system cannot find
Start with the failure you observed, not the file you expect to fix.
| Observation | Likely layer | Next evidence |
|---|---|---|
| The crawler receives a block or challenge | Edge access or robots policy | Response headers, robots group, CDN logs, provider documentation |
| The page returns an app shell with no answer | Rendering or content delivery | Initial HTML, rendered DOM, accessibility tree |
| The answer exists but its URL is hard to reach | Information architecture | Crawlable links, canonical, sitemap, redirects |
| The answer is wrong or stale | Content or data contract | Visible text, structured data, API response, update owner |
| The page is understood but the task fails | Controls, state, permissions, or tool contract | Browser trace, server result, retry and authorization logs |
For a crawler that is allowed in robots.txt but still receives a challenge, follow the Cloudflare crawler-access troubleshooting guide. It separates policy, request identity, and the security control that actually denied access.
If a successful response contains only an app shell, use the JavaScript rendering audit to compare the initial HTML, rendered content, and facts revealed by interaction.
Do not “fix” a task failure by opening more crawler paths. A readable, secure refusal can be the correct result when the requested action is private or consequential. For browser interactions, continue with the WebMCP security checklist.
Which file should you add?
- You want to manage automated fetches: review robots.txt and edge rules.
- You want to guide an agent through documentation: consider a maintained llms.txt or equivalent documentation index.
- You want a URL discovered: use ordinary crawlable links and a sitemap.
- You want a page excluded from search: use an appropriate noindex control while keeping the page crawlable long enough for the directive to be seen.
- You want data protected: implement authentication and authorization on the server.
- You want an agent to complete a task: test the rendered page, accessibility tree, feedback states, and final backend outcome.
Run the full readiness checklist when you need to connect these layers. If you manage a portfolio of sites, use the Agent Readiness API to collect passive findings and send each site’s owner the evidence to verify in a browser. The audit automation guide shows how to pace those scans and compare findings between runs.
That separation keeps your policy legible. Robots.txt says where a cooperating crawler may look. llms.txt can explain where an agent should start. Your application still has to deliver accurate answers and enforce every action.