llms.txt complete guide for 2026: does it actually work?

llms.txt is a proposed web standard introduced by Jeremy Howard in late 2024 to give AI crawlers a machine-readable summary of a site's identity, key pages, and topic taxonomy. The file lives at the root (https://yourdomain.com/llms.txt), uses a Markdown-derived format, and is intended to do for AI crawlers what robots.txt did for search engines. The standard has meaningful adoption in developer tooling — Cursor, Continue, Cline, and various MCP servers read llms.txt when pointed at documentation sites. It has near-zero adoption in consumer AI search: as we documented in our methodology change, the SERanking November 2025 study of 300,000 domains found zero measurable correlation between llms.txt presence and AI citation rates, and Google's John Mueller publicly confirmed Google Search doesn't read or act on the standard. This guide is the practical reference: how to write llms.txt correctly if you choose to ship it, what it actually accomplishes (and doesn't), and when the cost-benefit favors shipping versus skipping. We removed llms.txt from scored audit because the evidence didn't support its scoring weight; this guide helps brands decide whether to ship it anyway for the IDE-agent use cases where it does work.

What is llms.txt?

llms.txt is a structured text file declaring information about your site for AI crawlers and developer tools. The format uses Markdown headers, lists, and links. A minimal example:

# Acme Corp

> Acme Corp builds widgets for the global market. We help manufacturers reduce widget downtime by 40%.

## Key pages

- [Home](https://acme.example.com/): Overview of Acme Corp and our widget products.
- [About](https://acme.example.com/about): Company history, leadership, and mission.
- [Products](https://acme.example.com/products): Widget catalog with specifications.
- [Documentation](https://acme.example.com/docs): Developer documentation and API reference.

## Authors

- Sarah Chen, CEO (https://www.linkedin.com/in/sarahchen)
- Mike Patel, CTO (https://www.linkedin.com/in/mikepatel)

## Topic taxonomy

- Widget manufacturing
- Industrial IoT
- Predictive maintenance
- Equipment monitoring

The proposed standard also defines llms-full.txt for sites with extensive content — a longer-form companion file containing detailed page-by-page descriptions and substantive content excerpts.

The standard was designed with three use cases in mind:

1. Help AI crawlers understand the site quickly. Rather than crawl every page to build a model of the site, the crawler reads llms.txt for a structured overview.
2. Surface key context for AI agents. Developer tools like Cursor and Continue use llms.txt to understand which documentation pages matter when answering coding questions.
3. Provide entity disambiguation. The file declares brand identity, key personnel, and topic taxonomy in a way that helps AI systems distinguish the brand from similarly-named entities.

The proposed standard is well-designed and the use cases are real. The question is which AI consumers actually implement it.

Who actually reads llms.txt?

The empirical landscape as of mid-2026:

IDE agents read llms.txt. Cursor, Continue, Cline, and various MCP servers explicitly read llms.txt when pointed at documentation sites. For brands publishing SDKs, APIs, or developer documentation, shipping llms.txt provides real value because these tools surface the brand's content to developer users at higher rates when the file exists.

Major AI assistants do not read llms.txt in production. As of Q1 2026:

• Google publicly confirmed Google Search doesn't read or act on llms.txt. Gary Illyes from Google added Google doesn't support the standard and isn't planning to.
• OpenAI has not publicly committed to acting on llms.txt in ChatGPT or ChatGPT Search.
• Anthropic has not publicly committed to acting on llms.txt in Claude.
• Perplexity has not publicly committed to acting on llms.txt.
• xAI has not publicly committed to acting on llms.txt in Grok.

Common Crawl ingests llms.txt as part of the open web archive. This means the file's content is potentially available to AI training pipelines that consume Common Crawl, but no specific AI provider has confirmed that they read or weight llms.txt differently from other site content.

The combined picture: llms.txt is genuine infrastructure for IDE-agent attribution and provides no measurable lift on consumer-facing AI assistant citation. The SERanking November 2025 study found zero correlation between llms.txt presence and citation rates across ChatGPT, Claude, Gemini, and Perplexity.

Should you ship llms.txt?

The cost-benefit decision depends on your business model and target audience:

Ship llms.txt if:

• Your brand publishes an SDK, API, developer documentation, or technical content that IDE agents are likely to query.
• Your buyers include developers who use Cursor, Continue, Cline, or other AI-assisted IDE tools.
• You want to signal to the AEO community that your brand is AI-forward (some brands ship llms.txt as a positioning signal even when the technical value is limited).
• You're a documentation-heavy brand (technical writing platforms, knowledge base providers, learning platforms).

Skip llms.txt if:

• Your brand is consumer-facing service or product without significant developer audience.
• Your AEO budget is limited and the 30-60 minutes spent on llms.txt would be better invested elsewhere (HARO pitching, Wikidata submission, schema audit).
• You're prioritizing fast time-to-citation lift on the major AI assistants where llms.txt provides no measurable benefit.

In our 10-Point AI Citation Audit, llms.txt presence is reported as a hygiene flag but does not contribute to scoring. We surface the file's presence so brands can make an informed decision; we don't credit it with citation impact that the evidence doesn't support.

How to write llms.txt

If you decide to ship it, the proposed standard structure:

Required: Site identity block

# Site Name

> One-sentence description of the site's purpose and value proposition.

Recommended: Key pages section

## Key pages

- [Page title](https://yourdomain.com/path): Brief description of what's on the page.
- [Another page](https://yourdomain.com/another): Brief description.

List the 5-15 most important pages on the site. For each, include the page title (which should match the visible page title), the full URL (HTTPS, including the protocol), and a 1-2 sentence description.

Recommended: Author information

## Authors

- Author Name, Role (https://linkedin.com/in/author-handle)
- Co-author Name, Role (https://linkedin.com/in/coauthor)

Especially valuable for technical content brands where authors carry credibility.

Recommended: Topic taxonomy

## Topic taxonomy

- Primary topic
- Secondary topic
- Tertiary topic

A flat or hierarchical list of the topics your content covers. Helps AI agents understand topical scope.

Optional: External links

## External resources

- [Industry association](https://example.org/)
- [Reference standard](https://standards.example.org/)

Links to authoritative external resources you reference. Provides cross-validation context.

The complete pattern

Combining the sections above:

# Acme Corp

> Acme Corp builds widget monitoring systems for industrial manufacturing. We help manufacturers reduce widget downtime by 40% via real-time IoT telemetry and predictive maintenance.

## Key pages

- [Home](https://acme.example.com/): Overview of Acme Corp and our widget monitoring systems.
- [Products](https://acme.example.com/products): Widget monitoring product catalog including hardware, software, and integration services.
- [Documentation](https://acme.example.com/docs): Developer documentation for the Acme API, including authentication, endpoints, webhooks, and SDK examples.
- [Case studies](https://acme.example.com/case-studies): Customer outcomes from Acme deployments across automotive, electronics, and industrial manufacturing sectors.
- [Blog](https://acme.example.com/blog): Editorial covering industrial IoT, predictive maintenance, and equipment downtime reduction.

## Authors

- Sarah Chen, CEO (https://www.linkedin.com/in/sarahchen)
- Mike Patel, CTO (https://www.linkedin.com/in/mikepatel)
- Dr. Elena Rodriguez, Head of Research (https://www.linkedin.com/in/erodriguez)

## Topic taxonomy

- Industrial IoT
- Widget manufacturing
- Predictive maintenance
- Equipment monitoring
- API documentation
- Industrial automation

## External resources

- [International Widget Standards Association](https://iwsa.example.org/)
- [Industrial IoT Foundation](https://iiot.example.org/)

What about llms-full.txt?

The proposed standard also includes llms-full.txt — a longer-form companion file containing detailed page-by-page descriptions and substantive content excerpts. The intended use case: AI agents needing deep context can fetch llms-full.txt instead of crawling the entire site.

llms-full.txt adoption is even lower than llms.txt. The same IDE agents (Cursor, Continue, Cline) that read llms.txt typically don't read llms-full.txt consistently. Major AI assistants don't read either file in production.

Our recommendation: ship llms-full.txt only if you've already shipped llms.txt AND you're publishing a documentation-heavy site where the deeper context provides value. For most brands, the additional cost (30-60 minutes of authoring + ongoing maintenance) doesn't pay back.

Where to host llms.txt

Standard practice is to host llms.txt at the site root: https://yourdomain.com/llms.txt. The file should:

• Be plain text (Markdown format works) with Content-Type: text/markdown or Content-Type: text/plain.
• Be served over HTTPS.
• Be accessible without authentication, geographic restrictions, or rate-limiting.
• Match the canonical domain (avoid serving from a subdomain unless that's the canonical site).

Keep llms.txt under 10KB. For sites with extensive content, the link summaries should be brief; deeper detail belongs in llms-full.txt (if you ship one) or in the actual page content.

Frequently asked questions

Will llms.txt eventually be supported by major AI assistants?

Possibly. The standard is still under development. OpenAI, Anthropic, Google, or others may add support in future iterations. As of mid-2026, no major provider has publicly committed to production support. We're tracking the space; if support emerges, we'll update our methodology and re-introduce llms.txt scoring with appropriate weight.

How does llms.txt interact with robots.txt?

robots.txt controls crawler access (which paths bots can read). llms.txt is a content document (declaring site identity for AI consumers). They don't conflict. A site can ship both; the recommended setup is permissive robots.txt (see the AI bot robots.txt complete guide) plus optional llms.txt.

Should I include sensitive information in llms.txt?

No. llms.txt is publicly accessible. Treat it as marketing content — anything you wouldn't put in your About page shouldn't go in llms.txt.

How often should llms.txt be updated?

Quarterly is sufficient for most brands. The page descriptions and topic taxonomy should reflect current site structure; significant restructuring should trigger an update. Stale llms.txt is more harmful than no llms.txt because it sends misleading signals.

What about llms.txt for personal sites or solo professionals?

The standard works for any site — personal blogs, agent sites, consultants. The investment threshold is the same: 30-60 minutes for llms.txt if shipping. For personal brands where developer audience is limited, the cost-benefit usually favors investing the time in other AEO tactics (Wikidata entity, content geometry, brand mention engineering).

---

Companion guides: Why we removed llms.txt from our methodology · The AI bot robots.txt complete guide · Schema markup for AI search · The 10-Point AI Citation Framework.