llms.txt: Should Your Website Add One for AI Crawlers?

TL;DR: llms.txt is a proposed markdown file placed at your site’s root to summarise content for AI systems. No major LLM (ChatGPT, Claude, Gemini) is confirmed to read it at inference time. It takes under an hour to build and carries no downside, so add one, but don’t prioritise it over structured data or technical SEO.

What Is llms.txt and How Is It Different From robots.txt?

llms.txt is a proposal published by Jeremy Howard on September 3 2024 at llmstxt.org. It describes a Markdown navigation file that gives large language models context and guidance. It is not a blocking mechanism. As Derivatex puts it, llms.txt is “a Markdown navigation file, not a blocking tool.”

robots.txt controls crawler access by allowing or disallowing specific user-agents. llms.txt cannot block anything; it only offers information. Both files sit at the root domain in plain text, but their functions are opposites: robots.txt restricts, llms.txt invites.

Feature	robots.txt	llms.txt
Purpose	Control crawler access	Summarise site content for AI
Function	Block or allow bots	Provide guidance only
File format	Plain text	Markdown
Placement	Root directory	Root directory
Effect on AI crawlers	Can block them	Cannot block; purely informational

Do Any AI Systems Actually Use llms.txt Right Now?

llmstxt.org frames the concept as “a proposal to standardise,” not an adopted standard. No public confirmation exists from OpenAI, Google, Anthropic, or Perplexity that their consumer-facing models parse llms.txt at inference time. The 365Digital team implemented llms.txt on client sites and checked server logs for known AI crawler user-agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended). None of those crawlers fetched the file.

The real adopters cited in the proposal are developer-tooling and documentation projects, not end-user AI assistants. The FastHTML project follows the proposal for its documentation, and all nbdev projects now generate .md versions of pages by default. These are useful early signals, but they don’t mean consumer LLMs read llms.txt when answering user queries.

Adoption currently lives in the developer ecosystem, not in the AI search stack. To verify whether any AI system is fetching your own llms.txt, audit your server logs for the user-agents listed above. Until those logs show regular, meaningful requests, the file’s practical impact at inference time remains unconfirmed.

What Should You Include in an llms.txt File?

Your llms.txt must be a plain-text file written in Markdown and placed at the site root (yourdomain.com/llms.txt). Rankability identifies four core components: a Website Summary, Content Categories, Usage Guidelines, and Contact Information. The spec recommends linking to .md versions of key pages where possible, following the nbdev pattern, rather than linking only to HTML pages.

A minimal working example uses an H1 title, a blockquote summary, and H2 sections with markdown links:

# 365Digital Agency
> We help businesses grow with SEO, AI visibility, and automation.
## Services
- [SEO](/services/seo)
- [AI Visibility](/services/ai-search-visibility)
- [Automation](/services/automation)
## Usage
- You may use this content for training with attribution.
## Contact
- [Contact us](/contact-us)

Keep each section concise and structured so an AI can parse the hierarchy quickly. Avoid embedding large blocks of prose; the file is a navigation aid, not a content dump. Include only your most important pages and a clear statement of how you permit AI systems to use your material.

How Do You Create and Host an llms.txt File?

Start by writing the markdown file locally. Upload it to your site’s root directory via FTP or your hosting file manager, then confirm it loads at /llms.txt. On WordPress, the Hostinger Tools plugin can auto-generate the file for you.

For non-technical teams, a web development partner can handle the setup in minutes. The real maintenance challenge is dynamic or frequently updated sites. Unlike a static robots.txt that rarely changes, llms.txt needs a review cadence, monthly or quarterly, to stay accurate. Skip that cadence and the file quickly goes stale and loses any potential value.

Should You Add llms.txt to Your Website Right Now?

There is no known downside or ranking risk from adding an llms.txt file, and setup takes under an hour for most sites. But no major LLM is confirmed to use it at inference time, so the upside remains theoretical. The file complements, rather than replaces, the signals AI systems actually rely on today: structured data, clean HTML, and clear content hierarchy.

Treat llms.txt as low-cost insurance. Build it once, then put your real effort into technical SEO and content clarity, which have proven impact on AI visibility. Our AI Search Visibility (GEO) work consistently shows that structured data and well-organised content move the needle far more than speculative proposals. The verdict: add llms.txt, but don’t let it distract you from the fundamentals that already drive AI citations.

Key Takeaways

• llms.txt is a proposed standard from September 2024, not an adopted one.
• No major LLM (ChatGPT, Claude, Gemini) is confirmed to read llms.txt at inference time.
• llms.txt cannot block AI crawlers; robots.txt already does that job.
• Building one takes under an hour and carries no SEO downside.
• Structured data and clean content structure currently do more for AI visibility than llms.txt.

Frequently Asked Questions

What is llms.txt and why do I need it?

llms.txt is a proposed Markdown file that summarises your site’s content for AI systems. You don’t need it today, but adding it takes minimal effort and may help future-proof your site if LLMs adopt the standard.

Is llms.txt officially supported by Google or other LLMs yet?

No. No major LLM provider has confirmed that its consumer models parse llms.txt at inference time. Current adoption is limited to developer-tooling projects like FastHTML and nbdev.

What’s the difference between llms.txt and robots.txt?

robots.txt controls crawler access by allowing or disallowing bots. llms.txt provides content guidance in Markdown format and cannot block anything. They share a root-directory location but serve opposite functions.

Can llms.txt block AI crawlers?

No. llms.txt is purely informational. To block AI crawlers, use robots.txt with the appropriate user-agent directives.

Are there tools to help me create an llms.txt file?

You can write the file manually in any text editor. The Hostinger Tools plugin for WordPress can auto-generate one. Always validate the output against the llmstxt.org spec.

What happens if I don’t have an llms.txt file?

Nothing. No AI system currently requires the file or penalises its absence. Your site will function exactly as it does today.

Not sure whether llms.txt is worth your team’s time, or what actually moves your AI visibility? Talk to the 365Digital team.

About the author: Written by the 365Digital team, a group of SEO strategists, automation specialists, and content marketers helping businesses grow their organic and AI search visibility since 2013.