In the rapidly expanding world of LLMs, making your website easy for AI to interpret is more important than ever. That’s where llms.txt comes in— a simple, optional file that helps LLMs quickly grasp your site. This is important because at present LLMs struggle with long-term memory and can only keep so much context in their prompts. Below, you’ll better understand what llms.txt and llms-full.txt are, how to use them, and who came up with this initiative.
What Is llms.txt?
llms.txt is a proposed text file you place in your website’s root directory. Its main function is to provide a concise overview of what your site is about, giving LLMs a quick way to:
- Identify your site’s core topics or focus areas.
- Understand any unique terminology or organization.
- Adapt your site’s information into the LLM’s context window more effectively.
To check out an llms.txt files in the wild, head over to the directory.
Why Use llms.txt?
By offering a concise snapshot of your website without any HTML, you help LLMs understand what your website is about using less input tokens. Read more about tokenization here. As LLMs increasingly replace traditional search engines, clearer and more concise information means increasingly more accurate AI responses for users interacting with LLM-based tools or chatbots referencing your website.
Who Came Up with llms.txt?
The concept behind llms.txt was introduced by Jeremy Howard at llmstxt.org. It’s not an official internet standard, but rather an open initiative designed to help site owners and AI models work together more seamlessly. As such, anyone is free to adopt or ignore it.
An Expanded Option, llms-full.txt
In addition to llms.txt, the proposal also includes an expanded version, llms-full.txt, which serves as a more comprehensive resource for LLMs. While llms.txt provides a concise overview, llms-full.txt can include:
- Detailed Descriptions: A deeper dive into your site’s purpose, categories, or features.
- Supplementary Explanations: Extended notes about your documenation or any frequently asked questions.
- Structured Data References: Links or data points that are too extensive for a brief text file.
By placing an llms-full.txt file or folder in your website’s structure (e.g., https://example.com/llms-full/), you can offer LLMs a robust snapshot of your content. This can be particularly helpful if your site is large, covers multiple topics, or requires more in-depth context than a short text file allows.
Benefits of llms-full.txt
For websites that contain highly specialized or voluminous content, llms-full.txt provides LLMs with richer material to reference. There’s no strict format. If your site needs only a brief overview, llms.txt might be enough. But if you want to ensure LLMs have a thorough grasp of your resources, llms-full.txt is a handy extension.
How to Implement llms.txt
-
Create the File Use any text editor to create a file named
llms.txt. -
Provide Key Details Write a succinct description of your site, its main topics, and any important notes. Keep the language straightforward and clear. We have a draft generator here that you can use as a starting point.
-
Upload to the Root Directory Place the file in your site’s root (e.g.,
https://example.com/llms.txt). -
Update as Needed Whenever your website’s focus shifts or you add new features, update
llms.txtto ensure LLMs always have the latest context.
Common Questions
1. Is llms.txt mandatory for SEO?
No. It’s completely optional and specifically aimed at LLMs. Traditional search engines typically ignore llms.txt.
2. Will every LLM check llms.txt or llms-full? Adoption may vary. Some AI models might reference these files for improved context, while others may not support this feature yet.
3. Does llms.txt replace robots.txt?
No. robots.txt manages search engine crawling. llms.txt simply provides contextual information to LLMs.
Conclusion
llms.txt is a straightforward, community-driven method to give LLMs a quick, clear understanding of your website. Proposed by Jeremy Howard at llmstxt.org, it offers a practical way to improve AI-generated responses related to your content. If you want to ensure LLMs represent your site accurately, consider adding a llms.txt file to your site root today and then register it in the directory.