<p>In the rapidly expanding world of LLMs, making your website easy for AI to interpret is more important than ever. That’s where <code>llms.txt</code> comes in— a simple, optional file that helps LLMs quickly grasp your site. This is important because at present LLMs struggle with long-term memory and can only keep so much context in their prompts. Below, you’ll better understand what <code>llms.txt</code> and <code>llms-full.txt</code> are, how to use them, and who came up with this initiative.</p>
<h2>What Is llms.txt?</h2>
<p><code>llms.txt</code> is a proposed text file you place in your website’s root directory. Its main function is to provide a concise overview of what your site is about, giving LLMs a quick way to:</p>
<ul>
<li>Identify your site’s core topics or focus areas.</li>
<li>Understand any unique terminology or organization.</li>
<li>Adapt your site’s information into the LLM’s context window more effectively.</li>
</ul>
<p>To check out an <code>llms.txt</code> files in the wild, head over to the <a href="/directory">directory</a>.</p>
<h2>Why Use llms.txt?</h2>
<p>By offering a concise snapshot of your website without any HTML, you help LLMs understand what your website is about using less input tokens. Read more about tokenization <a href="https://en.wikipedia.org/wiki/Large_language_model#Dataset_preprocessing">here</a>. As LLMs increasingly replace traditional search engines, clearer and more concise information means increasingly more accurate AI responses for users interacting with LLM-based tools or chatbots referencing your website.</p>
<h2>Who Came Up with llms.txt?</h2>
<p>The concept behind <code>llms.txt</code> was introduced by Jeremy Howard at <a href="https://llmstxt.org/">llmstxt.org</a>. It’s not an official internet standard, but rather an open initiative designed to help site owners and AI models work together more seamlessly. As such, anyone is free to adopt or ignore it.</p>
<h2>An Expanded Option, <code>llms-full.txt</code></h2>
<p>In addition to <code>llms.txt</code>, the proposal also includes an expanded version, <code>llms-full.txt</code>, which serves as a more comprehensive resource for LLMs. While <code>llms.txt</code> provides a concise overview, <code>llms-full.txt</code> can include:</p>
<ul>
<li><strong>Detailed Descriptions</strong>: A deeper dive into your site’s purpose, categories, or features.</li>
<li><strong>Supplementary Explanations</strong>: Extended notes about your documenation or any frequently asked questions.</li>
<li><strong>Structured Data References</strong>: Links or data points that are too extensive for a brief text file.</li>
</ul>
<p>By placing an <code>llms-full.txt</code> file or folder in your website’s structure (e.g., <code>https://example.com/llms-full/</code>), you can offer LLMs a robust snapshot of your content. This can be particularly helpful if your site is large, covers multiple topics, or requires more in-depth context than a short text file allows.</p>
<h3>Benefits of <code>llms-full.txt</code></h3>
<p>For websites that contain highly specialized or voluminous content, <code>llms-full.txt</code> provides LLMs with richer material to reference. There’s no strict format. If your site needs only a brief overview, <code>llms.txt</code> might be enough. But if you want to ensure LLMs have a thorough grasp of your resources, <code>llms-full.txt</code> is a handy extension.</p>
<h2>How to Implement llms.txt</h2>
<ol>
<li>
<p><strong>Create the File</strong>
Use any text editor to create a file named <code>llms.txt</code>.</p>
</li>
<li>
<p><strong>Provide Key Details</strong>
Write a succinct description of your site, its main topics, and any important notes. Keep the language straightforward and clear. We have a draft generator <a href="/generate">here</a> that you can use as a starting point.</p>
</li>
<li>
<p><strong>Upload to the Root Directory</strong>
Place the file in your site’s root (e.g., <code>https://example.com/llms.txt</code>).</p>
</li>
<li>
<p><strong>Update as Needed</strong>
Whenever your website’s focus shifts or you add new features, update <code>llms.txt</code> to ensure LLMs always have the latest context.</p>
</li>
</ol>
<h2>Common Questions</h2>
<p><strong>1. Is llms.txt mandatory for SEO?</strong>
No. It’s completely optional and specifically aimed at LLMs. Traditional search engines typically ignore <code>llms.txt</code>.</p>
<p><strong>2. Will every LLM check llms.txt or llms-full?</strong>
Adoption may vary. Some AI models might reference these files for improved context, while others may not support this feature yet.</p>
<p><strong>3. Does llms.txt replace <code>robots.txt</code>?</strong>
No. <code>robots.txt</code> manages search engine crawling. <code>llms.txt</code> simply provides contextual information to LLMs.</p>
<hr>
<h2>Conclusion</h2>
<p><code>llms.txt</code> is a straightforward, community-driven method to give LLMs a quick, clear understanding of your website. Proposed by Jeremy Howard at <a href="https://llmstxt.org/">llmstxt.org</a>, it offers a practical way to improve AI-generated responses related to your content. If you want to ensure LLMs represent your site accurately, consider adding a <code>llms.txt</code> file to your site root today and then register it in the <a href="/directory">directory</a>.</p>


llms.txt is a proposed text file in a website’s root directory to provide **concise overview** for llms of a site

All Posts

Understanding llms.txt: A New Way for LLMs to Grasp Your Website