There’s been so much going on in the AI space lately, and I’ve been spending time catching up, both with Workers AI at work and just staying plugged in to what’s happening across the AI ecosystem. As always, I like writing down the things I’m learning, for future me and for anyone else who finds this stuff interesting.
So, let’s talk about something I recently discovered: llms.txt.
What even is llms.txt?
At first glance, it sounds a bit like robots.txt, and while there’s a tiny resemblance in concept, it actually serves a different purpose. llms.txt is a proposed standard, basically a markdown file you can place at the root of your website (like giftegwuenu.com/llms.txt) to help large language models (LLMs) better understand what your site is about.
Unlike web crawlers that index your site for search, this is meant for LLMs at inference time. So, not exactly for training, but for those moments when someone prompts an AI and it tries to pull context from your website. The idea is to provide a simple, structured overview of your content to make it easier for these models to find the important stuff.
Why does this matter?
LLMs aren’t great at digging through cluttered web pages. They can get distracted by navbars, ads, or just irrelevant content. And because they have a limited context window (aka how much they can “see” at once), giving them a shortcut to the good stuff just makes sense.
With llms.txt, you can:
- Highlight key sections like docs, tutorials, or guides
- Help AI tools give users better answers
- Control what parts of your site should be prioritized
Basically, this whole concept lets you say, “Hey AI, here’s what you should check out first.”
What does a llms.txt file look like?
Here’s a sample format for a llms.txt file:
# My Website
> A brief description of what the site is about.
## Docs
- [Getting Started](https://example.com/docs/start): A quickstart guide.
## Blog
- [Latest Post](https://example.com/blog/llms-txt): Covers how to use llms.txt.
## Optional
- [API Reference](https://example.com/api): Full list of endpoints.You can get fancier with more sections or use llms-full.txt if you want to go deeper, but the standard is intentionally lightweight.
How do you create one?
You can absolutely write one manually (it’s just markdown!), but if you want a head start, tools like Firecrawl make it super easy to get started. They’ll crawl your site and generate the file for you.
Who’s already using llms.txt?
This isn’t just a niche idea, some major players in the industry are already experimenting with llms.txt for their docs:
- Cloudflare, we’ve rolled it out on our docs too
- Anthropic, helping their AI models better parse internal resources
- Perplexity AI, making it easier for their LLM-powered search to reference accurate content
- ElevenLabs, guiding AI to their most relevant documentation
Curious who else is on board? Check out the llms.txt directory to browse a growing list of websites using this format.
Final thoughts
I think it’s a smart and simple way to make our websites more AI-accessible, without needing to change anything.As someone who builds and writes a lot on the web, it just feels like a nice step toward being intentional with how AI interacts with our content.
Especially in a world where LLMs are becoming the new interface to the internet. It’s also very easy to pass the file to ChatGPT, Cursor, or whatever AI tool you’re using, so it can quickly understand your site’s structure and context.
Would you consider adding llms.txt to your site?