← All posts
·4 min read

llms.txt Explained: Should Your Site Have One?

You've probably seen people adding an llms.txt file to their sites. Some swear it's the new robots.txt for AI. Others say it does nothing. Let's cut through the noise.

What llms.txt actually is

llms.txt is a proposed standard. It's a plain markdown file you place at the root of your site — yoursite.com/llms.txt — that points language models to your most important content.

Think of it as a curated map. Instead of making an AI crawl your entire site and guess what matters, you hand it a clean, structured summary up front.

The format is simple by design. It's markdown, not XML or JSON. Anyone can read it, and so can a model.

What the spec looks like

The proposed format is loose but has a rough shape:

  • An H1 with your site or project name.
  • A short blockquote-free summary describing what the site is about.
  • A few sections with links to key pages — docs, about, pricing, core articles — each link followed by a brief description.
A stripped-down example looks like this: 
# OptimizeCamp

> A tool that checks whether your content is ready to be cited by AI search engines.


Docs



About
There's also a convention of offering llms-full.txt — a larger file with the actual content inlined, not just links — so a model can ingest everything in one pass.

The problem it tries to solve

Here's the pitch. LLMs have limited context windows. Crawling a full website is messy — navigation, scripts, ads, boilerplate. A lot of tokens get wasted on stuff that isn't the point.

llms.txt is meant to fix that. You do the curation. You tell the model, "Here's what's worth reading, in clean markdown, no noise." In theory, that makes your content easier to understand and easier to cite.

It's the same instinct behind good generative engine optimization — make your content legible to machines, not just to Google's ranking algorithm.

The honest debate: does anyone use it yet?

This is where I'll be straight with you. As of now, there's no strong public confirmation that the major AI crawlers — the ones behind ChatGPT, Claude, or Perplexity — actually read llms.txt during answer generation.

The major providers haven't broadly committed to it the way search engines committed to robots.txt and sitemaps years ago. Adoption is grassroots. Plenty of dev-tool companies and docs sites have shipped one because it's cheap and might help. That's different from it being a confirmed ranking or retrieval signal.

So treat the breathless takes with skepticism. "llms.txt is the new SEO" is overselling it. Right now it's a low-cost bet, not a proven channel.

Who should actually bother

Given the uncertainty, here's my practical read:

  • Worth it now — Documentation sites, developer tools, and API products. Your content is already structured, your audience uses AI assistants heavily, and a clean llms.txt costs almost nothing to maintain.
  • Reasonable bet — Content sites and blogs where you want specific cornerstone articles to be the canonical reference an AI reaches for.
  • Skip for now — Brochure sites, small local businesses, anything where you'd spend more time maintaining the file than you'd ever get back.
The honest framing: this is insurance, not strategy. If crawlers do adopt it, you're early. If they don't, you've lost an hour.

How to write one

If you decide to ship one, keep it tight:

  • Step 1 — Start with the name and summary. One H1, one or two sentences on what your site is and who it's for.
  • Step 2 — List only your best pages. Don't dump your sitemap. Pick the pages you'd actually want an AI to cite or summarize.
  • Step 3 — Describe each link. A short phrase after every link telling the model what it'll find there. This is the part that adds real value.
  • Step 4 — Keep it in clean markdown. No HTML, no clever formatting. The whole point is legibility.
  • Step 5 — Maintain it. A stale llms.txt pointing at dead pages is worse than none. Update it when your content changes.
The same discipline that makes a good llms.txt — clear structure, plain language, strong summaries — is exactly what helps with getting cited by ChatGPT regardless of whether the file itself is read.

If you're going to invest in AI search readiness, start by knowing where you actually stand. That's why I built OptimizeCamp — it audits whether your content is structured and clear enough to get cited by AI search engines, llms.txt or not. Check your site, then decide if the file is worth your hour.

← All posts