Audit or draft an llms.txt

An llms.txt is a short markdown file at the site root that tells AI assistants what the site is and which pages to read first. It is the AI-era equivalent of a curated sitemap. Adoption by crawlers is uneven, so it is a cheap bet rather than a guarantee, but a good one takes an hour and a bad one misleads every assistant that does read it.

Sequel supplies the traffic data that decides what belongs in the file. Fetching the existing file and the pages uses your own browsing.

Steps

1. Fetch what exists. Request /llms.txt and /llms-full.txt. A 404 on the first means you are drafting; skip to step 4. If it exists, keep going.

2. Validate the structure. The spec expects: one H1 with the site or project name, a blockquote with a one-paragraph summary, optional prose, then H2 sections each containing a list of - [Title](url): description entries, with an optional final section named Optional for lower-priority links. Report deviations.

3. Validate the links and the coverage. Fetch every listed URL and check it returns 200, is not noindex, and is the canonical form. Then pull google_search_console.query with dimensions: ["page"], 90 days ending 3 days ago, rowLimit: 300, and list the top 50 pages by clicks that the file does not mention. Those are the omissions that matter.

4. Draft, if missing. From the same Search Console call, take the top pages by clicks and group them into sections that match how the site is organised (documentation, guides, product, pricing, blog). Fetch each chosen page to write a one-line description from its title and meta description in your own words. Keep the main file under about 100 links; push the rest to Optional or to llms-full.txt. Do not include gated, private, login or utility URLs.

5. Report. For an audit: the structural findings, the broken or non-canonical links, and the missing top pages. For a draft: the complete file in a markdown code block, ready to save at the site root, plus a note on where it goes and that it does not replace the XML sitemap. Continue with ai-crawler-access-check if the crawlers that would read it are blocked.

Gotchas

• Do not promise citations. The file helps assistants that read it; not all do. Present it as low-cost hygiene.
• Curate, do not enumerate. A file listing every URL is a sitemap with extra steps. The value is the selection and the descriptions.
• Keep it maintained. A stale llms.txt pointing at moved pages is worse than none. Suggest regenerating it when the top pages change.
• Markdown versions of pages are a separate convention. Some sites also serve .md copies of pages; that is optional and not required for llms.txt to be useful.
• Search Console lags 2 to 3 days. End the window 3 days back.