Markdown and llms.txt: should you serve AI a second site?

Markdown is a text format with simple marks: a hash in front of a heading, a dash in front of each list item. A model can process it without discarding menus, scripts and styles, which take up a large share of the code in an HTML page.

llms.txt is a Markdown file at the root of a site that summarises what the site is and links its important pages. Jeremy Howard of Answer.AI proposed it on 3 September 2024 at llmstxt.org, and the current version of the proposal, the second, was modified on 10 August 2026. It is a proposal, not an approved standard.

The third way is called content negotiation: the crawler requests the page with the header Accept: text/markdown, which roughly says “I prefer Markdown”, and the server answers with that version at the same address.

• Google: its guide to optimising for generative AI features on Google Search, last updated on 10 July 2026, says “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search”, because Google Search itself does not use them.
• OpenAI, Anthropic and Perplexity: the pages where they document their crawlers, such as OAI-SearchBot, Claude-User or PerplexityBot, explain what each one is for and do not say that they read llms.txt or Markdown versions. Checked on 13 November 2026.
• Those same pages do offer Markdown to people reading documentation: OpenAI’s points to its llms.txt and to a .md version of each page, and Perplexity’s links its llms.txt.
• Coding agents: Cloudflare wrote on 12 February 2026 that agents such as Claude Code and OpenCode already send the header that asks for Markdown.

An AI company publishing an llms.txt for its documentation does not mean its assistant uses the file to decide whom to cite. They are two separate things, and only the first is documented.

Chrome looks at the file as well: Lighthouse, its auditing tool, checks it as part of its agentic browsing audits. According to Chrome’s documentation, if a site has no llms.txt the audit is marked as not applicable, because providing the file is optional for now.

In the Ahrefs study, 28% of the 137,000 domains publish an llms.txt, and Ahrefs itself warns that its customers are more technical than the web at large, so that adoption is an upper bound. Of the requests that did reach an llms.txt, 96% came from bots.

SE Ranking tested the variable with statistical analysis and with a machine learning model, and removing llms.txt from the model made it more accurate. Its conclusion, in a study published on 15 October 2026 according to its own page, is that the file does not currently affect how AI systems see or cite content.

• The text in the initial HTML. If prices, FAQs or product pages are loaded later by JavaScript, a crawler that does not run it only sees the header and the footer.
• A clean heading hierarchy: one H1, an H2 for each question the page answers and H3s below. It is the same structure Markdown marks with hashes, so well-ordered HTML already gives almost everything a .md would.
• Tables in HTML, not in an image. No assistant can quote a figure inside a screenshot.
• Open access: neither robots.txt nor the firewall should block the search crawlers of each assistant. You can check whether AI crawlers can read your site for free and without giving an email.

Once the HTML is in order, what decides whether an assistant cites a page is how it is written. That part, with explicit definitions, sourced figures and sections that stand on their own, is covered in the guide to writing content that AI assistants cite.

One example: DoGeo’s guide on how to rank in ChatGPT has a Markdown version with the same text, and the dogeo.io llms.txt opens with DoGeo’s definition in Spanish, English and French before linking its pages.

The condition that matters is that both versions say the same thing, word for word. A .md version that strips menus and styles is the same page in another format; one that says things the page does not is a different site.

The line is drawn by the content, not the format. A crawler version with claims, prices or reviews that the page does not show comes close to what those spam policies describe.

There is a second, more practical risk: two versions maintained by hand end up saying different things. A price changed on the page and forgotten in the .md is a false figure that an agent can repeat to a customer. That is why the Markdown should come from the same content as the HTML.

• Technical documentation, APIs and developer software: both. These are the sites coding agents read, and OpenAI and Perplexity do the same with their own documentation.
• Services firms, law firms, clinics or agencies: the HTML first. llms.txt is optional; it is a place to write down the company’s definition, and in Google it neither adds nor subtracts, because Google ignores it.
• Online shops: first make sure every product page carries price, availability and description in the HTML. A .md version does not fix a product page that is built with JavaScript.
• Sites that depend on JavaScript to show their text: neither llms.txt nor Markdown is the priority. The first job is for the server to deliver the text.

The rule logs without blocking the requests of each crawler, identified by its user-agent, the name it presents itself with, and verified by IP when the provider publishes its ranges. This is what DoGeo sets up for each client.

Each type of crawler tells you something different. Search crawlers, such as OpenAI’s OAI-SearchBot or PerplexityBot, show which pages enter the assistant’s index. The ones that open a page because a person asked for it in a conversation, such as ChatGPT-User, Claude-User or Perplexity-User, are the closest signal that the assistant read that page to answer. GPTBot, which OpenAI uses for its models, only speaks to training.

The logs do not show what the assistant writes in its answer. That takes a fixed set of prompts, as explained in the guide to tracking rankings and AI visibility.