There’s been a lot of chatter recently in the technical SEO sphere around the benefits and practicalities of an llms.txt file and what this means for your website and SEO strategy.
Companies are spending lots of time and money in order to generate what they perceive to be the perfect llms.txt file that they believe is going to help them get discovered within AI tools. There are, however, common misconceptions of how a llms.txt file is used and, therefore, what benefits there are to be gained.
At Dig & Dig, our SEO team have put their heads together in order to explain exactly what an llms.txt file is, what those common misconceptions are and when it’s worthwhile creating an llms.txt file.
The llms.txt file is a proposed web standard (first published Sept 2024 by Jeremy Howard) for helping large language models (LLMs) and AI agents understand a website’s content. It lives at the site root (e.g. example.com/llms.txt) and is written in Markdown. The llms.txt file is meant to guide AI tools by providing a concise, curated map of the site, effectively “flattening” your site’s content into plain text summaries, making it easier for an LLM to parse without wrestling with complex HTML.
The idea is that when an AI agent on demand seeks information, it can fetch this llms.txt and jump straight to the relevant content, rather than crawling every page. The format is explicitly machine-readable (as Markdown with a rigid structure), so information can be extracted automatically.
Essentially, llms.txt should be thought of as a “table of contents” for LLMs, complementing sitemaps and robots.txt files by focusing on AI-friendly content, not web indexing rules.
The key things to remember are that the llms.txt is not a ranking or discovery signal. Both Google and other AI providers have made this clear, with Google’s guidance on the matter explicitly stating that you don’t need any special AI text files or markup in order to appear for generative AI features.
John Mueller, the Senior Search Analyst and advocate, emphasises that an llms.txt file was not meant to make a site discoverable, but to help once an agent is already on site.
You remember the old adage of “if everything is a priority, then nothing is” – well, the same applies in this situation. If every site makes the claim in their llms.txt file that their sites contain the best content on certain subject matters, then there’s no actual way that LLMs can trust that self-reported signal.
According to a recent study conducted by Ahrefs, they found that out of the 137,000 domains analysed, 97% of their llms.txt files received 0 hits from any crawler.
What’s even more shocking is that only 1% came from AI-related bots (such as ChatGPT, Claude, Perplexity etc).
It’s definitely worth checking your log files in order to see just how often AI bots are crawling your site (and by extension your llms.txt file, if you already have one).
Again, this is simply going against what a lot of the data is saying. Search Engine Land conducted a study where they tracked 10 diverse sites for 90 days before and after adding llms.txt.
The result was that 8 out of 10 saw no change in AI traffic, and the two that did see bumps (+12.5% and +25%) had introduced new content and PR campaigns at the same time.
In other words, llms.txt did not drive the gains – new valuable content and outreach did, an aspect that points towards true foundational SEO efforts that drive success.
As the evidence suggests above, the adoption of an llms.txt file remains extremely small across the web as a whole. However, the picture shifts considerably among developer-focused companies. Documentation-heavy SaaS businesses, including OpenAI, Anthropic, Stripe, Cloudflare and others, have either published llms.txt files or equivalent AI-readable documentation.
It’s because coding assistants like Claude Code, Cursor and GitHub Copilot genuinely consume structured documentation when helping developers write code.
What I do want to make clear is that I’m not wanting to discourage the use of llms.txt files; there are some legitimate use cases and contexts where it is useful.
Vercel reports that approximately 10% of their user signups come via ChatGPT. Their llms.txt includes clean API descriptions, so AI coding assistants know what content to fetch. In this “developer or AI agent” scenario, the efficiency gains matter: Markdown saves token count and simplifies parsing, which is valuable when AI tools (like GitHub Copilot, Cursor, or Claude Code) are integrated into a workflow.
Packaging up your best content for AI consumption so that an agent can answer a user’s question and find the right material is another ideal use case. OpenAI, Stripe and Cloudflare are real-world examples where their developer and help sites expose Markdown or text versions of docs and maintain llms.txt-style guides to support AI-based lookup.
As AI shopping assistants and autonomous agents become more capable, they may be searching ecommerce sites on behalf of users. Product comparisons, delivery options, and compatible accessories could be requested, in which case an llms.txt file could help agents locate these important aspects quickly, reducing the amount of HTML they are processing
Just like with every great SEO question, the answer boils down to: it depends.
For a standard marketing website, or a blog, an llms.txt file most likely shouldn’t be high up on your priorities list. The time used to create this could be better served by focusing on creating original, helpful content and fixing technical issues across your site.
For a site which is heavier on its technical documentation, however, if your users or tools rely on AI, then an llms.txt file is worth implementing. A SaaS platform, for example, that contains lots of product docs, code samples, and technical details might gain efficiency by aggregating its key pages into an llms.txt, especially when token savings matter.
Jenny Eckloff, SEO Account DirectorOne of the key things to remember is that Discovery isn’t handled by the llms.txt file – this is still handled by HTML and links. Just like a sitemap.xml, it can help machines parse your site efficiently, but that doesn’t make the actual content on the site authoritative.
Not only this, but interestingly enough, even though Google says that the llms.txt isn’t needed for AI visibility, Lighthouse now flags if a site does have one as part of its Agentic Browsing audit. Essentially, pointing towards the fact that this particular audit is to help AI agents interpret site structures, NOT for Google ranking.
If you’re looking for more information on whether an llms.txt file is right for your site, or if you’re intrigued about improving your on-page content or technical issues, then reach out to Dig & Dig and start the conversation at hello@diganddig.com and get put in contact with our expert SEO team.
Expertly led, expertly done. Our approach goes deep, and so does our experience.
SEO Account Director
Having worked in the SEO industry for 8+ years, where she has gained experience in content & Digital PR, Jenny has carved her niche and specialised in all things Technical SEO. Having worked on a range of clients (such as Argos, Skatehut, HiPP Organic, GAME) in an array of different industries during her career, she excels at problem-solving and all things led by data.
