llms.txt: What It Is, Whether It Works and How to Create the File for Your Website
Contents
- What is llms.txt and who proposed it
- What the file looks like: structure and example
- Does llms.txt work for SEO and search engines
- How it differs from robots.txt and sitemap.xml
- How to create llms.txt in one hour: a step-by-step plan
- What to include in the file and what to avoid
- Should you implement llms.txt right now
- Frequently asked questions
The llms.txt file is a Markdown text document placed in the website root directory for language models and autonomous AI agents. It contains a structured project sitemap with links to key materials and concise summaries of them. As a result, algorithms can quickly grasp the core of the resource without needing to download and strip complex HTML page markup.
Let's look into who proposed this format, whether modern search engines like Google consider it, how it differs from standard robots.txt and sitemap.xml files, and how to prepare a proper file for your own resource in one hour.
What is llms.txt and who proposed it
The idea of creating a dedicated file for large language models was introduced by programmer and researcher Jeremy Howard, founder of Answer.AI, in September 2024. The llms.txt format was conceived as an open developer community initiative. It is not an official standard from the W3C consortium or the IETF engineering group, but a practitioner-proposed agreement on how websites can serve key data to machines in a convenient format.
The main problem the format solves is web page clutter. When an autonomous agent or language model loads a standard HTML page, service data takes up most of the document size: analytics scripts, cascading style sheets, navigation menus, headers, footers, and banners. The valuable content of an article or service description often takes up only a fraction of the page. The model has to spend computing resources and its available context window filtering out noise.
The llms.txt format offers a simple solution: a website places a single file at the root at domain.com/llms.txt. The document is written in clean Markdown markup, which models understand without additional preprocessing. Inside the file, the webmaster lists key website sections, documentation, or blog articles with brief summaries. The agent loads this document, immediately sees a semantic map, and follows only the links needed to answer the user.
What the file looks like: structure and example
The llms.txt syntax relies on standard Markdown specification. The file consists of several required sections arranged in a strict hierarchy:
- An H1 heading with the exact name of the company, product, or project.
- A blockquote description right below the heading that explains the project's purpose in two sentences.
- H2 sections grouping links by topic: services, documentation, articles, contacts.
- Link lists in a specific format: page title, exact absolute URL, and a short description of what is inside.
Below is a compact example of a completed file for a fictional web studio from Kazakhstan.
# Example Studio > Web development and search engine optimization studio in Almaty. We create corporate websites, online stores, and set up end-to-end web analytics for business. ## Main services - [Website development](https://example.com/razrabotka): creating corporate websites, catalogs, and web services - [Search engine optimization](https://example.com/seo): technical audit, page structure and content optimization for search - [Analytics setup](https://example.com/analytics): web analytics, dashboards, and conversion tracking ## Expert materials - [Structured data guide](https://example.com/blog/schema): instructions for implementing Schema.org in JSON-LD format - [Technical audit checklist](https://example.com/blog/audit): steps for self-checking website accessibility ## About the company - [Contacts and legal details](https://example.com/contacts): office address in Almaty, phone numbers, working hours, and legal details Along with the base file, Howard's specification outlines an extended version: llms-full.txt. While the basic llms.txt acts as a table of contents with links, llms-full.txt provides a complete summary of the entire resource. It includes the full text of guides, API documentation, or key website sections without graphics and unnecessary code. This option is especially popular for open-source libraries, where a developer or AI assistant needs to quickly load all documentation into the context with a single request.
Does llms.txt work for SEO and search engines
Unreasonable expectations quickly formed around the llms.txt format. Some webmasters assumed that creating a text file would guarantee that their website appears in ChatGPT, Claude, or Google AI Overviews. The reality is quite different.
Major search engines have not officially confirmed that they consider the llms.txt file when generating search results or AI responses. Google representatives have explicitly stated that the company's crawler does not use the llms.txt file to rank websites and does not rely on it when choosing search sources. Search engines like Googlebot and YandexBot historically rely on crawling HTML page code, sitemap.xml sitemaps, and link connections.
Nevertheless, the file provides practical value in other scenarios:
- Autonomous AI agents and web scrapers. When a user sends an agent system to explore a specific company's website, the agent looks for structured cues first. If llms.txt is available at the root, the agent immediately understands the structure of services without wandering through dozens of links.
- Developer tools. Some code editors and AI assistants can pull in llms.txt when connecting documentation to a project.
- Local language models. Users who connect websites to their knowledge bases can feed them llms.txt as a ready-made, verified index.
Creating the file provides no advantage in traditional ranking, but it makes life easier for tools working with generative AI.
How it differs from robots.txt and sitemap.xml
To understand where the new format fits in a website's technical structure, let's compare it with standard utility files.
| Parameter | robots.txt | sitemap.xml | llms.txt |
|---|---|---|---|
| Primary purpose | Controlling search crawler access to website sections | Providing a complete list of URLs for indexing | Providing a concise table of contents for language models |
| Primary audience | Traditional search crawlers and scanners | Google, Yandex, Bing search engines | AI agents, language models, development environments |
| Data format | Plain text list of User-agent and Disallow directives | Structured XML document with update dates | Human-readable Markdown with links and descriptions |
| Standard status | Official internet standard (RFC 9309) | Widely accepted search engine protocol | Open developer community initiative |
| Mandatory status | Recommended for every website to save crawl budget | Highly recommended for fast indexing | Optional supplementary document |
These three files complement rather than replace each other. The robots.txt file answers where bots are allowed to go. The sitemap.xml file informs search engines which pages exist on the website. The llms.txt file explains to language models which pages are the most important and what useful information they contain.
How to create llms.txt in one hour: a step-by-step plan
Preparing a high-quality file does not require special programming skills. You can complete all the work in six consecutive steps.
- Select 10 to 25 key pages of the website. The list should include the home page, main commercial services, the company information page with contacts and legal details, as well as several of the best analytical articles or guides. You do not need to include utility sections like the cart, personal account, filter pages, or pagination.
- Formulate an accurate description of the website's core purpose. Write two concise sentences for the quote block under the main heading. Specify the company's line of business, key specialization, and city of presence.
- Write brief summaries for each page. For each URL, write a clear explanation 5 to 15 words long. Avoid generic marketing phrases and state concrete value: what exactly the user or algorithm will find inside that link.
- Assemble the document in a text editor. Use standard Markdown syntax: one H1 heading, a blockquote using the greater-than symbol, H2 headings for topic groups, and lists with links.
- Place the file on the server. Save the document named llms.txt in UTF-8 encoding and upload it to the website's root folder via the hosting file manager or FTP. The URL must strictly follow the domain.com/llms.txt format.
- Configure file serving on popular CMS platforms. If the website runs on WordPress, you can place the file directly in the root directory via hosting, bypassing the database. Make sure that installed redirect or caching plugins do not attempt to rewrite requests for the text file to a URL with a trailing slash or redirect them to a 404 HTML error page.
What to include in the file and what to avoid
The effectiveness of llms.txt depends on the accuracy of the selected data. If you overload the file with redundant information, it will lose its main advantage: conciseness.
What is useful to include in the document:
- direct links to current price lists or pages with detailed service descriptions;
- links to detailed guides and expert articles that answer frequent client questions;
- contacts, physical office address, phone numbers, and legal entity name;
- links to the full documentation file llms-full.txt, if such a file has been prepared for the website.
What you should strictly avoid:
- including hundreds of repetitive product pages of an online store (sitemap.xml exists for this);
- using advertising clichés and empty phrases with no factual information;
- adding links with dynamic parameters, redirects, or 404 status pages;
- blocking the file from reading in robots.txt (if the file is blocked by a Disallow directive, bots cannot read it).
Should you implement llms.txt right now
Creating an llms.txt file takes about an hour of a marketer's or webmaster's time. Implementation costs are minimal, and risks are minimal if the file is formatted properly and contains working links.
At the same time, it is important to keep a realistic view of AI search optimization. An llms.txt file cannot compensate for technical errors, slow loading speed, lack of indexing, or empty, low-quality content. The foundation remains indexing, page usefulness, and how the brand is covered on third-party platforms. Treat llms.txt as a cost-effective groundwork for the future, useful for interacting with artificial intelligence, but continue to develop the foundation of traditional SEO.
Frequently asked questions
Is it mandatory to create an llms.txt file for every website?
No, the file is not a mandatory requirement of search engines or internet standards. The website will continue to be properly indexed and ranked in Google and Yandex without this document. Creating it makes sense for companies with expert content, documentation, and services that want to make things easier for AI agents.
Will llms.txt help a website appear in ChatGPT answers faster?
There is no confirmation of such an effect. OpenAI has not stated that it considers llms.txt when selecting sources for ChatGPT search, and search-assisted responses rely on regular pages and search indexes. The file can be useful for agents and tools that read it directly, but you should not count on it to get into responses.
How does the llms-full.txt format differ from the basic llms.txt?
The llms.txt file contains only the website structure: a title, a short description, and links with annotations. The llms-full.txt file combines the full text of key pages or documentation into a single text document, allowing the language model to read the entire body of information without following additional links.
How do you set up llms.txt on a WordPress website?
The easiest way is to upload the finished llms.txt text file to the website root folder via the hosting control panel or FTP. After uploading, open domain.com/llms.txt in your browser and make sure that .htaccess rules or security plugins do not block serving text files.
Can having an llms.txt file hurt regular SEO in Google or Yandex?
No, as long as it contains working links to pages open for indexing, it does no harm. For search crawlers, it is just a regular static text file. The main thing is to ensure that the data in the file matches the actual website content and is updated on time.
