How AI engines read creator sites
llms.txt is hygiene. The written archive is the lever.
Short answer
Not on its own. Answer engines do not go looking for llms.txt files, and in the Ahrefs study almost none of the files received any traffic. Treat llms.txt as hygiene: a clean map that takes minutes and belongs on every creator site, which is why Archive Starter includes one. The lever that earns citations is what the map points to: a written archive of your videos on your own domain.
Why is llms.txt pitched as an AI shortcut?
An llms.txt file is a markdown file placed at the root of a domain that summarizes what the site offers and points toward its most important content. Defined in a community proposal (see our glossary), it was created to give automated systems a clean map of a website without forcing them to parse complex navigation trees.
The idea has since acquired a commercial pitch. The promise is that publishing an llms.txt file puts a website on the radar of answer engines and large language models. Under this view, a simple text file is a signal that attracts citations, so that answer engines notice your work and cite your pages.
The pitch appeals to creators for the same reason schema audits do: it sounds fast, cheap and technical. Adding a file takes minutes, requires no redesign, and feels like closing a gap. Schema advice makes the same mistake, as our note on what schema does for AI citations shows: both treat a label for content as if it were the content.
Do AI engines actually read or cite llms.txt files?
When researchers look at how automated systems behave in practice, the reality looks very different from the promotional pitch. In the Ahrefs study of llms.txt adoption, 28% of the 137K domains using Ahrefs Web Analytics publish an llms.txt file. Plenty of site owners have tried the format.
However, adoption has not translated into crawler activity or citation volume. In that same Ahrefs survey, 97% of those files received zero traffic in May 2026. Publishing the file and having it used are different things.
The reason is simple. As the Ahrefs investigation points out, answer tools do not go hunting for an llms.txt file, so publishing one does not put a site on their radar. Assistants and search crawlers find content the way they always have: by following links, querying search indices and retrieving documents during live search. They do not scour the web for root text files on domains they have no reason to inspect.
Publishing the file does not create discovery where discovery did not already exist. If an engine has no reason to examine your domain in the first place, adding an index file changes nothing.
What did our audit of creator websites find?
In The Dark Archive, our research into long-form YouTube creators, we scored 528 creator websites to measure how accessible and readable their material was to modern machines. As part of that investigation, our crawler flagged every domain that returned content at the standard file location.
When we re-fetched the files our crawler had flagged on a subset of creator domains, most of the addresses flagged were not genuine, creator-authored files at all.
Our inspection revealed three distinct patterns across those responses:
- Missing-page fallbacks. Many server configurations serve a generic error template or a homepage redirect instead of a proper not-found status when a nonexistent file is requested.
- Automated platform outputs. The file was generated by the site’s platform or a plugin, not written by the creator.
- Authored summaries. Only a small handful of the examined files showed signs of being deliberately drafted by the creator to summarize their own original work.
The lesson applies to any web audit: a server response at a specific path says nothing about whether the site owner made an intentional choice. In the subset we re-checked, a hand-written file was the rare exception; the rest were by-products of hosting software or fallback pages.
The full llms.txt and crawler audit, including our complete methodology and technical breakdown, is detailed in The Dark Archive report.
Why does every archive we build still ship an llms.txt file?
If the file earns no citations by itself, why does ExDarkMatter publish one on this domain, and why does every Archive Starter include one?
Because it is hygiene, like clean semantic markup and descriptive metadata: accurate, orderly and cheap. An llms.txt file offers an unambiguous summary of what an archive contains to any software agent built to look for it. It takes minutes to write and keep current. So we include it, done for you, and we tell you exactly what it is: a map of your archive, not a magnet for citations.
What we never do is sell the file as a shortcut to citations. A text file will not rescue a channel from obscurity. A map can organize knowledge that exists; it cannot invent authority or stand in for written substance.
Creators get competing advice about technical fixes. Some start with crawler policies, as discussed in our note on what blocking AI crawlers costs a creator’s domain. Others spend days fine-tuning markup or tweaking server headers. Technical hygiene pays off only after the harder problem is solved: substantial, original written answers on a domain the creator controls.
What earns citations, if llms.txt does not?
Start with how answer engines operate.
Answer engines do cite YouTube directly. As we explore in our analysis of which AI engines cite video, several major assistants frequently link directly to video URLs when responding to user questions. Citation behavior varies by platform and query: certain platforms include video links, others lean toward written pages.
More importantly, a single, nuanced point buried midway through an hour-long video is harder to cite than the same point stated in writing. When an assistant quotes a source, it looks for clear statements that directly resolve a specific query. A written article with a descriptive heading and an explicit answer near the top provides the exact shape that automated systems extract and credit.
The order of work is clear:
First, turn your videos into written articles. Every video essay, documentary or interview holds knowledge locked in audio and video. Readable articles on a website you own are the raw material machines quote.
Second, structure each article around a discrete question. State the answer clearly in the opening paragraph, provide supporting context beneath it, and link back to the exact timestamp in the original video.
Third, make sure your server does not block search crawlers, so they can index your written work.
Then add the finishing touches: semantic markup, structured data and an llms.txt file. Treat the finishing touch as the starting point and you get a well-labeled table of contents pointing to empty shelves.
Archive Starter does all of this in one order: your most-viewed videos become full articles, each linked to its video, with a hub page that ties them together and the llms.txt map included, ready to publish on your own domain.
Where should your archive start?
Start by measuring where your channel stands. The free Dark Matter Check gives you your tier on the Dark to Sovereign ladder and your machine readability score, by email, with no card needed.
For the full diagnosis, Report Pro breaks your score down component by component, checks whether Google’s Gemini and Anthropic’s Claude cite your channel, and hands you a prioritised fix list, so you know exactly what to do first.
The complete audit of machine readability, crawler permissions and tiers across every channel in the study is in The Dark Archive report.
Run the free Dark Matter Check and see how much of your channel is still dark.
Sources
- Ahrefs: llms.txt study, retrieved 2026-09-30