One trap when you name AI bots specifically is that is that robots.txt groups don’t merge. Crawlers will only follow the most specific group so if GPT bot has its own group named nothing under User-agent: * applies to it anymore so you need to repeat every disallowed line under each group.
MeshArc 2 days ago [-]
[flagged]
Hissigh 4 days ago [-]
Are llms.txt actually read by bots now? I found one experiment that showed that no bot has accessed the file in 3 months.
I get the odd one try on my personal site, but usually its just robots.txt... /shrug
literallyroy 4 days ago [-]
I once had oh-my-pi read a website’s llms.txt while doing research many months ago.
kenny_io 3 days ago [-]
[flagged]
3 days ago [-]
CqtGLRGcukpy 4 days ago [-]
Most of what the author is doing won't actually help.
> LLMs ignore Markdown files — log analysis showed no evidence that GPTBot, ClaudeBot, or similar crawlers request or prioritise .md content, even when listed in llms.txt.
why? You're helping AI companies to take your writing for free
mmcallister 4 days ago [-]
Whilst I agree with most criticism of AI/LLMs but most of the time at the other end of someone using Claude/ChatGPT there is still a human there, a human that is trying to solve a problem or learn something. Maybe I'm naive and it's just all bots scraping sites to build their corpus of data, but at least the way I use AI is to solve problems and build things I couldn't do before (either because of time, or it was out of my reach)
1shooner 4 days ago [-]
Sooo, are they reading the md?
lofaszvanitt 2 days ago [-]
Maybe read how webfetch and websearch works inside the harnesses.
https://www.rebootonline.com/geo/llms-txt-experiment/
> LLMs ignore Markdown files — log analysis showed no evidence that GPTBot, ClaudeBot, or similar crawlers request or prioritise .md content, even when listed in llms.txt.
https://www.longato.ch/llm-md-files/
> Across all 50 sites, across 200+ days, across every major AI user-agent logged - the `/llms.txt` endpoint was requested zero times.
https://mishamanko.com/research/what-ai-bots-actually-read