llms-txt-freshness: how should coverage handle intentionally excluded pages?

[mvvmm]

The llms-txt-freshness check compares llms.txt URLs against the sitemap to measure coverage, with thresholds at 95% (pass) and 80% (warn).

For our Cloudflare docs, I'm intentionally omitting certain pages that exist in our sitemap from llms.txt because they have no value for agents. For example, https://developers.cloudflare.com/workers/reference/ is a directory listing page — it contains a few links to other pages but no substantive content. I believe it belongs in the sitemap (it's a real page on the site) but not in llms.txt (the content on this page - links to other pages - already exist in llms.txt).

In our internal audits I'm currently working around this by just hardcoding lower thresholds (pass ≥ 75%, warn ≥ 60%) in a local patch, but I'm not sure what the right universal solution is. Does it make sense to treat every sitemap page as something that should be in llms.txt? If not, whats a reasonable and universal way to determine which ones should and shouldn't be included?

Open to discussion and whatever direction makes sense for the library.

[dacharyc]

Hah, you're the second person to file an issue about this, @mvvmm - someone filed #17 a few weeks ago for a similar question.

I think this is a change that affects both the spec and the implementation. When I added this check in the spec, it was for a case where the llms.txt was incomplete; where I observed this issue, there had been unintentional drift between the sitemap and llms.txt. You're right about it not handling the intentionally-curated case; I hadn't encountered that and didn't consider it when drafting the spec.

I've opened two new issues on the spec repo related to this:

• Rename to llms-txt-coverage (should be uncontroversial): Rename llms-txt-freshness check to llms-txt-coverage agent-docs-spec#16
• Explore design options to handle intentional exclusions: Coverage check: distinguish intentional curation from accidental gaps agent-docs-spec#17

If you have the time and inclination to weigh in on the design options conversation, I'd love to have your feedback there.

Meanwhile, I think it makes sense to leave this issue open while we discuss what the behavior and possible measurement mechanisms should be, and then we can bring the design decisions back here for implementation.