Skip to content

Generate llms.txt and per-page Markdown for the docs - #69

Merged
xhan97 merged 1 commit into
mainfrom
docs/llmstxt
Aug 15, 2026
Merged

Generate llms.txt and per-page Markdown for the docs#69
xhan97 merged 1 commit into
mainfrom
docs/llmstxt

Conversation

@xhan97

@xhan97 xhan97 commented Aug 15, 2026

Copy link
Copy Markdown
Collaborator

Adds mkdocs-llmstxt, which publishes /llms.txt as an index of the documentation plus a .md counterpart for every page.

Feeding these docs to an LLM currently means copying rendered HTML, which loses the structure that makes it useful.

What gets generated

llms.txt (4K) — an index, one entry per page, grouped by section:

# IKPyKit Docs

> Python library for Isolation Kernel Toolkit.

IKPyKit implements Isolation Kernel methods for anomaly detection, clustering,
and representation learning, over points, graphs, groups, streams, trajectories,
and time series. The estimators follow the scikit-learn API.

## API reference

- [INNE](https://isolationkernel.github.io/ikpykit/api/anomaly/inne.md)
- [IDKC](https://isolationkernel.github.io/ikpykit/api/cluster/idkc.md)
...

llms-full.txt (208K) — every page expanded, for pasting a whole corpus at once.

24 per-page .md files, one per documentation page.

Why it works on the API pages

The plugin renders each page to HTML and converts it back to Markdown, rather than reading the source. That distinction matters here. Every API page is a single directive:

::: ikpykit.anomaly.INNE

A source-level export would produce exactly that and nothing else. Going through the rendered output captures the full mkdocstrings result:

## ikpykit.anomaly.INNE

​```python
INNE(n_estimators=200, max_samples="auto", contamination="auto", random_state=None)
​```

Bases: `OutlierMixin`, `BaseEstimator`

Isolation-based anomaly detection using nearest-neighbor ensembles.
...

Parameters:

| Name           | Type  | Description                                    | Default |
| -------------- | ----- | ---------------------------------------------- | ------- |
| `n_estimators` | `int` | The number of base estimators in the ensemble. | `200`   |

This is also why the plugin has to run after mkdocstrings and mkdocs-jupyter, hence its position last in the plugins list. A comment in mkdocs.yml records that, since the ordering is not obvious and silently degrades if changed.

Coverage

Sections mirror the nav. Two entries needed care:

  • user_guides/inne.ipynb is listed explicitly — the user_guides/*.md glob does not match .ipynb
  • authors/authors.md was missing from my first pass

Verified by diffing every page in the nav against the generated index; the difference is now empty.

Caveats worth recording

The plugin is in maintenance mode. Its README states the author has moved to Zensical and is open to transferring maintainership. It works and is stable, but will not gain features. The configuration is about twenty lines in mkdocs.yml, so backing it out later is cheap.

Material has no "Copy as Markdown" button. squidfunk/mkdocs-material#8382 requested exactly that and was closed as a duplicate in August 2025, with the maintainer putting it on the backlog pending the end of feature freeze. So the .md URLs are the entry point for now, rather than an in-page button. If Material ships the button later, it composes with this rather than replacing it.

mike deploys per version, so these files land under each version prefix (/0.3.0/llms.txt) rather than at the site root. Whether a root-level /llms.txt is wanted is a separate question — it would need a redirect or a base_url override.

Minor conversion artifacts. The notebook page picks up one permalink and one Copied! from the rendered HTML. The plugin supports a preprocess hook to strip these, but for two occurrences in a 75-line file that seemed like more maintenance surface than it is worth.

Verification

mkdocs build --strict passes, uv lock --check is consistent, and all six pre-commit hooks pass.

Adds mkdocs-llmstxt, which publishes /llms.txt as an index of the
documentation and a .md counterpart for every page. Feeding docs to an
LLM currently means copying rendered HTML, which loses the structure that
makes it useful.

The plugin renders each page to HTML and converts it back to Markdown,
rather than reading the source. That matters here: the API pages are a
single `::: ikpykit.X` directive each, so a source-level export would
produce nothing. Going through the rendered output captures the full
mkdocstrings result -- signature, description, and parameter tables. It
also means the plugin has to run after mkdocstrings and mkdocs-jupyter,
hence its position last in the plugins list.

Sections mirror the nav. The INNE notebook is listed explicitly because
the user_guides/*.md glob does not match .ipynb, and authors.md is
included so every page in the nav is covered; verified by diffing the nav
against the generated index.

full_output produces llms-full.txt (208K) with every page expanded, for
pasting a whole corpus in one go.

Two caveats worth recording. The plugin is in maintenance mode upstream,
its author having moved to Zensical, so it works but will not gain
features. And mike deploys per version, so these files land under each
version prefix rather than at the site root.
@xhan97
xhan97 merged commit 21e5af6 into main Aug 15, 2026
17 checks passed
@xhan97
xhan97 deleted the docs/llmstxt branch August 15, 2026 18:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant