fix(context): measure visible link text when skipping tables of contents by JayOfTheKeyboard · Pull Request #190 · neuledge/context

JayOfTheKeyboard · 2026-10-07T01:13:56Z

Problem

isTableOfContents skips a section when more than half of its content is links. It counted each link's whole Markdown, URL included, as link text:

const links = content.match(linkPattern) || [];
const linkTextLength = links.reduce((sum, link) => sum + link.length, 0);
if (content.length > 0 && linkTextLength / content.length > TOC_LINK_RATIO) {

A prose paragraph with two or three long URLs therefore reads as "mostly links" and is left out of the index. tRPC's AWS Lambda adapter page is one example. Its section saying which API Gateway versions are supported has 5 links. A reader sees 59 characters of link text in 322 (18%), but counted with their URLs the links are 436 of 699 characters (62%), so the section is dropped. It opens:

The AWS Lambda adapter is supported for API Gateway [REST API(v1)](https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-rest-api.html) and [HTTP API(v2)](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api.html), and [Lambda Function URL](https://docs.aws.amazon.com/lambda/latest/dg/urls-configuration.html) use cases.

HTML is hit hardest, because turndown writes Sphinx's cross-references with relative paths and title attributes ([`PyObject`](structures.html#c.PyObject "PyObject")) and every definition has a [¶](#c.PyDict_Check "Link to this definition") permalink. Most Python C API pages are one long section, so the whole page goes.

I parsed every file with parseDocument on main and with this change. The Python run uses the registry definition's exclude_paths:

Docs	Sections (main → fix)	Tokens (main → fix)	Pages with no sections (main → fix)
Python 3.14 HTML (538 pages)	4,676 → 5,331	2,904,920 → 3,236,191	154 → 51
tRPC	397 → 413	148,757 → 150,693	3 → 3
Kysely	142 → 149	39,773 → 42,317	2 → 0
Drizzle ORM	1,933 → 1,943	879,207 → 880,006	6 → 4
NestJS	902 → 902	615,405 → 615,405	0 → 0

Python pages that produce no sections on main include the C API pages for dict (10,403 tokens), list, set, tuple, datetime, sequence and monitoring, plus library/errno.html, library/token.html, library/asyncio-exceptions.html and library/keyword.html. The 51 still empty with the fix are the chapter contents pages (library/index.html, contents.html, library/datatypes.html, ...) and Sphinx's generated index pages (genindex*.html, py-modindex.html).

In the Markdown sets, the sections this change brings back are prose: tRPC's AWS Lambda and edge-runtime introductions and its useUtils helper table, the Drizzle release notes' "Fixes" lists, and Kysely's merge examples. I read all of them.

End to end, building a package from c-api/dict.html, library/errno.html and tRPC's aws-lambda.md with buildPackage and then calling search:

Query	main	fix
`PyDict_SetDefault`	no results	`dict.html > Introduction (part 5)`
`ECONNREFUSED`	no results	`errno.html > Introduction (part 5)`
`API Gateway HTTP API v2`	no results	`AWS Lambda + API Gateway Adapter > AWS Lambda adapter`

The package has 3 sections (1,493 tokens) on main and 25 sections (16,943 tokens) with the fix.

Fix

Measure what a reader sees. Each link is replaced by its text, the link text is summed along the way, and the ratio is link text over the resulting visible text. The 50% threshold is unchanged. Because only the URL and bracket characters are removed, the new ratio is never higher than the old one: this change can only keep a section that was dropped, never drop one that was kept.

Sphinx's generated index pages (genindex*.html, py-modindex.html) are alphabetical link catalogs, and their list markup and plain-text entries hold them under the ratio once URLs no longer count. parseHtml now returns no sections for a page that has both a Sphinx index file name (genindex*.html, or a domain index such as py-modindex.html) and a tag from Sphinx's index templates (<table class="indextable">, <div class="genindex-jumpbox">). A page with only one of the two is still indexed. No content page in the Python docs has either class, and no page has fewer sections than on main.

Not changed

The 50% threshold and the title rules (table of contents, toc, contents, index).
Images are still matched as links, as before (![alt](src) counts its alt text as link text). Treating them separately changed no section in the Markdown sets and 11 of 5,437 in Python, so I left it out.

Test

html.test.ts: a genindex landing page, a letter page and py-modindex.html produce no sections, and a normal page with the same Sphinx chrome is still indexed. On the previous commit the three index cases fail. Three more pages stay indexed: a guide with its own indextable table, a guide that shows Sphinx's index markup in a code sample and a comment, and a page named genindex.html without Sphinx's markup. Dropping either half of the check fails one of them.

build.test.ts: a section of prose with two long AWS links is kept, and a section that is a bulleted list of three links is still skipped.

On the parent commit the test fails: the prose section is dropped ([] instead of ["AWS Lambda adapter"]). With the fix it passes.

I broke each part of the fix in turn, and each time the test went red:

Mutation	Result
Count the whole link (URL included) again, over visible text	prose section dropped
Count link text only, but over the raw content (URLs in the denominator)	link list kept as a section
Remove the link-ratio check	link list kept as a section

Validation

vitest run in packages/context: 18 files, 351 tests pass.
biome ci --error-on-warnings: clean.
tsc --noEmit -p tsconfig.build.json: clean.
Changeset: .changeset/toc-link-text-ratio.md (patch).

changeset-bot · 2026-10-07T01:14:01Z

🦋 Changeset detected

Latest commit: 151e483

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
@neuledge/context	Patch
@neuledge/registry	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

human0-reviewer

Changes requested — This fix restores useful prose, but Sphinx's generated Python index pages now enter search too. Please keep those pages out of the index before merging.

Changes behavior: yes — searches gain recovered prose and generated navigation catalogs.

Must fix

Keep Sphinx index pages out of search. (packages/context/src/build.ts:68)

This codebase is run by Human0.

human0-reviewer · 2026-10-07T20:11:41Z

+  // Check link ratio - TOC sections are mostly links. Measure the text a reader
+  // sees: counting each link's URL would make prose with a few long URLs look
+  // like a TOC and drop it.
  const linkPattern = /\[([^\]]*)\]\([^)]+\)/g;


Must fix: Keep generated Sphinx indexes out of search.

This change lets Sphinx's genindex and py-modindex pages through because their generated navigation and footer text lowers the link ratio. These alphabetical link catalogs can surface instead of useful documentation in search; exclude the affected pages or remove that generated text before applying the ratio.

JayOfTheKeyboard · 2026-10-10T08:29:02Z

Thanks, good catch. With visible link text, genindex.html, genindex-E, -O, -Symbols, -all and py-modindex.html all produced sections (95 sections, about 95k tokens, in the Python 3.14 docs). On main they produced none.

I tried your second suggestion first. Removing Sphinx's navigation bar and footer before measuring is not enough on its own: genindex-E, -O, -Symbols, -all and py-modindex.html still pass. Their list markup, the plain-text parent entries and the module synopses keep the link ratio under 50% without any chrome. Stripping the chrome also changes the content of every other HTML page, which is a bigger change than this PR needs.

So 888119a leaves the pages out instead. parseHtml returns no sections for a page that carries the classes Sphinx's index templates emit (indextable on the index tables, genindex-jumpbox on the split index's landing page). No content page in the Python docs has either class.

Python 3.14 docs, same exclude_paths as the registry:

	Sections	Tokens	Pages with no sections
`main`	4,676	2,904,920	154
this PR before	5,426	3,330,767	45
this PR now	5,331	3,236,191	51

The difference is exactly those six pages. Every other page keeps the sections this PR recovered, and no page has fewer sections than on main. genindex-all.html is split into 3 chunks by splitForParsing and none of them is indexed.

The new tests in html.test.ts fail without the change (a genindex landing page, a letter page and py-modindex.html each produce a section) and pass with it. A fourth test checks that a normal page with the same Sphinx chrome is still indexed. The changeset mentions the exclusion.

search.html gives one section on both main and this branch, so I left it alone here.

human0-reviewer

Changes requested — The link-ratio fix restores useful prose and excludes Sphinx index catalogs, but the new filter can also hide real HTML documentation pages.

Changes behavior: yes — searches now include prose with long links and omit generated Sphinx indexes.

Must fix

Only skip actual Sphinx index pages.

This codebase is run by Human0.

human0-reviewer · 2026-10-10T14:40:20Z

 * Markdown parser for section extraction and chunking.
 */
 export function parseHtml(source: string, filePath: string): ParsedDoc {
+  if (SPHINX_INDEX.test(source)) {


Must fix: Only skip actual Sphinx index pages.

This checks for a bare indextable class anywhere in the raw HTML and returns no sections before HTML cleanup. A real guide can contain that class on its own table, or show Sphinx markup in a code sample or comment; when parsed, the whole page disappears from search without an error. Match the actual Sphinx index template or gate the exclusion on index filenames, rather than matching this class anywhere in the source.

Done in a65a2f9. A page is now skipped only when it has both a Sphinx index file name (genindex*.html, or a domain index such as py-modindex.html) and an actual <table class="indextable"> or <div class="genindex-jumpbox"> tag. Escaped markup in a code sample doesn't match the tag pattern. A guide with its own indextable table, a guide showing the markup in a code sample and a comment, and a genindex.html without Sphinx's markup are all still indexed, with tests for each. The Python 3.14 docs give the same output as the previous commit: 538 pages, 5,331 sections, no page differs.

The ordinary-guide issue is improved but not fully settled: [a-z]+-[a-z]*index also matches names like api-index.html, so a guide with that name and an indextable can still disappear from search. I also found a separate performance issue in the raw-HTML check; the current review asks to narrow the filename match and bound the check.

This codebase is run by Human0.

The earlier changes request was a duplicate run; I’ve re-reviewed a65a2f9 and approved it.

This codebase is run by Human0.

human0-reviewer

Approved — Prose with long URLs now remains searchable without admitting generated Sphinx index catalogs; both earlier review concerns are resolved at this head.

Changes behavior: no — This corrects what counts as visible link text and preserves the existing exclusion of generated index catalogs; it adds no new user-facing capability or contract.

This codebase is run by Human0.

human0-reviewer

Changes requested — The link-text fix restores useful prose, but the new HTML filter can still hide ordinary pages and take excessive time on crafted input.

Changes behavior: yes — search now keeps prose with long URLs and excludes generated Sphinx index pages.

Must fix

Restrict the filter to actual Sphinx index pages.
Keep the raw HTML check bounded.

This codebase is run by Human0.

human0-reviewer · 2026-10-11T13:01:56Z

+// them by the file name Sphinx gives them and a tag its index templates emit; a guide
+// with only one of the two is still indexed.
+const SPHINX_INDEX_FILE =
+  /(?:^|\/)(?:genindex(?:-[^/]+)?|[a-z]+-[a-z]*index)\.html?$/;


Must fix: Restrict the filter to actual Sphinx index pages.

The [a-z]+-[a-z]*index pattern also matches ordinary names such as api-index.html because * allows no characters before index. If that guide contains its own indextable table, the filter returns no sections and silently removes its content from search; a literal matching tag in a comment can also satisfy the raw-source check. Match Sphinx's actual names, such as genindex*.html and *-modindex.html, and add a test that a non-Sphinx api-index.html page remains indexed.

human0-reviewer · 2026-10-11T13:01:56Z

+// with only one of the two is still indexed.
+const SPHINX_INDEX_FILE =
+  /(?:^|\/)(?:genindex(?:-[^/]+)?|[a-z]+-[a-z]*index)\.html?$/;
+const SPHINX_INDEX_MARKUP =


Must fix: Keep the raw HTML check bounded.

A truncated page with repeated <table or <div tokens makes this pattern repeatedly scan and backtrack to the end of the input; a 448 KB input took about 30 seconds in testing. Since the filter runs on raw HTML before parsing, indexing one crafted page can stall the whole build. Bound the scan or check tag heads in a linear pass.

human0-bot · 2026-10-11T13:04:31Z

Thanks for the fix. The reviewer found two remaining issues on a65a2f9: make sure an ordinary api-index.html guide with indextable markup stays searchable, and bound the raw-HTML check so malformed or truncated pages cannot make indexing stall. Please push an update for another review; the full test suite, lint, and build pass on the current head.

This codebase is run by Human0.

human0-reviewer

Approved — The index filter now preserves ordinary guides while excluding Sphinx catalogs, and the full Python 3.14 docs output matches the previous commit (538 pages, 5,331 sections).

Changes behavior: no — This corrects the promised visible-link measurement and keeps generated indexes out of search without adding a new user-facing capability or contract.

This codebase is run by Human0.

human0-bot · 2026-10-13T03:18:52Z

Thanks for the careful fix. Before this can merge, please address two remaining issues:

api-index.html still matches the domain-index filename pattern when it contains <table class="indextable">, so an ordinary guide can disappear from search. Please restrict the pattern to actual Sphinx index names and add a regression test for a normal api-index.html page.
The raw-HTML regex still rescans repeated, unclosed <table tags; 40,000 repetitions (280 KB) took about 10.9 seconds here. Please make this check linear or otherwise bounded so a malformed page cannot stall indexing.

This codebase is run by Human0.

isTableOfContents counted each link's whole Markdown, URL included, as link text, so a prose paragraph with a few long links read as "mostly links" and its section was dropped from the index. Compare the link text a reader sees with the rest of the visible text instead. A list of links is still skipped.

Measuring visible link text lets Sphinx's genindex pages and py-modindex.html through the table-of-contents check: their list markup, plain-text entries, module synopses and page chrome keep them under the 50% link ratio. Stripping Sphinx's navigation bar and footer does not change that for genindex-E, -O, -Symbols, -all or py-modindex.html, so recognise the pages by the classes Sphinx's index templates emit (indextable, genindex-jumpbox) and return no sections for them.

The previous check dropped any HTML page whose source contained an indextable or genindex-jumpbox class, including a guide that uses the class on its own table or shows Sphinx's markup in a code sample. Require both a Sphinx index file name (genindex*.html or a domain index such as py-modindex.html) and an actual <table class="indextable"> or <div class="genindex-jumpbox"> tag. The Python 3.14 docs give the same sections as before (538 pages, 5,331 sections, no page differs).

api-index.html matched the domain-index pattern because the part before "index" could be empty, so a guide with an indextable was dropped. Domain indexes are <domain>-<index>.html and the only built-in one is py-modindex, so match that. The markup pattern's [^>]* ran to the end of the page from every unclosed <table, which took about 11 seconds for 40,000 of them. It now stops at the next < or >.

JayOfTheKeyboard · 2026-10-13T07:08:16Z

Both fixed in 151e483, and the branch is rebased onto current main.

The file name pattern now matches only genindex*.html and <domain>-modindex.html. Python's module index is the only domain index Sphinx builds in, and the old [a-z]* let api-index.html through. There's a new test for a normal api-index.html guide with an indextable, and it fails on the previous commit.
The markup check no longer crosses < or >, so an unclosed <table can't make it scan the rest of the page. 40,000 unclosed <table tags took 11.5 seconds before and now finish in under 170 ms, turndown included. The new test fails on the previous commit too.

All 360 tests in packages/context pass.

human0-bot · 2026-10-13T13:11:48Z

Thanks for the update and for adding regression coverage for both cases. The reviewer will assess the new head before we decide on merging.

This codebase is run by Human0.

human0-reviewer

Changes requested — Readers can now find prose sections that were previously dropped as link-heavy, while generated Sphinx index pages remain excluded.

Changes behavior: no — This corrects the parser's handling of sections intended to be searchable; the explicit Sphinx-index exclusion preserves pages that were already omitted.

Must fix

Use a random changeset filename (.changeset/toc-link-text-ratio.md:1)

1 note, no action needed

Suggestion: Keep Sphinx index handling easy to find Sphinx-file filtering also exists in packages/registry/src/zip.ts, with a narrower filename-only rule. These run at different stages, but a comment linking the two policies could help future changes stay consistent. (packages/context/src/html.ts:96)

This codebase is run by Human0.

human0-reviewer · 2026-10-13T13:20:14Z

@@ -0,0 +1,5 @@
+---


Must fix: Use a random changeset filename

The repository's CLAUDE.md asks for a random changeset filename, but this one uses the feature name. Please rename it to a random name; its contents can remain descriptive.

human0-reviewer · 2026-10-13T13:20:14Z

 * Markdown parser for section extraction and chunking.
 */
 export function parseHtml(source: string, filePath: string): ParsedDoc {
+  if (SPHINX_INDEX_FILE.test(filePath) && SPHINX_INDEX_MARKUP.test(source)) {


Question: Could an HTML example trigger the Sphinx index skip?

A genuine page named genindex*.html that includes raw <table class="indextable"> markup in a code sample would be skipped too. Is that exclusion intentional?

human0-reviewer Bot requested changes Oct 7, 2026

View reviewed changes

github-actions Bot mentioned this pull request Oct 9, 2026

Maintenance watch: items needing attention #176

Open

human0-reviewer Bot requested changes Oct 10, 2026

View reviewed changes

human0-reviewer Bot approved these changes Oct 11, 2026

View reviewed changes

human0-reviewer Bot requested changes Oct 11, 2026

View reviewed changes

human0-reviewer Bot approved these changes Oct 11, 2026

View reviewed changes

JayOfTheKeyboard added 4 commits October 13, 2026 18:07

JayOfTheKeyboard force-pushed the fix/toc-link-ratio-urls branch from a65a2f9 to 151e483 Compare October 13, 2026 07:08

human0-reviewer Bot requested changes Oct 13, 2026

View reviewed changes

Conversation

JayOfTheKeyboard commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Not changed

Test

Validation

Uh oh!

changeset-bot Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

human0-reviewer Bot left a comment

Choose a reason for hiding this comment

Uh oh!

human0-reviewer Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

JayOfTheKeyboard commented Oct 10, 2026

Uh oh!

human0-reviewer Bot left a comment

Choose a reason for hiding this comment

Uh oh!

human0-reviewer Bot Oct 10, 2026

Choose a reason for hiding this comment

Uh oh!

JayOfTheKeyboard Oct 11, 2026

Choose a reason for hiding this comment

Uh oh!

human0-reviewer Bot Oct 11, 2026

Choose a reason for hiding this comment

Uh oh!

human0-reviewer Bot Oct 11, 2026

Choose a reason for hiding this comment

Uh oh!

human0-reviewer Bot left a comment

Choose a reason for hiding this comment

Uh oh!

human0-reviewer Bot left a comment

Choose a reason for hiding this comment

Uh oh!

human0-reviewer Bot Oct 11, 2026

Choose a reason for hiding this comment

Uh oh!

human0-reviewer Bot Oct 11, 2026

Choose a reason for hiding this comment

Uh oh!

human0-bot Bot commented Oct 11, 2026

Uh oh!

human0-reviewer Bot left a comment

Choose a reason for hiding this comment

Uh oh!

human0-bot Bot commented Oct 13, 2026

Uh oh!

JayOfTheKeyboard commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

human0-bot Bot commented Oct 13, 2026

Uh oh!

human0-reviewer Bot left a comment

Choose a reason for hiding this comment

Uh oh!

human0-reviewer Bot Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

human0-reviewer Bot Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

JayOfTheKeyboard commented Oct 7, 2026 •

edited

Loading

changeset-bot Bot commented Oct 7, 2026 •

edited

Loading

JayOfTheKeyboard commented Oct 13, 2026 •

edited

Loading