Include Markdown and notebook files with scoped references

[PiotrCzapla]

We needed a way to include Markdown in Markdown. Code blocks were perfect for that, but the included documents sometimes had clashing section names. We could fix all the clashes, but I thought it would be better to do it programmatically in mdhtml, as clashes in reference IDs are hard to notice when an include is added during DOCX generation.

This PR contains both the include function and the ability to scope references in included sections.
It feels that scope could be more generic and apply, for say, headers instead of a wrapping div.
But the generic version is harder to specify correctly.

LLM authored PR desc, but reads well:

include(path) loads Markdown files or exported notebook notes. During Markdown conversion, each include appends its scope to local IDs and links.

from mdhtml.fill import include

source = f'{include("camera.md")}\n\n<br type="page">\n\n{include("mic.ipynb")}'

The caller places page breaks. Ordinary placeholders are blank by default. keep=["port"] preserves a field; keep={"port": "camera_port"} also renames it. Code literals stay literal. Notebook loading excludes code and stored outputs.

Markdown and MDHTML

The blocks can also be written directly:

::: {.include scope="__camera"}
# Camera
## Setup {#sec-setup}
See [@sec-setup].
:::

::: {.include scope="__mic"}
# Microphone
## Setup {#sec-setup}
See [@sec-setup].
:::

md2mdhtml(source) produces:

<div class="include">
<h1>Camera</h1>
<h2 id="sec-setup__camera">Setup</h2>
<p>See <a href="#sec-setup__camera" data-ref=""></a>.</p>
</div>
<div class="include">
<h1>Microphone</h1>
<h2 id="sec-setup__mic">Setup</h2>
<p>See <a href="#sec-setup__mic" data-ref=""></a>.</p>
</div>

scope is the complete suffix. scope="__mic" produces sec-setup__mic by concatenation. The converter inserts no punctuation. The file helper defaults to __ followed by the filename stem.

The sec-, fig- and tbl- prefixes remain intact. Reference lookup uses its existing rules without stripping namespaces or adding reference syntax.

Inside the include, use [@sec-setup]. Outside, use [@sec-setup__mic]. An unqualified outside reference reports a missing target. Links to outside targets stay unchanged. Nested scopes append in order: sec-setup__lens__camera.

HTML export adds numbers and reference text. Each h1 resets the normal heading counter. Nested divs use the same ::: delimiter.

Implementation and checks

Include handling runs with the div renderer and processes include contents without reparsing the whole document. Invalid scope attributes produce parser warnings. Rust render and wiki2mdhtml return strings. The helper reuses the notebook conversion functions and Md return type.

Independent review approved the changes. All 963 Python tests and 42 Rust tests passed; the WASM target compiled. The combined converter suite passed 40 tests and 16 downstream integration tests passed. The default Python suite excludes one slow test.

[jph00]

I deleted the private links from this PR - be careful please.

I don't understand the jargon being used here. Please try to explain in simple concise terms.

[PiotrCzapla]

I deleted the private links from this PR - be careful please.

I don't understand the jargon being used here. Please try to explain in simple concise terms.

LLm didn't have instructions to watch out when publishing to a public repo; you got to the PR before I did. I need to find a better way to review the changes before they go to the public repo. Sorry about that. Thank you for catching that.