fix: handle CommonMark fenced code boundaries

[adityat210]

Related Issues

• None. This was found during a repository audit.

Proposed Changes

MarkdownHeaderSplitter previously matched an entire fenced code block with a regex whose closing delimiter had to be identical to the opener. Valid Markdown with a longer closing fence, up to three spaces of fence indentation, or no closing fence therefore exposed hash-prefixed code lines to the header matcher.

This PR:

• replaces the whole-block backreference regex with a linear line-state scanner;
• accepts closing fences that use the same marker and are at least as long as the opener;
• accepts the CommonMark allowance of up to three leading spaces;
• treats an unclosed fence as continuing to the end of the document;
• preserves the existing same-marker and minimum-length safeguards;
• adds regression coverage for backtick and tilde fences, each supported indentation, EOF handling, and shorter non-closing delimiters.

The behavior aligns with the CommonMark fenced code block specification.

Measured on the three minimal reproductions:

Case	Before headers	After headers
Longer closing fence	Real, fake, Next	Real, Next
Three-space-indented fence	Real, fake, Next	Real, Next
Unclosed fence	Real, fake	Real

False header metadata across these cases drops from 3 to 0. There is no public API change.

How did you test it?

• Targeted component suite in a Hatch-managed Python 3.10 environment: 46 passed in 0.29s.
• hatch run fmt-check haystack/components/preprocessors/markdown_header_splitter.py test/components/preprocessors/test_markdown_header_splitter.py
• hatch run pre-commit run --files haystack/components/preprocessors/markdown_header_splitter.py test/components/preprocessors/test_markdown_header_splitter.py releasenotes/notes/markdown-fence-boundaries-20f1dacc1efec6c8.yaml
• hatch run reno lint
• Manual before/after reproduction for longer, indented, and unclosed fences.

Notes for the reviewer

• The existing fence exclusion came from feat: MarkdownHeaderSplitter Add option to control which header depths create split boundaries and ignore # lines within code blocks #11011. The adjacent fix in fix: MarkdownHeaderSplitter drops content before an embedded header or code-fence comment #11920 prevents content loss during secondary splitting; it does not change fence boundary detection.
• Live searches for MarkdownHeaderSplitter, _code_block_spans, _code_block_pattern, closing/unclosed/longer fence terminology, and fenced-code header variants found no open overlapping issue or PR.
• This PR was fully generated with an AI assistant. The generated changes were inspected and the relevant tests and checks listed above were run locally.

Checklist

• I have read the contributors guidelines and code of conduct.
• There is no related issue to update.
• I have added focused unit tests and documented the non-obvious behavior.
• The PR title uses a conventional commit type.
• I generated and completed a release note.
• I ran the relevant pre-commit hooks and fixed all findings.

[vercel]

@adityat210 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

All committers have signed the CLA.

[HaystackBot]

Hi @adityat210, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

[HaystackBot]

Thanks for signing the CLA, @adityat210! 🎉 This PR is now ready for review again and the reviewer has been re-assigned.

[anakin87]

Thank you!

I found 2 minor opportunities for improvement.

[anakin87]

Let's also specify that now an unclosed fence is treated as extending to the end of the document, so hash lines after it are no longer split into chunks.

[anakin87]

Suggested change

	self._code_block_close_pattern = re.compile(r" {0,3}(?P<fence>`{3,}|~{3,})[ \t]*")
	self._code_block_close_pattern = re.compile(r" {0,3}(?P<fence>`{3,}|~{3,})\s*")

I'd keep using \s* as the previous regex: this should work well with documents containing page breaks.

Please also add a test similar to this:

def test_page_break_after_closing_fence(self):
    text = "# Real Header\n```python\n# not a header\n```\f\n## Real Sub\nContent.\n"
    ...