Skip to content

Expose the YAML frontmatter body in the grammar so a --- inside a value cannot truncate the metadata - #675

Merged
cscheid merged 1 commit into
mainfrom
bugfix/bd-mjo6ao32-frontmatter-delimiter-split
Sep 11, 2026
Merged

Expose the YAML frontmatter body in the grammar so a --- inside a value cannot truncate the metadata#675
cscheid merged 1 commit into
mainfrom
bugfix/bd-mjo6ao32-frontmatter-delimiter-split

Conversation

@cscheid

@cscheid cscheid commented Sep 10, 2026

Copy link
Copy Markdown
Member

Closes #671.

Problem

The reader located the frontmatter body by split("---")-ing the opaque metadata token's text and keeping the second piece, so the first --- inside a value ended the YAML there: a plain scalar silently dropped every later key, a quoted scalar failed with Q-0-99. Since #290 the qmd writer canonicalizes every em dash to ---, which made both shapes reachable from any document with an em dash in a frontmatter string (in the wild: quarto-web's 2026-04-14-chrome-headless-shell/index.qmd).

The tree-sitter scanner was never at fault — it closes the block only on a column-0 --- line — but it emitted the whole block as one external leaf token, which is why the Rust side had to rediscover the body by hand.

Fix

The grammar now parses the block as four external tokens and exposes the body as a yaml child under the body field:

(metadata body: (yaml))

pampa takes the YAML text and range straight from that node. extract_between_delimiters and the block.text.find(content) offset hazard are deleted; nothing in Rust knows about --- any more. The yaml node is also what a YAML injection query would target in external tooling.

Scanner details: _minus_metadata_start is the opening --- (it shares the thematic break's mark_end position so the confirming look-ahead can run speculatively), then _minus_metadata_open_newline, _minus_metadata_body, _minus_metadata_end, then the ordinary choice(_newline, _eof) tail — so a closing --- at EOF without a newline now also closes the block. A new STATE_IN_MINUS_METADATA bit gates the interior tokens against tree-sitter's mark-everything-valid error recovery.

Consequences worth reviewing

  • Fixtures/snapshots (please eyeball): the RawBlock quarto_minus_metadata carrier now holds the body text and the body range, so the body is no longer trimmed, the metadata span ends after the final line break (4..30 instead of 4..29), and the source-info chain is one layer shorter. Resolved key/value offsets are unchanged. That renumbers every s index in the 20 ts-packages/annotated-qmd/examples/*.json fixtures (regenerated with the documented loop; I verified every non-astContext difference is an s renumbering) and 4 insta snapshots under crates/pampa/snapshots/json/.
  • Block::BlockMetadata for a lexical block now spans the YAML body rather than the delimited block. Carrying both ranges would mean building BlockMetadata at tree-conversion time instead of via the RawBlock carrier — flagged as a follow-up in the plan, not done here.
  • parser.c regeneration renumbers LR states, so the state-keyed resources/error-corpus/_autogen-table.json was rebuilt (crates/pampa/scripts/build_error_table.ts); only the table changed.
  • One corpus expectation (div.txt: 2) gains the block_continuation child every block in a div already carries.

Tests

  • crates/pampa/tests/integration/test_frontmatter_delimiters.rs: the three shapes from the issue, an indented --- inside a block scalar, trailing whitespace on the closing line, CRLF, metadata nested in a div, exact key offsets after a ----containing value, qmd-writer round trips for single- and multi-line em-dash values, and the tree shape/range of the yaml node (including closing --- at EOF).
  • 5 new GH An em dash in a frontmatter value is written as ---, which truncates the YAML on re-read #671 cases in test/corpus/metadata.txt; 9 existing (metadata) expectations gained the child.
  • The three file-based tests in test_meta.rs now go through the real reader instead of handing whole documents to rawblock_to_config_value.
  • Full cargo xtask verify green locally.

End-to-end (output inspected):

$ printf -- '---\ndescription: "Hello — world"\nauthor: Z\n---\n\nx\n' | pampa -t qmd | pampa -t json | jq -c .meta
{"author": Z, "description": Hello — world}          # both keys, em dash restored
$ printf -- '---\ndescription: "a --- b"\nauthor: Z\n---\n\nx\n' | pampa -t json | jq -c .meta
{"author": Z, "description": a — b}                  # no Q-0-99
$ q2 render dash.qmd    # title/description with em dashes + author
<title>Dashes — everywhere</title> … <meta name="description" content="Hello — world"> … <meta name="author" content="Z">

Plan: claude-notes/plans/2026-09-10-frontmatter-delimiter-split-gh671.md (bd-mjo6ao32). Found along the way and filed separately: bd-wl58atds (pre-existing provenance warning on block scalars with multi-byte characters).

🤖 Generated with Claude Code

https://claude.ai/code/session_0132TvF41q7CA19fE29UX77X

…alue cannot truncate the metadata (bd-mjo6ao32, GH #671)

The reader located the frontmatter body by `split("---")`-ing the opaque
`metadata` token's text and keeping the second piece, so the first `---`
inside a value ended the YAML there: a plain scalar silently dropped every
later key, a quoted scalar failed with Q-0-99. Since PR #290 the qmd writer
canonicalizes every em dash to `---`, which made both shapes reachable from
any document with an em dash in a frontmatter string.

The tree-sitter scanner was never at fault (it closes the block only on a
column-0 `---` line), but it emitted the whole block as one external leaf
token, which is why the Rust side had to rediscover the body. The grammar
now parses the block as four external tokens and exposes the body as a
`yaml` child under the `body` field:

    (metadata body: (yaml))

pampa takes the YAML text and range straight from that node;
`extract_between_delimiters` and the `block.text.find(content)` offset
hazard are deleted. Nothing in Rust knows about `---` any more, and the
`yaml` node is also what a YAML injection query would target.

Scanner: `_minus_metadata_start` (the opening `---`, sharing the thematic
break's `mark_end` position so the confirming look-ahead can run
speculatively), `_minus_metadata_open_newline`, `_minus_metadata_body`,
`_minus_metadata_end`, then the ordinary `choice(_newline, _eof)` tail, so
a closing `---` at EOF without a newline now also closes the block. A new
`STATE_IN_MINUS_METADATA` bit gates the interior tokens against
tree-sitter's mark-everything-valid error recovery.

Consequences of the carrier change (`RawBlock quarto_minus_metadata` now
holds the body text and the body range): the body is no longer trimmed, the
metadata span ends after the final line break, and the source-info chain is
one layer shorter. Resolved key/value offsets are unchanged. That renumbers
every `s` index in the 20 `ts-packages/annotated-qmd/examples/*.json`
fixtures (regenerated with the documented loop; verified that every
non-`astContext` difference is an `s` renumbering) and updates 4 insta
snapshots under `crates/pampa/snapshots/json/` (002, and three others with
frontmatter) the same way. `Block::BlockMetadata` for a lexical block now
spans the YAML body rather than the delimited block; see the plan for the
follow-up that would carry both ranges.

Regenerating `parser.c` renumbers LR states, so the state-keyed
`resources/error-corpus/_autogen-table.json` was rebuilt with
`crates/pampa/scripts/build_error_table.ts`; only the table changed.

Tests: 12 reader/round-trip/tree-shape regressions in
`test_frontmatter_delimiters.rs`; 5 GH #671 corpus cases plus the
`body: (yaml)` child on 9 existing expectations (one, `div.txt: 2`, also
gains the `block_continuation` every block in a div carries). The three
file-based tests in `test_meta.rs` now go through the real reader instead
of handing whole documents to `rawblock_to_config_value`.

Plan: claude-notes/plans/2026-09-10-frontmatter-delimiter-split-gh671.md

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0132TvF41q7CA19fE29UX77X
@posit-snyk-bot

posit-snyk-bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Snyk checks have passed. No issues have been found so far.

Status Scan Engine Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues
Licenses 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@cscheid
cscheid merged commit 7ef5961 into main Sep 11, 2026
10 checks passed
@cscheid
cscheid deleted the bugfix/bd-mjo6ao32-frontmatter-delimiter-split branch September 11, 2026 13:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

An em dash in a frontmatter value is written as ---, which truncates the YAML on re-read

2 participants