Repository navigation
Expand Markdown fixture corpus and add golden-test -update flag - #13
Open
fairbearlab wants to merge 4 commits into
Open
fairbearlab wants to merge 4 commits into
fairbearlab wants to merge 4 commits into
Conversation
Grow engine/testdata/fixtures/ from 5 to 13 hand-synthesized HTML/MD golden pairs, covering Google Docs- and Word-style clipboard spans, images/captions, hr/strikethrough/sub-sup, definition lists, loose multi-paragraph list items, table colspan/alignment, and tracking link/anchor edge cases. TestMarkdown_Fixtures gains a -update flag to regenerate .md goldens from the converter's current output. Several new fixtures golden the engine's current (buggy) output on purpose rather than the ideal one; suspected bugs are listed in the PR description instead of being fixed here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
golangci-lint v2.14's gosec now taint-tracks the -update write path (mdFile derived from filepath.Glob) the same way it already does for the read path. Wrap the write in filepath.Clean(), matching the repo's documented gosec G304/G703 remedy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
filepath.Clean() alone didn't satisfy golangci-lint v2.14's gosec G703 taint check for os.WriteFile with a variable destination path. Switch to the OpenFile(O_CREATE|O_TRUNC)+Write pattern already established for this rule in rolodex's copyFile helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ntry Six new fixtures deliberately golden the converter's current (wrong) output. Record each pinned bug in TODOS.md so a future fix that changes those goldens is recognisably intended, rather than pointing at PR history. Add an Unreleased CHANGELOG line for the corpus + -update flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Grows
engine/testdata/fixtures/from 5 to 13 hand-synthesized (no scraped page content, no personal data) HTML → Markdown golden pairs, and adds a-updateflag toTestMarkdown_Fixturesso the goldens can be regenerated from the converter's current output.New fixtures:
google-docs-clipboard— Google Docs paste-style output (<b style="font-weight:normal" id="docs-internal-guid-...">wrapper,<span style="font-weight:700">bold, list items wrapped in<p>)word-clipboard— MS Word paste-style output (mso-*classes/styles,<o:p>, fake bullet lists built from<span style='mso-list:Ignore'>·</span>+ conditional comments, spacer paragraphs)images-and-captions—<figure>/<img>/<figcaption>, inline image in a paragraphrule-strikethrough-sub-sup—<hr>,<del>,<s>,<sub>,<sup>definition-list—<dl>/<dt>/<dd>loose-list-multi-paragraph— ordered list items containing multiple<p>blocks and a nested code blocktable-colspan-alignment— table withtext-alignhints and acolspan="3"summary rowtracking-links-and-anchors— UTM-tagged links, an anchor-only<a name>fragment target, a same-page#toplink,mailto:, and an emptyhref=""The 5 pre-existing fixtures are unchanged except for whitespace-only golden-file cleanup (trailing newline added; blank continuation lines within nested lists stripped of trailing spaces) picked up by the new
-updateflag — no behavioral difference, andTestMarkdown_Fixturesstill passes for all of them.Suspected engine bugs (not fixed here, goldened as-is per the task)
A few of the new fixtures intentionally golden the engine's current (arguably wrong) output rather than the ideal one, so the corpus can catch regressions and future fixes can update these goldens deliberately:
google-docs-clipboard): Google Docs wraps its whole clipboard payload in<b style="font-weight:normal">. The converter doesn't consider thestyleoverride and treats the outer<b>as literal bold, so the entire document comes out wrapped in**...**.google-docs-clipboard): the same Google Docs wrapper nests<ul><li><p>...</p></li></ul>inside a<b>(a<b>containing block-level content isn't valid HTML content model, but browsers/Docs still emit it). Under that nesting the converter drops the-bullet markers entirely, rendering list items as bare paragraphs. Note this is specific to the malformed nesting — the same<li><p>...</p></li>pattern converts correctly inloose-list-multi-paragraph, which doesn't have the outer<b>.word-clipboard): Word's clipboard HTML often has no real<ul>/<li>— just<p class=MsoListParagraph...>with anmso-liststyle and a literal·bullet glyph inside a<span style='mso-list:Ignore'>. The converter passes this through as literal·+ runs of rather than emitting-Markdown list syntax. -only spacer paragraphs leak into the output (word-clipboard): Word often inserts empty<p> </p>paragraphs for vertical spacing; these come through as stray lines containing a single non-breaking space rather than being collapsed away.rule-strikethrough-sub-sup):<del>and<s>content is unwrapped to plain text instead of being rendered as GFM~~strikethrough~~. The converter is built frombase+commonmark+tableplugins; wiring in the library'sstrikethroughplugin would likely fix this.definition-list):<dl>/<dt>/<dd>collapse to a sequence of plain paragraphs indistinguishable from unrelated prose — the term/definition relationship is lost entirely (no bolding of<dt>, no indentation of<dd>).colspanand alignment hints are ignored (table-colspan-alignment): acolspan="3"summary row has its content dumped into column 1 with columns 2–3 left as empty cells (structurally misleading — looks like missing data rather than a spanning cell), and per-columntext-alignCSS is ignored, so the generated alignment row is always---|---|---regardless of the source's alignment.tracking-links-and-anchors):<a name="top"></a>(nohref, no text — just a same-page jump target) comes out as a literal[]()in the Markdown rather than being dropped.None of these are fixed in this PR per the task instructions (record current output as golden, list suspected bugs for a human to triage/fix separately).
Test plan
cd engine && go test ./...— all pass, including the 13TestMarkdown_Fixturessubtestscd engine && go test ./transforms/... -run TestMarkdown_Fixtures -updateregenerates goldens byte-identical to the committed ones (re-run without-updatestill passes)go vet ./...,gofmt -l .(clean),go test -race ./...,go mod tidy(no diff),golangci-lint run(0 issues),govulncheck ./...(no vulnerabilities) — full local equivalent ofmake cifor the engine.github/workflows/*files touched🤖 Generated with Claude Code