diff --git a/CHANGELOG.md b/CHANGELOG.md index 90c34e4..67e7dc9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,9 @@ All notable changes to this project will be documented in this file. Swift app built in release configuration and uploaded as a workflow artifact - Added golangci-lint config, dependabot, and `make ci` (vet + lint + race tests + vulncheck) - Version normalized to 3-segment semver (`0.1.0`) across VERSION, Info.plist and CHANGELOG +- Engine tests: Markdown golden fixture corpus grown from 5 to 13 (Google Docs/Word clipboard, + images, strikethrough, definition lists, loose lists, colspan tables, anchors) and + `TestMarkdown_Fixtures` gained a `-update` flag; known converter bugs are pinned and tracked in TODOS.md ## [0.1.0] - 2026-04-08 diff --git a/TODOS.md b/TODOS.md index 1c70777..b3d19b5 100644 --- a/TODOS.md +++ b/TODOS.md @@ -26,6 +26,24 @@ **Priority:** P2 **Depends on:** None +### Fix converter bugs pinned by the golden fixtures + +**What:** Fix the HTML-to-Markdown behaviours that `engine/testdata/fixtures/` currently goldens as-is, then regenerate the affected goldens with `go test ./transforms/... -run TestMarkdown_Fixtures -update` and review the diff by hand. + +**Why:** Six fixtures deliberately pin the converter's *current* output so regressions are caught, but that output is wrong. A future fix will show up as a fixture "failure"; this list says which failures are the intended ones. + +**Context:** Pinned bugs, by fixture: +- `google-docs-clipboard`: the `` wrapper bolds the whole document, and list items under it lose their `-` markers. +- `word-clipboard`: `mso-list` fake bullets come through as literal `·` plus ` ` runs instead of `- ` items; `

 

` spacer paragraphs leak as lines holding a lone non-breaking space. +- `rule-strikethrough-sub-sup`: ``/`` are unwrapped to plain text (so "$49/month $39/month" reads as two prices); the library's `strikethrough` plugin is not wired in. ``/`` are flattened (H2O, reference1). +- `definition-list`: `
/
/
` become undifferentiated paragraphs. +- `table-colspan-alignment`: `colspan` content lands in column 1 with empty cells after it, and `text-align` is ignored in the alignment row. +- `tracking-links-and-anchors`: `` and `href=""` render as `[]()` / `[text]()`. + +**Effort:** M +**Priority:** P2 +**Depends on:** None + ## Design Review ### Design first-run onboarding flow @@ -56,7 +74,7 @@ ### Create Markdown fixture corpus from real-world sources -**Completed:** v0.1.0 (2026-04-08). 5 fixtures created in engine/testdata/fixtures/ covering Claude Code, ChatGPT, browser articles, nested lists/tables, and rich text mixed content. +**Completed:** v0.1.0 (2026-04-08), expanded 2026-09-26. 13 fixtures in engine/testdata/fixtures/ (hand-synthesized, no scraped page content): the original 5 (Claude Code, ChatGPT, browser articles, nested lists/tables, rich text mixed) plus Google Docs- and Word-style clipboard spans, images/captions, hr/strikethrough/sub/sup, definition lists, loose multi-paragraph list items, table colspan/alignment, and tracking-link/anchor edge cases. `TestMarkdown_Fixtures` in engine/transforms/markdown_test.go gained a `-update` flag to regenerate the golden .md files from current converter output. Several fixtures intentionally golden the engine's *current* (buggy) output rather than the ideal one — see "Fix converter bugs pinned by the golden fixtures" above. ### Write the choosepaste implementation spec diff --git a/engine/testdata/fixtures/browser-article.md b/engine/testdata/fixtures/browser-article.md index baa0573..1a83de0 100644 --- a/engine/testdata/fixtures/browser-article.md +++ b/engine/testdata/fixtures/browser-article.md @@ -38,4 +38,4 @@ func fetchAllUsers(ids: [Int]) async throws -> [User] { } ``` -See the [official Swift docs](https://docs.swift.org/swift-book/LanguageGuide/Concurrency.html) for more details. \ No newline at end of file +See the [official Swift docs](https://docs.swift.org/swift-book/LanguageGuide/Concurrency.html) for more details. diff --git a/engine/testdata/fixtures/chatgpt-code-explanation.md b/engine/testdata/fixtures/chatgpt-code-explanation.md index 2da44bb..b7ce7b7 100644 --- a/engine/testdata/fixtures/chatgpt-code-explanation.md +++ b/engine/testdata/fixtures/chatgpt-code-explanation.md @@ -28,4 +28,4 @@ func binarySearch(arr []int, target int) int { 4. If less, search the right half 5. If greater, search the left half -Time complexity: **O(log n)**. Space complexity: **O(1)**. \ No newline at end of file +Time complexity: **O(log n)**. Space complexity: **O(1)**. diff --git a/engine/testdata/fixtures/claude-code-response.md b/engine/testdata/fixtures/claude-code-response.md index a9f0b40..75fb60d 100644 --- a/engine/testdata/fixtures/claude-code-response.md +++ b/engine/testdata/fixtures/claude-code-response.md @@ -19,4 +19,4 @@ Key fields: - **data** — the response payload (nullable on error) - **tags** — array of strings, can be empty -> **Note:** The `id` field is always present, even on error responses. \ No newline at end of file +> **Note:** The `id` field is always present, even on error responses. diff --git a/engine/testdata/fixtures/definition-list.html b/engine/testdata/fixtures/definition-list.html new file mode 100644 index 0000000..6a158dd --- /dev/null +++ b/engine/testdata/fixtures/definition-list.html @@ -0,0 +1,7 @@ +

Glossary

+
+
Idempotent
+
An operation that produces the same result no matter how many times it is applied.
+
Backpressure
+
A mechanism for a consumer to signal a producer to slow down when overwhelmed.
+
diff --git a/engine/testdata/fixtures/definition-list.md b/engine/testdata/fixtures/definition-list.md new file mode 100644 index 0000000..6066db7 --- /dev/null +++ b/engine/testdata/fixtures/definition-list.md @@ -0,0 +1,9 @@ +## Glossary + +Idempotent + +An operation that produces the same result no matter how many times it is applied. + +Backpressure + +A mechanism for a consumer to signal a producer to slow down when overwhelmed. diff --git a/engine/testdata/fixtures/google-docs-clipboard.html b/engine/testdata/fixtures/google-docs-clipboard.html new file mode 100644 index 0000000..0e4f2b6 --- /dev/null +++ b/engine/testdata/fixtures/google-docs-clipboard.html @@ -0,0 +1 @@ +

Quarterly Planning Notes


This document tracks the priorities for the upcoming planning cycle. Owners should update their sections weekly.


  • Finalize the roadmap for Project Northwind

  • Review staffing for the design team

  • Confirm budget with the finance team


Open questions are tracked separately.

 diff --git a/engine/testdata/fixtures/google-docs-clipboard.md b/engine/testdata/fixtures/google-docs-clipboard.md new file mode 100644 index 0000000..094bdff --- /dev/null +++ b/engine/testdata/fixtures/google-docs-clipboard.md @@ -0,0 +1,11 @@ +**Quarterly Planning Notes** + +**This document tracks the priorities for the upcoming planning cycle. Owners should update their sections weekly.** + +**Finalize the roadmap for Project Northwind** + +**Review staffing for the design team** + +**Confirm budget with [the finance team](https://example.com/finance)** + +**Open questions are tracked separately.** diff --git a/engine/testdata/fixtures/images-and-captions.html b/engine/testdata/fixtures/images-and-captions.html new file mode 100644 index 0000000..83498b6 --- /dev/null +++ b/engine/testdata/fixtures/images-and-captions.html @@ -0,0 +1,7 @@ +

Architecture Overview

+

The diagram below shows the request flow through the system.

+
+Diagram showing a client sending a request through a load balancer to two backend services +
Figure 1: Request flow from client to backend
+
+

Each service reports health to a shared heartbeat icon monitor, which is polled every 30 seconds.

diff --git a/engine/testdata/fixtures/images-and-captions.md b/engine/testdata/fixtures/images-and-captions.md new file mode 100644 index 0000000..4f15309 --- /dev/null +++ b/engine/testdata/fixtures/images-and-captions.md @@ -0,0 +1,9 @@ +## Architecture Overview + +The diagram below shows the request flow through the system. + +![Diagram showing a client sending a request through a load balancer to two backend services](https://assets.example.com/diagrams/request-flow.png) + +Figure 1: Request flow from client to backend + +Each service reports health to a shared ![heartbeat icon](https://assets.example.com/icons/heartbeat.png) monitor, which is polled every 30 seconds. diff --git a/engine/testdata/fixtures/loose-list-multi-paragraph.html b/engine/testdata/fixtures/loose-list-multi-paragraph.html new file mode 100644 index 0000000..f6f5a7d --- /dev/null +++ b/engine/testdata/fixtures/loose-list-multi-paragraph.html @@ -0,0 +1,14 @@ +

Deployment Checklist

+
    +
  1. +

    Run the test suite locally.

    +
    make test
    +
    2. +
  2. +

    Tag the release and push it to the registry.

    +

    Double-check the tag matches the version in VERSION.

    +
    3. +
  3. +

    Notify the on-call engineer before rolling out.

    +
    4. +
diff --git a/engine/testdata/fixtures/loose-list-multi-paragraph.md b/engine/testdata/fixtures/loose-list-multi-paragraph.md new file mode 100644 index 0000000..8b3985b --- /dev/null +++ b/engine/testdata/fixtures/loose-list-multi-paragraph.md @@ -0,0 +1,11 @@ +## Deployment Checklist + +1. Run the test suite locally. + + ```bash + make test + ``` +2. Tag the release and push it to the registry. + + Double-check the tag matches the version in `VERSION`. +3. Notify the on-call engineer before rolling out. diff --git a/engine/testdata/fixtures/nested-lists-and-tables.md b/engine/testdata/fixtures/nested-lists-and-tables.md index 87f5fe8..305eddb 100644 --- a/engine/testdata/fixtures/nested-lists-and-tables.md +++ b/engine/testdata/fixtures/nested-lists-and-tables.md @@ -1,16 +1,16 @@ ## Project Structure - **Frontend** - + - React 18 with TypeScript - Tailwind CSS for styling - Components: - + - `Header.tsx` - `Sidebar.tsx` - `Dashboard.tsx` - **Backend** - + - Go 1.22 with Chi router - PostgreSQL database diff --git a/engine/testdata/fixtures/rich-text-mixed.md b/engine/testdata/fixtures/rich-text-mixed.md index 9d451f8..1d52e40 100644 --- a/engine/testdata/fixtures/rich-text-mixed.md +++ b/engine/testdata/fixtures/rich-text-mixed.md @@ -28,4 +28,4 @@ output: destination: stdout ``` -For questions, see the [migration docs](https://docs.example.com/migration) or open an [issue](https://github.com/example/repo/issues). \ No newline at end of file +For questions, see the [migration docs](https://docs.example.com/migration) or open an [issue](https://github.com/example/repo/issues). diff --git a/engine/testdata/fixtures/rule-strikethrough-sub-sup.html b/engine/testdata/fixtures/rule-strikethrough-sub-sup.html new file mode 100644 index 0000000..443b2f7 --- /dev/null +++ b/engine/testdata/fixtures/rule-strikethrough-sub-sup.html @@ -0,0 +1,5 @@ +

Pricing Notes

+

The base price is $49/month $39/month for early adopters.

+
+

Chemical formula example: H2O. Footnote reference1 below.

+

Deprecated tier removed in v3.

diff --git a/engine/testdata/fixtures/rule-strikethrough-sub-sup.md b/engine/testdata/fixtures/rule-strikethrough-sub-sup.md new file mode 100644 index 0000000..aa781b2 --- /dev/null +++ b/engine/testdata/fixtures/rule-strikethrough-sub-sup.md @@ -0,0 +1,9 @@ +## Pricing Notes + +The base price is $49/month **$39/month** for early adopters. + +* * * + +Chemical formula example: H2O. Footnote reference1 below. + +Deprecated tier removed in v3. diff --git a/engine/testdata/fixtures/table-colspan-alignment.html b/engine/testdata/fixtures/table-colspan-alignment.html new file mode 100644 index 0000000..f4ed4d7 --- /dev/null +++ b/engine/testdata/fixtures/table-colspan-alignment.html @@ -0,0 +1,11 @@ +

Regional Sales Summary

+ + + + + + + + + +
RegionQ1Q2
North120150
South9095
Totals shown are in thousands of units
diff --git a/engine/testdata/fixtures/table-colspan-alignment.md b/engine/testdata/fixtures/table-colspan-alignment.md new file mode 100644 index 0000000..3a873e6 --- /dev/null +++ b/engine/testdata/fixtures/table-colspan-alignment.md @@ -0,0 +1,7 @@ +## Regional Sales Summary + +| Region | Q1 | Q2 | +|--------------------------------------------|-----|-----| +| North | 120 | 150 | +| South | 90 | 95 | +| **Totals shown are in thousands of units** | | | diff --git a/engine/testdata/fixtures/tracking-links-and-anchors.html b/engine/testdata/fixtures/tracking-links-and-anchors.html new file mode 100644 index 0000000..b5e5492 --- /dev/null +++ b/engine/testdata/fixtures/tracking-links-and-anchors.html @@ -0,0 +1,4 @@ +

Further Reading

+

See the announcement post for background.

+

Jump back to the top of the page.

+

Reach the team at team@example.com or via an empty link left in by mistake.

diff --git a/engine/testdata/fixtures/tracking-links-and-anchors.md b/engine/testdata/fixtures/tracking-links-and-anchors.md new file mode 100644 index 0000000..594abaf --- /dev/null +++ b/engine/testdata/fixtures/tracking-links-and-anchors.md @@ -0,0 +1,7 @@ +## Further Reading + +See the [announcement post](https://example.com/blog/post?utm_source=newsletter&utm_medium=email&utm_campaign=fall2026) for background. + +[]()Jump back to the [top](#top) of the page. + +Reach the team at [team@example.com](mailto:team@example.com) or via [an empty link]() left in by mistake. diff --git a/engine/testdata/fixtures/word-clipboard.html b/engine/testdata/fixtures/word-clipboard.html new file mode 100644 index 0000000..a99a68a --- /dev/null +++ b/engine/testdata/fixtures/word-clipboard.html @@ -0,0 +1,14 @@ + + + +

Weekly Status Update

+

 

+

Summary of progress for the Atlas migration workstream this week.

+

·     Data migration script finished a full dry run

+

·     Rollback plan documented and reviewed

+

·     Blocked on staging credentials from IT

+

 

+

Next check-in is scheduled for Monday.

+ + + diff --git a/engine/testdata/fixtures/word-clipboard.md b/engine/testdata/fixtures/word-clipboard.md new file mode 100644 index 0000000..c280c24 --- /dev/null +++ b/engine/testdata/fixtures/word-clipboard.md @@ -0,0 +1,15 @@ +**Weekly Status Update** + +  + +Summary of progress for the **Atlas migration** workstream this week. + +·     Data migration script finished a full dry run + +·     Rollback plan documented and reviewed + +·     Blocked on *staging* credentials from IT + +  + +Next check-in is scheduled for Monday. diff --git a/engine/transforms/markdown_test.go b/engine/transforms/markdown_test.go index 1edcb14..c8a58bb 100644 --- a/engine/transforms/markdown_test.go +++ b/engine/transforms/markdown_test.go @@ -1,6 +1,7 @@ package transforms import ( + "flag" "os" "path/filepath" "regexp" @@ -8,6 +9,12 @@ import ( "testing" ) +// update regenerates the golden .md fixture files from the current output of +// the HTML-to-Markdown converter. Run with: +// +// go test ./transforms/... -run TestMarkdown_Fixtures -update +var update = flag.Bool("update", false, "update golden fixture files (testdata/fixtures/*.md) with current converter output") + // normalizeBlankLines strips trailing whitespace from each line. // This avoids false negatives when linters strip trailing spaces // from fixture files but the converter preserves indentation on blank lines. @@ -120,18 +127,39 @@ func TestMarkdown_Fixtures(t *testing.T) { t.Fatalf("failed to read %s: %v", htmlFile, err) } - expectedBytes, err := os.ReadFile(filepath.Clean(mdFile)) + got, err := Markdown(string(htmlBytes), "html") if err != nil { - t.Fatalf("failed to read %s: %v", mdFile, err) + t.Fatalf("transform error: %v", err) } + gotNorm := normalizeTrailingWS(got) - got, err := Markdown(string(htmlBytes), "html") + if *update { + // Use OpenFile+Write rather than os.WriteFile: gosec's G703 + // taint analysis flags a variable path passed straight to + // os.WriteFile even after filepath.Clean(); the equivalent + // OpenFile call with an explicit O_CREATE|O_TRUNC is the + // established fix for this rule elsewhere in fairbearlab Go + // repos (see rolodex's copyFile helper). + f, err := os.OpenFile(filepath.Clean(mdFile), os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600) + if err != nil { + t.Fatalf("failed to open %s for update: %v", mdFile, err) + } + if _, err := f.WriteString(gotNorm + "\n"); err != nil { + _ = f.Close() + t.Fatalf("failed to write %s: %v", mdFile, err) + } + if err := f.Close(); err != nil { + t.Fatalf("failed to close %s: %v", mdFile, err) + } + return + } + + expectedBytes, err := os.ReadFile(filepath.Clean(mdFile)) if err != nil { - t.Fatalf("transform error: %v", err) + t.Fatalf("failed to read %s: %v (run with -update to create it)", mdFile, err) } expected := normalizeTrailingWS(string(expectedBytes)) - gotNorm := normalizeTrailingWS(got) if gotNorm != expected { t.Errorf("fixture %s mismatch:\n--- got ---\n%s\n--- expected ---\n%s", name, gotNorm, expected) }