Repository navigation
fix(docx): render w:sym glyphs from the symbol-font code pages - #211
Open
rahul05ranjan wants to merge 3 commits into
Open
rahul05ranjan wants to merge 3 commits into
rahul05ranjan wants to merge 3 commits into
Conversation
The run walker dropped w:sym whole, so legacy bullets and Symbol-font operators vanished from the text (firecrawl#193). Add the code-page tables of Wingdings, Wingdings 2, Wingdings 3, Webdings and Symbol and an arm that emits the mapped Unicode text. Co-authored-by: rahul05ranjan <11793455+rahul05ranjan@users.noreply.github.com>
Co-authored-by: rahul05ranjan <11793455+rahul05ranjan@users.noreply.github.com>
Source: docx4j's SymbolMapper (Apache-2.0), cross-checked entry by entry against pandoc's independent table. Co-authored-by: rahul05ranjan <11793455+rahul05ranjan@users.noreply.github.com>
Author
|
Hi team — this fix for rendering w:sym glyphs from symbol-font code pages is ready for review. The cubic AI review passed, and all tests are green. Let me know if anything needs adjusting! |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
walk_run_contenthas no arm forw:sym, so a character drawn from a symbol font is dropped entirely (#193): the legacy bullets of old list templates vanish (the runs around them glue into a double space), and Symbol-font operators (p ≤ q) silently corrupt the sentence. The RTF and binary-DOC readers already decode the same characters; only the DOCX run walker has the gap.One correction to the issue's repro, which the source tables settle:
Wingdings F0B7is the one-o'clock clock glyph (U+1F550), not the round bullet — the round bullet of Word's legacy lists isSymbol F0B7(→•). Both fonts' code pages are covered here, so both legacy list styles get their markers back.Change
src/formats/docx/symbols.rs: the code pages of Wingdings, Wingdings 2, Wingdings 3, Webdings and Symbol, mapped to Unicode text. Source: docx4j'sSymbolMapper.java(Apache-2.0), which records Alan Wood's symbol-font tables. Every entry was cross-checked against pandoc's independent table (validation only, nothing taken from it): 1003 shared entries are identical; the differences are docx4j's deliberate corrections (Symbol's bracket codes map to real math characters instead of Adobe private-use codepoints,Wingdings F0D8→ U+27A2➢, Ohm → U+03A9).glyph()accepts both forms Word and converters write: the code byte and theU+F000..F0FFprivate-use form, with case-insensitive font names. A value too large to be a code byte loads as the character itself, but only for the five symbol fonts — the issue's converters-that-store-Unicode form (w:font="Symbol" w:char="2264"→≤) — and private-use and control values never load. Unknown font/code pairs log at debug level instead of vanishing silently;w:symof a non-symbol font (Arial, …) is dropped exactly as before.content.rsgains thesymarm, emitting the mapped text with the run's style.This generalizes #177's
symbolsmodule — same module, same arm placement, same rejection rules for malformed values. Where #177's four pinned checkbox codepoints differ from the source tables, the tables win on two glyphs (both sources agreeWingdings 0xFE→ U+1F5F9 andWingdings 2 0xA3→ U+2B1C, which still render as a checked and an empty box) and agree on the other two.Verification
No Rust toolchain on this host, so honestly:
glyph()logic was simulated line by line in Python against the same tables, running every scenario of the three new Rust regression tests: all 23 cases match the expected outputs (code-page forms, case-insensitivity, the four operators, the Unicode-valued form, private-use and malformed rejection, non-symbol fonts staying dropped).docx_partsstyle and run under CI'scargo test; they fail on HEAD (beforeafterwhere the glyph should be) and pass with the table.cargo fmt/clippynot run locally (no toolchain); formatting follows the codebase's style throughout.Fixes #193