fix(pdf): preserve a line-final hyphen that does not split a word (#4043)

[VSathveek]

Issue resolved by this Pull Request:
Resolves #4043

Problem

PageAssembleModel.sanitize_text() drops a line-final hyphen before joining two
lines, assuming it is a hyphen that splits a word across the break. The guard
deciding this inspects the last word token anywhere in the line rather than
the hyphen itself:

prev_words = re.findall(r"\b[\w]+\b", prev_line)
...
if len(prev_words) and len(line_words) and prev_words[-1].isalnum() and line_words[0].isalnum():
    lines[ix] = prev_line[:-1]   # drop the hyphen

\b[\w]+\b skips over whitespace, so for a line ending ... -pn 1 - the last token
is "1", the guard passes, and the hyphen is deleted even though it was never
attached to a word. The wrap reported in the issue is therefore extracted as
... -pn 1 prio 3 ..., the flag silently losing its dash.

The deletion is not specific to CLI syntax. Any line-final hyphen that follows
whitespace is affected, and the existing test corpus turns out to contain four
such documents where a real character is being discarded today.

Fix

Only a hyphen attached to the preceding word marks a split word, so the fix
requires the character immediately before it to be alphanumeric:

hyphen_attached_to_word = len(prev_line) > 1 and prev_line[-2].isalnum()

That condition only ever narrows the existing one, so nothing de-hyphenated
today stops being de-hyphenated: "algo-" + "rithms" → "algorithms" is
unchanged, as is the existing exclusion of underscore-bearing tokens
("foo_bar-" + "baz" stays "foo_bar-baz").

A detached hyphen is then treated as what it is — an ordinary character that
happens to sit at a line break — so it is kept and the lines are joined with a
space, exactly as the non-hyphen branch already does.

Reference data

Four documents change. Three recover a character that is being discarded today:

2203.01017v2  Computer Vision ECCV 2020    ->  Computer Vision - ECCV 2020
2206.01062    complex layouts rdcl2017     ->  complex layouts - rdcl2017
2305.03393v1  - -"C" cell a new table cell ->  - "C" cell - a new table cell

redp5110_sampled improves structurally. Four bullet items whose - marker sat
alone on its own cell were glued to the following word, which pushed them out of
the list and into numbered entries; they are now recognised as list items:

-2. -Human Resources can see the unmasked TAX\_ID of the employees.
+- Human Resources can see the unmasked TAX\_ID of the employees.

One line moves sideways rather than forwards, and I would rather point at it than
let it pass unnoticed. In 2305.03393v1 the author ORCIDs arrive as separate
cells ("[0000", "-", "0002", ...), so the bare hyphens now take a space:

-[0000 -0002 -3723 -6960]
+[0000 - 0002 - 3723 - 6960]

Neither form is the real [0000-0002-3723-6960]; the identifier is already
broken upstream by the per-cell split and the space this branch adds between
cells. No character is gained or lost, so I left it rather than special-casing
digits, but it is the one diff here that is not an improvement.

I ran the suite against unmodified main first to confirm this environment
reproduces the committed reference data exactly, so these diffs are attributable
to the change and not to local drift. Flagging them for the double review the
contributing guide asks for.

A limitation worth stating

Whether a detached hyphen should close up against the next token (-prio) or
keep its space (- ECCV) is not decidable at this layer: the caller .strip()s
each cell before sanitize_text sees it, so the trailing-space evidence is
already gone. All four corpus cases are separator dashes wanting the space, so
that is the default here, and the issue's flag case comes out as -pn 1 - prio 3
— the dash preserved but not closed up. That fixes the data loss the issue is
about without inventing a distinction the input no longer carries. Recovering it
properly means keeping the pre-strip whitespace, which is a larger change to the
caller and I would rather not fold it in here.

Tests

TestSanitizeTextHyphenation in tests/test_page_assemble_model.py covers both
directions: an attached hyphen still joins a split word, a non-alphanumeric
token is still left alone, and detached hyphens (wrapped flag, separator dash,
bullet marker, negative number) are preserved. The detached cases fail on main.

One related spot, left alone deliberately

ReadingOrderModel._merge_elements has a parallel end-of-line hyphen check for
merging elements across cluster boundaries and drops a detached hyphen the same
way. I have not touched it here, since the module has other work in flight
(#3891, #3955). Happy to follow up separately, or to fold it in if you prefer.

Checklist:

• Documentation has been updated, if necessary.
• Examples have been added, if necessary.
• Tests have been added, if necessary.

[github-actions]

✅ DCO Check Passed

Thanks @VSathveek, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🔴 1 of 2 protections blocking · waiting on 👀 reviews

	Protection	Waiting on
🔴	Require two reviewer for test updates	👀 reviews
🟢	Enforce conventional commit	—

🔴 Require two reviewer for test updates

Waiting for

• #approved-reviews-by >= 2

This rule is failing.

When test data is updated, we require two reviewers

• #approved-reviews-by >= 2

---

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?: