Skip to content

Count GPUI editor positions in UTF-16 code units like ProseMirror - #7538

Closed
devin-ai-integration[bot] wants to merge 9 commits into
chore/1789065520-editor-roundtrip-fixturesfrom
fix/1789066022-pm-utf16-positions
Closed

devin-ai-integration[bot] wants to merge 9 commits into
chore/1789065520-editor-roundtrip-fixturesfrom
fix/1789066022-pm-utf16-positions

Conversation

@devin-ai-integration

Copy link
Copy Markdown
Contributor

Summary

Problem: The round-trip harness (#7537) found that the GPUI ProseMirror port sizes text nodes in Unicode scalar values, while ProseMirror uses JS string length (UTF-16 code units). Any position exchanged with web-authored data — paste fixture offsets, comment anchor fromHint/toHint, future collaborative positions — is off by one per astral character (emoji, some CJK extension B).

Fix: pm/node.rs node_size/cut now count and slice by UTF-16 units (utf16_len, utf16_slice; a mid-surrogate index — which JS would express as a lone surrogate — clamps to the containing Rust char), and paste.rs position/caret convert byte offsets ↔ UTF-16 units instead of chars. model.rs stays byte-based internally as before. basic-unicode and markdown-table are removed from the harness's KNOWN_DIVERGENCES, so the web fixtures now assert nodeSize equality for them.

Stacked on #7537 (base branch is that PR's branch).

Verification

ubuntu-24.04 CI-equivalent container:

  • cargo test --locked -p desktop-gpui editor (70 passed, incl. paste.gen.json fixtures, roundtrip, new astral-text tests for sizes/cuts/resolve/caret)
  • cargo test --locked -p desktop-gpui (466 passed)
  • cargo clippy --locked -p desktop-gpui --all-targets --no-deps -- -D warnings
  • rustfmt --edition 2024 on changed files

Link to Devin session: https://app.devin.ai/sessions/f83ed73d85c648fcbdbd041cd4f15720
Open in Devin Desktop: https://app.devin.ai/desktop/session/f83ed73d85c648fcbdbd041cd4f15720?variant=devin
Requested by: @ComputelessComputer

devin-ai-integration Bot and others added 2 commits September 10, 2026 18:48
Co-Authored-By: John <john@fastrepl.com>
Co-Authored-By: John <john@fastrepl.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR that start with 'DevinAI' or '@devin'.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

…p-fixtures' into fix/1789066022-pm-utf16-positions
devin-ai-integration[bot]

This comment was marked as resolved.

…_slice

Co-Authored-By: John <john@fastrepl.com>
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 5 commits September 11, 2026 13:13
… the preceding one

Co-Authored-By: John <john@fastrepl.com>
…p-fixtures' into fix/1789066022-pm-utf16-positions
…p-fixtures' into fix/1789066022-pm-utf16-positions
…p-fixtures' into fix/1789066022-pm-utf16-positions
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

Superseded by the consolidated GPUI migration PR #7655, which lands this stack in one change on current main. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant