Repository navigation
Keep Unicode letters and numbers in slugs, so a Korean heading names its file in Korean instead of untitled - #20
Merged
Conversation
…its file
한국어-제목.md instead of falling back to untitled
slugify strips everything outside \p{L}\p{N} rather than [a-z0-9], so a heading
in any non-Latin script survives as a filename instead of collapsing into
separators. Hangul that arrives decomposed (NFD, as macOS input methods produce)
is recomposed to NFC after accent folding, so the file keeps composed syllables
and resolves to one name on every platform. Truncation counts code points
instead of UTF-16 units, so a long CJK heading is not cut mid-letter and stays
inside the 255-byte filename limit; MAX_SLUG is now documented as 60 code
points, which is at most 240 UTF-8 bytes.
The note and workspace names come from the same shared slug.ts that the iOS and
Android clients load, so all three platforms name files the same way.
docs/contributor/architecture.md (the trust boundary and the filesystem
invariants) and remote.md (why .server is dotted) stated "[a-z0-9-]" as the
slug invariant; both now say Unicode letters, numbers and hyphens.
note cannot overwrite a decomposed Korean filename on APFS, keep combining marks outside Latin script, and cap slugs at 200 UTF-8 bytes so a save's temp file fits on ext4
Collaborator
|
Thanks for this! The diagnosis and the shape of the change were right. Reviewing it turned up three problems that Unicode filenames introduce, so I pushed a commit on top rather than sending you another round:
|
Contributor
Author
|
Thanks for the thorough review and the additional fixes! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a bug fix rather than a feature, so per CONTRIBUTING's "Sending a
change" I sent the PR directly instead of opening an issue first. If you would
rather discuss the design before reviewing, I am happy to move it to an issue.
What changed
A note's filename comes from its first-line H1 through
slugifyinsrc/shared/slug.ts. That function kept only[a-z0-9]and turned everythingelse into a separator, so a heading in any non-Latin script slugged to nothing
and the note landed as
untitled.md. A Korean heading now names its file한국어-제목.md, and a Korean workspace display name creates a Korean folder.Three changes in
slugify:\p{L}\p{N}instead of[a-z0-9], so letters and numbersin any script survive. Dots, separators, colons and control characters are
still removed, so the "safe by construction" property is unchanged: the
emitted slug can never contain a path character or start with a dot.
normalize("NFC"). Without this, Korean text that arrives decomposed (NFD,which is what macOS input methods and the filesystem produce) would keep its
jamo and be written as a name no other platform matches.
MAX_SLUGnow counts code points rather than UTF-16 units, so along CJK heading is not cut in the middle of a surrogate pair. The comment
documents the limit as 60 code points, at most 240 UTF-8 bytes, which keeps
room under the 255-byte filename limit for the enumerated
-2suffix and theextension.
createManagedinsrc/bun/workspaces.tsuses the sameslugify, so workspacefolders inherit this with no separate code path. Both the desktop view and the
iOS/Android clients load
src/shared/slug.ts, so all three name files the sameway.
Why
A note titled
# 한국어 제목was saved asuntitled.md, and every followingKorean-titled note enumerated
untitled-2.md,untitled-3.mdand so on. Thetitle was still in the file, but the file on disk and the note browser had lost
the name the user wrote.
How verified
Tests land in this change:
src/shared/slug.test.ts: Korean and mixed-language headings, NFD inputproducing the same composed slug, unsafe path characters still removed
(
../../한국어/제목\초안:메모\0becomes한국어-제목-초안-메모), and a longCJK heading truncated without splitting a letter and staying under 255 bytes.
The old assertion that
日本語slugs tonullis replaced by one that a pureemoji heading (
🎉, which is neither\p{L}nor\p{N}) does.src/bun/notes.fs.test.ts: two Korean headings create한국어-제목.mdand한국어-제목-2.mdwithout clobbering, and an untitled note can be retitled toa Korean heading that moves the file.
src/bun/workspaces.fs.test.ts: a Korean display name creates a Korean folder.No UI behavior changed, so
bun run test:e2edoes not apply here and there is nonative seam to check live.
Docs rules changed
This is called out per CONTRIBUTING's "Sending a change": two normative pages
stated
[a-z0-9-]as the slug invariant, and they now say Unicode letters,numbers and hyphens.
docs/contributor/architecture.md— the trust-boundary bullet ("the view nevernames a file") and the filesystem-invariants subsection on the folder name
being the only validated name.
docs/contributor/remote.md— the note on why.serveris dotted, whichrelied on managed workspace slugs never being able to collide with it. The
reasoning still holds (a slug cannot start with a dot), but the stated range
was wrong.
docs/contributor/interactions.mdand the user manual needed no edit: thefilename rule is not user-facing text there.