docs: document the length chunker boundary settings - #475
Open
marevol wants to merge 5 commits into
Open
Conversation
Add content_chunker.length.boundary.enabled / lookback_percent / lookahead_percent to the system.properties reference table in config/search-semantic.rst, next to the existing content_chunker.length.overlap entry, across all 7 languages (en, ja, de, fr, es, ko, zh-cn). Also add a note explaining that content_chunker.length.chunk_size is now a target rather than a hard ceiling: the boundary search can overshoot it by up to lookahead_percent, a separate grapheme-cluster overshoot of up to 32 characters can occur independently of that setting, and chunks can come in up to lookback_percent shorter (affecting content_chunker.max_chunks_per_document). Setting boundary.enabled=false, or both percentages to 0, restores the previous exact fixed-length behavior.
Address 11 review findings on the search-semantic.rst boundary-settings addition, across all 7 languages (en, ja, de, fr, es, ko, zh-cn): - Add the missing "space" break kind to the boundary.enabled row and note, matching fess_config.properties' five-way enumeration (line break, sentence end, comma, space, script change). - en: escape the lookback/lookahead range annotations with `\ ` so they render without a space, matching the other 6 languages and en's own neighboring `-1``\ (unlimited)`` row. - ja: fix "近い" -> "最も近い" (approximately near -> nearest) so the note doesn't understate how precisely the cut is aligned. - ja: replace the invented terms "カット"/"切断位置" with "分割位置", matching the terminology already used throughout the page. - zh-cn: fix the note's direction word (向前搜索 -> 向后搜索) so it matches the table's own lookahead-vs-lookback convention. - es/fr: localize "joiner"/"joigneur" as "ensamblador de ancho cero (ZWJ)" / "jointeur de largeur nulle (ZWJ)". - fr: "tomberait sinon" -> "tomberait autrement"; "de jusqu'à" -> "d'au plus" (both occurrences). - de: rework "Satzwechsel"/"Teilsatzwechsel" into the grammatically correct "Satz-, Teilsatz- oder Zeilengrenze ... Schriftsystemwechsel". - ko: "경계를 얼마나 찾을 수 있는지" -> "얼마나 멀리까지 경계를 탐색할 수 있는지" (searches for how much boundary -> how far the search may look). - Add a short "(target, see note below)" pointer to the pre-existing chunk_size row in all 7 languages, since the target-vs-ceiling caveat lives in a note ~70 lines below the row. Verified: docutils parses all 7 files with the identical error/warning set (line-number-normalized) before and after; all touched lines stay within each file's pre-existing max line length; defaults/ranges/figures (true, 20 (0-50), 5 (0-25), 32, 841) are unchanged and consistent across languages.
Address 4 review findings on the search-semantic.rst boundary-settings
addition, across all 7 languages (en, ja, de, fr, es, ko, zh-cn):
- Replace the "moves to the nearest break" claim (row + note, all 7
languages) with the tiered behaviour ChunkBoundaryFinder actually
implements: it prefers a line break or sentence end (STRONG) over a
clause separator or space (WEAK), and those over a script change
(SCRIPT), within the lookback/lookahead search window -- not simply
the nearest candidate of any kind. A sentence end further back can
beat a closer space, so a chunk can end noticeably earlier than
chunk_size to land on a sentence boundary.
- de: reflow the note paragraph that a previous fix left with a
40-column orphan line ("Die Vorwaertssuche kann ``chunk_size``")
not forced by any long literal.
- zh-cn: fix a break that separated the measure word from its noun
(一篇 / 文档 across a line break); also reflow a similarly
unforced short line the same fix introduced.
- es: drop the article on "un espacio" (was the only articled item in
an otherwise bare-noun list) and replace "ensamblador" (reads as a
computing "assembler") with "conector de ancho cero"; fix the
resulting nested "))" from the ZWJ gloss in both es and fr.
- ko: drop the redundant 참고/참조 pairing on the chunk_size row's
target-value pointer.
Verified: docutils parses all 7 files with a message set identical
(line-number-normalized) to both the pre-task-8 baseline and the prior
fix rounds; defaults/ranges/figures (true, 20 (0-50), 5 (0-25), 32, 841)
unchanged and consistent across languages; every touched line stays
within its file's pre-existing maximum width.
Change of method for this round: every paragraph touched by the
length-chunker boundary-settings addition (the boundary.enabled table
cell and its note, across all 7 languages) was rewrapped from scratch
using an East-Asian-Width-aware greedy-fill algorithm, instead of
hand-patching individual lines. For ja/zh-cn the source text was
pre-segmented into non-breakable phrase units (compounds, and each
noun with its bound particle/classifier fused together) so the
wrapper can only place a line break after a particle or after
punctuation, never inside a word or between a classifier and its
noun.
Wrapping defects cleared (all introduced by the previous round's
line-patching):
- es: removed a 40-column orphan ("puede superar ``chunk_size`` en
hasta") in an otherwise 89-95-column paragraph.
- zh-cn: removed a 25-column orphan ("此外还可能出现第二种、") and a
split of 搜索窗口 across a line break (rendered as "搜索 窗口内").
- ja: removed a split of 検索ウィンドウ across a line break, and a
split that orphaned the particle を from それら.
Content/terminology fixes:
- de: restored the broader "clause separator" term (Teilsatztrenn-
zeichen) in place of "ein Komma", since isClauseSeparator covers
more than commas (; : - 、 ， ； ： ､ ･ ・ ‐ ‑ – — ― 〜 ～ among
others); removed a newly-introduced second colon in the same
sentence.
- es: made the boundary.enabled row/note use "límite" instead of
"ruptura" for the break, matching the pre-existing, untouched
lookback_percent/lookahead_percent rows that already say "límite".
- ko: replaced 검색 창 (reads as a search-box UI widget) with 탐색
terminology, matching how the rest of the file already refers to
this search (경계를 탐색할 수 있는지, 전방 탐색).
- en/de/fr/es/ko/zh-cn/ja: separated the no-character-loss guarantee
from the tier-preference clause it was previously (incorrectly)
presented as a consequence of; it now follows from the boundary
being moved rather than the text being discarded, stated as its own
sentence.
Verified: docutils message set identical (line-number-normalized) to
the pre-task-8 baseline for all 7 files; every compound/classifier+
particle term checked programmatically as never split across a line
boundary anywhere in the ja/zh-cn files; numbers/ranges/figures
unchanged and consistent across languages; every line's East-Asian-
Width-aware display width measured and confirmed within each file's
pre-existing maximum (123 columns, shared JSON code block, identical
in all 7 languages).
Two single-line fixes, no rewrapping or wording changes beyond these:
- de: "Verschoben wird dabei nur die Schnittstelle selbst" ->
"... der Schnittpunkt selbst". Schnittstelle means "interface"
(API), not "cut point" -- inconsistent with this file's own
Schnittpunkt/Schnitt usage and with every other language's "cut
point" wording.
- ja: moved the line break from after the full-width opening bracket
"(" to before it, so the source no longer ends a line on a bare
opening bracket (a hard line-break there survives into the
rendered <p> and the browser collapses it to a visible space just
inside the bracket -- also a 行末禁則 violation). Matches the
file's own convention of keeping "(" attached to what follows
(e.g. "(``m=16``" a few lines below).
Verified: docutils message set identical to the 02c5b80~1 baseline
in all 7 languages; EAW max still 123 in all 7 files, at or below
each file's pre-existing baseline; true/32/841 all still present in
every language; diff confirms only these two lines changed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documents the three
content_chunker.length.boundary.*settings added by the boundary-aware chunking change infess, and the resulting change to whatcontent_chunker.length.chunk_sizemeans.Added to
15.8/config/search-semantic.rstin all seven languages (en, ja, de, fr, es, ko, zh-cn), as new rows next to the existingcontent_chunker.length.overlapentry plus one note after the table. All seven already carried a fully parallel translation of that table, so leaving five of them untouched would have left them describingchunk_sizeas an exact per-chunk character count.What is documented
content_chunker.length.boundary.enabledtruecontent_chunker.length.boundary.lookback_percent20content_chunker.length.boundary.lookahead_percent5The note covers:
chunk_sizeis now a target rather than a hard ceiling. The forward sentence search may overshoot by up tolookahead_percent; separately, and not governed bylookahead_percent, moving off a grapheme cluster may overshoot by up to 32 characters. The two never occur on the same cut, so the worst case is about 841 characters at the shipped defaults.lookback_percentshorter, so a document may produce slightly more chunks -- relevant tocontent_chunker.max_chunks_per_document.boundary.enabled=false, or both percentages at0, restores the previous fixed-length behaviour.The
chunk_sizerow now points at that note, so a reader scanning the table does not miss the target-versus-ceiling caveat.Notes for review
The tier precedence and every figure were checked against the implementation, and the numbers are identical across all seven languages. Each file parses with docutils with exactly the same message set as before the change.
The non-English text was not reviewed by a native speaker; corrections welcome, particularly for the de, fr, es and ko notes.