From 47f693836c8231ecaca3faa927d1e3faa2828dfb Mon Sep 17 00:00:00 2001 From: Dmitry Ilyin <6576495+widgetii@users.noreply.github.com> Date: Sun, 4 Oct 2026 23:23:00 +0300 Subject: [PATCH 1/3] feat: read DSL files and group their lines into cards MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Layers one and two of the importer: bytes to lines, lines to cards. With the lexer and the sense builder already in, this reads the real 350 MB set end to end — 3,434,224 cards, matching the known count exactly, in 2.3 seconds under 10 MB of resident memory. **The line terminator is a code unit, not a byte pair.** In UTF-16LE a line feed is `0x000A`, which is the bytes `0A 00` *at an even offset*. `ਅ昀` — U+0A05 then U+6600 — is `05 0A 00 66`, so a scanner that searches the raw bytes breaks the line between the two bytes of a single character. The predecessor scanned for `0A 00` with no alignment check and got away with it: across all three files there are 10,302,687 aligned line feeds and **zero** misaligned occurrences of the pair, because a Chinese-Russian dictionary contains no Gurmukhi. So this is not a fix for an observed failure — it is the difference between code that is correct and code that happens to work on one input, and the aligned scan costs nothing. A short read is looped over rather than trusted, so the same misalignment cannot arrive by that route either. **`pread`, not `FileHandle`, and that is a measured decision.** Reading 116 MB through `FileHandle.read(upToCount:)` took peak resident size to 124 MB — the size of the file — because Foundation maps a large read instead of copying it, so resident size follows the input. The import has a 60 MB ceiling. The same read through `pread` into a reusable buffer costs 1 MiB whatever the file size, which is the entire point of reading in chunks. **UTF-16 is transcoded by hand, which is 7× faster.** Over 3.4 million lines, gathering the code units into a `[UInt16]` costs 0.13 s and handing them to `String(decoding:as: UTF16.self)` costs a further 3.05 s; writing UTF-8 directly, so `String` is handed its own native storage, costs 0.43 s. Output is byte-identical — 85,997,869 UTF-8 bytes either way — and it takes reading the whole set from 10 seconds to 1.2. Line endings are mixed *within* each file: exactly two CRLF lines, `#NAME` and `#INDEX_LANGUAGE`, among three million LF ones. Both are handled on every line. And all three files end with a line feed, which splitting the text turns into an empty final element — an unindented empty line is indistinguishable from a headword, and that is a headword this dictionary does not contain. **The card reader is positional and nothing else.** An indented line is article text, an unindented `#` line is a directive, anything else unindented is a headword. Indentation is tested first, so an indented `#` stays article text. Consecutive headwords are one card: BKRS never does it, but refusing it would be a parser bug waiting for the next dictionary. Each card records the byte offset of its headword line, which is the only place an interrupted import can restart without losing a card or writing one twice. **The file set is discovered, never hardcoded.** `dabkrs_1.dsl` names the other two in its own `#INCLUDE` directives; the predecessor hardcoded all three filenames and the dictionary's name, so it imported this one dictionary and no other and stopped working if a file was renamed. Includes are authoritative and followed with a visited set so a cycle cannot loop; only when a file declares none does discovery look at the directory, and then only for a contiguous run of numbered siblings. Picking volume two restarts from volume one, because that is where the directives live. The seam repair the plan asks for is here as two properties rather than a third of an implementation: a headword left dangling at the end of a file is reported and handed out as `trailingHeadwords`, and a reader can be given `leadingHeadwords` to complete it. Both are zero for this set, which is card-aligned; they are insurance against a set someone split with `split(1)`. **A 203-card fixture of real data**, with an extraction script so its provenance is documented and it can be regenerated. Thirteen cards are picked deliberately — 爱, 了, 一 and 打 for their sense structure, one per piece of markup, and the single card whose article breaks a line inside an open `[m1]` — and 190 are a contiguous run, which covers the shapes nobody thought to pick. Committed as UTF-8 so it stays diffable; the tests re-encode it to UTF-16LE+BOM and assert both readings produce identical cards, so the bytes are reviewable and the real encoding path is still exercised. 52 KB, well inside the 256 KB limit CI enforces, and permitted by the one exception CONTRIBUTING names — 大БКРС is not ours to redistribute. `DSLDiagnostic` moves to its own file now that two layers produce them, and gains a byte offset for the ones found at card level. Co-Authored-By: Claude Opus 5 --- Package.swift | 7 +- Scripts/extract-dsl-fixture.py | 207 ++++++ .../HanReaderCore/Format/DSLByteReader.swift | 535 +++++++++++++++ .../HanReaderCore/Format/DSLCardReader.swift | 253 ++++++++ .../HanReaderCore/Format/DSLDiagnostic.swift | 46 ++ Sources/HanReaderCore/Format/DSLFileSet.swift | 154 +++++ Sources/HanReaderCore/Format/DSLLexer.swift | 20 - .../DSLByteReaderTests.swift | 266 ++++++++ .../DSLCardReaderTests.swift | 394 +++++++++++ .../HanReaderCoreTests/DSLFixtureTests.swift | 244 +++++++ Tests/HanReaderCoreTests/Fixtures/README.md | 34 + .../Fixtures/bkrs-slice.dsl.txt | 614 ++++++++++++++++++ .../Fixtures/bkrs-slice.manifest.txt | 208 ++++++ 13 files changed, 2959 insertions(+), 23 deletions(-) create mode 100755 Scripts/extract-dsl-fixture.py create mode 100644 Sources/HanReaderCore/Format/DSLByteReader.swift create mode 100644 Sources/HanReaderCore/Format/DSLCardReader.swift create mode 100644 Sources/HanReaderCore/Format/DSLDiagnostic.swift create mode 100644 Sources/HanReaderCore/Format/DSLFileSet.swift create mode 100644 Tests/HanReaderCoreTests/DSLByteReaderTests.swift create mode 100644 Tests/HanReaderCoreTests/DSLCardReaderTests.swift create mode 100644 Tests/HanReaderCoreTests/DSLFixtureTests.swift create mode 100644 Tests/HanReaderCoreTests/Fixtures/bkrs-slice.dsl.txt create mode 100644 Tests/HanReaderCoreTests/Fixtures/bkrs-slice.manifest.txt diff --git a/Package.swift b/Package.swift index a02cea9..866cef0 100644 --- a/Package.swift +++ b/Package.swift @@ -144,9 +144,10 @@ let package = Package( .testTarget( name: "HanReaderCoreTests", dependencies: ["HanReaderCore"], - // Real CC-CEDICT data, a small slice extracted by - // Scripts/extract-cedict-fixture.py so its provenance is - // documented and it can be regenerated rather than hand-edited. + // Small slices of the real CC-CEDICT and 大БКРС data, each + // extracted by a committed script (Scripts/extract-cedict-fixture.py + // and Scripts/extract-dsl-fixture.py) so their provenance is + // documented and they can be regenerated rather than hand-edited. // // Kept inside the target directory rather than at Tests/Fixtures: // a resource path reaching outside the target works on the current diff --git a/Scripts/extract-dsl-fixture.py b/Scripts/extract-dsl-fixture.py new file mode 100755 index 0000000..7ec69c6 --- /dev/null +++ b/Scripts/extract-dsl-fixture.py @@ -0,0 +1,207 @@ +#!/usr/bin/env python3 +"""Extract the committed BKRS test fixture from a full 大БКРС DSL set. + +The fixture is a deliberately small slice of real DSL data, chosen to cover +every shape the importer has to handle. It is committed so the tests are +hermetic and fast; this script exists so its provenance is documented and the +slice can be regenerated rather than hand-maintained. + + ./Scripts/extract-dsl-fixture.py path/to/dabkrs_1.dsl + +`#INCLUDE` directives are followed, so pointing it at the master file is +enough. Writes Tests/HanReaderCoreTests/Fixtures/bkrs-slice.dsl.txt. + +The output is **UTF-8 with no byte-order mark**, where the source is +UTF-16LE+BOM. That is on purpose twice over: a UTF-8 file stays diffable in +git, and the tests re-encode it to UTF-16LE+BOM at runtime and assert both +readings produce identical cards — so the committed bytes are reviewable and +the real encoding path is still exercised. + +**Licensing.** 大БКРС is not freely licensed and is not ours to redistribute. +HanReader ships an importer, not the data. This slice exists under the one +exception CONTRIBUTING.md names: a ~200-card test fixture, regenerable by this +script. Do not enlarge it, and do not commit the source files. +""" + +from __future__ import annotations + +import pathlib +import re +import sys + +# Cards picked by headword, each for a stated reason, so a future edit can tell +# what it would be removing. +WANTED: dict[str, str] = { + # The worked examples from the design: a division heading, numbered senses + # under it, examples, and a second division. + "爱": "division + numbered senses + example; the plan's worked example", + # Homographs and multi-reading cards. The predecessor put the whole + # comma-separated reading above the word; one entry per reading is what + # replaces that. + "了": "two readings, le and liǎo; 30 senses", + "一": "57 senses; carries its own tone-sandhi note as prose", + "打": "57 senses, the deepest nesting in the set", + # The one card in 3,434,224 whose article is split across two physical + # lines, inside an open [m1]. + "乐芙兰": "article spans two lines inside an open [m1]", + # A run-together reading with the U+2019 apostrophe the set uses as a + # syllable boundary. + "土耳其斯坦": "run-together reading with a U+2019 syllable boundary", +} + +# Cards picked by what their markup contains. The first card matching each +# pattern is taken, so the slice stays deterministic. +PATTERNS: dict[str, tuple[str, str]] = { + "escaped-brackets": (r"\\\[", "escaped \\[ — 55,414 in the set; stripDSL ate them"), + "colour-argument": (r"\[c [a-z]+\]", "[c brown]: the only tag taking an argument"), + "level-four": (r"\[m4\]", "[m4]: 344 in the set, the deepest level used"), + "block-no-example": ( + r"\[\*\](?:(?!\[ex\]).)*\[/\*\]", + "[*] with no [ex] inside — 62 in the set, and that path must not crash", + ), + "reference": (r"\[ref\]", "[ref] cross-reference"), + "register": (r"\[p\](?:бот|ист|уст)\.", "a [p] field marker rather than a part of speech"), + "no-reading": (r"^_$", "`_` for the reading — 77% of the set"), +} + +HEADER = """#NAME "HanReader test slice of 大БКРС" +#INDEX_LANGUAGE "Chinese" +#CONTENTS_LANGUAGE "Russian" +""" + +# A run of consecutive cards taken verbatim after the anchor, on top of the +# picks above. Curated cards cover the shapes someone thought of; a contiguous +# run covers the ones nobody did, at a realistic density. 爱 is the anchor +# because it is already in the slice and its position is stable. +RUN_ANCHOR = "爱" +RUN_LENGTH = 190 + +# DSL has no comment syntax — a `#` line is a directive — so the reasons go in +# a sidecar instead of into the fixture, which stays a file the importer can +# read as-is. +MANIFEST_HEADER = """Why each card is in bkrs-slice.dsl.txt. + +Generated by Scripts/extract-dsl-fixture.py; do not hand-edit. DSL has no +comment syntax, so this cannot live inside the fixture itself. +""" + + +def read_dsl(path: pathlib.Path) -> list[str]: + """Decodes one DSL file into lines, honouring its byte-order mark.""" + raw = path.read_bytes() + if raw[:2] == b"\xff\xfe": + text = raw[2:].decode("utf-16-le") + elif raw[:2] == b"\xfe\xff": + text = raw[2:].decode("utf-16-be") + elif raw[:3] == b"\xef\xbb\xbf": + text = raw[3:].decode("utf-8") + else: + text = raw.decode("utf-8") + return [line.rstrip("\r") for line in text.split("\n")] + + +def file_set(master: pathlib.Path) -> list[pathlib.Path]: + """Follows #INCLUDE from the master file.""" + files = [master] + for line in read_dsl(master): + if not line.startswith("#"): + break + match = re.match(r'#INCLUDE\s+"([^"]+)"', line) + if match: + sibling = master.parent / match.group(1) + if sibling.exists(): + files.append(sibling) + return files + + +def cards(lines: list[str]): + """Yields (headwords, body_lines) for each card.""" + index, count = 0, len(lines) + while index < count: + line = lines[index] + if not line or line.startswith("#"): + index += 1 + continue + headwords = [] + while index < count and lines[index] and not lines[index][:1] in (" ", "\t", "#"): + headwords.append(lines[index]) + index += 1 + body = [] + while index < count and lines[index][:1] in (" ", "\t"): + body.append(lines[index]) + index += 1 + if headwords and body: + yield headwords, body + + +def main() -> int: + if len(sys.argv) != 2: + print(__doc__, file=sys.stderr) + return 2 + master = pathlib.Path(sys.argv[1]).expanduser() + if not master.exists(): + print(f"no such file: {master}", file=sys.stderr) + return 1 + + compiled = {key: re.compile(pattern) for key, (pattern, _) in PATTERNS.items()} + chosen: dict[str, tuple[list[str], list[str], str]] = {} + found_patterns: set[str] = set() + run_remaining = 0 + + for path in file_set(master): + for headwords, body in cards(read_dsl(path)): + key = headwords[0] + if run_remaining > 0 and key not in chosen: + run_remaining -= 1 + chosen[key] = (headwords, body, f"consecutive run after {RUN_ANCHOR}") + continue + if key in WANTED and key not in chosen: + chosen[key] = (headwords, body, WANTED[key]) + if key == RUN_ANCHOR: + run_remaining = RUN_LENGTH + continue + stripped = [line.strip() for line in body] + for name, regex in compiled.items(): + if name in found_patterns or key in chosen: + continue + if any(regex.search(line) for line in stripped): + found_patterns.add(name) + chosen[key] = (headwords, body, PATTERNS[name][1]) + + missing = sorted(set(WANTED) - set(chosen)) + if missing: + print(f"warning: headwords not found: {', '.join(missing)}", file=sys.stderr) + for name in sorted(set(PATTERNS) - found_patterns): + print(f"warning: no card matched {name}", file=sys.stderr) + + root = pathlib.Path(__file__).resolve().parent.parent + out = root / "Tests/HanReaderCoreTests/Fixtures/bkrs-slice.dsl.txt" + manifest = root / "Tests/HanReaderCoreTests/Fixtures/bkrs-slice.manifest.txt" + + with out.open("w", encoding="utf-8", newline="\n") as handle: + handle.write(HEADER) + handle.write("\n") + for key in sorted(chosen): + headwords, body, _ = chosen[key] + for headword in headwords: + handle.write(headword + "\n") + for line in body: + handle.write(line + "\n") + + with manifest.open("w", encoding="utf-8", newline="\n") as handle: + handle.write(MANIFEST_HEADER) + handle.write("\n") + for key in sorted(chosen): + handle.write(f"{key}\t{chosen[key][2]}\n") + + size = out.stat().st_size + print(f"wrote {out.relative_to(root)} and {manifest.relative_to(root)}") + print(f" {len(chosen)} cards, {size:,} bytes") + if size > 256 * 1024: + print(" error: over the 256 KB fixture limit CI enforces", file=sys.stderr) + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/Sources/HanReaderCore/Format/DSLByteReader.swift b/Sources/HanReaderCore/Format/DSLByteReader.swift new file mode 100644 index 0000000..0e23f51 --- /dev/null +++ b/Sources/HanReaderCore/Format/DSLByteReader.swift @@ -0,0 +1,535 @@ +// HanReader — MIT licensed. See LICENSE. + +public import Foundation + +/// How a DSL file's bytes spell its characters. +/// +/// ABBYY's own tools write UTF-16 with a byte-order mark, and all three files +/// of the BKRS set are UTF-16LE+BOM. Dictionaries assembled by hand are +/// routinely UTF-8, so both are first-class rather than one being a fallback. +public enum DSLByteEncoding: Hashable, Sendable { + case utf16LittleEndian + case utf16BigEndian + case utf8 + + /// Bytes per code unit. Line scanning, offset validation and + /// carriage-return stripping all turn on this. + var codeUnitWidth: Int { + self == .utf8 ? 1 : 2 + } +} + +/// One line of a DSL file, with where it started. +public struct DSLLine: Hashable, Sendable { + /// The line's text, with the line terminator and any carriage return + /// removed. + public let text: String + /// Byte offset of the line's first byte from the start of the file. + /// + /// A byte offset, not a line number, because this is what a resumed + /// import seeks to. It is also stable under a parser change in a way a + /// line count is not. + public let offset: Int + + public init(text: String, offset: Int) { + self.text = text + self.offset = offset + } +} + +/// Something that cannot be read as a DSL file at all. +/// +/// All three cases are about the file's shape rather than its content. A line +/// the parser dislikes is a diagnostic; a file whose bytes cannot form whole +/// characters is an error, because every offset derived from it would be a +/// guess. +public enum DSLByteError: Error, Hashable, Sendable, CustomStringConvertible { + /// A UTF-16 file whose length is not a whole number of code units, so its + /// last character is cut in half. + case oddByteLength(byteCount: Int) + /// A resume offset that does not sit on a code-unit boundary. + case misalignedOffset(Int) + /// A resume offset past the end of the file, which is what a stale + /// checkpoint against a replaced file looks like. + case offsetOutOfRange(offset: Int, byteCount: Int) + /// The file could not be opened or measured. + case cannotOpen(path: String, code: Int32) + /// A read failed part-way through the file. + case readFailed(offset: Int, code: Int32) + + public var description: String { + switch self { + case let .oddByteLength(byteCount): + "the file is \(byteCount) bytes long, which is not a whole number of " + + "UTF-16 code units — it is truncated or not UTF-16" + case let .misalignedOffset(offset): + "byte offset \(offset) is odd, so it falls inside a UTF-16 code unit" + case let .offsetOutOfRange(offset, byteCount): + "byte offset \(offset) is past the end of a \(byteCount)-byte file" + case let .cannotOpen(path, code): + "cannot open \(path): \(String(cString: strerror(code)))" + case let .readFailed(offset, code): + "read at byte \(offset) failed: \(String(cString: strerror(code)))" + } + } +} + +// MARK: - Byte sources + +/// Random-access bytes for `DSLByteReader`. +/// +/// Random access rather than a stream, because resuming an interrupted import +/// means starting at a recorded byte offset, and because it keeps the source +/// free of a read position that would have to be kept in step with the +/// reader's own. +public protocol DSLByteSource { + /// Total length of the file. Needed before reading in order to reject an + /// odd-length UTF-16 file rather than discovering it 300 MB in. + var byteCount: Int { get } + + /// Reads from `offset` into `buffer` starting at index `destination`, + /// filling at most to the end of `buffer`. + /// + /// Returns the number of bytes written, or 0 at end of file. A short read + /// is allowed: the reader loops until the buffer is full, so a source need + /// not return everything asked for. + func read(at offset: Int, into buffer: inout [UInt8], from destination: Int) throws -> Int +} + +/// Bytes already in memory, for fixtures and tests. +public struct DSLMemoryBytes: DSLByteSource, Sendable { + private let bytes: [UInt8] + + public init(_ bytes: [UInt8]) { + self.bytes = bytes + } + + public init(_ data: Data) { + bytes = [UInt8](data) + } + + public var byteCount: Int { + bytes.count + } + + public func read( + at offset: Int, + into buffer: inout [UInt8], + from destination: Int, + ) throws + -> Int + { + guard offset >= 0, offset < bytes.count, destination < buffer.count else { return 0 } + let count = min(buffer.count - destination, bytes.count - offset) + buffer.replaceSubrange( + destination ..< (destination + count), + with: bytes[offset ..< (offset + count)], + ) + return count + } +} + +/// Bytes read from a file a chunk at a time. +/// +/// `pread` rather than `FileHandle`, for a measured reason. The import has a +/// 60 MB ceiling over baseline, and reading 116 MB through +/// `FileHandle.read(upToCount:)` took peak resident size to 124 MB — the size +/// of the file — because Foundation maps a large read instead of copying it, +/// so resident size follows the input. The same read through `pread` into a +/// reusable buffer costs 1 MiB regardless of file size, which is the whole +/// point of reading in chunks. +/// +/// `pread` also takes the offset as an argument rather than as file state, so +/// there is no seek position to keep in step with the reader's own. +public final class DSLFileBytes: DSLByteSource { + private let descriptor: Int32 + public let byteCount: Int + + public init(url: URL) throws { + descriptor = open(url.path, O_RDONLY) + guard descriptor >= 0 else { + throw DSLByteError.cannotOpen(path: url.path, code: errno) + } + var status = stat() + guard fstat(descriptor, &status) == 0 else { + let code = errno + close(descriptor) + throw DSLByteError.cannotOpen(path: url.path, code: code) + } + byteCount = Int(status.st_size) + } + + deinit { + close(descriptor) + } + + public func read( + at offset: Int, + into buffer: inout [UInt8], + from destination: Int, + ) throws + -> Int + { + guard offset >= 0, offset < byteCount, destination < buffer.count else { return 0 } + let wanted = buffer.count - destination + return try buffer.withUnsafeMutableBytes { raw in + guard let base = raw.baseAddress else { return 0 } + let count = pread(descriptor, base.advanced(by: destination), wanted, off_t(offset)) + guard count >= 0 else { throw DSLByteError.readFailed(offset: offset, code: errno) } + return count + } + } +} + +// MARK: - The reader + +/// Reads a DSL file as lines, each with its byte offset. +/// +/// ### The line terminator is a code unit, not a byte pair +/// +/// In UTF-16LE a line feed is the code unit `0x000A`, which is the bytes +/// `0A 00` **at an even offset**. Scanning the raw bytes for that pair +/// without checking alignment is a latent corruption: a character in +/// `U+0A00`–`U+0AFF` puts `0A` at an odd offset, and if the next character's +/// low byte is zero the pair appears across the two of them. `ਅ昀` — +/// `U+0A05` then `U+6600` — is `05 0A 00 66`, and a byte scanner breaks the +/// line between the two bytes of a single character. +/// +/// The predecessor prototype scanned for `0A 00` with no alignment check. It +/// got away with it: across all three BKRS files there are 10,302,687 aligned +/// line feeds and **zero** misaligned occurrences of the pair, because a +/// Chinese-Russian dictionary contains no Gurmukhi. So this is not a bug fix +/// for an observed failure — it is the difference between code that is +/// correct and code that happens to work on one input, and the aligned scan +/// costs nothing. +/// +/// ### Line endings are mixed within a single file +/// +/// Each BKRS file has exactly two CRLF lines — `#NAME` and `#INDEX_LANGUAGE` +/// — and three million LF ones. A reader that splits on CRLF leaves a stray +/// carriage return on the dictionary's own name; one that splits on LF and +/// forgets to strip CR does the same. Both terminators are handled on every +/// line. +/// +/// ### A trailing newline does not make a final empty line +/// +/// All three files end with a line feed, so splitting the text yields an +/// empty final element. That empty string is not a line — and at the card +/// level an unindented empty line looks exactly like a headword, which is a +/// headword this dictionary does not contain. +public struct DSLByteReader { + /// 1 MiB. Large enough that a 350 MB file costs ~350 reads, small enough + /// to be irrelevant against the memory ceiling. + public static let defaultChunkSize = 1 << 20 + + /// How the file spells its characters, from its BOM or from a sniff. + public let encoding: DSLByteEncoding + /// Total file length. + public let byteCount: Int + + private let source: any DSLByteSource + /// The read buffer, allocated once and reused for the whole file. + private var chunk: [UInt8] + /// Absolute offset of `chunk[0]`. Always even for UTF-16, which is what + /// makes an index's parity within the chunk equal its parity in the file. + private var chunkStart = 0 + /// Index into `chunk` of the next unexamined byte. + private var cursor = 0 + /// Valid bytes in `chunk`. + private var available = 0 + /// Absolute offset of the next byte to read from the source. + private var nextOffset: Int + /// Bytes of the line being assembled, which may span chunks. + private var pending: [UInt8] = [] + /// Absolute offset of `pending`'s first byte. + private var pendingStart: Int + /// UTF-8 scratch for decoding, reused rather than allocated per line. + private var scratch: [UInt8] = [] + private var reachedEnd = false + + /// Opens a source for reading. + /// + /// - Parameters: + /// - source: the bytes to read. + /// - offset: where to start, for resuming an interrupted import. Nil + /// starts at the beginning, skipping the byte-order mark. A supplied + /// offset is taken literally — a resume point recorded by this reader + /// is already past the BOM. + /// - chunkSize: read-buffer size, rounded up to a whole number of + /// UTF-16 code units. + public init( + source: any DSLByteSource, + resumingAt offset: Int? = nil, + chunkSize: Int = Self.defaultChunkSize, + ) throws { + self.source = source + byteCount = source.byteCount + + var prefix = [UInt8](repeating: 0, count: min(64, max(byteCount, 0))) + if !prefix.isEmpty { + var filled = 0 + while filled < prefix.count { + let read = try source.read(at: filled, into: &prefix, from: filled) + guard read > 0 else { break } + filled += read + } + prefix.removeLast(prefix.count - filled) + } + let detected = Self.detectEncoding(prefix: prefix) + encoding = detected.encoding + + if encoding.codeUnitWidth == 2, !byteCount.isMultiple(of: 2) { + throw DSLByteError.oddByteLength(byteCount: byteCount) + } + + let start = offset ?? detected.bomLength + guard start >= 0, start <= byteCount else { + throw DSLByteError.offsetOutOfRange(offset: start, byteCount: byteCount) + } + guard start.isMultiple(of: encoding.codeUnitWidth) else { + throw DSLByteError.misalignedOffset(start) + } + + // Even, so a UTF-16 code unit never straddles two chunks. + chunk = [UInt8](repeating: 0, count: max(2, chunkSize + chunkSize % 2)) + nextOffset = start + pendingStart = start + pending.reserveCapacity(4096) + scratch.reserveCapacity(4096) + } + + public init( + url: URL, + resumingAt offset: Int? = nil, + chunkSize: Int = Self.defaultChunkSize, + ) throws { + try self.init( + source: DSLFileBytes(url: url), + resumingAt: offset, + chunkSize: chunkSize, + ) + } + + /// The next line, or nil at end of file. + public mutating func next() throws -> DSLLine? { + while true { + if cursor == available { + guard try fill() else { return finalLine() } + } + if let terminator = terminatorIndex(from: cursor) { + pending.append(contentsOf: chunk[cursor ..< terminator]) + cursor = terminator + encoding.codeUnitWidth + let line = takeLine() + pendingStart = chunkStart + cursor + return line + } + pending.append(contentsOf: chunk[cursor ..< available]) + cursor = available + } + } + + // MARK: - Reading + + /// Refills the chunk. Returns false at end of file. + /// + /// Loops until the buffer is full so that a source returning a short read + /// cannot leave half a UTF-16 code unit at the end of the chunk — which + /// would be the same misalignment bug arriving by a different route. + private mutating func fill() throws -> Bool { + guard !reachedEnd else { return false } + chunkStart = nextOffset + cursor = 0 + available = 0 + while available < chunk.count { + let read = try source.read(at: nextOffset, into: &chunk, from: available) + guard read > 0 else { + reachedEnd = true + break + } + available += read + nextOffset += read + } + return available > 0 + } + + /// Index within the chunk of the line terminator's first byte, if the + /// current chunk holds one. + /// + /// For UTF-16 the scan steps two bytes at a time from an even index, so + /// it can only ever match a whole code unit. `cursor` is even throughout: + /// `chunkStart` is even, a chunk holds whole code units, and the cursor + /// advances by the code-unit width or to the end of the chunk. + private func terminatorIndex(from start: Int) -> Int? { + switch encoding { + case .utf8: + var index = start + while index < available { + if chunk[index] == 0x0A { + return index + } + index += 1 + } + case .utf16LittleEndian: + var index = start + while index + 1 < available { + if chunk[index] == 0x0A, chunk[index + 1] == 0x00 { + return index + } + index += 2 + } + case .utf16BigEndian: + var index = start + while index + 1 < available { + if chunk[index] == 0x00, chunk[index + 1] == 0x0A { + return index + } + index += 2 + } + } + return nil + } + + private mutating func finalLine() -> DSLLine? { + guard !pending.isEmpty else { return nil } + return takeLine() + } + + private mutating func takeLine() -> DSLLine { + stripCarriageReturn() + let line = DSLLine(text: decodePending(), offset: pendingStart) + pending.removeAll(keepingCapacity: true) + return line + } + + private mutating func stripCarriageReturn() { + let width = encoding.codeUnitWidth + guard pending.count >= width else { return } + let last = pending.count - width + switch encoding { + case .utf8: + if pending[last] == 0x0D { + pending.removeLast(width) + } + case .utf16LittleEndian: + if pending[last] == 0x0D, pending[last + 1] == 0x00 { + pending.removeLast(width) + } + case .utf16BigEndian: + if pending[last] == 0x00, pending[last + 1] == 0x0D { + pending.removeLast(width) + } + } + } + + // The lint rule against a lossy decode is suppressed rather than obeyed, + // and only here. It is right in general — a silent U+FFFD hides a real + // encoding problem — but lossy is the specification at this one point: + // `String(bytes:encoding:)` returns nil for a line holding one bad code + // unit, which would cost the whole card. + // swiftlint:disable optional_data_string_conversion + + /// Decodes the pending bytes. + /// + /// Malformed input costs the character it mangles and nothing more: an + /// unpaired surrogate or a bad byte sequence becomes U+FFFD rather than + /// failing the line, the card around it or the import. That is the same + /// bargain the lexer makes with a tag it cannot read. + private mutating func decodePending() -> String { + guard encoding != .utf8 else { return String(decoding: pending, as: UTF8.self) } + transcodeUTF16ToUTF8() + return String(decoding: scratch, as: UTF8.self) + } + + // swiftlint:enable optional_data_string_conversion + + /// Rewrites the pending UTF-16 bytes into `scratch` as UTF-8. + /// + /// Hand-rolled for a measured reason. Over this dictionary's 3.4 million + /// lines, gathering the code units into a `[UInt16]` costs 0.13 s and + /// handing them to `String(decoding:as: UTF16.self)` costs a further + /// 3.05 s; writing UTF-8 directly, so that `String` is handed its own + /// native storage, costs 0.43 s in total. Byte-identical output, 7× the + /// speed, and it is the difference between a 10-second and a 2-second + /// read of the whole set. + private mutating func transcodeUTF16ToUTF8() { + let low = encoding == .utf16LittleEndian ? 0 : 1 + let high = 1 - low + scratch.removeAll(keepingCapacity: true) + var index = 0 + let end = pending.count + while index + 1 < end { + var scalar = UInt32(pending[index + low]) | (UInt32(pending[index + high]) << 8) + index += 2 + if scalar < 0x80 { + scratch.append(UInt8(scalar)) + continue + } + // A high surrogate followed by a low one is a single character + // above the basic plane. + if scalar >= 0xD800, scalar < 0xDC00, index + 1 < end { + let next = UInt32(pending[index + low]) | (UInt32(pending[index + high]) << 8) + if next >= 0xDC00, next < 0xE000 { + scalar = 0x10000 + ((scalar - 0xD800) << 10) + (next - 0xDC00) + index += 2 + } + } + switch scalar { + case ..<0x800: + scratch.append(UInt8(0xC0 | (scalar >> 6))) + scratch.append(UInt8(0x80 | (scalar & 0x3F))) + case ..<0x10000: + // A surrogate still standing alone here is not a character. + let value = scalar >= 0xD800 && scalar < 0xE000 ? 0xFFFD : scalar + scratch.append(UInt8(0xE0 | (value >> 12))) + scratch.append(UInt8(0x80 | ((value >> 6) & 0x3F))) + scratch.append(UInt8(0x80 | (value & 0x3F))) + default: + scratch.append(UInt8(0xF0 | (scalar >> 18))) + scratch.append(UInt8(0x80 | ((scalar >> 12) & 0x3F))) + scratch.append(UInt8(0x80 | ((scalar >> 6) & 0x3F))) + scratch.append(UInt8(0x80 | (scalar & 0x3F))) + } + } + } + + // MARK: - Encoding detection + + /// Picks an encoding from the start of the file. + /// + /// A byte-order mark settles it. Without one the rule is UTF-8, with one + /// exception: UTF-8 text cannot contain a NUL byte, so NUL bytes all on + /// odd offsets mean UTF-16LE and all on even offsets mean UTF-16BE. The + /// sniff needs one character below U+0100 in the first 64 bytes, which + /// the format guarantees — a DSL file opens with `#NAME`. + /// + /// Defaulting the no-BOM case to UTF-8 rather than refusing is deliberate: + /// hand-made DSL dictionaries are commonly UTF-8 with no mark. + static func detectEncoding(prefix: [UInt8]) -> (encoding: DSLByteEncoding, bomLength: Int) { + if prefix.count >= 3, prefix[0] == 0xEF, prefix[1] == 0xBB, prefix[2] == 0xBF { + return (.utf8, 3) + } + if prefix.count >= 2, prefix[0] == 0xFF, prefix[1] == 0xFE { + return (.utf16LittleEndian, 2) + } + if prefix.count >= 2, prefix[0] == 0xFE, prefix[1] == 0xFF { + return (.utf16BigEndian, 2) + } + + var evenZeros = 0 + var oddZeros = 0 + for (index, byte) in prefix.enumerated() where byte == 0 { + if index.isMultiple(of: 2) { + evenZeros += 1 + } else { + oddZeros += 1 + } + } + if oddZeros > 0, evenZeros == 0 { + return (.utf16LittleEndian, 0) + } + if evenZeros > 0, oddZeros == 0 { + return (.utf16BigEndian, 0) + } + return (.utf8, 0) + } +} diff --git a/Sources/HanReaderCore/Format/DSLCardReader.swift b/Sources/HanReaderCore/Format/DSLCardReader.swift new file mode 100644 index 0000000..31addb0 --- /dev/null +++ b/Sources/HanReaderCore/Format/DSLCardReader.swift @@ -0,0 +1,253 @@ +// HanReader — MIT licensed. See LICENSE. + +import Foundation + +/// What a DSL file's `#` directives say about itself. +/// +/// Read rather than assumed. The predecessor prototype hardcoded the three +/// BKRS filenames and took the dictionary's name from a constant in the source +/// — so importing any other DSL dictionary mislabelled it, and importing a +/// renamed copy of this one failed outright. Everything here comes out of the +/// file. +public struct DSLHeader: Hashable, Sendable { + /// `#NAME` — what the dictionary calls itself: `大БКРС - 250920 ( 1 / 3 )`. + public var name: String? + /// `#INDEX_LANGUAGE` — the language of the headwords. + public var indexLanguage: String? + /// `#CONTENTS_LANGUAGE` — the language of the definitions. This is what + /// decides whether a Russian-locale reader sees this dictionary first. + public var contentsLanguage: String? + /// `#INCLUDE` targets, as written: file names relative to this file. + public var includes: [String] = [] + /// Every other directive, keyed by name without the `#`. + /// + /// Kept rather than discarded, because a directive this version has not + /// heard of is still something the dictionary chose to state. + public var other: [String: String] = [:] + + public init() {} + + /// Reads a directive line. + /// + /// The name runs to the first space; the value is the rest, with one layer + /// of surrounding quotes removed. A directive with no value records the + /// empty string, which is distinguishable from absent. + mutating func consume(directive line: String) { + let rest = line.dropFirst() + let split = rest.firstIndex(of: " ") + let directive = String(split.map { rest[rest.startIndex ..< $0] } ?? rest) + let rawValue = split.map { String(rest[rest.index(after: $0)...]) } ?? "" + let value = Self.unquote(rawValue.trimmingCharacters(in: .whitespaces)) + + switch directive { + case "NAME": name = value + case "INDEX_LANGUAGE": indexLanguage = value + case "CONTENTS_LANGUAGE": contentsLanguage = value + case "INCLUDE": includes.append(value) + default: other[directive] = value + } + } + + private static func unquote(_ value: String) -> String { + guard value.count >= 2, value.hasPrefix("\""), value.hasSuffix("\"") else { return value } + return String(value.dropFirst().dropLast()) + } +} + +/// One card, and where it starts. +public struct DSLCardRecord: Hashable, Sendable { + public let card: DSLCard + /// Byte offset of the card's first headword line. + /// + /// The resume point. It is recorded at a card boundary rather than at a + /// line or a batch boundary because that is the only place the importer can + /// restart without either losing a card or writing one twice. + public let offset: Int + + public init(card: DSLCard, offset: Int) { + self.card = card + self.offset = offset + } +} + +/// Groups a DSL file's lines into cards. +/// +/// The structure is positional, and that is the whole of it: +/// +/// ``` +/// 爱 ← unindented: a headword +/// ài ← first indented line: the reading +/// [m1][b]I[/b] … ← the rest: the article +/// ``` +/// +/// ### Three line kinds, and the order the tests go in +/// +/// An indented line is article text; an unindented line starting with `#` is a +/// directive; anything else unindented is a headword. Indentation is tested +/// first, so an indented line beginning with `#` stays article text rather +/// than being read as a directive — the format escapes a literal leading `#`, +/// but only on an unindented line, where it would be ambiguous. +/// +/// ### Consecutive headwords are one card +/// +/// The DSL spec lets several spellings share one article, written as +/// consecutive unindented lines. BKRS never does — all 3,434,224 cards have +/// exactly one headword — but accepting it costs two lines and refusing it +/// would be a parser bug waiting for the next dictionary. +/// +/// ### A card's article is not always one line +/// +/// This set writes each card's entire sense run on a single physical line, +/// which is what made the predecessor's "one line is one sense" reading +/// produce exactly one sense per card. Exactly one card in the set — +/// 乐芙兰 — breaks a line *inside* an open `[m1]`, and most other DSL +/// dictionaries write one sense per line. So the article is a list of lines +/// and the sense structure is read from the markup, never from the line count. +public struct DSLCardReader { + /// What the file says about itself, filled in as directives are read. + /// Complete once the first card has been returned. + public private(set) var header = DSLHeader() + public private(set) var diagnostics: [DSLDiagnostic] = [] + /// Headwords found at the end of the file with no article under them. + /// + /// Empty for a card-aligned file. A file split on a byte boundary leaves + /// its last headword here, and the matching article arrives as + /// `bodyWithoutHeadword` diagnostics at the start of the next file; a + /// multi-file driver repairs the seam by passing these in as + /// `leadingHeadwords`. + public private(set) var trailingHeadwords: [String] = [] + + private var lines: DSLByteReader + private var headwords: [String] + private var body: [String] = [] + private var cardOffset = 0 + + public init( + source: any DSLByteSource, + resumingAt offset: Int? = nil, + leadingHeadwords: [String] = [], + chunkSize: Int = DSLByteReader.defaultChunkSize, + ) throws { + lines = try DSLByteReader( + source: source, + resumingAt: offset, + chunkSize: chunkSize, + ) + headwords = leadingHeadwords + cardOffset = offset ?? 0 + } + + /// How the file spells its characters. + public var encoding: DSLByteEncoding { + lines.encoding + } + + /// The next card, or nil at end of file. + public mutating func next() throws -> DSLCardRecord? { + while let line = try lines.next() { + let text = line.text + if text.isEmpty { + continue + } + + if let article = Self.stripIndent(text) { + guard !headwords.isEmpty else { + diagnostics.append(DSLDiagnostic( + kind: .bodyWithoutHeadword, + text: text, + offset: line.offset, + )) + continue + } + body.append(article) + continue + } + + if text.hasPrefix("#") { + header.consume(directive: text) + continue + } + + // A headword. If an article has already been collected, the card + // above this line is complete. + guard body.isEmpty else { + let record = takeCard() + headwords = [text] + cardOffset = line.offset + return record + } + if headwords.isEmpty { + cardOffset = line.offset + } + headwords.append(text) + } + return finalCard() + } + + // MARK: - Cards + + private mutating func takeCard() -> DSLCardRecord { + let record = DSLCardRecord( + card: DSLCard( + headwords: headwords, + pinyin: body.first ?? "", + body: Array(body.dropFirst()), + ), + offset: cardOffset, + ) + headwords.removeAll(keepingCapacity: true) + body.removeAll(keepingCapacity: true) + return record + } + + /// At end of file: a complete card, or a dangling headword reported and + /// handed to `trailingHeadwords`. + private mutating func finalCard() -> DSLCardRecord? { + guard !headwords.isEmpty else { return nil } + guard !body.isEmpty else { + for headword in headwords { + diagnostics.append(DSLDiagnostic( + kind: .headwordWithoutBody, + text: headword, + offset: cardOffset, + )) + } + trailingHeadwords = headwords + headwords.removeAll() + return nil + } + return takeCard() + } + + /// Returns the line without its indent, or nil if it was not indented. + /// + /// All leading horizontal whitespace goes, not just the first character. + /// DSL carries nesting in `[mN]`, never in indent depth, so a line indented + /// two spaces means exactly what one indented by one means — and leaving + /// the extra space in would put it inside the gloss. + static func stripIndent(_ line: String) -> String? { + guard let first = line.unicodeScalars.first, first == " " || first == "\t" else { + return nil + } + return String(line.drop { $0 == " " || $0 == "\t" }) + } +} + +extension DSLHeader { + /// Reads just the directive block at the top of a file. + /// + /// Stops at the first line that is neither blank nor a directive, so + /// discovering a file set costs one chunk rather than a pass over 116 MB. + public static func read(from source: any DSLByteSource) throws -> DSLHeader { + var reader = try DSLByteReader(source: source) + var header = DSLHeader() + while let line = try reader.next() { + if line.text.isEmpty { + continue + } + guard line.text.hasPrefix("#") else { break } + header.consume(directive: line.text) + } + return header + } +} diff --git a/Sources/HanReaderCore/Format/DSLDiagnostic.swift b/Sources/HanReaderCore/Format/DSLDiagnostic.swift new file mode 100644 index 0000000..9419c9a --- /dev/null +++ b/Sources/HanReaderCore/Format/DSLDiagnostic.swift @@ -0,0 +1,46 @@ +// HanReader — MIT licensed. See LICENSE. + +import Foundation + +/// Something a DSL file said that could not be acted on, reported rather than +/// hidden. +/// +/// The rule the whole importer follows is that nothing is ever deleted. A span +/// the lexer cannot identify is emitted as literal text *and* reported; a line +/// the card reader cannot place is reported *and* kept. A dictionary that +/// silently drops what it does not understand is worse than one that shows a +/// stray bracket, because the reader cannot tell "the dictionary says nothing +/// here" from "the importer ate it". +public struct DSLDiagnostic: Hashable, Sendable { + public enum Kind: Hashable, Sendable { + /// A bracketed span whose name is not in `DSLTagKind`. + case unknownTag(String) + /// A `[` with no matching `]` before the end of the line. + case unterminatedTag + /// An indented line with no headword above it. This is what a file + /// split on a byte boundary rather than a card boundary looks like + /// from the second half. + case bodyWithoutHeadword + /// A headword with no article under it — the other half of the same + /// split. + case headwordWithoutBody + /// An `#INCLUDE` naming a file that is not beside the one including + /// it. + case missingInclude(String) + } + + public let kind: Kind + /// The text as it appeared, which is also what was emitted or kept. + public let text: String + /// Byte offset in the file, where the reporting layer knows one. + /// + /// The lexer works a line at a time and does not, so this is nil for its + /// diagnostics and set for everything found at the card level. + public let offset: Int? + + public init(kind: Kind, text: String, offset: Int? = nil) { + self.kind = kind + self.text = text + self.offset = offset + } +} diff --git a/Sources/HanReaderCore/Format/DSLFileSet.swift b/Sources/HanReaderCore/Format/DSLFileSet.swift new file mode 100644 index 0000000..18b9484 --- /dev/null +++ b/Sources/HanReaderCore/Format/DSLFileSet.swift @@ -0,0 +1,154 @@ +// HanReader — MIT licensed. See LICENSE. + +public import Foundation + +/// The files one DSL dictionary is spread across. +/// +/// The BKRS set is three files, and `dabkrs_1.dsl` names the other two in its +/// own header: +/// +/// ``` +/// #NAME "大БКРС - 250920 ( 1 / 3 )" +/// #INCLUDE "dabkrs_2.dsl" +/// #INCLUDE "dabkrs_3.dsl" +/// ``` +/// +/// So the set is discovered by reading the file the reader picked, never by +/// knowing those names. The predecessor prototype hardcoded all three, which +/// means it imports this one dictionary and no other, and stops working if the +/// user renames a file. +/// +/// The three files are include-linked volumes rather than byte chunks: each +/// begins and ends on a complete card. They are parsed one at a time and never +/// concatenated, so a failure in the third volume leaves the first two +/// imported. +public struct DSLFileSet: Hashable, Sendable { + /// How the set was found, which the import screen reports so that + /// "importing 3 files when I picked 1" is explained rather than + /// surprising. + public enum Discovery: Hashable, Sendable { + /// The picked file named the others in `#INCLUDE` directives. + case includeDirectives + /// The picked file named nothing, but numbered siblings sit beside it. + case numberedSiblings + /// One file, standing alone. + case single + } + + /// The files to parse, in order, the one holding the directives first. + public let files: [URL] + /// The master file's header. + public let header: DSLHeader + public let discovery: Discovery + public let diagnostics: [DSLDiagnostic] + + public init( + files: [URL], + header: DSLHeader, + discovery: Discovery, + diagnostics: [DSLDiagnostic] = [], + ) { + self.files = files + self.header = header + self.discovery = discovery + self.diagnostics = diagnostics + } + + /// Follows at most this many `#INCLUDE` hops. A visited set already stops + /// a cycle; this stops a pathological file from being a denial of service. + static let fileLimit = 64 + + /// Works out which files belong with the one the reader picked. + /// + /// `#INCLUDE` first, and it is authoritative: a master file that names two + /// volumes gets exactly those two, in that order, and no directory scan + /// happens at all. An included file may include further files, which is + /// followed with a visited set so a cycle cannot loop. + /// + /// Only when there are no directives does it look at the directory, and + /// then only for files whose names differ from the picked one by a trailing + /// number. If that turns up a lower-numbered sibling — the reader picked + /// volume 2 of 3 — discovery restarts from the lowest, because that is + /// where the directives live. The restart terminates: the lowest file of a + /// numbered run either has directives or is its own lowest sibling. + public static func discover( + from url: URL, + fileManager: FileManager = .default, + ) throws + -> Self + { + let header = try DSLHeader.read(from: DSLFileBytes(url: url)) + + if !header.includes.isEmpty { + var files = [url.standardizedFileURL] + var visited: Set = [url.standardizedFileURL] + var diagnostics: [DSLDiagnostic] = [] + var queue = header.includes.map { (parent: url, name: $0) } + + while !queue.isEmpty, files.count < fileLimit { + let (parent, name) = queue.removeFirst() + let target = parent.deletingLastPathComponent() + .appendingPathComponent(name) + .standardizedFileURL + guard fileManager.fileExists(atPath: target.path) else { + diagnostics.append(DSLDiagnostic(kind: .missingInclude(name), text: name)) + continue + } + guard visited.insert(target).inserted else { continue } + files.append(target) + let nested = try DSLHeader.read(from: DSLFileBytes(url: target)) + queue += nested.includes.map { (parent: target, name: $0) } + } + return Self( + files: files, + header: header, + discovery: .includeDirectives, + diagnostics: diagnostics, + ) + } + + let siblings = numberedSiblings(of: url, fileManager: fileManager) + guard siblings.count > 1 else { + return Self(files: [url.standardizedFileURL], header: header, discovery: .single) + } + if siblings[0] != url.standardizedFileURL { + return try discover(from: siblings[0], fileManager: fileManager) + } + return Self(files: siblings, header: header, discovery: .numberedSiblings) + } + + /// Files beside `url` whose names differ only in a trailing number. + /// + /// `dabkrs_1.dsl` → `dabkrs_2.dsl`, `dabkrs_3.dsl`. The run is contiguous + /// from 1: a gap ends it, because a set missing its second volume is a set + /// of one, not a set of two with a hole. Zero padding is preserved, so + /// `vol_01` looks for `vol_02` and not `vol_2`. + static func numberedSiblings(of url: URL, fileManager: FileManager) -> [URL] { + let stem = url.deletingPathExtension().lastPathComponent + let digits = stem.reversed().prefix { $0.isASCII && $0.isNumber } + guard !digits.isEmpty else { return [url.standardizedFileURL] } + + let prefix = String(stem.dropLast(digits.count)) + let width = digits.count + let directory = url.deletingLastPathComponent() + let suffix = url.pathExtension.isEmpty ? "" : ".\(url.pathExtension)" + + var found: [URL] = [] + var number = 1 + while found.count < fileLimit { + let padded = String(number).count >= width + ? String(number) + : String(repeating: "0", count: width - String(number).count) + String(number) + let candidate = directory + .appendingPathComponent(prefix + padded + suffix) + .standardizedFileURL + guard fileManager.fileExists(atPath: candidate.path) else { break } + found.append(candidate) + number += 1 + } + // The picked file must be in the run; otherwise the names matched by + // accident and it is safer to import only what was asked for. + guard found.contains(url.standardizedFileURL) else { return [url.standardizedFileURL] } + return found + } +} diff --git a/Sources/HanReaderCore/Format/DSLLexer.swift b/Sources/HanReaderCore/Format/DSLLexer.swift index d67fee0..aae8bad 100644 --- a/Sources/HanReaderCore/Format/DSLLexer.swift +++ b/Sources/HanReaderCore/Format/DSLLexer.swift @@ -83,26 +83,6 @@ public enum DSLToken: Hashable, Sendable { case close(DSLTagKind) } -/// Something the lexer could not make sense of, reported rather than hidden. -public struct DSLDiagnostic: Hashable, Sendable { - public enum Kind: Hashable, Sendable { - /// A bracketed span whose name is not in `DSLTagKind`. - case unknownTag(String) - /// A `[` with no matching `]` before the end of the line. - case unterminatedTag - } - - public let kind: Kind - /// The text as it appeared, which is also what was emitted. Nothing is - /// dropped on the way to a diagnostic. - public let text: String - - public init(kind: Kind, text: String) { - self.kind = kind - self.text = text - } -} - /// Splits a line of DSL body text into text and markup. /// /// ### Escapes are handled before tags, and that ordering is the whole thing diff --git a/Tests/HanReaderCoreTests/DSLByteReaderTests.swift b/Tests/HanReaderCoreTests/DSLByteReaderTests.swift new file mode 100644 index 0000000..e225ff5 --- /dev/null +++ b/Tests/HanReaderCoreTests/DSLByteReaderTests.swift @@ -0,0 +1,266 @@ +// HanReader — MIT licensed. See LICENSE. + +import Foundation +import Testing +@testable import HanReaderCore + +// MARK: - Helpers + +/// Encodes text the way ABBYY writes a DSL file. +private func utf16(_ text: String, bigEndian: Bool = false, bom: Bool = true) -> [UInt8] { + var bytes: [UInt8] = [] + if bom { + bytes += bigEndian ? [0xFE, 0xFF] : [0xFF, 0xFE] + } + for unit in Array(text.utf16) { + let low = UInt8(unit & 0xFF) + let high = UInt8(unit >> 8) + bytes += bigEndian ? [high, low] : [low, high] + } + return bytes +} + +private func lines( + _ bytes: [UInt8], + chunkSize: Int = DSLByteReader.defaultChunkSize, + resumingAt offset: Int? = nil, +) throws + -> [DSLLine] +{ + var reader = try DSLByteReader( + source: DSLMemoryBytes(bytes), + resumingAt: offset, + chunkSize: chunkSize, + ) + var result: [DSLLine] = [] + while let line = try reader.next() { + result.append(line) + } + return result +} + +/// The real shape of a BKRS file: two CRLF header lines, a blank line, then +/// LF-terminated cards, ending with a line feed. +private let header = "#NAME \"大БКРС\"\r\n#INDEX_LANGUAGE \"Chinese\"\r\n" + + "#CONTENTS_LANGUAGE \"Russian\"\n#INCLUDE \"dabkrs_2.dsl\"\n\n" +private let cards = "爱\n ài\n [m1]любить[/m]\n了\n le, liǎo\n [m1]частица[/m]\n" + +@Suite("DSL byte reader") +struct DSLByteReaderTests { + // MARK: - The alignment trap + + /// In UTF-16LE a line feed is the code unit `0x000A` — the bytes `0A 00` + /// **at an even offset**. `ਅ` is U+0A05 and `昀` is U+6600, so the two + /// together are `05 0A 00 66`: the pair appears at an odd offset, inside + /// a character. A scanner that searches the raw bytes breaks the line + /// there and splits one character in half. + @Test("A misaligned line-feed byte pair is not a line break") + func misalignedPairIsNotABreak() throws { + let bytes = utf16("x\u{0A05}\u{6600}y\nz\n") + + // The trap is actually present in this input, so the assertion below + // is not vacuous. + let pair = (0 ..< (bytes.count - 1)).first { bytes[$0] == 0x0A && bytes[$0 + 1] == 0x00 } + let trap = try #require(pair) + #expect(!trap.isMultiple(of: 2)) + + let result = try lines(bytes) + #expect(result.map(\.text) == ["x\u{0A05}\u{6600}y", "z"]) + } + + /// Reading the same bytes two at a time puts the trap at the very end of + /// a chunk, which is where a scanner that forgets to realign after a + /// refill goes wrong. + @Test("The alignment holds across chunk boundaries", arguments: [2, 4, 6, 8, 16]) + func misalignedPairAcrossChunks(chunkSize: Int) throws { + let result = try lines(utf16("x\u{0A05}\u{6600}y\nz\n"), chunkSize: chunkSize) + #expect(result.map(\.text) == ["x\u{0A05}\u{6600}y", "z"]) + } + + // MARK: - Line endings + + /// Each BKRS file mixes exactly two CRLF lines in among three million LF + /// ones, so neither terminator is the one to split on. + @Test("Both line endings are handled in the same file") + func mixedLineEndings() throws { + let result = try lines(utf16(header + cards)) + #expect(result.map(\.text) == [ + "#NAME \"大БКРС\"", + "#INDEX_LANGUAGE \"Chinese\"", + "#CONTENTS_LANGUAGE \"Russian\"", + "#INCLUDE \"dabkrs_2.dsl\"", + "", + "爱", " ài", " [m1]любить[/m]", + "了", " le, liǎo", " [m1]частица[/m]", + ]) + } + + /// All three files end with a line feed. Splitting the text yields an + /// empty final element, and at the card level an unindented empty line is + /// indistinguishable from a headword. + @Test("A trailing newline does not produce an empty final line") + func trailingNewline() throws { + #expect(try lines(utf16("a\nb\n")).map(\.text) == ["a", "b"]) + } + + @Test("A file with no trailing newline still yields its last line") + func noTrailingNewline() throws { + #expect(try lines(utf16("a\nb")).map(\.text) == ["a", "b"]) + } + + /// A blank line inside the file is a line, not nothing — the header is + /// separated from the body by one, and the card reader needs to see it. + @Test("A blank line is reported") + func blankLine() throws { + #expect(try lines(utf16("a\n\nb\n")).map(\.text) == ["a", "", "b"]) + } + + @Test("An empty file yields no lines") + func emptyFile() throws { + #expect(try lines([]).isEmpty) + } + + @Test("A file that is nothing but a byte-order mark yields no lines") + func bomOnly() throws { + #expect(try lines(utf16("")).isEmpty) + } + + // MARK: - Offsets + + /// Byte offsets, because that is what a resumed import seeks to. + @Test("Each line carries the byte offset it started at") + func offsets() throws { + // BOM (2) + "a\n" (4) + "bb\n" (6). + let result = try lines(utf16("a\nbb\nc\n")) + #expect(result.map(\.offset) == [2, 6, 12]) + } + + @Test("Offsets are unaffected by the chunk size", arguments: [2, 4, 6, 10, 4096]) + func offsetsAcrossChunks(chunkSize: Int) throws { + let result = try lines(utf16(header + cards), chunkSize: chunkSize) + #expect(try result == lines(utf16(header + cards))) + } + + /// The resume contract: an offset this reader reported, handed back to a + /// new reader, continues from exactly that line. + @Test("Resuming at a reported offset continues from that line") + func resume() throws { + let bytes = utf16(header + cards) + let all = try lines(bytes) + let start = try #require(all.indices.last.map { $0 - 3 }) + let resumed = try lines(bytes, resumingAt: all[start].offset) + #expect(resumed == Array(all[start...])) + } + + /// A resume offset is taken literally rather than being re-sniffed for a + /// byte-order mark, so the first character after a checkpoint is not + /// mistaken for one. + @Test("A resume offset is not re-interpreted as a byte-order mark") + func resumeDoesNotSkipABOM() throws { + let bytes = utf16("\u{FEFF}a\nb\n") + let result = try lines(bytes, resumingAt: 2) + #expect(result.map(\.text) == ["\u{FEFF}a", "b"]) + } + + // MARK: - Malformed files + + /// The last character is cut in half, so every offset past it would be a + /// guess. This is the `kill -9`-during-a-copy case. + @Test("An odd-length UTF-16 file is refused") + func oddByteLength() throws { + var bytes = utf16("a\nb\n") + bytes.removeLast() + #expect(throws: DSLByteError.oddByteLength(byteCount: bytes.count)) { + _ = try lines(bytes) + } + } + + @Test("A resume offset inside a code unit is refused") + func misalignedOffset() throws { + #expect(throws: DSLByteError.misalignedOffset(5)) { + _ = try lines(utf16("a\nb\n"), resumingAt: 5) + } + } + + @Test("A resume offset past the end is refused") + func offsetOutOfRange() throws { + let bytes = utf16("a\n") + #expect(throws: DSLByteError.offsetOutOfRange(offset: 100, byteCount: bytes.count)) { + _ = try lines(bytes, resumingAt: 100) + } + } + + /// Nothing is dropped and nothing throws: one mangled character costs + /// that character, not the card around it. + @Test("An unpaired surrogate becomes a replacement character") + func unpairedSurrogate() throws { + var bytes = utf16("a") + bytes += [0x00, 0xD8] // U+D800, a high surrogate with no low one + bytes += utf16("b\n", bom: false) + #expect(try lines(bytes).map(\.text) == ["a\u{FFFD}b"]) + } + + // MARK: - Encoding detection + + @Test("A UTF-16LE byte-order mark is detected and skipped") + func littleEndianBOM() throws { + var reader = try DSLByteReader(source: DSLMemoryBytes(utf16("爱\n"))) + #expect(reader.encoding == .utf16LittleEndian) + #expect(try reader.next()?.text == "爱") + } + + @Test("A UTF-16BE byte-order mark is detected and skipped") + func bigEndianBOM() throws { + let bytes = utf16("#NAME\n爱\n", bigEndian: true) + var reader = try DSLByteReader(source: DSLMemoryBytes(bytes)) + #expect(reader.encoding == .utf16BigEndian) + #expect(try reader.next()?.text == "#NAME") + #expect(try reader.next()?.text == "爱") + } + + @Test("A UTF-8 byte-order mark is detected and skipped") + func utf8BOM() throws { + var bytes: [UInt8] = [0xEF, 0xBB, 0xBF] + bytes += Array("爱\n ài\n".utf8) + var reader = try DSLByteReader(source: DSLMemoryBytes(bytes)) + #expect(reader.encoding == .utf8) + #expect(try reader.next()?.text == "爱") + #expect(try reader.next()?.text == " ài") + } + + /// Hand-made DSL dictionaries are commonly UTF-8 with no mark, so that is + /// the default rather than a refusal. + @Test("A file with no mark is read as UTF-8") + func bomlessUTF8() throws { + let bytes = Array("爱\n ài\n [m1]любить[/m]\n".utf8) + var reader = try DSLByteReader(source: DSLMemoryBytes(bytes)) + #expect(reader.encoding == .utf8) + #expect(try reader.next()?.text == "爱") + } + + /// UTF-8 text cannot contain a NUL byte, so NUL bytes all at odd offsets + /// can only be the high halves of UTF-16LE code units. Without this a + /// mark-less UTF-16 file decodes to mojibake with no error at all. + @Test("A mark-less UTF-16 file is detected from its NUL bytes", arguments: [false, true]) + func bomlessUTF16(bigEndian: Bool) throws { + let bytes = utf16("#NAME \"x\"\n爱\n", bigEndian: bigEndian, bom: false) + var reader = try DSLByteReader(source: DSLMemoryBytes(bytes)) + #expect(reader.encoding == (bigEndian ? .utf16BigEndian : .utf16LittleEndian)) + #expect(try reader.next()?.text == "#NAME \"x\"") + #expect(try reader.next()?.text == "爱") + } + + /// Chinese characters have no zero byte in either half, so a mark-less + /// UTF-16 file that opens with one cannot be told from UTF-8. The format + /// rules it out — a DSL file opens with `#NAME` — and the sniff is + /// documented as needing that. + @Test("Detection reports where it cannot tell") + func detectionLimits() { + #expect(DSLByteReader.detectEncoding(prefix: []) == (.utf8, 0)) + #expect(DSLByteReader.detectEncoding(prefix: Array("爱 ài".utf8)) == (.utf8, 0)) + // 爱 in UTF-16LE is 1B 72 — no zero byte to sniff. + #expect(DSLByteReader.detectEncoding(prefix: [0x1B, 0x72]) == (.utf8, 0)) + // A zero byte in each half is not a byte order. + #expect(DSLByteReader.detectEncoding(prefix: [0x00, 0x00]) == (.utf8, 0)) + } +} diff --git a/Tests/HanReaderCoreTests/DSLCardReaderTests.swift b/Tests/HanReaderCoreTests/DSLCardReaderTests.swift new file mode 100644 index 0000000..7c974b2 --- /dev/null +++ b/Tests/HanReaderCoreTests/DSLCardReaderTests.swift @@ -0,0 +1,394 @@ +// HanReader — MIT licensed. See LICENSE. + +import Foundation +import Testing +@testable import HanReaderCore + +// MARK: - Helpers + +private func utf16(_ text: String) -> [UInt8] { + var bytes: [UInt8] = [0xFF, 0xFE] + for unit in Array(text.utf16) { + bytes += [UInt8(unit & 0xFF), UInt8(unit >> 8)] + } + return bytes +} + +private func read( + _ text: String, + leadingHeadwords: [String] = [], +) throws + -> (records: [DSLCardRecord], reader: DSLCardReader) +{ + var reader = try DSLCardReader( + source: DSLMemoryBytes(utf16(text)), + leadingHeadwords: leadingHeadwords, + ) + var records: [DSLCardRecord] = [] + while let record = try reader.next() { + records.append(record) + } + return (records, reader) +} + +private func cards(_ text: String) throws -> [DSLCard] { + try read(text).records.map(\.card) +} + +/// A file shaped like a real one: CRLF header lines, a blank separator, then +/// three-line cards. +private let file = """ +#NAME "大БКРС - 250920 ( 1 / 3 )"\r +#INDEX_LANGUAGE "Chinese"\r +#CONTENTS_LANGUAGE "Russian" +#INCLUDE "dabkrs_2.dsl" +#INCLUDE "dabkrs_3.dsl" + +爱 + ài + [m1]любить[/m] +了 + le, liǎo + [m1]частица[/m] + +""" + +@Suite("DSL card reader") +struct DSLCardReaderTests { + // MARK: - Card shape + + @Test("A file becomes its cards") + func wholeFile() throws { + let result = try cards(file) + #expect(result.count == 2) + #expect(result.map(\.headwords) == [["爱"], ["了"]]) + #expect(result.map(\.pinyin) == ["ài", "le, liǎo"]) + #expect(result.map(\.body) == [["[m1]любить[/m]"], ["[m1]частица[/m]"]]) + } + + /// The reading is the first indented line and the article is the rest, so + /// a card with no article still has its reading rather than shifting the + /// article into it. + @Test("A card with only a reading line keeps it as the reading") + func readingWithoutArticle() throws { + let result = try cards("爱\n ài\n") + #expect(result.map(\.pinyin) == ["ài"]) + #expect(result[0].body.isEmpty) + } + + /// Exactly one card in the set — 乐芙兰 — breaks a line inside an open + /// `[m1]`, and most other DSL dictionaries write one sense per line. The + /// article is a list of lines; the structure comes from the markup. + @Test("An article spread over several lines keeps all of them") + func multiLineArticle() throws { + let result = try cards("乐芙兰\n _\n [m1]Ле Блан\n (чемпион из Лиги Легенд)[/m]\n") + #expect(result[0].body == ["[m1]Ле Блан", "(чемпион из Лиги Легенд)[/m]"]) + } + + /// And the sense that article belongs to comes out as one gloss with the + /// line break rendered as a space, not as `Ле Блан(чемпион…)`. + @Test("A sense broken across lines reads as one sentence") + func multiLineSense() throws { + let card = try #require( + try cards("乐芙兰\n _\n [m1]Ле Блан\n (чемпион из Лиги Легенд)[/m]\n").first, + ) + let built = DSLCardBuilder(syllableBases: []).entries(from: card) + let senses = try #require(built.entries.first?.senses) + #expect(senses.count == 1) + #expect(senses[0].gloss.text == "Ле Блан (чемпион из Лиги Легенд)") + } + + /// The DSL spec lets several spellings share one article. BKRS never does + /// — all 3,434,224 cards have one headword — but refusing it would be a + /// parser bug waiting for the next dictionary. + @Test("Consecutive unindented lines are one card") + func sharedArticle() throws { + let result = try cards("爱\n愛\n ài\n [m1]любить[/m]\n") + #expect(result.count == 1) + #expect(result[0].headwords == ["爱", "愛"]) + } + + @Test("A blank line between cards is not a headword") + func blankLineBetweenCards() throws { + let result = try cards("爱\n ài\n [m1]x[/m]\n\n了\n le\n [m1]y[/m]\n") + #expect(result.map(\.headwords) == [["爱"], ["了"]]) + } + + @Test("A file with no trailing newline still yields its last card") + func noTrailingNewline() throws { + #expect(try cards("爱\n ài\n [m1]любить[/m]").count == 1) + } + + @Test("A file of nothing but a header yields no cards") + func headerOnly() throws { + let result = try read("#NAME \"x\"\n#INDEX_LANGUAGE \"Chinese\"\n\n") + #expect(result.records.isEmpty) + #expect(result.reader.header.name == "x") + } + + // MARK: - Line kinds + + /// Indentation is tested before the `#`, so an indented line beginning + /// with `#` is article text. The format escapes a literal leading `#` + /// only where it would be ambiguous, which is on an unindented line. + @Test("An indented line starting with a hash is article text") + func indentedHash() throws { + let result = try cards("爱\n ài\n #1 значение\n") + #expect(result[0].body == ["#1 значение"]) + } + + /// Nesting is carried by `[mN]`, never by indent depth, so a line + /// indented by two spaces means what one indented by one means — and + /// leaving the extra space in would put it inside the gloss. + @Test("Indent depth carries no meaning", arguments: [" ", " ", "\t", " \t "]) + func indentDepth(indent: String) throws { + let result = try cards("爱\n\(indent)ài\n\(indent)[m1]любить[/m]\n") + #expect(result[0].pinyin == "ài") + #expect(result[0].body == ["[m1]любить[/m]"]) + } + + @Test("stripIndent reports an unindented line as unindented") + func stripIndentOnUnindented() { + #expect(DSLCardReader.stripIndent("爱") == nil) + #expect(DSLCardReader.stripIndent(" ài") == "ài") + #expect(DSLCardReader.stripIndent("") == nil) + } + + // MARK: - Offsets + + /// Byte offset of the headword line, because a card boundary is the only + /// place an interrupted import can restart without losing a card or + /// writing one twice. + @Test("Each card records the byte offset of its headword line") + func offsets() throws { + let result = try read(file).records + let bytes = utf16(file) + for record in result { + let headword = record.card.headwords[0] + let expected = Array(headword.utf16).flatMap { [UInt8($0 & 0xFF), UInt8($0 >> 8)] } + #expect(Array(bytes[record.offset ..< (record.offset + expected.count)]) == expected) + } + } + + /// The resume contract, end to end: an offset this reader reported, handed + /// back to a new reader, yields exactly the cards from there on. + @Test("Resuming at a card's offset continues from that card") + func resume() throws { + let all = try read(file).records + let second = try #require(all.last) + var reader = try DSLCardReader( + source: DSLMemoryBytes(utf16(file)), + resumingAt: second.offset, + ) + var resumed: [DSLCardRecord] = [] + while let record = try reader.next() { + resumed.append(record) + } + #expect(resumed == [second]) + } + + // MARK: - Header + + @Test("Directives are read from the file, never assumed") + func header() throws { + let reader = try read(file).reader + #expect(reader.header.name == "大БКРС - 250920 ( 1 / 3 )") + #expect(reader.header.indexLanguage == "Chinese") + #expect(reader.header.contentsLanguage == "Russian") + #expect(reader.header.includes == ["dabkrs_2.dsl", "dabkrs_3.dsl"]) + } + + /// A directive this version has not heard of is still something the + /// dictionary chose to state. + @Test("An unrecognised directive is kept rather than dropped") + func unknownDirective() throws { + let reader = try read("#ICON_FILE \"x.bmp\"\n#SOUND_DICTIONARY\n\n").reader + #expect(reader.header.other == ["ICON_FILE": "x.bmp", "SOUND_DICTIONARY": ""]) + } + + @Test("Reading only the header stops at the first card") + func headerOnlyRead() throws { + let header = try DSLHeader.read(from: DSLMemoryBytes(utf16(file))) + #expect(header.includes.count == 2) + #expect(header.other.isEmpty) + } + + // MARK: - A seam split on the wrong boundary + + /// The BKRS files are include-linked volumes, each beginning and ending on + /// a complete card, so none of this fires for them. It is insurance + /// against a set someone split with `split(1)`. + @Test("An article with no headword above it is reported, not dropped") + func bodyWithoutHeadword() throws { + let result = try read(" ài\n [m1]любить[/m]\n了\n le\n [m1]x[/m]\n") + #expect(result.records.count == 1) + #expect(result.reader.diagnostics.count == 2) + #expect(result.reader.diagnostics.allSatisfy { $0.kind == .bodyWithoutHeadword }) + #expect(result.reader.diagnostics[0].text == " ài") + #expect(result.reader.diagnostics[0].offset == 2) + } + + @Test("A headword with no article is reported and handed on") + func headwordWithoutBody() throws { + let result = try read("爱\n ài\n [m1]x[/m]\n了\n") + #expect(result.records.count == 1) + #expect(result.reader.trailingHeadwords == ["了"]) + #expect(result.reader.diagnostics.map(\.kind) == [.headwordWithoutBody]) + } + + /// The repair: the headword left over from the previous file is passed in, + /// and the article that opens this one completes its card. + @Test("A headword carried in from the previous file completes its card") + func leadingHeadwords() throws { + let result = try read(" le\n [m1]частица[/m]\n", leadingHeadwords: ["了"]) + #expect(result.records.count == 1) + #expect(result.records[0].card.headwords == ["了"]) + #expect(result.records[0].card.pinyin == "le") + #expect(result.reader.diagnostics.isEmpty) + } +} + +// MARK: - File-set discovery + +@Suite("DSL file-set discovery") +struct DSLFileSetTests { + /// Writes a throwaway directory of DSL files. + private func directory( + _ files: [String: String], + _ body: (URL) throws -> Void, + ) throws { + let root = URL(fileURLWithPath: NSTemporaryDirectory()) + .appendingPathComponent("hanreader-dsl-\(UUID().uuidString)") + try FileManager.default.createDirectory(at: root, withIntermediateDirectories: true) + defer { try? FileManager.default.removeItem(at: root) } + for (name, text) in files { + try Data(utf16(text)).write(to: root.appendingPathComponent(name)) + } + try body(root) + } + + /// The master file names its volumes, so the set is read out of the file + /// rather than out of a hardcoded list of names — which is what makes the + /// importer work on a second dictionary, and on a renamed copy of this + /// one. + @Test("The set comes from the master file's include directives") + func includeDirectives() throws { + try directory([ + "a.dsl": "#NAME \"A\"\n#INCLUDE \"b.dsl\"\n#INCLUDE \"c.dsl\"\n\n爱\n ài\n [m1]x[/m]\n", + "b.dsl": "#NAME \"B\"\n\n了\n le\n [m1]y[/m]\n", + "c.dsl": "#NAME \"C\"\n\n一\n yī\n [m1]z[/m]\n", + ]) { root in + let set = try DSLFileSet.discover(from: root.appendingPathComponent("a.dsl")) + #expect(set.discovery == .includeDirectives) + #expect(set.files.map(\.lastPathComponent) == ["a.dsl", "b.dsl", "c.dsl"]) + #expect(set.header.name == "A") + #expect(set.diagnostics.isEmpty) + } + } + + @Test("An include naming a file that is not there is reported") + func missingInclude() throws { + try directory([ + "a.dsl": "#NAME \"A\"\n#INCLUDE \"gone.dsl\"\n\n爱\n ài\n [m1]x[/m]\n", + ]) { root in + let set = try DSLFileSet.discover(from: root.appendingPathComponent("a.dsl")) + #expect(set.files.count == 1) + #expect(set.diagnostics.map(\.kind) == [.missingInclude("gone.dsl")]) + } + } + + @Test("An include cycle does not loop") + func includeCycle() throws { + try directory([ + "a.dsl": "#NAME \"A\"\n#INCLUDE \"b.dsl\"\n\n爱\n ài\n [m1]x[/m]\n", + "b.dsl": "#NAME \"B\"\n#INCLUDE \"a.dsl\"\n\n了\n le\n [m1]y[/m]\n", + ]) { root in + let set = try DSLFileSet.discover(from: root.appendingPathComponent("a.dsl")) + #expect(set.files.map(\.lastPathComponent) == ["a.dsl", "b.dsl"]) + } + } + + /// Only when there are no directives at all. + @Test("Numbered siblings are found when nothing is declared") + func numberedSiblings() throws { + try directory([ + "dict_1.dsl": "#NAME \"1\"\n\n爱\n ài\n [m1]x[/m]\n", + "dict_2.dsl": "#NAME \"2\"\n\n了\n le\n [m1]y[/m]\n", + "dict_3.dsl": "#NAME \"3\"\n\n一\n yī\n [m1]z[/m]\n", + ]) { root in + let set = try DSLFileSet.discover(from: root.appendingPathComponent("dict_1.dsl")) + #expect(set.discovery == .numberedSiblings) + #expect(set.files.map(\.lastPathComponent) == [ + "dict_1.dsl", + "dict_2.dsl", + "dict_3.dsl", + ]) + } + } + + /// Picking volume two has to find volume one, because that is where the + /// directives and the dictionary's own name live. + @Test("Picking a later volume restarts discovery from the first") + func picksLowestSibling() throws { + try directory([ + "dict_1.dsl": "#NAME \"whole thing\"\n#INCLUDE \"dict_2.dsl\"\n\n爱\n ài\n [m1]x[/m]\n", + "dict_2.dsl": "#NAME \"volume 2\"\n\n了\n le\n [m1]y[/m]\n", + ]) { root in + let set = try DSLFileSet.discover(from: root.appendingPathComponent("dict_2.dsl")) + #expect(set.discovery == .includeDirectives) + #expect(set.header.name == "whole thing") + #expect(set.files.map(\.lastPathComponent) == ["dict_1.dsl", "dict_2.dsl"]) + } + } + + /// A set missing its second volume is a set of one, not a set of two with + /// a hole in it. + @Test("A gap in the numbering ends the run") + func gapEndsTheRun() throws { + try directory([ + "dict_1.dsl": "#NAME \"1\"\n\n爱\n ài\n [m1]x[/m]\n", + "dict_3.dsl": "#NAME \"3\"\n\n一\n yī\n [m1]z[/m]\n", + ]) { root in + let set = try DSLFileSet.discover(from: root.appendingPathComponent("dict_1.dsl")) + #expect(set.files.map(\.lastPathComponent) == ["dict_1.dsl"]) + #expect(set.discovery == .single) + } + } + + @Test("Zero padding in the numbering is preserved") + func zeroPadding() throws { + try directory([ + "vol_01.dsl": "#NAME \"1\"\n\n爱\n ài\n [m1]x[/m]\n", + "vol_02.dsl": "#NAME \"2\"\n\n了\n le\n [m1]y[/m]\n", + ]) { root in + let set = try DSLFileSet.discover(from: root.appendingPathComponent("vol_01.dsl")) + #expect(set.files.map(\.lastPathComponent) == ["vol_01.dsl", "vol_02.dsl"]) + } + } + + /// A file whose name happens to end in a digit is not part of a run it is + /// not in; importing files the reader did not pick needs a better reason + /// than a coincidence of naming. + @Test("A file outside the numbered run is imported alone") + func unrelatedNumbering() throws { + try directory([ + "dict_1.dsl": "#NAME \"1\"\n\n爱\n ài\n [m1]x[/m]\n", + "dict_9.dsl": "#NAME \"9\"\n\n了\n le\n [m1]y[/m]\n", + ]) { root in + let set = try DSLFileSet.discover(from: root.appendingPathComponent("dict_9.dsl")) + #expect(set.files.map(\.lastPathComponent) == ["dict_9.dsl"]) + #expect(set.discovery == .single) + } + } + + @Test("A name with no trailing number stands alone") + func noNumber() throws { + try directory([ + "dict.dsl": "#NAME \"d\"\n\n爱\n ài\n [m1]x[/m]\n", + ]) { root in + let set = try DSLFileSet.discover(from: root.appendingPathComponent("dict.dsl")) + #expect(set.discovery == .single) + #expect(set.files.map(\.lastPathComponent) == ["dict.dsl"]) + } + } +} diff --git a/Tests/HanReaderCoreTests/DSLFixtureTests.swift b/Tests/HanReaderCoreTests/DSLFixtureTests.swift new file mode 100644 index 0000000..f5a160d --- /dev/null +++ b/Tests/HanReaderCoreTests/DSLFixtureTests.swift @@ -0,0 +1,244 @@ +// HanReader — MIT licensed. See LICENSE. + +import Foundation +import Testing +@testable import HanReaderCore + +/// Loads the committed slice of real 大БКРС data. +/// +/// Real cards rather than hand-written ones, because the shapes that break a +/// parser are the ones nobody thinks to invent — 一's prose sandhi note, the +/// `[*]` block with no `[ex]` inside it, the one card whose article is split +/// across two lines. Extracted by `Scripts/extract-dsl-fixture.py`; the +/// sidecar manifest says why each card is in there. +private enum Fixture { + static let text: String = { + guard let url = Bundle.module.url( + forResource: "Fixtures/bkrs-slice.dsl", + withExtension: "txt", + ), + let data = try? Data(contentsOf: url), + let text = String(data: data, encoding: .utf8) + else { + Issue.record("could not load the BKRS fixture") + return "" + } + return text + }() + + /// The committed bytes: UTF-8, no byte-order mark, so the slice stays + /// diffable in a pull request. + static let utf8: [UInt8] = Array(text.utf8) + + /// The same content as the source files are actually written: UTF-16LE + /// with a byte-order mark. Re-encoded here rather than committed, so the + /// real encoding path is exercised without a binary blob in the + /// repository. + static let utf16: [UInt8] = { + var bytes: [UInt8] = [0xFF, 0xFE] + for unit in Array(text.utf16) { + bytes += [UInt8(unit & 0xFF), UInt8(unit >> 8)] + } + return bytes + }() + + struct Reading { + let records: [DSLCardRecord] + let header: DSLHeader + let diagnostics: [DSLDiagnostic] + } + + static func read(_ bytes: [UInt8]) throws -> Reading { + var reader = try DSLCardReader(source: DSLMemoryBytes(bytes)) + var records: [DSLCardRecord] = [] + while let record = try reader.next() { + records.append(record) + } + return Reading( + records: records, + header: reader.header, + diagnostics: reader.diagnostics, + ) + } + + static func card(_ headword: String, in records: [DSLCardRecord]) -> DSLCard? { + records.first { $0.card.headwords.first == headword }?.card + } + + static func senses(_ headword: String, in records: [DSLCardRecord]) -> [Sense] { + guard let card = card(headword, in: records) else { return [] } + return DSLCardBuilder(syllableBases: []).entries(from: card).entries.first?.senses ?? [] + } +} + +@Suite("DSL fixture") +struct DSLFixtureTests { + /// The committed slice is UTF-8 and the real files are UTF-16LE+BOM. Both + /// have to come out as the same cards, or the fixture is testing a + /// different dictionary from the one the app imports. + @Test("UTF-8 and UTF-16 readings of the same content agree") + func encodingsAgree() throws { + let fromUTF8 = try Fixture.read(Fixture.utf8) + let fromUTF16 = try Fixture.read(Fixture.utf16) + #expect(fromUTF8.records.map(\.card) == fromUTF16.records.map(\.card)) + #expect(fromUTF8.header == fromUTF16.header) + // Offsets differ — two bytes per character against one — which is + // exactly why a resume point is only meaningful against its own file. + #expect(fromUTF8.records[1].offset != fromUTF16.records[1].offset) + } + + /// 203 real cards through the whole stack with nothing the parser cannot + /// account for. The tag set was established by a full scan of all three + /// files; a diagnostic here means the slice found something that scan + /// missed. + @Test("The whole slice parses with no diagnostics") + func noDiagnostics() throws { + let result = try Fixture.read(Fixture.utf16) + #expect(result.records.count == 203) + #expect(result.diagnostics.isEmpty) + + let builder = DSLCardBuilder(syllableBases: []) + var diagnostics: [DSLDiagnostic] = [] + var senses = 0 + var multiSense = 0 + for record in result.records { + let built = builder.entries(from: record.card) + diagnostics += built.diagnostics + let count = built.entries.first?.senses.count ?? 0 + senses += count + if count > 1 { + multiSense += 1 + } + } + #expect(diagnostics.isEmpty) + // Measured over this slice, and asserted exactly rather than as a + // threshold: the fixture is committed and deterministic, so a change + // here is either a parser regression or a deliberate regeneration, + // and both are worth a line in the diff. + #expect(senses == 523) + #expect(multiSense == 67) + } + + @Test("The header is read from the file") + func header() throws { + let header = try Fixture.read(Fixture.utf16).header + #expect(header.indexLanguage == "Chinese") + #expect(header.contentsLanguage == "Russian") + // The fixture is one file, so it declares no includes — and the + // extraction script writes no stray directives. + #expect(header.includes.isEmpty) + #expect(header.other.isEmpty) + } + + // MARK: - The cards the plan names + + @Test("爱 comes out as divisions with numbered senses under them") + func ai() throws { + let senses = try Fixture.senses("爱", in: Fixture.read(Fixture.utf16).records) + #expect(senses.count > 10) + #expect(senses[0].label == "I") + #expect(senses[0].partOfSpeech == ["гл."]) + #expect(senses[1].label == "1)") + #expect(senses[1].level == 1) + #expect(senses[1].gloss.text.hasPrefix("любить")) + // A second division, so the Roman numerals are not being flattened. + #expect(senses.contains { $0.label == "II" && $0.level == 0 }) + } + + /// The predecessor produced one sense for this card, and put the whole + /// string `le, liǎo, liào` above the word on the page. Three readings + /// means three entries sharing the sense run, so the reading-selection + /// rule can choose between them. + @Test("了 has many senses and one entry per reading") + func le() throws { + let records = try Fixture.read(Fixture.utf16).records + let card = try #require(Fixture.card("了", in: records)) + #expect(card.pinyin == "le, liǎo, liào") + let built = DSLCardBuilder(syllableBases: ["le", "liao"]).entries(from: card) + #expect(built.entries.map(\.readingDisplay) == ["le", "liǎo", "liào"]) + #expect(built.entries[0].senses.count >= 20) + #expect(built.entries.allSatisfy { $0.senses == built.entries[0].senses }) + } + + /// 一 carries its own tone-sandhi note as prose at the end of the card. + /// Keeping it is free and faithful; computing sandhi ourselves is not + /// something this app does. + @Test("一 keeps its sandhi note") + func yi() throws { + let senses = try Fixture.senses("一", in: Fixture.read(Fixture.utf16).records) + #expect(senses.count > 40) + #expect(senses.contains { $0.gloss.text.contains("произносится") }) + } + + @Test("打 is the deepest card in the slice") + func da() throws { + let senses = try Fixture.senses("打", in: Fixture.read(Fixture.utf16).records) + #expect(senses.count > 40) + #expect(senses.contains { $0.level >= 2 }) + } + + /// The one card in 3,434,224 whose article breaks a line inside an open + /// `[m1]`. + @Test("乐芙兰's split article reads as one sentence") + func leBlanc() throws { + let records = try Fixture.read(Fixture.utf16).records + let card = try #require(Fixture.card("乐芙兰", in: records)) + #expect(card.body.count == 2) + let senses = Fixture.senses("乐芙兰", in: records) + #expect(senses.count == 1) + #expect(senses[0].gloss.text == "Ле Блан (чемпион из Лиги Легенд)") + } + + // MARK: - Markup the predecessor lost + + /// 55,414 escaped brackets in the set. `stripDSL` treated every `[` as a + /// tag and deleted to the next `]`, so these and their contents vanished. + @Test("Escaped brackets survive as literal brackets") + func escapedBrackets() throws { + let senses = try Fixture.senses("一二", in: Fixture.read(Fixture.utf16).records) + #expect(senses.contains { $0.gloss.text.contains("[целого]") }) + } + + /// 62 `[*]` spans in the set contain no `[ex]`, and that path must neither + /// crash nor lose the text. + @Test("An example block with no example keeps its text") + func blockWithoutExample() throws { + let senses = try Fixture.senses("栓", in: Fixture.read(Fixture.utf16).records) + #expect(!senses.isEmpty) + #expect(senses.allSatisfy { !$0.gloss.text.isEmpty || !$0.examples.isEmpty }) + } + + @Test("A cross-reference records the word it points at") + func reference() throws { + let senses = try Fixture.senses("上海市", in: Fixture.read(Fixture.utf16).records) + #expect(senses.contains { !$0.gloss.references.isEmpty }) + } + + /// `_` means the dictionary has no reading — 77% of the set. An entry + /// claiming its pronunciation is the empty string is worse than one + /// admitting it has none. + @Test("A card with no reading produces an entry with no reading") + func noReading() throws { + let records = try Fixture.read(Fixture.utf16).records + let card = try #require(Fixture.card("碟仙", in: records)) + #expect(card.pinyin == "_") + let built = DSLCardBuilder(syllableBases: []).entries(from: card) + #expect(built.entries.count == 1) + #expect(built.entries[0].reading.isEmpty) + #expect(built.entries[0].readingDisplay.isEmpty) + } + + /// Every card in the slice yields at least one entry: whatever the markup + /// does, a headword the dictionary defines is a headword the reader can + /// look up. + @Test("Every card yields an entry") + func everyCardYieldsAnEntry() throws { + let records = try Fixture.read(Fixture.utf16).records + let builder = DSLCardBuilder(syllableBases: []) + for record in records { + let built = builder.entries(from: record.card) + #expect(!built.entries.isEmpty, "\(record.card.headwords) produced no entry") + #expect(built.entries.allSatisfy { !$0.senses.isEmpty }) + } + } +} diff --git a/Tests/HanReaderCoreTests/Fixtures/README.md b/Tests/HanReaderCoreTests/Fixtures/README.md index cea1551..a389c19 100644 --- a/Tests/HanReaderCoreTests/Fixtures/README.md +++ b/Tests/HanReaderCoreTests/Fixtures/README.md @@ -24,5 +24,39 @@ for the full attribution, and [`../../../Dictionaries/cc-cedict/LICENSE`](../../../Dictionaries/cc-cedict/LICENSE) for the license text. +## `bkrs-slice.dsl.txt` + +203 cards of real **大БКРС** data, used by the DSL importer tests, with +`bkrs-slice.manifest.txt` recording why each one is in there. + +Two kinds of card, because they catch different things. Thirteen are picked +deliberately — 爱, 了, 一 and 打 for their sense structure, plus one card for +each piece of markup the format uses, including the single card in 3,434,224 +whose article breaks a line inside an open `[m1]`. The other 190 are a +contiguous run taken verbatim, which covers the shapes nobody thought to pick +at a realistic density. + +Regenerate with: + +```sh +./Scripts/extract-dsl-fixture.py path/to/dabkrs_1.dsl +``` + +**UTF-8 with no byte-order mark**, where the source files are UTF-16LE+BOM. A +UTF-8 file stays diffable in a pull request; the tests re-encode it to +UTF-16LE+BOM at runtime and assert both readings produce identical cards, so +the committed bytes are reviewable and the real encoding path is still +exercised. DSL has no comment syntax — a `#` line is a directive — which is +why the reasons are in the manifest rather than inline. + +**Licensing.** 大БКРС is **not** freely licensed and is not ours to +redistribute; HanReader ships an importer, not the data. This slice exists +under the one exception +[`CONTRIBUTING.md`](../../../CONTRIBUTING.md#do-not-add-dictionary-data-to-this-repository) +names. Do not +enlarge it, and do not commit the source files. + +--- + Keep fixtures small — CI fails any file here over 256 KB. If a test needs more data than that, it wants a generator, not a bigger fixture. diff --git a/Tests/HanReaderCoreTests/Fixtures/bkrs-slice.dsl.txt b/Tests/HanReaderCoreTests/Fixtures/bkrs-slice.dsl.txt new file mode 100644 index 0000000..4f0729b --- /dev/null +++ b/Tests/HanReaderCoreTests/Fixtures/bkrs-slice.dsl.txt @@ -0,0 +1,614 @@ +#NAME "HanReader test slice of 大БКРС" +#INDEX_LANGUAGE "Chinese" +#CONTENTS_LANGUAGE "Russian" + +一 + yī + [m1][b]I[/b][/m][m2]1) 1, один, единица, один раз[/m][m3][*][ex]一一等于一 одиножды один — один[/ex][/*][/m][m3][*][ex]一败二胜 единожды потерпеть поражение и дважды одержать победу[/ex][/*][/m][m3][*][ex]一氨基酸 моноаминокислота[/ex][/*][/m][m2]2) первый ([i]по порядку[/i]), первый раз[/m][m3][*][ex]一年级 первый курс (класс)[/ex][/*][/m][m3][*][ex]一见如故 встретиться впервые[/ex][/*][/m][m2]3) одинаковый, единый, один и тот же[/m][m3][*][ex]咱们是一家人 мы - одна семья[/ex][/*][/m][m3][*][ex]你们一路走 идите одной дорогой[/ex][/*][/m][m3][*][ex]这不是一码事 это не одно и то же[/ex][/*][/m][m2]4) другой, иной, иначе[/m][m3][*][ex]一波未平，一波又起 волна ещё не улеглась, как уже поднимается другая[/ex][/*][/m][m3][*][ex]番茄一名西红柿 другое название помидора - томат[/ex][/*][/m][m2]5) весь, сплошной, целый, целиком[/m][m3][*][ex]一冬 вся зима[/ex][/*][/m][m3][*][ex]一生 вся жизнь[/ex][/*][/m][m3][*][ex]一身的汗 весь в поту[/ex][/*][/m][m3][*][ex]一家之人 все члены семьи, весь дом, вся семья[/ex][/*][/m][m2]6) каждый, отдельный[/m][m3][*][ex]用十辆卡车, 一车坐五十人 взять 10 грузовиков и на каждый посадить по 50 человек[/ex][/*][/m][m2]7) единство, единение, объединять, едино-, моно-, уни-[/m][m3][*][ex]孰能一之？ кто может объединить её ([i]Поднебесную[/i])?[/ex][/*][/m][m3][*][ex]统一 объединение[/ex][/*][/m][m2]8) некий, какой-то, один ([i]показатель неопределённости последующего существительного[/i])[/m][m3][*][ex]一人 некий (какой-то) человек[/ex][/*][/m][m2]9) [p]фил.[/p] один, одно \[единое\], \[нерасчленённое, безатрибутное\] единство, первобытное, всё[/m][m3][*][ex]道生一，一生二 Дао рождает одно (нерасчленённое единство), одно рождает два (раздвоение)[/ex][/*][/m][m3][*][ex]一人 некий (какой-то) человек[/ex][/*][/m][m2]10) \[один\] раз, разок, единожды ([i]о единичном или кратковременном действии или попытке действия, используется а) в удвоенном глаголе из одного иероглифа, б) между глаголом и его счетным словом[/i])[/m][m3][*][ex]等一等 подождать (немного); обожди![/ex][/*][/m][m3][*][ex]看一看 взглянуть; взгляни!; посмотри![/ex][/*][/m][m3][*][ex]笑一声 усмехнуться, хохотнуть[/ex][/*][/m][m3][*][ex]看一眼 взглянуть разок, взглянуть одним глазом[/ex][/*][/m][m2]11) [i]перед глаголом или его счетным словом означает первичное действие, далее по контексту результат этого действия[/i][/m][m3][*][ex]一跳跳了过去 прискакал сюда[/ex][/*][/m][m3][*][ex]他在旁边一站，再也不说什么 он стоял рядом, не произнося ни слова[/ex][/*][/m][m2]12) резко, вдруг, сразу ([i]резкий переход к новому состоянию или действию, выраженному последующим глаголом[/i])[/m][m3][*][ex]天气一凉 резко похолодало[/ex][/*][/m][m3][*][ex]他累得蒙头一睡 он так устал, что едва укрылся с головой, как мгновенно (сразу же) и заснул[/ex][/*][/m][m3][*][ex]他都不管甩手一走 он ни с чем не посчитался и, махнув рукой, пошёл вдруг прочь[/ex][/*][/m][m2]13) раз...; как только...; всякий раз, как (когда)...; если \[бы\] вдруг; если уж ([i]быстрая смена действий, часто с последующим 就, 便, 即[/i])[/m][m3][*][ex]你一叫，我就来 как только ты позовёшь, я приду[/ex][/*][/m][m3][*][ex]一听就懂 как только послушал, так сейчас же и понял[/ex][/*][/m][m3][*][ex]一不小心 по не осторожности[/ex][/*][/m][m2]14) 一... 一... или, или; и; то..., то...[/m][m3][*][ex]一胜一负 то выигрывать, то проигрывать[/ex][/*][/m][m3][*][ex]一红一白 то краснеть, то бледнеть[/ex][/*][/m][m2]15) [i]модальная усилительная частица[/i], очень, весьма[/m][m3][*][ex]一至于此 вот в конце концов до чего дошло![/ex][/*][/m][m3][*][ex]史呼一何怒 насколько же гневно чиновники эти кричали...[/ex][/*][/m][m3][*][ex]余一怪之 я искренне удивился этому[/ex][/*][/m][m1][b]II[/b][/m][m2]1) знак в древних китайских нотах[/m][m2]2) ключ иероглифа ([p]см.[/p] [ref]横[/ref])[/m][m1]-----[/m][i]Примечание: 一 перед иероглифом с нисходящим или нейтральным тоном произносится[/i] [b]yí[/b]; [i]перед иероглифом с ровным, восходящим, нисходяще-восходящим тоном произносится[/i] [b]yì[/b]; [i]тон не изменяется, когда: а) во фразе или в предложении происходит перечисление или указание порядкового номера; б) 一 оказывается в конце фразы.[/i] +一二 + yīèr + [m1]1) один-два, несколько, немного; что-нибудь, сколько-нибудь; хоть немножко[/m][m2][*][ex]除一二老人外，都不认识他 кроме немногих (нескольких) стариков, мало кто его знает[/ex][/*][/m][m2][*][ex]我略知一二 я кое-что слышал; я несколько в курсе дела[/ex][/*][/m][m1]2) частица, небольшая часть \[целого\]; меньшинство[/m][m1]3) один за одним, по порядку[/m][m1]4) лавировать; уклончивый[/m][m1]5) 12[/m][m1]6) двенадцатый месяц, декабрь[/m][m1]7) 2 января[/m] +一般性讨论 + yībānxìng tǎolùn + [m1]общие прения[/m] +三摩呾咤 + sānmódázhà + [m1][p]ист.[/p] Саматата ([i]государство в Вост. Индии в эпоху Тан[/i])[/m] +上海市 + shànghǎi shì + [m1][p]см.[/p] [ref]上海[/ref][/m] +乐芙兰 + _ + [m1]Ле Блан + (чемпион из Лиги Легенд)[/m] +了 + le, liǎo, liào + [m1][b]I[/b] [b]le[/b][/m][m2]1) [i]после глагола или прилагательного означает завершения действия или изменение состояния[/i][/m][m3][*][ex]写完了 написал[/ex][/*][/m][m3][*][ex]水位已经低了两米 уровень воды снизился на два метра[/ex][/*][/m][m2]2) [i]в конце предложения или паузы означает перемену, создание новой ситуации[/i][/m][m3][*][ex]下雨了 пошёл дождь, уже идёт дождь[/ex][/*][/m][m3][*][ex]我现在明白他的意思了 теперь я понимаю о чем он[/ex][/*][/m][m1][b]II[/b] [b]liǎo[/b][/m][m2]1) заканчивать, завершать, улаживать, справляться[/m][m3][*][ex]一了百了 разобравшись в одном — разобраться в остальном[/ex][/*][/m][m3][*][ex]这事儿已经了啦 это дело уже закончилось[/ex][/*][/m][m3][*][ex]这场纠纷还是他出头给了了 этот конфликт был улажен всё же благодаря его вмешательству[/ex][/*][/m][m2]2) [i]после глагола, вместе с 得 или 不 означает соответственно возможность или невозможность данного действия:[/i] \[не\]смочь, \[не\]возможно[/m][m3][*][ex]来不了 не могу прийти[/ex][/*][/m][m3][*][ex]办得了 возможно сделать[/ex][/*][/m][m2]3) понимать, получать ясное представление, ясный[/m][m3][*][ex]明了 отчетливо понять[/ex][/*][/m][m3][*][ex]一目了然 понятный с одного взгляда[/ex][/*][/m][m2]4) в полной мере, абсолютно; совершенно, полностью ([i]с отрицанием[/i])[/m][m3][*][ex]了不相涉 не иметь никакого отношения[/ex][/*][/m][m3][*][ex]了无惧色 не проявлять и тени страха[/ex][/*][/m][m2]5) умный, сообразительный[/m][m3][*][ex]了干 умный и способный[/ex][/*][/m][m2]6) светлый, блестящий[/m][m2]7) в конце \[концов\], в итоге[/m][m3][*][ex]了复何益？ в итоге какая польза?[/ex][/*][/m][m1][b]III[/b] [b]liào[/b] [p]упрощ.[/p] [ref]瞭[/ref][/m][m2]1) смотреть вдаль; смотреть издали[/m][m3][*][ex]上高处了一了 посмотреть ввысь[/ex][/*][/m][m2]2) смотреть, коситься[/m] +代表国家集团 + dàibiǎo guójiā jítuán + [m1]делегация, выступающая от имени группы государств[/m] +僾 + ài + [m1][b]I [/b][c][i]вводн. слово[/i][/c][/m][m2]* как будто, вроде, словно бы, похоже, что... ([i]также [/i]僾然)[/m][m1][b]II[/b][c][i] гл.[/i][/c][/m][m2]* терять дыхание, задыхаться[/m][m3][*][ex]如彼遡风, 亦孔之僾 Словно под этим бы ветром противным Тоже дыхание сильно теряешь...[/ex][/*][/m] +单宁植物 + _ + [m1]танидонос[/m] +叆靅 + àifèi + [m1]пасмурный, облачный, сумрачный, тусклый, туманный[/m] +后不久 + hòubùjiǔ + [m1]вскоре после ([i]этого[/i]), спустя немного времени, через короткий промежуток времени[/m] +后不圆音 + hòubùyuányīn + [m1][p]лингв.[/p] нелабилизованные гласные заднего ряда[/m] +后乘 + hòuchéng + [m1]1) экипаж свиты[/m][m1]2) сопровождающие всадники[/m] +后代 + hòudài + [m1]1) последующие эпохи (века); позднейшие времена; поздний период[/m][m1]2) грядущие поколения; потомки, потомство[/m] +后劲 + hòujìn + [m1]1) результат, последствие; последующее действие; постэффект[/m][m1]2) эффект задержки, поздний эффект, отсроченное воздействие[/m][m1]3) выносливость, стойкость, выдержка[/m][m1]4) [p]уст.[/p] сильный резерв (арьергард); мощная поддержка (с тыла)[/m] +后勾儿 + hòugōur + [m1]: 拉后勾儿 [b]lāhòugōur[/b] отложить на время окончание ([c][i]чего-л.[/i][/c]); отложить «на-пот[c red]о[/c]м»[/m] +后半夜儿 + hòubanyèr, hòubànyèr + [m1]во второй половине ночи; глубокой ночью; за полночь[/m] +后半晌儿 + hòubànshǎngr, hòubanshǎngr + [m1]после обеда, после полудня; во второй половине дня[/m] +后半期 + hòubànqī + [m1]вторая половина ([i]отрезка времени[/i])[/m] +后卫 + hòuwèi + [m1]1)[c][i] воен.[/i][/c] тыловое охранение, арьергард; арьергардный[/m][m1]2) [p]спорт[/p] защитник[/m] +后厦 + hòushà + [m1]крытая галерея (навес) позади дома[/m] +后咎 + hòujiù + [m1]дальнейшие дурные последствия; отрицательные результаты в будущем[/m] +后堂 + hòutáng + [m1]1) женская половина; гинекей[/m][m1]2) зал в задней части дома[/m] +后备军 + hòubèijūn + [m1]резервные войска, войска запаса; ополчение; резерв[/m] +后天性免疫 + hòutiānxìng miǎnyì + [m1]приобретённый иммунитет[/m] +后夫 + hòufū + [m1]1) второй муж[/m][m1]2) потомки[/m] +后头叶 + hòutóuxié + [m1][p]анат.[/p] затылочная доля ([i]головного мозга[/i])[/m] +后奏曲 + hòuzòuqǔ + [m1][p]муз.[/p] эпилог[/m] +后婚 + hòuhūn + [m1]вторичный брак[/m] +后学 + hòuxué + [m1]1) последующие исследователи; будущие ученики; последователи (учения)[/m][m1]2)[c][i] вежл.[/c] я,[/i] Ваш ученик[/m] +后尘 + hòuchén + [m1]1) за собой вздымать пыль (по дороге)[/m][m2][*][ex]逸翮后尘，翱翥先路 вздымая пыль стремительными крыльями, летят вперёд[/ex][/*][/m][m1]2) [p]обр.[/p] тот, кто следует за другими; идущий проторенной дорогой[/m][m2][*][ex]步人后尘 [p]обр.[/p] идти по проторенной дорожке[/ex][/*][/m][m2][*][ex]不步后尘 не идти по стопам других[/ex][/*][/m][m2][*][ex]步安敦使者的后尘 те, кто шли по стопам посланников Марка Аврелия[/ex][/*][/m][m1]3) [p]уничижит.[/p] о себе: плетущийся следом, недостойный[/m][m1]余虽不敏，请寻后尘 И хотя я не умён, прошу \[Вас\] следовать за \[мной, \] недостойным[/m] +后尾 + hòuwěi + [m1]1) задняя (хвостовая) часть; корма; хвост ([c][i]напр.[/c] колонны, самолёта[/i]); тыл[/m][m1]2) в хвосте, сзади, позади ([i]послелог[/i])[/m][m2][*][ex]纵队后尾儿 в хвосте колонны, позади колонны[/ex][/*][/m] +后席 + hòuxí + [m1]1) задние места[/m][m1]2) отодвигать циновку назад; отодвигаться назад[/m] +后序 + hòuxù + [m1]послесловие; эпилог[/m] +后庭 + hòutíng + [m1]1) задний двор[/m][m1]2) императорский гарем[/m][m1]3) задний проход, анус[/m] +后开 + hòukāi + [m1][p]эпист.[/p] нижеперечисленные; ниже[/m] +后得智 + hòudézhì + [m1][p]будд.[/p] конкретное (детальное) знание, проистекающее из интуитивного знания[/m] +后心 + hòuxīn + [m1]1) последующая (возникшая позже) идея (мысль, настроение, намерение)[/m][m1]2) часть спины напротив сердца; середина спины; под левой лопаткой; между лопатками[/m] +后怕 + hòupà + [m1]испытывать запоздалый страх; содрогаться при воспоминании о...; испугаться задним числом[/m] +后成河 + hòuchénghé + [m1][p]геол.[/p] последующие реки[/m] +后房水 + hòufángshuǐ + [m1][p]биол.[/p] влага задней камеры глаза; камерная влага[/m] +后手儿 + hòushǒur + [m1]запас, резерв \[времени, места\]; простор; поле \[для дальнейших действий\]; шанс; возможность к отступлению[/m][m2][*][ex]留后手儿 оставить за собой путь к отступлению[/ex][/*][/m] +后把拉 + hòubala + [m1][c][i]сев.-вост. диал.[/c],[c] см.[/i][/c] 后边[/m] +后拒 + hòujù + [m1]арьергард; тыловое охранение; резерв; подкрепление[/m] +后接词 + hòujiēcí + [m1][p]лингв.[/p] проклитика[/m] +后支索 + hòuzhīsuǒ + [m1][p]мор.[/p] бакштаг; бурундук ([i]снасть[/i])[/m] +后日 + hòurì + [m1]послезавтра[/m] +后期 + hòuqī + [m1]1) просрочить время, опоздать[/m][m1]2) поздний период; поздняя стадия; поздний, запоздалый[/m][m1]3) последующий[/m][m1]4) второй срок ([c][i]напр.[/c] второй семестр[/i]); вторая половина; во второй половине, в конце[/m][m1]5) [p]биол.[/p] анафаза[/m][m1]6) постпроцессинг[/m][m1]7) [i]в сложных терминах соответствует морфеме[/i] пост-, после-[/m][m2][*][ex]后期制作 пост-продакшн[/ex][/*][/m] +后来 + hòulái + [m1]1) впоследствии, в дальнейшем, потом, после, затем; последующий[/m][m1]2) прийти после всех (последним); пришедший (полученный) позже[/m] +后果 + hòuguǒ + [m1]последствия, результаты[/m] +后槽 + hòucáo + [m1]1) конюшня[/m][m1]2) конюх[/m] +后檐 + hòuyán + [m1]задняя стреха крыши[/m] +后檐墙 + hòuyánqiáng + [m1]задняя стена дома[/m] +后此 + hòucǐ + [m1]после этого, затем; позднее, позже[/m] +后殿 + hòudiàn + [m1]1) задняя (женская) часть дома; гарем[/m][m1]2) тыловой заслон, арьергард[/m][m1]3) сокровищница в храме[/m] +后淋 + hòulìn + [m1][p]мед.[/p] хронический уретрит[/m] +后溲 + hòusōu + [m1]испражняться[/m] +后灯 + hòudēng + [m1][p]тех.[/p] задний (буферный) фонарь, катафот[/m] +后熟 + hòushú + [m1][c][i]с.-х.[/i][/c] дозревать; дозревание[/m] +后父 + hòufù + [m1]отчим[/m] +后生可畏 + hòu shēng kě wèi + [m1]молодое поколение дышит в затылок; молодой да ранний, умен не по годам[/m] +后甲板 + hòujiǎbǎn + [m1][p]мор.[/p] кормовая палуба, ют, шканцы[/m] +后知 + hòuzhī + [m1]поздно (позднее) познавшие ([c][i]напр.[/c] истину[/i])[/m] +后继 + hòujì + [m1]1) преемник, наследник[/m][m1]2)[c][i] уст. воен.[/i][/c] подкрепление[/m][m1]3) вторая ([i]по второму браку[/i]) жена[/m] +后缀 + hòuzhuì + [m1]1) [p]лингв.[/p] суффикс, постфикс[/m][m1]2) [p]комп.[/p] расширение[/m] +后置性 + hòuzhìxìng + [m1][p]лингв.[/p] постпозитивность[/m] +后肠 + hòucháng + [m1][p]зоол.[/p] задняя кишка[/m] +后脑 + hòunǎo + [m1]1) [c][i]анат.[/i][/c] задний мозг, ромбовидный мозг, ромбэнцефалон[/m][m1]2) [c][i]сокр.[/i][/c] затылок[/m] +后脑杓子 + hòunǎosháozi + [m1]затылок[/m] +后腰 + hòuyāo + [m1]1) поясничная область спины, поясница[/m][m1]2) [p]спорт[/p] опорный полузащитник[/m] +后腿儿 + hòutuǐr + [m1][p]см.[/p] [ref]后腿[/ref][/m] +后膛 + hòutáng + [m1]казённик, казённая часть ([i]ружья, орудия[/i])[/m] +后臼齿 + hòujiùchǐ + [m1][p]см.[/p] [ref]大臼齿[/ref][/m] +后艰 + hòujiān + [m1][p]см.[/p] 后难[/m] +后街 + hòujiē + [m1]переулок (улица) позади дома; задняя ([i]обычно северная[/i]) улица[/m] +后补 + hòubǔ + [m1]вводить дополнительно; кооптировать; кооптированный[/m] +后裔 + hòuyì + [m1]потомки, отпрыски; потомство; младшее поколение[/m] +后见人 + hòujiànrén + [m1][p]яп.[/p] опекун, попечитель ([i]несовершеннолетнего[/i])[/m] +后赶着儿 + hòugǎnzher + [m1]1) следовать за...; вдогонку[/m][m1]2) в спешке, второпях[/m] +后起之秀 + hòuqǐ zhī xiù + [m1]восходящее светило; [p]обр.[/p] молодой талант[/m] +后足 + hòuzú + [m1]задние лапы (ноги)[/m] +后跟 + hòugēn + [m1]задник, каблук ([i]обуви[/i]); пятка[/m] +后身 + hòushēn + [m1]1) сзади, со спины[/m][m1]2) задняя часть ([c][i]напр.[/c] подола одежды[/i]); фалды[/m][m1]3)[c][i] воен.[/i][/c] казённая часть ([i]оружия[/i])[/m][m1]4) за, позади; в глубине помещения (двора, тупика, улицы)[/m][m1]5)[c][i] будд.[/i][/c] перевоплощение в будущей жизни; перерождение[/m][m1]6) последующая форма ([i]организации, учреждения[/i])[/m] +后车 + hòuchē + [m1]1) экипаж свиты государя; колесница охраны[/m][m1]2) задняя повозка, экипаж, идущий сзади[/m] +后轴 + hòuzhóu + [m1]задняя ось[/m] +后辈儿 + hòubèir + [m1]потомки; будущие (грядущие) поколения[/m] +后边儿 + hòubiānr + [m1][p]см.[/p] [ref]后边[/ref][/m] +后进 + hòujìn + [m1]1) подрастающее поколение, молодёжь[/m][m1]2) отстающий, отсталый[/m] +后送 + hòusòng + [m1][p]воен.[/p] отправлять (направлять) в тыл, эвакуировать; эвакуация[/m] +后部 + hòubù + [m1]задняя часть; кормовая часть; зад; задний; донная часть [i](снаряда)[/i][/m] +后锯筋 + hòujùjīn + [m1][p]анат.[/p] задняя зубчатая мышца[/m] +后队 + hòuduì + [m1]замыкающий отряд, тыльный отряд, арьергард[/m] +后院 + hòuyuàn + [m1]1) задний (внутренний) двор[/m][m1]2) [p]перен.[/p] вотчина; сфера влияния[/m][m2][*][ex]拉美地区长期被视为“美国后院” Латинская Америка давно считается «задним двором США».[/ex][/*][/m] +后难 + hòunán + [m1]будущие осложнения (бедствия), предстоящие трудности (лишения), последующие страдания[/m] +后音儿 + hòuyīnr + [m1]заключительная нота (аккорд); отзвук[/m] +后韵 + hòuyùn + [m1][c][i]см.[/c] [/i]后元音[/m] +后项 + hòuxiàng + [m1]1)[c][i] мат.[/i][/c] последующий член отношения; второй член пропорции[/m][m1]2) следующий абзац (параграф)[/m][m1]3) затылок, загривок[/m] +后顾之忧 + hòugùzhīyōu + [m1]1) проблемы внутри (дома, организации); проблемы[/m][m1]2) беспокойство (опасение) за свой тыл[/m] +后首里 + hòushǒuli + [m1]после, потом, затем; впоследствии[/m] +嗳 + ǎi; ài; āi + [m1][b]I[/b] [p]сущ.[/p] [b] ǎi, [/b] [p]книжн.[/p] [b] ài[/b][/m][m2]тяжёлый горячий выдох; отрыжка[/m][m1][b]II [/b] [p]междом[/p][/m][m2]1) [b]ǎi[/b] ну!, м-м!, нет! ([i]междометие отрицания, несогласия[/i])[/m][m3][*][ex]嗳, 话不是那未说! Ну, нет! ничего подобного![/ex][/*][/m][m2]2) [b]ài[/b], [b]āi[/b] ([i]также[/i] 哎) ай!, ох!, эх! ([i]междометие испуга, нетерпения, досады)[/i][/m][m3][*][ex]嗳, 今儿偏偏来了刘老老, 我自己多事, 为她跑了半日 Эх, сегодня как нарочно приходила бабушка Лю, и мне при всей куче своих дел ещё пришлось провозиться с нею полдня![/ex][/*][/m] +土耳其斯坦 + tǔ’ěrqísītǎn + [m1][p]см.[/p] [ref]突厥斯坦[/ref][/m] +处世 + chǔshì + [m1]жить \[на свете\]; общаться с людьми; жизнь, существование; житейский[/m] +处之泰然 + chǔzhī tàirán + [m1]спокойно отнестись; как ни в чём не бывало[/m] +处事 + chǔshì + [m1]1) ведать делами; решать (улаживать) дела; вести дела, управлять делами[/m][m1]2)[c][i] уст.[/i][/c] указательная категория иероглифов[/m] +处以 + chǔyǐ + [m1]приговаривать к...; налагать штраф в размере...[/m][m2][*][ex]处以死刑 приговаривать к смертной казни[/ex][/*][/m] +处决 + chǔjué + [m1]1) приводить в исполнение смертный приговор, казнить[/m][m1]2) окончательно уладить (решить) дело; разделаться; \[окончательное\] решение; резолюция[/m] +处刑 + chǔxíng + [m1]подвергать наказанию, приводить приговор в исполнение; казнить, карать, наказывать; казнь, кара[/m] +处办 + chǔbàn + [m1]решать и приводить в исполнение; распоряжаться; исполнять; исполнение, распоряжение[/m] +处务 + chǔwù + [m1]решать дела, управлять (ведать) делами[/m][m2][*][ex]处务细则[c][i] офиц.[/i][/c] подробная инструкция, правила ([i]работы[/i])[/ex][/*][/m] +处变 + chǔbiàn + [m1]жить в треволнениях (в необычных условиях); попадать в передрягу[/m] +处和 + chǔhé + [m1]примирять, улаживать; быть посредником[/m] +处境 + chǔjìng + [m1]положение, обстановка, ситуация[/m] +处士 + chǔshì + [m1]1) чиновник не у дел; человек, удалившийся от общественной жизни; учёный-затворник; анахорет; оппозиционер ([c][i]напр.[/c] правителю[/i])[/m][m1]2)[c][i] кит. астр.[/i][/c] Чуши ([i]созвездие из четырёх звезд к западу от конфигурации Тайвэй [/i]太微)[/m] +处处 + chùchù + [m1]повсюду, везде, повсеместно; в разных местах; во всех отношениях, во всём[/m] +处女 + chǔnǚ + [m1]1) девственница, дева; девушка; девственный; девственность[/m][m1]2) вернисаж, дебют, премьера; первый, впервые[/m][m2][*][ex]处女表演 дебют, премьера[/ex][/*][/m][m2][*][ex]处女航海 первое плавание[/ex][/*][/m][m1]3) [p]см.[/p] [ref]处女座[/ref][/m] +处女官 + chǔnǚguān + [m1][p]астр.[/p] Дева ([i]6-й из 12 знаков зодиака[/i])[/m] +处女带 + chǔnǚdài + [m1]головная повязка (лента) девушки ([i]вид украшения[/i])[/m] +处女性 + chǔnǚxìng + [m1]девственность[/m] +处女林 + chǔnǚlín + [m1]девственные леса[/m] +处女航 + chǔnǚháng + [m1]первый рейс (судна)[/m] +处子 + chǔzǐ + [m1]1) девственник; девственница, дева; девушка[/m][m2][*][ex]搂处子 сманить (соблазнить) невинную девушку[/ex][/*][/m][m1]2) человек, удалившийся от общественной жизни; анахорет[/m] +处己 + chǔjǐ + [m1]вести себя; управлять собой[/m] +处心积虑 + chǔxīn jīlǜ + [m1]вынашивать, давно замышлять; строить козни; только и думать, что о... ([i]обычно о дурном[/i]), давно питать намерение...[/m] +处所 + chùsuǒ + [m1]место, местонахождение; местопребывание[/m] +处断 + chǔduàn + [m1]\[окончательно\] решать ([i]дело[/i])[i],[/i] выносить решение, приговаривать; принимать решительные меры; решение, определение[/m] +处方 + chǔfāng + [m1]1) рецепт ([i]врача[/i])[/m][m2][*][ex]处方说明 правила пользования ([i]лекарством[/i])[i],[/i] сигнатура[/ex][/*][/m][m1]2) выписывать рецепт, назначать лечение[/m][m2][*][ex]给病人处方退烧药 назначать пациенту жаропонижающие средства[/ex][/*][/m] +处方笺 + chǔfāngjiān + [m1]бланк рецепта, рецепт[/m] +处治 + chǔzhì, chǔzhi + [m1]1)[c][i] см.[/i][/c] 处置[/m][m1]2) приступать к лечению \[больного\][/m] +处理品 + chǔlǐpǐn + [m1]уценённые товары; залежавшиеся товары, требующие реализации; товарные остатки[/m] +处置 + chǔzhì + [m1]1) упорядочивать, налаживать; принимать меры, утилизовать; справляться; меры, шаги к упорядочению, обращение[/m][m1]2) налагать взыскание, подвергать наказанию (штрафу), наказывать; расправляться с ([c][i]кем-л.[/i][/c])[/m][m1]3) располагать, размещать; диспозиция[/m][m1]4) распоряжаться [i](об имуществе)[/i][/m][m2][*][ex]处置上述货物 распоряжаться вышеуказанным товаром[/ex][/*][/m] +处舍 + chǔshè + [m1]лагерь, бивак[/m] +处裁 + chǔcái + [m1]решать, определять[/m] +处身 + chǔshēn + [m1]жить среди людей, блюсти себя; соблюдать правила общежития[/m] +夜 + yè + [m1]ночь; ночью, вечером; ночной, вечерний; поздно[/m][m2][*][ex]深夜 глубокая ночь[/ex][/*][/m][m2][*][ex]他两夜\[儿\]工夫没睡 он не спал две ночи[/ex][/*][/m][m2][*][ex]夜船 ночной пароход[/ex][/*][/m][m2][*][ex]夜礼服 вечерний костюм; смокинг[/ex][/*][/m][m2][*][ex]夜行军 ночной марш (переход)[/ex][/*][/m][m2][*][ex]夜大学 вечерний университет (институт)[/ex][/*][/m][m2][*][ex]夜寝 поздно ложиться[/ex][/*][/m][m1]2) мрак, темнота; тёмный[/m][m2][*][ex]昼夜 день и ночь, светлое и тёмное время суток; сутки[/ex][/*][/m][m2][*][ex]夜室 тёмная комната, мрачное помещение[/ex][/*][/m][m1]3) Е ([i]фамилия[/i])[/m] +夜合 + yèhé + [m1][p]бот.[/p][/m][m1]1) горец многоцветный ([i]лат. Polygonum multiflorum[/i])[/m][m1]2) черёмуха обыкновенная ([i]лат. Padus racemosa[/i])[/m][m1]3) [p]см.[/p] [ref]合欢[/ref]3)[/m] +夜学 + yèxué + [m1]1) вечерняя школа (курсы); вечерние занятия (уроки)[/m][m1]2) учиться (заниматься) по ночам[/m] +夜客 + yèkè + [m1][p]поэт.[/p] ночной гость ([c][i]обр. в знач.:[/c] грабитель, вор-взломщик, бандит[/i])[/m] +夜春 + yèchūn + [m1]ночная весна ([i]хлебороб о погоде, когда днём[/i] ― [i]солнце, а ночью[/i] ― [i]дождь[/i])[/m] +夜格 + yègé + [m1][p]диал.[/p] вчера[/m] +夜泣 + yèqì + [m1]плакать по ночам[/m] +夜猫子 + yèmāozi + [m1]1) [p]диал.[/p] [ref]猫头鹰[/ref][/m][m1]2) полуночник, сова; [p]букв.[/p] полуночный кот [i](о человеке, не спящем ночью или поздно ложащемся спать)[/i][/m] +夜盲 + yèmáng + [m1][p]мед.[/p] гемералопия, куриная слепота[/m] +夜直 + yèzhí + [m1]ночное дежурство (служба); дежурить ночью[/m] +夜郎自大 + yèláng zìdà + [m1]Елан считает себя великим ([i]по преданию эпохи Хань о том, как в княжество Елан, находившееся на юге Китая, приехал китайский посол, и тамошний правитель с высокомерием спросил его, не думает ли тот, что Китай больше его собственного княжества; [p]обр.[/p] невежественная самонадеянность; бесстыдное хвастовство; мания величия; в подполье и мышь геройствует; на своей улочке храбра и курочка[/i])[/m] +夜静 + yèjìng + [m1]ночная тишина ([c][i]обр. в знач.:[/c] глубокая ночь; глубокой ночью; среди ночи[/i])[/m] +够面子 + gòu miànzi + [m1]соблюдать полное достоинство, полностью сохранять лицо (честь, репутацию); \[ваше\] достоинство соблюдено; \[ваше\] самолюбие не страдает[/m] +媚 + mèi + [m1][b]I [/b] [p]гл.[/p][/m][m2]1) льстить, угождать, заискивать, низкопоклонничать, пресмыкаться[/m][m3][*][ex]媚上骄下 заискивать перед высшими и третировать низших[/ex][/*][/m][m4][*][ex]媚富 заискивать перед богатыми[/ex][/*][/m][m2]2) любить, ласкать[/m][m3][*][ex]媚兹一人 любить его одного[/ex][/*][/m][m1][b]II[/b] [p]прил.[/p][/m][m2]1) льстивый, угодливый[/m][m3][*][ex]媚语 льстивые слова, медовые речи[/ex][/*][/m][m2]2) привлекательный, пленительный, миловидный; кокетливый[/m][m3][*][ex]媚眼 кокетливый взгляд[/ex][/*][/m] +布里斯托尔 + bùlǐsītuō’ěr + [m1]Бристоль ([i]город в Англии[/i])[/m] +帕氏刺隆头鱼 + _ + [m1]акантолабрус, колючий губан ([i]лат. Acantholabrus palloni[/i])[/m] +打 + dǎ, dá + [m1][b]I[/b] [p]гл.[/p] [b]dǎ[/b][/m][m2]1) бить, ударять; колотить; драться[/m][m3][*][ex]不许打人 нельзя прибегать к побоям[/ex][/*][/m][m3][*][ex]用枪托一打 ударить прикладом[/ex][/*][/m][m3][*][ex]打锣鼓 бить в гонги и барабаны[/ex][/*][/m][m3][*][ex]两个小孩儿打起来了 два мальчугана подрались[/ex][/*][/m][m2]2) бить, разбивать; раздроблять; портить[/m][m3][*][ex]打了一个碗 разбить \[одну\] пиалу[/ex][/*][/m][m3][*][ex]打玻璃 бить стёкла[/ex][/*][/m][m3][*][ex]帽子叫虫子打了 шапка изъедена молью[/ex][/*][/m][m2]3) отбить, прозвонить; подать сигнал \[звоном\][/m][m3][*][ex]钟打六点 часы бьют шесть[/ex][/*][/m][m2]4) избивать; убивать[/m][m3][*][ex]这只野鸭子是怎么打的 как была убита эта утка?[/ex][/*][/m][m2]5) акробатика [i](в Пекинской опере)[/i][/m][m2]6) [i]в сочетании с дополнением-существительным выступает в качестве слова-заместителя целого ряда глаголов с весьма широким значением. Для примера здесь могут быть выделены следующие смысловые рубрики[/i][/m][m2]а) делать, вырабатывать; работать; изготовлять; приготовлять, подготавливать[/m][m3][*][ex]打了两辆车 изготовить две повозки[/ex][/*][/m][m3][*][ex]打刀 сделать нож[/ex][/*][/m][m3][*][ex]夜打 работать ночью ([i]по ночам[/i])[/ex][/*][/m][m2]б) извлекать \[жидкость\]; вынимать, доставать; раздобывать; приобретать, покупать[/m][m3][*][ex]打了不少的鱼 выловил немало рыбы[/ex][/*][/m][m3][*][ex]我给你打了四两酒 я раздобыл для тебя 4 ляна вина[/ex][/*][/m][m3][*][ex]打油 купить (раздобыть) \[жидкого\] масла[/ex][/*][/m][m3][*][ex]打火车票 приобрести железнодорожный билет[/ex][/*][/m][m2]в) вести; организовать, создавать, устраивать[/m][m3][*][ex]打掩护 создать прикрытие, организовать заслон[/ex][/*][/m][m3][*][ex]打游击\[战\] вести партизанскую войну, партизанить[/ex][/*][/m][m2]г) поднимать, высоко держать[/m][m3][*][ex]打旗子 высоко поднять флаг[/ex][/*][/m][m3][*][ex]打灯笼 держать в руке \[зажжённый\] фонарь ([i]бумажный или из ткани[/i])[/ex][/*][/m][m2]д) снимать, собирать (урожай)[/m][m3][*][ex]一亩地打多少粮食? какой урожай снимается с одного му земли?[/ex][/*][/m][m2]е) быть; становиться; являться ([c][i]кем-л.[/i][/c])[/m][m3][*][ex]他还是打光棍的 он ещё ходит в холостяках[/ex][/*][/m][m2]ж) производить ([i]действие[/i]) [i],[/i] пускать ([c][i]что-л.[/i][/c]) по назначению, пустить в ход[/m][m3][*][ex]打电话 позвонить по телефону[/ex][/*][/m][m3][*][ex]打电\[报\] отправить телеграмму[/ex][/*][/m][m3][*][ex]打主意 принять решение[/ex][/*][/m][m3][*][ex]打路条 выправить пропуск ([i]для проезда из одного района в другой[/i])[/ex][/*][/m][m3][*][ex]打短工 работать подённо[/ex][/*][/m][m3][*][ex]打基础 закладывать фундамент[/ex][/*][/m][m3][*][ex]打交道 сталкиваться, иметь связь[/ex][/*][/m][m3][*][ex]打走 изгнать, выбить ([c][i]напр.[/c] неприятеля[/i])[/ex][/*][/m][m2]6) играть ([c][i]на чем-л.[/c], во[c] что-л.[/i][/c])[/m][m3][*][ex]打麻将 играть в мацзян[/ex][/*][/m][m3][*][ex]打钢琴 играть на рояле[/ex][/*][/m][m1][b]II [/b] [c][i]счётное слово[/i][/c] [b] dá[/b][/m][m2][p]сокр.[/p] ([i]от[c] англ.[/c][/i] dozen) дюжина[/m][m3][*][ex]买一打手帕 купить дюжину носовых платков[/ex][/*][/m][m1][b]III [/b] [p]предлог[/p] [b]dǎ[/b][/m][m2]1) [i]указывает на исходное место действия:[/i] из, от, с[/m][m3][*][ex]刚打天津 только что прибыть из Тянь-цзиня[/ex][/*][/m][m3][*][ex]打这里\[儿\] отсюда[/ex][/*][/m][m3][*][ex]打这儿走 проходите здесь[/ex][/*][/m][m2]2) [i]указывает на исходное время действия:[/i] с[/m][m3][*][ex]打今天算起 считать (считая) с сегодняшнего дня[/ex][/*][/m] +打睡一觉 + dǎshuì yījiào + [m1]поспать немного[/m] +操弄 + cāonòng + [m1]манипуляции; манипулировать[/m] +支农工业 + zhīnónggōngyè + [m1]работающие на сельское хозяйство отрасли промышленности[/m] +改性橡碗栲胶 + _ + [m1]изомеризованный экстракт валонеи[/m] +有害做法 + yǒuhài zuòfa + [m1]порочная практика[/m] +杠铃架 + gànglíngjià + [m1]стойка под штангу[/m] +柴郡 + cháijùn + [m1][p]графство[/p] Чешир ([p]столица[/p] — Честер, [c][i]крупн. город[/i][/c] — Уоррингтон) см. [ref]柴郡猫[/ref][/m] +栓 + shuān + [m1]1) деревянная затычка; втулка; пробка ([i]напр.[/i] бутылки); вентиль, кран ([i]напр.[/i] пожарный)[/m][m1][*] - [ref]木栓[/ref][/*][/m][m1]2) затвор ([i]напр.[/i] ружейный)[/m][m1]3) [p]мед.[/p] тромб, эмбол[/m] +滥赏 + lànshàng + [m1]засып[c red]а[/c]ть наградами, неумеренно награждать[/m] +爱 + ài + [m1][b]I[/b] [p]гл.[/p][/m][m2]1) любить, быть влюбленным в ([i]кого-л.[/i]), быть привязанным, расположенным к ([i]кому-л.[/i])[/m][m3][*][ex]爱儿童 любить ребёнка (детей)[/ex][/*][/m][m3][*][ex]他说他爱我 он говорит, что любит меня[/ex][/*][/m][m2]2) пристраститься к ([i]чему-л.[/i]), иметь пристрастие к..., находить удовольствие в...; любить; дорожить; нравится ([i]что-л.[/i])[/m][m3][*][ex]爱酒 пристраститься к вину[/ex][/*][/m][m3][*][ex]我爱钓鱼 я люблю (мне нравится) ловить рыбу[/ex][/*][/m][m3][*][ex]我不爱这个 мне не нравится это[/ex][/*][/m][m3][*][ex]尔爱其羊, 我爱其礼 тебе дорога овца, мне же дорог этикет (ритуал)[/ex][/*][/m][m2]3) беречь, оберегать; щадить; экономить; дорожить, ценить[/m][m3][*][ex]爱公物 беречь общественное имущество[/ex][/*][/m][m3][*][ex]爱名誉 дорожить славой (репутацией)[/ex][/*][/m][m2]4) легко (быстро, скоро) поддаваться, ([i]чему-л.[/i]), быть склонным[/m][m3][*][ex]他爱害臊 очень застенчив[/ex][/*][/m][m3][*][ex]这个东西爱坏 эта вещь легко ломается ([i]очень хрупка[/i])[/ex][/*][/m][m3][*][ex]天冷, 花就爱死 в холодную погоду цветы быстро вянут[/ex][/*][/m][m2]5) жалеть ([i]что-л.[/i]), быть жадным до ([i]чего-л.[/i]); скупиться; скупой, жадный[/m][m3][*][ex]不爱钱 не скупиться на деньги, не быть жадным до денег[/ex][/*][/m][m3][*][ex]百姓皆以王为爱也 весь народ считает вас, государь, скупым[/ex][/*][/m][m1][b]II[/b] [p]сущ.[/p][/m][m2]1) любовь, страсть; привязанность; любимый, дорогой, милый[/m][m3][*][ex]吾爱 любовь моя ([i]обращение[/i])[/ex][/*][/m][m3][*][ex]幸爱 фаворит, любимец[/ex][/*][/m][m2]2) * доброта, милость, благодеяние, великодушие, милосердие, сострадание, гуманность[/m][m3][*][ex]古之遗爱也 \[обладать\] великодушием, наследованным от древних[/ex][/*][/m][m2]3) [p]будд.[/p] тяга (страсть) к мирскому; жажда мирских благ[/m][m2]4) * [p]вм.[/p] 嫒 ([i]\[Ваша\] дочь[/i])[/m][m1][b]III [/b] [p]вводн. слово[/p][/m][m2]* [p]вм.[/p] 僾 ([i]как будто, словно[/i])[/m][m1][b]IV[/b] [p]собств.[/p][/m][m2]1) [p]сокр.[/p] Эйре, Ирландия[/m][m2]2) Ай ([i]фамилия[/i])[/m] +爱乐 + àiyuè + [m1]любить музыку, филармонический[/m][m2][*][ex]柏林爱乐音乐厅 Берлинская филармония[/ex][/*][/m] +爱人 + àiren, àirén + [m1][b]I, àiren[/b][/m][m2]1) муж, жена[/m][m2]2) возлюбленный, возлюбленная[/m][m2]3) любовник, любовница[/m][m1][b]II, ài rén[/b][/m][m2]любить людей[/m] +爱他主义 + àitāzhǔyì + [m1]альтруизм[/m] +爱克司射线 + àikèsīshèxiàn + [m1][p]физ.[/p] Х-лучи, рентген[/m] +爱克斯射线 + àikèsīshèxiàn + [m1][p]физ.[/p] Х-лучи, рентген[/m] +爱冷 + àilěng + [m1]любить холод; морозоустойчивый[/m] +爱别离苦 + àibiélíkǔ + [m1][p]будд.[/p] страдания разлуки \[с любимым человеком\] ([i]одно из восьми страданий,[/i] 八苦)[/m] +爱力 + àilì + [m1]1)[c][i] хим.[/i][/c] сродство[/m][m1]2) беречь силы (энергию)[/m] +爱堉 + àiyù + [m1]любимый зять, зятёк[/m] +爱女 + àinǚ + [m1]1) любимая дочь; дочь-любимица[/m][m1]2) 贪爱女色[/m] +爱好 + àihào, àihǎo + [m1]I [b]àihào[/b][/m][m2]1) любить; иметь влечение к...; симпатизировать, симпатия[/m][m3][*][ex]爱好和平 миролюбивый[/ex][/*][/m][m2]2) хобби, любимое занятие; увлечение, пристрастия, вкусы[/m][m1]II [b]àihǎo[/b][/m][m2]1) [p]диал.[/p] любить хорошо выглядеть, любить хорошо одеваться[/m][m2]2) по-хорошему, полюбовно[/m][m3][*][ex]爱好做亲 жить вместе по-дружески[/ex][/*][/m] +爱妻 + àiqī + [m1]1) любимая жена[/m][m1]2) любить жену[/m] +爱少 + àishào + [m1]любить детей (молодое поколение); любовь к детям (молодёжи)[/m] +爱尔兰自治 + àiěrlánzìzhì + [m1][c][i]полит.[/c],[c] ист.[/i][/c] гомруль[/m] +爱尚 + àishàng + [m1]ценить, отдавать предпочтение, предпочитать, любить; нравится ([c][i]что-л.[/i][/c])[/m] +爱己 + àijǐ + [m1]любить себя; себялюбивый, эгоистичный; самовлюблённый[/m] +爱巴物儿 + àibāwùr + [m1]любимая вещь (игрушка, безделушка)[/m] +爱幸 + àixìng + [m1]1) фаворит, любимец ([i]государя[/i])[/m][m1]2) любить, жаловать; быть привязанным к ([c][i]кому-л.[/i][/c])[i],[/i] привязаться; любовь, привязанность[/m] +爱异 + àiyì + [m1]любить и отличать, сильно любить и предпочитать другим; сильная любовь[/m] +爱情 + àiqíng + [m1]1) любовь, чувство любви; привязанность[/m][m1]2) мелодрама (жанр)[/m] +爱惠 + àihuì + [m1]1) благодеяние, милость, расположение[/m][m1]2) быть тронутым ([i]чьей-л.[/i]) добротой (милостью)[/m] +爱意 + àiyì + [m1]1) с радостью (охотно) соглашаться; желать[/m][m1]2) любовь, чувство любви[/m][m2][*][ex]他向她表白了爱意 он объяснился ей в любви[/ex][/*][/m] +爱慕 + àimù + [m1]1) радоваться и восхищаться; восхищённый[/m][m1]2) нравиться; любить; иметь чувства[/m] +爱护 + àihù + [m1]заботиться, ухаживать, беречь, ревниво оберегать, лелеять, пестовать, любить ([c][i]напр.[/c] родину[/i])[i],[/i] ревностно присматривать за ([c][i]кем-л.[/i][/c]); сбережение, уход[/m][m2][*][ex]爱护环境 беречь окружающую среду[/ex][/*][/m][m2][*][ex]爱护花草树木 ухаживать за растениями[/ex][/*][/m][m2][*][ex]爱护老年人 заботиться о стариках[/ex][/*][/m] +爱服盖独氏定律 + àifúgàidúshìdìnglǜ + [m1][p]хим.[/p] закон Авогадро[/m] +爱根 + àigēn + [m1][p]будд.[/p] корень желаний, основа страстей ([i]как причина всех бед[/i])[/m] +爱焰 + àiyàn + [m1][p]см.[/p] 爱火 2)[/m] +爱照 + àizhào + [m1]любовно заботиться[/m] +爱玛克 + àimǎkè + [m1][p]монг.[/p] аймак[/m] +爱玩 + àiwán + [m1]2) любить развлекаться; предаваться любимым забавам; питать страсть, склонность; любимый конёк[/m][m1]1) вести любовную игру, забавляться любовью, флиртовать[/m] +爱神 + àishén + [m1][p]миф.[/p] бог любви, Амур, Купидон; Эрот; Эрос[/m] +爱神星 + àishénxīng + [m1]Эрос, Эрот ([i]околоземный астероид из группы Амура[/i])[/m] +爱美 + àiměi + [m1]1) любить красоту (прекрасное); быть эстетом[/m][m1]2) любить наряжаться[/m] +爱美剧 + àiměijù + [m1]1) любительское представление, любительский спектакль[/m][m2][*][ex]爱美剧运动 движение за обновление театра через любительские постановки ([i]Китай, 20-е годы XX в.[/i])[/ex][/*][/m][m1]2) [b]ài měijù[/b] любить американские телесериалы[/m] +爱视 + àishì + [m1]благоволить, отдавать предпочтение, быть расположенным (к[c][i] кому-л.[/i][/c]); расположение, предпочтение, благоволение[/m] +爱说 + àiyuè + [m1]любить; нравится ([c][i]что-л.[/i][/c]); быть расположенным к; плениться, наслаждаться ([i]чем-л.[/i])[/m] +瑷珲条约 + àihún tiáoyuē + [m1]Айгунский договор, Айгуньский договор ([i]1858 г.[/i])[/m] +碟仙 + _ + [m1]碟仙：顾名思义就是在碟子上的神仙。这是一种比喻的称谓，阴阳不测为之神·《易经》，人们在用碟子占卜时，发现碟子会自我移动，回答问题，变化莫测，因为不知道其中的原因，故称为碟仙。[/m] +薆薆 + àiài + [m1]1) пышный, густой[/m][m1]2) тенистый; скрытый[/m] +薆障 + àizhàng + [m1]закрывать, заслонять, скрывать[/m] +铃片架 + língpiànjià + [m1]стойка для блинов [i](штанги)[/i][/m] +雨密度 + yǔmìdù + [m1][p]метео.[/p] плотность дождя[/m] diff --git a/Tests/HanReaderCoreTests/Fixtures/bkrs-slice.manifest.txt b/Tests/HanReaderCoreTests/Fixtures/bkrs-slice.manifest.txt new file mode 100644 index 0000000..584b873 --- /dev/null +++ b/Tests/HanReaderCoreTests/Fixtures/bkrs-slice.manifest.txt @@ -0,0 +1,208 @@ +Why each card is in bkrs-slice.dsl.txt. + +Generated by Scripts/extract-dsl-fixture.py; do not hand-edit. DSL has no +comment syntax, so this cannot live inside the fixture itself. + +一 57 senses; carries its own tone-sandhi note as prose +一二 escaped \[ — 55,414 in the set; stripDSL ate them +一般性讨论 consecutive run after 爱 +三摩呾咤 a [p] field marker rather than a part of speech +上海市 [ref] cross-reference +乐芙兰 article spans two lines inside an open [m1] +了 two readings, le and liǎo; 30 senses +代表国家集团 consecutive run after 爱 +僾 consecutive run after 爱 +单宁植物 consecutive run after 爱 +叆靅 consecutive run after 爱 +后不久 consecutive run after 爱 +后不圆音 consecutive run after 爱 +后乘 consecutive run after 爱 +后代 consecutive run after 爱 +后劲 consecutive run after 爱 +后勾儿 consecutive run after 爱 +后半夜儿 consecutive run after 爱 +后半晌儿 consecutive run after 爱 +后半期 consecutive run after 爱 +后卫 consecutive run after 爱 +后厦 consecutive run after 爱 +后咎 consecutive run after 爱 +后堂 consecutive run after 爱 +后备军 consecutive run after 爱 +后天性免疫 consecutive run after 爱 +后夫 consecutive run after 爱 +后头叶 consecutive run after 爱 +后奏曲 consecutive run after 爱 +后婚 consecutive run after 爱 +后学 consecutive run after 爱 +后尘 consecutive run after 爱 +后尾 consecutive run after 爱 +后席 consecutive run after 爱 +后序 consecutive run after 爱 +后庭 consecutive run after 爱 +后开 consecutive run after 爱 +后得智 consecutive run after 爱 +后心 consecutive run after 爱 +后怕 consecutive run after 爱 +后成河 consecutive run after 爱 +后房水 consecutive run after 爱 +后手儿 consecutive run after 爱 +后把拉 consecutive run after 爱 +后拒 consecutive run after 爱 +后接词 consecutive run after 爱 +后支索 consecutive run after 爱 +后日 consecutive run after 爱 +后期 consecutive run after 爱 +后来 consecutive run after 爱 +后果 consecutive run after 爱 +后槽 consecutive run after 爱 +后檐 consecutive run after 爱 +后檐墙 consecutive run after 爱 +后此 consecutive run after 爱 +后殿 consecutive run after 爱 +后淋 consecutive run after 爱 +后溲 consecutive run after 爱 +后灯 consecutive run after 爱 +后熟 consecutive run after 爱 +后父 consecutive run after 爱 +后生可畏 consecutive run after 爱 +后甲板 consecutive run after 爱 +后知 consecutive run after 爱 +后继 consecutive run after 爱 +后缀 consecutive run after 爱 +后置性 consecutive run after 爱 +后肠 consecutive run after 爱 +后脑 consecutive run after 爱 +后脑杓子 consecutive run after 爱 +后腰 consecutive run after 爱 +后腿儿 consecutive run after 爱 +后膛 consecutive run after 爱 +后臼齿 consecutive run after 爱 +后艰 consecutive run after 爱 +后街 consecutive run after 爱 +后补 consecutive run after 爱 +后裔 consecutive run after 爱 +后见人 consecutive run after 爱 +后赶着儿 consecutive run after 爱 +后起之秀 consecutive run after 爱 +后足 consecutive run after 爱 +后跟 consecutive run after 爱 +后身 consecutive run after 爱 +后车 consecutive run after 爱 +后轴 consecutive run after 爱 +后辈儿 consecutive run after 爱 +后边儿 consecutive run after 爱 +后进 consecutive run after 爱 +后送 consecutive run after 爱 +后部 consecutive run after 爱 +后锯筋 consecutive run after 爱 +后队 consecutive run after 爱 +后院 consecutive run after 爱 +后难 consecutive run after 爱 +后音儿 consecutive run after 爱 +后韵 consecutive run after 爱 +后项 consecutive run after 爱 +后顾之忧 consecutive run after 爱 +后首里 consecutive run after 爱 +嗳 consecutive run after 爱 +土耳其斯坦 run-together reading with a U+2019 syllable boundary +处世 consecutive run after 爱 +处之泰然 consecutive run after 爱 +处事 consecutive run after 爱 +处以 consecutive run after 爱 +处决 consecutive run after 爱 +处刑 consecutive run after 爱 +处办 consecutive run after 爱 +处务 consecutive run after 爱 +处变 consecutive run after 爱 +处和 consecutive run after 爱 +处境 consecutive run after 爱 +处士 consecutive run after 爱 +处处 consecutive run after 爱 +处女 consecutive run after 爱 +处女官 consecutive run after 爱 +处女带 consecutive run after 爱 +处女性 consecutive run after 爱 +处女林 consecutive run after 爱 +处女航 consecutive run after 爱 +处子 consecutive run after 爱 +处己 consecutive run after 爱 +处心积虑 consecutive run after 爱 +处所 consecutive run after 爱 +处断 consecutive run after 爱 +处方 consecutive run after 爱 +处方笺 consecutive run after 爱 +处治 consecutive run after 爱 +处理品 consecutive run after 爱 +处置 consecutive run after 爱 +处舍 consecutive run after 爱 +处裁 consecutive run after 爱 +处身 consecutive run after 爱 +夜 consecutive run after 爱 +夜合 consecutive run after 爱 +夜学 consecutive run after 爱 +夜客 consecutive run after 爱 +夜春 consecutive run after 爱 +夜格 consecutive run after 爱 +夜泣 consecutive run after 爱 +夜猫子 consecutive run after 爱 +夜盲 consecutive run after 爱 +夜直 consecutive run after 爱 +夜郎自大 consecutive run after 爱 +夜静 consecutive run after 爱 +够面子 consecutive run after 爱 +媚 [m4]: 344 in the set, the deepest level used +布里斯托尔 consecutive run after 爱 +帕氏刺隆头鱼 consecutive run after 爱 +打 57 senses, the deepest nesting in the set +打睡一觉 consecutive run after 爱 +操弄 consecutive run after 爱 +支农工业 consecutive run after 爱 +改性橡碗栲胶 consecutive run after 爱 +有害做法 consecutive run after 爱 +杠铃架 consecutive run after 爱 +柴郡 consecutive run after 爱 +栓 [*] with no [ex] inside — 62 in the set, and that path must not crash +滥赏 [c brown]: the only tag taking an argument +爱 division + numbered senses + example; the plan's worked example +爱乐 consecutive run after 爱 +爱人 consecutive run after 爱 +爱他主义 consecutive run after 爱 +爱克司射线 consecutive run after 爱 +爱克斯射线 consecutive run after 爱 +爱冷 consecutive run after 爱 +爱别离苦 consecutive run after 爱 +爱力 consecutive run after 爱 +爱堉 consecutive run after 爱 +爱女 consecutive run after 爱 +爱好 consecutive run after 爱 +爱妻 consecutive run after 爱 +爱少 consecutive run after 爱 +爱尔兰自治 consecutive run after 爱 +爱尚 consecutive run after 爱 +爱己 consecutive run after 爱 +爱巴物儿 consecutive run after 爱 +爱幸 consecutive run after 爱 +爱异 consecutive run after 爱 +爱情 consecutive run after 爱 +爱惠 consecutive run after 爱 +爱意 consecutive run after 爱 +爱慕 consecutive run after 爱 +爱护 consecutive run after 爱 +爱服盖独氏定律 consecutive run after 爱 +爱根 consecutive run after 爱 +爱焰 consecutive run after 爱 +爱照 consecutive run after 爱 +爱玛克 consecutive run after 爱 +爱玩 consecutive run after 爱 +爱神 consecutive run after 爱 +爱神星 consecutive run after 爱 +爱美 consecutive run after 爱 +爱美剧 consecutive run after 爱 +爱视 consecutive run after 爱 +爱说 consecutive run after 爱 +瑷珲条约 consecutive run after 爱 +碟仙 `_` for the reading — 77% of the set +薆薆 consecutive run after 爱 +薆障 consecutive run after 爱 +铃片架 consecutive run after 爱 +雨密度 consecutive run after 爱 From dce6ffc1a8d8cac124111e6954b2af684eb76ebe Mon Sep 17 00:00:00 2001 From: Dmitry Ilyin <6576495+widgetii@users.noreply.github.com> Date: Mon, 5 Oct 2026 00:06:17 +0300 Subject: [PATCH 2/3] fix: detect the reading line, and stop losing it quietly MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Six correctness fixes raised in review, each measured against the full 3,434,224-card set. **A reading line is a convention, not a rule of the format.** The card reader took the first indented line as the reading unconditionally, so a DSL dictionary that writes no readings — the format does not require them — lost the first line of every article to a field it does not have, and `DSLCardBuilder` received a card with no senses in it. It is now detected, by a rule the whole set agrees with: the first indented line is the reading unless it contains a `[mN]` tag. **Zero of 3,434,224 reading lines do**, and an article's first line effectively always does. The test has to be the sense tag specifically rather than markup in general, because 287 reading lines *do* carry other markup — `lùzhóu, [i]разг.[/i] liùzhóu`, a register note between two readings — and a markup test would have read all 287 as articles. A single indented line is now article text rather than a reading. That case is genuinely ambiguous and the two mistakes are not equal: a reading with no definition is not an entry worth having, while a definition read as a reading leaves the entry saying nothing at all. No BKRS card has one, so nothing in this set changes. **A card completed across a file seam offered offset zero as its resume point**, which restarts at the next file's byte-order mark and loses the carried headword entirely. `DSLCardRecord.offset` is now optional and nil for exactly that card: no offset in the file it was completed in restarts it, and the one that does is in the previous file, which an importer already has. **Each dangling headword now reports its own line.** A run of them at the end of a file all carried the first one's offset, pointing every diagnostic but the first at the wrong place in the file — worse than no offset at all. **Discovery says when it stops at its file limit.** Returning 64 volumes of a 65-volume dictionary as though it were complete is the worst of the three possible behaviours: it imports, it looks right, and the missing volume's words simply cannot be found. Both the include walk and the numbered-sibling scan now report `tooManyFiles` instead. **A directive may be separated by a tab.** `#INCLUDE\t"b.dsl"` was read as an unknown directive, which leaves the include list empty and the other volumes undiscovered — the one failure that turns a three-volume dictionary into a third of one with no error at all. **A file that ends before the length it reported is refused.** `fill()` took a zero-byte read as end of file even when the file said there was more, so a source shortened between being measured and being read produced a dictionary quietly missing its tail — and in UTF-16, a final half-character dropped without the odd-length check ever running. Counts over the whole set are unchanged: 3,434,224 cards, 1 multi-line article, 0 trailing headwords, 0 diagnostics. Co-Authored-By: Claude Opus 5 --- .../HanReaderCore/Format/DSLByteReader.swift | 14 ++ .../HanReaderCore/Format/DSLCardReader.swift | 99 ++++++++++--- .../HanReaderCore/Format/DSLDiagnostic.swift | 3 + Sources/HanReaderCore/Format/DSLFileSet.swift | 47 ++++++- .../DSLByteReaderTests.swift | 35 +++++ .../DSLCardReaderTests.swift | 133 ++++++++++++++++-- 6 files changed, 297 insertions(+), 34 deletions(-) diff --git a/Sources/HanReaderCore/Format/DSLByteReader.swift b/Sources/HanReaderCore/Format/DSLByteReader.swift index 0e23f51..fda544c 100644 --- a/Sources/HanReaderCore/Format/DSLByteReader.swift +++ b/Sources/HanReaderCore/Format/DSLByteReader.swift @@ -52,6 +52,9 @@ public enum DSLByteError: Error, Hashable, Sendable, CustomStringConvertible { /// A resume offset past the end of the file, which is what a stale /// checkpoint against a replaced file looks like. case offsetOutOfRange(offset: Int, byteCount: Int) + /// The file ended before the length it reported, so it was shortened + /// while being read. + case truncated(read: Int, expected: Int) /// The file could not be opened or measured. case cannotOpen(path: String, code: Int32) /// A read failed part-way through the file. @@ -66,6 +69,9 @@ public enum DSLByteError: Error, Hashable, Sendable, CustomStringConvertible { "byte offset \(offset) is odd, so it falls inside a UTF-16 code unit" case let .offsetOutOfRange(offset, byteCount): "byte offset \(offset) is past the end of a \(byteCount)-byte file" + case let .truncated(read, expected): + "the file ended after \(read) bytes but reported \(expected) — it was " + + "shortened while being read" case let .cannotOpen(path, code): "cannot open \(path): \(String(cString: strerror(code)))" case let .readFailed(offset, code): @@ -343,6 +349,14 @@ public struct DSLByteReader { while available < chunk.count { let read = try source.read(at: nextOffset, into: &chunk, from: available) guard read > 0 else { + // End of file where the file said there was more. Something + // shortened it since it was measured, and the alternative to + // saying so is a dictionary quietly missing its tail — or, in + // UTF-16, a final half-character dropped without the + // odd-length check ever running. + guard nextOffset >= byteCount else { + throw DSLByteError.truncated(read: nextOffset, expected: byteCount) + } reachedEnd = true break } diff --git a/Sources/HanReaderCore/Format/DSLCardReader.swift b/Sources/HanReaderCore/Format/DSLCardReader.swift index 31addb0..d0532cd 100644 --- a/Sources/HanReaderCore/Format/DSLCardReader.swift +++ b/Sources/HanReaderCore/Format/DSLCardReader.swift @@ -34,7 +34,10 @@ public struct DSLHeader: Hashable, Sendable { /// empty string, which is distinguishable from absent. mutating func consume(directive line: String) { let rest = line.dropFirst() - let split = rest.firstIndex(of: " ") + // Any horizontal whitespace, not a literal space. `#INCLUDE\t"b.dsl"` + // is a legal directive, and reading it as an unknown one leaves the + // include list empty and the other volumes undiscovered. + let split = rest.firstIndex { $0 == " " || $0 == "\t" } let directive = String(split.map { rest[rest.startIndex ..< $0] } ?? rest) let rawValue = split.map { String(rest[rest.index(after: $0)...]) } ?? "" let value = Self.unquote(rawValue.trimmingCharacters(in: .whitespaces)) @@ -57,14 +60,21 @@ public struct DSLHeader: Hashable, Sendable { /// One card, and where it starts. public struct DSLCardRecord: Hashable, Sendable { public let card: DSLCard - /// Byte offset of the card's first headword line. + /// Byte offset of the card's first headword line, or nil when the card has + /// no resume point in this file. /// - /// The resume point. It is recorded at a card boundary rather than at a - /// line or a batch boundary because that is the only place the importer can - /// restart without either losing a card or writing one twice. - public let offset: Int + /// The resume point is a card boundary rather than a line or a batch + /// boundary because that is the only place an importer can restart without + /// either losing a card or writing one twice. + /// + /// Nil for one case: a card whose headword was carried in from the + /// previous file of the set. No offset in *this* file restarts at it, and + /// an importer that checkpointed there would resume past the headword and + /// lose the card. A reader that needs a resume point for such a card wants + /// the previous file's, which it already has. + public let offset: Int? - public init(card: DSLCard, offset: Int) { + public init(card: DSLCard, offset: Int?) { self.card = card self.offset = offset } @@ -119,8 +129,11 @@ public struct DSLCardReader { private var lines: DSLByteReader private var headwords: [String] + /// Byte offset of each headword line, parallel to `headwords`. Shorter + /// than it when headwords were carried in from the previous file, which is + /// how a seam-completed card knows it has no resume point of its own. + private var headwordOffsets: [Int] = [] private var body: [String] = [] - private var cardOffset = 0 public init( source: any DSLByteSource, @@ -134,7 +147,6 @@ public struct DSLCardReader { chunkSize: chunkSize, ) headwords = leadingHeadwords - cardOffset = offset ?? 0 } /// How the file spells its characters. @@ -173,13 +185,11 @@ public struct DSLCardReader { guard body.isEmpty else { let record = takeCard() headwords = [text] - cardOffset = line.offset + headwordOffsets = [line.offset] return record } - if headwords.isEmpty { - cardOffset = line.offset - } headwords.append(text) + headwordOffsets.append(line.offset) } return finalCard() } @@ -187,33 +197,86 @@ public struct DSLCardReader { // MARK: - Cards private mutating func takeCard() -> DSLCardRecord { + let split = Self.readingLineCount(in: body) let record = DSLCardRecord( card: DSLCard( headwords: headwords, - pinyin: body.first ?? "", - body: Array(body.dropFirst()), + pinyin: split == 1 ? body[0] : "", + body: Array(body.dropFirst(split)), ), - offset: cardOffset, + // Nil when the headwords came from the previous file: there is no + // offset in *this* file that would restart at this card, and + // offering zero would restart at the file's byte-order mark and + // lose the headword entirely. + offset: headwordOffsets.count == headwords.count ? headwordOffsets.first : nil, ) headwords.removeAll(keepingCapacity: true) + headwordOffsets.removeAll(keepingCapacity: true) body.removeAll(keepingCapacity: true) return record } + /// Whether the article's first line is a reading, as 1 or 0. + /// + /// A reading line is a convention of Chinese dictionaries rather than a + /// rule of the format, so it is detected rather than assumed. Two + /// conditions, both measured against all 3,434,224 BKRS cards: + /// + /// - **It contains no `[mN]`.** Zero reading lines in the set do, and an + /// article's first line effectively always does. 287 reading lines *do* + /// carry other markup — `lùzhóu, [i]разг.[/i] liùzhóu` — so the test has + /// to be the sense tag specifically and not markup in general. + /// - **Something follows it.** A card with one indented line and no + /// markup is genuinely ambiguous, and the two mistakes are not equal: a + /// reading with no definition is not an entry worth having, while a + /// definition read as a reading leaves the entry saying nothing at all. + /// + /// Without the first condition a dictionary that writes no readings — the + /// format does not require them — loses its first line of every article. + static func readingLineCount(in body: [String]) -> Int { + guard body.count > 1, let first = body.first, !containsSenseTag(first) else { return 0 } + return 1 + } + + /// Whether a line opens a sense. Scans scalars, as all markup must: `]` + /// followed by a zero-width non-joiner is a single `Character`. + private static func containsSenseTag(_ line: String) -> Bool { + let scalars = Array(line.unicodeScalars) + guard scalars.count >= 4 else { return false } + for index in 0 ... (scalars.count - 4) { + guard scalars[index] == "[", scalars[index + 1] == "m", + scalars[index + 2].properties.numericType != nil, + scalars[index + 2].isASCII, + scalars[index + 3] == "]" + else { continue } + // An escaped `\[` is a literal bracket, not a tag. + if index > 0, scalars[index - 1] == "\\" { + continue + } + return true + } + return false + } + /// At end of file: a complete card, or a dangling headword reported and /// handed to `trailingHeadwords`. private mutating func finalCard() -> DSLCardRecord? { guard !headwords.isEmpty else { return nil } guard !body.isEmpty else { - for headword in headwords { + // Each headword reports its own line. Sharing the first one's + // offset across a run of them points every diagnostic but the + // first at the wrong place in the file. + let carried = headwords.count - headwordOffsets.count + for (index, headword) in headwords.enumerated() { diagnostics.append(DSLDiagnostic( kind: .headwordWithoutBody, text: headword, - offset: cardOffset, + offset: index < carried ? nil : headwordOffsets[index - carried], )) } trailingHeadwords = headwords headwords.removeAll() + headwordOffsets.removeAll() return nil } return takeCard() diff --git a/Sources/HanReaderCore/Format/DSLDiagnostic.swift b/Sources/HanReaderCore/Format/DSLDiagnostic.swift index 9419c9a..095b970 100644 --- a/Sources/HanReaderCore/Format/DSLDiagnostic.swift +++ b/Sources/HanReaderCore/Format/DSLDiagnostic.swift @@ -27,6 +27,9 @@ public struct DSLDiagnostic: Hashable, Sendable { /// An `#INCLUDE` naming a file that is not beside the one including /// it. case missingInclude(String) + /// Discovery stopped at its file limit with files still to follow, so + /// the set it returned is incomplete. + case tooManyFiles(limit: Int) } public let kind: Kind diff --git a/Sources/HanReaderCore/Format/DSLFileSet.swift b/Sources/HanReaderCore/Format/DSLFileSet.swift index 18b9484..f132872 100644 --- a/Sources/HanReaderCore/Format/DSLFileSet.swift +++ b/Sources/HanReaderCore/Format/DSLFileSet.swift @@ -56,7 +56,7 @@ public struct DSLFileSet: Hashable, Sendable { /// Follows at most this many `#INCLUDE` hops. A visited set already stops /// a cycle; this stops a pathological file from being a denial of service. - static let fileLimit = 64 + public static let fileLimit = 64 /// Works out which files belong with the one the reader picked. /// @@ -85,7 +85,18 @@ public struct DSLFileSet: Hashable, Sendable { var diagnostics: [DSLDiagnostic] = [] var queue = header.includes.map { (parent: url, name: $0) } - while !queue.isEmpty, files.count < fileLimit { + while !queue.isEmpty { + guard files.count < fileLimit else { + // Returning 64 volumes of a 65-volume dictionary as + // though it were complete is the worst of the three + // possible behaviours: it imports, it looks right, and the + // missing volume's words simply cannot be found. + diagnostics.append(DSLDiagnostic( + kind: .tooManyFiles(limit: fileLimit), + text: queue.map(\.name).joined(separator: ", "), + )) + break + } let (parent, name) = queue.removeFirst() let target = parent.deletingLastPathComponent() .appendingPathComponent(name) @@ -107,14 +118,25 @@ public struct DSLFileSet: Hashable, Sendable { ) } - let siblings = numberedSiblings(of: url, fileManager: fileManager) + var truncated = false + let siblings = numberedSiblings(of: url, fileManager: fileManager, truncated: &truncated) guard siblings.count > 1 else { return Self(files: [url.standardizedFileURL], header: header, discovery: .single) } if siblings[0] != url.standardizedFileURL { return try discover(from: siblings[0], fileManager: fileManager) } - return Self(files: siblings, header: header, discovery: .numberedSiblings) + return Self( + files: siblings, + header: header, + discovery: .numberedSiblings, + diagnostics: truncated + ? [DSLDiagnostic( + kind: .tooManyFiles(limit: fileLimit), + text: url.lastPathComponent, + )] + : [], + ) } /// Files beside `url` whose names differ only in a trailing number. @@ -123,7 +145,13 @@ public struct DSLFileSet: Hashable, Sendable { /// from 1: a gap ends it, because a set missing its second volume is a set /// of one, not a set of two with a hole. Zero padding is preserved, so /// `vol_01` looks for `vol_02` and not `vol_2`. - static func numberedSiblings(of url: URL, fileManager: FileManager) -> [URL] { + static func numberedSiblings( + of url: URL, + fileManager: FileManager, + truncated: inout Bool, + ) + -> [URL] + { let stem = url.deletingPathExtension().lastPathComponent let digits = stem.reversed().prefix { $0.isASCII && $0.isNumber } guard !digits.isEmpty else { return [url.standardizedFileURL] } @@ -135,7 +163,7 @@ public struct DSLFileSet: Hashable, Sendable { var found: [URL] = [] var number = 1 - while found.count < fileLimit { + while true { let padded = String(number).count >= width ? String(number) : String(repeating: "0", count: width - String(number).count) + String(number) @@ -145,6 +173,13 @@ public struct DSLFileSet: Hashable, Sendable { guard fileManager.fileExists(atPath: candidate.path) else { break } found.append(candidate) number += 1 + // The run is capped for the same reason the include walk is, and + // reported the same way: the caller is told the set is longer than + // this rather than handed a silently short one. + if found.count >= fileLimit { + truncated = true + break + } } // The picked file must be in the run; otherwise the names matched by // accident and it is safer to import only what was asked for. diff --git a/Tests/HanReaderCoreTests/DSLByteReaderTests.swift b/Tests/HanReaderCoreTests/DSLByteReaderTests.swift index e225ff5..308c824 100644 --- a/Tests/HanReaderCoreTests/DSLByteReaderTests.swift +++ b/Tests/HanReaderCoreTests/DSLByteReaderTests.swift @@ -39,6 +39,27 @@ private func lines( return result } +/// A source that reports its full length but stops delivering part way, which +/// is what a file truncated between being measured and being read looks like. +private struct ShrinkingBytes: DSLByteSource { + let bytes: [UInt8] + let delivering: Int + + var byteCount: Int { + bytes.count + } + + func read(at offset: Int, into buffer: inout [UInt8], from destination: Int) throws -> Int { + guard offset >= 0, offset < delivering, destination < buffer.count else { return 0 } + let count = min(buffer.count - destination, delivering - offset) + buffer.replaceSubrange( + destination ..< (destination + count), + with: bytes[offset ..< (offset + count)], + ) + return count + } +} + /// The real shape of a BKRS file: two CRLF header lines, a blank line, then /// LF-terminated cards, ending with a line feed. private let header = "#NAME \"大БКРС\"\r\n#INDEX_LANGUAGE \"Chinese\"\r\n" @@ -182,6 +203,20 @@ struct DSLByteReaderTests { } } + /// A source that reports one length and then delivers less was shortened + /// while being read. Accepting it means a dictionary quietly missing its + /// tail — and in UTF-16, a final half-character dropped without the + /// odd-length check ever running. + @Test("A file that ends early is refused") + func truncatedWhileReading() throws { + let bytes = utf16("爱\n了\n一\n") + let source = ShrinkingBytes(bytes: bytes, delivering: 8) + var reader = try DSLByteReader(source: source, chunkSize: 4) + #expect(throws: DSLByteError.truncated(read: 8, expected: bytes.count)) { + while try reader.next() != nil {} + } + } + @Test("A resume offset past the end is refused") func offsetOutOfRange() throws { let bytes = utf16("a\n") diff --git a/Tests/HanReaderCoreTests/DSLCardReaderTests.swift b/Tests/HanReaderCoreTests/DSLCardReaderTests.swift index 7c974b2..f949d02 100644 --- a/Tests/HanReaderCoreTests/DSLCardReaderTests.swift +++ b/Tests/HanReaderCoreTests/DSLCardReaderTests.swift @@ -66,14 +66,44 @@ struct DSLCardReaderTests { #expect(result.map(\.body) == [["[m1]любить[/m]"], ["[m1]частица[/m]"]]) } - /// The reading is the first indented line and the article is the rest, so - /// a card with no article still has its reading rather than shifting the - /// article into it. - @Test("A card with only a reading line keeps it as the reading") - func readingWithoutArticle() throws { - let result = try cards("爱\n ài\n") - #expect(result.map(\.pinyin) == ["ài"]) - #expect(result[0].body.isEmpty) + // MARK: - Telling a reading line from an article line + + /// A reading line is a convention of Chinese dictionaries, not a rule of + /// the format, so it is detected rather than assumed. A dictionary that + /// writes no readings — which DSL permits — would otherwise lose the first + /// line of every article to a reading field. + @Test("A card whose article opens with a sense tag has no reading") + func articleWithoutReading() throws { + let result = try cards("dog\n [m1]animal[/m]\n [m1]пёс[/m]\n") + #expect(result[0].pinyin.isEmpty) + #expect(result[0].body == ["[m1]animal[/m]", "[m1]пёс[/m]"]) + } + + /// 287 reading lines in the set carry markup — an italic register note + /// between two readings — so the test has to be the sense tag + /// specifically, not markup in general. + @Test("A reading line carrying other markup is still a reading") + func readingWithMarkup() throws { + let result = try cards("陆轴\n lùzhóu, [i]разг.[/i] liùzhóu\n [m1]каток[/m]\n") + #expect(result[0].pinyin == "lùzhóu, [i]разг.[/i] liùzhóu") + #expect(result[0].body == ["[m1]каток[/m]"]) + } + + /// One indented line with no markup is genuinely ambiguous, and the two + /// mistakes are not equal: a reading with no definition is not an entry + /// worth having, while a definition read as a reading leaves the entry + /// saying nothing at all. + @Test("A single indented line is article text, not a reading") + func singleLineIsArticle() throws { + let result = try cards("爱\n любить\n") + #expect(result[0].pinyin.isEmpty) + #expect(result[0].body == ["любить"]) + } + + @Test("An escaped bracket does not look like a sense tag") + func escapedBracketIsNotATag() throws { + let result = try cards("爱\n \\[m1\\] ài\n [m1]любить[/m]\n") + #expect(result[0].pinyin == "\\[m1\\] ài") } /// Exactly one card in the set — 乐芙兰 — breaks a line inside an open @@ -166,7 +196,8 @@ struct DSLCardReaderTests { for record in result { let headword = record.card.headwords[0] let expected = Array(headword.utf16).flatMap { [UInt8($0 & 0xFF), UInt8($0 >> 8)] } - #expect(Array(bytes[record.offset ..< (record.offset + expected.count)]) == expected) + let offset = try #require(record.offset) + #expect(Array(bytes[offset ..< (offset + expected.count)]) == expected) } } @@ -206,6 +237,18 @@ struct DSLCardReaderTests { #expect(reader.header.other == ["ICON_FILE": "x.bmp", "SOUND_DICTIONARY": ""]) } + /// The format separates a directive from its value with whitespace, not + /// with a space specifically. Read as an unknown directive, a + /// tab-separated `#INCLUDE` leaves the include list empty and the other + /// volumes undiscovered. + @Test("A directive separated by a tab is still a directive") + func tabSeparatedDirective() throws { + let reader = try read("#INCLUDE\t\"b.dsl\"\n#NAME\t\"A\"\n\n").reader + #expect(reader.header.includes == ["b.dsl"]) + #expect(reader.header.name == "A") + #expect(reader.header.other.isEmpty) + } + @Test("Reading only the header stops at the first card") func headerOnlyRead() throws { let header = try DSLHeader.read(from: DSLMemoryBytes(utf16(file))) @@ -236,6 +279,39 @@ struct DSLCardReaderTests { #expect(result.reader.diagnostics.map(\.kind) == [.headwordWithoutBody]) } + /// Each headword reports its own line. Sharing the first one's offset + /// across a run of them points every diagnostic but the first at the wrong + /// place in the file, which is worse than no offset at all. + @Test("Each dangling headword reports its own line") + func danglingHeadwordOffsets() throws { + let text = "爱\n ài\n [m1]x[/m]\n了\n一\n" + let result = try read(text) + let bytes = utf16(text) + let offsets = result.reader.diagnostics.compactMap(\.offset) + #expect(offsets.count == 2) + #expect(offsets[0] != offsets[1]) + for (offset, headword) in zip(offsets, ["了", "一"]) { + let expected = Array(headword.utf16).flatMap { [UInt8($0 & 0xFF), UInt8($0 >> 8)] } + #expect(Array(bytes[offset ..< (offset + expected.count)]) == expected) + } + } + + /// A card completed across a file seam has no resume point in the file it + /// was completed in: offset zero there restarts at the byte-order mark and + /// loses the carried headword entirely. + @Test("A card completed across a seam offers no resume point") + func seamCardHasNoOffset() throws { + let result = try read( + " le\n [m1]частица[/m]\n一\n yī\n [m1]один[/m]\n", + leadingHeadwords: ["了"], + ) + #expect(result.records.count == 2) + #expect(result.records[0].card.headwords == ["了"]) + #expect(result.records[0].offset == nil) + // The card that does start in this file keeps its offset. + #expect(result.records[1].offset != nil) + } + /// The repair: the headword left over from the previous file is passed in, /// and the article that opens this one completes its card. @Test("A headword carried in from the previous file completes its card") @@ -253,7 +329,7 @@ struct DSLCardReaderTests { @Suite("DSL file-set discovery") struct DSLFileSetTests { /// Writes a throwaway directory of DSL files. - private func directory( + func directory( _ files: [String: String], _ body: (URL) throws -> Void, ) throws { @@ -381,6 +457,43 @@ struct DSLFileSetTests { } } + /// Returning 64 volumes of a 65-volume dictionary as though it were + /// complete is the worst of the three possible behaviours: it imports, it + /// looks right, and the missing volume's words simply cannot be found. + @Test("Discovery says so when it stops at its file limit") + func tooManyIncludes() throws { + var files: [String: String] = [:] + let includes = (1 ... DSLFileSet.fileLimit + 2) + .map { "#INCLUDE \"v\($0).dsl\"\n" } + .joined() + files["a.dsl"] = "#NAME \"A\"\n" + includes + "\n爱\n ài\n [m1]x[/m]\n" + for index in 1 ... DSLFileSet.fileLimit + 2 { + files["v\(index).dsl"] = "#NAME \"\(index)\"\n\n了\n le\n [m1]y[/m]\n" + } + try directory(files) { root in + let set = try DSLFileSet.discover(from: root.appendingPathComponent("a.dsl")) + #expect(set.files.count == DSLFileSet.fileLimit) + #expect(set.diagnostics.contains { + $0.kind == .tooManyFiles(limit: DSLFileSet.fileLimit) + }) + } + } + + @Test("A numbered run that hits the limit says so too") + func tooManySiblings() throws { + var files: [String: String] = [:] + for index in 1 ... DSLFileSet.fileLimit + 2 { + files["vol_\(index).dsl"] = "#NAME \"\(index)\"\n\n了\n le\n [m1]y[/m]\n" + } + try directory(files) { root in + let set = try DSLFileSet.discover(from: root.appendingPathComponent("vol_1.dsl")) + #expect(set.files.count == DSLFileSet.fileLimit) + #expect(set.diagnostics.contains { + $0.kind == .tooManyFiles(limit: DSLFileSet.fileLimit) + }) + } + } + @Test("A name with no trailing number stands alone") func noNumber() throws { try directory([ From 212474f59c8ab38b2125caff3cfd2e31c33d9dbd Mon Sep 17 00:00:00 2001 From: Dmitry Ilyin <6576495+widgetii@users.noreply.github.com> Date: Mon, 5 Oct 2026 00:22:08 +0300 Subject: [PATCH 3/3] refactor: lift the include walk out of discovery MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `discover` was doing three things in one body — reading the header, walking the `#INCLUDE` graph, and scanning for numbered siblings — and the walk is the part with the loop, the visited set and the file limit in it. Moved to its own function, which is also what gets it back under the length the linter enforces. No behaviour change. Co-Authored-By: Claude Opus 5 --- Sources/HanReaderCore/Format/DSLFileSet.swift | 89 ++++++++++++------- 1 file changed, 55 insertions(+), 34 deletions(-) diff --git a/Sources/HanReaderCore/Format/DSLFileSet.swift b/Sources/HanReaderCore/Format/DSLFileSet.swift index f132872..17b5949 100644 --- a/Sources/HanReaderCore/Format/DSLFileSet.swift +++ b/Sources/HanReaderCore/Format/DSLFileSet.swift @@ -80,41 +80,11 @@ public struct DSLFileSet: Hashable, Sendable { let header = try DSLHeader.read(from: DSLFileBytes(url: url)) if !header.includes.isEmpty { - var files = [url.standardizedFileURL] - var visited: Set = [url.standardizedFileURL] - var diagnostics: [DSLDiagnostic] = [] - var queue = header.includes.map { (parent: url, name: $0) } - - while !queue.isEmpty { - guard files.count < fileLimit else { - // Returning 64 volumes of a 65-volume dictionary as - // though it were complete is the worst of the three - // possible behaviours: it imports, it looks right, and the - // missing volume's words simply cannot be found. - diagnostics.append(DSLDiagnostic( - kind: .tooManyFiles(limit: fileLimit), - text: queue.map(\.name).joined(separator: ", "), - )) - break - } - let (parent, name) = queue.removeFirst() - let target = parent.deletingLastPathComponent() - .appendingPathComponent(name) - .standardizedFileURL - guard fileManager.fileExists(atPath: target.path) else { - diagnostics.append(DSLDiagnostic(kind: .missingInclude(name), text: name)) - continue - } - guard visited.insert(target).inserted else { continue } - files.append(target) - let nested = try DSLHeader.read(from: DSLFileBytes(url: target)) - queue += nested.includes.map { (parent: target, name: $0) } - } - return Self( - files: files, + return try following( + header.includes, + from: url, header: header, - discovery: .includeDirectives, - diagnostics: diagnostics, + fileManager: fileManager, ) } @@ -139,6 +109,57 @@ public struct DSLFileSet: Hashable, Sendable { ) } + /// Walks the `#INCLUDE` graph breadth-first from the master file. + /// + /// A visited set rather than a depth limit, so a cycle cannot loop; the + /// file limit is a separate guard against a pathological file, and + /// reaching it is reported rather than hidden. + private static func following( + _ includes: [String], + from url: URL, + header: DSLHeader, + fileManager: FileManager, + ) throws + -> Self + { + var files = [url.standardizedFileURL] + var visited: Set = [url.standardizedFileURL] + var diagnostics: [DSLDiagnostic] = [] + var queue = includes.map { (parent: url, name: $0) } + + while !queue.isEmpty { + guard files.count < fileLimit else { + // Returning 64 volumes of a 65-volume dictionary as though it + // were complete is the worst of the three possible + // behaviours: it imports, it looks right, and the missing + // volume's words simply cannot be found. + diagnostics.append(DSLDiagnostic( + kind: .tooManyFiles(limit: fileLimit), + text: queue.map(\.name).joined(separator: ", "), + )) + break + } + let (parent, name) = queue.removeFirst() + let target = parent.deletingLastPathComponent() + .appendingPathComponent(name) + .standardizedFileURL + guard fileManager.fileExists(atPath: target.path) else { + diagnostics.append(DSLDiagnostic(kind: .missingInclude(name), text: name)) + continue + } + guard visited.insert(target).inserted else { continue } + files.append(target) + let nested = try DSLHeader.read(from: DSLFileBytes(url: target)) + queue += nested.includes.map { (parent: target, name: $0) } + } + return Self( + files: files, + header: header, + discovery: .includeDirectives, + diagnostics: diagnostics, + ) + } + /// Files beside `url` whose names differ only in a trailing number. /// /// `dabkrs_1.dsl` → `dabkrs_2.dsl`, `dabkrs_3.dsl`. The run is contiguous