Skip to content

improvements.md

Latest commit

 

History

History
648 lines (569 loc) · 40.9 KB

improvements.md

File metadata and controls

648 lines (569 loc) · 40.9 KB
Raw

Morpheus: measured improvements since the fork

Author of the improvements: John Baima (QuantForgeSoftware) — john.baima@quantforge.software Report compiled 2026-10-14 from the round records in completed.md, python/README.md, and fresh runs on the current build; Round 10 additions appended 2026-10-15; Round 11 (AGLDT pooling) appended 2026-10-15; Round 14 (entity extraction) appended 2026-10-15; Round 15 (learned + adaptive defaults), Round 16 (recovery pre-pass), Round 17 (recovery tail pass + PapyGreek gate entry) and Round 18 (canonical-lemma scoring + learned retrain) appended 2026-10-15; Round 19 (PG error audit, adaptive numerals, Unicode-native C input, MLP head-to-head) appended 2026-10-16.

This file summarizes how much the analyzer improved, after the fork, when parsing different Greek texts. It is an estimate built from measurements, not a claim of absolute accuracy; the caveats at the end matter.

What "before" means

The fork point is the stock Perseus analyzer (perseids-tools/morpheus). Before this work:

  • It accepted only lowercase beta code and printed Perseus-format strings. Fed a real Unicode Greek text, it echoes the token and produces no analysis at all (verified: θεοῦ and ΛΟΓΟΣ in → nothing out; qeo/s in → a parse).
  • Its candidate analyses were in alphabetical lemma order, so the correct reading was frequently not first.

Two separable gains follow: enablement (from ~0% usable on real text to ~94–100% coverage) and accuracy (picking the right reading among candidates).

Coverage — can the analyzer handle the text at all

Greek text (tokens) Fork baseline Now (default) Now (opt-in flags)
Septuagint / LXX (623,685) 94.85% 100.0% 100.0%
Classical Greek, AGLDT (469,254) 92.8%¹ 99.9% 99.9%
SBLGNT / NT (137,554) 93.7% 95.3% 100% (unknown_as_proper)
Apostolic Fathers (63,222) 90.7% 91.3% 99.35% (-n + elided stems)
PapyGreek papyri (36,455) 89.9% 97.4% 99.4% (--numerals)
PROIEL Greek NT — independent (132,366) — 99.9% —
PROIEL Herodotus — independent (83,819) — 99.8% —

Round 16: the recovery pre-pass is now on by default; the default cells above are the recovery-on measurements — PapyGreek gains the most (+4.5: 92.9% → 97.4%), the literary corpora rise 0.04–0.24pp, the LXX reaches 100%, and the SBLGNT/AF analysis rates reach 99.95%/99.4%. See the Round 16 section below.

¹ Generous "before": that figure is with the new converter but before the elision fix. A pure forked analyzer scores 0% on Unicode AGLDT, since word-final U+0313 (δ̓, ~34K tokens) and Unicode itself both fail. After the fix, Homer goes 90.3% → 100.0%.

Missed tokens overall fell from ~6% of tokens to ~1% by default, or ~0.2–0.7% with the opt-in fallbacks — a ~4–25× reduction in misses — driven by new stemlib material (nom.lxx ~14.8K lines, nom.biblical, nom.papyri, nom.elided) that did not exist at the fork. Since Round 16 the recovery pre-pass (default) cuts the residual further: the LXX and SBLGNT analyze at ~100%, AF at 99.4%, and PapyGreek's default misses fall from 4.9% to 2.6% (0.6% with --numerals).

Accuracy — top-1 lemma (is the gold reading first?)

Greek text (tokens) Before (Morpheus order) After (frequency ranking) Gain Error cut
SBLGNT / NT (137,554)ᵃ 88.1% 94.6% +6.5 55%
Apostolic Fathers, all (63,222)ᵃ 84.8% 90.1% +5.3 35%
— AF, MorphGNT-sourced subset (52,094) 89.8% 95.9% +6.1 60%
Classical Greek, AGLDT (469,254) 79.9% 91.4% +11.5 57%
PROIEL Greek NT — independent (132,366) 78.0% 90.7% +12.7 58%
PROIEL Herodotus — independent (83,819) 79.9% 90.2% +10.3 51%
PapyGreek papyri (36,455) 78.3% 85.9% +7.6 35%
LXX (623,685, corpus-agreement) 90.25% 94.23% +4.0 41%

ᵃ Re-measured 2026-10-14 on the current build. The AF row reproduces the committed README exactly; the SBLGNT row is now better than the README's, which was never re-run after the proper-name additions.

Round 15 update: since the B-item sprint the shipped default is learned re-ranking + adaptive table selection, worth another +7.2 on AGLDT (86.56 → 93.74), +0.5 on PROIEL (91.66 → 92.19), +2.3 on the LXX (→ 90.7) and +4.5 on PapyGreek (→ 85.7) — see the Round 15 section below.

Round 16 update: the recovery pre-pass (default) resolves previously unanalyzed tokens; on PapyGreek it adds +1.8 learned top-1 (87.9 → 89.7 exact) and lifts coverage 95.1% → 99.2%; on the literary corpora it is neutral-to-slightly-positive (+0.02–0.05) — see the Round 16 section below.

Round 17 update: the recovery tail pass (pairwise variants, nasal assimilation, vowel-less numeral rule, three curated nouns) lifts PapyGreek to 99.4% coverage and 89.9 learned exact, with the default path unchanged at 97.4%; the four existing gate corpora are unchanged (additive-only changes) — see the Round 17 section below.

Notes on the before/after:

  • The "after" for SBLGNT / AF / AGLDT / PROIEL-Herodotus is plain frequency ranking with bundled domain tables (lemma_frequencies.json, _af, _grc). The domain table matters for independent texts: on PROIEL Herodotus the classical table beats the Koine one by +4 points (90.2 vs 86.3).
  • The LXX row uses the --dictionary-forms --base-lemmas convention layer (natural output is 90.25%); it measures agreement with the reference corpus, so treat it as a floor.

Opt-in contextual re-ranking (--homographs) adds a further +0.6 to +1.6 points on top, and it transfers to unseen text: SBLGNT 93.6→94.4, AF 89.0→89.7, PapyGreek 83.8→85.4, independent PROIEL 92.4→93.0.

Round 10 additions (2026-10-15)

The tables above use the default frequency path and are unchanged (drift from the new stemlib is within build noise). Round 10 added further measured gains — the learned re-ranker and adaptive table selection (both shipped as defaults in Round 15) plus correctness fixes — all on the current build:

  • Learned re-ranker (now the default; Round 15 numbers below): top-1 lemma on top of frequency ranking — uniformly positive, including out-of-register classical (every AGLDT author).
  • Adaptive domain-table selection (now the default): picks the best-fitting bundled table automatically. On AGLDT classical it recovers +6.1 points (85.3 → 91.4) that a user who left the Koine default would lose; on the Apostolic Fathers it selects the AF table (+0.4).
  • Confidence calibration: the reported confidence is now temperature-scaled on held-out gold (top-1 ECE 0.21 → 0.14 on SBLGNT and AF).
  • Correctness fixes: interjections now carry POS I; the aorist passive of -έλλω verbs resolves (ἀπηγγέλη → ἀπαγγέλλω); the pers_name -πλ lemma artifact is gone (SBLGNT 4/40K and AGLDT 26/120K bogus top-1 lemmas → 0, no coverage loss).
  • Reproducible gate: scripts/benchmark.py + benchmarks/expected_greek.json (offline, tolerance-gated, bootstrap CIs); analysis parallelizes 3.06× at jobs=4.

Round 11 additions (2026-10-15)

The AGLDT v2.1 Greek treebank (which was used only to evaluate and to build the classical frequency table) is now also pooled into the learned re-ranker. Trained on SBLGNT + Apostolic Fathers + AGLDT (385,758 tokens; --agldt-limit 200000 balances the pool), the model is much stronger on classical text while keeping the Koine corpora within build noise:

corpus frequency ranking learned (Koine model) learned (pooled)
AGLDT v2.1 Greek (classical) 91.4% 92.1% 93.2%
PROIEL greek-nt (independent) 91.7% 91.5% 92.2%
SBLGNT 94.6% 95.0% 94.7%
Apostolic Fathers 90.1% 90.6% 90.3%

(A ~0.3pp Koine drift, inside the ±0.5pp build-noise band; the unbalanced pool cost ~0.7pp — hence the --agldt-limit balance.) The same gold now also builds a classical POS bigram model (data/pos_bigram_grc.json); it is the better bigram for classical text if the Viterbi tagger is used (Homer POS 87.3% vs 84.8%), but the tagger still trails plain frequency-ranked POS (85.2% on AGLDT) — the long-standing negative result stands. The offline benchmark thresholds were re-baselined to this model.

Round 14 additions (2026-10-15): entity extraction on documentary papyri

Round 14 evaluated Morpheus as the morphology layer under an external entity-extraction task (economic entities in documentary papyri, measured on a public 115-document gold set; measurement only — no corpus text is committed). The analyzer is not the extractor; it is the canonicalize / soft-validate layer on top of a fine-tuned NER model. Measured on that gold:

  • Canonicalization gain: matching entities in Morpheus lemma space (instead of surface strings) lifts micro F1 on the NER route 0.580 → 0.594 and on a raw-LLM route 0.229 → 0.274 — the analyzer recovers agreement between inflected variants. After gold fine-tuning (strict 0.717) the gain is +0.8pp (0.717 → 0.725): real, but it shrinks as the extractor improves.
  • Soft validation is informative but must not filter: predicted entities whose morphology is fully known strict-match gold 62.7% of the time vs 55.3% for the rest (53% of predictions are fully known, 31% not at all). Hard-filtering to known words drops strict F1 0.580 → 0.422, so validation stays a confidence signal, never a filter. On the fine-tuned model the same pattern holds more weakly (70.5% vs 66.4%; filter 0.717 → 0.522).
  • Candidate generation for recall: 46% of the extractor's missed entities (425/931) contain a content word Morpheus flags (unanalysed or numeral); a whitelist-filtered LLM pass over just those candidates raises strict recall +4.6pp (strict F1 0.580 → 0.585, relaxed 0.683 → 0.689) at one extra LLM call per document — versus a naive unfiltered pass, which is F1-negative. On a stronger (gold-fine-tuned) extractor the same pass is net-negative (0.717 → 0.700): the boost is a crutch for a weak extractor, not a permanent layer.

Two toolkit gaps surfaced (carried as follow-ups, not fixed here):

  • Letter-numerals are opt-in and incomplete: the PapyGreek numeral table (Morpheus(numerals=True), 50 forms) resolves common letter-numerals, but the lowercase letter for 1 was absent (its capital form is present) and the table is off by default. (Round 15 fixed the case gap — α and ε now fall back to their capital entries; the table stays opt-in.)
  • False-friend analyses outrank name proposals: an unanalysed token gets a proper-name proposal, but a capitalized genitive name that collides with a rare stemlib verb entry keeps the verb lemma — name-like entity keys therefore need a surface fallback (implemented in the Round 14 canonicalizer, not in the toolkit).

Round 15 additions (2026-10-15): the learned + adaptive defaults (B-sprint)

Round 15 shipped the B1 default flip: the learned re-ranker and adaptive frequency-table selection are now the Greek defaults — Morpheus() and scripts/benchmark.py get both with no flags. Opt out with Morpheus(use_learned=False, adaptive_tables=False) / --no-learned --no-adaptive-tables; caller-supplied frequencies= are never overridden. Validated on every local corpus against the old default (frequency ranking with the fixed Koine table), on identical candidate sets:

corpus (tokens) old default new default gain
SBLGNT / NT (137,554) 94.56% 94.67% +0.1
Apostolic Fathers (63,227) 90.14% 90.21% +0.1
Classical Greek, AGLDT (469,254) 86.56% 93.74% +7.2
PROIEL Greek NT — independent (132,366) 91.66% 92.19% +0.5
LXX (623,685, exact) 88.4% 90.7% +2.3
PapyGreek papyri (36,455, exact) 81.2% 85.7% +4.5
— PapyGreek + --numerals (95.1% coverage) 81.2% 87.8% +6.6
  • Adaptive selection picked the right register on every corpus (fit margins): SBLGNT → Koine (0.91 vs 0.89 classical), Apostolic Fathers → AF (0.89 vs 0.86), AGLDT → classical (0.95 vs 0.79), PROIEL → Koine (0.91 vs 0.90), LXX → LXX (0.93 vs 0.88), PapyGreek → papyri (0.72 vs 0.68).
  • The NT-family corpora gain least (they were already served by the Koine default); the flip is worth +7.2 on classical, +4.5 on papyri, +2.3 on the LXX — the corpora a reader beyond the NT actually needs.
  • The gate (scripts/benchmark.py + benchmarks/expected_greek.json) was re-baselined from the shipped-default run; the old frequency path stays available (--no-learned --no-adaptive, plus --no-gate for diagnostic runs).
  • One measured interaction, kept for the record: on the Apostolic Fathers the learned path gives back a fraction of the table-selection gain (learned + AF table 90.21 vs frequency + AF table 90.45) — the model's features were trained against the default table, so a selected table helps its fallback order more than its scores. Net vs the old default is still positive.

Also in this round:

  • Confidence follows the score path. The bundled softmax temperature (T=0.08) was fitted on frequency scores; the learned path now gets its own (temperature_learned=1.0 — the learned scores turn out natively calibrated). Top-1 expected calibration error on the shipped path: 0.005 (SBLGNT) / 0.002 (AF), vs 0.142 / 0.139 for the calibrated frequency path (raw 0.208 / 0.203) — the default path's confidence is ~30× tighter than pre-flip.

  • Lowercase letter-numerals fixed (Round 14 gap): α and ε resolve via a case fallback to the table's capital entries. PapyGreek --numerals numeral misses drop 853 → 38 of 1,454; coverage 92.9% → 95.1%.

  • Homograph re-ranking stays opt-in, now measured on the shipped default: SBLGNT 94.7 → 94.9, AF 90.2 → 90.7, PROIEL 92.2 → 93.1, but AGLDT 93.7 → 93.3 (the table is Koine/NT-trained; classical ὃ/τε/ἢ usage differs), so it does not meet the "never worse" bar for a default flip.

  • B8 head-to-head (measurement only, not shipped). On identical ambiguous-token example sets (≥2 candidates, gold among them), bundled learned vs an MLP head over frozen pranaydeeps/Ancient-Greek-BERT context, trained on SBLGNT:

    ambiguous tokens freq bundled learned LM-MLP combined
    SBLGNT test split (4,320) 97.7% 97.9% 99.4% 99.4%
    Apostolic Fathers (21,589) 96.9% 97.3% 98.0% 98.0%
    PROIEL NT, accent-insensitive (72,141) 95.4% 96.4% 92.3% 92.7%

    Corpus-level projection: SBLGNT test 94.0/94.1/94.6/94.6, AF 90.1/90.3/90.5/90.5, PROIEL 91.7/92.2/89.9/90.2 (the AF and PROIEL projections reproduce the benchmark exactly, so the harness is comparable). The LM head has real headroom on accented text (+0.5pp corpus-level over the bundled learned on the SBLGNT test slice, +0.2 on AF) but regresses on the shipped accent-insensitive pipeline: PROIEL analyzed with ignore_accents is 59.4% multi-candidate (vs 37.8% with accents; 38.8% vs 22.1% distinct-lemma), i.e. dominated by accent-variant homographs the LM head was never trained on. The combined model ≈ LM (the learned-rank feature adds nothing). Not shipped (torch + ~440 MB GPL-3.0-derived weights); the actionable finding is that an LM layer must be trained on the shipped candidate distribution (including ignore_accents examples), not just on accented gold text.

  • B10 verified. "Logion" (Cowen-Breen et al., Machine-Learning Based Detection and Correction of Textual Errors in Greek Philology, ALP 2023; charliecb/Logion; HF cabrooks/LOGION-*) is a scribal-error BERT, not a corpus: its 70M-word training corpus is request-only, and its models derive from a GPL-3.0 base with no licence card — not a clean corpus/LM candidate.

Round 16 additions (2026-10-15): the recovery pre-pass (spelling variants + names)

Papyri spell words the way people spoke; the stemlib holds the literary forms. Round 16 adds a recovery pre-pass that fires only on tokens the stemlib could not analyze at all — analyzed tokens are byte-identical with it off or on — and proposes stemlib-analyzed spelling variants (itacism, gemination, sandhi, dropped iota subscripts) and rule-generated name nominatives (Δωρίωνος → Δωρίων). Every proposal is flagged (proposed=True, recovered_from) and goes through the regular ranking. Measured on PapyGreek (all 36,455 word tokens; learned + adaptive defaults with the papyri table and --numerals; recovery off → on):

metric off on gain
coverage 95.1% 99.2% +4.1
top-1 ranked (exact / acc-insens) 85.9 / 87.3% 87.9 / 89.3% +2.0
top-1 learned (exact / acc-insens) 87.9 / 89.4% 89.7 / 91.2% +1.8
unanalyzed tokens 1,781 296 −83%

(The off column is re-measured under the same harness as the on column; it is 0.1pp above the Round 15 record because gold lemmas now have |num:N|/|gap:N| annotation spans stripped — the annotation-lemma hygiene, not the pre-pass.)

  • What the pre-pass recovers: 1,265 distinct forms — 390 from spelling variants alone, 376 from name candidates alone, 499 from both. Exact top-1 hits among recovered tokens (ranked) split 588 spelling / 134 name (token-weighted). The miss profile before → after: editorially marked/truncated forms 1,291 → 189, numerals outside the table 38 → 31, and the whole itacism/gemination/sandhi family that no stemlib entry covers (ὑγειαίνῃς → ὑγιαίνω via a restored iota subscript, μὲγ → μέν, ῥαύδους → ῥάβδος, Δωρίωνος → Δωρίων).
  • Coarse POS moves down slightly (80.1% → 79.7%) by construction: POS is measured over covered tokens, and the ~1,485 tokens that recovery brings into the denominator agree at ~70%. That is a completeness artifact, not a regression — coverage and top-1 can only rise (a miss has no candidates to lose; a proposal can only win if ranking puts it first).
  • The cruncher's accent-insensitive mode does not ignore the iota subscript (u(giai/nhs → 0 analyses, u(giai/nh|s → ὑγιαίνω), so the variant generator restores the subscript on single η positions; without it, tier-1 variants of documentary subjunctives are all dead.
  • Convention floor (documented, not fixable by rules): 45 tokens carry editorial annotation spans in the gold lemma (ια|num:11|, βαυβ|gap:2| — the hygiene strips them before matching), 35 carry hyphenated compound lemmas (σύν-προσγίγνομαι), and 5,598 (15.4%) have a gold lemma equal to the surface itself (indeclinable/name convention). Degree suppletion (πλεῖστα → πολύς) and particle conventions (κε → ἄν) can never match a stemlib lemma. These bound the achievable agreement on this corpus.
  • Gate (SBLGNT / AF / AGLDT / PROIEL): coverage, top-1 ranked and top-1 learned up-or-equal on all four corpora (SBLGNT learned 94.67 → 94.70, AF 90.21 → 90.26, AGLDT 93.74 → 93.76, PROIEL 92.19 → 92.24; coverage +0.02–0.06 each). The analysis rate jumps where the misses were name-heavy: AF 98.37% → 99.4%, SBLGNT 99.79% → 99.95%. The one metric that moves down is coarse POS on AF (75.94 → 75.31) — and it is the completeness artifact above: pos_n is measured over analyzed tokens, the ~650 newly analyzed AF tokens agree at ~15%, and every previously analyzed token's order is byte-identical (the arithmetic reproduces 75.31 exactly). Thresholds re-baselined with --update from the shipped-default run.
  • PapyGreek remains the top-priority corpus; the pre-pass is most valuable exactly where the stemlib is weakest.

Round 17 additions (2026-10-15): recovery tail pass + PapyGreek gate entry

Round 17 closes the measured tail of the recovery pre-pass and puts the top-priority corpus into the offline regression gate:

  • Operation-based variant generation (recovery.py): each normalization rule is now a primitive edit list on the normalized form — all single operations first, then every eligible operation pair merged (slot+slot, slot+geminate, slot+global), capped at 96 and ordered vowel-first. This makes multi-deviation forms reachable (θρυδάκεια → θριδάκια: υ→ι + ει→ι; ὑγειαίνησς → ὑγιαίνῃς: ει→ι + σσ→σ), adds internal nasal assimilation (νπ→μπ, νβ→μβ, νφ→μφ, νκ→γκ, νχ→γχ, νγ→γγ) and extends geminate reduction to ββ γγ δδ θθ φφ χχ.
  • Compositional letter-numerals: beside the 50-form table, a run of numeral letters with no vowel (≥2 letters) is a number — real Greek words always contain a vowel — proposed with the table's surface-lemma convention and the letter sum in the value feature; single letters fall back across case in both directions. Under the existing --numerals opt-in.
  • Three curated stemlib nouns in the generated nom.papyri (each justified by LSJ / PG attestation): παστοφόρος (LSJ via παστοφόριον/ἀρχιπαστοφόρος), πόκιον (PG-attested diminutive of πόκος), ἐρίδιον (LSJ headword, Dim. of ἔριον, long iota). θριδάκιον/πιττάκιον were already covered by lsj.nom and ἀρχισωματοφύλαξ already analyzed — those misses were spelling issues.
  • PapyGreek in the gate (scripts/benchmark.py): optional PAPYGREEK_DIR corpus, reader mirroring the tmp harness (original spelling, marker-stripped; regularized gold lemma; per-document grouping; POS classes inverted from the harness's TB2CLASS/MORPH2CLASS maps including the DC bridge and X/I equivalence). The shipped default is gated (no --numerals).

Measured on PapyGreek (all 36,455 word tokens; learned + adaptive defaults with the papyri table and --numerals; Round 16 → Round 17):

metric Round 16 Round 17 gain
coverage 99.2% 99.4% +0.2
top-1 ranked (exact / acc-insens) 87.9 / 89.3% 88.1 / 89.5% +0.2
top-1 learned (exact / acc-insens) 89.7 / 91.2% 89.9 / 91.4% +0.2
unanalyzed tokens 296 235 −21%
numeral tokens missed 31 / 1,454 14 / 1,454 −17
  • What the tail pass recovers: 1,295 forms (422 spelling-only, 331 name-only, 542 both) — 30 more than Round 16; exact top-1 hits among recovered tokens 634 spelling / 132 name (+46 spelling). The new rules reach συνπροσγενόμενος → συμπροσγενόμενος (nasal assimilation), θρυδάκεια → θριδάκια (pair), ὑγειαίνησς → ὑγιαίνῃς (pair), geminate pairs, the six vowel-less numeral forms (χμγ 643, ψπ 780, ρξβ 162 …) and the capital single letters (Β, Ζ, Δ, Τ, Φ).
  • The residual is now: 157 editorially marked/truncated forms, 14 numeral tokens (Ατ = 1300 is a decimal-lemma table exception; single-letter conjunctions like ἰ are excluded from the rule by design), and single-token scribal forms (γεγράφεκα, γραψατας, πειτακείων — each needs 3+ operations including insertions, or an ε/η interchange measured at 3 tokens total and deferred).
  • The default path (no --numerals): 97.4% coverage, ranked 85.9/87.3, learned 87.6/89.1, 942 unanalyzed (Round 16: 97.4%, 87.5/89.0).
  • Gate: PapyGreek joins the offline gate (shipped default: 97.4% analyzed, ranked 87.7, learned 88.1, POS 81.3 — the gate's variant-based comparison reads slightly higher than the harness's exact column, see lessons.md); the four existing corpora are unchanged-or-up (AGLDT learned −0.01pp: the recovery's new variants alter a few previously-recovered tokens' candidate sets; SBLGNT identical on every metric). Thresholds re-baselined with --update and confirmed green by a follow-up run.
  • Proposal confidence (audit, carried to B4): softmax confidence is conditional on the generated candidate set — lone proposals report 1.0 and all-proposed tokens average 0.84 on PapyGreek vs ~0.59 measured top-1 accuracy for recovered forms. A fitted per-strategy prior belongs in calibrate.py; the proposed / recovered_from flags are the actionable signal meanwhile.

Round 18 additions (2026-10-15): canonical-lemma scoring + learned retrain

Round 18 fixes the scoring primitive that credited a candidate with a form element's frequency, retrains the learned model on the shipped candidate distribution, and closes the confidence-calibration and profile items from the B list:

  • Canonical-lemma scoring (ranking.py, learned.py, tables.py, homographs.py): Morpheus's form,lemma field carries the forms the stem matched before the lemma, and an alternate stem can make one entry's form element equal a different word's lemma — πηρός's Doric stem lists παρά, so variant-list scoring credited πηρός with παρά's frequency (366) and promoted it above the actual preposition on PapyGreek. Scoring now uses the canonical lemma only; matching (the gold readers, the Strong's fallback) still considers every variant. Measured: PG plain path +1.3pp ranked (88.1 → 89.4 exact, 89.5 → 90.8 accent-insensitive), recovered-form top-1 634 → 660; SBLGNT a wash.
  • Retrained learned model (711 features, 411,615 analyzable tokens vs Round 15's 385,758): the trainer now runs the same recovery pre-pass the shipped pipeline uses (proposals are ranked like any other candidate), pools PapyGreek (34,998 tokens) under a document-level held-out split — 79 documents / 7,474 tokens, never trained on — and prefers canonical-lemma gold matches. Held-out PG: frequency 94.2% → learned 96.1% top-1. Model metadata records the split (papygreek_holdout).

Measured on PapyGreek (all 36,455 word tokens; learned + adaptive defaults with the papyri table and --numerals; Round 17 → Round 18):

metric Round 17 Round 18 gain
coverage 99.4% 99.4% —
top-1 ranked (exact / acc-insens) 88.1 / 89.5% 89.4 / 90.8% +1.3
top-1 learned (exact / acc-insens) 89.9 / 91.4% 90.2 / 91.7% +0.3
recovered-form top-1 (ranked, exact) 634 660 +26
default path (no --numerals) learned 87.6 / 89.1% 88.0 / 89.5% +0.4
  • Confidence calibration (B4): the joint (lemma, POS) score path got its own fitted temperature (T=0.12, ECE 0.198 → 0.137); the reported pos_confidence is now calibrated — the class mass multiplied by a per-class reliability fitted on SBLGNT (ECE 0.19 → 0.002 on the learned path, 0.021 frequency, 0.014 joint; AF transfer 0.003/0.020/0.016) — and all-proposed tokens (recovery / numerals / proper) are scaled by fitted per-strategy priors, so a lone proposal reports its strategy's measured rate (spelling 0.49, name 0.27) instead of 1.0.
  • Profiles (B5): the analyzer-guarded base-lemmas mode (compound verbs lemmatize to the base verb when the remainder is an attested verb, contracted compounds tail-matched) is now a registry profile, comma-combinable with morphgnt/lxx.
  • Tokenizer (B7): editorial markers inside words no longer split them (ἀδελφ[οῦ] → ἀδελφοῦ, πα|gap:1|ρά → παρά) and elision apostrophes stay attached, so ἀλλʼ analyses as ἀλλά and δʼ as δέ; crasis stays a single token (κἀγώ → ἐγώ, τοὐμόν → ἐμός, τἀληθές → ἀληθής, χὠ → ὁ; μοὐ recovers μοι).
  • Gate (18.4): full offline gate re-run, re-baselined with --update and confirmed green — every per-corpus adaptive pick unchanged, no metric regressed; POS agreement improved on every corpus (PapyGreek 81.3 → 82.8, AGLDT 85.2 → 86.0, SBLGNT 77.8 → 78.1, AF 75.3 → 75.5, PROIEL 81.5 → 81.7); PapyGreek learned 88.07 → 88.25 (the gate's variant-aware ranked column is flat by construction — the demoted candidates already "matched" the gold through their form elements there).

Round 19 additions (2026-10-16): PG error audit, adaptive numerals, Unicode-native input, MLP head-to-head

  • PapyGreek top-1 error audit (19.1, measurement only). Under the shipped path (adaptive papyri table + learned + recovery, numerals off; 36,455 word tokens, 97.4% covered), learned top-1 exact/acc-insensitive = 88.1/89.6% — matching the Round 18 record's default path (88.0/89.5) and confirming the adaptive papyri pick is worth ~2.5pp over the default table (85.6/88.2). The 2,857 remaining errors split into 1,725 gold-absent (1,533 editorial/regularization conventions no ranking change can fix — regularization to a different lexeme, orthographic/editorial spellings, dialectal/name variants, truncation; 192 covered-misanalyzed numerals worth ~+0.75pp with a papyri-scoped numeral-candidate rule) and 1,132 gold-present — the only ranking-fixable bucket: 48% gold at rank 2, function-word homographs (ἀλλά→ἄλλος 86, ὅς→ὁ 69+, εἰμί→ἐάν/οὖν 43, πᾶς→πάντῃ 24) and same-stem derivative confusions (ἀποδίδωμι→ἄπους 28). Ceiling for better ranking without candidate changes: +3.1pp. This bounded the B8-lite expectation before its training run.
  • Adaptive numerals (19.2, shipped). numerals="auto" (CLI --numerals-auto): the papyrus-numeral proposals apply only when adaptive table selection picks the papyri table, so numeral-shaped misses in other registers stay misses. Runs after selection so proposals cannot skew the table pick; inert with caller-supplied frequencies or adaptive_tables=False. Measured: PG identical to --numerals (89.4/90.8); SBLGNT/AF/AGLDT/PROIEL identical to the default. 5 new tests (105 total).
  • B3: Unicode-native cruncher input (19.4, shipped). cruncher -u converts UTF-8 Greek to lowercase beta directly in C (src/anal/gkunicode.c + generated gkunicode_tab.c), replicating the beta-code package's algorithm (NFC normalize → per-codepoint map → pass-through) with every table generated from the same JSON the Python converter uses. Parity oracle over 110,927 forms (6,623 synthetic + every distinct form in MorphGNT/AF/AGLDT/PROIEL/PapyGreek): 0 mismatches; ~2M forms/s. bin/gkconvert oracle binary + pytest guards. Default (no -u) byte-identical; the toolkit switch-over is carried.
  • CSNTM prep (19.5). Demo refreshed with current numbers (SBLGNT 94.6/94.7 exact/acc-insensitive; PG 89.4/90.8 with numerals) and the John 1:1–2 worked example now shows the calibrated confidences; outreach email drafted in docs/csntm-outreach.md for the user to send (Round 13.2).
  • B8-lite MLP head-to-head (19.3, measured; not shipped). train_mlp() was added to scripts/build_learned_model.py (opt-in --mlp; listwise softmax CE, bucketed by candidate count, CPU torch, best-held-out state; pure-Python payload ready if it had shipped). On the same shipped candidate distribution as the Round 18 retrain (411,615 analyzable tokens; 6,936-token PG document-level held-out split), the perceptron reproduces 96.08% (its weights come out byte-identical to the shipped model) and the MLP (64 hidden, 8 epochs) reaches 96.29% — +0.21pp ≈ 15 held-out tokens, below the pre-set ≥0.3pp ship gate and within noise (SE ≈ 0.24pp at n=6,936). The nonlinearity over the existing features buys fractions of a point, exactly as the audit predicted (the ranking-fixable bucket is 3.1% of tokens with a +3.1pp total ceiling); not shipped, carried with the next step being features (the audit's function-word homograph families) rather than model capacity.

Round 20 additions (2026-10-17): NT passage alignment (the CSNTM layer)

A new capability rather than a corpus-accuracy change: morpheus-toolkit align locates a transcription in a caller-supplied base text, classes every difference (equal / orthographic / partial / substantive / lacuna / extra), restores the edition's readings where the letters agree, and analyses the restored forms — a substantive variant is analysed exactly as the manuscript wrote it, never replaced by the edition.

  • Restoration payoff (42 synthetic uncial pages cut from MorphGNT; light / heavy / scriptio continua damage; ignore_accents, adaptive tables pinned off so both sides use one table): as-is 78.5% → restored 92.1% top-1 accent-insensitive lemma on light pages (n=2,159); 70.7% → 92.5% on heavy (n=2,076); 90.4% restored on continua (n=2,267). The restored column keeps injected variants as written, so it is a floor for this design — not the prototype's inflated 62.5% → 90.3%.
  • Safety: across all conditions, 0 substantive variants were reported as equal/orthographic/partial; 0 untouched words were flagged substantive or extra in the text modes (11/1,854 in continua). Every injected orthographic, partial, and substantive form in continua was flagged by some record — none went silent (a boundary shift moves the label to a neighbour instead).
  • Operational: identification anchors 41/42 pages in the right passage (the miss is a 10-token formula that occurs verbatim elsewhere); Mark pages avoid synoptic confusion 15/15; scrambled text refuses 8/8; the confidence gate refuses nothing on the healthy synthetic pages.
  • Caveat: synthetic pages only — no real transcription with a checked licence was available; nomina sacra are matched by the shipped table. The harness is committed (scripts/passage_eval.py) and reads MorphGNT at $MORPHGNT_DIR; nothing derived is written into the repo.

Bottom line

  • Coverage: ~94% of tokens analyzable at the fork's best → ~99% now, and ~100% on the literary corpora since Round 16's recovery pre-pass (PapyGreek 97.4% default / 99.4% with --numerals); on true Unicode input the fork was 0%.
  • Top-1 lemma accuracy: token-weighted across ~1.55M tokens, roughly 84.8% → 92.5% — the top-1 lemma error rate is about halved (15.2% → 7.5%), with per-corpus gains of +4 to +13 points. Since Round 15 the shipped default (learned + adaptive) adds a further +7.2 on classical, +4.5 on papyri, +2.3 on the LXX, and +0.1–0.5 on the NT-family corpora; Round 16's recovery pre-pass adds +1.8 on papyri (its top-priority corpus) and is neutral-to-slightly- positive on the literary corpora, and Round 17's tail pass adds +0.2 more there. Round 18's canonical-lemma scoring adds +1.3 more on papyri (harness exact column; the gate's variant-aware comparison is flat by construction) and the retrained model a further +0.3 learned there, while POS agreement gains +0.2 to +1.5 on every gate corpus.
  • Passage alignment (Round 20): on synthetic uncial pages, aligning to a base text and restoring its readings lifts bare-text lemmatization from 78.5/70.7% to 92.1/92.5% (light/heavy), with no substantive variant silently replaced (0 hidden) and every continua variant flagged by some record. Synthetic-only, and not a CSNTM-page result.

Caveats

  • "Before" for top-1 is "raw Morpheus order", measured on the current stemlib; the pure forked analyzer was a touch lower still (SBLGNT 86.7 at fork-time toolkit).
  • LXX and the LXX/AF/PG tables are corpus-trained, so their agreement numbers measure domain consistency, not absolute accuracy. The two PROIEL corpora are independent gold and show the same magnitude of gain, which is the strongest evidence the ranking is real and not convention-fitted.
  • POS is the weak dimension (~74–85%), because Morpheus collapses noun/adjective/pronoun/article into N; lemma accuracy is the headline. Since Round 16 it is also measured over the analyzed set, so additive coverage gains (the recovery pre-pass) move it down slightly by denominator growth — a completeness artifact, not a regression (see the Round 16 section).
  • PapyGreek numerals: the opt-in --numerals pass (table + vowel-less rule) resolves 1,440/1,454 numeral tokens (the Round 14 lowercase α/ε gap is fixed); bare-letter numeral false hits remain deliberately out of scope.

Reproducing the spot checks

# SBLGNT (current build: coverage 95.2%, top-1 88.1 -> 94.6, POS 77.4)
.venv/bin/python scripts/agreement_report.py --morphgnt-dir /path/to/morphgnt

# Apostolic Fathers (coverage 91.2%, top-1 84.8 -> 90.1, POS 74.3)
.venv/bin/python scripts/agreement_report.py \
    ../apostolic-fathers/data/morph/*.txt \
    --lemma-field 7 --pos-field 2 --source-field 9 --lang-field 8 --by-source

The classical (AGLDT), PROIEL, and PapyGreek figures come from the per-round evaluators recorded in completed.md (Rounds 8–9).

The whole table (raw → adaptive-selected frequency ranking → learned, plus coarse POS, with bootstrap CIs and a tolerance gate) is reproducible in one command — the shipped defaults:

MORPHGNT_DIR=/path/to/morphgnt AF_CORPUS_DIR=/path/to/apostolic-fathers/data/morph \
AGLDT_DIR=/path/to/agldt/v2.1/Greek/texts PROIEL_DIR=/path/to/greek-nt.conll \
    .venv/bin/python scripts/benchmark.py --bootstrap 300

(--no-learned --no-adaptive --no-recovery reproduces the old frequency-only default; --no-gate skips the threshold comparison for such diagnostic runs.)

PROIEL readiness engineering — 2026-11-02 checkpoint

These changes improve correctness and annotation-loss prevention; they do not measure new annotation accuracy, human throughput or accepted gold/silver. Both original and development checkouts pass 978 Python tests / 863 PROIEL tests, including the real pinned CLI and recorded synthetic editor proof. The completed record is in completed.md; active approval/editor/pilot/scale-out gates remain in todo.md and the PROIEL plan.

  • Canonical allocation precedes filtering: all 2,710 pilot IDs/attributes match full-source identities. Persistent locked reservations, historical migration and typed accepted lineage preserve new/retired IDs without reuse or fork mixing.
  • Graph validation catches disconnected cycles and protects virtual-root/empty/ secondary-edge semantics. Source-bound CoNLL-U/native imports preserve metadata, reject unsupported edits and publish atomically under cooperative source locks.
  • Strict builds and all-token seed provenance bind raw/code/schema/output hashes. Fresh actual full/pilot builds and XSD/CLI/manifest/sidecar verification pass; canonical XML/editing views are unchanged. Raw-input/store namespace poisoning, physical/case aliases, forged multi-source views and resealed invalid baselines fail.
  • Capped proposal stores enforce exact changed scope/workload, ownership, current approval guards and complete recovery audits before HEAD advances. Observed edit histories and ID-only lineage reports preserve evidence without certifying actors, linguistic sidecar compatibility or tiers. Actual sidecar rebinding stays open.
  • Independent surface/graph/uncertainty diagnostics expose missingness instead of false perfection. Draft overlap packs, tier-link checks and declared activity timing prepare human work without manufacturing annotation or independence.
  • Public CoNLL-U/CoNLL-X projections document losses separately from editing. The tested editor API bridge needs explicit root decisions; UI/native/provenance acceptance remains open. Constructed fixtures and mapping/input evidence prepare domain/legal review without substituting for it.

Full-size synthetic split→V insertion→native acceptance preserves 137,554 surface tokens, adds one empty and replays three versions offline; the pinned CLI and latest original lineage CLI pass. Performance remains a scale-out constraint: native-only format2 v8 inspection was 12.01s/about868MiB peak; typed format3 v3 inspection was 66.59s/about896MiB peak. A separate transaction-local audit reuse observed 34.92s under uncontrolled parallel load; see scale evidence for exact code hashes and limits. A later exact accepted-proof reuse removes duplicate reconstruction for approved bytes within the same lock: fresh-process inspection observed 32.52s before / 21.26s after on the same store. Every actual payload is rehashed; pending/rejected reconstruction and full version/lineage replay remain costs. These are single observations, not controlled general gains. No persistent cache or weakened recovery checks was introduced.

No source was staged/committed, pushed or sent through CI. No human annotation, qualification, manual/tokenization approval, training-input clearance, output license or gold/silver certification is inferred. All detailed temporary checkpoint notes are retained in development-copy tmp/improvements-checkpoint-history.md.