Add provenance-checked spoken French training corpus

corpus/sources/PARIS_TRAINING.ja.md

@@ -0,0 +1,70 @@
+<!-- SPDX-License-Identifier: MIT -->
+# Paris Stories training専用profile
+
+既存の[validation専用pilot](PARIS_STORIES.md)をtrainingへ流用しない。
+別recipe/profileで同じ固定revisionのupstream trainだけを取り込む。
+tool/test/本文書はMIT。入力本文はCC BY-SA 4.0、生成modelの配布条件は未決定。
+
+## 元データと固定条件
+
+- repository: `UniversalDependencies/UD_French-ParisStories`
+- revision: `dec76f7a1731318b033c578d534410b0a3d4ea5c`
+- [README](https://github.com/UniversalDependencies/UD_French-ParisStories/blob/dec76f7a1731318b033c578d534410b0a3d4ea5c/README.md)
+  は本文を含む会話corpusとCC BY-SA 4.0、contributorsを記載。
+- [LICENSE](https://github.com/UniversalDependencies/UD_French-ParisStories/blob/dec76f7a1731318b033c578d534410b0a3d4ea5c/LICENSE.txt)
+  をREADMEとともに出力へ保存する。
+- `fr_parisstories-ud-train.conllu`: 2,208,034 bytes、SHA-256
+  `558c125ecb84f4e16ef8d8871368e24b8c6054446a06da9a5211fc67abb65a1d`
+- notices込みの取得対象は3file、計2,214,251 bytes。test・音声・外部URLは取得しない。
+- 既存validation manifest hashはrecipeで固定し、metadataだけを取込時に照合する。
+
+## 実際に見つかった分割上の問題
+
+最初のstrict取込では録音ID欠落によって停止した。全1,387文中27文で `sound_url` がなく、
+全て `ParisStories_2020_maisonAbondonnee` に属していた。IDを推測せず、recipeに27件の
+sentence IDを列挙して隔離する。隔離対象は必ず録音metadataが欠落し、text/tokenが存在する
+ことを検証する。未知の欠落を一括skipする機構ではない。
+
+残る1,360文・33録音についても、既存validationと録音IDが一致する1録音38文があった。
+そのidentity hashは `3d6643a3c3dc8219b68ff1a45c1d48c35635be5dea1ca7ffd61a17d8c9b0235c`。
+この録音全体をtrainingから隔離する。recipeの指定と実際のvalidationとの一致がなければ失敗する。
+
+隔離はraw cacheの削除ではない。元fileを保持し、理由・sentence ID・recording ID等を
+ingestion reportへ記録する。既存validationの本文・splitは変更しない。
+既知の除外後も、sentence ID・source hash・originのcross-split重複を拒否してから出力する。
+encoding変換の成否やdetectorスコアによる文章選別は行わない。
+
+## 再現
+
+```sh
+uv run --no-project python corpus/sources/paris_training.py fetch \
+  corpus/sources/paris-training.json /disk/paris-training/raw
+uv run --no-project python corpus/sources/paris_training.py ingest \
+  corpus/sources/paris-training.json /disk/paris-training/raw \
+  --validation-manifest /disk/paris-validation/manifest.json \
+  --output /disk/paris-training/input
+uv run --no-project python corpus/framework.py generate \
+  /disk/paris-training/input/config.json /disk/paris-training/generated \
+  --failure-policy record-and-continue
+uv run --no-project python corpus/overlap.py \
+  /disk/paris-training/generated/manifest.json /disk/paris-validation/manifest.json \
+  --output /disk/paris-training/overlap.json
+```
+
+元のvalidation parserの2 MiB上限は既定値として維持し、trainingだけ明示的に4 MiBを指定する。
+full textのUTF-8/cp1252をstrict往復で生成し、変換不能はframeworkに記録させる。
+取込先directoryは既存なら拒否する。途中出力を自動削除する処理はない。
+
+## 初回の観測（2026-09-21）
+
+- 採用入力: 32録音、1,322文、UTF-8原文87,681 bytes。
+- UTF-8/cp1252の64 variants全て成功、変換不能skipは0。
+- 別出力先の取込・生成について全fileのbyte一致を確認。
+- training 32 + validation 16 = 48 source、1,128 pairの既存char5近似重複診断は候補0。
+- training manifest: `fdd8e32a85cd6c4604929f62d5402f50bfc69b59b81cc3d08b85fe9494998466`
+- overlap report: `33f50a11f9dfd879273c781da9c5663611dfbaa07183de62f068349fa5304b22`
+
+近似候補0でも翻訳・部分転載・同一話者や意味内容の独立性は保証しない。
+会話という同一domain内のvalidationであり、独立評価を代替しない。
+このPRではmodel再生成・検出精度評価・独立holdout予測は行っていない。
+本文やmodelをrepositoryへ公開せず、P01の保留作業も再開しない。

corpus/sources/paris-training.json

@@ -0,0 +1,30 @@
+{
+  "recipe_version": 1,
+  "repository": "UniversalDependencies/UD_French-ParisStories",
+  "revision": "dec76f7a1731318b033c578d534410b0a3d4ea5c",
+  "license": "CC-BY-SA-4.0",
+  "upstream_split": "train",
+  "quarantine_recording_ids": ["3d6643a3c3dc8219b68ff1a45c1d48c35635be5dea1ca7ffd61a17d8c9b0235c"],
+  "quarantine_sentence_ids": [
+    "ParisStories_2020_maisonAbondonnee_1", "ParisStories_2020_maisonAbondonnee_2",
+    "ParisStories_2020_maisonAbondonnee_3", "ParisStories_2020_maisonAbondonnee_4",
+    "ParisStories_2020_maisonAbondonnee_5", "ParisStories_2020_maisonAbondonnee_6",
+    "ParisStories_2020_maisonAbondonnee_7", "ParisStories_2020_maisonAbondonnee_8",
+    "ParisStories_2020_maisonAbondonnee_9", "ParisStories_2020_maisonAbondonnee_10",
+    "ParisStories_2020_maisonAbondonnee_11", "ParisStories_2020_maisonAbondonnee_12",
+    "ParisStories_2020_maisonAbondonnee_13", "ParisStories_2020_maisonAbondonnee_14",
+    "ParisStories_2020_maisonAbondonnee_15", "ParisStories_2020_maisonAbondonnee_16",
+    "ParisStories_2020_maisonAbondonnee_17", "ParisStories_2020_maisonAbondonnee_18",
+    "ParisStories_2020_maisonAbondonnee_19", "ParisStories_2020_maisonAbondonnee_20",
+    "ParisStories_2020_maisonAbondonnee_21", "ParisStories_2020_maisonAbondonnee_22",
+    "ParisStories_2020_maisonAbondonnee_23", "ParisStories_2020_maisonAbondonnee_24",
+    "ParisStories_2020_maisonAbondonnee_25", "ParisStories_2020_maisonAbondonnee_26",
+    "ParisStories_2020_maisonAbondonnee_27"
+  ],
+  "validation_manifest_content_hash": "6340aff3b3d424fced87b782d75036c64152b4840eec98a05dc9a5b4f90216aa",
+  "files": [
+    ["LICENSE.txt", "899b1804a12ebc090b96339614eede1b64b686721b650a71430b55b5235f7f79", 202],
+    ["README.md", "3a6381529a560d09ffa9746fa6279b1d19ada33bb96386a2fa95e61e74205e91", 6015],
+    ["fr_parisstories-ud-train.conllu", "558c125ecb84f4e16ef8d8871368e24b8c6054446a06da9a5211fc67abb65a1d", 2208034]
+  ]
+}

corpus/sources/paris_stories.py

@@ -61,8 +61,10 @@ def fetch(recipe, root):
     return {"transferred_bytes": transferred, "verified_bytes": sum(r[2] for r in recipe["files"])}
 
 
-def documents(raw):
-    if len(raw) > MAX_FILE_BYTES:
+def documents(raw, *, max_bytes=MAX_FILE_BYTES):
+    if type(max_bytes) is not int or not 0 < max_bytes <= 4 * 1024 * 1024:
+        raise ValueError("invalid source byte budget")
+    if len(raw) > max_bytes:
         raise ValueError("source byte budget exceeded")
     groups, seen_ids = {}, set()
     metadata, token_count = {}, 0

corpus/sources/paris_training.py

@@ -0,0 +1,182 @@
+# SPDX-License-Identifier: MIT
+"""Separate upstream-train ingestion; preserve the existing validation-only pilot."""
+import argparse
+import json
+from pathlib import Path
+import re
+import urllib.request
+
+import paris_stories as pilot
+from artifact import write_idempotent
+from framework import audit_splits, content_hash, digest, safe_path
+from acquire import NoRedirect, serialized
+
+TEXT_FILE = "fr_parisstories-ud-train.conllu"
+MAX_FILE_BYTES = 4 * 1024 * 1024
+PROFILE = "paris-stories-recording-training-v1"
+
+
+def validate_recipe(recipe):
+    if (recipe.get("recipe_version") != 1 or recipe.get("repository") != pilot.REPOSITORY or
+            recipe.get("license") != "CC-BY-SA-4.0" or recipe.get("upstream_split") != "train"):
+        raise ValueError("unsupported training recipe")
+    if not re.fullmatch(r"[0-9a-f]{40}", recipe["revision"]):
+        raise ValueError("immutable revision required")
+    if not re.fullmatch(r"[0-9a-f]{64}", recipe["validation_manifest_content_hash"]):
+        raise ValueError("fixed validation manifest required")
+    files = recipe["files"]
+    if len(files) != 3 or {row[0] for row in files} != {"README.md", "LICENSE.txt", TEXT_FILE}:
+        raise ValueError("only train text and notices are allowed")
+    for _, sha, size in files:
+        if not re.fullmatch(r"[0-9a-f]{64}", sha) or type(size) is not int or not 0 < size <= MAX_FILE_BYTES:
+            raise ValueError("invalid source hash/size")
+    quarantine = recipe.get("quarantine_sentence_ids", [])
+    if (not isinstance(quarantine, list) or len(quarantine) > 100 or
+            any(not isinstance(s, str) or not s.startswith("ParisStories_") for s in quarantine) or
+            len(set(quarantine)) != len(quarantine)):
+        raise ValueError("invalid explicit quarantine identities")
+    recordings = recipe.get("quarantine_recording_ids", [])
+    if (not isinstance(recordings, list) or len(recordings) > 100 or
+            any(not isinstance(s, str) or not re.fullmatch(r"[0-9a-f]{64}", s) for s in recordings) or
+            len(set(recordings)) != len(recordings)):
+        raise ValueError("invalid explicit recording quarantine")
+
+
+def partition(raw, identities):
+    """Quarantine only explicitly listed records whose recording metadata is absent."""
+    if len(raw) > MAX_FILE_BYTES:
+        raise ValueError("source byte budget exceeded")
+    expected, seen, kept, quarantine = set(identities), set(), [], []
+    for block in re.split(r"\r?\n\r?\n", raw.decode("utf-8", errors="strict")):
+        if not block.strip():
+            continue
+        lines = block.splitlines()
+        ids = [line.removeprefix("# sent_id = ") for line in lines if line.startswith("# sent_id = ")]
+        if len(ids) != 1 or ids[0] not in expected:
+            kept.append(block)
+            continue
+        identity = ids[0]
+        tokens = [line for line in lines if line and not line.startswith("#")]
+        texts = [line for line in lines if line.startswith("# text = ")]
+        if (identity in seen or any(line.startswith("# sound_url") for line in lines) or
+                len(texts) != 1 or not texts[0].removeprefix("# text = ").strip() or
+                not tokens or any(len(line.split("\t")) != 10 for line in tokens)):
+            raise ValueError("quarantine record differs from missing-recording policy")
+        seen.add(identity)
+        quarantine.append(dict(sentence_id=identity, reason="MISSING_RECORDING_IDENTITY",
+                               block_sha256=digest(block.encode("utf-8"))))
+    if seen != expected:
+        raise ValueError("explicit quarantine identity missing from input")
+    return ("\n\n".join(kept) + "\n\n").encode("utf-8"), quarantine
+
+
+def fetch(recipe, root):
+    validate_recipe(recipe)
+    opener = urllib.request.build_opener(NoRedirect)
+    transferred = 0
+    for name, sha, size in recipe["files"]:
+        path = safe_path(root, name)
+        if path.exists():
+            pilot.read_verified(path, sha, size)
+            continue
+        url = f"https://raw.githubusercontent.com/{pilot.REPOSITORY}/{recipe['revision']}/{name}"
+        with opener.open(url, timeout=30) as response:
+            data = response.read(size + 1)
+        if len(data) != size or digest(data) != sha:
+            raise ValueError("download hash/size mismatch")
+        write_idempotent(path, data)
+        transferred += len(data)
+    return dict(transferred_bytes=transferred, verified_bytes=sum(row[2] for row in recipe["files"]))
+
+
+def ingest(recipe, root, validation, output):
+    validate_recipe(recipe)
+    if output.exists():
+        raise ValueError("output must not exist")
+    audit_splits([validation])  # Metadata only; no validation body or prediction is read.
+    if validation["content_hash"] != recipe["validation_manifest_content_hash"]:
+        raise ValueError("validation manifest differs from frozen recipe")
+    if not validation["sources"] or any(s["split"] != "validation" for s in validation["sources"]):
+        raise ValueError("validation-only reference required")
+    cached = {name: pilot.read_verified(safe_path(root, name), sha, size)
+              for name, sha, size in recipe["files"]}
+    admitted, quarantine = partition(cached[TEXT_FILE], recipe.get("quarantine_sentence_ids", []))
+    groups = pilot.documents(admitted, max_bytes=MAX_FILE_BYTES)
+    validation_origins = {s["origin"] for s in validation["sources"]}
+    for identity in recipe.get("quarantine_recording_ids", []):
+        if identity not in groups or f"parisstories:recording:{identity}" not in validation_origins:
+            raise ValueError("quarantined recording is not a validation overlap")
+        group = groups.pop(identity)
+        quarantine.append(dict(recording_identity_sha256=identity,
+                               sentence_ids=group["sentence_ids"],
+                               reason="VALIDATION_RECORDING_OVERLAP"))
+    if not groups:
+        raise ValueError("no admissible training recordings")
+    validation_ids = {sid for s in validation["sources"] for sid in s.get("sentence_ids", [])}
+    sources, payloads = [], {}
+    base = f"https://github.com/{pilot.REPOSITORY}/blob/{recipe['revision']}"
+    for identity, group in sorted(groups.items()):
+        if validation_ids.intersection(group["sentence_ids"]):
+            raise ValueError("training/validation sentence identity overlap")
+        source_id = f"paris-fr-{identity}"
+        relative = f"texts/{source_id}.txt"
+        data = ("\n".join(group["texts"]) + "\n").encode("utf-8")
+        payloads[relative] = data
+        sources.append(dict(
+            id=source_id, path=relative, language="fr", kind="natural",
+            license=recipe["license"], license_reference=f"{base}/LICENSE.txt",
+            revision=recipe["revision"], origin=f"parisstories:recording:{identity}",
+            sha256=digest(data), split="training", source_url=f"{base}/{TEXT_FILE}",
+            source_file_sha256=digest(cached[TEXT_FILE]), upstream_split="train",
+            source_kind="spoken-transcript", extraction_profile=PROFILE,
+            sentence_ids=group["sentence_ids"], recording_identity_sha256=identity,
+        ))
+    # Reject identical source content or recording identity across splits before any output.
+    prospective = dict(schema_version=1, sources=sources)
+    prospective["content_hash"] = content_hash(prospective)
+    audit_splits([validation, prospective])
+    config = dict(sources=sources, encodings=["utf-8", "cp1252"], formats=["text"],
+                  boundaries=["complete"], byte_limits=[None])
+    report = dict(profile=PROFILE, recipe=recipe, documents=len(sources),
+                  sentences=sum(len(g["sentence_ids"]) for g in groups.values()),
+                  text_bytes=sum(map(len, payloads.values())), source_split="training",
+                  validation_manifest_content_hash=validation["content_hash"],
+                  split_identity_overlap=False, semantic_independence="NOT_ESTABLISHED",
+                  legacy_representability_filter=False, audio_fetched=False,
+                  native_predictions=False, dependencies={
+                      p.name: digest(p.read_bytes()) for p in (Path(__file__), Path(pilot.__file__))
+                  }, quarantined_records=quarantine,
+                  quarantine_policy="explicit missing-recording/validation-overlap identities; raw retained")
+    output.mkdir(parents=True)
+    for relative, data in payloads.items():
+        write_idempotent(safe_path(output, relative), data)
+    for name in ("README.md", "LICENSE.txt"):
+        write_idempotent(output / "notices" / name, cached[name])
+    write_idempotent(output / "config.json", serialized(config))
+    write_idempotent(output / "ingestion-report.json", serialized(report))
+    return report
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("command", choices=("fetch", "ingest"))
+    parser.add_argument("recipe", type=Path)
+    parser.add_argument("cache", type=Path)
+    parser.add_argument("--validation-manifest", type=Path)
+    parser.add_argument("--output", type=Path)
+    args = parser.parse_args()
+    recipe = json.loads(args.recipe.read_text(encoding="utf-8"))
+    if args.command == "fetch":
+        if args.output or args.validation_manifest:
+            parser.error("fetch does not use output/validation")
+        result = fetch(recipe, args.cache)
+    else:
+        if not args.output or not args.validation_manifest:
+            parser.error("ingest requires output and validation-manifest")
+        validation = json.loads(args.validation_manifest.read_text(encoding="utf-8"))
+        result = ingest(recipe, args.cache, validation, args.output)
+    print(json.dumps(result, ensure_ascii=False, sort_keys=True, indent=2))
+
+
+if __name__ == "__main__":
+    main()