Incremental company knowledge bases from primary sources

Replay a company's primary-source documents (FCA NSM disclosures today; analyst notes, news, filings later) in chronological order into an editable handbook, so that each document either updates the knowledge base (genuinely new info) or does nothing (already reflected). The handbook's git history becomes a changelog of what was actually new — the basis for an all-signal/no-noise investor feed.

The premise: most documents about a company contain no new information. The value is identifying the few that do.

Pipeline

extract            replay                 render
─────────          ──────                 ──────
sources/<src>  →   handbook (per subject) → CHANGELOG.md
(per source)       edit-in-place, cited    (committed in the handbook repo)

1. Extract — pipeline/extract/<source>.py pulls a subject's documents from a source and parses them to text + structured facts. (nsm.py today.)
2. Replay — pipeline/driver.py feeds each document, oldest first, to an agent that maintains the handbook under a strict novelty discipline (commit only when a document moves the knowledge base). One agent call per document; resumable.
3. Render — pipeline/changelog.py turns the handbook's commit diffs into a StreetAccount-style CHANGELOG.md (signal headline + grounded body + source link).

Layout

pipeline/            reusable engine (tracked)
  extract/<source>.py    one extractor per source
  driver.py              replay documents → handbook
  changelog.py           handbook commits → CHANGELOG.md
templates/handbook/  SKILL.md (+ ontology.md) — instantiate per subject (tracked)
docs/DESIGN.md       design rationale + v2 tuning notes (tracked)
subjects/<subject>/
  subject.toml           identifiers + run params (tracked)
  CHANGELOG.md           rendered deliverable (tracked)
  state/processed.tsv    run ledger — resume + audit (tracked; state/logs/ ignored)
  sources/<source>/      extraction outputs — IGNORED (regenerable)
  handbook/              PURE skill: SKILL.md + entries/ — own git repo (ignored here)

One handbook per subject (all sources feed one knowledge base). The handbook is its own git repo kept as pure, installable skill content (SKILL.md + entries/) — a standalone, pushable deliverable the tooling repo ignores (no submodule). Its rendered CHANGELOG.md and run ledger live one level up at the subject level (so regenerating the changelog never churns the handbook's knowledge history). Extraction outputs under sources/ are regenerable and ignored.

Adding a subject

mkdir -p subjects/<slug>/sources
cp -r templates/handbook subjects/<slug>/handbook   # then fill SKILL.md placeholders
git -C subjects/<slug>/handbook init
# write subjects/<slug>/subject.toml with identifiers + sources
python3 pipeline/extract/nsm.py  --lei … --out subjects/<slug>/sources/nsm --parse
python3 pipeline/driver.py       --handbook subjects/<slug>/handbook \
    --disclosures subjects/<slug>/sources/nsm/disclosures.csv \
    --manifest    subjects/<slug>/sources/nsm/parsed_manifest.csv
python3 pipeline/changelog.py    --handbook subjects/<slug>/handbook

Adding a new source to an existing subject = a new sources/<source>/ feeding the same handbook.

Reference subject

subjects/henryboot/ — Henry Boot PLC (LSE: BOOT), 2010→2026: 460 NSM disclosures → 148 knowledge-bearing commits (65% of parsed documents were no-ops) → a 148-entry changelog. See docs/DESIGN.md for results and tuning notes.