feat: adding senzing sink and documentation

.gitignore

@@ -115,6 +115,11 @@ cache/
 
 /docs/design/
 /bank/
+# Output directories the documented examples write to
+/chain/
+/platform/
+/records/
+/lasvegas/
 
 # pages generated from README.md by docs/_readme_pages.py at build time
 docs/get-started/install.md

HISTORY.rst

@@ -157,6 +157,20 @@ A real register, and what measuring one changed.
   ``GROUP BY address`` is one PO box with 99,524 companies at it, with
   ``3225 Mcleod Dr`` and ``3225 McLeod Dr Ste 100`` as separate rows a little
   below.
+* A real register loads into LadybugDB. A relationship in one does not
+  always run between the same two kinds of node: in the Las Vegas open data
+  ``SITE_OF`` runs organisation to organisation 814,029 times and person to
+  organisation 5,079 times, because a few thousand people are filed as a
+  company's location, and ``HEADQUARTERS_OF`` does the same 2,089 times. The
+  sink declared one pair per relationship, so the whole register wrote itself
+  to Parquet and then died on the database with "unable to find primary key
+  value". It now declares every pair a relationship actually connects and
+  copies each pair separately, which is what LadybugDB wants. The 2,026,444
+  records of that file become 1,668,368 organisations and 358,076 people in
+  36 seconds. ``infer_endpoint_pairs`` is the new shared helper, ordered
+  commonest first so a sink that can only express one pair loses the least.
+  GraphFaker's own datasets never produce this shape, which is why nothing
+  caught it until a real register went in.
 * A dataset with no edges no longer breaks the DuckDB property graph.
   ``EDGE TABLES ()`` is a syntax error, so a cut of a register that happens
   to contain only companies loaded every table and then failed on the last

README.md

@@ -208,7 +208,7 @@ Sinks put the same tables into a database's own loader format:
 | `write_duckdb`, `graphfaker load duckdb` | the tables as DuckDB tables plus a `CREATE PROPERTY GRAPH` for SQL/PGQ pattern queries; DuckDB reads the Parquet itself | no graph database at all: SQL, with graph patterns, on the same files; see [docs/duckdb.md](docs/duckdb.md) |
 | `to_hetero_data`, `write_pyg`, `graphfaker generate ... --sink pyg` | a PyTorch Geometric `HeteroData`: features encoded, `y` and `decoy` on accounts, `y` and `edge_time` on transactions, stratified train/val/test masks | training and evaluating GNNs against a known answer; see [docs/pyg.md](docs/pyg.md) |
 | `write_neo4j_admin` | typed CSVs and the `neo4j-admin database import` command | the fastest path into Neo4j, offline, at any scale |
-| `write_senzing`, `graphfaker generate ... --sink senzing` | Senzing-format records: the same company several times under several spellings, at addresses it shares with others, plus `entities.parquet` as the answer key | running Senzing or any other resolver on data where the right answer is known; see [docs/senzing.md](docs/senzing.md) |
+| `write_senzing`, `graphfaker generate ... --sink senzing` | Senzing-format records: the same company several times under several spellings, at addresses it shares with others, plus `entities.parquet` as the answer key. Reads back through `graphfaker register` into LadybugDB, DuckDB or GraphML | running Senzing or any other resolver on data where the right answer is known; see [docs/senzing.md](docs/senzing.md) |
 | `write_gen_fraud_graph` | gen-fraud-graph's `accounts/`, `transactions/`, `fraud/` layout | pipelines built on that generator |
 | `export_csv`, `export_neo4j_csv`, `export_cypher` | from a NetworkX graph: CSV, neo4j-admin CSV, Cypher, openCypher, ISO GQL | Memgraph, Neptune, TigerGraph, any bulk loader |
 | `export_graph` | GraphML | Gephi, Cytoscape, igraph |

docs/notebooks/build_register.py

@@ -59,8 +59,11 @@ def unwrap(text: str) -> str:
 | 1.02% of name stems are shared by two companies | names are nearly unique, so near-matches are mostly the same firm | name reuse held at that rate at any scale |
 | `LLC` 53.9%, `L.L.C` 2.6%, comma 50/50 | the same company is written several ways | the measured spellings, and `variants` to write them |
 
-The real register is optional. Set `GRAPHFAKER_REGISTER` to a Senzing-format file, directory or zip and the last
-section runs; without it everything else still does. Nothing from a real register is printed here beyond counts,
+Along the way it writes the dataset out in Senzing's format and loads it into LadybugDB, because handing it to a
+resolver that is not this one is the other half of the point.
+
+The real register is optional. Set `GRAPHFAKER_REGISTER` to a Senzing-format file, a directory of shards or a zip
+and the last section runs; without it everything else still does. Nothing from a real register is printed here beyond counts,
 because it is someone else's data and real people's.
 """)
 
@@ -281,18 +284,133 @@ def score(label, **kwargs):
 """)
 
 md("""
-## 6. The same resolver on a real register
+## 6. Hand it to something else
+
+Scoring a resolver in this notebook is one use. The other is handing the dataset to a resolver that is not this one,
+which is what `--sink senzing` is for: records in the format Senzing takes, and the answer key in a second file.
+
+```bash
+graphfaker generate supply_chain --scale 0.01 --seed 11 --out ./chain --sink senzing
+```
+
+The records read back through GraphFaker's own register loader, so the generated dataset can go into a graph
+database and be queried like a real register. That round trip is the point: the same code path reads our records and
+somebody else's.
+""")
+
+code("""
+import tempfile
+
+from graphfaker.fetchers.senzing import read_tables
+from graphfaker.sinks.ladybug import connect, write_ladybug
+from graphfaker.sinks.senzing import write_senzing
+
+out = Path(tempfile.mkdtemp()) / "senzing"
+export = write_senzing(run, out)
+print(export.summary())
+""")
+
+code("""
+# The records, read back as a register and loaded into LadybugDB. This is what
+# `graphfaker register records.jsonl --out ./records --sink ladybug` does.
+records = read_tables(out / "records.jsonl")
+print({label: frame.height for label, frame in records.nodes.items()})
+print({label: frame.height for label, frame in records.edges.items()})
+
+write_ladybug(records, out / "db", db_path=out / "db" / "graph.lbdb", truth=None)
+conn = connect(out / "db" / "graph.lbdb")
+
+
+def ask(cypher):
+    # The column names come from the result, so a RETURN ... AS alias survives
+    # into the table instead of arriving as column_0.
+    result = conn.execute(cypher)
+    names = result.get_column_names()
+    rows = []
+    while result.has_next():
+        rows.append(result.get_next())
+    return pl.DataFrame(rows, schema=names, orient="row") if rows else pl.DataFrame()
+""")
+
+md("""
+**Who is the registered agent.** One address, many companies. This is the query that makes address useless as a
+blocking key, and a dataset where every company had its own address would return a flat list of ones.
+""")
+
+code("""
+ask(\"\"\"
+MATCH (o:Organization)
+WHERE o.ADDR_LINE1 IS NOT NULL
+RETURN o.ADDR_LINE1 AS address, o.ADDR_CITY AS city, count(*) AS records
+ORDER BY records DESC LIMIT 5
+\"\"\")
+""")
+
+md("""
+**One company, as a register holds it.** Whichever company has the most filed against it, and every record of it.
+""")
+
+code("""
+ask(\"\"\"
+MATCH (r:Organization)-[]->(c:Organization)
+WITH c, count(*) AS filings
+ORDER BY filings DESC LIMIT 1
+MATCH (r:Organization)-[e]->(c)
+RETURN r.id AS record, r.NAME_ORG AS filed_as, label(e) AS role, r.ADDR_LINE1 AS at
+\"\"\")
+""")
+
+md("""
+One company, one name spelled several ways, several addresses. Nothing in those records says they belong together;
+`entities.parquet` does.
+
+**The trap, counted.** Pairs of records at the same address that are not the same company.
+""")
+
+code("""
+ask(\"\"\"
+MATCH (a:Organization), (b:Organization)
+WHERE a.ADDR_LINE1 = b.ADDR_LINE1 AND a.id < b.id AND a.NAME_ORG <> b.NAME_ORG
+RETURN a.ADDR_LINE1 AS address, count(*) AS wrong_pairs
+ORDER BY wrong_pairs DESC LIMIT 3
+\"\"\")
+""")
+
+md("""
+**And the signal people actually want from a register:** somebody on more than one board. In a register that is
+several rows written several ways, and whether they are one person is the question.
+""")
+
+code("""
+ask(\"\"\"
+MATCH (p:Person)-[:EXECUTIVE_AT]->(c:Organization)
+WITH p.NAME_FULL AS person, count(*) AS boards
+WHERE boards > 1
+RETURN person, boards ORDER BY boards DESC LIMIT 5
+\"\"\")
+""")
+
+md("""
+## 7. The same resolver on a real register
 
 Everything above had an answer key. This section does not, which is the point of running it: the output is a shape,
 not a score, and no amount of looking at it tells you how much was missed.
 
-Set `GRAPHFAKER_REGISTER` to a Senzing-format file, directory or zip to run it. The cell prints counts only.
+Set `GRAPHFAKER_REGISTER` to a Senzing-format register and re-run. A single JSONL file works, and so does a
+directory of shards or a zip: the outputs below came from the national open data export, 2,493 shards and 330
+million records in one 21 GB archive, read without unpacking it.
+
+```bash
+GRAPHFAKER_REGISTER=~/Downloads/ODO_SENZING_20260305.zip python docs/notebooks/build_register.py
+```
+
+The cell prints counts only. A register is real people, and nothing from one belongs in a committed file.
 """)
 
 code("""
 if not REGISTER_PATH:
     print("GRAPHFAKER_REGISTER is not set, so this section is skipped.")
-    print("Point it at a register and re-run:  graphfaker register <path> --limit 200000")
+    print("Point it at a register and re-run. A zip is fine; it is read without unpacking.")
 """)
 
 code("""