feat: Gemara v1 schemas as Pydantic v2 models

[jpower432]

Summary

This PR adapts complytime-labs/gemara-py (unreleased) as a new distribution.

gemara-python on the index, imported as gemara.v1. gemara is a PEP 420 namespace package, so a
future gemara.v2 can ship separately and install side by side.

Approach

How Types are generated

graph TD;
A[Gemara Spec Update] --> B[Schema Sync]
B --> C[JSON Schema Creation, Fixture Vendoring, Provenance Capture]
C --> D[Generated `_models.py` and `_registry.py`]

Loading

There is only one package for v1 so the tooling is designed to be forward compatible. Unknown fields (additive, optional) are ignored instead of generating an error. Pydantic v2 types are used to do structural and syntactic validation. datamode-codegen is used to create the Pydantic types from JSON Schemas.

Basic loading capabilities are included to mimic the behavior of gemara.Load in go-gemara. Conversion capabilities are not included on this first iteration.

Related Issues

Closes gemaraproj/community#12

This code was generated with the AI assistance, but directed by a human. Architecture, technology, and approach choices were all human driven. Code was reviewed by a human (@jpower432) and fixed applies in a multi-turn review resulting in the final four commits.

[jpower432]

Keeping this in draft pending a final review and more exhaustive local testing.

[eddie-knight]

🤖 Multi-model review: three independent reviewers (haiku, sonnet, opus) worked from a local worktree at cb89523, then every claim was re-verified against source, the pinned toolchain, and GitHub's docs before posting. Local run on Python 3.11: 117 tests pass, ruff check and mypy --strict clean, poe generate produces no drift, ruff format --check . fails. GitHub CI has not executed on this PR at all yet, so none of the workflows have been exercised. Inline comments are limited to findings with a drop-in suggestion; everything else is here.

Questions for the author

1. What does the Catalog/Log runtime tier (commit 532865c, no body) give a consumer? At runtime every subclass redeclares every base field, so nothing is inherited, and the README never mentions either name. If it is isinstance narrowing across catalogs, a sentence in the README and a test that fails without the tier would settle it; otherwise it looks removable. Details under Major below.
2. The description says py-gemara and _registery.py; the package is gemara-python since commit 6290126 and the file is _registry.py. Worth fixing before merge so the merged description is not the first stale doc.

Focus questions

• PEP 8 / idiom / typing / packaging: substantively clean. PEP 420 namespace + src layout, py.typed, Self, pydantic v2 idiom, mypy strict all check out. One deviation the repo's own formatter catches: _loader.py:1-2 (inline).
• Linters in CI, CI vs local: everything is wired, but three seams: the two workflows cancelling each other (inline on zizmor.yml:16 and ci.yml:9), a CONTRIBUTING instruction that cannot run because poethepoet is in the wrong dependency group (Major below), and the format check that CI runs but no local task or doc mentions, which is how the docstring failure got through. Poe is used for some commands and bypassed for others; pick one.
• DRY: docs duplicate the sync procedure in two files and one copy already contradicts the test file it describes (Major below). Four copies of the skip-guard shell block, two of the drift gate, two sys.path.insert (inline on pyproject.toml:77), one duplicated decode branch, and two suppression mechanisms for one lint line that neither fires (inline on __init__.py:6 and pyproject.toml:59-61).
• YAGNI / readability for new maintainers: the Catalog/Log tier above; two .gitignore entries for AI-tooling scratch dirs that do not exist in the repo (inline); a temp-file round trip in run_codegen that its own docstring says does not exist; and prose that points at a commit, a file path, and four numbered "Defects" that exist nowhere in this repo.

Blocking

The format failure on _loader.py:1-2 (CI quality job is red as submitted, inline). The concurrency-group collision (inline) will start biting on the first PR that touches a workflow.

Major (no drop-in fix, so not inline)

• CONTRIBUTING.md:13-14 vs pyproject.toml:32,36 — "uv sync --only-group lint is enough to run the linters" cannot work: poethepoet is declared only in the codegen group, so the documented uv run poe lint / poe typecheck fail with no poe on the path. CI hits the same seam: the test job runs uv run pytest raw, the quality job runs ruff format --check . raw, and the drift job can call poe generate only because codegen happens to carry it. Two clean options: put poethepoet in a group every documented caller includes (or plain dev) and drop the --only-group lint claim; or drop poe entirely, since the six tasks are each a one-word command that CONTRIBUTING can list directly, as CI already does for two of them.
• schemas/README.md:48-53 — states that SEMANTIC_GAPS entries "are known CUE cross-field rules that JSON Schema cannot currently express", which tests/test_fixtures.py:30-40 explicitly denies: not every entry is that, two former entries turned out to be codegen fidelity losses, and "still in this set" means "not yet proven recoverable". A maintainer reading the README stops investigating exactly the entries the test file asks them to investigate. Lines 20-40 are also a second copy of CONTRIBUTING.md:16-53, same three-line bash block verbatim. Pick one owner for the procedure and link from the other; for the policy, point at the comment in test_fixtures.py rather than restating it.
• tools/generate.py:102-110 (prioritize_category_bases) — nothing in the PR says what the Catalog/Log tier buys. Commit 532865c has no body, README.md never mentions either name, and at runtime the sharing is empty: all ten Catalog/Log subclasses redeclare every base field, so the set of fields inherited-without-redeclaring is empty for each (checked via model_fields against each class's own annotations). The tier costs this pass (no reference to it under tests/), the replace at lines 297-298, and test_registry.py:49-58, which asserts the inheritance exists but not that it does anything. Either state the consumer benefit and add a test that fails without the tier, or delete this function, the two replace lines, and the two registry tests; every document model then sits on GemaraDocumentModel with identical fields and behaviour.

Minor (no drop-in fix, so not inline)

• tools/generate.py:297-298 — same root cause as the tier above. If it stays, this replace needs the count() != 1 guard that lines 300-302 give the document models; today a changed declaration shape silently no-ops and surfaces as a confusing test_registry failure instead of a GenerateError.
• tools/generate.py:272-285 (run_codegen) — the docstring says nothing is serialised to disk, but the body writes to a TemporaryDirectory and reads it back. datamodel_code_generator.generate returns the source as str when output is omitted (its own docstring: "When output is None and single module: str"). Dropping the temp dir, output=, read_text and the tempfile import passes mypy strict and ruff and regenerates byte-identical _models.py/_registry.py (checked in the worktree). The return type is str | GeneratedModules | None, so keep one isinstance(source, str) guard that raises GenerateError otherwise.
• src/gemara/v1/_loader.py:104-110 — a copy of _loads_as lines 57-60, comment included. One _text(text) -> str helper (return text if isinstance(text, str) else _decode(text)) collapses both: loads becomes _dispatch(_parse(_text(text))) and _loads_as becomes model.model_validate(_parse(_text(text))).
• .github/workflows/ci.yml:32-40 — the skip-guard shell block appears four times (release.yml:33-42, publish-testpypi.yml:26-33 and :64-73), the drift gate twice (release.yml:43-48), and the tomllib version one-liner twice with different interpreters (release.yml:28 via uv run python, publish-testpypi.yml:55 bare python). The skip guard belongs in the test suite, not in shell around it. A tests/conftest.py pytest_sessionfinish hook that sets session.exitstatus = pytest.ExitCode.TESTS_FAILED when terminalreporter.stats.get("skipped") is non-empty does it in four lines (verified: exit 1 with a skip, 0 without; mypy strict and ruff clean). Every job then runs plain pytest, and summary.txt and its .gitignore entry go away.
• tests/tools/test_sync_schema.py:11 and test_generate.py:12 — both sys.path.insert lines and their # noqa: E402 go away with the pythonpath change suggested inline on pyproject.toml:77.
• CONTRIBUTING.md:47-49 — points at a "known limitation documented on recover_array_allof_element_type" that commit 3347a49 deleted in this same PR; tools/generate.py:213-218 is now a five-line docstring with no limitation text. Restore the one sentence there or drop the pointer.
• tests/tools/test_schema.py:7-8 — commit a21d87e is not in this repository's history (this PR is the initial import), and tests/test_sync_schema.py does not exist (the file is tests/tools/test_sync_schema.py). Same pattern with "Defect 1", "Defect 3" and "Defect 4" at ci.yml:34, release.yml:26,35, test_generate.py:79 and test_registry.py:34, cited against a numbered list that exists nowhere in the repo. Replace each with the fact it stands for or delete it.
• tools/sync_schema.py:119-120 — the root-vs-nested conflict guard is the head commit's behavioural change and has no test; test_merge_exports_rejects_conflicting_nested_defs (tests/tools/test_sync_schema.py:51-57) exercises only the nested-vs-nested branch at 113-114. One mirror test with a root export that disagrees with a stored nested def covers it.
• tools/generate.py:359 and tools/sync_schema.py:180 — main() -> int plus sys.exit(main()) promises an exit-code contract, but every failure path raises the tool's own error uncaught, so the return value is always 0 and a maintainer gets a traceback. Either catch GenerateError/SyncError in __main__ and return 1 with the message, or drop the wrapper and let the exception be the interface.

Recorded so they don't get re-raised

• Tooling security: subprocess.run with argument lists, no shell=True; yaml.safe_load. Fine.
• Workflow supply chain: every action SHA-pinned, persist-credentials: false everywhere, contents: read default, uv cache off on the publish path. Fine.
• Drift gate: regeneration in the worktree is byte-identical.
• Error surface: no yaml.YAMLError or UnicodeDecodeError escapes loads; tests cover tab-indented JSON and undecodable bytes.
• from_text/from_file go beyond "mimic gemara.Load" but are documented and tested; keep.
• The 218-line .gitignore template predates this PR (only 9 lines were added); trimming it is a follow-up, not a PR finding.
• Two reviewers rated the docstring format failure Critical; downgraded here because it is a one-line, zero-runtime-effect fix.
• One reviewer counted the sync pipeline as documented three times including the README's Known limitations section; that section does not describe it. Two copies.
• sync_schema.py:41-47 has the same multi-line-summary docstring shape as _loader.py but ruff does not flag it; left out under "no nitpicking".

[eddie-knight]

🤖 Minor — tests/tools/test_generate.py:12 and test_sync_schema.py:11 each hand-roll sys.path.insert(0, .../"tools") plus a # noqa: E402 on the import that follows, while this setting already does the job for tests. Adding tools here and deleting both sys.path lines keeps all 28 tool tests and mypy strict green (checked; mypy_path already includes tools).

Suggested change

	pythonpath = ["tests"]
	pythonpath = ["tests", "tools"]

[sonupreetam]

Complementing @eddie-knight's review. I tested the library end-to-end against every fixture and API path, and runtime behavior is solid.

[sonupreetam]

Minor — Every other function in this module has a docstring. _dispatch is the core routing logic and would benefit from one for consistency:

Suggested change

	def _dispatch(raw: Any) -> GemaraDocument:
	def _dispatch(raw: Any) -> GemaraDocument:
	"""Route a parsed mapping to the model selected by metadata.type."""

[sonupreetam]

Minor — The package exposes SCHEMA_VERSION (upstream schema version) but not its own package version. For a PyPI-distributed library, gemara.v1.__version__ is a common expectation for troubleshooting and CI version checks.

from importlib.metadata import version as _pkg_version
__version__ = _pkg_version("gemara-python")

[sonupreetam]

Minor — publish-testpypi.yml has a verify-published job that installs from the index and runs tests against the published artifact. The production release.yml has no equivalent — a packaging misconfiguration (wrong module path, missing py.typed) would ship to real users undetected. Consider adding a mirrored verification step with a brief delay for PyPI index propagation.

[jpower432]

Interesting. Certainly something to consider. The purpose of the testpypi workflow is to uncovered hidden issues with the published packaged. Maybe we could create an internal action that does validation from a package from a given index.

[sonupreetam]

Minor — vendor_fixtures unconditionally deletes schemas/fixtures/ via shutil.rmtree before re-populating. If the subsequent clone or copy fails (network error, disk full, tag missing test/test-data), the working tree is left with an empty directory and poe test fails with no clear recovery path.

Consider writing to a temp directory first, then swapping:

with tempfile.TemporaryDirectory(dir=SCHEMA_DIR) as staging:
    # copy fixtures into staging/
    if FIXTURE_DIR.exists():
        shutil.rmtree(FIXTURE_DIR)
    Path(staging).rename(FIXTURE_DIR)

Or at minimum, mention git checkout -- schemas/fixtures/ in the error output.

[sonupreetam]

Minor — This test calls load(path) but makes no assertion on the result — "no exception" is the entire pass criterion. A minimal assertion would make the test self-documenting:

def test_good_fixture_validates(path: Path) -> None:
    doc = load(path)
    assert doc.metadata.type in DOCUMENT_TYPES

[sonupreetam]

Minor — There's a test for {"metadata": {}} (missing type) but none for a mapping with no metadata key at all (e.g., {}), which exercises the isinstance(metadata, dict) → False branch in _dispatch. Consider adding:

def test_missing_metadata_key_raises_unknown_document_type() -> None:
    with pytest.raises(UnknownDocumentTypeError) as excinfo:
        loads(json.dumps({}))
    assert excinfo.value.value is None

[sonupreetam]

Nit — GemaraDocument is the return type of load/loads and is exported in __all__, but isn't mentioned here. Users who type-annotate their code need to know about it:

- `GemaraDocument` is the union type of all document models — the return type of `load` and `loads`.

[jpower432]

Thanks @eddie-knight and @sonupreetam. You are awesome! Working through the review feedback now.

[jpower432]

Questions for the author

1. What does the Catalog/Log runtime tier (commit 532865c, no body) give a consumer? At runtime every subclass redeclares every base field, so nothing is inherited, and the README never mentions either name. If it is isinstance narrowing across catalogs, a sentence in the README and a test that fails without the tier would settle it; otherwise it looks removable. Details under Major below.

Yep. This was for isinstance narrowing for working with common fields. Seems most useful for catalogs where we may want to process imports or extensions generically. Could also be deferred if we don't think we need this yet.

2. The description says py-gemara and _registery.py; the package is gemara-python since commit 6290126 and the file is _registry.py. Worth fixing before merge so the merged description is not the first stale doc.

Fixed! @eddie-knight after our discussion on ecosystem convention for package names, I looked a bit more and found the *-python seemed to be the leading.

[sonupreetam]

Re-reviewed at 8fdb9b7. All CI gates pass locally (tests, lint, format, typecheck, drift). Ran E2E tests and integration tests against the updated code. Everything is solid.