-
Notifications
You must be signed in to change notification settings - Fork 2
feat: Gemara v1 schemas as Pydantic v2 models #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
e6a2978
feat: py-gemara, Gemara v1 schemas as Pydantic v2 models
jpower432 28c6572
ci: harden workflows against zizmor findings, add audit job and depen…
jpower432 4c88606
refactor: make pyyaml a core dependency, split docs and release workf…
jpower432 0230821
refactor: call datamodel-code-generator in-process, scope dependency …
jpower432 38e2db6
ci: cancel superseded runs and install only what each job needs
jpower432 6740679
feat: add typed document loading constructors
jpower432 6290126
build: rename distribution to gemara-python
jpower432 6e038e4
docs: document package usage and schema maintenance
jpower432 309b747
ci: verify TestPyPI artifacts after publishing
jpower432 a03b590
chore: simplify Dependabot configuration
jpower432 3347a49
docs: shorten array allOf recovery docstring
jpower432 532865c
refactor: share catalog and log runtime bases
jpower432 cb89523
fix: guard root-vs-nested def conflicts and fix stale README example
jpower432 9825d5d
chore: apply suggestions from code review on workflows
jpower432 9102e46
chore: apply suggestions from code review
jpower432 710a0ff
test: fail suites that skip fixture tests
jpower432 bb8e065
ci: verify generated and released packages
jpower432 09e6c5d
build: share development tooling dependencies
jpower432 ef038f0
refactor: generate models without temporary files
jpower432 b9b5c10
fix: preserve fixtures when schema sync fails
jpower432 72f2458
feat: expose package version and document categories
jpower432 e9a2a1d
refactor: centralize loader text decoding
jpower432 ba9a1b3
test: confirm fixtures dispatch to registered models
jpower432 0d2e943
docs: clarify schema update and validation guidance
jpower432 8bc3bdc
test: cover missing vendored fixture failure
jpower432 5a2fd40
ci: share published package verification
jpower432 8fdb9b7
fix: add supression on ruff finding for public API rexport
jpower432 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| name: Check generated models | ||
| description: Fail when committed generated models differ from code generation output. | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - shell: bash | ||
| run: | | ||
| git diff --exit-code src/gemara/v1/_models.py src/gemara/v1/_registry.py \ | ||
| || { echo "::error::generated files are stale; run 'uv run poe generate'"; exit 1; } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| name: Verify published package | ||
| description: Install the published package from an index and run its package-level tests. | ||
|
|
||
| inputs: | ||
| index-url: | ||
| description: Package index URL to install from. | ||
| required: false | ||
| extra-index-url: | ||
| description: Additional package index URL to resolve dependencies from. | ||
| required: false | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - name: Install the published package | ||
| shell: bash | ||
| env: | ||
| INDEX_URL: ${{ inputs.index-url }} | ||
| EXTRA_INDEX_URL: ${{ inputs.extra-index-url }} | ||
| run: | | ||
| version="$(python -c 'import tomllib, pathlib; print(tomllib.loads(pathlib.Path("pyproject.toml").read_text())["project"]["version"])')" | ||
| venv=.published-package-venv | ||
| index_args=() | ||
| if [[ -n "$INDEX_URL" ]]; then | ||
| index_args+=(--index-url "$INDEX_URL") | ||
| fi | ||
| if [[ -n "$EXTRA_INDEX_URL" ]]; then | ||
| index_args+=(--extra-index-url "$EXTRA_INDEX_URL") | ||
| fi | ||
| uv venv --clear "$venv" | ||
| uv pip install --python "$venv/bin/python" "${index_args[@]}" "gemara-python==$version" pytest | ||
| - name: Verify imports use the installed artifact | ||
| shell: bash | ||
| run: | | ||
| .published-package-venv/bin/python -c 'import gemara.v1, pathlib; assert pathlib.Path(gemara.v1.__file__).is_relative_to(pathlib.Path.cwd() / ".published-package-venv")' | ||
| - name: Run tests against the published package | ||
| shell: bash | ||
| run: .published-package-venv/bin/python -m pytest -q tests/gemara/v1/test_loader.py tests/gemara/v1/test_registry.py tests/test_fixtures.py |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| version: 2 | ||
|
|
||
| updates: | ||
| - package-ecosystem: github-actions | ||
| directory: / | ||
| schedule: | ||
| interval: weekly | ||
| day: monday | ||
| open-pull-requests-limit: 5 | ||
| cooldown: | ||
| default-days: 7 | ||
| commit-message: | ||
| prefix: "ci" | ||
| groups: | ||
| actions: | ||
| patterns: ["*"] | ||
| update-types: ["minor", "patch"] | ||
|
|
||
| # Resolves against pyproject.toml and keeps uv.lock in step, which the | ||
| # `uv sync --frozen` steps in CI require. | ||
| - package-ecosystem: uv | ||
| directory: / | ||
| schedule: | ||
| interval: weekly | ||
| day: monday | ||
| open-pull-requests-limit: 5 | ||
| cooldown: | ||
| default-days: 7 | ||
| commit-message: | ||
| prefix: "deps" | ||
| prefix-development: "chore" | ||
| groups: | ||
| python-minor-patch: | ||
| patterns: ["*"] | ||
| update-types: ["minor", "patch"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| name: CI | ||
|
|
||
| on: | ||
| push: | ||
| branches: [main] | ||
| pull_request: | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.ref }} | ||
| # `main` is excluded so pushes there always produce a complete record. | ||
| cancel-in-progress: ${{ github.ref != 'refs/heads/main' }} | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| test: | ||
| name: Tests | ||
| runs-on: ubuntu-latest | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| python-version: ["3.11", "3.12", "3.13", "3.14"] | ||
| steps: | ||
| - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 | ||
| with: | ||
| persist-credentials: false | ||
| - uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0 | ||
| with: | ||
| enable-cache: true | ||
| - run: uv sync --frozen --no-default-groups --group test --python ${{ matrix.python-version }} | ||
| - run: uv run pytest -q | ||
|
|
||
| quality: | ||
| name: Lint, format and types | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 | ||
| with: | ||
| persist-credentials: false | ||
| - uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0 | ||
| with: | ||
| enable-cache: true | ||
| - run: uv sync --frozen | ||
| - run: uv run poe lint | ||
| - run: uv run ruff format --check . | ||
| - run: uv run poe typecheck | ||
|
|
||
| drift: | ||
| name: Generated-model drift | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 | ||
| with: | ||
| persist-credentials: false | ||
| - uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0 | ||
| with: | ||
| enable-cache: true | ||
| - run: uv sync --frozen --no-default-groups --group codegen --group lint | ||
| - name: Regenerate models from the vendored schema | ||
| run: uv run poe generate | ||
| - uses: ./.github/actions/check-generated-models |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| name: Publish to TestPyPI | ||
|
|
||
| on: | ||
| workflow_dispatch: | ||
| push: | ||
| tags: [ "test-v*" ] | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| publish: | ||
| name: Build and publish to TestPyPI | ||
| runs-on: ubuntu-latest | ||
| environment: testpypi | ||
| permissions: | ||
| id-token: write # For trusted publishing | ||
| steps: | ||
| - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 | ||
| with: | ||
| persist-credentials: false | ||
| - uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0 | ||
| with: | ||
| enable-cache: false | ||
| - run: uv sync --frozen | ||
| - run: uv run pytest -q | ||
| - run: uv build | ||
| - uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2 | ||
| with: | ||
| repository-url: https://test.pypi.org/legacy/ | ||
| # Rehearsals get re-run against an unchanged version; that should be a | ||
| # no-op rather than a failure. | ||
| skip-existing: true | ||
|
|
||
| verify-published: | ||
| name: Test the TestPyPI package | ||
| needs: publish | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 | ||
| with: | ||
| persist-credentials: false | ||
| - uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0 | ||
| with: | ||
| enable-cache: false | ||
| - uses: ./.github/actions/verify-published-package | ||
| with: | ||
| index-url: https://test.pypi.org/simple | ||
| extra-index-url: https://pypi.org/simple |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| name: Release | ||
|
|
||
| on: | ||
| push: | ||
| tags: ["v*"] | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| build: | ||
| name: Build and verify | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 | ||
| with: | ||
| persist-credentials: false | ||
| - uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0 | ||
| with: | ||
| # The release path builds the artifacts that get published, so it must | ||
| # not restore a cache that a pull request workflow could have poisoned. | ||
| enable-cache: false | ||
| - run: uv sync --frozen | ||
| - name: Verify the tag matches the static version | ||
| run: | | ||
| # Artifacts must never ship as 0.0.0, and the tag is the only thing | ||
| # that should ever disagree with pyproject.toml. | ||
| declared="$(uv run python -c 'import tomllib,pathlib; print(tomllib.loads(pathlib.Path("pyproject.toml").read_text())["project"]["version"])')" | ||
| tagged="${GITHUB_REF_NAME#v}" | ||
| test "$declared" != "0.0.0" || { echo "::error::version is 0.0.0"; exit 1; } | ||
| test "$declared" = "$tagged" || { | ||
| echo "::error::tag $tagged does not match pyproject version $declared"; exit 1; } | ||
| - run: uv run pytest -q | ||
| - name: Regenerate models from the vendored schema | ||
| run: uv run poe generate | ||
| - uses: ./.github/actions/check-generated-models | ||
| - run: uv run poe lint | ||
| - run: uv run ruff format --check . | ||
| - run: uv run poe typecheck | ||
| - run: uv build | ||
| - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 | ||
| with: | ||
| name: dist | ||
| path: dist/ | ||
|
|
||
| publish-pypi: | ||
| name: Publish to PyPI | ||
| needs: build | ||
| runs-on: ubuntu-latest | ||
| environment: pypi | ||
| permissions: | ||
| # Trusted publishing: mint a short-lived OIDC token instead of holding a | ||
| # long-lived PyPI API token in repository secrets. | ||
| id-token: write | ||
| steps: | ||
| - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 | ||
| with: | ||
| name: dist | ||
| path: dist/ | ||
| - uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2 | ||
|
jpower432 marked this conversation as resolved.
|
||
|
|
||
| verify-published: | ||
| name: Test the PyPI package | ||
| needs: publish-pypi | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 | ||
| with: | ||
| persist-credentials: false | ||
| - uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0 | ||
| with: | ||
| enable-cache: false | ||
| - name: Wait for PyPI index propagation | ||
| run: sleep 60 | ||
| - uses: ./.github/actions/verify-published-package | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| name: Audit Workflow Changes | ||
| on: | ||
| push: | ||
| branches: | ||
| - main | ||
| paths: | ||
| - '.github/workflows/**' # Triggers only when workflow files change | ||
| - '.github/actions/**' | ||
| pull_request: | ||
| branches: | ||
| - main | ||
| paths: | ||
| - '.github/workflows/**' | ||
|
jpower432 marked this conversation as resolved.
|
||
| - '.github/actions/**' | ||
| - '.github/dependabot.yml' | ||
|
|
||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.ref }} | ||
| # `main` is excluded so pushes there always produce a complete record. | ||
| cancel-in-progress: ${{ github.ref != 'refs/heads/main' }} | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| workflows: | ||
| name: Workflow security audit | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 | ||
| with: | ||
| persist-credentials: false | ||
| - name: Audit workflows and dependabot config with zizmor | ||
| uses: zizmorcore/zizmor-action@70fb788f84895a7701f5643d103d587e460b5c99 # v0.6.3 | ||
| with: | ||
| # Emit inline annotations rather than SARIF: results land on the PR | ||
| # without depending on code scanning being enabled for the repo. | ||
| # The two options are mutually exclusive. | ||
| advanced-security: false | ||
| annotations: true | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| # Contributing | ||
|
|
||
| ```bash | ||
| uv sync | ||
| uv run poe test # pytest | ||
| uv run poe lint # ruff check | ||
| uv run poe typecheck # mypy --strict | ||
| uv run poe format # ruff format | ||
| ``` | ||
|
|
||
| Dependencies are split into purpose-scoped groups, so a job or a contributor can | ||
| install only what it needs: `test`, `lint`, `codegen` (regenerating the models), | ||
| and `dev`, which includes all three. `uv sync` installs `dev`; `uv sync --only-group lint` | ||
| is enough to run the linters. | ||
|
|
||
| ## How the models are produced | ||
|
|
||
| This consists of two steps. | ||
|
|
||
| 1. `poe sync-schema`: needs `cue` on PATH plus network | ||
| access. It exports every `#Definition` from the upstream CUE module as JSON | ||
| Schema, merges them into `schemas/gemara-v1.schema.json`, records the exact ref | ||
| and digest in `schemas/provenance.json`, and re-vendors the upstream | ||
| `good-*`/`bad-*` corpus into `schemas/fixtures/`. | ||
|
|
||
| 2. `poe generate`: reads the vendored schema, applies its repair passes, | ||
| calls `datamodel-code-generator` in-process, and writes `src/gemara/v1/_models.py` | ||
| and `_registry.py`. | ||
|
|
||
| Both generated files are committed. **Never edit them by hand**: CI regenerates | ||
| them and fails on any diff, so a hand edit is reverted on the next run. | ||
|
|
||
| The codegen invocation is fixed at | ||
| `--preset practical-py311-20260619 --schema-version 2020-12`, and | ||
| `datamodel-code-generator` and `ruff` are pinned exactly. Both touch generated | ||
| bytes, so an unpinned bump would churn thousands of committed lines and fail the | ||
| drift gate for no semantic reason. Do not add hand-picked generator flags. | ||
|
|
||
| ## Bumping the schema version | ||
|
|
||
| ```bash | ||
| uv run poe sync-schema # optionally --ref vX.Y.Z | ||
| uv run poe generate | ||
| uv run poe test | ||
| ``` | ||
|
|
||
| Review the diff to `schemas/` and `src/gemara/v1/_models.py` together. A field | ||
| that got *looser* is the thing to watch for: a repair pass in `tools/generate.py` | ||
| may drop a constraint it cannot merge, and the only signal is a looser generated | ||
| type. | ||
|
|
||
| Update `SEMANTIC_GAPS` in `tests/test_fixtures.py` if the corpus changed. Read | ||
| the comment on that set for what it means and when to move an entry. | ||
|
|
||
| ## Tests | ||
|
|
||
| The fixture corpus is vendored, so the suite runs anywhere with no `cue` and no | ||
| warm cache. CI fails the build if any test is skipped. | ||
|
|
||
| ## Releasing | ||
|
|
||
| Publishing uses Trusted Publishing (OIDC); no API tokens are stored. The | ||
| `testpypi` and `pypi` GitHub environments must exist with a matching pending | ||
| publisher registered on each index. | ||
|
|
||
| - **Rehearse:** run the *Publish to TestPyPI* workflow manually | ||
| (`workflow_dispatch`) against any ref or push a test tag matching `test-vX.Y.X`. | ||
| - **Release:** set `version` in `pyproject.toml`, then push a matching `vX.Y.Z` | ||
| tag. The release workflow refuses a tag that disagrees with that version, and | ||
| refuses `0.0.0` outright. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.