ProofFrame

A validator that tells you when it did not run, and hands you the evidence when it did.

ProofFrame is a Rust data-quality engine for PyArrow, Pandas, Polars, CSV, Parquet and Arrow streams. It compiles a contract against the physical schema before the first row is read, scans record batches without turning cells into Python objects, and returns a decision you can verify months later without the data in front of you.

What is different

The answer has three values, not two. accepted, rejected, unknown. A file that could not be opened, a limit that was exhausted, a schema that did not match — none of those become an acceptance. Most validators have nowhere to put the check did not run, so it lands on one of the two real outcomes, and which one you inherit is an accident of how the error was caught.

The verdict travels with what produced it. Evidence V2 binds the dataset fingerprint, the resolved Arrow schema, the contract source, the compiled plan and the resources the scan actually used. Verification is offline and rescans nothing — it proves the envelope, which is what lets it tell you that the contract you are reading now is not the contract that ran.

Counts are exact while output stays bounded. You get the true number of violations and a capped set of example rows, rather than a sampled estimate or an unbounded dump. A truncated finding list says it was truncated; the count beside it is still the count.

The limits are stated before the scan, not discovered during it. 512 MiB of memory and 4 GiB of temporary storage by default, charged before each allocation rather than observed afterwards. A scan that would exceed them fails instead of growing, and spill="never" refuses rather than touching the disk at all. Corrupt temporary data, ambiguous contracts and incompatible schemas fail closed.

One compiled engine behind every surface. The CLI, the Python package, the browser build and the scheduler operators call the same compiled functions. The offline review you save from a browser tab is byte for byte the document the CLI writes, because no second renderer exists to disagree with the first.

A new rule cannot invalidate an old receipt. A rule joins the compiled plan digest only when a contract uses it, and the V1 contract format and both fingerprint protocols stay frozen. Receipts written before a release keep verifying after it, and there is a test for each rule that says so.

0.7.2 — The contract moves into the scheduler

Airflow and Dagster get real packages instead of a sample DAG. proofframe-airflow ships ProofFrameAcceptOperator and ProofFrameVerifyOperator; proofframe-dagster ships build_acceptance_check. Both call the bindings in process rather than shelling out to the CLI, so the library's exception types survive and the engine's own memory and temporary-storage limits are the ones that apply.

Acceptance answers with three statuses and each scheduler offers two, so neither mapping is left to the caller. Dagster reports accepted as a pass, rejected as a failed check at ERROR, and unknown at WARN — a scan that did not complete decided nothing, and reporting it at ERROR would claim the data failed. Airflow raises the non-retryable failure on rejected, because the same bytes under the same contract cannot reach a different decision, and leaves unknown to on_unknown, which is the one status a retry can legitimately change.

The Airflow operator pushes a summary rather than a bundle. XCom lives in the scheduler's metadata database and a bundle carries the full report and the Evidence V2 envelope; the decision, the digests that identify it, and the path the bundle was written to go through instead.

0.7.1 — Nine more rules, and a review every surface can write

Ordering, step, exclusivity, totals, mean, standard deviation, conditional uniqueness, ledger balance, category dominance, row-count drift, text length, and dates written as dates. Each joins the plan digest only when a contract uses it, so existing receipts keep verifying. Uniqueness no longer needs a writable directory, so it runs on a read-only filesystem and in a browser, failing closed on the memory budget instead of on a missing folder. A profile drops the one count it cannot finish rather than the whole scan, and says which. review_html and review_markdown are part of the crate, so every surface renders the same report from the same code.

Earlier releases are in the changelog.

Try it without installing anything

Drop a CSV into the browser linter →

The engine is compiled to WebAssembly and runs in the tab: it reads the file with Arrow, infers a draft contract, refuses to execute that draft until you have reviewed it, and renders the same HTML review proofframe review writes. Nothing is uploaded, and the page reports the engine time it measured on your machine.

Uniqueness and the dataset-level exact rules run there too, in memory, failing closed on the budget rather than on a missing folder. Only references is refused, because it resolves against a second dataset the page cannot bind; a contract that uses it is still checked for everything else, and that result is reported as incomplete and carries no evidence.

Review a dataset

proofframe review orders.parquet --contract contract.json --out review-run
# Open review-run/index.html. To include finding details, use a NEW folder:
proofframe review orders.parquet --contract contract.json --out review-details --max-samples 20

CI: review exits 1 on contract violations; existing evidence still exits 0 when evidence generation succeeds. Samples default to zero. Contracts and column names are not redacted. Nothing is uploaded. Read the review guide for limits, privacy, signing, output publication and the runnable demo.

Accept or reject a delivery

A review tells you what the data looks like. Acceptance turns that into a decision an application can act on.

proofframe accept orders.csv --contract contract.json --policy policy.json   --csv-options reader.json --output acceptance.json
proofframe verify-acceptance acceptance.json

bundle = pf.accept_file("orders.csv", contract, policy=policy, csv_options=options)
bundle["payload"]["decision"]["status"]   # "accepted", "rejected" or "unknown"

Three answers, and the third is a real one: a missing file, a parse failure or an exhausted resource limit produces unknown, never a quiet accepted.

Zero violations is not the same as a result. A policy can require that named columns actually had values evaluated, so a file cannot be accepted because nothing was asked of it. The limit is stated plainly: those counters are per column, not proof that every rule ran.

CSV delimiter, encoding, decimal separator, column types and null tokens are chosen by you and recorded in the bundle under their own identity, so two systems reading the same file either agree or disagree visibly. 1.234 is not silently guessed.

verify_acceptance checks the bundle offline without rescanning, and checks its shape before it trusts any digest — a hash proves that what is present was not edited and says nothing about what is absent. Optional Ed25519 signing (proofframe[signing]) covers the whole payload.

Read the acceptance guide for the decision contract and its limits, and the runnable example for an accepted delivery, a rejected one, one that could not be read, and a tampered bundle that fails verification.

In GitHub Actions

Check a delivered file on the way in, and fail the job when it does not match the contract. Nothing is uploaded anywhere: the check runs on the runner.

- uses: emirhuseynrmx/proofframe@v1
  with:
    path: data/orders.csv
    contract: contracts/orders.json

The default mode writes an offline HTML review, a Markdown summary, report JSON and Evidence V2, uploads them as a build artifact, puts the summary in the job summary, and exits non-zero on a violation.

accept mode answers a different question — whether to take the delivery at all:

- id: gate
  uses: emirhuseynrmx/proofframe@v1
  with:
    path: data/orders.csv
    contract: contracts/orders.json
    policy: contracts/acceptance.json
    mode: accept
    out: acceptance.json

- if: steps.gate.outputs.status == 'accepted'
  run: ./load-into-warehouse.sh

status is accepted, rejected or unknown. A missing file, a parse failure or an exhausted limit answers unknown and fails the step; it never becomes an acceptance. Set fail-on-unknown: false to handle that case yourself — to retry a delivery that has not landed yet, for example.

Findings are not sampled by default: a sample can carry values out of your data and into a build artifact. Set max-samples when you want them.

In Airflow and Dagster

The same acceptance scan, inside the worker rather than in a subprocess.

from proofframe_airflow import ProofFrameAcceptOperator

validate = ProofFrameAcceptOperator(
    task_id="validate_orders",
    path="/opt/airflow/data/orders.parquet",
    contract="/opt/airflow/contracts/orders.json",
    output_path="/opt/airflow/evidence/{{ ds }}/orders.json",
)

accepted succeeds and returns a summary. rejected raises Airflow's non-retryable failure, because the same bytes under the same contract cannot decide differently. unknown is the one status a retry can change, so it is the one left configurable through on_unknown. ProofFrameVerifyOperator checks a bundle a previous task wrote, and rescans nothing.

In Dagster the result belongs on the asset that produced the file:

from proofframe_dagster import build_acceptance_check

orders_accepted = build_acceptance_check(
    asset=orders,
    path="data/orders.parquet",
    contract="contracts/orders.json",
    output_path="evidence/orders.json",
)

accepted passes; rejected fails the check at ERROR; unknown fails it at WARN, because a scan that never completed decided nothing about the rows. The check is blocking by default, so downstream assets wait for a verdict.

Both packages put the decision, the digests and the bundle path in the scheduler's own metadata, and leave the report and the Evidence V2 envelope on disk.

proofframe-airflow requires apache-airflow>=2.7,<3. The operators import BaseOperator from airflow.models, which is where Airflow 2 keeps it; Airflow 3 moves the provider surface to airflow.sdk. The ceiling is there so a resolver is not told that a version nothing has been tested against will work, and it lifts in the release that adds a 3.x job to CI beside the 2.10.5 one.

Install

Python 3.10–3.13:

pip install proofframe==0.7.2

Rust 1.85 or newer:

cargo add proofframe@0.7.2

The scheduler integrations are separate distributions, so the core package keeps its dependency surface and you install only the one you run:

pip install proofframe-airflow==0.7.2
pip install proofframe-dagster==0.7.2

The 30-second demo

import pyarrow as pa
import proofframe as pf

orders = pa.table({
    "order_id": [101, 102, 103],
    "subtotal": [12.50, 8.00, 10.00],
    "total": [12.50, 7.50, 10.00],
})

contract = {
    "version": "proofframe.contract.v2",
    "columns": {},
    "row_rules": [{
        "name": "total_covers_subtotal",
        "compare": {
            "left": {"column": "total"},
            "op": "gte",
            "right": {"column": "subtotal"},
        },
    }],
    "dataset_rules": {
        "row_count": {"min": 1},
        "distinct_ratio": {"order_id": {"min": 1.0}},
    },
}

report = pf.check(
    orders,
    contract,
    max_memory=64 << 20,
    max_temp=512 << 20,
    max_samples=20,
)

assert report["valid"] is False
assert report["violation_count"] == 1

The contract is compiled before scanning. Unknown fields, missing required columns, invalid bounds, and rules that do not match the Arrow type are rejected before the first row is processed. violation_count remains exact even when the retained findings sample is truncated.

Start from a reviewable draft

Use the native suggestion scanner to create a V2 draft, then inspect it before activation:

proofframe suggest data.parquet > contract.json
# Review contract.json and set "status" to "active" before checking it.
proofframe check data.parquet --contract contract.json

suggest performs its own Arrow scan so it can preserve exact integer bounds. It infers types and non-null columns by default; uniqueness, required columns, and category allowlists are explicit opt-ins. Timestamp and monotonically increasing numeric ranges are deliberately omitted and recorded in suggested_from.review. A draft is rejected with PF_DRAFT_CONTRACT until a reviewer changes its status to active. See the five-minute guide and contract reference.

Cross-column and conditional rules

V2 compares Arrow values in their physical type. It does not cast through Python objects or parse an expression language at runtime.

shipments = pa.table({
    "ordered_at": [1, 3],
    "delivered_at": [2, 2],
    "status": ["delivered", "pending"],
    "tracking_id": ["TR-1", None],
})

contract = {
    "version": "proofframe.contract.v2",
    "columns": {},
    "row_rules": [
        {
            "name": "delivery_window",
            "compare": {
                "left": {"column": "ordered_at"},
                "op": "lte",
                "right": {"column": "delivered_at"},
            },
        },
        {
            "name": "delivered_has_tracking",
            "when": {
                "left": {"column": "status"},
                "op": "eq",
                "right": {"literal": "delivered"},
            },
            "assert": {"column": "tracking_id", "not_null": True},
        },
    ],
}

report = pf.check(shipments, contract)

Comparisons support signed and unsigned integers, floats, booleans, UTF-8, dates, timestamps, and decimal128 where the Arrow types are compatible. Null behavior is explicit. Conditional assertions cover nullability, numeric bounds, allowlists, patterns, and NaN policy without building a row mask.

Dataset-level rules and partitions

{
  "dataset_rules": {
    "monotonicity": [{ "name": "clock", "column": "ts", "direction": "strictly_increasing" }],
    "gap_detection": [{ "name": "bars", "column": "ts", "expected_step": 60 }],
    "mutually_exclusive": [{ "name": "tax_id", "columns": ["tckn", "vkn"] }],
    "sum": [{ "name": "turnover", "column": "amount", "max": 50000000 }],
    "mean": [{ "name": "latency", "column": "latency_ms", "max": 45 }],
    "std_dev": [{ "name": "spread", "column": "latency_ms", "max": 15 }],
    "balance_equal": [{ "name": "books", "left_column": "debit", "right_column": "credit" }],
    "max_dominant_value_ratio": [{ "name": "skew", "column": "country", "max": 0.7 }],
    "conditional_unique": [
      {
        "name": "live_ids",
        "columns": ["id"],
        "when": { "left": { "column": "is_deleted" }, "op": "eq", "right": { "literal": false } }
      }
    ]
  }
}

A step is measured in the column's own units and findings name them. A total is counted in 128-bit integers, or with compensated addition for floats, so it does not change with the reader's batch size. A statistic over no values is reported as such rather than as zero.

Dataset rules keep exact state across record-batch and partition boundaries. Distinct and composite keys use canonical values, not hash-only identity. When the memory budget is reached, sorted, checksummed runs spill under the configured temporary-storage limit.

partitions = [
    pa.table({"order_id": [101, 101], "line_id": [1, 2]}),
    pa.table({"order_id": [102], "line_id": [1]}),
]

contract = {
    "version": "proofframe.contract.v2",
    "columns": {},
    "dataset_rules": {
        "row_count": {"min": 3},
        "distinct_ratio": {"order_id": {"min": 0.5}},
        "composite_unique": [{
            "name": "line_key",
            "columns": ["order_id", "line_id"],
        }],
    },
}

report = pf.check_partitions(partitions, contract, threads=2)
assert report["valid"] is True

pf.check_partitions_with_evidence additionally returns an ordered manifest binding every partition's V2 fingerprint, row count, schema, contract, compiled plan, result contribution, global result, and resource settings. Reordering, omission, duplication, or mixed identities fails verification.

Check a foreign key across datasets

orders = pa.table({"customer_id": [1, 99, 2]})
customers = pa.table({"id": [1, 2]})

contract = {
    "version": "proofframe.contract.v2",
    "dataset_rules": {
        "references": [{
            "name": "orders_customer_fk",
            "columns": ["customer_id"],
            "reference": "customers",
            "reference_columns": ["id"],
        }],
    },
}

report = pf.check(orders, contract, references={"customers": customers})
assert report["valid"] is False
assert report["findings"][0]["row"] == 1
assert report["references"][0]["reference_fingerprint"].startswith("pf-fp-v2:")

The contract names the reference; the caller supplies it. A declared reference with no bound dataset, and a bound dataset no rule uses, are both errors: a foreign key that is never evaluated would otherwise report as one that held. The report records the fingerprint of the dataset the keys resolved against, because "the key held" is not verifiable without saying against what.

One engine, several proof operations

Fingerprint a dataset

legacy = pf.fingerprint(orders, version="v1")
current = pf.fingerprint(orders, version="v2")

Fingerprints bind schema, row and column order, nulls, type tags, and canonical values. They do not depend on Arrow display formatting and remain stable across record-batch boundaries. V1 is frozen for existing proofs; V2 is a separate protocol for new evidence.

Diff by business key

changes = pf.diff(
    before,
    after,
    keys="order_id",
    max_memory=256 << 20,
    max_temp=2 << 30,
    max_samples=100,
    output="changes.jsonl",
    spill="auto",
)

Counts are exact. Samples stay bounded, while full change records can be written atomically as JSON Lines or Arrow IPC. Duplicate keys and schema mismatches fail loudly.

Create verifiable evidence

checked = pf.check_with_evidence(orders, contract, max_samples=20)
report = checked["report"]
evidence = checked["evidence"]

assert report["valid"] is False
assert evidence["schema"] == "proofframe.evidence.v2"

Evidence V2 binds the dataset fingerprint, canonical contract source, compiled plan, Arrow schema, engine version, resource limits, and result. Signed receipts use Ed25519 and separate cryptographic validity from signer trust. Keep signing material in a secret manager and verify against a public key obtained independently from the receipt.

Find PII and train/test leakage

pii = pf.scan_pii(customers)
overlap = pf.detect_leakage(train, test, keys="user_id")

PII findings contain the class, column, row, confidence, and a keyed fingerprint—not the matched value. Leakage reports support business keys or full-row identity and expose only bounded hashed samples.

Arrow-native by design

Pandas / Polars / PyArrow / Arrow C Stream / CSV / Parquet
                           |
                           v
                  Arrow record batches
                           |
          +----------------+----------------+
          |                |                |
       contracts       fingerprints      keyed diff
          |                |                |
          +----------------+----------------+
                           |
                           v
              deterministic JSON evidence

Known DataFrame containers provide exact row and logical-byte hints. Stream-only inputs remain streaming. The Rust scan releases the Python GIL, and the ProofFrame crate itself uses #![forbid(unsafe_code)].

Where ProofFrame fits

ProofFrame is strongest when you need Arrow-native, exact checks with bounded resources and evidence that records both the contract source and executable plan. It is deliberately smaller than established data-quality platforms.

Tool	Prefer it when	ProofFrame trade-off
Pandera	You want Python-first dataframe schemas, typing, and familiar pandas workflows.	ProofFrame prioritizes Arrow streams, exact global rules, and evidence over dataframe typing ergonomics.
Great Expectations	You need a large expectation library, data docs, and broad orchestration connectors.	ProofFrame has a narrower rule surface and a smaller connector set — GitHub Actions, Airflow, Dagster and dbt — but keeps validation and proof artifacts compact.
Soda	You want monitors, alerting, and a mature data-observability workflow.	ProofFrame is a library/CLI for deterministic checks; it does not replace an observability platform.
Deequ	Your data platform is Spark/Scala and you value its constraint-suggestion ecosystem.	ProofFrame avoids a Spark dependency and works directly with Arrow, but does not provide Deequ's Spark ecosystem.

These tools can coexist: use ProofFrame at an Arrow boundary when repeatable checks, resource limits, and independently verifiable evidence matter.

CLI

proofframe check data.parquet --contract contract.json --max-memory 256MiB --max-temp 2GiB
proofframe fingerprint data.csv --fingerprint-version v2
proofframe diff old.parquet new.parquet --key order_id --output changes.jsonl
proofframe evidence data.parquet --contract contract.json --output evidence.json
proofframe verify receipt.json --expected-public-key "$PROOFFRAME_PUBLIC_KEY"

Exit code	Meaning
0	Operation succeeded; check or receipt is valid
1	Contract violation or invalid receipt
2	Invalid input, contract, or command configuration
3	Engine, I/O, schema, or corrupt-data failure
4	Resource limit exceeded

JSON is emitted only after a successful operation. File outputs use same-directory temporary files, fsync, and atomic replacement.

Rust core

The default crate has no Python dependency. Rust users get the same compiled contracts, typed Arrow kernels, fingerprints, evidence, receipt verification, resource accounting, and spill engine used by the Python wheels.

Since 0.7.1 that includes the review itself. review_html and review_markdown render the report and the CI summary from a report, its evidence and the contract, so the document a Rust caller produces is the document proofframe review writes.

let html = proofframe::review_html(&report, &evidence, &contract, "orders.parquet", &columns)?;

See the crate guide and API documentation.

Compatibility and performance evidence

Version 0.7.2 preserves V1 fingerprints and the established compatibility entry points. New work should use check, explicit fingerprint versions, Evidence V2, and Receipt V2.

Performance claims are tied to raw samples, dataset hashes, compiler and package versions, and machine metadata. The committed smoke harness is deterministic; the pinned 7,645,034-row Bitcoin comparison remains a dedicated-runner gate rather than a published benchmark claim. See testing and benchmark methodology.

Release integrity

The release workflow packages the exact wheel, sdist, and crate subjects before publication. It emits deterministic SHA-256 checksums, SPDX JSON SBOMs, GitHub build-provenance attestations, and SBOM attestations. PyPI uses trusted publishing; crates.io publication is gated by the same tagged commit and CI evidence. These controls support provenance verification, but they are not a claim of formal SLSA certification.

Development

cargo test --locked --all-targets --all-features
cargo clippy --locked --all-targets --all-features -- -D warnings
maturin develop --release --locked
python -m pytest -q

CI covers Rust 1.85, Python 3.10–3.13 on Linux, macOS, and Windows, portable wheels, release-mode allocation contracts, Miri-compatible state machines, fuzz targets, source-package hygiene, coverage, DeepSource, and SonarCloud.

License and security

ProofFrame is licensed under Apache-2.0. Report vulnerabilities through the process in SECURITY.md. Sponsorship is available through GitHub Sponsors.