Data augmentation: declarative recipes materialised at export

[JArmandoAnaya]

Draft — planning seed, not an implementation ticket. Captures partial information as of
2026-08-07; a dedicated design session will produce the real spec. Do not pick up for
implementation. Seven of the nine open decisions were ratified on 2026-08-07 — see the decision comments. Only the design-session details named in the stamps remain open.

The problem

Offline data augmentation — materialising transformed copies of a training set — is table
stakes for a dataset platform: every commercial competitor ships it, and a user with a small
dataset (the typical open-source case) expects it. On its own it is not a differentiator, and
it is not even the dominant practice — modern training frameworks apply transforms per-epoch
inside the loop, and VisionSet is not a training framework. What VisionSet can own is the
half a trainer cannot: the augmentation recipe as a declarative, seeded, versioned artifact
— the same shape SplitRecipe already has, so that "release v1.2 was trained with these
transforms at this seed" is a fact the workspace can state rather than a note in somebody's
README. Online augmentation stays out of scope as an execution concern.

Where it goes, and where it does not

Verified against HEAD (5aebfc6).

• Not at ingest. Asset identity is the SHA-256 of the content
  (uq_asset_project_content_hash), and IngestService dedups on it. Synthetic pixels in the
  content-address space of source data corrupts what an asset is.
• Not in the trunk. batch-lifecycle decision 8 (G5): "An Annotation hangs off its
  asset_id and nothing else… Do not add supersession links, per-round filtering, or
  annotation ids on DatasetMember." Derived assets would need "hide derived" filtering in
  every gallery, stat and batch surface, forever.
• Not in the blob store. BlobStore has no delete() — blobs are never deleted — so a
  variant written there is permanent workspace growth. Proposed invariant: the augmentation
  executor never calls BlobStore.put. Augmented pixels live in export output, which is
  disposable and regenerable.
• A recipe on the release, materialised at export. Export already commits pixels in a
  trainer's format, already runs in a worker (visionset.jobs.export, job type
  export.release), and already refuses to lose information silently.

The recipe attaches to the release row, not to the manifest

The draft this issue came from proposed putting the recipe inside the frozen manifest. That
is wrong by the manifest's own doctrine (kernel/domain/release.py module docstring):

The manifest is a pure function of content. Nothing time-, machine- or
identity-specific goes inside it… That is what makes "publish twice from an unchanged
dataset and the bytes agree" a property of the design rather than something to engineer.

SplitRecipe is already an instance of that rule being applied — a recipe is a statement
about how to use content, not about what the content is, so it lives on the release row
(_tables.py:565, a nullable JSON column) and not in the document. An augmentation recipe
passes the same test. Putting one in the manifest would need a MANIFEST_VERSION bump and
would stop two releases of identical content sharing a manifest blob.

The named migration implication

Migration 6, release_augmentation — one nullable JSON column on release.
FORMAT_VERSION 5 → 6.

• Same shape as the split column, one for one: Mapped[dict[str, Any] | None],
  mapped_column(JSON, nullable=True).
• An ALTER TABLE ADD COLUMN, not a rebuild, because it is nullable. ReleaseRow's
  docstring notes a NOT NULL column here would force a rebuild — that does not bind, and
  must not be dodged with a server_default.
• Two standing rules a migration task gets wrong: the new column is declared last on
  ReleaseRow (the ALTER-adds-last rule), and its undo goes into _at_generation_one —
  a column-adding migration whose undo is omitted never runs in the fresh-vs-migrated test and
  that test then passes vacuously.
• ReleaseOut gains one nullable field; openapi.json and the generated TS client are
  regenerated.

This cost is contingent on the attachment decision below: Option A costs zero migrations.

The blocker a design session must resolve first

The split is resolved inside the format plugin, not before it — so an augmentation stage
cannot simply be inserted, and the obvious implementation causes the exact val/test
contamination augmentation is supposed to avoid.

• The port is export(release, manifest, dest, *, content: ContentReader) -> None
  (kernel/ports/exporter.py). It receives no fold information.
• Each image-laying-out exporter calls folds_of(release, manifest) itself
  (formats/_layout.py:61; call site at formats/yolo/__init__.py:133, likewise COCO and
  VOC), which re-runs assign_split over the manifest it was handed.
• assign_split keys on content_hash. A variant has a new content hash — so if an
  executor widens the manifest with variants, the plugin assigns each variant a fold by
  hash, independently of its source asset. Variants of a training image land in val and
  test, and because counts come from largest-remainder apportionment over a larger set,
  original assets change folds too: the same release exports two different splits
  depending on whether augmentation was on.

So "the executor receives assets after split resolution, and variants inherit their source's
fold trivially" describes a pipeline that does not exist. Four routes, none free:

1. Widen Exporter.export with an optional precomputed fold map. Signature-compatible,
   not behaviour-compatible: a third-party plugin ignoring it contaminates silently.
2. Move fold resolution above the plugin and pass a fold map as a required argument. Clean;
   a breaking port change (cf. #365).
3. Post-process the export directory — dest/images/train/ is the train fold on disk, but
   labels are format-specific, so per-format label rewriting reappears, which is the
   duplication _layout.py exists to prevent.
4. Augment inside each plugin. Rejected: N formats × the same logic.

The good news in the same place. ReleaseService.export composes the ContentReader
and passes the manifest as a value — the plugin never sees a workspace or the frozen blob.
So an executor can legitimately hand down a derived manifest plus a reader that resolves
variant hashes from a temporary directory, leaving the release's manifest hash untouched. The
seam exists. Only fold resolution is in the wrong place.

Declared loss: a third input to the existing gate, not a second mechanism

Rotation on an axis-aligned box does not survive exactly — re-fitting inflates it. Crop
truncates. Occlusion dropout can erase a whole object and orphan its label. VisionSet already
refuses to lose information silently, and the existing machinery is the right shape:
_compatibility(release, manifest, exporter) judges one format, one release, per class and
produces ClassCompatibility with SUPPORTED / DEGRADED / DROPPED plus a reason.
Recipe loss is the same shape with a different second argument — rotation on a bbox class is
precisely DEGRADED, reason "axis-aligned box re-fit inflates the box".

Two consequences:

• Applicability must be judged per class, never against the schema's geometry union. The
  kernel already warns about this trap for annotations (annotation_service.py,
  errors.py): the union "would let a polygon through under a bbox class". A schema with
  one polygon class and one bbox class would report "rotation is fine" while every box
  degrades.
• Expect a fourth gate word. This repo keeps confirm=, allow_destructive= and
  allow_lossy= deliberately separate, "never one except". "This format cannot carry my
  polygons" and "my recipe degrades my boxes" are different consents with different remedies.
• visionset export --check must see recipe loss too, or it breaks its own promise that
  --check && export means something.

Library options — licenses verified 2026-08-07

Option	License	New runtime dep	Reach	Cost
A′ — Pillow only	MIT-CMU; already a hard dependency (pillow>=11.0)	none	flip, rotate, affine/perspective, crop, scale, brightness/contrast/colour, blur, JPEG artifacts. Not weather, not heavy noise.	Lowest. No extra, no optional-import vocabulary. Annotation math is still the real work.
A — own implementation on OpenCV + numpy	OpenCV Apache-2.0, numpy BSD-3-Clause	opencv-python-headless, numpy	everything incl. weather/occlusion	Highest build; full control
B — legacy albumentations 2.0.8	MIT	albumentations (→ numpy, opencv)	broad, immediately	Archived — see below
C — vendor selected transforms from the MIT-era code	MIT + attribution	as taken	chosen subset	Medium; own the code, skip re-deriving the math
D — torchvision / Kornia	BSD / Apache	PyTorch	broad	Disproportionate: a multi-GB tensor stack inside a dataset tool for offline CPU work
AlbumentationsX	AGPL-3.0-only	—	—	Excluded on license — incompatible with shipping as a hard dependency of an Apache-2.0 distribution. Recorded so it is not re-proposed.

Evidence: AlbumentationsX PyPI license_expression: AGPL-3.0-only, repo active
(archived: false, last push 2026-08-06). Legacy albumentations 2.0.8 is MIT and the repo
is archived (archived: true, last push 2025-06-25). opencv-python* wheels bundle
FFmpeg under LGPL-2.1 in all wheels (Qt5 LGPL-3 in non-headless Linux wheels) — not a
blocker, but a distribution fact, and an argument for -headless if OpenCV is chosen.

Two facts that reshape this table:

• Pillow is already shipped, so a meaningful subset of augmentation needs no new
  dependency and no extra at all. A staged answer (A′ for v1, an extra later for
  weather/noise) is available and costs nothing to keep open.
• This distribution has no pip extras. pyproject.toml has no
  [project.optional-dependencies]; the yolo and coco entries are PEP 735
  [dependency-groups], i.e. dev groups — pip install visionset[yolo] installs nothing.
  visionset[augment] would be the first extra ever shipped here, and there is no
  "missing extra, one-sentence refusal" vocabulary to reuse. The model to copy is
  MediaToolUnavailable (ffmpeg): checked lazily per call, deliberately outside the error
  family it sits next to, carrying the install hint in its message.

Architecture sketch (not ratified)

• The kernel models the recipe: a frozen pydantic model in kernel/domain/ — ordered
  transform specs, parameters, probabilities, variant count, seed, target-split policy, with a
  recipe_version — validated the way SplitRecipe is, so an invalid recipe cannot be
  constructed at all. No numpy, no OpenCV, ever.
• Execution lives outside the kernel, behind a narrow port. The precedent is
  visionset/jobs/export.py's own words: "Turning a format name into an Exporter means
  visionset.formats.registry, and import-linter forbids visionset.kernel from importing
  it… that single fact places visionset.jobs where it is." Open: a visionset.augment
  sibling (needs a new entry in contract 1's forbidden_modules plus a contract-3 mirror) or
  a module inside visionset.jobs (needs no contract change at all) — the answer depends on
  whether any caller other than the export handler is planned.
• Determinism: per-asset seed from recipe_seed + asset.content_hash — content_hash is a
  required field on ManifestAsset and is already what assign_split keys on, so it is
  available at materialisation. The honest promise is byte-identical within one build of the
  imaging library, matching what thumbnails and video frame extraction already promise; not
  across builds.
• No release-level capabilities. allowed_actions exists on BatchOut, JobOut and
  BatchAssetOut only; a release is immutable and declares none. The question anyone asks of
  a release — "what would this cost?" — is already check_export →ExportCompatibility,
  which is also the right home for recipe loss.

Known limitations to state rather than engineer around

• Video frames. ManifestAsset carries asset_id, content_hash, uri, width,
  height, annotations and nothing else — no frame_index, no frame_timestamp, no
  source_id. An executor working from the manifest structurally cannot tell a frame from a
  still. Good for leakage; it also means 200 consecutive frames of one clip would get 200
  independently sampled transforms, which is defensible for detection and wrong for anything
  temporal. cf. #364.
• Multi-image mixing (mixup, CutMix, mosaic, copy-paste) is cross-sample and
  overwhelmingly online-only. Out of scope for offline materialisation.
• Augmentation multiplies information, it does not add any — it replicates annotation
  errors N times over. Aggressive transforms produce out-of-distribution images and degraded
  labels; the mitigation is declared loss plus preview, not a better default.
• Progress. The export handler consults its ProgressReporter once before starting,
  because Exporter has no progress channel. Augmentation would be the first export-side work
  with a natural per-item unit, so a long augmented export would show no progress at all.
• 3D. GeometryType reserves CUBOID_3D and POLYLINE_3D and Asset.modality is typed
  to extend. A recipe schema designed 2D-only would need reopening. cf. #363.

Open decisions (Armando)

Seven of the nine items below were ratified on 2026-08-07 — see the decision comments on
this issue. Each is stamped with its outcome; the original trade-off analysis is left in place
as the reasoning behind the call. Only the design-session details named in the stamps remain
open.

1. Milestone placement. Resolved — see decision comment (2026-08-07): 0.1.0. The
   post-0.2.0 framing of the earlier decision comment was a working presumption, never a
   ratified decision, and is superseded. Sequencing consequence: the Exporter port change in
   item 5 ships with this feature in 0.1.0, and cf. #365 inherits the already-changed port
   rather than carrying the change itself.

2. Recipe attachment model. Resolved — see decision comment (2026-08-07): Option B, one
   recipe.json format at two commitment levels. The sub-question is resolved too: with a
   frozen recipe on the release and an ad-hoc one at export, the export refuses with one
   sentence unless an explicit override flag is passed — neither source ever wins silently.
   Still open: the override flag's spelling (design-session detail).

   • Option A — ad-hoc only. visionset export --augment recipe.json; the release stores
     nothing. Zero migrations. The cost is provenance: a Release today records
     manifest_hash, schema_version, the counts, split, created_at and
     visionset_version, and verify re-hashes every blob against them — the release can
     prove what it was made of. An ad-hoc recipe is recorded nowhere in the workspace, so "v1.2
     was trained with rotation ±15° at seed 42" would be true and unprovable, and a re-export a
     month later could silently differ.
   • Option B — same file format, two commitment levels. Attachable at release publish
     (frozen, provenance-bearing, migration 6) and passable ad-hoc at export for
     experimentation. This is split's exact shape. Sub-question that must be answered in
     the same session: when a release carries a recipe and the export passes one, does it
     refuse, or does one win?
   • The --augment recipe.json syntax is available under both.

3. Backend library. Resolved — see decision comment (2026-08-07): an own implementation
   on opencv-python-headless + numpy, as a hard runtime dependency — not an extra. A′
   (Pillow only) is declined, as is any staged rollout; full technique reach ships on one
   backend from v1.

4. UI scope of v1. Resolved — see decision comment (2026-08-07): a visual recipe editor
   with live per-transform preview, executed server-side through the same executor, with
   full UI/CLI parity over one recipe.json. Still open: editor scope details
   (design-session detail).

5. The Exporter port question. Resolved — see decision comment (2026-08-07): route 2
   — fold resolution moves above the format plugin and the port takes a precomputed fold map as
   a required argument. Route 1 is rejected: a third-party plugin ignoring an optional
   kwarg would contaminate val/test silently. Sequenced with the Exporter contract work in
   cf. #365 so the port breaks once.

6. Package placement. Resolved — see decision comment (2026-08-07): a visionset.augment
   sibling package with its own import contracts, mirroring the visionset.jobs shape. The
   second caller that justifies it exists by decision — the server-side preview path.

7. The fourth gate word. Resolved — see decision comment (2026-08-07): recipe-declared
   loss gets its own consent word, never folded into allow_lossy. Still open: the word's
   spelling (design-session detail).

8. Video frames. Resolved — see decision comment (2026-08-07): v1 states the
   temporal limitation in docs; temporal-consistent augmentation is scoped out to cf. #364.
   No engineering around it in v1.

9. Where recipe-loss reporting lives. ExportCompatibility.format_name /
   format_is_lossy are format-shaped fields, and the report is round-tripped through
   AliasChoices and written to disk. Carrying recipe loss means either widening
   ExportCompatibility (a wire change and an on-disk report-format change) or a sibling
   per-class document. Open — design-session decision. cf. the durable-findings comment.

Standing constraint

Per the dependency policy on #81 (cf. #81), third-party inference/labeling frameworks were
not evaluated as options here.

References

• cf. #365 — 0.2.0 scope: more formats. Owns the lossiness contract this feature extends and
  the Exporter port any fold-resolution change touches.
• cf. #411 — 0.2.0 scope: import of external datasets. The other half of
  visionset.formats; shares the "declare what is carried, refuse silent drops" principle.
• cf. #363 — 0.2.0 scope: 3D and point clouds. Decides whether the recipe schema may be
  designed 2D-only.
• cf. #364 — 0.2.0 scope: video tracking. The temporal-consistency limitation above.
• cf. #352 — per-vertex annotation data; any transform-applicability table over polylines
  inherits it.
• cf. #375 — geometry categories; the applicability matrix is a per-geometry table and would
  group the same way.
• cf. #81 — AI-assist substrate. Adjacent, neither blocks the other.

[JArmandoAnaya]

Ratified decisions

Decision (Armando, 2026-08-07): Backend is an own implementation on opencv-python-headless + numpy, added as a hard runtime dependency — not a pip extra, not a Pillow-first staging. Full technique reach (geometric, photometric, weather, noise) is available from v1 on one backend. Rationale: VisionSet is a computer-vision tool, so OpenCV being always present is coherent with what the product is, and no missing-extra refusal vocabulary needs to be invented. The A′/staged option from the analysis is declined. AlbumentationsX remains excluded on license (AGPL-3.0-only); third-party inference/labeling frameworks remain excluded per the dependency policy on #81 (cf. #81). — supersedes any prior direction

Decision (Armando, 2026-08-07): Fold-resolution conflict resolved via route 2: fold resolution moves above the format plugin, and the Exporter port receives a precomputed fold map as a required argument. Route 1 (optional kwarg) is rejected because a third-party plugin ignoring it would contaminate val/test silently — the exact failure class this product refuses. The port change is sequenced with the Exporter contract work in cf. #365, so the port breaks once, not twice. This confirms augmentation as post-0.2.0 work; no milestone is assigned by this decision. — supersedes any prior direction

Decision (Armando, 2026-08-07): Recipe attachment is Option B: one recipe.json format, two commitment levels. The recipe is attachable at release publish (frozen on the release row beside split — migration 6, one nullable JSON column, never inside the manifest) and passable ad-hoc at export --augment recipe.json for experimentation. Both-present rule: an export that receives an ad-hoc recipe while the release carries a frozen one refuses with one sentence, unless an explicit override flag is passed; neither source ever wins silently. The override flag's spelling is a design-session detail. — supersedes any prior direction

Decision (Armando, 2026-08-07): Package placement is a visionset.augment sibling package with its own import contracts (mirroring the visionset.jobs shape), not a module inside visionset.jobs. The second caller that justifies the sibling exists by decision: the UI ships a visual recipe editor with live per-transform preview, and preview execution is server-side through the same executor — a preview path outside the export job. No transform math is ever reimplemented in the frontend; UI and CLI operate on the same recipe.json (full UI/CLI parity from v1). — supersedes any prior direction

Decision (Armando, 2026-08-07): Recipe-declared loss gets its own consent gate word, never folded into allow_lossy. "This format cannot carry my polygons" and "my recipe degrades my boxes" are different consents with different remedies; the standing three-words-three-reasons rule extends to a fourth. The word's spelling is a design-session detail. Recipe loss reporting reuses the per-class compatibility-report family (per-class judgment, never the geometry union), and --check must see recipe-declared loss — a --check blind to it would lie. — supersedes any prior direction

Decision (Armando, 2026-08-07): Video frames: v1 states the temporal limitation (independent per-frame transforms are wrong for temporal tasks) in docs and scopes temporal-consistent augmentation out to cf. #364. No engineering around it in v1. — supersedes any prior direction

Remaining open: milestone placement (post-0.2.0 presumption, Armando assigns), plus the design-session details named above (override flag spelling, gate word spelling, UI editor scope details).

[JArmandoAnaya]

Durable findings from the analysis report

The findings the recording brief named as durable — the migration-6 specification, the
BlobStore.put-never invariant, the fold-contamination mechanics, the per-class applicability
rule, the --check requirement and the determinism caveat — were already in the body above
and are not repeated. What follows is the remainder of the analysis working file that exists
nowhere else, recorded before that file is discarded. No decision is implied by anything
here.

• ExportCompatibility is format-shaped, so recipe loss still needs a home. format_name
  and format_is_lossy are fields about a format (kernel/domain/release.py:531, round-tripped
  through AliasChoices). The 2026-08-07 decision settles that recipe loss reuses the per-class
  compatibility family — it does not settle where the document lives: widening
  ExportCompatibility is a wire change and an on-disk report-format change, and a sibling
  document is the alternative. Design-session input.
• numpy and opencv-python are already resolved in uv.lock — transitively, and shipped to
  nobody. They arrive via the yolo (ultralytics) and coco (matplotlib/pycocotools) PEP 735
  dependency groups. The backend decision moves opencv-python-headless from transitive-dev to
  direct runtime; CI already installs these in a dedicated job, so a test matrix has somewhere
  to live. The three-day dependency cool-down (cf. #399) now governs whatever pin is chosen.
• opencv-python* wheels bundle FFmpeg under LGPL-2.1 — in all wheels (and Qt5 under
  LGPL-3 in non-headless Linux wheels). Not a blocker: ordinary dynamic linking inside a
  redistributed wheel. It is the argument for -headless, which the decision already takes.
• Export idempotency survives variants, with one condition. ExportResult.file_count and
  total_bytes are computed by walking dest after the plugin returns, so they stay correct
  when variants are written; and jobs/export.py clears its own destination first, so re-running
  an augmented export is idempotent — provided the recipe seed is stable.
• Frontend geometry gating is not the Python union. The annotator tool strip uses
  drawableGeometry from @visionset/annotator
  (frontend/ui-core/src/annotator/ToolPalette.tsx); SchemaService.allowed_geometries is a
  separate kernel-side union. A recipe editor's applicability display should follow the per-class
  rule, not either union.

Library evidence as fetched 2026-08-07, for the record: AlbumentationsX 2.3.9
(license_expression: AGPL-3.0-only, repo active, last push 2026-08-06) · albumentations
2.0.8 (MIT, repo archived, last push 2025-06-25) · opencv-python-headless 5.0.0.93
(OpenCV Apache-2.0, packaging MIT) · numpy 2.5.1 (BSD-3-Clause and others) · Pillow already
shipped at pillow>=11.0, resolved 12.3.0.

[JArmandoAnaya]

Decision (Armando, 2026-08-07): Data augmentation lands in 0.1.0. The post-0.2.0 framing in the earlier decision comment is superseded — it was a working presumption, never a ratified decision. Sequencing consequence: the Exporter port change (fold resolution above the plugin, fold map as required argument) ships with this feature in 0.1.0; the formats work in cf. #365 inherits the already-changed port rather than carrying the change itself. — supersedes any prior direction