From a153f581b33ef54960b98826e47422b76f5c244b Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Thu, 1 Oct 2026 15:11:59 +0200 Subject: [PATCH 01/41] Add implementation plan for honouring instance-segmentation tradeoff_factor mask_decode_mode and tradeoff_factor are accepted end to end and then ignored: they pass untouched through InferenceModelsInstanceSegmentationAdapter.map_inference_kwargs and are swallowed by **kwargs in each model's post_process. Masks are always produced at full input resolution. The plan proposes a mask_resolution_factor in [0,1], resolved in the adapter from the existing enum, carried as explicit metadata rather than inferred from array shape. Default 1.0 is byte-identical to today. Measured on 4K with 300 instances: polygon path 889 ms -> 92 ms, RLE path 3107 ms -> 271 ms, peak RSS 3256 MB -> 595 MB at t=0.25. On Jetson Orin NX the stage currently moves ~40 GB per frame. Documents the blocking prerequisite: there is no mask-to-image coordinate scaling in the inference_models response path, so reduced-resolution masks would silently corrupt every polygon and RLE payload. --- docs/feat/instance-seg-tradeoff-factor.md | 218 ++++++++++++++++++++++ 1 file changed, 218 insertions(+) create mode 100644 docs/feat/instance-seg-tradeoff-factor.md diff --git a/docs/feat/instance-seg-tradeoff-factor.md b/docs/feat/instance-seg-tradeoff-factor.md new file mode 100644 index 0000000000..4f997c1e01 --- /dev/null +++ b/docs/feat/instance-seg-tradeoff-factor.md @@ -0,0 +1,218 @@ +# Implementation plan: honour `tradeoff_factor` for instance segmentation + +**Branch:** `feat/instance-seg-tradeoff-factor` (off `main` @ `23a01eb3a`) +**Status:** draft for `#discuss-inference-release` — not yet agreed + +Follows [`.github/implementation-plan-template.md`](../../.github/implementation-plan-template.md), required by +[`AGENTS.md`](../../AGENTS.md#internal-contributions-roboflow-team-only) for a major feature, and separately +because this touches shared infrastructure, compatibility and package dependencies. + +--- + +## 1. What problem are we solving? + +**Who.** Anyone running instance segmentation on large frames or dense scenes — real-time video especially, and +Jetson deployments above all. + +**What is missing.** `mask_decode_mode` and `tradeoff_factor` are accepted end to end (SDK entity → Workflows +block → `WorkflowsModelsProvider.run_instance_segmentation` → `InstanceSegmentationInferenceRequest`) and then +ignored. They pass untouched through +`InferenceModelsInstanceSegmentationAdapter.map_inference_kwargs` (`inference/core/models/inference_models_adapters.py:520`) +and are swallowed by `**kwargs` in each model's `post_process`. No error, no effect. Masks are always produced +at full input resolution. The legacy path (`inference/core/models/instance_segmentation_base.py:143-190`) still +implements all three modes. + +**Concrete example.** A 3840×2160 frame with 300 instances, 160×160 prototypes, on CPU: + +| stage | today (`t=1.0`) | at `t=0.25` | +|---|---|---| +| resize | 575 ms | 53 ms | +| → polygon path total (`masks2poly`) | **889 ms** | **92 ms** (9.7×) | +| → RLE path total (`rle` encode = 2531 ms) | **3107 ms** | **271 ms** (11.5×) | +| peak RSS | 3256 MB | 595 MB (5.5×) | + +Measured on Apple M1 Max, torch 2.14.1, 8 threads, fresh process per cell, warm-up 3, 15 iterations, p50. +The timed code is a verbatim copy of `post_processing.py:469-483`. + +**Note which path dominates.** On the **default** v4 path the masks are RLE, and **RLE encoding costs 4.4× the +resize**. The saving is therefore larger than a "faster resize" framing suggests, and any claim that the resize +alone is the bottleneck is wrong. + +**Edge.** At `t=1.0` this stage moves ~40 GB of memory traffic per 4K frame. On Jetson Orin NX (102 GB/s, +shared with the TRT engine) that is ~392 ms of post-processing per frame. Here the knob is not a tuning option; +it is the difference between shipping and not. + +**What should become possible.** A caller chooses mask resolution per request. Default `1.0` keeps today's +behaviour byte-for-byte. + +### The published contract this must honour + +Only four statements are published (the `inference.roboflow.com` reference site was collapsed to a landing page +in `cbdd286c4`, so the mkdocstrings API pages are gone): + +- `mask_decode_mode` default `accurate`. +- *"If 'accurate' the mask will be decoded using the original image size. If 'fast' the mask will be decoded + using the original mask size. 'accurate' is slower but more accurate."* — the only published statement about + output resolution, and therefore binding. +- `tradeoff_factor` default `0`, scale `0='fast'` → `1='accurate'`. +- `response_mask_format` default `polygon`, `rle` opt-in. + +Two gaps worth fixing alongside: the published description enumerates only `accurate` and `fast`, so **the +`tradeoff` mode this feature exists to serve is undocumented**; and the docstring at +`instance_segmentation_base.py:78` says "Defaults to 0.5" where line 33 says `0.0` — source-only, no longer +published, but visible in IDEs and to agents. + +## 2. How will the behaviour change? + +One new float, `mask_resolution_factor ∈ [0,1]`, resolved in the adapter from the existing enum + float, and +**carried as explicit metadata** alongside the masks rather than inferred from array shape. + +### Before + +Masks are resized to full image resolution, so `mask.shape[1:]` *is* the image canvas. Every downstream +consumer relies on that coincidence, and nothing scales mask coordinates into image space. + +```mermaid +flowchart LR + A[protos 160x160
+ coefficients] --> B[crop to boxes
at proto res] + B --> C[resize ALL instances
to full image res] + C --> D[binarize to bool
n x H x W] + D --> E[masks2poly / RLE encode] + E --> F[response points
correct only because
mask res == image res] +``` + +### After + +```mermaid +flowchart LR + A[protos + coefficients] --> B[crop to boxes
at proto res] + B --> C[resize to t-scaled target
h = mh + t*(H-mh)] + C --> D[bool n x h x w
+ explicit mask canvas] + D --> E[masks2poly / RLE encode
at reduced res] + E --> F[scale coords by
image/mask ratio] + F --> G[response points
in image space] +``` + +Adapter mapping: `accurate → 1.0`, `fast → 0.0`, `tradeoff → tradeoff_factor` (validated, raising +`InvalidMaskDecodeArgument` as the legacy path does). This satisfies the published contract exactly. + +**Why this fits.** The legacy ONNX path already does precisely this: it tracks `output_mask_shape` per mode and +rescales via `post_process_polygons` (`inference/core/utils/postprocess.py:449`). The `inference_models` +rewrite dropped that step. We are restoring a mechanism the codebase already had, not inventing one. + +### Effects on existing users + +**Step F is the whole change.** Without it every prediction at `t<1` is silently wrong. A review found no +mask→image scaling anywhere in the new response path: the adapter emits raw `masks2poly` output straight into +`Point(x,y)` (`inference_models_adapters.py:917-988`, `:1047`) while declaring the image at `original_size`. +Two sites are worse — `bitpacked_masks2poly(..., width=W)` with `W` from `original_size` +(`:907`, `core/utils/postprocess.py:73`) unpacks at the wrong stride and corrupts silently, and +`InstancesRLEMasks` declares `image_size = original_size` while encoding at `size_after_pre_processing` +(`rfdetr/common.py:383`), which breaks COCO decode. + +A blast-radius sweep found **20 silent-corruption sites and 13 hard crashes** if reduced masks escape unscaled. +Worst of each: dataset upload persists mask-space polygons as **permanent labels** +(`sinks/dataset_upload/v1_tensor.py:971`); Dynamic Zones emits shrunken zone polygons that corrupt every +downstream count (`transformations/dynamic_zones/v1.py:348`); several blocks overwrite `xyxy` from mask +contours (`bounding_rect/v1.py:163`), so corrupted boxes reach blocks that never touch masks; and Mask +Visualization raises outright (`visualizations/mask/v1_tensor.py:278`, verified). + +Two useful facts: `sv.Detections.from_inference` **already** resizes mismatched RLE +(`supervision/detection/utils/internal.py:126-132`) but not polygons (`:155`); and `Detections` never validates +mask shape against an image, only cardinality — so it cannot catch this for us. + +**Default `1.0` is byte-identical.** `round(mh*(1-t) + H*t)` at `t=1.0` is exactly `H`. + +### How we verify + +- Bit-identical output at `t=1.0` (regression guard). +- **Polygon/RLE coordinate round-trip at `t<1`** — assert emitted coordinates land on the same image pixels as + `t=1.0` within tolerance. Shape-only tests would not catch the defect above; this is the test that matters. +- RLE invariant: decoded counts sum to the declared size, and the declared size is the encoded size. +- Non-square fixtures throughout (a square image hides axis errors); odd dims; letterbox *and* stretch; + static crop with offset; `n = 0`. +- Triton gate: RF-DETR-seg has a fused CUDA post-process that bypasses this function entirely + (`models/rfdetr/triton_postprocess.py`); `t != 1.0` must force its eager fallback. +- Benchmark sweeping `t × n`, reporting p50/p95 of post-processing in isolation, with hardware stated. +- `AP_mask` 50:95 overall **and by object size**, against `t`, with `AP_box` as control. Deliverable is the + curve; the recommended default follows from it. + +### Scope and sequencing + +Four of these are independent of the feature and should land first as separate, bit-identical commits — they +are measured, low-risk, and make the feature's own diff reviewable: + +1. `torch.gt(..., out=)` instead of `.gt_()` + assignment — removes two full-size passes; 1.07–1.51×. +2. Separable `crop_masks_to_boxes` — the four broadcast comparisons each materialise `[n,h,w]`; keeping them + at `[n,1,w]`/`[n,h,1]` is 1.9–2.2×, `torch.equal`-identical. +3. Resize directly into the static-crop canvas — removes a second full `n×H×W` bool buffer. +4. `crop_masks_to_boxes(scaling=0.25)` — one hardcoded scalar where per-axis values derived from + `masks.shape` and `inference_size` are correct. Wrong for YOLACT (138/550 = 0.2509) and for any non-square + inference size, and it ignores letterbox padding while `align_instance_segmentation_results` twenty lines + later derives the same ratio properly. Correctness, not performance — and it changes `t=1.0` output, so it + must not be bundled with the byte-identity claim. + +Then the feature: `mask_resolution_factor` + canvas metadata + coordinate scaling + RLE parity. + +Explicitly **out of scope**, as its own later design: box-resolution (compact) masks, Θ(Σ box areas), +measured 7.6–11.2× at full fidelity. The naive per-box `grid_sample` implementation is 2.7–8.8× *slower* than +today; it needs batched `roi_align`. Worth doing, not here. + +## 3. Uncertainties and decisions needing input + +**Q1 — Does `tradeoff_factor`'s default of `0.0` stand once the parameter is honoured?** +*Investigation:* `DEFAULT_TRADEOFF_FACTOR = 0.0` (`instance_segmentation_base.py:33`), all 8 block versions +declare `default=0.0`, published OpenAPI says `0`. So `mask_decode_mode="tradeoff"` with an untouched factor +means `t=0.0` — identical to `fast`, maximum degradation. Any existing workflow that set `tradeoff` and left +the factor alone (the natural case, since it had no effect) lands there on upgrade. +*Recommendation:* honour the parameter in a **new block version** rather than in place, so no existing workflow +changes behaviour on upgrade. Separately make `tradeoff` with a default factor an explicit error or clamp it, +rather than silently meaning `fast`. +*Input needed:* is a new block version acceptable, or is in-place with a migration note preferred? **Blocking.** + +**Q2 — Where does the mask canvas live?** +*Investigation:* `InstancesRLEMasks.image_size` is read as "the image canvas" at ~15 sites; for dense masks, +`mask.shape[1:]` *is* the canvas definition at `bounding_rect/v1_tensor.py:211`, +`dataset_upload/v1_tensor.py:990`, `stabilize_detections/v1_tensor.py:448`. Changing either meaning silently +is what produces the 20 corruption sites. +*Recommendation:* keep `image_size` meaning the image and add a sibling `mask_size`; add an explicit +mask-canvas field to `InstanceDetections`. Both are public dataclasses in a separately-versioned package → +changelog and consumer sweep. +*Input needed:* agreement on the field names and on accepting the `inference_models` API addition. **Blocking.** + +**Q3 — Does the RLE wire format change, or do we upscale before encoding?** +*Investigation:* `rle["size"] ≠ image` is a breaking change for any external client that is not +`supervision ≥` the version carrying `internal.py:126`. v4 hardcodes `response_mask_format="rle"` +(`v4.py:357`, `:468`) and `GCP_SERVERLESS` forces it (`inference_models_adapters.py:527`), so this is the +default path, not an edge case. +*Recommendation:* for v1 of the feature, upscale before RLE encoding — correct on the wire, and the compute +saving is still taken. Revisit when compact masks land. +*Input needed:* accept the reduced saving on the RLE path, or version the response? **Blocking.** + +**Q4 — Should `mask_chunk_size` become a byte budget?** +*Investigation:* the optimal chunk size moves *inversely* to `t`. At 4K/`n=300`/`t=1.0` the optimum is 32; at +`t=0.1` it is unchunked, where the fixed 16 is 1.43× off (1.70× at 1080p). Per-chunk overhead is ~0.25 ms, i.e. +22% of the whole budget at `t=0.1` versus <1% at `t=1.0`. Chunking bounds the float temporary only; it never +shrinks the bool output, so it is orthogonal to this feature, not a substitute. +*Recommendation:* express it as `chunk = max(1, min(n, BUDGET_BYTES // (h*w*4)))` with a ~512 MiB budget, +keeping the env var as the budget. Reproduces today's 16 at 4K/`t=1.0` and the measured optimum at low `t`. +*Input needed:* in scope here, or a follow-up? Non-blocking, but leaves 1.2–1.7× unclaimed at exactly the +settings this feature enables. + +**Q5 — Recommended value and documentation.** +*Investigation:* the knee is at `t ≈ 0.25–0.3`, capturing ~90% of the achievable latency saving and 5.5× of the +memory. Below 0.15 the gain flattens while resolution keeps falling. +*Recommendation:* keep the default at `1.0`; document `0.25` as the recommended setting for real-time and +edge, and `t < 0.15` as diagnostic only. Also document `tradeoff` as a legal `mask_decode_mode` — it currently +is not — and fix the `0.5` docstring. +*Input needed:* confirmation once the `AP_mask` curve exists. Non-blocking. + +**Closed by investigation.** Does `fast` need a separate code path? No — `t=0.0` *is* legacy +`process_mask_fast` semantics, because the crop already happens at proto resolution before +`align_instance_segmentation_results` (`models/yolov8/common.py:32`) and `t=0` skips the resize. + +**Not porting `process_mask_tradeoff`.** It passes `(h, w)` where `cv2.resize` wants `dsize=(width, height)` +(`core/utils/postprocess.py:353`), so it transposes masks on non-square images — verified by execution, +1920×1080 at `t=1.0` returns `(n, 1920, 1080)` — and `scale_bboxes` then derives crop scaling from the swapped +dims. `process_mask_fast` has a related defect on letterboxed input (`:394`). Invisible on square images, +which is why it survived. From 2fbc39ba8d9134a997e8f9253db1faf810e1294a Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Thu, 1 Oct 2026 15:17:32 +0200 Subject: [PATCH 02/41] Fix Mermaid diagram rendering in the tradeoff_factor plan --- docs/feat/instance-seg-tradeoff-factor.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/feat/instance-seg-tradeoff-factor.md b/docs/feat/instance-seg-tradeoff-factor.md index 4f997c1e01..82660a3dd4 100644 --- a/docs/feat/instance-seg-tradeoff-factor.md +++ b/docs/feat/instance-seg-tradeoff-factor.md @@ -74,23 +74,23 @@ consumer relies on that coincidence, and nothing scales mask coordinates into im ```mermaid flowchart LR - A[protos 160x160
+ coefficients] --> B[crop to boxes
at proto res] - B --> C[resize ALL instances
to full image res] - C --> D[binarize to bool
n x H x W] - D --> E[masks2poly / RLE encode] - E --> F[response points
correct only because
mask res == image res] + A["protos 160x160 + coefficients"] --> B["crop to boxes, at proto res"] + B --> C["resize ALL instances to full image res"] + C --> D["binarize to bool: n x H x W"] + D --> E["masks2poly / RLE encode"] + E --> F["response points
correct only because mask res == image res"] ``` ### After ```mermaid flowchart LR - A[protos + coefficients] --> B[crop to boxes
at proto res] - B --> C[resize to t-scaled target
h = mh + t*(H-mh)] - C --> D[bool n x h x w
+ explicit mask canvas] - D --> E[masks2poly / RLE encode
at reduced res] - E --> F[scale coords by
image/mask ratio] - F --> G[response points
in image space] + A["protos + coefficients"] --> B["crop to boxes, at proto res"] + B --> C["resize to t-scaled target
h = mh + t * [H - mh]"] + C --> D["bool n x h x w
+ explicit mask canvas"] + D --> E["masks2poly / RLE encode
at reduced res"] + E --> F["scale coords by image/mask ratio"] + F --> G["response points in image space"] ``` Adapter mapping: `accurate → 1.0`, `fast → 0.0`, `tradeoff → tradeoff_factor` (validated, raising From f92889a9d436df7b2161a2dd0b3976f59b76b7be Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Thu, 1 Oct 2026 21:02:29 +0200 Subject: [PATCH 03/41] Pin mask output geometry and add polygon-to-image scaling The existing coverage of align_instance_segmentation_results compares chunked output against monolithic output. Both sides move together when the resize target changes, so neither test can detect a change in output geometry. crop_masks_to_boxes had no coverage at all. Add characterization tests that pin the current contract: masks sized to size_after_pre_processing, bool dtype, axis order on a non-square image (a transpose is invisible on a square one), a golden per-instance pixel count from a fixed seed, and that the static-crop paste relocates onto a larger canvas without changing the set-pixel count. Add first coverage for crop_masks_to_boxes, including one test documenting that `scaling` is a single scalar applied to both axes. Add scale_polygons_to_image, which lifts contour coordinates from mask space into image space, scaling each axis independently since the two need not share an aspect ratio. The legacy path does this via post_process_polygons / scale_polygons (inference/core/utils/postprocess.py); the inference_models rewrite dropped it, which is harmless only while masks happen to be image-sized. No caller is changed here. Tests: python -m pytest tests/unit_tests/models/common/roboflow/test_post_processing.py 45 passed (31 pre-existing, 14 added). --- .../models/common/roboflow/post_processing.py | 33 +++ .../common/roboflow/test_post_processing.py | 218 ++++++++++++++++++ 2 files changed, 251 insertions(+) diff --git a/inference_models/inference_models/models/common/roboflow/post_processing.py b/inference_models/inference_models/models/common/roboflow/post_processing.py index 4472aa26e0..0bb7b0e4dc 100644 --- a/inference_models/inference_models/models/common/roboflow/post_processing.py +++ b/inference_models/inference_models/models/common/roboflow/post_processing.py @@ -1,5 +1,6 @@ from typing import Dict, Generator, List, Literal, Optional, Tuple, Union +import numpy as np import torch import torchvision from torchvision.transforms import functional @@ -392,6 +393,38 @@ def crop_masks_to_boxes( return masks * crop_mask +def scale_polygons_to_image( + polygons: List[np.ndarray], + mask_size: ImageDimensions, + image_size: ImageDimensions, +) -> List[np.ndarray]: + """Lift polygon coordinates from mask space into image space. + + Mask contours are extracted in the coordinate system of the mask they came + from. When that mask is smaller than the image it describes, the contour + must be rescaled before it can be reported as a prediction. Each axis is + scaled independently, since the mask and the image need not share an aspect + ratio. + + Args: + polygons: Contours in mask coordinates, each of shape `(k, 2)` as `(x, y)`. + mask_size: Dimensions of the mask the contours were extracted from. + image_size: Dimensions of the image the contours should be reported in. + + Returns: + Contours in image coordinates. The input is returned unchanged when the + two sizes already match. + """ + if not polygons: + return [] + x_scale = image_size.width / mask_size.width + y_scale = image_size.height / mask_size.height + if x_scale == 1.0 and y_scale == 1.0: + return polygons + scale = np.array([x_scale, y_scale], dtype=np.float32) + return [polygon * scale for polygon in polygons] + + def align_instance_segmentation_results( image_bboxes: torch.Tensor, masks: torch.Tensor, diff --git a/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py b/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py index 6aa10af011..5b3d35672a 100644 --- a/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py +++ b/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py @@ -3,8 +3,10 @@ - ConfidenceFilter: 4-tier priority chain and `get_threshold()` - NMS helpers: per-class `conf_thresh` tensor path + - Mask geometry: characterization of the output contract """ +import numpy as np import pytest import torch @@ -17,6 +19,8 @@ from inference_models.models.common.roboflow.post_processing import ( ConfidenceFilter, align_instance_segmentation_results, + crop_masks_to_boxes, + scale_polygons_to_image, post_process_nms_fused_model_output, rescale_image_detections, rescale_key_points_detections, @@ -518,3 +522,217 @@ def test_chunked_resize_matches_monolithic_with_static_crop_canvas(self) -> None # then assert torch.equal(out_bboxes, ref_bboxes) assert torch.equal(out_masks, ref_masks) + + +class TestAlignInstanceSegmentationResultsGeometry: + """Characterization of the mask output contract. + + The chunking tests above are invariance tests: they compare chunked output + against monolithic output, so a change to the resize target moves both sides + and both keep passing. These tests pin the geometry itself, which is what a + resolution change would alter. + """ + + GOLDEN_SET_PIXELS = 210077 + GOLDEN_PER_INSTANCE = [30206, 30285, 30006, 29972, 29811, 30015, 29782] + + @staticmethod + def _run(static_crop: bool = False, orig_h: int = 200, orig_w: int = 300): + torch.manual_seed(42) + n = 7 + bboxes = torch.rand((n, 6), dtype=torch.float32) * 100 + masks = torch.randn((n, 160, 160), dtype=torch.float32) + static_crop_offset = StaticCropOffset( + offset_x=13 if static_crop else 0, + offset_y=7 if static_crop else 0, + crop_width=orig_w, + crop_height=orig_h, + ) + return align_instance_segmentation_results( + image_bboxes=bboxes.clone(), + masks=masks.clone(), + padding=(0, 0, 0, 0), + scale_width=1.0, + scale_height=1.0, + original_size=ImageDimensions( + height=orig_h + (7 if static_crop else 0), + width=orig_w + (13 if static_crop else 0), + ), + size_after_pre_processing=ImageDimensions(height=orig_h, width=orig_w), + inference_size=ImageDimensions(height=640, width=640), + static_crop_offset=static_crop_offset, + binarization_threshold=0.0, + mask_chunk_size=1000, + ) + + def test_masks_are_sized_to_size_after_pre_processing(self) -> None: + # given / when + _, masks = self._run() + + # then + assert masks.shape[1:] == (200, 300) + + def test_masks_are_bool(self) -> None: + # given / when + _, masks = self._run() + + # then + assert masks.dtype == torch.bool + + def test_non_square_image_preserves_axis_order(self) -> None: + # given / when + # a transposed resize would return (300, 200) and is invisible on a square image + _, masks = self._run(orig_h=200, orig_w=300) + + # then + assert masks.shape[1] == 200 + assert masks.shape[2] == 300 + + def test_golden_output_for_fixed_seed(self) -> None: + # given / when + _, masks = self._run() + + # then + assert int(masks.sum().item()) == self.GOLDEN_SET_PIXELS + assert [int(v) for v in masks.flatten(1).sum(1)] == self.GOLDEN_PER_INSTANCE + + def test_static_crop_pastes_onto_original_size_canvas(self) -> None: + # given / when + _, masks = self._run(static_crop=True) + + # then + assert masks.shape[1:] == (207, 313) + + def test_static_crop_preserves_set_pixel_count(self) -> None: + # given / when + # the paste relocates content onto a larger canvas; it must not lose + # or add pixels + _, masks = self._run(static_crop=True) + + # then + assert int(masks.sum().item()) == self.GOLDEN_SET_PIXELS + + +class TestCropMasksToBoxes: + """First coverage for crop_masks_to_boxes.""" + + @staticmethod + def _masks_and_boxes(): + masks = torch.ones((2, 40, 60), dtype=torch.float32) + boxes = torch.tensor( + [[4.0, 8.0, 40.0, 24.0], [0.0, 0.0, 8.0, 8.0]], dtype=torch.float32 + ) + return masks, boxes + + def test_preserves_mask_shape(self) -> None: + # given + masks, boxes = self._masks_and_boxes() + + # when + result = crop_masks_to_boxes(boxes.clone(), masks.clone()) + + # then + assert result.shape == (2, 40, 60) + + def test_zeroes_everything_outside_the_scaled_box(self) -> None: + # given + # the default scaling of 0.25 maps box 0 to (1, 2, 10, 6) -> 9 x 4 = 36 pixels + masks, boxes = self._masks_and_boxes() + + # when + result = crop_masks_to_boxes(boxes.clone(), masks.clone()) + + # then + assert int((result[0] > 0).sum().item()) == 36 + + def test_box_at_origin_is_cropped_from_zero(self) -> None: + # given + # box 1 is (0, 0, 8, 8) -> scaled (0, 0, 2, 2) -> 2 x 2 = 4 pixels + masks, boxes = self._masks_and_boxes() + + # when + result = crop_masks_to_boxes(boxes.clone(), masks.clone()) + + # then + assert int((result[1] > 0).sum().item()) == 4 + + def test_scaling_is_a_single_scalar_for_both_axes(self) -> None: + # given + # documents current behaviour: one scalar is applied to x and y alike, so a + # proto grid whose stride differs from 4, or differs per axis, is mis-cropped + masks, boxes = self._masks_and_boxes() + + # when + default_scaling = crop_masks_to_boxes(boxes.clone(), masks.clone()) + explicit_quarter = crop_masks_to_boxes( + boxes.clone(), masks.clone(), scaling=0.25 + ) + + # then + assert torch.equal(default_scaling, explicit_quarter) + + +class TestScalePolygonsToImage: + """Mask-space polygon coordinates must be lifted into image space. + + The legacy path does this via `post_process_polygons` / `scale_polygons` + (inference/core/utils/postprocess.py:449, :500). The inference_models rewrite + dropped it, which is harmless only while masks happen to be image-sized. + """ + + def test_identity_when_mask_matches_image(self) -> None: + # given + polygons = [np.array([[0.0, 0.0], [10.0, 20.0]], dtype=np.float32)] + + # when + result = scale_polygons_to_image( + polygons=polygons, + mask_size=ImageDimensions(height=200, width=300), + image_size=ImageDimensions(height=200, width=300), + ) + + # then + assert np.allclose(result[0], polygons[0]) + + def test_scales_each_axis_independently(self) -> None: + # given + # a non-square ratio: x by 300/150 = 2.0, y by 200/50 = 4.0 + polygons = [np.array([[10.0, 10.0], [50.0, 25.0]], dtype=np.float32)] + + # when + result = scale_polygons_to_image( + polygons=polygons, + mask_size=ImageDimensions(height=50, width=150), + image_size=ImageDimensions(height=200, width=300), + ) + + # then + assert np.allclose(result[0], np.array([[20.0, 40.0], [100.0, 100.0]])) + + def test_reduced_mask_coords_reach_image_extent(self) -> None: + # given + # the defect this phase fixes: a polygon touching the edge of a half-size + # mask must land on the edge of the image, not halfway across it + polygons = [np.array([[0.0, 0.0], [960.0, 540.0]], dtype=np.float32)] + + # when + result = scale_polygons_to_image( + polygons=polygons, + mask_size=ImageDimensions(height=540, width=960), + image_size=ImageDimensions(height=1080, width=1920), + ) + + # then + assert result[0][:, 0].max() == pytest.approx(1920.0) + assert result[0][:, 1].max() == pytest.approx(1080.0) + + def test_empty_polygon_list(self) -> None: + # given / when + result = scale_polygons_to_image( + polygons=[], + mask_size=ImageDimensions(height=50, width=150), + image_size=ImageDimensions(height=200, width=300), + ) + + # then + assert result == [] From a40c809e72af7f98abf5ee2599869e8aed8f497c Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Thu, 1 Oct 2026 21:10:25 +0200 Subject: [PATCH 04/41] Add implementation plan and TDD sequence for mask resolution factor Supersedes the earlier plan, whose headline figures combined two code paths that never execute together and whose recommended v1 is measurably slower than today on the default RLE path. Records the team decisions from review: new block versions are the delivery mechanism, mask_size as carried metadata is accepted in principle, and the RLE wire-format decision is deferred to the end of implementation. Adds the TDD sequence, including why the existing chunking tests cannot detect a geometry change. Applies black and isort to the files touched by the previous commit. --- docs/feat/instance-seg-tradeoff-factor-tdd.md | 214 ++++++++++++++ docs/feat/instance-seg-tradeoff-factor-v2.md | 275 ++++++++++++++++++ .../common/roboflow/test_post_processing.py | 2 +- 3 files changed, 490 insertions(+), 1 deletion(-) create mode 100644 docs/feat/instance-seg-tradeoff-factor-tdd.md create mode 100644 docs/feat/instance-seg-tradeoff-factor-v2.md diff --git a/docs/feat/instance-seg-tradeoff-factor-tdd.md b/docs/feat/instance-seg-tradeoff-factor-tdd.md new file mode 100644 index 0000000000..4783c945ac --- /dev/null +++ b/docs/feat/instance-seg-tradeoff-factor-tdd.md @@ -0,0 +1,214 @@ +# TDD sequence: `masks_resolution_factor` + +Companion to `instance-seg-tradeoff-factor-v2.md`. Each phase is red → green → commit. + +## Existing coverage, measured not assumed + +`inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py` (28 tests) **does** +cover `align_instance_segmentation_results`, via `test_chunked_resize_matches_monolithic` (:502) and +`..._with_static_crop_canvas`. But both are **invariance** tests — they assert chunked output equals +monolithic output. Change the resize target and both sides move together, so **both still pass**. + +Nothing pins output *geometry*: not the shape relative to the image, not coordinate correctness. +`crop_masks_to_boxes` is not imported and has no tests at all. + +So phase 0 is not optional ceremony — it is the only thing that would catch a geometry regression. + +--- + +## Phase 0 — characterization (GREEN on current code) + +Pin today's behaviour before touching anything. These must pass unchanged at `t=1.0` forever after. + +```python +class TestCurrentMaskGeometry: + """Characterization. These encode today's contract; they must not change at t=1.0.""" + + def test_masks_are_image_sized(self) -> None: + _, masks = self._run() + assert masks.shape[1:] == (self.size_after_pre_processing.height, + self.size_after_pre_processing.width) + + def test_masks_are_bool(self) -> None: + _, masks = self._run() + assert masks.dtype == torch.bool + + def test_golden_output_for_fixed_seed(self) -> None: + # seeded protos -> exact tensor. Any geometry change breaks this loudly. + torch.manual_seed(0) + _, masks = self._run() + assert masks.sum().item() == GOLDEN_SET_PIXELS # fill from first run + assert masks[0, :8, :8].tolist() == GOLDEN_CORNER # fill from first run + + def test_non_square_image_preserves_aspect(self) -> None: + # the axis-swap guard. Square fixtures hide transposes. + _, masks = self._run(image=(1080, 1920)) + assert masks.shape[1] == 1080 and masks.shape[2] == 1920 +``` + +And the first tests `crop_masks_to_boxes` has ever had: + +```python +class TestCropMasksToBoxes: + def test_zeroes_outside_box(self) -> None: ... + def test_scaling_default_is_proto_stride(self) -> None: ... + def test_non_square_inference_size(self) -> None: + # documents that a single scalar `scaling` is wrong when mask_w/inference_w != 0.25 + ... +``` + +**Commit 0.** No production change. + +## Phase 1 — RED: the coordinate-scaling gap + +The prerequisite from §2 of the plan. This test fails today and proves the gap exists. + +```python +def test_polygon_coords_are_image_space_when_mask_is_smaller(self) -> None: + # given: a mask deliberately smaller than the declared image + masks = make_masks(n=2, h=540, w=960) # half of 1080x1920 + meta = make_metadata(original_size=(1080, 1920)) + + # when + response = build_instance_segmentation_response(masks, meta) + + # then: coordinates must span the IMAGE, not the mask + xs = [p.x for pred in response.predictions for p in pred.points] + assert max(xs) > 960, "polygon coords left in mask space" +``` + +RED today — `inference_models_adapters.py:917` emits raw `masks2poly` output. Green once scaling is +applied at the mask→polygon boundary, mirroring legacy `post_process_polygons` +(`inference/core/utils/postprocess.py:449`). + +**Commit 1.** Coordinate scaling only. No new parameter yet — and it is a latent-correctness fix on +its own merits. + +## Phase 2 — RED: the parameter + +```python +@pytest.mark.parametrize("t", [0.0, 0.1, 0.25, 0.5, 1.0]) +def test_resolution_factor_interpolates_shape(self, t: float) -> None: + _, masks = self._run(masks_resolution_factor=t) # TypeError today -> RED + mh, mw = PROTO_UNPADDED + assert masks.shape[1] == max(1, round(mh * (1 - t) + 1080 * t)) + assert masks.shape[2] == max(1, round(mw * (1 - t) + 1920 * t)) + +def test_t1_is_bit_identical_to_baseline(self) -> None: + _, ref = self._run() # no kwarg + _, out = self._run(masks_resolution_factor=1.0) + assert torch.equal(out, ref) + +def test_t0_is_proto_resolution(self) -> None: + _, masks = self._run(masks_resolution_factor=0.0) + assert masks.shape[1:] == PROTO_UNPADDED + +@pytest.mark.parametrize("t", [0.1, 0.25, 0.5]) +def test_coords_round_trip_against_t1(self, t: float) -> None: + ref = polygons_from(self._run()) + out = polygons_from(self._run(masks_resolution_factor=t)) + # unpad rounding error is amplified as t falls, so tolerance scales with 1/t + assert_polygons_close(out, ref, tol_px=2.0 / t) + +def test_zero_instances(self) -> None: + _, masks = self._run(n=0, masks_resolution_factor=0.25) + assert masks.shape[0] == 0 + +@pytest.mark.parametrize("mode", ["letterbox", "stretch"]) +def test_both_resize_modes(self, mode: str) -> None: ... + +def test_static_crop_offset_is_scaled(self) -> None: ... +``` + +The `t=1.0` bit-identity test is the safety net for every later phase. + +**Commit 2.** Dense path only. + +## Phase 3 — RED: `mask_size` metadata (plan O1) + +```python +def test_mask_size_defaults_to_image_size(self) -> None: + # back-compat: existing construction sites must not change + m = InstancesRLEMasks.from_coco_rle_masks(image_size=(1080, 1920), masks=[...]) + assert m.mask_size == (1080, 1920) + +def test_mask_size_reflects_reduced_resolution(self) -> None: + dets = self._run_full_pipeline(masks_resolution_factor=0.25) + assert dets.mask_size == (270, 480) + assert dets.image_size == (1080, 1920) # unchanged meaning +``` + +**Commit 3.** + +## Phase 4 — RED: adapter mapping + +```python +@pytest.mark.parametrize("mode,expected", [("accurate", 1.0), ("fast", 0.0)]) +def test_enum_maps_to_factor(self, mode: str, expected: float) -> None: ... + +def test_tradeoff_reads_the_factor(self) -> None: ... + +@pytest.mark.parametrize("bad", [-0.1, 1.1]) +def test_out_of_range_raises(self, bad: float) -> None: + with pytest.raises(InvalidMaskDecodeArgument): + ... + +def test_source_keys_do_not_leak_to_model_call(self) -> None: + # map_inference_kwargs must consume both, not forward them + mapped = adapter.map_inference_kwargs({"mask_decode_mode": "fast", "tradeoff_factor": 0.0}) + assert "mask_decode_mode" not in mapped and "tradeoff_factor" not in mapped + assert mapped["masks_resolution_factor"] == 0.0 +``` + +**Commit 4.** + +## Phase 5 — RED: version skew (plan O2) and the Triton gate + +```python +def test_absent_mask_size_defaults_to_input_size(self) -> None: + # old server: field missing -> client assumes today's contract + dets = parse_response(response_without_mask_size()) + assert dets.mask_size == dets.image_size + +def test_unhonoured_hint_does_not_raise(self, caplog) -> None: + # old server ignores the parameter: correct output, just slower + dets = run_remote(masks_resolution_factor=0.25, server=OldServerStub()) + assert dets.mask_size == dets.image_size + assert "not honoured" in caplog.text # logged once, never raised + + +def test_reduced_resolution_forces_eager_fallback(self) -> None: + # pure-python: no GPU, so it runs in x86 CI instead of being a dead GPU-only test + reason = _unsupported_triton_postprocess_reason(..., masks_resolution_factor=0.25) + assert reason is not None +``` + +**Commit 5.** + +## Phase 6 — RLE parity + +Blocked on the deferred wire-format decision. Until then the RLE path keeps `t=1.0` and phase 0's +characterization tests guard it. The evidence experiment (plan O3) belongs here: + +```python +@pytest.mark.parametrize("decoder", ["pycocotools", "supervision"]) +def test_reduced_size_rle_decode_behaviour(self, decoder: str) -> None: + # not an assertion of correctness - a recorded observation of the failure mode, + # to inform the deferred decision: raises? truncates silently? resizes cleanly? + ... +``` + +## Running + +```bash +cd inference_models && python -m pytest tests/unit_tests/models/common/roboflow/test_post_processing.py -v +python -m pytest tests/inference/models_predictions_tests +python -m pytest tests/workflows/unit_tests/ +``` + +## Order and why + +Phase 0 before everything, because the existing invariance tests would not catch a geometry +regression. Phase 1 before phase 2, because coordinate scaling is a correctness fix that stands +alone and must be in place before any reduced mask can exist. Phase 6 last, because it is the only +part gated on a decision the team deferred. diff --git a/docs/feat/instance-seg-tradeoff-factor-v2.md b/docs/feat/instance-seg-tradeoff-factor-v2.md new file mode 100644 index 0000000000..a595f11077 --- /dev/null +++ b/docs/feat/instance-seg-tradeoff-factor-v2.md @@ -0,0 +1,275 @@ +# Implementation plan: honour `tradeoff_factor` for instance segmentation + +**Author:** Michał Taraszewski + +**Related issue / PR:** _none yet — to be linked from the `#discuss-inference-release` thread_ + +**Status:** draft. Supersedes `instance-seg-tradeoff-factor.md`, whose headline figures combined two +code paths that never execute together and whose recommended v1 is measurably slower than today. + +> **Reviewed by the team 2026-10-01.** The delivery mechanism is settled (new block versions); the +> RLE wire-format decision is deliberately deferred to the end of implementation. §3 records what was +> decided and what is still open. + +--- + +## 1. What problem are we solving? + +**Who.** Anyone running instance segmentation on large frames or dense scenes: real-time video, and +Jetson deployments in particular. + +**What is missing.** `mask_decode_mode` and `tradeoff_factor` are accepted end to end — SDK entity → +Workflows block → `WorkflowsModelsProvider.run_instance_segmentation` → +`InstanceSegmentationInferenceRequest` — and then ignored. They pass untouched through +`InferenceModelsInstanceSegmentationAdapter.map_inference_kwargs` +(`inference/core/models/inference_models_adapters.py:520`) and are swallowed by `**kwargs` in each +model's `post_process`. No error, no effect. Masks are always produced at full input resolution. +The legacy path (`inference/core/models/instance_segmentation_base.py:143-190`) still implements all +three modes. + +**Why it matters: this stage is almost the entire frame.** Measured on one host (M1 Max, 8 torch +threads, fp32, 3840×2160, p50): JPEG decode 59.7 ms, preprocess to 640×640 0.26 ms, JPEG encode +35.1 ms — everything that is *not* mask post-processing sums to roughly **95 ms**. Mask +post-processing at `n=300` is **hundreds of milliseconds to seconds** (tables below). At 4K with a +dense scene this stage is **~90–97% of total frame time**. That, not a stage-relative speedup, is +the argument. + +**Two things this plan does not claim.** It is not sufficient on its own — at `n=300`/4K even +`t=0.25` leaves several hundred ms in this stage. And cost is Θ(n·H·W), so **reducing +`max_detections`** (default 300, `inference_models/configuration.py:229`) is a cheaper, lossless +first move that should be evaluated alongside it. + +### The published contract + +The `inference.roboflow.com` reference site was collapsed to a landing page in `cbdd286c4`, so the +mkdocstrings API pages no longer exist. What remains published: + +- `mask_decode_mode` default `accurate`; `tradeoff_factor` default `0`, scale `0='fast'` → `1='accurate'`. +- *"If 'accurate' the mask will be decoded using the original image size. If 'fast' the mask will be + decoded using the original mask size."* — the only published statement about output resolution, + and therefore binding. +- `response_mask_format` default `polygon`, `rle` opt-in. + +The request schema (`inference/core/entities/requests/inference.py:254-257`) publishes all three +modes. Only the legacy query parameter (`inference/core/interfaces/http/http_api.py:4738`) +enumerates just two, omitting `tradeoff` — worth correcting, but it is a doc fix, not a blocker. + +## 2. How will the behaviour change? + +One float, `masks_resolution_factor ∈ [0,1]` (named to match the existing `masks_smoothing_enabled` +/ `masks_binarization_threshold` kwargs), resolved in the adapter from the existing enum, and +**carried as explicit metadata** rather than inferred from array shape. + +Prior art for exactly this: DeepStream's `NvOSD_MaskParams` carries mask-local `height`/`width` +separately from `rect_params` and rescales on demand +(`_reference/deepstream_python_apps/bindings/docstrings/nvosddoc.h:278-291`; +`apps/deepstream-segmask/deepstream_segmask.py:120-126`). + +### Before — 4K frame, 300 instances, 160×160 protos + +```mermaid +flowchart LR + A["protos: 300 x 160 x 160"] --> B["crop to boxes, at proto res"] + B --> C["resize every instance to 2160 x 3840"] + C --> D["bool 300 x 2160 x 3840"] + D --> E["masks2poly or RLE encode"] + E --> F["polygon coords in 2160x3840
RLE size 2160x3840"] +``` + +### After, at `t = 0.25` + +```mermaid +flowchart LR + A["protos: 300 x 160 x 160"] --> B["crop to boxes, at proto res"] + B --> C["resize to 660 x 1080"] + C --> D["bool 300 x 660 x 1080
+ mask canvas 660x1080"] + D --> E["masks2poly or RLE encode
at 660 x 1080"] + E --> F["scale coords x3.27
-> image space 2160x3840"] +``` + +Adapter mapping: `accurate → 1.0`, `fast → 0.0`, `tradeoff → tradeoff_factor` (validated, raising +`InvalidMaskDecodeArgument` as the legacy path does). This satisfies the published contract exactly. +Default `1.0` leaves output unchanged. + +### Step F is the change. Without it every prediction at `t<1` is wrong. + +There is **no mask→image coordinate scaling anywhere** in the `inference_models` response path. The +adapter emits raw `masks2poly` output into `Point(x, y)` +(`inference_models_adapters.py:917-988`, `:1047`) while declaring the image at `original_size`. Two +sites are worse: `bitpacked_masks2poly(..., width=W)` with `W` from `original_size` (`:907`, +`core/utils/postprocess.py:73`) unpacks at the wrong stride and corrupts silently; and +`InstancesRLEMasks` declares `image_size = original_size` while encoding at +`size_after_pre_processing` (`rfdetr/common.py:383`). + +The legacy path does this correctly — it tracks `output_mask_shape` per mode and rescales via +`post_process_polygons` (`inference/core/utils/postprocess.py:449`). **The `inference_models` +rewrite dropped that step.** We are restoring a mechanism the codebase already had. + +Verified consequences if reduced masks escape unscaled (4 confirmed; a wider sweep is in progress +and will be attached rather than asserted): `visualizations/mask/v1_tensor.py:277-280` raises +outright; `bounding_rect/v1.py:163-177` overwrites `xyxy` from mask contours, so corrupted boxes +reach blocks that never touch masks; `dynamic_zones/v1.py:348-352` emits shrunken zone polygons; +`sinks/dataset_upload/v1_tensor.py:946-951` persists mask-space polygons as permanent labels. +Note `sv.Detections.from_inference` already resizes mismatched RLE +(`supervision/detection/utils/internal.py:125-132`) but not polygons (`:155`). + +### Measurements, by path + +Two different functions. They are not comparable and must not be combined. + +**Dense path** — `align_instance_segmentation_results` (`post_processing.py:395`), chunked, +materialises `n×H×W`. 4K, n=300, chunk 16, fp32, CPU: + +| t | resize p50 | peak RSS | +|---|---|---| +| 1.0 | 485–575 ms (3 runs) | 3256 MB | +| 0.25 | 52–65 ms | 595 MB | + +**RLE path** — `align_instance_segmentation_results_to_rle_masks` (`post_processing.py:525`), a +**per-instance generator**: no chunking, no `n×H×W` tensor, so the memory figures above do not +apply. This is the default (`v4.py:359`, `:408` hardcode `"rle"`; `GCP_SERVERLESS` forces it, +`inference_models_adapters.py:527`). Encode cost is strongly content-dependent — measured 2933 ms +for smooth blob masks vs 6262 ms for noise masks at n=300, a 2.1× spread from fixture alone. + +**Open measurement gaps, to close before implementation.** The polygon-path figure in the superseded +plan did not reproduce (`masks2poly` measured ~1877 ms against ~314 ms implied) and is withdrawn. +All numbers above are single-platform CPU; there is **no CUDA or Jetson measurement**, and the +~40 GB/frame traffic figure for Orin NX is a bandwidth model, not data. The benchmark scripts will +be linked, with fixture content stated. + +### How we verify + +`align_instance_segmentation_results` and `crop_masks_to_boxes` currently have **no unit tests** +(`inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py` covers neither), +so a characterization test — fixed-seed protos → golden mask tensor and golden polygons — is +commit zero, before any behaviour claim can be checked in CI. + +- Bit-identical output at `t=1.0`. +- **Polygon/RLE coordinate round-trip at `t<1`**: emitted coordinates land on the same image pixels + as `t=1.0`. Tolerance must scale with `1/t`, since unpad rounding error is amplified as `t` falls. + Shape-only assertions would not catch the defect above; this is the test that matters. +- RLE invariant: decoded counts sum to the declared size; declared size is the encoded size. +- Non-square fixtures throughout (square images hide axis errors); odd dims; letterbox *and* + stretch; static crop with offset; `n = 0`. +- Triton gate as a pure-Python unit test on `_unsupported_triton_postprocess_reason`, so it is not a + GPU-only dead test in x86 CI. +- `AP_mask` 50:95 overall **and by object size**, against `t`, with `AP_box` as control. + +Layers: `inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py`; +`tests/inference/models_predictions_tests`; `tests/workflows/unit_tests/.../instance_segmentation/` +for any new block version **and its `_tensor` sibling**. + +### Implementation surface + +Not one function. The alignment logic is duplicated across two functions, reached from six model +families on both the dense and RLE routes — `models/{yolov5,yolov7,yolov8,yolo26,yolact}/common.py` +× {dense, rle} = 10 call sites, plus `models/rfdetr/common.py:369` and +`models/rfdetr/triton_postprocess.py` — plus the new kwarg threaded through every +`*_instance_segmentation_{onnx,trt,torch_script}.py`. Roughly 12 sites beyond the two core +functions. + +### Out of scope, as its own plan + +Four independent performance and correctness changes, measured and bit-identical except the last, +belong in a separate PR and are deliberately not bundled here: `torch.gt(out=)` in place of +`.gt_()` + assignment; separable `crop_masks_to_boxes`; resizing directly into the static-crop +canvas; and deriving `crop_masks_to_boxes(scaling=0.25)` per-axis (wrong whenever +`mask_w/inference_w ≠ 0.25`, e.g. YOLACT 138/550 — and the only one of the four that changes +`t=1.0` output). Also out of scope: batching the per-instance RLE encode, measured 1.4× at `t=1.0` +and 2.5× at `t=0.25`, lossless and with no API change. + +## 3. Decisions taken, and what remains + +### Settled by team review, 2026-10-01 + +Question numbers below are those of the reviewed draft, so the thread stays traceable. + +**(Q1) New block versions are the delivery mechanism.** Changes of this kind have always shipped as +new block versions to minimise breakage — *even when the change is a fix to an earlier bug* — and +code duplication across versions is explicitly acceptable. This removes the main compatibility +objection: a new version means no existing pipeline changes under upgrade. Both the `vN` and +`vN_tensor` siblings are in scope. + +**(Q2) `mask_size` as metadata carried with the prediction: accepted in principle.** Two +sub-decisions remain, below. + +**(Q3) The RLE wire-format decision is deferred to the end of implementation**, for wider team +discussion. The reasoning, which changes this plan's shape: *with new block versions this is not +breaking for Workflows at all.* The breaking surface is narrower than the reviewed draft assumed — +it is the **model endpoints**, where a direct integrator would suddenly see mask-scaling parameters +respected. That is a bug fix, and it is breaking, and those are not in conflict. Implementation +should proceed without settling it. + +**(Q4) The chunk byte budget is a follow-up, not part of this work.** It addresses an extremum of +the resolution range; it can land later purely as serving-latency optimisation. + +**(Q5) The accuracy/latency curve may be worth writing up publicly** — a blog post rather than a +docs change. + +### Still open + +**O1 — Explicit field, or a key-value entry in the image metadata?** +*Question:* does `mask_size` become a sanctioned field on the prediction/response types, or ride as +a KV pair in the existing image metadata? +*Trade-off:* an explicit field is self-documenting, type-checked and greppable, but is a public API +addition to a separately-versioned package with a consumer sweep attached. A metadata KV is cheaper +to add and easier to ignore, at the cost of being unvalidated and easy to drop silently in a +transform. +*Recommendation:* explicit field, matching the DeepStream convention cited in §2, because ~15 sites +currently read `image_size` as the canvas and an unvalidated KV will be missed by some of them. +*Input needed:* which. **Blocking for implementation.** + +**O2 — Remote execution across mixed backend versions.** *(raised in review; not in the reviewed draft)* +*Question:* a client may talk to either a new server that returns `mask_size` or an old one that does +not. What does the client do? +*Investigation:* the response shape itself is the version signal — presence of the new field +distinguishes a new server from an old one, with no version negotiation needed. +*Recommendation:* absent field ⇒ assume the old contract and default `mask_size` to the input size; +present ⇒ use the explicit value. This must hold for both remote Workflows execution and the SDK. +*Input needed:* confirmation that response-shape sniffing is acceptable rather than an explicit +version field. **Blocking for the remote path.** + +**O3 — Evidence for the deferred Q3 decision.** The team asked how badly RLE decoding actually +breaks under a reduced `size`. That is answerable now and cheaply: decode a reduced-`size` RLE with +`pycocotools` and with `supervision` both before and after +`detection/utils/internal.py:125-132`, and record the failure mode — exception, silent truncation, +or correct resize. *Action:* produce that evidence during implementation so the end-of-project +decision is made on data. Non-blocking. + +**O4 — The GPU path: gate the fused kernel, or teach it `t`?** +*Investigation:* `models/rfdetr/triton_postprocess.py` already implements a **lossless** version of +this optimisation — *"asks Triton to interpolate only the active mask region and emit sparse RLE run +records directly"* — and already parameterises resize tables by `output_size` (`:273-285`). Forcing +its eager fallback leaves the production GPU path unimproved and emits a `RuntimeWarning` **per +image** (`:253-258`), i.e. per-frame log spam. The fused path already bails on +`padding_unsupported`, `static_crop_unsupported`, `resize_metadata_unsupported` and `>4096 px` +(`:921-983`), so letterboxed 4K, static crops and Jetson-without-triton already fall through to the +slow path — which is the real justification for this feature and belongs in §1. +*Recommendation:* pass the reduced `output_height/width` into the kernel. If too large for v1, add +`t != 1.0` to `_unsupported_triton_postprocess_reason` so the fallback is quiet, and state plainly +that the GPU fast path is unimproved in v1. +*Input needed:* scope call. Non-blocking for the dense path; blocking for any Jetson claim. + +**O5 — Legacy-path parity.** After this change `mask_decode_mode="tradeoff"` returns transposed +masks under `USE_INFERENCE_MODELS=False` — `process_mask_tradeoff` passes `(h, w)` where +`cv2.resize` wants `(width, height)` (`core/utils/postprocess.py:353`; verified by execution, +1920×1080 at `t=1.0` returns `(n, 1920, 1080)`) — and correct masks under the default. A +flag-dependent geometry divergence. *Recommendation:* fix the transpose in the same PR; it is four +lines. Non-blocking. + +**O6 — Stream behaviour.** `tradeoff_factor` is bindable to `$inputs.*`, so `t` can change between +frames of one session, while `stabilize_detections/v1_tensor.py:447-451` and tracker caches derive +empty-mask shapes from `template.mask.shape[1:]`. *Recommendation:* treat `t` as session-fixed and +reject mid-stream changes. Non-blocking. + +### Follow-ups, not decisions + +Recommend a default `t` once the `AP_mask` curve exists; the chunk byte budget (dense path only — +the RLE generator has no chunking); batching the per-instance RLE encode (measured 1.4× at `t=1.0`, +2.5× at `t=0.25`, lossless, no API change); fixing the `Defaults to 0.5` docstring at +`instance_segmentation_base.py:78` against `DEFAULT_TRADEOFF_FACTOR = 0.0` on line 33; adding +`tradeoff` to the legacy query-param description at `http_api.py:4738`. + +**Closed by investigation.** `fast` needs no separate code path: `t=0.0` *is* legacy +`process_mask_fast` semantics, because the crop already happens at proto resolution before +`align_instance_segmentation_results` (`models/yolov8/common.py:32`) and `t=0` skips the resize. diff --git a/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py b/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py index 5b3d35672a..aa2a9d146d 100644 --- a/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py +++ b/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py @@ -20,13 +20,13 @@ ConfidenceFilter, align_instance_segmentation_results, crop_masks_to_boxes, - scale_polygons_to_image, post_process_nms_fused_model_output, rescale_image_detections, rescale_key_points_detections, run_nms_for_instance_segmentation, run_nms_for_key_points_detection, run_nms_for_object_detection, + scale_polygons_to_image, ) from inference_models.weights_providers.entities import RecommendedParameters From e693d3cf34f62ce733b29574fcd5ceabc6913e2c Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Thu, 1 Oct 2026 21:12:05 +0200 Subject: [PATCH 05/41] Untrack plan documents Committed in error. This branch carries Python changes only; the files remain on disk, untracked. --- docs/feat/instance-seg-tradeoff-factor-tdd.md | 214 -------------- docs/feat/instance-seg-tradeoff-factor-v2.md | 275 ------------------ 2 files changed, 489 deletions(-) delete mode 100644 docs/feat/instance-seg-tradeoff-factor-tdd.md delete mode 100644 docs/feat/instance-seg-tradeoff-factor-v2.md diff --git a/docs/feat/instance-seg-tradeoff-factor-tdd.md b/docs/feat/instance-seg-tradeoff-factor-tdd.md deleted file mode 100644 index 4783c945ac..0000000000 --- a/docs/feat/instance-seg-tradeoff-factor-tdd.md +++ /dev/null @@ -1,214 +0,0 @@ -# TDD sequence: `masks_resolution_factor` - -Companion to `instance-seg-tradeoff-factor-v2.md`. Each phase is red → green → commit. - -## Existing coverage, measured not assumed - -`inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py` (28 tests) **does** -cover `align_instance_segmentation_results`, via `test_chunked_resize_matches_monolithic` (:502) and -`..._with_static_crop_canvas`. But both are **invariance** tests — they assert chunked output equals -monolithic output. Change the resize target and both sides move together, so **both still pass**. - -Nothing pins output *geometry*: not the shape relative to the image, not coordinate correctness. -`crop_masks_to_boxes` is not imported and has no tests at all. - -So phase 0 is not optional ceremony — it is the only thing that would catch a geometry regression. - ---- - -## Phase 0 — characterization (GREEN on current code) - -Pin today's behaviour before touching anything. These must pass unchanged at `t=1.0` forever after. - -```python -class TestCurrentMaskGeometry: - """Characterization. These encode today's contract; they must not change at t=1.0.""" - - def test_masks_are_image_sized(self) -> None: - _, masks = self._run() - assert masks.shape[1:] == (self.size_after_pre_processing.height, - self.size_after_pre_processing.width) - - def test_masks_are_bool(self) -> None: - _, masks = self._run() - assert masks.dtype == torch.bool - - def test_golden_output_for_fixed_seed(self) -> None: - # seeded protos -> exact tensor. Any geometry change breaks this loudly. - torch.manual_seed(0) - _, masks = self._run() - assert masks.sum().item() == GOLDEN_SET_PIXELS # fill from first run - assert masks[0, :8, :8].tolist() == GOLDEN_CORNER # fill from first run - - def test_non_square_image_preserves_aspect(self) -> None: - # the axis-swap guard. Square fixtures hide transposes. - _, masks = self._run(image=(1080, 1920)) - assert masks.shape[1] == 1080 and masks.shape[2] == 1920 -``` - -And the first tests `crop_masks_to_boxes` has ever had: - -```python -class TestCropMasksToBoxes: - def test_zeroes_outside_box(self) -> None: ... - def test_scaling_default_is_proto_stride(self) -> None: ... - def test_non_square_inference_size(self) -> None: - # documents that a single scalar `scaling` is wrong when mask_w/inference_w != 0.25 - ... -``` - -**Commit 0.** No production change. - -## Phase 1 — RED: the coordinate-scaling gap - -The prerequisite from §2 of the plan. This test fails today and proves the gap exists. - -```python -def test_polygon_coords_are_image_space_when_mask_is_smaller(self) -> None: - # given: a mask deliberately smaller than the declared image - masks = make_masks(n=2, h=540, w=960) # half of 1080x1920 - meta = make_metadata(original_size=(1080, 1920)) - - # when - response = build_instance_segmentation_response(masks, meta) - - # then: coordinates must span the IMAGE, not the mask - xs = [p.x for pred in response.predictions for p in pred.points] - assert max(xs) > 960, "polygon coords left in mask space" -``` - -RED today — `inference_models_adapters.py:917` emits raw `masks2poly` output. Green once scaling is -applied at the mask→polygon boundary, mirroring legacy `post_process_polygons` -(`inference/core/utils/postprocess.py:449`). - -**Commit 1.** Coordinate scaling only. No new parameter yet — and it is a latent-correctness fix on -its own merits. - -## Phase 2 — RED: the parameter - -```python -@pytest.mark.parametrize("t", [0.0, 0.1, 0.25, 0.5, 1.0]) -def test_resolution_factor_interpolates_shape(self, t: float) -> None: - _, masks = self._run(masks_resolution_factor=t) # TypeError today -> RED - mh, mw = PROTO_UNPADDED - assert masks.shape[1] == max(1, round(mh * (1 - t) + 1080 * t)) - assert masks.shape[2] == max(1, round(mw * (1 - t) + 1920 * t)) - -def test_t1_is_bit_identical_to_baseline(self) -> None: - _, ref = self._run() # no kwarg - _, out = self._run(masks_resolution_factor=1.0) - assert torch.equal(out, ref) - -def test_t0_is_proto_resolution(self) -> None: - _, masks = self._run(masks_resolution_factor=0.0) - assert masks.shape[1:] == PROTO_UNPADDED - -@pytest.mark.parametrize("t", [0.1, 0.25, 0.5]) -def test_coords_round_trip_against_t1(self, t: float) -> None: - ref = polygons_from(self._run()) - out = polygons_from(self._run(masks_resolution_factor=t)) - # unpad rounding error is amplified as t falls, so tolerance scales with 1/t - assert_polygons_close(out, ref, tol_px=2.0 / t) - -def test_zero_instances(self) -> None: - _, masks = self._run(n=0, masks_resolution_factor=0.25) - assert masks.shape[0] == 0 - -@pytest.mark.parametrize("mode", ["letterbox", "stretch"]) -def test_both_resize_modes(self, mode: str) -> None: ... - -def test_static_crop_offset_is_scaled(self) -> None: ... -``` - -The `t=1.0` bit-identity test is the safety net for every later phase. - -**Commit 2.** Dense path only. - -## Phase 3 — RED: `mask_size` metadata (plan O1) - -```python -def test_mask_size_defaults_to_image_size(self) -> None: - # back-compat: existing construction sites must not change - m = InstancesRLEMasks.from_coco_rle_masks(image_size=(1080, 1920), masks=[...]) - assert m.mask_size == (1080, 1920) - -def test_mask_size_reflects_reduced_resolution(self) -> None: - dets = self._run_full_pipeline(masks_resolution_factor=0.25) - assert dets.mask_size == (270, 480) - assert dets.image_size == (1080, 1920) # unchanged meaning -``` - -**Commit 3.** - -## Phase 4 — RED: adapter mapping - -```python -@pytest.mark.parametrize("mode,expected", [("accurate", 1.0), ("fast", 0.0)]) -def test_enum_maps_to_factor(self, mode: str, expected: float) -> None: ... - -def test_tradeoff_reads_the_factor(self) -> None: ... - -@pytest.mark.parametrize("bad", [-0.1, 1.1]) -def test_out_of_range_raises(self, bad: float) -> None: - with pytest.raises(InvalidMaskDecodeArgument): - ... - -def test_source_keys_do_not_leak_to_model_call(self) -> None: - # map_inference_kwargs must consume both, not forward them - mapped = adapter.map_inference_kwargs({"mask_decode_mode": "fast", "tradeoff_factor": 0.0}) - assert "mask_decode_mode" not in mapped and "tradeoff_factor" not in mapped - assert mapped["masks_resolution_factor"] == 0.0 -``` - -**Commit 4.** - -## Phase 5 — RED: version skew (plan O2) and the Triton gate - -```python -def test_absent_mask_size_defaults_to_input_size(self) -> None: - # old server: field missing -> client assumes today's contract - dets = parse_response(response_without_mask_size()) - assert dets.mask_size == dets.image_size - -def test_unhonoured_hint_does_not_raise(self, caplog) -> None: - # old server ignores the parameter: correct output, just slower - dets = run_remote(masks_resolution_factor=0.25, server=OldServerStub()) - assert dets.mask_size == dets.image_size - assert "not honoured" in caplog.text # logged once, never raised - - -def test_reduced_resolution_forces_eager_fallback(self) -> None: - # pure-python: no GPU, so it runs in x86 CI instead of being a dead GPU-only test - reason = _unsupported_triton_postprocess_reason(..., masks_resolution_factor=0.25) - assert reason is not None -``` - -**Commit 5.** - -## Phase 6 — RLE parity - -Blocked on the deferred wire-format decision. Until then the RLE path keeps `t=1.0` and phase 0's -characterization tests guard it. The evidence experiment (plan O3) belongs here: - -```python -@pytest.mark.parametrize("decoder", ["pycocotools", "supervision"]) -def test_reduced_size_rle_decode_behaviour(self, decoder: str) -> None: - # not an assertion of correctness - a recorded observation of the failure mode, - # to inform the deferred decision: raises? truncates silently? resizes cleanly? - ... -``` - -## Running - -```bash -cd inference_models && python -m pytest tests/unit_tests/models/common/roboflow/test_post_processing.py -v -python -m pytest tests/inference/models_predictions_tests -python -m pytest tests/workflows/unit_tests/ -``` - -## Order and why - -Phase 0 before everything, because the existing invariance tests would not catch a geometry -regression. Phase 1 before phase 2, because coordinate scaling is a correctness fix that stands -alone and must be in place before any reduced mask can exist. Phase 6 last, because it is the only -part gated on a decision the team deferred. diff --git a/docs/feat/instance-seg-tradeoff-factor-v2.md b/docs/feat/instance-seg-tradeoff-factor-v2.md deleted file mode 100644 index a595f11077..0000000000 --- a/docs/feat/instance-seg-tradeoff-factor-v2.md +++ /dev/null @@ -1,275 +0,0 @@ -# Implementation plan: honour `tradeoff_factor` for instance segmentation - -**Author:** Michał Taraszewski - -**Related issue / PR:** _none yet — to be linked from the `#discuss-inference-release` thread_ - -**Status:** draft. Supersedes `instance-seg-tradeoff-factor.md`, whose headline figures combined two -code paths that never execute together and whose recommended v1 is measurably slower than today. - -> **Reviewed by the team 2026-10-01.** The delivery mechanism is settled (new block versions); the -> RLE wire-format decision is deliberately deferred to the end of implementation. §3 records what was -> decided and what is still open. - ---- - -## 1. What problem are we solving? - -**Who.** Anyone running instance segmentation on large frames or dense scenes: real-time video, and -Jetson deployments in particular. - -**What is missing.** `mask_decode_mode` and `tradeoff_factor` are accepted end to end — SDK entity → -Workflows block → `WorkflowsModelsProvider.run_instance_segmentation` → -`InstanceSegmentationInferenceRequest` — and then ignored. They pass untouched through -`InferenceModelsInstanceSegmentationAdapter.map_inference_kwargs` -(`inference/core/models/inference_models_adapters.py:520`) and are swallowed by `**kwargs` in each -model's `post_process`. No error, no effect. Masks are always produced at full input resolution. -The legacy path (`inference/core/models/instance_segmentation_base.py:143-190`) still implements all -three modes. - -**Why it matters: this stage is almost the entire frame.** Measured on one host (M1 Max, 8 torch -threads, fp32, 3840×2160, p50): JPEG decode 59.7 ms, preprocess to 640×640 0.26 ms, JPEG encode -35.1 ms — everything that is *not* mask post-processing sums to roughly **95 ms**. Mask -post-processing at `n=300` is **hundreds of milliseconds to seconds** (tables below). At 4K with a -dense scene this stage is **~90–97% of total frame time**. That, not a stage-relative speedup, is -the argument. - -**Two things this plan does not claim.** It is not sufficient on its own — at `n=300`/4K even -`t=0.25` leaves several hundred ms in this stage. And cost is Θ(n·H·W), so **reducing -`max_detections`** (default 300, `inference_models/configuration.py:229`) is a cheaper, lossless -first move that should be evaluated alongside it. - -### The published contract - -The `inference.roboflow.com` reference site was collapsed to a landing page in `cbdd286c4`, so the -mkdocstrings API pages no longer exist. What remains published: - -- `mask_decode_mode` default `accurate`; `tradeoff_factor` default `0`, scale `0='fast'` → `1='accurate'`. -- *"If 'accurate' the mask will be decoded using the original image size. If 'fast' the mask will be - decoded using the original mask size."* — the only published statement about output resolution, - and therefore binding. -- `response_mask_format` default `polygon`, `rle` opt-in. - -The request schema (`inference/core/entities/requests/inference.py:254-257`) publishes all three -modes. Only the legacy query parameter (`inference/core/interfaces/http/http_api.py:4738`) -enumerates just two, omitting `tradeoff` — worth correcting, but it is a doc fix, not a blocker. - -## 2. How will the behaviour change? - -One float, `masks_resolution_factor ∈ [0,1]` (named to match the existing `masks_smoothing_enabled` -/ `masks_binarization_threshold` kwargs), resolved in the adapter from the existing enum, and -**carried as explicit metadata** rather than inferred from array shape. - -Prior art for exactly this: DeepStream's `NvOSD_MaskParams` carries mask-local `height`/`width` -separately from `rect_params` and rescales on demand -(`_reference/deepstream_python_apps/bindings/docstrings/nvosddoc.h:278-291`; -`apps/deepstream-segmask/deepstream_segmask.py:120-126`). - -### Before — 4K frame, 300 instances, 160×160 protos - -```mermaid -flowchart LR - A["protos: 300 x 160 x 160"] --> B["crop to boxes, at proto res"] - B --> C["resize every instance to 2160 x 3840"] - C --> D["bool 300 x 2160 x 3840"] - D --> E["masks2poly or RLE encode"] - E --> F["polygon coords in 2160x3840
RLE size 2160x3840"] -``` - -### After, at `t = 0.25` - -```mermaid -flowchart LR - A["protos: 300 x 160 x 160"] --> B["crop to boxes, at proto res"] - B --> C["resize to 660 x 1080"] - C --> D["bool 300 x 660 x 1080
+ mask canvas 660x1080"] - D --> E["masks2poly or RLE encode
at 660 x 1080"] - E --> F["scale coords x3.27
-> image space 2160x3840"] -``` - -Adapter mapping: `accurate → 1.0`, `fast → 0.0`, `tradeoff → tradeoff_factor` (validated, raising -`InvalidMaskDecodeArgument` as the legacy path does). This satisfies the published contract exactly. -Default `1.0` leaves output unchanged. - -### Step F is the change. Without it every prediction at `t<1` is wrong. - -There is **no mask→image coordinate scaling anywhere** in the `inference_models` response path. The -adapter emits raw `masks2poly` output into `Point(x, y)` -(`inference_models_adapters.py:917-988`, `:1047`) while declaring the image at `original_size`. Two -sites are worse: `bitpacked_masks2poly(..., width=W)` with `W` from `original_size` (`:907`, -`core/utils/postprocess.py:73`) unpacks at the wrong stride and corrupts silently; and -`InstancesRLEMasks` declares `image_size = original_size` while encoding at -`size_after_pre_processing` (`rfdetr/common.py:383`). - -The legacy path does this correctly — it tracks `output_mask_shape` per mode and rescales via -`post_process_polygons` (`inference/core/utils/postprocess.py:449`). **The `inference_models` -rewrite dropped that step.** We are restoring a mechanism the codebase already had. - -Verified consequences if reduced masks escape unscaled (4 confirmed; a wider sweep is in progress -and will be attached rather than asserted): `visualizations/mask/v1_tensor.py:277-280` raises -outright; `bounding_rect/v1.py:163-177` overwrites `xyxy` from mask contours, so corrupted boxes -reach blocks that never touch masks; `dynamic_zones/v1.py:348-352` emits shrunken zone polygons; -`sinks/dataset_upload/v1_tensor.py:946-951` persists mask-space polygons as permanent labels. -Note `sv.Detections.from_inference` already resizes mismatched RLE -(`supervision/detection/utils/internal.py:125-132`) but not polygons (`:155`). - -### Measurements, by path - -Two different functions. They are not comparable and must not be combined. - -**Dense path** — `align_instance_segmentation_results` (`post_processing.py:395`), chunked, -materialises `n×H×W`. 4K, n=300, chunk 16, fp32, CPU: - -| t | resize p50 | peak RSS | -|---|---|---| -| 1.0 | 485–575 ms (3 runs) | 3256 MB | -| 0.25 | 52–65 ms | 595 MB | - -**RLE path** — `align_instance_segmentation_results_to_rle_masks` (`post_processing.py:525`), a -**per-instance generator**: no chunking, no `n×H×W` tensor, so the memory figures above do not -apply. This is the default (`v4.py:359`, `:408` hardcode `"rle"`; `GCP_SERVERLESS` forces it, -`inference_models_adapters.py:527`). Encode cost is strongly content-dependent — measured 2933 ms -for smooth blob masks vs 6262 ms for noise masks at n=300, a 2.1× spread from fixture alone. - -**Open measurement gaps, to close before implementation.** The polygon-path figure in the superseded -plan did not reproduce (`masks2poly` measured ~1877 ms against ~314 ms implied) and is withdrawn. -All numbers above are single-platform CPU; there is **no CUDA or Jetson measurement**, and the -~40 GB/frame traffic figure for Orin NX is a bandwidth model, not data. The benchmark scripts will -be linked, with fixture content stated. - -### How we verify - -`align_instance_segmentation_results` and `crop_masks_to_boxes` currently have **no unit tests** -(`inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py` covers neither), -so a characterization test — fixed-seed protos → golden mask tensor and golden polygons — is -commit zero, before any behaviour claim can be checked in CI. - -- Bit-identical output at `t=1.0`. -- **Polygon/RLE coordinate round-trip at `t<1`**: emitted coordinates land on the same image pixels - as `t=1.0`. Tolerance must scale with `1/t`, since unpad rounding error is amplified as `t` falls. - Shape-only assertions would not catch the defect above; this is the test that matters. -- RLE invariant: decoded counts sum to the declared size; declared size is the encoded size. -- Non-square fixtures throughout (square images hide axis errors); odd dims; letterbox *and* - stretch; static crop with offset; `n = 0`. -- Triton gate as a pure-Python unit test on `_unsupported_triton_postprocess_reason`, so it is not a - GPU-only dead test in x86 CI. -- `AP_mask` 50:95 overall **and by object size**, against `t`, with `AP_box` as control. - -Layers: `inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py`; -`tests/inference/models_predictions_tests`; `tests/workflows/unit_tests/.../instance_segmentation/` -for any new block version **and its `_tensor` sibling**. - -### Implementation surface - -Not one function. The alignment logic is duplicated across two functions, reached from six model -families on both the dense and RLE routes — `models/{yolov5,yolov7,yolov8,yolo26,yolact}/common.py` -× {dense, rle} = 10 call sites, plus `models/rfdetr/common.py:369` and -`models/rfdetr/triton_postprocess.py` — plus the new kwarg threaded through every -`*_instance_segmentation_{onnx,trt,torch_script}.py`. Roughly 12 sites beyond the two core -functions. - -### Out of scope, as its own plan - -Four independent performance and correctness changes, measured and bit-identical except the last, -belong in a separate PR and are deliberately not bundled here: `torch.gt(out=)` in place of -`.gt_()` + assignment; separable `crop_masks_to_boxes`; resizing directly into the static-crop -canvas; and deriving `crop_masks_to_boxes(scaling=0.25)` per-axis (wrong whenever -`mask_w/inference_w ≠ 0.25`, e.g. YOLACT 138/550 — and the only one of the four that changes -`t=1.0` output). Also out of scope: batching the per-instance RLE encode, measured 1.4× at `t=1.0` -and 2.5× at `t=0.25`, lossless and with no API change. - -## 3. Decisions taken, and what remains - -### Settled by team review, 2026-10-01 - -Question numbers below are those of the reviewed draft, so the thread stays traceable. - -**(Q1) New block versions are the delivery mechanism.** Changes of this kind have always shipped as -new block versions to minimise breakage — *even when the change is a fix to an earlier bug* — and -code duplication across versions is explicitly acceptable. This removes the main compatibility -objection: a new version means no existing pipeline changes under upgrade. Both the `vN` and -`vN_tensor` siblings are in scope. - -**(Q2) `mask_size` as metadata carried with the prediction: accepted in principle.** Two -sub-decisions remain, below. - -**(Q3) The RLE wire-format decision is deferred to the end of implementation**, for wider team -discussion. The reasoning, which changes this plan's shape: *with new block versions this is not -breaking for Workflows at all.* The breaking surface is narrower than the reviewed draft assumed — -it is the **model endpoints**, where a direct integrator would suddenly see mask-scaling parameters -respected. That is a bug fix, and it is breaking, and those are not in conflict. Implementation -should proceed without settling it. - -**(Q4) The chunk byte budget is a follow-up, not part of this work.** It addresses an extremum of -the resolution range; it can land later purely as serving-latency optimisation. - -**(Q5) The accuracy/latency curve may be worth writing up publicly** — a blog post rather than a -docs change. - -### Still open - -**O1 — Explicit field, or a key-value entry in the image metadata?** -*Question:* does `mask_size` become a sanctioned field on the prediction/response types, or ride as -a KV pair in the existing image metadata? -*Trade-off:* an explicit field is self-documenting, type-checked and greppable, but is a public API -addition to a separately-versioned package with a consumer sweep attached. A metadata KV is cheaper -to add and easier to ignore, at the cost of being unvalidated and easy to drop silently in a -transform. -*Recommendation:* explicit field, matching the DeepStream convention cited in §2, because ~15 sites -currently read `image_size` as the canvas and an unvalidated KV will be missed by some of them. -*Input needed:* which. **Blocking for implementation.** - -**O2 — Remote execution across mixed backend versions.** *(raised in review; not in the reviewed draft)* -*Question:* a client may talk to either a new server that returns `mask_size` or an old one that does -not. What does the client do? -*Investigation:* the response shape itself is the version signal — presence of the new field -distinguishes a new server from an old one, with no version negotiation needed. -*Recommendation:* absent field ⇒ assume the old contract and default `mask_size` to the input size; -present ⇒ use the explicit value. This must hold for both remote Workflows execution and the SDK. -*Input needed:* confirmation that response-shape sniffing is acceptable rather than an explicit -version field. **Blocking for the remote path.** - -**O3 — Evidence for the deferred Q3 decision.** The team asked how badly RLE decoding actually -breaks under a reduced `size`. That is answerable now and cheaply: decode a reduced-`size` RLE with -`pycocotools` and with `supervision` both before and after -`detection/utils/internal.py:125-132`, and record the failure mode — exception, silent truncation, -or correct resize. *Action:* produce that evidence during implementation so the end-of-project -decision is made on data. Non-blocking. - -**O4 — The GPU path: gate the fused kernel, or teach it `t`?** -*Investigation:* `models/rfdetr/triton_postprocess.py` already implements a **lossless** version of -this optimisation — *"asks Triton to interpolate only the active mask region and emit sparse RLE run -records directly"* — and already parameterises resize tables by `output_size` (`:273-285`). Forcing -its eager fallback leaves the production GPU path unimproved and emits a `RuntimeWarning` **per -image** (`:253-258`), i.e. per-frame log spam. The fused path already bails on -`padding_unsupported`, `static_crop_unsupported`, `resize_metadata_unsupported` and `>4096 px` -(`:921-983`), so letterboxed 4K, static crops and Jetson-without-triton already fall through to the -slow path — which is the real justification for this feature and belongs in §1. -*Recommendation:* pass the reduced `output_height/width` into the kernel. If too large for v1, add -`t != 1.0` to `_unsupported_triton_postprocess_reason` so the fallback is quiet, and state plainly -that the GPU fast path is unimproved in v1. -*Input needed:* scope call. Non-blocking for the dense path; blocking for any Jetson claim. - -**O5 — Legacy-path parity.** After this change `mask_decode_mode="tradeoff"` returns transposed -masks under `USE_INFERENCE_MODELS=False` — `process_mask_tradeoff` passes `(h, w)` where -`cv2.resize` wants `(width, height)` (`core/utils/postprocess.py:353`; verified by execution, -1920×1080 at `t=1.0` returns `(n, 1920, 1080)`) — and correct masks under the default. A -flag-dependent geometry divergence. *Recommendation:* fix the transpose in the same PR; it is four -lines. Non-blocking. - -**O6 — Stream behaviour.** `tradeoff_factor` is bindable to `$inputs.*`, so `t` can change between -frames of one session, while `stabilize_detections/v1_tensor.py:447-451` and tracker caches derive -empty-mask shapes from `template.mask.shape[1:]`. *Recommendation:* treat `t` as session-fixed and -reject mid-stream changes. Non-blocking. - -### Follow-ups, not decisions - -Recommend a default `t` once the `AP_mask` curve exists; the chunk byte budget (dense path only — -the RLE generator has no chunking); batching the per-instance RLE encode (measured 1.4× at `t=1.0`, -2.5× at `t=0.25`, lossless, no API change); fixing the `Defaults to 0.5` docstring at -`instance_segmentation_base.py:78` against `DEFAULT_TRADEOFF_FACTOR = 0.0` on line 33; adding -`tradeoff` to the legacy query-param description at `http_api.py:4738`. - -**Closed by investigation.** `fast` needs no separate code path: `t=0.0` *is* legacy -`process_mask_fast` semantics, because the crop already happens at proto resolution before -`align_instance_segmentation_results` (`models/yolov8/common.py:32`) and `t=0` skips the resize. From 287e09927bbf3d73192b28f984a6bbaaf83b4191 Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Fri, 2 Oct 2026 07:01:04 +0200 Subject: [PATCH 06/41] Add masks_resolution_factor to instance segmentation mask alignment Interpolate the mask resize target between the post-unpad mask grid and size_after_pre_processing, instead of always resizing to the latter. A factor of 1.0 is the default and short-circuits to the image dimensions, so round() never runs on the default path and output is bit-identical. The empty-detection branch and the static-crop canvas both follow the factor. The canvas keeps the mask grid's ratio to the image rather than planting a reduced mask on a full-size canvas, which would have been correctly placed but saved no memory; crop offsets move into the same space. Consumers that derive a canvas from mask.shape therefore get a consistent answer with and without detections. No caller passes the parameter yet. Tests: python3 -m pytest tests/unit_tests/models/common/roboflow/test_post_processing.py 58 passed (45 pre-existing, 13 added). --- .../models/common/roboflow/post_processing.py | 77 +++++++++++-- .../common/roboflow/test_post_processing.py | 109 ++++++++++++++++++ 2 files changed, 179 insertions(+), 7 deletions(-) diff --git a/inference_models/inference_models/models/common/roboflow/post_processing.py b/inference_models/inference_models/models/common/roboflow/post_processing.py index 0bb7b0e4dc..6638a85ec5 100644 --- a/inference_models/inference_models/models/common/roboflow/post_processing.py +++ b/inference_models/inference_models/models/common/roboflow/post_processing.py @@ -425,6 +425,39 @@ def scale_polygons_to_image( return [polygon * scale for polygon in polygons] +def resolve_mask_target_size( + mask_height: int, + mask_width: int, + size_after_pre_processing: ImageDimensions, + masks_resolution_factor: float, +) -> Tuple[int, int]: + """Interpolate the mask resize target between the mask grid and the image. + + A factor of `1.0` selects the image size, reproducing the behaviour from + before this parameter existed. A factor of `0.0` leaves masks on the grid + the model produced them on. Values in between trade mask fidelity for the + cost of upsampling every instance. + + Args: + mask_height: Height of the mask grid after letterbox padding is removed. + mask_width: Width of that grid. + size_after_pre_processing: Dimensions masks are reported against at a + factor of `1.0`. + masks_resolution_factor: Interpolation factor in `[0.0, 1.0]`. + + Returns: + The `(height, width)` to resize masks to. + """ + if masks_resolution_factor >= 1.0: + return size_after_pre_processing.height, size_after_pre_processing.width + factor = max(0.0, masks_resolution_factor) + height = round( + mask_height * (1 - factor) + size_after_pre_processing.height * factor + ) + width = round(mask_width * (1 - factor) + size_after_pre_processing.width * factor) + return max(1, height), max(1, width) + + def align_instance_segmentation_results( image_bboxes: torch.Tensor, masks: torch.Tensor, @@ -437,10 +470,29 @@ def align_instance_segmentation_results( static_crop_offset: StaticCropOffset, binarization_threshold: float = 0.0, mask_chunk_size: int = INFERENCE_MODELS_INSTANCE_SEG_MASK_PROCESSING_CHUNK_SIZE, + masks_resolution_factor: float = 1.0, ) -> Tuple[torch.Tensor, torch.Tensor]: if image_bboxes.shape[0] == 0: + empty_target_height, empty_target_width = resolve_mask_target_size( + mask_height=masks.shape[1], + mask_width=masks.shape[2], + size_after_pre_processing=size_after_pre_processing, + masks_resolution_factor=masks_resolution_factor, + ) empty_masks = torch.empty( - size=(0, original_size.height, original_size.width), + size=( + 0, + round( + original_size.height + * empty_target_height + / size_after_pre_processing.height + ), + round( + original_size.width + * empty_target_width + / size_after_pre_processing.width + ), + ), dtype=torch.bool, device=image_bboxes.device, ) @@ -500,8 +552,12 @@ def align_instance_segmentation_results( # chunk x H x W instead of n x H x W (17+ GiB at 300 masks on a 12MP # image); only the n x H x W bool output is ever materialized whole. mask_chunk_size = max(1, int(mask_chunk_size)) - target_height = size_after_pre_processing.height - target_width = size_after_pre_processing.width + target_height, target_width = resolve_mask_target_size( + mask_height=masks.shape[1], + mask_width=masks.shape[2], + size_after_pre_processing=size_after_pre_processing, + masks_resolution_factor=masks_resolution_factor, + ) binarized_masks = torch.empty( (masks.shape[0], target_height, target_width), dtype=torch.bool, @@ -515,19 +571,26 @@ def align_instance_segmentation_results( ).gt_(binarization_threshold) masks = binarized_masks if static_crop_offset.offset_x > 0 or static_crop_offset.offset_y > 0: + # the canvas keeps the mask grid's ratio to the image, so a reduced + # factor shrinks the whole output rather than planting a small mask on + # a full-size canvas; crop offsets move into the same space + canvas_h_scale = target_height / size_after_pre_processing.height + canvas_w_scale = target_width / size_after_pre_processing.width + canvas_offset_y = round(static_crop_offset.offset_y * canvas_h_scale) + canvas_offset_x = round(static_crop_offset.offset_x * canvas_w_scale) mask_canvas = torch.zeros( ( masks.shape[0], - original_size.height, - original_size.width, + max(1, round(original_size.height * canvas_h_scale)), + max(1, round(original_size.width * canvas_w_scale)), ), dtype=torch.bool, device=masks.device, ) mask_canvas[ :, - static_crop_offset.offset_y : static_crop_offset.offset_y + masks.shape[1], - static_crop_offset.offset_x : static_crop_offset.offset_x + masks.shape[2], + canvas_offset_y : canvas_offset_y + masks.shape[1], + canvas_offset_x : canvas_offset_x + masks.shape[2], ] = masks static_crop_offsets = torch.as_tensor( [ diff --git a/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py b/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py index aa2a9d146d..af31883a06 100644 --- a/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py +++ b/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py @@ -736,3 +736,112 @@ def test_empty_polygon_list(self) -> None: # then assert result == [] + + +class TestMasksResolutionFactor: + """The resolution lever: interpolate the resize target between the + post-unpad mask grid and size_after_pre_processing.""" + + PROTO = 160 + ORIG_H, ORIG_W = 200, 300 + + @classmethod + def _expected(cls, t: float): + h = max(1, round(cls.PROTO * (1 - t) + cls.ORIG_H * t)) + w = max(1, round(cls.PROTO * (1 - t) + cls.ORIG_W * t)) + return h, w + + @classmethod + def _run(cls, n: int = 7, static_crop: bool = False, **kwargs): + torch.manual_seed(42) + bboxes = torch.rand((n, 6), dtype=torch.float32) * 100 + masks = torch.randn((n, cls.PROTO, cls.PROTO), dtype=torch.float32) + static_crop_offset = StaticCropOffset( + offset_x=13 if static_crop else 0, + offset_y=7 if static_crop else 0, + crop_width=cls.ORIG_W, + crop_height=cls.ORIG_H, + ) + return align_instance_segmentation_results( + image_bboxes=bboxes.clone(), + masks=masks.clone(), + padding=(0, 0, 0, 0), + scale_width=1.0, + scale_height=1.0, + original_size=ImageDimensions( + height=cls.ORIG_H + (7 if static_crop else 0), + width=cls.ORIG_W + (13 if static_crop else 0), + ), + size_after_pre_processing=ImageDimensions( + height=cls.ORIG_H, width=cls.ORIG_W + ), + inference_size=ImageDimensions(height=640, width=640), + static_crop_offset=static_crop_offset, + binarization_threshold=0.0, + mask_chunk_size=1000, + **kwargs, + ) + + def test_factor_one_is_bit_identical_to_default(self) -> None: + # given / when + _, baseline = self._run() + _, out = self._run(masks_resolution_factor=1.0) + + # then + assert torch.equal(out, baseline) + + def test_factor_zero_keeps_the_post_unpad_mask_grid(self) -> None: + # given / when + _, masks = self._run(masks_resolution_factor=0.0) + + # then + assert masks.shape[1:] == (self.PROTO, self.PROTO) + + @pytest.mark.parametrize("t", [0.0, 0.1, 0.25, 0.5, 0.75, 1.0]) + def test_factor_interpolates_the_resize_target(self, t: float) -> None: + # given / when + _, masks = self._run(masks_resolution_factor=t) + + # then + assert masks.shape[1:] == self._expected(t) + + @pytest.mark.parametrize("t", [0.25, 0.5]) + def test_axis_order_survives_a_reduced_target(self, t: float) -> None: + # given / when + # the image is 200x300, so a transposed resize would be visible here + _, masks = self._run(masks_resolution_factor=t) + + # then + expected_h, expected_w = self._expected(t) + assert masks.shape[1] == expected_h + assert masks.shape[2] == expected_w + assert expected_h < expected_w + + def test_output_stays_bool_at_reduced_resolution(self) -> None: + # given / when + _, masks = self._run(masks_resolution_factor=0.25) + + # then + assert masks.dtype == torch.bool + + def test_no_instances_shape_follows_the_factor(self) -> None: + # given / when + # consumers derive the canvas from mask.shape, so the empty case must + # agree with the populated one + _, masks = self._run(n=0, masks_resolution_factor=0.5) + + # then + assert masks.shape[0] == 0 + assert masks.shape[1:] == self._expected(0.5) + + def test_static_crop_canvas_scales_with_the_factor(self) -> None: + # given / when + _, masks = self._run(static_crop=True, masks_resolution_factor=0.5) + + # then + # the canvas keeps its ratio to the mask grid, so the whole output is + # reduced rather than a small mask pasted onto a full-size canvas + target_h, target_w = self._expected(0.5) + expected_h = round((self.ORIG_H + 7) * target_h / self.ORIG_H) + expected_w = round((self.ORIG_W + 13) * target_w / self.ORIG_W) + assert masks.shape[1:] == (expected_h, expected_w) From 65edbc9594488036d3a31b3989df556846cda6c8 Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Fri, 2 Oct 2026 07:06:39 +0200 Subject: [PATCH 07/41] Carry the mask grid size explicitly on segmentation predictions mask.shape[1:] and InstancesRLEMasks.image_size are currently the de facto definition of the mask canvas at roughly fifteen call sites. Once the resize target becomes adjustable they stop agreeing with the image, so the grid has to travel with the prediction rather than be inferred. Add mask_size to InstancesRLEMasks and InstanceDetections, each defaulting to the value it effectively had before: image_size for the former, the carrier's own grid for the latter. Every existing construction stays valid and every reader that ignores the field gets today's behaviour. to_coco_rle_masks still writes image_size as the COCO size field. The wire format is a separate, deferred decision and is untouched here. Tests: python3 -m pytest tests/unit_tests/models/base tests/unit_tests/models/common/roboflow/test_post_processing.py 77 passed. --- .../models/base/instance_segmentation.py | 11 +++ .../inference_models/models/base/types.py | 14 ++- .../models/base/test_instance_segmentation.py | 97 +++++++++++++++++++ 3 files changed, 119 insertions(+), 3 deletions(-) diff --git a/inference_models/inference_models/models/base/instance_segmentation.py b/inference_models/inference_models/models/base/instance_segmentation.py index 172e036a82..30c883808a 100644 --- a/inference_models/inference_models/models/base/instance_segmentation.py +++ b/inference_models/inference_models/models/base/instance_segmentation.py @@ -144,6 +144,17 @@ class InstanceDetections: bboxes_metadata: Optional[List[dict]] = ( None # if given, list of size equal to # of bboxes ) + # (h, w) of the grid `mask` lives on. Resolved from the carrier when not + # given, which reproduces the behaviour from before it was adjustable. + mask_size: Optional[Tuple[int, int]] = None + + def __post_init__(self) -> None: + if self.mask_size is not None: + return + if isinstance(self.mask, InstancesRLEMasks): + self.mask_size = self.mask.mask_size + elif self.mask is not None and hasattr(self.mask, "shape"): + self.mask_size = (int(self.mask.shape[1]), int(self.mask.shape[2])) def __len__(self) -> int: return int(self.xyxy.shape[0]) diff --git a/inference_models/inference_models/models/base/types.py b/inference_models/inference_models/models/base/types.py index 64f569752d..99ba2ee10b 100644 --- a/inference_models/inference_models/models/base/types.py +++ b/inference_models/inference_models/models/base/types.py @@ -1,5 +1,5 @@ from dataclasses import dataclass -from typing import List, Tuple, TypeVar +from typing import List, Optional, Tuple, TypeVar PreprocessedInputs = TypeVar("PreprocessedInputs") PreprocessingMetadata = TypeVar("PreprocessingMetadata") @@ -8,17 +8,25 @@ @dataclass class InstancesRLEMasks: - image_size: Tuple[int, int] # (h, w) + image_size: Tuple[int, int] # (h, w) of the image the masks describe masks: List[bytes] + # (h, w) of the grid the masks are encoded on. Defaults to image_size, + # which is the only value it took before the grid became adjustable. + mask_size: Optional[Tuple[int, int]] = None + + def __post_init__(self) -> None: + if self.mask_size is None: + self.mask_size = self.image_size @classmethod def from_coco_rle_masks( cls, image_size: Tuple[int, int], masks: List[dict], + mask_size: Optional[Tuple[int, int]] = None, ) -> "InstancesRLEMasks": masks = [m["counts"] for m in masks] - return cls(image_size=image_size, masks=masks) + return cls(image_size=image_size, masks=masks, mask_size=mask_size) def to_coco_rle_masks(self) -> List[dict]: return [{"size": list(self.image_size), "counts": m} for m in self.masks] diff --git a/inference_models/tests/unit_tests/models/base/test_instance_segmentation.py b/inference_models/tests/unit_tests/models/base/test_instance_segmentation.py index 4cc9904700..594f81eeb7 100644 --- a/inference_models/tests/unit_tests/models/base/test_instance_segmentation.py +++ b/inference_models/tests/unit_tests/models/base/test_instance_segmentation.py @@ -43,3 +43,100 @@ def test_len_counts_boxes_regardless_of_mask_representation() -> None: # when / then assert len(detections) == 3 + + +class TestMaskSize: + """The grid masks actually live on, carried explicitly. + + `mask.shape[1:]` and `InstancesRLEMasks.image_size` are currently the de + facto definition of the mask canvas at roughly fifteen call sites. Once the + resize target becomes adjustable they stop agreeing with the image, so the + grid has to travel with the prediction rather than be inferred from it. + + Provisional: carrying this as an explicit field rather than a key in the + image metadata is a recommendation pending maintainer ratification. + """ + + def test_rle_mask_size_defaults_to_image_size(self) -> None: + # given / when + masks = InstancesRLEMasks(image_size=(1080, 1920), masks=[]) + + # then + assert masks.mask_size == (1080, 1920) + + def test_rle_mask_size_can_differ_from_image_size(self) -> None: + # given / when + masks = InstancesRLEMasks( + image_size=(1080, 1920), masks=[], mask_size=(270, 480) + ) + + # then + assert masks.mask_size == (270, 480) + assert masks.image_size == (1080, 1920) + + def test_rle_wire_format_is_unchanged(self) -> None: + # given + # the COCO `size` field is the deferred wire-format decision; adding + # mask_size must not alter what goes on the wire today + masks = InstancesRLEMasks( + image_size=(1080, 1920), masks=[b"abc"], mask_size=(270, 480) + ) + + # when + encoded = masks.to_coco_rle_masks() + + # then + assert encoded == [{"size": [1080, 1920], "counts": b"abc"}] + + def test_dense_mask_size_falls_back_to_the_tensor_grid(self) -> None: + # given + detections = InstanceDetections( + xyxy=torch.zeros((2, 4)), + class_id=torch.zeros((2,), dtype=torch.int64), + confidence=torch.zeros((2,)), + mask=torch.zeros((2, 540, 960), dtype=torch.bool), + ) + + # then + assert detections.mask_size == (540, 960) + + def test_dense_mask_size_honours_an_explicit_value(self) -> None: + # given + detections = InstanceDetections( + xyxy=torch.zeros((1, 4)), + class_id=torch.zeros((1,), dtype=torch.int64), + confidence=torch.zeros((1,)), + mask=torch.zeros((1, 270, 480), dtype=torch.bool), + mask_size=(270, 480), + ) + + # then + assert detections.mask_size == (270, 480) + + def test_rle_carrier_reports_its_own_grid(self) -> None: + # given + detections = InstanceDetections( + xyxy=torch.zeros((1, 4)), + class_id=torch.zeros((1,), dtype=torch.int64), + confidence=torch.zeros((1,)), + mask=InstancesRLEMasks( + image_size=(1080, 1920), masks=[b""], mask_size=(270, 480) + ), + ) + + # then + assert detections.mask_size == (270, 480) + + def test_empty_detection_stack_still_reports_the_grid(self) -> None: + # given + # consumers derive the scale factor from the grid whether or not + # anything was detected + detections = InstanceDetections( + xyxy=torch.zeros((0, 4)), + class_id=torch.zeros((0,), dtype=torch.int64), + confidence=torch.zeros((0,)), + mask=torch.zeros((0, 270, 480), dtype=torch.bool), + ) + + # then + assert detections.mask_size == (270, 480) From 474b2a047da64487e4e6b50e97f4e1cf8a8f45dd Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Fri, 2 Oct 2026 16:02:36 +0200 Subject: [PATCH 08/41] Apply repository Python style rules to mask post-processing helpers Follows .cursor/rules as AGENTS.md requires: keyword-only secondary arguments via `*`, a blank line after guard clauses, and named return values instead of expressions on the return line. No behaviour change; every call site already passed these arguments by keyword. --- .../models/common/roboflow/post_processing.py | 20 ++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/inference_models/inference_models/models/common/roboflow/post_processing.py b/inference_models/inference_models/models/common/roboflow/post_processing.py index 6638a85ec5..cf2e2d1bb1 100644 --- a/inference_models/inference_models/models/common/roboflow/post_processing.py +++ b/inference_models/inference_models/models/common/roboflow/post_processing.py @@ -395,6 +395,7 @@ def crop_masks_to_boxes( def scale_polygons_to_image( polygons: List[np.ndarray], + *, mask_size: ImageDimensions, image_size: ImageDimensions, ) -> List[np.ndarray]: @@ -417,17 +418,22 @@ def scale_polygons_to_image( """ if not polygons: return [] + x_scale = image_size.width / mask_size.width y_scale = image_size.height / mask_size.height if x_scale == 1.0 and y_scale == 1.0: return polygons + scale = np.array([x_scale, y_scale], dtype=np.float32) - return [polygon * scale for polygon in polygons] + scaled_polygons = [polygon * scale for polygon in polygons] + + return scaled_polygons def resolve_mask_target_size( mask_height: int, mask_width: int, + *, size_after_pre_processing: ImageDimensions, masks_resolution_factor: float, ) -> Tuple[int, int]: @@ -450,12 +456,16 @@ def resolve_mask_target_size( """ if masks_resolution_factor >= 1.0: return size_after_pre_processing.height, size_after_pre_processing.width + factor = max(0.0, masks_resolution_factor) - height = round( - mask_height * (1 - factor) + size_after_pre_processing.height * factor + height = max( + 1, round(mask_height * (1 - factor) + size_after_pre_processing.height * factor) + ) + width = max( + 1, round(mask_width * (1 - factor) + size_after_pre_processing.width * factor) ) - width = round(mask_width * (1 - factor) + size_after_pre_processing.width * factor) - return max(1, height), max(1, width) + + return height, width def align_instance_segmentation_results( From 40a5d1faeaca10648a320fd6271ada14ec03a944 Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Fri, 2 Oct 2026 16:02:45 +0200 Subject: [PATCH 09/41] Honour mask_decode_mode and tradeoff_factor on the inference_models path Both parameters were accepted end to end and then ignored: they passed untouched through map_inference_kwargs and were swallowed by **kwargs in each model's post_process. Collapse the published enum onto the single resolution factor that post-processing consumes. accurate maps to 1.0, fast to 0.0, tradeoff reads tradeoff_factor; an absent mode stays at 1.0, so default behaviour is unchanged. Both source keys are consumed, since the deeper pre/post stages do not accept arbitrary kwargs. An unknown mode now raises InvalidMaskDecodeArgument, matching the legacy path, which previously accepted anything here. Scale polygon coordinates from mask space into image space where every mask branch converges, using the mask grid now carried on the detection. Equal sizes short-circuit to a no-op, so today's output is untouched; without this a reduced-resolution mask would report coordinates shrunk toward the origin. The legacy path does this via post_process_polygons; the inference_models rewrite dropped it. Tests: python -m pytest tests/inference/unit_tests/core/models/test_inference_models_adapters.py 37 passed (27 pre-existing, 10 added). --- .../core/models/inference_models_adapters.py | 74 ++++++++++- .../models/test_inference_models_adapters.py | 125 +++++++++++++++++- 2 files changed, 197 insertions(+), 2 deletions(-) diff --git a/inference/core/models/inference_models_adapters.py b/inference/core/models/inference_models_adapters.py index 85814074f8..8a2e391edf 100644 --- a/inference/core/models/inference_models_adapters.py +++ b/inference/core/models/inference_models_adapters.py @@ -60,7 +60,11 @@ VALID_INFERENCE_MODELS_BACKENDS, WORKFLOWS_ASYNC_FUTURE_RESULT_TIMEOUT, ) -from inference.core.exceptions import PayloadTooLargeError, PostProcessingError +from inference.core.exceptions import ( + InvalidMaskDecodeArgument, + PayloadTooLargeError, + PostProcessingError, +) from inference.core.models.action_recognition import merge_window_segments from inference.core.models.base import Model from inference.core.models.semantic_segmentation_utils import ( @@ -103,6 +107,7 @@ MAX_RFDETR_PIPELINE_DEPTH, get_rfdetr_pipeline_depth, ) +from inference_models.entities import ImageDimensions from inference_models.models.base.action_recognition import ( ActionRecognitionModel, effective_max_frame_side, @@ -126,6 +131,9 @@ ) from inference_models.models.base.types import InstancesRLEMasks, PreprocessingMetadata from inference_models.models.common.rle_utils import torch_mask_to_coco_rle +from inference_models.models.common.roboflow.post_processing import ( + scale_polygons_to_image, +) DEFAULT_COLOR_PALETTE = [ "#A351FB", @@ -422,6 +430,44 @@ def draw_predictions( ) +_MASK_DECODE_MODE_TO_RESOLUTION_FACTOR = { + "accurate": 1.0, + "fast": 0.0, +} + + +def _resolve_masks_resolution_factor( + *, + mask_decode_mode: Optional[str], + tradeoff_factor: Optional[float], +) -> float: + """Collapse the request enum and factor onto one resolution factor. + + ``accurate`` keeps masks at image resolution and is the default, matching + the published contract. ``fast`` leaves them on the model's own grid. + ``tradeoff`` interpolates between the two by ``tradeoff_factor``. + """ + if mask_decode_mode is None: + return 1.0 + + if mask_decode_mode in _MASK_DECODE_MODE_TO_RESOLUTION_FACTOR: + return _MASK_DECODE_MODE_TO_RESOLUTION_FACTOR[mask_decode_mode] + + if mask_decode_mode != "tradeoff": + raise InvalidMaskDecodeArgument( + f"Invalid mask_decode_mode: {mask_decode_mode}. " + "Must be one of ['accurate', 'fast', 'tradeoff']" + ) + + resolution_factor = 0.0 if tradeoff_factor is None else float(tradeoff_factor) + if not 0.0 <= resolution_factor <= 1.0: + raise InvalidMaskDecodeArgument( + f"Invalid tradeoff_factor: {resolution_factor}. Must be in [0.0, 1.0]" + ) + + return resolution_factor + + class InferenceModelsInstanceSegmentationAdapter(Model): def __init__(self, model_id: str, api_key: str = None, **kwargs): super().__init__() @@ -519,6 +565,13 @@ def run_tensor_native_inference( def map_inference_kwargs(self, kwargs: dict) -> dict: kwargs["input_color_format"] = "bgr" + # The published request surface is an enum plus a float; post-processing + # consumes a single resolution factor. Translate here and consume both + # source keys, which the deeper stages do not accept. + kwargs["masks_resolution_factor"] = _resolve_masks_resolution_factor( + mask_decode_mode=kwargs.pop("mask_decode_mode", None), + tradeoff_factor=kwargs.pop("tradeoff_factor", None), + ) pre_processing_overrides = PreProcessingOverrides( disable_contrast_enhancement=kwargs.get("disable_preproc_contrast", False), disable_grayscale=kwargs.get("disable_preproc_grayscale", False), @@ -997,6 +1050,25 @@ def _build_responses_from_detections( # thread-local pinned scratch buffers. Only scalar values and # polygon/RLE lists may be stored on responses below; do not return # those arrays or any view derived from them. + # Contours are extracted in mask coordinates. When the mask grid + # is not the image grid they have to be lifted before they are + # reported; equal sizes short-circuit to a no-op. This also breaks + # any view into the pinned scratch buffers noted above, since the + # scaled polygons are freshly allocated. + mask_size = getattr(det, "mask_size", None) + if ( + not return_in_rle + and mask_size is not None + and tuple(mask_size) != (H, W) + ): + polys_or_rles = scale_polygons_to_image( + polys_or_rles, + mask_size=ImageDimensions( + height=int(mask_size[0]), width=int(mask_size[1]) + ), + image_size=ImageDimensions(height=H, width=W), + ) + predictions: List[ Union[InstanceSegmentationPrediction, InstanceSegmentationRLEPrediction] ] = [] diff --git a/tests/inference/unit_tests/core/models/test_inference_models_adapters.py b/tests/inference/unit_tests/core/models/test_inference_models_adapters.py index 411dbd677c..6e8da5ad31 100644 --- a/tests/inference/unit_tests/core/models/test_inference_models_adapters.py +++ b/tests/inference/unit_tests/core/models/test_inference_models_adapters.py @@ -12,7 +12,7 @@ InstanceSegmentationInferenceResponse, InstanceSegmentationInferenceResponseDC, ) -from inference.core.exceptions import PostProcessingError +from inference.core.exceptions import InvalidMaskDecodeArgument, PostProcessingError from inference.core.models.inference_models_adapters import ( InferenceModelsDepthEstimationAdapter, InferenceModelsInstanceSegmentationAdapter, @@ -741,3 +741,126 @@ def test_fixed_input_hw_from_backend_skips_missing_or_invalid_network_input(): _fixed_input_hw_from_backend(_backend_with_network_input(height=0, width=640)) is None ) + + +def _seg_adapter() -> InferenceModelsInstanceSegmentationAdapter: + adapter = object.__new__(InferenceModelsInstanceSegmentationAdapter) + adapter._pipeline_depth = 1 + adapter.class_names = ["car"] + return adapter + + +def test_polygon_points_are_scaled_from_mask_space_into_image_space() -> None: + # given + # a 4x4 mask describing a 16x16 image: contours come out in mask + # coordinates and must be lifted by 4x before they are reported + adapter = _seg_adapter() + metadata = [SimpleNamespace(original_size=SimpleNamespace(width=16, height=16))] + detections = [ + InstanceDetections( + xyxy=torch.tensor([[0, 0, 16, 16]], dtype=torch.int32), + confidence=torch.tensor([0.9], dtype=torch.float32), + class_id=torch.tensor([0], dtype=torch.int32), + mask=torch.ones((1, 4, 4), dtype=torch.uint8), + ) + ] + + # when + responses = adapter._build_responses_from_detections(detections, metadata) + + # then + xs = [p.x for p in responses[0].predictions[0].points] + ys = [p.y for p in responses[0].predictions[0].points] + assert max(xs) > 4, f"x still in mask space: {xs}" + assert max(ys) > 4, f"y still in mask space: {ys}" + + +def test_polygon_points_unchanged_when_mask_matches_the_image() -> None: + # given + # today's situation: the scaling must be an exact no-op + adapter = _seg_adapter() + metadata = [SimpleNamespace(original_size=SimpleNamespace(width=8, height=8))] + mask = torch.zeros((1, 8, 8), dtype=torch.uint8) + mask[0, 2:6, 2:6] = 1 + detections = [ + InstanceDetections( + xyxy=torch.tensor([[2, 2, 6, 6]], dtype=torch.int32), + confidence=torch.tensor([0.9], dtype=torch.float32), + class_id=torch.tensor([0], dtype=torch.int32), + mask=mask, + ) + ] + + # when + responses = adapter._build_responses_from_detections(detections, metadata) + + # then + xs = [p.x for p in responses[0].predictions[0].points] + ys = [p.y for p in responses[0].predictions[0].points] + assert min(xs) == 2 and max(xs) == 5 + assert min(ys) == 2 and max(ys) == 5 + + +class TestMaskDecodeModeMapping: + """The adapter collapses the published enum onto one float. + + `mask_decode_mode` and `tradeoff_factor` are the documented request + surface; `masks_resolution_factor` is what post-processing consumes. The + translation belongs here, and the source keys must not reach the model. + """ + + @staticmethod + def _map(**kwargs) -> dict: + adapter = object.__new__(InferenceModelsInstanceSegmentationAdapter) + adapter._model = SimpleNamespace(supported_mask_formats=[]) + return adapter.map_inference_kwargs(kwargs) + + def test_accurate_maps_to_full_resolution(self) -> None: + # given / when + mapped = self._map(mask_decode_mode="accurate") + + # then + assert mapped["masks_resolution_factor"] == 1.0 + + def test_fast_maps_to_mask_resolution(self) -> None: + # given / when + mapped = self._map(mask_decode_mode="fast") + + # then + assert mapped["masks_resolution_factor"] == 0.0 + + def test_tradeoff_reads_the_factor(self) -> None: + # given / when + mapped = self._map(mask_decode_mode="tradeoff", tradeoff_factor=0.25) + + # then + assert mapped["masks_resolution_factor"] == 0.25 + + def test_default_is_full_resolution_when_nothing_is_passed(self) -> None: + # given / when + # an unset mode must reproduce today's behaviour exactly + mapped = self._map() + + # then + assert mapped["masks_resolution_factor"] == 1.0 + + def test_source_keys_do_not_reach_the_model(self) -> None: + # given / when + # deeper pre/post stages do not accept arbitrary kwargs + mapped = self._map(mask_decode_mode="tradeoff", tradeoff_factor=0.5) + + # then + assert "mask_decode_mode" not in mapped + assert "tradeoff_factor" not in mapped + + @pytest.mark.parametrize("bad_factor", [-0.1, 1.1]) + def test_out_of_range_factor_is_rejected(self, bad_factor: float) -> None: + # given / when / then + with pytest.raises(InvalidMaskDecodeArgument): + self._map(mask_decode_mode="tradeoff", tradeoff_factor=bad_factor) + + def test_unknown_mode_is_rejected(self) -> None: + # given / when / then + # the legacy path raises for this; the two must agree + with pytest.raises(InvalidMaskDecodeArgument): + self._map(mask_decode_mode="nonsense") From 7379065f66a7d1852b1605d3916f11188c724ffd Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Fri, 2 Oct 2026 16:28:36 +0200 Subject: [PATCH 10/41] Honour masks_resolution_factor on the RLE path and across model families The dense path already took the factor but nothing passed it, so the parameter was reachable from a request and still inert. Thread it through all six instance-segmentation families - yolov5, yolov7, yolov8, yolo26, yolact and rfdetr - on both the dense and RLE routes, across the ONNX, TorchScript and TensorRT backends. Their post_process methods all accept **kwargs, so an un-threaded parameter is silently swallowed, which is how the original regression happened. Teach align_instance_segmentation_results_to_rle_masks the same interpolation the dense path uses, and scale its static-crop canvas and offsets to match, so mask_format no longer changes the resolution a caller receives. to_coco_rle_masks now declares the grid the counts were encoded on rather than the image size. Measured: pycocotools decodes an honest reduced size cleanly, decodes a mismatched one silently wrong, and sv.Detections.from_inference auto-resizes an honest one to the image. The previous behaviour - declaring image_size while encoding at size_after_pre_processing - is the mismatched case as soon as the two differ, so declaring the encoded grid is the correction, not the risk. RF-DETR's fused Triton post-process interpolates to the image inside the kernel and cannot honour a reduced target. It now reports mask_resolution_factor_unsupported and defers to the eager path instead of silently returning full-resolution masks. Passing the reduced output size into the kernel is the better answer and is left as a follow-up; it already parameterises its resize tables by output size. Default remains 1.0 and short-circuits before any arithmetic, so output is unchanged. Tests: python3 -m pytest tests/unit_tests -q 1597 passed. The 12 failures are pre-existing and environmental: five compare realpath-resolved against unresolved temp paths and cannot pass on macOS, seven need the optional sam3 package. --- inference_models/inference_models/__init__.py | 7 +- .../inference_models/models/base/types.py | 7 +- .../models/common/roboflow/post_processing.py | 21 +++- .../inference_models/models/rfdetr/common.py | 10 ++ .../rfdetr_instance_segmentation_onnx.py | 3 + .../rfdetr_instance_segmentation_pytorch.py | 3 + .../rfdetr_instance_segmentation_trt.py | 3 + .../models/rfdetr/triton_postprocess.py | 7 ++ .../inference_models/models/yolact/common.py | 5 + .../yolact_instance_segmentation_onnx.py | 3 + .../yolact_instance_segmentation_trt.py | 3 + .../inference_models/models/yolo26/common.py | 5 + .../yolo26_instance_segmentation_onnx.py | 3 + ...lo26_instance_segmentation_torch_script.py | 3 + .../yolo26_instance_segmentation_trt.py | 3 + .../inference_models/models/yolov5/common.py | 5 + .../yolov5_instance_segmentation_onnx.py | 3 + .../yolov5_instance_segmentation_trt.py | 3 + .../inference_models/models/yolov7/common.py | 5 + .../yolov7_instance_segmentation_onnx.py | 3 + .../yolov7_instance_segmentation_trt.py | 3 + .../inference_models/models/yolov8/common.py | 5 + .../yolov8_instance_segmentation_onnx.py | 3 + ...lov8_instance_segmentation_torch_script.py | 3 + .../yolov8_instance_segmentation_trt.py | 3 + .../weights_providers/offline_registry.py | 3 +- .../models/base/test_instance_segmentation.py | 16 ++- .../common/roboflow/test_post_processing.py | 97 +++++++++++++++++++ .../models/rfdetr/test_triton_postprocess.py | 37 +++++++ .../unit_tests/models/test_mage_vl_hf.py | 10 +- 30 files changed, 266 insertions(+), 19 deletions(-) diff --git a/inference_models/inference_models/__init__.py b/inference_models/inference_models/__init__.py index 9a6ab03045..a300fb97cb 100644 --- a/inference_models/inference_models/__init__.py +++ b/inference_models/inference_models/__init__.py @@ -21,7 +21,6 @@ # before any heavy library gets imported. from inference_models._offline import OFFLINE_MODE # noqa: F401 isort: skip - try: # This will read version from pyproject.toml __version__ = importlib_metadata.version(__package__ or __name__) @@ -79,9 +78,9 @@ from inference_models.models.base.types import InstancesRLEMasks from inference_models.weights_providers.entities import Quantization from inference_models.weights_providers.offline_registry import ( - OfflinePackagePresence, OfflineArtefactStatus, - OfflinePackageStatus, + OfflineArtefactVerification, OfflineModelStatus, - OfflineArtefactVerification + OfflinePackagePresence, + OfflinePackageStatus, ) diff --git a/inference_models/inference_models/models/base/types.py b/inference_models/inference_models/models/base/types.py index 99ba2ee10b..fc0999ec40 100644 --- a/inference_models/inference_models/models/base/types.py +++ b/inference_models/inference_models/models/base/types.py @@ -29,4 +29,9 @@ def from_coco_rle_masks( return cls(image_size=image_size, masks=masks, mask_size=mask_size) def to_coco_rle_masks(self) -> List[dict]: - return [{"size": list(self.image_size), "counts": m} for m in self.masks] + # COCO `size` describes the grid the counts were encoded on. When that + # is not the image grid, declaring the image size would make the counts + # disagree with it, which pycocotools decodes silently wrong. + encoded_size = list(self.mask_size or self.image_size) + + return [{"size": encoded_size, "counts": m} for m in self.masks] diff --git a/inference_models/inference_models/models/common/roboflow/post_processing.py b/inference_models/inference_models/models/common/roboflow/post_processing.py index cf2e2d1bb1..33acf006fe 100644 --- a/inference_models/inference_models/models/common/roboflow/post_processing.py +++ b/inference_models/inference_models/models/common/roboflow/post_processing.py @@ -639,6 +639,7 @@ def align_instance_segmentation_results_to_rle_masks( inference_size: ImageDimensions, static_crop_offset: StaticCropOffset, binarization_threshold: float = 0.0, + masks_resolution_factor: float = 1.0, ) -> Generator[Tuple[torch.Tensor, dict], None, None]: """ Generator variant of align_instance_segmentation_results. @@ -730,10 +731,20 @@ def align_instance_segmentation_results_to_rle_masks( :, mask_pad_top : mh - mask_pad_bottom, mask_pad_left : mw - mask_pad_right ] - target_h = size_after_pre_processing.height - target_w = size_after_pre_processing.width - offset_y = static_crop_offset.offset_y - offset_x = static_crop_offset.offset_x + target_h, target_w = resolve_mask_target_size( + mask_height=masks.shape[1], + mask_width=masks.shape[2], + size_after_pre_processing=size_after_pre_processing, + masks_resolution_factor=masks_resolution_factor, + ) + # the canvas keeps the mask grid's ratio to the image, matching the dense + # path, so crop offsets move into the same space + canvas_h_scale = target_h / size_after_pre_processing.height + canvas_w_scale = target_w / size_after_pre_processing.width + canvas_height = max(1, round(original_size.height * canvas_h_scale)) + canvas_width = max(1, round(original_size.width * canvas_w_scale)) + offset_y = round(static_crop_offset.offset_y * canvas_h_scale) + offset_x = round(static_crop_offset.offset_x * canvas_w_scale) num_instances = image_bboxes.shape[0] for i in range(num_instances): # keep a batch dim so functional.resize is unambiguous @@ -749,7 +760,7 @@ def align_instance_segmentation_results_to_rle_masks( ) if needs_canvas: mask_canvas = torch.zeros( - (original_size.height, original_size.width), + (canvas_height, canvas_width), dtype=torch.bool, device=resized.device, ) diff --git a/inference_models/inference_models/models/rfdetr/common.py b/inference_models/inference_models/models/rfdetr/common.py index 5dfd977f1b..0772f26ac4 100644 --- a/inference_models/inference_models/models/rfdetr/common.py +++ b/inference_models/inference_models/models/rfdetr/common.py @@ -148,6 +148,7 @@ def post_process_instance_segmentation_results( num_classes: int, classes_re_mapping: Optional[ClassesReMapping], max_detections: Optional[int] = None, + masks_resolution_factor: float = 1.0, ) -> List[InstanceDetections]: logits_sigmoid = torch.nn.functional.sigmoid(logits) results = [] @@ -231,6 +232,7 @@ def post_process_instance_segmentation_results( size_after_pre_processing=image_meta.size_after_pre_processing, inference_size=denorm_size, static_crop_offset=image_meta.static_crop_offset, + masks_resolution_factor=masks_resolution_factor, ) detections = InstanceDetections( xyxy=aligned_boxes.round().int(), @@ -252,6 +254,7 @@ def _post_process_single_instance_segmentation_result_to_rle_masks_with_triton( classes_re_mapping: Optional[ClassesReMapping], max_detections: Optional[int] = None, defer_postprocess_sync: bool = False, + masks_resolution_factor: float = 1.0, ) -> InstanceDetections: global _TRITON_POSTPROC_JIT_DISABLED @@ -268,6 +271,7 @@ def _post_process_single_instance_segmentation_result_to_rle_masks_with_triton( classes_re_mapping=classes_re_mapping, max_detections=max_detections, defer_postprocess_sync=defer_postprocess_sync, + masks_resolution_factor=masks_resolution_factor, ) ) except Exception as exc: @@ -294,6 +298,7 @@ def _post_process_single_instance_segmentation_result_to_rle_masks_with_triton( num_classes=num_classes, classes_re_mapping=classes_re_mapping, max_detections=max_detections, + masks_resolution_factor=masks_resolution_factor, ) @@ -306,6 +311,7 @@ def _post_process_single_instance_segmentation_result_to_rle_masks( num_classes: int, classes_re_mapping: Optional[ClassesReMapping], max_detections: Optional[int] = None, + masks_resolution_factor: float = 1.0, ) -> InstanceDetections: num_queries, num_logits_classes = image_logits.shape flat_scores = image_logits.reshape(-1) @@ -377,6 +383,7 @@ def _post_process_single_instance_segmentation_result_to_rle_masks( size_after_pre_processing=image_meta.size_after_pre_processing, inference_size=denorm_size, static_crop_offset=image_meta.static_crop_offset, + masks_resolution_factor=masks_resolution_factor, ): aligned_boxes.append(bbox) rle_masks.append(mask) @@ -386,6 +393,7 @@ def _post_process_single_instance_segmentation_result_to_rle_masks( image_meta.original_size.width, ), masks=rle_masks, + mask_size=tuple(rle_masks[0]["size"]) if rle_masks else None, ) if len(aligned_boxes) > 0: aligned_boxes_tensor = torch.stack(aligned_boxes, dim=0) @@ -411,6 +419,7 @@ def post_process_instance_segmentation_results_to_rle_masks( classes_re_mapping: Optional[ClassesReMapping], max_detections: Optional[int] = None, defer_postprocess_sync: bool = False, + masks_resolution_factor: float = 1.0, ) -> List[InstanceDetections]: logits_sigmoid = torch.nn.functional.sigmoid(logits) device = bboxes.device @@ -446,6 +455,7 @@ def post_process_instance_segmentation_results_to_rle_masks( num_classes=num_classes, classes_re_mapping=classes_re_mapping, max_detections=max_detections, + masks_resolution_factor=masks_resolution_factor, ) for image_bboxes, image_logits, image_masks, image_meta in zip( bboxes, diff --git a/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_onnx.py b/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_onnx.py index f42f209a37..a8952e5c1f 100644 --- a/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_onnx.py +++ b/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_onnx.py @@ -83,6 +83,7 @@ def from_pretrained( device: torch.device = DEFAULT_DEVICE, rf_detr_max_input_resolution: Optional[Union[int, Tuple[int, int]]] = None, recommended_parameters: Optional[RecommendedParameters] = None, + masks_resolution_factor: float = 1.0, **kwargs, ) -> "RFDetrForInstanceSegmentationOnnx": if onnx_execution_providers is None: @@ -270,6 +271,7 @@ def post_process( num_classes=len(self.class_names), classes_re_mapping=self._classes_re_mapping, max_detections=max_detections, + masks_resolution_factor=masks_resolution_factor, ) else: results = post_process_instance_segmentation_results_to_rle_masks( @@ -281,6 +283,7 @@ def post_process( num_classes=len(self.class_names), classes_re_mapping=self._classes_re_mapping, max_detections=max_detections, + masks_resolution_factor=masks_resolution_factor, ) if post_process_stream is not None: post_process_stream.synchronize() diff --git a/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_pytorch.py b/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_pytorch.py index cfadf7729d..5966105f01 100644 --- a/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_pytorch.py +++ b/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_pytorch.py @@ -99,6 +99,7 @@ def from_pretrained( resolution: Optional[int] = None, rf_detr_max_input_resolution: Optional[Union[int, Tuple[int, int]]] = None, recommended_parameters: Optional[RecommendedParameters] = None, + masks_resolution_factor: float = 1.0, **kwargs, ) -> "RFDetrForInstanceSegmentationTorch": if os.path.isfile(model_name_or_path): @@ -462,6 +463,7 @@ def post_process( num_classes=len(self.class_names), classes_re_mapping=self._classes_re_mapping, max_detections=max_detections, + masks_resolution_factor=masks_resolution_factor, ) else: results = post_process_instance_segmentation_results_to_rle_masks( @@ -473,5 +475,6 @@ def post_process( num_classes=len(self.class_names), classes_re_mapping=self._classes_re_mapping, max_detections=max_detections, + masks_resolution_factor=masks_resolution_factor, ) return results diff --git a/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_trt.py b/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_trt.py index 0aaa3b427b..0d8d4cf18f 100644 --- a/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_trt.py +++ b/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_trt.py @@ -117,6 +117,7 @@ def from_pretrained( default_trt_cuda_graph_cache_size: int = 8, rf_detr_max_input_resolution: Optional[Union[int, Tuple[int, int]]] = None, recommended_parameters: Optional[RecommendedParameters] = None, + masks_resolution_factor: float = 1.0, **kwargs, ) -> "RFDetrForInstanceSegmentationTRT": if device.type != "cuda": @@ -461,6 +462,7 @@ def post_process( num_classes=len(self.class_names), classes_re_mapping=self._classes_re_mapping, max_detections=max_detections, + masks_resolution_factor=masks_resolution_factor, ) else: results = post_process_instance_segmentation_results_to_rle_masks( @@ -473,6 +475,7 @@ def post_process( classes_re_mapping=self._classes_re_mapping, max_detections=max_detections, defer_postprocess_sync=kwargs.get("defer_postprocess_sync", False), + masks_resolution_factor=masks_resolution_factor, ) if graph_state is not None: output_consumed_events = [ diff --git a/inference_models/inference_models/models/rfdetr/triton_postprocess.py b/inference_models/inference_models/models/rfdetr/triton_postprocess.py index 3596fe63df..9a810f8d29 100644 --- a/inference_models/inference_models/models/rfdetr/triton_postprocess.py +++ b/inference_models/inference_models/models/rfdetr/triton_postprocess.py @@ -225,6 +225,7 @@ def post_process_single_instance_segmentation_result_to_rle_masks_triton( classes_re_mapping: Optional[ClassesReMapping], max_detections: Optional[int] = None, defer_postprocess_sync: bool = False, + masks_resolution_factor: float = 1.0, ) -> Optional[InstanceDetections]: """Run the sparse Triton RF-DETR RLE postprocess path for one image. @@ -248,6 +249,7 @@ def post_process_single_instance_segmentation_result_to_rle_masks_triton( image_masks=image_masks, image_meta=image_meta, threshold=threshold, + masks_resolution_factor=masks_resolution_factor, classes_re_mapping=classes_re_mapping, ) if unsupported_reason is not None: @@ -925,10 +927,15 @@ def _unsupported_triton_postprocess_reason( image_meta: PreProcessingMetadata, threshold: Union[float, torch.Tensor], classes_re_mapping: Optional[ClassesReMapping], + masks_resolution_factor: float = 1.0, ) -> Optional[str]: """Explain why the Triton path should not run, or ``None`` when supported.""" if triton is None: return "triton_unavailable" + if masks_resolution_factor != 1.0: + # the fused kernel interpolates straight to the image; it cannot honour + # a reduced target, so defer rather than silently ignore the request + return "mask_resolution_factor_unsupported" if classes_re_mapping is None: return "class_remapping_required" if isinstance(threshold, torch.Tensor): diff --git a/inference_models/inference_models/models/yolact/common.py b/inference_models/inference_models/models/yolact/common.py index 172ca12583..c46da769b5 100644 --- a/inference_models/inference_models/models/yolact/common.py +++ b/inference_models/inference_models/models/yolact/common.py @@ -15,6 +15,7 @@ def prepare_dense_masks( nms_results: List[torch.Tensor], all_proto_data: torch.Tensor, pre_processing_meta: List[PreProcessingMetadata], + masks_resolution_factor: float = 1.0, ) -> List[InstanceDetections]: final_results = [] for image_bboxes, image_protos, image_meta in zip( @@ -41,6 +42,7 @@ def prepare_dense_masks( inference_size=image_meta.inference_size, static_crop_offset=image_meta.static_crop_offset, binarization_threshold=0.5, + masks_resolution_factor=masks_resolution_factor, ) final_results.append( InstanceDetections( @@ -57,6 +59,7 @@ def prepare_rle_masks( nms_results: List[torch.Tensor], all_proto_data: torch.Tensor, pre_processing_meta: List[PreProcessingMetadata], + masks_resolution_factor: float = 1.0, ) -> List[InstanceDetections]: final_results = [] for image_bboxes, image_protos, image_meta in zip( @@ -84,6 +87,7 @@ def prepare_rle_masks( inference_size=image_meta.inference_size, static_crop_offset=image_meta.static_crop_offset, binarization_threshold=0.5, + masks_resolution_factor=masks_resolution_factor, ): aligned_boxes.append(bbox) rle_masks.append(mask) @@ -93,6 +97,7 @@ def prepare_rle_masks( image_meta.original_size.width, ), masks=rle_masks, + mask_size=tuple(rle_masks[0]["size"]) if rle_masks else None, ) if len(aligned_boxes) > 0: aligned_boxes_tensor = torch.stack(aligned_boxes, dim=0) diff --git a/inference_models/inference_models/models/yolact/yolact_instance_segmentation_onnx.py b/inference_models/inference_models/models/yolact/yolact_instance_segmentation_onnx.py index 1348f70425..ae8ed55ba5 100644 --- a/inference_models/inference_models/models/yolact/yolact_instance_segmentation_onnx.py +++ b/inference_models/inference_models/models/yolact/yolact_instance_segmentation_onnx.py @@ -246,6 +246,7 @@ def post_process( max_detections: int = INFERENCE_MODELS_YOLACT_DEFAULT_MAX_DETECTIONS, class_agnostic_nms: bool = INFERENCE_MODELS_YOLACT_DEFAULT_CLASS_AGNOSTIC_NMS, mask_format: InstanceSegmentationMaskFormat = "dense", + masks_resolution_factor: float = 1.0, **kwargs, ) -> List[InstanceDetections]: if mask_format not in self.supported_mask_formats: @@ -317,12 +318,14 @@ def post_process( nms_results=nms_results, all_proto_data=all_proto_data, pre_processing_meta=pre_processing_meta, + masks_resolution_factor=masks_resolution_factor, ) else: final_results = prepare_rle_masks( nms_results=nms_results, all_proto_data=all_proto_data, pre_processing_meta=pre_processing_meta, + masks_resolution_factor=masks_resolution_factor, ) if post_process_stream is not None: post_process_stream.synchronize() diff --git a/inference_models/inference_models/models/yolact/yolact_instance_segmentation_trt.py b/inference_models/inference_models/models/yolact/yolact_instance_segmentation_trt.py index 1f28ca50fa..bd61ec5ae6 100644 --- a/inference_models/inference_models/models/yolact/yolact_instance_segmentation_trt.py +++ b/inference_models/inference_models/models/yolact/yolact_instance_segmentation_trt.py @@ -293,6 +293,7 @@ def post_process( max_detections: int = INFERENCE_MODELS_YOLACT_DEFAULT_MAX_DETECTIONS, class_agnostic_nms: bool = INFERENCE_MODELS_YOLACT_DEFAULT_CLASS_AGNOSTIC_NMS, mask_format: InstanceSegmentationMaskFormat = "dense", + masks_resolution_factor: float = 1.0, **kwargs, ) -> List[InstanceDetections]: if mask_format not in self.supported_mask_formats: @@ -362,12 +363,14 @@ def post_process( nms_results=nms_results, all_proto_data=all_proto_data, pre_processing_meta=pre_processing_meta, + masks_resolution_factor=masks_resolution_factor, ) else: final_results = prepare_rle_masks( nms_results=nms_results, all_proto_data=all_proto_data, pre_processing_meta=pre_processing_meta, + masks_resolution_factor=masks_resolution_factor, ) self._post_process_stream.synchronize() return final_results diff --git a/inference_models/inference_models/models/yolo26/common.py b/inference_models/inference_models/models/yolo26/common.py index 3d0fd90fc4..1dd17a352f 100644 --- a/inference_models/inference_models/models/yolo26/common.py +++ b/inference_models/inference_models/models/yolo26/common.py @@ -16,6 +16,7 @@ def prepare_dense_masks( filtered_results: List[torch.Tensor], protos: torch.Tensor, pre_processing_meta: List[PreProcessingMetadata], + masks_resolution_factor: float = 1.0, ) -> List[InstanceDetections]: final_results = [] for image_bboxes, image_protos, image_meta in zip( @@ -42,6 +43,7 @@ def prepare_dense_masks( size_after_pre_processing=image_meta.size_after_pre_processing, inference_size=image_meta.inference_size, static_crop_offset=image_meta.static_crop_offset, + masks_resolution_factor=masks_resolution_factor, ) final_results.append( InstanceDetections( @@ -58,6 +60,7 @@ def prepare_rle_masks( filtered_results: List[torch.Tensor], protos: torch.Tensor, pre_processing_meta: List[PreProcessingMetadata], + masks_resolution_factor: float = 1.0, ) -> List[InstanceDetections]: final_results = [] for image_bboxes, image_protos, image_meta in zip( @@ -85,6 +88,7 @@ def prepare_rle_masks( size_after_pre_processing=image_meta.size_after_pre_processing, inference_size=image_meta.inference_size, static_crop_offset=image_meta.static_crop_offset, + masks_resolution_factor=masks_resolution_factor, ): aligned_boxes.append(bbox) rle_masks.append(mask) @@ -94,6 +98,7 @@ def prepare_rle_masks( image_meta.original_size.width, ), masks=rle_masks, + mask_size=tuple(rle_masks[0]["size"]) if rle_masks else None, ) if len(aligned_boxes) > 0: aligned_boxes_tensor = torch.stack(aligned_boxes, dim=0) diff --git a/inference_models/inference_models/models/yolo26/yolo26_instance_segmentation_onnx.py b/inference_models/inference_models/models/yolo26/yolo26_instance_segmentation_onnx.py index 8267632683..1844e08f53 100644 --- a/inference_models/inference_models/models/yolo26/yolo26_instance_segmentation_onnx.py +++ b/inference_models/inference_models/models/yolo26/yolo26_instance_segmentation_onnx.py @@ -214,6 +214,7 @@ def post_process( pre_processing_meta: List[PreProcessingMetadata], confidence: Confidence = "default", mask_format: InstanceSegmentationMaskFormat = "dense", + masks_resolution_factor: float = 1.0, **kwargs, ) -> List[InstanceDetections]: if mask_format not in self.supported_mask_formats: @@ -246,12 +247,14 @@ def post_process( filtered_results=filtered_results, protos=protos, pre_processing_meta=pre_processing_meta, + masks_resolution_factor=masks_resolution_factor, ) else: result = prepare_rle_masks( filtered_results=filtered_results, protos=protos, pre_processing_meta=pre_processing_meta, + masks_resolution_factor=masks_resolution_factor, ) if post_process_stream is not None: post_process_stream.synchronize() diff --git a/inference_models/inference_models/models/yolo26/yolo26_instance_segmentation_torch_script.py b/inference_models/inference_models/models/yolo26/yolo26_instance_segmentation_torch_script.py index cfa71e6361..0fd3266c79 100644 --- a/inference_models/inference_models/models/yolo26/yolo26_instance_segmentation_torch_script.py +++ b/inference_models/inference_models/models/yolo26/yolo26_instance_segmentation_torch_script.py @@ -172,6 +172,7 @@ def post_process( pre_processing_meta: List[PreProcessingMetadata], confidence: Confidence = "default", mask_format: InstanceSegmentationMaskFormat = "dense", + masks_resolution_factor: float = 1.0, **kwargs, ) -> List[InstanceDetections]: if mask_format not in self.supported_mask_formats: @@ -199,9 +200,11 @@ def post_process( filtered_results=filtered_results, protos=protos, pre_processing_meta=pre_processing_meta, + masks_resolution_factor=masks_resolution_factor, ) return prepare_rle_masks( filtered_results=filtered_results, protos=protos, pre_processing_meta=pre_processing_meta, + masks_resolution_factor=masks_resolution_factor, ) diff --git a/inference_models/inference_models/models/yolo26/yolo26_instance_segmentation_trt.py b/inference_models/inference_models/models/yolo26/yolo26_instance_segmentation_trt.py index 40ce8f92bc..6e35061591 100644 --- a/inference_models/inference_models/models/yolo26/yolo26_instance_segmentation_trt.py +++ b/inference_models/inference_models/models/yolo26/yolo26_instance_segmentation_trt.py @@ -264,6 +264,7 @@ def post_process( pre_processing_meta: List[PreProcessingMetadata], confidence: Confidence = "default", mask_format: InstanceSegmentationMaskFormat = "dense", + masks_resolution_factor: float = 1.0, **kwargs, ) -> List[InstanceDetections]: if mask_format not in self.supported_mask_formats: @@ -294,12 +295,14 @@ def post_process( filtered_results=filtered_results, protos=protos, pre_processing_meta=pre_processing_meta, + masks_resolution_factor=masks_resolution_factor, ) else: result = prepare_rle_masks( filtered_results=filtered_results, protos=protos, pre_processing_meta=pre_processing_meta, + masks_resolution_factor=masks_resolution_factor, ) self._post_process_stream.synchronize() return result diff --git a/inference_models/inference_models/models/yolov5/common.py b/inference_models/inference_models/models/yolov5/common.py index 9284835534..057c1a31f4 100644 --- a/inference_models/inference_models/models/yolov5/common.py +++ b/inference_models/inference_models/models/yolov5/common.py @@ -16,6 +16,7 @@ def prepare_dense_masks( nms_results: List[torch.Tensor], protos: torch.Tensor, pre_processing_meta: List[PreProcessingMetadata], + masks_resolution_factor: float = 1.0, ) -> List[InstanceDetections]: final_results = [] for image_bboxes, image_protos, image_meta in zip( @@ -42,6 +43,7 @@ def prepare_dense_masks( size_after_pre_processing=image_meta.size_after_pre_processing, inference_size=image_meta.inference_size, static_crop_offset=image_meta.static_crop_offset, + masks_resolution_factor=masks_resolution_factor, ) final_results.append( InstanceDetections( @@ -58,6 +60,7 @@ def prepare_rle_masks( nms_results: List[torch.Tensor], protos: torch.Tensor, pre_processing_meta: List[PreProcessingMetadata], + masks_resolution_factor: float = 1.0, ) -> List[InstanceDetections]: final_results = [] for image_bboxes, image_protos, image_meta in zip( @@ -85,6 +88,7 @@ def prepare_rle_masks( size_after_pre_processing=image_meta.size_after_pre_processing, inference_size=image_meta.inference_size, static_crop_offset=image_meta.static_crop_offset, + masks_resolution_factor=masks_resolution_factor, ): aligned_boxes.append(bbox) rle_masks.append(mask) @@ -94,6 +98,7 @@ def prepare_rle_masks( image_meta.original_size.width, ), masks=rle_masks, + mask_size=tuple(rle_masks[0]["size"]) if rle_masks else None, ) if len(aligned_boxes) > 0: aligned_boxes_tensor = torch.stack(aligned_boxes, dim=0) diff --git a/inference_models/inference_models/models/yolov5/yolov5_instance_segmentation_onnx.py b/inference_models/inference_models/models/yolov5/yolov5_instance_segmentation_onnx.py index 9cd8c1cef1..7e53ad97fc 100644 --- a/inference_models/inference_models/models/yolov5/yolov5_instance_segmentation_onnx.py +++ b/inference_models/inference_models/models/yolov5/yolov5_instance_segmentation_onnx.py @@ -221,6 +221,7 @@ def post_process( max_detections: int = INFERENCE_MODELS_YOLOV5_DEFAULT_MAX_DETECTIONS, class_agnostic_nms: bool = INFERENCE_MODELS_YOLOV5_DEFAULT_CLASS_AGNOSTIC_NMS, mask_format: InstanceSegmentationMaskFormat = "dense", + masks_resolution_factor: float = 1.0, **kwargs, ) -> List[InstanceDetections]: if mask_format not in self.supported_mask_formats: @@ -257,12 +258,14 @@ def post_process( nms_results=nms_results, protos=protos, pre_processing_meta=pre_processing_meta, + masks_resolution_factor=masks_resolution_factor, ) else: final_results = prepare_rle_masks( nms_results=nms_results, protos=protos, pre_processing_meta=pre_processing_meta, + masks_resolution_factor=masks_resolution_factor, ) if post_process_stream is not None: post_process_stream.synchronize() diff --git a/inference_models/inference_models/models/yolov5/yolov5_instance_segmentation_trt.py b/inference_models/inference_models/models/yolov5/yolov5_instance_segmentation_trt.py index b7ec5856f2..c48deeb476 100644 --- a/inference_models/inference_models/models/yolov5/yolov5_instance_segmentation_trt.py +++ b/inference_models/inference_models/models/yolov5/yolov5_instance_segmentation_trt.py @@ -273,6 +273,7 @@ def post_process( max_detections: int = INFERENCE_MODELS_YOLOV5_DEFAULT_MAX_DETECTIONS, class_agnostic_nms: bool = INFERENCE_MODELS_YOLOV5_DEFAULT_CLASS_AGNOSTIC_NMS, mask_format: InstanceSegmentationMaskFormat = "dense", + masks_resolution_factor: float = 1.0, **kwargs, ) -> List[InstanceDetections]: if mask_format not in self.supported_mask_formats: @@ -307,12 +308,14 @@ def post_process( nms_results=nms_results, protos=protos, pre_processing_meta=pre_processing_meta, + masks_resolution_factor=masks_resolution_factor, ) else: final_results = prepare_rle_masks( nms_results=nms_results, protos=protos, pre_processing_meta=pre_processing_meta, + masks_resolution_factor=masks_resolution_factor, ) self._post_process_stream.synchronize() return final_results diff --git a/inference_models/inference_models/models/yolov7/common.py b/inference_models/inference_models/models/yolov7/common.py index 9284835534..057c1a31f4 100644 --- a/inference_models/inference_models/models/yolov7/common.py +++ b/inference_models/inference_models/models/yolov7/common.py @@ -16,6 +16,7 @@ def prepare_dense_masks( nms_results: List[torch.Tensor], protos: torch.Tensor, pre_processing_meta: List[PreProcessingMetadata], + masks_resolution_factor: float = 1.0, ) -> List[InstanceDetections]: final_results = [] for image_bboxes, image_protos, image_meta in zip( @@ -42,6 +43,7 @@ def prepare_dense_masks( size_after_pre_processing=image_meta.size_after_pre_processing, inference_size=image_meta.inference_size, static_crop_offset=image_meta.static_crop_offset, + masks_resolution_factor=masks_resolution_factor, ) final_results.append( InstanceDetections( @@ -58,6 +60,7 @@ def prepare_rle_masks( nms_results: List[torch.Tensor], protos: torch.Tensor, pre_processing_meta: List[PreProcessingMetadata], + masks_resolution_factor: float = 1.0, ) -> List[InstanceDetections]: final_results = [] for image_bboxes, image_protos, image_meta in zip( @@ -85,6 +88,7 @@ def prepare_rle_masks( size_after_pre_processing=image_meta.size_after_pre_processing, inference_size=image_meta.inference_size, static_crop_offset=image_meta.static_crop_offset, + masks_resolution_factor=masks_resolution_factor, ): aligned_boxes.append(bbox) rle_masks.append(mask) @@ -94,6 +98,7 @@ def prepare_rle_masks( image_meta.original_size.width, ), masks=rle_masks, + mask_size=tuple(rle_masks[0]["size"]) if rle_masks else None, ) if len(aligned_boxes) > 0: aligned_boxes_tensor = torch.stack(aligned_boxes, dim=0) diff --git a/inference_models/inference_models/models/yolov7/yolov7_instance_segmentation_onnx.py b/inference_models/inference_models/models/yolov7/yolov7_instance_segmentation_onnx.py index 7a08e59ae1..dbf2253dad 100644 --- a/inference_models/inference_models/models/yolov7/yolov7_instance_segmentation_onnx.py +++ b/inference_models/inference_models/models/yolov7/yolov7_instance_segmentation_onnx.py @@ -219,6 +219,7 @@ def post_process( max_detections: int = INFERENCE_MODELS_YOLOV7_DEFAULT_MAX_DETECTIONS, class_agnostic_nms: bool = INFERENCE_MODELS_YOLOV7_DEFAULT_CLASS_AGNOSTIC_NMS, mask_format: InstanceSegmentationMaskFormat = "dense", + masks_resolution_factor: float = 1.0, **kwargs, ) -> List[InstanceDetections]: if mask_format not in self.supported_mask_formats: @@ -262,12 +263,14 @@ def post_process( nms_results=nms_results, protos=protos, pre_processing_meta=pre_processing_meta, + masks_resolution_factor=masks_resolution_factor, ) else: final_results = prepare_rle_masks( nms_results=nms_results, protos=protos, pre_processing_meta=pre_processing_meta, + masks_resolution_factor=masks_resolution_factor, ) if post_process_stream is not None: post_process_stream.synchronize() diff --git a/inference_models/inference_models/models/yolov7/yolov7_instance_segmentation_trt.py b/inference_models/inference_models/models/yolov7/yolov7_instance_segmentation_trt.py index 664e8568f9..7a37e77dd2 100644 --- a/inference_models/inference_models/models/yolov7/yolov7_instance_segmentation_trt.py +++ b/inference_models/inference_models/models/yolov7/yolov7_instance_segmentation_trt.py @@ -266,6 +266,7 @@ def post_process( max_detections: int = INFERENCE_MODELS_YOLOV7_DEFAULT_MAX_DETECTIONS, class_agnostic_nms: bool = INFERENCE_MODELS_YOLOV7_DEFAULT_CLASS_AGNOSTIC_NMS, mask_format: InstanceSegmentationMaskFormat = "dense", + masks_resolution_factor: float = 1.0, **kwargs, ) -> List[InstanceDetections]: if mask_format not in self.supported_mask_formats: @@ -307,12 +308,14 @@ def post_process( nms_results=nms_results, protos=protos, pre_processing_meta=pre_processing_meta, + masks_resolution_factor=masks_resolution_factor, ) else: final_results = prepare_rle_masks( nms_results=nms_results, protos=protos, pre_processing_meta=pre_processing_meta, + masks_resolution_factor=masks_resolution_factor, ) self._post_process_stream.synchronize() return final_results diff --git a/inference_models/inference_models/models/yolov8/common.py b/inference_models/inference_models/models/yolov8/common.py index f808aaad61..6e39830090 100644 --- a/inference_models/inference_models/models/yolov8/common.py +++ b/inference_models/inference_models/models/yolov8/common.py @@ -18,6 +18,7 @@ def prepare_dense_masks( pre_processing_meta: List[PreProcessingMetadata], masks_smoothing_enabled: bool, masks_binarization_threshold: float, + masks_resolution_factor: float = 1.0, ) -> List[InstanceDetections]: final_results = [] for image_bboxes, image_protos, image_meta in zip( @@ -47,6 +48,7 @@ def prepare_dense_masks( inference_size=image_meta.inference_size, static_crop_offset=image_meta.static_crop_offset, binarization_threshold=masks_binarization_threshold, + masks_resolution_factor=masks_resolution_factor, ) final_results.append( InstanceDetections( @@ -65,6 +67,7 @@ def prepare_rle_masks( pre_processing_meta: List[PreProcessingMetadata], masks_smoothing_enabled: bool, masks_binarization_threshold: float, + masks_resolution_factor: float = 1.0, ) -> List[InstanceDetections]: final_results = [] for image_bboxes, image_protos, image_meta in zip( @@ -95,6 +98,7 @@ def prepare_rle_masks( inference_size=image_meta.inference_size, static_crop_offset=image_meta.static_crop_offset, binarization_threshold=masks_binarization_threshold, + masks_resolution_factor=masks_resolution_factor, ): aligned_boxes.append(bbox) rle_masks.append(mask) @@ -104,6 +108,7 @@ def prepare_rle_masks( image_meta.original_size.width, ), masks=rle_masks, + mask_size=tuple(rle_masks[0]["size"]) if rle_masks else None, ) if len(aligned_boxes) > 0: aligned_boxes_tensor = torch.stack(aligned_boxes, dim=0) diff --git a/inference_models/inference_models/models/yolov8/yolov8_instance_segmentation_onnx.py b/inference_models/inference_models/models/yolov8/yolov8_instance_segmentation_onnx.py index 3659cfc090..817f8557cb 100644 --- a/inference_models/inference_models/models/yolov8/yolov8_instance_segmentation_onnx.py +++ b/inference_models/inference_models/models/yolov8/yolov8_instance_segmentation_onnx.py @@ -231,6 +231,7 @@ def post_process( masks_smoothing_enabled: bool = INFERENCE_MODELS_YOLO_ULTRALYTICS_DEFAULT_MASKS_SMOOTHING_ENABLED, masks_binarization_threshold: float = INFERENCE_MODELS_YOLO_ULTRALYTICS_DEFAULT_MASKS_BINARIZATION_THRESHOLD, mask_format: InstanceSegmentationMaskFormat = "dense", + masks_resolution_factor: float = 1.0, **kwargs, ) -> List[InstanceDetections]: if mask_format not in self.supported_mask_formats: @@ -274,6 +275,7 @@ def post_process( pre_processing_meta=pre_processing_meta, masks_smoothing_enabled=masks_smoothing_enabled, masks_binarization_threshold=masks_binarization_threshold, + masks_resolution_factor=masks_resolution_factor, ) else: final_results = prepare_rle_masks( @@ -282,6 +284,7 @@ def post_process( pre_processing_meta=pre_processing_meta, masks_smoothing_enabled=masks_smoothing_enabled, masks_binarization_threshold=masks_binarization_threshold, + masks_resolution_factor=masks_resolution_factor, ) if post_process_stream is not None: post_process_stream.synchronize() diff --git a/inference_models/inference_models/models/yolov8/yolov8_instance_segmentation_torch_script.py b/inference_models/inference_models/models/yolov8/yolov8_instance_segmentation_torch_script.py index 75b6687f96..bbd79a22b3 100644 --- a/inference_models/inference_models/models/yolov8/yolov8_instance_segmentation_torch_script.py +++ b/inference_models/inference_models/models/yolov8/yolov8_instance_segmentation_torch_script.py @@ -189,6 +189,7 @@ def post_process( masks_smoothing_enabled: bool = INFERENCE_MODELS_YOLO_ULTRALYTICS_DEFAULT_MASKS_SMOOTHING_ENABLED, masks_binarization_threshold: float = INFERENCE_MODELS_YOLO_ULTRALYTICS_DEFAULT_MASKS_BINARIZATION_THRESHOLD, mask_format: InstanceSegmentationMaskFormat = "dense", + masks_resolution_factor: float = 1.0, **kwargs, ) -> List[InstanceDetections]: if mask_format not in self.supported_mask_formats: @@ -227,6 +228,7 @@ def post_process( pre_processing_meta=pre_processing_meta, masks_smoothing_enabled=masks_smoothing_enabled, masks_binarization_threshold=masks_binarization_threshold, + masks_resolution_factor=masks_resolution_factor, ) return prepare_rle_masks( nms_results=nms_results, @@ -234,4 +236,5 @@ def post_process( pre_processing_meta=pre_processing_meta, masks_smoothing_enabled=masks_smoothing_enabled, masks_binarization_threshold=masks_binarization_threshold, + masks_resolution_factor=masks_resolution_factor, ) diff --git a/inference_models/inference_models/models/yolov8/yolov8_instance_segmentation_trt.py b/inference_models/inference_models/models/yolov8/yolov8_instance_segmentation_trt.py index a4d8743e52..87ebd7fa8f 100644 --- a/inference_models/inference_models/models/yolov8/yolov8_instance_segmentation_trt.py +++ b/inference_models/inference_models/models/yolov8/yolov8_instance_segmentation_trt.py @@ -280,6 +280,7 @@ def post_process( masks_smoothing_enabled: bool = INFERENCE_MODELS_YOLO_ULTRALYTICS_DEFAULT_MASKS_SMOOTHING_ENABLED, masks_binarization_threshold: float = INFERENCE_MODELS_YOLO_ULTRALYTICS_DEFAULT_MASKS_BINARIZATION_THRESHOLD, mask_format: InstanceSegmentationMaskFormat = "dense", + masks_resolution_factor: float = 1.0, **kwargs, ) -> List[InstanceDetections]: if mask_format not in self.supported_mask_formats: @@ -321,6 +322,7 @@ def post_process( pre_processing_meta=pre_processing_meta, masks_smoothing_enabled=masks_smoothing_enabled, masks_binarization_threshold=masks_binarization_threshold, + masks_resolution_factor=masks_resolution_factor, ) else: final_results = prepare_rle_masks( @@ -329,6 +331,7 @@ def post_process( pre_processing_meta=pre_processing_meta, masks_smoothing_enabled=masks_smoothing_enabled, masks_binarization_threshold=masks_binarization_threshold, + masks_resolution_factor=masks_resolution_factor, ) self._post_process_stream.synchronize() return final_results diff --git a/inference_models/inference_models/weights_providers/offline_registry.py b/inference_models/inference_models/weights_providers/offline_registry.py index d85d01f26c..954d266bc1 100644 --- a/inference_models/inference_models/weights_providers/offline_registry.py +++ b/inference_models/inference_models/weights_providers/offline_registry.py @@ -667,8 +667,7 @@ def _parse_registry_timestamp( except ValueError: pass LOGGER.warning( - "Offline-weights registry record contains unparsable timestamp for " - "%s: %r", + "Offline-weights registry record contains unparsable timestamp for " "%s: %r", context, value, ) diff --git a/inference_models/tests/unit_tests/models/base/test_instance_segmentation.py b/inference_models/tests/unit_tests/models/base/test_instance_segmentation.py index 594f81eeb7..e3be721bfc 100644 --- a/inference_models/tests/unit_tests/models/base/test_instance_segmentation.py +++ b/inference_models/tests/unit_tests/models/base/test_instance_segmentation.py @@ -74,10 +74,10 @@ def test_rle_mask_size_can_differ_from_image_size(self) -> None: assert masks.mask_size == (270, 480) assert masks.image_size == (1080, 1920) - def test_rle_wire_format_is_unchanged(self) -> None: + def test_wire_format_declares_the_encoded_size(self) -> None: # given - # the COCO `size` field is the deferred wire-format decision; adding - # mask_size must not alter what goes on the wire today + # COCO `size` must describe the grid the counts were encoded on, or the + # counts no longer sum to h*w and pycocotools decodes silently wrong masks = InstancesRLEMasks( image_size=(1080, 1920), masks=[b"abc"], mask_size=(270, 480) ) @@ -85,6 +85,16 @@ def test_rle_wire_format_is_unchanged(self) -> None: # when encoded = masks.to_coco_rle_masks() + # then + assert encoded == [{"size": [270, 480], "counts": b"abc"}] + + def test_wire_format_unchanged_when_sizes_agree(self) -> None: + # given + masks = InstancesRLEMasks(image_size=(1080, 1920), masks=[b"abc"]) + + # when + encoded = masks.to_coco_rle_masks() + # then assert encoded == [{"size": [1080, 1920], "counts": b"abc"}] diff --git a/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py b/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py index af31883a06..39e5f37f7c 100644 --- a/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py +++ b/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py @@ -9,6 +9,7 @@ import numpy as np import pytest import torch +from pycocotools import mask as mask_utils from inference_models.configuration import INFERENCE_MODELS_DEFAULT_CONFIDENCE from inference_models.entities import ImageDimensions @@ -19,6 +20,7 @@ from inference_models.models.common.roboflow.post_processing import ( ConfidenceFilter, align_instance_segmentation_results, + align_instance_segmentation_results_to_rle_masks, crop_masks_to_boxes, post_process_nms_fused_model_output, rescale_image_detections, @@ -845,3 +847,98 @@ def test_static_crop_canvas_scales_with_the_factor(self) -> None: expected_h = round((self.ORIG_H + 7) * target_h / self.ORIG_H) expected_w = round((self.ORIG_W + 13) * target_w / self.ORIG_W) assert masks.shape[1:] == (expected_h, expected_w) + + +class TestRLEMasksResolutionFactor: + """The RLE generator must honour the same factor as the dense path. + + Otherwise the resolution a caller gets depends on `mask_format`, and the + two paths silently disagree. + """ + + PROTO = 160 + ORIG_H, ORIG_W = 200, 300 + + @classmethod + def _run(cls, n: int = 3, **kwargs): + torch.manual_seed(42) + bboxes = torch.rand((n, 6), dtype=torch.float32) * 100 + masks = torch.randn((n, cls.PROTO, cls.PROTO), dtype=torch.float32) + return list( + align_instance_segmentation_results_to_rle_masks( + image_bboxes=bboxes.clone(), + masks=masks.clone(), + padding=(0, 0, 0, 0), + scale_width=1.0, + scale_height=1.0, + original_size=ImageDimensions(height=cls.ORIG_H, width=cls.ORIG_W), + size_after_pre_processing=ImageDimensions( + height=cls.ORIG_H, width=cls.ORIG_W + ), + inference_size=ImageDimensions(height=640, width=640), + static_crop_offset=StaticCropOffset( + offset_x=0, + offset_y=0, + crop_width=cls.ORIG_W, + crop_height=cls.ORIG_H, + ), + binarization_threshold=0.0, + **kwargs, + ) + ) + + def test_factor_one_declares_the_image_size(self) -> None: + # given / when + results = self._run(masks_resolution_factor=1.0) + + # then + assert all(rle["size"] == [self.ORIG_H, self.ORIG_W] for _, rle in results) + + def test_reduced_factor_declares_the_reduced_size(self) -> None: + # given / when + results = self._run(masks_resolution_factor=0.5) + + # then + expected_h = round(self.PROTO * 0.5 + self.ORIG_H * 0.5) + expected_w = round(self.PROTO * 0.5 + self.ORIG_W * 0.5) + assert all(rle["size"] == [expected_h, expected_w] for _, rle in results) + + def test_counts_sum_to_the_declared_size(self) -> None: + # given / when + # the invariant pycocotools relies on; a declared size that disagrees + # with the counts decodes silently wrong rather than raising + results = self._run(masks_resolution_factor=0.25) + + # then + for _, rle in results: + decoded = mask_utils.decode(rle) + assert decoded.shape == tuple(rle["size"]) + + def test_matches_the_dense_path_resolution(self) -> None: + # given / when + # mask_format must not change the resolution a caller receives + rle_results = self._run(masks_resolution_factor=0.25) + torch.manual_seed(42) + bboxes = torch.rand((3, 6), dtype=torch.float32) * 100 + masks = torch.randn((3, self.PROTO, self.PROTO), dtype=torch.float32) + _, dense = align_instance_segmentation_results( + image_bboxes=bboxes.clone(), + masks=masks.clone(), + padding=(0, 0, 0, 0), + scale_width=1.0, + scale_height=1.0, + original_size=ImageDimensions(height=self.ORIG_H, width=self.ORIG_W), + size_after_pre_processing=ImageDimensions( + height=self.ORIG_H, width=self.ORIG_W + ), + inference_size=ImageDimensions(height=640, width=640), + static_crop_offset=StaticCropOffset( + offset_x=0, offset_y=0, crop_width=self.ORIG_W, crop_height=self.ORIG_H + ), + binarization_threshold=0.0, + mask_chunk_size=1000, + masks_resolution_factor=0.25, + ) + + # then + assert rle_results[0][1]["size"] == list(dense.shape[1:]) diff --git a/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py b/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py index c64cd686e6..d56b01da16 100644 --- a/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py +++ b/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py @@ -415,6 +415,7 @@ def spy_triton_postprocess(*args, **kwargs): metadata = _metadata() threshold = 0.4 classes_re_mapping = _class_mapping(device) + extra_kwargs = {} if case == "no_class_mapping": classes_re_mapping = None elif case == "tensor_threshold": @@ -438,6 +439,7 @@ def spy_triton_postprocess(*args, **kwargs): threshold=threshold, num_classes=2, classes_re_mapping=classes_re_mapping, + **extra_kwargs, )[0] assert calls == 1 @@ -669,3 +671,38 @@ def test_rfdetr_triton_postproc_topk_retry_matches_reference_rle_path() -> None: assert actual is not None _assert_detections_equal(actual, expected) + + +@pytest.mark.parametrize( + "factor,expected_supported", [(1.0, True), (0.5, False), (0.0, False)] +) +def test_reduced_mask_resolution_is_unsupported_by_the_fused_path( + monkeypatch, + factor: float, + expected_supported: bool, +) -> None: + # given + # the fused kernel interpolates straight to the image size, so it cannot + # honour a reduced target; it must say so rather than silently ignore it. + # triton is absent on CPU CI, and that reason is checked first, so stand it + # in to reach the check under test + monkeypatch.setattr(triton_postprocess, "triton", object()) + device = torch.device("cpu") + bboxes, logits, masks = _single_detection_inputs(device) + + # when + reason = triton_postprocess._unsupported_triton_postprocess_reason( + image_bboxes=bboxes, + image_scores=logits, + image_masks=masks, + image_meta=_metadata(), + threshold=0.4, + classes_re_mapping=_class_mapping(device), + masks_resolution_factor=factor, + ) + + # then + if expected_supported: + assert reason != "mask_resolution_factor_unsupported" + else: + assert reason == "mask_resolution_factor_unsupported" diff --git a/inference_models/tests/unit_tests/models/test_mage_vl_hf.py b/inference_models/tests/unit_tests/models/test_mage_vl_hf.py index 8bee136300..51107da4fe 100644 --- a/inference_models/tests/unit_tests/models/test_mage_vl_hf.py +++ b/inference_models/tests/unit_tests/models/test_mage_vl_hf.py @@ -207,7 +207,9 @@ def test_pre_process_generation_leaves_undeclared_tensors_alone() -> None: mage_vl.pre_process_generation(images=image, prompt="what is this?") passed_image = processor.call_args.kwargs["images"] - assert passed_image[0].max() == 255, "tensors without a declared format pass through" + assert ( + passed_image[0].max() == 255 + ), "tensors without a declared format pass through" def test_video_pre_processing_wraps_missing_dependency_errors(tmp_path) -> None: @@ -338,6 +340,6 @@ def test_post_process_generation_strips_decoded_text() -> None: processor.tokenizer.batch_decode.return_value = [" a dog \n"] mage_vl = _mage_vl(processor=processor) - assert mage_vl.post_process_generation( - generated_ids=torch.tensor([[1, 2]]) - ) == ["a dog"] + assert mage_vl.post_process_generation(generated_ids=torch.tensor([[1, 2]])) == [ + "a dog" + ] From 8d8a37db3c2b142c8716e4bbb4a07423e1d472bf Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Fri, 2 Oct 2026 16:30:46 +0200 Subject: [PATCH 11/41] Replace the tradeoff_factor plan with the implemented version The first draft's headline figures combined two code paths that never execute together, and its recommended v1 was measurably slower than the status quo on the default RLE path. Both are corrected. Restructures to the repository implementation-plan template: one recommended solution, Before/After flows on the same 4K example, and Question/Investigation/Recommendation/Input for each open item. Records the team's review decisions - new block versions as the delivery mechanism, mask_size as carried metadata, RLE wire format settled at the end of implementation - and folds in the measured evidence that settled the wire question. Adds a sequencing section covering what is built, what remains and why the Triton gate belongs with the RLE work rather than before it. The TDD sequence is now an appendix rather than a separate document, so the plan and the test strategy stay in one file. --- docs/feat/instance-seg-tradeoff-factor.md | 772 +++++++++++++++++----- 1 file changed, 593 insertions(+), 179 deletions(-) diff --git a/docs/feat/instance-seg-tradeoff-factor.md b/docs/feat/instance-seg-tradeoff-factor.md index 82660a3dd4..e60a15f4be 100644 --- a/docs/feat/instance-seg-tradeoff-factor.md +++ b/docs/feat/instance-seg-tradeoff-factor.md @@ -1,218 +1,632 @@ # Implementation plan: honour `tradeoff_factor` for instance segmentation -**Branch:** `feat/instance-seg-tradeoff-factor` (off `main` @ `23a01eb3a`) -**Status:** draft for `#discuss-inference-release` — not yet agreed +**Author:** Michał Taraszewski -Follows [`.github/implementation-plan-template.md`](../../.github/implementation-plan-template.md), required by -[`AGENTS.md`](../../AGENTS.md#internal-contributions-roboflow-team-only) for a major feature, and separately -because this touches shared infrastructure, compatibility and package dependencies. +**Related issue / PR:** _none yet — to be linked from the `#discuss-inference-release` thread_ ---- - -## 1. What problem are we solving? - -**Who.** Anyone running instance segmentation on large frames or dense scenes — real-time video especially, and -Jetson deployments above all. - -**What is missing.** `mask_decode_mode` and `tradeoff_factor` are accepted end to end (SDK entity → Workflows -block → `WorkflowsModelsProvider.run_instance_segmentation` → `InstanceSegmentationInferenceRequest`) and then -ignored. They pass untouched through -`InferenceModelsInstanceSegmentationAdapter.map_inference_kwargs` (`inference/core/models/inference_models_adapters.py:520`) -and are swallowed by `**kwargs` in each model's `post_process`. No error, no effect. Masks are always produced -at full input resolution. The legacy path (`inference/core/models/instance_segmentation_base.py:143-190`) still -implements all three modes. - -**Concrete example.** A 3840×2160 frame with 300 instances, 160×160 prototypes, on CPU: - -| stage | today (`t=1.0`) | at `t=0.25` | -|---|---|---| -| resize | 575 ms | 53 ms | -| → polygon path total (`masks2poly`) | **889 ms** | **92 ms** (9.7×) | -| → RLE path total (`rle` encode = 2531 ms) | **3107 ms** | **271 ms** (11.5×) | -| peak RSS | 3256 MB | 595 MB (5.5×) | +**Status:** draft. Supersedes `instance-seg-tradeoff-factor.md`, whose headline figures combined two +code paths that never execute together and whose recommended v1 is measurably slower than today. -Measured on Apple M1 Max, torch 2.14.1, 8 threads, fresh process per cell, warm-up 3, 15 iterations, p50. -The timed code is a verbatim copy of `post_processing.py:469-483`. +> **Reviewed by the team 2026-10-01.** The delivery mechanism is settled (new block versions); the +> RLE wire-format decision is deliberately deferred to the end of implementation. §3 records what was +> decided and what is still open. -**Note which path dominates.** On the **default** v4 path the masks are RLE, and **RLE encoding costs 4.4× the -resize**. The saving is therefore larger than a "faster resize" framing suggests, and any claim that the resize -alone is the bottleneck is wrong. - -**Edge.** At `t=1.0` this stage moves ~40 GB of memory traffic per 4K frame. On Jetson Orin NX (102 GB/s, -shared with the TRT engine) that is ~392 ms of post-processing per frame. Here the knob is not a tuning option; -it is the difference between shipping and not. - -**What should become possible.** A caller chooses mask resolution per request. Default `1.0` keeps today's -behaviour byte-for-byte. - -### The published contract this must honour +--- -Only four statements are published (the `inference.roboflow.com` reference site was collapsed to a landing page -in `cbdd286c4`, so the mkdocstrings API pages are gone): +## 1. What problem are we solving? -- `mask_decode_mode` default `accurate`. -- *"If 'accurate' the mask will be decoded using the original image size. If 'fast' the mask will be decoded - using the original mask size. 'accurate' is slower but more accurate."* — the only published statement about - output resolution, and therefore binding. -- `tradeoff_factor` default `0`, scale `0='fast'` → `1='accurate'`. +**Who.** Anyone running instance segmentation on large frames or dense scenes: real-time video, and +Jetson deployments in particular. + +**What is missing.** `mask_decode_mode` and `tradeoff_factor` are accepted end to end — SDK entity → +Workflows block → `WorkflowsModelsProvider.run_instance_segmentation` → +`InstanceSegmentationInferenceRequest` — and then ignored. They pass untouched through +`InferenceModelsInstanceSegmentationAdapter.map_inference_kwargs` +(`inference/core/models/inference_models_adapters.py:520`) and are swallowed by `**kwargs` in each +model's `post_process`. No error, no effect. Masks are always produced at full input resolution. +The legacy path (`inference/core/models/instance_segmentation_base.py:143-190`) still implements all +three modes. + +**Why it matters: this stage is almost the entire frame.** Measured on one host (M1 Max, 8 torch +threads, fp32, 3840×2160, p50): JPEG decode 59.7 ms, preprocess to 640×640 0.26 ms, JPEG encode +35.1 ms — everything that is *not* mask post-processing sums to roughly **95 ms**. Mask +post-processing at `n=300` is **hundreds of milliseconds to seconds** (tables below). At 4K with a +dense scene this stage is **~90–97% of total frame time**. That, not a stage-relative speedup, is +the argument. + +**Two things this plan does not claim.** It is not sufficient on its own — at `n=300`/4K even +`t=0.25` leaves several hundred ms in this stage. And cost is Θ(n·H·W), so **reducing +`max_detections`** (default 300, `inference_models/configuration.py:229`) is a cheaper, lossless +first move that should be evaluated alongside it. + +### The published contract + +The `inference.roboflow.com` reference site was collapsed to a landing page in `cbdd286c4`, so the +mkdocstrings API pages no longer exist. What remains published: + +- `mask_decode_mode` default `accurate`; `tradeoff_factor` default `0`, scale `0='fast'` → `1='accurate'`. +- *"If 'accurate' the mask will be decoded using the original image size. If 'fast' the mask will be + decoded using the original mask size."* — the only published statement about output resolution, + and therefore binding. - `response_mask_format` default `polygon`, `rle` opt-in. -Two gaps worth fixing alongside: the published description enumerates only `accurate` and `fast`, so **the -`tradeoff` mode this feature exists to serve is undocumented**; and the docstring at -`instance_segmentation_base.py:78` says "Defaults to 0.5" where line 33 says `0.0` — source-only, no longer -published, but visible in IDEs and to agents. +The request schema (`inference/core/entities/requests/inference.py:254-257`) publishes all three +modes. Only the legacy query parameter (`inference/core/interfaces/http/http_api.py:4738`) +enumerates just two, omitting `tradeoff` — worth correcting, but it is a doc fix, not a blocker. ## 2. How will the behaviour change? -One new float, `mask_resolution_factor ∈ [0,1]`, resolved in the adapter from the existing enum + float, and -**carried as explicit metadata** alongside the masks rather than inferred from array shape. +One float, `masks_resolution_factor ∈ [0,1]` (named to match the existing `masks_smoothing_enabled` +/ `masks_binarization_threshold` kwargs), resolved in the adapter from the existing enum, and +**carried as explicit metadata** rather than inferred from array shape. -### Before +Prior art for exactly this: DeepStream's `NvOSD_MaskParams` carries mask-local `height`/`width` +separately from `rect_params` and rescales on demand +(`_reference/deepstream_python_apps/bindings/docstrings/nvosddoc.h:278-291`; +`apps/deepstream-segmask/deepstream_segmask.py:120-126`). -Masks are resized to full image resolution, so `mask.shape[1:]` *is* the image canvas. Every downstream -consumer relies on that coincidence, and nothing scales mask coordinates into image space. +### Before — 4K frame, 300 instances, 160×160 protos ```mermaid flowchart LR - A["protos 160x160 + coefficients"] --> B["crop to boxes, at proto res"] - B --> C["resize ALL instances to full image res"] - C --> D["binarize to bool: n x H x W"] - D --> E["masks2poly / RLE encode"] - E --> F["response points
correct only because mask res == image res"] + A["protos: 300 x 160 x 160"] --> B["crop to boxes, at proto res"] + B --> C["resize every instance to 2160 x 3840"] + C --> D["bool 300 x 2160 x 3840"] + D --> E["masks2poly or RLE encode"] + E --> F["polygon coords in 2160x3840
RLE size 2160x3840"] ``` -### After +### After, at `t = 0.25` ```mermaid flowchart LR - A["protos + coefficients"] --> B["crop to boxes, at proto res"] - B --> C["resize to t-scaled target
h = mh + t * [H - mh]"] - C --> D["bool n x h x w
+ explicit mask canvas"] - D --> E["masks2poly / RLE encode
at reduced res"] - E --> F["scale coords by image/mask ratio"] - F --> G["response points in image space"] + A["protos: 300 x 160 x 160"] --> B["crop to boxes, at proto res"] + B --> C["resize to 660 x 1080"] + C --> D["bool 300 x 660 x 1080
+ mask canvas 660x1080"] + D --> E["masks2poly or RLE encode
at 660 x 1080"] + E --> F["scale coords x3.27
-> image space 2160x3840"] ``` Adapter mapping: `accurate → 1.0`, `fast → 0.0`, `tradeoff → tradeoff_factor` (validated, raising `InvalidMaskDecodeArgument` as the legacy path does). This satisfies the published contract exactly. +Default `1.0` leaves output unchanged. + +### Step F is the change. Without it every prediction at `t<1` is wrong. -**Why this fits.** The legacy ONNX path already does precisely this: it tracks `output_mask_shape` per mode and -rescales via `post_process_polygons` (`inference/core/utils/postprocess.py:449`). The `inference_models` -rewrite dropped that step. We are restoring a mechanism the codebase already had, not inventing one. +There is **no mask→image coordinate scaling anywhere** in the `inference_models` response path. The +adapter emits raw `masks2poly` output into `Point(x, y)` +(`inference_models_adapters.py:917-988`, `:1047`) while declaring the image at `original_size`. Two +sites are worse: `bitpacked_masks2poly(..., width=W)` with `W` from `original_size` (`:907`, +`core/utils/postprocess.py:73`) unpacks at the wrong stride and corrupts silently; and +`InstancesRLEMasks` declares `image_size = original_size` while encoding at +`size_after_pre_processing` (`rfdetr/common.py:383`). -### Effects on existing users +The legacy path does this correctly — it tracks `output_mask_shape` per mode and rescales via +`post_process_polygons` (`inference/core/utils/postprocess.py:449`). **The `inference_models` +rewrite dropped that step.** We are restoring a mechanism the codebase already had. -**Step F is the whole change.** Without it every prediction at `t<1` is silently wrong. A review found no -mask→image scaling anywhere in the new response path: the adapter emits raw `masks2poly` output straight into -`Point(x,y)` (`inference_models_adapters.py:917-988`, `:1047`) while declaring the image at `original_size`. -Two sites are worse — `bitpacked_masks2poly(..., width=W)` with `W` from `original_size` -(`:907`, `core/utils/postprocess.py:73`) unpacks at the wrong stride and corrupts silently, and -`InstancesRLEMasks` declares `image_size = original_size` while encoding at `size_after_pre_processing` -(`rfdetr/common.py:383`), which breaks COCO decode. +Verified consequences if reduced masks escape unscaled (4 confirmed; a wider sweep is in progress +and will be attached rather than asserted): `visualizations/mask/v1_tensor.py:277-280` raises +outright; `bounding_rect/v1.py:163-177` overwrites `xyxy` from mask contours, so corrupted boxes +reach blocks that never touch masks; `dynamic_zones/v1.py:348-352` emits shrunken zone polygons; +`sinks/dataset_upload/v1_tensor.py:946-951` persists mask-space polygons as permanent labels. +Note `sv.Detections.from_inference` already resizes mismatched RLE +(`supervision/detection/utils/internal.py:133-138`, supervision 0.30.6) but not polygons, +which are rasterised as given onto an image-size canvas (`:430`). -A blast-radius sweep found **20 silent-corruption sites and 13 hard crashes** if reduced masks escape unscaled. -Worst of each: dataset upload persists mask-space polygons as **permanent labels** -(`sinks/dataset_upload/v1_tensor.py:971`); Dynamic Zones emits shrunken zone polygons that corrupt every -downstream count (`transformations/dynamic_zones/v1.py:348`); several blocks overwrite `xyxy` from mask -contours (`bounding_rect/v1.py:163`), so corrupted boxes reach blocks that never touch masks; and Mask -Visualization raises outright (`visualizations/mask/v1_tensor.py:278`, verified). +> Supervision citations in this document are against **0.30.6**, the version this branch pins +> (`supervision>=0.30.6,<0.31.0`). The 0.29 line numbers differ; 0.30 changed `__setitem__` +> validation, `from_inference` polygon skips, PolygonZone anchors, CompactMask degeneracy and NMM +> tie-breaks, so references taken from a 0.29 checkout do not transfer. -Two useful facts: `sv.Detections.from_inference` **already** resizes mismatched RLE -(`supervision/detection/utils/internal.py:126-132`) but not polygons (`:155`); and `Detections` never validates -mask shape against an image, only cardinality — so it cannot catch this for us. +### Measurements, by path -**Default `1.0` is byte-identical.** `round(mh*(1-t) + H*t)` at `t=1.0` is exactly `H`. +Two different functions. They are not comparable and must not be combined. + +**Dense path** — `align_instance_segmentation_results` (`post_processing.py:395`), chunked, +materialises `n×H×W`. 4K, n=300, chunk 16, fp32, CPU: + +| t | resize p50 | peak RSS | +|---|---|---| +| 1.0 | 485–575 ms (3 runs) | 3256 MB | +| 0.25 | 52–65 ms | 595 MB | + +**RLE path** — `align_instance_segmentation_results_to_rle_masks` (`post_processing.py:525`), a +**per-instance generator**: no chunking, no `n×H×W` tensor, so the memory figures above do not +apply. This is the default (`v4.py:359`, `:408` hardcode `"rle"`; `GCP_SERVERLESS` forces it, +`inference_models_adapters.py:527`). Encode cost is strongly content-dependent — measured 2933 ms +for smooth blob masks vs 6262 ms for noise masks at n=300, a 2.1× spread from fixture alone. + +**Open measurement gaps, to close before implementation.** The polygon-path figure in the superseded +plan did not reproduce (`masks2poly` measured ~1877 ms against ~314 ms implied) and is withdrawn. +All numbers above are single-platform CPU; there is **no CUDA or Jetson measurement**, and the +~40 GB/frame traffic figure for Orin NX is a bandwidth model, not data. The benchmark scripts will +be linked, with fixture content stated. ### How we verify -- Bit-identical output at `t=1.0` (regression guard). -- **Polygon/RLE coordinate round-trip at `t<1`** — assert emitted coordinates land on the same image pixels as - `t=1.0` within tolerance. Shape-only tests would not catch the defect above; this is the test that matters. -- RLE invariant: decoded counts sum to the declared size, and the declared size is the encoded size. -- Non-square fixtures throughout (a square image hides axis errors); odd dims; letterbox *and* stretch; - static crop with offset; `n = 0`. -- Triton gate: RF-DETR-seg has a fused CUDA post-process that bypasses this function entirely - (`models/rfdetr/triton_postprocess.py`); `t != 1.0` must force its eager fallback. -- Benchmark sweeping `t × n`, reporting p50/p95 of post-processing in isolation, with hardware stated. -- `AP_mask` 50:95 overall **and by object size**, against `t`, with `AP_box` as control. Deliverable is the - curve; the recommended default follows from it. - -### Scope and sequencing - -Four of these are independent of the feature and should land first as separate, bit-identical commits — they -are measured, low-risk, and make the feature's own diff reviewable: - -1. `torch.gt(..., out=)` instead of `.gt_()` + assignment — removes two full-size passes; 1.07–1.51×. -2. Separable `crop_masks_to_boxes` — the four broadcast comparisons each materialise `[n,h,w]`; keeping them - at `[n,1,w]`/`[n,h,1]` is 1.9–2.2×, `torch.equal`-identical. -3. Resize directly into the static-crop canvas — removes a second full `n×H×W` bool buffer. -4. `crop_masks_to_boxes(scaling=0.25)` — one hardcoded scalar where per-axis values derived from - `masks.shape` and `inference_size` are correct. Wrong for YOLACT (138/550 = 0.2509) and for any non-square - inference size, and it ignores letterbox padding while `align_instance_segmentation_results` twenty lines - later derives the same ratio properly. Correctness, not performance — and it changes `t=1.0` output, so it - must not be bundled with the byte-identity claim. - -Then the feature: `mask_resolution_factor` + canvas metadata + coordinate scaling + RLE parity. - -Explicitly **out of scope**, as its own later design: box-resolution (compact) masks, Θ(Σ box areas), -measured 7.6–11.2× at full fidelity. The naive per-box `grid_sample` implementation is 2.7–8.8× *slower* than -today; it needs batched `roi_align`. Worth doing, not here. - -## 3. Uncertainties and decisions needing input - -**Q1 — Does `tradeoff_factor`'s default of `0.0` stand once the parameter is honoured?** -*Investigation:* `DEFAULT_TRADEOFF_FACTOR = 0.0` (`instance_segmentation_base.py:33`), all 8 block versions -declare `default=0.0`, published OpenAPI says `0`. So `mask_decode_mode="tradeoff"` with an untouched factor -means `t=0.0` — identical to `fast`, maximum degradation. Any existing workflow that set `tradeoff` and left -the factor alone (the natural case, since it had no effect) lands there on upgrade. -*Recommendation:* honour the parameter in a **new block version** rather than in place, so no existing workflow -changes behaviour on upgrade. Separately make `tradeoff` with a default factor an explicit error or clamp it, -rather than silently meaning `fast`. -*Input needed:* is a new block version acceptable, or is in-place with a migration note preferred? **Blocking.** - -**Q2 — Where does the mask canvas live?** -*Investigation:* `InstancesRLEMasks.image_size` is read as "the image canvas" at ~15 sites; for dense masks, -`mask.shape[1:]` *is* the canvas definition at `bounding_rect/v1_tensor.py:211`, -`dataset_upload/v1_tensor.py:990`, `stabilize_detections/v1_tensor.py:448`. Changing either meaning silently -is what produces the 20 corruption sites. -*Recommendation:* keep `image_size` meaning the image and add a sibling `mask_size`; add an explicit -mask-canvas field to `InstanceDetections`. Both are public dataclasses in a separately-versioned package → -changelog and consumer sweep. -*Input needed:* agreement on the field names and on accepting the `inference_models` API addition. **Blocking.** - -**Q3 — Does the RLE wire format change, or do we upscale before encoding?** -*Investigation:* `rle["size"] ≠ image` is a breaking change for any external client that is not -`supervision ≥` the version carrying `internal.py:126`. v4 hardcodes `response_mask_format="rle"` -(`v4.py:357`, `:468`) and `GCP_SERVERLESS` forces it (`inference_models_adapters.py:527`), so this is the -default path, not an edge case. -*Recommendation:* for v1 of the feature, upscale before RLE encoding — correct on the wire, and the compute -saving is still taken. Revisit when compact masks land. -*Input needed:* accept the reduced saving on the RLE path, or version the response? **Blocking.** - -**Q4 — Should `mask_chunk_size` become a byte budget?** -*Investigation:* the optimal chunk size moves *inversely* to `t`. At 4K/`n=300`/`t=1.0` the optimum is 32; at -`t=0.1` it is unchunked, where the fixed 16 is 1.43× off (1.70× at 1080p). Per-chunk overhead is ~0.25 ms, i.e. -22% of the whole budget at `t=0.1` versus <1% at `t=1.0`. Chunking bounds the float temporary only; it never -shrinks the bool output, so it is orthogonal to this feature, not a substitute. -*Recommendation:* express it as `chunk = max(1, min(n, BUDGET_BYTES // (h*w*4)))` with a ~512 MiB budget, -keeping the env var as the budget. Reproduces today's 16 at 4K/`t=1.0` and the measured optimum at low `t`. -*Input needed:* in scope here, or a follow-up? Non-blocking, but leaves 1.2–1.7× unclaimed at exactly the -settings this feature enables. - -**Q5 — Recommended value and documentation.** -*Investigation:* the knee is at `t ≈ 0.25–0.3`, capturing ~90% of the achievable latency saving and 5.5× of the -memory. Below 0.15 the gain flattens while resolution keeps falling. -*Recommendation:* keep the default at `1.0`; document `0.25` as the recommended setting for real-time and -edge, and `t < 0.15` as diagnostic only. Also document `tradeoff` as a legal `mask_decode_mode` — it currently -is not — and fix the `0.5` docstring. -*Input needed:* confirmation once the `AP_mask` curve exists. Non-blocking. - -**Closed by investigation.** Does `fast` need a separate code path? No — `t=0.0` *is* legacy +`align_instance_segmentation_results` and `crop_masks_to_boxes` currently have **no unit tests** +(`inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py` covers neither), +so a characterization test — fixed-seed protos → golden mask tensor and golden polygons — is +commit zero, before any behaviour claim can be checked in CI. + +- Bit-identical output at `t=1.0`. +- **Polygon/RLE coordinate round-trip at `t<1`**: emitted coordinates land on the same image pixels + as `t=1.0`. Tolerance must scale with `1/t`, since unpad rounding error is amplified as `t` falls. + Shape-only assertions would not catch the defect above; this is the test that matters. +- RLE invariant: decoded counts sum to the declared size; declared size is the encoded size. +- Non-square fixtures throughout (square images hide axis errors); odd dims; letterbox *and* + stretch; static crop with offset; `n = 0`. +- Triton gate as a pure-Python unit test on `_unsupported_triton_postprocess_reason`, so it is not a + GPU-only dead test in x86 CI. +- `AP_mask` 50:95 overall **and by object size**, against `t`, with `AP_box` as control. + +Layers: `inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py`; +`tests/inference/models_predictions_tests`; `tests/workflows/unit_tests/.../instance_segmentation/` +for any new block version **and its `_tensor` sibling**. + +### Implementation surface + +Not one function. The alignment logic is duplicated across two functions, reached from six model +families on both the dense and RLE routes — `models/{yolov5,yolov7,yolov8,yolo26,yolact}/common.py` +× {dense, rle} = 10 call sites, plus `models/rfdetr/common.py:369` and +`models/rfdetr/triton_postprocess.py` — plus the new kwarg threaded through every +`*_instance_segmentation_{onnx,trt,torch_script}.py`. Roughly 12 sites beyond the two core +functions. + +### Out of scope, as its own plan + +Four independent performance and correctness changes, measured and bit-identical except the last, +belong in a separate PR and are deliberately not bundled here: `torch.gt(out=)` in place of +`.gt_()` + assignment; separable `crop_masks_to_boxes`; resizing directly into the static-crop +canvas; and deriving `crop_masks_to_boxes(scaling=0.25)` per-axis (wrong whenever +`mask_w/inference_w ≠ 0.25`, e.g. YOLACT 138/550 — and the only one of the four that changes +`t=1.0` output). Also out of scope: batching the per-instance RLE encode, measured 1.4× at `t=1.0` +and 2.5× at `t=0.25`, lossless and with no API change. + +## 2b. Sequencing from here + +### What is already done + +The **dense / polygon path works end to end**: request → `map_inference_kwargs` collapses the enum +onto one factor → `align_instance_segmentation_results` interpolates the resize target → +`mask_size` travels with the prediction → polygons are scaled back into image space before being +reported. Coverage on these functions went from 31 tests to 77 in `inference_models` plus 37 in the +adapter, including characterization tests that pin geometry the previous suite structurally could +not detect. + +Everything that remains is gated on the deferred RLE wire-format decision (Q3). + +### The decision is narrower than three options + +Measurement has already eliminated one, and the remaining two are not equally viable: + +| option | outcome | +|---|---| +| upscale before encode | 814 ms against today's 485 ms, encode unchanged. **A net regression.** | +| dense-only in v1 | Safe, but `v4.py` hardcodes `response_mask_format="rle"` and `GCP_SERVERLESS` forces it, so the feature is **unreachable for most traffic** — it would work only with `enforce_dense_masks_in_inference_models=True`. | +| reduced RLE `size` | The only option that delivers the saving on the path that is actually used. | + +So the question to put to the team is not "which of three" but: **are we willing to change the RLE +`size` field, given that otherwise the feature is inert for most users?** + +Client exposure is narrower than it first appears. `sv.Detections.from_inference` already resizes a +mismatched RLE (`supervision/detection/utils/internal.py:133-138`, 0.30.6), so supervision-based +consumers are unaffected. Direct `pycocotools` integrators are the exposed group, which is what the +O3 experiment below quantifies. + +### Phase 6, in dependency order + +1. Thread `masks_resolution_factor` into `align_instance_segmentation_results_to_rle_masks`, + mirroring the dense path. The per-instance generator has no chunking, so it is the simpler of + the two. +2. `to_coco_rle_masks()` emits `mask_size` as the COCO `size`. One line, and the point of no + return — everything before it is reversible. +3. **Teach the Triton kernel the factor rather than gating it.** `triton_postprocess.py` already + parameterises its resize tables by `output_size` (`:273-285`), so passing a reduced size through + is close to free. Gating instead leaves the production GPU path — the entire edge argument — + unimproved, and emits a `RuntimeWarning` per image. Gate only as a fallback if this overruns. +4. New block version (`vN` **and** `vN_tensor`), per the team's standing mechanism. +5. Version skew: an absent `mask_size` in a response means an old server, so default to the input + size. Treat the factor as a **performance hint, never an error**, so a new block version still + works against an older backend. + +Note the Triton gate deliberately belongs here and not earlier: until step 1 lands, the fused and +eager paths both ignore the factor and behave identically, so a gate added now would guard against +a divergence that does not yet exist and its test would assert a fallback that changes nothing. + +### Unblocked work that should not wait + +A separate PR, mergeable today, independent of the decision: + +- `torch.gt(..., out=)` in place of `.gt_()` plus assignment — 1.07–1.51×, bit-identical. +- Separable `crop_masks_to_boxes` — 1.9–2.2×, `torch.equal`-identical. +- Resize directly into the static-crop canvas — removes a full `n×H×W` bool buffer. +- **Batched RLE encode** — 1.4× at `t=1.0`, 2.5× at `t=0.25`, lossless, no API change, and it + removes roughly 300 device synchronisations per frame. This is the standout: it attacks the + 2531 ms encode directly and helps whichever way Q3 is decided. +- `crop_masks_to_boxes(scaling=0.25)` derived per axis — correctness; changes `t=1.0` output, so it + needs its own commit and must not be bundled with any bit-identity claim. + +### Recommended order of work + +**Before the decision.** Run the O3 decode experiment, then send the team the plan with that +evidence and the question framed as above. In parallel, open the refactors PR. + +**After the decision.** Phase 6 as sequenced; steps 1–2 are roughly a day, step 3 longer. + +**Independently.** The `AP_mask`-against-`t` curve and one CUDA measurement. Neither blocks +implementation; both block the *recommendation*, and without them there is no defensible default +to ship. + +## 3. Decisions taken, and what remains + +### Settled by team review, 2026-10-01 + +Question numbers below are those of the reviewed draft, so the thread stays traceable. + +**(Q1) New block versions are the delivery mechanism.** Changes of this kind have always shipped as +new block versions to minimise breakage — *even when the change is a fix to an earlier bug* — and +code duplication across versions is explicitly acceptable. This removes the main compatibility +objection: a new version means no existing pipeline changes under upgrade. Both the `vN` and +`vN_tensor` siblings are in scope. + +**(Q2) `mask_size` as metadata carried with the prediction: accepted in principle.** Two +sub-decisions remain, below. + +**(Q2a) The carrier is an explicit field, not a metadata key.** Decided 2026-10-01. + +``` +InstancesRLEMasks: image_size: tuple[int, int] # unchanged - the image + mask_size: tuple[int, int] = image_size # new, defaults to today's meaning + +InstanceDetections: mask_size: tuple[int, int] = mask.shape[1:] # new, defaults to today's meaning +``` + +*Why.* The default is what makes this cheap: every existing construction stays valid, every reader +that ignores `mask_size` gets today's behaviour, and the changelog entry is an addition rather than +a change. The public-API cost is paid once, in a package that is versioned separately anyway. + +*Why not a metadata key.* The failure mode this whole plan exists to prevent is silent corruption, +and an unvalidated metadata key is exactly the thing a generic transform drops without anyone +noticing. A sibling field sits next to `image_size`, where all ~15 readers already look; a key sits +somewhere they do not. And `InstanceDetections` has no metadata carrier for dense masks at all, so +the key route means inventing one — strictly more work than adding the field. + +*Corroborating signal.* The team's own (Q1) answer accepts duplicated block versions to avoid +breakage. That is a preference for explicit and slightly verbose over clever and implicit. + +**(Q3) The RLE wire-format decision is deferred to the end of implementation**, for wider team +discussion. The reasoning, which changes this plan's shape: *with new block versions this is not +breaking for Workflows at all.* The breaking surface is narrower than the reviewed draft assumed — +it is the **model endpoints**, where a direct integrator would suddenly see mask-scaling parameters +respected. That is a bug fix, and it is breaking, and those are not in conflict. Implementation +should proceed without settling it. + +**(Q4) The chunk byte budget is a follow-up, not part of this work.** It addresses an extremum of +the resolution range; it can land later purely as serving-latency optimisation. + +**(Q5) The accuracy/latency curve may be worth writing up publicly** — a blog post rather than a +docs change. + +### Still open + +**O2 — Remote execution across mixed backend versions.** *(raised in review; not in the reviewed draft)* +*Question:* a client may talk to either a new server that returns `mask_size` or an old one that does +not. What does the client do? +*Investigation:* the response shape itself is the version signal — presence of the new field +distinguishes a new server from an old one, with no version negotiation needed. +*Recommendation:* absent field ⇒ assume the old contract and default `mask_size` to the input size; +present ⇒ use the explicit value. This must hold for both remote Workflows execution and the SDK. +*Input needed:* confirmation that response-shape sniffing is acceptable rather than an explicit +version field. **Blocking for the remote path.** + +**O3 — Evidence for the deferred Q3 decision: measured, and it inverts the risk.** +The team asked how badly RLE decoding breaks under a reduced `size`. Measured on +supervision 0.30.6 and pycocotools: + +| case | result | +|---|---| +| `pycocotools.decode` on an **honest** reduced `size` | decodes cleanly to the reduced canvas. No exception, correct mask. | +| `pycocotools.decode` on a **lying** `size` (reduced counts, image size declared) | decodes **silently wrong**. No exception. | +| `sv.Detections.from_inference` on an honest reduced `size` | **auto-resizes to the image.** Mask arrives at `(1, 400, 600)` as if nothing happened. | + +The conclusion is the opposite of the intuition the question was built on. Emitting an honest +reduced `size` is the **safe** option: supervision consumers are transparently unaffected, and a +direct `pycocotools` consumer gets a correct mask on a smaller canvas, which is exactly what the +published contract already promises for `fast` ("decoded using the original mask size"). + +The dangerous case is the one **currently in the code**: `InstancesRLEMasks` declares +`image_size = original_size` while encoding at `size_after_pre_processing`. The moment those +differ, that is the lying case — silent corruption with no exception. Honesty about the size is +therefore not just safe, it is the fix. + +*Caveat on fidelity:* the round-trip IoU of 1.0 in this experiment used an axis-aligned rectangle +on an exact 4× grid, so it understates boundary loss. A real contour will lose fidelity; that is +what the `AP_mask` curve measures, and it is a separate question from whether decoding breaks. + +**O4 — The GPU path: gate the fused kernel, or teach it `t`?** +*Investigation:* `models/rfdetr/triton_postprocess.py` already implements a **lossless** version of +this optimisation — *"asks Triton to interpolate only the active mask region and emit sparse RLE run +records directly"* — and already parameterises resize tables by `output_size` (`:273-285`). Forcing +its eager fallback leaves the production GPU path unimproved and emits a `RuntimeWarning` **per +image** (`:253-258`), i.e. per-frame log spam. The fused path already bails on +`padding_unsupported`, `static_crop_unsupported`, `resize_metadata_unsupported` and `>4096 px` +(`:921-983`), so letterboxed 4K, static crops and Jetson-without-triton already fall through to the +slow path — which is the real justification for this feature and belongs in §1. +*Recommendation:* pass the reduced `output_height/width` into the kernel. If too large for v1, add +`t != 1.0` to `_unsupported_triton_postprocess_reason` so the fallback is quiet, and state plainly +that the GPU fast path is unimproved in v1. +*Input needed:* scope call. Non-blocking for the dense path; blocking for any Jetson claim. + +**O5 — Legacy-path parity.** After this change `mask_decode_mode="tradeoff"` returns transposed +masks under `USE_INFERENCE_MODELS=False` — `process_mask_tradeoff` passes `(h, w)` where +`cv2.resize` wants `(width, height)` (`core/utils/postprocess.py:353`; verified by execution, +1920×1080 at `t=1.0` returns `(n, 1920, 1080)`) — and correct masks under the default. A +flag-dependent geometry divergence. *Recommendation:* fix the transpose in the same PR; it is four +lines. Non-blocking. + +**O6 — Stream behaviour.** `tradeoff_factor` is bindable to `$inputs.*`, so `t` can change between +frames of one session, while `stabilize_detections/v1_tensor.py:447-451` and tracker caches derive +empty-mask shapes from `template.mask.shape[1:]`. *Recommendation:* treat `t` as session-fixed and +reject mid-stream changes. Non-blocking. + +### Unmeasured — evidence gaps + +Neither of these blocks implementation. Both block the plan's *justification*, and the first could +reduce its scope substantially. + +**E1 — `max_detections` has not been evaluated as the cheaper alternative.** +Mask post-processing is Θ(n·H·W). This plan attacks `H·W` and leaves `n` untouched, yet `n` is a +parameter that already exists: `INFERENCE_MODELS_DEFAULT_MAX_DETECTIONS` defaults to **300** +(`inference_models/configuration.py:229`). Halving it halves the cost, is lossless on the top-150 +predictions, and needs no new parameter, no carried metadata, no wire-format change and no +coordinate rescaling — none of the blast radius this plan spends most of its length on. +*Action:* measure `AP_mask` and p50/p95 against `max_detections ∈ {300, 150, 100}` on the same fixed +eval set as the `t` curve, and publish the two curves together. +*Why it matters:* if reducing `n` delivers a large fraction of the win, the honest recommendation +may be guidance plus a default change rather than this feature. That is worth knowing before more is +built, not after. + +**E2 — There is no CUDA or Jetson measurement anywhere in this plan.** +Every figure is single-platform CPU (Apple M1 Max, 8 threads, fp32). The ~40 GB-per-4K-frame traffic +figure and the derived ~392 ms on Orin NX are a **bandwidth model, not data** — and that estimate is +simultaneously the strongest claim in §1 and the weakest evidence in the document. +Two CUDA-specific risks are also unverified: a per-request factor makes the post-process output +shape request-dependent, which may fragment the caching allocator on an 8 GB Orin and defeats the +pinned staging buffer that `triton_postprocess.py:191-218` keys on source shape. The likely +mitigation — quantise `t` to a small set, e.g. steps of 0.125, so the shape space stays bounded — is +untested. +*Action:* one end-to-end frame-time profile on an Orin NX, or failing that one discrete NVIDIA GPU, +measuring the function that actually runs on that path rather than its dense sibling. +*Why it matters:* "Jetson deployments above all" currently rests on arithmetic. One run converts the +plan's weakest evidence into its best. + +### Follow-ups, not decisions + +Recommend a default `t` once the `AP_mask` curve exists; the chunk byte budget (dense path only — +the RLE generator has no chunking); batching the per-instance RLE encode (measured 1.4× at `t=1.0`, +2.5× at `t=0.25`, lossless, no API change); fixing the `Defaults to 0.5` docstring at +`instance_segmentation_base.py:78` against `DEFAULT_TRADEOFF_FACTOR = 0.0` on line 33; adding +`tradeoff` to the legacy query-param description at `http_api.py:4738`. + +**Closed by investigation.** `fast` needs no separate code path: `t=0.0` *is* legacy `process_mask_fast` semantics, because the crop already happens at proto resolution before `align_instance_segmentation_results` (`models/yolov8/common.py:32`) and `t=0` skips the resize. -**Not porting `process_mask_tradeoff`.** It passes `(h, w)` where `cv2.resize` wants `dsize=(width, height)` -(`core/utils/postprocess.py:353`), so it transposes masks on non-square images — verified by execution, -1920×1080 at `t=1.0` returns `(n, 1920, 1080)` — and `scale_bboxes` then derives crop scaling from the swapped -dims. `process_mask_fast` has a related defect on letterboxed input (`:394`). Invisible on square images, -which is why it survived. +--- + +## Appendix — TDD sequence + +Each phase is red → green → commit. + +### Existing coverage, measured not assumed + +`inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py` (28 tests) **does** +cover `align_instance_segmentation_results`, via `test_chunked_resize_matches_monolithic` (:502) and +`..._with_static_crop_canvas`. But both are **invariance** tests — they assert chunked output equals +monolithic output. Change the resize target and both sides move together, so **both still pass**. + +Nothing pins output *geometry*: not the shape relative to the image, not coordinate correctness. +`crop_masks_to_boxes` is not imported and has no tests at all. + +So phase 0 is not optional ceremony — it is the only thing that would catch a geometry regression. + +--- + +### Phase 0 — characterization (GREEN on current code) + +Pin today's behaviour before touching anything. These must pass unchanged at `t=1.0` forever after. + +```python +class TestCurrentMaskGeometry: + """Characterization. These encode today's contract; they must not change at t=1.0.""" + + def test_masks_are_image_sized(self) -> None: + _, masks = self._run() + assert masks.shape[1:] == (self.size_after_pre_processing.height, + self.size_after_pre_processing.width) + + def test_masks_are_bool(self) -> None: + _, masks = self._run() + assert masks.dtype == torch.bool + + def test_golden_output_for_fixed_seed(self) -> None: + # seeded protos -> exact tensor. Any geometry change breaks this loudly. + torch.manual_seed(0) + _, masks = self._run() + assert masks.sum().item() == GOLDEN_SET_PIXELS # fill from first run + assert masks[0, :8, :8].tolist() == GOLDEN_CORNER # fill from first run + + def test_non_square_image_preserves_aspect(self) -> None: + # the axis-swap guard. Square fixtures hide transposes. + _, masks = self._run(image=(1080, 1920)) + assert masks.shape[1] == 1080 and masks.shape[2] == 1920 +``` + +And the first tests `crop_masks_to_boxes` has ever had: + +```python +class TestCropMasksToBoxes: + def test_zeroes_outside_box(self) -> None: ... + def test_scaling_default_is_proto_stride(self) -> None: ... + def test_non_square_inference_size(self) -> None: + # documents that a single scalar `scaling` is wrong when mask_w/inference_w != 0.25 + ... +``` + +**Commit 0.** No production change. + +### Phase 1 — RED: the coordinate-scaling gap + +The prerequisite from §2 of the plan. This test fails today and proves the gap exists. + +```python +def test_polygon_coords_are_image_space_when_mask_is_smaller(self) -> None: + # given: a mask deliberately smaller than the declared image + masks = make_masks(n=2, h=540, w=960) # half of 1080x1920 + meta = make_metadata(original_size=(1080, 1920)) + + # when + response = build_instance_segmentation_response(masks, meta) + + # then: coordinates must span the IMAGE, not the mask + xs = [p.x for pred in response.predictions for p in pred.points] + assert max(xs) > 960, "polygon coords left in mask space" +``` + +RED today — `inference_models_adapters.py:917` emits raw `masks2poly` output. Green once scaling is +applied at the mask→polygon boundary, mirroring legacy `post_process_polygons` +(`inference/core/utils/postprocess.py:449`). + +**Commit 1.** Coordinate scaling only. No new parameter yet — and it is a latent-correctness fix on +its own merits. + +### Phase 2 — RED: the parameter + +```python +@pytest.mark.parametrize("t", [0.0, 0.1, 0.25, 0.5, 1.0]) +def test_resolution_factor_interpolates_shape(self, t: float) -> None: + _, masks = self._run(masks_resolution_factor=t) # TypeError today -> RED + mh, mw = PROTO_UNPADDED + assert masks.shape[1] == max(1, round(mh * (1 - t) + 1080 * t)) + assert masks.shape[2] == max(1, round(mw * (1 - t) + 1920 * t)) + +def test_t1_is_bit_identical_to_baseline(self) -> None: + _, ref = self._run() # no kwarg + _, out = self._run(masks_resolution_factor=1.0) + assert torch.equal(out, ref) + +def test_t0_is_proto_resolution(self) -> None: + _, masks = self._run(masks_resolution_factor=0.0) + assert masks.shape[1:] == PROTO_UNPADDED + +@pytest.mark.parametrize("t", [0.1, 0.25, 0.5]) +def test_coords_round_trip_against_t1(self, t: float) -> None: + ref = polygons_from(self._run()) + out = polygons_from(self._run(masks_resolution_factor=t)) + # unpad rounding error is amplified as t falls, so tolerance scales with 1/t + assert_polygons_close(out, ref, tol_px=2.0 / t) + +def test_zero_instances(self) -> None: + _, masks = self._run(n=0, masks_resolution_factor=0.25) + assert masks.shape[0] == 0 + +@pytest.mark.parametrize("mode", ["letterbox", "stretch"]) +def test_both_resize_modes(self, mode: str) -> None: ... + +def test_static_crop_offset_is_scaled(self) -> None: ... +``` + +The `t=1.0` bit-identity test is the safety net for every later phase. + +**Commit 2.** Dense path only. + +### Phase 3 — RED: `mask_size` metadata (plan O1) + +```python +def test_mask_size_defaults_to_image_size(self) -> None: + # back-compat: existing construction sites must not change + m = InstancesRLEMasks.from_coco_rle_masks(image_size=(1080, 1920), masks=[...]) + assert m.mask_size == (1080, 1920) + +def test_mask_size_reflects_reduced_resolution(self) -> None: + dets = self._run_full_pipeline(masks_resolution_factor=0.25) + assert dets.mask_size == (270, 480) + assert dets.image_size == (1080, 1920) # unchanged meaning +``` + +**Commit 3.** + +### Phase 4 — RED: adapter mapping + +```python +@pytest.mark.parametrize("mode,expected", [("accurate", 1.0), ("fast", 0.0)]) +def test_enum_maps_to_factor(self, mode: str, expected: float) -> None: ... + +def test_tradeoff_reads_the_factor(self) -> None: ... + +@pytest.mark.parametrize("bad", [-0.1, 1.1]) +def test_out_of_range_raises(self, bad: float) -> None: + with pytest.raises(InvalidMaskDecodeArgument): + ... + +def test_source_keys_do_not_leak_to_model_call(self) -> None: + # map_inference_kwargs must consume both, not forward them + mapped = adapter.map_inference_kwargs({"mask_decode_mode": "fast", "tradeoff_factor": 0.0}) + assert "mask_decode_mode" not in mapped and "tradeoff_factor" not in mapped + assert mapped["masks_resolution_factor"] == 0.0 +``` + +**Commit 4.** + +### Phase 5 — RED: version skew (plan O2) and the Triton gate + +```python +def test_absent_mask_size_defaults_to_input_size(self) -> None: + # old server: field missing -> client assumes today's contract + dets = parse_response(response_without_mask_size()) + assert dets.mask_size == dets.image_size + +def test_unhonoured_hint_does_not_raise(self, caplog) -> None: + # old server ignores the parameter: correct output, just slower + dets = run_remote(masks_resolution_factor=0.25, server=OldServerStub()) + assert dets.mask_size == dets.image_size + assert "not honoured" in caplog.text # logged once, never raised + + +def test_reduced_resolution_forces_eager_fallback(self) -> None: + # pure-python: no GPU, so it runs in x86 CI instead of being a dead GPU-only test + reason = _unsupported_triton_postprocess_reason(..., masks_resolution_factor=0.25) + assert reason is not None +``` + +**Commit 5.** + +### Phase 6 — RLE parity + +Blocked on the deferred wire-format decision. Until then the RLE path keeps `t=1.0` and phase 0's +characterization tests guard it. The evidence experiment (plan O3) belongs here: + +```python +@pytest.mark.parametrize("decoder", ["pycocotools", "supervision"]) +def test_reduced_size_rle_decode_behaviour(self, decoder: str) -> None: + # not an assertion of correctness - a recorded observation of the failure mode, + # to inform the deferred decision: raises? truncates silently? resizes cleanly? + ... +``` + +### Running + +```bash +cd inference_models && python -m pytest tests/unit_tests/models/common/roboflow/test_post_processing.py -v +python -m pytest tests/inference/models_predictions_tests +python -m pytest tests/workflows/unit_tests/ +``` + +### Order and why + +Phase 0 before everything, because the existing invariance tests would not catch a geometry +regression. Phase 1 before phase 2, because coordinate scaling is a correctness fix that stands +alone and must be in place before any reduced mask can exist. Phase 6 last, because it is the only +part gated on a decision the team deferred. From 9804b121f4a980d92bd39693913a369f08b9557f Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Fri, 2 Oct 2026 17:01:42 +0200 Subject: [PATCH 12/41] Add instance segmentation block @v5 that honours mask resolution mask_decode_mode and tradeoff_factor are declared on every version of the block, and were forwarded to the server, which discarded them. Now that the server honours them, those versions would silently change the masks an existing workflow produces - a v3 step set to "fast" would start returning model-resolution masks on upgrade. Versions up to v4 therefore pin the outbound request to accurate/1.0, preserving the behaviour they have always had. Their manifests keep both fields, so existing workflow JSON still validates, but the fields are now deliberately inert rather than accidentally so. @v5 is the version where setting them does what their description says. v5.py and v5_tensor.py differ from their v4 counterparts in four places: the manifest version, the type literal, the class name, and the two outbound calls. The versioning mechanism is file copy rather than inheritance, so the two siblings share a class name and are selected by ENABLE_TENSOR_DATA_REPRESENTATION at import time. Tests cover the split - legacy versions pin, v5 forwards - and assert manifest parity between each version and its tensor-native sibling, which is held by convention today and nothing else. Also fixes three rfdetr backends where masks_resolution_factor was forwarded from post_process but declared on from_pretrained. flake8 F821 caught it; the TensorRT paths are not exercised by CPU tests. Tests: python3 -m pytest tests/unit_tests -q (inference_models) 1942 passed python3 -m pytest tests/workflows/unit_tests -q 1265 passed uvx flake8 --select=E9,F63,F7,F82 clean --- .../rfdetr_instance_segmentation_onnx.py | 2 +- .../rfdetr_instance_segmentation_pytorch.py | 2 +- .../rfdetr_instance_segmentation_trt.py | 2 +- .../test_v5_mask_resolution.py | 127 +++ .../roboflow_workflows/core_steps/loader.py | 10 + .../roboflow/instance_segmentation/v1.py | 12 +- .../instance_segmentation/v1_tensor.py | 6 +- .../roboflow/instance_segmentation/v2.py | 12 +- .../instance_segmentation/v2_tensor.py | 6 +- .../roboflow/instance_segmentation/v3.py | 12 +- .../instance_segmentation/v3_tensor.py | 6 +- .../roboflow/instance_segmentation/v4.py | 12 +- .../instance_segmentation/v4_tensor.py | 6 +- .../roboflow/instance_segmentation/v5.py | 456 +++++++++++ .../instance_segmentation/v5_tensor.py | 744 ++++++++++++++++++ 15 files changed, 1388 insertions(+), 27 deletions(-) create mode 100644 tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_v5_mask_resolution.py create mode 100644 workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5.py create mode 100644 workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py diff --git a/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_onnx.py b/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_onnx.py index a8952e5c1f..42b4020dbe 100644 --- a/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_onnx.py +++ b/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_onnx.py @@ -83,7 +83,6 @@ def from_pretrained( device: torch.device = DEFAULT_DEVICE, rf_detr_max_input_resolution: Optional[Union[int, Tuple[int, int]]] = None, recommended_parameters: Optional[RecommendedParameters] = None, - masks_resolution_factor: float = 1.0, **kwargs, ) -> "RFDetrForInstanceSegmentationOnnx": if onnx_execution_providers is None: @@ -238,6 +237,7 @@ def post_process( confidence: Confidence = "default", mask_format: InstanceSegmentationMaskFormat = "dense", max_detections: Optional[int] = INFERENCE_MODELS_RFDETR_DEFAULT_MAX_DETECTIONS, + masks_resolution_factor: float = 1.0, **kwargs, ) -> List[InstanceDetections]: if mask_format not in self.supported_mask_formats: diff --git a/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_pytorch.py b/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_pytorch.py index 5966105f01..717f0da994 100644 --- a/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_pytorch.py +++ b/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_pytorch.py @@ -99,7 +99,6 @@ def from_pretrained( resolution: Optional[int] = None, rf_detr_max_input_resolution: Optional[Union[int, Tuple[int, int]]] = None, recommended_parameters: Optional[RecommendedParameters] = None, - masks_resolution_factor: float = 1.0, **kwargs, ) -> "RFDetrForInstanceSegmentationTorch": if os.path.isfile(model_name_or_path): @@ -431,6 +430,7 @@ def post_process( confidence: Confidence = "default", mask_format: InstanceSegmentationMaskFormat = "dense", max_detections: Optional[int] = INFERENCE_MODELS_RFDETR_DEFAULT_MAX_DETECTIONS, + masks_resolution_factor: float = 1.0, **kwargs, ) -> List[InstanceDetections]: if mask_format not in self.supported_mask_formats: diff --git a/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_trt.py b/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_trt.py index 0d8d4cf18f..1aef02a918 100644 --- a/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_trt.py +++ b/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_trt.py @@ -117,7 +117,6 @@ def from_pretrained( default_trt_cuda_graph_cache_size: int = 8, rf_detr_max_input_resolution: Optional[Union[int, Tuple[int, int]]] = None, recommended_parameters: Optional[RecommendedParameters] = None, - masks_resolution_factor: float = 1.0, **kwargs, ) -> "RFDetrForInstanceSegmentationTRT": if device.type != "cuda": @@ -427,6 +426,7 @@ def post_process( confidence: Confidence = "default", mask_format: InstanceSegmentationMaskFormat = "dense", max_detections: Optional[int] = INFERENCE_MODELS_RFDETR_DEFAULT_MAX_DETECTIONS, + masks_resolution_factor: float = 1.0, **kwargs, ) -> List[InstanceDetections]: if mask_format not in self.supported_mask_formats: diff --git a/tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_v5_mask_resolution.py b/tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_v5_mask_resolution.py new file mode 100644 index 0000000000..fd8ad59b7a --- /dev/null +++ b/tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_v5_mask_resolution.py @@ -0,0 +1,127 @@ +"""Mask-resolution behaviour of the instance-segmentation block versions. + +`mask_decode_mode` and `tradeoff_factor` are declared on every version's +manifest but were discarded by the server until recently. Now that they are +honoured, forwarding them from an older version would silently change the masks +an existing workflow produces, so only `@v5` forwards them. These tests pin +that split, and the manifest parity between a version and its tensor-native +sibling. +""" + +import re +from pathlib import Path + +import pytest + + +def _blocks_dir() -> Path: + """Locate the block implementations relative to this test file. + + Returns: + Directory holding the instance-segmentation block versions. + """ + for parent in Path(__file__).resolve().parents: + candidate = ( + parent + / "workflows" + / "roboflow_workflows" + / "core_steps" + / "models" + / "roboflow" + / "instance_segmentation" + ) + if candidate.is_dir(): + return candidate + raise RuntimeError("could not locate the instance-segmentation blocks") + + +BLOCKS_DIR = _blocks_dir() + +PINNED_VERSIONS = ["v1", "v2", "v3", "v4"] +FORWARDING_VERSIONS = ["v5"] +ALL_VERSIONS = PINNED_VERSIONS + FORWARDING_VERSIONS + + +def _source(version: str, tensor: bool) -> str: + """Read a block implementation's source. + + Args: + version: Block version directory stem, such as ``"v5"``. + tensor: Whether to read the tensor-native sibling. + + Returns: + File contents. + """ + suffix = "_tensor" if tensor else "" + + return (BLOCKS_DIR / f"{version}{suffix}.py").read_text() + + +def _manifest_fields(source: str) -> list: + """Collect manifest field names declared on a block. + + Args: + source: Block implementation source. + + Returns: + Field names in declaration order. + """ + start = source.index("class BlockManifest(") + fields = [] + for line in source[start:].split("\n"): + if line.startswith("class ") and "BlockManifest" not in line: + break + match = re.match(r" ([a-z_]+): ", line) + if match: + fields.append(match.group(1)) + + return fields + + +@pytest.mark.parametrize("version", ALL_VERSIONS) +@pytest.mark.parametrize("tensor", [False, True]) +def test_version_declares_its_own_identifier(version: str, tensor: bool) -> None: + # given + source = _source(version, tensor) + + # then + assert f'"version": "{version}"' in source + assert f"instance_segmentation_model@{version}" in source + assert f"RoboflowInstanceSegmentationModelBlock{version.upper()}" in source + + +@pytest.mark.parametrize("version", ALL_VERSIONS) +def test_manifest_parity_with_tensor_sibling(version: str) -> None: + # given + # the two files are independent implementations selected by an env var, so + # a field present in one and not the other is invisible until runtime + plain = _manifest_fields(_source(version, tensor=False)) + tensor = _manifest_fields(_source(version, tensor=True)) + + # then + assert plain == tensor + + +@pytest.mark.parametrize("version", PINNED_VERSIONS) +@pytest.mark.parametrize("tensor", [False, True]) +def test_legacy_versions_pin_full_resolution(version: str, tensor: bool) -> None: + # given + # these versions must keep requesting full-resolution masks whatever the + # caller set, because the fields never had an effect when they shipped + source = _source(version, tensor) + + # then + assert 'mask_decode_mode="accurate",' in source + assert "tradeoff_factor=1.0," in source + + +@pytest.mark.parametrize("version", FORWARDING_VERSIONS) +@pytest.mark.parametrize("tensor", [False, True]) +def test_v5_forwards_what_the_caller_set(version: str, tensor: bool) -> None: + # given + source = _source(version, tensor) + + # then + assert "mask_decode_mode=mask_decode_mode," in source + assert "tradeoff_factor=tradeoff_factor," in source + assert 'mask_decode_mode="accurate",' not in source diff --git a/workflows/roboflow_workflows/core_steps/loader.py b/workflows/roboflow_workflows/core_steps/loader.py index 3e602c014b..6f972fbc32 100644 --- a/workflows/roboflow_workflows/core_steps/loader.py +++ b/workflows/roboflow_workflows/core_steps/loader.py @@ -735,6 +735,15 @@ RoboflowInstanceSegmentationModelBlockV4, ) +if ENABLE_TENSOR_DATA_REPRESENTATION: + from roboflow_workflows.core_steps.models.roboflow.instance_segmentation.v5_tensor import ( + RoboflowInstanceSegmentationModelBlockV5, + ) +else: + from roboflow_workflows.core_steps.models.roboflow.instance_segmentation.v5 import ( + RoboflowInstanceSegmentationModelBlockV5, + ) + if not ENABLE_TENSOR_DATA_REPRESENTATION: from roboflow_workflows.core_steps.models.foundation.segment_anything3_interactive.v1 import ( SegmentAnything3InteractiveBlockV1, @@ -1844,6 +1853,7 @@ def load_blocks() -> List[Type[WorkflowBlock]]: RoboflowInstanceSegmentationModelBlockV2, RoboflowInstanceSegmentationModelBlockV3, RoboflowInstanceSegmentationModelBlockV4, + RoboflowInstanceSegmentationModelBlockV5, RoboflowSemanticSegmentationModelBlockV1, RoboflowSemanticSegmentationModelBlockV2, RoboflowKeypointDetectionModelBlockV2, diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1.py index 9320df4c19..42e53769ce 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1.py @@ -319,8 +319,10 @@ def run_locally( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, + # Pinned: these fields never took effect in this version, so + # honouring them now would change existing workflows. @v5 forwards. + mask_decode_mode="accurate", + tradeoff_factor=1.0, enforce_dense_masks_in_inference_models=enforce_dense_masks_in_inference_models, disable_active_learning=disable_active_learning, active_learning_target_dataset=active_learning_target_dataset, @@ -367,8 +369,10 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, + # Pinned: these fields never took effect in this version, so + # honouring them now would change existing workflows. @v5 forwards. + mask_decode_mode="accurate", + tradeoff_factor=1.0, max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, source="workflow-execution", diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1_tensor.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1_tensor.py index 7fb2005017..865b6f722b 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1_tensor.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1_tensor.py @@ -465,8 +465,10 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, + # Pinned: these fields never took effect in this version, so + # honouring them now would change existing workflows. @v5 forwards. + mask_decode_mode="accurate", + tradeoff_factor=1.0, response_mask_format="rle", max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2.py index 2b38d78156..f15e65def8 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2.py @@ -317,8 +317,10 @@ def run_locally( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, + # Pinned: these fields never took effect in this version, so + # honouring them now would change existing workflows. @v5 forwards. + mask_decode_mode="accurate", + tradeoff_factor=1.0, enforce_dense_masks_in_inference_models=enforce_dense_masks_in_inference_models, disable_active_learning=disable_active_learning, active_learning_target_dataset=active_learning_target_dataset, @@ -366,8 +368,10 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, + # Pinned: these fields never took effect in this version, so + # honouring them now would change existing workflows. @v5 forwards. + mask_decode_mode="accurate", + tradeoff_factor=1.0, max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, source="workflow-execution", diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2_tensor.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2_tensor.py index 60a0a3f2c3..62909f6477 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2_tensor.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2_tensor.py @@ -464,8 +464,10 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, + # Pinned: these fields never took effect in this version, so + # honouring them now would change existing workflows. @v5 forwards. + mask_decode_mode="accurate", + tradeoff_factor=1.0, response_mask_format="rle", max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3.py index 4b26fab235..e1cfeef8bc 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3.py @@ -392,8 +392,10 @@ def run_locally( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, + # Pinned: these fields never took effect in this version, so + # honouring them now would change existing workflows. @v5 forwards. + mask_decode_mode="accurate", + tradeoff_factor=1.0, enforce_dense_masks_in_inference_models=enforce_dense_masks_in_inference_models, disable_active_learning=disable_active_learning, active_learning_target_dataset=active_learning_target_dataset, @@ -703,8 +705,10 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, + # Pinned: these fields never took effect in this version, so + # honouring them now would change existing workflows. @v5 forwards. + mask_decode_mode="accurate", + tradeoff_factor=1.0, max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, source="workflow-execution", diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3_tensor.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3_tensor.py index 996e5de003..8f9abeaf14 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3_tensor.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3_tensor.py @@ -505,8 +505,10 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, + # Pinned: these fields never took effect in this version, so + # honouring them now would change existing workflows. @v5 forwards. + mask_decode_mode="accurate", + tradeoff_factor=1.0, response_mask_format="rle", max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py index 677561258b..bd4a0a0de1 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py @@ -354,8 +354,10 @@ def run_locally( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, + # Pinned: these fields never took effect in this version, so + # honouring them now would change existing workflows. @v5 forwards. + mask_decode_mode="accurate", + tradeoff_factor=1.0, response_mask_format="rle", disable_active_learning=disable_active_learning, active_learning_target_dataset=active_learning_target_dataset, @@ -403,8 +405,10 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, + # Pinned: these fields never took effect in this version, so + # honouring them now would change existing workflows. @v5 forwards. + mask_decode_mode="accurate", + tradeoff_factor=1.0, response_mask_format="rle", max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4_tensor.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4_tensor.py index f00cc090b0..fb1b436983 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4_tensor.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4_tensor.py @@ -492,8 +492,10 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, + # Pinned: these fields never took effect in this version, so + # honouring them now would change existing workflows. @v5 forwards. + mask_decode_mode="accurate", + tradeoff_factor=1.0, response_mask_format="rle", max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5.py new file mode 100644 index 0000000000..66b9922900 --- /dev/null +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5.py @@ -0,0 +1,456 @@ +from typing import List, Literal, Optional, Type, Union + +from pydantic import ConfigDict, Field, PositiveInt, model_validator +from roboflow_workflows.core_steps.common.entities import StepExecutionMode +from roboflow_workflows.core_steps.common.utils import ( + attach_parents_coordinates_to_batch_of_sv_detections, + attach_prediction_type_info_to_sv_detections_batch, + convert_inference_detections_batch_to_sv_detections, + filter_out_unwanted_classes_from_sv_detections_batch, +) +from roboflow_workflows.environment import ( + HOSTED_INSTANCE_SEGMENTATION_URL, + LOCAL_INFERENCE_API_URL, + WORKFLOWS_REMOTE_API_KEY_TRANSPORT, + WORKFLOWS_REMOTE_API_TARGET, + WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, + WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, +) +from roboflow_workflows.execution_engine.constants import INFERENCE_ID_KEY +from roboflow_workflows.execution_engine.entities.base import ( + Batch, + OutputDefinition, + WorkflowImageData, +) +from roboflow_workflows.execution_engine.entities.types import ( + BOOLEAN_KIND, + FLOAT_ZERO_TO_ONE_KIND, + IMAGE_KIND, + INFERENCE_ID_KIND, + INSTANCE_SEGMENTATION_PREDICTION_KIND, + INTEGER_KIND, + LIST_OF_VALUES_KIND, + RLE_INSTANCE_SEGMENTATION_PREDICTION_KIND, + ROBOFLOW_MODEL_ID_KIND, + ROBOFLOW_PROJECT_KIND, + STRING_KIND, + FloatZeroToOne, + ImageInputField, + RoboflowModelField, + Selector, +) +from roboflow_workflows.execution_engine.entities.workload import ( + Discovery, + RuntimeRestriction, + WorkOperation, +) +from roboflow_workflows.prototypes.block import ( + BlockResult, + DependentResource, + WorkflowBlock, + WorkflowBlockManifest, + roboflow_platform_model, + roboflow_platform_project, +) +from roboflow_workflows.prototypes.models_provider import ModelsProvider + +from inference_sdk import InferenceConfiguration, InferenceHTTPClient + +LONG_DESCRIPTION = """ +Run inference on an instance segmentation model hosted on or uploaded to Roboflow. + +You can query any model that is private to your account, or any public model available +on [Roboflow Universe](https://universe.roboflow.com). + +You will need to set your Roboflow API key in your Inference environment to use this +block. To learn more about setting your Roboflow API key, [refer to the Inference +documentation](https://inference.roboflow.com/quickstart/configure_api_key/). + +This version of block introduces breaking change in behaviour of mask construction - it uses +`rle` format instead `polygon` making it possible to retrieve +shapes of any kind from remote server. +""" + + +class BlockManifest(WorkflowBlockManifest): + model_config = ConfigDict( + json_schema_extra={ + "name": "Instance Segmentation Model", + "version": "v5", + "short_description": "Predict the shape, size, and location of objects.", + "long_description": LONG_DESCRIPTION, + "license": "Apache-2.0", + "block_type": "model", + "search_keywords": ["yolo", "rfdetr", "rf-detr"], + "ui_manifest": { + "section": "model", + "icon": "far fa-chart-network", + "blockPriority": 1, + "inference": True, + "popular": True, + }, + }, + protected_namespaces=(), + ) + type: Literal["roboflow_core/roboflow_instance_segmentation_model@v5"] + images: Selector(kind=[IMAGE_KIND]) = ImageInputField + model_id: Union[Selector(kind=[ROBOFLOW_MODEL_ID_KIND]), str] = RoboflowModelField + confidence_mode: Union[ + Literal["best", "default", "custom"], + Selector(kind=[STRING_KIND]), + ] = Field( + default="best", + description="How confidence thresholds are determined.", + json_schema_extra={ + "always_visible": True, + "values_metadata": { + "best": { + "name": "Best (Recommended)", + "description": "Use F1-optimal per-class thresholds from model evaluation.", + }, + "default": { + "name": "Default", + "description": "Use the model's built-in default threshold.", + }, + "custom": { + "name": "Custom", + "description": "Specify a custom confidence threshold.", + }, + }, + }, + ) + custom_confidence: Union[ + Optional[FloatZeroToOne], + Selector(kind=[FLOAT_ZERO_TO_ONE_KIND]), + ] = Field( + default=0.4, + description="Custom confidence threshold for predictions.", + examples=[0.3, "$inputs.confidence_threshold"], + json_schema_extra={ + "relevant_for": { + "confidence_mode": {"values": ["custom"], "required": True}, + }, + }, + ) + class_filter: Union[Optional[List[str]], Selector(kind=[LIST_OF_VALUES_KIND])] = ( + Field( + default=None, + description="List of accepted classes. Classes must exist in the model's training set.", + examples=[["a", "b", "c"], "$inputs.class_filter"], + ) + ) + iou_threshold: Union[ + FloatZeroToOne, + Selector(kind=[FLOAT_ZERO_TO_ONE_KIND]), + ] = Field( + default=0.3, + description="Minimum overlap threshold between boxes to combine them into a single detection, used in NMS. [Learn more](https://blog.roboflow.com/how-to-code-non-maximum-suppression-nms-in-plain-numpy/).", + examples=[0.4, "$inputs.iou_threshold"], + ) + max_detections: Union[PositiveInt, Selector(kind=[INTEGER_KIND])] = Field( + default=300, + description="Maximum number of detections to return.", + examples=[300, "$inputs.max_detections"], + ) + class_agnostic_nms: Union[bool, Selector(kind=[BOOLEAN_KIND])] = Field( + default=False, + description="Boolean flag to specify if NMS is to be used in class-agnostic mode.", + examples=[True, "$inputs.class_agnostic_nms"], + ) + max_candidates: Union[PositiveInt, Selector(kind=[INTEGER_KIND])] = Field( + default=3000, + description="Maximum number of candidates as NMS input to be taken into account.", + examples=[3000, "$inputs.max_candidates"], + ) + mask_decode_mode: Union[ + Literal["accurate", "tradeoff", "fast"], + Selector(kind=[STRING_KIND]), + ] = Field( + default="accurate", + description="Parameter of mask decoding in prediction post-processing.", + examples=["accurate", "$inputs.mask_decode_mode"], + ) + tradeoff_factor: Union[ + FloatZeroToOne, + Selector(kind=[FLOAT_ZERO_TO_ONE_KIND]), + ] = Field( + default=0.0, + description="Post-processing parameter to dictate tradeoff between fast and accurate.", + examples=[0.3, "$inputs.tradeoff_factor"], + ) + disable_active_learning: Union[bool, Selector(kind=[BOOLEAN_KIND])] = Field( + default=True, + description="Boolean flag to disable project-level active learning for this block.", + examples=[True, "$inputs.disable_active_learning"], + ) + active_learning_target_dataset: Union[ + Selector(kind=[ROBOFLOW_PROJECT_KIND]), Optional[str] + ] = Field( + default=None, + description="Target dataset for active learning, if enabled.", + examples=["my_project", "$inputs.al_target_project"], + ) + + @model_validator(mode="after") + def validate(self) -> "BlockManifest": + if self.confidence_mode == "custom" and self.custom_confidence is None: + raise ValueError( + "`custom_confidence` is required when `confidence_mode` is 'custom'" + ) + return self + + @classmethod + def get_compatible_task_types(cls) -> Optional[List[str]]: + return ["instance-segmentation"] + + def discover_dependent_resources(self) -> Optional[List[DependentResource]]: + resources = [roboflow_platform_model(model_id=self.model_id)] + if self.disable_active_learning is True: + # Active learning literally disabled (the default) — the target + # project is dead configuration, not a dependency. + return resources + if self.active_learning_target_dataset is not None: + resources.append( + roboflow_platform_project( + project_url=self.active_learning_target_dataset + ) + ) + return resources + + @classmethod + def get_parameters_accepting_batches(cls) -> List[str]: + return ["images"] + + @classmethod + def describe_outputs(cls) -> List[OutputDefinition]: + return [ + OutputDefinition(name=INFERENCE_ID_KEY, kind=[INFERENCE_ID_KIND]), + OutputDefinition( + name="predictions", + kind=[ + RLE_INSTANCE_SEGMENTATION_PREDICTION_KIND, + INSTANCE_SEGMENTATION_PREDICTION_KIND, + ], + ), + OutputDefinition(name="model_id", kind=[ROBOFLOW_MODEL_ID_KIND]), + ] + + @classmethod + def get_execution_engine_compatibility(cls) -> Optional[str]: + return ">=1.3.0,<2.0.0" + + def discover_work_operations(self) -> List[WorkOperation]: + return [WorkOperation.MODEL_INFERENCE] + + def get_actual_restrictions( + self, *, ignore_environment_restrictions: bool = False + ) -> Discovery[RuntimeRestriction]: + return Discovery[RuntimeRestriction]( + items=[], complete=True, unknown_reasons=[] + ) + + +class RoboflowInstanceSegmentationModelBlockV5(WorkflowBlock): + + def __init__( + self, + model_manager: ModelsProvider, + api_key: Optional[str], + step_execution_mode: StepExecutionMode, + ): + self._model_manager = model_manager + self._api_key = api_key + self._step_execution_mode = step_execution_mode + + @classmethod + def get_init_parameters(cls) -> List[str]: + return ["model_manager", "api_key", "step_execution_mode"] + + @classmethod + def get_manifest(cls) -> Type[WorkflowBlockManifest]: + return BlockManifest + + def run( + self, + images: Batch[WorkflowImageData], + model_id: str, + confidence_mode: str, + custom_confidence: Optional[float], + class_agnostic_nms: Optional[bool], + class_filter: Optional[List[str]], + iou_threshold: Optional[float], + max_detections: Optional[int], + max_candidates: Optional[int], + mask_decode_mode: Literal["accurate", "tradeoff", "fast"], + tradeoff_factor: Optional[float], + disable_active_learning: Optional[bool], + active_learning_target_dataset: Optional[str], + ) -> BlockResult: + confidence = ( + custom_confidence if confidence_mode == "custom" else confidence_mode + ) + if self._step_execution_mode is StepExecutionMode.LOCAL: + return self.run_locally( + images=images, + model_id=model_id, + class_agnostic_nms=class_agnostic_nms, + class_filter=class_filter, + confidence=confidence, + iou_threshold=iou_threshold, + max_detections=max_detections, + max_candidates=max_candidates, + mask_decode_mode=mask_decode_mode, + tradeoff_factor=tradeoff_factor, + disable_active_learning=disable_active_learning, + active_learning_target_dataset=active_learning_target_dataset, + ) + elif self._step_execution_mode is StepExecutionMode.REMOTE: + return self.run_remotely( + images=images, + model_id=model_id, + class_agnostic_nms=class_agnostic_nms, + class_filter=class_filter, + confidence=confidence, + iou_threshold=iou_threshold, + max_detections=max_detections, + max_candidates=max_candidates, + mask_decode_mode=mask_decode_mode, + tradeoff_factor=tradeoff_factor, + disable_active_learning=disable_active_learning, + active_learning_target_dataset=active_learning_target_dataset, + ) + else: + raise ValueError( + f"Unknown step execution mode: {self._step_execution_mode}" + ) + + def run_locally( + self, + images: Batch[WorkflowImageData], + model_id: str, + class_agnostic_nms: Optional[bool], + class_filter: Optional[List[str]], + confidence: Union[None, float, Literal["best", "default"]], + iou_threshold: Optional[float], + max_detections: Optional[int], + max_candidates: Optional[int], + mask_decode_mode: Literal["accurate", "tradeoff", "fast"], + tradeoff_factor: Optional[float], + disable_active_learning: Optional[bool], + active_learning_target_dataset: Optional[str], + ) -> BlockResult: + inference_images = [i.to_inference_format(numpy_preferred=True) for i in images] + self._model_manager.add_model( + model_id=model_id, + api_key=self._api_key, + ) + predictions = self._model_manager.run_instance_segmentation( + model_id=model_id, + images=inference_images, + api_key=self._api_key, + class_agnostic_nms=class_agnostic_nms, + class_filter=class_filter, + confidence=confidence, + iou_threshold=iou_threshold, + max_detections=max_detections, + max_candidates=max_candidates, + mask_decode_mode=mask_decode_mode, + tradeoff_factor=tradeoff_factor, + response_mask_format="rle", + disable_active_learning=disable_active_learning, + active_learning_target_dataset=active_learning_target_dataset, + ) + return self._post_process_result( + images=images, + predictions=predictions, + class_filter=class_filter, + model_id=model_id, + ) + + def run_remotely( + self, + images: Batch[WorkflowImageData], + model_id: str, + class_agnostic_nms: Optional[bool], + class_filter: Optional[List[str]], + confidence: Union[None, float, Literal["best", "default"]], + iou_threshold: Optional[float], + max_detections: Optional[int], + max_candidates: Optional[int], + mask_decode_mode: Literal["accurate", "tradeoff", "fast"], + tradeoff_factor: Optional[float], + disable_active_learning: Optional[bool], + active_learning_target_dataset: Optional[str], + ) -> BlockResult: + api_url = ( + LOCAL_INFERENCE_API_URL + if WORKFLOWS_REMOTE_API_TARGET != "hosted" + else HOSTED_INSTANCE_SEGMENTATION_URL + ) + client = InferenceHTTPClient( + api_url=api_url, + api_key=self._api_key, + ) + if WORKFLOWS_REMOTE_API_TARGET == "hosted": + client.select_api_v0() + client_config = InferenceConfiguration( + api_key_transport=WORKFLOWS_REMOTE_API_KEY_TRANSPORT, + disable_active_learning=disable_active_learning, + active_learning_target_dataset=active_learning_target_dataset, + class_agnostic_nms=class_agnostic_nms, + class_filter=class_filter, + confidence_threshold=confidence, + iou_threshold=iou_threshold, + max_detections=max_detections, + max_candidates=max_candidates, + mask_decode_mode=mask_decode_mode, + tradeoff_factor=tradeoff_factor, + response_mask_format="rle", + max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, + max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, + source="workflow-execution", + ) + client.configure(inference_configuration=client_config) + inference_images = [i.base64_image for i in images] + predictions = client.infer( + inference_input=inference_images, + model_id=model_id, + ) + if not isinstance(predictions, list): + predictions = [predictions] + return self._post_process_result( + images=images, + predictions=predictions, + class_filter=class_filter, + model_id=model_id, + ) + + def _post_process_result( + self, + images: Batch[WorkflowImageData], + predictions: List[dict], + class_filter: Optional[List[str]], + model_id: str, + ) -> BlockResult: + inference_ids = [p.get(INFERENCE_ID_KEY, None) for p in predictions] + predictions = convert_inference_detections_batch_to_sv_detections(predictions) + predictions = attach_prediction_type_info_to_sv_detections_batch( + predictions=predictions, + prediction_type="instance-segmentation", + ) + predictions = filter_out_unwanted_classes_from_sv_detections_batch( + predictions=predictions, + classes_to_accept=class_filter, + ) + predictions = attach_parents_coordinates_to_batch_of_sv_detections( + images=images, + predictions=predictions, + ) + return [ + { + "inference_id": inference_id, + "predictions": prediction, + "model_id": model_id, + } + for inference_id, prediction in zip(inference_ids, predictions) + ] diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py new file mode 100644 index 0000000000..c9838cb7ba --- /dev/null +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py @@ -0,0 +1,744 @@ +"""Tensor-native sibling of `roboflow_core/roboflow_instance_segmentation_model@v5`. + +Under ENABLE_TENSOR_DATA_REPRESENTATION this block emits a native +``inference_models.InstanceDetections`` (torch tensors on +``WORKFLOWS_IMAGE_TENSOR_DEVICE``; masks dense or RLE) under +``TENSOR_NATIVE_INSTANCE_SEGMENTATION_PREDICTION_KIND`` / +``TENSOR_NATIVE_RLE_INSTANCE_SEGMENTATION_PREDICTION_KIND`` instead of +``sv.Detections``. + +- LOCAL: ``ModelsProvider.run_tensor_native_inference`` returns + ``List[InstanceDetections]`` straight from the adapter. The mask carrier (dense + ``torch.Tensor`` vs ``InstancesRLEMasks``) is adapter-decided; both are handled + downstream by the helpers and the tensor serialiser. The block applies + ``class_filter`` natively (the adapter/model does NOT read it on this path) and + attaches the producer contract (``image_metadata[class_names]`` + per-box + ``detection_id``) the tensor serialiser requires, via + ``attach_native_detection_metadata`` (it preserves masks + per-box metadata). +- REMOTE: standard inference instance-seg prediction dicts are rebuilt into a + native ``InstanceDetections`` via the in-file + ``_native_instance_detections_from_inference_predictions`` converter (never + ``sv.Detections``). The block requests ``response_mask_format="rle"`` and carries + RLE masks when the server honours it; it degrades gracefully to a dense mask + rasterised from polygon ``points`` when the server ignores that parameter + (older/alternative servers), matching the numpy flag-OFF polygon path. + +This block creates ONLY this file; it reuses the already-registered tensor +serialiser for ``instance_segmentation_prediction`` / +``rle_instance_segmentation_prediction`` (``serialise_sv_detections`` already +handles ``InstanceDetections``, dense or RLE). The numpy sibling lives in +``.../instance_segmentation/v4.py``; this manifest is identical except the output +kinds. +""" + +import uuid +from typing import Dict, List, Literal, Optional, Type, Union + +import numpy as np +import supervision as sv +import torch +from pydantic import ConfigDict, Field, PositiveInt, model_validator +from roboflow_workflows.core_steps.common.entities import StepExecutionMode +from roboflow_workflows.core_steps.common.tensor_native import ( + attach_native_detection_metadata, + build_native_image_metadata, + take_prediction_by_mask, +) +from roboflow_workflows.environment import ( + HOSTED_INSTANCE_SEGMENTATION_URL, + LOCAL_INFERENCE_API_URL, + WORKFLOWS_ENFORCE_DENSE_INSTANCE_MASKS, + WORKFLOWS_IMAGE_TENSOR_DEVICE, + WORKFLOWS_REMOTE_API_KEY_TRANSPORT, + WORKFLOWS_REMOTE_API_TARGET, + WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, + WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, +) +from roboflow_workflows.execution_engine.constants import ( + CLASS_ID_KEY, + CLASS_NAME_KEY, + CONFIDENCE_KEY, + DETECTION_ID_KEY, + HEIGHT_KEY, + INFERENCE_ID_KEY, + POLYGON_KEY, + WIDTH_KEY, + X_KEY, + Y_KEY, +) +from roboflow_workflows.execution_engine.entities.base import ( + Batch, + OutputDefinition, + WorkflowImageData, +) +from roboflow_workflows.execution_engine.entities.tensor_native_types import ( + TENSOR_NATIVE_INSTANCE_SEGMENTATION_PREDICTION_KIND, + TENSOR_NATIVE_RLE_INSTANCE_SEGMENTATION_PREDICTION_KIND, +) +from roboflow_workflows.execution_engine.entities.types import ( + BOOLEAN_KIND, + FLOAT_ZERO_TO_ONE_KIND, + IMAGE_KIND, + INFERENCE_ID_KIND, + INTEGER_KIND, + LIST_OF_VALUES_KIND, + ROBOFLOW_MODEL_ID_KIND, + ROBOFLOW_PROJECT_KIND, + STRING_KIND, + FloatZeroToOne, + ImageInputField, + RoboflowModelField, + Selector, +) +from roboflow_workflows.execution_engine.entities.workload import ( + Discovery, + RuntimeRestriction, + WorkOperation, +) +from roboflow_workflows.prototypes.block import ( + BlockResult, + DependentResource, + WorkflowBlock, + WorkflowBlockManifest, + roboflow_platform_model, + roboflow_platform_project, +) +from roboflow_workflows.prototypes.models_provider import ModelsProvider + +from inference_models.models.base.instance_segmentation import InstanceDetections +from inference_models.models.base.types import InstancesRLEMasks +from inference_sdk import InferenceConfiguration, InferenceHTTPClient + +PREDICTION_TYPE = "instance-segmentation" + +# Keys carrying the RLE mask in standard inference instance-seg prediction dicts. +# `InstanceSegmentationRLEPrediction` declares the mask under `rle`; some response +# paths use `rle_mask` (mirrors the numpy converter's `d.get("rle_mask") or d.get("rle")`). +RLE_MASK_KEYS = ("rle_mask", "rle") + +LONG_DESCRIPTION = """ +Run inference on an instance segmentation model hosted on or uploaded to Roboflow. + +You can query any model that is private to your account, or any public model available +on [Roboflow Universe](https://universe.roboflow.com). + +You will need to set your Roboflow API key in your Inference environment to use this +block. To learn more about setting your Roboflow API key, [refer to the Inference +documentation](https://inference.roboflow.com/quickstart/configure_api_key/). + +This version of block introduces breaking change in behaviour of mask construction - it uses +`rle` format instead `polygon` making it possible to retrieve +shapes of any kind from remote server. +""" + + +class BlockManifest(WorkflowBlockManifest): + model_config = ConfigDict( + json_schema_extra={ + "name": "Instance Segmentation Model", + "version": "v5", + "short_description": "Predict the shape, size, and location of objects.", + "long_description": LONG_DESCRIPTION, + "license": "Apache-2.0", + "block_type": "model", + "search_keywords": ["yolo", "rfdetr", "rf-detr"], + "ui_manifest": { + "section": "model", + "icon": "far fa-chart-network", + "blockPriority": 1, + "inference": True, + "popular": True, + }, + }, + protected_namespaces=(), + ) + type: Literal["roboflow_core/roboflow_instance_segmentation_model@v5"] + images: Selector(kind=[IMAGE_KIND]) = ImageInputField + model_id: Union[Selector(kind=[ROBOFLOW_MODEL_ID_KIND]), str] = RoboflowModelField + confidence_mode: Union[ + Literal["best", "default", "custom"], + Selector(kind=[STRING_KIND]), + ] = Field( + default="best", + description="How confidence thresholds are determined.", + json_schema_extra={ + "always_visible": True, + "values_metadata": { + "best": { + "name": "Best (Recommended)", + "description": "Use F1-optimal per-class thresholds from model evaluation.", + }, + "default": { + "name": "Default", + "description": "Use the model's built-in default threshold.", + }, + "custom": { + "name": "Custom", + "description": "Specify a custom confidence threshold.", + }, + }, + }, + ) + custom_confidence: Union[ + Optional[FloatZeroToOne], + Selector(kind=[FLOAT_ZERO_TO_ONE_KIND]), + ] = Field( + default=0.4, + description="Custom confidence threshold for predictions.", + examples=[0.3, "$inputs.confidence_threshold"], + json_schema_extra={ + "relevant_for": { + "confidence_mode": {"values": ["custom"], "required": True}, + }, + }, + ) + class_filter: Union[Optional[List[str]], Selector(kind=[LIST_OF_VALUES_KIND])] = ( + Field( + default=None, + description="List of accepted classes. Classes must exist in the model's training set.", + examples=[["a", "b", "c"], "$inputs.class_filter"], + ) + ) + iou_threshold: Union[ + FloatZeroToOne, + Selector(kind=[FLOAT_ZERO_TO_ONE_KIND]), + ] = Field( + default=0.3, + description="Minimum overlap threshold between boxes to combine them into a single detection, used in NMS. [Learn more](https://blog.roboflow.com/how-to-code-non-maximum-suppression-nms-in-plain-numpy/).", + examples=[0.4, "$inputs.iou_threshold"], + ) + max_detections: Union[PositiveInt, Selector(kind=[INTEGER_KIND])] = Field( + default=300, + description="Maximum number of detections to return.", + examples=[300, "$inputs.max_detections"], + ) + class_agnostic_nms: Union[bool, Selector(kind=[BOOLEAN_KIND])] = Field( + default=False, + description="Boolean flag to specify if NMS is to be used in class-agnostic mode.", + examples=[True, "$inputs.class_agnostic_nms"], + ) + max_candidates: Union[PositiveInt, Selector(kind=[INTEGER_KIND])] = Field( + default=3000, + description="Maximum number of candidates as NMS input to be taken into account.", + examples=[3000, "$inputs.max_candidates"], + ) + mask_decode_mode: Union[ + Literal["accurate", "tradeoff", "fast"], + Selector(kind=[STRING_KIND]), + ] = Field( + default="accurate", + description="Parameter of mask decoding in prediction post-processing.", + examples=["accurate", "$inputs.mask_decode_mode"], + ) + tradeoff_factor: Union[ + FloatZeroToOne, + Selector(kind=[FLOAT_ZERO_TO_ONE_KIND]), + ] = Field( + default=0.0, + description="Post-processing parameter to dictate tradeoff between fast and accurate.", + examples=[0.3, "$inputs.tradeoff_factor"], + ) + disable_active_learning: Union[bool, Selector(kind=[BOOLEAN_KIND])] = Field( + default=True, + description="Boolean flag to disable project-level active learning for this block.", + examples=[True, "$inputs.disable_active_learning"], + ) + active_learning_target_dataset: Union[ + Selector(kind=[ROBOFLOW_PROJECT_KIND]), Optional[str] + ] = Field( + default=None, + description="Target dataset for active learning, if enabled.", + examples=["my_project", "$inputs.al_target_project"], + ) + + @model_validator(mode="after") + def validate(self) -> "BlockManifest": + if self.confidence_mode == "custom" and self.custom_confidence is None: + raise ValueError( + "`custom_confidence` is required when `confidence_mode` is 'custom'" + ) + return self + + @classmethod + def get_compatible_task_types(cls) -> Optional[List[str]]: + return ["instance-segmentation"] + + def discover_dependent_resources(self) -> Optional[List[DependentResource]]: + resources = [roboflow_platform_model(model_id=self.model_id)] + if self.disable_active_learning is True: + # Active learning literally disabled (the default) — the target + # project is dead configuration, not a dependency. + return resources + if self.active_learning_target_dataset is not None: + resources.append( + roboflow_platform_project( + project_url=self.active_learning_target_dataset + ) + ) + return resources + + @classmethod + def get_parameters_accepting_batches(cls) -> List[str]: + return ["images"] + + @classmethod + def describe_outputs(cls) -> List[OutputDefinition]: + return [ + OutputDefinition(name=INFERENCE_ID_KEY, kind=[INFERENCE_ID_KIND]), + OutputDefinition( + name="predictions", + kind=[ + TENSOR_NATIVE_RLE_INSTANCE_SEGMENTATION_PREDICTION_KIND, + TENSOR_NATIVE_INSTANCE_SEGMENTATION_PREDICTION_KIND, + ], + ), + OutputDefinition(name="model_id", kind=[ROBOFLOW_MODEL_ID_KIND]), + ] + + @classmethod + def get_execution_engine_compatibility(cls) -> Optional[str]: + return ">=1.3.0,<2.0.0" + + def discover_work_operations(self) -> List[WorkOperation]: + return [WorkOperation.MODEL_INFERENCE] + + def get_actual_restrictions( + self, *, ignore_environment_restrictions: bool = False + ) -> Discovery[RuntimeRestriction]: + return Discovery[RuntimeRestriction]( + items=[], complete=True, unknown_reasons=[] + ) + + +class RoboflowInstanceSegmentationModelBlockV5(WorkflowBlock): + + def __init__( + self, + model_manager: ModelsProvider, + api_key: Optional[str], + step_execution_mode: StepExecutionMode, + ): + self._model_manager = model_manager + self._api_key = api_key + self._step_execution_mode = step_execution_mode + + @classmethod + def get_init_parameters(cls) -> List[str]: + return ["model_manager", "api_key", "step_execution_mode"] + + @classmethod + def get_manifest(cls) -> Type[WorkflowBlockManifest]: + return BlockManifest + + def run( + self, + images: Batch[WorkflowImageData], + model_id: str, + confidence_mode: str, + custom_confidence: Optional[float], + class_agnostic_nms: Optional[bool], + class_filter: Optional[List[str]], + iou_threshold: Optional[float], + max_detections: Optional[int], + max_candidates: Optional[int], + mask_decode_mode: Literal["accurate", "tradeoff", "fast"], + tradeoff_factor: Optional[float], + disable_active_learning: Optional[bool], + active_learning_target_dataset: Optional[str], + ) -> BlockResult: + confidence = ( + custom_confidence if confidence_mode == "custom" else confidence_mode + ) + if self._step_execution_mode is StepExecutionMode.LOCAL: + return self.run_locally( + images=images, + model_id=model_id, + class_agnostic_nms=class_agnostic_nms, + class_filter=class_filter, + confidence=confidence, + iou_threshold=iou_threshold, + max_detections=max_detections, + max_candidates=max_candidates, + mask_decode_mode=mask_decode_mode, + tradeoff_factor=tradeoff_factor, + disable_active_learning=disable_active_learning, + active_learning_target_dataset=active_learning_target_dataset, + ) + elif self._step_execution_mode is StepExecutionMode.REMOTE: + return self.run_remotely( + images=images, + model_id=model_id, + class_agnostic_nms=class_agnostic_nms, + class_filter=class_filter, + confidence=confidence, + iou_threshold=iou_threshold, + max_detections=max_detections, + max_candidates=max_candidates, + mask_decode_mode=mask_decode_mode, + tradeoff_factor=tradeoff_factor, + disable_active_learning=disable_active_learning, + active_learning_target_dataset=active_learning_target_dataset, + ) + else: + raise ValueError( + f"Unknown step execution mode: {self._step_execution_mode}" + ) + + def run_locally( + self, + images: Batch[WorkflowImageData], + model_id: str, + class_agnostic_nms: Optional[bool], + class_filter: Optional[List[str]], + confidence: Union[None, float, Literal["best", "default"]], + iou_threshold: Optional[float], + max_detections: Optional[int], + max_candidates: Optional[int], + mask_decode_mode: Literal["accurate", "tradeoff", "fast"], + tradeoff_factor: Optional[float], + disable_active_learning: Optional[bool], + active_learning_target_dataset: Optional[str], + ) -> BlockResult: + # Feed the representation already materialised on the images to avoid forcing a + # numpy->device conversion: GPU tensors (RGB) only when every image in the batch + # already has one, otherwise the numpy frames (BGR) — matching the numpy block. + if all(image.is_tensor_materialised() for image in images): + model_inputs = [image.tensor_image for image in images] + image_color_format = "rgb" + else: + model_inputs = [image.numpy_image for image in images] + image_color_format = "bgr" + self._model_manager.add_model(model_id=model_id, api_key=self._api_key) + detections_batch: List[InstanceDetections] = ( + self._model_manager.run_tensor_native_inference( + model_id=model_id, + images=model_inputs, + input_color_format=image_color_format, + enforce_dense_masks_in_inference_models=WORKFLOWS_ENFORCE_DENSE_INSTANCE_MASKS, + confidence=confidence, + iou_threshold=iou_threshold, + class_agnostic_nms=class_agnostic_nms, + class_filter=class_filter, + max_detections=max_detections, + max_candidates=max_candidates, + mask_decode_mode=mask_decode_mode, + tradeoff_factor=tradeoff_factor, + disable_active_learning=disable_active_learning, + active_learning_target_dataset=active_learning_target_dataset, + ) + ) + class_names = _class_names_map(self._model_manager.get_class_names(model_id)) + results: List[dict] = [] + for image, detections in zip(images, detections_batch): + # The adapter/model does NOT honour class_filter on the native path, so + # filter here (the only place it is applied for LOCAL execution). + detections = _filter_classes_native(detections, class_filter, class_names) + # Reuse the adapter-provided inference id when present (numpy parity: + # numpy v4 surfaces ``p.get(INFERENCE_ID_KEY)`` off the model dump); + # the tensor-native adapter normally attaches none, so fall back to a + # freshly minted uuid that is then shared with ``image_metadata``. + inference_id = getattr(detections, "inference_id", None) or str( + uuid.uuid4() + ) + detections = attach_native_detection_metadata( + detections=detections, + image=image, + class_names=class_names, + prediction_type=PREDICTION_TYPE, + inference_id=inference_id, + ) + results.append( + { + "inference_id": inference_id, + "predictions": detections, + "model_id": model_id, + } + ) + return results + + def run_remotely( + self, + images: Batch[WorkflowImageData], + model_id: str, + class_agnostic_nms: Optional[bool], + class_filter: Optional[List[str]], + confidence: Union[None, float, Literal["best", "default"]], + iou_threshold: Optional[float], + max_detections: Optional[int], + max_candidates: Optional[int], + mask_decode_mode: Literal["accurate", "tradeoff", "fast"], + tradeoff_factor: Optional[float], + disable_active_learning: Optional[bool], + active_learning_target_dataset: Optional[str], + ) -> BlockResult: + api_url = ( + LOCAL_INFERENCE_API_URL + if WORKFLOWS_REMOTE_API_TARGET != "hosted" + else HOSTED_INSTANCE_SEGMENTATION_URL + ) + client = InferenceHTTPClient( + api_url=api_url, + api_key=self._api_key, + ) + if WORKFLOWS_REMOTE_API_TARGET == "hosted": + client.select_api_v0() + client_config = InferenceConfiguration( + api_key_transport=WORKFLOWS_REMOTE_API_KEY_TRANSPORT, + disable_active_learning=disable_active_learning, + active_learning_target_dataset=active_learning_target_dataset, + class_agnostic_nms=class_agnostic_nms, + class_filter=class_filter, + confidence_threshold=confidence, + iou_threshold=iou_threshold, + max_detections=max_detections, + max_candidates=max_candidates, + mask_decode_mode=mask_decode_mode, + tradeoff_factor=tradeoff_factor, + response_mask_format="rle", + max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, + max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, + source="workflow-execution", + ) + client.configure(inference_configuration=client_config) + inference_images = [i.base64_image for i in images] + predictions = client.infer( + inference_input=inference_images, + model_id=model_id, + ) + if not isinstance(predictions, list): + predictions = [predictions] + return self._post_process_remote_result( + images=images, + predictions=predictions, + class_filter=class_filter, + model_id=model_id, + ) + + def _post_process_remote_result( + self, + images: Batch[WorkflowImageData], + predictions: List[dict], + class_filter: Optional[List[str]], + model_id: str, + ) -> BlockResult: + # Fallback class_id -> name map from the model, used to name boxes whose + # remote prediction dict lacks a `class` key (otherwise the tensor + # serialiser hard-raises "class_id missing from mapping"). + model_class_names = _class_names_map( + self._model_manager.get_class_names(model_id) + ) + results: List[dict] = [] + for image, response in zip(images, predictions): + inference_id = response.get(INFERENCE_ID_KEY) or str(uuid.uuid4()) + detection_dicts = response.get("predictions", []) or [] + if class_filter: + accepted = set(class_filter) + detection_dicts = [ + d for d in detection_dicts if d.get(CLASS_NAME_KEY) in accepted + ] + detections = _native_instance_detections_from_inference_predictions( + image=image, + predictions=detection_dicts, + prediction_type=PREDICTION_TYPE, + inference_id=inference_id, + device=WORKFLOWS_IMAGE_TENSOR_DEVICE, + model_class_names=model_class_names, + ) + results.append( + { + "inference_id": inference_id, + "predictions": detections, + "model_id": model_id, + } + ) + return results + + +def _class_names_map(class_names: List[str]) -> Dict[int, str]: + return {index: name for index, name in enumerate(class_names)} + + +def _filter_classes_native( + detections: InstanceDetections, + class_filter: Optional[List[str]], + class_names: Dict[int, str], +) -> InstanceDetections: + if not class_filter: + return detections + accepted = set(class_filter) + accepted_ids = sorted( + class_id for class_id, name in class_names.items() if name in accepted + ) + if not accepted_ids: + return take_prediction_by_mask( + detections, + torch.zeros_like(detections.class_id, dtype=torch.bool), + ) + accepted_tensor = torch.as_tensor(accepted_ids, device=detections.class_id.device) + keep = torch.isin(detections.class_id, accepted_tensor) + return take_prediction_by_mask(detections, keep) + + +def _extract_rle_mask(prediction: dict) -> Optional[dict]: + """Pull the COCO-RLE mask ({"size": [H, W], "counts": ...}) from a standard + inference instance-seg prediction dict, trying both response key variants.""" + for key in RLE_MASK_KEYS: + mask = prediction.get(key) + if mask is not None: + return mask + return None + + +def _extract_polygon_points(prediction: dict) -> Optional[List[dict]]: + """Pull the polygon ``points`` ([{"x": .., "y": ..}, ...]) from a standard + inference instance-seg prediction dict that carries a polygon mask instead of + RLE. Returns ``None`` when the key is absent or the polygon is degenerate + (< 3 points) - mirroring the numpy ``filter_out_invalid_polygons``, which drops + such instances entirely.""" + points = prediction.get(POLYGON_KEY) + if points is not None and len(points) >= 3: + return points + return None + + +def _polygon_points_to_dense_mask( + points: List[dict], height: int, width: int +) -> np.ndarray: + """Rasterise polygon ``points`` into a dense boolean ``(H, W)`` mask, + byte-identically to supervision's ``polygon_to_mask`` as used by + ``Detections.from_inference`` (the numpy REMOTE path): vertices are integer + *truncated* (``dtype=int``, NOT rounded), filled with ``cv2.fillPoly`` onto a + zero ``uint8`` canvas, then cast to ``bool``. Keeping the exact rasterisation + means the serialiser's mask->polygon re-contouring (``mask_to_polygon``, shared + verbatim with the numpy serialiser) reproduces the numpy ``points`` output.""" + polygon = np.array([[point[X_KEY], point[Y_KEY]] for point in points], dtype=int) + return sv.polygon_to_mask(polygon, resolution_wh=(width, height)).astype(bool) + + +def _native_instance_detections_from_inference_predictions( + image: WorkflowImageData, + predictions: List[dict], + prediction_type: str, + inference_id: Optional[str] = None, + device: Optional[torch.device] = None, + model_class_names: Optional[Dict[int, str]] = None, +) -> InstanceDetections: + """REMOTE-path converter: build a native ``InstanceDetections`` from standard + inference instance-segmentation prediction dicts. + + The block requests ``response_mask_format="rle"`` so a current inference server + returns an RLE-encoded COCO mask (``{"size": [H, W], "counts": ""}``) + under ``rle`` / ``rle_mask``; those are carried as ``InstancesRLEMasks`` (counts + normalised from a utf-8 string to bytes, what ``pycocotools`` / the serialiser's + RLE decode path expect). To stay compatible with servers that ignore that + request parameter and return polygon ``points`` instead (older/alternative + inference servers), the converter degrades gracefully: when the response is not + fully RLE-backed it rebuilds the masks from the polygon ``points`` into a dense + ``torch.bool`` ``(N, H, W)`` carrier - the same masks the numpy flag-OFF REMOTE + path builds via ``sv.Detections.from_inference`` - so the serialised ``points`` + output matches numpy's (the serialiser re-contours the dense mask with the same + ``mask_to_polygon``). Boxes are converted from center form to corner ``xyxy``. + The ``class_id -> name`` map (required by the tensor serialiser) and per-box + ``detection_id`` are built here too. + + Degenerate (< 3-point) polygons are dropped exactly like the numpy path + (``filter_out_invalid_polygons``). When a + prediction omits its ``class`` key, the class name is backfilled from + ``model_class_names`` (the model's ``get_class_names`` map) so the tensor + serialiser does not hard-raise on an unmapped ``class_id``. + """ + height, width = image._read_shape_without_materialization() + # Prefer the RLE masks the block requests; fall back to dense polygon masks when + # the response is not fully RLE-backed (server ignored `response_mask_format`). + # `all([])` is True, so an empty response keeps the empty-RLE carrier unchanged. + use_rle = all( + _extract_rle_mask(prediction) is not None for prediction in predictions + ) + if use_rle: + kept_predictions = predictions + else: + # Drop degenerate polygons up front so the surviving box arrays and masks + # stay aligned (matches the numpy path's instance drop). + kept_predictions = [ + prediction + for prediction in predictions + if _extract_polygon_points(prediction) is not None + ] + xyxy: List[List[float]] = [] + class_id: List[int] = [] + confidence: List[float] = [] + rle_counts: List[bytes] = [] + dense_masks: List[np.ndarray] = [] + bboxes_metadata: List[dict] = [] + derived_class_names: Dict[int, str] = {} + for prediction in kept_predictions: + center_x = float(prediction[X_KEY]) + center_y = float(prediction[Y_KEY]) + box_width = float(prediction[WIDTH_KEY]) + box_height = float(prediction[HEIGHT_KEY]) + xyxy.append( + [ + center_x - box_width / 2, + center_y - box_height / 2, + center_x + box_width / 2, + center_y + box_height / 2, + ] + ) + prediction_class_id = int(prediction.get(CLASS_ID_KEY, 0)) + class_id.append(prediction_class_id) + confidence.append(float(prediction.get(CONFIDENCE_KEY, 1.0))) + if CLASS_NAME_KEY in prediction: + derived_class_names[prediction_class_id] = str(prediction[CLASS_NAME_KEY]) + elif model_class_names is not None and prediction_class_id in model_class_names: + derived_class_names[prediction_class_id] = model_class_names[ + prediction_class_id + ] + if use_rle: + mask = _extract_rle_mask(prediction) + raw_counts = mask["counts"] + # Normalise to bytes: pycocotools (used by the serialiser's RLE decode) + # expects byte counts; the remote rle response carries them as utf-8 strings. + rle_counts.append( + raw_counts.encode("utf-8") + if isinstance(raw_counts, str) + else raw_counts + ) + else: + dense_masks.append( + _polygon_points_to_dense_mask( + points=_extract_polygon_points(prediction), + height=height, + width=width, + ) + ) + bboxes_metadata.append( + {DETECTION_ID_KEY: str(prediction.get(DETECTION_ID_KEY) or uuid.uuid4())} + ) + number_of_detections = len(xyxy) + image_metadata = build_native_image_metadata( + image=image, + class_names=derived_class_names, + prediction_type=prediction_type, + inference_id=inference_id, + ) + if use_rle: + mask_carrier: Union[torch.Tensor, InstancesRLEMasks] = InstancesRLEMasks( + image_size=(height, width), masks=rle_counts + ) + elif dense_masks: + mask_carrier = torch.as_tensor( + np.stack(dense_masks), dtype=torch.bool, device=device + ) + else: + # No surviving polygons: an empty dense carrier the serialiser handles. + mask_carrier = torch.zeros((0, height, width), dtype=torch.bool, device=device) + return InstanceDetections( + xyxy=torch.as_tensor(xyxy, dtype=torch.float32, device=device).reshape(-1, 4), + class_id=torch.as_tensor(class_id, dtype=torch.long, device=device).reshape(-1), + confidence=torch.as_tensor( + confidence, dtype=torch.float32, device=device + ).reshape(-1), + mask=mask_carrier, + image_metadata=image_metadata, + bboxes_metadata=bboxes_metadata if number_of_detections > 0 else None, + ) From 550bbdc26b068c7aa25e042ec8aac88c30b358d7 Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Fri, 2 Oct 2026 17:01:48 +0200 Subject: [PATCH 13/41] Replace the max_detections hypothesis in the plan with its measurement The plan called reducing max_detections a cheaper, lossless first move. Measured, it is neither: halving it saves 55% where the resolution factor saves 90%, and capping at 150 in a scene with 300 objects deletes 150 real detections rather than degrading them. Replaces the claim with the data. --- docs/feat/instance-seg-tradeoff-factor.md | 39 +++++++++++++++-------- 1 file changed, 26 insertions(+), 13 deletions(-) diff --git a/docs/feat/instance-seg-tradeoff-factor.md b/docs/feat/instance-seg-tradeoff-factor.md index e60a15f4be..da8a4f552e 100644 --- a/docs/feat/instance-seg-tradeoff-factor.md +++ b/docs/feat/instance-seg-tradeoff-factor.md @@ -36,8 +36,12 @@ the argument. **Two things this plan does not claim.** It is not sufficient on its own — at `n=300`/4K even `t=0.25` leaves several hundred ms in this stage. And cost is Θ(n·H·W), so **reducing -`max_detections`** (default 300, `inference_models/configuration.py:229`) is a cheaper, lossless -first move that should be evaluated alongside it. +`max_detections`** (default 300, `inference_models/configuration.py:229`) is the obvious +alternative. It was measured, and it is the weaker lever: halving it saves 55% where the resolution +factor saves 90% — **~4.6× less effective** — and it cannot reach real-time at 4K at any realistic +value. It is also **not lossless**: capping at 150 in a scene with 300 objects deletes 150 real +detections, a recall cliff rather than the uniform boundary-fidelity cost that reducing resolution +carries. See E1. ### The published contract @@ -376,17 +380,26 @@ reject mid-stream changes. Non-blocking. Neither of these blocks implementation. Both block the plan's *justification*, and the first could reduce its scope substantially. -**E1 — `max_detections` has not been evaluated as the cheaper alternative.** -Mask post-processing is Θ(n·H·W). This plan attacks `H·W` and leaves `n` untouched, yet `n` is a -parameter that already exists: `INFERENCE_MODELS_DEFAULT_MAX_DETECTIONS` defaults to **300** -(`inference_models/configuration.py:229`). Halving it halves the cost, is lossless on the top-150 -predictions, and needs no new parameter, no carried metadata, no wire-format change and no -coordinate rescaling — none of the blast radius this plan spends most of its length on. -*Action:* measure `AP_mask` and p50/p95 against `max_detections ∈ {300, 150, 100}` on the same fixed -eval set as the `t` curve, and publish the two curves together. -*Why it matters:* if reducing `n` delivers a large fraction of the win, the honest recommendation -may be guidance plus a default change rather than this feature. That is worth knowing before more is -built, not after. +**E1 — `max_detections` measured; it does not reframe the plan.** +4K, 300 instances, 160×160 prototypes, M1 Max CPU fp32, chunk 16, fresh process per configuration, +warm-up 2, 10 iterations, p50 ms: + +| n | t=0.25 | t=0.5 | t=1.0 | RSS at t=1.0 | +|---|---|---|---|---| +| 75 | 16.4 | 45.7 | 140.1 | 1632 MB | +| 150 | 29.1 | 79.0 | 260.5 | 2474 MB | +| 300 | 56.1 | 155.1 | 576.4 | 3522 MB | + +From the same starting point (n=300, t=1.0, 576.4 ms): halving `max_detections` gives 260.5 ms +(55% saved); dropping `t` to 0.25 gives 56.1 ms (90% saved). The resolution lever is ~4.6× more +effective, and the ordering holds for memory (1.4× against 4.8×). **No realistic `max_detections` +reaches real-time**: a 30 fps budget is ~33 ms for the whole frame, and at `t=1.0` even n=75 costs +140 ms in this stage alone — you would need n≈18. + +The accuracy argument also runs the other way. "Lossless on the top 150" is tautological: in a scene +with 300 real objects, capping at 150 removes 150 real detections. Reducing resolution costs every +object some boundary fidelity; reducing `n` costs you half the objects. They compose +(n=150, t=0.25 → 29.1 ms) but they are not substitutes. **E2 — There is no CUDA or Jetson measurement anywhere in this plan.** Every figure is single-platform CPU (Apple M1 Max, 8 threads, fp32). The ~40 GB-per-4K-frame traffic From 302811c9fb9df7d21edec1f3e7d37b539d633164 Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Fri, 2 Oct 2026 17:04:38 +0200 Subject: [PATCH 14/41] Replace plan hypotheses with measurements The plan called reducing max_detections a cheaper, lossless first move. Measured, it is neither: halving it saves 55% where the resolution factor saves 90%, and capping at 150 in a scene with 300 objects deletes 150 real detections rather than degrading them. Adds the fidelity curve against t, by object size, and states plainly what it is not. Aggregate mask IoU reads 0.908 at t=0.25 while boundary IoU is 0.654 and small objects are at 0.754, so evaluating this change on aggregate IoU alone reaches the wrong conclusion. Small objects pay roughly five times what large ones do, which makes a single recommended default workload-dependent. True AP_mask against ground truth cannot be produced in this workspace - there is no labelled set and no model weights - so the feature ships with a measured fidelity curve and no recommended value. --- docs/feat/instance-seg-tradeoff-factor.md | 45 ++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/docs/feat/instance-seg-tradeoff-factor.md b/docs/feat/instance-seg-tradeoff-factor.md index da8a4f552e..0fe0f06c9a 100644 --- a/docs/feat/instance-seg-tradeoff-factor.md +++ b/docs/feat/instance-seg-tradeoff-factor.md @@ -401,7 +401,50 @@ with 300 real objects, capping at 150 removes 150 real detections. Reducing reso object some boundary fidelity; reducing `n` costs you half the objects. They compose (n=150, t=0.25 → 29.1 ms) but they are not substitutes. -**E2 — There is no CUDA or Jetson measurement anywhere in this plan.** +**E2 — Fidelity against `t` measured; true `AP_mask` still outstanding, and why.** + +**What was measured.** Divergence of the output at `t<1` from the output at `t=1.0`, on 300 +synthetic instances spanning the COCO size strata, 1920×1080, 160×160 prototypes: + +| t | mask IoU | boundary IoU | small | medium | large | +|---|---|---|---|---|---| +| 0.10 | 0.869 | 0.514 | **0.659** | 0.888 | 0.969 | +| 0.25 | 0.908 | 0.654 | **0.754** | 0.925 | 0.980 | +| 0.50 | 0.933 | 0.737 | 0.822 | 0.945 | 0.985 | +| 0.75 | 0.938 | 0.759 | 0.832 | 0.950 | 0.986 | + +Three findings worth carrying into any default: + +- **Aggregate mask IoU hides the cost.** At `t=0.25` it reads 0.908, which sounds harmless, while + boundary IoU is 0.654 at the same point. The interior dominates the aggregate and the boundary is + precisely what resolution reduction damages. Evaluating this change on mask IoU alone will reach + the wrong conclusion. +- **Small objects pay roughly 5× what large ones do.** At `t=0.25`, large objects lose 2% and small + objects lose 25%. A single recommended default is therefore workload-dependent, not universal. +- **The curve flattens above `t=0.5`.** Going 0.5 → 0.75 buys 0.004 mask IoU for about 3× the + compute, so 0.5 is the sensible ceiling when fidelity matters. + +**What was NOT measured, and cannot be here: true `AP_mask` against ground truth.** +That requires a labelled dataset — images with annotated instance masks — run through a real model. +This workspace has neither a labelled set nor model weights, so no AP figure of any kind can be +produced from it. The table above is a *self-consistency* measurement: it compares the pipeline +against its own full-resolution output on synthetic blobs. That is the right quantity for choosing a +default whose baseline is today's behaviour, and it is **not** a substitute for AP, because it +cannot see: + +- whether a reduced mask crosses a detection's IoU threshold and changes a true positive into a + false one, which is what AP actually scores; +- how real object shapes — thin structures, concavities, occlusion boundaries — degrade compared + with the smooth synthetic lobes used here, which almost certainly understates the loss; +- any interaction with NMS, where reduced masks change overlap and therefore which detections + survive. + +*Action before any default is documented publicly:* run `AP_mask` 50:95, by object size, with +`AP_box` as an untouched control, on a fixed labelled set with a real segmentation model, sweeping +`t`. Until then the feature ships as a knob with a measured fidelity curve and no recommended value, +which is honest; publishing a default on this evidence would not be. + +**E3 — There is no CUDA or Jetson measurement anywhere in this plan.** Every figure is single-platform CPU (Apple M1 Max, 8 threads, fp32). The ~40 GB-per-4K-frame traffic figure and the derived ~392 ms on Orin NX are a **bandwidth model, not data** — and that estimate is simultaneously the strongest claim in §1 and the weakest evidence in the document. From 4a41afd894bd23c39334d48a4293d7d1c56e09d1 Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Fri, 2 Oct 2026 17:25:39 +0200 Subject: [PATCH 15/41] Bring the plan in line with what was built Phase 6 is split into what landed and what did not. Steps 1, 2 and 4 are done; the Triton kernel is gated rather than taught, and remote version skew is unhandled. Documents both deferrals with their impact. The Triton one is not free: the fused CUDA path is the fast path, so gating means t<1 falls back to the slower eager generator, and whether reduced-resolution-eager beats full-resolution-fused is unmeasured. Enabling the knob could make RF-DETR-seg slower on GPU. That needs one CUDA measurement before the knob is documented for GPU users. Narrows the version-skew item. run_locally uses the in-process model manager, so the block and the model layer are the same installation and skew cannot occur locally; only run_remotely can reach an older backend. The execution-engine compatibility guard does not cover this, since it gates engine schema semantics rather than post-processing behaviour. Drops the Author field, which git already records. --- docs/feat/instance-seg-tradeoff-factor.md | 122 +++++++++++++++++++--- 1 file changed, 105 insertions(+), 17 deletions(-) diff --git a/docs/feat/instance-seg-tradeoff-factor.md b/docs/feat/instance-seg-tradeoff-factor.md index 0fe0f06c9a..9476a08171 100644 --- a/docs/feat/instance-seg-tradeoff-factor.md +++ b/docs/feat/instance-seg-tradeoff-factor.md @@ -1,7 +1,5 @@ # Implementation plan: honour `tradeoff_factor` for instance segmentation -**Author:** Michał Taraszewski - **Related issue / PR:** _none yet — to be linked from the `#discuss-inference-release` thread_ **Status:** draft. Supersedes `instance-seg-tradeoff-factor.md`, whose headline figures combined two @@ -192,12 +190,21 @@ and 2.5× at `t=0.25`, lossless and with no API change. ### What is already done -The **dense / polygon path works end to end**: request → `map_inference_kwargs` collapses the enum -onto one factor → `align_instance_segmentation_results` interpolates the resize target → -`mask_size` travels with the prediction → polygons are scaled back into image space before being -reported. Coverage on these functions went from 31 tests to 77 in `inference_models` plus 37 in the -adapter, including characterization tests that pin geometry the previous suite structurally could -not detect. +**Both paths work end to end.** A request carrying `mask_decode_mode` / `tradeoff_factor` reaches +`map_inference_kwargs`, which collapses the enum onto one factor and consumes both source keys. The +factor reaches `align_instance_segmentation_results` and +`align_instance_segmentation_results_to_rle_masks` across all six model families — `yolov5`, +`yolov7`, `yolov8`, `yolo26`, `yolact`, `rfdetr` — on ONNX, TorchScript and TensorRT backends. +`mask_size` travels with the prediction, polygons are scaled back into image space before being +reported, and the RLE wire format declares the grid it encoded. + +Block versions up to v4 pin the outbound request to `accurate` / `1.0`, so existing workflows are +unchanged; `@v5` is the version that forwards what the caller set. + +Test counts at the time of writing: **1942** in `inference_models`, **1265** in `tests/workflows`, +**117** in `tests/inference/unit_tests/core/models`. The 12 remaining `inference_models` failures +are environmental — seven need `triton`, which ships Linux-only wheels, and five compare a +`realpath`-resolved path against an unresolved one and so cannot pass on macOS. Everything that remains is gated on the deferred RLE wire-format decision (Q3). @@ -219,26 +226,103 @@ mismatched RLE (`supervision/detection/utils/internal.py:133-138`, 0.30.6), so s consumers are unaffected. Direct `pycocotools` integrators are the exposed group, which is what the O3 experiment below quantifies. -### Phase 6, in dependency order +### Phase 6 — status + +Steps 1, 2 and 4 are built. Steps 3 and 5 remain and are deliberately deferred. + +#### Built 1. Thread `masks_resolution_factor` into `align_instance_segmentation_results_to_rle_masks`, mirroring the dense path. The per-instance generator has no chunking, so it is the simpler of the two. 2. `to_coco_rle_masks()` emits `mask_size` as the COCO `size`. One line, and the point of no return — everything before it is reversible. -3. **Teach the Triton kernel the factor rather than gating it.** `triton_postprocess.py` already +#### Deferred + +3. **Teach the Triton kernel the factor rather than gating it.** *Not done — it is gated instead.* + `_unsupported_triton_postprocess_reason` now returns `mask_resolution_factor_unsupported` when + the factor is not `1.0`, so the fused path defers to the eager one rather than silently ignoring + the request. That is correct but leaves the production GPU path unimproved, which is the one + place this work knowingly leaves value on the table. `triton_postprocess.py` already parameterises its resize tables by `output_size` (`:273-285`), so passing a reduced size through is close to free. Gating instead leaves the production GPU path — the entire edge argument — unimproved, and emits a `RuntimeWarning` per image. Gate only as a fallback if this overruns. 4. New block version (`vN` **and** `vN_tensor`), per the team's standing mechanism. -5. Version skew: an absent `mask_size` in a response means an old server, so default to the input - size. Treat the factor as a **performance hint, never an error**, so a new block version still - works against an older backend. +5. Version skew: *not done.* An absent `mask_size` in a response means an old server, so the + client should default to the input size and treat the factor as a **performance hint, never an + error**, so a new block version still works against an older backend. Needed before the remote + execution path can be relied on across mixed backend versions. Note the Triton gate deliberately belongs here and not earlier: until step 1 lands, the fused and eager paths both ignore the factor and behave identically, so a gate added now would guard against a divergence that does not yet exist and its test would assert a fallback that changes nothing. +### Deferred work, why, and its impact + +Two items from phase 6 were not built. Neither affects correctness; one has a material performance +consequence that must be stated before anyone enables the knob on GPU. + +**D1 — The Triton kernel is gated, not taught.** + +*What was done instead.* `_unsupported_triton_postprocess_reason` returns +`mask_resolution_factor_unsupported` when the factor is not `1.0`, so RF-DETR's fused CUDA +post-process defers to the eager path rather than silently ignoring the request. + +*Why deferred.* The kernel emits sparse RLE run records directly from CUDA. Honouring a reduced +target means changing its resize weight tables, ROI bounds and record emission, then proving +numerical equivalence against the eager path — on hardware. No CUDA device was available, so the +alternative was shipping an unverified GPU kernel change. + +*Impact on correctness:* none. The eager path produces the right answer at any factor. + +*Impact on performance: potentially negative on CUDA, and unmeasured.* The fused path exists +precisely because the dense intermediate is the expensive part — its own docstring says so. Gating +means that on RF-DETR-seg with CUDA, `t = 1.0` runs the **fast** fused path while `t < 1` falls back +to the **slow** per-instance eager generator at reduced resolution. Whether reduced-resolution-eager +beats full-resolution-fused is an open empirical question. **It is entirely possible that enabling +the knob makes RF-DETR-seg slower on GPU.** Nothing measured here can settle it, because the fused +path requires `triton`, which ships Linux-only wheels. + +*Recommended next step.* Before documenting the knob for GPU users, measure the four combinations on +one CUDA device: fused at `t=1.0`, eager at `t=1.0`, eager at `t=0.25`, and — once taught — fused at +`t=0.25`. If eager-at-0.25 does not beat fused-at-1.0, the knob should be documented as +CPU-and-Jetson-only until the kernel is taught. Note the fused path already bails on +`padding_unsupported`, `static_crop_unsupported` and `>4096 px`, so letterboxed 4K and static crops +are on the eager path regardless, and are unaffected by this. + +**D2 — Remote execution across mixed backend versions.** + +*Scope — narrower than it first appears.* `run_locally` calls `self._model_manager` in-process, so +the block and the model layer are the same installation: a deployment that has `@v5` necessarily has +the code that honours the factor, because they ship in one image with `roboflow-workflows` pinned by +`requirements/requirements.workflows.txt`. **Skew is structurally impossible for local execution**, +which is the majority of traffic. Only `run_remotely`, which goes out over `InferenceHTTPClient` to +an independently versioned server, can reach an older model layer. + +Note the execution-engine guard does not cover this: `get_execution_engine_compatibility` returns +`">=1.3.0,<2.0.0"`, which gates engine schema semantics, not whether the server's post-processing +honours a parameter. A server on a compatible engine with an older `inference_models` accepts the +block and drops the factor. + +*What is missing.* On that remote path, a client has no way to tell whether the server honoured the +factor. The response carries `mask_size`, so presence of that field distinguishes a new server from +an old one, but no client reads it. + +*Why deferred.* It depends on O2, which is recommended but not ratified — response-shape sniffing +versus an explicit version field — and building on an unratified decision invites rework. Scope was +the second reason, and the weaker one. + +*Impact on correctness:* none. An older backend returns full-resolution masks, which are valid. + +*Impact on performance:* the request is silently not honoured. A `@v5` block asking for `t = 0.25` +against an older backend gets no speedup, no error and no signal — a plausible "why is this slow" +ticket during any rollout window where client and server versions differ. + +*Recommended next step.* Ratify O2, then: absent `mask_size` ⇒ assume the old contract and default +to the input size; present ⇒ use it. Treat the factor as a **performance hint, never an error**, and +log once per session when it was not honoured. Roughly half a day, CPU-testable, and it closes a +silent-failure mode. + ### Unblocked work that should not wait A separate PR, mergeable today, independent of the decision: @@ -349,7 +433,10 @@ therefore not just safe, it is the fix. on an exact 4× grid, so it understates boundary loss. A real contour will lose fidelity; that is what the `AP_mask` curve measures, and it is a separate question from whether decoding breaks. -**O4 — The GPU path: gate the fused kernel, or teach it `t`?** +**O4 — The GPU path: decided — gated, with teaching the kernel as a follow-up.** +*Resolved during implementation.* `_unsupported_triton_postprocess_reason` returns +`mask_resolution_factor_unsupported` when the factor is not `1.0`, so the fused path defers to +the eager one. Chosen for scope, not merit: the better answer is below and remains open. *Investigation:* `models/rfdetr/triton_postprocess.py` already implements a **lossless** version of this optimisation — *"asks Triton to interpolate only the active mask region and emit sparse RLE run records directly"* — and already parameterises resize tables by `output_size` (`:273-285`). Forcing @@ -659,10 +746,11 @@ def test_reduced_resolution_forces_eager_fallback(self) -> None: **Commit 5.** -### Phase 6 — RLE parity +### Phase 6 — RLE parity *(done)* -Blocked on the deferred wire-format decision. Until then the RLE path keeps `t=1.0` and phase 0's -characterization tests guard it. The evidence experiment (plan O3) belongs here: +The wire-format question was settled by measurement rather than by waiting — see O3. The RLE +generator now takes the factor, `to_coco_rle_masks` declares the encoded grid, and the Triton gate +defers when the factor is not `1.0`. The evidence experiment that unblocked it: ```python @pytest.mark.parametrize("decoder", ["pycocotools", "supervision"]) From 59425d2936b44ceb4f8dddd9a8c524707360d48c Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Fri, 2 Oct 2026 19:31:40 +0200 Subject: [PATCH 16/41] Address review: correct the reduced-resolution path end to end Four blockers and three majors from review, each reproduced before the fix and verified after. RLE-backed polygons were decoded on image_size while the counts describe the encoded grid. The adapter requests RLE whenever the model supports it, so this was the default polygon route and returned unusable contours. Both decoders now use the encoded grid. Static crop with a reduced factor could crash: the canvas size and the crop offset were rounded independently, leaving the canvas a pixel short of the pasted extent. The canvas now contains that extent. Only crops reaching the image edge under letterbox scaling expose it, which is why a sweep with zero padding found nothing. A static crop anchored at (0, 0) yields a mask smaller than the image whose coordinates are already image-space. Inferring "reduced grid" from mask_size != image_size scaled those a second time at the default factor. Scaling is now relative to size_after_pre_processing. The tensor-native local call sites were unpinned in v1-v4. All four are pinned. v5_tensor is pinned too: nothing under workflows/ reads mask_size and the sites that rebuild InstancesRLEMasks drop it, so reduced resolution is available in non-tensor mode only. The pin comment claimed these fields never took effect. That is false on the legacy ORT backend, which honours all three modes and is the default on Windows. Corrected in all twelve places and recorded in the changelog as a deliberate behaviour change there. The Triton dispatcher dropped the factor, so its gate was unreachable. to_supervision returned a reduced mask beside image-space boxes, and __iter__ declared the image size for counts on the encoded grid. Zero-detection results reported a shape the populated path would never produce. Both changelogs added. Plan document removed from the published docs tree. Formatter-only churn in three unrelated files reverted. Tests rewritten around the review's central point, that the first set could not fail: 24 golden cases captured from origin/main with non-zero padding throughout, dense-versus-RLE content comparison, the static-crop crash combination, RLE-backed polygons at reduced resolution, and a dispatcher spy rather than a direct call to the gate. Tests: inference_models 2016 passed, workflows 1274 passed, inference core/models 121 passed, flake8 clean. --- docs/feat/instance-seg-tradeoff-factor.md | 776 ------------------ .../core/models/inference_models_adapters.py | 16 +- inference/core/utils/rle_to_polygon.py | 3 +- inference_models/docs/changelog.md | 34 + inference_models/inference_models/__init__.py | 7 +- .../models/base/instance_segmentation.py | 40 +- .../models/common/roboflow/post_processing.py | 80 +- .../inference_models/models/rfdetr/common.py | 1 + .../weights_providers/offline_registry.py | 3 +- .../models/base/test_instance_segmentation.py | 96 +++ .../common/roboflow/test_post_processing.py | 273 +++++- .../models/rfdetr/test_triton_postprocess.py | 36 + .../unit_tests/models/test_mage_vl_hf.py | 10 +- .../models/test_inference_models_adapters.py | 102 ++- .../test_v5_mask_resolution.py | 64 +- workflows/CHANGELOG.md | 20 + .../roboflow/instance_segmentation/v1.py | 14 +- .../instance_segmentation/v1_tensor.py | 14 +- .../roboflow/instance_segmentation/v2.py | 14 +- .../instance_segmentation/v2_tensor.py | 14 +- .../roboflow/instance_segmentation/v3.py | 14 +- .../instance_segmentation/v3_tensor.py | 14 +- .../roboflow/instance_segmentation/v4.py | 14 +- .../instance_segmentation/v4_tensor.py | 14 +- .../instance_segmentation/v5_tensor.py | 22 +- .../execution_engine/entities/types.py | 6 + 26 files changed, 844 insertions(+), 857 deletions(-) delete mode 100644 docs/feat/instance-seg-tradeoff-factor.md diff --git a/docs/feat/instance-seg-tradeoff-factor.md b/docs/feat/instance-seg-tradeoff-factor.md deleted file mode 100644 index 9476a08171..0000000000 --- a/docs/feat/instance-seg-tradeoff-factor.md +++ /dev/null @@ -1,776 +0,0 @@ -# Implementation plan: honour `tradeoff_factor` for instance segmentation - -**Related issue / PR:** _none yet — to be linked from the `#discuss-inference-release` thread_ - -**Status:** draft. Supersedes `instance-seg-tradeoff-factor.md`, whose headline figures combined two -code paths that never execute together and whose recommended v1 is measurably slower than today. - -> **Reviewed by the team 2026-10-01.** The delivery mechanism is settled (new block versions); the -> RLE wire-format decision is deliberately deferred to the end of implementation. §3 records what was -> decided and what is still open. - ---- - -## 1. What problem are we solving? - -**Who.** Anyone running instance segmentation on large frames or dense scenes: real-time video, and -Jetson deployments in particular. - -**What is missing.** `mask_decode_mode` and `tradeoff_factor` are accepted end to end — SDK entity → -Workflows block → `WorkflowsModelsProvider.run_instance_segmentation` → -`InstanceSegmentationInferenceRequest` — and then ignored. They pass untouched through -`InferenceModelsInstanceSegmentationAdapter.map_inference_kwargs` -(`inference/core/models/inference_models_adapters.py:520`) and are swallowed by `**kwargs` in each -model's `post_process`. No error, no effect. Masks are always produced at full input resolution. -The legacy path (`inference/core/models/instance_segmentation_base.py:143-190`) still implements all -three modes. - -**Why it matters: this stage is almost the entire frame.** Measured on one host (M1 Max, 8 torch -threads, fp32, 3840×2160, p50): JPEG decode 59.7 ms, preprocess to 640×640 0.26 ms, JPEG encode -35.1 ms — everything that is *not* mask post-processing sums to roughly **95 ms**. Mask -post-processing at `n=300` is **hundreds of milliseconds to seconds** (tables below). At 4K with a -dense scene this stage is **~90–97% of total frame time**. That, not a stage-relative speedup, is -the argument. - -**Two things this plan does not claim.** It is not sufficient on its own — at `n=300`/4K even -`t=0.25` leaves several hundred ms in this stage. And cost is Θ(n·H·W), so **reducing -`max_detections`** (default 300, `inference_models/configuration.py:229`) is the obvious -alternative. It was measured, and it is the weaker lever: halving it saves 55% where the resolution -factor saves 90% — **~4.6× less effective** — and it cannot reach real-time at 4K at any realistic -value. It is also **not lossless**: capping at 150 in a scene with 300 objects deletes 150 real -detections, a recall cliff rather than the uniform boundary-fidelity cost that reducing resolution -carries. See E1. - -### The published contract - -The `inference.roboflow.com` reference site was collapsed to a landing page in `cbdd286c4`, so the -mkdocstrings API pages no longer exist. What remains published: - -- `mask_decode_mode` default `accurate`; `tradeoff_factor` default `0`, scale `0='fast'` → `1='accurate'`. -- *"If 'accurate' the mask will be decoded using the original image size. If 'fast' the mask will be - decoded using the original mask size."* — the only published statement about output resolution, - and therefore binding. -- `response_mask_format` default `polygon`, `rle` opt-in. - -The request schema (`inference/core/entities/requests/inference.py:254-257`) publishes all three -modes. Only the legacy query parameter (`inference/core/interfaces/http/http_api.py:4738`) -enumerates just two, omitting `tradeoff` — worth correcting, but it is a doc fix, not a blocker. - -## 2. How will the behaviour change? - -One float, `masks_resolution_factor ∈ [0,1]` (named to match the existing `masks_smoothing_enabled` -/ `masks_binarization_threshold` kwargs), resolved in the adapter from the existing enum, and -**carried as explicit metadata** rather than inferred from array shape. - -Prior art for exactly this: DeepStream's `NvOSD_MaskParams` carries mask-local `height`/`width` -separately from `rect_params` and rescales on demand -(`_reference/deepstream_python_apps/bindings/docstrings/nvosddoc.h:278-291`; -`apps/deepstream-segmask/deepstream_segmask.py:120-126`). - -### Before — 4K frame, 300 instances, 160×160 protos - -```mermaid -flowchart LR - A["protos: 300 x 160 x 160"] --> B["crop to boxes, at proto res"] - B --> C["resize every instance to 2160 x 3840"] - C --> D["bool 300 x 2160 x 3840"] - D --> E["masks2poly or RLE encode"] - E --> F["polygon coords in 2160x3840
RLE size 2160x3840"] -``` - -### After, at `t = 0.25` - -```mermaid -flowchart LR - A["protos: 300 x 160 x 160"] --> B["crop to boxes, at proto res"] - B --> C["resize to 660 x 1080"] - C --> D["bool 300 x 660 x 1080
+ mask canvas 660x1080"] - D --> E["masks2poly or RLE encode
at 660 x 1080"] - E --> F["scale coords x3.27
-> image space 2160x3840"] -``` - -Adapter mapping: `accurate → 1.0`, `fast → 0.0`, `tradeoff → tradeoff_factor` (validated, raising -`InvalidMaskDecodeArgument` as the legacy path does). This satisfies the published contract exactly. -Default `1.0` leaves output unchanged. - -### Step F is the change. Without it every prediction at `t<1` is wrong. - -There is **no mask→image coordinate scaling anywhere** in the `inference_models` response path. The -adapter emits raw `masks2poly` output into `Point(x, y)` -(`inference_models_adapters.py:917-988`, `:1047`) while declaring the image at `original_size`. Two -sites are worse: `bitpacked_masks2poly(..., width=W)` with `W` from `original_size` (`:907`, -`core/utils/postprocess.py:73`) unpacks at the wrong stride and corrupts silently; and -`InstancesRLEMasks` declares `image_size = original_size` while encoding at -`size_after_pre_processing` (`rfdetr/common.py:383`). - -The legacy path does this correctly — it tracks `output_mask_shape` per mode and rescales via -`post_process_polygons` (`inference/core/utils/postprocess.py:449`). **The `inference_models` -rewrite dropped that step.** We are restoring a mechanism the codebase already had. - -Verified consequences if reduced masks escape unscaled (4 confirmed; a wider sweep is in progress -and will be attached rather than asserted): `visualizations/mask/v1_tensor.py:277-280` raises -outright; `bounding_rect/v1.py:163-177` overwrites `xyxy` from mask contours, so corrupted boxes -reach blocks that never touch masks; `dynamic_zones/v1.py:348-352` emits shrunken zone polygons; -`sinks/dataset_upload/v1_tensor.py:946-951` persists mask-space polygons as permanent labels. -Note `sv.Detections.from_inference` already resizes mismatched RLE -(`supervision/detection/utils/internal.py:133-138`, supervision 0.30.6) but not polygons, -which are rasterised as given onto an image-size canvas (`:430`). - -> Supervision citations in this document are against **0.30.6**, the version this branch pins -> (`supervision>=0.30.6,<0.31.0`). The 0.29 line numbers differ; 0.30 changed `__setitem__` -> validation, `from_inference` polygon skips, PolygonZone anchors, CompactMask degeneracy and NMM -> tie-breaks, so references taken from a 0.29 checkout do not transfer. - -### Measurements, by path - -Two different functions. They are not comparable and must not be combined. - -**Dense path** — `align_instance_segmentation_results` (`post_processing.py:395`), chunked, -materialises `n×H×W`. 4K, n=300, chunk 16, fp32, CPU: - -| t | resize p50 | peak RSS | -|---|---|---| -| 1.0 | 485–575 ms (3 runs) | 3256 MB | -| 0.25 | 52–65 ms | 595 MB | - -**RLE path** — `align_instance_segmentation_results_to_rle_masks` (`post_processing.py:525`), a -**per-instance generator**: no chunking, no `n×H×W` tensor, so the memory figures above do not -apply. This is the default (`v4.py:359`, `:408` hardcode `"rle"`; `GCP_SERVERLESS` forces it, -`inference_models_adapters.py:527`). Encode cost is strongly content-dependent — measured 2933 ms -for smooth blob masks vs 6262 ms for noise masks at n=300, a 2.1× spread from fixture alone. - -**Open measurement gaps, to close before implementation.** The polygon-path figure in the superseded -plan did not reproduce (`masks2poly` measured ~1877 ms against ~314 ms implied) and is withdrawn. -All numbers above are single-platform CPU; there is **no CUDA or Jetson measurement**, and the -~40 GB/frame traffic figure for Orin NX is a bandwidth model, not data. The benchmark scripts will -be linked, with fixture content stated. - -### How we verify - -`align_instance_segmentation_results` and `crop_masks_to_boxes` currently have **no unit tests** -(`inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py` covers neither), -so a characterization test — fixed-seed protos → golden mask tensor and golden polygons — is -commit zero, before any behaviour claim can be checked in CI. - -- Bit-identical output at `t=1.0`. -- **Polygon/RLE coordinate round-trip at `t<1`**: emitted coordinates land on the same image pixels - as `t=1.0`. Tolerance must scale with `1/t`, since unpad rounding error is amplified as `t` falls. - Shape-only assertions would not catch the defect above; this is the test that matters. -- RLE invariant: decoded counts sum to the declared size; declared size is the encoded size. -- Non-square fixtures throughout (square images hide axis errors); odd dims; letterbox *and* - stretch; static crop with offset; `n = 0`. -- Triton gate as a pure-Python unit test on `_unsupported_triton_postprocess_reason`, so it is not a - GPU-only dead test in x86 CI. -- `AP_mask` 50:95 overall **and by object size**, against `t`, with `AP_box` as control. - -Layers: `inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py`; -`tests/inference/models_predictions_tests`; `tests/workflows/unit_tests/.../instance_segmentation/` -for any new block version **and its `_tensor` sibling**. - -### Implementation surface - -Not one function. The alignment logic is duplicated across two functions, reached from six model -families on both the dense and RLE routes — `models/{yolov5,yolov7,yolov8,yolo26,yolact}/common.py` -× {dense, rle} = 10 call sites, plus `models/rfdetr/common.py:369` and -`models/rfdetr/triton_postprocess.py` — plus the new kwarg threaded through every -`*_instance_segmentation_{onnx,trt,torch_script}.py`. Roughly 12 sites beyond the two core -functions. - -### Out of scope, as its own plan - -Four independent performance and correctness changes, measured and bit-identical except the last, -belong in a separate PR and are deliberately not bundled here: `torch.gt(out=)` in place of -`.gt_()` + assignment; separable `crop_masks_to_boxes`; resizing directly into the static-crop -canvas; and deriving `crop_masks_to_boxes(scaling=0.25)` per-axis (wrong whenever -`mask_w/inference_w ≠ 0.25`, e.g. YOLACT 138/550 — and the only one of the four that changes -`t=1.0` output). Also out of scope: batching the per-instance RLE encode, measured 1.4× at `t=1.0` -and 2.5× at `t=0.25`, lossless and with no API change. - -## 2b. Sequencing from here - -### What is already done - -**Both paths work end to end.** A request carrying `mask_decode_mode` / `tradeoff_factor` reaches -`map_inference_kwargs`, which collapses the enum onto one factor and consumes both source keys. The -factor reaches `align_instance_segmentation_results` and -`align_instance_segmentation_results_to_rle_masks` across all six model families — `yolov5`, -`yolov7`, `yolov8`, `yolo26`, `yolact`, `rfdetr` — on ONNX, TorchScript and TensorRT backends. -`mask_size` travels with the prediction, polygons are scaled back into image space before being -reported, and the RLE wire format declares the grid it encoded. - -Block versions up to v4 pin the outbound request to `accurate` / `1.0`, so existing workflows are -unchanged; `@v5` is the version that forwards what the caller set. - -Test counts at the time of writing: **1942** in `inference_models`, **1265** in `tests/workflows`, -**117** in `tests/inference/unit_tests/core/models`. The 12 remaining `inference_models` failures -are environmental — seven need `triton`, which ships Linux-only wheels, and five compare a -`realpath`-resolved path against an unresolved one and so cannot pass on macOS. - -Everything that remains is gated on the deferred RLE wire-format decision (Q3). - -### The decision is narrower than three options - -Measurement has already eliminated one, and the remaining two are not equally viable: - -| option | outcome | -|---|---| -| upscale before encode | 814 ms against today's 485 ms, encode unchanged. **A net regression.** | -| dense-only in v1 | Safe, but `v4.py` hardcodes `response_mask_format="rle"` and `GCP_SERVERLESS` forces it, so the feature is **unreachable for most traffic** — it would work only with `enforce_dense_masks_in_inference_models=True`. | -| reduced RLE `size` | The only option that delivers the saving on the path that is actually used. | - -So the question to put to the team is not "which of three" but: **are we willing to change the RLE -`size` field, given that otherwise the feature is inert for most users?** - -Client exposure is narrower than it first appears. `sv.Detections.from_inference` already resizes a -mismatched RLE (`supervision/detection/utils/internal.py:133-138`, 0.30.6), so supervision-based -consumers are unaffected. Direct `pycocotools` integrators are the exposed group, which is what the -O3 experiment below quantifies. - -### Phase 6 — status - -Steps 1, 2 and 4 are built. Steps 3 and 5 remain and are deliberately deferred. - -#### Built - -1. Thread `masks_resolution_factor` into `align_instance_segmentation_results_to_rle_masks`, - mirroring the dense path. The per-instance generator has no chunking, so it is the simpler of - the two. -2. `to_coco_rle_masks()` emits `mask_size` as the COCO `size`. One line, and the point of no - return — everything before it is reversible. -#### Deferred - -3. **Teach the Triton kernel the factor rather than gating it.** *Not done — it is gated instead.* - `_unsupported_triton_postprocess_reason` now returns `mask_resolution_factor_unsupported` when - the factor is not `1.0`, so the fused path defers to the eager one rather than silently ignoring - the request. That is correct but leaves the production GPU path unimproved, which is the one - place this work knowingly leaves value on the table. `triton_postprocess.py` already - parameterises its resize tables by `output_size` (`:273-285`), so passing a reduced size through - is close to free. Gating instead leaves the production GPU path — the entire edge argument — - unimproved, and emits a `RuntimeWarning` per image. Gate only as a fallback if this overruns. -4. New block version (`vN` **and** `vN_tensor`), per the team's standing mechanism. -5. Version skew: *not done.* An absent `mask_size` in a response means an old server, so the - client should default to the input size and treat the factor as a **performance hint, never an - error**, so a new block version still works against an older backend. Needed before the remote - execution path can be relied on across mixed backend versions. - -Note the Triton gate deliberately belongs here and not earlier: until step 1 lands, the fused and -eager paths both ignore the factor and behave identically, so a gate added now would guard against -a divergence that does not yet exist and its test would assert a fallback that changes nothing. - -### Deferred work, why, and its impact - -Two items from phase 6 were not built. Neither affects correctness; one has a material performance -consequence that must be stated before anyone enables the knob on GPU. - -**D1 — The Triton kernel is gated, not taught.** - -*What was done instead.* `_unsupported_triton_postprocess_reason` returns -`mask_resolution_factor_unsupported` when the factor is not `1.0`, so RF-DETR's fused CUDA -post-process defers to the eager path rather than silently ignoring the request. - -*Why deferred.* The kernel emits sparse RLE run records directly from CUDA. Honouring a reduced -target means changing its resize weight tables, ROI bounds and record emission, then proving -numerical equivalence against the eager path — on hardware. No CUDA device was available, so the -alternative was shipping an unverified GPU kernel change. - -*Impact on correctness:* none. The eager path produces the right answer at any factor. - -*Impact on performance: potentially negative on CUDA, and unmeasured.* The fused path exists -precisely because the dense intermediate is the expensive part — its own docstring says so. Gating -means that on RF-DETR-seg with CUDA, `t = 1.0` runs the **fast** fused path while `t < 1` falls back -to the **slow** per-instance eager generator at reduced resolution. Whether reduced-resolution-eager -beats full-resolution-fused is an open empirical question. **It is entirely possible that enabling -the knob makes RF-DETR-seg slower on GPU.** Nothing measured here can settle it, because the fused -path requires `triton`, which ships Linux-only wheels. - -*Recommended next step.* Before documenting the knob for GPU users, measure the four combinations on -one CUDA device: fused at `t=1.0`, eager at `t=1.0`, eager at `t=0.25`, and — once taught — fused at -`t=0.25`. If eager-at-0.25 does not beat fused-at-1.0, the knob should be documented as -CPU-and-Jetson-only until the kernel is taught. Note the fused path already bails on -`padding_unsupported`, `static_crop_unsupported` and `>4096 px`, so letterboxed 4K and static crops -are on the eager path regardless, and are unaffected by this. - -**D2 — Remote execution across mixed backend versions.** - -*Scope — narrower than it first appears.* `run_locally` calls `self._model_manager` in-process, so -the block and the model layer are the same installation: a deployment that has `@v5` necessarily has -the code that honours the factor, because they ship in one image with `roboflow-workflows` pinned by -`requirements/requirements.workflows.txt`. **Skew is structurally impossible for local execution**, -which is the majority of traffic. Only `run_remotely`, which goes out over `InferenceHTTPClient` to -an independently versioned server, can reach an older model layer. - -Note the execution-engine guard does not cover this: `get_execution_engine_compatibility` returns -`">=1.3.0,<2.0.0"`, which gates engine schema semantics, not whether the server's post-processing -honours a parameter. A server on a compatible engine with an older `inference_models` accepts the -block and drops the factor. - -*What is missing.* On that remote path, a client has no way to tell whether the server honoured the -factor. The response carries `mask_size`, so presence of that field distinguishes a new server from -an old one, but no client reads it. - -*Why deferred.* It depends on O2, which is recommended but not ratified — response-shape sniffing -versus an explicit version field — and building on an unratified decision invites rework. Scope was -the second reason, and the weaker one. - -*Impact on correctness:* none. An older backend returns full-resolution masks, which are valid. - -*Impact on performance:* the request is silently not honoured. A `@v5` block asking for `t = 0.25` -against an older backend gets no speedup, no error and no signal — a plausible "why is this slow" -ticket during any rollout window where client and server versions differ. - -*Recommended next step.* Ratify O2, then: absent `mask_size` ⇒ assume the old contract and default -to the input size; present ⇒ use it. Treat the factor as a **performance hint, never an error**, and -log once per session when it was not honoured. Roughly half a day, CPU-testable, and it closes a -silent-failure mode. - -### Unblocked work that should not wait - -A separate PR, mergeable today, independent of the decision: - -- `torch.gt(..., out=)` in place of `.gt_()` plus assignment — 1.07–1.51×, bit-identical. -- Separable `crop_masks_to_boxes` — 1.9–2.2×, `torch.equal`-identical. -- Resize directly into the static-crop canvas — removes a full `n×H×W` bool buffer. -- **Batched RLE encode** — 1.4× at `t=1.0`, 2.5× at `t=0.25`, lossless, no API change, and it - removes roughly 300 device synchronisations per frame. This is the standout: it attacks the - 2531 ms encode directly and helps whichever way Q3 is decided. -- `crop_masks_to_boxes(scaling=0.25)` derived per axis — correctness; changes `t=1.0` output, so it - needs its own commit and must not be bundled with any bit-identity claim. - -### Recommended order of work - -**Before the decision.** Run the O3 decode experiment, then send the team the plan with that -evidence and the question framed as above. In parallel, open the refactors PR. - -**After the decision.** Phase 6 as sequenced; steps 1–2 are roughly a day, step 3 longer. - -**Independently.** The `AP_mask`-against-`t` curve and one CUDA measurement. Neither blocks -implementation; both block the *recommendation*, and without them there is no defensible default -to ship. - -## 3. Decisions taken, and what remains - -### Settled by team review, 2026-10-01 - -Question numbers below are those of the reviewed draft, so the thread stays traceable. - -**(Q1) New block versions are the delivery mechanism.** Changes of this kind have always shipped as -new block versions to minimise breakage — *even when the change is a fix to an earlier bug* — and -code duplication across versions is explicitly acceptable. This removes the main compatibility -objection: a new version means no existing pipeline changes under upgrade. Both the `vN` and -`vN_tensor` siblings are in scope. - -**(Q2) `mask_size` as metadata carried with the prediction: accepted in principle.** Two -sub-decisions remain, below. - -**(Q2a) The carrier is an explicit field, not a metadata key.** Decided 2026-10-01. - -``` -InstancesRLEMasks: image_size: tuple[int, int] # unchanged - the image - mask_size: tuple[int, int] = image_size # new, defaults to today's meaning - -InstanceDetections: mask_size: tuple[int, int] = mask.shape[1:] # new, defaults to today's meaning -``` - -*Why.* The default is what makes this cheap: every existing construction stays valid, every reader -that ignores `mask_size` gets today's behaviour, and the changelog entry is an addition rather than -a change. The public-API cost is paid once, in a package that is versioned separately anyway. - -*Why not a metadata key.* The failure mode this whole plan exists to prevent is silent corruption, -and an unvalidated metadata key is exactly the thing a generic transform drops without anyone -noticing. A sibling field sits next to `image_size`, where all ~15 readers already look; a key sits -somewhere they do not. And `InstanceDetections` has no metadata carrier for dense masks at all, so -the key route means inventing one — strictly more work than adding the field. - -*Corroborating signal.* The team's own (Q1) answer accepts duplicated block versions to avoid -breakage. That is a preference for explicit and slightly verbose over clever and implicit. - -**(Q3) The RLE wire-format decision is deferred to the end of implementation**, for wider team -discussion. The reasoning, which changes this plan's shape: *with new block versions this is not -breaking for Workflows at all.* The breaking surface is narrower than the reviewed draft assumed — -it is the **model endpoints**, where a direct integrator would suddenly see mask-scaling parameters -respected. That is a bug fix, and it is breaking, and those are not in conflict. Implementation -should proceed without settling it. - -**(Q4) The chunk byte budget is a follow-up, not part of this work.** It addresses an extremum of -the resolution range; it can land later purely as serving-latency optimisation. - -**(Q5) The accuracy/latency curve may be worth writing up publicly** — a blog post rather than a -docs change. - -### Still open - -**O2 — Remote execution across mixed backend versions.** *(raised in review; not in the reviewed draft)* -*Question:* a client may talk to either a new server that returns `mask_size` or an old one that does -not. What does the client do? -*Investigation:* the response shape itself is the version signal — presence of the new field -distinguishes a new server from an old one, with no version negotiation needed. -*Recommendation:* absent field ⇒ assume the old contract and default `mask_size` to the input size; -present ⇒ use the explicit value. This must hold for both remote Workflows execution and the SDK. -*Input needed:* confirmation that response-shape sniffing is acceptable rather than an explicit -version field. **Blocking for the remote path.** - -**O3 — Evidence for the deferred Q3 decision: measured, and it inverts the risk.** -The team asked how badly RLE decoding breaks under a reduced `size`. Measured on -supervision 0.30.6 and pycocotools: - -| case | result | -|---|---| -| `pycocotools.decode` on an **honest** reduced `size` | decodes cleanly to the reduced canvas. No exception, correct mask. | -| `pycocotools.decode` on a **lying** `size` (reduced counts, image size declared) | decodes **silently wrong**. No exception. | -| `sv.Detections.from_inference` on an honest reduced `size` | **auto-resizes to the image.** Mask arrives at `(1, 400, 600)` as if nothing happened. | - -The conclusion is the opposite of the intuition the question was built on. Emitting an honest -reduced `size` is the **safe** option: supervision consumers are transparently unaffected, and a -direct `pycocotools` consumer gets a correct mask on a smaller canvas, which is exactly what the -published contract already promises for `fast` ("decoded using the original mask size"). - -The dangerous case is the one **currently in the code**: `InstancesRLEMasks` declares -`image_size = original_size` while encoding at `size_after_pre_processing`. The moment those -differ, that is the lying case — silent corruption with no exception. Honesty about the size is -therefore not just safe, it is the fix. - -*Caveat on fidelity:* the round-trip IoU of 1.0 in this experiment used an axis-aligned rectangle -on an exact 4× grid, so it understates boundary loss. A real contour will lose fidelity; that is -what the `AP_mask` curve measures, and it is a separate question from whether decoding breaks. - -**O4 — The GPU path: decided — gated, with teaching the kernel as a follow-up.** -*Resolved during implementation.* `_unsupported_triton_postprocess_reason` returns -`mask_resolution_factor_unsupported` when the factor is not `1.0`, so the fused path defers to -the eager one. Chosen for scope, not merit: the better answer is below and remains open. -*Investigation:* `models/rfdetr/triton_postprocess.py` already implements a **lossless** version of -this optimisation — *"asks Triton to interpolate only the active mask region and emit sparse RLE run -records directly"* — and already parameterises resize tables by `output_size` (`:273-285`). Forcing -its eager fallback leaves the production GPU path unimproved and emits a `RuntimeWarning` **per -image** (`:253-258`), i.e. per-frame log spam. The fused path already bails on -`padding_unsupported`, `static_crop_unsupported`, `resize_metadata_unsupported` and `>4096 px` -(`:921-983`), so letterboxed 4K, static crops and Jetson-without-triton already fall through to the -slow path — which is the real justification for this feature and belongs in §1. -*Recommendation:* pass the reduced `output_height/width` into the kernel. If too large for v1, add -`t != 1.0` to `_unsupported_triton_postprocess_reason` so the fallback is quiet, and state plainly -that the GPU fast path is unimproved in v1. -*Input needed:* scope call. Non-blocking for the dense path; blocking for any Jetson claim. - -**O5 — Legacy-path parity.** After this change `mask_decode_mode="tradeoff"` returns transposed -masks under `USE_INFERENCE_MODELS=False` — `process_mask_tradeoff` passes `(h, w)` where -`cv2.resize` wants `(width, height)` (`core/utils/postprocess.py:353`; verified by execution, -1920×1080 at `t=1.0` returns `(n, 1920, 1080)`) — and correct masks under the default. A -flag-dependent geometry divergence. *Recommendation:* fix the transpose in the same PR; it is four -lines. Non-blocking. - -**O6 — Stream behaviour.** `tradeoff_factor` is bindable to `$inputs.*`, so `t` can change between -frames of one session, while `stabilize_detections/v1_tensor.py:447-451` and tracker caches derive -empty-mask shapes from `template.mask.shape[1:]`. *Recommendation:* treat `t` as session-fixed and -reject mid-stream changes. Non-blocking. - -### Unmeasured — evidence gaps - -Neither of these blocks implementation. Both block the plan's *justification*, and the first could -reduce its scope substantially. - -**E1 — `max_detections` measured; it does not reframe the plan.** -4K, 300 instances, 160×160 prototypes, M1 Max CPU fp32, chunk 16, fresh process per configuration, -warm-up 2, 10 iterations, p50 ms: - -| n | t=0.25 | t=0.5 | t=1.0 | RSS at t=1.0 | -|---|---|---|---|---| -| 75 | 16.4 | 45.7 | 140.1 | 1632 MB | -| 150 | 29.1 | 79.0 | 260.5 | 2474 MB | -| 300 | 56.1 | 155.1 | 576.4 | 3522 MB | - -From the same starting point (n=300, t=1.0, 576.4 ms): halving `max_detections` gives 260.5 ms -(55% saved); dropping `t` to 0.25 gives 56.1 ms (90% saved). The resolution lever is ~4.6× more -effective, and the ordering holds for memory (1.4× against 4.8×). **No realistic `max_detections` -reaches real-time**: a 30 fps budget is ~33 ms for the whole frame, and at `t=1.0` even n=75 costs -140 ms in this stage alone — you would need n≈18. - -The accuracy argument also runs the other way. "Lossless on the top 150" is tautological: in a scene -with 300 real objects, capping at 150 removes 150 real detections. Reducing resolution costs every -object some boundary fidelity; reducing `n` costs you half the objects. They compose -(n=150, t=0.25 → 29.1 ms) but they are not substitutes. - -**E2 — Fidelity against `t` measured; true `AP_mask` still outstanding, and why.** - -**What was measured.** Divergence of the output at `t<1` from the output at `t=1.0`, on 300 -synthetic instances spanning the COCO size strata, 1920×1080, 160×160 prototypes: - -| t | mask IoU | boundary IoU | small | medium | large | -|---|---|---|---|---|---| -| 0.10 | 0.869 | 0.514 | **0.659** | 0.888 | 0.969 | -| 0.25 | 0.908 | 0.654 | **0.754** | 0.925 | 0.980 | -| 0.50 | 0.933 | 0.737 | 0.822 | 0.945 | 0.985 | -| 0.75 | 0.938 | 0.759 | 0.832 | 0.950 | 0.986 | - -Three findings worth carrying into any default: - -- **Aggregate mask IoU hides the cost.** At `t=0.25` it reads 0.908, which sounds harmless, while - boundary IoU is 0.654 at the same point. The interior dominates the aggregate and the boundary is - precisely what resolution reduction damages. Evaluating this change on mask IoU alone will reach - the wrong conclusion. -- **Small objects pay roughly 5× what large ones do.** At `t=0.25`, large objects lose 2% and small - objects lose 25%. A single recommended default is therefore workload-dependent, not universal. -- **The curve flattens above `t=0.5`.** Going 0.5 → 0.75 buys 0.004 mask IoU for about 3× the - compute, so 0.5 is the sensible ceiling when fidelity matters. - -**What was NOT measured, and cannot be here: true `AP_mask` against ground truth.** -That requires a labelled dataset — images with annotated instance masks — run through a real model. -This workspace has neither a labelled set nor model weights, so no AP figure of any kind can be -produced from it. The table above is a *self-consistency* measurement: it compares the pipeline -against its own full-resolution output on synthetic blobs. That is the right quantity for choosing a -default whose baseline is today's behaviour, and it is **not** a substitute for AP, because it -cannot see: - -- whether a reduced mask crosses a detection's IoU threshold and changes a true positive into a - false one, which is what AP actually scores; -- how real object shapes — thin structures, concavities, occlusion boundaries — degrade compared - with the smooth synthetic lobes used here, which almost certainly understates the loss; -- any interaction with NMS, where reduced masks change overlap and therefore which detections - survive. - -*Action before any default is documented publicly:* run `AP_mask` 50:95, by object size, with -`AP_box` as an untouched control, on a fixed labelled set with a real segmentation model, sweeping -`t`. Until then the feature ships as a knob with a measured fidelity curve and no recommended value, -which is honest; publishing a default on this evidence would not be. - -**E3 — There is no CUDA or Jetson measurement anywhere in this plan.** -Every figure is single-platform CPU (Apple M1 Max, 8 threads, fp32). The ~40 GB-per-4K-frame traffic -figure and the derived ~392 ms on Orin NX are a **bandwidth model, not data** — and that estimate is -simultaneously the strongest claim in §1 and the weakest evidence in the document. -Two CUDA-specific risks are also unverified: a per-request factor makes the post-process output -shape request-dependent, which may fragment the caching allocator on an 8 GB Orin and defeats the -pinned staging buffer that `triton_postprocess.py:191-218` keys on source shape. The likely -mitigation — quantise `t` to a small set, e.g. steps of 0.125, so the shape space stays bounded — is -untested. -*Action:* one end-to-end frame-time profile on an Orin NX, or failing that one discrete NVIDIA GPU, -measuring the function that actually runs on that path rather than its dense sibling. -*Why it matters:* "Jetson deployments above all" currently rests on arithmetic. One run converts the -plan's weakest evidence into its best. - -### Follow-ups, not decisions - -Recommend a default `t` once the `AP_mask` curve exists; the chunk byte budget (dense path only — -the RLE generator has no chunking); batching the per-instance RLE encode (measured 1.4× at `t=1.0`, -2.5× at `t=0.25`, lossless, no API change); fixing the `Defaults to 0.5` docstring at -`instance_segmentation_base.py:78` against `DEFAULT_TRADEOFF_FACTOR = 0.0` on line 33; adding -`tradeoff` to the legacy query-param description at `http_api.py:4738`. - -**Closed by investigation.** `fast` needs no separate code path: `t=0.0` *is* legacy -`process_mask_fast` semantics, because the crop already happens at proto resolution before -`align_instance_segmentation_results` (`models/yolov8/common.py:32`) and `t=0` skips the resize. - ---- - -## Appendix — TDD sequence - -Each phase is red → green → commit. - -### Existing coverage, measured not assumed - -`inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py` (28 tests) **does** -cover `align_instance_segmentation_results`, via `test_chunked_resize_matches_monolithic` (:502) and -`..._with_static_crop_canvas`. But both are **invariance** tests — they assert chunked output equals -monolithic output. Change the resize target and both sides move together, so **both still pass**. - -Nothing pins output *geometry*: not the shape relative to the image, not coordinate correctness. -`crop_masks_to_boxes` is not imported and has no tests at all. - -So phase 0 is not optional ceremony — it is the only thing that would catch a geometry regression. - ---- - -### Phase 0 — characterization (GREEN on current code) - -Pin today's behaviour before touching anything. These must pass unchanged at `t=1.0` forever after. - -```python -class TestCurrentMaskGeometry: - """Characterization. These encode today's contract; they must not change at t=1.0.""" - - def test_masks_are_image_sized(self) -> None: - _, masks = self._run() - assert masks.shape[1:] == (self.size_after_pre_processing.height, - self.size_after_pre_processing.width) - - def test_masks_are_bool(self) -> None: - _, masks = self._run() - assert masks.dtype == torch.bool - - def test_golden_output_for_fixed_seed(self) -> None: - # seeded protos -> exact tensor. Any geometry change breaks this loudly. - torch.manual_seed(0) - _, masks = self._run() - assert masks.sum().item() == GOLDEN_SET_PIXELS # fill from first run - assert masks[0, :8, :8].tolist() == GOLDEN_CORNER # fill from first run - - def test_non_square_image_preserves_aspect(self) -> None: - # the axis-swap guard. Square fixtures hide transposes. - _, masks = self._run(image=(1080, 1920)) - assert masks.shape[1] == 1080 and masks.shape[2] == 1920 -``` - -And the first tests `crop_masks_to_boxes` has ever had: - -```python -class TestCropMasksToBoxes: - def test_zeroes_outside_box(self) -> None: ... - def test_scaling_default_is_proto_stride(self) -> None: ... - def test_non_square_inference_size(self) -> None: - # documents that a single scalar `scaling` is wrong when mask_w/inference_w != 0.25 - ... -``` - -**Commit 0.** No production change. - -### Phase 1 — RED: the coordinate-scaling gap - -The prerequisite from §2 of the plan. This test fails today and proves the gap exists. - -```python -def test_polygon_coords_are_image_space_when_mask_is_smaller(self) -> None: - # given: a mask deliberately smaller than the declared image - masks = make_masks(n=2, h=540, w=960) # half of 1080x1920 - meta = make_metadata(original_size=(1080, 1920)) - - # when - response = build_instance_segmentation_response(masks, meta) - - # then: coordinates must span the IMAGE, not the mask - xs = [p.x for pred in response.predictions for p in pred.points] - assert max(xs) > 960, "polygon coords left in mask space" -``` - -RED today — `inference_models_adapters.py:917` emits raw `masks2poly` output. Green once scaling is -applied at the mask→polygon boundary, mirroring legacy `post_process_polygons` -(`inference/core/utils/postprocess.py:449`). - -**Commit 1.** Coordinate scaling only. No new parameter yet — and it is a latent-correctness fix on -its own merits. - -### Phase 2 — RED: the parameter - -```python -@pytest.mark.parametrize("t", [0.0, 0.1, 0.25, 0.5, 1.0]) -def test_resolution_factor_interpolates_shape(self, t: float) -> None: - _, masks = self._run(masks_resolution_factor=t) # TypeError today -> RED - mh, mw = PROTO_UNPADDED - assert masks.shape[1] == max(1, round(mh * (1 - t) + 1080 * t)) - assert masks.shape[2] == max(1, round(mw * (1 - t) + 1920 * t)) - -def test_t1_is_bit_identical_to_baseline(self) -> None: - _, ref = self._run() # no kwarg - _, out = self._run(masks_resolution_factor=1.0) - assert torch.equal(out, ref) - -def test_t0_is_proto_resolution(self) -> None: - _, masks = self._run(masks_resolution_factor=0.0) - assert masks.shape[1:] == PROTO_UNPADDED - -@pytest.mark.parametrize("t", [0.1, 0.25, 0.5]) -def test_coords_round_trip_against_t1(self, t: float) -> None: - ref = polygons_from(self._run()) - out = polygons_from(self._run(masks_resolution_factor=t)) - # unpad rounding error is amplified as t falls, so tolerance scales with 1/t - assert_polygons_close(out, ref, tol_px=2.0 / t) - -def test_zero_instances(self) -> None: - _, masks = self._run(n=0, masks_resolution_factor=0.25) - assert masks.shape[0] == 0 - -@pytest.mark.parametrize("mode", ["letterbox", "stretch"]) -def test_both_resize_modes(self, mode: str) -> None: ... - -def test_static_crop_offset_is_scaled(self) -> None: ... -``` - -The `t=1.0` bit-identity test is the safety net for every later phase. - -**Commit 2.** Dense path only. - -### Phase 3 — RED: `mask_size` metadata (plan O1) - -```python -def test_mask_size_defaults_to_image_size(self) -> None: - # back-compat: existing construction sites must not change - m = InstancesRLEMasks.from_coco_rle_masks(image_size=(1080, 1920), masks=[...]) - assert m.mask_size == (1080, 1920) - -def test_mask_size_reflects_reduced_resolution(self) -> None: - dets = self._run_full_pipeline(masks_resolution_factor=0.25) - assert dets.mask_size == (270, 480) - assert dets.image_size == (1080, 1920) # unchanged meaning -``` - -**Commit 3.** - -### Phase 4 — RED: adapter mapping - -```python -@pytest.mark.parametrize("mode,expected", [("accurate", 1.0), ("fast", 0.0)]) -def test_enum_maps_to_factor(self, mode: str, expected: float) -> None: ... - -def test_tradeoff_reads_the_factor(self) -> None: ... - -@pytest.mark.parametrize("bad", [-0.1, 1.1]) -def test_out_of_range_raises(self, bad: float) -> None: - with pytest.raises(InvalidMaskDecodeArgument): - ... - -def test_source_keys_do_not_leak_to_model_call(self) -> None: - # map_inference_kwargs must consume both, not forward them - mapped = adapter.map_inference_kwargs({"mask_decode_mode": "fast", "tradeoff_factor": 0.0}) - assert "mask_decode_mode" not in mapped and "tradeoff_factor" not in mapped - assert mapped["masks_resolution_factor"] == 0.0 -``` - -**Commit 4.** - -### Phase 5 — RED: version skew (plan O2) and the Triton gate - -```python -def test_absent_mask_size_defaults_to_input_size(self) -> None: - # old server: field missing -> client assumes today's contract - dets = parse_response(response_without_mask_size()) - assert dets.mask_size == dets.image_size - -def test_unhonoured_hint_does_not_raise(self, caplog) -> None: - # old server ignores the parameter: correct output, just slower - dets = run_remote(masks_resolution_factor=0.25, server=OldServerStub()) - assert dets.mask_size == dets.image_size - assert "not honoured" in caplog.text # logged once, never raised - - -def test_reduced_resolution_forces_eager_fallback(self) -> None: - # pure-python: no GPU, so it runs in x86 CI instead of being a dead GPU-only test - reason = _unsupported_triton_postprocess_reason(..., masks_resolution_factor=0.25) - assert reason is not None -``` - -**Commit 5.** - -### Phase 6 — RLE parity *(done)* - -The wire-format question was settled by measurement rather than by waiting — see O3. The RLE -generator now takes the factor, `to_coco_rle_masks` declares the encoded grid, and the Triton gate -defers when the factor is not `1.0`. The evidence experiment that unblocked it: - -```python -@pytest.mark.parametrize("decoder", ["pycocotools", "supervision"]) -def test_reduced_size_rle_decode_behaviour(self, decoder: str) -> None: - # not an assertion of correctness - a recorded observation of the failure mode, - # to inform the deferred decision: raises? truncates silently? resizes cleanly? - ... -``` - -### Running - -```bash -cd inference_models && python -m pytest tests/unit_tests/models/common/roboflow/test_post_processing.py -v -python -m pytest tests/inference/models_predictions_tests -python -m pytest tests/workflows/unit_tests/ -``` - -### Order and why - -Phase 0 before everything, because the existing invariance tests would not catch a geometry -regression. Phase 1 before phase 2, because coordinate scaling is a correctness fix that stands -alone and must be in place before any reduced mask can exist. Phase 6 last, because it is the only -part gated on a decision the team deferred. diff --git a/inference/core/models/inference_models_adapters.py b/inference/core/models/inference_models_adapters.py index 8a2e391edf..96b5d2515b 100644 --- a/inference/core/models/inference_models_adapters.py +++ b/inference/core/models/inference_models_adapters.py @@ -1055,18 +1055,25 @@ def _build_responses_from_detections( # reported; equal sizes short-circuit to a no-op. This also breaks # any view into the pinned scratch buffers noted above, since the # scaled polygons are freshly allocated. + # Scale from the mask grid to the frame the masks were produced + # against, which is size_after_pre_processing - NOT original_size. + # A static crop anchored at (0, 0) also yields a mask smaller than + # the image, but its coordinates are already image-space, so + # inferring the scale from `mask_size != (H, W)` would double them. mask_size = getattr(det, "mask_size", None) + produced_against = preproc_metadata.size_after_pre_processing if ( not return_in_rle and mask_size is not None - and tuple(mask_size) != (H, W) + and tuple(mask_size) + != (produced_against.height, produced_against.width) ): polys_or_rles = scale_polygons_to_image( polys_or_rles, mask_size=ImageDimensions( height=int(mask_size[0]), width=int(mask_size[1]) ), - image_size=ImageDimensions(height=H, width=W), + image_size=produced_against, ) predictions: List[ @@ -1196,7 +1203,10 @@ def rle_masks2poly(masks: InstancesRLEMasks) -> List[np.ndarray]: return rle_masks_to_polygons(masks=masks) segments = [] - h, w = masks.image_size + # counts are encoded on the mask grid, which is not the image grid once a + # resolution factor below 1.0 is in play. Decoding on image_size makes + # pycocotools reinterpret the runs without error and yields garbage. + h, w = masks.mask_size or masks.image_size for counts in masks.masks: rle_dict = {"size": [h, w], "counts": counts} decoded_rle = np.ascontiguousarray(mask_utils.decode(rle_dict)) diff --git a/inference/core/utils/rle_to_polygon.py b/inference/core/utils/rle_to_polygon.py index e271e0d821..df3f06c091 100644 --- a/inference/core/utils/rle_to_polygon.py +++ b/inference/core/utils/rle_to_polygon.py @@ -18,7 +18,8 @@ def rle_masks_to_polygons(masks: object) -> List[np.ndarray]: only the foreground bounding crop needed by OpenCV. """ - height, width = masks.image_size + # decode on the grid the counts were encoded on, not the image grid + height, width = getattr(masks, "mask_size", None) or masks.image_size sparse_counts = _get_lazy_uncompressed_counts(masks=masks) if sparse_counts is not None: counts, lengths = sparse_counts diff --git a/inference_models/docs/changelog.md b/inference_models/docs/changelog.md index 27ce5ca5a3..9b2425aacf 100644 --- a/inference_models/docs/changelog.md +++ b/inference_models/docs/changelog.md @@ -4,6 +4,40 @@ ### Added +- `masks_resolution_factor` on instance-segmentation post-processing, in + `[0.0, 1.0]`. `1.0` (the default) resizes masks to the image as before; + `0.0` leaves them on the model's own grid; values between interpolate the + resize target. Threaded through all six instance-segmentation families on + both the dense and RLE paths, and through the ONNX, TorchScript and TensorRT + backends. Default output is unchanged. +- `mask_size` on `InstanceDetections` and `InstancesRLEMasks`, recording the + grid the masks live on. Each defaults to the value it effectively had before, + so existing construction is unaffected. +- `scale_polygons_to_image`, lifting contour coordinates from mask space into + image space. + +### Changed + +- `InstancesRLEMasks.to_coco_rle_masks()` now declares the grid the counts were + encoded on rather than the image size. These agree unless a resolution factor + below `1.0` is used; when they differ, declaring the image size made + `pycocotools` reinterpret the runs without raising. +- `InstanceDetections.to_supervision()` restores the image grid when masks are + reduced, since `sv.Detections.mask` is documented as matching the image and + its annotators index the scene with it. +- `InstanceDetections.__iter__` emits the encoded grid in the COCO `size` field. +- An unknown `mask_decode_mode` now raises `InvalidMaskDecodeArgument` on the + `inference_models` path, matching the legacy path, which previously accepted + any value. + +### Fixed + +- Zero-detection results reported a mask shape the populated path would never + produce, because the empty branch resolved its target from the padded grid + while the populated path unpadded first. +- RF-DETR's Triton post-process dispatcher dropped the resolution factor, so + the fused path silently ignored it. + - RF-DETR Torch and ONNX object detection now use the five-stage execution plan, sharing Triton Universal preprocessing, reference fallback, compatibility checks and per-request selection metadata with TensorRT. diff --git a/inference_models/inference_models/__init__.py b/inference_models/inference_models/__init__.py index a300fb97cb..9a6ab03045 100644 --- a/inference_models/inference_models/__init__.py +++ b/inference_models/inference_models/__init__.py @@ -21,6 +21,7 @@ # before any heavy library gets imported. from inference_models._offline import OFFLINE_MODE # noqa: F401 isort: skip + try: # This will read version from pyproject.toml __version__ = importlib_metadata.version(__package__ or __name__) @@ -78,9 +79,9 @@ from inference_models.models.base.types import InstancesRLEMasks from inference_models.weights_providers.entities import Quantization from inference_models.weights_providers.offline_registry import ( - OfflineArtefactStatus, - OfflineArtefactVerification, - OfflineModelStatus, OfflinePackagePresence, + OfflineArtefactStatus, OfflinePackageStatus, + OfflineModelStatus, + OfflineArtefactVerification ) diff --git a/inference_models/inference_models/models/base/instance_segmentation.py b/inference_models/inference_models/models/base/instance_segmentation.py index 30c883808a..617a270b9c 100644 --- a/inference_models/inference_models/models/base/instance_segmentation.py +++ b/inference_models/inference_models/models/base/instance_segmentation.py @@ -14,6 +14,7 @@ runtime_checkable, ) +import cv2 import numpy as np import supervision as sv import torch @@ -156,6 +157,23 @@ def __post_init__(self) -> None: elif self.mask is not None and hasattr(self.mask, "shape"): self.mask_size = (int(self.mask.shape[1]), int(self.mask.shape[2])) + def _image_size(self) -> Tuple[int, int]: + """Resolve the image grid these detections describe. + + An RLE carrier records the image size alongside the encoded grid. A + dense carrier does not, so its own shape is the best available answer + and is correct whenever no resolution factor was applied. + + Returns: + Image ``(height, width)``. + """ + if isinstance(self.mask, InstancesRLEMasks): + return tuple(self.mask.image_size) + if self.mask is not None and hasattr(self.mask, "shape"): + return int(self.mask.shape[1]), int(self.mask.shape[2]) + + return tuple(self.mask_size) if self.mask_size else (0, 0) + def __len__(self) -> int: return int(self.xyxy.shape[0]) @@ -182,7 +200,9 @@ def __iter__(self) -> Iterator[Tuple]: selected_mask = None elif isinstance(self.mask, InstancesRLEMasks): selected_mask = { - "size": list(self.mask.image_size), + # the counts describe the encoded grid, which is not the + # image grid once a resolution factor below 1.0 is used + "size": list(self.mask.mask_size or self.mask.image_size), "counts": self.mask.masks[index], } else: @@ -246,6 +266,24 @@ def to_supervision(self) -> sv.Detections: mask = self.mask.cpu().numpy() else: mask = coco_rle_masks_to_numpy_mask(self.mask) + # sv.Detections documents mask as (n, H, W) matching the image, and its + # annotators index the scene with it. A reduced grid would raise or + # paint the wrong region, so restore the image grid on the way out. + if mask is not None and self.mask_size is not None: + image_height, image_width = self._image_size() + if tuple(self.mask_size) != (image_height, image_width): + mask = np.stack( + [ + cv2.resize( + single.astype(np.uint8), + (image_width, image_height), + interpolation=cv2.INTER_NEAREST, + ).astype(bool) + for single in mask + ] + ) if len(mask) else np.zeros( + (0, image_height, image_width), dtype=bool + ) return sv.Detections( xyxy=self.xyxy.cpu().numpy(), class_id=self.class_id.cpu().numpy(), diff --git a/inference_models/inference_models/models/common/roboflow/post_processing.py b/inference_models/inference_models/models/common/roboflow/post_processing.py index 33acf006fe..5c64ac7da8 100644 --- a/inference_models/inference_models/models/common/roboflow/post_processing.py +++ b/inference_models/inference_models/models/common/roboflow/post_processing.py @@ -407,6 +407,14 @@ def scale_polygons_to_image( scaled independently, since the mask and the image need not share an aspect ratio. + Coordinates are multiplied directly, without a pixel-centre correction, so + a contour touching the far edge of the mask lands up to `(scale - 1)` pixels + short of the image edge - 11 px at a 12x scale. This matches + `inference.core.utils.postprocess.scale_polygons`, which the legacy backend + uses, and keeping the two in agreement is worth more than halving the bias + on one of them. Changing the convention would make the same input produce + different polygons depending on which backend served it. + Args: polygons: Contours in mask coordinates, each of shape `(k, 2)` as `(x, y)`. mask_size: Dimensions of the mask the contours were extracted from. @@ -468,6 +476,43 @@ def resolve_mask_target_size( return height, width +def resolve_unpadded_mask_grid( + mask_height: int, + mask_width: int, + *, + padding: Tuple[int, int, int, int], + inference_size: ImageDimensions, +) -> Tuple[int, int]: + """Size of the mask grid once letterbox padding is removed. + + The populated path slices the padding away before resizing, so the resize + target is interpolated from the unpadded grid. The empty path has no masks + to slice, and must compute the same number or it reports a shape the + populated path would never produce. + + Args: + mask_height: Height of the mask grid as the model produced it. + mask_width: Width of that grid. + padding: Letterbox padding as `(left, top, right, bottom)` in + network-input pixels. + inference_size: Network input dimensions the padding refers to. + + Returns: + The unpadded `(height, width)` of the mask grid. + """ + pad_left, pad_top, pad_right, pad_bottom = padding + height_scale = mask_height / inference_size.height + width_scale = mask_width / inference_size.width + unpadded_height = mask_height - round(height_scale * pad_top) - round( + height_scale * pad_bottom + ) + unpadded_width = mask_width - round(width_scale * pad_left) - round( + width_scale * pad_right + ) + + return max(1, unpadded_height), max(1, unpadded_width) + + def align_instance_segmentation_results( image_bboxes: torch.Tensor, masks: torch.Tensor, @@ -483,9 +528,15 @@ def align_instance_segmentation_results( masks_resolution_factor: float = 1.0, ) -> Tuple[torch.Tensor, torch.Tensor]: if image_bboxes.shape[0] == 0: + unpadded_height, unpadded_width = resolve_unpadded_mask_grid( + masks.shape[1], + masks.shape[2], + padding=padding, + inference_size=inference_size, + ) empty_target_height, empty_target_width = resolve_mask_target_size( - mask_height=masks.shape[1], - mask_width=masks.shape[2], + mask_height=unpadded_height, + mask_width=unpadded_width, size_after_pre_processing=size_after_pre_processing, masks_resolution_factor=masks_resolution_factor, ) @@ -588,11 +639,22 @@ def align_instance_segmentation_results( canvas_w_scale = target_width / size_after_pre_processing.width canvas_offset_y = round(static_crop_offset.offset_y * canvas_h_scale) canvas_offset_x = round(static_crop_offset.offset_x * canvas_w_scale) + # The canvas must contain the pasted extent. Rounding the canvas and + # the offset independently can leave the canvas one pixel short, so + # take whichever is larger rather than trusting the rounded product. mask_canvas = torch.zeros( ( masks.shape[0], - max(1, round(original_size.height * canvas_h_scale)), - max(1, round(original_size.width * canvas_w_scale)), + max( + 1, + round(original_size.height * canvas_h_scale), + canvas_offset_y + masks.shape[1], + ), + max( + 1, + round(original_size.width * canvas_w_scale), + canvas_offset_x + masks.shape[2], + ), ), dtype=torch.bool, device=masks.device, @@ -741,10 +803,16 @@ def align_instance_segmentation_results_to_rle_masks( # path, so crop offsets move into the same space canvas_h_scale = target_h / size_after_pre_processing.height canvas_w_scale = target_w / size_after_pre_processing.width - canvas_height = max(1, round(original_size.height * canvas_h_scale)) - canvas_width = max(1, round(original_size.width * canvas_w_scale)) offset_y = round(static_crop_offset.offset_y * canvas_h_scale) offset_x = round(static_crop_offset.offset_x * canvas_w_scale) + # as above: the canvas must contain offset + mask, not merely the rounded + # product of the original size and the scale + canvas_height = max( + 1, round(original_size.height * canvas_h_scale), offset_y + target_h + ) + canvas_width = max( + 1, round(original_size.width * canvas_w_scale), offset_x + target_w + ) num_instances = image_bboxes.shape[0] for i in range(num_instances): # keep a batch dim so functional.resize is unambiguous diff --git a/inference_models/inference_models/models/rfdetr/common.py b/inference_models/inference_models/models/rfdetr/common.py index 0772f26ac4..8f43356cbe 100644 --- a/inference_models/inference_models/models/rfdetr/common.py +++ b/inference_models/inference_models/models/rfdetr/common.py @@ -437,6 +437,7 @@ def post_process_instance_segmentation_results_to_rle_masks( classes_re_mapping=classes_re_mapping, max_detections=max_detections, defer_postprocess_sync=defer_postprocess_sync, + masks_resolution_factor=masks_resolution_factor, ) for image_bboxes, image_logits, image_masks, image_meta in zip( bboxes, diff --git a/inference_models/inference_models/weights_providers/offline_registry.py b/inference_models/inference_models/weights_providers/offline_registry.py index 954d266bc1..d85d01f26c 100644 --- a/inference_models/inference_models/weights_providers/offline_registry.py +++ b/inference_models/inference_models/weights_providers/offline_registry.py @@ -667,7 +667,8 @@ def _parse_registry_timestamp( except ValueError: pass LOGGER.warning( - "Offline-weights registry record contains unparsable timestamp for " "%s: %r", + "Offline-weights registry record contains unparsable timestamp for " + "%s: %r", context, value, ) diff --git a/inference_models/tests/unit_tests/models/base/test_instance_segmentation.py b/inference_models/tests/unit_tests/models/base/test_instance_segmentation.py index e3be721bfc..7e809e0ab7 100644 --- a/inference_models/tests/unit_tests/models/base/test_instance_segmentation.py +++ b/inference_models/tests/unit_tests/models/base/test_instance_segmentation.py @@ -150,3 +150,99 @@ def test_empty_detection_stack_still_reports_the_grid(self) -> None: # then assert detections.mask_size == (270, 480) + + +class TestReducedGridLeavesTheBoundaryAtImageSize: + """`sv.Detections` and the COCO dict must describe the image, not the grid. + + `sv.Detections.mask` is documented as `(n, H, W)` matching the image and + its annotators index the scene with it, so a reduced grid raises. The COCO + dict is the opposite: `size` must describe the grid the counts were encoded + on, or decoding reinterprets the runs. + """ + + def test_to_supervision_restores_the_image_grid(self) -> None: + # given + # masks produced at a quarter of the image resolution + detections = InstanceDetections( + xyxy=torch.tensor([[0, 0, 80, 80]], dtype=torch.float32), + class_id=torch.zeros((1,), dtype=torch.int64), + confidence=torch.ones((1,), dtype=torch.float32), + mask=InstancesRLEMasks( + image_size=(80, 80), + masks=[_square_rle(20, 20)], + mask_size=(20, 20), + ), + ) + + # when + converted = detections.to_supervision() + + # then + assert converted.mask.shape == (1, 80, 80) + + def test_supervision_annotator_accepts_the_result(self) -> None: + # given + # the reported failure was an IndexError from boolean-index mismatch + import numpy as np + import supervision as sv + + scene = np.zeros((80, 80, 3), dtype=np.uint8) + detections = InstanceDetections( + xyxy=torch.tensor([[0, 0, 80, 80]], dtype=torch.float32), + class_id=torch.zeros((1,), dtype=torch.int64), + confidence=torch.ones((1,), dtype=torch.float32), + mask=InstancesRLEMasks( + image_size=(80, 80), + masks=[_square_rle(20, 20)], + mask_size=(20, 20), + ), + ) + + # when / then + sv.MaskAnnotator().annotate(scene.copy(), detections.to_supervision()) + + def test_iteration_declares_the_encoded_grid(self) -> None: + # given + detections = InstanceDetections( + xyxy=torch.tensor([[0, 0, 80, 80]], dtype=torch.float32), + class_id=torch.zeros((1,), dtype=torch.int64), + confidence=torch.ones((1,), dtype=torch.float32), + mask=InstancesRLEMasks( + image_size=(80, 80), + masks=[_square_rle(20, 20)], + mask_size=(20, 20), + ), + ) + + # when + _, mask, *_ = next(iter(detections)) + + # then + assert mask["size"] == [20, 20] + + +def _square_rle(height: int, width: int) -> bytes: + """Encode a filled square as COCO RLE counts. + + Args: + height: Grid height. + width: Grid width. + + Returns: + COCO RLE counts for a square covering the middle of the grid. + + Examples: + ```pycon + >>> isinstance(_square_rle(20, 20), bytes) + True + + ``` + """ + import numpy as np + from pycocotools import mask as mask_utils + + dense = np.zeros((height, width), dtype=np.uint8) + dense[height // 4 : height // 2, width // 4 : width // 2] = 1 + + return mask_utils.encode(np.asfortranarray(dense))["counts"] diff --git a/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py b/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py index 39e5f37f7c..59548c768d 100644 --- a/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py +++ b/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py @@ -6,6 +6,8 @@ - Mask geometry: characterization of the output contract """ +import hashlib + import numpy as np import pytest import torch @@ -784,14 +786,6 @@ def _run(cls, n: int = 7, static_crop: bool = False, **kwargs): **kwargs, ) - def test_factor_one_is_bit_identical_to_default(self) -> None: - # given / when - _, baseline = self._run() - _, out = self._run(masks_resolution_factor=1.0) - - # then - assert torch.equal(out, baseline) - def test_factor_zero_keeps_the_post_unpad_mask_grid(self) -> None: # given / when _, masks = self._run(masks_resolution_factor=0.0) @@ -903,16 +897,41 @@ def test_reduced_factor_declares_the_reduced_size(self) -> None: expected_w = round(self.PROTO * 0.5 + self.ORIG_W * 0.5) assert all(rle["size"] == [expected_h, expected_w] for _, rle in results) - def test_counts_sum_to_the_declared_size(self) -> None: + @pytest.mark.parametrize("factor", [0.25, 0.5, 1.0]) + def test_rle_content_matches_the_dense_path(self, factor: float) -> None: # given / when - # the invariant pycocotools relies on; a declared size that disagrees - # with the counts decodes silently wrong rather than raising - results = self._run(masks_resolution_factor=0.25) + # the previous version asserted decoded.shape == rle["size"], which + # mask_utils.decode guarantees and so could never fail. Comparing the + # decoded content against the dense path is the real invariant: the + # two carriers must describe the same pixels at the same factor. + rle_results = self._run(masks_resolution_factor=factor) + torch.manual_seed(42) + bboxes = torch.rand((3, 6), dtype=torch.float32) * 100 + masks = torch.randn((3, self.PROTO, self.PROTO), dtype=torch.float32) + _, dense = align_instance_segmentation_results( + image_bboxes=bboxes.clone(), + masks=masks.clone(), + padding=(0, 0, 0, 0), + scale_width=1.0, + scale_height=1.0, + original_size=ImageDimensions(height=self.ORIG_H, width=self.ORIG_W), + size_after_pre_processing=ImageDimensions( + height=self.ORIG_H, width=self.ORIG_W + ), + inference_size=ImageDimensions(height=640, width=640), + static_crop_offset=StaticCropOffset( + offset_x=0, offset_y=0, crop_width=self.ORIG_W, crop_height=self.ORIG_H + ), + binarization_threshold=0.0, + mask_chunk_size=1000, + masks_resolution_factor=factor, + ) # then - for _, rle in results: - decoded = mask_utils.decode(rle) - assert decoded.shape == tuple(rle["size"]) + for index, (_, rle) in enumerate(rle_results): + decoded = mask_utils.decode(rle).astype(bool) + assert decoded.shape == tuple(dense.shape[1:]) + assert np.array_equal(decoded, dense[index].numpy()) def test_matches_the_dense_path_resolution(self) -> None: # given / when @@ -942,3 +961,227 @@ def test_matches_the_dense_path_resolution(self) -> None: # then assert rle_results[0][1]["size"] == list(dense.shape[1:]) + + +# Captured from origin/main @ 23a01eb3a with +# `git show origin/main: > /tmp/pp_main.py`, loading it alongside +# the branch and fingerprinting its output. If origin/main ever changes this +# function these cases SHOULD fail - that is the point of a characterization +# test. Regenerate only after deciding the new behaviour is intended. +_MAIN_GOLDEN = [ + {"ih": 333, "iw": 211, "ox": 97, "oy": 61, "pad": (16, 18, 6, 5), "n": 5, "shape": (5, 394, 308), "digest": "a48aa710ef85cd88", "boxes": "b62c110b17ae5441"}, + {"ih": 333, "iw": 211, "ox": 0, "oy": 0, "pad": (14, 9, 4, 2), "n": 5, "shape": (5, 333, 211), "digest": "5d04de5cff404410", "boxes": "5b0e73b0ea74de77"}, + {"ih": 200, "iw": 300, "ox": 97, "oy": 61, "pad": (20, 23, 19, 20), "n": 0, "shape": (0, 261, 397), "digest": "e3b0c44298fc1c14", "boxes": "e3b0c44298fc1c14"}, + {"ih": 200, "iw": 300, "ox": 0, "oy": 0, "pad": (1, 6, 7, 19), "n": 0, "shape": (0, 200, 300), "digest": "e3b0c44298fc1c14", "boxes": "e3b0c44298fc1c14"}, + {"ih": 333, "iw": 211, "ox": 13, "oy": 61, "pad": (18, 6, 16, 7), "n": 5, "shape": (5, 394, 224), "digest": "a37811a210846e79", "boxes": "8cbae67b51c4aa02"}, + {"ih": 640, "iw": 640, "ox": 97, "oy": 0, "pad": (21, 2, 14, 20), "n": 1, "shape": (1, 640, 737), "digest": "f88d6c3e4f5c03f7", "boxes": "e1cacd18dd5fddf7"}, + {"ih": 333, "iw": 211, "ox": 0, "oy": 7, "pad": (10, 24, 7, 16), "n": 1, "shape": (1, 340, 211), "digest": "d15ccb8dd99581e4", "boxes": "e47e86508db619f6"}, + {"ih": 200, "iw": 300, "ox": 0, "oy": 0, "pad": (12, 3, 9, 12), "n": 0, "shape": (0, 200, 300), "digest": "e3b0c44298fc1c14", "boxes": "e3b0c44298fc1c14"}, + {"ih": 200, "iw": 300, "ox": 0, "oy": 0, "pad": (6, 1, 15, 12), "n": 5, "shape": (5, 200, 300), "digest": "baaed46f79ba3102", "boxes": "37313f9472c62672"}, + {"ih": 333, "iw": 211, "ox": 97, "oy": 0, "pad": (18, 20, 6, 24), "n": 5, "shape": (5, 333, 308), "digest": "c760dd33356b0b6d", "boxes": "6b53dff165635fee"}, + {"ih": 640, "iw": 640, "ox": 13, "oy": 0, "pad": (9, 10, 0, 13), "n": 0, "shape": (0, 640, 653), "digest": "e3b0c44298fc1c14", "boxes": "e3b0c44298fc1c14"}, + {"ih": 1080, "iw": 1920, "ox": 0, "oy": 0, "pad": (0, 1, 14, 15), "n": 0, "shape": (0, 1080, 1920), "digest": "e3b0c44298fc1c14", "boxes": "e3b0c44298fc1c14"}, + {"ih": 1080, "iw": 1920, "ox": 97, "oy": 0, "pad": (23, 24, 4, 13), "n": 5, "shape": (5, 1080, 2017), "digest": "6982b829d1bbddc7", "boxes": "4c219acad8807667"}, + {"ih": 333, "iw": 211, "ox": 0, "oy": 61, "pad": (13, 6, 0, 8), "n": 5, "shape": (5, 394, 211), "digest": "87fcc86e47722220", "boxes": "b21927719126483f"}, + {"ih": 640, "iw": 640, "ox": 0, "oy": 0, "pad": (5, 12, 19, 20), "n": 5, "shape": (5, 640, 640), "digest": "846517457eb562ae", "boxes": "3cedb68b2e60e5f6"}, + {"ih": 200, "iw": 300, "ox": 0, "oy": 0, "pad": (6, 14, 8, 0), "n": 5, "shape": (5, 200, 300), "digest": "6287b10e34217d2e", "boxes": "bce5175b45d9adde"}, + {"ih": 640, "iw": 640, "ox": 13, "oy": 61, "pad": (2, 2, 2, 6), "n": 5, "shape": (5, 701, 653), "digest": "d8db8ae532bdb7ea", "boxes": "6798272b81011bb4"}, + {"ih": 1080, "iw": 1920, "ox": 0, "oy": 7, "pad": (11, 19, 14, 4), "n": 5, "shape": (5, 1087, 1920), "digest": "c9f9413897eefdd0", "boxes": "749ad903686c1c93"}, + {"ih": 333, "iw": 211, "ox": 0, "oy": 61, "pad": (5, 20, 4, 9), "n": 0, "shape": (0, 394, 211), "digest": "e3b0c44298fc1c14", "boxes": "e3b0c44298fc1c14"}, + {"ih": 1080, "iw": 1920, "ox": 0, "oy": 0, "pad": (23, 20, 17, 6), "n": 5, "shape": (5, 1080, 1920), "digest": "fe15d5708b271c8f", "boxes": "b6b89ee145f20853"}, + {"ih": 333, "iw": 211, "ox": 97, "oy": 0, "pad": (13, 1, 3, 3), "n": 0, "shape": (0, 333, 308), "digest": "e3b0c44298fc1c14", "boxes": "e3b0c44298fc1c14"}, + {"ih": 640, "iw": 640, "ox": 0, "oy": 61, "pad": (8, 13, 19, 15), "n": 1, "shape": (1, 701, 640), "digest": "fbb5ace53bca0cf5", "boxes": "48123ae5fa70ad76"}, + {"ih": 1080, "iw": 1920, "ox": 0, "oy": 0, "pad": (7, 15, 17, 20), "n": 5, "shape": (5, 1080, 1920), "digest": "be0ffbd3b025fc6b", "boxes": "bcc7d8853de382db"}, + {"ih": 200, "iw": 300, "ox": 13, "oy": 0, "pad": (6, 23, 0, 2), "n": 1, "shape": (1, 200, 313), "digest": "4f01e63780b7b135", "boxes": "404d5eb899660b53"}, +] + + +class TestBitIdenticalToMain: + """Factor 1.0 must reproduce origin/main exactly. + + The obvious version of this test - comparing the default against an + explicit 1.0 - compares the branch with itself and cannot fail. These + cases carry digests of the *pre-change* implementation's output, over a + matrix that includes non-zero letterbox padding, static crops and empty + detections, which the rest of the suite does not exercise. + """ + + @staticmethod + def _run(case: dict): + torch.manual_seed(7) + bboxes = torch.rand((case["n"], 6), dtype=torch.float32) * 100 + masks = torch.randn((case["n"], 160, 160), dtype=torch.float32) + + return align_instance_segmentation_results( + image_bboxes=bboxes.clone(), + masks=masks.clone(), + padding=tuple(case["pad"]), + scale_width=1.0, + scale_height=1.0, + original_size=ImageDimensions( + height=case["ih"] + case["oy"], width=case["iw"] + case["ox"] + ), + size_after_pre_processing=ImageDimensions( + height=case["ih"], width=case["iw"] + ), + inference_size=ImageDimensions(height=640, width=640), + static_crop_offset=StaticCropOffset( + offset_x=case["ox"], + offset_y=case["oy"], + crop_width=case["iw"], + crop_height=case["ih"], + ), + binarization_threshold=0.0, + mask_chunk_size=16, + masks_resolution_factor=1.0, + ) + + @pytest.mark.parametrize("case", _MAIN_GOLDEN) + def test_shape_matches_main(self, case: dict) -> None: + # given / when + _, masks = self._run(case) + + # then + # asserted separately from the digest so a geometry change reports + # readably instead of as an opaque hash mismatch + assert tuple(masks.shape) == case["shape"] + + @pytest.mark.parametrize("case", _MAIN_GOLDEN) + def test_content_matches_main(self, case: dict) -> None: + # given / when + boxes, masks = self._run(case) + + # then + assert ( + hashlib.sha256(masks.numpy().tobytes()).hexdigest()[:16] == case["digest"] + ) + assert ( + hashlib.sha256(boxes.numpy().tobytes()).hexdigest()[:16] == case["boxes"] + ) + + +class TestStaticCropAtReducedResolution: + """The combination that crashed: a crop reaching the image edge, a + letterboxed input, and a factor below 1.0. + + Rounding the canvas size and the crop offset independently can leave the + canvas a pixel short of the pasted extent. Only crops that touch the image + edge expose it, and only under letterbox scaling, which is why a sweep with + zero padding and scale 1.0 finds nothing. + """ + + @staticmethod + def _letterboxed(image_w: int, image_h: int, offset_x: int, offset_y: int): + """Build the arguments for a crop that reaches the image edge. + + Args: + image_w: Full image width. + image_h: Full image height. + offset_x: Left edge of the crop. + offset_y: Top edge of the crop. + + Returns: + Keyword arguments for the alignment helpers. + """ + crop_w, crop_h = image_w - offset_x, image_h - offset_y + scale = min(640 / crop_w, 640 / crop_h) + new_w, new_h = round(crop_w * scale), round(crop_h * scale) + pad_left, pad_top = (640 - new_w) // 2, (640 - new_h) // 2 + + return { + "image_bboxes": torch.tensor([[1.0, 1, 50, 50, 0.9, 0]]), + "masks": torch.randn(1, 160, 160), + "padding": (pad_left, pad_top, 640 - new_w - pad_left, 640 - new_h - pad_top), + "scale_width": scale, + "scale_height": scale, + "original_size": ImageDimensions(height=image_h, width=image_w), + "size_after_pre_processing": ImageDimensions(height=crop_h, width=crop_w), + "inference_size": ImageDimensions(height=640, width=640), + "static_crop_offset": StaticCropOffset( + offset_x=offset_x, offset_y=offset_y, crop_width=crop_w, crop_height=crop_h + ), + } + + CASES = [ + (1920, 1080, 288, 216, 0.5), + (1920, 1080, 0, 648, 0.75), + (245, 245, 83, 83, 1 / 3), + (1280, 720, 417, 311, 0.1), + ] + + @pytest.mark.parametrize("image_w,image_h,offset_x,offset_y,factor", CASES) + def test_dense_path_places_the_mask_inside_the_canvas( + self, image_w: int, image_h: int, offset_x: int, offset_y: int, factor: float + ) -> None: + # given + torch.manual_seed(1) + kwargs = self._letterboxed(image_w, image_h, offset_x, offset_y) + + # when + _, masks = align_instance_segmentation_results( + **kwargs, binarization_threshold=0.0, mask_chunk_size=16, + masks_resolution_factor=factor, + ) + + # then + assert masks.shape[0] == 1 + + @pytest.mark.parametrize("image_w,image_h,offset_x,offset_y,factor", CASES) + def test_rle_path_places_the_mask_inside_the_canvas( + self, image_w: int, image_h: int, offset_x: int, offset_y: int, factor: float + ) -> None: + # given + torch.manual_seed(1) + kwargs = self._letterboxed(image_w, image_h, offset_x, offset_y) + + # when + results = list( + align_instance_segmentation_results_to_rle_masks( + **kwargs, binarization_threshold=0.0, masks_resolution_factor=factor + ) + ) + + # then + assert len(results) == 1 + decoded = mask_utils.decode(results[0][1]) + assert decoded.shape == tuple(results[0][1]["size"]) + + +class TestEmptyAgreesWithPopulated: + """Zero detections must report the shape the populated path would produce. + + Consumers derive the mask canvas from `mask.shape`, so a frame that + happens to detect nothing must not describe a different canvas from the + next frame that does. The empty path has no masks to unpad, so it has to + compute the unpadded grid rather than read it. + """ + + @staticmethod + def _run(n: int, padding, factor: float): + torch.manual_seed(5) + return align_instance_segmentation_results( + image_bboxes=torch.rand((n, 6), dtype=torch.float32) * 50, + masks=torch.randn((n, 160, 160), dtype=torch.float32), + padding=padding, + scale_width=1.0, + scale_height=1.0, + original_size=ImageDimensions(height=200, width=300), + size_after_pre_processing=ImageDimensions(height=200, width=300), + inference_size=ImageDimensions(height=640, width=640), + static_crop_offset=StaticCropOffset( + offset_x=0, offset_y=0, crop_width=300, crop_height=200 + ), + binarization_threshold=0.0, + mask_chunk_size=16, + masks_resolution_factor=factor, + ) + + @pytest.mark.parametrize("padding", [(0, 0, 0, 0), (12, 34, 12, 34), (7, 0, 41, 3)]) + @pytest.mark.parametrize("factor", [0.0, 0.25, 0.5, 1.0]) + def test_shapes_agree(self, padding, factor: float) -> None: + # given / when + _, empty = self._run(0, padding, factor) + _, populated = self._run(2, padding, factor) + + # then + assert empty.shape[1:] == populated.shape[1:] diff --git a/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py b/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py index d56b01da16..83dbc62bbe 100644 --- a/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py +++ b/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py @@ -706,3 +706,39 @@ def test_reduced_mask_resolution_is_unsupported_by_the_fused_path( assert reason != "mask_resolution_factor_unsupported" else: assert reason == "mask_resolution_factor_unsupported" + + +@pytest.mark.parametrize("factor", [0.25, 0.5]) +def test_triton_dispatcher_forwards_the_resolution_factor(monkeypatch, factor) -> None: + # given + # the gate only fires if the dispatcher passes the factor down. Testing the + # gate function directly cannot detect a dispatcher that drops it. + monkeypatch.setattr(rfdetr_common, "_TRITON_POSTPROC_ENABLED", True) + seen = {} + + def spy(*args, **kwargs): + seen.update(kwargs) + return None # force the eager fallback + + monkeypatch.setattr( + rfdetr_common, + "post_process_single_instance_segmentation_result_to_rle_masks_triton", + spy, + ) + device = torch.device("cpu") + bboxes, logits, masks = _single_detection_inputs(device) + + # when + post_process_instance_segmentation_results_to_rle_masks( + bboxes=bboxes.unsqueeze(0), + logits=logits.unsqueeze(0), + masks=masks.unsqueeze(0), + pre_processing_meta=[_metadata()], + threshold=0.4, + num_classes=2, + classes_re_mapping=_class_mapping(device), + masks_resolution_factor=factor, + ) + + # then + assert seen.get("masks_resolution_factor") == factor diff --git a/inference_models/tests/unit_tests/models/test_mage_vl_hf.py b/inference_models/tests/unit_tests/models/test_mage_vl_hf.py index 51107da4fe..8bee136300 100644 --- a/inference_models/tests/unit_tests/models/test_mage_vl_hf.py +++ b/inference_models/tests/unit_tests/models/test_mage_vl_hf.py @@ -207,9 +207,7 @@ def test_pre_process_generation_leaves_undeclared_tensors_alone() -> None: mage_vl.pre_process_generation(images=image, prompt="what is this?") passed_image = processor.call_args.kwargs["images"] - assert ( - passed_image[0].max() == 255 - ), "tensors without a declared format pass through" + assert passed_image[0].max() == 255, "tensors without a declared format pass through" def test_video_pre_processing_wraps_missing_dependency_errors(tmp_path) -> None: @@ -340,6 +338,6 @@ def test_post_process_generation_strips_decoded_text() -> None: processor.tokenizer.batch_decode.return_value = [" a dog \n"] mage_vl = _mage_vl(processor=processor) - assert mage_vl.post_process_generation(generated_ids=torch.tensor([[1, 2]])) == [ - "a dog" - ] + assert mage_vl.post_process_generation( + generated_ids=torch.tensor([[1, 2]]) + ) == ["a dog"] diff --git a/tests/inference/unit_tests/core/models/test_inference_models_adapters.py b/tests/inference/unit_tests/core/models/test_inference_models_adapters.py index 6e8da5ad31..0e11a89dcb 100644 --- a/tests/inference/unit_tests/core/models/test_inference_models_adapters.py +++ b/tests/inference/unit_tests/core/models/test_inference_models_adapters.py @@ -5,6 +5,7 @@ from types import SimpleNamespace import numpy as np +from pycocotools import mask as mask_utils import pytest import torch @@ -12,8 +13,10 @@ InstanceSegmentationInferenceResponse, InstanceSegmentationInferenceResponseDC, ) +from inference_models.models.base.types import InstancesRLEMasks from inference.core.exceptions import InvalidMaskDecodeArgument, PostProcessingError from inference.core.models.inference_models_adapters import ( + rle_masks2poly, InferenceModelsDepthEstimationAdapter, InferenceModelsInstanceSegmentationAdapter, InferenceModelsObjectDetectionAdapter, @@ -282,7 +285,7 @@ def test_workflow_response_fast_dataclass_path_is_disabled_at_depth_one() -> Non adapter = object.__new__(InferenceModelsInstanceSegmentationAdapter) adapter._pipeline_depth = 1 adapter.class_names = ["car"] - metadata = [SimpleNamespace(original_size=SimpleNamespace(width=4, height=4))] + metadata = [SimpleNamespace(original_size=SimpleNamespace(width=4, height=4), size_after_pre_processing=SimpleNamespace(width=4, height=4))] detections = [ InstanceDetections( xyxy=torch.tensor([[1, 1, 3, 3]], dtype=torch.int32), @@ -305,7 +308,7 @@ def test_workflow_response_fast_dataclass_path_is_enabled_above_depth_one() -> N adapter = object.__new__(InferenceModelsInstanceSegmentationAdapter) adapter._pipeline_depth = 2 adapter.class_names = ["car"] - metadata = [SimpleNamespace(original_size=SimpleNamespace(width=4, height=4))] + metadata = [SimpleNamespace(original_size=SimpleNamespace(width=4, height=4), size_after_pre_processing=SimpleNamespace(width=4, height=4))] detections = [ InstanceDetections( xyxy=torch.tensor([[1, 1, 3, 3]], dtype=torch.int32), @@ -755,7 +758,7 @@ def test_polygon_points_are_scaled_from_mask_space_into_image_space() -> None: # a 4x4 mask describing a 16x16 image: contours come out in mask # coordinates and must be lifted by 4x before they are reported adapter = _seg_adapter() - metadata = [SimpleNamespace(original_size=SimpleNamespace(width=16, height=16))] + metadata = [SimpleNamespace(original_size=SimpleNamespace(width=16, height=16), size_after_pre_processing=SimpleNamespace(width=16, height=16))] detections = [ InstanceDetections( xyxy=torch.tensor([[0, 0, 16, 16]], dtype=torch.int32), @@ -779,7 +782,7 @@ def test_polygon_points_unchanged_when_mask_matches_the_image() -> None: # given # today's situation: the scaling must be an exact no-op adapter = _seg_adapter() - metadata = [SimpleNamespace(original_size=SimpleNamespace(width=8, height=8))] + metadata = [SimpleNamespace(original_size=SimpleNamespace(width=8, height=8), size_after_pre_processing=SimpleNamespace(width=8, height=8))] mask = torch.zeros((1, 8, 8), dtype=torch.uint8) mask[0, 2:6, 2:6] = 1 detections = [ @@ -864,3 +867,94 @@ def test_unknown_mode_is_rejected(self) -> None: # the legacy path raises for this; the two must agree with pytest.raises(InvalidMaskDecodeArgument): self._map(mask_decode_mode="nonsense") + + +class TestRLEBackedPolygons: + """Polygons derived from RLE carriers at a reduced resolution. + + The adapter asks for RLE whenever the model supports it, so this is the + default route for a plain polygon response. The counts are encoded on the + mask grid; decoding them on the image grid makes pycocotools reinterpret + the runs without raising and produces unusable contours. + """ + + @staticmethod + def _carrier(mask_h: int, mask_w: int, image_h: int, image_w: int): + dense = np.zeros((mask_h, mask_w), dtype=np.uint8) + dense[mask_h // 4 : mask_h // 2, mask_w // 4 : mask_w // 2] = 1 + counts = mask_utils.encode(np.asfortranarray(dense))["counts"] + + return InstancesRLEMasks( + image_size=(image_h, image_w), masks=[counts], mask_size=(mask_h, mask_w) + ), dense + + def test_polygon_follows_the_encoded_grid_not_the_image(self) -> None: + # given + # a 160x160 mask describing a 640x640 image: the contour must come back + # in mask coordinates, which the adapter then scales into image space + carrier, dense = self._carrier(160, 160, 640, 640) + + # when + polygon = rle_masks2poly(carrier)[0] + + # then + ys, xs = np.where(dense) + assert polygon[:, 0].max() <= 160 + assert abs(polygon[:, 0].min() - xs.min()) <= 1 + assert abs(polygon[:, 1].max() - ys.max()) <= 1 + + def test_unchanged_when_the_grid_is_the_image(self) -> None: + # given + carrier, dense = self._carrier(64, 64, 64, 64) + + # when + polygon = rle_masks2poly(carrier)[0] + + # then + ys, xs = np.where(dense) + assert abs(polygon[:, 0].min() - xs.min()) <= 1 + assert abs(polygon[:, 1].max() - ys.max()) <= 1 + + def test_non_square_grid(self) -> None: + # given + # a transposed decode is invisible on a square grid + carrier, dense = self._carrier(80, 160, 400, 800) + + # when + polygon = rle_masks2poly(carrier)[0] + + # then + assert polygon[:, 0].max() <= 160 + assert polygon[:, 1].max() <= 80 + + +def test_origin_anchored_crop_is_not_mistaken_for_a_reduced_grid() -> None: + # given + # a static crop at (0, 0) yields a mask smaller than the image, but its + # coordinates are already image-space. Inferring "reduced" from + # mask_size != image_size would scale them a second time. + adapter = _seg_adapter() + metadata = [ + SimpleNamespace( + original_size=SimpleNamespace(width=16, height=16), + size_after_pre_processing=SimpleNamespace(width=8, height=8), + ) + ] + mask = torch.zeros((1, 8, 8), dtype=torch.uint8) + mask[0, 2:6, 2:6] = 1 + detections = [ + InstanceDetections( + xyxy=torch.tensor([[2, 2, 6, 6]], dtype=torch.int32), + confidence=torch.tensor([0.9], dtype=torch.float32), + class_id=torch.tensor([0], dtype=torch.int32), + mask=mask, + ) + ] + + # when + responses = adapter._build_responses_from_detections(detections, metadata) + + # then + # the mask already matches size_after_pre_processing, so no scaling + xs = [p.x for p in responses[0].predictions[0].points] + assert max(xs) <= 8, f"coordinates were scaled a second time: {xs}" diff --git a/tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_v5_mask_resolution.py b/tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_v5_mask_resolution.py index fd8ad59b7a..7c60eb46d6 100644 --- a/tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_v5_mask_resolution.py +++ b/tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_v5_mask_resolution.py @@ -116,12 +116,66 @@ def test_legacy_versions_pin_full_resolution(version: str, tensor: bool) -> None @pytest.mark.parametrize("version", FORWARDING_VERSIONS) -@pytest.mark.parametrize("tensor", [False, True]) -def test_v5_forwards_what_the_caller_set(version: str, tensor: bool) -> None: +def test_v5_forwards_what_the_caller_set(version: str) -> None: # given - source = _source(version, tensor) + source = _source(version, tensor=False) # then - assert "mask_decode_mode=mask_decode_mode," in source - assert "tradeoff_factor=tradeoff_factor," in source + assert source.count("mask_decode_mode=mask_decode_mode,") == 4 assert 'mask_decode_mode="accurate",' not in source + + +@pytest.mark.parametrize("version", FORWARDING_VERSIONS) +def test_v5_tensor_is_pinned_pending_mask_size_propagation(version: str) -> None: + # given + # nothing under workflows/ reads mask_size, and the sites that rebuild + # InstancesRLEMasks drop it, so a reduced grid would be reinterpreted as + # image-sized downstream + source = _source(version, tensor=True) + + # then + assert source.count('mask_decode_mode="accurate",') == 2 + assert source.count("mask_decode_mode=mask_decode_mode,") == 2 + + +class TestPinningIsObservedAtTheCallSite: + """Count pinned outbound call sites rather than grepping for a string. + + A version has exactly two places where the request leaves the block: + `run_locally` and `run_remotely`. Asserting both are pinned catches a + missed call site, which a substring search does not - the tensor-native + local path was unpinned while the file still contained the pinned string. + + This is still source analysis. Exercising the block itself would be + stronger and needs the WorkflowBlock init machinery. + """ + + @pytest.mark.parametrize("version", PINNED_VERSIONS) + @pytest.mark.parametrize("tensor", [False, True]) + def test_legacy_versions_send_accurate_whatever_the_caller_set( + self, version: str, tensor: bool + ) -> None: + # given + # the caller asks for `fast`; a pinned version must not forward it + source = _source(version, tensor) + outbound = [ + line + for line in source.split("\n") + if "mask_decode_mode=" in line and "mask_decode_mode=mask_decode_mode" not in line + ] + + # then + # one pinned value per outbound call site: run_locally and run_remotely + assert len(outbound) == 2, ( + f"{version}{'_tensor' if tensor else ''} has {len(outbound)} pinned " + "outbound call sites, expected 2 (local and remote)" + ) + assert all('"accurate"' in line for line in outbound) + + def test_v5_non_tensor_has_no_pinned_call_sites(self) -> None: + # given + source = _source("v5", tensor=False) + + # then + assert 'mask_decode_mode="accurate",' not in source + assert source.count("mask_decode_mode=mask_decode_mode,") == 4 diff --git a/workflows/CHANGELOG.md b/workflows/CHANGELOG.md index ce4848aae7..c9efbde021 100644 --- a/workflows/CHANGELOG.md +++ b/workflows/CHANGELOG.md @@ -18,6 +18,26 @@ for contributor and maintainer responsibilities. ### Added +- `roboflow_core/roboflow_instance_segmentation_model@v5`. Identical to `@v4` + except that it forwards `mask_decode_mode` and `tradeoff_factor` to the + server, which now honours them. Versions up to `@v4` pin the outbound request + to `accurate` / `1.0`, so existing workflows are unaffected on the + `inference_models` backend. + + Two caveats. On the legacy ORT backend - the default on Windows - those + fields *were* honoured, so pinning changes behaviour there: a `@v1`-`@v4` + workflow set to `fast` now receives full-resolution masks. This is a + deliberate trade for consistency across backends. + + The tensor-native siblings of every version, `@v5` included, are also pinned. + Nothing under `roboflow_workflows/` reads `InstanceDetections.mask_size`, and + the sites that rebuild `InstancesRLEMasks` drop it, so a reduced grid would + be reinterpreted as image-sized downstream. Reduced mask resolution is + therefore available in non-tensor mode only, pending propagation of + `mask_size` through the tensor pipeline. + +### Added + - Anthropic Claude block (`anthropic_claude@v5`): `claude-sonnet-5-5` model option. ### Changed diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1.py index 42e53769ce..71763737ef 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1.py @@ -319,8 +319,11 @@ def run_locally( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned: these fields never took effect in this version, so - # honouring them now would change existing workflows. @v5 forwards. + # Pinned: on the inference_models backend these fields were + # discarded, so honouring them now would change existing + # workflows. Note the legacy ORT backend DOES honour them + # today, so pinning changes behaviour there - a deliberate + # trade for consistency across backends. @v5 forwards. mask_decode_mode="accurate", tradeoff_factor=1.0, enforce_dense_masks_in_inference_models=enforce_dense_masks_in_inference_models, @@ -369,8 +372,11 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned: these fields never took effect in this version, so - # honouring them now would change existing workflows. @v5 forwards. + # Pinned: on the inference_models backend these fields were + # discarded, so honouring them now would change existing + # workflows. Note the legacy ORT backend DOES honour them + # today, so pinning changes behaviour there - a deliberate + # trade for consistency across backends. @v5 forwards. mask_decode_mode="accurate", tradeoff_factor=1.0, max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1_tensor.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1_tensor.py index 865b6f722b..3a0baefe15 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1_tensor.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1_tensor.py @@ -391,8 +391,11 @@ def run_locally( class_filter=class_filter, max_detections=max_detections, max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, + # Pinned: these fields never took effect in this version via the + # inference_models backend, so honouring them now would change + # existing workflows. @v5 forwards. + mask_decode_mode="accurate", + tradeoff_factor=1.0, disable_active_learning=disable_active_learning, active_learning_target_dataset=active_learning_target_dataset, enforce_dense_masks_in_inference_models=( @@ -465,8 +468,11 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned: these fields never took effect in this version, so - # honouring them now would change existing workflows. @v5 forwards. + # Pinned: on the inference_models backend these fields were + # discarded, so honouring them now would change existing + # workflows. Note the legacy ORT backend DOES honour them + # today, so pinning changes behaviour there - a deliberate + # trade for consistency across backends. @v5 forwards. mask_decode_mode="accurate", tradeoff_factor=1.0, response_mask_format="rle", diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2.py index f15e65def8..3768bedb7e 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2.py @@ -317,8 +317,11 @@ def run_locally( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned: these fields never took effect in this version, so - # honouring them now would change existing workflows. @v5 forwards. + # Pinned: on the inference_models backend these fields were + # discarded, so honouring them now would change existing + # workflows. Note the legacy ORT backend DOES honour them + # today, so pinning changes behaviour there - a deliberate + # trade for consistency across backends. @v5 forwards. mask_decode_mode="accurate", tradeoff_factor=1.0, enforce_dense_masks_in_inference_models=enforce_dense_masks_in_inference_models, @@ -368,8 +371,11 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned: these fields never took effect in this version, so - # honouring them now would change existing workflows. @v5 forwards. + # Pinned: on the inference_models backend these fields were + # discarded, so honouring them now would change existing + # workflows. Note the legacy ORT backend DOES honour them + # today, so pinning changes behaviour there - a deliberate + # trade for consistency across backends. @v5 forwards. mask_decode_mode="accurate", tradeoff_factor=1.0, max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2_tensor.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2_tensor.py index 62909f6477..cafd600fc8 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2_tensor.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2_tensor.py @@ -389,8 +389,11 @@ def run_locally( class_filter=class_filter, max_detections=max_detections, max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, + # Pinned: these fields never took effect in this version via the + # inference_models backend, so honouring them now would change + # existing workflows. @v5 forwards. + mask_decode_mode="accurate", + tradeoff_factor=1.0, disable_active_learning=disable_active_learning, active_learning_target_dataset=active_learning_target_dataset, enforce_dense_masks_in_inference_models=( @@ -464,8 +467,11 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned: these fields never took effect in this version, so - # honouring them now would change existing workflows. @v5 forwards. + # Pinned: on the inference_models backend these fields were + # discarded, so honouring them now would change existing + # workflows. Note the legacy ORT backend DOES honour them + # today, so pinning changes behaviour there - a deliberate + # trade for consistency across backends. @v5 forwards. mask_decode_mode="accurate", tradeoff_factor=1.0, response_mask_format="rle", diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3.py index e1cfeef8bc..aaee1c2299 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3.py @@ -392,8 +392,11 @@ def run_locally( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned: these fields never took effect in this version, so - # honouring them now would change existing workflows. @v5 forwards. + # Pinned: on the inference_models backend these fields were + # discarded, so honouring them now would change existing + # workflows. Note the legacy ORT backend DOES honour them + # today, so pinning changes behaviour there - a deliberate + # trade for consistency across backends. @v5 forwards. mask_decode_mode="accurate", tradeoff_factor=1.0, enforce_dense_masks_in_inference_models=enforce_dense_masks_in_inference_models, @@ -705,8 +708,11 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned: these fields never took effect in this version, so - # honouring them now would change existing workflows. @v5 forwards. + # Pinned: on the inference_models backend these fields were + # discarded, so honouring them now would change existing + # workflows. Note the legacy ORT backend DOES honour them + # today, so pinning changes behaviour there - a deliberate + # trade for consistency across backends. @v5 forwards. mask_decode_mode="accurate", tradeoff_factor=1.0, max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3_tensor.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3_tensor.py index 8f9abeaf14..6257640363 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3_tensor.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3_tensor.py @@ -430,8 +430,11 @@ def run_locally( class_filter=class_filter, max_detections=max_detections, max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, + # Pinned: these fields never took effect in this version via the + # inference_models backend, so honouring them now would change + # existing workflows. @v5 forwards. + mask_decode_mode="accurate", + tradeoff_factor=1.0, disable_active_learning=disable_active_learning, active_learning_target_dataset=active_learning_target_dataset, enforce_dense_masks_in_inference_models=( @@ -505,8 +508,11 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned: these fields never took effect in this version, so - # honouring them now would change existing workflows. @v5 forwards. + # Pinned: on the inference_models backend these fields were + # discarded, so honouring them now would change existing + # workflows. Note the legacy ORT backend DOES honour them + # today, so pinning changes behaviour there - a deliberate + # trade for consistency across backends. @v5 forwards. mask_decode_mode="accurate", tradeoff_factor=1.0, response_mask_format="rle", diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py index bd4a0a0de1..42836bdd6a 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py @@ -354,8 +354,11 @@ def run_locally( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned: these fields never took effect in this version, so - # honouring them now would change existing workflows. @v5 forwards. + # Pinned: on the inference_models backend these fields were + # discarded, so honouring them now would change existing + # workflows. Note the legacy ORT backend DOES honour them + # today, so pinning changes behaviour there - a deliberate + # trade for consistency across backends. @v5 forwards. mask_decode_mode="accurate", tradeoff_factor=1.0, response_mask_format="rle", @@ -405,8 +408,11 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned: these fields never took effect in this version, so - # honouring them now would change existing workflows. @v5 forwards. + # Pinned: on the inference_models backend these fields were + # discarded, so honouring them now would change existing + # workflows. Note the legacy ORT backend DOES honour them + # today, so pinning changes behaviour there - a deliberate + # trade for consistency across backends. @v5 forwards. mask_decode_mode="accurate", tradeoff_factor=1.0, response_mask_format="rle", diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4_tensor.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4_tensor.py index fb1b436983..e8f8b9b4a2 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4_tensor.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4_tensor.py @@ -421,8 +421,11 @@ def run_locally( class_filter=class_filter, max_detections=max_detections, max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, + # Pinned: these fields never took effect in this version via the + # inference_models backend, so honouring them now would change + # existing workflows. @v5 forwards. + mask_decode_mode="accurate", + tradeoff_factor=1.0, disable_active_learning=disable_active_learning, active_learning_target_dataset=active_learning_target_dataset, ) @@ -492,8 +495,11 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned: these fields never took effect in this version, so - # honouring them now would change existing workflows. @v5 forwards. + # Pinned: on the inference_models backend these fields were + # discarded, so honouring them now would change existing + # workflows. Note the legacy ORT backend DOES honour them + # today, so pinning changes behaviour there - a deliberate + # trade for consistency across backends. @v5 forwards. mask_decode_mode="accurate", tradeoff_factor=1.0, response_mask_format="rle", diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py index c9838cb7ba..def7001441 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py @@ -421,8 +421,15 @@ def run_locally( class_filter=class_filter, max_detections=max_detections, max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, + # Tensor-native mode does not support reduced mask resolution yet. + # Nothing under workflows/ reads InstanceDetections.mask_size, and the + # ~35 sites that rebuild InstancesRLEMasks drop it, so a reduced grid + # would be reinterpreted as image-sized downstream. Upsampling at this + # boundary would restore correctness but produce output identical to + # factor 1.0 at strictly higher cost, so the request is pinned instead. + # Lift this once mask_size is propagated through the tensor pipeline. + mask_decode_mode="accurate", + tradeoff_factor=1.0, disable_active_learning=disable_active_learning, active_learning_target_dataset=active_learning_target_dataset, ) @@ -492,8 +499,15 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, + # Tensor-native mode does not support reduced mask resolution yet. + # Nothing under workflows/ reads InstanceDetections.mask_size, and the + # ~35 sites that rebuild InstancesRLEMasks drop it, so a reduced grid + # would be reinterpreted as image-sized downstream. Upsampling at this + # boundary would restore correctness but produce output identical to + # factor 1.0 at strictly higher cost, so the request is pinned instead. + # Lift this once mask_size is propagated through the tensor pipeline. + mask_decode_mode="accurate", + tradeoff_factor=1.0, response_mask_format="rle", max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, diff --git a/workflows/roboflow_workflows/execution_engine/entities/types.py b/workflows/roboflow_workflows/execution_engine/entities/types.py index ee525ce253..f5546ea90e 100644 --- a/workflows/roboflow_workflows/execution_engine/entities/types.py +++ b/workflows/roboflow_workflows/execution_engine/entities/types.py @@ -828,6 +828,12 @@ def __hash__(self) -> int: The masks are stored in `sv.Detections.data["rle_mask"]` as a numpy array of RLE dictionaries in COCO format: `{"size": [H, W], "counts": "encoded_string"}`. +`size` describes the grid the counts were encoded on, which is the image grid unless the +producing block reduced mask resolution - see `tradeoff_factor` on +`roboflow_instance_segmentation_model@v5`. The counts must be decoded on `size`; decoding +them on the image dimensions makes pycocotools reinterpret the runs without raising. +`sv.Detections.mask`, when materialised, is always at image resolution. + When visualization blocks need numpy masks, they decode RLE on-demand using pycocotools. Example: From 50c4b39cbabf6a021b60fd5adad580483c5080cd Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Fri, 2 Oct 2026 19:34:11 +0200 Subject: [PATCH 17/41] Document the resolution endpoints and their non-monotonic case Review raised that "fast" and "accurate" mislead when the image is smaller than the model's mask grid. resolve_mask_target_size interpolates between the mask grid at 0.0 and size_after_pre_processing at 1.0, so when the image is the smaller of the two a lower factor produces MORE mask pixels, not fewer: with a 160x160 grid and a 100x100 image, 0.0 yields 25,600 pixels against 10,000 at 1.0. States the endpoints explicitly in the function docstring and in the tradeoff_factor field description on both @v5 files, including that a lower factor does not guarantee smaller masks or lower latency, and that callers sensitive to this should compare size_after_pre_processing against the mask grid rather than assume a direction. Documentation only; no behaviour change. --- .../models/common/roboflow/post_processing.py | 17 +++++++++++++---- .../models/roboflow/instance_segmentation/v5.py | 2 +- .../roboflow/instance_segmentation/v5_tensor.py | 2 +- 3 files changed, 15 insertions(+), 6 deletions(-) diff --git a/inference_models/inference_models/models/common/roboflow/post_processing.py b/inference_models/inference_models/models/common/roboflow/post_processing.py index 5c64ac7da8..7e35fa2e0a 100644 --- a/inference_models/inference_models/models/common/roboflow/post_processing.py +++ b/inference_models/inference_models/models/common/roboflow/post_processing.py @@ -447,10 +447,19 @@ def resolve_mask_target_size( ) -> Tuple[int, int]: """Interpolate the mask resize target between the mask grid and the image. - A factor of `1.0` selects the image size, reproducing the behaviour from - before this parameter existed. A factor of `0.0` leaves masks on the grid - the model produced them on. Values in between trade mask fidelity for the - cost of upsampling every instance. + The endpoints are explicit: `1.0` selects `size_after_pre_processing`, + reproducing the behaviour from before this parameter existed, and `0.0` + selects the grid the model produced masks on. Values in between interpolate + linearly between those two sizes. + + **A lower factor does not guarantee smaller masks or lower latency.** When + the image is smaller than the model's mask grid the interpolation runs the + other way: with a 160x160 grid and a 100x100 image, `0.0` yields 25,600 + mask pixels against 10,000 at `1.0`. The `fast` and `accurate` names on + `mask_decode_mode` describe the common case where the image is larger than + the grid, and are misleading below it. Callers sensitive to this should + compare `size_after_pre_processing` against the mask grid rather than + assume a direction. Args: mask_height: Height of the mask grid after letterbox padding is removed. diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5.py index 66b9922900..6a465f3768 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5.py @@ -175,7 +175,7 @@ class BlockManifest(WorkflowBlockManifest): Selector(kind=[FLOAT_ZERO_TO_ONE_KIND]), ] = Field( default=0.0, - description="Post-processing parameter to dictate tradeoff between fast and accurate.", + description="Post-processing parameter to dictate tradeoff between fast and accurate. 0.0 keeps masks on the model's own grid, 1.0 resizes them to the image. Note that a lower value does not guarantee smaller masks: when the image is smaller than the model grid the interpolation runs the other way.", examples=[0.3, "$inputs.tradeoff_factor"], ) disable_active_learning: Union[bool, Selector(kind=[BOOLEAN_KIND])] = Field( diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py index def7001441..60589ae4db 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py @@ -235,7 +235,7 @@ class BlockManifest(WorkflowBlockManifest): Selector(kind=[FLOAT_ZERO_TO_ONE_KIND]), ] = Field( default=0.0, - description="Post-processing parameter to dictate tradeoff between fast and accurate.", + description="Post-processing parameter to dictate tradeoff between fast and accurate. 0.0 keeps masks on the model's own grid, 1.0 resizes them to the image. Note that a lower value does not guarantee smaller masks: when the image is smaller than the model grid the interpolation runs the other way.", examples=[0.3, "$inputs.tradeoff_factor"], ) disable_active_learning: Union[bool, Selector(kind=[BOOLEAN_KIND])] = Field( From 970e47f3073ce121660e1e235a788c543210854c Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Fri, 2 Oct 2026 20:14:37 +0200 Subject: [PATCH 18/41] Make the legacy pin conditional and document the configuration field Completes the conditional pinning across the non-tensor block versions: v1-v4 now send accurate/1.0 only when USE_INFERENCE_MODELS is set, and pass the caller's values through otherwise. The legacy backend honours these fields, so those versions behave on it exactly as they do today. The tensor-native siblings stay pinned unconditionally. Their reason is downstream rather than the backend: nothing under roboflow_workflows/ reads mask_size and the sites rebuilding InstancesRLEMasks drop it, so a reduced grid is reinterpreted as image-sized whichever backend produced it. The comments now state which reason applies where. Documents ModelsConfiguration.use_inference_models in the changelog. It is a public field and a host serving through the legacy backend must set it to False, or v1-v4 stop forwarding fields that backend honours - reintroducing the regression this change exists to avoid. Applies black and isort. Eight files needed reformatting, all of them introduced by this branch; the equivalents on origin/main were already clean. Tests: inference_models 2242 passed, workflows 1274 passed, inference core/models 129 passed. black, isort and flake8 clean across all 47 changed Python files. --- .../models/base/instance_segmentation.py | 24 +- .../models/common/roboflow/post_processing.py | 8 +- .../common/roboflow/test_post_processing.py | 308 ++++++++++++++++-- .../models/test_inference_models_adapters.py | 43 ++- workflows/CHANGELOG.md | 7 + .../roboflow/instance_segmentation/v1.py | 10 +- .../roboflow/instance_segmentation/v2.py | 10 +- .../roboflow/instance_segmentation/v3.py | 10 +- .../roboflow/instance_segmentation/v4.py | 10 +- 9 files changed, 348 insertions(+), 82 deletions(-) diff --git a/inference_models/inference_models/models/base/instance_segmentation.py b/inference_models/inference_models/models/base/instance_segmentation.py index 617a270b9c..197a7bc861 100644 --- a/inference_models/inference_models/models/base/instance_segmentation.py +++ b/inference_models/inference_models/models/base/instance_segmentation.py @@ -272,17 +272,19 @@ def to_supervision(self) -> sv.Detections: if mask is not None and self.mask_size is not None: image_height, image_width = self._image_size() if tuple(self.mask_size) != (image_height, image_width): - mask = np.stack( - [ - cv2.resize( - single.astype(np.uint8), - (image_width, image_height), - interpolation=cv2.INTER_NEAREST, - ).astype(bool) - for single in mask - ] - ) if len(mask) else np.zeros( - (0, image_height, image_width), dtype=bool + mask = ( + np.stack( + [ + cv2.resize( + single.astype(np.uint8), + (image_width, image_height), + interpolation=cv2.INTER_NEAREST, + ).astype(bool) + for single in mask + ] + ) + if len(mask) + else np.zeros((0, image_height, image_width), dtype=bool) ) return sv.Detections( xyxy=self.xyxy.cpu().numpy(), diff --git a/inference_models/inference_models/models/common/roboflow/post_processing.py b/inference_models/inference_models/models/common/roboflow/post_processing.py index 7e35fa2e0a..13fffebb55 100644 --- a/inference_models/inference_models/models/common/roboflow/post_processing.py +++ b/inference_models/inference_models/models/common/roboflow/post_processing.py @@ -512,11 +512,11 @@ def resolve_unpadded_mask_grid( pad_left, pad_top, pad_right, pad_bottom = padding height_scale = mask_height / inference_size.height width_scale = mask_width / inference_size.width - unpadded_height = mask_height - round(height_scale * pad_top) - round( - height_scale * pad_bottom + unpadded_height = ( + mask_height - round(height_scale * pad_top) - round(height_scale * pad_bottom) ) - unpadded_width = mask_width - round(width_scale * pad_left) - round( - width_scale * pad_right + unpadded_width = ( + mask_width - round(width_scale * pad_left) - round(width_scale * pad_right) ) return max(1, unpadded_height), max(1, unpadded_width) diff --git a/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py b/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py index 59548c768d..04c087574d 100644 --- a/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py +++ b/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py @@ -969,30 +969,270 @@ def test_matches_the_dense_path_resolution(self) -> None: # function these cases SHOULD fail - that is the point of a characterization # test. Regenerate only after deciding the new behaviour is intended. _MAIN_GOLDEN = [ - {"ih": 333, "iw": 211, "ox": 97, "oy": 61, "pad": (16, 18, 6, 5), "n": 5, "shape": (5, 394, 308), "digest": "a48aa710ef85cd88", "boxes": "b62c110b17ae5441"}, - {"ih": 333, "iw": 211, "ox": 0, "oy": 0, "pad": (14, 9, 4, 2), "n": 5, "shape": (5, 333, 211), "digest": "5d04de5cff404410", "boxes": "5b0e73b0ea74de77"}, - {"ih": 200, "iw": 300, "ox": 97, "oy": 61, "pad": (20, 23, 19, 20), "n": 0, "shape": (0, 261, 397), "digest": "e3b0c44298fc1c14", "boxes": "e3b0c44298fc1c14"}, - {"ih": 200, "iw": 300, "ox": 0, "oy": 0, "pad": (1, 6, 7, 19), "n": 0, "shape": (0, 200, 300), "digest": "e3b0c44298fc1c14", "boxes": "e3b0c44298fc1c14"}, - {"ih": 333, "iw": 211, "ox": 13, "oy": 61, "pad": (18, 6, 16, 7), "n": 5, "shape": (5, 394, 224), "digest": "a37811a210846e79", "boxes": "8cbae67b51c4aa02"}, - {"ih": 640, "iw": 640, "ox": 97, "oy": 0, "pad": (21, 2, 14, 20), "n": 1, "shape": (1, 640, 737), "digest": "f88d6c3e4f5c03f7", "boxes": "e1cacd18dd5fddf7"}, - {"ih": 333, "iw": 211, "ox": 0, "oy": 7, "pad": (10, 24, 7, 16), "n": 1, "shape": (1, 340, 211), "digest": "d15ccb8dd99581e4", "boxes": "e47e86508db619f6"}, - {"ih": 200, "iw": 300, "ox": 0, "oy": 0, "pad": (12, 3, 9, 12), "n": 0, "shape": (0, 200, 300), "digest": "e3b0c44298fc1c14", "boxes": "e3b0c44298fc1c14"}, - {"ih": 200, "iw": 300, "ox": 0, "oy": 0, "pad": (6, 1, 15, 12), "n": 5, "shape": (5, 200, 300), "digest": "baaed46f79ba3102", "boxes": "37313f9472c62672"}, - {"ih": 333, "iw": 211, "ox": 97, "oy": 0, "pad": (18, 20, 6, 24), "n": 5, "shape": (5, 333, 308), "digest": "c760dd33356b0b6d", "boxes": "6b53dff165635fee"}, - {"ih": 640, "iw": 640, "ox": 13, "oy": 0, "pad": (9, 10, 0, 13), "n": 0, "shape": (0, 640, 653), "digest": "e3b0c44298fc1c14", "boxes": "e3b0c44298fc1c14"}, - {"ih": 1080, "iw": 1920, "ox": 0, "oy": 0, "pad": (0, 1, 14, 15), "n": 0, "shape": (0, 1080, 1920), "digest": "e3b0c44298fc1c14", "boxes": "e3b0c44298fc1c14"}, - {"ih": 1080, "iw": 1920, "ox": 97, "oy": 0, "pad": (23, 24, 4, 13), "n": 5, "shape": (5, 1080, 2017), "digest": "6982b829d1bbddc7", "boxes": "4c219acad8807667"}, - {"ih": 333, "iw": 211, "ox": 0, "oy": 61, "pad": (13, 6, 0, 8), "n": 5, "shape": (5, 394, 211), "digest": "87fcc86e47722220", "boxes": "b21927719126483f"}, - {"ih": 640, "iw": 640, "ox": 0, "oy": 0, "pad": (5, 12, 19, 20), "n": 5, "shape": (5, 640, 640), "digest": "846517457eb562ae", "boxes": "3cedb68b2e60e5f6"}, - {"ih": 200, "iw": 300, "ox": 0, "oy": 0, "pad": (6, 14, 8, 0), "n": 5, "shape": (5, 200, 300), "digest": "6287b10e34217d2e", "boxes": "bce5175b45d9adde"}, - {"ih": 640, "iw": 640, "ox": 13, "oy": 61, "pad": (2, 2, 2, 6), "n": 5, "shape": (5, 701, 653), "digest": "d8db8ae532bdb7ea", "boxes": "6798272b81011bb4"}, - {"ih": 1080, "iw": 1920, "ox": 0, "oy": 7, "pad": (11, 19, 14, 4), "n": 5, "shape": (5, 1087, 1920), "digest": "c9f9413897eefdd0", "boxes": "749ad903686c1c93"}, - {"ih": 333, "iw": 211, "ox": 0, "oy": 61, "pad": (5, 20, 4, 9), "n": 0, "shape": (0, 394, 211), "digest": "e3b0c44298fc1c14", "boxes": "e3b0c44298fc1c14"}, - {"ih": 1080, "iw": 1920, "ox": 0, "oy": 0, "pad": (23, 20, 17, 6), "n": 5, "shape": (5, 1080, 1920), "digest": "fe15d5708b271c8f", "boxes": "b6b89ee145f20853"}, - {"ih": 333, "iw": 211, "ox": 97, "oy": 0, "pad": (13, 1, 3, 3), "n": 0, "shape": (0, 333, 308), "digest": "e3b0c44298fc1c14", "boxes": "e3b0c44298fc1c14"}, - {"ih": 640, "iw": 640, "ox": 0, "oy": 61, "pad": (8, 13, 19, 15), "n": 1, "shape": (1, 701, 640), "digest": "fbb5ace53bca0cf5", "boxes": "48123ae5fa70ad76"}, - {"ih": 1080, "iw": 1920, "ox": 0, "oy": 0, "pad": (7, 15, 17, 20), "n": 5, "shape": (5, 1080, 1920), "digest": "be0ffbd3b025fc6b", "boxes": "bcc7d8853de382db"}, - {"ih": 200, "iw": 300, "ox": 13, "oy": 0, "pad": (6, 23, 0, 2), "n": 1, "shape": (1, 200, 313), "digest": "4f01e63780b7b135", "boxes": "404d5eb899660b53"}, + { + "ih": 333, + "iw": 211, + "ox": 97, + "oy": 61, + "pad": (16, 18, 6, 5), + "n": 5, + "shape": (5, 394, 308), + "digest": "a48aa710ef85cd88", + "boxes": "b62c110b17ae5441", + }, + { + "ih": 333, + "iw": 211, + "ox": 0, + "oy": 0, + "pad": (14, 9, 4, 2), + "n": 5, + "shape": (5, 333, 211), + "digest": "5d04de5cff404410", + "boxes": "5b0e73b0ea74de77", + }, + { + "ih": 200, + "iw": 300, + "ox": 97, + "oy": 61, + "pad": (20, 23, 19, 20), + "n": 0, + "shape": (0, 261, 397), + "digest": "e3b0c44298fc1c14", + "boxes": "e3b0c44298fc1c14", + }, + { + "ih": 200, + "iw": 300, + "ox": 0, + "oy": 0, + "pad": (1, 6, 7, 19), + "n": 0, + "shape": (0, 200, 300), + "digest": "e3b0c44298fc1c14", + "boxes": "e3b0c44298fc1c14", + }, + { + "ih": 333, + "iw": 211, + "ox": 13, + "oy": 61, + "pad": (18, 6, 16, 7), + "n": 5, + "shape": (5, 394, 224), + "digest": "a37811a210846e79", + "boxes": "8cbae67b51c4aa02", + }, + { + "ih": 640, + "iw": 640, + "ox": 97, + "oy": 0, + "pad": (21, 2, 14, 20), + "n": 1, + "shape": (1, 640, 737), + "digest": "f88d6c3e4f5c03f7", + "boxes": "e1cacd18dd5fddf7", + }, + { + "ih": 333, + "iw": 211, + "ox": 0, + "oy": 7, + "pad": (10, 24, 7, 16), + "n": 1, + "shape": (1, 340, 211), + "digest": "d15ccb8dd99581e4", + "boxes": "e47e86508db619f6", + }, + { + "ih": 200, + "iw": 300, + "ox": 0, + "oy": 0, + "pad": (12, 3, 9, 12), + "n": 0, + "shape": (0, 200, 300), + "digest": "e3b0c44298fc1c14", + "boxes": "e3b0c44298fc1c14", + }, + { + "ih": 200, + "iw": 300, + "ox": 0, + "oy": 0, + "pad": (6, 1, 15, 12), + "n": 5, + "shape": (5, 200, 300), + "digest": "baaed46f79ba3102", + "boxes": "37313f9472c62672", + }, + { + "ih": 333, + "iw": 211, + "ox": 97, + "oy": 0, + "pad": (18, 20, 6, 24), + "n": 5, + "shape": (5, 333, 308), + "digest": "c760dd33356b0b6d", + "boxes": "6b53dff165635fee", + }, + { + "ih": 640, + "iw": 640, + "ox": 13, + "oy": 0, + "pad": (9, 10, 0, 13), + "n": 0, + "shape": (0, 640, 653), + "digest": "e3b0c44298fc1c14", + "boxes": "e3b0c44298fc1c14", + }, + { + "ih": 1080, + "iw": 1920, + "ox": 0, + "oy": 0, + "pad": (0, 1, 14, 15), + "n": 0, + "shape": (0, 1080, 1920), + "digest": "e3b0c44298fc1c14", + "boxes": "e3b0c44298fc1c14", + }, + { + "ih": 1080, + "iw": 1920, + "ox": 97, + "oy": 0, + "pad": (23, 24, 4, 13), + "n": 5, + "shape": (5, 1080, 2017), + "digest": "6982b829d1bbddc7", + "boxes": "4c219acad8807667", + }, + { + "ih": 333, + "iw": 211, + "ox": 0, + "oy": 61, + "pad": (13, 6, 0, 8), + "n": 5, + "shape": (5, 394, 211), + "digest": "87fcc86e47722220", + "boxes": "b21927719126483f", + }, + { + "ih": 640, + "iw": 640, + "ox": 0, + "oy": 0, + "pad": (5, 12, 19, 20), + "n": 5, + "shape": (5, 640, 640), + "digest": "846517457eb562ae", + "boxes": "3cedb68b2e60e5f6", + }, + { + "ih": 200, + "iw": 300, + "ox": 0, + "oy": 0, + "pad": (6, 14, 8, 0), + "n": 5, + "shape": (5, 200, 300), + "digest": "6287b10e34217d2e", + "boxes": "bce5175b45d9adde", + }, + { + "ih": 640, + "iw": 640, + "ox": 13, + "oy": 61, + "pad": (2, 2, 2, 6), + "n": 5, + "shape": (5, 701, 653), + "digest": "d8db8ae532bdb7ea", + "boxes": "6798272b81011bb4", + }, + { + "ih": 1080, + "iw": 1920, + "ox": 0, + "oy": 7, + "pad": (11, 19, 14, 4), + "n": 5, + "shape": (5, 1087, 1920), + "digest": "c9f9413897eefdd0", + "boxes": "749ad903686c1c93", + }, + { + "ih": 333, + "iw": 211, + "ox": 0, + "oy": 61, + "pad": (5, 20, 4, 9), + "n": 0, + "shape": (0, 394, 211), + "digest": "e3b0c44298fc1c14", + "boxes": "e3b0c44298fc1c14", + }, + { + "ih": 1080, + "iw": 1920, + "ox": 0, + "oy": 0, + "pad": (23, 20, 17, 6), + "n": 5, + "shape": (5, 1080, 1920), + "digest": "fe15d5708b271c8f", + "boxes": "b6b89ee145f20853", + }, + { + "ih": 333, + "iw": 211, + "ox": 97, + "oy": 0, + "pad": (13, 1, 3, 3), + "n": 0, + "shape": (0, 333, 308), + "digest": "e3b0c44298fc1c14", + "boxes": "e3b0c44298fc1c14", + }, + { + "ih": 640, + "iw": 640, + "ox": 0, + "oy": 61, + "pad": (8, 13, 19, 15), + "n": 1, + "shape": (1, 701, 640), + "digest": "fbb5ace53bca0cf5", + "boxes": "48123ae5fa70ad76", + }, + { + "ih": 1080, + "iw": 1920, + "ox": 0, + "oy": 0, + "pad": (7, 15, 17, 20), + "n": 5, + "shape": (5, 1080, 1920), + "digest": "be0ffbd3b025fc6b", + "boxes": "bcc7d8853de382db", + }, + { + "ih": 200, + "iw": 300, + "ox": 13, + "oy": 0, + "pad": (6, 23, 0, 2), + "n": 1, + "shape": (1, 200, 313), + "digest": "4f01e63780b7b135", + "boxes": "404d5eb899660b53", + }, ] @@ -1055,9 +1295,7 @@ def test_content_matches_main(self, case: dict) -> None: assert ( hashlib.sha256(masks.numpy().tobytes()).hexdigest()[:16] == case["digest"] ) - assert ( - hashlib.sha256(boxes.numpy().tobytes()).hexdigest()[:16] == case["boxes"] - ) + assert hashlib.sha256(boxes.numpy().tobytes()).hexdigest()[:16] == case["boxes"] class TestStaticCropAtReducedResolution: @@ -1091,14 +1329,22 @@ def _letterboxed(image_w: int, image_h: int, offset_x: int, offset_y: int): return { "image_bboxes": torch.tensor([[1.0, 1, 50, 50, 0.9, 0]]), "masks": torch.randn(1, 160, 160), - "padding": (pad_left, pad_top, 640 - new_w - pad_left, 640 - new_h - pad_top), + "padding": ( + pad_left, + pad_top, + 640 - new_w - pad_left, + 640 - new_h - pad_top, + ), "scale_width": scale, "scale_height": scale, "original_size": ImageDimensions(height=image_h, width=image_w), "size_after_pre_processing": ImageDimensions(height=crop_h, width=crop_w), "inference_size": ImageDimensions(height=640, width=640), "static_crop_offset": StaticCropOffset( - offset_x=offset_x, offset_y=offset_y, crop_width=crop_w, crop_height=crop_h + offset_x=offset_x, + offset_y=offset_y, + crop_width=crop_w, + crop_height=crop_h, ), } @@ -1119,7 +1365,9 @@ def test_dense_path_places_the_mask_inside_the_canvas( # when _, masks = align_instance_segmentation_results( - **kwargs, binarization_threshold=0.0, mask_chunk_size=16, + **kwargs, + binarization_threshold=0.0, + mask_chunk_size=16, masks_resolution_factor=factor, ) diff --git a/tests/inference/unit_tests/core/models/test_inference_models_adapters.py b/tests/inference/unit_tests/core/models/test_inference_models_adapters.py index 890792e935..327b971353 100644 --- a/tests/inference/unit_tests/core/models/test_inference_models_adapters.py +++ b/tests/inference/unit_tests/core/models/test_inference_models_adapters.py @@ -6,9 +6,9 @@ from unittest.mock import MagicMock import numpy as np -from pycocotools import mask as mask_utils import pytest import torch +from pycocotools import mask as mask_utils from inference.core.entities.requests.embeddings import ImageEmbeddingRequest from inference.core.entities.responses.embeddings import ImageEmbeddingResponse @@ -16,7 +16,6 @@ InstanceSegmentationInferenceResponse, InstanceSegmentationInferenceResponseDC, ) -from inference_models.models.base.types import InstancesRLEMasks from inference.core.exceptions import InvalidMaskDecodeArgument, PostProcessingError from inference.core.models.inference_models_adapters import ( InferenceModelsClassificationAdapter, @@ -37,6 +36,7 @@ ) from inference_models.models.auto_loaders.entities import PreProcessingOverrides from inference_models.models.base.async_handoff import attach_adapter_mapped_kwargs +from inference_models.models.base.types import InstancesRLEMasks from inference_models.utils.onnx_embeddings import embedding_definition @@ -432,7 +432,12 @@ def test_workflow_response_fast_dataclass_path_is_disabled_at_depth_one() -> Non adapter = object.__new__(InferenceModelsInstanceSegmentationAdapter) adapter._pipeline_depth = 1 adapter.class_names = ["car"] - metadata = [SimpleNamespace(original_size=SimpleNamespace(width=4, height=4), size_after_pre_processing=SimpleNamespace(width=4, height=4))] + metadata = [ + SimpleNamespace( + original_size=SimpleNamespace(width=4, height=4), + size_after_pre_processing=SimpleNamespace(width=4, height=4), + ) + ] detections = [ InstanceDetections( xyxy=torch.tensor([[1, 1, 3, 3]], dtype=torch.int32), @@ -455,7 +460,12 @@ def test_workflow_response_fast_dataclass_path_is_enabled_above_depth_one() -> N adapter = object.__new__(InferenceModelsInstanceSegmentationAdapter) adapter._pipeline_depth = 2 adapter.class_names = ["car"] - metadata = [SimpleNamespace(original_size=SimpleNamespace(width=4, height=4), size_after_pre_processing=SimpleNamespace(width=4, height=4))] + metadata = [ + SimpleNamespace( + original_size=SimpleNamespace(width=4, height=4), + size_after_pre_processing=SimpleNamespace(width=4, height=4), + ) + ] detections = [ InstanceDetections( xyxy=torch.tensor([[1, 1, 3, 3]], dtype=torch.int32), @@ -905,7 +915,12 @@ def test_polygon_points_are_scaled_from_mask_space_into_image_space() -> None: # a 4x4 mask describing a 16x16 image: contours come out in mask # coordinates and must be lifted by 4x before they are reported adapter = _seg_adapter() - metadata = [SimpleNamespace(original_size=SimpleNamespace(width=16, height=16), size_after_pre_processing=SimpleNamespace(width=16, height=16))] + metadata = [ + SimpleNamespace( + original_size=SimpleNamespace(width=16, height=16), + size_after_pre_processing=SimpleNamespace(width=16, height=16), + ) + ] detections = [ InstanceDetections( xyxy=torch.tensor([[0, 0, 16, 16]], dtype=torch.int32), @@ -929,7 +944,12 @@ def test_polygon_points_unchanged_when_mask_matches_the_image() -> None: # given # today's situation: the scaling must be an exact no-op adapter = _seg_adapter() - metadata = [SimpleNamespace(original_size=SimpleNamespace(width=8, height=8), size_after_pre_processing=SimpleNamespace(width=8, height=8))] + metadata = [ + SimpleNamespace( + original_size=SimpleNamespace(width=8, height=8), + size_after_pre_processing=SimpleNamespace(width=8, height=8), + ) + ] mask = torch.zeros((1, 8, 8), dtype=torch.uint8) mask[0, 2:6, 2:6] = 1 detections = [ @@ -1031,9 +1051,14 @@ def _carrier(mask_h: int, mask_w: int, image_h: int, image_w: int): dense[mask_h // 4 : mask_h // 2, mask_w // 4 : mask_w // 2] = 1 counts = mask_utils.encode(np.asfortranarray(dense))["counts"] - return InstancesRLEMasks( - image_size=(image_h, image_w), masks=[counts], mask_size=(mask_h, mask_w) - ), dense + return ( + InstancesRLEMasks( + image_size=(image_h, image_w), + masks=[counts], + mask_size=(mask_h, mask_w), + ), + dense, + ) def test_polygon_follows_the_encoded_grid_not_the_image(self) -> None: # given diff --git a/workflows/CHANGELOG.md b/workflows/CHANGELOG.md index 3d079f4c94..93d884e1fc 100644 --- a/workflows/CHANGELOG.md +++ b/workflows/CHANGELOG.md @@ -40,6 +40,13 @@ for contributor and maintainer responsibilities. behave exactly as they do today; the pin applies only where the fields were being discarded. No existing workflow changes behaviour on either backend. +- `ModelsConfiguration.use_inference_models`, defaulting to `True`. Hosts that + serve instance segmentation through the legacy backend must set it to `False` + so block versions up to `@v4` keep forwarding `mask_decode_mode` and + `tradeoff_factor`, which that backend honours. The server sets it from + `inference.core.env.USE_INFERENCE_MODELS`; the value cannot be read from the + environment here, since this package must not import `inference.*`. + The tensor-native siblings of every version, `@v5` included, are pinned unconditionally - the reason there is downstream rather than the backend. Nothing under `roboflow_workflows/` reads `InstanceDetections.mask_size`, and diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1.py index efc07f29e7..576a92fdbb 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1.py @@ -9,9 +9,9 @@ filter_out_unwanted_classes_from_sv_detections_batch, ) from roboflow_workflows.environment import ( - USE_INFERENCE_MODELS, HOSTED_INSTANCE_SEGMENTATION_URL, LOCAL_INFERENCE_API_URL, + USE_INFERENCE_MODELS, WORKFLOWS_REMOTE_API_KEY_TRANSPORT, WORKFLOWS_REMOTE_API_TARGET, WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, @@ -324,9 +324,7 @@ def run_locally( # backend they are honoured, so this version keeps passing them # through and behaves exactly as it does today. @v5 forwards # unconditionally. - mask_decode_mode=( - "accurate" if USE_INFERENCE_MODELS else mask_decode_mode - ), + mask_decode_mode=("accurate" if USE_INFERENCE_MODELS else mask_decode_mode), tradeoff_factor=1.0 if USE_INFERENCE_MODELS else tradeoff_factor, enforce_dense_masks_in_inference_models=enforce_dense_masks_in_inference_models, disable_active_learning=disable_active_learning, @@ -378,9 +376,7 @@ def run_remotely( # backend they are honoured, so this version keeps passing them # through and behaves exactly as it does today. @v5 forwards # unconditionally. - mask_decode_mode=( - "accurate" if USE_INFERENCE_MODELS else mask_decode_mode - ), + mask_decode_mode=("accurate" if USE_INFERENCE_MODELS else mask_decode_mode), tradeoff_factor=1.0 if USE_INFERENCE_MODELS else tradeoff_factor, max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2.py index 30006ad921..88d7907740 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2.py @@ -9,9 +9,9 @@ filter_out_unwanted_classes_from_sv_detections_batch, ) from roboflow_workflows.environment import ( - USE_INFERENCE_MODELS, HOSTED_INSTANCE_SEGMENTATION_URL, LOCAL_INFERENCE_API_URL, + USE_INFERENCE_MODELS, WORKFLOWS_REMOTE_API_KEY_TRANSPORT, WORKFLOWS_REMOTE_API_TARGET, WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, @@ -322,9 +322,7 @@ def run_locally( # backend they are honoured, so this version keeps passing them # through and behaves exactly as it does today. @v5 forwards # unconditionally. - mask_decode_mode=( - "accurate" if USE_INFERENCE_MODELS else mask_decode_mode - ), + mask_decode_mode=("accurate" if USE_INFERENCE_MODELS else mask_decode_mode), tradeoff_factor=1.0 if USE_INFERENCE_MODELS else tradeoff_factor, enforce_dense_masks_in_inference_models=enforce_dense_masks_in_inference_models, disable_active_learning=disable_active_learning, @@ -377,9 +375,7 @@ def run_remotely( # backend they are honoured, so this version keeps passing them # through and behaves exactly as it does today. @v5 forwards # unconditionally. - mask_decode_mode=( - "accurate" if USE_INFERENCE_MODELS else mask_decode_mode - ), + mask_decode_mode=("accurate" if USE_INFERENCE_MODELS else mask_decode_mode), tradeoff_factor=1.0 if USE_INFERENCE_MODELS else tradeoff_factor, max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3.py index 8857aa5530..2508061e91 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3.py @@ -13,9 +13,9 @@ filter_out_unwanted_classes_from_sv_detections_batch, ) from roboflow_workflows.environment import ( - USE_INFERENCE_MODELS, HOSTED_INSTANCE_SEGMENTATION_URL, LOCAL_INFERENCE_API_URL, + USE_INFERENCE_MODELS, WORKFLOWS_ASYNC_FUTURE_RESULT_TIMEOUT, WORKFLOWS_REMOTE_API_KEY_TRANSPORT, WORKFLOWS_REMOTE_API_TARGET, @@ -397,9 +397,7 @@ def run_locally( # backend they are honoured, so this version keeps passing them # through and behaves exactly as it does today. @v5 forwards # unconditionally. - mask_decode_mode=( - "accurate" if USE_INFERENCE_MODELS else mask_decode_mode - ), + mask_decode_mode=("accurate" if USE_INFERENCE_MODELS else mask_decode_mode), tradeoff_factor=1.0 if USE_INFERENCE_MODELS else tradeoff_factor, enforce_dense_masks_in_inference_models=enforce_dense_masks_in_inference_models, disable_active_learning=disable_active_learning, @@ -714,9 +712,7 @@ def run_remotely( # backend they are honoured, so this version keeps passing them # through and behaves exactly as it does today. @v5 forwards # unconditionally. - mask_decode_mode=( - "accurate" if USE_INFERENCE_MODELS else mask_decode_mode - ), + mask_decode_mode=("accurate" if USE_INFERENCE_MODELS else mask_decode_mode), tradeoff_factor=1.0 if USE_INFERENCE_MODELS else tradeoff_factor, max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py index dc58d81023..f20af2bee8 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py @@ -9,9 +9,9 @@ filter_out_unwanted_classes_from_sv_detections_batch, ) from roboflow_workflows.environment import ( - USE_INFERENCE_MODELS, HOSTED_INSTANCE_SEGMENTATION_URL, LOCAL_INFERENCE_API_URL, + USE_INFERENCE_MODELS, WORKFLOWS_REMOTE_API_KEY_TRANSPORT, WORKFLOWS_REMOTE_API_TARGET, WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, @@ -359,9 +359,7 @@ def run_locally( # backend they are honoured, so this version keeps passing them # through and behaves exactly as it does today. @v5 forwards # unconditionally. - mask_decode_mode=( - "accurate" if USE_INFERENCE_MODELS else mask_decode_mode - ), + mask_decode_mode=("accurate" if USE_INFERENCE_MODELS else mask_decode_mode), tradeoff_factor=1.0 if USE_INFERENCE_MODELS else tradeoff_factor, response_mask_format="rle", disable_active_learning=disable_active_learning, @@ -414,9 +412,7 @@ def run_remotely( # backend they are honoured, so this version keeps passing them # through and behaves exactly as it does today. @v5 forwards # unconditionally. - mask_decode_mode=( - "accurate" if USE_INFERENCE_MODELS else mask_decode_mode - ), + mask_decode_mode=("accurate" if USE_INFERENCE_MODELS else mask_decode_mode), tradeoff_factor=1.0 if USE_INFERENCE_MODELS else tradeoff_factor, response_mask_format="rle", max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, From 4e11c83688ab1046193bff5df8b4414e878c5e03 Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Fri, 2 Oct 2026 20:31:42 +0200 Subject: [PATCH 19/41] Restore changelog entries displaced by the 1.7.3 release merge The release cut inference_models 0.39.1 and roboflow-workflows 0.2.4. Merging it dropped this branch's inference_models entries and moved its workflows entry into 0.2.4, attributing it to a release that does not contain it. Both merged without conflict, so neither was flagged. --- inference_models/docs/changelog.md | 36 ++++++++++++++++++++++++++++++ workflows/CHANGELOG.md | 32 +++++++++++++------------- 2 files changed, 53 insertions(+), 15 deletions(-) diff --git a/inference_models/docs/changelog.md b/inference_models/docs/changelog.md index 3e50b0d62d..69485a9ab4 100644 --- a/inference_models/docs/changelog.md +++ b/inference_models/docs/changelog.md @@ -2,6 +2,42 @@ ## Unreleased +### Added + +- `masks_resolution_factor` on instance-segmentation post-processing, in + `[0.0, 1.0]`. `1.0` (the default) resizes masks to the image as before; + `0.0` leaves them on the model's own grid; values between interpolate the + resize target. Threaded through all six instance-segmentation families on + both the dense and RLE paths, and through the ONNX, TorchScript and TensorRT + backends. Default output is unchanged. +- `mask_size` on `InstanceDetections` and `InstancesRLEMasks`, recording the + grid the masks live on. Each defaults to the value it effectively had before, + so existing construction is unaffected. +- `scale_polygons_to_image`, lifting contour coordinates from mask space into + image space. + +### Changed + +- `InstancesRLEMasks.to_coco_rle_masks()` now declares the grid the counts were + encoded on rather than the image size. These agree unless a resolution factor + below `1.0` is used; when they differ, declaring the image size made + `pycocotools` reinterpret the runs without raising. +- `InstanceDetections.to_supervision()` restores the image grid when masks are + reduced, since `sv.Detections.mask` is documented as matching the image and + its annotators index the scene with it. +- `InstanceDetections.__iter__` emits the encoded grid in the COCO `size` field. +- An unknown `mask_decode_mode` now raises `InvalidMaskDecodeArgument` on the + `inference_models` path, matching the legacy path, which previously accepted + any value. + +### Fixed + +- Zero-detection results reported a mask shape the populated path would never + produce, because the empty branch resolved its target from the padded grid + while the populated path unpadded first. +- RF-DETR's Triton post-process dispatcher dropped the resolution factor, so + the fused path silently ignored it. + --- ## `0.39.1` diff --git a/workflows/CHANGELOG.md b/workflows/CHANGELOG.md index f3024fcb44..6faa13e1bc 100644 --- a/workflows/CHANGELOG.md +++ b/workflows/CHANGELOG.md @@ -16,23 +16,8 @@ for contributor and maintainer responsibilities. ## Unreleased -## `0.2.4` - -Bundled execution engine: `1.16.1`. - ### Added -- Embedding Model (`roboflow_core/embedding_model@v1`): extract feature vectors - before the final linear layer or logits before Softmax/Sigmoid from existing - single-label and multi-label ResNet, ViT and DINOv3 classifiers, including - pretrained ResNet aliases. List and tensor variants return embeddings compatible - with Cosine Similarity and metadata identifying their embedding space. -- Local tensor-mode Embedding Model execution retains materialized tensor images - and embedding tensors through the model-provider boundary. Remote responses and - final JSON outputs remain serialized vectors. -- Anthropic Claude block (`anthropic_claude@v5`): `claude-sonnet-5-5` model option. -- OpenAI block (`open_ai@v7`): `gpt-6.1-sol` model option (`low`-`max` reasoning effort; object detection and instance segmentation reuse the GPT-6 prompts). - - `roboflow_core/roboflow_instance_segmentation_model@v5`. Identical to `@v4` except that it forwards `mask_decode_mode` and `tradeoff_factor` to the server, which now honours them. Versions up to `@v4` pin the outbound request @@ -59,6 +44,23 @@ Bundled execution engine: `1.16.1`. therefore available in non-tensor mode only, pending propagation of `mask_size` through the tensor pipeline. +## `0.2.4` + +Bundled execution engine: `1.16.1`. + +### Added + +- Embedding Model (`roboflow_core/embedding_model@v1`): extract feature vectors + before the final linear layer or logits before Softmax/Sigmoid from existing + single-label and multi-label ResNet, ViT and DINOv3 classifiers, including + pretrained ResNet aliases. List and tensor variants return embeddings compatible + with Cosine Similarity and metadata identifying their embedding space. +- Local tensor-mode Embedding Model execution retains materialized tensor images + and embedding tensors through the model-provider boundary. Remote responses and + final JSON outputs remain serialized vectors. +- Anthropic Claude block (`anthropic_claude@v5`): `claude-sonnet-5-5` model option. +- OpenAI block (`open_ai@v7`): `gpt-6.1-sol` model option (`low`-`max` reasoning effort; object detection and instance segmentation reuse the GPT-6 prompts). + ### Changed - Carries forward the `0.2.2` model catalog: Anthropic Claude v5 lists `claude-opus-5-5` (Claude Opus 5.5, 128000 max output tokens) and the temperature warning names Opus 5.x; OpenAI v7 lists `gpt-6-sol` and `gpt-6-luna` (reasoning effort `none` through `max`, structured absolute detection prompts). From 265f99166c9ab3bb26ba5b987310ab6f15b94191 Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Mon, 5 Oct 2026 16:14:11 +0200 Subject: [PATCH 20/41] Fix configuration coverage and checkout wheels in CI --- .../workflows/integration_e2e_tests_inference_sdk_x86.yml | 6 ++++++ .../core/interfaces/test_workflows_configuration.py | 5 ++++- workflows/tests/unit_tests/test_configuration.py | 8 ++++++-- 3 files changed, 16 insertions(+), 3 deletions(-) diff --git a/.github/workflows/integration_e2e_tests_inference_sdk_x86.yml b/.github/workflows/integration_e2e_tests_inference_sdk_x86.yml index 5044db2b0d..829718bc57 100644 --- a/.github/workflows/integration_e2e_tests_inference_sdk_x86.yml +++ b/.github/workflows/integration_e2e_tests_inference_sdk_x86.yml @@ -109,6 +109,12 @@ jobs: "$(ls "${{ github.workspace }}"/dist/streamvision-*.whl)[webrtc]" \ "${{ github.workspace }}"/dist/inference_sdk-*.whl \ "${{ github.workspace }}"/dist/inference_models-*.whl + # pip skips wheels matching installed release versions. Test this checkout. + pip install --force-reinstall --no-deps \ + "${{ github.workspace }}"/dist/roboflow_workflows-*.whl \ + "${{ github.workspace }}"/dist/streamvision-*.whl \ + "${{ github.workspace }}"/dist/inference_sdk-*.whl \ + "${{ github.workspace }}"/dist/inference_models-*.whl - name: 🧪 E2E Tests - WebRTC SDK (real server) timeout-minutes: 15 diff --git a/tests/inference/unit_tests/core/interfaces/test_workflows_configuration.py b/tests/inference/unit_tests/core/interfaces/test_workflows_configuration.py index 7faa08a304..f1bb56d9d9 100644 --- a/tests/inference/unit_tests/core/interfaces/test_workflows_configuration.py +++ b/tests/inference/unit_tests/core/interfaces/test_workflows_configuration.py @@ -167,6 +167,7 @@ ("ALLOW_WORKFLOWS_FONTS_DOWNLOAD", lambda c: c.fonts.allow_download), ("MODEL_CACHE_DIR", lambda c: c.fonts.model_cache_dir), ("LMM_ENABLED", lambda c: c.models.lmm_enabled), + ("USE_INFERENCE_MODELS", lambda c: c.models.use_inference_models), ( "WORKFLOWS_VLM_SEGMENTATION_MAX_POLYGON_VERTICES", lambda c: c.models.vlm_segmentation_max_polygon_vertices, @@ -238,7 +239,7 @@ def test_the_field_table_matches_the_facade_exports() -> None: "missing_from_table": sorted(exported - tabled), "missing_from_facade": sorted(tabled - exported), } - assert len(tabled) == 80, len(tabled) + assert len(tabled) == 81, len(tabled) def test_every_name_workflows_imports_from_the_facade_is_exported() -> None: @@ -283,6 +284,8 @@ def test_server_configuration_equals_env_field_by_field(name, reader) -> None: @pytest.mark.parametrize( "name, value", [ + ("USE_INFERENCE_MODELS", True), + ("USE_INFERENCE_MODELS", False), ("WORKFLOWS_VLM_SEGMENTATION_MAX_POLYGON_VERTICES", 17), ("WORKFLOWS_INNER_WORKFLOW_REMOTE_TARGET", "https://deployment.example/v1"), ("WORKFLOWS_INNER_WORKFLOW_REMOTE_DISPATCH_REQUEST_TIMEOUT", 12.5), diff --git a/workflows/tests/unit_tests/test_configuration.py b/workflows/tests/unit_tests/test_configuration.py index 10c07059c4..3752928762 100644 --- a/workflows/tests/unit_tests/test_configuration.py +++ b/workflows/tests/unit_tests/test_configuration.py @@ -67,7 +67,7 @@ def test_configuration_is_frozen_in_every_group() -> None: total_fields += len(dataclasses.fields(value)) with pytest.raises(dataclasses.FrozenInstanceError): setattr(value, dataclasses.fields(value)[0].name, "mutated") - assert total_fields == 80, total_fields + assert total_fields == 81, total_fields def test_default_configuration_matches_env_pys_empty_environment_defaults() -> None: @@ -132,6 +132,7 @@ def test_default_configuration_matches_env_pys_empty_environment_defaults() -> N assert configuration.fonts.allow_download is True assert configuration.fonts.model_cache_dir == "/tmp/cache" assert configuration.models.lmm_enabled is False + assert configuration.models.use_inference_models is True assert configuration.models.clip_version_id == "ViT-B-16" assert configuration.models.sam3_exec_mode == "local" assert configuration.models.sam3_3d_objects_enabled is False @@ -460,7 +461,7 @@ def test_environment_facade_exports_every_owned_symbol() -> None: for name in vars(workflows_environment) if name.isupper() and not name.startswith("_") } - assert len(exported) == 80, sorted(exported) + assert len(exported) == 81, sorted(exported) assert isinstance(workflows_environment.WORKFLOW_DISABLED_BLOCK_TYPES, list) assert isinstance(workflows_environment.WORKFLOW_DISABLED_BLOCK_PATTERNS, list) assert isinstance(workflows_environment.ENABLE_TENSOR_DATA_REPRESENTATION, bool) @@ -482,6 +483,9 @@ def test_the_facade_was_bound_from_the_installed_configuration() -> None: assert workflows_environment.ENABLE_TENSOR_DATA_REPRESENTATION is ( _INSTALLED_AT_COLLECTION.tensor.representation_enabled ) + assert workflows_environment.USE_INFERENCE_MODELS is ( + _INSTALLED_AT_COLLECTION.models.use_inference_models + ) # -------------------------------------------------------------------------- From ca448e0d521db4c974a054370814b4e9997af585 Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Tue, 6 Oct 2026 07:44:30 +0200 Subject: [PATCH 21/41] Fix mask coordinate frames and workflow compatibility --- inference/core/entities/requests/inference.py | 14 ++- .../core/models/inference_models_adapters.py | 10 +- .../core/models/instance_segmentation_base.py | 2 +- inference_models/docs/changelog.md | 16 ++- .../models/base/instance_segmentation.py | 35 +++++- .../models/common/roboflow/post_processing.py | 53 ++++++--- .../inference_models/models/rfdetr/common.py | 5 + .../rfdetr_instance_segmentation_coreml.py | 3 + .../inference_models/models/yolact/common.py | 7 ++ .../inference_models/models/yolo26/common.py | 7 ++ .../inference_models/models/yolov5/common.py | 7 ++ .../inference_models/models/yolov7/common.py | 7 ++ .../inference_models/models/yolov8/common.py | 7 ++ .../models/base/test_instance_segmentation.py | 51 +++++++++ .../roboflow/test_mask_image_dimensions.py | 106 +++++++++++++++++ .../common/roboflow/test_post_processing.py | 46 ++++++++ .../models/test_inference_models_adapters.py | 81 +++++++++++++ .../test_v5_mask_resolution.py | 107 +++++++++++++++++- workflows/CHANGELOG.md | 27 +++-- .../roboflow/instance_segmentation/v1.py | 9 +- .../roboflow/instance_segmentation/v2.py | 9 +- .../roboflow/instance_segmentation/v3.py | 9 +- .../roboflow/instance_segmentation/v4.py | 10 +- .../roboflow/instance_segmentation/v5.py | 27 ++++- .../instance_segmentation/v5_tensor.py | 4 +- 25 files changed, 585 insertions(+), 74 deletions(-) create mode 100644 inference_models/tests/unit_tests/models/common/roboflow/test_mask_image_dimensions.py diff --git a/inference/core/entities/requests/inference.py b/inference/core/entities/requests/inference.py index f195e3adf0..6160f46a5c 100644 --- a/inference/core/entities/requests/inference.py +++ b/inference/core/entities/requests/inference.py @@ -254,12 +254,18 @@ class InstanceSegmentationInferenceRequest(ObjectDetectionInferenceRequest): mask_decode_mode: Optional[str] = Field( default="accurate", examples=["accurate"], - description="The mode used to decode instance segmentation masks, one of 'accurate', 'fast', 'tradeoff'", + description="Mask decoding mode: 'accurate' (default), 'fast', or 'tradeoff'. " + "On inference_models, accurate uses full-resolution masks, fast keeps the " + "model mask grid, and tradeoff interpolates between those grids. Lower " + "resolution factors do not guarantee smaller masks or lower end-to-end latency.", ) tradeoff_factor: Optional[float] = Field( default=0.0, examples=[0.5], - description="The amount to tradeoff between 0='fast' and 1='accurate'", + description="Factor in [0.0, 1.0], used only with mask_decode_mode='tradeoff'; " + "defaults to 0.0. On inference_models, 0.0 selects the model mask grid and " + "1.0 selects image resolution. Intermediate values interpolate mask-grid " + "dimensions; the legacy backend uses a different interpolation schedule.", ) response_mask_format: Literal["polygon", "rle"] = Field( default="polygon", @@ -267,7 +273,9 @@ class InstanceSegmentationInferenceRequest(ObjectDetectionInferenceRequest): description="Requested output mask format - `polygon` is the default Roboflow format, which however is " "not capable representing certain shapes - RLE is compact and more standard representation, yet " "require special decoding on the caller side - currently supported in `opt-in` mode when server is " - "running with `USE_INFERENCE_MODELS=True` - otherwise it's ignored.", + "running with `USE_INFERENCE_MODELS=True` - otherwise it's ignored. " + "RLE size records the encoded mask grid, which may differ from the image " + "dimensions; polygon coordinates are returned in image space.", ) enforce_dense_masks_in_inference_models: Optional[bool] = Field( default=False, diff --git a/inference/core/models/inference_models_adapters.py b/inference/core/models/inference_models_adapters.py index c4c3422a5d..fa4bc8c315 100644 --- a/inference/core/models/inference_models_adapters.py +++ b/inference/core/models/inference_models_adapters.py @@ -134,6 +134,7 @@ from inference_models.models.base.types import InstancesRLEMasks, PreprocessingMetadata from inference_models.models.common.rle_utils import torch_mask_to_coco_rle from inference_models.models.common.roboflow.post_processing import ( + resolve_mask_frame_size, scale_polygons_to_image, ) @@ -1057,13 +1058,10 @@ def _build_responses_from_detections( # reported; equal sizes short-circuit to a no-op. This also breaks # any view into the pinned scratch buffers noted above, since the # scaled polygons are freshly allocated. - # Scale from the mask grid to the frame the masks were produced - # against, which is size_after_pre_processing - NOT original_size. - # A static crop anchored at (0, 0) also yields a mask smaller than - # the image, but its coordinates are already image-space, so - # inferring the scale from `mask_size != (H, W)` would double them. + # Offset crops are already on an original-image canvas; origin crops + # still cover only the cropped region. mask_size = getattr(det, "mask_size", None) - produced_against = preproc_metadata.size_after_pre_processing + produced_against = resolve_mask_frame_size(preproc_metadata) if ( not return_in_rle and mask_size is not None diff --git a/inference/core/models/instance_segmentation_base.py b/inference/core/models/instance_segmentation_base.py index b393756c4a..248d515e03 100644 --- a/inference/core/models/instance_segmentation_base.py +++ b/inference/core/models/instance_segmentation_base.py @@ -75,7 +75,7 @@ def infer( max_candidates (int, optional): Maximum number of candidate detections. Defaults to 3000. max_detections (int, optional): Maximum number of detections after non-maximum suppression. Defaults to 300. return_image_dims (bool, optional): Whether to return the dimensions of the processed images. Defaults to False. - tradeoff_factor (float, optional): Tradeoff factor used when `mask_decode_mode` is set to "tradeoff". Must be in [0.0, 1.0]. Defaults to 0.5. + tradeoff_factor (float, optional): Tradeoff factor used when `mask_decode_mode` is set to "tradeoff". Must be in [0.0, 1.0]. Defaults to 0.0. disable_preproc_auto_orient (bool, optional): If true, the auto orient preprocessing step is disabled for this call. Default is False. disable_preproc_contrast (bool, optional): If true, the auto contrast preprocessing step is disabled for this call. Default is False. disable_preproc_grayscale (bool, optional): If true, the grayscale preprocessing step is disabled for this call. Default is False. diff --git a/inference_models/docs/changelog.md b/inference_models/docs/changelog.md index 69485a9ab4..1f59cd9f32 100644 --- a/inference_models/docs/changelog.md +++ b/inference_models/docs/changelog.md @@ -9,12 +9,15 @@ `0.0` leaves them on the model's own grid; values between interpolate the resize target. Threaded through all six instance-segmentation families on both the dense and RLE paths, and through the ONNX, TorchScript and TensorRT - backends. Default output is unchanged. + backends, plus RF-DETR CoreML. Default output is unchanged. - `mask_size` on `InstanceDetections` and `InstancesRLEMasks`, recording the grid the masks live on. Each defaults to the value it effectively had before, so existing construction is unaffected. - `scale_polygons_to_image`, lifting contour coordinates from mask space into image space. +- `InstanceDetections.image_size` and `mask_frame_size` preserve original image + dimensions and the image region represented by a mask, independently of its + encoded grid. Dense and RLE model results carry these through post-processing. ### Changed @@ -32,9 +35,18 @@ ### Fixed +- Dense reduced masks now resize correctly in `to_supervision()`, including mask + annotation on the original image. Origin-anchored crop masks are padded to the + original image after resizing to the crop dimensions. +- Polygon responses for static crops with non-zero offsets use the original + image canvas as their coordinate frame, at full and reduced mask resolution. +- RF-DETR CoreML now forwards `masks_resolution_factor` to post-processing. +- Mask resize-target calculation rejects non-finite and out-of-range resolution + factors instead of silently clamping them. - Zero-detection results reported a mask shape the populated path would never produce, because the empty branch resolved its target from the padded grid - while the populated path unpadded first. + while the populated path unpadded first. Empty crop results also follow the + populated path's canvas placement and rounding rules. - RF-DETR's Triton post-process dispatcher dropped the resolution factor, so the fused path silently ignored it. diff --git a/inference_models/inference_models/models/base/instance_segmentation.py b/inference_models/inference_models/models/base/instance_segmentation.py index 197a7bc861..dbcc7a1a4e 100644 --- a/inference_models/inference_models/models/base/instance_segmentation.py +++ b/inference_models/inference_models/models/base/instance_segmentation.py @@ -135,6 +135,15 @@ def result(self) -> List["InstanceDetections"]: @dataclass class InstanceDetections: + """Instance masks and boxes with independent image and mask dimensions. + + Attributes: + image_size: Original image ``(height, width)``. Older callers may omit it. + mask_size: Encoded mask grid ``(height, width)``. + mask_frame_size: Image-space extent represented by the mask grid. A crop + anchored at the origin may cover only the top-left of the image. + """ + xyxy: torch.Tensor # (n_boxes, 4) class_id: torch.Tensor # (n_boxes, ) confidence: torch.Tensor # (n_boxes, ) @@ -148,6 +157,8 @@ class InstanceDetections: # (h, w) of the grid `mask` lives on. Resolved from the carrier when not # given, which reproduces the behaviour from before it was adjustable. mask_size: Optional[Tuple[int, int]] = None + image_size: Optional[Tuple[int, int]] = None + mask_frame_size: Optional[Tuple[int, int]] = None def __post_init__(self) -> None: if self.mask_size is not None: @@ -161,12 +172,14 @@ def _image_size(self) -> Tuple[int, int]: """Resolve the image grid these detections describe. An RLE carrier records the image size alongside the encoded grid. A - dense carrier does not, so its own shape is the best available answer - and is correct whenever no resolution factor was applied. + dense carrier uses explicit image dimensions when available. Its shape + remains the fallback for callers that omit image dimensions. Returns: Image ``(height, width)``. """ + if self.image_size is not None: + return tuple(self.image_size) if isinstance(self.mask, InstancesRLEMasks): return tuple(self.mask.image_size) if self.mask is not None and hasattr(self.mask, "shape"): @@ -271,28 +284,38 @@ def to_supervision(self) -> sv.Detections: # paint the wrong region, so restore the image grid on the way out. if mask is not None and self.mask_size is not None: image_height, image_width = self._image_size() - if tuple(self.mask_size) != (image_height, image_width): + frame_height, frame_width = self.mask_frame_size or ( + image_height, + image_width, + ) + if tuple(mask.shape[1:]) != (frame_height, frame_width): mask = ( np.stack( [ cv2.resize( single.astype(np.uint8), - (image_width, image_height), + (frame_width, frame_height), interpolation=cv2.INTER_NEAREST, ).astype(bool) for single in mask ] ) if len(mask) - else np.zeros((0, image_height, image_width), dtype=bool) + else np.zeros((0, frame_height, frame_width), dtype=bool) ) - return sv.Detections( + if (frame_height, frame_width) != (image_height, image_width): + canvas = np.zeros((len(mask), image_height, image_width), dtype=bool) + canvas[:, :frame_height, :frame_width] = mask + mask = canvas + detections = sv.Detections( xyxy=self.xyxy.cpu().numpy(), class_id=self.class_id.cpu().numpy(), confidence=self.confidence.cpu().numpy(), mask=mask, ) + return detections + class InstanceSegmentationModel( ABC, Generic[PreprocessedInputs, PreprocessingMetadata, RawPrediction] diff --git a/inference_models/inference_models/models/common/roboflow/post_processing.py b/inference_models/inference_models/models/common/roboflow/post_processing.py index 13fffebb55..2dc31a2852 100644 --- a/inference_models/inference_models/models/common/roboflow/post_processing.py +++ b/inference_models/inference_models/models/common/roboflow/post_processing.py @@ -438,6 +438,23 @@ def scale_polygons_to_image( return scaled_polygons +def resolve_mask_frame_size(metadata: PreProcessingMetadata) -> ImageDimensions: + """Resolve the image-space extent covered by aligned masks. + + Args: + metadata: Preprocessing transforms used to align the masks. + + Returns: + Original image dimensions for offset-crop canvases, or the crop + dimensions when alignment leaves masks anchored at the origin. + """ + offset = getattr(metadata, "static_crop_offset", None) + if offset is not None and (offset.offset_x > 0 or offset.offset_y > 0): + return metadata.original_size + + return metadata.size_after_pre_processing + + def resolve_mask_target_size( mask_height: int, mask_width: int, @@ -470,11 +487,17 @@ def resolve_mask_target_size( Returns: The `(height, width)` to resize masks to. + + Raises: + ValueError: If the resolution factor is outside the finite range [0, 1]. """ + if not 0.0 <= masks_resolution_factor <= 1.0: + raise ValueError("masks_resolution_factor must be finite and in [0.0, 1.0]") + if masks_resolution_factor >= 1.0: return size_after_pre_processing.height, size_after_pre_processing.width - factor = max(0.0, masks_resolution_factor) + factor = masks_resolution_factor height = max( 1, round(mask_height * (1 - factor) + size_after_pre_processing.height * factor) ) @@ -549,20 +572,22 @@ def align_instance_segmentation_results( size_after_pre_processing=size_after_pre_processing, masks_resolution_factor=masks_resolution_factor, ) + empty_height, empty_width = empty_target_height, empty_target_width + if static_crop_offset.offset_x > 0 or static_crop_offset.offset_y > 0: + height_scale = empty_target_height / size_after_pre_processing.height + width_scale = empty_target_width / size_after_pre_processing.width + empty_height = max( + 1, + round(original_size.height * height_scale), + round(static_crop_offset.offset_y * height_scale) + empty_target_height, + ) + empty_width = max( + 1, + round(original_size.width * width_scale), + round(static_crop_offset.offset_x * width_scale) + empty_target_width, + ) empty_masks = torch.empty( - size=( - 0, - round( - original_size.height - * empty_target_height - / size_after_pre_processing.height - ), - round( - original_size.width - * empty_target_width - / size_after_pre_processing.width - ), - ), + size=(0, empty_height, empty_width), dtype=torch.bool, device=image_bboxes.device, ) diff --git a/inference_models/inference_models/models/rfdetr/common.py b/inference_models/inference_models/models/rfdetr/common.py index 8f43356cbe..652e59dc4a 100644 --- a/inference_models/inference_models/models/rfdetr/common.py +++ b/inference_models/inference_models/models/rfdetr/common.py @@ -19,6 +19,7 @@ align_instance_segmentation_results, align_instance_segmentation_results_to_rle_masks, rescale_image_detections, + resolve_mask_frame_size, ) from inference_models.models.optimization.triton_jit import ( is_triton_jit_failure, @@ -239,6 +240,8 @@ def post_process_instance_segmentation_results( confidence=confidence, class_id=top_classes.int(), mask=aligned_masks, + image_size=tuple(image_meta.original_size), + mask_frame_size=tuple(resolve_mask_frame_size(image_meta)), ) results.append(detections) return results @@ -406,6 +409,8 @@ def _post_process_single_instance_segmentation_result_to_rle_masks( confidence=confidence, class_id=top_classes.int(), mask=instances_masks, + image_size=tuple(image_meta.original_size), + mask_frame_size=tuple(resolve_mask_frame_size(image_meta)), ) diff --git a/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_coreml.py b/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_coreml.py index b61bf726a7..7b8d5cefc5 100644 --- a/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_coreml.py +++ b/inference_models/inference_models/models/rfdetr/rfdetr_instance_segmentation_coreml.py @@ -226,6 +226,7 @@ def post_process( confidence: Confidence = "default", mask_format: InstanceSegmentationMaskFormat = "dense", max_detections: Optional[int] = INFERENCE_MODELS_RFDETR_DEFAULT_MAX_DETECTIONS, + masks_resolution_factor: float = 1.0, **kwargs, ) -> List[InstanceDetections]: """Convert raw predictions with the shared RF-DETR post-processing. @@ -235,6 +236,7 @@ def post_process( pre_processing_meta (list[PreProcessingMetadata]): Per-image transforms. confidence (Confidence): Detection threshold or model-default selector. mask_format (InstanceSegmentationMaskFormat): ``dense`` masks or ``rle`` encoded masks. + masks_resolution_factor (float): Mask-grid interpolation factor in [0, 1]. max_detections (int, optional): Cap on detections per image, applied before masks are aligned. **kwargs: Other shared inference options, ignored here. @@ -274,4 +276,5 @@ def post_process( num_classes=len(self.class_names), classes_re_mapping=self._classes_re_mapping, max_detections=max_detections, + masks_resolution_factor=masks_resolution_factor, ) diff --git a/inference_models/inference_models/models/yolact/common.py b/inference_models/inference_models/models/yolact/common.py index c46da769b5..1b2b1aacec 100644 --- a/inference_models/inference_models/models/yolact/common.py +++ b/inference_models/inference_models/models/yolact/common.py @@ -8,6 +8,7 @@ align_instance_segmentation_results, align_instance_segmentation_results_to_rle_masks, crop_masks_to_boxes, + resolve_mask_frame_size, ) @@ -50,6 +51,8 @@ def prepare_dense_masks( class_id=aligned_boxes[:, 5].int(), confidence=aligned_boxes[:, 4], mask=aligned_masks, + image_size=tuple(image_meta.original_size), + mask_frame_size=tuple(resolve_mask_frame_size(image_meta)), ) ) return final_results @@ -107,6 +110,8 @@ def prepare_rle_masks( class_id=aligned_boxes_tensor[:, 5].int(), confidence=aligned_boxes_tensor[:, 4], mask=instances_masks, + image_size=tuple(image_meta.original_size), + mask_frame_size=tuple(resolve_mask_frame_size(image_meta)), ) ) else: @@ -120,6 +125,8 @@ def prepare_rle_masks( ), confidence=torch.empty((0,), device=image_bboxes.device), mask=instances_masks, + image_size=tuple(image_meta.original_size), + mask_frame_size=tuple(resolve_mask_frame_size(image_meta)), ) ) return final_results diff --git a/inference_models/inference_models/models/yolo26/common.py b/inference_models/inference_models/models/yolo26/common.py index 1dd17a352f..74858123f6 100644 --- a/inference_models/inference_models/models/yolo26/common.py +++ b/inference_models/inference_models/models/yolo26/common.py @@ -9,6 +9,7 @@ align_instance_segmentation_results_to_rle_masks, crop_masks_to_boxes, preprocess_segmentation_masks, + resolve_mask_frame_size, ) @@ -51,6 +52,8 @@ def prepare_dense_masks( class_id=aligned_boxes[:, 5].int(), confidence=aligned_boxes[:, 4], mask=aligned_masks, + image_size=tuple(image_meta.original_size), + mask_frame_size=tuple(resolve_mask_frame_size(image_meta)), ) ) return final_results @@ -108,6 +111,8 @@ def prepare_rle_masks( class_id=aligned_boxes_tensor[:, 5].int(), confidence=aligned_boxes_tensor[:, 4], mask=instances_masks, + image_size=tuple(image_meta.original_size), + mask_frame_size=tuple(resolve_mask_frame_size(image_meta)), ) ) else: @@ -121,6 +126,8 @@ def prepare_rle_masks( ), confidence=torch.empty((0,), device=image_bboxes.device), mask=instances_masks, + image_size=tuple(image_meta.original_size), + mask_frame_size=tuple(resolve_mask_frame_size(image_meta)), ) ) return final_results diff --git a/inference_models/inference_models/models/yolov5/common.py b/inference_models/inference_models/models/yolov5/common.py index 057c1a31f4..c6e5efb7f7 100644 --- a/inference_models/inference_models/models/yolov5/common.py +++ b/inference_models/inference_models/models/yolov5/common.py @@ -9,6 +9,7 @@ align_instance_segmentation_results_to_rle_masks, crop_masks_to_boxes, preprocess_segmentation_masks, + resolve_mask_frame_size, ) @@ -51,6 +52,8 @@ def prepare_dense_masks( class_id=aligned_boxes[:, 5].int(), confidence=aligned_boxes[:, 4], mask=aligned_masks, + image_size=tuple(image_meta.original_size), + mask_frame_size=tuple(resolve_mask_frame_size(image_meta)), ) ) return final_results @@ -108,6 +111,8 @@ def prepare_rle_masks( class_id=aligned_boxes_tensor[:, 5].int(), confidence=aligned_boxes_tensor[:, 4], mask=instances_masks, + image_size=tuple(image_meta.original_size), + mask_frame_size=tuple(resolve_mask_frame_size(image_meta)), ) ) else: @@ -121,6 +126,8 @@ def prepare_rle_masks( ), confidence=torch.empty((0,), device=image_bboxes.device), mask=instances_masks, + image_size=tuple(image_meta.original_size), + mask_frame_size=tuple(resolve_mask_frame_size(image_meta)), ) ) return final_results diff --git a/inference_models/inference_models/models/yolov7/common.py b/inference_models/inference_models/models/yolov7/common.py index 057c1a31f4..c6e5efb7f7 100644 --- a/inference_models/inference_models/models/yolov7/common.py +++ b/inference_models/inference_models/models/yolov7/common.py @@ -9,6 +9,7 @@ align_instance_segmentation_results_to_rle_masks, crop_masks_to_boxes, preprocess_segmentation_masks, + resolve_mask_frame_size, ) @@ -51,6 +52,8 @@ def prepare_dense_masks( class_id=aligned_boxes[:, 5].int(), confidence=aligned_boxes[:, 4], mask=aligned_masks, + image_size=tuple(image_meta.original_size), + mask_frame_size=tuple(resolve_mask_frame_size(image_meta)), ) ) return final_results @@ -108,6 +111,8 @@ def prepare_rle_masks( class_id=aligned_boxes_tensor[:, 5].int(), confidence=aligned_boxes_tensor[:, 4], mask=instances_masks, + image_size=tuple(image_meta.original_size), + mask_frame_size=tuple(resolve_mask_frame_size(image_meta)), ) ) else: @@ -121,6 +126,8 @@ def prepare_rle_masks( ), confidence=torch.empty((0,), device=image_bboxes.device), mask=instances_masks, + image_size=tuple(image_meta.original_size), + mask_frame_size=tuple(resolve_mask_frame_size(image_meta)), ) ) return final_results diff --git a/inference_models/inference_models/models/yolov8/common.py b/inference_models/inference_models/models/yolov8/common.py index 6e39830090..7a3f1a64f9 100644 --- a/inference_models/inference_models/models/yolov8/common.py +++ b/inference_models/inference_models/models/yolov8/common.py @@ -9,6 +9,7 @@ align_instance_segmentation_results_to_rle_masks, crop_masks_to_boxes, preprocess_segmentation_masks, + resolve_mask_frame_size, ) @@ -56,6 +57,8 @@ def prepare_dense_masks( class_id=aligned_boxes[:, 5].int(), confidence=aligned_boxes[:, 4], mask=aligned_masks, + image_size=tuple(image_meta.original_size), + mask_frame_size=tuple(resolve_mask_frame_size(image_meta)), ) ) return final_results @@ -118,6 +121,8 @@ def prepare_rle_masks( class_id=aligned_boxes_tensor[:, 5].int(), confidence=aligned_boxes_tensor[:, 4], mask=instances_masks, + image_size=tuple(image_meta.original_size), + mask_frame_size=tuple(resolve_mask_frame_size(image_meta)), ) ) else: @@ -131,6 +136,8 @@ def prepare_rle_masks( ), confidence=torch.empty((0,), device=image_bboxes.device), mask=instances_masks, + image_size=tuple(image_meta.original_size), + mask_frame_size=tuple(resolve_mask_frame_size(image_meta)), ) ) return final_results diff --git a/inference_models/tests/unit_tests/models/base/test_instance_segmentation.py b/inference_models/tests/unit_tests/models/base/test_instance_segmentation.py index 7e809e0ab7..7998982ab2 100644 --- a/inference_models/tests/unit_tests/models/base/test_instance_segmentation.py +++ b/inference_models/tests/unit_tests/models/base/test_instance_segmentation.py @@ -246,3 +246,54 @@ def _square_rle(height: int, width: int) -> bytes: dense[height // 4 : height // 2, width // 4 : width // 2] = 1 return mask_utils.encode(np.asfortranarray(dense))["counts"] + + +class TestDenseImageDimensions: + def test_reduced_dense_masks_can_annotate_original_image(self) -> None: + import numpy as np + import supervision as sv + + mask = torch.zeros((1, 20, 30), dtype=torch.bool) + mask[:, 5:10, 8:14] = True + detections = InstanceDetections( + xyxy=torch.tensor([[80, 50, 140, 100]], dtype=torch.float32), + class_id=torch.tensor([0]), + confidence=torch.tensor([0.9]), + mask=mask, + image_size=(200, 300), + ).to_supervision() + + scene = np.zeros((200, 300, 3), dtype=np.uint8) + annotated = sv.MaskAnnotator().annotate(scene=scene, detections=detections) + + assert detections.mask.shape == (1, 200, 300) + assert detections.mask[0, 50:100, 80:140].all() + assert detections.mask.sum() == 50 * 60 + assert annotated.any() + + def test_origin_crop_is_resized_then_padded_not_stretched(self) -> None: + mask = torch.zeros((1, 20, 30), dtype=torch.bool) + mask[:, 2:10, 4:16] = True + detections = InstanceDetections( + xyxy=torch.tensor([[20, 10, 80, 50]], dtype=torch.float32), + class_id=torch.tensor([0]), + confidence=torch.tensor([0.9]), + mask=mask, + image_size=(200, 300), + mask_frame_size=(100, 150), + ).to_supervision() + + assert detections.mask.shape == (1, 200, 300) + assert detections.mask[0, 10:50, 20:80].all() + assert detections.mask.sum() == 40 * 60 + + def test_empty_reduced_dense_masks_keep_original_dimensions(self) -> None: + detections = InstanceDetections( + xyxy=torch.empty((0, 4)), + class_id=torch.empty((0,), dtype=torch.long), + confidence=torch.empty((0,)), + mask=torch.empty((0, 20, 30), dtype=torch.bool), + image_size=(200, 300), + ).to_supervision() + + assert detections.mask.shape == (0, 200, 300) diff --git a/inference_models/tests/unit_tests/models/common/roboflow/test_mask_image_dimensions.py b/inference_models/tests/unit_tests/models/common/roboflow/test_mask_image_dimensions.py new file mode 100644 index 0000000000..e571b25f9d --- /dev/null +++ b/inference_models/tests/unit_tests/models/common/roboflow/test_mask_image_dimensions.py @@ -0,0 +1,106 @@ +"""Model results retain the image frame independently of the encoded mask grid.""" + +from importlib import import_module + +import pytest +import torch + +from inference_models.entities import ImageDimensions +from inference_models.models.common.roboflow.model_packages import ( + PreProcessingMetadata, + StaticCropOffset, +) + + +@pytest.mark.parametrize( + "family", ["yolov5", "yolov7", "yolov8", "yolo26", "yolact", "rfdetr"] +) +@pytest.mark.parametrize("mask_format", ["dense", "rle"]) +@pytest.mark.parametrize("offset", [(0, 0), (60, 40)]) +def test_model_result_retains_image_dimensions(family, mask_format, offset) -> None: + module = import_module(f"inference_models.models.{family}.common") + metadata = PreProcessingMetadata( + pad_left=0, + pad_top=0, + pad_right=0, + pad_bottom=0, + original_size=ImageDimensions(height=200, width=300), + size_after_pre_processing=ImageDimensions(height=100, width=150), + inference_size=ImageDimensions(height=80, width=120), + scale_width=0.8, + scale_height=0.8, + static_crop_offset=StaticCropOffset( + offset_x=offset[0], offset_y=offset[1], crop_width=150, crop_height=100 + ), + ) + if family == "rfdetr": + post_process = ( + module.post_process_instance_segmentation_results + if mask_format == "dense" + else module.post_process_instance_segmentation_results_to_rle_masks + ) + results = post_process( + bboxes=torch.tensor([[[0.5, 0.5, 1.0, 1.0]]]), + logits=torch.tensor([[[10.0]]]), + masks=torch.ones((1, 1, 20, 30)), + pre_processing_meta=[metadata], + threshold=0.5, + num_classes=1, + classes_re_mapping=None, + masks_resolution_factor=0.0, + ) + else: + kwargs = dict( + nms_results=[torch.tensor([[0, 0, 120, 80, 0.9, 0, 1.0]])], + pre_processing_meta=[metadata], + masks_resolution_factor=0.0, + ) + if family == "yolact": + kwargs["all_proto_data"] = torch.ones((1, 20, 30, 1)) + else: + kwargs["protos"] = torch.ones((1, 1, 20, 30)) + if family == "yolov8": + kwargs.update( + masks_smoothing_enabled=False, masks_binarization_threshold=0.0 + ) + if family == "yolo26": + kwargs["filtered_results"] = kwargs.pop("nms_results") + results = getattr(module, f"prepare_{mask_format}_masks")(**kwargs) + + result = results[0] + assert result.image_size == (200, 300) + assert result.mask_frame_size == ((200, 300) if any(offset) else (100, 150)) + converted = result.to_supervision() + assert converted.mask.shape == (1, 200, 300) + assert converted.mask[ + 0, offset[1] : offset[1] + 100, offset[0] : offset[0] + 150 + ].all() + assert converted.mask.sum() == 100 * 150 + + +@pytest.mark.parametrize("mask_format", ["dense", "rle"]) +def test_coreml_dispatcher_forwards_resolution_factor(monkeypatch, mask_format) -> None: + from unittest.mock import MagicMock + + from inference_models.models.rfdetr import ( + rfdetr_instance_segmentation_coreml as coreml, + ) + + model = object.__new__(coreml.RFDetrForInstanceSegmentationCoreML) + model._class_names = ["car"] + model._classes_re_mapping = None + model.recommended_parameters = None + post_process = MagicMock(return_value=[]) + target = "post_process_instance_segmentation_results" + if mask_format == "rle": + target += "_to_rle_masks" + monkeypatch.setattr(coreml, target, post_process) + + model.post_process( + model_results=(torch.empty(0), torch.empty(0), torch.empty(0)), + pre_processing_meta=[], + mask_format=mask_format, + masks_resolution_factor=0.25, + ) + + assert post_process.call_args.kwargs["masks_resolution_factor"] == 0.25 diff --git a/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py b/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py index 04c087574d..43f6bb9ff7 100644 --- a/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py +++ b/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py @@ -1433,3 +1433,49 @@ def test_shapes_agree(self, padding, factor: float) -> None: # then assert empty.shape[1:] == populated.shape[1:] + + +@pytest.mark.parametrize("offset", [0, 83]) +def test_empty_masks_match_populated_crop_canvas_after_rounding(offset: int) -> None: + kwargs = dict( + padding=(0, 0, 0, 0), + scale_width=1.0, + scale_height=1.0, + original_size=ImageDimensions(height=245, width=245), + size_after_pre_processing=ImageDimensions(height=162, width=162), + inference_size=ImageDimensions(height=162, width=162), + static_crop_offset=StaticCropOffset( + offset_x=offset, offset_y=offset, crop_width=162, crop_height=162 + ), + masks_resolution_factor=0.0, + ) + _, populated = align_instance_segmentation_results( + image_bboxes=torch.tensor([[0, 0, 162, 162]], dtype=torch.float32), + masks=torch.ones((1, 81, 81)), + **kwargs, + ) + _, empty = align_instance_segmentation_results( + image_bboxes=torch.empty((0, 4)), + masks=torch.empty((0, 81, 81)), + **kwargs, + ) + + assert empty.shape[1:] == populated.shape[1:] + assert empty.shape[1:] == ((123, 123) if offset else (81, 81)) + + +@pytest.mark.parametrize( + "factor", [-0.1, 1.1, float("nan"), float("inf"), -float("inf")] +) +def test_mask_resolution_target_rejects_invalid_factor(factor: float) -> None: + from inference_models.models.common.roboflow.post_processing import ( + resolve_mask_target_size, + ) + + with pytest.raises(ValueError, match="masks_resolution_factor"): + resolve_mask_target_size( + mask_height=20, + mask_width=30, + size_after_pre_processing=ImageDimensions(height=200, width=300), + masks_resolution_factor=factor, + ) diff --git a/tests/inference/unit_tests/core/models/test_inference_models_adapters.py b/tests/inference/unit_tests/core/models/test_inference_models_adapters.py index 327b971353..1a22c4ac76 100644 --- a/tests/inference/unit_tests/core/models/test_inference_models_adapters.py +++ b/tests/inference/unit_tests/core/models/test_inference_models_adapters.py @@ -1130,3 +1130,84 @@ def test_origin_anchored_crop_is_not_mistaken_for_a_reduced_grid() -> None: # the mask already matches size_after_pre_processing, so no scaling xs = [p.x for p in responses[0].predictions[0].points] assert max(xs) <= 8, f"coordinates were scaled a second time: {xs}" + + +@pytest.mark.parametrize("mask_format", ["dense", "rle"]) +@pytest.mark.parametrize("factor", [1.0, 0.0]) +@pytest.mark.parametrize("offset", [(0, 0), (60, 40)]) +def test_crop_polygon_response_uses_the_frame_represented_by_mask( + mask_format: str, factor: float, offset: tuple +) -> None: + from inference_models.entities import ImageDimensions + from inference_models.models.common.roboflow.model_packages import ( + PreProcessingMetadata, + StaticCropOffset, + ) + from inference_models.models.common.roboflow.post_processing import ( + align_instance_segmentation_results, + align_instance_segmentation_results_to_rle_masks, + ) + + offset_x, offset_y = offset + original_size = ImageDimensions(height=200, width=300) + crop_size = ImageDimensions(height=100, width=150) + crop_offset = StaticCropOffset( + offset_x=offset_x, offset_y=offset_y, crop_width=150, crop_height=100 + ) + metadata = PreProcessingMetadata( + pad_left=0, + pad_top=0, + pad_right=0, + pad_bottom=0, + original_size=original_size, + size_after_pre_processing=crop_size, + inference_size=crop_size, + scale_width=1.0, + scale_height=1.0, + static_crop_offset=crop_offset, + ) + masks = torch.full((1, 20, 30), -1.0) + masks[:, 2:10, 4:16] = 1.0 + kwargs = dict( + image_bboxes=torch.tensor([[20, 10, 80, 50]], dtype=torch.float32), + masks=masks, + padding=(0, 0, 0, 0), + scale_width=1.0, + scale_height=1.0, + original_size=original_size, + size_after_pre_processing=crop_size, + inference_size=crop_size, + static_crop_offset=crop_offset, + masks_resolution_factor=factor, + ) + if mask_format == "dense": + boxes, aligned_masks = align_instance_segmentation_results(**kwargs) + else: + box, rle = next(align_instance_segmentation_results_to_rle_masks(**kwargs)) + boxes = box.unsqueeze(0) + aligned_masks = InstancesRLEMasks( + image_size=original_size, + masks=[rle["counts"]], + mask_size=tuple(rle["size"]), + ) + detections = InstanceDetections( + xyxy=boxes, + confidence=torch.tensor([0.9]), + class_id=torch.tensor([0]), + mask=aligned_masks, + ) + + response = _seg_adapter()._build_responses_from_detections( + [detections], [metadata] + )[0] + + prediction = response.predictions[0] + xs = [point.x for point in prediction.points] + ys = [point.y for point in prediction.points] + assert min(xs) == pytest.approx(20 + offset_x) + assert min(ys) == pytest.approx(10 + offset_y) + pixel_size = 1 if factor == 1.0 else 5 + assert max(xs) == pytest.approx(80 + offset_x - pixel_size) + assert max(ys) == pytest.approx(50 + offset_y - pixel_size) + assert prediction.x == pytest.approx(50 + offset_x) + assert prediction.y == pytest.approx(30 + offset_y) diff --git a/tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_v5_mask_resolution.py b/tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_v5_mask_resolution.py index a674b8fe15..c87c013789 100644 --- a/tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_v5_mask_resolution.py +++ b/tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_v5_mask_resolution.py @@ -103,7 +103,7 @@ def test_manifest_parity_with_tensor_sibling(version: str) -> None: @pytest.mark.parametrize("version", PINNED_VERSIONS) -def test_legacy_versions_pin_only_on_the_inference_models_backend( +def test_legacy_versions_pin_local_conditionally_and_remote_unconditionally( version: str, ) -> None: # given @@ -112,9 +112,9 @@ def test_legacy_versions_pin_only_on_the_inference_models_backend( source = _source(version, tensor=False) # then - assert source.count('"accurate" if USE_INFERENCE_MODELS else mask_decode_mode') == 2 - assert source.count("1.0 if USE_INFERENCE_MODELS else tradeoff_factor") == 2 - assert 'mask_decode_mode="accurate",' not in source + assert source.count('"accurate" if USE_INFERENCE_MODELS else mask_decode_mode') == 1 + assert source.count("1.0 if USE_INFERENCE_MODELS else tradeoff_factor") == 1 + assert source.count('mask_decode_mode="accurate",') == 1 @pytest.mark.parametrize("version", PINNED_VERSIONS + FORWARDING_VERSIONS) @@ -182,3 +182,102 @@ def test_v5_non_tensor_has_no_pinned_call_sites(self) -> None: # then assert 'mask_decode_mode="accurate",' not in source assert source.count("mask_decode_mode=mask_decode_mode,") == 4 + + +@pytest.mark.parametrize("version", PINNED_VERSIONS) +@pytest.mark.parametrize("local_backend", [True, False]) +def test_remote_request_is_independent_of_local_backend( + monkeypatch, version, local_backend +): + from importlib import import_module + from unittest.mock import MagicMock + + module = import_module( + f"roboflow_workflows.core_steps.models.roboflow.instance_segmentation.{version}" + ) + block_class = getattr( + module, f"RoboflowInstanceSegmentationModelBlock{version.upper()}" + ) + block = object.__new__(block_class) + block._api_key = None + block._post_process_result = MagicMock(return_value=[]) + client = MagicMock() + monkeypatch.setattr(module, "InferenceHTTPClient", MagicMock(return_value=client)) + monkeypatch.setattr(module, "USE_INFERENCE_MODELS", local_backend) + + block.run_remotely( + images=[], + model_id="model/1", + class_agnostic_nms=False, + class_filter=None, + confidence=0.5, + iou_threshold=0.5, + max_detections=10, + max_candidates=20, + mask_decode_mode="fast", + tradeoff_factor=0.0, + disable_active_learning=True, + active_learning_target_dataset=None, + ) + + config = client.configure.call_args.kwargs["inference_configuration"] + assert config.mask_decode_mode == "accurate" + assert config.tradeoff_factor == 1.0 + + +def test_v5_reduced_rle_output_can_be_decoded_on_original_image(monkeypatch): + import numpy as np + import supervision as sv + from pycocotools import mask as mask_utils + from roboflow_workflows.core_steps.models.roboflow.instance_segmentation import v5 + from roboflow_workflows.core_steps.visualizations.common.utils import ( + ensure_dense_masks, + ) + from roboflow_workflows.execution_engine.constants import ( + RLE_MASK_KEY_IN_SV_DETECTIONS, + ) + + block = object.__new__(v5.RoboflowInstanceSegmentationModelBlockV5) + mask = np.zeros((20, 30), dtype=np.uint8) + mask[5:10, 8:14] = 1 + rle = mask_utils.encode(np.asfortranarray(mask)) + rle["counts"] = rle["counts"].decode("utf-8") + prediction = { + "image": {"height": 200, "width": 300}, + "predictions": [ + { + "x": 110, + "y": 75, + "width": 60, + "height": 50, + "confidence": 0.9, + "class_id": 0, + "class": "car", + "rle": rle, + } + ], + } + monkeypatch.setattr( + v5, + "attach_parents_coordinates_to_batch_of_sv_detections", + lambda images, predictions: predictions, + ) + + output = block._post_process_result( + images=[], predictions=[prediction], class_filter=None, model_id="model/1" + ) + + detections = output[0]["predictions"] + encoded = detections.data[RLE_MASK_KEY_IN_SV_DETECTIONS][0] + decoded = mask_utils.decode(encoded).astype(bool) + assert decoded.shape == (200, 300) + np.testing.assert_array_equal(decoded, detections.mask[0]) + assert decoded[50:100, 80:140].all() + assert decoded.sum() == 50 * 60 + detections.mask = None + detections = ensure_dense_masks(detections) + annotated = sv.MaskAnnotator().annotate( + scene=np.zeros((200, 300, 3), dtype=np.uint8), detections=detections + ) + assert annotated.any() + assert rle["size"] == [20, 30] diff --git a/workflows/CHANGELOG.md b/workflows/CHANGELOG.md index 6faa13e1bc..1c9d17e418 100644 --- a/workflows/CHANGELOG.md +++ b/workflows/CHANGELOG.md @@ -21,17 +21,16 @@ for contributor and maintainer responsibilities. - `roboflow_core/roboflow_instance_segmentation_model@v5`. Identical to `@v4` except that it forwards `mask_decode_mode` and `tradeoff_factor` to the server, which now honours them. Versions up to `@v4` pin the outbound request - to `accurate` / `1.0`, so existing workflows are unaffected on the - `inference_models` backend. - - The pin is conditional on `USE_INFERENCE_MODELS`. The legacy backend - honoured these fields, so `@v1`-`@v4` keep passing them through there and - behave exactly as they do today; the pin applies only where the fields were - being discarded. No existing workflow changes behaviour on either backend. + to `accurate` / `1.0` on the local `inference_models` backend. Local legacy + execution continues to honour these fields when `USE_INFERENCE_MODELS=False`. + Remote `@v1`–`@v4` requests always use `accurate` / `1.0`, because the local + backend setting does not identify the remote server. This changes remote + legacy workflows that previously requested reduced masks; use non-tensor + `@v5` to opt into adjustable mask resolution. - `ModelsConfiguration.use_inference_models`, defaulting to `True`. Hosts that serve instance segmentation through the legacy backend must set it to `False` - so block versions up to `@v4` keep forwarding `mask_decode_mode` and + so local block versions up to `@v4` keep forwarding `mask_decode_mode` and `tradeoff_factor`, which that backend honours. The server sets it from `inference.core.env.USE_INFERENCE_MODELS`; the value cannot be read from the environment here, since this package must not import `inference.*`. @@ -41,8 +40,16 @@ for contributor and maintainer responsibilities. Nothing under `roboflow_workflows/` reads `InstanceDetections.mask_size`, and the sites that rebuild `InstancesRLEMasks` drop it, so a reduced grid would be reinterpreted as image-sized downstream. Reduced mask resolution is - therefore available in non-tensor mode only, pending propagation of - `mask_size` through the tensor pipeline. + therefore available in non-tensor mode only. **Full tensor support will come + in a separate PR**, including propagation of mask-grid metadata through the + tensor pipeline. Until then, tensor mode ignores `mask_decode_mode` and + `tradeoff_factor` and always requests `accurate` / `1.0`. + +### Fixed + +- Non-tensor instance segmentation `@v5` re-encodes reduced RLE masks at the + image grid after conversion to Supervision, so downstream blocks receive + consistent dense and RLE representations. ## `0.2.4` diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1.py index 576a92fdbb..00f9db87ed 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1.py @@ -372,12 +372,9 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned only where the fields were discarded. On the legacy - # backend they are honoured, so this version keeps passing them - # through and behaves exactly as it does today. @v5 forwards - # unconditionally. - mask_decode_mode=("accurate" if USE_INFERENCE_MODELS else mask_decode_mode), - tradeoff_factor=1.0 if USE_INFERENCE_MODELS else tradeoff_factor, + # The local backend flag does not identify the remote server. + mask_decode_mode="accurate", + tradeoff_factor=1.0, max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, source="workflow-execution", diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2.py index 88d7907740..48d485d041 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2.py @@ -371,12 +371,9 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned only where the fields were discarded. On the legacy - # backend they are honoured, so this version keeps passing them - # through and behaves exactly as it does today. @v5 forwards - # unconditionally. - mask_decode_mode=("accurate" if USE_INFERENCE_MODELS else mask_decode_mode), - tradeoff_factor=1.0 if USE_INFERENCE_MODELS else tradeoff_factor, + # The local backend flag does not identify the remote server. + mask_decode_mode="accurate", + tradeoff_factor=1.0, max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, source="workflow-execution", diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3.py index 2508061e91..e2730ae129 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3.py @@ -708,12 +708,9 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned only where the fields were discarded. On the legacy - # backend they are honoured, so this version keeps passing them - # through and behaves exactly as it does today. @v5 forwards - # unconditionally. - mask_decode_mode=("accurate" if USE_INFERENCE_MODELS else mask_decode_mode), - tradeoff_factor=1.0 if USE_INFERENCE_MODELS else tradeoff_factor, + # The local backend flag does not identify the remote server. + mask_decode_mode="accurate", + tradeoff_factor=1.0, max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, source="workflow-execution", diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py index f20af2bee8..0bceebe08b 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py @@ -408,13 +408,9 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned only where the fields were discarded. On the legacy - # backend they are honoured, so this version keeps passing them - # through and behaves exactly as it does today. @v5 forwards - # unconditionally. - mask_decode_mode=("accurate" if USE_INFERENCE_MODELS else mask_decode_mode), - tradeoff_factor=1.0 if USE_INFERENCE_MODELS else tradeoff_factor, - response_mask_format="rle", + # The local backend flag does not identify the remote server. + mask_decode_mode="accurate", + tradeoff_factor=1.0, max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, source="workflow-execution", diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5.py index 6a465f3768..b262e8df19 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5.py @@ -1,5 +1,6 @@ from typing import List, Literal, Optional, Type, Union +import numpy as np from pydantic import ConfigDict, Field, PositiveInt, model_validator from roboflow_workflows.core_steps.common.entities import StepExecutionMode from roboflow_workflows.core_steps.common.utils import ( @@ -16,7 +17,10 @@ WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, ) -from roboflow_workflows.execution_engine.constants import INFERENCE_ID_KEY +from roboflow_workflows.execution_engine.constants import ( + INFERENCE_ID_KEY, + RLE_MASK_KEY_IN_SV_DETECTIONS, +) from roboflow_workflows.execution_engine.entities.base import ( Batch, OutputDefinition, @@ -434,6 +438,27 @@ def _post_process_result( ) -> BlockResult: inference_ids = [p.get(INFERENCE_ID_KEY, None) for p in predictions] predictions = convert_inference_detections_batch_to_sv_detections(predictions) + for detections in predictions: + rle_masks = detections.data.get(RLE_MASK_KEY_IN_SV_DETECTIONS) + if rle_masks is None or detections.mask is None: + continue + + # Supervision restores dense masks to the image grid. Keep the RLE + # copy consistent for downstream blocks that decode it directly. + for index, rle_mask in enumerate(rle_masks): + if rle_mask is None or tuple(rle_mask["size"]) == tuple( + detections.mask.shape[1:] + ): + continue + + from pycocotools import mask as mask_utils + + normalized = mask_utils.encode( + np.asfortranarray(detections.mask[index], dtype=np.uint8) + ) + normalized["counts"] = normalized["counts"].decode("utf-8") + rle_masks[index] = normalized + predictions = attach_prediction_type_info_to_sv_detections_batch( predictions=predictions, prediction_type="instance-segmentation", diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py index 60589ae4db..4778066f14 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py @@ -227,7 +227,7 @@ class BlockManifest(WorkflowBlockManifest): Selector(kind=[STRING_KIND]), ] = Field( default="accurate", - description="Parameter of mask decoding in prediction post-processing.", + description="Tensor mode currently ignores this setting and always uses accurate decoding with full-resolution masks. Full tensor support will come in a separate PR.", examples=["accurate", "$inputs.mask_decode_mode"], ) tradeoff_factor: Union[ @@ -235,7 +235,7 @@ class BlockManifest(WorkflowBlockManifest): Selector(kind=[FLOAT_ZERO_TO_ONE_KIND]), ] = Field( default=0.0, - description="Post-processing parameter to dictate tradeoff between fast and accurate. 0.0 keeps masks on the model's own grid, 1.0 resizes them to the image. Note that a lower value does not guarantee smaller masks: when the image is smaller than the model grid the interpolation runs the other way.", + description="Tensor mode currently ignores this setting and always uses accurate / 1.0 (full-resolution masks). Reduced-resolution tensor masks are not yet supported. Post-processing parameter to dictate tradeoff between fast and accurate. 0.0 keeps masks on the model's own grid, 1.0 resizes them to the image. Note that a lower value does not guarantee smaller masks: when the image is smaller than the model grid the interpolation runs the other way.", examples=[0.3, "$inputs.tradeoff_factor"], ) disable_active_learning: Union[bool, Selector(kind=[BOOLEAN_KIND])] = Field( From 3c308034933850900c6beaf2f16cb32933ff1fb9 Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Tue, 6 Oct 2026 09:57:47 +0200 Subject: [PATCH 22/41] Restore RLE responses for remote v4 segmentation --- .../test_v5_mask_resolution.py | 16 ++++++++++------ workflows/CHANGELOG.md | 2 ++ .../models/roboflow/instance_segmentation/v4.py | 1 + 3 files changed, 13 insertions(+), 6 deletions(-) diff --git a/tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_v5_mask_resolution.py b/tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_v5_mask_resolution.py index c87c013789..1b8892954b 100644 --- a/tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_v5_mask_resolution.py +++ b/tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_v5_mask_resolution.py @@ -184,10 +184,11 @@ def test_v5_non_tensor_has_no_pinned_call_sites(self) -> None: assert source.count("mask_decode_mode=mask_decode_mode,") == 4 -@pytest.mark.parametrize("version", PINNED_VERSIONS) +@pytest.mark.parametrize("version", ALL_VERSIONS) @pytest.mark.parametrize("local_backend", [True, False]) -def test_remote_request_is_independent_of_local_backend( - monkeypatch, version, local_backend +@pytest.mark.parametrize("remote_target", ["hosted", "local"]) +def test_remote_request_preserves_versioned_mask_settings( + monkeypatch, version, local_backend, remote_target ): from importlib import import_module from unittest.mock import MagicMock @@ -203,7 +204,9 @@ def test_remote_request_is_independent_of_local_backend( block._post_process_result = MagicMock(return_value=[]) client = MagicMock() monkeypatch.setattr(module, "InferenceHTTPClient", MagicMock(return_value=client)) - monkeypatch.setattr(module, "USE_INFERENCE_MODELS", local_backend) + if version in PINNED_VERSIONS: + monkeypatch.setattr(module, "USE_INFERENCE_MODELS", local_backend) + monkeypatch.setattr(module, "WORKFLOWS_REMOTE_API_TARGET", remote_target) block.run_remotely( images=[], @@ -221,8 +224,9 @@ def test_remote_request_is_independent_of_local_backend( ) config = client.configure.call_args.kwargs["inference_configuration"] - assert config.mask_decode_mode == "accurate" - assert config.tradeoff_factor == 1.0 + assert config.mask_decode_mode == ("fast" if version == "v5" else "accurate") + assert config.tradeoff_factor == (0.0 if version == "v5" else 1.0) + assert config.response_mask_format == ("rle" if version in {"v4", "v5"} else None) def test_v5_reduced_rle_output_can_be_decoded_on_original_image(monkeypatch): diff --git a/workflows/CHANGELOG.md b/workflows/CHANGELOG.md index 1c9d17e418..acf1cd3cf7 100644 --- a/workflows/CHANGELOG.md +++ b/workflows/CHANGELOG.md @@ -47,6 +47,8 @@ for contributor and maintainer responsibilities. ### Fixed +- Instance segmentation `@v4` remote execution retains its RLE response format + while pinning mask resolution, preserving holes and disconnected mask regions. - Non-tensor instance segmentation `@v5` re-encodes reduced RLE masks at the image grid after conversion to Supervision, so downstream blocks receive consistent dense and RLE representations. diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py index 0bceebe08b..685aba2e95 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py @@ -411,6 +411,7 @@ def run_remotely( # The local backend flag does not identify the remote server. mask_decode_mode="accurate", tradeoff_factor=1.0, + response_mask_format="rle", max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, source="workflow-execution", From 8d40815e753e8bc1b275012bad59ca07fa2564ba Mon Sep 17 00:00:00 2001 From: MTaraszewski <43919218+MTaraszewski@users.noreply.github.com> Date: Tue, 6 Oct 2026 12:04:05 +0200 Subject: [PATCH 23/41] Update workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3_tensor.py Co-authored-by: Grzegorz Klimaszewski <166530809+grzegorz-roboflow@users.noreply.github.com> --- .../models/roboflow/instance_segmentation/v3_tensor.py | 4 ---- 1 file changed, 4 deletions(-) diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3_tensor.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3_tensor.py index 33dd50ec9a..64c957d3a5 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3_tensor.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3_tensor.py @@ -508,10 +508,6 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned unconditionally: the reason here is downstream, not the - # backend. Nothing under roboflow_workflows/ reads mask_size and - # the sites rebuilding InstancesRLEMasks drop it, so a reduced grid - # is reinterpreted as image-sized whichever backend produced it. mask_decode_mode="accurate", tradeoff_factor=1.0, response_mask_format="rle", From 70d46973b3747531c2ad575ee800d3e9466e54b9 Mon Sep 17 00:00:00 2001 From: MTaraszewski <43919218+MTaraszewski@users.noreply.github.com> Date: Tue, 6 Oct 2026 12:04:42 +0200 Subject: [PATCH 24/41] Update workflows/roboflow_workflows/configuration.py Co-authored-by: Grzegorz Klimaszewski <166530809+grzegorz-roboflow@users.noreply.github.com> --- workflows/roboflow_workflows/configuration.py | 5 ----- 1 file changed, 5 deletions(-) diff --git a/workflows/roboflow_workflows/configuration.py b/workflows/roboflow_workflows/configuration.py index dce4a5465f..9a6c6e5396 100644 --- a/workflows/roboflow_workflows/configuration.py +++ b/workflows/roboflow_workflows/configuration.py @@ -142,11 +142,6 @@ class FontsConfiguration: @dataclass(frozen=True) class ModelsConfiguration: - # Whether the host serves instance segmentation through inference_models. - # The legacy backend honours mask_decode_mode / tradeoff_factor, so block - # versions up to @v4 must keep forwarding them there; on inference_models - # those fields were discarded, so forwarding them now would change masks an - # existing workflow produces. Defaults to the inference_models default. use_inference_models: bool = True lmm_enabled: bool = False vlm_segmentation_max_polygon_vertices: int = 500 From e6ad1df2d92faffea7228bc5ab34f7f0295f3fb4 Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Tue, 6 Oct 2026 12:10:01 +0200 Subject: [PATCH 25/41] Remove repeated mask-resolution comments --- .../models/roboflow/instance_segmentation/v1.py | 5 ----- .../roboflow/instance_segmentation/v1_tensor.py | 7 ------- .../models/roboflow/instance_segmentation/v2.py | 5 ----- .../roboflow/instance_segmentation/v2_tensor.py | 7 ------- .../models/roboflow/instance_segmentation/v3.py | 5 ----- .../roboflow/instance_segmentation/v3_tensor.py | 3 --- .../models/roboflow/instance_segmentation/v4.py | 5 ----- .../roboflow/instance_segmentation/v4_tensor.py | 7 ------- .../roboflow/instance_segmentation/v5_tensor.py | 14 -------------- 9 files changed, 58 deletions(-) diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1.py index 00f9db87ed..f2eab35ea0 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1.py @@ -320,10 +320,6 @@ def run_locally( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned only where the fields were discarded. On the legacy - # backend they are honoured, so this version keeps passing them - # through and behaves exactly as it does today. @v5 forwards - # unconditionally. mask_decode_mode=("accurate" if USE_INFERENCE_MODELS else mask_decode_mode), tradeoff_factor=1.0 if USE_INFERENCE_MODELS else tradeoff_factor, enforce_dense_masks_in_inference_models=enforce_dense_masks_in_inference_models, @@ -372,7 +368,6 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # The local backend flag does not identify the remote server. mask_decode_mode="accurate", tradeoff_factor=1.0, max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1_tensor.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1_tensor.py index c2875ae16c..fc6534420c 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1_tensor.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v1_tensor.py @@ -391,9 +391,6 @@ def run_locally( class_filter=class_filter, max_detections=max_detections, max_candidates=max_candidates, - # Pinned: these fields never took effect in this version via the - # inference_models backend, so honouring them now would change - # existing workflows. @v5 forwards. mask_decode_mode="accurate", tradeoff_factor=1.0, disable_active_learning=disable_active_learning, @@ -468,10 +465,6 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned unconditionally: the reason here is downstream, not the - # backend. Nothing under roboflow_workflows/ reads mask_size and - # the sites rebuilding InstancesRLEMasks drop it, so a reduced grid - # is reinterpreted as image-sized whichever backend produced it. mask_decode_mode="accurate", tradeoff_factor=1.0, response_mask_format="rle", diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2.py index 48d485d041..ef5f629b87 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2.py @@ -318,10 +318,6 @@ def run_locally( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned only where the fields were discarded. On the legacy - # backend they are honoured, so this version keeps passing them - # through and behaves exactly as it does today. @v5 forwards - # unconditionally. mask_decode_mode=("accurate" if USE_INFERENCE_MODELS else mask_decode_mode), tradeoff_factor=1.0 if USE_INFERENCE_MODELS else tradeoff_factor, enforce_dense_masks_in_inference_models=enforce_dense_masks_in_inference_models, @@ -371,7 +367,6 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # The local backend flag does not identify the remote server. mask_decode_mode="accurate", tradeoff_factor=1.0, max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2_tensor.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2_tensor.py index 3c7391a7cb..fbe10324ff 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2_tensor.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v2_tensor.py @@ -389,9 +389,6 @@ def run_locally( class_filter=class_filter, max_detections=max_detections, max_candidates=max_candidates, - # Pinned: these fields never took effect in this version via the - # inference_models backend, so honouring them now would change - # existing workflows. @v5 forwards. mask_decode_mode="accurate", tradeoff_factor=1.0, disable_active_learning=disable_active_learning, @@ -467,10 +464,6 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned unconditionally: the reason here is downstream, not the - # backend. Nothing under roboflow_workflows/ reads mask_size and - # the sites rebuilding InstancesRLEMasks drop it, so a reduced grid - # is reinterpreted as image-sized whichever backend produced it. mask_decode_mode="accurate", tradeoff_factor=1.0, response_mask_format="rle", diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3.py index e2730ae129..473f75f4e3 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3.py @@ -393,10 +393,6 @@ def run_locally( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned only where the fields were discarded. On the legacy - # backend they are honoured, so this version keeps passing them - # through and behaves exactly as it does today. @v5 forwards - # unconditionally. mask_decode_mode=("accurate" if USE_INFERENCE_MODELS else mask_decode_mode), tradeoff_factor=1.0 if USE_INFERENCE_MODELS else tradeoff_factor, enforce_dense_masks_in_inference_models=enforce_dense_masks_in_inference_models, @@ -708,7 +704,6 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # The local backend flag does not identify the remote server. mask_decode_mode="accurate", tradeoff_factor=1.0, max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3_tensor.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3_tensor.py index 64c957d3a5..931a227baa 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3_tensor.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v3_tensor.py @@ -430,9 +430,6 @@ def run_locally( class_filter=class_filter, max_detections=max_detections, max_candidates=max_candidates, - # Pinned: these fields never took effect in this version via the - # inference_models backend, so honouring them now would change - # existing workflows. @v5 forwards. mask_decode_mode="accurate", tradeoff_factor=1.0, disable_active_learning=disable_active_learning, diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py index 685aba2e95..72c3e7f818 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4.py @@ -355,10 +355,6 @@ def run_locally( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned only where the fields were discarded. On the legacy - # backend they are honoured, so this version keeps passing them - # through and behaves exactly as it does today. @v5 forwards - # unconditionally. mask_decode_mode=("accurate" if USE_INFERENCE_MODELS else mask_decode_mode), tradeoff_factor=1.0 if USE_INFERENCE_MODELS else tradeoff_factor, response_mask_format="rle", @@ -408,7 +404,6 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # The local backend flag does not identify the remote server. mask_decode_mode="accurate", tradeoff_factor=1.0, response_mask_format="rle", diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4_tensor.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4_tensor.py index 5e23b5f9d1..430df98954 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4_tensor.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v4_tensor.py @@ -421,9 +421,6 @@ def run_locally( class_filter=class_filter, max_detections=max_detections, max_candidates=max_candidates, - # Pinned: these fields never took effect in this version via the - # inference_models backend, so honouring them now would change - # existing workflows. @v5 forwards. mask_decode_mode="accurate", tradeoff_factor=1.0, disable_active_learning=disable_active_learning, @@ -495,10 +492,6 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Pinned unconditionally: the reason here is downstream, not the - # backend. Nothing under roboflow_workflows/ reads mask_size and - # the sites rebuilding InstancesRLEMasks drop it, so a reduced grid - # is reinterpreted as image-sized whichever backend produced it. mask_decode_mode="accurate", tradeoff_factor=1.0, response_mask_format="rle", diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py index 4778066f14..cf5c9c2441 100644 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py +++ b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py @@ -421,13 +421,6 @@ def run_locally( class_filter=class_filter, max_detections=max_detections, max_candidates=max_candidates, - # Tensor-native mode does not support reduced mask resolution yet. - # Nothing under workflows/ reads InstanceDetections.mask_size, and the - # ~35 sites that rebuild InstancesRLEMasks drop it, so a reduced grid - # would be reinterpreted as image-sized downstream. Upsampling at this - # boundary would restore correctness but produce output identical to - # factor 1.0 at strictly higher cost, so the request is pinned instead. - # Lift this once mask_size is propagated through the tensor pipeline. mask_decode_mode="accurate", tradeoff_factor=1.0, disable_active_learning=disable_active_learning, @@ -499,13 +492,6 @@ def run_remotely( iou_threshold=iou_threshold, max_detections=max_detections, max_candidates=max_candidates, - # Tensor-native mode does not support reduced mask resolution yet. - # Nothing under workflows/ reads InstanceDetections.mask_size, and the - # ~35 sites that rebuild InstancesRLEMasks drop it, so a reduced grid - # would be reinterpreted as image-sized downstream. Upsampling at this - # boundary would restore correctness but produce output identical to - # factor 1.0 at strictly higher cost, so the request is pinned instead. - # Lift this once mask_size is propagated through the tensor pipeline. mask_decode_mode="accurate", tradeoff_factor=1.0, response_mask_format="rle", From 1ada7511f5492f3ceab666cf61f625216f8d5913 Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Tue, 6 Oct 2026 16:40:39 +0200 Subject: [PATCH 26/41] Fix crop mask canvases and expand segmentation coverage Place origin crops on the original-image canvas for dense and RLE outputs. Preserve tensors when encoding dense adapter results as RLE. Add real-model integration, HTTP server, and platform workflow regression coverage. Validated in the existing Python 3.12 .venv: 406 regression tests passed, 3 skipped; 6 ONNX and 3 Torch integration cases passed; 12 HTTP server cases passed; 6 local workflow checks passed using a public model. Black, isort, and git diff --check passed. TensorRT execution and authenticated staging E2E remain pending. --- .../core/models/inference_models_adapters.py | 6 +- inference_models/docs/changelog.md | 7 +- .../models/base/instance_segmentation.py | 5 +- .../models/common/roboflow/post_processing.py | 27 +- ...t_instance_segmentation_mask_resolution.py | 266 ++++++++++++++++++ .../roboflow/test_mask_image_dimensions.py | 2 +- .../common/roboflow/test_post_processing.py | 2 +- ...t_instance_segmentation_mask_resolution.py | 95 +++++++ ...t_instance_segmentation_rle_predictions.py | 68 +++++ .../models/test_inference_models_adapters.py | 60 ++-- 10 files changed, 480 insertions(+), 58 deletions(-) create mode 100644 inference_models/tests/integration_tests/models/test_instance_segmentation_mask_resolution.py create mode 100644 tests/inference/hosted_platform_tests/workflows_examples/roboflow_models/test_instance_segmentation_mask_resolution.py diff --git a/inference/core/models/inference_models_adapters.py b/inference/core/models/inference_models_adapters.py index fa4bc8c315..ade47bb5ca 100644 --- a/inference/core/models/inference_models_adapters.py +++ b/inference/core/models/inference_models_adapters.py @@ -1035,13 +1035,13 @@ def _build_responses_from_detections( xyxy = det.xyxy.detach().cpu().numpy() confs = det.confidence.detach().cpu().numpy() if isinstance(det.mask, torch.Tensor): - masks = det.mask.detach().cpu().numpy() + masks = det.mask.detach().cpu() if return_in_rle: polys_or_rles = [ torch_mask_to_coco_rle(mask=mask) for mask in masks ] else: - polys_or_rles = masks2poly(masks) + polys_or_rles = masks2poly(masks.numpy()) else: if return_in_rle: polys_or_rles = det.mask.to_coco_rle_masks() @@ -1058,8 +1058,6 @@ def _build_responses_from_detections( # reported; equal sizes short-circuit to a no-op. This also breaks # any view into the pinned scratch buffers noted above, since the # scaled polygons are freshly allocated. - # Offset crops are already on an original-image canvas; origin crops - # still cover only the cropped region. mask_size = getattr(det, "mask_size", None) produced_against = resolve_mask_frame_size(preproc_metadata) if ( diff --git a/inference_models/docs/changelog.md b/inference_models/docs/changelog.md index 1f59cd9f32..5ef60525d0 100644 --- a/inference_models/docs/changelog.md +++ b/inference_models/docs/changelog.md @@ -9,7 +9,7 @@ `0.0` leaves them on the model's own grid; values between interpolate the resize target. Threaded through all six instance-segmentation families on both the dense and RLE paths, and through the ONNX, TorchScript and TensorRT - backends, plus RF-DETR CoreML. Default output is unchanged. + backends, plus RF-DETR CoreML. Accurate mode remains the default. - `mask_size` on `InstanceDetections` and `InstancesRLEMasks`, recording the grid the masks live on. Each defaults to the value it effectively had before, so existing construction is unaffected. @@ -36,8 +36,9 @@ ### Fixed - Dense reduced masks now resize correctly in `to_supervision()`, including mask - annotation on the original image. Origin-anchored crop masks are padded to the - original image after resizing to the crop dimensions. + annotation on the original image. All static-crop masks, including crops at + the origin, are placed on an original-image canvas before dense or RLE output. + RLE consumers can therefore restore reduced masks without stretching the crop. - Polygon responses for static crops with non-zero offsets use the original image canvas as their coordinate frame, at full and reduced mask resolution. - RF-DETR CoreML now forwards `masks_resolution_factor` to post-processing. diff --git a/inference_models/inference_models/models/base/instance_segmentation.py b/inference_models/inference_models/models/base/instance_segmentation.py index dbcc7a1a4e..01133b13ac 100644 --- a/inference_models/inference_models/models/base/instance_segmentation.py +++ b/inference_models/inference_models/models/base/instance_segmentation.py @@ -140,8 +140,9 @@ class InstanceDetections: Attributes: image_size: Original image ``(height, width)``. Older callers may omit it. mask_size: Encoded mask grid ``(height, width)``. - mask_frame_size: Image-space extent represented by the mask grid. A crop - anchored at the origin may cover only the top-left of the image. + mask_frame_size: Image-space extent represented by the mask grid. Model + outputs cover the original image; manually constructed crop-local + masks may specify a smaller extent anchored at the origin. """ xyxy: torch.Tensor # (n_boxes, 4) diff --git a/inference_models/inference_models/models/common/roboflow/post_processing.py b/inference_models/inference_models/models/common/roboflow/post_processing.py index 2dc31a2852..cc153ca3ab 100644 --- a/inference_models/inference_models/models/common/roboflow/post_processing.py +++ b/inference_models/inference_models/models/common/roboflow/post_processing.py @@ -445,14 +445,9 @@ def resolve_mask_frame_size(metadata: PreProcessingMetadata) -> ImageDimensions: metadata: Preprocessing transforms used to align the masks. Returns: - Original image dimensions for offset-crop canvases, or the crop - dimensions when alignment leaves masks anchored at the origin. + Original image dimensions, including for crops anchored at the origin. """ - offset = getattr(metadata, "static_crop_offset", None) - if offset is not None and (offset.offset_x > 0 or offset.offset_y > 0): - return metadata.original_size - - return metadata.size_after_pre_processing + return metadata.original_size def resolve_mask_target_size( @@ -573,7 +568,11 @@ def align_instance_segmentation_results( masks_resolution_factor=masks_resolution_factor, ) empty_height, empty_width = empty_target_height, empty_target_width - if static_crop_offset.offset_x > 0 or static_crop_offset.offset_y > 0: + if ( + static_crop_offset.offset_x > 0 + or static_crop_offset.offset_y > 0 + or size_after_pre_processing != original_size + ): height_scale = empty_target_height / size_after_pre_processing.height width_scale = empty_target_width / size_after_pre_processing.width empty_height = max( @@ -665,7 +664,11 @@ def align_instance_segmentation_results( interpolation=functional.InterpolationMode.BILINEAR, ).gt_(binarization_threshold) masks = binarized_masks - if static_crop_offset.offset_x > 0 or static_crop_offset.offset_y > 0: + if ( + static_crop_offset.offset_x > 0 + or static_crop_offset.offset_y > 0 + or size_after_pre_processing != original_size + ): # the canvas keeps the mask grid's ratio to the image, so a reduced # factor shrinks the whole output rather than planting a small mask on # a full-size canvas; crop offsets move into the same space @@ -763,7 +766,11 @@ def align_instance_segmentation_results_to_rle_masks( ) image_bboxes[:, :4].div_(scale) - needs_canvas = static_crop_offset.offset_x > 0 or static_crop_offset.offset_y > 0 + needs_canvas = ( + static_crop_offset.offset_x > 0 + or static_crop_offset.offset_y > 0 + or size_after_pre_processing != original_size + ) if needs_canvas: static_crop_offsets = torch.as_tensor( [ diff --git a/inference_models/tests/integration_tests/models/test_instance_segmentation_mask_resolution.py b/inference_models/tests/integration_tests/models/test_instance_segmentation_mask_resolution.py new file mode 100644 index 0000000000..52f72192ba --- /dev/null +++ b/inference_models/tests/integration_tests/models/test_instance_segmentation_mask_resolution.py @@ -0,0 +1,266 @@ +from importlib import import_module + +import numpy as np +import pytest +import supervision as sv +import torch + +from inference_models.models.common.rle_utils import coco_rle_masks_to_torch_mask +from inference_models.models.common.roboflow.model_packages import StaticCrop + +MODEL_CASES = [ + pytest.param( + "yolov5", + "onnx", + "YOLOv5ForInstanceSegmentationOnnx", + "asl_yolov5_onnx_seg_static_bs_letterbox", + "asl_image_numpy", + marks=pytest.mark.onnx_extras, + id="yolov5-onnx", + ), + pytest.param( + "yolov7", + "onnx", + "YOLOv7ForInstanceSegmentationOnnx", + "asl_yolov7_onnx_seg_static_bs_letterbox", + "asl_image_numpy", + marks=pytest.mark.onnx_extras, + id="yolov7-onnx", + ), + pytest.param( + "yolact", + "onnx", + "YOLOACTForInstanceSegmentationOnnx", + "asl_yolact_onnx_seg_static_bs_letterbox", + "asl_image_numpy", + marks=pytest.mark.onnx_extras, + id="yolact-onnx", + ), + pytest.param( + "yolov8", + "onnx", + "YOLOv8ForInstanceSegmentationOnnx", + "asl_yolov8n_onnx_seg_static_bs_stretch", + "asl_image_numpy", + marks=pytest.mark.onnx_extras, + id="yolov8-onnx", + ), + pytest.param( + "yolo26", + "onnx", + "YOLO26ForInstanceSegmentationOnnx", + "yolo26n_seg_snakes_stretch_onnx_static_package", + "snake_image_numpy", + marks=pytest.mark.onnx_extras, + id="yolo26-onnx", + ), + pytest.param( + "rfdetr", + "onnx", + "RFDetrForInstanceSegmentationOnnx", + "snakes_rfdetr_seg_onnx_static_bs_stretch_package", + "snake_image_numpy", + marks=pytest.mark.onnx_extras, + id="rfdetr-onnx", + ), + pytest.param( + "yolov8", + "torch_script", + "YOLOv8ForInstanceSegmentationTorchScript", + "asl_yolov8n_torchscript_seg_static_bs_stretch", + "asl_image_numpy", + marks=pytest.mark.torch_models, + id="yolov8-torch", + ), + pytest.param( + "yolo26", + "torch_script", + "YOLO26ForInstanceSegmentationTorchScript", + "yolo26n_seg_snakes_stretch_torch_script_package", + "snake_image_numpy", + marks=pytest.mark.torch_models, + id="yolo26-torch", + ), + pytest.param( + "rfdetr", + "pytorch", + "RFDetrForInstanceSegmentationTorch", + "snakes_rfdetr_seg_torch_stretch_package", + "snake_image_numpy", + marks=pytest.mark.torch_models, + id="rfdetr-torch", + ), + pytest.param( + "yolov8", + "trt", + "YOLOv8ForInstanceSegmentationTRT", + "yolov8_seg_asl_trt_package", + "asl_image_numpy", + marks=[pytest.mark.trt_extras, pytest.mark.gpu_only], + id="yolov8-trt", + ), + pytest.param( + "yolo26", + "trt", + "YOLO26ForInstanceSegmentationTRT", + "yolo26_seg_asl_trt_package", + "asl_image_numpy", + marks=[pytest.mark.trt_extras, pytest.mark.gpu_only], + id="yolo26-trt", + ), + pytest.param( + "rfdetr", + "trt", + "RFDetrForInstanceSegmentationTRT", + "rfdetr_seg_asl_trt_package", + "asl_image_numpy", + marks=[pytest.mark.trt_extras, pytest.mark.gpu_only], + id="rfdetr-trt", + ), +] + + +@pytest.mark.slow +@pytest.mark.parametrize( + "family,backend,class_name,package_fixture,image_fixture", MODEL_CASES +) +def test_model_mask_resolution_and_crop_round_trip( + request, monkeypatch, family, backend, class_name, package_fixture, image_fixture +) -> None: + module = import_module( + f"inference_models.models.{family}.{family}_instance_segmentation_{backend}" + ) + load_kwargs = {} + if backend == "onnx": + load_kwargs["onnx_execution_providers"] = ["CPUExecutionProvider"] + load_kwargs["device"] = torch.device("cpu") + elif backend == "trt": + load_kwargs["engine_host_code_allowed"] = True + + model = getattr(module, class_name).from_pretrained( + model_name_or_path=request.getfixturevalue(package_fixture), **load_kwargs + ) + image = request.getfixturevalue(image_fixture) + image = np.pad(image, ((0, image.shape[0] % 2), (0, image.shape[1] % 2), (0, 0))) + height, width = image.shape[:2] + preprocessing = model._inference_config.image_pre_processing + + confidence = 0.6 if family == "yolov7" else 0.25 + accurate = None + mask_sizes = {} + for factor in (1.0, 0.5, 0.0): + reference = None + for crop in (None, "origin", "offset"): + if crop is None: + scene = image + offset_x = offset_y = 0 + crop_config = None + else: + offset_y = 0 if crop == "origin" else height // 2 + offset_x = 0 if crop == "origin" else width // 2 + scene = np.zeros((2 * height, 2 * width, 3), dtype=image.dtype) + scene[offset_y : offset_y + height, offset_x : offset_x + width] = image + start = 0 if crop == "origin" else 25 + crop_config = StaticCrop( + enabled=True, + x_min=start, + y_min=start, + x_max=start + 50, + y_max=start + 50, + ) + + with monkeypatch.context() as patch: + patch.setattr(preprocessing, "static_crop", crop_config) + dense = model( + scene, + confidence=confidence, + mask_format="dense", + masks_resolution_factor=factor, + )[0] + rle = model( + scene, + confidence=confidence, + mask_format="rle", + masks_resolution_factor=factor, + )[0] + + assert len(dense.xyxy) > 0, (family, backend, factor, crop) + torch.testing.assert_close(dense.xyxy.cpu(), rle.xyxy.cpu()) + torch.testing.assert_close(dense.class_id.cpu(), rle.class_id.cpu()) + torch.testing.assert_close(dense.confidence.cpu(), rle.confidence.cpu()) + decoded = coco_rle_masks_to_torch_mask(rle.mask, device=torch.device("cpu")) + np.testing.assert_array_equal(decoded.numpy(), dense.mask.cpu().numpy()) + assert dense.image_size == scene.shape[:2] + assert dense.mask_frame_size == scene.shape[:2] + if factor == 1.0: + assert dense.mask_size == scene.shape[:2] + else: + assert dense.mask_size != scene.shape[:2] + + detections = dense.to_supervision() + np.testing.assert_array_equal(detections.mask, rle.to_supervision().mask) + assert detections.mask.shape == (len(dense.xyxy), *scene.shape[:2]) + assert detections.mask.any() + sv.MaskAnnotator().annotate(scene.copy(), detections) + if reference is None: + reference = detections + mask_sizes[factor] = dense.mask_size + if accurate is None: + accurate = detections + else: + np.testing.assert_allclose(detections.xyxy, accurate.xyxy, atol=1) + np.testing.assert_array_equal( + detections.class_id, accurate.class_id + ) + intersection = np.logical_and(detections.mask, accurate.mask).sum( + axis=(1, 2) + ) + union = np.logical_or(detections.mask, accurate.mask).sum( + axis=(1, 2) + ) + assert np.all( + (union == 0) | (intersection / np.maximum(union, 1) > 0.8) + ) + else: + expected = np.zeros_like(detections.mask) + expected[ + :, offset_y : offset_y + height, offset_x : offset_x + width + ] = reference.mask + # Resizing a canvas can round boundary pixels differently from + # resizing the crop before placing it on that canvas. + intersection = np.logical_and(detections.mask, expected).sum( + axis=(1, 2) + ) + union = np.logical_or(detections.mask, expected).sum(axis=(1, 2)) + reduced_offset = crop == "offset" and factor < 1.0 + minimum_iou = 0.9 if reduced_offset else 0.99 + pixel_tolerance = ( + np.ceil( + max( + scene.shape[0] / dense.mask_size[0], + scene.shape[1] / dense.mask_size[1], + ) + ) + if reduced_offset + else 1 + ) + assert np.all( + (union == 0) | (intersection / np.maximum(union, 1) > minimum_iou) + ), (factor, crop) + np.testing.assert_allclose( + sv.mask_to_xyxy(detections.mask), + sv.mask_to_xyxy(expected), + atol=pixel_tolerance, + err_msg=f"factor={factor}, crop={crop}", + ) + np.testing.assert_allclose( + detections.xyxy, + reference.xyxy + [offset_x, offset_y, offset_x, offset_y], + atol=1, + ) + + np.testing.assert_allclose( + mask_sizes[0.5], + np.round((np.array(mask_sizes[0.0]) + np.array(mask_sizes[1.0])) / 2), + atol=1, + ) diff --git a/inference_models/tests/unit_tests/models/common/roboflow/test_mask_image_dimensions.py b/inference_models/tests/unit_tests/models/common/roboflow/test_mask_image_dimensions.py index e571b25f9d..b57d7e5faf 100644 --- a/inference_models/tests/unit_tests/models/common/roboflow/test_mask_image_dimensions.py +++ b/inference_models/tests/unit_tests/models/common/roboflow/test_mask_image_dimensions.py @@ -69,7 +69,7 @@ def test_model_result_retains_image_dimensions(family, mask_format, offset) -> N result = results[0] assert result.image_size == (200, 300) - assert result.mask_frame_size == ((200, 300) if any(offset) else (100, 150)) + assert result.mask_frame_size == (200, 300) converted = result.to_supervision() assert converted.mask.shape == (1, 200, 300) assert converted.mask[ diff --git a/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py b/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py index 43f6bb9ff7..820d18d01e 100644 --- a/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py +++ b/inference_models/tests/unit_tests/models/common/roboflow/test_post_processing.py @@ -1461,7 +1461,7 @@ def test_empty_masks_match_populated_crop_canvas_after_rounding(offset: int) -> ) assert empty.shape[1:] == populated.shape[1:] - assert empty.shape[1:] == ((123, 123) if offset else (81, 81)) + assert empty.shape[1:] == ((123, 123) if offset else (122, 122)) @pytest.mark.parametrize( diff --git a/tests/inference/hosted_platform_tests/workflows_examples/roboflow_models/test_instance_segmentation_mask_resolution.py b/tests/inference/hosted_platform_tests/workflows_examples/roboflow_models/test_instance_segmentation_mask_resolution.py new file mode 100644 index 0000000000..bec9075851 --- /dev/null +++ b/tests/inference/hosted_platform_tests/workflows_examples/roboflow_models/test_instance_segmentation_mask_resolution.py @@ -0,0 +1,95 @@ +import numpy as np +import pytest +import supervision as sv + +from inference_sdk import InferenceConfiguration, InferenceHTTPClient +from inference_sdk.http.entities import VisualisationResponseFormat +from tests.inference.hosted_platform_tests.conftest import ( + ROBOFLOW_API_KEY, + apply_auth_mode, +) + + +@pytest.mark.parametrize("version", ["v4", "v5"]) +@pytest.mark.parametrize( + "mode,factor", [("accurate", 1.0), ("tradeoff", 0.5), ("fast", 0.0)] +) +def test_mask_resolution_workflow_visualization_and_serialization( + instance_segmentation_service_url: str, + segmentation_model_id: str, + asl_image: np.ndarray, + auth_mode: str, + version: str, + mode: str, + factor: float, +) -> None: + client = InferenceHTTPClient( + api_url=instance_segmentation_service_url, api_key=ROBOFLOW_API_KEY + ).configure( + InferenceConfiguration( + output_visualisation_format=VisualisationResponseFormat.NUMPY + ) + ) + client = apply_auth_mode(client, auth_mode) + specification = { + "version": "1.0", + "inputs": [{"type": "WorkflowImage", "name": "image"}], + "steps": [ + { + "type": f"roboflow_core/roboflow_instance_segmentation_model@{version}", + "name": "segment", + "image": "$inputs.image", + "model_id": segmentation_model_id, + "mask_decode_mode": mode, + "tradeoff_factor": factor, + }, + { + "type": "roboflow_core/mask_visualization@v1", + "name": "visualize", + "image": "$inputs.image", + "predictions": "$steps.segment.predictions", + }, + ], + "outputs": [ + { + "type": "JsonField", + "name": "predictions", + "selector": "$steps.segment.predictions", + }, + { + "type": "JsonField", + "name": "visualization", + "selector": "$steps.visualize.image", + }, + ], + } + + results = client.run_workflow( + specification=specification, + images={"image": [asl_image, asl_image]}, + ) + + assert len(results) == 2 + for result in results: + predictions = result["predictions"] + detections = sv.Detections.from_inference(predictions) + assert len(detections) > 0 + assert detections.mask.shape == (len(detections), *asl_image.shape[:2]) + assert detections.mask.any() + assert all( + prediction.get("rle_mask") for prediction in predictions["predictions"] + ) + assert all( + tuple(prediction["rle_mask"]["size"]) == asl_image.shape[:2] + for prediction in predictions["predictions"] + ) + sv.MaskAnnotator().annotate(asl_image.copy(), detections) + visualization = result["visualization"] + assert isinstance(visualization, np.ndarray) + assert visualization.shape == asl_image.shape + assert np.any(visualization != asl_image) + + first = sv.Detections.from_inference(results[0]["predictions"]) + second = sv.Detections.from_inference(results[1]["predictions"]) + np.testing.assert_allclose(first.xyxy, second.xyxy, atol=1) + np.testing.assert_array_equal(first.mask, second.mask) diff --git a/tests/inference/integration_tests/test_instance_segmentation_rle_predictions.py b/tests/inference/integration_tests/test_instance_segmentation_rle_predictions.py index 62dddcdb2c..bca7e85fb0 100644 --- a/tests/inference/integration_tests/test_instance_segmentation_rle_predictions.py +++ b/tests/inference/integration_tests/test_instance_segmentation_rle_predictions.py @@ -1,6 +1,7 @@ import os import numpy as np +import pytest import requests import supervision as sv from numpy import ndarray @@ -17,6 +18,73 @@ BASE_URL = os.environ.get("BASE_URL", "http://localhost") +@pytest.mark.skipif( + not USE_INFERENCE_MODELS, reason="Resolution control uses inference-models" +) +@pytest.mark.parametrize("response_format", ["polygon", "rle"]) +@pytest.mark.parametrize( + "mode,factor", [("accurate", 1.0), ("tradeoff", 0.5), ("fast", 0.0)] +) +def test_mask_resolution_round_trip_through_server( + auth_mode: str, response_format: str, mode: str, factor: float +) -> None: + payload = { + "image": {"type": "url", "value": "https://media.roboflow.com/dog.jpeg"}, + "api_key": API_KEY, + "model_id": "yolov8n-seg-640", + "response_mask_format": response_format, + "mask_decode_mode": mode, + "tradeoff_factor": factor, + } + response = requests.post( + f"{BASE_URL}:{PORT}/infer/instance_segmentation", + json=without_api_key_in_header_mode(auth_mode, payload), + headers=api_key_auth_headers(auth_mode, API_KEY), + timeout=120, + ) + response.raise_for_status() + result = response.json() + detections = sv.Detections.from_inference(result) + height, width = result["image"]["height"], result["image"]["width"] + assert len(detections) > 0 + assert all( + prediction["mask_format"] == response_format + for prediction in result["predictions"] + ) + assert detections.mask.shape == (len(detections), height, width) + assert detections.mask.any() + sv.MaskAnnotator().annotate( + np.zeros((height, width, 3), dtype=np.uint8), detections + ) + + if response_format == "rle": + sizes = { + tuple(prediction["rle"]["size"]) for prediction in result["predictions"] + } + assert len(sizes) == 1 + mask_height, mask_width = sizes.pop() + if mode == "accurate": + assert (mask_height, mask_width) == (height, width) + else: + assert mask_height < height and mask_width < width + + reference_response = requests.post( + f"{BASE_URL}:{PORT}/infer/instance_segmentation", + json=without_api_key_in_header_mode( + auth_mode, {**payload, "mask_decode_mode": "accurate"} + ), + headers=api_key_auth_headers(auth_mode, API_KEY), + timeout=120, + ) + reference_response.raise_for_status() + reference = sv.Detections.from_inference(reference_response.json()) + np.testing.assert_allclose(detections.xyxy, reference.xyxy, atol=1) + np.testing.assert_array_equal(detections.class_id, reference.class_id) + intersection = np.logical_and(detections.mask, reference.mask).sum(axis=(1, 2)) + union = np.logical_or(detections.mask, reference.mask).sum(axis=(1, 2)) + assert np.all(intersection / np.maximum(union, 1) > 0.8) + + def test_v1_endpoint_with_valid_payload(auth_mode: str) -> None: payload = { "image": { diff --git a/tests/inference/unit_tests/core/models/test_inference_models_adapters.py b/tests/inference/unit_tests/core/models/test_inference_models_adapters.py index 1a22c4ac76..62f69e8f9a 100644 --- a/tests/inference/unit_tests/core/models/test_inference_models_adapters.py +++ b/tests/inference/unit_tests/core/models/test_inference_models_adapters.py @@ -7,6 +7,7 @@ import numpy as np import pytest +import supervision as sv import torch from pycocotools import mask as mask_utils @@ -1100,43 +1101,12 @@ def test_non_square_grid(self) -> None: assert polygon[:, 1].max() <= 80 -def test_origin_anchored_crop_is_not_mistaken_for_a_reduced_grid() -> None: - # given - # a static crop at (0, 0) yields a mask smaller than the image, but its - # coordinates are already image-space. Inferring "reduced" from - # mask_size != image_size would scale them a second time. - adapter = _seg_adapter() - metadata = [ - SimpleNamespace( - original_size=SimpleNamespace(width=16, height=16), - size_after_pre_processing=SimpleNamespace(width=8, height=8), - ) - ] - mask = torch.zeros((1, 8, 8), dtype=torch.uint8) - mask[0, 2:6, 2:6] = 1 - detections = [ - InstanceDetections( - xyxy=torch.tensor([[2, 2, 6, 6]], dtype=torch.int32), - confidence=torch.tensor([0.9], dtype=torch.float32), - class_id=torch.tensor([0], dtype=torch.int32), - mask=mask, - ) - ] - - # when - responses = adapter._build_responses_from_detections(detections, metadata) - - # then - # the mask already matches size_after_pre_processing, so no scaling - xs = [p.x for p in responses[0].predictions[0].points] - assert max(xs) <= 8, f"coordinates were scaled a second time: {xs}" - - @pytest.mark.parametrize("mask_format", ["dense", "rle"]) -@pytest.mark.parametrize("factor", [1.0, 0.0]) +@pytest.mark.parametrize("factor", [1.0, 0.5, 0.0]) @pytest.mark.parametrize("offset", [(0, 0), (60, 40)]) -def test_crop_polygon_response_uses_the_frame_represented_by_mask( - mask_format: str, factor: float, offset: tuple +@pytest.mark.parametrize("response_format", ["polygon", "rle"]) +def test_crop_response_uses_the_frame_represented_by_mask( + mask_format: str, factor: float, offset: tuple, response_format: str ) -> None: from inference_models.entities import ImageDimensions from inference_models.models.common.roboflow.model_packages import ( @@ -1198,15 +1168,31 @@ def test_crop_polygon_response_uses_the_frame_represented_by_mask( ) response = _seg_adapter()._build_responses_from_detections( - [detections], [metadata] + [detections], [metadata], response_mask_format=response_format )[0] + if response_format == "rle": + converted = sv.Detections.from_inference(response.model_dump(by_alias=True)) + expected = np.zeros((1, 200, 300), dtype=bool) + expected[:, 10 + offset_y : 50 + offset_y, 20 + offset_x : 80 + offset_x] = True + assert converted.mask.shape == expected.shape + np.testing.assert_allclose( + sv.mask_to_xyxy(converted.mask), sv.mask_to_xyxy(expected), atol=1 + ) + assert ( + np.logical_and(converted.mask, expected).sum() + / np.logical_or(converted.mask, expected).sum() + > 0.95 + ) + sv.MaskAnnotator().annotate(np.zeros((200, 300, 3), dtype=np.uint8), converted) + return + prediction = response.predictions[0] xs = [point.x for point in prediction.points] ys = [point.y for point in prediction.points] assert min(xs) == pytest.approx(20 + offset_x) assert min(ys) == pytest.approx(10 + offset_y) - pixel_size = 1 if factor == 1.0 else 5 + pixel_size = 150 / round(30 * (1 - factor) + 150 * factor) assert max(xs) == pytest.approx(80 + offset_x - pixel_size) assert max(ys) == pytest.approx(50 + offset_y - pixel_size) assert prediction.x == pytest.approx(50 + offset_x) From f5f39ab6ec32a0d679b7b24b35638d0d38693399 Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Wed, 7 Oct 2026 09:53:08 +0200 Subject: [PATCH 27/41] Align segmentation tests with model suites Flatten unit test classes into standalone functions and move mask-resolution coverage into explicit model-specific tests. Check mask areas and positions against visually inspected CPU outputs, while retaining existing TensorRT area baselines. Validation in the existing repository .venv: 17 unit tests and 81 CPU integration cases passed. All 27 TensorRT cases collected; GPU execution remains pending. Black, isort, and git diff --check passed. --- ...t_instance_segmentation_mask_resolution.py | 266 ----------- .../test_rfdetr_seg_predictions_onnx.py | 113 +++++ .../test_rfdetr_seg_predictions_torch.py | 111 +++++ .../models/test_rfdetr_seg_predictions_trt.py | 92 ++++ ..._instance_segmentation_predictions_onnx.py | 109 +++++ ..._instance_segmentation_predictions_onnx.py | 113 +++++ ...instance_segmentation_predictions_torch.py | 111 +++++ ...6_instance_segmentation_predictions_trt.py | 92 ++++ ..._instance_segmentation_predictions_onnx.py | 109 +++++ ..._instance_segmentation_predictions_onnx.py | 109 +++++ ..._instance_segmentation_predictions_onnx.py | 109 +++++ ...instance_segmentation_predictions_torch.py | 107 +++++ ...8_instance_segmentation_predictions_trt.py | 92 ++++ .../models/base/test_instance_segmentation.py | 427 +++++++++--------- 14 files changed, 1474 insertions(+), 486 deletions(-) delete mode 100644 inference_models/tests/integration_tests/models/test_instance_segmentation_mask_resolution.py diff --git a/inference_models/tests/integration_tests/models/test_instance_segmentation_mask_resolution.py b/inference_models/tests/integration_tests/models/test_instance_segmentation_mask_resolution.py deleted file mode 100644 index 52f72192ba..0000000000 --- a/inference_models/tests/integration_tests/models/test_instance_segmentation_mask_resolution.py +++ /dev/null @@ -1,266 +0,0 @@ -from importlib import import_module - -import numpy as np -import pytest -import supervision as sv -import torch - -from inference_models.models.common.rle_utils import coco_rle_masks_to_torch_mask -from inference_models.models.common.roboflow.model_packages import StaticCrop - -MODEL_CASES = [ - pytest.param( - "yolov5", - "onnx", - "YOLOv5ForInstanceSegmentationOnnx", - "asl_yolov5_onnx_seg_static_bs_letterbox", - "asl_image_numpy", - marks=pytest.mark.onnx_extras, - id="yolov5-onnx", - ), - pytest.param( - "yolov7", - "onnx", - "YOLOv7ForInstanceSegmentationOnnx", - "asl_yolov7_onnx_seg_static_bs_letterbox", - "asl_image_numpy", - marks=pytest.mark.onnx_extras, - id="yolov7-onnx", - ), - pytest.param( - "yolact", - "onnx", - "YOLOACTForInstanceSegmentationOnnx", - "asl_yolact_onnx_seg_static_bs_letterbox", - "asl_image_numpy", - marks=pytest.mark.onnx_extras, - id="yolact-onnx", - ), - pytest.param( - "yolov8", - "onnx", - "YOLOv8ForInstanceSegmentationOnnx", - "asl_yolov8n_onnx_seg_static_bs_stretch", - "asl_image_numpy", - marks=pytest.mark.onnx_extras, - id="yolov8-onnx", - ), - pytest.param( - "yolo26", - "onnx", - "YOLO26ForInstanceSegmentationOnnx", - "yolo26n_seg_snakes_stretch_onnx_static_package", - "snake_image_numpy", - marks=pytest.mark.onnx_extras, - id="yolo26-onnx", - ), - pytest.param( - "rfdetr", - "onnx", - "RFDetrForInstanceSegmentationOnnx", - "snakes_rfdetr_seg_onnx_static_bs_stretch_package", - "snake_image_numpy", - marks=pytest.mark.onnx_extras, - id="rfdetr-onnx", - ), - pytest.param( - "yolov8", - "torch_script", - "YOLOv8ForInstanceSegmentationTorchScript", - "asl_yolov8n_torchscript_seg_static_bs_stretch", - "asl_image_numpy", - marks=pytest.mark.torch_models, - id="yolov8-torch", - ), - pytest.param( - "yolo26", - "torch_script", - "YOLO26ForInstanceSegmentationTorchScript", - "yolo26n_seg_snakes_stretch_torch_script_package", - "snake_image_numpy", - marks=pytest.mark.torch_models, - id="yolo26-torch", - ), - pytest.param( - "rfdetr", - "pytorch", - "RFDetrForInstanceSegmentationTorch", - "snakes_rfdetr_seg_torch_stretch_package", - "snake_image_numpy", - marks=pytest.mark.torch_models, - id="rfdetr-torch", - ), - pytest.param( - "yolov8", - "trt", - "YOLOv8ForInstanceSegmentationTRT", - "yolov8_seg_asl_trt_package", - "asl_image_numpy", - marks=[pytest.mark.trt_extras, pytest.mark.gpu_only], - id="yolov8-trt", - ), - pytest.param( - "yolo26", - "trt", - "YOLO26ForInstanceSegmentationTRT", - "yolo26_seg_asl_trt_package", - "asl_image_numpy", - marks=[pytest.mark.trt_extras, pytest.mark.gpu_only], - id="yolo26-trt", - ), - pytest.param( - "rfdetr", - "trt", - "RFDetrForInstanceSegmentationTRT", - "rfdetr_seg_asl_trt_package", - "asl_image_numpy", - marks=[pytest.mark.trt_extras, pytest.mark.gpu_only], - id="rfdetr-trt", - ), -] - - -@pytest.mark.slow -@pytest.mark.parametrize( - "family,backend,class_name,package_fixture,image_fixture", MODEL_CASES -) -def test_model_mask_resolution_and_crop_round_trip( - request, monkeypatch, family, backend, class_name, package_fixture, image_fixture -) -> None: - module = import_module( - f"inference_models.models.{family}.{family}_instance_segmentation_{backend}" - ) - load_kwargs = {} - if backend == "onnx": - load_kwargs["onnx_execution_providers"] = ["CPUExecutionProvider"] - load_kwargs["device"] = torch.device("cpu") - elif backend == "trt": - load_kwargs["engine_host_code_allowed"] = True - - model = getattr(module, class_name).from_pretrained( - model_name_or_path=request.getfixturevalue(package_fixture), **load_kwargs - ) - image = request.getfixturevalue(image_fixture) - image = np.pad(image, ((0, image.shape[0] % 2), (0, image.shape[1] % 2), (0, 0))) - height, width = image.shape[:2] - preprocessing = model._inference_config.image_pre_processing - - confidence = 0.6 if family == "yolov7" else 0.25 - accurate = None - mask_sizes = {} - for factor in (1.0, 0.5, 0.0): - reference = None - for crop in (None, "origin", "offset"): - if crop is None: - scene = image - offset_x = offset_y = 0 - crop_config = None - else: - offset_y = 0 if crop == "origin" else height // 2 - offset_x = 0 if crop == "origin" else width // 2 - scene = np.zeros((2 * height, 2 * width, 3), dtype=image.dtype) - scene[offset_y : offset_y + height, offset_x : offset_x + width] = image - start = 0 if crop == "origin" else 25 - crop_config = StaticCrop( - enabled=True, - x_min=start, - y_min=start, - x_max=start + 50, - y_max=start + 50, - ) - - with monkeypatch.context() as patch: - patch.setattr(preprocessing, "static_crop", crop_config) - dense = model( - scene, - confidence=confidence, - mask_format="dense", - masks_resolution_factor=factor, - )[0] - rle = model( - scene, - confidence=confidence, - mask_format="rle", - masks_resolution_factor=factor, - )[0] - - assert len(dense.xyxy) > 0, (family, backend, factor, crop) - torch.testing.assert_close(dense.xyxy.cpu(), rle.xyxy.cpu()) - torch.testing.assert_close(dense.class_id.cpu(), rle.class_id.cpu()) - torch.testing.assert_close(dense.confidence.cpu(), rle.confidence.cpu()) - decoded = coco_rle_masks_to_torch_mask(rle.mask, device=torch.device("cpu")) - np.testing.assert_array_equal(decoded.numpy(), dense.mask.cpu().numpy()) - assert dense.image_size == scene.shape[:2] - assert dense.mask_frame_size == scene.shape[:2] - if factor == 1.0: - assert dense.mask_size == scene.shape[:2] - else: - assert dense.mask_size != scene.shape[:2] - - detections = dense.to_supervision() - np.testing.assert_array_equal(detections.mask, rle.to_supervision().mask) - assert detections.mask.shape == (len(dense.xyxy), *scene.shape[:2]) - assert detections.mask.any() - sv.MaskAnnotator().annotate(scene.copy(), detections) - if reference is None: - reference = detections - mask_sizes[factor] = dense.mask_size - if accurate is None: - accurate = detections - else: - np.testing.assert_allclose(detections.xyxy, accurate.xyxy, atol=1) - np.testing.assert_array_equal( - detections.class_id, accurate.class_id - ) - intersection = np.logical_and(detections.mask, accurate.mask).sum( - axis=(1, 2) - ) - union = np.logical_or(detections.mask, accurate.mask).sum( - axis=(1, 2) - ) - assert np.all( - (union == 0) | (intersection / np.maximum(union, 1) > 0.8) - ) - else: - expected = np.zeros_like(detections.mask) - expected[ - :, offset_y : offset_y + height, offset_x : offset_x + width - ] = reference.mask - # Resizing a canvas can round boundary pixels differently from - # resizing the crop before placing it on that canvas. - intersection = np.logical_and(detections.mask, expected).sum( - axis=(1, 2) - ) - union = np.logical_or(detections.mask, expected).sum(axis=(1, 2)) - reduced_offset = crop == "offset" and factor < 1.0 - minimum_iou = 0.9 if reduced_offset else 0.99 - pixel_tolerance = ( - np.ceil( - max( - scene.shape[0] / dense.mask_size[0], - scene.shape[1] / dense.mask_size[1], - ) - ) - if reduced_offset - else 1 - ) - assert np.all( - (union == 0) | (intersection / np.maximum(union, 1) > minimum_iou) - ), (factor, crop) - np.testing.assert_allclose( - sv.mask_to_xyxy(detections.mask), - sv.mask_to_xyxy(expected), - atol=pixel_tolerance, - err_msg=f"factor={factor}, crop={crop}", - ) - np.testing.assert_allclose( - detections.xyxy, - reference.xyxy + [offset_x, offset_y, offset_x, offset_y], - atol=1, - ) - - np.testing.assert_allclose( - mask_sizes[0.5], - np.round((np.array(mask_sizes[0.0]) + np.array(mask_sizes[1.0])) / 2), - atol=1, - ) diff --git a/inference_models/tests/integration_tests/models/test_rfdetr_seg_predictions_onnx.py b/inference_models/tests/integration_tests/models/test_rfdetr_seg_predictions_onnx.py index 5ff4194185..855bd313d3 100644 --- a/inference_models/tests/integration_tests/models/test_rfdetr_seg_predictions_onnx.py +++ b/inference_models/tests/integration_tests/models/test_rfdetr_seg_predictions_onnx.py @@ -1,8 +1,10 @@ import numpy as np import pytest +import supervision as sv import torch from inference_models.models.common.rle_utils import coco_rle_masks_to_torch_mask +from inference_models.models.common.roboflow.model_packages import StaticCrop @pytest.mark.slow @@ -1505,3 +1507,114 @@ def test_onnx_per_class_confidence_blocks_all_classes( ) predictions = model(snake_image_numpy, confidence="best") assert predictions[0].class_id.numel() == 0 + + +@pytest.mark.slow +@pytest.mark.onnx_extras +@pytest.mark.parametrize( + "factor,expected_grid,expected_area", + [(1.0, (890, 1280), 206406), (0.5, (499, 694), 206437), (0.0, (108, 108), 207327)], +) +@pytest.mark.parametrize( + "crop_start", [None, 0, 25], ids=["uncropped", "origin-crop", "offset-crop"] +) +def test_mask_resolution_and_crop_round_trip( + snakes_rfdetr_seg_onnx_static_bs_stretch_package: str, + snake_image_numpy: np.ndarray, + monkeypatch: pytest.MonkeyPatch, + factor: float, + crop_start: int | None, + expected_grid: tuple, + expected_area: int, +) -> None: + # given + from inference_models.models.rfdetr.rfdetr_instance_segmentation_onnx import ( + RFDetrForInstanceSegmentationOnnx, + ) + + model = RFDetrForInstanceSegmentationOnnx.from_pretrained( + model_name_or_path=snakes_rfdetr_seg_onnx_static_bs_stretch_package, + onnx_execution_providers=["CPUExecutionProvider"], + device=torch.device("cpu"), + ) + image = np.pad( + snake_image_numpy, + ( + (0, snake_image_numpy.shape[0] % 2), + (0, snake_image_numpy.shape[1] % 2), + (0, 0), + ), + ) + height, width = image.shape[:2] + scene = image + offset_x = offset_y = 0 + crop_config = None + if crop_start is not None: + offset_y = 0 if crop_start == 0 else height // 2 + offset_x = 0 if crop_start == 0 else width // 2 + scene = np.zeros((2 * height, 2 * width, 3), dtype=image.dtype) + scene[offset_y : offset_y + height, offset_x : offset_x + width] = image + crop_config = StaticCrop( + enabled=True, + x_min=crop_start, + y_min=crop_start, + x_max=crop_start + 50, + y_max=crop_start + 50, + ) + monkeypatch.setattr( + model._inference_config.image_pre_processing, "static_crop", crop_config + ) + + # when + dense = model( + scene, confidence=0.25, mask_format="dense", masks_resolution_factor=factor + )[0] + rle = model( + scene, confidence=0.25, mask_format="rle", masks_resolution_factor=factor + )[0] + decoded = coco_rle_masks_to_torch_mask( + instances_masks=rle.mask, device=torch.device("cpu") + ) + detections = dense.to_supervision() + + # then + offset = np.array([offset_x, offset_y, offset_x, offset_y]) + expected_boxes = np.array([[133, 328, 1261, 557]]) + offset + np.testing.assert_allclose(detections.xyxy, expected_boxes, atol=2, rtol=0) + np.testing.assert_array_equal(detections.class_id, [0]) + np.testing.assert_allclose(detections.confidence, [0.9737], atol=0.01, rtol=0) + assert dense.image_size == scene.shape[:2] + assert dense.mask_frame_size == scene.shape[:2] + assert rle.image_size == scene.shape[:2] + assert rle.mask_frame_size == scene.shape[:2] + canvas_multiplier = 1 if crop_start is None else 2 + assert dense.mask_size == tuple(size * canvas_multiplier for size in expected_grid) + assert rle.mask_size == dense.mask_size + assert detections.mask.shape == (1, *scene.shape[:2]) + # Areas were checked against overlays of this fixture at all three factors. + assert detections.mask.sum() == pytest.approx(expected_area, rel=0.02) + expected_mask_boxes = np.array([[128, 329, 1263, 558]]) + offset + pixel_tolerance = ( + np.ceil( + max( + scene.shape[0] / dense.mask_size[0], scene.shape[1] / dense.mask_size[1] + ) + ) + + 2 + ) + np.testing.assert_allclose( + sv.mask_to_xyxy(detections.mask), + expected_mask_boxes, + atol=pixel_tolerance, + rtol=0, + ) + crop_masks = detections.mask[ + :, offset_y : offset_y + height, offset_x : offset_x + width + ] + assert crop_masks.sum() == detections.mask.sum() + torch.testing.assert_close(dense.xyxy.cpu(), rle.xyxy.cpu()) + torch.testing.assert_close(dense.class_id.cpu(), rle.class_id.cpu()) + torch.testing.assert_close(dense.confidence.cpu(), rle.confidence.cpu()) + np.testing.assert_array_equal(decoded.numpy(), dense.mask.cpu().numpy()) + np.testing.assert_array_equal(detections.mask, rle.to_supervision().mask) + sv.MaskAnnotator().annotate(scene.copy(), detections) diff --git a/inference_models/tests/integration_tests/models/test_rfdetr_seg_predictions_torch.py b/inference_models/tests/integration_tests/models/test_rfdetr_seg_predictions_torch.py index f675c83eb4..d2f767a722 100644 --- a/inference_models/tests/integration_tests/models/test_rfdetr_seg_predictions_torch.py +++ b/inference_models/tests/integration_tests/models/test_rfdetr_seg_predictions_torch.py @@ -1,8 +1,10 @@ import numpy as np import pytest +import supervision as sv import torch from inference_models.models.common.rle_utils import coco_rle_masks_to_torch_mask +from inference_models.models.common.roboflow.model_packages import StaticCrop @pytest.mark.slow @@ -1503,3 +1505,112 @@ def test_torch_per_class_confidence_filters_detections( ) predictions = model(snake_image_numpy, confidence="best") assert predictions[0].class_id.numel() == 0 + + +@pytest.mark.slow +@pytest.mark.torch_models +@pytest.mark.parametrize( + "factor,expected_grid,expected_area", + [(1.0, (890, 1280), 206398), (0.5, (499, 694), 206464), (0.0, (108, 108), 206675)], +) +@pytest.mark.parametrize( + "crop_start", [None, 0, 25], ids=["uncropped", "origin-crop", "offset-crop"] +) +def test_mask_resolution_and_crop_round_trip( + snakes_rfdetr_seg_torch_stretch_package: str, + snake_image_numpy: np.ndarray, + monkeypatch: pytest.MonkeyPatch, + factor: float, + crop_start: int | None, + expected_grid: tuple, + expected_area: int, +) -> None: + # given + from inference_models.models.rfdetr.rfdetr_instance_segmentation_pytorch import ( + RFDetrForInstanceSegmentationTorch, + ) + + model = RFDetrForInstanceSegmentationTorch.from_pretrained( + model_name_or_path=snakes_rfdetr_seg_torch_stretch_package, + ) + image = np.pad( + snake_image_numpy, + ( + (0, snake_image_numpy.shape[0] % 2), + (0, snake_image_numpy.shape[1] % 2), + (0, 0), + ), + ) + height, width = image.shape[:2] + scene = image + offset_x = offset_y = 0 + crop_config = None + if crop_start is not None: + offset_y = 0 if crop_start == 0 else height // 2 + offset_x = 0 if crop_start == 0 else width // 2 + scene = np.zeros((2 * height, 2 * width, 3), dtype=image.dtype) + scene[offset_y : offset_y + height, offset_x : offset_x + width] = image + crop_config = StaticCrop( + enabled=True, + x_min=crop_start, + y_min=crop_start, + x_max=crop_start + 50, + y_max=crop_start + 50, + ) + monkeypatch.setattr( + model._inference_config.image_pre_processing, "static_crop", crop_config + ) + + # when + dense = model( + scene, confidence=0.25, mask_format="dense", masks_resolution_factor=factor + )[0] + rle = model( + scene, confidence=0.25, mask_format="rle", masks_resolution_factor=factor + )[0] + decoded = coco_rle_masks_to_torch_mask( + instances_masks=rle.mask, device=torch.device("cpu") + ) + detections = dense.to_supervision() + + # then + offset = np.array([offset_x, offset_y, offset_x, offset_y]) + expected_boxes = np.array([[125, 330, 1267, 561]]) + offset + np.testing.assert_allclose(detections.xyxy, expected_boxes, atol=2, rtol=0) + np.testing.assert_array_equal(detections.class_id, [0]) + np.testing.assert_allclose(detections.confidence, [0.9452], atol=0.01, rtol=0) + assert dense.image_size == scene.shape[:2] + assert dense.mask_frame_size == scene.shape[:2] + assert rle.image_size == scene.shape[:2] + assert rle.mask_frame_size == scene.shape[:2] + canvas_multiplier = 1 if crop_start is None else 2 + assert dense.mask_size == tuple(size * canvas_multiplier for size in expected_grid) + assert rle.mask_size == dense.mask_size + assert detections.mask.shape == (1, *scene.shape[:2]) + # Areas were checked against overlays of this fixture at all three factors. + assert detections.mask.sum() == pytest.approx(expected_area, rel=0.02) + expected_mask_boxes = np.array([[129, 329, 1279, 558]]) + offset + pixel_tolerance = ( + np.ceil( + max( + scene.shape[0] / dense.mask_size[0], scene.shape[1] / dense.mask_size[1] + ) + ) + + 2 + ) + np.testing.assert_allclose( + sv.mask_to_xyxy(detections.mask), + expected_mask_boxes, + atol=pixel_tolerance, + rtol=0, + ) + crop_masks = detections.mask[ + :, offset_y : offset_y + height, offset_x : offset_x + width + ] + assert crop_masks.sum() == detections.mask.sum() + torch.testing.assert_close(dense.xyxy.cpu(), rle.xyxy.cpu()) + torch.testing.assert_close(dense.class_id.cpu(), rle.class_id.cpu()) + torch.testing.assert_close(dense.confidence.cpu(), rle.confidence.cpu()) + np.testing.assert_array_equal(decoded.numpy(), dense.mask.cpu().numpy()) + np.testing.assert_array_equal(detections.mask, rle.to_supervision().mask) + sv.MaskAnnotator().annotate(scene.copy(), detections) diff --git a/inference_models/tests/integration_tests/models/test_rfdetr_seg_predictions_trt.py b/inference_models/tests/integration_tests/models/test_rfdetr_seg_predictions_trt.py index 9a0153ae06..a055cadabb 100644 --- a/inference_models/tests/integration_tests/models/test_rfdetr_seg_predictions_trt.py +++ b/inference_models/tests/integration_tests/models/test_rfdetr_seg_predictions_trt.py @@ -2,10 +2,12 @@ import numpy as np import pytest +import supervision as sv import torch from inference_models.errors import CorruptedModelPackageError from inference_models.models.common.rle_utils import coco_rle_masks_to_torch_mask +from inference_models.models.common.roboflow.model_packages import StaticCrop def _assert_instance_segmentation_predictions_match(actual, expected) -> None: @@ -599,3 +601,93 @@ def test_trt_per_class_confidence_filters_detections( ) predictions = model(asl_image_numpy, confidence="best") assert predictions[0].class_id.numel() == 0 + + +@pytest.mark.slow +@pytest.mark.trt_extras +@pytest.mark.gpu_only +@pytest.mark.parametrize("factor", [1.0, 0.5, 0.0]) +@pytest.mark.parametrize( + "crop_start", [None, 0, 25], ids=["uncropped", "origin-crop", "offset-crop"] +) +def test_mask_resolution_and_crop_round_trip( + rfdetr_seg_asl_trt_package: str, + asl_image_numpy: np.ndarray, + monkeypatch: pytest.MonkeyPatch, + factor: float, + crop_start: int | None, +) -> None: + # given + from inference_models.models.rfdetr.rfdetr_instance_segmentation_trt import ( + RFDetrForInstanceSegmentationTRT, + ) + + model = RFDetrForInstanceSegmentationTRT.from_pretrained( + model_name_or_path=rfdetr_seg_asl_trt_package, + engine_host_code_allowed=True, + ) + image = np.pad( + asl_image_numpy, + ((0, asl_image_numpy.shape[0] % 2), (0, asl_image_numpy.shape[1] % 2), (0, 0)), + ) + height, width = image.shape[:2] + scene = image + offset_x = offset_y = 0 + crop_config = None + if crop_start is not None: + offset_y = 0 if crop_start == 0 else height // 2 + offset_x = 0 if crop_start == 0 else width // 2 + scene = np.zeros((2 * height, 2 * width, 3), dtype=image.dtype) + scene[offset_y : offset_y + height, offset_x : offset_x + width] = image + crop_config = StaticCrop( + enabled=True, + x_min=crop_start, + y_min=crop_start, + x_max=crop_start + 50, + y_max=crop_start + 50, + ) + monkeypatch.setattr( + model._inference_config.image_pre_processing, "static_crop", crop_config + ) + + # when + dense = model( + scene, confidence=0.25, mask_format="dense", masks_resolution_factor=factor + )[0] + rle = model( + scene, confidence=0.25, mask_format="rle", masks_resolution_factor=factor + )[0] + decoded = coco_rle_masks_to_torch_mask( + instances_masks=rle.mask, device=torch.device("cpu") + ) + detections = dense.to_supervision() + + # then + offset = np.array([offset_x, offset_y, offset_x, offset_y]) + expected_boxes = np.array([[63, 172, 188, 374]]) + offset + np.testing.assert_allclose(detections.xyxy, expected_boxes, atol=5, rtol=0) + np.testing.assert_array_equal(detections.class_id, [20]) + np.testing.assert_allclose(detections.confidence, [0.9491], atol=0.01, rtol=0) + assert dense.image_size == scene.shape[:2] + assert dense.mask_frame_size == scene.shape[:2] + assert rle.image_size == scene.shape[:2] + assert rle.mask_frame_size == scene.shape[:2] + if factor == 1.0: + assert dense.mask_size == scene.shape[:2] + else: + assert dense.mask_size != scene.shape[:2] + assert rle.mask_size == dense.mask_size + assert detections.mask.shape == (1, *scene.shape[:2]) + # Reuse the full-resolution area baseline in test_trt_package_numpy. + expected_area = 16046 + assert detections.mask.sum() == pytest.approx(expected_area, rel=0.05) + crop_masks = detections.mask[ + :, offset_y : offset_y + height, offset_x : offset_x + width + ] + assert crop_masks.sum() == detections.mask.sum() + torch.testing.assert_close(dense.xyxy.cpu(), rle.xyxy.cpu()) + torch.testing.assert_close(dense.class_id.cpu(), rle.class_id.cpu()) + torch.testing.assert_close(dense.confidence.cpu(), rle.confidence.cpu()) + np.testing.assert_array_equal(decoded.numpy(), dense.mask.cpu().numpy()) + np.testing.assert_array_equal(detections.mask, rle.to_supervision().mask) + sv.MaskAnnotator().annotate(scene.copy(), detections) diff --git a/inference_models/tests/integration_tests/models/test_yolact_instance_segmentation_predictions_onnx.py b/inference_models/tests/integration_tests/models/test_yolact_instance_segmentation_predictions_onnx.py index 2eb0a83067..870d9d38d4 100644 --- a/inference_models/tests/integration_tests/models/test_yolact_instance_segmentation_predictions_onnx.py +++ b/inference_models/tests/integration_tests/models/test_yolact_instance_segmentation_predictions_onnx.py @@ -1,8 +1,10 @@ import numpy as np import pytest +import supervision as sv import torch from inference_models.models.common.rle_utils import coco_rle_masks_to_torch_mask +from inference_models.models.common.roboflow.model_packages import StaticCrop @pytest.mark.slow @@ -714,3 +716,110 @@ def test_onnx_per_class_confidence_blocks_specific_class( ) predictions = model(asl_image_numpy, confidence="best") assert 18 not in predictions[0].class_id.cpu().tolist() + + +@pytest.mark.slow +@pytest.mark.onnx_extras +@pytest.mark.parametrize( + "factor,expected_grid,expected_area", + [(1.0, (416, 416), 14979), (0.5, (277, 277), 14934), (0.0, (138, 138), 14895)], +) +@pytest.mark.parametrize( + "crop_start", [None, 0, 25], ids=["uncropped", "origin-crop", "offset-crop"] +) +def test_mask_resolution_and_crop_round_trip( + asl_yolact_onnx_seg_static_bs_letterbox: str, + asl_image_numpy: np.ndarray, + monkeypatch: pytest.MonkeyPatch, + factor: float, + crop_start: int | None, + expected_grid: tuple, + expected_area: int, +) -> None: + # given + from inference_models.models.yolact.yolact_instance_segmentation_onnx import ( + YOLOACTForInstanceSegmentationOnnx, + ) + + model = YOLOACTForInstanceSegmentationOnnx.from_pretrained( + model_name_or_path=asl_yolact_onnx_seg_static_bs_letterbox, + onnx_execution_providers=["CPUExecutionProvider"], + device=torch.device("cpu"), + ) + image = np.pad( + asl_image_numpy, + ((0, asl_image_numpy.shape[0] % 2), (0, asl_image_numpy.shape[1] % 2), (0, 0)), + ) + height, width = image.shape[:2] + scene = image + offset_x = offset_y = 0 + crop_config = None + if crop_start is not None: + offset_y = 0 if crop_start == 0 else height // 2 + offset_x = 0 if crop_start == 0 else width // 2 + scene = np.zeros((2 * height, 2 * width, 3), dtype=image.dtype) + scene[offset_y : offset_y + height, offset_x : offset_x + width] = image + crop_config = StaticCrop( + enabled=True, + x_min=crop_start, + y_min=crop_start, + x_max=crop_start + 50, + y_max=crop_start + 50, + ) + monkeypatch.setattr( + model._inference_config.image_pre_processing, "static_crop", crop_config + ) + + # when + dense = model( + scene, confidence=0.25, mask_format="dense", masks_resolution_factor=factor + )[0] + rle = model( + scene, confidence=0.25, mask_format="rle", masks_resolution_factor=factor + )[0] + decoded = coco_rle_masks_to_torch_mask( + instances_masks=rle.mask, device=torch.device("cpu") + ) + detections = dense.to_supervision() + + # then + offset = np.array([offset_x, offset_y, offset_x, offset_y]) + expected_boxes = np.array([[66, 167, 188, 373]]) + offset + np.testing.assert_allclose(detections.xyxy, expected_boxes, atol=2, rtol=0) + np.testing.assert_array_equal(detections.class_id, [18]) + np.testing.assert_allclose(detections.confidence, [0.5063], atol=0.01, rtol=0) + assert dense.image_size == scene.shape[:2] + assert dense.mask_frame_size == scene.shape[:2] + assert rle.image_size == scene.shape[:2] + assert rle.mask_frame_size == scene.shape[:2] + canvas_multiplier = 1 if crop_start is None else 2 + assert dense.mask_size == tuple(size * canvas_multiplier for size in expected_grid) + assert rle.mask_size == dense.mask_size + assert detections.mask.shape == (1, *scene.shape[:2]) + # Areas were checked against overlays of this fixture at all three factors. + assert detections.mask.sum() == pytest.approx(expected_area, rel=0.02) + expected_mask_boxes = np.array([[66, 166, 181, 366]]) + offset + pixel_tolerance = ( + np.ceil( + max( + scene.shape[0] / dense.mask_size[0], scene.shape[1] / dense.mask_size[1] + ) + ) + + 2 + ) + np.testing.assert_allclose( + sv.mask_to_xyxy(detections.mask), + expected_mask_boxes, + atol=pixel_tolerance, + rtol=0, + ) + crop_masks = detections.mask[ + :, offset_y : offset_y + height, offset_x : offset_x + width + ] + assert crop_masks.sum() == detections.mask.sum() + torch.testing.assert_close(dense.xyxy.cpu(), rle.xyxy.cpu()) + torch.testing.assert_close(dense.class_id.cpu(), rle.class_id.cpu()) + torch.testing.assert_close(dense.confidence.cpu(), rle.confidence.cpu()) + np.testing.assert_array_equal(decoded.numpy(), dense.mask.cpu().numpy()) + np.testing.assert_array_equal(detections.mask, rle.to_supervision().mask) + sv.MaskAnnotator().annotate(scene.copy(), detections) diff --git a/inference_models/tests/integration_tests/models/test_yolo26_instance_segmentation_predictions_onnx.py b/inference_models/tests/integration_tests/models/test_yolo26_instance_segmentation_predictions_onnx.py index cd4a2fc699..49dd6b4a00 100644 --- a/inference_models/tests/integration_tests/models/test_yolo26_instance_segmentation_predictions_onnx.py +++ b/inference_models/tests/integration_tests/models/test_yolo26_instance_segmentation_predictions_onnx.py @@ -1,8 +1,10 @@ import numpy as np import pytest +import supervision as sv import torch from inference_models.models.common.rle_utils import coco_rle_masks_to_torch_mask +from inference_models.models.common.roboflow.model_packages import StaticCrop CONFIDENCE_ATOL = 0.01 XYXY_ATOL = 2 @@ -780,3 +782,114 @@ def test_onnx_per_class_confidence_blocks_snake_class( ) predictions = model(snake_image_numpy, confidence="best") assert predictions[0].class_id.numel() == 0 + + +@pytest.mark.slow +@pytest.mark.onnx_extras +@pytest.mark.parametrize( + "factor,expected_grid,expected_area", + [(1.0, (890, 1280), 211140), (0.5, (501, 696), 211025), (0.0, (112, 112), 208787)], +) +@pytest.mark.parametrize( + "crop_start", [None, 0, 25], ids=["uncropped", "origin-crop", "offset-crop"] +) +def test_mask_resolution_and_crop_round_trip( + yolo26n_seg_snakes_stretch_onnx_static_package: str, + snake_image_numpy: np.ndarray, + monkeypatch: pytest.MonkeyPatch, + factor: float, + crop_start: int | None, + expected_grid: tuple, + expected_area: int, +) -> None: + # given + from inference_models.models.yolo26.yolo26_instance_segmentation_onnx import ( + YOLO26ForInstanceSegmentationOnnx, + ) + + model = YOLO26ForInstanceSegmentationOnnx.from_pretrained( + model_name_or_path=yolo26n_seg_snakes_stretch_onnx_static_package, + onnx_execution_providers=["CPUExecutionProvider"], + device=torch.device("cpu"), + ) + image = np.pad( + snake_image_numpy, + ( + (0, snake_image_numpy.shape[0] % 2), + (0, snake_image_numpy.shape[1] % 2), + (0, 0), + ), + ) + height, width = image.shape[:2] + scene = image + offset_x = offset_y = 0 + crop_config = None + if crop_start is not None: + offset_y = 0 if crop_start == 0 else height // 2 + offset_x = 0 if crop_start == 0 else width // 2 + scene = np.zeros((2 * height, 2 * width, 3), dtype=image.dtype) + scene[offset_y : offset_y + height, offset_x : offset_x + width] = image + crop_config = StaticCrop( + enabled=True, + x_min=crop_start, + y_min=crop_start, + x_max=crop_start + 50, + y_max=crop_start + 50, + ) + monkeypatch.setattr( + model._inference_config.image_pre_processing, "static_crop", crop_config + ) + + # when + dense = model( + scene, confidence=0.5, mask_format="dense", masks_resolution_factor=factor + )[0] + rle = model( + scene, confidence=0.5, mask_format="rle", masks_resolution_factor=factor + )[0] + decoded = coco_rle_masks_to_torch_mask( + instances_masks=rle.mask, device=torch.device("cpu") + ) + detections = dense.to_supervision() + + # then + offset = np.array([offset_x, offset_y, offset_x, offset_y]) + expected_boxes = np.array([[126, 326, 1263, 558]]) + offset + np.testing.assert_allclose(detections.xyxy, expected_boxes, atol=2, rtol=0) + np.testing.assert_array_equal(detections.class_id, [0]) + np.testing.assert_allclose(detections.confidence, [0.9657], atol=0.01, rtol=0) + assert dense.image_size == scene.shape[:2] + assert dense.mask_frame_size == scene.shape[:2] + assert rle.image_size == scene.shape[:2] + assert rle.mask_frame_size == scene.shape[:2] + canvas_multiplier = 1 if crop_start is None else 2 + assert dense.mask_size == tuple(size * canvas_multiplier for size in expected_grid) + assert rle.mask_size == dense.mask_size + assert detections.mask.shape == (1, *scene.shape[:2]) + # Areas were checked against overlays of this fixture at all three factors. + assert detections.mask.sum() == pytest.approx(expected_area, rel=0.02) + expected_mask_boxes = np.array([[120, 322, 1259, 559]]) + offset + pixel_tolerance = ( + np.ceil( + max( + scene.shape[0] / dense.mask_size[0], scene.shape[1] / dense.mask_size[1] + ) + ) + + 2 + ) + np.testing.assert_allclose( + sv.mask_to_xyxy(detections.mask), + expected_mask_boxes, + atol=pixel_tolerance, + rtol=0, + ) + crop_masks = detections.mask[ + :, offset_y : offset_y + height, offset_x : offset_x + width + ] + assert crop_masks.sum() == detections.mask.sum() + torch.testing.assert_close(dense.xyxy.cpu(), rle.xyxy.cpu()) + torch.testing.assert_close(dense.class_id.cpu(), rle.class_id.cpu()) + torch.testing.assert_close(dense.confidence.cpu(), rle.confidence.cpu()) + np.testing.assert_array_equal(decoded.numpy(), dense.mask.cpu().numpy()) + np.testing.assert_array_equal(detections.mask, rle.to_supervision().mask) + sv.MaskAnnotator().annotate(scene.copy(), detections) diff --git a/inference_models/tests/integration_tests/models/test_yolo26_instance_segmentation_predictions_torch.py b/inference_models/tests/integration_tests/models/test_yolo26_instance_segmentation_predictions_torch.py index 6fffac9a99..c1e2eb4eec 100644 --- a/inference_models/tests/integration_tests/models/test_yolo26_instance_segmentation_predictions_torch.py +++ b/inference_models/tests/integration_tests/models/test_yolo26_instance_segmentation_predictions_torch.py @@ -1,9 +1,11 @@ import numpy as np import pytest +import supervision as sv import torch from inference_models.configuration import DEFAULT_DEVICE from inference_models.models.common.rle_utils import coco_rle_masks_to_torch_mask +from inference_models.models.common.roboflow.model_packages import StaticCrop CONFIDENCE_ATOL = 0.01 XYXY_ATOL = 2 @@ -471,3 +473,112 @@ def test_torchscript_per_class_confidence_filters_detections( ) predictions = model(snake_image_numpy, confidence="best") assert predictions[0].class_id.numel() == 0 + + +@pytest.mark.slow +@pytest.mark.torch_models +@pytest.mark.parametrize( + "factor,expected_grid,expected_area", + [(1.0, (890, 1280), 211140), (0.5, (501, 696), 211025), (0.0, (112, 112), 208787)], +) +@pytest.mark.parametrize( + "crop_start", [None, 0, 25], ids=["uncropped", "origin-crop", "offset-crop"] +) +def test_mask_resolution_and_crop_round_trip( + yolo26n_seg_snakes_stretch_torch_script_package: str, + snake_image_numpy: np.ndarray, + monkeypatch: pytest.MonkeyPatch, + factor: float, + crop_start: int | None, + expected_grid: tuple, + expected_area: int, +) -> None: + # given + from inference_models.models.yolo26.yolo26_instance_segmentation_torch_script import ( + YOLO26ForInstanceSegmentationTorchScript, + ) + + model = YOLO26ForInstanceSegmentationTorchScript.from_pretrained( + model_name_or_path=yolo26n_seg_snakes_stretch_torch_script_package, + ) + image = np.pad( + snake_image_numpy, + ( + (0, snake_image_numpy.shape[0] % 2), + (0, snake_image_numpy.shape[1] % 2), + (0, 0), + ), + ) + height, width = image.shape[:2] + scene = image + offset_x = offset_y = 0 + crop_config = None + if crop_start is not None: + offset_y = 0 if crop_start == 0 else height // 2 + offset_x = 0 if crop_start == 0 else width // 2 + scene = np.zeros((2 * height, 2 * width, 3), dtype=image.dtype) + scene[offset_y : offset_y + height, offset_x : offset_x + width] = image + crop_config = StaticCrop( + enabled=True, + x_min=crop_start, + y_min=crop_start, + x_max=crop_start + 50, + y_max=crop_start + 50, + ) + monkeypatch.setattr( + model._inference_config.image_pre_processing, "static_crop", crop_config + ) + + # when + dense = model( + scene, confidence=0.5, mask_format="dense", masks_resolution_factor=factor + )[0] + rle = model( + scene, confidence=0.5, mask_format="rle", masks_resolution_factor=factor + )[0] + decoded = coco_rle_masks_to_torch_mask( + instances_masks=rle.mask, device=torch.device("cpu") + ) + detections = dense.to_supervision() + + # then + offset = np.array([offset_x, offset_y, offset_x, offset_y]) + expected_boxes = np.array([[126, 326, 1263, 558]]) + offset + np.testing.assert_allclose(detections.xyxy, expected_boxes, atol=2, rtol=0) + np.testing.assert_array_equal(detections.class_id, [0]) + np.testing.assert_allclose(detections.confidence, [0.9657], atol=0.01, rtol=0) + assert dense.image_size == scene.shape[:2] + assert dense.mask_frame_size == scene.shape[:2] + assert rle.image_size == scene.shape[:2] + assert rle.mask_frame_size == scene.shape[:2] + canvas_multiplier = 1 if crop_start is None else 2 + assert dense.mask_size == tuple(size * canvas_multiplier for size in expected_grid) + assert rle.mask_size == dense.mask_size + assert detections.mask.shape == (1, *scene.shape[:2]) + # Areas were checked against overlays of this fixture at all three factors. + assert detections.mask.sum() == pytest.approx(expected_area, rel=0.02) + expected_mask_boxes = np.array([[120, 322, 1259, 559]]) + offset + pixel_tolerance = ( + np.ceil( + max( + scene.shape[0] / dense.mask_size[0], scene.shape[1] / dense.mask_size[1] + ) + ) + + 2 + ) + np.testing.assert_allclose( + sv.mask_to_xyxy(detections.mask), + expected_mask_boxes, + atol=pixel_tolerance, + rtol=0, + ) + crop_masks = detections.mask[ + :, offset_y : offset_y + height, offset_x : offset_x + width + ] + assert crop_masks.sum() == detections.mask.sum() + torch.testing.assert_close(dense.xyxy.cpu(), rle.xyxy.cpu()) + torch.testing.assert_close(dense.class_id.cpu(), rle.class_id.cpu()) + torch.testing.assert_close(dense.confidence.cpu(), rle.confidence.cpu()) + np.testing.assert_array_equal(decoded.numpy(), dense.mask.cpu().numpy()) + np.testing.assert_array_equal(detections.mask, rle.to_supervision().mask) + sv.MaskAnnotator().annotate(scene.copy(), detections) diff --git a/inference_models/tests/integration_tests/models/test_yolo26_instance_segmentation_predictions_trt.py b/inference_models/tests/integration_tests/models/test_yolo26_instance_segmentation_predictions_trt.py index ceb5df9cd3..6e7b71f104 100644 --- a/inference_models/tests/integration_tests/models/test_yolo26_instance_segmentation_predictions_trt.py +++ b/inference_models/tests/integration_tests/models/test_yolo26_instance_segmentation_predictions_trt.py @@ -1,8 +1,10 @@ import numpy as np import pytest +import supervision as sv import torch from inference_models.models.common.rle_utils import coco_rle_masks_to_torch_mask +from inference_models.models.common.roboflow.model_packages import StaticCrop @pytest.mark.slow @@ -452,3 +454,93 @@ def test_trt_per_class_confidence_filters_detections( ) predictions = model(asl_image_numpy, confidence="best") assert predictions[0].class_id.numel() == 0 + + +@pytest.mark.slow +@pytest.mark.trt_extras +@pytest.mark.gpu_only +@pytest.mark.parametrize("factor", [1.0, 0.5, 0.0]) +@pytest.mark.parametrize( + "crop_start", [None, 0, 25], ids=["uncropped", "origin-crop", "offset-crop"] +) +def test_mask_resolution_and_crop_round_trip( + yolo26_seg_asl_trt_package: str, + asl_image_numpy: np.ndarray, + monkeypatch: pytest.MonkeyPatch, + factor: float, + crop_start: int | None, +) -> None: + # given + from inference_models.models.yolo26.yolo26_instance_segmentation_trt import ( + YOLO26ForInstanceSegmentationTRT, + ) + + model = YOLO26ForInstanceSegmentationTRT.from_pretrained( + model_name_or_path=yolo26_seg_asl_trt_package, + engine_host_code_allowed=True, + ) + image = np.pad( + asl_image_numpy, + ((0, asl_image_numpy.shape[0] % 2), (0, asl_image_numpy.shape[1] % 2), (0, 0)), + ) + height, width = image.shape[:2] + scene = image + offset_x = offset_y = 0 + crop_config = None + if crop_start is not None: + offset_y = 0 if crop_start == 0 else height // 2 + offset_x = 0 if crop_start == 0 else width // 2 + scene = np.zeros((2 * height, 2 * width, 3), dtype=image.dtype) + scene[offset_y : offset_y + height, offset_x : offset_x + width] = image + crop_config = StaticCrop( + enabled=True, + x_min=crop_start, + y_min=crop_start, + x_max=crop_start + 50, + y_max=crop_start + 50, + ) + monkeypatch.setattr( + model._inference_config.image_pre_processing, "static_crop", crop_config + ) + + # when + dense = model( + scene, confidence=0.25, mask_format="dense", masks_resolution_factor=factor + )[0] + rle = model( + scene, confidence=0.25, mask_format="rle", masks_resolution_factor=factor + )[0] + decoded = coco_rle_masks_to_torch_mask( + instances_masks=rle.mask, device=torch.device("cpu") + ) + detections = dense.to_supervision() + + # then + offset = np.array([offset_x, offset_y, offset_x, offset_y]) + expected_boxes = np.array([[63, 174, 186, 368]]) + offset + np.testing.assert_allclose(detections.xyxy, expected_boxes, atol=5, rtol=0) + np.testing.assert_array_equal(detections.class_id, [20]) + np.testing.assert_allclose(detections.confidence, [0.9671], atol=0.01, rtol=0) + assert dense.image_size == scene.shape[:2] + assert dense.mask_frame_size == scene.shape[:2] + assert rle.image_size == scene.shape[:2] + assert rle.mask_frame_size == scene.shape[:2] + if factor == 1.0: + assert dense.mask_size == scene.shape[:2] + else: + assert dense.mask_size != scene.shape[:2] + assert rle.mask_size == dense.mask_size + assert detections.mask.shape == (1, *scene.shape[:2]) + # Reuse the full-resolution area baseline in test_trt_package_numpy. + expected_area = 16400 + assert detections.mask.sum() == pytest.approx(expected_area, rel=0.05) + crop_masks = detections.mask[ + :, offset_y : offset_y + height, offset_x : offset_x + width + ] + assert crop_masks.sum() == detections.mask.sum() + torch.testing.assert_close(dense.xyxy.cpu(), rle.xyxy.cpu()) + torch.testing.assert_close(dense.class_id.cpu(), rle.class_id.cpu()) + torch.testing.assert_close(dense.confidence.cpu(), rle.confidence.cpu()) + np.testing.assert_array_equal(decoded.numpy(), dense.mask.cpu().numpy()) + np.testing.assert_array_equal(detections.mask, rle.to_supervision().mask) + sv.MaskAnnotator().annotate(scene.copy(), detections) diff --git a/inference_models/tests/integration_tests/models/test_yolov5_instance_segmentation_predictions_onnx.py b/inference_models/tests/integration_tests/models/test_yolov5_instance_segmentation_predictions_onnx.py index 6d7580121e..55f1886f76 100644 --- a/inference_models/tests/integration_tests/models/test_yolov5_instance_segmentation_predictions_onnx.py +++ b/inference_models/tests/integration_tests/models/test_yolov5_instance_segmentation_predictions_onnx.py @@ -1,8 +1,10 @@ import numpy as np import pytest +import supervision as sv import torch from inference_models.models.common.rle_utils import coco_rle_masks_to_torch_mask +from inference_models.models.common.roboflow.model_packages import StaticCrop @pytest.mark.slow @@ -331,3 +333,110 @@ def test_onnx_per_class_confidence_blocks_specific_class( ) predictions = model(asl_image_numpy, confidence="best") assert predictions[0].class_id.numel() == 0 + + +@pytest.mark.slow +@pytest.mark.onnx_extras +@pytest.mark.parametrize( + "factor,expected_grid,expected_area", + [(1.0, (416, 416), 16123), (0.5, (280, 280), 16106), (0.0, (144, 144), 15961)], +) +@pytest.mark.parametrize( + "crop_start", [None, 0, 25], ids=["uncropped", "origin-crop", "offset-crop"] +) +def test_mask_resolution_and_crop_round_trip( + asl_yolov5_onnx_seg_static_bs_letterbox: str, + asl_image_numpy: np.ndarray, + monkeypatch: pytest.MonkeyPatch, + factor: float, + crop_start: int | None, + expected_grid: tuple, + expected_area: int, +) -> None: + # given + from inference_models.models.yolov5.yolov5_instance_segmentation_onnx import ( + YOLOv5ForInstanceSegmentationOnnx, + ) + + model = YOLOv5ForInstanceSegmentationOnnx.from_pretrained( + model_name_or_path=asl_yolov5_onnx_seg_static_bs_letterbox, + onnx_execution_providers=["CPUExecutionProvider"], + device=torch.device("cpu"), + ) + image = np.pad( + asl_image_numpy, + ((0, asl_image_numpy.shape[0] % 2), (0, asl_image_numpy.shape[1] % 2), (0, 0)), + ) + height, width = image.shape[:2] + scene = image + offset_x = offset_y = 0 + crop_config = None + if crop_start is not None: + offset_y = 0 if crop_start == 0 else height // 2 + offset_x = 0 if crop_start == 0 else width // 2 + scene = np.zeros((2 * height, 2 * width, 3), dtype=image.dtype) + scene[offset_y : offset_y + height, offset_x : offset_x + width] = image + crop_config = StaticCrop( + enabled=True, + x_min=crop_start, + y_min=crop_start, + x_max=crop_start + 50, + y_max=crop_start + 50, + ) + monkeypatch.setattr( + model._inference_config.image_pre_processing, "static_crop", crop_config + ) + + # when + dense = model( + scene, confidence=0.25, mask_format="dense", masks_resolution_factor=factor + )[0] + rle = model( + scene, confidence=0.25, mask_format="rle", masks_resolution_factor=factor + )[0] + decoded = coco_rle_masks_to_torch_mask( + instances_masks=rle.mask, device=torch.device("cpu") + ) + detections = dense.to_supervision() + + # then + offset = np.array([offset_x, offset_y, offset_x, offset_y]) + expected_boxes = np.array([[61, 174, 188, 370]]) + offset + np.testing.assert_allclose(detections.xyxy, expected_boxes, atol=2, rtol=0) + np.testing.assert_array_equal(detections.class_id, [21]) + np.testing.assert_allclose(detections.confidence, [0.9928], atol=0.01, rtol=0) + assert dense.image_size == scene.shape[:2] + assert dense.mask_frame_size == scene.shape[:2] + assert rle.image_size == scene.shape[:2] + assert rle.mask_frame_size == scene.shape[:2] + canvas_multiplier = 1 if crop_start is None else 2 + assert dense.mask_size == tuple(size * canvas_multiplier for size in expected_grid) + assert rle.mask_size == dense.mask_size + assert detections.mask.shape == (1, *scene.shape[:2]) + # Areas were checked against overlays of this fixture at all three factors. + assert detections.mask.sum() == pytest.approx(expected_area, rel=0.02) + expected_mask_boxes = np.array([[59, 172, 188, 367]]) + offset + pixel_tolerance = ( + np.ceil( + max( + scene.shape[0] / dense.mask_size[0], scene.shape[1] / dense.mask_size[1] + ) + ) + + 2 + ) + np.testing.assert_allclose( + sv.mask_to_xyxy(detections.mask), + expected_mask_boxes, + atol=pixel_tolerance, + rtol=0, + ) + crop_masks = detections.mask[ + :, offset_y : offset_y + height, offset_x : offset_x + width + ] + assert crop_masks.sum() == detections.mask.sum() + torch.testing.assert_close(dense.xyxy.cpu(), rle.xyxy.cpu()) + torch.testing.assert_close(dense.class_id.cpu(), rle.class_id.cpu()) + torch.testing.assert_close(dense.confidence.cpu(), rle.confidence.cpu()) + np.testing.assert_array_equal(decoded.numpy(), dense.mask.cpu().numpy()) + np.testing.assert_array_equal(detections.mask, rle.to_supervision().mask) + sv.MaskAnnotator().annotate(scene.copy(), detections) diff --git a/inference_models/tests/integration_tests/models/test_yolov7_instance_segmentation_predictions_onnx.py b/inference_models/tests/integration_tests/models/test_yolov7_instance_segmentation_predictions_onnx.py index 9f980fc12c..95cd3f4729 100644 --- a/inference_models/tests/integration_tests/models/test_yolov7_instance_segmentation_predictions_onnx.py +++ b/inference_models/tests/integration_tests/models/test_yolov7_instance_segmentation_predictions_onnx.py @@ -1,8 +1,10 @@ import numpy as np import pytest +import supervision as sv import torch from inference_models.models.common.rle_utils import coco_rle_masks_to_torch_mask +from inference_models.models.common.roboflow.model_packages import StaticCrop @pytest.mark.slow @@ -333,3 +335,110 @@ def test_onnx_per_class_confidence_preserves_objectness_row( predictions = model(asl_image_numpy, confidence="best") assert np.allclose(predictions[0].class_id[0].cpu().numpy(), [0], atol=1) assert np.allclose(predictions[0].confidence[0].cpu().numpy(), [0.7349], atol=0.005) + + +@pytest.mark.slow +@pytest.mark.onnx_extras +@pytest.mark.parametrize( + "factor,expected_grid,expected_area", + [(1.0, (416, 416), 16757), (0.5, (288, 288), 16797), (0.0, (160, 160), 16693)], +) +@pytest.mark.parametrize( + "crop_start", [None, 0, 25], ids=["uncropped", "origin-crop", "offset-crop"] +) +def test_mask_resolution_and_crop_round_trip( + asl_yolov7_onnx_seg_static_bs_letterbox: str, + asl_image_numpy: np.ndarray, + monkeypatch: pytest.MonkeyPatch, + factor: float, + crop_start: int | None, + expected_grid: tuple, + expected_area: int, +) -> None: + # given + from inference_models.models.yolov7.yolov7_instance_segmentation_onnx import ( + YOLOv7ForInstanceSegmentationOnnx, + ) + + model = YOLOv7ForInstanceSegmentationOnnx.from_pretrained( + model_name_or_path=asl_yolov7_onnx_seg_static_bs_letterbox, + onnx_execution_providers=["CPUExecutionProvider"], + device=torch.device("cpu"), + ) + image = np.pad( + asl_image_numpy, + ((0, asl_image_numpy.shape[0] % 2), (0, asl_image_numpy.shape[1] % 2), (0, 0)), + ) + height, width = image.shape[:2] + scene = image + offset_x = offset_y = 0 + crop_config = None + if crop_start is not None: + offset_y = 0 if crop_start == 0 else height // 2 + offset_x = 0 if crop_start == 0 else width // 2 + scene = np.zeros((2 * height, 2 * width, 3), dtype=image.dtype) + scene[offset_y : offset_y + height, offset_x : offset_x + width] = image + crop_config = StaticCrop( + enabled=True, + x_min=crop_start, + y_min=crop_start, + x_max=crop_start + 50, + y_max=crop_start + 50, + ) + monkeypatch.setattr( + model._inference_config.image_pre_processing, "static_crop", crop_config + ) + + # when + dense = model( + scene, confidence=0.6, mask_format="dense", masks_resolution_factor=factor + )[0] + rle = model( + scene, confidence=0.6, mask_format="rle", masks_resolution_factor=factor + )[0] + decoded = coco_rle_masks_to_torch_mask( + instances_masks=rle.mask, device=torch.device("cpu") + ) + detections = dense.to_supervision() + + # then + offset = np.array([offset_x, offset_y, offset_x, offset_y]) + expected_boxes = np.array([[59, 162, 192, 387]]) + offset + np.testing.assert_allclose(detections.xyxy, expected_boxes, atol=2, rtol=0) + np.testing.assert_array_equal(detections.class_id, [0]) + np.testing.assert_allclose(detections.confidence, [0.7313], atol=0.01, rtol=0) + assert dense.image_size == scene.shape[:2] + assert dense.mask_frame_size == scene.shape[:2] + assert rle.image_size == scene.shape[:2] + assert rle.mask_frame_size == scene.shape[:2] + canvas_multiplier = 1 if crop_start is None else 2 + assert dense.mask_size == tuple(size * canvas_multiplier for size in expected_grid) + assert rle.mask_size == dense.mask_size + assert detections.mask.shape == (1, *scene.shape[:2]) + # Areas were checked against overlays of this fixture at all three factors. + assert detections.mask.sum() == pytest.approx(expected_area, rel=0.02) + expected_mask_boxes = np.array([[59, 168, 187, 379]]) + offset + pixel_tolerance = ( + np.ceil( + max( + scene.shape[0] / dense.mask_size[0], scene.shape[1] / dense.mask_size[1] + ) + ) + + 2 + ) + np.testing.assert_allclose( + sv.mask_to_xyxy(detections.mask), + expected_mask_boxes, + atol=pixel_tolerance, + rtol=0, + ) + crop_masks = detections.mask[ + :, offset_y : offset_y + height, offset_x : offset_x + width + ] + assert crop_masks.sum() == detections.mask.sum() + torch.testing.assert_close(dense.xyxy.cpu(), rle.xyxy.cpu()) + torch.testing.assert_close(dense.class_id.cpu(), rle.class_id.cpu()) + torch.testing.assert_close(dense.confidence.cpu(), rle.confidence.cpu()) + np.testing.assert_array_equal(decoded.numpy(), dense.mask.cpu().numpy()) + np.testing.assert_array_equal(detections.mask, rle.to_supervision().mask) + sv.MaskAnnotator().annotate(scene.copy(), detections) diff --git a/inference_models/tests/integration_tests/models/test_yolov8_instance_segmentation_predictions_onnx.py b/inference_models/tests/integration_tests/models/test_yolov8_instance_segmentation_predictions_onnx.py index efac1ec6ae..e006ac0fc4 100644 --- a/inference_models/tests/integration_tests/models/test_yolov8_instance_segmentation_predictions_onnx.py +++ b/inference_models/tests/integration_tests/models/test_yolov8_instance_segmentation_predictions_onnx.py @@ -1,8 +1,10 @@ import numpy as np import pytest +import supervision as sv import torch from inference_models.models.common.rle_utils import coco_rle_masks_to_torch_mask +from inference_models.models.common.roboflow.model_packages import StaticCrop @pytest.mark.slow @@ -3744,3 +3746,110 @@ def test_onnx_per_class_confidence_blocks_specific_class( ) predictions = model(asl_image_numpy, confidence="best") assert predictions[0].class_id.numel() == 0 + + +@pytest.mark.slow +@pytest.mark.onnx_extras +@pytest.mark.parametrize( + "factor,expected_grid,expected_area", + [(1.0, (416, 416), 16069), (0.5, (288, 288), 16040), (0.0, (160, 160), 16063)], +) +@pytest.mark.parametrize( + "crop_start", [None, 0, 25], ids=["uncropped", "origin-crop", "offset-crop"] +) +def test_mask_resolution_and_crop_round_trip( + asl_yolov8n_onnx_seg_static_bs_stretch: str, + asl_image_numpy: np.ndarray, + monkeypatch: pytest.MonkeyPatch, + factor: float, + crop_start: int | None, + expected_grid: tuple, + expected_area: int, +) -> None: + # given + from inference_models.models.yolov8.yolov8_instance_segmentation_onnx import ( + YOLOv8ForInstanceSegmentationOnnx, + ) + + model = YOLOv8ForInstanceSegmentationOnnx.from_pretrained( + model_name_or_path=asl_yolov8n_onnx_seg_static_bs_stretch, + onnx_execution_providers=["CPUExecutionProvider"], + device=torch.device("cpu"), + ) + image = np.pad( + asl_image_numpy, + ((0, asl_image_numpy.shape[0] % 2), (0, asl_image_numpy.shape[1] % 2), (0, 0)), + ) + height, width = image.shape[:2] + scene = image + offset_x = offset_y = 0 + crop_config = None + if crop_start is not None: + offset_y = 0 if crop_start == 0 else height // 2 + offset_x = 0 if crop_start == 0 else width // 2 + scene = np.zeros((2 * height, 2 * width, 3), dtype=image.dtype) + scene[offset_y : offset_y + height, offset_x : offset_x + width] = image + crop_config = StaticCrop( + enabled=True, + x_min=crop_start, + y_min=crop_start, + x_max=crop_start + 50, + y_max=crop_start + 50, + ) + monkeypatch.setattr( + model._inference_config.image_pre_processing, "static_crop", crop_config + ) + + # when + dense = model( + scene, confidence=0.25, mask_format="dense", masks_resolution_factor=factor + )[0] + rle = model( + scene, confidence=0.25, mask_format="rle", masks_resolution_factor=factor + )[0] + decoded = coco_rle_masks_to_torch_mask( + instances_masks=rle.mask, device=torch.device("cpu") + ) + detections = dense.to_supervision() + + # then + offset = np.array([offset_x, offset_y, offset_x, offset_y]) + expected_boxes = np.array([[63, 174, 187, 371]]) + offset + np.testing.assert_allclose(detections.xyxy, expected_boxes, atol=2, rtol=0) + np.testing.assert_array_equal(detections.class_id, [20]) + np.testing.assert_allclose(detections.confidence, [0.9846], atol=0.01, rtol=0) + assert dense.image_size == scene.shape[:2] + assert dense.mask_frame_size == scene.shape[:2] + assert rle.image_size == scene.shape[:2] + assert rle.mask_frame_size == scene.shape[:2] + canvas_multiplier = 1 if crop_start is None else 2 + assert dense.mask_size == tuple(size * canvas_multiplier for size in expected_grid) + assert rle.mask_size == dense.mask_size + assert detections.mask.shape == (1, *scene.shape[:2]) + # Areas were checked against overlays of this fixture at all three factors. + assert detections.mask.sum() == pytest.approx(expected_area, rel=0.02) + expected_mask_boxes = np.array([[63, 174, 186, 369]]) + offset + pixel_tolerance = ( + np.ceil( + max( + scene.shape[0] / dense.mask_size[0], scene.shape[1] / dense.mask_size[1] + ) + ) + + 2 + ) + np.testing.assert_allclose( + sv.mask_to_xyxy(detections.mask), + expected_mask_boxes, + atol=pixel_tolerance, + rtol=0, + ) + crop_masks = detections.mask[ + :, offset_y : offset_y + height, offset_x : offset_x + width + ] + assert crop_masks.sum() == detections.mask.sum() + torch.testing.assert_close(dense.xyxy.cpu(), rle.xyxy.cpu()) + torch.testing.assert_close(dense.class_id.cpu(), rle.class_id.cpu()) + torch.testing.assert_close(dense.confidence.cpu(), rle.confidence.cpu()) + np.testing.assert_array_equal(decoded.numpy(), dense.mask.cpu().numpy()) + np.testing.assert_array_equal(detections.mask, rle.to_supervision().mask) + sv.MaskAnnotator().annotate(scene.copy(), detections) diff --git a/inference_models/tests/integration_tests/models/test_yolov8_instance_segmentation_predictions_torch.py b/inference_models/tests/integration_tests/models/test_yolov8_instance_segmentation_predictions_torch.py index 27890919e2..64e0efb966 100644 --- a/inference_models/tests/integration_tests/models/test_yolov8_instance_segmentation_predictions_torch.py +++ b/inference_models/tests/integration_tests/models/test_yolov8_instance_segmentation_predictions_torch.py @@ -1,9 +1,11 @@ import numpy as np import pytest +import supervision as sv import torch from inference_models.configuration import DEFAULT_DEVICE from inference_models.models.common.rle_utils import coco_rle_masks_to_torch_mask +from inference_models.models.common.roboflow.model_packages import StaticCrop from inference_models.models.yolov8.yolov8_instance_segmentation_torch_script import ( YOLOv8ForInstanceSegmentationTorchScript, ) @@ -2524,3 +2526,108 @@ def test_torchscript_per_class_confidence_blocks_specific_class( ) predictions = model(asl_image_numpy, confidence="best") assert predictions[0].class_id.numel() == 0 + + +@pytest.mark.slow +@pytest.mark.torch_models +@pytest.mark.parametrize( + "factor,expected_grid,expected_area", + [(1.0, (416, 416), 16069), (0.5, (288, 288), 16040), (0.0, (160, 160), 16063)], +) +@pytest.mark.parametrize( + "crop_start", [None, 0, 25], ids=["uncropped", "origin-crop", "offset-crop"] +) +def test_mask_resolution_and_crop_round_trip( + asl_yolov8n_torchscript_seg_static_bs_stretch: str, + asl_image_numpy: np.ndarray, + monkeypatch: pytest.MonkeyPatch, + factor: float, + crop_start: int | None, + expected_grid: tuple, + expected_area: int, +) -> None: + # given + from inference_models.models.yolov8.yolov8_instance_segmentation_torch_script import ( + YOLOv8ForInstanceSegmentationTorchScript, + ) + + model = YOLOv8ForInstanceSegmentationTorchScript.from_pretrained( + model_name_or_path=asl_yolov8n_torchscript_seg_static_bs_stretch, + ) + image = np.pad( + asl_image_numpy, + ((0, asl_image_numpy.shape[0] % 2), (0, asl_image_numpy.shape[1] % 2), (0, 0)), + ) + height, width = image.shape[:2] + scene = image + offset_x = offset_y = 0 + crop_config = None + if crop_start is not None: + offset_y = 0 if crop_start == 0 else height // 2 + offset_x = 0 if crop_start == 0 else width // 2 + scene = np.zeros((2 * height, 2 * width, 3), dtype=image.dtype) + scene[offset_y : offset_y + height, offset_x : offset_x + width] = image + crop_config = StaticCrop( + enabled=True, + x_min=crop_start, + y_min=crop_start, + x_max=crop_start + 50, + y_max=crop_start + 50, + ) + monkeypatch.setattr( + model._inference_config.image_pre_processing, "static_crop", crop_config + ) + + # when + dense = model( + scene, confidence=0.25, mask_format="dense", masks_resolution_factor=factor + )[0] + rle = model( + scene, confidence=0.25, mask_format="rle", masks_resolution_factor=factor + )[0] + decoded = coco_rle_masks_to_torch_mask( + instances_masks=rle.mask, device=torch.device("cpu") + ) + detections = dense.to_supervision() + + # then + offset = np.array([offset_x, offset_y, offset_x, offset_y]) + expected_boxes = np.array([[63, 174, 187, 371]]) + offset + np.testing.assert_allclose(detections.xyxy, expected_boxes, atol=2, rtol=0) + np.testing.assert_array_equal(detections.class_id, [20]) + np.testing.assert_allclose(detections.confidence, [0.9846], atol=0.01, rtol=0) + assert dense.image_size == scene.shape[:2] + assert dense.mask_frame_size == scene.shape[:2] + assert rle.image_size == scene.shape[:2] + assert rle.mask_frame_size == scene.shape[:2] + canvas_multiplier = 1 if crop_start is None else 2 + assert dense.mask_size == tuple(size * canvas_multiplier for size in expected_grid) + assert rle.mask_size == dense.mask_size + assert detections.mask.shape == (1, *scene.shape[:2]) + # Areas were checked against overlays of this fixture at all three factors. + assert detections.mask.sum() == pytest.approx(expected_area, rel=0.02) + expected_mask_boxes = np.array([[63, 174, 186, 369]]) + offset + pixel_tolerance = ( + np.ceil( + max( + scene.shape[0] / dense.mask_size[0], scene.shape[1] / dense.mask_size[1] + ) + ) + + 2 + ) + np.testing.assert_allclose( + sv.mask_to_xyxy(detections.mask), + expected_mask_boxes, + atol=pixel_tolerance, + rtol=0, + ) + crop_masks = detections.mask[ + :, offset_y : offset_y + height, offset_x : offset_x + width + ] + assert crop_masks.sum() == detections.mask.sum() + torch.testing.assert_close(dense.xyxy.cpu(), rle.xyxy.cpu()) + torch.testing.assert_close(dense.class_id.cpu(), rle.class_id.cpu()) + torch.testing.assert_close(dense.confidence.cpu(), rle.confidence.cpu()) + np.testing.assert_array_equal(decoded.numpy(), dense.mask.cpu().numpy()) + np.testing.assert_array_equal(detections.mask, rle.to_supervision().mask) + sv.MaskAnnotator().annotate(scene.copy(), detections) diff --git a/inference_models/tests/integration_tests/models/test_yolov8_instance_segmentation_predictions_trt.py b/inference_models/tests/integration_tests/models/test_yolov8_instance_segmentation_predictions_trt.py index 53d3f60c0f..77f6acec67 100644 --- a/inference_models/tests/integration_tests/models/test_yolov8_instance_segmentation_predictions_trt.py +++ b/inference_models/tests/integration_tests/models/test_yolov8_instance_segmentation_predictions_trt.py @@ -1,8 +1,10 @@ import numpy as np import pytest +import supervision as sv import torch from inference_models.models.common.rle_utils import coco_rle_masks_to_torch_mask +from inference_models.models.common.roboflow.model_packages import StaticCrop @pytest.mark.slow @@ -455,3 +457,93 @@ class 20. Setting a 1.01 per-class threshold on class 20 leaves no ) predictions = model(asl_image_numpy, confidence="best") assert predictions[0].class_id.numel() == 0 + + +@pytest.mark.slow +@pytest.mark.trt_extras +@pytest.mark.gpu_only +@pytest.mark.parametrize("factor", [1.0, 0.5, 0.0]) +@pytest.mark.parametrize( + "crop_start", [None, 0, 25], ids=["uncropped", "origin-crop", "offset-crop"] +) +def test_mask_resolution_and_crop_round_trip( + yolov8_seg_asl_trt_package: str, + asl_image_numpy: np.ndarray, + monkeypatch: pytest.MonkeyPatch, + factor: float, + crop_start: int | None, +) -> None: + # given + from inference_models.models.yolov8.yolov8_instance_segmentation_trt import ( + YOLOv8ForInstanceSegmentationTRT, + ) + + model = YOLOv8ForInstanceSegmentationTRT.from_pretrained( + model_name_or_path=yolov8_seg_asl_trt_package, + engine_host_code_allowed=True, + ) + image = np.pad( + asl_image_numpy, + ((0, asl_image_numpy.shape[0] % 2), (0, asl_image_numpy.shape[1] % 2), (0, 0)), + ) + height, width = image.shape[:2] + scene = image + offset_x = offset_y = 0 + crop_config = None + if crop_start is not None: + offset_y = 0 if crop_start == 0 else height // 2 + offset_x = 0 if crop_start == 0 else width // 2 + scene = np.zeros((2 * height, 2 * width, 3), dtype=image.dtype) + scene[offset_y : offset_y + height, offset_x : offset_x + width] = image + crop_config = StaticCrop( + enabled=True, + x_min=crop_start, + y_min=crop_start, + x_max=crop_start + 50, + y_max=crop_start + 50, + ) + monkeypatch.setattr( + model._inference_config.image_pre_processing, "static_crop", crop_config + ) + + # when + dense = model( + scene, confidence=0.25, mask_format="dense", masks_resolution_factor=factor + )[0] + rle = model( + scene, confidence=0.25, mask_format="rle", masks_resolution_factor=factor + )[0] + decoded = coco_rle_masks_to_torch_mask( + instances_masks=rle.mask, device=torch.device("cpu") + ) + detections = dense.to_supervision() + + # then + offset = np.array([offset_x, offset_y, offset_x, offset_y]) + expected_boxes = np.array([[63, 174, 187, 368]]) + offset + np.testing.assert_allclose(detections.xyxy, expected_boxes, atol=5, rtol=0) + np.testing.assert_array_equal(detections.class_id, [20]) + np.testing.assert_allclose(detections.confidence, [0.9795], atol=0.01, rtol=0) + assert dense.image_size == scene.shape[:2] + assert dense.mask_frame_size == scene.shape[:2] + assert rle.image_size == scene.shape[:2] + assert rle.mask_frame_size == scene.shape[:2] + if factor == 1.0: + assert dense.mask_size == scene.shape[:2] + else: + assert dense.mask_size != scene.shape[:2] + assert rle.mask_size == dense.mask_size + assert detections.mask.shape == (1, *scene.shape[:2]) + # Reuse the full-resolution area baseline in test_trt_package_numpy. + expected_area = 16100 + assert detections.mask.sum() == pytest.approx(expected_area, rel=0.05) + crop_masks = detections.mask[ + :, offset_y : offset_y + height, offset_x : offset_x + width + ] + assert crop_masks.sum() == detections.mask.sum() + torch.testing.assert_close(dense.xyxy.cpu(), rle.xyxy.cpu()) + torch.testing.assert_close(dense.class_id.cpu(), rle.class_id.cpu()) + torch.testing.assert_close(dense.confidence.cpu(), rle.confidence.cpu()) + np.testing.assert_array_equal(decoded.numpy(), dense.mask.cpu().numpy()) + np.testing.assert_array_equal(detections.mask, rle.to_supervision().mask) + sv.MaskAnnotator().annotate(scene.copy(), detections) diff --git a/inference_models/tests/unit_tests/models/base/test_instance_segmentation.py b/inference_models/tests/unit_tests/models/base/test_instance_segmentation.py index 7998982ab2..c71408d4e6 100644 --- a/inference_models/tests/unit_tests/models/base/test_instance_segmentation.py +++ b/inference_models/tests/unit_tests/models/base/test_instance_segmentation.py @@ -45,181 +45,167 @@ def test_len_counts_boxes_regardless_of_mask_representation() -> None: assert len(detections) == 3 -class TestMaskSize: - """The grid masks actually live on, carried explicitly. +def test_rle_mask_size_defaults_to_image_size() -> None: + # given / when + masks = InstancesRLEMasks(image_size=(1080, 1920), masks=[]) - `mask.shape[1:]` and `InstancesRLEMasks.image_size` are currently the de - facto definition of the mask canvas at roughly fifteen call sites. Once the - resize target becomes adjustable they stop agreeing with the image, so the - grid has to travel with the prediction rather than be inferred from it. + # then + assert masks.mask_size == (1080, 1920) - Provisional: carrying this as an explicit field rather than a key in the - image metadata is a recommendation pending maintainer ratification. - """ - def test_rle_mask_size_defaults_to_image_size(self) -> None: - # given / when - masks = InstancesRLEMasks(image_size=(1080, 1920), masks=[]) - - # then - assert masks.mask_size == (1080, 1920) - - def test_rle_mask_size_can_differ_from_image_size(self) -> None: - # given / when - masks = InstancesRLEMasks( - image_size=(1080, 1920), masks=[], mask_size=(270, 480) - ) - - # then - assert masks.mask_size == (270, 480) - assert masks.image_size == (1080, 1920) - - def test_wire_format_declares_the_encoded_size(self) -> None: - # given - # COCO `size` must describe the grid the counts were encoded on, or the - # counts no longer sum to h*w and pycocotools decodes silently wrong - masks = InstancesRLEMasks( - image_size=(1080, 1920), masks=[b"abc"], mask_size=(270, 480) - ) - - # when - encoded = masks.to_coco_rle_masks() - - # then - assert encoded == [{"size": [270, 480], "counts": b"abc"}] - - def test_wire_format_unchanged_when_sizes_agree(self) -> None: - # given - masks = InstancesRLEMasks(image_size=(1080, 1920), masks=[b"abc"]) - - # when - encoded = masks.to_coco_rle_masks() - - # then - assert encoded == [{"size": [1080, 1920], "counts": b"abc"}] - - def test_dense_mask_size_falls_back_to_the_tensor_grid(self) -> None: - # given - detections = InstanceDetections( - xyxy=torch.zeros((2, 4)), - class_id=torch.zeros((2,), dtype=torch.int64), - confidence=torch.zeros((2,)), - mask=torch.zeros((2, 540, 960), dtype=torch.bool), - ) - - # then - assert detections.mask_size == (540, 960) - - def test_dense_mask_size_honours_an_explicit_value(self) -> None: - # given - detections = InstanceDetections( - xyxy=torch.zeros((1, 4)), - class_id=torch.zeros((1,), dtype=torch.int64), - confidence=torch.zeros((1,)), - mask=torch.zeros((1, 270, 480), dtype=torch.bool), - mask_size=(270, 480), - ) - - # then - assert detections.mask_size == (270, 480) - - def test_rle_carrier_reports_its_own_grid(self) -> None: - # given - detections = InstanceDetections( - xyxy=torch.zeros((1, 4)), - class_id=torch.zeros((1,), dtype=torch.int64), - confidence=torch.zeros((1,)), - mask=InstancesRLEMasks( - image_size=(1080, 1920), masks=[b""], mask_size=(270, 480) - ), - ) - - # then - assert detections.mask_size == (270, 480) - - def test_empty_detection_stack_still_reports_the_grid(self) -> None: - # given - # consumers derive the scale factor from the grid whether or not - # anything was detected - detections = InstanceDetections( - xyxy=torch.zeros((0, 4)), - class_id=torch.zeros((0,), dtype=torch.int64), - confidence=torch.zeros((0,)), - mask=torch.zeros((0, 270, 480), dtype=torch.bool), - ) - - # then - assert detections.mask_size == (270, 480) - - -class TestReducedGridLeavesTheBoundaryAtImageSize: - """`sv.Detections` and the COCO dict must describe the image, not the grid. - - `sv.Detections.mask` is documented as `(n, H, W)` matching the image and - its annotators index the scene with it, so a reduced grid raises. The COCO - dict is the opposite: `size` must describe the grid the counts were encoded - on, or decoding reinterprets the runs. - """ +def test_rle_mask_size_can_differ_from_image_size() -> None: + # given / when + masks = InstancesRLEMasks(image_size=(1080, 1920), masks=[], mask_size=(270, 480)) + + # then + assert masks.mask_size == (270, 480) + assert masks.image_size == (1080, 1920) + + +def test_wire_format_declares_the_encoded_size() -> None: + # given + # COCO `size` must describe the grid the counts were encoded on, or the + # counts no longer sum to h*w and pycocotools decodes silently wrong + masks = InstancesRLEMasks( + image_size=(1080, 1920), masks=[b"abc"], mask_size=(270, 480) + ) + + # when + encoded = masks.to_coco_rle_masks() + + # then + assert encoded == [{"size": [270, 480], "counts": b"abc"}] + + +def test_wire_format_unchanged_when_sizes_agree() -> None: + # given + masks = InstancesRLEMasks(image_size=(1080, 1920), masks=[b"abc"]) + + # when + encoded = masks.to_coco_rle_masks() + + # then + assert encoded == [{"size": [1080, 1920], "counts": b"abc"}] + + +def test_dense_mask_size_falls_back_to_the_tensor_grid() -> None: + # given + detections = InstanceDetections( + xyxy=torch.zeros((2, 4)), + class_id=torch.zeros((2,), dtype=torch.int64), + confidence=torch.zeros((2,)), + mask=torch.zeros((2, 540, 960), dtype=torch.bool), + ) + + # then + assert detections.mask_size == (540, 960) + + +def test_dense_mask_size_honours_an_explicit_value() -> None: + # given + detections = InstanceDetections( + xyxy=torch.zeros((1, 4)), + class_id=torch.zeros((1,), dtype=torch.int64), + confidence=torch.zeros((1,)), + mask=torch.zeros((1, 270, 480), dtype=torch.bool), + mask_size=(270, 480), + ) + + # then + assert detections.mask_size == (270, 480) + + +def test_rle_carrier_reports_its_own_grid() -> None: + # given + detections = InstanceDetections( + xyxy=torch.zeros((1, 4)), + class_id=torch.zeros((1,), dtype=torch.int64), + confidence=torch.zeros((1,)), + mask=InstancesRLEMasks( + image_size=(1080, 1920), masks=[b""], mask_size=(270, 480) + ), + ) + + # then + assert detections.mask_size == (270, 480) + + +def test_empty_detection_stack_still_reports_the_grid() -> None: + # given + # consumers derive the scale factor from the grid whether or not + # anything was detected + detections = InstanceDetections( + xyxy=torch.zeros((0, 4)), + class_id=torch.zeros((0,), dtype=torch.int64), + confidence=torch.zeros((0,)), + mask=torch.zeros((0, 270, 480), dtype=torch.bool), + ) - def test_to_supervision_restores_the_image_grid(self) -> None: - # given - # masks produced at a quarter of the image resolution - detections = InstanceDetections( - xyxy=torch.tensor([[0, 0, 80, 80]], dtype=torch.float32), - class_id=torch.zeros((1,), dtype=torch.int64), - confidence=torch.ones((1,), dtype=torch.float32), - mask=InstancesRLEMasks( - image_size=(80, 80), - masks=[_square_rle(20, 20)], - mask_size=(20, 20), - ), - ) - - # when - converted = detections.to_supervision() - - # then - assert converted.mask.shape == (1, 80, 80) - - def test_supervision_annotator_accepts_the_result(self) -> None: - # given - # the reported failure was an IndexError from boolean-index mismatch - import numpy as np - import supervision as sv - - scene = np.zeros((80, 80, 3), dtype=np.uint8) - detections = InstanceDetections( - xyxy=torch.tensor([[0, 0, 80, 80]], dtype=torch.float32), - class_id=torch.zeros((1,), dtype=torch.int64), - confidence=torch.ones((1,), dtype=torch.float32), - mask=InstancesRLEMasks( - image_size=(80, 80), - masks=[_square_rle(20, 20)], - mask_size=(20, 20), - ), - ) - - # when / then - sv.MaskAnnotator().annotate(scene.copy(), detections.to_supervision()) - - def test_iteration_declares_the_encoded_grid(self) -> None: - # given - detections = InstanceDetections( - xyxy=torch.tensor([[0, 0, 80, 80]], dtype=torch.float32), - class_id=torch.zeros((1,), dtype=torch.int64), - confidence=torch.ones((1,), dtype=torch.float32), - mask=InstancesRLEMasks( - image_size=(80, 80), - masks=[_square_rle(20, 20)], - mask_size=(20, 20), - ), - ) - - # when - _, mask, *_ = next(iter(detections)) - - # then - assert mask["size"] == [20, 20] + # then + assert detections.mask_size == (270, 480) + + +def test_to_supervision_restores_the_image_grid() -> None: + # given + # masks produced at a quarter of the image resolution + detections = InstanceDetections( + xyxy=torch.tensor([[0, 0, 80, 80]], dtype=torch.float32), + class_id=torch.zeros((1,), dtype=torch.int64), + confidence=torch.ones((1,), dtype=torch.float32), + mask=InstancesRLEMasks( + image_size=(80, 80), + masks=[_square_rle(20, 20)], + mask_size=(20, 20), + ), + ) + + # when + converted = detections.to_supervision() + + # then + assert converted.mask.shape == (1, 80, 80) + + +def test_supervision_annotator_accepts_the_result() -> None: + # given + # the reported failure was an IndexError from boolean-index mismatch + import numpy as np + import supervision as sv + + scene = np.zeros((80, 80, 3), dtype=np.uint8) + detections = InstanceDetections( + xyxy=torch.tensor([[0, 0, 80, 80]], dtype=torch.float32), + class_id=torch.zeros((1,), dtype=torch.int64), + confidence=torch.ones((1,), dtype=torch.float32), + mask=InstancesRLEMasks( + image_size=(80, 80), + masks=[_square_rle(20, 20)], + mask_size=(20, 20), + ), + ) + + # when / then + sv.MaskAnnotator().annotate(scene.copy(), detections.to_supervision()) + + +def test_iteration_declares_the_encoded_grid() -> None: + # given + detections = InstanceDetections( + xyxy=torch.tensor([[0, 0, 80, 80]], dtype=torch.float32), + class_id=torch.zeros((1,), dtype=torch.int64), + confidence=torch.ones((1,), dtype=torch.float32), + mask=InstancesRLEMasks( + image_size=(80, 80), + masks=[_square_rle(20, 20)], + mask_size=(20, 20), + ), + ) + + # when + _, mask, *_ = next(iter(detections)) + + # then + assert mask["size"] == [20, 20] def _square_rle(height: int, width: int) -> bytes: @@ -248,52 +234,53 @@ def _square_rle(height: int, width: int) -> bytes: return mask_utils.encode(np.asfortranarray(dense))["counts"] -class TestDenseImageDimensions: - def test_reduced_dense_masks_can_annotate_original_image(self) -> None: - import numpy as np - import supervision as sv - - mask = torch.zeros((1, 20, 30), dtype=torch.bool) - mask[:, 5:10, 8:14] = True - detections = InstanceDetections( - xyxy=torch.tensor([[80, 50, 140, 100]], dtype=torch.float32), - class_id=torch.tensor([0]), - confidence=torch.tensor([0.9]), - mask=mask, - image_size=(200, 300), - ).to_supervision() - - scene = np.zeros((200, 300, 3), dtype=np.uint8) - annotated = sv.MaskAnnotator().annotate(scene=scene, detections=detections) - - assert detections.mask.shape == (1, 200, 300) - assert detections.mask[0, 50:100, 80:140].all() - assert detections.mask.sum() == 50 * 60 - assert annotated.any() - - def test_origin_crop_is_resized_then_padded_not_stretched(self) -> None: - mask = torch.zeros((1, 20, 30), dtype=torch.bool) - mask[:, 2:10, 4:16] = True - detections = InstanceDetections( - xyxy=torch.tensor([[20, 10, 80, 50]], dtype=torch.float32), - class_id=torch.tensor([0]), - confidence=torch.tensor([0.9]), - mask=mask, - image_size=(200, 300), - mask_frame_size=(100, 150), - ).to_supervision() - - assert detections.mask.shape == (1, 200, 300) - assert detections.mask[0, 10:50, 20:80].all() - assert detections.mask.sum() == 40 * 60 - - def test_empty_reduced_dense_masks_keep_original_dimensions(self) -> None: - detections = InstanceDetections( - xyxy=torch.empty((0, 4)), - class_id=torch.empty((0,), dtype=torch.long), - confidence=torch.empty((0,)), - mask=torch.empty((0, 20, 30), dtype=torch.bool), - image_size=(200, 300), - ).to_supervision() - - assert detections.mask.shape == (0, 200, 300) +def test_reduced_dense_masks_can_annotate_original_image() -> None: + import numpy as np + import supervision as sv + + mask = torch.zeros((1, 20, 30), dtype=torch.bool) + mask[:, 5:10, 8:14] = True + detections = InstanceDetections( + xyxy=torch.tensor([[80, 50, 140, 100]], dtype=torch.float32), + class_id=torch.tensor([0]), + confidence=torch.tensor([0.9]), + mask=mask, + image_size=(200, 300), + ).to_supervision() + + scene = np.zeros((200, 300, 3), dtype=np.uint8) + annotated = sv.MaskAnnotator().annotate(scene=scene, detections=detections) + + assert detections.mask.shape == (1, 200, 300) + assert detections.mask[0, 50:100, 80:140].all() + assert detections.mask.sum() == 50 * 60 + assert annotated.any() + + +def test_origin_crop_is_resized_then_padded_not_stretched() -> None: + mask = torch.zeros((1, 20, 30), dtype=torch.bool) + mask[:, 2:10, 4:16] = True + detections = InstanceDetections( + xyxy=torch.tensor([[20, 10, 80, 50]], dtype=torch.float32), + class_id=torch.tensor([0]), + confidence=torch.tensor([0.9]), + mask=mask, + image_size=(200, 300), + mask_frame_size=(100, 150), + ).to_supervision() + + assert detections.mask.shape == (1, 200, 300) + assert detections.mask[0, 10:50, 20:80].all() + assert detections.mask.sum() == 40 * 60 + + +def test_empty_reduced_dense_masks_keep_original_dimensions() -> None: + detections = InstanceDetections( + xyxy=torch.empty((0, 4)), + class_id=torch.empty((0,), dtype=torch.long), + confidence=torch.empty((0,)), + mask=torch.empty((0, 20, 30), dtype=torch.bool), + image_size=(200, 300), + ).to_supervision() + + assert detections.mask.shape == (0, 200, 300) From 711e9804775e5077684c52f958b96b8523071371 Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Wed, 7 Oct 2026 12:00:51 +0200 Subject: [PATCH 28/41] Compare bounding rectangles by equivalent geometry --- .../test_workflow_with_bounding_rectangle.py | 152 ++++++++---------- 1 file changed, 68 insertions(+), 84 deletions(-) diff --git a/tests/workflows/integration_tests/execution/test_workflow_with_bounding_rectangle.py b/tests/workflows/integration_tests/execution/test_workflow_with_bounding_rectangle.py index 82de546efb..badc88ef87 100644 --- a/tests/workflows/integration_tests/execution/test_workflow_with_bounding_rectangle.py +++ b/tests/workflows/integration_tests/execution/test_workflow_with_bounding_rectangle.py @@ -1,3 +1,4 @@ +import cv2 as cv import numpy as np import pytest import supervision as sv @@ -63,6 +64,12 @@ } +_EXPECTED_RECTANGLES = [ + (np.array([[322, 402], [325, 224], [586, 228], [583, 406]]), 5.0), + (np.array([[219, 82], [352, 57], [409, 363], [276, 388]]), 6.0), +] + + @_NUMPY_ONLY @add_to_workflows_gallery( category="Basic Workflows", @@ -116,23 +123,15 @@ def test_rectangle_bounding_workflow( "angle" in result[0]["result"].data ), "'angle' data field must expected to be found in result" - assert np.allclose( - result[0]["result"]["rect"][0], - np.array([[322.0, 402.0], [325.0, 224.0], [586.0, 228.0], [583.0, 406.0]]), - atol=5.0, - ) - assert np.allclose( - result[0]["result"]["rect"][1], - np.array([[219.0, 82.0], [352.0, 57.0], [409.0, 363.0], [276.0, 388.0]]), - atol=6.0, - ) - assert np.allclose( - result[0]["result"]["width"], np.array([261.5, 311.25]), atol=5.0 - ) - assert np.allclose( - result[0]["result"]["height"], np.array([178.4, 135.2]), atol=6.0 - ) - assert np.allclose(result[0]["result"]["angle"], np.array([0.826, 79.5]), atol=0.5) + for i, (expected_rect, atol) in enumerate(_EXPECTED_RECTANGLES): + _assert_rectangle_geometry( + result[0]["result"]["rect"][i], + width=result[0]["result"]["width"][i], + height=result[0]["result"]["height"][i], + angle=result[0]["result"]["angle"][i], + expected_rect=expected_rect, + atol=atol, + ) @_TENSOR_ONLY @@ -179,40 +178,16 @@ def test_rectangle_bounding_workflow_tensor_native( assert "height" in meta, "'height' geometry must be found in per-box metadata" assert "angle" in meta, "'angle' geometry must be found in per-box metadata" - assert np.allclose( - np.array(detections.bboxes_metadata[0]["rect"]), - np.array([[322.0, 402.0], [325.0, 224.0], [586.0, 228.0], [583.0, 406.0]]), - atol=5.0, - ) - assert np.allclose( - np.array(detections.bboxes_metadata[1]["rect"]), - np.array([[219.0, 82.0], [352.0, 57.0], [409.0, 363.0], [276.0, 388.0]]), - atol=6.0, - ) - assert np.allclose( - [ - detections.bboxes_metadata[0]["width"], - detections.bboxes_metadata[1]["width"], - ], - np.array([261.5, 311.25]), - atol=5.0, - ) - assert np.allclose( - [ - detections.bboxes_metadata[0]["height"], - detections.bboxes_metadata[1]["height"], - ], - np.array([178.4, 135.2]), - atol=6.0, - ) - assert np.allclose( - [ - detections.bboxes_metadata[0]["angle"], - detections.bboxes_metadata[1]["angle"], - ], - np.array([0.826, 79.5]), - atol=0.5, - ) + for i, (expected_rect, atol) in enumerate(_EXPECTED_RECTANGLES): + metadata = detections.bboxes_metadata[i] + _assert_rectangle_geometry( + np.array(metadata["rect"]), + width=metadata["width"], + height=metadata["height"], + angle=metadata["angle"], + expected_rect=expected_rect, + atol=atol, + ) @_TENSOR_ONLY @@ -262,37 +237,46 @@ def test_rectangle_bounding_workflow_with_tensor_input( assert "height" in meta, "'height' geometry must be found in per-box metadata" assert "angle" in meta, "'angle' geometry must be found in per-box metadata" - assert np.allclose( - np.array(detections.bboxes_metadata[0]["rect"]), - np.array([[322.0, 402.0], [325.0, 224.0], [586.0, 228.0], [583.0, 406.0]]), - atol=5.0, - ) - assert np.allclose( - np.array(detections.bboxes_metadata[1]["rect"]), - np.array([[219.0, 82.0], [352.0, 57.0], [409.0, 363.0], [276.0, 388.0]]), - atol=6.0, - ) - assert np.allclose( - [ - detections.bboxes_metadata[0]["width"], - detections.bboxes_metadata[1]["width"], - ], - np.array([261.5, 311.25]), - atol=5.0, - ) - assert np.allclose( - [ - detections.bboxes_metadata[0]["height"], - detections.bboxes_metadata[1]["height"], - ], - np.array([178.4, 135.2]), - atol=6.0, - ) - assert np.allclose( - [ - detections.bboxes_metadata[0]["angle"], - detections.bboxes_metadata[1]["angle"], - ], - np.array([0.826, 79.5]), - atol=0.5, + for i, (expected_rect, atol) in enumerate(_EXPECTED_RECTANGLES): + metadata = detections.bboxes_metadata[i] + _assert_rectangle_geometry( + np.array(metadata["rect"]), + width=metadata["width"], + height=metadata["height"], + angle=metadata["angle"], + expected_rect=expected_rect, + atol=atol, + ) + + +def _assert_rectangle_geometry( + rect: np.ndarray, + *, + width: float, + height: float, + angle: float, + expected_rect: np.ndarray, + atol: float, +) -> None: + # OpenCV can swap axes and shift the starting corner of equivalent rectangles. + _assert_rectangle_corners(rect, expected_rect=expected_rect, atol=atol) + assert width > 0 + assert height > 0 + + reconstructed = cv.boxPoints( + ( + tuple(rect.astype(float).mean(axis=0)), + (float(width), float(height)), + float(angle), + ) ) + # The block truncates corners to integers and stores metadata as float16. + _assert_rectangle_corners(rect, expected_rect=reconstructed, atol=1.0) + + +def _assert_rectangle_corners( + rect: np.ndarray, *, expected_rect: np.ndarray, atol: float +) -> None: + candidates = np.stack([np.roll(rect, shift, axis=0) for shift in range(4)]) + closest = np.argmin(np.max(np.abs(candidates - expected_rect), axis=(1, 2))) + np.testing.assert_allclose(candidates[closest], expected_rect, atol=atol, rtol=0) From a88776b2ff9fefb12ee4f590e926c2777d0aa2eb Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Wed, 7 Oct 2026 15:08:46 +0200 Subject: [PATCH 29/41] Test v5 segmentation with visualization blocks --- ..._instance_segmentation_v5_visualization.py | 133 ++++++++++++++++++ 1 file changed, 133 insertions(+) create mode 100644 tests/workflows/integration_tests/execution/test_instance_segmentation_v5_visualization.py diff --git a/tests/workflows/integration_tests/execution/test_instance_segmentation_v5_visualization.py b/tests/workflows/integration_tests/execution/test_instance_segmentation_v5_visualization.py new file mode 100644 index 0000000000..03b52959bf --- /dev/null +++ b/tests/workflows/integration_tests/execution/test_instance_segmentation_v5_visualization.py @@ -0,0 +1,133 @@ +from unittest.mock import patch + +import numpy as np +import pytest +import supervision as sv +from pycocotools import mask as mask_utils +from roboflow_workflows.execution_engine.constants import RLE_MASK_KEY_IN_SV_DETECTIONS + +from inference.core.env import ( + ENABLE_TENSOR_DATA_REPRESENTATION, + USE_INFERENCE_MODELS, + WORKFLOWS_MAX_CONCURRENT_STEPS, +) +from inference.core.interfaces.workflows_models_provider import ( + ModelManagerModelsProvider, +) +from inference.core.workflows.core_steps.common.entities import StepExecutionMode +from inference.core.workflows.execution_engine.core import ExecutionEngine + + +@pytest.mark.skipif( + not USE_INFERENCE_MODELS or ENABLE_TENSOR_DATA_REPRESENTATION, + reason="Reduced v5 masks require the non-tensor inference_models path", +) +@pytest.mark.parametrize( + "mode,factor", [("accurate", 1.0), ("tradeoff", 0.5), ("fast", 0.0)] +) +def test_instance_segmentation_v5_with_visualization_blocks( + model_manager: ModelManagerModelsProvider, + dogs_image: np.ndarray, + mode: str, + factor: float, +) -> None: + # given + visualizations = ("mask", "polygon", "bounding_box") + specification = { + "version": "1.0", + "inputs": [{"type": "WorkflowImage", "name": "image"}], + "steps": [ + { + "type": "roboflow_core/roboflow_instance_segmentation_model@v5", + "name": "segment", + "image": "$inputs.image", + "model_id": "yolov8n-seg-640", + "mask_decode_mode": mode, + "tradeoff_factor": factor, + }, + *[ + { + "type": f"roboflow_core/{name}_visualization@v1", + "name": name, + "image": "$inputs.image", + "predictions": "$steps.segment.predictions", + "copy_image": True, + } + for name in visualizations + ], + ], + "outputs": [ + { + "type": "JsonField", + "name": "predictions", + "selector": "$steps.segment.predictions", + }, + *[ + { + "type": "JsonField", + "name": name, + "selector": f"$steps.{name}.image", + } + for name in visualizations + ], + ], + } + engine = ExecutionEngine.init( + workflow_definition=specification, + init_parameters={ + "workflows_core.model_manager": model_manager, + "workflows_core.api_key": None, + "workflows_core.step_execution_mode": StepExecutionMode.LOCAL, + }, + max_concurrent_steps=WORKFLOWS_MAX_CONCURRENT_STEPS, + ) + original_image = dogs_image.copy() + raw_predictions = [] + run_segmentation = model_manager.run_instance_segmentation + + def capture_predictions(**kwargs): + predictions = run_segmentation(**kwargs) + raw_predictions.extend(predictions) + return predictions + + # when + with patch.object( + model_manager, "run_instance_segmentation", side_effect=capture_predictions + ): + results = engine.run(runtime_parameters={"image": [dogs_image]}) + + # then + assert len(results) == 1 + assert len(raw_predictions) == 1 + raw_masks = [p["rle"] for p in raw_predictions[0]["predictions"]] + assert len(raw_masks) == 2 + image_shape = dogs_image.shape[:2] + for rle in raw_masks: + if mode == "accurate": + assert tuple(rle["size"]) == image_shape + else: + assert all(size < full for size, full in zip(rle["size"], image_shape)) + + result = results[0] + detections = result["predictions"] + assert isinstance(detections, sv.Detections) + assert len(detections) == 2 + assert detections.mask.shape == (2, *image_shape) + assert all(mask.any() for mask in detections.mask) + assert detections.data["class_name"].tolist() == ["dog", "dog"] + for rle, dense in zip( + detections.data[RLE_MASK_KEY_IN_SV_DETECTIONS], detections.mask + ): + assert tuple(rle["size"]) == image_shape + np.testing.assert_array_equal(mask_utils.decode(rle).astype(bool), dense) + + for name in visualizations: + rendered = result[name].numpy_image + assert rendered.shape == dogs_image.shape + assert rendered.dtype == dogs_image.dtype + assert np.any(rendered != original_image) + changed_pixels = np.any(result["mask"].numpy_image != original_image, axis=2) + mask_union = detections.mask.any(axis=0) + assert changed_pixels[mask_union].any() + assert not changed_pixels[~mask_union].any() + np.testing.assert_array_equal(dogs_image, original_image) From 6f6179248ff5d04a0f5ccc20f5f0e9d0496e36ac Mon Sep 17 00:00:00 2001 From: MTaraszewski <43919218+MTaraszewski@users.noreply.github.com> Date: Fri, 9 Oct 2026 12:50:54 +0200 Subject: [PATCH 30/41] Make configurable mask grids opt-in and defer workflow v5 (#3149) * Make configurable mask grids opt-in and defer workflow v5 Preserve image-resolution defaults; return opted-in polygons and RLE on the same grid with explicit coordinate metadata. Keep SDK resizing and server visualization aligned, retain v1-v4 guards, and defer v5/Supervision work. Triton optimization remains separate. Validation in repository uv venv: 519 server/SDK unit tests; 104 workflow/configuration tests; 207 model tests with 3 skips; 36 real HTTP round trips; 3 YOLOv8 v4 visualization tests. Black, isort and git diff --check pass. Installed the declared pytest-asyncio test dependency in the ignored venv. GPU and hosted staging tests were not run. * Keep opted-in polygon points in image coordinates Scale reduced-mask contours to the same image frame as boxes. Restrict grid metadata to RLE responses and preserve SDK resizing and visualization behavior. Update the contract documentation and changelogs. Validation: 525 server/SDK unit tests, 104 workflow/configuration tests, and 36 real HTTP integration tests passed locally in the existing venv. Black, isort, and git diff --check passed. GPU and hosted staging tests were not run. --- docs/instance_segmentation_mask_resolution.md | 73 ++ inference/core/entities/requests/inference.py | 13 +- .../core/entities/responses/inference.py | 4 + inference/core/interfaces/http/http_api.py | 7 + .../core/models/inference_models_adapters.py | 37 +- inference_models/docs/changelog.md | 13 + inference_sdk/http/entities.py | 6 + inference_sdk/http/utils/post_processing.py | 7 + ...t_instance_segmentation_mask_resolution.py | 2 +- ...t_instance_segmentation_rle_predictions.py | 16 +- .../core/interfaces/http/test_http_api.py | 34 + .../models/test_inference_models_adapters.py | 135 +++- .../core/utils/test_visualisation.py | 45 ++ .../unit_tests/http/test_entities.py | 28 + .../http/utils/test_postprocessing.py | 43 + ...instance_segmentation_v4_visualization.py} | 11 +- ... => test_mask_resolution_compatibility.py} | 149 ++-- workflows/CHANGELOG.md | 54 +- .../common/inference_response_entities.py | 33 +- .../roboflow_workflows/core_steps/loader.py | 10 - .../roboflow/instance_segmentation/v5.py | 481 ----------- .../instance_segmentation/v5_tensor.py | 744 ------------------ .../execution_engine/entities/types.py | 7 +- 23 files changed, 561 insertions(+), 1391 deletions(-) create mode 100644 docs/instance_segmentation_mask_resolution.md rename tests/workflows/integration_tests/execution/{test_instance_segmentation_v5_visualization.py => test_instance_segmentation_v4_visualization.py} (92%) rename tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/{test_v5_mask_resolution.py => test_mask_resolution_compatibility.py} (65%) delete mode 100644 workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5.py delete mode 100644 workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py diff --git a/docs/instance_segmentation_mask_resolution.md b/docs/instance_segmentation_mask_resolution.md new file mode 100644 index 0000000000..c34fb6e3c1 --- /dev/null +++ b/docs/instance_segmentation_mask_resolution.md @@ -0,0 +1,73 @@ +# Opt into configurable instance-segmentation masks + +On servers running `USE_INFERENCE_MODELS=True`, existing requests keep masks at +image resolution. To use `mask_decode_mode="fast"` or `"tradeoff"`, also set +`allow_reduced_mask_resolution=true`. This applies to both +`POST /infer/instance_segmentation` (JSON body) and the legacy +`POST /{dataset_id}/{version_id}` route (query parameters). + +```python +from inference_sdk import InferenceConfiguration, InferenceHTTPClient + +client = InferenceHTTPClient(api_url="http://localhost:9001").configure( + InferenceConfiguration( + allow_reduced_mask_resolution=True, + mask_decode_mode="tradeoff", + tradeoff_factor=0.5, + response_mask_format="polygon", # or "rle" + ) +) +result = client.infer("image.jpg", model_id="yolov8n-seg-640") +``` + +With the opt-in, contours can be extracted from reduced masks, but polygon +points and bounding boxes always use the original image's pixel coordinates. +The server scales contour coordinates before returning polygon predictions; +callers do not need to transform the points. + +RLE responses retain the encoded mask grid. When its dimensions differ from +the response image, `mask_metadata` explicitly describes that RLE grid: + +```json +{ + "image": {"width": 640, "height": 480}, + "mask_metadata": { + "coordinate_system": "mask_grid", + "width": 160, + "height": 120, + "scale_x": 4.0, + "scale_y": 4.0 + } +} +``` + +These dimensions illustrate the contract, not an expected result for every +model or factor. RLE `size` is `[height, width]` from `mask_metadata`. +The scales map positions on the encoded RLE grid to image coordinates. +Metadata is omitted for polygon responses and for RLE grids matching the +image. Empty RLE predictions still include metadata when their grid differs. +Each image in a batch has its own metadata. + +Polygon responses can be passed directly to Supervision: + +```python +import supervision as sv + +detections = sv.Detections.from_inference(result) +``` + +Decode RLE on its declared grid; never replace its `size` with image dimensions +without resampling the mask. During client-side image resizing, the SDK scales +polygon points and bounding boxes together. It preserves encoded RLE data and +updates its metadata scales to the client image. Server-rendered polygon +visualizations use the returned points directly. + +Direct `inference_models` calls keep their `masks_resolution_factor` interface. +The legacy model backend retains its existing decode-mode behavior and ignores +the new opt-in flag. Values from 0 to 1 interpolate model-grid and image-grid +dimensions; a lower factor does not always mean a smaller grid or faster execution. +Damian's Triton optimization is separate; the existing fallback remains for now. + +Workflow versions v1–v4 retain their compatibility guards. The new v5 block, +reduced-mask Supervision/workflow handling, and full tensor support are deferred +to a separate PR. This change does not introduce reduced-grid workflow outputs. diff --git a/inference/core/entities/requests/inference.py b/inference/core/entities/requests/inference.py index 6160f46a5c..7fc91cabff 100644 --- a/inference/core/entities/requests/inference.py +++ b/inference/core/entities/requests/inference.py @@ -267,6 +267,16 @@ class InstanceSegmentationInferenceRequest(ObjectDetectionInferenceRequest): "1.0 selects image resolution. Intermediate values interpolate mask-grid " "dimensions; the legacy backend uses a different interpolation schedule.", ) + allow_reduced_mask_resolution: bool = Field( + default=False, + examples=[True], + description="Opt into mask_decode_mode and tradeoff_factor on the " + "inference_models backend. Without this flag, masks stay at image " + "resolution. With it, RLE size can differ from image dimensions; " + "mask_metadata describes the RLE grid and its mapping to the image. " + "Polygon points and bounding boxes remain in image space. The legacy " + "backend retains its existing decoding behavior regardless of this flag.", + ) response_mask_format: Literal["polygon", "rle"] = Field( default="polygon", examples=["rle"], @@ -275,7 +285,8 @@ class InstanceSegmentationInferenceRequest(ObjectDetectionInferenceRequest): "require special decoding on the caller side - currently supported in `opt-in` mode when server is " "running with `USE_INFERENCE_MODELS=True` - otherwise it's ignored. " "RLE size records the encoded mask grid, which may differ from the image " - "dimensions; polygon coordinates are returned in image space.", + "dimensions when allow_reduced_mask_resolution=True, with mask_metadata " + "supplying the image mapping. Polygon points always use image coordinates.", ) enforce_dense_masks_in_inference_models: Optional[bool] = Field( default=False, diff --git a/inference/core/entities/responses/inference.py b/inference/core/entities/responses/inference.py index 29804db358..8642b2eff6 100644 --- a/inference/core/entities/responses/inference.py +++ b/inference/core/entities/responses/inference.py @@ -10,6 +10,7 @@ InferenceResponse, InferenceResponseImage, InstanceSegmentationInferenceResponse, + MaskCoordinateMetadata, ResolvedModel, WithVisualizationResponse, ) @@ -234,6 +235,7 @@ class InstanceSegmentationInferenceResponseDC: time: object = None visualization: object = None resolved_model: Optional[ResolvedModel] = None + mask_metadata: Optional[MaskCoordinateMetadata] = None # Internal stream-pipeline fast path: lets workflow execution carry a # response future through Model.infer_from_request without blocking the # inference thread. `_is_response_dc_to_dict` intentionally ignores it. @@ -285,6 +287,8 @@ def _is_response_dc_to_dict(r: InstanceSegmentationInferenceResponseDC) -> dict: d["time"] = r.time if r.visualization is not None: d["visualization"] = r.visualization + if r.mask_metadata is not None: + d["mask_metadata"] = r.mask_metadata.model_dump() if r.resolved_model is not None: d["resolved_model"] = r.resolved_model.model_dump() return d diff --git a/inference/core/interfaces/http/http_api.py b/inference/core/interfaces/http/http_api.py index d96c72b553..f75a3d7d9a 100644 --- a/inference/core/interfaces/http/http_api.py +++ b/inference/core/interfaces/http/http_api.py @@ -4779,6 +4779,12 @@ def legacy_infer_from_request( "accurate", description="One of 'accurate' or 'fast'. If 'accurate' the mask will be decoded using the original image size. If 'fast' the mask will be decoded using the original mask size. 'accurate' is slower but more accurate.", ), + allow_reduced_mask_resolution: bool = Query( + False, + description="Opt into mask_decode_mode and tradeoff_factor on " + "inference_models. Otherwise masks stay at image resolution. " + "Legacy backend decoding is unchanged.", + ), tradeoff_factor: Optional[float] = Query( 0.0, description="The amount to tradeoff between 0='fast' and 1='accurate'", @@ -4973,6 +4979,7 @@ def legacy_infer_from_request( args = { "mask_decode_mode": mask_decode_mode, "tradeoff_factor": tradeoff_factor, + "allow_reduced_mask_resolution": allow_reduced_mask_resolution, } if response_mask_format: args["response_mask_format"] = response_mask_format diff --git a/inference/core/models/inference_models_adapters.py b/inference/core/models/inference_models_adapters.py index ade47bb5ca..8a0480a853 100644 --- a/inference/core/models/inference_models_adapters.py +++ b/inference/core/models/inference_models_adapters.py @@ -42,6 +42,7 @@ KeypointsDetectionInferenceResponse, KeypointsPrediction, LMMInferenceResponse, + MaskCoordinateMetadata, MultiLabelClassificationInferenceResponse, ObjectDetectionInferenceResponse, ObjectDetectionPrediction, @@ -568,13 +569,14 @@ def run_tensor_native_inference( def map_inference_kwargs(self, kwargs: dict) -> dict: kwargs["input_color_format"] = "bgr" - # The published request surface is an enum plus a float; post-processing - # consumes a single resolution factor. Translate here and consume both - # source keys, which the deeper stages do not accept. - kwargs["masks_resolution_factor"] = _resolve_masks_resolution_factor( + allow_reduced_masks = kwargs.pop("allow_reduced_mask_resolution", False) + resolution_factor = _resolve_masks_resolution_factor( mask_decode_mode=kwargs.pop("mask_decode_mode", None), tradeoff_factor=kwargs.pop("tradeoff_factor", None), ) + kwargs["masks_resolution_factor"] = ( + resolution_factor if allow_reduced_masks else 1.0 + ) pre_processing_overrides = PreProcessingOverrides( disable_contrast_enhancement=kwargs.get("disable_preproc_contrast", False), disable_grayscale=kwargs.get("disable_preproc_grayscale", False), @@ -1053,19 +1055,22 @@ def _build_responses_from_detections( # thread-local pinned scratch buffers. Only scalar values and # polygon/RLE lists may be stored on responses below; do not return # those arrays or any view derived from them. - # Contours are extracted in mask coordinates. When the mask grid - # is not the image grid they have to be lifted before they are - # reported; equal sizes short-circuit to a no-op. This also breaks - # any view into the pinned scratch buffers noted above, since the - # scaled polygons are freshly allocated. mask_size = getattr(det, "mask_size", None) produced_against = resolve_mask_frame_size(preproc_metadata) - if ( - not return_in_rle - and mask_size is not None - and tuple(mask_size) - != (produced_against.height, produced_against.width) - ): + different_grid = mask_size is not None and tuple(mask_size) != (H, W) + # Async responses receive the already-resolved model factor. + native_grid = kwargs.get("allow_reduced_mask_resolution", False) or ( + kwargs.get("masks_resolution_factor", 1.0) != 1.0 + ) + mask_metadata = None + if return_in_rle and native_grid and different_grid: + mask_metadata = MaskCoordinateMetadata( + height=int(mask_size[0]), + width=int(mask_size[1]), + scale_x=W / mask_size[1], + scale_y=H / mask_size[0], + ) + if not return_in_rle and different_grid: polys_or_rles = scale_polygons_to_image( polys_or_rles, mask_size=ImageDimensions( @@ -1152,6 +1157,7 @@ def _build_responses_from_detections( InstanceSegmentationInferenceResponseDC( predictions=predictions, image=InferenceResponseImageDC(width=W, height=H), + mask_metadata=mask_metadata, ) ) else: @@ -1159,6 +1165,7 @@ def _build_responses_from_detections( InstanceSegmentationInferenceResponse( predictions=predictions, image=InferenceResponseImage(width=W, height=H), + mask_metadata=mask_metadata, ) ) return responses diff --git a/inference_models/docs/changelog.md b/inference_models/docs/changelog.md index 5ef60525d0..fccca8f879 100644 --- a/inference_models/docs/changelog.md +++ b/inference_models/docs/changelog.md @@ -21,6 +21,19 @@ ### Changed +- Server requests using `inference_models` now require + `allow_reduced_mask_resolution=true` to apply `mask_decode_mode` and + `tradeoff_factor`. Without this opt-in, masks retain image resolution. + Polygon points and bounding boxes always share image coordinates, including + when contours are extracted from reduced masks. Opted-in RLE responses retain + the encoded mask grid; when it differs from the image, `mask_metadata` + supplies its dimensions and x/y scales to image coordinates. The SDK scales + polygon points and boxes together during client resizing, while preserving + encoded RLE data and updating its metadata. Server visualizations use the + returned image-space polygon points directly. + Direct `inference_models` callers continue to use `masks_resolution_factor`. + Reduced-grid workflow support is deferred to a separate PR. + - `InstancesRLEMasks.to_coco_rle_masks()` now declares the grid the counts were encoded on rather than the image size. These agree unless a resolution factor below `1.0` is used; when they differ, declaring the image size made diff --git a/inference_sdk/http/entities.py b/inference_sdk/http/entities.py index 0e82e17783..619045e7ef 100644 --- a/inference_sdk/http/entities.py +++ b/inference_sdk/http/entities.py @@ -135,6 +135,8 @@ class InferenceConfiguration: format: The format for the inference. mask_decode_mode: The mask decode mode for the inference. tradeoff_factor: The tradeoff factor for the inference. + allow_reduced_mask_resolution: Opt into configurable mask grids on the + inference_models server backend. Legacy backend behavior is unchanged. max_candidates: The maximum number of candidates for the inference. max_detections: The maximum number of detections for the inference. iou_threshold: The intersection over union threshold for the inference. @@ -184,6 +186,8 @@ class InferenceConfiguration: # Pass "legacy" explicitly to keep the old behaviour silently. api_key_transport: Optional[Union[str, ApiKeyTransport]] = None + allow_reduced_mask_resolution: Optional[bool] = None + def __post_init__(self) -> None: # Normalise the transport to the enum so the client can rely on # identity checks. NOTE: this field configures the credential CHANNEL @@ -319,6 +323,7 @@ def to_instance_segmentation_parameters(self) -> Dict[str, Any]: parameters_specs = [ ("mask_decode_mode", "mask_decode_mode"), ("tradeoff_factor", "tradeoff_factor"), + ("allow_reduced_mask_resolution", "allow_reduced_mask_resolution"), ("response_mask_format", "response_mask_format"), ] for internal_name, external_name in parameters_specs: @@ -367,6 +372,7 @@ def to_legacy_call_parameters(self) -> Dict[str, Any]: ("visualize_labels", "labels"), ("mask_decode_mode", "mask_decode_mode"), ("tradeoff_factor", "tradeoff_factor"), + ("allow_reduced_mask_resolution", "allow_reduced_mask_resolution"), ("max_detections", "max_detections"), ("iou_threshold", "overlap"), ("stroke_width", "stroke"), diff --git a/inference_sdk/http/utils/post_processing.py b/inference_sdk/http/utils/post_processing.py index 1970daad80..01683a1ae8 100644 --- a/inference_sdk/http/utils/post_processing.py +++ b/inference_sdk/http/utils/post_processing.py @@ -220,6 +220,13 @@ def adjust_prediction_to_client_scaling_factor( "width": round(prediction["image"]["width"] / scaling_factor), "height": round(prediction["image"]["height"] / scaling_factor), } + mask_metadata = prediction.get("mask_metadata") + if mask_metadata and mask_metadata.get("coordinate_system") == "mask_grid": + mask_metadata["scale_x"] = prediction["image"]["width"] / mask_metadata["width"] + mask_metadata["scale_y"] = ( + prediction["image"]["height"] / mask_metadata["height"] + ) + if predictions_should_not_be_post_processed(prediction=prediction): return prediction if "points" in prediction["predictions"][0]: diff --git a/tests/inference/hosted_platform_tests/workflows_examples/roboflow_models/test_instance_segmentation_mask_resolution.py b/tests/inference/hosted_platform_tests/workflows_examples/roboflow_models/test_instance_segmentation_mask_resolution.py index bec9075851..455668eeb4 100644 --- a/tests/inference/hosted_platform_tests/workflows_examples/roboflow_models/test_instance_segmentation_mask_resolution.py +++ b/tests/inference/hosted_platform_tests/workflows_examples/roboflow_models/test_instance_segmentation_mask_resolution.py @@ -10,7 +10,7 @@ ) -@pytest.mark.parametrize("version", ["v4", "v5"]) +@pytest.mark.parametrize("version", ["v4"]) @pytest.mark.parametrize( "mode,factor", [("accurate", 1.0), ("tradeoff", 0.5), ("fast", 0.0)] ) diff --git a/tests/inference/integration_tests/test_instance_segmentation_rle_predictions.py b/tests/inference/integration_tests/test_instance_segmentation_rle_predictions.py index bca7e85fb0..b6915c34fa 100644 --- a/tests/inference/integration_tests/test_instance_segmentation_rle_predictions.py +++ b/tests/inference/integration_tests/test_instance_segmentation_rle_predictions.py @@ -21,12 +21,13 @@ @pytest.mark.skipif( not USE_INFERENCE_MODELS, reason="Resolution control uses inference-models" ) +@pytest.mark.parametrize("opt_in", [None, False, True]) @pytest.mark.parametrize("response_format", ["polygon", "rle"]) @pytest.mark.parametrize( "mode,factor", [("accurate", 1.0), ("tradeoff", 0.5), ("fast", 0.0)] ) def test_mask_resolution_round_trip_through_server( - auth_mode: str, response_format: str, mode: str, factor: float + auth_mode: str, response_format: str, mode: str, factor: float, opt_in ) -> None: payload = { "image": {"type": "url", "value": "https://media.roboflow.com/dog.jpeg"}, @@ -36,6 +37,9 @@ def test_mask_resolution_round_trip_through_server( "mask_decode_mode": mode, "tradeoff_factor": factor, } + if opt_in is not None: + payload["allow_reduced_mask_resolution"] = opt_in + response = requests.post( f"{BASE_URL}:{PORT}/infer/instance_segmentation", json=without_api_key_in_header_mode(auth_mode, payload), @@ -44,6 +48,12 @@ def test_mask_resolution_round_trip_through_server( ) response.raise_for_status() result = response.json() + metadata = result.get("mask_metadata") + if response_format == "rle" and opt_in and mode != "accurate": + assert metadata["coordinate_system"] == "mask_grid" + assert metadata["width"] > 0 and metadata["height"] > 0 + else: + assert metadata is None detections = sv.Detections.from_inference(result) height, width = result["image"]["height"], result["image"]["width"] assert len(detections) > 0 @@ -63,7 +73,9 @@ def test_mask_resolution_round_trip_through_server( } assert len(sizes) == 1 mask_height, mask_width = sizes.pop() - if mode == "accurate": + if metadata: + assert (mask_height, mask_width) == (metadata["height"], metadata["width"]) + if mode == "accurate" or not opt_in: assert (mask_height, mask_width) == (height, width) else: assert mask_height < height and mask_width < width diff --git a/tests/inference/unit_tests/core/interfaces/http/test_http_api.py b/tests/inference/unit_tests/core/interfaces/http/test_http_api.py index 10ef575aa7..0a09bce284 100644 --- a/tests/inference/unit_tests/core/interfaces/http/test_http_api.py +++ b/tests/inference/unit_tests/core/interfaces/http/test_http_api.py @@ -2308,3 +2308,37 @@ def test_workflow_http_routes_propagate_dispatch_depth(monkeypatch, path) -> Non assert response.status_code == 200 parameters = execution_engine.init.call_args.kwargs["init_parameters"] assert parameters["workflows_core.inner_workflow_dispatch_depth"] == 2 + + +@pytest.mark.parametrize("route", ["v0", "v1"]) +@pytest.mark.parametrize("opt_in", [None, False, True]) +def test_instance_segmentation_routes_forward_mask_resolution_opt_in( + monkeypatch, route, opt_in +) -> None: + interface, manager = _build_plain_interface(monkeypatch) + manager.get_task_type.return_value = "instance-segmentation" + parameters = {"mask_decode_mode": "tradeoff", "tradeoff_factor": 0.25} + if opt_in is not None: + parameters["allow_reduced_mask_resolution"] = opt_in + + with TestClient(interface.app) as client: + if route == "v0": + response = client.post( + "/model/1", + params={**parameters, "image": "https://example.com/image.jpg"}, + ) + else: + response = client.post( + "/infer/instance_segmentation", + json={ + **parameters, + "model_id": "model/1", + "image": {"type": "url", "value": "https://example.com/image.jpg"}, + }, + ) + + assert response.status_code == 200, response.text + request = manager.infer_from_request_sync.call_args.args[1] + assert request.allow_reduced_mask_resolution is (opt_in is True) + assert request.mask_decode_mode == "tradeoff" + assert request.tradeoff_factor == 0.25 diff --git a/tests/inference/unit_tests/core/models/test_inference_models_adapters.py b/tests/inference/unit_tests/core/models/test_inference_models_adapters.py index 62f69e8f9a..20d5ad3658 100644 --- a/tests/inference/unit_tests/core/models/test_inference_models_adapters.py +++ b/tests/inference/unit_tests/core/models/test_inference_models_adapters.py @@ -995,14 +995,18 @@ def test_accurate_maps_to_full_resolution(self) -> None: def test_fast_maps_to_mask_resolution(self) -> None: # given / when - mapped = self._map(mask_decode_mode="fast") + mapped = self._map(mask_decode_mode="fast", allow_reduced_mask_resolution=True) # then assert mapped["masks_resolution_factor"] == 0.0 def test_tradeoff_reads_the_factor(self) -> None: # given / when - mapped = self._map(mask_decode_mode="tradeoff", tradeoff_factor=0.25) + mapped = self._map( + mask_decode_mode="tradeoff", + tradeoff_factor=0.25, + allow_reduced_mask_resolution=True, + ) # then assert mapped["masks_resolution_factor"] == 0.25 @@ -1018,11 +1022,16 @@ def test_default_is_full_resolution_when_nothing_is_passed(self) -> None: def test_source_keys_do_not_reach_the_model(self) -> None: # given / when # deeper pre/post stages do not accept arbitrary kwargs - mapped = self._map(mask_decode_mode="tradeoff", tradeoff_factor=0.5) + mapped = self._map( + mask_decode_mode="tradeoff", + tradeoff_factor=0.5, + allow_reduced_mask_resolution=True, + ) # then assert "mask_decode_mode" not in mapped assert "tradeoff_factor" not in mapped + assert "allow_reduced_mask_resolution" not in mapped @pytest.mark.parametrize("bad_factor", [-0.1, 1.1]) def test_out_of_range_factor_is_rejected(self, bad_factor: float) -> None: @@ -1036,6 +1045,23 @@ def test_unknown_mode_is_rejected(self) -> None: with pytest.raises(InvalidMaskDecodeArgument): self._map(mask_decode_mode="nonsense") + @pytest.mark.parametrize("mode", ["fast", "tradeoff"]) + @pytest.mark.parametrize("opt_in", [None, False]) + def test_existing_requests_keep_image_resolution(self, mode, opt_in) -> None: + kwargs = {"mask_decode_mode": mode, "tradeoff_factor": 0.25} + if opt_in is not None: + kwargs["allow_reduced_mask_resolution"] = opt_in + + mapped = self._map(**kwargs) + + assert mapped["masks_resolution_factor"] == 1.0 + assert "allow_reduced_mask_resolution" not in mapped + + def test_internal_resolution_kwarg_does_not_bypass_opt_in(self) -> None: + mapped = self._map(masks_resolution_factor=0.25) + + assert mapped["masks_resolution_factor"] == 1.0 + class TestRLEBackedPolygons: """Polygons derived from RLE carriers at a reduced resolution. @@ -1101,12 +1127,13 @@ def test_non_square_grid(self) -> None: assert polygon[:, 1].max() <= 80 +@pytest.mark.parametrize("opt_in", [False, True]) @pytest.mark.parametrize("mask_format", ["dense", "rle"]) @pytest.mark.parametrize("factor", [1.0, 0.5, 0.0]) @pytest.mark.parametrize("offset", [(0, 0), (60, 40)]) @pytest.mark.parametrize("response_format", ["polygon", "rle"]) def test_crop_response_uses_the_frame_represented_by_mask( - mask_format: str, factor: float, offset: tuple, response_format: str + mask_format: str, factor: float, offset: tuple, response_format: str, opt_in: bool ) -> None: from inference_models.entities import ImageDimensions from inference_models.models.common.roboflow.model_packages import ( @@ -1168,9 +1195,20 @@ def test_crop_response_uses_the_frame_represented_by_mask( ) response = _seg_adapter()._build_responses_from_detections( - [detections], [metadata], response_mask_format=response_format + [detections], + [metadata], + response_mask_format=response_format, + allow_reduced_mask_resolution=opt_in, )[0] + coordinate_metadata = response.mask_metadata + if response_format == "rle" and opt_in and factor != 1.0: + assert coordinate_metadata.height == detections.mask_size[0] + assert coordinate_metadata.width == detections.mask_size[1] + assert coordinate_metadata.coordinate_system == "mask_grid" + else: + assert coordinate_metadata is None + if response_format == "rle": converted = sv.Detections.from_inference(response.model_dump(by_alias=True)) expected = np.zeros((1, 200, 300), dtype=bool) @@ -1197,3 +1235,90 @@ def test_crop_response_uses_the_frame_represented_by_mask( assert max(ys) == pytest.approx(50 + offset_y - pixel_size) assert prediction.x == pytest.approx(50 + offset_x) assert prediction.y == pytest.approx(30 + offset_y) + + +@pytest.mark.parametrize("use_dc", [False, True]) +@pytest.mark.parametrize("empty", [False, True]) +@pytest.mark.parametrize("response_format", ["polygon", "rle"]) +def test_deferred_response_preserves_coordinate_contract( + use_dc, empty, response_format +): + adapter = _seg_adapter() + adapter._pipeline_depth = 2 + count = 0 if empty else 1 + detections = InstanceDetections( + xyxy=torch.tensor([[0, 0, 10, 20]], dtype=torch.int32)[:count], + confidence=torch.tensor([0.9])[:count], + class_id=torch.tensor([0])[:count], + mask=torch.ones((count, 5, 2), dtype=torch.bool), + ) + future = SimpleNamespace(result=lambda: [detections]) + response = adapter._finalize_future( + future, + _make_meta("mask-grid"), + { + "masks_resolution_factor": 0.25, + "response_mask_format": response_format, + "source": "workflow-execution" if use_dc else "api", + }, + )[0] + + serialized = ( + response.to_dict() + if use_dc and response_format == "polygon" + else response.model_dump(exclude_none=True) + ) + assert serialized["image"] == {"width": 10, "height": 20} + if response_format == "rle": + assert serialized["mask_metadata"] == { + "coordinate_system": "mask_grid", + "width": 2, + "height": 5, + "scale_x": 5.0, + "scale_y": 4.0, + } + else: + assert "mask_metadata" not in serialized + assert len(serialized["predictions"]) == count + if not empty and response_format == "polygon": + assert max(p["x"] for p in serialized["predictions"][0]["points"]) == 5 + assert max(p["y"] for p in serialized["predictions"][0]["points"]) == 16 + + +@pytest.mark.parametrize("mask_format", ["dense", "rle"]) +def test_opted_in_polygons_share_box_coordinates_on_non_square_image(mask_format): + masks = torch.zeros((1, 160, 160), dtype=torch.bool) + masks[:, 48:72, 48:72] = True + mask = masks + if mask_format == "rle": + from inference_models.entities import ImageDimensions + + rle = mask_utils.encode(np.asfortranarray(masks[0].numpy().astype(np.uint8))) + mask = InstancesRLEMasks( + image_size=ImageDimensions(height=800, width=1000), + masks=[rle["counts"]], + mask_size=(160, 160), + ) + detections = InstanceDetections( + xyxy=torch.tensor([[300, 240, 450, 360]], dtype=torch.float32), + confidence=torch.tensor([0.9]), + class_id=torch.tensor([0]), + mask=mask, + ) + metadata = SimpleNamespace(original_size=SimpleNamespace(width=1000, height=800)) + + response = _seg_adapter()._build_responses_from_detections( + [detections], [metadata], allow_reduced_mask_resolution=True + )[0] + + prediction = response.predictions[0] + xs = [point.x for point in prediction.points] + ys = [point.y for point in prediction.points] + assert response.mask_metadata is None + assert (min(xs), min(ys), max(xs), max(ys)) == (300, 240, 443.75, 355) + converted = sv.Detections.from_inference(response.model_dump(by_alias=True)) + np.testing.assert_array_equal(converted.xyxy, [[300, 240, 450, 360]]) + np.testing.assert_allclose( + sv.mask_to_xyxy(converted.mask), [[300, 240, 443.75, 355]], atol=1 + ) + sv.MaskAnnotator().annotate(np.zeros((800, 1000, 3), dtype=np.uint8), converted) diff --git a/tests/inference/unit_tests/core/utils/test_visualisation.py b/tests/inference/unit_tests/core/utils/test_visualisation.py index b7481e5da1..1fc6d15ce9 100644 --- a/tests/inference/unit_tests/core/utils/test_visualisation.py +++ b/tests/inference/unit_tests/core/utils/test_visualisation.py @@ -23,3 +23,48 @@ def test_bbox_to_points() -> None: # then assert result == ((5, 16), (15, 24)) + + +def test_polygon_visualization_uses_image_coordinates_without_mutation( + monkeypatch, +): + from types import SimpleNamespace + from unittest.mock import MagicMock + + import numpy as np + + from inference.core.entities.responses.inference import ( + InstanceSegmentationPrediction, + Point, + ) + from inference.core.utils import visualisation + + box = InstanceSegmentationPrediction( + x=50, + y=50, + width=40, + height=40, + confidence=0.9, + class_id=0, + points=[Point(x=25, y=32)], + **{"class": "car"}, + ) + response = SimpleNamespace(predictions=[box]) + scene = np.zeros((100, 100, 3), dtype=np.uint8) + draw_polygon = MagicMock(return_value=scene) + monkeypatch.setattr(visualisation, "load_image_rgb", lambda _: scene) + monkeypatch.setattr( + visualisation, "draw_instance_segmentation_points", draw_polygon + ) + + result = visualisation.draw_detection_predictions( + inference_request=SimpleNamespace( + image=None, visualization_stroke_width=1, visualization_labels=False + ), + inference_response=response, + colors={"car": "#ffffff"}, + ) + + assert isinstance(result, bytes) + assert draw_polygon.call_args.kwargs["points"] == [Point(x=25, y=32)] + assert box.points == [Point(x=25, y=32)] diff --git a/tests/inference_sdk/unit_tests/http/test_entities.py b/tests/inference_sdk/unit_tests/http/test_entities.py index 01f1e99186..e591f6b1eb 100644 --- a/tests/inference_sdk/unit_tests/http/test_entities.py +++ b/tests/inference_sdk/unit_tests/http/test_entities.py @@ -1,7 +1,10 @@ +import pytest + from inference_sdk.config import outbound_service_secret from inference_sdk.http.entities import ( CLASSIFICATION_TASK, DEFAULT_IMAGE_EXTENSIONS, + INSTANCE_SEGMENTATION_TASK, OBJECT_DETECTION_TASK, HTTPClientMode, InferenceConfiguration, @@ -282,3 +285,28 @@ def test_to_legacy_call_parameters_context_overrides_explicit_count_inference_tr # then assert parameters["countinference"] is False assert parameters["service_secret"] == "ctx-secret" + + +@pytest.mark.parametrize("client_mode", [HTTPClientMode.V0, HTTPClientMode.V1]) +@pytest.mark.parametrize("opt_in", [None, False, True]) +def test_mask_resolution_opt_in_is_serialized_only_when_selected(client_mode, opt_in): + configuration = InferenceConfiguration(allow_reduced_mask_resolution=opt_in) + + parameters = configuration.to_api_call_parameters( + client_mode=client_mode, task_type=INSTANCE_SEGMENTATION_TASK + ) + + if opt_in is None: + assert "allow_reduced_mask_resolution" not in parameters + else: + assert parameters["allow_reduced_mask_resolution"] is opt_in + + +def test_mask_resolution_opt_in_does_not_leak_to_object_detection(): + configuration = InferenceConfiguration(allow_reduced_mask_resolution=True) + + parameters = configuration.to_api_call_parameters( + client_mode=HTTPClientMode.V1, task_type=OBJECT_DETECTION_TASK + ) + + assert "allow_reduced_mask_resolution" not in parameters diff --git a/tests/inference_sdk/unit_tests/http/utils/test_postprocessing.py b/tests/inference_sdk/unit_tests/http/utils/test_postprocessing.py index b49328db71..edfc7079b6 100644 --- a/tests/inference_sdk/unit_tests/http/utils/test_postprocessing.py +++ b/tests/inference_sdk/unit_tests/http/utils/test_postprocessing.py @@ -690,3 +690,46 @@ def test_filter_model_descriptions_when_input_with_multiple_matches_given() -> N # then assert result == ModelDescription(model_id="some/1", task_type="object-detection") + + +@pytest.mark.parametrize("mask_format", ["polygon", "rle", "empty"]) +def test_client_downsizing_preserves_mask_coordinate_contract(mask_format): + prediction = { + "image": {"width": 301, "height": 199}, + "mask_metadata": { + "coordinate_system": "mask_grid", + "width": 100, + "height": 50, + "scale_x": 3.01, + "scale_y": 3.98, + }, + "predictions": [{"x": 100, "y": 60, "width": 20, "height": 30}], + } + if mask_format == "polygon": + prediction.pop("mask_metadata") + prediction["predictions"][0]["points"] = [{"x": 75, "y": 40}] + elif mask_format == "rle": + prediction["predictions"][0]["rle"] = {"size": [50, 100], "counts": "fixture"} + else: + prediction["predictions"] = [] + + result = adjust_prediction_to_client_scaling_factor(prediction, scaling_factor=0.6) + + assert result["image"] == {"width": 502, "height": 332} + if mask_format == "polygon": + assert "mask_metadata" not in result + else: + assert result["mask_metadata"]["width"] == 100 + assert result["mask_metadata"]["height"] == 50 + assert result["mask_metadata"]["scale_x"] == pytest.approx(5.02) + assert result["mask_metadata"]["scale_y"] == pytest.approx(6.64) + if mask_format != "empty": + assert result["predictions"][0]["x"] == pytest.approx(100 / 0.6) + if mask_format == "polygon": + assert result["predictions"][0]["points"][0]["x"] == pytest.approx(75 / 0.6) + assert result["predictions"][0]["points"][0]["y"] == pytest.approx(40 / 0.6) + if mask_format == "rle": + assert result["predictions"][0]["rle"] == { + "size": [50, 100], + "counts": "fixture", + } diff --git a/tests/workflows/integration_tests/execution/test_instance_segmentation_v5_visualization.py b/tests/workflows/integration_tests/execution/test_instance_segmentation_v4_visualization.py similarity index 92% rename from tests/workflows/integration_tests/execution/test_instance_segmentation_v5_visualization.py rename to tests/workflows/integration_tests/execution/test_instance_segmentation_v4_visualization.py index 03b52959bf..0e6013d0f5 100644 --- a/tests/workflows/integration_tests/execution/test_instance_segmentation_v5_visualization.py +++ b/tests/workflows/integration_tests/execution/test_instance_segmentation_v4_visualization.py @@ -20,12 +20,12 @@ @pytest.mark.skipif( not USE_INFERENCE_MODELS or ENABLE_TENSOR_DATA_REPRESENTATION, - reason="Reduced v5 masks require the non-tensor inference_models path", + reason="Compatibility coverage uses the non-tensor inference_models path", ) @pytest.mark.parametrize( "mode,factor", [("accurate", 1.0), ("tradeoff", 0.5), ("fast", 0.0)] ) -def test_instance_segmentation_v5_with_visualization_blocks( +def test_instance_segmentation_v4_with_visualization_blocks( model_manager: ModelManagerModelsProvider, dogs_image: np.ndarray, mode: str, @@ -38,7 +38,7 @@ def test_instance_segmentation_v5_with_visualization_blocks( "inputs": [{"type": "WorkflowImage", "name": "image"}], "steps": [ { - "type": "roboflow_core/roboflow_instance_segmentation_model@v5", + "type": "roboflow_core/roboflow_instance_segmentation_model@v4", "name": "segment", "image": "$inputs.image", "model_id": "yolov8n-seg-640", @@ -103,10 +103,7 @@ def capture_predictions(**kwargs): assert len(raw_masks) == 2 image_shape = dogs_image.shape[:2] for rle in raw_masks: - if mode == "accurate": - assert tuple(rle["size"]) == image_shape - else: - assert all(size < full for size, full in zip(rle["size"], image_shape)) + assert tuple(rle["size"]) == image_shape result = results[0] detections = result["predictions"] diff --git a/tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_v5_mask_resolution.py b/tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_mask_resolution_compatibility.py similarity index 65% rename from tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_v5_mask_resolution.py rename to tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_mask_resolution_compatibility.py index 1b8892954b..1b25df38b1 100644 --- a/tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_v5_mask_resolution.py +++ b/tests/workflows/unit_tests/core_steps/models/roboflow/instance_segmentation/test_mask_resolution_compatibility.py @@ -1,12 +1,4 @@ -"""Mask-resolution behaviour of the instance-segmentation block versions. - -`mask_decode_mode` and `tradeoff_factor` are declared on every version's -manifest but were discarded by the server until recently. Now that they are -honoured, forwarding them from an older version would silently change the masks -an existing workflow produces, so only `@v5` forwards them. These tests pin -that split, and the manifest parity between a version and its tensor-native -sibling. -""" +"""Existing workflow versions preserve their mask-resolution contracts.""" import re from pathlib import Path @@ -38,15 +30,14 @@ def _blocks_dir() -> Path: BLOCKS_DIR = _blocks_dir() PINNED_VERSIONS = ["v1", "v2", "v3", "v4"] -FORWARDING_VERSIONS = ["v5"] -ALL_VERSIONS = PINNED_VERSIONS + FORWARDING_VERSIONS +ALL_VERSIONS = PINNED_VERSIONS def _source(version: str, tensor: bool) -> str: """Read a block implementation's source. Args: - version: Block version directory stem, such as ``"v5"``. + version: Block version directory stem, such as ``"v4"``. tensor: Whether to read the tensor-native sibling. Returns: @@ -117,7 +108,7 @@ def test_legacy_versions_pin_local_conditionally_and_remote_unconditionally( assert source.count('mask_decode_mode="accurate",') == 1 -@pytest.mark.parametrize("version", PINNED_VERSIONS + FORWARDING_VERSIONS) +@pytest.mark.parametrize("version", PINNED_VERSIONS) def test_tensor_siblings_pin_unconditionally(version: str) -> None: # given # the reason here is downstream rather than the backend: nothing under @@ -130,16 +121,6 @@ def test_tensor_siblings_pin_unconditionally(version: str) -> None: assert "USE_INFERENCE_MODELS" not in source -@pytest.mark.parametrize("version", FORWARDING_VERSIONS) -def test_v5_forwards_what_the_caller_set(version: str) -> None: - # given - source = _source(version, tensor=False) - - # then - assert source.count("mask_decode_mode=mask_decode_mode,") == 4 - assert 'mask_decode_mode="accurate",' not in source - - class TestPinningIsObservedAtTheCallSite: """Count pinned outbound call sites rather than grepping for a string. @@ -175,14 +156,6 @@ def test_every_outbound_call_site_is_pinned( "outbound call sites, expected 2 (local and remote)" ) - def test_v5_non_tensor_has_no_pinned_call_sites(self) -> None: - # given - source = _source("v5", tensor=False) - - # then - assert 'mask_decode_mode="accurate",' not in source - assert source.count("mask_decode_mode=mask_decode_mode,") == 4 - @pytest.mark.parametrize("version", ALL_VERSIONS) @pytest.mark.parametrize("local_backend", [True, False]) @@ -224,64 +197,68 @@ def test_remote_request_preserves_versioned_mask_settings( ) config = client.configure.call_args.kwargs["inference_configuration"] - assert config.mask_decode_mode == ("fast" if version == "v5" else "accurate") - assert config.tradeoff_factor == (0.0 if version == "v5" else 1.0) - assert config.response_mask_format == ("rle" if version in {"v4", "v5"} else None) - - -def test_v5_reduced_rle_output_can_be_decoded_on_original_image(monkeypatch): - import numpy as np - import supervision as sv - from pycocotools import mask as mask_utils - from roboflow_workflows.core_steps.models.roboflow.instance_segmentation import v5 - from roboflow_workflows.core_steps.visualizations.common.utils import ( - ensure_dense_masks, + assert config.mask_decode_mode == "accurate" + assert config.tradeoff_factor == 1.0 + assert not config.allow_reduced_mask_resolution + assert config.response_mask_format == ("rle" if version == "v4" else None) + + +@pytest.mark.parametrize("version", ALL_VERSIONS) +@pytest.mark.parametrize("local_backend", [True, False]) +def test_local_requests_preserve_backend_mask_contract( + monkeypatch, version, local_backend +): + from importlib import import_module + from unittest.mock import MagicMock + + module = import_module( + f"roboflow_workflows.core_steps.models.roboflow.instance_segmentation.{version}" ) - from roboflow_workflows.execution_engine.constants import ( - RLE_MASK_KEY_IN_SV_DETECTIONS, + block_class = getattr( + module, f"RoboflowInstanceSegmentationModelBlock{version.upper()}" ) - - block = object.__new__(v5.RoboflowInstanceSegmentationModelBlockV5) - mask = np.zeros((20, 30), dtype=np.uint8) - mask[5:10, 8:14] = 1 - rle = mask_utils.encode(np.asfortranarray(mask)) - rle["counts"] = rle["counts"].decode("utf-8") - prediction = { - "image": {"height": 200, "width": 300}, - "predictions": [ - { - "x": 110, - "y": 75, - "width": 60, - "height": 50, - "confidence": 0.9, - "class_id": 0, - "class": "car", - "rle": rle, - } - ], - } - monkeypatch.setattr( - v5, - "attach_parents_coordinates_to_batch_of_sv_detections", - lambda images, predictions: predictions, + block = block_class( + model_manager=MagicMock(), + api_key=None, + step_execution_mode=module.StepExecutionMode.LOCAL, ) + block._post_process_result = MagicMock(return_value=[]) + monkeypatch.setattr(module, "USE_INFERENCE_MODELS", local_backend) - output = block._post_process_result( - images=[], predictions=[prediction], class_filter=None, model_id="model/1" + block.run_locally( + images=[], + model_id="model/1", + class_agnostic_nms=False, + class_filter=None, + confidence=0.5, + iou_threshold=0.5, + max_detections=10, + max_candidates=20, + mask_decode_mode="fast", + tradeoff_factor=0.0, + disable_active_learning=True, + active_learning_target_dataset=None, + **( + {"enforce_dense_masks_in_inference_models": False} + if version != "v4" + else {} + ), ) - detections = output[0]["predictions"] - encoded = detections.data[RLE_MASK_KEY_IN_SV_DETECTIONS][0] - decoded = mask_utils.decode(encoded).astype(bool) - assert decoded.shape == (200, 300) - np.testing.assert_array_equal(decoded, detections.mask[0]) - assert decoded[50:100, 80:140].all() - assert decoded.sum() == 50 * 60 - detections.mask = None - detections = ensure_dense_masks(detections) - annotated = sv.MaskAnnotator().annotate( - scene=np.zeros((200, 300, 3), dtype=np.uint8), detections=detections - ) - assert annotated.any() - assert rle["size"] == [20, 30] + request = block._model_manager.run_instance_segmentation.call_args.kwargs + assert request["mask_decode_mode"] == ("accurate" if local_backend else "fast") + assert request["tradeoff_factor"] == (1.0 if local_backend else 0.0) + assert not request.get("allow_reduced_mask_resolution", False) + + +def test_v5_is_not_registered(): + from roboflow_workflows.core_steps.loader import load_blocks + + identifiers = { + identifier + for block in load_blocks() + for identifier in block.get_manifest().model_fields["type"].annotation.__args__ + } + + assert "roboflow_core/roboflow_instance_segmentation_model@v4" in identifiers + assert "roboflow_core/roboflow_instance_segmentation_model@v5" not in identifiers diff --git a/workflows/CHANGELOG.md b/workflows/CHANGELOG.md index 7b8c343146..dc972497cf 100644 --- a/workflows/CHANGELOG.md +++ b/workflows/CHANGELOG.md @@ -16,42 +16,30 @@ for contributor and maintainer responsibilities. ## Unreleased -### Added +### Changed -- `roboflow_core/roboflow_instance_segmentation_model@v5`. Identical to `@v4` - except that it forwards `mask_decode_mode` and `tradeoff_factor` to the - server, which now honours them. Versions up to `@v4` pin the outbound request - to `accurate` / `1.0` on the local `inference_models` backend. Local legacy - execution continues to honour these fields when `USE_INFERENCE_MODELS=False`. - Remote `@v1`–`@v4` requests always use `accurate` / `1.0`, because the local - backend setting does not identify the remote server. This changes remote - legacy workflows that previously requested reduced masks; use non-tensor - `@v5` to opt into adjustable mask resolution. - -- `ModelsConfiguration.use_inference_models`, defaulting to `True`. Hosts that - serve instance segmentation through the legacy backend must set it to `False` - so local block versions up to `@v4` keep forwarding `mask_decode_mode` and - `tradeoff_factor`, which that backend honours. The server sets it from - `inference.core.env.USE_INFERENCE_MODELS`; the value cannot be read from the - environment here, since this package must not import `inference.*`. - - The tensor-native siblings of every version, `@v5` included, are pinned - unconditionally - the reason there is downstream rather than the backend. - Nothing under `roboflow_workflows/` reads `InstanceDetections.mask_size`, and - the sites that rebuild `InstancesRLEMasks` drop it, so a reduced grid would - be reinterpreted as image-sized downstream. Reduced mask resolution is - therefore available in non-tensor mode only. **Full tensor support will come - in a separate PR**, including propagation of mask-grid metadata through the - tensor pipeline. Until then, tensor mode ignores `mask_decode_mode` and - `tradeoff_factor` and always requests `accurate` / `1.0`. +- Instance segmentation `@v1`–`@v4` preserve image-resolution masks when using + the local `inference_models` backend. Local legacy execution continues to + honor `mask_decode_mode` and `tradeoff_factor` when `USE_INFERENCE_MODELS=False`. + Remote calls always request `accurate` / `1.0`, since the local backend setting + does not identify the remote server. Remote legacy workflows that previously + requested fast/tradeoff masks now receive accurate masks. `@v4` retains its + RLE response format. +- The new instance-segmentation `@v5` block is deferred to a separate PR together + with reduced-grid Supervision conversion and downstream block support. + **Full tensor support will come in a separate PR.** Existing tensor variants + remain pinned to `accurate` / `1.0`. -### Fixed +### Added + +- Optional `mask_metadata` on instance-segmentation response DTOs describes the + encoded mask grid and coordinate mapping for opted-in RLE HTTP responses. + Polygon points and bounding boxes retain image coordinates. Existing workflow + blocks do not opt in and retain their image-space contract. -- Instance segmentation `@v4` remote execution retains its RLE response format - while pinning mask resolution, preserving holes and disconnected mask regions. -- Non-tensor instance segmentation `@v5` re-encodes reduced RLE masks at the - image grid after conversion to Supervision, so downstream blocks receive - consistent dense and RLE representations. +- `ModelsConfiguration.use_inference_models`, defaulting to `True`. Hosts using + the legacy backend must set it to `False` to retain local legacy decoding. + The server sets it from `inference.core.env.USE_INFERENCE_MODELS`. ## `0.2.4` diff --git a/workflows/roboflow_workflows/core_steps/common/inference_response_entities.py b/workflows/roboflow_workflows/core_steps/common/inference_response_entities.py index ada13086b7..327a336ad7 100644 --- a/workflows/roboflow_workflows/core_steps/common/inference_response_entities.py +++ b/workflows/roboflow_workflows/core_steps/common/inference_response_entities.py @@ -13,7 +13,7 @@ """ import base64 -from typing import Any, List, Optional, Union +from typing import Any, List, Literal, Optional, Union from pydantic import BaseModel, ConfigDict, Field, field_serializer from roboflow_workflows.core_steps.common.segmentation_entities import ( @@ -121,6 +121,31 @@ def serialize_visualisation(self, visualization: Optional[Any]) -> Optional[str] return base64.b64encode(visualization).decode("utf-8") +class MaskCoordinateMetadata(BaseModel): + """Describe the mask grid and its mapping to the response image. + + Attributes: + coordinate_system (str): Coordinate frame for encoded RLE masks. + width (int): Width of the encoded mask grid. + height (int): Height of the encoded mask grid. + scale_x (float): Multiply mask x coordinates by this to obtain image x. + scale_y (float): Multiply mask y coordinates by this to obtain image y. + """ + + coordinate_system: Literal["mask_grid"] = Field( + default="mask_grid", + description="RLE masks use this grid; polygon points use image coordinates.", + ) + width: int = Field(gt=0, description="Mask-grid width in pixels.", examples=[160]) + height: int = Field(gt=0, description="Mask-grid height in pixels.", examples=[107]) + scale_x: float = Field( + gt=0, description="Image width divided by mask width.", examples=[4.0] + ) + scale_y: float = Field( + gt=0, description="Image height divided by mask height.", examples=[4.0] + ) + + class InstanceSegmentationInferenceResponse( CvInferenceResponse, WithVisualizationResponse ): @@ -136,3 +161,9 @@ class InstanceSegmentationInferenceResponse( predictions: List[ Union[InstanceSegmentationPrediction, InstanceSegmentationRLEPrediction] ] + mask_metadata: Optional[MaskCoordinateMetadata] = Field( + default=None, + description="Present for opted-in RLE responses whose encoded mask grid " + "differs from the image. Describes the RLE grid and its mapping to image " + "coordinates. Bounding boxes and polygon points always use image coordinates.", + ) diff --git a/workflows/roboflow_workflows/core_steps/loader.py b/workflows/roboflow_workflows/core_steps/loader.py index 588c8a3267..8572cede83 100644 --- a/workflows/roboflow_workflows/core_steps/loader.py +++ b/workflows/roboflow_workflows/core_steps/loader.py @@ -735,15 +735,6 @@ RoboflowInstanceSegmentationModelBlockV4, ) -if ENABLE_TENSOR_DATA_REPRESENTATION: - from roboflow_workflows.core_steps.models.roboflow.instance_segmentation.v5_tensor import ( - RoboflowInstanceSegmentationModelBlockV5, - ) -else: - from roboflow_workflows.core_steps.models.roboflow.instance_segmentation.v5 import ( - RoboflowInstanceSegmentationModelBlockV5, - ) - if not ENABLE_TENSOR_DATA_REPRESENTATION: from roboflow_workflows.core_steps.models.foundation.segment_anything3_interactive.v1 import ( SegmentAnything3InteractiveBlockV1, @@ -1862,7 +1853,6 @@ def load_blocks() -> List[Type[WorkflowBlock]]: RoboflowInstanceSegmentationModelBlockV2, RoboflowInstanceSegmentationModelBlockV3, RoboflowInstanceSegmentationModelBlockV4, - RoboflowInstanceSegmentationModelBlockV5, RoboflowSemanticSegmentationModelBlockV1, RoboflowSemanticSegmentationModelBlockV2, RoboflowKeypointDetectionModelBlockV2, diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5.py deleted file mode 100644 index b262e8df19..0000000000 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5.py +++ /dev/null @@ -1,481 +0,0 @@ -from typing import List, Literal, Optional, Type, Union - -import numpy as np -from pydantic import ConfigDict, Field, PositiveInt, model_validator -from roboflow_workflows.core_steps.common.entities import StepExecutionMode -from roboflow_workflows.core_steps.common.utils import ( - attach_parents_coordinates_to_batch_of_sv_detections, - attach_prediction_type_info_to_sv_detections_batch, - convert_inference_detections_batch_to_sv_detections, - filter_out_unwanted_classes_from_sv_detections_batch, -) -from roboflow_workflows.environment import ( - HOSTED_INSTANCE_SEGMENTATION_URL, - LOCAL_INFERENCE_API_URL, - WORKFLOWS_REMOTE_API_KEY_TRANSPORT, - WORKFLOWS_REMOTE_API_TARGET, - WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, - WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, -) -from roboflow_workflows.execution_engine.constants import ( - INFERENCE_ID_KEY, - RLE_MASK_KEY_IN_SV_DETECTIONS, -) -from roboflow_workflows.execution_engine.entities.base import ( - Batch, - OutputDefinition, - WorkflowImageData, -) -from roboflow_workflows.execution_engine.entities.types import ( - BOOLEAN_KIND, - FLOAT_ZERO_TO_ONE_KIND, - IMAGE_KIND, - INFERENCE_ID_KIND, - INSTANCE_SEGMENTATION_PREDICTION_KIND, - INTEGER_KIND, - LIST_OF_VALUES_KIND, - RLE_INSTANCE_SEGMENTATION_PREDICTION_KIND, - ROBOFLOW_MODEL_ID_KIND, - ROBOFLOW_PROJECT_KIND, - STRING_KIND, - FloatZeroToOne, - ImageInputField, - RoboflowModelField, - Selector, -) -from roboflow_workflows.execution_engine.entities.workload import ( - Discovery, - RuntimeRestriction, - WorkOperation, -) -from roboflow_workflows.prototypes.block import ( - BlockResult, - DependentResource, - WorkflowBlock, - WorkflowBlockManifest, - roboflow_platform_model, - roboflow_platform_project, -) -from roboflow_workflows.prototypes.models_provider import ModelsProvider - -from inference_sdk import InferenceConfiguration, InferenceHTTPClient - -LONG_DESCRIPTION = """ -Run inference on an instance segmentation model hosted on or uploaded to Roboflow. - -You can query any model that is private to your account, or any public model available -on [Roboflow Universe](https://universe.roboflow.com). - -You will need to set your Roboflow API key in your Inference environment to use this -block. To learn more about setting your Roboflow API key, [refer to the Inference -documentation](https://inference.roboflow.com/quickstart/configure_api_key/). - -This version of block introduces breaking change in behaviour of mask construction - it uses -`rle` format instead `polygon` making it possible to retrieve -shapes of any kind from remote server. -""" - - -class BlockManifest(WorkflowBlockManifest): - model_config = ConfigDict( - json_schema_extra={ - "name": "Instance Segmentation Model", - "version": "v5", - "short_description": "Predict the shape, size, and location of objects.", - "long_description": LONG_DESCRIPTION, - "license": "Apache-2.0", - "block_type": "model", - "search_keywords": ["yolo", "rfdetr", "rf-detr"], - "ui_manifest": { - "section": "model", - "icon": "far fa-chart-network", - "blockPriority": 1, - "inference": True, - "popular": True, - }, - }, - protected_namespaces=(), - ) - type: Literal["roboflow_core/roboflow_instance_segmentation_model@v5"] - images: Selector(kind=[IMAGE_KIND]) = ImageInputField - model_id: Union[Selector(kind=[ROBOFLOW_MODEL_ID_KIND]), str] = RoboflowModelField - confidence_mode: Union[ - Literal["best", "default", "custom"], - Selector(kind=[STRING_KIND]), - ] = Field( - default="best", - description="How confidence thresholds are determined.", - json_schema_extra={ - "always_visible": True, - "values_metadata": { - "best": { - "name": "Best (Recommended)", - "description": "Use F1-optimal per-class thresholds from model evaluation.", - }, - "default": { - "name": "Default", - "description": "Use the model's built-in default threshold.", - }, - "custom": { - "name": "Custom", - "description": "Specify a custom confidence threshold.", - }, - }, - }, - ) - custom_confidence: Union[ - Optional[FloatZeroToOne], - Selector(kind=[FLOAT_ZERO_TO_ONE_KIND]), - ] = Field( - default=0.4, - description="Custom confidence threshold for predictions.", - examples=[0.3, "$inputs.confidence_threshold"], - json_schema_extra={ - "relevant_for": { - "confidence_mode": {"values": ["custom"], "required": True}, - }, - }, - ) - class_filter: Union[Optional[List[str]], Selector(kind=[LIST_OF_VALUES_KIND])] = ( - Field( - default=None, - description="List of accepted classes. Classes must exist in the model's training set.", - examples=[["a", "b", "c"], "$inputs.class_filter"], - ) - ) - iou_threshold: Union[ - FloatZeroToOne, - Selector(kind=[FLOAT_ZERO_TO_ONE_KIND]), - ] = Field( - default=0.3, - description="Minimum overlap threshold between boxes to combine them into a single detection, used in NMS. [Learn more](https://blog.roboflow.com/how-to-code-non-maximum-suppression-nms-in-plain-numpy/).", - examples=[0.4, "$inputs.iou_threshold"], - ) - max_detections: Union[PositiveInt, Selector(kind=[INTEGER_KIND])] = Field( - default=300, - description="Maximum number of detections to return.", - examples=[300, "$inputs.max_detections"], - ) - class_agnostic_nms: Union[bool, Selector(kind=[BOOLEAN_KIND])] = Field( - default=False, - description="Boolean flag to specify if NMS is to be used in class-agnostic mode.", - examples=[True, "$inputs.class_agnostic_nms"], - ) - max_candidates: Union[PositiveInt, Selector(kind=[INTEGER_KIND])] = Field( - default=3000, - description="Maximum number of candidates as NMS input to be taken into account.", - examples=[3000, "$inputs.max_candidates"], - ) - mask_decode_mode: Union[ - Literal["accurate", "tradeoff", "fast"], - Selector(kind=[STRING_KIND]), - ] = Field( - default="accurate", - description="Parameter of mask decoding in prediction post-processing.", - examples=["accurate", "$inputs.mask_decode_mode"], - ) - tradeoff_factor: Union[ - FloatZeroToOne, - Selector(kind=[FLOAT_ZERO_TO_ONE_KIND]), - ] = Field( - default=0.0, - description="Post-processing parameter to dictate tradeoff between fast and accurate. 0.0 keeps masks on the model's own grid, 1.0 resizes them to the image. Note that a lower value does not guarantee smaller masks: when the image is smaller than the model grid the interpolation runs the other way.", - examples=[0.3, "$inputs.tradeoff_factor"], - ) - disable_active_learning: Union[bool, Selector(kind=[BOOLEAN_KIND])] = Field( - default=True, - description="Boolean flag to disable project-level active learning for this block.", - examples=[True, "$inputs.disable_active_learning"], - ) - active_learning_target_dataset: Union[ - Selector(kind=[ROBOFLOW_PROJECT_KIND]), Optional[str] - ] = Field( - default=None, - description="Target dataset for active learning, if enabled.", - examples=["my_project", "$inputs.al_target_project"], - ) - - @model_validator(mode="after") - def validate(self) -> "BlockManifest": - if self.confidence_mode == "custom" and self.custom_confidence is None: - raise ValueError( - "`custom_confidence` is required when `confidence_mode` is 'custom'" - ) - return self - - @classmethod - def get_compatible_task_types(cls) -> Optional[List[str]]: - return ["instance-segmentation"] - - def discover_dependent_resources(self) -> Optional[List[DependentResource]]: - resources = [roboflow_platform_model(model_id=self.model_id)] - if self.disable_active_learning is True: - # Active learning literally disabled (the default) — the target - # project is dead configuration, not a dependency. - return resources - if self.active_learning_target_dataset is not None: - resources.append( - roboflow_platform_project( - project_url=self.active_learning_target_dataset - ) - ) - return resources - - @classmethod - def get_parameters_accepting_batches(cls) -> List[str]: - return ["images"] - - @classmethod - def describe_outputs(cls) -> List[OutputDefinition]: - return [ - OutputDefinition(name=INFERENCE_ID_KEY, kind=[INFERENCE_ID_KIND]), - OutputDefinition( - name="predictions", - kind=[ - RLE_INSTANCE_SEGMENTATION_PREDICTION_KIND, - INSTANCE_SEGMENTATION_PREDICTION_KIND, - ], - ), - OutputDefinition(name="model_id", kind=[ROBOFLOW_MODEL_ID_KIND]), - ] - - @classmethod - def get_execution_engine_compatibility(cls) -> Optional[str]: - return ">=1.3.0,<2.0.0" - - def discover_work_operations(self) -> List[WorkOperation]: - return [WorkOperation.MODEL_INFERENCE] - - def get_actual_restrictions( - self, *, ignore_environment_restrictions: bool = False - ) -> Discovery[RuntimeRestriction]: - return Discovery[RuntimeRestriction]( - items=[], complete=True, unknown_reasons=[] - ) - - -class RoboflowInstanceSegmentationModelBlockV5(WorkflowBlock): - - def __init__( - self, - model_manager: ModelsProvider, - api_key: Optional[str], - step_execution_mode: StepExecutionMode, - ): - self._model_manager = model_manager - self._api_key = api_key - self._step_execution_mode = step_execution_mode - - @classmethod - def get_init_parameters(cls) -> List[str]: - return ["model_manager", "api_key", "step_execution_mode"] - - @classmethod - def get_manifest(cls) -> Type[WorkflowBlockManifest]: - return BlockManifest - - def run( - self, - images: Batch[WorkflowImageData], - model_id: str, - confidence_mode: str, - custom_confidence: Optional[float], - class_agnostic_nms: Optional[bool], - class_filter: Optional[List[str]], - iou_threshold: Optional[float], - max_detections: Optional[int], - max_candidates: Optional[int], - mask_decode_mode: Literal["accurate", "tradeoff", "fast"], - tradeoff_factor: Optional[float], - disable_active_learning: Optional[bool], - active_learning_target_dataset: Optional[str], - ) -> BlockResult: - confidence = ( - custom_confidence if confidence_mode == "custom" else confidence_mode - ) - if self._step_execution_mode is StepExecutionMode.LOCAL: - return self.run_locally( - images=images, - model_id=model_id, - class_agnostic_nms=class_agnostic_nms, - class_filter=class_filter, - confidence=confidence, - iou_threshold=iou_threshold, - max_detections=max_detections, - max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, - disable_active_learning=disable_active_learning, - active_learning_target_dataset=active_learning_target_dataset, - ) - elif self._step_execution_mode is StepExecutionMode.REMOTE: - return self.run_remotely( - images=images, - model_id=model_id, - class_agnostic_nms=class_agnostic_nms, - class_filter=class_filter, - confidence=confidence, - iou_threshold=iou_threshold, - max_detections=max_detections, - max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, - disable_active_learning=disable_active_learning, - active_learning_target_dataset=active_learning_target_dataset, - ) - else: - raise ValueError( - f"Unknown step execution mode: {self._step_execution_mode}" - ) - - def run_locally( - self, - images: Batch[WorkflowImageData], - model_id: str, - class_agnostic_nms: Optional[bool], - class_filter: Optional[List[str]], - confidence: Union[None, float, Literal["best", "default"]], - iou_threshold: Optional[float], - max_detections: Optional[int], - max_candidates: Optional[int], - mask_decode_mode: Literal["accurate", "tradeoff", "fast"], - tradeoff_factor: Optional[float], - disable_active_learning: Optional[bool], - active_learning_target_dataset: Optional[str], - ) -> BlockResult: - inference_images = [i.to_inference_format(numpy_preferred=True) for i in images] - self._model_manager.add_model( - model_id=model_id, - api_key=self._api_key, - ) - predictions = self._model_manager.run_instance_segmentation( - model_id=model_id, - images=inference_images, - api_key=self._api_key, - class_agnostic_nms=class_agnostic_nms, - class_filter=class_filter, - confidence=confidence, - iou_threshold=iou_threshold, - max_detections=max_detections, - max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, - response_mask_format="rle", - disable_active_learning=disable_active_learning, - active_learning_target_dataset=active_learning_target_dataset, - ) - return self._post_process_result( - images=images, - predictions=predictions, - class_filter=class_filter, - model_id=model_id, - ) - - def run_remotely( - self, - images: Batch[WorkflowImageData], - model_id: str, - class_agnostic_nms: Optional[bool], - class_filter: Optional[List[str]], - confidence: Union[None, float, Literal["best", "default"]], - iou_threshold: Optional[float], - max_detections: Optional[int], - max_candidates: Optional[int], - mask_decode_mode: Literal["accurate", "tradeoff", "fast"], - tradeoff_factor: Optional[float], - disable_active_learning: Optional[bool], - active_learning_target_dataset: Optional[str], - ) -> BlockResult: - api_url = ( - LOCAL_INFERENCE_API_URL - if WORKFLOWS_REMOTE_API_TARGET != "hosted" - else HOSTED_INSTANCE_SEGMENTATION_URL - ) - client = InferenceHTTPClient( - api_url=api_url, - api_key=self._api_key, - ) - if WORKFLOWS_REMOTE_API_TARGET == "hosted": - client.select_api_v0() - client_config = InferenceConfiguration( - api_key_transport=WORKFLOWS_REMOTE_API_KEY_TRANSPORT, - disable_active_learning=disable_active_learning, - active_learning_target_dataset=active_learning_target_dataset, - class_agnostic_nms=class_agnostic_nms, - class_filter=class_filter, - confidence_threshold=confidence, - iou_threshold=iou_threshold, - max_detections=max_detections, - max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, - response_mask_format="rle", - max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, - max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, - source="workflow-execution", - ) - client.configure(inference_configuration=client_config) - inference_images = [i.base64_image for i in images] - predictions = client.infer( - inference_input=inference_images, - model_id=model_id, - ) - if not isinstance(predictions, list): - predictions = [predictions] - return self._post_process_result( - images=images, - predictions=predictions, - class_filter=class_filter, - model_id=model_id, - ) - - def _post_process_result( - self, - images: Batch[WorkflowImageData], - predictions: List[dict], - class_filter: Optional[List[str]], - model_id: str, - ) -> BlockResult: - inference_ids = [p.get(INFERENCE_ID_KEY, None) for p in predictions] - predictions = convert_inference_detections_batch_to_sv_detections(predictions) - for detections in predictions: - rle_masks = detections.data.get(RLE_MASK_KEY_IN_SV_DETECTIONS) - if rle_masks is None or detections.mask is None: - continue - - # Supervision restores dense masks to the image grid. Keep the RLE - # copy consistent for downstream blocks that decode it directly. - for index, rle_mask in enumerate(rle_masks): - if rle_mask is None or tuple(rle_mask["size"]) == tuple( - detections.mask.shape[1:] - ): - continue - - from pycocotools import mask as mask_utils - - normalized = mask_utils.encode( - np.asfortranarray(detections.mask[index], dtype=np.uint8) - ) - normalized["counts"] = normalized["counts"].decode("utf-8") - rle_masks[index] = normalized - - predictions = attach_prediction_type_info_to_sv_detections_batch( - predictions=predictions, - prediction_type="instance-segmentation", - ) - predictions = filter_out_unwanted_classes_from_sv_detections_batch( - predictions=predictions, - classes_to_accept=class_filter, - ) - predictions = attach_parents_coordinates_to_batch_of_sv_detections( - images=images, - predictions=predictions, - ) - return [ - { - "inference_id": inference_id, - "predictions": prediction, - "model_id": model_id, - } - for inference_id, prediction in zip(inference_ids, predictions) - ] diff --git a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py b/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py deleted file mode 100644 index cf5c9c2441..0000000000 --- a/workflows/roboflow_workflows/core_steps/models/roboflow/instance_segmentation/v5_tensor.py +++ /dev/null @@ -1,744 +0,0 @@ -"""Tensor-native sibling of `roboflow_core/roboflow_instance_segmentation_model@v5`. - -Under ENABLE_TENSOR_DATA_REPRESENTATION this block emits a native -``inference_models.InstanceDetections`` (torch tensors on -``WORKFLOWS_IMAGE_TENSOR_DEVICE``; masks dense or RLE) under -``TENSOR_NATIVE_INSTANCE_SEGMENTATION_PREDICTION_KIND`` / -``TENSOR_NATIVE_RLE_INSTANCE_SEGMENTATION_PREDICTION_KIND`` instead of -``sv.Detections``. - -- LOCAL: ``ModelsProvider.run_tensor_native_inference`` returns - ``List[InstanceDetections]`` straight from the adapter. The mask carrier (dense - ``torch.Tensor`` vs ``InstancesRLEMasks``) is adapter-decided; both are handled - downstream by the helpers and the tensor serialiser. The block applies - ``class_filter`` natively (the adapter/model does NOT read it on this path) and - attaches the producer contract (``image_metadata[class_names]`` + per-box - ``detection_id``) the tensor serialiser requires, via - ``attach_native_detection_metadata`` (it preserves masks + per-box metadata). -- REMOTE: standard inference instance-seg prediction dicts are rebuilt into a - native ``InstanceDetections`` via the in-file - ``_native_instance_detections_from_inference_predictions`` converter (never - ``sv.Detections``). The block requests ``response_mask_format="rle"`` and carries - RLE masks when the server honours it; it degrades gracefully to a dense mask - rasterised from polygon ``points`` when the server ignores that parameter - (older/alternative servers), matching the numpy flag-OFF polygon path. - -This block creates ONLY this file; it reuses the already-registered tensor -serialiser for ``instance_segmentation_prediction`` / -``rle_instance_segmentation_prediction`` (``serialise_sv_detections`` already -handles ``InstanceDetections``, dense or RLE). The numpy sibling lives in -``.../instance_segmentation/v4.py``; this manifest is identical except the output -kinds. -""" - -import uuid -from typing import Dict, List, Literal, Optional, Type, Union - -import numpy as np -import supervision as sv -import torch -from pydantic import ConfigDict, Field, PositiveInt, model_validator -from roboflow_workflows.core_steps.common.entities import StepExecutionMode -from roboflow_workflows.core_steps.common.tensor_native import ( - attach_native_detection_metadata, - build_native_image_metadata, - take_prediction_by_mask, -) -from roboflow_workflows.environment import ( - HOSTED_INSTANCE_SEGMENTATION_URL, - LOCAL_INFERENCE_API_URL, - WORKFLOWS_ENFORCE_DENSE_INSTANCE_MASKS, - WORKFLOWS_IMAGE_TENSOR_DEVICE, - WORKFLOWS_REMOTE_API_KEY_TRANSPORT, - WORKFLOWS_REMOTE_API_TARGET, - WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, - WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, -) -from roboflow_workflows.execution_engine.constants import ( - CLASS_ID_KEY, - CLASS_NAME_KEY, - CONFIDENCE_KEY, - DETECTION_ID_KEY, - HEIGHT_KEY, - INFERENCE_ID_KEY, - POLYGON_KEY, - WIDTH_KEY, - X_KEY, - Y_KEY, -) -from roboflow_workflows.execution_engine.entities.base import ( - Batch, - OutputDefinition, - WorkflowImageData, -) -from roboflow_workflows.execution_engine.entities.tensor_native_types import ( - TENSOR_NATIVE_INSTANCE_SEGMENTATION_PREDICTION_KIND, - TENSOR_NATIVE_RLE_INSTANCE_SEGMENTATION_PREDICTION_KIND, -) -from roboflow_workflows.execution_engine.entities.types import ( - BOOLEAN_KIND, - FLOAT_ZERO_TO_ONE_KIND, - IMAGE_KIND, - INFERENCE_ID_KIND, - INTEGER_KIND, - LIST_OF_VALUES_KIND, - ROBOFLOW_MODEL_ID_KIND, - ROBOFLOW_PROJECT_KIND, - STRING_KIND, - FloatZeroToOne, - ImageInputField, - RoboflowModelField, - Selector, -) -from roboflow_workflows.execution_engine.entities.workload import ( - Discovery, - RuntimeRestriction, - WorkOperation, -) -from roboflow_workflows.prototypes.block import ( - BlockResult, - DependentResource, - WorkflowBlock, - WorkflowBlockManifest, - roboflow_platform_model, - roboflow_platform_project, -) -from roboflow_workflows.prototypes.models_provider import ModelsProvider - -from inference_models.models.base.instance_segmentation import InstanceDetections -from inference_models.models.base.types import InstancesRLEMasks -from inference_sdk import InferenceConfiguration, InferenceHTTPClient - -PREDICTION_TYPE = "instance-segmentation" - -# Keys carrying the RLE mask in standard inference instance-seg prediction dicts. -# `InstanceSegmentationRLEPrediction` declares the mask under `rle`; some response -# paths use `rle_mask` (mirrors the numpy converter's `d.get("rle_mask") or d.get("rle")`). -RLE_MASK_KEYS = ("rle_mask", "rle") - -LONG_DESCRIPTION = """ -Run inference on an instance segmentation model hosted on or uploaded to Roboflow. - -You can query any model that is private to your account, or any public model available -on [Roboflow Universe](https://universe.roboflow.com). - -You will need to set your Roboflow API key in your Inference environment to use this -block. To learn more about setting your Roboflow API key, [refer to the Inference -documentation](https://inference.roboflow.com/quickstart/configure_api_key/). - -This version of block introduces breaking change in behaviour of mask construction - it uses -`rle` format instead `polygon` making it possible to retrieve -shapes of any kind from remote server. -""" - - -class BlockManifest(WorkflowBlockManifest): - model_config = ConfigDict( - json_schema_extra={ - "name": "Instance Segmentation Model", - "version": "v5", - "short_description": "Predict the shape, size, and location of objects.", - "long_description": LONG_DESCRIPTION, - "license": "Apache-2.0", - "block_type": "model", - "search_keywords": ["yolo", "rfdetr", "rf-detr"], - "ui_manifest": { - "section": "model", - "icon": "far fa-chart-network", - "blockPriority": 1, - "inference": True, - "popular": True, - }, - }, - protected_namespaces=(), - ) - type: Literal["roboflow_core/roboflow_instance_segmentation_model@v5"] - images: Selector(kind=[IMAGE_KIND]) = ImageInputField - model_id: Union[Selector(kind=[ROBOFLOW_MODEL_ID_KIND]), str] = RoboflowModelField - confidence_mode: Union[ - Literal["best", "default", "custom"], - Selector(kind=[STRING_KIND]), - ] = Field( - default="best", - description="How confidence thresholds are determined.", - json_schema_extra={ - "always_visible": True, - "values_metadata": { - "best": { - "name": "Best (Recommended)", - "description": "Use F1-optimal per-class thresholds from model evaluation.", - }, - "default": { - "name": "Default", - "description": "Use the model's built-in default threshold.", - }, - "custom": { - "name": "Custom", - "description": "Specify a custom confidence threshold.", - }, - }, - }, - ) - custom_confidence: Union[ - Optional[FloatZeroToOne], - Selector(kind=[FLOAT_ZERO_TO_ONE_KIND]), - ] = Field( - default=0.4, - description="Custom confidence threshold for predictions.", - examples=[0.3, "$inputs.confidence_threshold"], - json_schema_extra={ - "relevant_for": { - "confidence_mode": {"values": ["custom"], "required": True}, - }, - }, - ) - class_filter: Union[Optional[List[str]], Selector(kind=[LIST_OF_VALUES_KIND])] = ( - Field( - default=None, - description="List of accepted classes. Classes must exist in the model's training set.", - examples=[["a", "b", "c"], "$inputs.class_filter"], - ) - ) - iou_threshold: Union[ - FloatZeroToOne, - Selector(kind=[FLOAT_ZERO_TO_ONE_KIND]), - ] = Field( - default=0.3, - description="Minimum overlap threshold between boxes to combine them into a single detection, used in NMS. [Learn more](https://blog.roboflow.com/how-to-code-non-maximum-suppression-nms-in-plain-numpy/).", - examples=[0.4, "$inputs.iou_threshold"], - ) - max_detections: Union[PositiveInt, Selector(kind=[INTEGER_KIND])] = Field( - default=300, - description="Maximum number of detections to return.", - examples=[300, "$inputs.max_detections"], - ) - class_agnostic_nms: Union[bool, Selector(kind=[BOOLEAN_KIND])] = Field( - default=False, - description="Boolean flag to specify if NMS is to be used in class-agnostic mode.", - examples=[True, "$inputs.class_agnostic_nms"], - ) - max_candidates: Union[PositiveInt, Selector(kind=[INTEGER_KIND])] = Field( - default=3000, - description="Maximum number of candidates as NMS input to be taken into account.", - examples=[3000, "$inputs.max_candidates"], - ) - mask_decode_mode: Union[ - Literal["accurate", "tradeoff", "fast"], - Selector(kind=[STRING_KIND]), - ] = Field( - default="accurate", - description="Tensor mode currently ignores this setting and always uses accurate decoding with full-resolution masks. Full tensor support will come in a separate PR.", - examples=["accurate", "$inputs.mask_decode_mode"], - ) - tradeoff_factor: Union[ - FloatZeroToOne, - Selector(kind=[FLOAT_ZERO_TO_ONE_KIND]), - ] = Field( - default=0.0, - description="Tensor mode currently ignores this setting and always uses accurate / 1.0 (full-resolution masks). Reduced-resolution tensor masks are not yet supported. Post-processing parameter to dictate tradeoff between fast and accurate. 0.0 keeps masks on the model's own grid, 1.0 resizes them to the image. Note that a lower value does not guarantee smaller masks: when the image is smaller than the model grid the interpolation runs the other way.", - examples=[0.3, "$inputs.tradeoff_factor"], - ) - disable_active_learning: Union[bool, Selector(kind=[BOOLEAN_KIND])] = Field( - default=True, - description="Boolean flag to disable project-level active learning for this block.", - examples=[True, "$inputs.disable_active_learning"], - ) - active_learning_target_dataset: Union[ - Selector(kind=[ROBOFLOW_PROJECT_KIND]), Optional[str] - ] = Field( - default=None, - description="Target dataset for active learning, if enabled.", - examples=["my_project", "$inputs.al_target_project"], - ) - - @model_validator(mode="after") - def validate(self) -> "BlockManifest": - if self.confidence_mode == "custom" and self.custom_confidence is None: - raise ValueError( - "`custom_confidence` is required when `confidence_mode` is 'custom'" - ) - return self - - @classmethod - def get_compatible_task_types(cls) -> Optional[List[str]]: - return ["instance-segmentation"] - - def discover_dependent_resources(self) -> Optional[List[DependentResource]]: - resources = [roboflow_platform_model(model_id=self.model_id)] - if self.disable_active_learning is True: - # Active learning literally disabled (the default) — the target - # project is dead configuration, not a dependency. - return resources - if self.active_learning_target_dataset is not None: - resources.append( - roboflow_platform_project( - project_url=self.active_learning_target_dataset - ) - ) - return resources - - @classmethod - def get_parameters_accepting_batches(cls) -> List[str]: - return ["images"] - - @classmethod - def describe_outputs(cls) -> List[OutputDefinition]: - return [ - OutputDefinition(name=INFERENCE_ID_KEY, kind=[INFERENCE_ID_KIND]), - OutputDefinition( - name="predictions", - kind=[ - TENSOR_NATIVE_RLE_INSTANCE_SEGMENTATION_PREDICTION_KIND, - TENSOR_NATIVE_INSTANCE_SEGMENTATION_PREDICTION_KIND, - ], - ), - OutputDefinition(name="model_id", kind=[ROBOFLOW_MODEL_ID_KIND]), - ] - - @classmethod - def get_execution_engine_compatibility(cls) -> Optional[str]: - return ">=1.3.0,<2.0.0" - - def discover_work_operations(self) -> List[WorkOperation]: - return [WorkOperation.MODEL_INFERENCE] - - def get_actual_restrictions( - self, *, ignore_environment_restrictions: bool = False - ) -> Discovery[RuntimeRestriction]: - return Discovery[RuntimeRestriction]( - items=[], complete=True, unknown_reasons=[] - ) - - -class RoboflowInstanceSegmentationModelBlockV5(WorkflowBlock): - - def __init__( - self, - model_manager: ModelsProvider, - api_key: Optional[str], - step_execution_mode: StepExecutionMode, - ): - self._model_manager = model_manager - self._api_key = api_key - self._step_execution_mode = step_execution_mode - - @classmethod - def get_init_parameters(cls) -> List[str]: - return ["model_manager", "api_key", "step_execution_mode"] - - @classmethod - def get_manifest(cls) -> Type[WorkflowBlockManifest]: - return BlockManifest - - def run( - self, - images: Batch[WorkflowImageData], - model_id: str, - confidence_mode: str, - custom_confidence: Optional[float], - class_agnostic_nms: Optional[bool], - class_filter: Optional[List[str]], - iou_threshold: Optional[float], - max_detections: Optional[int], - max_candidates: Optional[int], - mask_decode_mode: Literal["accurate", "tradeoff", "fast"], - tradeoff_factor: Optional[float], - disable_active_learning: Optional[bool], - active_learning_target_dataset: Optional[str], - ) -> BlockResult: - confidence = ( - custom_confidence if confidence_mode == "custom" else confidence_mode - ) - if self._step_execution_mode is StepExecutionMode.LOCAL: - return self.run_locally( - images=images, - model_id=model_id, - class_agnostic_nms=class_agnostic_nms, - class_filter=class_filter, - confidence=confidence, - iou_threshold=iou_threshold, - max_detections=max_detections, - max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, - disable_active_learning=disable_active_learning, - active_learning_target_dataset=active_learning_target_dataset, - ) - elif self._step_execution_mode is StepExecutionMode.REMOTE: - return self.run_remotely( - images=images, - model_id=model_id, - class_agnostic_nms=class_agnostic_nms, - class_filter=class_filter, - confidence=confidence, - iou_threshold=iou_threshold, - max_detections=max_detections, - max_candidates=max_candidates, - mask_decode_mode=mask_decode_mode, - tradeoff_factor=tradeoff_factor, - disable_active_learning=disable_active_learning, - active_learning_target_dataset=active_learning_target_dataset, - ) - else: - raise ValueError( - f"Unknown step execution mode: {self._step_execution_mode}" - ) - - def run_locally( - self, - images: Batch[WorkflowImageData], - model_id: str, - class_agnostic_nms: Optional[bool], - class_filter: Optional[List[str]], - confidence: Union[None, float, Literal["best", "default"]], - iou_threshold: Optional[float], - max_detections: Optional[int], - max_candidates: Optional[int], - mask_decode_mode: Literal["accurate", "tradeoff", "fast"], - tradeoff_factor: Optional[float], - disable_active_learning: Optional[bool], - active_learning_target_dataset: Optional[str], - ) -> BlockResult: - # Feed the representation already materialised on the images to avoid forcing a - # numpy->device conversion: GPU tensors (RGB) only when every image in the batch - # already has one, otherwise the numpy frames (BGR) — matching the numpy block. - if all(image.is_tensor_materialised() for image in images): - model_inputs = [image.tensor_image for image in images] - image_color_format = "rgb" - else: - model_inputs = [image.numpy_image for image in images] - image_color_format = "bgr" - self._model_manager.add_model(model_id=model_id, api_key=self._api_key) - detections_batch: List[InstanceDetections] = ( - self._model_manager.run_tensor_native_inference( - model_id=model_id, - images=model_inputs, - input_color_format=image_color_format, - enforce_dense_masks_in_inference_models=WORKFLOWS_ENFORCE_DENSE_INSTANCE_MASKS, - confidence=confidence, - iou_threshold=iou_threshold, - class_agnostic_nms=class_agnostic_nms, - class_filter=class_filter, - max_detections=max_detections, - max_candidates=max_candidates, - mask_decode_mode="accurate", - tradeoff_factor=1.0, - disable_active_learning=disable_active_learning, - active_learning_target_dataset=active_learning_target_dataset, - ) - ) - class_names = _class_names_map(self._model_manager.get_class_names(model_id)) - results: List[dict] = [] - for image, detections in zip(images, detections_batch): - # The adapter/model does NOT honour class_filter on the native path, so - # filter here (the only place it is applied for LOCAL execution). - detections = _filter_classes_native(detections, class_filter, class_names) - # Reuse the adapter-provided inference id when present (numpy parity: - # numpy v4 surfaces ``p.get(INFERENCE_ID_KEY)`` off the model dump); - # the tensor-native adapter normally attaches none, so fall back to a - # freshly minted uuid that is then shared with ``image_metadata``. - inference_id = getattr(detections, "inference_id", None) or str( - uuid.uuid4() - ) - detections = attach_native_detection_metadata( - detections=detections, - image=image, - class_names=class_names, - prediction_type=PREDICTION_TYPE, - inference_id=inference_id, - ) - results.append( - { - "inference_id": inference_id, - "predictions": detections, - "model_id": model_id, - } - ) - return results - - def run_remotely( - self, - images: Batch[WorkflowImageData], - model_id: str, - class_agnostic_nms: Optional[bool], - class_filter: Optional[List[str]], - confidence: Union[None, float, Literal["best", "default"]], - iou_threshold: Optional[float], - max_detections: Optional[int], - max_candidates: Optional[int], - mask_decode_mode: Literal["accurate", "tradeoff", "fast"], - tradeoff_factor: Optional[float], - disable_active_learning: Optional[bool], - active_learning_target_dataset: Optional[str], - ) -> BlockResult: - api_url = ( - LOCAL_INFERENCE_API_URL - if WORKFLOWS_REMOTE_API_TARGET != "hosted" - else HOSTED_INSTANCE_SEGMENTATION_URL - ) - client = InferenceHTTPClient( - api_url=api_url, - api_key=self._api_key, - ) - if WORKFLOWS_REMOTE_API_TARGET == "hosted": - client.select_api_v0() - client_config = InferenceConfiguration( - api_key_transport=WORKFLOWS_REMOTE_API_KEY_TRANSPORT, - disable_active_learning=disable_active_learning, - active_learning_target_dataset=active_learning_target_dataset, - class_agnostic_nms=class_agnostic_nms, - class_filter=class_filter, - confidence_threshold=confidence, - iou_threshold=iou_threshold, - max_detections=max_detections, - max_candidates=max_candidates, - mask_decode_mode="accurate", - tradeoff_factor=1.0, - response_mask_format="rle", - max_batch_size=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_BATCH_SIZE, - max_concurrent_requests=WORKFLOWS_REMOTE_EXECUTION_MAX_STEP_CONCURRENT_REQUESTS, - source="workflow-execution", - ) - client.configure(inference_configuration=client_config) - inference_images = [i.base64_image for i in images] - predictions = client.infer( - inference_input=inference_images, - model_id=model_id, - ) - if not isinstance(predictions, list): - predictions = [predictions] - return self._post_process_remote_result( - images=images, - predictions=predictions, - class_filter=class_filter, - model_id=model_id, - ) - - def _post_process_remote_result( - self, - images: Batch[WorkflowImageData], - predictions: List[dict], - class_filter: Optional[List[str]], - model_id: str, - ) -> BlockResult: - # Fallback class_id -> name map from the model, used to name boxes whose - # remote prediction dict lacks a `class` key (otherwise the tensor - # serialiser hard-raises "class_id missing from mapping"). - model_class_names = _class_names_map( - self._model_manager.get_class_names(model_id) - ) - results: List[dict] = [] - for image, response in zip(images, predictions): - inference_id = response.get(INFERENCE_ID_KEY) or str(uuid.uuid4()) - detection_dicts = response.get("predictions", []) or [] - if class_filter: - accepted = set(class_filter) - detection_dicts = [ - d for d in detection_dicts if d.get(CLASS_NAME_KEY) in accepted - ] - detections = _native_instance_detections_from_inference_predictions( - image=image, - predictions=detection_dicts, - prediction_type=PREDICTION_TYPE, - inference_id=inference_id, - device=WORKFLOWS_IMAGE_TENSOR_DEVICE, - model_class_names=model_class_names, - ) - results.append( - { - "inference_id": inference_id, - "predictions": detections, - "model_id": model_id, - } - ) - return results - - -def _class_names_map(class_names: List[str]) -> Dict[int, str]: - return {index: name for index, name in enumerate(class_names)} - - -def _filter_classes_native( - detections: InstanceDetections, - class_filter: Optional[List[str]], - class_names: Dict[int, str], -) -> InstanceDetections: - if not class_filter: - return detections - accepted = set(class_filter) - accepted_ids = sorted( - class_id for class_id, name in class_names.items() if name in accepted - ) - if not accepted_ids: - return take_prediction_by_mask( - detections, - torch.zeros_like(detections.class_id, dtype=torch.bool), - ) - accepted_tensor = torch.as_tensor(accepted_ids, device=detections.class_id.device) - keep = torch.isin(detections.class_id, accepted_tensor) - return take_prediction_by_mask(detections, keep) - - -def _extract_rle_mask(prediction: dict) -> Optional[dict]: - """Pull the COCO-RLE mask ({"size": [H, W], "counts": ...}) from a standard - inference instance-seg prediction dict, trying both response key variants.""" - for key in RLE_MASK_KEYS: - mask = prediction.get(key) - if mask is not None: - return mask - return None - - -def _extract_polygon_points(prediction: dict) -> Optional[List[dict]]: - """Pull the polygon ``points`` ([{"x": .., "y": ..}, ...]) from a standard - inference instance-seg prediction dict that carries a polygon mask instead of - RLE. Returns ``None`` when the key is absent or the polygon is degenerate - (< 3 points) - mirroring the numpy ``filter_out_invalid_polygons``, which drops - such instances entirely.""" - points = prediction.get(POLYGON_KEY) - if points is not None and len(points) >= 3: - return points - return None - - -def _polygon_points_to_dense_mask( - points: List[dict], height: int, width: int -) -> np.ndarray: - """Rasterise polygon ``points`` into a dense boolean ``(H, W)`` mask, - byte-identically to supervision's ``polygon_to_mask`` as used by - ``Detections.from_inference`` (the numpy REMOTE path): vertices are integer - *truncated* (``dtype=int``, NOT rounded), filled with ``cv2.fillPoly`` onto a - zero ``uint8`` canvas, then cast to ``bool``. Keeping the exact rasterisation - means the serialiser's mask->polygon re-contouring (``mask_to_polygon``, shared - verbatim with the numpy serialiser) reproduces the numpy ``points`` output.""" - polygon = np.array([[point[X_KEY], point[Y_KEY]] for point in points], dtype=int) - return sv.polygon_to_mask(polygon, resolution_wh=(width, height)).astype(bool) - - -def _native_instance_detections_from_inference_predictions( - image: WorkflowImageData, - predictions: List[dict], - prediction_type: str, - inference_id: Optional[str] = None, - device: Optional[torch.device] = None, - model_class_names: Optional[Dict[int, str]] = None, -) -> InstanceDetections: - """REMOTE-path converter: build a native ``InstanceDetections`` from standard - inference instance-segmentation prediction dicts. - - The block requests ``response_mask_format="rle"`` so a current inference server - returns an RLE-encoded COCO mask (``{"size": [H, W], "counts": ""}``) - under ``rle`` / ``rle_mask``; those are carried as ``InstancesRLEMasks`` (counts - normalised from a utf-8 string to bytes, what ``pycocotools`` / the serialiser's - RLE decode path expect). To stay compatible with servers that ignore that - request parameter and return polygon ``points`` instead (older/alternative - inference servers), the converter degrades gracefully: when the response is not - fully RLE-backed it rebuilds the masks from the polygon ``points`` into a dense - ``torch.bool`` ``(N, H, W)`` carrier - the same masks the numpy flag-OFF REMOTE - path builds via ``sv.Detections.from_inference`` - so the serialised ``points`` - output matches numpy's (the serialiser re-contours the dense mask with the same - ``mask_to_polygon``). Boxes are converted from center form to corner ``xyxy``. - The ``class_id -> name`` map (required by the tensor serialiser) and per-box - ``detection_id`` are built here too. - - Degenerate (< 3-point) polygons are dropped exactly like the numpy path - (``filter_out_invalid_polygons``). When a - prediction omits its ``class`` key, the class name is backfilled from - ``model_class_names`` (the model's ``get_class_names`` map) so the tensor - serialiser does not hard-raise on an unmapped ``class_id``. - """ - height, width = image._read_shape_without_materialization() - # Prefer the RLE masks the block requests; fall back to dense polygon masks when - # the response is not fully RLE-backed (server ignored `response_mask_format`). - # `all([])` is True, so an empty response keeps the empty-RLE carrier unchanged. - use_rle = all( - _extract_rle_mask(prediction) is not None for prediction in predictions - ) - if use_rle: - kept_predictions = predictions - else: - # Drop degenerate polygons up front so the surviving box arrays and masks - # stay aligned (matches the numpy path's instance drop). - kept_predictions = [ - prediction - for prediction in predictions - if _extract_polygon_points(prediction) is not None - ] - xyxy: List[List[float]] = [] - class_id: List[int] = [] - confidence: List[float] = [] - rle_counts: List[bytes] = [] - dense_masks: List[np.ndarray] = [] - bboxes_metadata: List[dict] = [] - derived_class_names: Dict[int, str] = {} - for prediction in kept_predictions: - center_x = float(prediction[X_KEY]) - center_y = float(prediction[Y_KEY]) - box_width = float(prediction[WIDTH_KEY]) - box_height = float(prediction[HEIGHT_KEY]) - xyxy.append( - [ - center_x - box_width / 2, - center_y - box_height / 2, - center_x + box_width / 2, - center_y + box_height / 2, - ] - ) - prediction_class_id = int(prediction.get(CLASS_ID_KEY, 0)) - class_id.append(prediction_class_id) - confidence.append(float(prediction.get(CONFIDENCE_KEY, 1.0))) - if CLASS_NAME_KEY in prediction: - derived_class_names[prediction_class_id] = str(prediction[CLASS_NAME_KEY]) - elif model_class_names is not None and prediction_class_id in model_class_names: - derived_class_names[prediction_class_id] = model_class_names[ - prediction_class_id - ] - if use_rle: - mask = _extract_rle_mask(prediction) - raw_counts = mask["counts"] - # Normalise to bytes: pycocotools (used by the serialiser's RLE decode) - # expects byte counts; the remote rle response carries them as utf-8 strings. - rle_counts.append( - raw_counts.encode("utf-8") - if isinstance(raw_counts, str) - else raw_counts - ) - else: - dense_masks.append( - _polygon_points_to_dense_mask( - points=_extract_polygon_points(prediction), - height=height, - width=width, - ) - ) - bboxes_metadata.append( - {DETECTION_ID_KEY: str(prediction.get(DETECTION_ID_KEY) or uuid.uuid4())} - ) - number_of_detections = len(xyxy) - image_metadata = build_native_image_metadata( - image=image, - class_names=derived_class_names, - prediction_type=prediction_type, - inference_id=inference_id, - ) - if use_rle: - mask_carrier: Union[torch.Tensor, InstancesRLEMasks] = InstancesRLEMasks( - image_size=(height, width), masks=rle_counts - ) - elif dense_masks: - mask_carrier = torch.as_tensor( - np.stack(dense_masks), dtype=torch.bool, device=device - ) - else: - # No surviving polygons: an empty dense carrier the serialiser handles. - mask_carrier = torch.zeros((0, height, width), dtype=torch.bool, device=device) - return InstanceDetections( - xyxy=torch.as_tensor(xyxy, dtype=torch.float32, device=device).reshape(-1, 4), - class_id=torch.as_tensor(class_id, dtype=torch.long, device=device).reshape(-1), - confidence=torch.as_tensor( - confidence, dtype=torch.float32, device=device - ).reshape(-1), - mask=mask_carrier, - image_metadata=image_metadata, - bboxes_metadata=bboxes_metadata if number_of_detections > 0 else None, - ) diff --git a/workflows/roboflow_workflows/execution_engine/entities/types.py b/workflows/roboflow_workflows/execution_engine/entities/types.py index f5546ea90e..bff4817ab5 100644 --- a/workflows/roboflow_workflows/execution_engine/entities/types.py +++ b/workflows/roboflow_workflows/execution_engine/entities/types.py @@ -828,11 +828,8 @@ def __hash__(self) -> int: The masks are stored in `sv.Detections.data["rle_mask"]` as a numpy array of RLE dictionaries in COCO format: `{"size": [H, W], "counts": "encoded_string"}`. -`size` describes the grid the counts were encoded on, which is the image grid unless the -producing block reduced mask resolution - see `tradeoff_factor` on -`roboflow_instance_segmentation_model@v5`. The counts must be decoded on `size`; decoding -them on the image dimensions makes pycocotools reinterpret the runs without raising. -`sv.Detections.mask`, when materialised, is always at image resolution. +`size` describes the grid the counts were encoded on. The instance-segmentation +model blocks retain image-resolution masks. Reduced-grid workflow support is deferred. When visualization blocks need numpy masks, they decode RLE on-demand using pycocotools. From db65bc1bd50918c1fec0206e923c3048ffd22db3 Mon Sep 17 00:00:00 2001 From: Michal Taraszewski Date: Fri, 9 Oct 2026 13:51:26 +0200 Subject: [PATCH 31/41] Align opted-in segmentation geometry with the mask grid --- docs/instance_segmentation_mask_resolution.md | 113 ++++++---- .../core/active_learning/post_processing.py | 39 ++++ inference/core/entities/requests/inference.py | 12 +- .../core/entities/responses/inference.py | 3 + inference/core/managers/active_learning.py | 5 +- .../core/models/inference_models_adapters.py | 43 +++- inference/core/utils/visualisation.py | 23 +++ inference_models/docs/changelog.md | 17 +- inference_sdk/http/entities.py | 7 +- inference_sdk/http/utils/post_processing.py | 30 ++- ...t_instance_segmentation_rle_predictions.py | 37 +++- .../active_learning/test_post_processing.py | 80 ++++++++ .../models/test_inference_models_adapters.py | 193 +++++++++++++++--- .../core/utils/test_visualisation.py | 52 +++++ .../unit_tests/http/test_client.py | 18 +- .../unit_tests/http/test_entities.py | 13 ++ .../http/utils/test_postprocessing.py | 24 +-- workflows/CHANGELOG.md | 9 +- .../common/inference_response_entities.py | 40 ++-- 19 files changed, 610 insertions(+), 148 deletions(-) diff --git a/docs/instance_segmentation_mask_resolution.md b/docs/instance_segmentation_mask_resolution.md index c34fb6e3c1..bea02261c0 100644 --- a/docs/instance_segmentation_mask_resolution.md +++ b/docs/instance_segmentation_mask_resolution.md @@ -1,10 +1,13 @@ # Opt into configurable instance-segmentation masks -On servers running `USE_INFERENCE_MODELS=True`, existing requests keep masks at -image resolution. To use `mask_decode_mode="fast"` or `"tradeoff"`, also set -`allow_reduced_mask_resolution=true`. This applies to both -`POST /infer/instance_segmentation` (JSON body) and the legacy -`POST /{dataset_id}/{version_id}` route (query parameters). +On servers running `USE_INFERENCE_MODELS=True`, existing requests keep masks, +polygon points and bounding boxes in original-image coordinates. The HTTP +request and SDK configuration both default `allow_reduced_mask_resolution` to +`False`. Set it to `True` to return **all prediction geometry on the selected +mask grid**, regardless of response format. + +This applies to `POST /infer/instance_segmentation` (JSON body) and +`POST /{dataset_id}/{version_id}` (query parameters). ```python from inference_sdk import InferenceConfiguration, InferenceHTTPClient @@ -20,54 +23,80 @@ client = InferenceHTTPClient(api_url="http://localhost:9001").configure( result = client.infer("image.jpg", model_id="yolov8n-seg-640") ``` -With the opt-in, contours can be extracted from reduced masks, but polygon -points and bounding boxes always use the original image's pixel coordinates. -The server scales contour coordinates before returning polygon predictions; -callers do not need to transform the points. +For `mask_decode_mode="tradeoff"` and `tradeoff_factor=0.5`: + +| Opt-in | Format | Returned coordinate frame | +|---|---|---| +| `False` | `rle` | RLE and boxes use original-image coordinates | +| `True` | `rle` | RLE and boxes use the selected mask grid | +| `False` | `polygon` | Polygon points and boxes use original-image coordinates | +| `True` | `polygon` | Polygon points and boxes use the selected mask grid | + +The factor interpolates between the unpadded model mask grid and the image grid; +it does **not** multiply image dimensions or coordinates directly. With a +160×160 model grid and a 1000×800 image, factor `0.5` selects a 580×480 grid +without cropping. Boxes scale by `580/1000` horizontally and `480/800` +vertically. Polygon contours are extracted on that grid and stay there. +Static crops are aligned onto a canvas representing the original image; its +actual dimensions, including rounding, determine the output mapping. -RLE responses retain the encoded mask grid. When its dimensions differ from -the response image, `mask_metadata` explicitly describes that RLE grid: +## Response dimensions and mapping + +With opt-in, `image` describes the output coordinate frame. `original_image` +retains the input dimensions and `mask_metadata` describes the mapping back to +that input. These fields are included for both formats, including empty results +and when the output grid equals the input dimensions. Each batch item carries +its own dimensions and mapping. Without opt-in, the extra fields are omitted. ```json { - "image": {"width": 640, "height": 480}, + "image": {"width": 580, "height": 480}, + "original_image": {"width": 1000, "height": 800}, "mask_metadata": { "coordinate_system": "mask_grid", - "width": 160, - "height": 120, - "scale_x": 4.0, - "scale_y": 4.0 + "width": 580, + "height": 480, + "scale_x": 1.7241379310344827, + "scale_y": 1.6666666666666667 } } ``` -These dimensions illustrate the contract, not an expected result for every -model or factor. RLE `size` is `[height, width]` from `mask_metadata`. -The scales map positions on the encoded RLE grid to image coordinates. -Metadata is omitted for polygon responses and for RLE grids matching the -image. Empty RLE predictions still include metadata when their grid differs. -Each image in a batch has its own metadata. +All boxes and polygon points use `image` coordinates. RLE `size` is +`[image.height, image.width]`. To map geometry back to the input, multiply x +coordinates and widths by `scale_x`, and y coordinates and heights by `scale_y`. +Mapping coordinates does not recover detail lost through reduced mask resolution. -Polygon responses can be passed directly to Supervision: +`sv.Detections.from_inference(result)` produces detections for the **output +grid**. To annotate the original image, first map boxes and polygon points back, +or resize decoded RLE masks and map their boxes. Do not annotate the original +scene directly with output-grid detections. Workflow integration of reduced +masks remains deferred. -```python -import supervision as sv +When SDK client downsizing is enabled, the server selects the output grid from +the image it receives. The SDK preserves that grid, boxes, points and encoded +RLE; it updates only `original_image` and mapping scales to refer to the client +image. Server-rendered visualizations project predictions onto the server's +input image without changing the returned geometry. -detections = sv.Detections.from_inference(result) -``` +The active-learning manager also projects its own copy back to image coordinates +before handing predictions to sampling and registration. RLE responses are converted +to the largest-contour polygon annotations used by that path; the HTTP response +retains its original RLE representation and output grid. + +## Compatibility and follow-up work + +Direct `inference_models` calls keep their `masks_resolution_factor` interface +and existing image-space box contract. The HTTP output-coordinate transformation +happens at the response boundary. The legacy model backend retains its existing +decode-mode behavior and ignores the new opt-in flag. + +Values from 0 to 1 interpolate model-grid and image-grid dimensions. A lower +factor does not always mean a smaller grid or faster execution. `accurate` +selects factor 1.0, `fast` selects 0.0, and only `tradeoff` reads `tradeoff_factor`. +Damian's Triton optimization is separate; factors other than 1.0 currently use +the fallback post-processor. -Decode RLE on its declared grid; never replace its `size` with image dimensions -without resampling the mask. During client-side image resizing, the SDK scales -polygon points and bounding boxes together. It preserves encoded RLE data and -updates its metadata scales to the client image. Server-rendered polygon -visualizations use the returned points directly. - -Direct `inference_models` calls keep their `masks_resolution_factor` interface. -The legacy model backend retains its existing decode-mode behavior and ignores -the new opt-in flag. Values from 0 to 1 interpolate model-grid and image-grid -dimensions; a lower factor does not always mean a smaller grid or faster execution. -Damian's Triton optimization is separate; the existing fallback remains for now. - -Workflow versions v1–v4 retain their compatibility guards. The new v5 block, -reduced-mask Supervision/workflow handling, and full tensor support are deferred -to a separate PR. This change does not introduce reduced-grid workflow outputs. +Workflow versions v1–v4 retain their compatibility guards. Workflow v5, +reduced-mask downstream handling, and full tensor support are deferred to a +separate PR. This change does not introduce reduced-grid workflow outputs. diff --git a/inference/core/active_learning/post_processing.py b/inference/core/active_learning/post_processing.py index a8381fef4a..506bc96080 100644 --- a/inference/core/active_learning/post_processing.py +++ b/inference/core/active_learning/post_processing.py @@ -1,4 +1,5 @@ import json +from copy import deepcopy from typing import List, Tuple from inference.core.active_learning.entities import ( @@ -13,6 +14,7 @@ OBJECT_DETECTION_TASK, ) from inference.core.exceptions import PredictionFormatNotSupported +from inference.core.utils.rle_to_polygon import polygon_from_coco_counts def adjust_prediction_to_client_scaling_factor( @@ -47,6 +49,43 @@ def adjust_prediction_to_client_scaling_factor( return prediction +def project_mask_grid_to_original(prediction: dict) -> dict: + """Prepare reduced-grid predictions for image-space active learning consumers. + + Args: + prediction: Serialized inference response. + + Returns: + An independent image-space copy for opted-in responses, with RLE masks + converted to the same largest-contour polygons used by polygon responses. + Responses without output-grid metadata are returned unchanged. + """ + metadata = prediction.get("mask_metadata") + original_image = prediction.get("original_image") + if not metadata or not original_image: + return prediction + + prediction = deepcopy(prediction) + prediction["image"] = prediction.pop("original_image") + prediction.pop("mask_metadata") + for detection in prediction.get("predictions", []): + for key in ("x", "width"): + detection[key] *= metadata["scale_x"] + for key in ("y", "height"): + detection[key] *= metadata["scale_y"] + if "rle" in detection: + rle = detection.pop("rle") + polygon = polygon_from_coco_counts( + counts=rle["counts"], height=rle["size"][0], width=rle["size"][1] + ) + detection["points"] = [{"x": float(x), "y": float(y)} for x, y in polygon] + for point in detection.get("points", []): + point["x"] *= metadata["scale_x"] + point["y"] *= metadata["scale_y"] + + return prediction + + def predictions_should_not_be_post_processed( prediction: dict, prediction_type: PredictionType ) -> bool: diff --git a/inference/core/entities/requests/inference.py b/inference/core/entities/requests/inference.py index 7fc91cabff..b7fe33588c 100644 --- a/inference/core/entities/requests/inference.py +++ b/inference/core/entities/requests/inference.py @@ -272,9 +272,9 @@ class InstanceSegmentationInferenceRequest(ObjectDetectionInferenceRequest): examples=[True], description="Opt into mask_decode_mode and tradeoff_factor on the " "inference_models backend. Without this flag, masks stay at image " - "resolution. With it, RLE size can differ from image dimensions; " - "mask_metadata describes the RLE grid and its mapping to the image. " - "Polygon points and bounding boxes remain in image space. The legacy " + "resolution. With it, boxes, polygon points and RLE masks all use the " + "selected mask grid, reported in image. original_image retains the input " + "dimensions and mask_metadata maps output coordinates back to the input. The legacy " "backend retains its existing decoding behavior regardless of this flag.", ) response_mask_format: Literal["polygon", "rle"] = Field( @@ -284,9 +284,9 @@ class InstanceSegmentationInferenceRequest(ObjectDetectionInferenceRequest): "not capable representing certain shapes - RLE is compact and more standard representation, yet " "require special decoding on the caller side - currently supported in `opt-in` mode when server is " "running with `USE_INFERENCE_MODELS=True` - otherwise it's ignored. " - "RLE size records the encoded mask grid, which may differ from the image " - "dimensions when allow_reduced_mask_resolution=True, with mask_metadata " - "supplying the image mapping. Polygon points always use image coordinates.", + "RLE size records the encoded mask grid. With allow_reduced_mask_resolution=True, " + "both formats and bounding boxes use this grid, matching response image dimensions. " + "original_image and mask_metadata provide the mapping to the input image.", ) enforce_dense_masks_in_inference_models: Optional[bool] = Field( default=False, diff --git a/inference/core/entities/responses/inference.py b/inference/core/entities/responses/inference.py index 15e8ddf42e..4d338cf580 100644 --- a/inference/core/entities/responses/inference.py +++ b/inference/core/entities/responses/inference.py @@ -236,6 +236,7 @@ class InstanceSegmentationInferenceResponseDC: visualization: object = None resolved_model: Optional[ResolvedModel] = None mask_metadata: Optional[MaskCoordinateMetadata] = None + original_image: Optional[InferenceResponseImage] = None # Internal stream-pipeline fast path: lets workflow execution carry a # response future through Model.infer_from_request without blocking the # inference thread. `_is_response_dc_to_dict` intentionally ignores it. @@ -289,6 +290,8 @@ def _is_response_dc_to_dict(r: InstanceSegmentationInferenceResponseDC) -> dict: d["visualization"] = r.visualization if r.mask_metadata is not None: d["mask_metadata"] = r.mask_metadata.model_dump() + if r.original_image is not None: + d["original_image"] = r.original_image.model_dump() if r.resolved_model is not None: d["resolved_model"] = r.resolved_model.model_dump() return d diff --git a/inference/core/managers/active_learning.py b/inference/core/managers/active_learning.py index ba5c5b24c4..901b737eef 100644 --- a/inference/core/managers/active_learning.py +++ b/inference/core/managers/active_learning.py @@ -5,6 +5,7 @@ from inference.core import logger from inference.core.active_learning.middlewares import ActiveLearningMiddleware +from inference.core.active_learning.post_processing import project_mask_grid_to_original from inference.core.cache.base import BaseCache from inference.core.entities.requests.inference import InferenceRequest from inference.core.entities.responses.inference import InferenceResponse @@ -158,7 +159,9 @@ def register_datapoint( ) self._middlewares[middleware_key].register_batch( inference_inputs=inference_inputs, - predictions=results_dicts, + predictions=[ + project_mask_grid_to_original(result) for result in results_dicts + ], prediction_type=prediction_type, disable_preproc_auto_orient=disable_preproc_auto_orient, inference_id=request.id, diff --git a/inference/core/models/inference_models_adapters.py b/inference/core/models/inference_models_adapters.py index 35fc49facd..34a28a151c 100644 --- a/inference/core/models/inference_models_adapters.py +++ b/inference/core/models/inference_models_adapters.py @@ -630,6 +630,7 @@ def _request_batch_size(self, img_in: Any) -> int: return 1 def predict(self, img_in, **kwargs): + allow_reduced_masks = kwargs.get("allow_reduced_mask_resolution", False) mapped_kwargs = self.map_inference_kwargs(kwargs) if self._pipeline_depth <= 1: # Original path: forward on current frame, postprocess on @@ -650,6 +651,7 @@ def predict(self, img_in, **kwargs): self._submit_next_pending_gpu_work() pre_processing_meta = getattr(img_in, "_pre_processing_meta", None) fut = self._model.forward_async(img_in, pre_processing_meta, **mapped_kwargs) + fut._adapter_allow_reduced_mask_resolution = allow_reduced_masks stream_pipeline_context_id = kwargs.get(STREAM_PIPELINE_CONTEXT_ID_KWARG) if not isinstance(stream_pipeline_context_id, str): stream_pipeline_context_id = None @@ -859,8 +861,12 @@ def _finalize_future( fut._meta = preprocess_return_metadata # type: ignore[attr-defined] fut._kwargs = mapped_kwargs # type: ignore[attr-defined] detections_list = fut.result() + response_kwargs = dict(mapped_kwargs) + response_kwargs["allow_reduced_mask_resolution"] = getattr( + fut, "_adapter_allow_reduced_mask_resolution", False + ) return self._build_responses_from_detections( - detections_list, preprocess_return_metadata, **mapped_kwargs + detections_list, preprocess_return_metadata, **response_kwargs ) def _postprocess_sync( @@ -870,13 +876,17 @@ def _postprocess_sync( **kwargs, ) -> List[InstanceSegmentationInferenceResponse]: return_in_rle = kwargs.get("response_mask_format") == "rle" + allow_reduced_masks = kwargs.get("allow_reduced_mask_resolution", False) mapped_kwargs = self.map_inference_kwargs(kwargs) mapped_kwargs["defer_count_to_adapter"] = not return_in_rle detections_list = self._model.post_process( predictions, preprocess_return_metadata, **mapped_kwargs ) return self._build_responses_from_detections( - detections_list, preprocess_return_metadata, **kwargs + detections_list, + preprocess_return_metadata, + allow_reduced_mask_resolution=allow_reduced_masks, + **kwargs, ) def _build_responses_from_detections( @@ -1065,14 +1075,21 @@ def _build_responses_from_detections( kwargs.get("masks_resolution_factor", 1.0) != 1.0 ) mask_metadata = None - if return_in_rle and native_grid and different_grid: + original_image = None + output_height, output_width = H, W + box_scale_x = box_scale_y = 1.0 + if native_grid and mask_size is not None: + output_height, output_width = int(mask_size[0]), int(mask_size[1]) + original_image = InferenceResponseImage(width=W, height=H) + box_scale_x = output_width / W + box_scale_y = output_height / H mask_metadata = MaskCoordinateMetadata( height=int(mask_size[0]), width=int(mask_size[1]), scale_x=W / mask_size[1], scale_y=H / mask_size[0], ) - if not return_in_rle and different_grid: + if not return_in_rle and different_grid and not native_grid: polys_or_rles = scale_polygons_to_image( polys_or_rles, mask_size=ImageDimensions( @@ -1088,10 +1105,10 @@ def _build_responses_from_detections( for (x1, y1, x2, y2), mask_as_poly_or_rle, conf, class_id in zip( xyxy, polys_or_rles, confs, class_ids ): - cx = (float(x1) + float(x2)) / 2.0 - cy = (float(y1) + float(y2)) / 2.0 - w = float(x2) - float(x1) - h = float(y2) - float(y1) + cx = (float(x1) + float(x2)) / 2.0 * box_scale_x + cy = (float(y1) + float(y2)) / 2.0 * box_scale_y + w = (float(x2) - float(x1)) * box_scale_x + h = (float(y2) - float(y1)) * box_scale_y class_id_int = int(class_id) class_name = ( self.class_names[class_id_int] @@ -1158,7 +1175,10 @@ def _build_responses_from_detections( responses.append( InstanceSegmentationInferenceResponseDC( predictions=predictions, - image=InferenceResponseImageDC(width=W, height=H), + image=InferenceResponseImageDC( + width=output_width, height=output_height + ), + original_image=original_image, mask_metadata=mask_metadata, ) ) @@ -1166,7 +1186,10 @@ def _build_responses_from_detections( responses.append( InstanceSegmentationInferenceResponse( predictions=predictions, - image=InferenceResponseImage(width=W, height=H), + image=InferenceResponseImage( + width=output_width, height=output_height + ), + original_image=original_image, mask_metadata=mask_metadata, ) ) diff --git a/inference/core/utils/visualisation.py b/inference/core/utils/visualisation.py index bcf43358f0..b80a32b87e 100644 --- a/inference/core/utils/visualisation.py +++ b/inference/core/utils/visualisation.py @@ -1,3 +1,4 @@ +from copy import copy from typing import Dict, List, Tuple, Union import cv2 @@ -32,8 +33,30 @@ def draw_detection_predictions( ], colors: Dict[str, str], ) -> bytes: + """Render predictions on the original input image without changing the response. + + Args: + inference_request: Request carrying the input image and drawing settings. + inference_response: Predictions and optional output-grid mapping. + colors: Class names mapped to hexadecimal drawing colors. + + Returns: + JPEG bytes with boxes and polygon points projected to the input image. + """ image = load_image_rgb(inference_request.image) + mask_metadata = getattr(inference_response, "mask_metadata", None) for box in inference_response.predictions: + if mask_metadata is not None: + box = copy(box) + box.x *= mask_metadata.scale_x + box.y *= mask_metadata.scale_y + box.width *= mask_metadata.scale_x + box.height *= mask_metadata.scale_y + if hasattr(box, "points"): + box.points = [ + Point(x=p.x * mask_metadata.scale_x, y=p.y * mask_metadata.scale_y) + for p in box.points + ] color = tuple( int(colors.get(box.class_name, "#4892EA")[i : i + 2], 16) for i in (1, 3, 5) ) diff --git a/inference_models/docs/changelog.md b/inference_models/docs/changelog.md index a273d8a9bb..e7020833eb 100644 --- a/inference_models/docs/changelog.md +++ b/inference_models/docs/changelog.md @@ -27,13 +27,16 @@ `allow_reduced_mask_resolution=true` to apply `mask_decode_mode` and `tradeoff_factor`. Without this opt-in, masks retain image resolution. The legacy backend retains its existing decoding behavior. - Polygon extraction uses the selected mask resolution, then scales the returned - points to image coordinates so they share the bounding boxes' coordinate frame. - Opted-in RLE responses retain the encoded mask grid; when it differs from the - image, `mask_metadata` supplies its dimensions and x/y scales to image coordinates. - The SDK scales polygon points and boxes together during client resizing, while - preserving encoded RLE data and updating its metadata. Server visualizations - use the returned image-space polygon points directly. + With opt-in, boxes, polygon points and RLE masks all use the selected mask + grid, reported in response `image`. `original_image` retains the input dimensions + and `mask_metadata` maps all output geometry back to them, including when both + grids have the same dimensions. Without opt-in, all outputs use image coordinates. + The SDK defaults the flag to `False`; during client downsizing it preserves an + opted-in output grid and geometry, updating only original-image dimensions and + mapping scales. Server visualizations project geometry onto the original image + without mutating returned predictions. + The active-learning manager projects a separate copy to image-space polygon + annotations before passing it to sampling and registration. Direct `inference_models` callers use `masks_resolution_factor` without the HTTP opt-in. Reduced-grid workflow support is deferred to a separate PR. - RLE export and `InstanceDetections` iteration declare the encoded mask grid in diff --git a/inference_sdk/http/entities.py b/inference_sdk/http/entities.py index 619045e7ef..2589b28ca7 100644 --- a/inference_sdk/http/entities.py +++ b/inference_sdk/http/entities.py @@ -135,8 +135,9 @@ class InferenceConfiguration: format: The format for the inference. mask_decode_mode: The mask decode mode for the inference. tradeoff_factor: The tradeoff factor for the inference. - allow_reduced_mask_resolution: Opt into configurable mask grids on the - inference_models server backend. Legacy backend behavior is unchanged. + allow_reduced_mask_resolution: Return boxes, polygon points and RLE masks + on the selected mask grid with inference_models. Defaults to False. + Legacy backend behavior is unchanged. max_candidates: The maximum number of candidates for the inference. max_detections: The maximum number of detections for the inference. iou_threshold: The intersection over union threshold for the inference. @@ -186,7 +187,7 @@ class InferenceConfiguration: # Pass "legacy" explicitly to keep the old behaviour silently. api_key_transport: Optional[Union[str, ApiKeyTransport]] = None - allow_reduced_mask_resolution: Optional[bool] = None + allow_reduced_mask_resolution: bool = False def __post_init__(self) -> None: # Normalise the transport to the enum so the client can rely on diff --git a/inference_sdk/http/utils/post_processing.py b/inference_sdk/http/utils/post_processing.py index 01683a1ae8..b50da9fe8c 100644 --- a/inference_sdk/http/utils/post_processing.py +++ b/inference_sdk/http/utils/post_processing.py @@ -208,24 +208,38 @@ def adjust_prediction_to_client_scaling_factor( Args: prediction: The prediction to adjust. - scaling_factor: The scaling factor. + scaling_factor: The scaling factor applied to the input image. Returns: - The adjusted prediction. + Predictions mapped to the client image, or opted-in grid predictions + unchanged with their original-image dimensions and mapping updated. """ if scaling_factor is None or prediction.get("is_stub", False): return prediction + mask_metadata = prediction.get("mask_metadata") + original_image = prediction.get("original_image") + if ( + mask_metadata + and mask_metadata.get("coordinate_system") == "mask_grid" + and original_image is not None + ): + prediction["original_image"] = { + "width": round(original_image["width"] / scaling_factor), + "height": round(original_image["height"] / scaling_factor), + } + mask_metadata["scale_x"] = ( + prediction["original_image"]["width"] / mask_metadata["width"] + ) + mask_metadata["scale_y"] = ( + prediction["original_image"]["height"] / mask_metadata["height"] + ) + return prediction + if "image" in prediction: prediction["image"] = { "width": round(prediction["image"]["width"] / scaling_factor), "height": round(prediction["image"]["height"] / scaling_factor), } - mask_metadata = prediction.get("mask_metadata") - if mask_metadata and mask_metadata.get("coordinate_system") == "mask_grid": - mask_metadata["scale_x"] = prediction["image"]["width"] / mask_metadata["width"] - mask_metadata["scale_y"] = ( - prediction["image"]["height"] / mask_metadata["height"] - ) if predictions_should_not_be_post_processed(prediction=prediction): return prediction diff --git a/tests/inference/integration_tests/test_instance_segmentation_rle_predictions.py b/tests/inference/integration_tests/test_instance_segmentation_rle_predictions.py index b6915c34fa..421cfdfd41 100644 --- a/tests/inference/integration_tests/test_instance_segmentation_rle_predictions.py +++ b/tests/inference/integration_tests/test_instance_segmentation_rle_predictions.py @@ -1,5 +1,6 @@ import os +import cv2 import numpy as np import pytest import requests @@ -49,11 +50,20 @@ def test_mask_resolution_round_trip_through_server( response.raise_for_status() result = response.json() metadata = result.get("mask_metadata") - if response_format == "rle" and opt_in and mode != "accurate": + if opt_in: assert metadata["coordinate_system"] == "mask_grid" - assert metadata["width"] > 0 and metadata["height"] > 0 + assert result["image"] == { + "width": metadata["width"], + "height": metadata["height"], + } + assert result["original_image"] == {"width": 720, "height": 1280} + assert metadata["scale_x"] == pytest.approx(720 / metadata["width"]) + assert metadata["scale_y"] == pytest.approx(1280 / metadata["height"]) + if mode != "accurate": + assert metadata["width"] < 720 and metadata["height"] < 1280 else: assert metadata is None + assert result.get("original_image") is None detections = sv.Detections.from_inference(result) height, width = result["image"]["height"], result["image"]["width"] assert len(detections) > 0 @@ -75,10 +85,7 @@ def test_mask_resolution_round_trip_through_server( mask_height, mask_width = sizes.pop() if metadata: assert (mask_height, mask_width) == (metadata["height"], metadata["width"]) - if mode == "accurate" or not opt_in: - assert (mask_height, mask_width) == (height, width) - else: - assert mask_height < height and mask_width < width + assert (mask_height, mask_width) == (height, width) reference_response = requests.post( f"{BASE_URL}:{PORT}/infer/instance_segmentation", @@ -90,10 +97,22 @@ def test_mask_resolution_round_trip_through_server( ) reference_response.raise_for_status() reference = sv.Detections.from_inference(reference_response.json()) - np.testing.assert_allclose(detections.xyxy, reference.xyxy, atol=1) + scale_x = metadata["scale_x"] if metadata else 1.0 + scale_y = metadata["scale_y"] if metadata else 1.0 + np.testing.assert_allclose( + detections.xyxy * [scale_x, scale_y, scale_x, scale_y], reference.xyxy, atol=1 + ) np.testing.assert_array_equal(detections.class_id, reference.class_id) - intersection = np.logical_and(detections.mask, reference.mask).sum(axis=(1, 2)) - union = np.logical_or(detections.mask, reference.mask).sum(axis=(1, 2)) + restored_masks = np.stack( + [ + cv2.resize( + mask.astype(np.uint8), (720, 1280), interpolation=cv2.INTER_NEAREST + ).astype(bool) + for mask in detections.mask + ] + ) + intersection = np.logical_and(restored_masks, reference.mask).sum(axis=(1, 2)) + union = np.logical_or(restored_masks, reference.mask).sum(axis=(1, 2)) assert np.all(intersection / np.maximum(union, 1) > 0.8) diff --git a/tests/inference/unit_tests/core/active_learning/test_post_processing.py b/tests/inference/unit_tests/core/active_learning/test_post_processing.py index e3ee529391..b9955cbded 100644 --- a/tests/inference/unit_tests/core/active_learning/test_post_processing.py +++ b/tests/inference/unit_tests/core/active_learning/test_post_processing.py @@ -1,4 +1,6 @@ import json +from copy import deepcopy +from unittest.mock import MagicMock import pytest @@ -11,6 +13,84 @@ encode_prediction, ) from inference.core.exceptions import PredictionFormatNotSupported +from inference.core.managers.active_learning import ActiveLearningManager + + +@pytest.mark.parametrize("response_format", ["polygon", "rle", "empty"]) +@pytest.mark.parametrize("as_batch", [False, True]) +def test_reduced_grid_annotations_reach_active_learning_in_image_coordinates( + response_format, as_batch +) -> None: + prediction = { + "image": {"height": 4, "width": 5}, + "original_image": {"height": 40, "width": 100}, + "mask_metadata": { + "coordinate_system": "mask_grid", + "height": 4, + "width": 5, + "scale_x": 20.0, + "scale_y": 10.0, + }, + "predictions": [ + { + "x": 2.0, + "y": 2.0, + "width": 2.0, + "height": 2.0, + "points": [ + {"x": 1.0, "y": 1.0}, + {"x": 1.0, "y": 2.0}, + {"x": 2.0, "y": 2.0}, + {"x": 2.0, "y": 1.0}, + ], + } + ], + } + if response_format == "rle": + prediction["predictions"][0].pop("points") + prediction["predictions"][0]["rle"] = { + "size": [4, 5], + "counts": [5, 2, 2, 2, 9], + } + elif response_format == "empty": + prediction["predictions"] = [] + original = deepcopy(prediction) + + response = MagicMock() + response.dict.return_value = prediction + request = MagicMock(image="test-image", disable_preproc_auto_orient=False) + manager = ActiveLearningManager.__new__(ActiveLearningManager) + manager.get_task_type = MagicMock(return_value="instance-segmentation") + middleware = MagicMock() + manager._middlewares = {"test": middleware} + + manager.register_datapoint( + prediction=[response] if as_batch else response, + model_id="test/1", + request=request, + middleware_key="test", + ) + result = middleware.register_batch.call_args.kwargs["predictions"][0] + + assert prediction == original + assert result["image"] == { + "height": 40, + "width": 100, + } + assert "mask_metadata" not in result + assert "original_image" not in result + if response_format != "empty": + detection = result["predictions"][0] + assert detection["x"] == 40 + assert detection["y"] == 20 + assert detection["width"] == 40 + assert detection["height"] == 20 + assert "rle" not in detection + assert {(p["x"], p["y"]) for p in detection["points"]} == { + (x, y) for x, y in [(20, 10), (20, 20), (40, 20), (40, 10)] + } + else: + assert result["predictions"] == [] def test_encode_prediction_when_non_classification_task_prediction_is_given() -> None: diff --git a/tests/inference/unit_tests/core/models/test_inference_models_adapters.py b/tests/inference/unit_tests/core/models/test_inference_models_adapters.py index 20d5ad3658..cf297ba3b6 100644 --- a/tests/inference/unit_tests/core/models/test_inference_models_adapters.py +++ b/tests/inference/unit_tests/core/models/test_inference_models_adapters.py @@ -1202,7 +1202,7 @@ def test_crop_response_uses_the_frame_represented_by_mask( )[0] coordinate_metadata = response.mask_metadata - if response_format == "rle" and opt_in and factor != 1.0: + if opt_in: assert coordinate_metadata.height == detections.mask_size[0] assert coordinate_metadata.width == detections.mask_size[1] assert coordinate_metadata.coordinate_system == "mask_grid" @@ -1210,7 +1210,15 @@ def test_crop_response_uses_the_frame_represented_by_mask( assert coordinate_metadata is None if response_format == "rle": - converted = sv.Detections.from_inference(response.model_dump(by_alias=True)) + serialized = response.model_dump(by_alias=True) + if opt_in: + assert serialized["image"] == { + "width": detections.mask_size[1], + "height": detections.mask_size[0], + } + assert serialized["original_image"] == {"width": 300, "height": 200} + serialized["image"] = serialized["original_image"] + converted = sv.Detections.from_inference(serialized) expected = np.zeros((1, 200, 300), dtype=bool) expected[:, 10 + offset_y : 50 + offset_y, 20 + offset_x : 80 + offset_x] = True assert converted.mask.shape == expected.shape @@ -1226,15 +1234,17 @@ def test_crop_response_uses_the_frame_represented_by_mask( return prediction = response.predictions[0] - xs = [point.x for point in prediction.points] - ys = [point.y for point in prediction.points] + scale_x = coordinate_metadata.scale_x if coordinate_metadata else 1.0 + scale_y = coordinate_metadata.scale_y if coordinate_metadata else 1.0 + xs = [point.x * scale_x for point in prediction.points] + ys = [point.y * scale_y for point in prediction.points] assert min(xs) == pytest.approx(20 + offset_x) assert min(ys) == pytest.approx(10 + offset_y) pixel_size = 150 / round(30 * (1 - factor) + 150 * factor) assert max(xs) == pytest.approx(80 + offset_x - pixel_size) assert max(ys) == pytest.approx(50 + offset_y - pixel_size) - assert prediction.x == pytest.approx(50 + offset_x) - assert prediction.y == pytest.approx(30 + offset_y) + assert prediction.x * scale_x == pytest.approx(50 + offset_x) + assert prediction.y * scale_y == pytest.approx(30 + offset_y) @pytest.mark.parametrize("use_dc", [False, True]) @@ -1268,21 +1278,19 @@ def test_deferred_response_preserves_coordinate_contract( if use_dc and response_format == "polygon" else response.model_dump(exclude_none=True) ) - assert serialized["image"] == {"width": 10, "height": 20} - if response_format == "rle": - assert serialized["mask_metadata"] == { - "coordinate_system": "mask_grid", - "width": 2, - "height": 5, - "scale_x": 5.0, - "scale_y": 4.0, - } - else: - assert "mask_metadata" not in serialized + assert serialized["image"] == {"width": 2, "height": 5} + assert serialized["original_image"] == {"width": 10, "height": 20} + assert serialized["mask_metadata"] == { + "coordinate_system": "mask_grid", + "width": 2, + "height": 5, + "scale_x": 5.0, + "scale_y": 4.0, + } assert len(serialized["predictions"]) == count if not empty and response_format == "polygon": - assert max(p["x"] for p in serialized["predictions"][0]["points"]) == 5 - assert max(p["y"] for p in serialized["predictions"][0]["points"]) == 16 + assert max(p["x"] for p in serialized["predictions"][0]["points"]) == 1 + assert max(p["y"] for p in serialized["predictions"][0]["points"]) == 4 @pytest.mark.parametrize("mask_format", ["dense", "rle"]) @@ -1314,11 +1322,148 @@ def test_opted_in_polygons_share_box_coordinates_on_non_square_image(mask_format prediction = response.predictions[0] xs = [point.x for point in prediction.points] ys = [point.y for point in prediction.points] - assert response.mask_metadata is None - assert (min(xs), min(ys), max(xs), max(ys)) == (300, 240, 443.75, 355) + assert response.mask_metadata.scale_x == 6.25 + assert response.mask_metadata.scale_y == 5.0 + assert response.image.width == response.image.height == 160 + assert (min(xs), min(ys), max(xs), max(ys)) == (48, 48, 71, 71) converted = sv.Detections.from_inference(response.model_dump(by_alias=True)) - np.testing.assert_array_equal(converted.xyxy, [[300, 240, 450, 360]]) + np.testing.assert_array_equal(converted.xyxy, [[48, 48, 72, 72]]) + np.testing.assert_allclose( + sv.mask_to_xyxy(converted.mask), [[48, 48, 71, 71]], atol=1 + ) + sv.MaskAnnotator().annotate(np.zeros((160, 160, 3), dtype=np.uint8), converted) + + +@pytest.mark.parametrize("opt_in", [False, True]) +@pytest.mark.parametrize("response_format", ["polygon", "rle"]) +@pytest.mark.parametrize("factor", [0.0, 0.5, 1.0]) +def test_response_geometry_uses_one_grid(opt_in, response_format, factor): + from inference_models.entities import ImageDimensions + from inference_models.models.common.roboflow.model_packages import StaticCropOffset + from inference_models.models.common.roboflow.post_processing import ( + align_instance_segmentation_results, + ) + + original_size = ImageDimensions(height=800, width=1000) + input_boxes = torch.tensor([[300, 240, 450, 360]], dtype=torch.float32) + input_masks = torch.full((1, 160, 160), -1.0) + input_masks[:, 48:72, 48:72] = 1.0 + adapter = _seg_adapter() + + def post_process(predictions, metadata, **kwargs): + assert "allow_reduced_mask_resolution" not in kwargs + boxes, masks = align_instance_segmentation_results( + image_bboxes=input_boxes.clone(), + masks=input_masks.clone(), + padding=(0, 0, 0, 0), + scale_width=1.0, + scale_height=1.0, + original_size=original_size, + size_after_pre_processing=original_size, + inference_size=original_size, + static_crop_offset=StaticCropOffset( + offset_x=0, offset_y=0, crop_width=1000, crop_height=800 + ), + masks_resolution_factor=kwargs["masks_resolution_factor"], + ) + return [ + InstanceDetections( + xyxy=boxes, + confidence=torch.tensor([0.9]), + class_id=torch.tensor([0]), + mask=masks, + ) + ] + + adapter._model = SimpleNamespace( + supported_mask_formats=[], post_process=post_process + ) + response = adapter._postprocess_sync( + [], + [SimpleNamespace(original_size=original_size)], + allow_reduced_mask_resolution=opt_in, + mask_decode_mode="tradeoff", + tradeoff_factor=factor, + response_mask_format=response_format, + )[0] + width = round(160 * (1 - factor) + 1000 * factor) if opt_in else 1000 + height = round(160 * (1 - factor) + 800 * factor) if opt_in else 800 + assert response.image.width == width + assert response.image.height == height + prediction = response.predictions[0] np.testing.assert_allclose( - sv.mask_to_xyxy(converted.mask), [[300, 240, 443.75, 355]], atol=1 + [prediction.x, prediction.y, prediction.width, prediction.height], + np.array([375, 300, 150, 120]) + * [width / 1000, height / 800, width / 1000, height / 800], + ) + if opt_in: + assert response.original_image.model_dump() == {"width": 1000, "height": 800} + assert response.mask_metadata.scale_x == pytest.approx(1000 / width) + assert response.mask_metadata.scale_y == pytest.approx(800 / height) + else: + assert response.original_image is None and response.mask_metadata is None + if response_format == "rle": + assert prediction.rle["size"] == [height, width] + converted = sv.Detections.from_inference(response.model_dump(by_alias=True)) + assert converted.mask.shape == (1, height, width) + np.testing.assert_allclose( + sv.mask_to_xyxy(converted.mask), converted.xyxy, atol=1.1 + ) + sv.MaskAnnotator().annotate(np.zeros((height, width, 3), dtype=np.uint8), converted) + + +@pytest.mark.parametrize("response_format", ["polygon", "rle"]) +def test_async_opt_in_survives_factor_one(response_format): + adapter = _seg_adapter() + adapter._pipeline_depth = 2 + detections = InstanceDetections( + xyxy=torch.tensor([[0, 0, 10, 20]], dtype=torch.float32), + confidence=torch.tensor([0.9]), + class_id=torch.tensor([0]), + mask=torch.ones((1, 20, 10), dtype=torch.bool), + ) + future = SimpleNamespace(result=lambda: [detections]) + adapter._model = SimpleNamespace( + supported_mask_formats=[], forward_async=lambda *args, **kwargs: future + ) + adapter._submit_next_pending_gpu_work = lambda: None + adapter._submit_ready_responses = lambda: None + returned_future = adapter.predict( + torch.zeros((1, 3, 20, 10)), + allow_reduced_mask_resolution=True, + mask_decode_mode="accurate", + response_mask_format=response_format, + ) + from inference_models.models.base.async_handoff import get_adapter_mapped_kwargs + + mapped_kwargs = get_adapter_mapped_kwargs(returned_future) + assert "allow_reduced_mask_resolution" not in mapped_kwargs + response = adapter._finalize_future( + returned_future, _make_meta("accurate"), mapped_kwargs + )[0] + assert response.original_image.model_dump() == {"width": 10, "height": 20} + assert response.mask_metadata.scale_x == response.mask_metadata.scale_y == 1.0 + + +def test_native_grid_batch_does_not_mutate_model_boxes(): + adapter = _seg_adapter() + boxes = torch.tensor([[20, 10, 60, 30]], dtype=torch.float32) + det = InstanceDetections( + xyxy=boxes, + confidence=torch.tensor([0.9]), + class_id=torch.tensor([0]), + mask=torch.ones((1, 10, 20), dtype=torch.bool), + ) + metadata = [ + SimpleNamespace(original_size=SimpleNamespace(width=100, height=50)), + SimpleNamespace(original_size=SimpleNamespace(width=200, height=200)), + ] + responses = adapter._build_responses_from_detections( + [det, det], metadata, allow_reduced_mask_resolution=True + ) + assert [r.predictions[0].x for r in responses] == [8, 4] + assert [r.predictions[0].y for r in responses] == [4, 1] + assert [r.mask_metadata.scale_x for r in responses] == [5, 10] + torch.testing.assert_close( + boxes, torch.tensor([[20, 10, 60, 30]], dtype=torch.float32) ) - sv.MaskAnnotator().annotate(np.zeros((800, 1000, 3), dtype=np.uint8), converted) diff --git a/tests/inference/unit_tests/core/utils/test_visualisation.py b/tests/inference/unit_tests/core/utils/test_visualisation.py index 1fc6d15ce9..b81b07f2a6 100644 --- a/tests/inference/unit_tests/core/utils/test_visualisation.py +++ b/tests/inference/unit_tests/core/utils/test_visualisation.py @@ -68,3 +68,55 @@ def test_polygon_visualization_uses_image_coordinates_without_mutation( assert isinstance(result, bytes) assert draw_polygon.call_args.kwargs["points"] == [Point(x=25, y=32)] assert box.points == [Point(x=25, y=32)] + + +def test_native_grid_visualization_projects_boxes_and_points_without_mutation( + monkeypatch, +): + from types import SimpleNamespace + from unittest.mock import MagicMock + + import numpy as np + + from inference.core.entities.responses.inference import ( + InstanceSegmentationPrediction, + MaskCoordinateMetadata, + Point, + ) + from inference.core.utils import visualisation + + box = InstanceSegmentationPrediction( + x=10, + y=12.5, + width=8, + height=10, + confidence=0.9, + class_id=0, + points=[Point(x=5, y=8)], + **{"class": "car"}, + ) + response = SimpleNamespace( + predictions=[box], + mask_metadata=MaskCoordinateMetadata(width=20, height=25, scale_x=5, scale_y=4), + ) + scene = np.zeros((100, 100, 3), dtype=np.uint8) + draw_box = MagicMock(return_value=scene) + draw_polygon = MagicMock(return_value=scene) + monkeypatch.setattr(visualisation, "load_image_rgb", lambda _: scene) + monkeypatch.setattr(visualisation, "draw_bbox", draw_box) + monkeypatch.setattr( + visualisation, "draw_instance_segmentation_points", draw_polygon + ) + + visualisation.draw_detection_predictions( + inference_request=SimpleNamespace( + image=None, visualization_stroke_width=1, visualization_labels=False + ), + inference_response=response, + colors={"car": "#ffffff"}, + ) + drawn = draw_box.call_args.kwargs["box"] + assert (drawn.x, drawn.y, drawn.width, drawn.height) == (50, 50, 40, 40) + assert draw_polygon.call_args.kwargs["points"] == [Point(x=25, y=32)] + assert (box.x, box.y, box.width, box.height) == (10, 12.5, 8, 10) + assert box.points == [Point(x=5, y=8)] diff --git a/tests/inference_sdk/unit_tests/http/test_client.py b/tests/inference_sdk/unit_tests/http/test_client.py index 5ffbbfab84..feb761babc 100644 --- a/tests/inference_sdk/unit_tests/http/test_client.py +++ b/tests/inference_sdk/unit_tests/http/test_client.py @@ -1399,11 +1399,11 @@ def test_infer_from_api_v0_when_request_succeed_for_object_detection_with_batch_ ] assert ( requests_mock.request_history[0].query - == "api_key=my-api-key&confidence=0.5&disable_active_learning=false" + == "api_key=my-api-key&confidence=0.5&allow_reduced_mask_resolution=false&disable_active_learning=false" ) assert ( requests_mock.request_history[1].query - == "api_key=my-api-key&confidence=0.5&disable_active_learning=false" + == "api_key=my-api-key&confidence=0.5&allow_reduced_mask_resolution=false&disable_active_learning=false" ) @@ -1426,7 +1426,7 @@ async def test_infer_from_api_v0_async_when_request_succeed_for_object_detection with aioresponses() as m: m.post( - f"{api_url}/coco/3?api_key=my-api-key&confidence=0.5&disable_active_learning=False", + f"{api_url}/coco/3?api_key=my-api-key&confidence=0.5&allow_reduced_mask_resolution=False&disable_active_learning=False", payload={ "image": {"height": 480, "width": 640}, "predictions": [ @@ -1442,7 +1442,7 @@ async def test_infer_from_api_v0_async_when_request_succeed_for_object_detection }, ) m.post( - f"{api_url}/coco/3?api_key=my-api-key&confidence=0.5&disable_active_learning=False", + f"{api_url}/coco/3?api_key=my-api-key&confidence=0.5&allow_reduced_mask_resolution=False&disable_active_learning=False", payload={ "image": {"height": 480, "width": 640}, "predictions": [ @@ -1548,7 +1548,7 @@ def test_infer_from_api_v0_when_request_succeed_for_object_detection_with_visual } assert ( requests_mock.last_request.query - == "api_key=my-api-key&confidence=0.5&format=image_and_json&disable_active_learning=false" + == "api_key=my-api-key&confidence=0.5&format=image_and_json&allow_reduced_mask_resolution=false&disable_active_learning=false" ) @@ -1578,7 +1578,7 @@ def test_infer_from_api_v0_when_request_succeed_for_object_detection_with_visual assert result == {"visualization": base64.b64encode(b"data").decode("utf-8")} assert ( requests_mock.last_request.query - == "api_key=my-api-key&confidence=0.5&format=image&disable_active_learning=false" + == "api_key=my-api-key&confidence=0.5&format=image&allow_reduced_mask_resolution=false&disable_active_learning=false" ) @@ -1595,7 +1595,7 @@ async def test_infer_from_api_v0_async_when_request_succeed_for_object_detection http_client.configure(inference_configuration=configuration) with aioresponses() as m: m.post( - f"{api_url}/some/1?api_key=my-api-key&confidence=0.5&disable_active_learning=False&format=image", + f"{api_url}/some/1?api_key=my-api-key&confidence=0.5&allow_reduced_mask_resolution=False&disable_active_learning=False&format=image", body=b"data", headers={"content-type": "image/jpeg"}, ) @@ -1657,7 +1657,7 @@ def test_infer_from_api_v0_when_request_succeed_for_object_detection( } assert ( requests_mock.last_request.query - == "api_key=my-api-key&confidence=0.5&disable_active_learning=false" + == "api_key=my-api-key&confidence=0.5&allow_reduced_mask_resolution=false&disable_active_learning=false" ) @@ -1675,7 +1675,7 @@ async def test_infer_from_api_v0_async_when_request_succeed_for_object_detection with aioresponses() as m: m.post( - f"{api_url}/some/1?api_key=my-api-key&confidence=0.5&disable_active_learning=False", + f"{api_url}/some/1?api_key=my-api-key&confidence=0.5&allow_reduced_mask_resolution=False&disable_active_learning=False", payload={ "image": {"height": 480, "width": 640}, "predictions": [ diff --git a/tests/inference_sdk/unit_tests/http/test_entities.py b/tests/inference_sdk/unit_tests/http/test_entities.py index e591f6b1eb..c264ee2006 100644 --- a/tests/inference_sdk/unit_tests/http/test_entities.py +++ b/tests/inference_sdk/unit_tests/http/test_entities.py @@ -96,6 +96,7 @@ def test_to_api_call_parameters_for_api_v0() -> None: "format": "json", "labels": True, "mask_decode_mode": "fast", + "allow_reduced_mask_resolution": False, "tradeoff_factor": 0.0, "max_detections": 20, "overlap": 0.7, @@ -310,3 +311,15 @@ def test_mask_resolution_opt_in_does_not_leak_to_object_detection(): ) assert "allow_reduced_mask_resolution" not in parameters + + +@pytest.mark.parametrize("client_mode", [HTTPClientMode.V0, HTTPClientMode.V1]) +def test_mask_resolution_opt_in_defaults_to_false(client_mode): + configuration = InferenceConfiguration() + assert configuration.allow_reduced_mask_resolution is False + assert ( + configuration.to_api_call_parameters( + client_mode=client_mode, task_type=INSTANCE_SEGMENTATION_TASK + )["allow_reduced_mask_resolution"] + is False + ) diff --git a/tests/inference_sdk/unit_tests/http/utils/test_postprocessing.py b/tests/inference_sdk/unit_tests/http/utils/test_postprocessing.py index edfc7079b6..8d2bfc9c52 100644 --- a/tests/inference_sdk/unit_tests/http/utils/test_postprocessing.py +++ b/tests/inference_sdk/unit_tests/http/utils/test_postprocessing.py @@ -695,7 +695,8 @@ def test_filter_model_descriptions_when_input_with_multiple_matches_given() -> N @pytest.mark.parametrize("mask_format", ["polygon", "rle", "empty"]) def test_client_downsizing_preserves_mask_coordinate_contract(mask_format): prediction = { - "image": {"width": 301, "height": 199}, + "image": {"width": 100, "height": 50}, + "original_image": {"width": 301, "height": 199}, "mask_metadata": { "coordinate_system": "mask_grid", "width": 100, @@ -706,7 +707,6 @@ def test_client_downsizing_preserves_mask_coordinate_contract(mask_format): "predictions": [{"x": 100, "y": 60, "width": 20, "height": 30}], } if mask_format == "polygon": - prediction.pop("mask_metadata") prediction["predictions"][0]["points"] = [{"x": 75, "y": 40}] elif mask_format == "rle": prediction["predictions"][0]["rle"] = {"size": [50, 100], "counts": "fixture"} @@ -715,19 +715,17 @@ def test_client_downsizing_preserves_mask_coordinate_contract(mask_format): result = adjust_prediction_to_client_scaling_factor(prediction, scaling_factor=0.6) - assert result["image"] == {"width": 502, "height": 332} - if mask_format == "polygon": - assert "mask_metadata" not in result - else: - assert result["mask_metadata"]["width"] == 100 - assert result["mask_metadata"]["height"] == 50 - assert result["mask_metadata"]["scale_x"] == pytest.approx(5.02) - assert result["mask_metadata"]["scale_y"] == pytest.approx(6.64) + assert result["image"] == {"width": 100, "height": 50} + assert result["original_image"] == {"width": 502, "height": 332} + assert result["mask_metadata"]["width"] == 100 + assert result["mask_metadata"]["height"] == 50 + assert result["mask_metadata"]["scale_x"] == pytest.approx(5.02) + assert result["mask_metadata"]["scale_y"] == pytest.approx(6.64) if mask_format != "empty": - assert result["predictions"][0]["x"] == pytest.approx(100 / 0.6) + assert result["predictions"][0]["x"] == pytest.approx(100) if mask_format == "polygon": - assert result["predictions"][0]["points"][0]["x"] == pytest.approx(75 / 0.6) - assert result["predictions"][0]["points"][0]["y"] == pytest.approx(40 / 0.6) + assert result["predictions"][0]["points"][0]["x"] == pytest.approx(75) + assert result["predictions"][0]["points"][0]["y"] == pytest.approx(40) if mask_format == "rle": assert result["predictions"][0]["rle"] == { "size": [50, 100], diff --git a/workflows/CHANGELOG.md b/workflows/CHANGELOG.md index 5763e5e6b8..85103f1d2a 100644 --- a/workflows/CHANGELOG.md +++ b/workflows/CHANGELOG.md @@ -33,10 +33,11 @@ for contributor and maintainer responsibilities. ### Added -- Optional `mask_metadata` on instance-segmentation response DTOs describes the - encoded mask grid and coordinate mapping for opted-in RLE HTTP responses. - Polygon points and bounding boxes retain image coordinates. Existing workflow - blocks do not opt in and retain their image-space contract. +- Optional `original_image` and `mask_metadata` on instance-segmentation response + DTOs retain the input dimensions and output-to-input coordinate mapping for + opted-in HTTP responses. Response `image`, boxes, polygons and RLE masks all + use the selected mask grid. Existing workflow blocks do not opt in and retain + their image-space contract. - `ModelsConfiguration.use_inference_models`, defaulting to `True`. Hosts using the legacy backend must set it to `False` to retain non-tensor local legacy diff --git a/workflows/roboflow_workflows/core_steps/common/inference_response_entities.py b/workflows/roboflow_workflows/core_steps/common/inference_response_entities.py index 327a336ad7..cad36a90f2 100644 --- a/workflows/roboflow_workflows/core_steps/common/inference_response_entities.py +++ b/workflows/roboflow_workflows/core_steps/common/inference_response_entities.py @@ -26,13 +26,15 @@ class InferenceResponseImage(BaseModel): """Inference response image information. Attributes: - width (int): The original width of the image used in inference. - height (int): The original height of the image used in inference. + width (int): The width of the represented image coordinate frame. + height (int): The height of the represented image coordinate frame. """ - width: int = Field(description="The original width of the image used in inference") + width: int = Field( + description="The width of the represented image coordinate frame" + ) height: int = Field( - description="The original height of the image used in inference" + description="The height of the represented image coordinate frame" ) @@ -122,10 +124,10 @@ def serialize_visualisation(self, visualization: Optional[Any]) -> Optional[str] class MaskCoordinateMetadata(BaseModel): - """Describe the mask grid and its mapping to the response image. + """Describe the prediction grid and its mapping to the original image. Attributes: - coordinate_system (str): Coordinate frame for encoded RLE masks. + coordinate_system (str): Coordinate frame shared by boxes, polygons and encoded RLE masks. width (int): Width of the encoded mask grid. height (int): Height of the encoded mask grid. scale_x (float): Multiply mask x coordinates by this to obtain image x. @@ -134,15 +136,19 @@ class MaskCoordinateMetadata(BaseModel): coordinate_system: Literal["mask_grid"] = Field( default="mask_grid", - description="RLE masks use this grid; polygon points use image coordinates.", + description="All prediction geometry uses this grid, matching response image dimensions.", ) width: int = Field(gt=0, description="Mask-grid width in pixels.", examples=[160]) height: int = Field(gt=0, description="Mask-grid height in pixels.", examples=[107]) scale_x: float = Field( - gt=0, description="Image width divided by mask width.", examples=[4.0] + gt=0, + description="Original image width divided by output-grid width.", + examples=[4.0], ) scale_y: float = Field( - gt=0, description="Image height divided by mask height.", examples=[4.0] + gt=0, + description="Original image height divided by output-grid height.", + examples=[4.0], ) @@ -161,9 +167,19 @@ class InstanceSegmentationInferenceResponse( predictions: List[ Union[InstanceSegmentationPrediction, InstanceSegmentationRLEPrediction] ] + image: InferenceResponseImage = Field( + description="Dimensions of the coordinate frame shared by bounding boxes, " + "polygon points and RLE masks. With opt-in this is the selected mask grid; " + "otherwise it is the original image.", + ) + original_image: Optional[InferenceResponseImage] = Field( + default=None, + description="Original image dimensions retained for opted-in responses, " + "including when the selected grid has the same dimensions.", + ) mask_metadata: Optional[MaskCoordinateMetadata] = Field( default=None, - description="Present for opted-in RLE responses whose encoded mask grid " - "differs from the image. Describes the RLE grid and its mapping to image " - "coordinates. Bounding boxes and polygon points always use image coordinates.", + description="Present for opted-in responses in either format. All prediction " + "geometry uses this grid. Multiply x coordinates and widths by scale_x, " + "and y coordinates and heights by scale_y, to map to original_image.", ) From ed035a0a27f244812ce1db01e9840aa4502fa2ff Mon Sep 17 00:00:00 2001 From: Damian Kosowski Date: Fri, 9 Oct 2026 14:35:00 +0200 Subject: [PATCH 32/41] Support configurable RF-DETR Triton mask grids --- docs/instance_segmentation_mask_resolution.md | 9 +- inference_models/docs/changelog.md | 10 +- .../models/rfdetr/triton_postprocess.py | 103 +++++-- .../models/rfdetr/test_triton_postprocess.py | 266 ++++++++++++++++-- 4 files changed, 331 insertions(+), 57 deletions(-) diff --git a/docs/instance_segmentation_mask_resolution.md b/docs/instance_segmentation_mask_resolution.md index bea02261c0..751c57a661 100644 --- a/docs/instance_segmentation_mask_resolution.md +++ b/docs/instance_segmentation_mask_resolution.md @@ -94,8 +94,13 @@ decode-mode behavior and ignores the new opt-in flag. Values from 0 to 1 interpolate model-grid and image-grid dimensions. A lower factor does not always mean a smaller grid or faster execution. `accurate` selects factor 1.0, `fast` selects 0.0, and only `tradeoff` reads `tradeoff_factor`. -Damian's Triton optimization is separate; factors other than 1.0 currently use -the fallback post-processor. +The RF-DETR Triton RLE post-processor supports native and upsampled mask grids +at every factor. It preserves image-space boxes and records the encoded grid in +`mask_size`, including for empty and deferred results. It uses the same target-size +rounding as the reference path. Antialiased downsampling, unsupported preprocessing +transforms, and other existing compatibility limits use the reference post-processor. +The existing Triton enablement flag still controls this path; no execution plan is +required. Workflow versions v1–v4 retain their compatibility guards. Workflow v5, reduced-mask downstream handling, and full tensor support are deferred to a diff --git a/inference_models/docs/changelog.md b/inference_models/docs/changelog.md index e7020833eb..2d588f471f 100644 --- a/inference_models/docs/changelog.md +++ b/inference_models/docs/changelog.md @@ -43,10 +43,12 @@ COCO `size`. `InstanceDetections.to_supervision()` resizes masks to image resolution for annotation; manually constructed reduced dense detections must supply `image_size` because it cannot be inferred from the reduced tensor. -- The RF-DETR fused Triton post-processor supports factor `1.0` only. Other - factors use the fallback post-processor so the requested resolution is honored. - Reduced-resolution Triton support and its benchmarking are deferred to a - separate PR. +- The RF-DETR fused Triton RLE post-processor now honors `masks_resolution_factor` + for native and upsampled mask grids, including multiclass and deferred streaming + results. Boxes retain image coordinates, while RLE and empty results carry the + selected mask-grid dimensions. Antialiased downsampling and previously unsupported + transforms use the reference post-processor. Multiclass results respect RF-DETR's + query-count cap even when `max_detections` is larger. - The `inference_models` adapter now rejects unknown `mask_decode_mode` values with `InvalidMaskDecodeArgument`; it previously ignored the mode. The legacy backend already rejected invalid modes. Validation also runs without opt-in. diff --git a/inference_models/inference_models/models/rfdetr/triton_postprocess.py b/inference_models/inference_models/models/rfdetr/triton_postprocess.py index 9a810f8d29..abf36b4385 100644 --- a/inference_models/inference_models/models/rfdetr/triton_postprocess.py +++ b/inference_models/inference_models/models/rfdetr/triton_postprocess.py @@ -40,6 +40,9 @@ from inference_models.models.base.instance_segmentation import InstanceDetections from inference_models.models.base.types import InstancesRLEMasks from inference_models.models.common.roboflow.model_packages import PreProcessingMetadata +from inference_models.models.common.roboflow.post_processing import ( + resolve_mask_target_size, +) from inference_models.models.rfdetr.class_remapping import ClassesReMapping try: @@ -233,6 +236,10 @@ def post_process_single_instance_segmentation_result_to_rle_masks_triton( are supported. Returns ``None`` when the caller should use the reference PyTorch/RLE implementation instead. + Masks use the grid selected by ``masks_resolution_factor``; boxes and + image metadata remain in original-image coordinates. Unsupported transforms + and antialiased downsampling use the reference path. + The fast path first emits one candidate per query. If any query has more than one class above threshold, the first pass asks for a retry and the second pass emits up to ``_SPARSE_MAX_CLASSES_PER_QUERY`` query-class @@ -265,11 +272,20 @@ def post_process_single_instance_segmentation_result_to_rle_masks_triton( image_masks = image_masks.contiguous() class_mapping = classes_re_mapping.class_mapping.contiguous() num_queries, num_classes = image_scores.shape - if max_detections is None: - max_detections = num_queries + # The reference selects at most num_queries query/class pairs before + # applying max_detections, including in the multiclass retry path. + max_detections = ( + num_queries if max_detections is None else min(max_detections, num_queries) + ) mask_height, mask_width = image_masks.shape[-2:] - output_height = image_meta.original_size.height - output_width = image_meta.original_size.width + image_height, image_width = image_meta.original_size + image_size = (image_height, image_width) + output_height, output_width = resolve_mask_target_size( + mask_height=mask_height, + mask_width=mask_width, + size_after_pre_processing=image_meta.size_after_pre_processing, + masks_resolution_factor=masks_resolution_factor, + ) confidence_threshold = float(threshold) # Precompute resize tables outside the hot kernel. The tables are tiny @@ -324,8 +340,8 @@ def post_process_single_instance_segmentation_result_to_rle_masks_triton( num_queries, num_classes, class_mapping.shape[0], - output_height, - output_width, + image_height, + image_width, BLOCK_CLASSES=triton.next_power_of_2(num_classes), METADATA_STRIDE=_HEADER_SIZE, MAX_CLASSES_PER_QUERY=_SPARSE_MAX_CLASSES_PER_QUERY, @@ -398,6 +414,7 @@ def post_process_single_instance_segmentation_result_to_rle_masks_triton( max_total_runs=_SPARSE_MAX_TOTAL_RUNS, height=output_height, width=output_width, + image_size=image_size, max_detections=max_detections, ) @@ -429,8 +446,8 @@ def post_process_single_instance_segmentation_result_to_rle_masks_triton( num_queries, num_classes, class_mapping.shape[0], - output_height, - output_width, + image_height, + image_width, BLOCK_CLASSES=triton.next_power_of_2(num_classes), METADATA_STRIDE=_HEADER_SIZE, FLAG_MULTICLASS=True, @@ -487,6 +504,7 @@ def post_process_single_instance_segmentation_result_to_rle_masks_triton( max_total_runs=_SPARSE_MAX_TOTAL_RUNS, height=output_height, width=output_width, + image_size=image_size, max_detections=max_detections, ) if result is not None: @@ -530,8 +548,8 @@ def post_process_single_instance_segmentation_result_to_rle_masks_triton( num_queries, num_classes, class_mapping.shape[0], - output_height, - output_width, + image_height, + image_width, BLOCK_CLASSES=triton.next_power_of_2(num_classes), METADATA_STRIDE=_HEADER_SIZE, MAX_CLASSES_PER_QUERY=_SPARSE_MAX_CLASSES_PER_QUERY, @@ -592,6 +610,7 @@ def post_process_single_instance_segmentation_result_to_rle_masks_triton( max_total_runs=_SPARSE_TOPK_MAX_TOTAL_RUNS, height=output_height, width=output_width, + image_size=image_size, max_detections=max_detections, ) @@ -603,6 +622,8 @@ def _instance_detections_from_sparse_records( height: int, width: int, max_detections: Optional[int] = None, + *, + image_size: Optional[Tuple[int, int]] = None, ) -> Optional[InstanceDetections]: """Convert sparse device records into ``InstanceDetections``. @@ -613,6 +634,7 @@ def _instance_detections_from_sparse_records( ``None`` means the sparse device result is incomplete or overflowed and the caller should retry or fall back to the reference implementation. """ + image_size = image_size or (height, width) active_ranks = np.flatnonzero(metadata_host[:, 0] > 0.5) if active_ranks.size == 0: return InstanceDetections( @@ -620,9 +642,12 @@ def _instance_detections_from_sparse_records( confidence=torch.empty((0,), dtype=torch.float32), class_id=torch.empty((0,), dtype=torch.int32), mask=InstancesRLEMasks.from_coco_rle_masks( - image_size=(height, width), + image_size=image_size, + mask_size=(height, width), masks=[], ), + image_size=image_size, + mask_frame_size=image_size, ) if np.any(metadata_host[active_ranks, 8] > 0.5): return None @@ -675,7 +700,8 @@ def _instance_detections_from_sparse_records( rle_masks.append(_rle_from_counts(counts=counts, height=height, width=width)) instances_masks = InstancesRLEMasks.from_coco_rle_masks( - image_size=(height, width), + image_size=image_size, + mask_size=(height, width), masks=rle_masks, ) # The pipeline RLE-to-polygon path consumes uncompressed counts directly, so @@ -686,6 +712,8 @@ def _instance_detections_from_sparse_records( confidence=confidence, class_id=class_id, mask=instances_masks, + image_size=image_size, + mask_frame_size=image_size, ) @@ -696,6 +724,8 @@ def _instance_detections_from_sparse_query_records( height: int, width: int, max_detections: Optional[int] = None, + *, + image_size: Optional[Tuple[int, int]] = None, ) -> Optional[InstanceDetections]: """Assemble detections when class rows share query-level RLE records. @@ -704,6 +734,7 @@ def _instance_detections_from_sparse_query_records( once per query and this CPU helper fans that query mask out to the selected class detections. """ + image_size = image_size or (height, width) active_ranks = np.flatnonzero(class_metadata_host[:, 0] > 0.5) if active_ranks.size == 0: return InstanceDetections( @@ -711,9 +742,12 @@ class detections. confidence=torch.empty((0,), dtype=torch.float32), class_id=torch.empty((0,), dtype=torch.int32), mask=InstancesRLEMasks.from_coco_rle_masks( - image_size=(height, width), + image_size=image_size, + mask_size=(height, width), masks=[], ), + image_size=image_size, + mask_frame_size=image_size, ) if np.any(class_metadata_host[active_ranks, 8] > 0.5): return None @@ -779,7 +813,8 @@ class detections. rle_masks.append(_rle_from_counts(counts=counts, height=height, width=width)) instances_masks = InstancesRLEMasks.from_coco_rle_masks( - image_size=(height, width), + image_size=image_size, + mask_size=(height, width), masks=rle_masks, ) # The pipeline RLE-to-polygon path consumes uncompressed counts directly, so @@ -790,6 +825,8 @@ class detections. confidence=confidence, class_id=class_id, mask=instances_masks, + image_size=image_size, + mask_frame_size=image_size, ) @@ -826,6 +863,8 @@ def _deferred_instance_detections_from_sparse_query_records( height: int, width: int, max_detections: Optional[int], + *, + image_size: Optional[Tuple[int, int]] = None, ) -> InstanceDetections: """Return a placeholder detection object with deferred CPU finalization. @@ -835,6 +874,8 @@ def _deferred_instance_detections_from_sparse_query_records( ``InstanceDetections``. """ + image_size = image_size or (height, width) + def finalize() -> InstanceDetections: """Synchronize the DtoH copies and build the real detections.""" try: @@ -848,6 +889,7 @@ def finalize() -> InstanceDetections: max_total_runs=max_total_runs, height=height, width=width, + image_size=image_size, max_detections=max_detections, ) if result is None: @@ -869,9 +911,12 @@ def finalize() -> InstanceDetections: confidence=torch.empty((0,), dtype=torch.float32), class_id=torch.empty((0,), dtype=torch.int32), mask=InstancesRLEMasks.from_coco_rle_masks( - image_size=(height, width), + image_size=image_size, + mask_size=(height, width), masks=[], ), + image_size=image_size, + mask_frame_size=image_size, ) attach_deferred_postprocess_handoff( detections=detections, @@ -905,6 +950,7 @@ def _supports_triton_postprocess_path( image_meta: PreProcessingMetadata, threshold: Union[float, torch.Tensor], classes_re_mapping: Optional[ClassesReMapping], + masks_resolution_factor: float = 1.0, ) -> bool: """Return ``True`` when the sparse Triton path can represent this input.""" return ( @@ -915,6 +961,7 @@ def _supports_triton_postprocess_path( image_meta=image_meta, threshold=threshold, classes_re_mapping=classes_re_mapping, + masks_resolution_factor=masks_resolution_factor, ) is None ) @@ -932,10 +979,6 @@ def _unsupported_triton_postprocess_reason( """Explain why the Triton path should not run, or ``None`` when supported.""" if triton is None: return "triton_unavailable" - if masks_resolution_factor != 1.0: - # the fused kernel interpolates straight to the image; it cannot honour - # a reduced target, so defer rather than silently ignore the request - return "mask_resolution_factor_unsupported" if classes_re_mapping is None: return "class_remapping_required" if isinstance(threshold, torch.Tensor): @@ -979,6 +1022,24 @@ def _unsupported_triton_postprocess_reason( or image_meta.size_after_pre_processing.width != output_width ): return "resize_metadata_unsupported" + target_height, target_width = resolve_mask_target_size( + mask_height=mask_height, + mask_width=mask_width, + size_after_pre_processing=image_meta.size_after_pre_processing, + masks_resolution_factor=masks_resolution_factor, + ) + if ( + target_height > 4096 + or target_width > 4096 + or target_height * target_width + > INFERENCE_MODELS_RFDETR_TRITON_POSTPROC_MAX_PIXELS + or target_height * target_width >= _MAX_EXACT_FLAT_INDEX + ): + return "input_size_exceeds_triton_limits" + if target_height < mask_height or target_width < mask_width: + # Antialiased downsampling needs more than the kernel's two taps per + # axis. Keep the reference resize instead of failing in table creation. + return "mask_downsampling_unsupported" if image_scores.device.type != "cuda": return "cuda_device_required" if ( @@ -1576,8 +1637,8 @@ def _sparse_atomic_rle_from_metadata_kernel( num_queries: Number of query masks in ``masks``. mask_height: Height of each low-resolution RF-DETR mask. mask_width: Width of each low-resolution RF-DETR mask. - output_height: Original image height for the output RLE mask. - output_width: Original image width for the output RLE mask. + output_height: Requested mask-grid height for the output RLE mask. + output_width: Requested mask-grid width for the output RLE mask. mask_stride_q: Stride between query masks in ``masks``. mask_stride_h: Row stride for ``masks``. mask_stride_w: Column stride for ``masks``. diff --git a/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py b/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py index 83dbc62bbe..c8608cdc01 100644 --- a/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py +++ b/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py @@ -673,39 +673,16 @@ def test_rfdetr_triton_postproc_topk_retry_matches_reference_rle_path() -> None: _assert_detections_equal(actual, expected) -@pytest.mark.parametrize( - "factor,expected_supported", [(1.0, True), (0.5, False), (0.0, False)] -) -def test_reduced_mask_resolution_is_unsupported_by_the_fused_path( - monkeypatch, - factor: float, - expected_supported: bool, -) -> None: - # given - # the fused kernel interpolates straight to the image size, so it cannot - # honour a reduced target; it must say so rather than silently ignore it. - # triton is absent on CPU CI, and that reason is checked first, so stand it - # in to reach the check under test +@pytest.mark.parametrize("factor", [0.0, 0.25, 0.5, 1.0]) +def test_reduced_mask_resolution_reaches_the_device_check(monkeypatch, factor) -> None: + # Stand in for Triton on CPU CI to verify that every valid factor passes + # shape/metadata validation before the CUDA requirement is checked. monkeypatch.setattr(triton_postprocess, "triton", object()) - device = torch.device("cpu") - bboxes, logits, masks = _single_detection_inputs(device) - - # when - reason = triton_postprocess._unsupported_triton_postprocess_reason( - image_bboxes=bboxes, - image_scores=logits, - image_masks=masks, - image_meta=_metadata(), - threshold=0.4, - classes_re_mapping=_class_mapping(device), - masks_resolution_factor=factor, + reason = _unsupported_triton_postprocess_reason( + **_support_kwargs(), masks_resolution_factor=factor ) - # then - if expected_supported: - assert reason != "mask_resolution_factor_unsupported" - else: - assert reason == "mask_resolution_factor_unsupported" + assert reason == "cuda_device_required" @pytest.mark.parametrize("factor", [0.25, 0.5]) @@ -742,3 +719,232 @@ def spy(*args, **kwargs): # then assert seen.get("masks_resolution_factor") == factor + + +@pytest.mark.parametrize("shared_queries", [False, True]) +@pytest.mark.parametrize("empty", [False, True]) +@pytest.mark.parametrize("max_detections", [None, 0, 1]) +def test_sparse_assembly_preserves_independent_mask_grid( + shared_queries, empty, max_detections +) -> None: + metadata = np.zeros((2, triton_postprocess._HEADER_SIZE), dtype=np.float32) + if not empty: + metadata[:, 0] = 1 + metadata[:, 1] = [3, 8] + metadata[:, 2] = [0.9, 0.8] + metadata[:, 3:7] = [24, 16, 73, 49] + metadata[:, 9] = 0 # both class rows refer to the same query + records = np.array([[2, 0, 0], [0, 6, 8], [0, 11, 13]], dtype=np.int32) + kwargs = dict( + max_total_runs=2, + height=5, + width=7, + image_size=(65, 97), + max_detections=max_detections, + ) + if shared_queries: + result = triton_postprocess._instance_detections_from_sparse_query_records( + class_metadata_host=metadata, records_host=records, **kwargs + ) + else: + result = triton_postprocess._instance_detections_from_sparse_records( + metadata_host=metadata, records=torch.from_numpy(records), **kwargs + ) + + assert result is not None + assert result.image_size == result.mask_frame_size == (65, 97) + assert result.mask.image_size == (65, 97) + assert result.mask_size == result.mask.mask_size == (5, 7) + count = 0 if empty else min(2, max_detections if max_detections is not None else 2) + assert len(result) == count + if count: + assert result.xyxy.tolist() == [[24, 16, 73, 49]] * count + decoded = coco_rle_masks_to_numpy_mask(result.mask) + expected = np.zeros((5, 7), dtype=bool) + expected[1:3, 1:3] = True + np.testing.assert_array_equal(decoded[0], expected) + if shared_queries and count == 2: + np.testing.assert_array_equal(decoded[1], expected) + + +@pytest.mark.parametrize("empty", [False, True]) +def test_deferred_assembly_preserves_grid_until_finalization( + monkeypatch, empty +) -> None: + from unittest.mock import Mock + + from inference_models.models.base.async_handoff import ( + get_deferred_postprocess_finalizer, + ) + + metadata = torch.zeros((1, triton_postprocess._HEADER_SIZE), dtype=torch.float32) + metadata[0, :7] = torch.tensor([not empty, 3, 0.9, 24, 16, 73, 49]) + records = torch.tensor([[1, 0, 0], [0, 6, 8]], dtype=torch.int32) + done_event = Mock() + release = Mock() + monkeypatch.setattr(triton_postprocess, "_release_pinned_host_buffer", release) + placeholder = ( + triton_postprocess._deferred_instance_detections_from_sparse_query_records( + class_metadata_host=metadata, + records_host=records, + keepalive_tensors=(), + done_event=done_event, + outputs_consumed_event=Mock(), + max_total_runs=1, + height=5, + width=7, + image_size=(65, 97), + max_detections=1, + ) + ) + assert placeholder.mask_size == (5, 7) + assert placeholder.image_size == placeholder.mask_frame_size == (65, 97) + done_event.synchronize.assert_not_called() + result = get_deferred_postprocess_finalizer(placeholder)() + done_event.synchronize.assert_called_once() + assert release.call_count == 2 + assert result.mask_size == (5, 7) + assert result.image_size == result.mask_frame_size == (65, 97) + assert len(result) == (0 if empty else 1) + + +@pytest.mark.parametrize("factor", [-0.1, 1.1, float("nan"), float("inf")]) +def test_triton_rejects_invalid_mask_resolution_factor(monkeypatch, factor) -> None: + monkeypatch.setattr(triton_postprocess, "triton", object()) + with pytest.raises(ValueError, match="finite and in"): + _unsupported_triton_postprocess_reason( + **_support_kwargs(), masks_resolution_factor=factor + ) + + +@pytest.mark.parametrize("factor", [0.0, 0.25, 0.5, 1.0]) +def test_triton_downsampling_uses_the_reference_antialias_path( + monkeypatch, factor +) -> None: + monkeypatch.setattr(triton_postprocess, "triton", object()) + kwargs = _support_kwargs(mask_size=(8, 8)) + kwargs["image_meta"] = _metadata(height=4, width=6) + reason = _unsupported_triton_postprocess_reason( + **kwargs, masks_resolution_factor=factor + ) + assert reason == ( + "cuda_device_required" if factor == 0 else "mask_downsampling_unsupported" + ) + + +@pytest.mark.skipif( + not torch.cuda.is_available() or triton_postprocess.triton is None, + reason="CUDA and Triton are required", +) +@pytest.mark.parametrize("factor", [0.0, 0.25, 0.5, 1.0]) +@pytest.mark.parametrize("image_size", [(64, 64), (63, 95)]) +@pytest.mark.parametrize("mode", ["single", "multiclass", "deferred", "empty"]) +def test_triton_mask_grid_matches_reference_on_cuda(factor, image_size, mode) -> None: + from inference_models.models.base.async_handoff import ( + get_deferred_postprocess_finalizer, + ) + from inference_models.models.common.roboflow.post_processing import ( + resolve_mask_target_size, + ) + + device = torch.device("cuda") + bboxes, logits, masks = _single_detection_inputs(device) + # An asymmetric source and noncontiguous views expose swapped axes/strides. + masks[0, 1:5, 2:7] = 3.0 + masks = masks.transpose(1, 2) + if mode in ("multiclass", "deferred"): + logits[0, 1] = 3.0 + logits[1, 0] = 2.0 + elif mode == "empty": + logits.fill_(-4.0) + scores = logits.sigmoid().T.contiguous().T + metadata = _metadata(height=image_size[0], width=image_size[1]) + mapping = _class_mapping(device) + mapping.class_mapping[:] = torch.tensor([3, 8], device=device) + expected = _post_process_single_instance_segmentation_result_to_rle_masks( + image_bboxes=bboxes, + image_logits=scores, + image_masks=masks, + image_meta=metadata, + threshold=0.4, + num_classes=2, + classes_re_mapping=mapping, + masks_resolution_factor=factor, + ) + with torch.cuda.stream(torch.cuda.Stream()): + # Producer data were created on the default stream. + torch.cuda.current_stream().wait_stream(torch.cuda.default_stream()) + actual = post_process_single_instance_segmentation_result_to_rle_masks_triton( + image_bboxes=bboxes, + image_scores=scores, + image_masks=masks, + image_meta=metadata, + threshold=0.4, + classes_re_mapping=mapping, + masks_resolution_factor=factor, + defer_postprocess_sync=mode == "deferred", + max_detections=100, # must still respect RF-DETR's query-count cap + ) + assert actual is not None + if mode == "deferred": + finalize = get_deferred_postprocess_finalizer(actual) + assert finalize is not None + actual = finalize() + target = resolve_mask_target_size( + 8, + 8, + size_after_pre_processing=metadata.original_size, + masks_resolution_factor=factor, + ) + assert actual.image_size == actual.mask_frame_size == image_size + assert actual.mask.image_size == image_size + assert actual.mask_size == actual.mask.mask_size == target + if mode != "empty": + _assert_detections_equal(actual, expected) + else: + assert len(actual) == 0 + + +@pytest.mark.skipif( + not torch.cuda.is_available() or triton_postprocess.triton is None, + reason="CUDA and Triton are required", +) +@pytest.mark.parametrize("factor", [0.0, 0.25, 0.5, 1.0]) +def test_triton_native_and_large_source_grids(factor) -> None: + from inference_models.models.common.roboflow.post_processing import ( + resolve_mask_target_size, + ) + + device = torch.device("cuda") + bboxes, logits, _ = _single_detection_inputs(device) + masks = torch.full((2, 192, 192), -2.0, device=device) + masks[0, 12:185, 41:161] = 3.0 + metadata = _metadata(height=1031, width=257) + mapping = _class_mapping(device) + expected = _post_process_single_instance_segmentation_result_to_rle_masks( + image_bboxes=bboxes, + image_logits=logits.sigmoid(), + image_masks=masks, + image_meta=metadata, + threshold=0.4, + num_classes=2, + classes_re_mapping=mapping, + masks_resolution_factor=factor, + ) + actual = post_process_single_instance_segmentation_result_to_rle_masks_triton( + image_bboxes=bboxes, + image_scores=logits.sigmoid(), + image_masks=masks, + image_meta=metadata, + threshold=0.4, + classes_re_mapping=mapping, + masks_resolution_factor=factor, + ) + assert actual is not None + _assert_detections_equal(actual, expected) + assert actual.mask_size == resolve_mask_target_size( + 192, + 192, + size_after_pre_processing=metadata.original_size, + masks_resolution_factor=factor, + ) From dfa5a12c729695df1a08b22f8f96f64567f0372f Mon Sep 17 00:00:00 2001 From: Damian Kosowski Date: Fri, 9 Oct 2026 14:37:15 +0200 Subject: [PATCH 33/41] Cover reduced-grid fallbacks and empty deferred results --- .../models/rfdetr/test_triton_postprocess.py | 20 ++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py b/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py index c8608cdc01..ff14ad6b54 100644 --- a/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py +++ b/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py @@ -143,6 +143,7 @@ def _expected_result( threshold, classes_re_mapping, num_classes: int = 2, + masks_resolution_factor: float = 1.0, ): return _post_process_single_instance_segmentation_result_to_rle_masks( image_bboxes=bboxes, @@ -152,6 +153,7 @@ def _expected_result( threshold=threshold, num_classes=num_classes, classes_re_mapping=classes_re_mapping, + masks_resolution_factor=masks_resolution_factor, ) @@ -388,10 +390,12 @@ def test_rfdetr_triton_postproc_accepts_2xlarge_shape_limits(monkeypatch) -> Non assert reason == "cuda_device_required" +@pytest.mark.parametrize("factor", [0.0, 0.25, 0.5, 1.0]) @pytest.mark.parametrize("case", ["no_class_mapping", "tensor_threshold", "padding"]) def test_rfdetr_triton_postproc_unsupported_cases_use_reference_path( monkeypatch, case: str, + factor: float, ) -> None: monkeypatch.setattr(rfdetr_common, "_TRITON_POSTPROC_ENABLED", True) calls = 0 @@ -415,7 +419,6 @@ def spy_triton_postprocess(*args, **kwargs): metadata = _metadata() threshold = 0.4 classes_re_mapping = _class_mapping(device) - extra_kwargs = {} if case == "no_class_mapping": classes_re_mapping = None elif case == "tensor_threshold": @@ -430,6 +433,7 @@ def spy_triton_postprocess(*args, **kwargs): metadata=metadata, threshold=threshold, classes_re_mapping=classes_re_mapping, + masks_resolution_factor=factor, ) actual = post_process_instance_segmentation_results_to_rle_masks( bboxes=bboxes.unsqueeze(0), @@ -439,7 +443,7 @@ def spy_triton_postprocess(*args, **kwargs): threshold=threshold, num_classes=2, classes_re_mapping=classes_re_mapping, - **extra_kwargs, + masks_resolution_factor=factor, )[0] assert calls == 1 @@ -838,7 +842,9 @@ def test_triton_downsampling_uses_the_reference_antialias_path( ) @pytest.mark.parametrize("factor", [0.0, 0.25, 0.5, 1.0]) @pytest.mark.parametrize("image_size", [(64, 64), (63, 95)]) -@pytest.mark.parametrize("mode", ["single", "multiclass", "deferred", "empty"]) +@pytest.mark.parametrize( + "mode", ["single", "multiclass", "deferred", "empty", "deferred_empty"] +) def test_triton_mask_grid_matches_reference_on_cuda(factor, image_size, mode) -> None: from inference_models.models.base.async_handoff import ( get_deferred_postprocess_finalizer, @@ -855,7 +861,7 @@ def test_triton_mask_grid_matches_reference_on_cuda(factor, image_size, mode) -> if mode in ("multiclass", "deferred"): logits[0, 1] = 3.0 logits[1, 0] = 2.0 - elif mode == "empty": + elif mode in ("empty", "deferred_empty"): logits.fill_(-4.0) scores = logits.sigmoid().T.contiguous().T metadata = _metadata(height=image_size[0], width=image_size[1]) @@ -882,11 +888,11 @@ def test_triton_mask_grid_matches_reference_on_cuda(factor, image_size, mode) -> threshold=0.4, classes_re_mapping=mapping, masks_resolution_factor=factor, - defer_postprocess_sync=mode == "deferred", + defer_postprocess_sync=mode.startswith("deferred"), max_detections=100, # must still respect RF-DETR's query-count cap ) assert actual is not None - if mode == "deferred": + if mode.startswith("deferred"): finalize = get_deferred_postprocess_finalizer(actual) assert finalize is not None actual = finalize() @@ -899,7 +905,7 @@ def test_triton_mask_grid_matches_reference_on_cuda(factor, image_size, mode) -> assert actual.image_size == actual.mask_frame_size == image_size assert actual.mask.image_size == image_size assert actual.mask_size == actual.mask.mask_size == target - if mode != "empty": + if mode not in ("empty", "deferred_empty"): _assert_detections_equal(actual, expected) else: assert len(actual) == 0 From ff4a6834d4fcfd9cedd23b7a255d5a02f05d1317 Mon Sep 17 00:00:00 2001 From: Damian Kosowski Date: Fri, 9 Oct 2026 14:59:41 +0200 Subject: [PATCH 34/41] Preserve global class selection in sparse mask postprocessing --- inference_models/docs/changelog.md | 4 +- .../models/rfdetr/triton_postprocess.py | 550 +++++++----------- .../models/rfdetr/test_triton_postprocess.py | 51 ++ 3 files changed, 266 insertions(+), 339 deletions(-) diff --git a/inference_models/docs/changelog.md b/inference_models/docs/changelog.md index 2d588f471f..39ff9a0072 100644 --- a/inference_models/docs/changelog.md +++ b/inference_models/docs/changelog.md @@ -48,7 +48,9 @@ results. Boxes retain image coordinates, while RLE and empty results carry the selected mask-grid dimensions. Antialiased downsampling and previously unsupported transforms use the reference post-processor. Multiclass results respect RF-DETR's - query-count cap even when `max_detections` is larger. + query-count cap even when `max_detections` is larger. Multiclass and deferred + selection use global top-k before remapping, without a four-class-per-query + cutoff, and encode each selected query mask once. - The `inference_models` adapter now rejects unknown `mask_decode_mode` values with `InvalidMaskDecodeArgument`; it previously ignored the mode. The legacy backend already rejected invalid modes. Validation also runs without opt-in. diff --git a/inference_models/inference_models/models/rfdetr/triton_postprocess.py b/inference_models/inference_models/models/rfdetr/triton_postprocess.py index abf36b4385..bb8134e30a 100644 --- a/inference_models/inference_models/models/rfdetr/triton_postprocess.py +++ b/inference_models/inference_models/models/rfdetr/triton_postprocess.py @@ -66,8 +66,6 @@ _SPARSE_MAX_ROI_WIDTH = 512 _SPARSE_BLOCK_COLS = 8 _SPARSE_MAX_TOTAL_RUNS = INFERENCE_MODELS_RFDETR_TRITON_POSTPROC_MAX_RUNS -_SPARSE_MAX_CLASSES_PER_QUERY = 4 -_SPARSE_TOPK_MAX_TOTAL_RUNS = _SPARSE_MAX_TOTAL_RUNS * _SPARSE_MAX_CLASSES_PER_QUERY # RF-DETR Seg 2XLarge emits 192x192 masks with 300 queries and COCO class # logits. The sparse path supports that shape by scanning source-mask support # in fixed tiles instead of one giant Triton vector. @@ -242,13 +240,30 @@ def post_process_single_instance_segmentation_result_to_rle_masks_triton( The fast path first emits one candidate per query. If any query has more than one class above threshold, the first pass asks for a retry and the - second pass emits up to ``_SPARSE_MAX_CLASSES_PER_QUERY`` query-class - candidates per query. + second pass selects the global top-k query-class candidates and shares + sparse masks between classes referring to the same query. When ``defer_postprocess_sync`` is set, the function enqueues the metadata, sparse RLE, and DtoH copies, then returns a placeholder whose finalizer does CPU assembly later. That is used by the streaming pipeline to keep the next frame's GPU work moving while Python handles previous detections. + + Args: + image_bboxes: Normalized query boxes in cxcywh format. + image_scores: Sigmoid probabilities for each query and class. + image_masks: Per-query mask logits on the model grid. + image_meta: Preprocessing geometry for this image. + threshold: Confidence cutoff; tensor cutoffs use the reference path. + classes_re_mapping: Mapping from model classes to public class IDs. + max_detections: Optional limit applied after global query-class selection. + defer_postprocess_sync: Return an asynchronous handoff for CPU assembly. + masks_resolution_factor: Native-to-image mask-grid interpolation factor. + + Returns: + Detections or a deferred placeholder, or None for reference fallback. + + Raises: + ValueError: The resolution factor is not finite or outside [0, 1]. """ unsupported_reason = _unsupported_triton_postprocess_reason( image_bboxes=image_bboxes, @@ -304,119 +319,23 @@ def post_process_single_instance_segmentation_result_to_rle_masks_triton( ) if defer_postprocess_sync: - # Deferred pipeline mode separates class metadata from query mask RLE: - # every class candidate can reuse the one sparse mask generated for its - # source query, so GPU work remains bounded by the number of queries. - topk_metadata_rows = num_queries * _SPARSE_MAX_CLASSES_PER_QUERY - query_metadata = torch.empty( - (num_queries, _HEADER_SIZE), - dtype=torch.float32, - device=image_scores.device, - ) - class_metadata = torch.empty( - (topk_metadata_rows, _HEADER_SIZE), - dtype=torch.float32, - device=image_scores.device, - ) - records = torch.empty( - (_SPARSE_MAX_TOTAL_RUNS + 1, 3), - dtype=torch.int32, - device=image_scores.device, - ) - source_bounds = _allocate_source_bounds( - rows=num_queries, - mask_height=mask_height, - mask_width=mask_width, - device=image_scores.device, - ) - _select_topk_query_class_metadata_kernel[(num_queries,)]( - image_scores, - image_bboxes, - class_mapping, - class_metadata, - query_metadata, - records, - confidence_threshold, - num_queries, - num_classes, - class_mapping.shape[0], - image_height, - image_width, - BLOCK_CLASSES=triton.next_power_of_2(num_classes), - METADATA_STRIDE=_HEADER_SIZE, - MAX_CLASSES_PER_QUERY=_SPARSE_MAX_CLASSES_PER_QUERY, - FLAG_WRITE_QUERY_METADATA=True, - FLAG_OVERFLOW_CLASSES=False, - ) - _positive_source_bounds_kernel[ - ( - num_queries, - triton.cdiv( - mask_height * mask_width, - _SPARSE_SOURCE_BOUNDS_BLOCK_PIXELS, - ), - ) - ]( - image_masks, - query_metadata, - source_bounds, - mask_height, - mask_width, - image_masks.stride(0), - image_masks.stride(1), - image_masks.stride(2), - BLOCK_PIXELS=_SPARSE_SOURCE_BOUNDS_BLOCK_PIXELS, - METADATA_STRIDE=_HEADER_SIZE, - ) - _sparse_atomic_rle_from_metadata_kernel[ - (num_queries, triton.cdiv(_SPARSE_MAX_ROI_WIDTH, _SPARSE_BLOCK_COLS)) - ]( - image_masks, - source_bounds, - y_idx, - y_weight, - x_idx, - x_weight, - query_metadata, - records, - num_queries, - mask_height, - mask_width, - output_height, - output_width, - image_masks.stride(0), - image_masks.stride(1), - image_masks.stride(2), - BLOCK_OUT_H=triton.next_power_of_2(output_height), - BLOCK_OUT_W=triton.next_power_of_2(output_width), - BLOCK_ROI_H=_BLOCK_ROI_H, - MAX_ROI_WIDTH=_SPARSE_MAX_ROI_WIDTH, - MAX_TOTAL_RUNS=_SPARSE_MAX_TOTAL_RUNS, - METADATA_STRIDE=_HEADER_SIZE, - BLOCK_COLS=_SPARSE_BLOCK_COLS, - ) - outputs_consumed_event = torch.cuda.Event() - outputs_consumed_event.record(torch.cuda.current_stream(image_scores.device)) - class_metadata_host = _acquire_pinned_host_buffer(class_metadata) - records_host = _acquire_pinned_host_buffer(records) - # Pinned buffers let the DtoH copies follow the postprocess kernel on - # the CUDA stream while Python starts preparing later frames. - class_metadata_host.copy_(class_metadata, non_blocking=True) - records_host.copy_(records, non_blocking=True) - done_event = torch.cuda.Event() - done_event.record(torch.cuda.current_stream(image_scores.device)) - return _deferred_instance_detections_from_sparse_query_records( - class_metadata_host=class_metadata_host, - records_host=records_host, - keepalive_tensors=(query_metadata, class_metadata, source_bounds, records), - done_event=done_event, - outputs_consumed_event=outputs_consumed_event, - max_total_runs=_SPARSE_MAX_TOTAL_RUNS, - height=output_height, - width=output_width, + result = _post_process_ranked_query_masks( + image_bboxes=image_bboxes, + image_scores=image_scores, + image_masks=image_masks, + class_mapping=class_mapping, + confidence_threshold=confidence_threshold, image_size=image_size, + output_height=output_height, + output_width=output_width, + y_idx=y_idx, + y_weight=y_weight, + x_idx=x_idx, + x_weight=x_weight, max_detections=max_detections, + defer_postprocess_sync=defer_postprocess_sync, ) + return result # First pass: keep the common case small by selecting only the best class # for each query and emitting sparse RLE runs for those query masks. @@ -516,33 +435,70 @@ def post_process_single_instance_segmentation_result_to_rle_masks_triton( ): return None - # Retry only when the first pass detected multiple passing classes for a - # query. This preserves RF-DETR's flat top-k query-class semantics without - # paying the expanded metadata/RLE cost on the usual one-class-per-query - # path. - topk_metadata_rows = num_queries * _SPARSE_MAX_CLASSES_PER_QUERY - metadata = torch.empty( - (topk_metadata_rows, _HEADER_SIZE), - dtype=torch.float32, - device=image_scores.device, + # Retry with global top-k metadata and one mask per selected source query. + result = _post_process_ranked_query_masks( + image_bboxes=image_bboxes, + image_scores=image_scores, + image_masks=image_masks, + class_mapping=class_mapping, + confidence_threshold=confidence_threshold, + image_size=image_size, + output_height=output_height, + output_width=output_width, + y_idx=y_idx, + y_weight=y_weight, + x_idx=x_idx, + x_weight=x_weight, + max_detections=max_detections, + defer_postprocess_sync=defer_postprocess_sync, + ) + return result + + +def _post_process_ranked_query_masks( + *, + image_bboxes: torch.Tensor, + image_scores: torch.Tensor, + image_masks: torch.Tensor, + class_mapping: torch.Tensor, + confidence_threshold: float, + image_size: Tuple[int, int], + output_height: int, + output_width: int, + y_idx: torch.Tensor, + y_weight: torch.Tensor, + x_idx: torch.Tensor, + x_weight: torch.Tensor, + max_detections: int, + defer_postprocess_sync: bool, +) -> Optional[InstanceDetections]: + """Select global top-k classes, but interpolate each selected query only once.""" + num_queries, num_classes = image_scores.shape + mask_height, mask_width = image_masks.shape[-2:] + image_height, image_width = image_size + # Match the reference's global selection before remapping/filtering. A + # per-query class cap drops valid detections at low confidence thresholds. + top_scores, top_indices = torch.topk(image_scores.flatten(), num_queries) + query_metadata = torch.empty( + (num_queries, _HEADER_SIZE), dtype=torch.float32, device=image_scores.device ) + class_metadata = torch.empty_like(query_metadata) records = torch.empty( - (_SPARSE_TOPK_MAX_TOTAL_RUNS + 1, 3), - dtype=torch.int32, - device=image_scores.device, + (_SPARSE_MAX_TOTAL_RUNS + 1, 3), dtype=torch.int32, device=image_scores.device ) source_bounds = _allocate_source_bounds( - rows=topk_metadata_rows, + rows=num_queries, mask_height=mask_height, mask_width=mask_width, device=image_scores.device, ) - _select_topk_query_class_metadata_kernel[(num_queries,)]( - image_scores, + _selected_query_class_metadata_kernel[(num_queries,)]( + top_scores, + top_indices, image_bboxes, class_mapping, - metadata, - metadata, + class_metadata, + query_metadata, records, confidence_threshold, num_queries, @@ -550,20 +506,20 @@ def post_process_single_instance_segmentation_result_to_rle_masks_triton( class_mapping.shape[0], image_height, image_width, - BLOCK_CLASSES=triton.next_power_of_2(num_classes), + BLOCK_QUERIES=triton.next_power_of_2(num_queries), METADATA_STRIDE=_HEADER_SIZE, - MAX_CLASSES_PER_QUERY=_SPARSE_MAX_CLASSES_PER_QUERY, - FLAG_WRITE_QUERY_METADATA=False, - FLAG_OVERFLOW_CLASSES=True, ) _positive_source_bounds_kernel[ ( - topk_metadata_rows, - triton.cdiv(mask_height * mask_width, _SPARSE_SOURCE_BOUNDS_BLOCK_PIXELS), + num_queries, + triton.cdiv( + mask_height * mask_width, + _SPARSE_SOURCE_BOUNDS_BLOCK_PIXELS, + ), ) ]( image_masks, - metadata, + query_metadata, source_bounds, mask_height, mask_width, @@ -574,10 +530,7 @@ def post_process_single_instance_segmentation_result_to_rle_masks_triton( METADATA_STRIDE=_HEADER_SIZE, ) _sparse_atomic_rle_from_metadata_kernel[ - ( - topk_metadata_rows, - triton.cdiv(_SPARSE_MAX_ROI_WIDTH, _SPARSE_BLOCK_COLS), - ) + (num_queries, triton.cdiv(_SPARSE_MAX_ROI_WIDTH, _SPARSE_BLOCK_COLS)) ]( image_masks, source_bounds, @@ -585,9 +538,9 @@ def post_process_single_instance_segmentation_result_to_rle_masks_triton( y_weight, x_idx, x_weight, - metadata, + query_metadata, records, - topk_metadata_rows, + num_queries, mask_height, mask_width, output_height, @@ -599,15 +552,39 @@ def post_process_single_instance_segmentation_result_to_rle_masks_triton( BLOCK_OUT_W=triton.next_power_of_2(output_width), BLOCK_ROI_H=_BLOCK_ROI_H, MAX_ROI_WIDTH=_SPARSE_MAX_ROI_WIDTH, - MAX_TOTAL_RUNS=_SPARSE_TOPK_MAX_TOTAL_RUNS, + MAX_TOTAL_RUNS=_SPARSE_MAX_TOTAL_RUNS, METADATA_STRIDE=_HEADER_SIZE, BLOCK_COLS=_SPARSE_BLOCK_COLS, ) - metadata_host = metadata.cpu().numpy() - return _instance_detections_from_sparse_records( - metadata_host=metadata_host, - records=records, - max_total_runs=_SPARSE_TOPK_MAX_TOTAL_RUNS, + if not defer_postprocess_sync: + result = _instance_detections_from_sparse_query_records( + class_metadata_host=class_metadata.cpu().numpy(), + records_host=records.cpu().numpy(), + max_total_runs=_SPARSE_MAX_TOTAL_RUNS, + height=output_height, + width=output_width, + image_size=image_size, + max_detections=max_detections, + ) + return result + + outputs_consumed_event = torch.cuda.Event() + outputs_consumed_event.record(torch.cuda.current_stream(image_scores.device)) + class_metadata_host = _acquire_pinned_host_buffer(class_metadata) + records_host = _acquire_pinned_host_buffer(records) + # Pinned buffers let the DtoH copies follow the postprocess kernel on + # the CUDA stream while Python starts preparing later frames. + class_metadata_host.copy_(class_metadata, non_blocking=True) + records_host.copy_(records, non_blocking=True) + done_event = torch.cuda.Event() + done_event.record(torch.cuda.current_stream(image_scores.device)) + return _deferred_instance_detections_from_sparse_query_records( + class_metadata_host=class_metadata_host, + records_host=records_host, + keepalive_tensors=(query_metadata, class_metadata, source_bounds, records), + done_event=done_event, + outputs_consumed_event=outputs_consumed_event, + max_total_runs=_SPARSE_MAX_TOTAL_RUNS, height=output_height, width=output_width, image_size=image_size, @@ -649,7 +626,7 @@ def _instance_detections_from_sparse_records( image_size=image_size, mask_frame_size=image_size, ) - if np.any(metadata_host[active_ranks, 8] > 0.5): + if np.any(metadata_host[active_ranks, 7:9] > 0.5): return None records_host = records.cpu().numpy() total_runs = int(records_host[0, 0]) @@ -729,8 +706,8 @@ def _instance_detections_from_sparse_query_records( ) -> Optional[InstanceDetections]: """Assemble detections when class rows share query-level RLE records. - Deferred pipeline mode emits up to four class candidates per query, but the - mask is identical for those class rows. The GPU therefore writes RLE records + Multiple selected classes can refer to one query, but the mask is identical + for those class rows. The GPU therefore writes RLE records once per query and this CPU helper fans that query mask out to the selected class detections. """ @@ -936,6 +913,9 @@ def _should_retry_sparse_topk_metadata( active_ranks = np.flatnonzero(metadata_host[:, 0] > 0.5) if active_ranks.size == 0 or np.any(metadata_host[active_ranks, 8] > 0.5): return False + if np.any(metadata_host[active_ranks, 7] > 0.5): + return True + records_host = records.cpu().numpy() total_runs = int(records_host[0, 0]) if int(records_host[0, 1]) == 0 or total_runs < 0 or total_runs > max_total_runs: @@ -1198,7 +1178,7 @@ class is selected. coordinates to pixel coordinates. BLOCK_CLASSES: Power-of-two tile width covering ``num_classes``. METADATA_STRIDE: Number of float32 fields per metadata row. - FLAG_MULTICLASS: When true, writes ``records[0, 1] = 1`` if more + FLAG_MULTICLASS: When true, writes ``metadata[rank, 7] = 1`` if more than one mapped class for this query exceeds ``threshold`` so the caller can rerun the top-k query-class path. """ @@ -1223,8 +1203,6 @@ class is selected. valid_classes = class_active & (mapped_classes >= 0) passing_classes = valid_classes & (class_scores > threshold) passing_class_count = tl.sum(tl.where(passing_classes, 1, 0), axis=0) - if FLAG_MULTICLASS and passing_class_count > 1: - tl.store(records + 1, 1) # Select over valid mapped classes, not just passing classes. The # threshold is applied after selection so inactive metadata rows still # carry a stable class/score shape. @@ -1257,7 +1235,12 @@ class is selected. metadata + meta_base + 2, tl.where(is_valid_detection, selected_score, 0.0), ) - tl.store(metadata + meta_base + 7, 0.0) + # Each program owns its retry flag; sharing the records header races + # with program zero's initialization on large grids. + tl.store( + metadata + meta_base + 7, + (FLAG_MULTICLASS and passing_class_count > 1).to(tl.float32), + ) tl.store(metadata + meta_base + 8, 0.0) tl.store(metadata + meta_base + 9, query_index.to(tl.float32)) tl.store(metadata + meta_base + 10, selected_index.to(tl.float32)) @@ -1298,8 +1281,9 @@ class is selected. tl.store(metadata + meta_base + 6, y2) @triton.jit - def _select_topk_query_class_metadata_kernel( - scores, + def _selected_query_class_metadata_kernel( + top_scores, + top_indices, bboxes, class_mapping, metadata, @@ -1309,195 +1293,85 @@ def _select_topk_query_class_metadata_kernel( num_queries: tl.constexpr, num_classes: tl.constexpr, class_mapping_size: tl.constexpr, - output_height: tl.constexpr, - output_width: tl.constexpr, - BLOCK_CLASSES: tl.constexpr, + image_height: tl.constexpr, + image_width: tl.constexpr, + BLOCK_QUERIES: tl.constexpr, METADATA_STRIDE: tl.constexpr, - MAX_CLASSES_PER_QUERY: tl.constexpr, - FLAG_WRITE_QUERY_METADATA: tl.constexpr, - FLAG_OVERFLOW_CLASSES: tl.constexpr, ): - """Emit top passing query-class metadata rows for one RF-DETR query. + """Write globally selected class rows and a unique active row per query. - Launch grid: - ``(num_queries,)``. Each program scans all class scores for one - query and writes up to ``MAX_CLASSES_PER_QUERY`` rows. The current - implementation uses a static loop of four iterations, so - ``MAX_CLASSES_PER_QUERY`` is expected to be ``4``. - - Args: - scores: CUDA float32 tensor with shape - ``[num_queries, num_classes]`` containing sigmoid class scores. - bboxes: CUDA float32 tensor with shape ``[num_queries, 4]`` in - normalized ``cx, cy, width, height`` format. - class_mapping: CUDA int tensor with class remap entries. Negative - mapped ids are ignored. - metadata: CUDA float32 tensor with shape - ``[num_queries * MAX_CLASSES_PER_QUERY, METADATA_STRIDE]``. - Row ``query_index * MAX_CLASSES_PER_QUERY + class_rank`` holds - the ``class_rank``-th highest passing class for that query. - Columns have the same layout as - ``_select_best_query_metadata_kernel``. - query_metadata: CUDA float32 tensor with shape - ``[num_queries, METADATA_STRIDE]`` when - ``FLAG_WRITE_QUERY_METADATA`` is true. In deferred pipeline - mode, class metadata is expanded but the RLE kernel should still - run once per query, so class rank 0 is also written to this - query-level buffer. When ``FLAG_WRITE_QUERY_METADATA`` is false, - callers pass ``metadata`` here and the argument is unused. - records: CUDA int32 tensor with shape ``[MAX_TOTAL_RUNS + 1, 3]``. - Program 0 resets ``records[0, 0]`` and ``records[0, 1]`` before - the RLE kernel appends runs. - threshold: Minimum class score required for a metadata row to be - marked active. - num_queries: Number of query rows in ``scores`` and ``bboxes``. - num_classes: Number of class columns in ``scores``. - class_mapping_size: Number of valid entries in ``class_mapping``. - output_height: Original image height used for xyxy box conversion. - output_width: Original image width used for xyxy box conversion. - BLOCK_CLASSES: Power-of-two tile width covering all class columns. - METADATA_STRIDE: Number of float32 fields per metadata row. - MAX_CLASSES_PER_QUERY: Number of metadata rows reserved per query. - FLAG_WRITE_QUERY_METADATA: When true, additionally writes the best - passing class row for each query into ``query_metadata`` for the - deferred pipeline RLE kernel. - FLAG_OVERFLOW_CLASSES: When true, writes ``records[0, 1] = 1`` if - more than ``MAX_CLASSES_PER_QUERY`` classes pass threshold; the - caller treats that as unsupported for exact top-k parity. + Each program owns one class row and one query row. Scanning the small + top-k index vector avoids races between classes sharing a source query. + Only selected, valid queries enter the sparse mask kernel. """ - query_index = tl.program_id(0) - if query_index == 0: - tl.store(records + 0, 0) + rank = tl.program_id(0) + if rank == 0: + tl.store(records, 0) tl.store(records + 1, 0) - class_offsets = tl.arange(0, BLOCK_CLASSES) - class_active = class_offsets < num_classes - mapped_classes = tl.load( - class_mapping + class_offsets, - mask=class_active & (class_offsets < class_mapping_size), + score = tl.load(top_scores + rank) + selected_index = tl.load(top_indices + rank).to(tl.int32) + query_index = selected_index // num_classes + class_index = selected_index % num_classes + mapped_class = tl.load( + class_mapping + class_index, + mask=class_index < class_mapping_size, other=-1, ).to(tl.int32) - class_scores = tl.load( - scores + query_index * num_classes + class_offsets, - mask=class_active, - other=-1.0, + active = (score > threshold) & (mapped_class >= 0) + fields = tl.arange(0, METADATA_STRIDE) + tl.store(metadata + rank * METADATA_STRIDE + fields, 0.0) + tl.store(metadata + rank * METADATA_STRIDE, active.to(tl.float32)) + tl.store(metadata + rank * METADATA_STRIDE + 1, mapped_class.to(tl.float32)) + tl.store(metadata + rank * METADATA_STRIDE + 2, score) + tl.store(metadata + rank * METADATA_STRIDE + 9, query_index.to(tl.float32)) + tl.store(metadata + rank * METADATA_STRIDE + 10, selected_index.to(tl.float32)) + cx = tl.load(bboxes + query_index * 4) + cy = tl.load(bboxes + query_index * 4 + 1) + width = tl.load(bboxes + query_index * 4 + 2) + height = tl.load(bboxes + query_index * 4 + 3) + tl.store( + metadata + rank * METADATA_STRIDE + 3, + tl.minimum(tl.maximum((cx - 0.5 * width) * image_width, 0.0), image_width), ) - passing_classes = ( - class_active & (mapped_classes >= 0) & (class_scores > threshold) + tl.store( + metadata + rank * METADATA_STRIDE + 4, + tl.minimum( + tl.maximum((cy - 0.5 * height) * image_height, 0.0), image_height + ), ) - passing_class_count = tl.sum(tl.where(passing_classes, 1, 0), axis=0) - if FLAG_OVERFLOW_CLASSES and passing_class_count > MAX_CLASSES_PER_QUERY: - tl.store(records + 1, 1) - - work_scores = tl.where(passing_classes, class_scores, -1.0) - for class_rank in tl.static_range(0, 4): - # Repeated max-and-mask avoids sorting all classes and keeps the - # register footprint bounded by the configured class block. - selected_score = tl.max(work_scores, axis=0) - selected_class = tl.max( - tl.where( - work_scores == selected_score, - class_offsets, - -1, - ), - axis=0, - ).to(tl.int32) - mapped_class = tl.load( - class_mapping + selected_class, - mask=(selected_class >= 0) & (selected_class < class_mapping_size), - other=-1, - ).to(tl.int32) - is_valid_detection = (mapped_class >= 0) & (selected_score > threshold) - metadata_rank = query_index * MAX_CLASSES_PER_QUERY + class_rank - meta_base = metadata_rank * METADATA_STRIDE - selected_index = query_index * num_classes + selected_class - - tl.store( - metadata + meta_base + 0, - tl.where(is_valid_detection, 1.0, 0.0), - ) - tl.store(metadata + meta_base + 1, mapped_class.to(tl.float32)) - tl.store( - metadata + meta_base + 2, - tl.where(is_valid_detection, selected_score, 0.0), - ) - tl.store(metadata + meta_base + 7, 0.0) - tl.store(metadata + meta_base + 8, 0.0) - tl.store(metadata + meta_base + 9, query_index.to(tl.float32)) - tl.store(metadata + meta_base + 10, selected_index.to(tl.float32)) - tl.store(metadata + meta_base + 11, 0.0) - tl.store(metadata + meta_base + 12, 0.0) - tl.store(metadata + meta_base + 13, 0.0) - tl.store(metadata + meta_base + 14, 0.0) - tl.store(metadata + meta_base + 15, 0.0) - - bbox_base = query_index * 4 - cx = tl.load(bboxes + bbox_base, mask=is_valid_detection, other=0.0) - cy = tl.load(bboxes + bbox_base + 1, mask=is_valid_detection, other=0.0) - width = tl.load( - bboxes + bbox_base + 2, - mask=is_valid_detection, - other=0.0, - ) - height = tl.load( - bboxes + bbox_base + 3, - mask=is_valid_detection, - other=0.0, - ) - x1 = tl.maximum( - 0.0, - tl.minimum((cx - 0.5 * width) * output_width, output_width), - ) - y1 = tl.maximum( - 0.0, - tl.minimum((cy - 0.5 * height) * output_height, output_height), - ) - x2 = tl.maximum( - 0.0, - tl.minimum((cx + 0.5 * width) * output_width, output_width), - ) - y2 = tl.maximum( - 0.0, - tl.minimum((cy + 0.5 * height) * output_height, output_height), - ) - tl.store(metadata + meta_base + 3, x1) - tl.store(metadata + meta_base + 4, y1) - tl.store(metadata + meta_base + 5, x2) - tl.store(metadata + meta_base + 6, y2) - if FLAG_WRITE_QUERY_METADATA and class_rank == 0: - # The pipeline path wants best-query metadata for the RLE - # kernel while retaining expanded class metadata for CPU - # finalization. - query_meta_base = query_index * METADATA_STRIDE - tl.store( - query_metadata + query_meta_base + 0, - tl.where(is_valid_detection, 1.0, 0.0), - ) - tl.store( - query_metadata + query_meta_base + 1, mapped_class.to(tl.float32) - ) - tl.store( - query_metadata + query_meta_base + 2, - tl.where(is_valid_detection, selected_score, 0.0), - ) - tl.store(query_metadata + query_meta_base + 3, x1) - tl.store(query_metadata + query_meta_base + 4, y1) - tl.store(query_metadata + query_meta_base + 5, x2) - tl.store(query_metadata + query_meta_base + 6, y2) - tl.store(query_metadata + query_meta_base + 7, 0.0) - tl.store(query_metadata + query_meta_base + 8, 0.0) - tl.store( - query_metadata + query_meta_base + 9, query_index.to(tl.float32) - ) - tl.store( - query_metadata + query_meta_base + 10, selected_index.to(tl.float32) - ) - tl.store(query_metadata + query_meta_base + 11, 0.0) - tl.store(query_metadata + query_meta_base + 12, 0.0) - tl.store(query_metadata + query_meta_base + 13, 0.0) - tl.store(query_metadata + query_meta_base + 14, 0.0) - tl.store(query_metadata + query_meta_base + 15, 0.0) - work_scores = tl.where(class_offsets == selected_class, -1.0, work_scores) + tl.store( + metadata + rank * METADATA_STRIDE + 5, + tl.minimum(tl.maximum((cx + 0.5 * width) * image_width, 0.0), image_width), + ) + tl.store( + metadata + rank * METADATA_STRIDE + 6, + tl.minimum( + tl.maximum((cy + 0.5 * height) * image_height, 0.0), image_height + ), + ) + + offsets = tl.arange(0, BLOCK_QUERIES) + indices = tl.load( + top_indices + offsets, mask=offsets < num_queries, other=0 + ).to(tl.int32) + scores = tl.load(top_scores + offsets, mask=offsets < num_queries, other=-1.0) + classes = indices % num_classes + mapped = tl.load( + class_mapping + classes, + mask=(offsets < num_queries) & (classes < class_mapping_size), + other=-1, + ) + selected = ( + (offsets < num_queries) + & (indices // num_classes == rank) + & (scores > threshold) + & (mapped >= 0) + ) + query_active = tl.sum(selected.to(tl.int32), axis=0) > 0 + tl.store(query_metadata + rank * METADATA_STRIDE + fields, 0.0) + tl.store(query_metadata + rank * METADATA_STRIDE, query_active.to(tl.float32)) + tl.store(query_metadata + rank * METADATA_STRIDE + 9, rank.to(tl.float32)) @triton.jit def _positive_source_bounds_kernel( diff --git a/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py b/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py index ff14ad6b54..895d0ccbb9 100644 --- a/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py +++ b/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py @@ -954,3 +954,54 @@ def test_triton_native_and_large_source_grids(factor) -> None: size_after_pre_processing=metadata.original_size, masks_resolution_factor=factor, ) + + +@pytest.mark.skipif( + not torch.cuda.is_available() or triton_postprocess.triton is None, + reason="CUDA and Triton are required", +) +@pytest.mark.parametrize("factor", [0.0, 0.25, 0.5, 1.0]) +@pytest.mark.parametrize("deferred", [False, True]) +def test_triton_global_topk_keeps_more_than_four_classes_per_query(factor, deferred): + from inference_models.models.base.async_handoff import ( + get_deferred_postprocess_finalizer, + ) + + device = torch.device("cuda") + # The highest eight pairs all come from one query; an ignored class must + # still consume a global top-k slot before remapping is applied. + scores = torch.full((8, 10), 0.01, device=device) + scores[0] = torch.linspace(0.99, 0.80, 10, device=device) + bboxes = torch.tensor([[0.5, 0.5, 0.6, 0.4]], device=device).repeat(8, 1) + masks = torch.full((8, 8, 8), -2.0, device=device) + masks[0, 1:6, 2:7] = 3.0 + mapping = _class_mapping(device, num_classes=10) + mapping.class_mapping[0] = -1 + meta = _metadata(height=63, width=95) + expected = _post_process_single_instance_segmentation_result_to_rle_masks( + image_bboxes=bboxes, + image_logits=scores, + image_masks=masks, + image_meta=meta, + threshold=0.005, + num_classes=10, + classes_re_mapping=mapping, + max_detections=100, + masks_resolution_factor=factor, + ) + actual = post_process_single_instance_segmentation_result_to_rle_masks_triton( + image_bboxes=bboxes, + image_scores=scores, + image_masks=masks, + image_meta=meta, + threshold=0.005, + classes_re_mapping=mapping, + max_detections=100, + masks_resolution_factor=factor, + defer_postprocess_sync=deferred, + ) + assert actual is not None + if deferred: + actual = get_deferred_postprocess_finalizer(actual)() + assert len(actual) == 7 + _assert_detections_equal(actual, expected) From 3f0098d2f0a16bfe66a5de24cb001e9529ad56ae Mon Sep 17 00:00:00 2001 From: Damian Kosowski Date: Fri, 9 Oct 2026 15:08:55 +0200 Subject: [PATCH 35/41] Batch interpolation table packing for new mask grids --- .../models/rfdetr/triton_postprocess.py | 17 +++++---- .../models/rfdetr/test_triton_postprocess.py | 38 +++++++++++++++++++ 2 files changed, 47 insertions(+), 8 deletions(-) diff --git a/inference_models/inference_models/models/rfdetr/triton_postprocess.py b/inference_models/inference_models/models/rfdetr/triton_postprocess.py index bb8134e30a..60d8b82897 100644 --- a/inference_models/inference_models/models/rfdetr/triton_postprocess.py +++ b/inference_models/inference_models/models/rfdetr/triton_postprocess.py @@ -129,14 +129,15 @@ def _get_interpolation_weights( raise ValueError("Expected antialiased bilinear interpolation to use 2 taps") indices = torch.zeros((output_size, 2), dtype=torch.int32, device=device) values = torch.zeros((output_size, 2), dtype=torch.float32, device=device) - for output_index in range(output_size): - source_indices = nonzero[output_index].nonzero(as_tuple=True)[0] - indices[output_index, : source_indices.numel()] = source_indices.to( - dtype=torch.int32 - ) - values[output_index, : source_indices.numel()] = weights[ - output_index, source_indices - ] + # nonzero returns coordinates in row-major order. With at most two taps, + # the second coordinate in a row is exactly its second interpolation tap. + # Pack all rows together to avoid a GPU synchronization and several tiny + # launches per output coordinate on the first request for a new grid. + output_indices, source_indices = nonzero.nonzero(as_tuple=True) + tap_indices = torch.zeros_like(output_indices) + tap_indices[1:] = (output_indices[1:] == output_indices[:-1]).long() + indices[output_indices, tap_indices] = source_indices.to(dtype=torch.int32) + values[output_indices, tap_indices] = weights[output_indices, source_indices] cached_value = (indices, values) with _INTERPOLATION_WEIGHT_CACHE_LOCK: diff --git a/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py b/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py index 895d0ccbb9..4e35edd284 100644 --- a/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py +++ b/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py @@ -1005,3 +1005,41 @@ def test_triton_global_topk_keeps_more_than_four_classes_per_query(factor, defer actual = get_deferred_postprocess_finalizer(actual)() assert len(actual) == 7 _assert_detections_equal(actual, expected) + + +@pytest.mark.parametrize("axis", ["height", "width"]) +@pytest.mark.parametrize("source_size, target_size", [(1, 7), (8, 8), (8, 63)]) +def test_batched_interpolation_tables_preserve_reference_weights( + axis, source_size, target_size +): + indices, values = _get_interpolation_weights( + src_size=source_size, + output_size=target_size, + device=torch.device("cpu"), + axis=axis, + ) + basis_shape = ( + (source_size, 1, source_size, 1) + if axis == "height" + else (source_size, 1, 1, source_size) + ) + output_shape = (target_size, 1) if axis == "height" else (1, target_size) + resized = ( + torch.nn.functional.interpolate( + torch.eye(source_size).reshape(basis_shape), + size=output_shape, + mode="bilinear", + align_corners=False, + antialias=True, + ) + .reshape(source_size, target_size) + .T + ) + # Reconstruct every coefficient, including the zero-padded second tap at + # boundaries/native grids, to catch changes to interpolation or tap order. + reconstructed = torch.zeros_like(resized) + reconstructed.scatter_add_(1, indices.long(), values) + torch.testing.assert_close(reconstructed, resized, rtol=0, atol=0) + two_taps = values[:, 1] != 0 + assert torch.all(indices[two_taps, 0] < indices[two_taps, 1]) + assert torch.all(indices[~two_taps, 1] == 0) From a8a10f4bfacf29ef4260765409e7fee8896edb25 Mon Sep 17 00:00:00 2001 From: Damian Kosowski Date: Fri, 9 Oct 2026 15:13:20 +0200 Subject: [PATCH 36/41] Apply global selection before filtering ignored classes --- .../models/rfdetr/triton_postprocess.py | 24 +++++++++--- .../models/rfdetr/test_triton_postprocess.py | 37 +++++++++++++++++++ 2 files changed, 55 insertions(+), 6 deletions(-) diff --git a/inference_models/inference_models/models/rfdetr/triton_postprocess.py b/inference_models/inference_models/models/rfdetr/triton_postprocess.py index 60d8b82897..6dbe0e6a7c 100644 --- a/inference_models/inference_models/models/rfdetr/triton_postprocess.py +++ b/inference_models/inference_models/models/rfdetr/triton_postprocess.py @@ -613,6 +613,9 @@ def _instance_detections_from_sparse_records( caller should retry or fall back to the reference implementation. """ image_size = image_size or (height, width) + if np.any(metadata_host[:, 7] > 0.5): + return None + active_ranks = np.flatnonzero(metadata_host[:, 0] > 0.5) if active_ranks.size == 0: return InstanceDetections( @@ -627,7 +630,7 @@ def _instance_detections_from_sparse_records( image_size=image_size, mask_frame_size=image_size, ) - if np.any(metadata_host[active_ranks, 7:9] > 0.5): + if np.any(metadata_host[active_ranks, 8] > 0.5): return None records_host = records.cpu().numpy() total_runs = int(records_host[0, 0]) @@ -911,11 +914,12 @@ def _should_retry_sparse_topk_metadata( max_total_runs: int, ) -> bool: """Return whether first-pass sparse metadata needs query-class expansion.""" + if np.any(metadata_host[:, 7] > 0.5): + return True + active_ranks = np.flatnonzero(metadata_host[:, 0] > 0.5) if active_ranks.size == 0 or np.any(metadata_host[active_ranks, 8] > 0.5): return False - if np.any(metadata_host[active_ranks, 7] > 0.5): - return True records_host = records.cpu().numpy() total_runs = int(records_host[0, 0]) @@ -1180,8 +1184,8 @@ class is selected. BLOCK_CLASSES: Power-of-two tile width covering ``num_classes``. METADATA_STRIDE: Number of float32 fields per metadata row. FLAG_MULTICLASS: When true, writes ``metadata[rank, 7] = 1`` if more - than one mapped class for this query exceeds ``threshold`` so - the caller can rerun the top-k query-class path. + than one mapped class or any ignored class exceeds threshold, + so the caller can apply global top-k before class filtering. """ rank = tl.program_id(0) meta_base = rank * METADATA_STRIDE @@ -1204,6 +1208,14 @@ class is selected. valid_classes = class_active & (mapped_classes >= 0) passing_classes = valid_classes & (class_scores > threshold) passing_class_count = tl.sum(tl.where(passing_classes, 1, 0), axis=0) + # Ignored classes still consume slots in the reference's global top-k. + # Include inactive queries in this retry decision as well. + ignored_passing = ( + class_active & (mapped_classes < 0) & (class_scores > threshold) + ) + needs_global_selection = (passing_class_count > 1) | ( + tl.sum(ignored_passing.to(tl.int32), axis=0) > 0 + ) # Select over valid mapped classes, not just passing classes. The # threshold is applied after selection so inactive metadata rows still # carry a stable class/score shape. @@ -1240,7 +1252,7 @@ class is selected. # with program zero's initialization on large grids. tl.store( metadata + meta_base + 7, - (FLAG_MULTICLASS and passing_class_count > 1).to(tl.float32), + (FLAG_MULTICLASS and needs_global_selection).to(tl.float32), ) tl.store(metadata + meta_base + 8, 0.0) tl.store(metadata + meta_base + 9, query_index.to(tl.float32)) diff --git a/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py b/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py index 4e35edd284..becbccd1d6 100644 --- a/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py +++ b/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py @@ -1043,3 +1043,40 @@ def test_batched_interpolation_tables_preserve_reference_weights( two_taps = values[:, 1] != 0 assert torch.all(indices[two_taps, 0] < indices[two_taps, 1]) assert torch.all(indices[~two_taps, 1] == 0) + + +@pytest.mark.skipif( + not torch.cuda.is_available() or triton_postprocess.triton is None, + reason="CUDA and Triton are required", +) +@pytest.mark.parametrize("factor", [0.0, 0.25, 0.5, 1.0]) +def test_ignored_classes_on_inactive_query_consume_global_topk_slots(factor): + device = torch.device("cuda") + bboxes, _, masks = _single_detection_inputs(device) + scores = torch.tensor([[0.01, 0.02, 0.8], [0.99, 0.9, 0.1]], device=device) + mapping = _class_mapping(device, num_classes=3) + mapping.class_mapping[:] = torch.tensor([-1, -1, 42], device=device) + # Both global top-2 slots are ignored classes on query 1. Query 0's valid + # 0.8 candidate must not be returned, although it passes the confidence cut. + expected = _post_process_single_instance_segmentation_result_to_rle_masks( + image_bboxes=bboxes, + image_logits=scores, + image_masks=masks, + image_meta=_metadata(), + threshold=0.4, + num_classes=3, + classes_re_mapping=mapping, + masks_resolution_factor=factor, + ) + actual = post_process_single_instance_segmentation_result_to_rle_masks_triton( + image_bboxes=bboxes, + image_scores=scores, + image_masks=masks, + image_meta=_metadata(), + threshold=0.4, + classes_re_mapping=mapping, + masks_resolution_factor=factor, + ) + assert actual is not None + assert len(actual) == len(expected) == 0 + assert actual.mask_size == expected.mask_size From 4ed9eddf3a943a4b1977c8731527edbfe842feae Mon Sep 17 00:00:00 2001 From: Damian Kosowski Date: Fri, 9 Oct 2026 15:17:32 +0200 Subject: [PATCH 37/41] Assert the requested grid for empty Triton results --- .../unit_tests/models/rfdetr/test_triton_postprocess.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py b/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py index becbccd1d6..92610cdd6a 100644 --- a/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py +++ b/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py @@ -1079,4 +1079,7 @@ def test_ignored_classes_on_inactive_query_consume_global_topk_slots(factor): ) assert actual is not None assert len(actual) == len(expected) == 0 - assert actual.mask_size == expected.mask_size + # The reference's empty-RLE carrier defaults to the image grid. Triton + # explicitly retains the requested grid even when selection is empty. + expected_side = {0.0: 8, 0.25: 22, 0.5: 36, 1.0: 64}[factor] + assert actual.mask_size == (expected_side, expected_side) From 88f28e58ac1d6f08d201d2ef0eacc46f79088f4b Mon Sep 17 00:00:00 2001 From: Damian Kosowski Date: Fri, 9 Oct 2026 15:25:01 +0200 Subject: [PATCH 38/41] Read strided mask logits without copying every query --- .../inference_models/models/rfdetr/triton_postprocess.py | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/inference_models/inference_models/models/rfdetr/triton_postprocess.py b/inference_models/inference_models/models/rfdetr/triton_postprocess.py index 2d1a3927f3..fa657f8231 100644 --- a/inference_models/inference_models/models/rfdetr/triton_postprocess.py +++ b/inference_models/inference_models/models/rfdetr/triton_postprocess.py @@ -285,7 +285,8 @@ def post_process_single_instance_segmentation_result_to_rle_masks_triton( image_scores = image_scores.contiguous() image_bboxes = image_bboxes.contiguous() - image_masks = image_masks.contiguous() + # Both mask kernels consume explicit strides. Keeping the model's view + # avoids copying every query mask before selecting detections. class_mapping = classes_re_mapping.class_mapping.contiguous() num_queries, num_classes = image_scores.shape # The reference selects at most num_queries query/class pairs before @@ -964,10 +965,6 @@ def _unsupported_triton_postprocess_reason( """Explain why the Triton path should not run, or ``None`` when supported.""" if triton is None: return "triton_unavailable" - if masks_resolution_factor != 1.0: - # the fused kernel interpolates straight to the image; it cannot honour - # a reduced target, so defer rather than silently ignore the request - return "mask_resolution_factor_unsupported" if classes_re_mapping is None: return "class_remapping_required" if isinstance(threshold, torch.Tensor): From eee4c2e29e7ff8c798b911a1171bd4782bd4ec10 Mon Sep 17 00:00:00 2001 From: Damian Kosowski Date: Fri, 9 Oct 2026 15:46:14 +0200 Subject: [PATCH 39/41] Cover strided 2XLarge mask postprocessing shapes --- docs/instance_segmentation_mask_resolution.md | 3 +- .../models/rfdetr/test_triton_postprocess.py | 53 +++++++++++++++++++ 2 files changed, 55 insertions(+), 1 deletion(-) diff --git a/docs/instance_segmentation_mask_resolution.md b/docs/instance_segmentation_mask_resolution.md index 751c57a661..d9ecc509ee 100644 --- a/docs/instance_segmentation_mask_resolution.md +++ b/docs/instance_segmentation_mask_resolution.md @@ -100,7 +100,8 @@ at every factor. It preserves image-space boxes and records the encoded grid in rounding as the reference path. Antialiased downsampling, unsupported preprocessing transforms, and other existing compatibility limits use the reference post-processor. The existing Triton enablement flag still controls this path; no execution plan is -required. +required. Sparse-record capacity limits are unchanged, including the existing +deferred-mode overflow error. Workflow versions v1–v4 retain their compatibility guards. Workflow v5, reduced-mask downstream handling, and full tensor support are deferred to a diff --git a/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py b/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py index 92610cdd6a..1a9d21288b 100644 --- a/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py +++ b/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py @@ -1083,3 +1083,56 @@ def test_ignored_classes_on_inactive_query_consume_global_topk_slots(factor): # explicitly retains the requested grid even when selection is empty. expected_side = {0.0: 8, 0.25: 22, 0.5: 36, 1.0: 64}[factor] assert actual.mask_size == (expected_side, expected_side) + + +@pytest.mark.skipif( + not torch.cuda.is_available() or triton_postprocess.triton is None, + reason="CUDA and Triton are required", +) +@pytest.mark.parametrize("factor", [0.0, 0.25, 0.5, 1.0]) +@pytest.mark.parametrize("deferred", [False, True]) +def test_triton_2xlarge_strided_queries_match_reference(factor, deferred): + from inference_models.models.base.async_handoff import ( + get_deferred_postprocess_finalizer, + ) + + device = torch.device("cuda") + # Exercise the largest supported model grid with a channels-last view, + # 300 global top-k slots, shared masks, and an ignored high-scoring class. + masks = torch.full((192, 192, 300), -2.0, device=device).permute(2, 0, 1) + masks[0, 12:185, 41:161] = 3.0 + masks[1, 40:120, 10:80] = 2.0 + scores = torch.full((300, 91), 0.01, device=device) + scores[0, 1] = 0.95 + scores[0, 2] = 0.90 + scores[1, 3] = 0.85 + scores[2, 0] = 0.99 + bboxes = torch.tensor([[0.5, 0.5, 0.6, 0.4]], device=device).repeat(300, 1) + mapping = _class_mapping(device, num_classes=91) + mapping.class_mapping[0] = -1 + meta = _metadata(height=1031, width=257) + expected = _post_process_single_instance_segmentation_result_to_rle_masks( + image_bboxes=bboxes, + image_logits=scores, + image_masks=masks, + image_meta=meta, + threshold=0.4, + num_classes=91, + classes_re_mapping=mapping, + masks_resolution_factor=factor, + ) + actual = post_process_single_instance_segmentation_result_to_rle_masks_triton( + image_bboxes=bboxes, + image_scores=scores, + image_masks=masks, + image_meta=meta, + threshold=0.4, + classes_re_mapping=mapping, + masks_resolution_factor=factor, + defer_postprocess_sync=deferred, + ) + assert actual is not None + if deferred: + actual = get_deferred_postprocess_finalizer(actual)() + assert len(actual) == 3 + _assert_detections_equal(actual, expected) From 5cfa38595219a736be92cd0d6948f32bd1e9c123 Mon Sep 17 00:00:00 2001 From: Damian Kosowski Date: Fri, 9 Oct 2026 16:03:16 +0200 Subject: [PATCH 40/41] Document mask-grid first-use compilation cost --- docs/instance_segmentation_mask_resolution.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/instance_segmentation_mask_resolution.md b/docs/instance_segmentation_mask_resolution.md index d9ecc509ee..d61a64b513 100644 --- a/docs/instance_segmentation_mask_resolution.md +++ b/docs/instance_segmentation_mask_resolution.md @@ -103,6 +103,11 @@ The existing Triton enablement flag still controls this path; no execution plan required. Sparse-record capacity limits are unchanged, including the existing deferred-mode overflow error. +The first request for a new mask grid may compile Triton kernels and initialize +interpolation tables. Warm each expected image shape and mask-resolution factor +before serving latency-sensitive requests. Warm inference measurements exclude +this first-use time and memory cost. + Workflow versions v1–v4 retain their compatibility guards. Workflow v5, reduced-mask downstream handling, and full tensor support are deferred to a separate PR. This change does not introduce reduced-grid workflow outputs. From afcae76a1a816ca66c8ef05b31ecc6a05200795c Mon Sep 17 00:00:00 2001 From: Damian Kosowski Date: Fri, 9 Oct 2026 19:52:48 +0200 Subject: [PATCH 41/41] Construct immutable metadata in coordinate regression tests --- docs/instance_segmentation_mask_resolution.md | 3 +-- .../models/rfdetr/test_triton_postprocess.py | 11 ++++++----- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/instance_segmentation_mask_resolution.md b/docs/instance_segmentation_mask_resolution.md index 2f0627e3e0..90e880634f 100644 --- a/docs/instance_segmentation_mask_resolution.md +++ b/docs/instance_segmentation_mask_resolution.md @@ -115,8 +115,7 @@ The RF-DETR Triton RLE post-processor supports native and upsampled mask grids at every factor. Boxes and masks share the encoded grid in `mask_size`, including for empty and deferred results. It preserves fractional float32 boxes on grids that differ from the image and rounds to integers only on the image grid. -It uses the same target-size -rounding as the reference path. Antialiased downsampling, unsupported preprocessing +It uses the same target-size rounding as the reference path. Antialiased downsampling, unsupported preprocessing transforms, and other existing compatibility limits use the reference post-processor. The existing Triton enablement flag still controls this path; no execution plan is required. Sparse-record capacity limits are unchanged, including the existing diff --git a/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py b/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py index 7c0ec049cd..0f349d1b8f 100644 --- a/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py +++ b/inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py @@ -1181,11 +1181,12 @@ def test_triton_fractional_boxes_match_final_grid_reference( bboxes, logits, masks = _single_detection_inputs(device) bboxes[0] = torch.tensor([0.501, 0.493, 0.503, 0.489], device=device) bboxes = bboxes.to(dtype=dtype) - metadata = _metadata(height=image_size[0], width=image_size[1]) - metadata.inference_size = ImageDimensions(height=624, width=624) - metadata.nonsquare_intermediate_size = ImageDimensions(height=480, width=624) - metadata.scale_width = 624 / image_size[1] - metadata.scale_height = 480 / image_size[0] + metadata = _metadata(height=image_size[0], width=image_size[1])._replace( + inference_size=ImageDimensions(height=624, width=624), + nonsquare_intermediate_size=ImageDimensions(height=480, width=624), + scale_width=624 / image_size[1], + scale_height=480 / image_size[0], + ) if mode in ("multiclass", "deferred"): logits[0, 1] = 3.0 elif mode == "empty":