[probicheaux]

Description

Workflows had no way to run a PatchCore/FoundAD anomaly model and show its heatmap: the classification block drops anomaly_map, and the existing Heatmap Visualization block is built for detections (customer ask from Flex via #discuss-models; remaining GA item on MODELS-1194). Standard blocks, no new kinds, no new dependencies (as discussed in #discuss-inference-release).

This PR also makes the request confidence decide an anomaly model's verdict, so a caller can tune the operating point the same way as for any other model.

Anomaly confidence contract (inference_models + both servers)

An anomaly model's anomalous confidence is sigmoid((score - threshold) / scale), with the threshold and scale calibrated at training, so it is exactly 0.5 at the saved threshold. The verdict is now is_anomalous = anomalous_confidence >= confidence instead of always score >= threshold.

• confidence resolves like the other models: a number in [0, 1], "best" (the model-eval recommendation, via recommended_parameters, which PatchCore/FoundAD now accept), or "default" (0.5, the calibrated decision).
• A classification request that omits confidence gets "best" for anomaly models (the request field defaults to 0.4, which would move the decision). The v1 body route checks model_fields_set; the legacy /{dataset}/{version} route now leaves confidence unset for classification when the query omits it. Plain classifiers keep their 0.4 default.
• The response gains anomaly_scale. anomaly_threshold stays the saved threshold, so a caller can re-decide against their own threshold with the same rule.
• The new inference_server / inference_model_manager stack applies the same contract: build_task_params forwards a classification confidence only when the caller sent one, the legacy router mirrors the legacy server, AnomalyDetectionResponse and the typed serializers carry anomaly_scale, the classification action declares confidence, and the workflows provider's run_classification accepts include_anomaly_map like the protocol.

The companion platform change registers 0.5 as the recommended confidence for anomaly evals (roboflow/roboflow#18033), so "best" reproduces the calibrated decision.

PatchCore memory under concurrent requests

PatchCore scored patches in 4096-row chunks against the whole memory bank. A 640px model has 6400 patches per image and a ~140k-entry bank, so each chunk was a 2.13 GiB distance matrix plus temporaries, per request. Model eval's 16 concurrent requests ran the serverless GPU out of memory and got 500 Internal error, failing the PatchCore evals for paul-guerrie-tang1/anomaly-det-algbq-1-patchcore-t2 and paul-guerrie/anomaly-det-1-patchcore-t2. The chunk is now bounded by elements (2^26, 256 MiB in fp32) and the distances are computed in place with addmm, so one chunk-sized buffer is live. Reproduced on an L4 with a cold server (16 concurrent first requests: 16× 500, CUDA OOM in score_patches); with the fix, 16× 200 and identical scores and verdicts.

roboflow_core/anomaly_detection_model@v1

Runs an anomaly model (served on the classification route, see #3007/#3144) and outputs:

output	kind	notes
predictions	classification_prediction	the usual classification dict plus anomaly_score, anomaly_threshold, anomaly_scale, is_anomalous; never carries the raw map
anomaly_score	float	raw score, larger = more anomalous
anomaly_threshold	float	the threshold in effect (saved with the model, or the override)
anomaly_decision_threshold	float	the raw score at which the anomalous confidence equals confidence: anomaly_threshold + scale * logit(confidence), anomaly_threshold itself at 0.5
is_anomalous	boolean	the anomalous confidence reached confidence
anomaly_map	numpy_array	float32 (h, w) at the network input resolution (square), None when include_anomaly_map is off
inference_id, model_id		as the classification block

Inputs: images, model_id, include_anomaly_map (default True), confidence (default 0.5, sent to the server), anomaly_threshold (optional raw-score override), disable_active_learning, active_learning_target_dataset. With an override the block re-decides with the same rule relative to it, recomputing both class confidences from anomaly_scale, so the two inputs compose. A plain classifier fed to the block raises with a message pointing at the classification block. LOCAL goes through ModelsProvider.run_classification(..., include_anomaly_map=...) (set on the request only when True so the request-shape parity test keeps passing); REMOTE through InferenceConfiguration(include_anomaly_map=..., confidence_threshold=...), accepting both the SDK-decoded ndarray and the payload dict. The tensor sibling emits a native ClassificationPrediction with the anomaly fields in images_metadata[0]; serialise_native_classification passes them through with the decided class first. Output kind order is identical on both siblings.

roboflow_core/anomaly_heatmap_visualization@v1

Stretches the map onto its image (bilinear) and blends it on a scale anchored to anomaly_threshold, t = max(v, 0) / (max(v, 0) + anchor). Connected to the model block's anomaly_decision_threshold, 0.5 marks the block's verdict at any confidence. A very low confidence can put the decision at or below zero; the ramp then saturates every positive value, the limit of the formula, instead of rejecting it. color_map is ROBOFLOW (the ramp the Roboflow app draws in its inference preview, with its alpha stops) or an OpenCV map (JET, TURBO, INFERNO, VIRIDIS, HOT); opacity default 0.75; copy_image as the other visualizations. A missing map (model block run with include_anomaly_map=False) raises rather than passing the image through.

Shared helpers: core_steps/common/anomaly_detection.py (payload decoder duplicated from inference_sdk, since the workflows package cannot import inference.core; the override and decision-threshold rules), visualizations/common/annotators/anomaly_heatmap.py (ramp + annotator, shared by both viz siblings). Both blocks are registered in both loader.py branches; workflows/CHANGELOG.md has the entries under Unreleased → Added. Block docs are generated from the manifests.

Reference models for the tests

FoundAD (10 epochs) and PatchCore trained on the same anomaly dataset in the reference workspaces whose keys CI already injects: paul-guerrie/anomaly-det (staging, LOAD_TEST_STAGING_API_KEY) and paul-guerrie-tang1/anomaly-det-algbq (prod, LOAD_TEST_PRODUCTION_API_KEY). The prod project is public on Universe, so the workflow integration suite reaches it with API_KEY like its other public models. Test images: one clearly anomalous and one clearly normal image from that dataset (tests/workflows/integration_tests/execution/assets/anomaly_*.jpg), each at least 20% from the threshold on all four models.

Type of change

• New feature (non-breaking change which adds functionality)

How has this change been tested, please provide a testcase or example of how you tested the change?

• Workflow integration test (tests/workflows/integration_tests/execution/test_workflow_with_anomaly_detection.py): the model block into the heatmap against the public FoundAD and PatchCore models, run locally with a key from a different workspace, under ENABLE_TENSOR_DATA_REPRESENTATION=False and True: 6 passed each. Covers the verdict on both images, confidence 0/1 flipping it, and the override composing with confidence.
• Hosted platform E2E (tests/inference/hosted_platform_tests/test_anomaly_detection_models.py, models registered in MODELS_TO_BE_USED for staging and prod): verdicts at the default confidence, the served class confidence matching sigmoid((score - threshold) / scale), a request confidence flipping the verdict without moving the threshold, and the map at network resolution, through API v0 and v1 and both auth modes. Run against a local server from this branch with the prod load-test key (HOSTED_PLATFORM_TESTS_PROJECT=roboflow-platform-localhost): 20 passed. Against hosted staging/prod they pass once this ships, since the deployed servers do not yet serve anomaly_scale.
• Unit tests: inference_models anomaly post-processing (confidence modes, recommended parameters, invalid confidence) 27 passed; legacy server anomaly + HTTP interface 598 passed; workflows core_steps/common, anomaly blocks and visualizations 1243 passed (the 27 errors are font-download tests that cannot reach GitHub from my machine); inference_server unit tests 9961 passed; inference_model_manager unit tests 956 passed.
• black / isort / flake8 (error selections) clean on the touched files.

Any specific deployment considerations

• The next inference-models release must precede inference and roboflow-workflows: the block requires anomaly_scale, which the anomaly models now put in their metadata.
• On the legacy v0 route a confidence of 1 or more is divided by 100 (existing behavior), so confidence=1.0 there means 0.01.

Docs

• Block pages are generated from the manifests (development/docs/build_block_docs.py).

🤖 Generated with Claude Code

[github-actions]

👋 Thanks for the pull request! Here is how automated Claude review works here, so you spend credits (and reviewer time) wisely.

🚦 This PR is marked Ready for review, so automated Claude review will run — and every pass spends real credits.

Warning

💸 The Claude reviewer bills in credits, not vibes

Automated review spins up a real agent that reads real code and spends real credits on every pass. It is glad to help — but it is not a rubber duck, a linter you poke in a loop, or a substitute for reading the contributing guide. Treat it like an expensive senior reviewer whose time you booked, and show up prepared.

Draft when unsure, Ready when you mean it:

• 🌱 Not sure the PR is in good shape yet? Keep it (or set it back) as a draft — drafts pause review, so you can push and iterate without burning credits on a moving target.
• 💪 Feel strong about the contents? Mark it Ready for review and the reviewer will take a look.

However you get there, arrive prepared:

• 🧱 Bring a SOLID, thorough PR. Point your local agent at our skills/ to tune it to our guidelines first — or, if you are one of those fabled carbon-based contributors, read them yourself. A half-baked diff costs exactly the same to review as a finished one.
• ✅ Resolve every comment before you re-request review. Re-requesting with threads still open means paying twice for the same conversation.
• 🔁 Do not use CI review as an inner loop for a local agent. The reviewer is not a step-by-step debugger — do the unfolding locally and arrive with the answer, not the search.
• 🙋 If something looks off, ask a human. One question to a maintainer is cheaper and faster than three rounds of agent re-review chasing a misread.

Reviews are not free. A draft costs nothing to review; a Ready PR is a promise that it is worth reviewing.

• Prefer to skip automated review entirely? Add the skip-claude-review label.

[github-actions]

🤖 Claude review started at commit 18bc3742a304df04f302774259c707b9e990d943.

New commits are not auto-reviewed. Add the claude-review label to request a re-review — the label is consumed when the review starts, so just add it again next time.

[github-actions]

@probicheaux — please answer the question below.

⏳ This PR is on hold pending your answers — the review will not advance to sign-off until the IMPORTANT questions below are answered.

IMPORTANT questions

1. Which inference-sdk release first shipped InferenceConfiguration.include_anomaly_map?
   Both REMOTE paths (workflows/roboflow_workflows/core_steps/models/roboflow/anomaly_detection/v1.py run_remotely and the same method in v1_tensor.py) always pass include_anomaly_map=... to InferenceConfiguration(...), including when it is False. InferenceConfiguration is a frozen @dataclass (inference_sdk/http/entities.py), so an SDK without that field raises TypeError: __init__() got an unexpected keyword argument on every remote call. workflows/pyproject.toml still allows inference-sdk>=1.7.0, and workflows/uv.lock resolves 1.7.2. The block's own coerce_anomaly_map also accepts the undecoded payload dict "from older SDKs", which suggests older SDKs are expected to work with this block.
   What would change the outcome: if the field shipped after 1.7.0 (or is still unreleased), the inference-sdk floor in workflows/pyproject.toml needs raising (or the kwarg needs guarding). Otherwise standalone roboflow-workflows installs with REMOTE step execution fail on this block. If it shipped in ≤1.7.0, this is resolved.

⚠️ Unanswered questions may keep this PR out of a release.

Re-review: new commits are NOT auto-reviewed. Add the claude-review label to request a re-review. The label is consumed when the review starts, so each new request is a plain add.

Reviewed at HEAD: 18bc374

[github-actions]

Claude review summary

Skills: review-workflows-blocks, review-core-infra, review-topic-backward-compat-and-versioning, review-topic-local-vs-remote-execution, review-topic-prediction-integrity, review-topic-test-hygiene

No high-confidence blocking findings at this commit. This is not a sign-off: the open IMPORTANT question above has to be answered first.

Traced and verified:

• LOCAL numpy path: run_classification(include_anomaly_map=True) sets the field on ClassificationInferenceRequest, which reaches InferenceModelsAnomalyDetectionAdapter.postprocess and post_process_anomaly_scores. With the default False the request is unchanged for existing classification blocks.
• LOCAL tensor path: run_tensor_native_inference forwards the kwarg into model(...) and then post_process(include_anomaly_map=...). The map arrives as a CPU numpy array, so coerce_anomaly_map handles it.
• apply_threshold_override and _decided_anomaly_class_first rely on the normal/anomalous class names and the 0/1 ids, which match inference_models/.../common/anomaly_detection.py::ANOMALY_CLASS_NAMES.
• The two manifests are identical apart from the expected tensor-native prediction kind. Output-kind order matches, and both siblings are registered in both loader.py branches. __init__.py files exist, discover_dependent_resources is present on both siblings, and copy_image=False calls declare_numpy_image_mutated() on both viz siblings.
• The shared serializers_tensor change only fires when is_anomalous is in the image metadata, which brings the tensor classification output in line with the numpy AnomalyDetectionResponse dump.

Tests to add (non-blocking):

• A workflow execution integration test (tests/workflows/integration_tests/execution/) that compiles and runs anomaly_detection_model@v1 feeding anomaly_heatmap_visualization@v1 with a stubbed models provider returning an anomaly response. It should check that the anomaly_map/anomaly_threshold selectors bind and that the output image has the input's shape, under both ENABLE_TENSOR_DATA_REPRESENTATION values.

Minor doubts (non-blocking): the threshold override accepts zero or negative values, which the heatmap block then rejects at runtime. Anomaly models only load under the inference_models runtime, and the block declares no RuntimeRestriction for that, which matches how the existing classification and keypoint blocks treat inference_models-only models. ModelsProvider.run_classification gains a kwarg that third-party host providers must accept to run the new block.

Commands that informed the review: gh pr view 3201, gh pr diff 3201, the paginated gh api reads of issue comments, review comments, reviews and commits, plus Read/Grep over review-source/ (block, adapter, SDK entities.py, inference_models anomaly post-processing, workflows/pyproject.toml, workflows/uv.lock).

Reviewed at HEAD: 18bc374

[github-actions]

Release coordination

• Contributor: the workflows/CHANGELOG.md entries under ## Unreleased → ### Added are present. Nothing more is needed from you on versioning.
• Maintainers: roboflow-workflows needs a version change at release (new blocks, a new ModelsProvider.run_classification kwarg and the tensor serializer change). This PR doesn't require an Execution Engine or inference-models version change.

Reviewed at HEAD: 18bc374

[probicheaux]

Answer to question 1. InferenceConfiguration.include_anomaly_map first shipped in inference-sdk 1.6.1 (#3007), and on the 1.7 line in 1.7.2. It is absent from 1.7.0 and 1.7.1, which were cut before #3007 merged (neither tag contains the commit; inference_sdk/http/entities.py at both tags has no such field). So the floor was wrong: 46c79d0 raises workflows/pyproject.toml to inference-sdk>=1.7.2 (the lock already resolved 1.7.2). The payload-dict branch in coerce_anomaly_map covers the LOCAL path, where the response is dumped undecoded, not older SDKs on the REMOTE path.

[github-actions]

🤖 Claude review started at commit 46c79d0b4680128a7db7438add5be41d6d5b5e9d.

New commits are not auto-reviewed. Add the claude-review label to request a re-review — the label is consumed when the review starts, so just add it again next time.

[github-actions]

High — workflows/uv.lock is now stale, so uv lock --check will fail.

The floor moved to >=1.7.2, but workflows/uv.lock was not regenerated. It still records the root package's requirement as { name = "inference-sdk", specifier = ">=1.7.0" } (workflows/uv.lock:4152). uv lock --check compares the lock's recorded requires-dist against pyproject.toml and fails when they differ. Resolving 1.7.2 doesn't help, because the specifier itself doesn't match.

make create_workflows_wheel runs cd workflows && uv lock --check (Makefile:47) before uv build. create_isolation_wheels depends on that target. The target is used by the unit/integration workflow jobs, unit_tests_inference_x86, unit_tests_inference_cli_x86, integration_e2e_tests_inference_sdk_x86 and publish.pypi.roboflow_workflows.yml. As a result, all of those jobs and the package publish fail at the wheel-build step.

Fix: run cd workflows && uv lock and commit the updated uv.lock. The only expected change is that specifier line, since 1.7.2 is already the resolved version.

[probicheaux]

Fixed in 3e57b3b: workflows/uv.lock now records inference-sdk>=1.7.2 (the locked version was already 1.7.2, so only the requires-dist line changed). uv lock --check passes with uv 0.11.31, the version CI pins.

[github-actions]

Claude review summary (re-run)

Skills: review-workflows-blocks, review-topic-backward-compat-and-versioning, review-packaging-ci, review-topic-test-hygiene

Delta since 18bc3742: a merge from main (1fae8747) and 46c79d0 (inference-sdk>=1.7.2 in workflows/pyproject.toml).

Question 1 (SDK version with include_anomaly_map): resolved. The answer is concrete and responsive, and the diff matches it: the floor is now >=1.7.2, and the lock already resolves inference-sdk 1.7.2. The clarification about the payload-dict branch also matches the code: the LOCAL path dumps the response without decoding the map.

Blocking (1 open):

• High: workflows/uv.lock was not regenerated after the floor bump. It still records specifier = ">=1.7.0" at line 4152, so uv lock --check in make create_workflows_wheel fails. That breaks the wheel build in every CI job that uses it and in the PyPI publish. Fix: run cd workflows && uv lock and commit the lock (see the inline comment).

The non-blocking test suggestion from the previous summary still applies: an end-to-end workflow integration test chaining the model block into the heatmap block under both ENABLE_TENSOR_DATA_REPRESENTATION values.

This is not a sign-off. Fix the lockfile and add the claude-review label to request a re-review.

Commands that informed the review: gh pr view 3201, gh pr diff 3201, the paginated gh api reads of issue comments, review comments, reviews and commits, and Read/Grep over review-source/ (workflows/pyproject.toml, workflows/uv.lock, Makefile, .github/workflows/*.yml, inference_sdk/http/entities.py).

Reviewed at HEAD: 46c79d0

[github-actions]

🤖 Claude review started at commit 3e57b3b64fb7fe876a458eb8282d222b747cbfd4.

New commits are not auto-reviewed. Add the claude-review label to request a re-review — the label is consumed when the review starts, so just add it again next time.

[github-actions]

Claude review summary (re-run)

Skills: review-packaging-ci, review-topic-backward-compat-and-versioning

Delta since 46c79d0b: 3e57b3b6, a one-line change to workflows/uv.lock.

Blocker resolved: the stale workflows/uv.lock. The root package's requires-dist entry now reads { name = "inference-sdk", specifier = ">=1.7.2" } (workflows/uv.lock:4152), matching workflows/pyproject.toml:27. The locked inference-sdk package is still 1.7.2 (workflows/uv.lock:1577-1578), and no other requires-dist entry for inference-sdk exists. The lock is now consistent with the manifest, so the uv lock --check step in make create_workflows_wheel should no longer fail on this mismatch.

Question 1 (the SDK version that ships include_anomaly_map) stayed resolved in the previous run. No blocking findings or IMPORTANT questions remain open. The earlier release-coordination notice still applies unchanged: roboflow-workflows needs a release-time version change. The test suggestion is non-blocking and still stands: an end-to-end workflow test that chains the model block into the heatmap block under both ENABLE_TENSOR_DATA_REPRESENTATION values.

Commands that informed the review: gh pr diff 3201, gh pr view 3201, the paginated gh api reads of issue comments, review comments, reviews and commits, gh api repos/roboflow/inference/commits/3e57b3b6… (the delta's patch), and grep over review-source/workflows/{pyproject.toml,uv.lock}.

Reviewed at HEAD: 3e57b3b

[github-actions]

😎 PR passes the vibe-check and trust-me-bro verification.

[github-actions]

Maintainer review requested.

Final approval and merge remain in GitHub.

[PawelPeczek-Roboflow]

Hi there.
Code itself looks fine - I need integration test for workflows suite and E2E test against the platform on staging + prod. We need models in the reference projects we use for all tests, such that injected secrets match.

[probicheaux]

The below is claude generated

@PawelPeczek-Roboflow addressed:

• Workflows integration test: tests/workflows/integration_tests/execution/test_workflow_with_anomaly_detection.py chains the model block into the heatmap against real FoundAD and PatchCore models, under both ENABLE_TENSOR_DATA_REPRESENTATION values (6/6 each locally).
• E2E against the platform (staging + prod): tests/inference/hosted_platform_tests/test_anomaly_detection_models.py, both models registered in MODELS_TO_BE_USED, through API v0 and v1 and both auth modes (20/20 against a local server from this branch with the prod load-test key).
• Models in the reference projects, so injected secrets match: FoundAD and PatchCore are trained in paul-guerrie/anomaly-det (staging, LOAD_TEST_STAGING_API_KEY) and paul-guerrie-tang1/anomaly-det-algbq (prod, LOAD_TEST_PRODUCTION_API_KEY). The prod project is public on Universe, so the workflows suite reaches it with API_KEY the same way it uses its other public models.

The hosted E2E tests will pass on staging/prod once this ships: the PR also adds anomaly_scale to the response (the request confidence now decides the anomaly verdict; see the description), which the deployed servers don't serve yet.

[probicheaux]

Retested e2e with latest changes, everything looking good!

[PawelPeczek-Roboflow]

LGTM, fixed minor issues and prepared for release