[iurisilvio]

What does this PR do?

Adds an optional reference_examples input to roboflow_core/google_gemini@v6 and roboflow_core/open_ai@v7 for the object-detection task: labeled example images that go before the image to label in every request. With class names alone the models often know the word but not what the person who labeled the data means by it: where the box goes, how big it is, which of two similar things it is. The examples show that.

Each example is {"image": <URL, base64 or {"type", "value"}>, "boxes": [{"x", "y", "width", "height", "class"}]}, the Roboflow prediction format: box centers and sizes in pixels of the example image, class one of the block's labels. Up to 10 examples. Literal examples are validated when the manifest is parsed, and a selector value such as $inputs.reference_examples is validated when the block runs.

How each block sends them (chosen from the benchmark below):

• Gemini v6: each example image with its boxes and labels drawn on it (longest edge 1024 px), numbered, after a one-line intro.
• OpenAI v7: up to 3 crops per label (largest boxes first, 15% margin, 128–384 px), grouped by label (Examples of "cat":). Box labels are the output labels, so output_classes when set.

In both, the references come first, then Target image: and the image to label, then the block's existing detection prompt with "in this image" pointed at the target and one sentence saying the references are not part of the task. Prompt builders and decoders are unchanged; the block adds parts around what they return. The references are the same for every image of a run, so they form a prefix the provider can cache. The rendered parts are also cached in the process (an LRU of 4 keyed by the examples JSON), so a job does not fetch and redraw the same examples for every image.

Example images load through the host's image codec (roboflow_workflows.utils.images.load_image), with the same URL and local-file policy as any workflow image. Other tasks reject the input. Without it the request bodies are the same as today.

The field lands in place, as other optional fields with a behavior-preserving default did: service_tier on v7 (#3194), output_classes on v7 (#3036), RLE output on Gemini v6.

Main elements:

• workflows/roboflow_workflows/core_steps/common/vlm_reference_examples.py (new): validation, loading, the two renderings, the content-part builders and the merge into the request.
• google_gemini/v6.py, openai/v7.py: the reference_examples field, manifest validation, and the parts added in run_gemini_prompting / run_openai_prompting.
• Tests and the workflows/CHANGELOG.md entry.

Why these renderings

A benchmark on three RF20-VL datasets where text prompts fail (hand X-rays, wood defects, railway fishplates; 40 test images each, 952 boxes, the same 3–4 example images for every method), against the production providers, F1 at IoU 0.5 averaged over the three datasets:

	Text	+ crops	+ boxed images
Gemini 4 Argon	0.68	0.66	0.76
GPT-6.1 Sol	0.46	0.69	0.62

The gain is large where class names do not say what the object looks like (X-rays: Argon 0.65 → 0.85, Sol 0.33 → 0.67; railway: Argon 0.74 → 0.88, Sol 0.59 → 0.86) and absent for Argon on small wood defects (0.63 → 0.57), so this stays opt-in. Gemini bills about 1,100 tokens per image whatever its size, so 12–18 crops cost more than three or four whole images; OpenAI crops are cheap. At list price per 1,000 images: Argon $5.8 → $12.6 with boxed images, Sol $6.8 → $8.2 with crops. OWLv2 few-shot (/owlv2/infer with the same examples) scored 0.17.

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactoring (no functional changes)
• Other:

Testing

Unit tests

• workflows/tests/unit_tests/core_steps/common/test_vlm_reference_examples.py (new, 17 tests): validation (no examples, more than 10, another task, labels the block does not have, the malformed field named in the error), box clipping, crop choice, order and size, boxed rendering (downscale, label color), the part builders, the cache, and the merge for both prompt layouts (text after the image, text before it).
• test_google_gemini_v6.py (+8) and test_openai_v7.py (+8): the manifest accepts a literal list and a selector and rejects other tasks and unknown labels (output labels on v7); run sends the parts in order with the retargeted prompt, including OpenAI's legacy normalized style; requests without examples keep their two parts; resolved examples are rejected for other tasks.

Integration tests

Ran the branch's blocks locally against the staging proxy with the benchmark's examples (6 images, both models, with and without references). Every call succeeded and decoded, and the input token counts match the benchmark's requests (for example 4,826–4,853 tokens for Argon with references on X-rays).

Other

Run locally on macOS with Python 3.12:

• cd workflows && python -m pytest -n 8 tests/unit_tests/core_steps tests/unit_tests/execution_engine: 8828 passed, 181 skipped. The only errors are visualizations/test_fonts_registry.py::test_provisioned_font_asset_is_valid[*], which need the downloaded font assets that this checkout does not have.
• black --check, isort --check-only and the fatal-error flake8 selection from make check_code_quality are clean on the changed files.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code where necessary, particularly in hard-to-understand areas
• My changes generate no new warnings or errors
• I have updated the documentation accordingly (if applicable)

Additional Context

Draft so maintainers can agree on the approach first: the shape of reference_examples and rendering per provider.

Not in this PR: anthropic_claude@v5, other tasks (instance segmentation), and explicit prompt caching for the reference prefix. Labels are drawn with Pillow's bundled font, which covers Latin scripts; labels in other scripts show missing glyphs on the boxed images, while the prompt text still lists them.

🤖 Generated with Claude Code

[github-actions]

👋 Thanks for the pull request! Here is how automated Claude review works here, so you spend credits (and reviewer time) wisely.

🚧 Right now this is a draft, so automated Claude review is paused — nothing is being spent yet. Mark it Ready for review to trigger it.

Warning

💸 The Claude reviewer bills in credits, not vibes

Automated review spins up a real agent that reads real code and spends real credits on every pass. It is glad to help — but it is not a rubber duck, a linter you poke in a loop, or a substitute for reading the contributing guide. Treat it like an expensive senior reviewer whose time you booked, and show up prepared.

Draft when unsure, Ready when you mean it:

• 🌱 Not sure the PR is in good shape yet? Keep it (or set it back) as a draft — drafts pause review, so you can push and iterate without burning credits on a moving target.
• 💪 Feel strong about the contents? Mark it Ready for review and the reviewer will take a look.

However you get there, arrive prepared:

• 🧱 Bring a SOLID, thorough PR. Point your local agent at our skills/ to tune it to our guidelines first — or, if you are one of those fabled carbon-based contributors, read them yourself. A half-baked diff costs exactly the same to review as a finished one.
• ✅ Resolve every comment before you re-request review. Re-requesting with threads still open means paying twice for the same conversation.
• 🔁 Do not use CI review as an inner loop for a local agent. The reviewer is not a step-by-step debugger — do the unfolding locally and arrive with the answer, not the search.
• 🙋 If something looks off, ask a human. One question to a maintainer is cheaper and faster than three rounds of agent re-review chasing a misread.

Reviews are not free. A draft costs nothing to review; a Ready PR is a promise that it is worth reviewing.

• Prefer to skip automated review entirely? Add the skip-claude-review label.