VLM Judge Experiments

Static report and reproducible artifacts for comparing VLM judges against specialist visual verifier stacks.

The hosted report is deployed from site/ using GitHub Pages:

https://parsewave.github.io/vlm-judge-experiments/

What This Tests

The experiment asks whether non-VLM or model-assisted specialist stacks can replace a frontier VLM judge for image/video rubric checks.

Four splits are included:

Split	Cases	What it tests
Controlled synthetic	24	Fixed-label generated image/video checks for object presence, count, segmentation, layout, color, charts, timing, and pose.
Sourced assets	24	Similar checks using permissively licensed game/art assets rather than purely synthetic drawings.
Reverse/negative	24	Prompts ask for absent or wrong visual conditions; useful for false-positive behavior.
Real-world stress	12	Photographic scenes with compound requirements, object relations, semantic context, and hard negatives.

Current aggregate from the report:

Verifier	Score
Specialist stack used in each split	80/84
Gemini 2.5 Pro VLM judge	82/84

Stack Comparison

The report includes a dedicated real-world section explaining each stack in plain English.

The real-world stack comparison was run without rerunning Gemini:

Stack	Score	Notes
CLIP single prompt	8/12	Whole-image image-text similarity. Weak on negation and multi-part requirements.
CLIP prompt ensemble	9/12	Same CLIP model with multiple positive/negative prompt phrasings.
SigLIP single prompt	6/12	Modern CLIP-like embedding model; underperformed on this split.
SigLIP prompt ensemble	6/12	Prompt variants did not improve this small split.
OWL-ViT object rules	8/12	Open-vocabulary object detection plus small rule checks.
BLIP-VQA	10/12	Strongest single local model, but this is VQA/VLM-style judging.
ViLT-VQA	7/12	Weaker local VQA baseline.
Routed hybrid, post-hoc	12/12	Feasibility signal only; not a fair benchmark score unless routing is predeclared on held-out data.

References:

• OpenAI CLIP: https://github.com/openai/CLIP
• Google SigLIP: https://huggingface.co/google/siglip-base-patch16-224
• Google OWL-ViT: https://huggingface.co/google/owlvit-base-patch32
• Salesforce BLIP-VQA: https://huggingface.co/Salesforce/blip-vqa-base
• ViLT VQA: https://huggingface.co/dandelin/vilt-b32-finetuned-vqa
• OpenCV: https://github.com/opencv/opencv
• ffmpeg: https://ffmpeg.org/
• Gemini API docs: https://ai.google.dev/gemini-api/docs

Visual Fixes

The report thumbnails are client-facing, so chart cards were redrawn with a separate title band to avoid text/graph overlap.

Real-World Examples

The real-world section shows the actual images, prompts, expected labels, specialist result, Gemini result, and source link.

Repository Layout

.
├── site/                         # GitHub Pages static report
│   ├── index.html
│   ├── data.js
│   ├── script.js
│   ├── style.css
│   └── assets/
├── experiments/
│   ├── scripts/                  # Generation/scoring scripts
│   ├── controlled/               # Controlled split images/videos/results
│   ├── sourced/                  # Sourced-asset split sources/images/videos/results
│   ├── reverse/                  # Reverse split results
│   └── realworld/                # Real-world images/results/stack comparison
└── docs/images/                  # README screenshots

Reproducing

The scripts are in experiments/scripts/.

Typical environment:

python -m venv .venv
source .venv/bin/activate
pip install pillow requests torch transformers scikit-image opencv-python imagehash matplotlib

Gemini/OpenRouter scoring requires an API key:

Set OPENROUTER_API_KEY in your shell before running Gemini-backed scripts.

Run examples:

python experiments/scripts/visual_eval_v5_realworld.py
python experiments/scripts/compare_realworld_stacks.py

Notes:

• site/ is already generated and does not require Python to view.
• experiments/realworld/results/stack_comparison.json contains the local stack comparison details.
• The sourced-asset split includes the copied source assets used for this report under experiments/sourced/source_assets/.
• Learned model stacks such as CLIP, SigLIP, OWL-ViT, BLIP, and ViLT are repeatable only when model weights, preprocessing, thresholds, and runtime are pinned. They should be described as model-assisted checks, not strict deterministic tests.