Problem
Currently, the single_buyer_product_seller category has 28 tasks, but the breakdown by modality is highly skewed:
| Type |
Tasks |
Can run with LLM-only? |
| Text-only, price-only |
Task1, Task2, Task3 |
✅ Yes |
| Multimodal + contract |
Task4–Task28 (25 tasks) |
❌ Requires VLM (image input) |
Users who run local LLMs via vLLM/SGLang without vision support (e.g., --language-model-only) can only evaluate on 3 out of 28 single-buyer-single-seller tasks. This makes the benchmark nearly unusable for text-model evaluation or RL training with local models.
Request
Add more text-only, 1v1 negotiation tasks that:
- Do not require image input — no
product_info.image_url, runnable with BaseLLM / OpenAILLM / CustomLLM / VLLMLLM
- Cover more diverse scenarios — the 4 scenario families (E-commerce, Taxi, Food Delivery, Rent House) already have rich text metadata per task (product descriptions, pricing rules, surcharge breakdowns) that work perfectly fine without images
- Optionally include contract_config — multi-dimensional contracts can also be done purely via text, since all
field_descriptions, continuous_bounds, discrete_options are already text-encoded in the prompt
Proposed approach
The simplest way would be to provide "text-only variants" of the existing s1–s25 scenario tasks — same contract_config, same product_info text, same scoring, just dropping the product_info.image_url line. These could be registered under environment IDs like Task_s1_text-v0.
Alternatively, a batch of 10–20 new text-only tasks with varied difficulty levels (close-price, wide-spread, market-aligned, multi-round grinding) would be very helpful for systematic evaluation of text-only models.
Impact
This would significantly lower the barrier for:
- Researchers evaluating open-weight LLMs (Qwen, LLaMA, DeepSeek) without VLM capability
- RL training experiments (e.g., GRPO/PPO on negotiation) that use text-only environments
- Reproducibility studies on commodity GPU setups where VLM inference is prohibitively expensive
(written by deepseek)
Problem
Currently, the
single_buyer_product_sellercategory has 28 tasks, but the breakdown by modality is highly skewed:Users who run local LLMs via vLLM/SGLang without vision support (e.g.,
--language-model-only) can only evaluate on 3 out of 28 single-buyer-single-seller tasks. This makes the benchmark nearly unusable for text-model evaluation or RL training with local models.Request
Add more text-only, 1v1 negotiation tasks that:
product_info.image_url, runnable withBaseLLM/OpenAILLM/CustomLLM/VLLMLLMfield_descriptions,continuous_bounds,discrete_optionsare already text-encoded in the promptProposed approach
The simplest way would be to provide "text-only variants" of the existing s1–s25 scenario tasks — same
contract_config, sameproduct_infotext, same scoring, just dropping theproduct_info.image_urlline. These could be registered under environment IDs likeTask_s1_text-v0.Alternatively, a batch of 10–20 new text-only tasks with varied difficulty levels (close-price, wide-spread, market-aligned, multi-round grinding) would be very helpful for systematic evaluation of text-only models.
Impact
This would significantly lower the barrier for:
(written by deepseek)