Skip to content

test(agentic): act with Ling 3.0 Flash VL instead of GPT-6 Luna - #111

Merged
Programcode merged 2 commits into
mainfrom
test/agentic-cheap-actor
Oct 7, 2026
Merged

Programcode merged 2 commits into
mainfrom
test/agentic-cheap-actor

Conversation

@Programcode

@Programcode Programcode commented Oct 6, 2026 •

Copy link
Copy Markdown
Owner

Follows up Adam's review comment on #110: use a much cheaper actor model for the agentic tests.

The actor drives agent.act on a replay-cache miss and in explorations. The default changes from openai/gpt-6-luna to inclusionai/ling-3.0-flash-vl ($0.021/$0.062 per million tokens on Novita, DeepInfra as fallback, about a fifth of Luna). The judge stays Muse Spark. E2E_AGENT_MODEL still overrides, and no repository variable sets it, so CI picks up the new default.

Trials on the local fake-mode stack, with --no-cache so every act step hit the model:

  • Gemma 3 12B, all five actor-heavy specs (first-debate, case-generation, practice, ai-tools, workspace), both viewports: 19 of 24 failed. It wandered, and DeepInfra rejected some of its tool-result messages.
  • first-debate and case-generation on desktop: Ling, GPT-5 nano and Qwen 3.5 9B all passed, costing $0.003, $0.012 and $0.027.
  • Ling, all five specs, both viewports: 23 of 24 passed, $0.024 in total. The failure was on desktop: Ling set both languages but left the language dialog open, so the judge couldn't see the Settings page. Rerunning that test twice with Ling passed both times.

Not run: the page sweep and the other specs, which make no agent.act calls, so the actor doesn't touch them. No CI run yet.

No replay entries changed: the model isn't part of the replay key.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Chores
    • Automated end-to-end tests now use Ling 3.0 Flash VL as the default acting model instead of GPT-6 Luna. The judge model remains unchanged.
  • Documentation
    • Updated testing guidance with the model’s pricing, fallback options, and estimated costs and runtime for test runs.

Ling costs $0.021/$0.062 per million tokens on OpenRouter, about a fifth
of Luna. With the replay cache off it passed 23 of 24 actor-heavy tests
on both viewports for $0.024; the one failure, a dialog left open, passed
twice on rerun. Gemma 3 12B, cheaper still, failed 19 of 24.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
debate-navigator Ready Ready Preview Oct 7, 2026 12:25pm UTC
debatelab-staging Ready Ready Preview Oct 7, 2026 12:25pm UTC

Copilot AI balanced review requested due to automatic review settings October 6, 2026 13:57

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@coderabbitai

coderabbitai Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Warning

Review limit reached

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

Next included review available in 27 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Repository YAML (base), Organization UI (inherited)
  • Review profile: CHILL
  • Plan: Free
  • Run ID: d3ea2377-c812-477e-a71c-3b05436fc4ae
📥 Commits

Reviewing files that changed from the base of the PR and between eacfb43 and 9eb6d6f.

📒 Files selected for processing (2)
  • agentic/replay/1273d0cd9f711117b0949152cdb1a8e25ce37cffe17db05cc6132e7a94550551.json
  • docs/testing.md
ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository YAML (base), Organization UI (inherited)
  • Review profile: CHILL
  • Plan: Free
  • Run ID: 06f0475b-2410-4ae9-854f-a77ada74dddb
📥 Commits

Reviewing files that changed from the base of the PR and between 4025827 and eacfb43.

📒 Files selected for processing (5)
  • .claude/skills/verify-debatelab/SKILL.md
  • .claude/skills/verify-debatelab/bin/verify.sh
  • AGENTS.md
  • docs/testing.md
  • e2e.config.ts

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The default E2E acting model changed to inclusionai/ling-3.0-flash-vl. The E2E configuration, command output, and agentic-testing guidance reflect the change. The judge model and environment-variable override remain unchanged.

Changes

Agent model default

Layer / File(s) Summary
Acting-model default and guidance
e2e.config.ts, .claude/skills/verify-debatelab/SKILL.md, .claude/skills/verify-debatelab/bin/verify.sh, AGENTS.md, docs/testing.md
The E2E fallback, command output, and agentic-testing guidance identify Ling 3.0 Flash VL as the acting model. The documentation also describes its provider pricing, fallback, and usage. The judge model remains unchanged.

Estimated code review effort: 2 (Simple) | ~8 minutes

  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Essentials by visiting https://app.coderabbit.ai/settings/billing.

Comment @coderabbitai help to get the list of available commands.

Desktop had no recording for "Pause the source", so CI sent it to the
actor on every run. Uncached, Ling failed it 2 of 7 times: it calls
`check` on the switch without `checked: false` and reports the step
done. Mobile already had a recording. Note the weakness in testing.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Programcode

Copy link
Copy Markdown
Owner Author

None of the seven failing checks ever started. Each one failed in 2 to 4 seconds with no steps, and GitHub's annotation says: "The job was not started because recent account payments have failed or your spending limit needs to be increased." That is the Actions billing limit, not a code failure. No review feedback asks for a change: Copilot didn't review because of its quota, and CodeRabbit left no comments.

My own verification turned up one real gap, which 9eb6d6f fixes. The desktop "Pause the source" step had no replay recording, so CI sent it to the actor on every run. With the cache off, Ling failed that step 2 of 7 times. It calls check on the switch without checked: false (once it even sent "target":"n69019checkedfalse") and then reports the step done. The commit adds the desktop recording (mobile already had one) and a sentence about this weakness in docs/testing.md.

Local results on 9eb6d6f, fake-provider stack. These are local passes, not CI passes.

  • npm run verify (run at eacfb43, which differs only in the replay entry and one docs sentence): exit 0. That covers the docs check, lint, typecheck, Vitest (27 files, 273 tests), collaboration (5 files, 30 tests), Edge tests (318 passed), the build, the service-worker offline check, and the Playwright desktop-chromium @smoke set (29 passed). npm run docs:check passes again at 9eb6d6f.
  • Agentic suite (desktop, journeys) and suite (mobile, journeys), run with CI's invocation: 33/33 passed on each, 16 steps replayed, 0 actor calls.
  • Actor with the cache off (workspace and ai-tools on desktop): 7 of 8 tests passed. The failure is the one above, and the pause test alone passed 5 of 6 repeats. 56 actor calls cost about half a cent.
  • Not run: the two pages legs, browser-smoke, quality as a CI job, and anything on staging. The pages legs are judge-only, and this PR doesn't change the judge.

The repo has no E2E_AGENT_MODEL variable, so CI will use the new default. OpenRouter lists inclusionai/ling-3.0-flash-vl with tool calling and image input. Each agent.act keeps e2e's default limit of 25 steps and 25 model calls.

🤖 Generated with Claude Code

@Programcode
Programcode merged commit 70357d7 into main Oct 7, 2026
8 of 15 checks passed
@Programcode
Programcode deleted the test/agentic-cheap-actor branch October 7, 2026 15:49

This branch was successfully deployed

2 active deployments
Preview – debate-navigator — 9eb6d6f6 Deployed Oct 7, 2026 by vercel[bot]
Preview – debatelab-staging — 9eb6d6f6 Deployed Oct 7, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants