Skip to content

Move saved Opus 5.5 settings to Sonnet 5.5 (data migration, after #164) - #180

Merged
Programcode merged 4 commits into
mainfrom
cost/opus-to-sonnet-migration
Oct 22, 2026
Merged

Programcode merged 4 commits into
mainfrom
cost/opus-to-sonnet-migration

Conversation

@Programcode

@Programcode Programcode commented Oct 22, 2026 •

Copy link
Copy Markdown
Owner

Follow-up to #164. Opus 5.5 was the Pro default, so most Pro accounts still have it saved for every tool. A saved setting can't tell the old default from a deliberate pick, so this data migration moves every saved Opus 5.5 tool setting to Sonnet 5.5. Opus stays in the picker, and anyone can choose it again.

Do not merge before #164. This PR targets cost/sonnet-pro-default. Once #164 is merged, retarget it to main.

What the migration does

supabase/migrations/20261022120000_opus_default_to_sonnet.sql:

  • For each user_settings row, it changes every tool entry in feature_configs whose model is exactly anthropic/claude-opus-5.5 to anthropic/claude-sonnet-5.5. Other fields (reasoning effort, provider) and other tools stay as they are.
  • It copies the original and the new feature_configs into a new server-only table, private.opus_default_backup, before the update. The table has RLS on and no client grants, and its rows are deleted with the account.
  • It skips any user who already has a backup row. A second run therefore changes nothing, even for someone who picked Opus again after the first run.
  • It does not touch retired Opus IDs (anthropic/claude-opus-5, which the client maps to Opus 5.5 as an explicit pick), empty settings, or malformed values.

It covers every account with Opus 5.5 saved, not only accounts that are Pro right now. Pro access depends on the clock and on Stripe rows, so SQL can't check it reliably. For a Basic or lapsed account the change has no visible effect, because the client already shows Haiku for any Pro-only model. If that account becomes Pro again, it starts on Sonnet rather than Opus.

Rollback

supabase/rollbacks/20261022120000_opus_default_to_sonnet.down.sql. It isn't in supabase/migrations, so db push never runs it. It sets a tool back to Opus 5.5 only where the migration changed that tool from Opus to Sonnet and the tool still holds Sonnet. Changes a user made after the migration stay. It then empties the backup table, so a second run does nothing. It contains no DROP. The rollback needs private.opus_default_backup, so keep that table until it's clear no rollback will be needed; dropping it is a separate decision.

To roll back: copy the file into supabase/migrations/ under a new timestamp, open a PR, and apply it like any other migration. This follows the forward-corrective approach in docs/operations.md.

Order and apply steps

Done. Do not re-apply or redeploy. The migration was applied to production on 2026-10-22 at 16:38:59 Budapest (14:38:59 UTC) and is live. Production's Edge Functions are already newer than this PR's commit. The staging-sync timer applied it to staging earlier, at bf4c34e. A second run would change nothing, but don't run it again.

The steps below are kept as a record of how it was meant to go out.

  1. Production serves Make Claude Sonnet 5.5 the Pro default and Opus opt-in only #164's frontend first. Under the old frontend, an account with no saved settings would still get Opus on its first load.
  2. The migration runs from a checkout of the newest reviewed main, never from this PR's merge commit, which lacks later migrations. Before pushing, read the full backend:status pending list. Use --include-all for both the dry run and the push only when production already has a migration newer than 20261022120000.
  3. Afterwards, private.opus_default_backup holds one row per account that was moved, and no saved tool setting holds anthropic/claude-opus-5.5.

Edge Functions are a separate step from this migration and come only from the newest reviewed main. Never deploy them from #164's or this PR's commit, because later PRs changed them.

Known limitation

The client writes all six tool settings at once, without checking updated_at, whenever someone changes a setting or the theme. It can therefore write Opus back for the other tools when its in-memory settings are older than the migration:

  • a tab left open since before the migration;
  • a fresh load whose settings read or Pro check fails, so the client keeps the settings it cached in the browser from its last session.

The client's own one-off settings cleanup does check updated_at, and the migration bumps updated_at, so that path can't write Opus back. The rollback leaves rows written back this way alone, because those tools no longer hold the migrated Sonnet. Closing the gap would need a client change; this PR doesn't make one.

Verification (local; GitHub Actions is down repo-wide)

  • ./.superset/db-gate.sh --db-only: all pgTAP tests passed (including account deletion and server-only table grants), plus the database scripts.
  • The new scripts/test-opus-default-migration.sh, now part of npm run test:db, seeds six accounts: all Opus, mixed tools, retired Opus ID, never saved, Basic with Opus, and a malformed value. It runs the migration, runs it again after one user picks Opus again, changes settings, then runs the rollback twice. It checks every row after each step, and everything runs in one transaction that is rolled back.
  • I broke the migration and the rollback three ways on purpose to check that the test catches it. Without the skip-already-migrated guard, the second-run check fails. If the rollback ignores later picks, the "model picked after the migration is kept" check fails. If the rollback keeps the backup, the "rollback empties the backup" check fails.
  • After the Opus review, the test also seeds a tool that was already on Sonnet before the migration and an old updated_at. Two more deliberate breaks now fail it: a rollback without the "was Opus before" check fails "a tool already on Sonnet before it is kept", and a migration without the updated_at bump fails "migrated rows get a new updated_at". At 8ca1e46 the full database set (pgTAP plus all database scripts) passed on a private scripts/agent-stack.sh stack, which now also runs this test.
  • npm run docs:check and npm run lint (including the repo rules) passed.
  • Not applied to staging or production. No CI run.

🤖 Generated with Claude Code

Programcode and others added 2 commits October 22, 2026 10:55
New Pro accounts start on Sonnet 5.5. Auto Pro routes between Sonnet 5.5
and GPT-6.1 Sol and falls back to Sonnet, so it never picks Opus. Opus
runs only when someone selects it. Retired Opus IDs still move to Opus
5.5, since those were explicit picks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Opus 5.5 was the Pro default, and a saved setting can't tell the default
from a pick. This data migration moves every saved Opus 5.5 tool setting
to Sonnet 5.5, backs the original rows up in private.opus_default_backup,
and skips users it has already migrated. A rollback script restores Opus
wherever the migrated Sonnet is still saved. A database test runs both
twice against seeded rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 22, 2026 12:14
@vercel

vercel Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
debate-navigator Ready Ready Preview Oct 22, 2026 1:11pm UTC
debatelab-staging Ready Ready Preview Oct 22, 2026 1:11pm UTC

@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

Next included review available in 29 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
📥 Commits

Reviewing files that changed from the base of the PR and between 988b870 and 9565d73.

📒 Files selected for processing: 7
  • .claude/skills/verify-debatelab/features/model-defaults.md
  • package.json
  • scripts/agent-stack.sh
  • scripts/test-opus-default-migration.sh
  • supabase/README.md
  • supabase/migrations/20261022120000_opus_default_to_sonnet.sql
  • supabase/rollbacks/20261022120000_opus_default_to_sonnet.down.sql
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Essentials by visiting https://app.coderabbit.ai/settings/billing.

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@Programcode

Copy link
Copy Markdown
Owner Author

The GitHub Actions failures on 220b367 never ran the code. Each job stopped within 2 to 3 seconds with "The job was not started because recent account payments have failed or your spending limit needs to be increased", which is the repo-wide billing block. Local results at 220b367: ./.superset/db-gate.sh --db-only passed (pgTAP and the database scripts, including the new scripts/test-opus-default-migration.sh), and npm run docs:check and npm run lint passed. That is a local pass, not a CI pass.

The test now seeds a tool already on Sonnet before the migration, which the
rollback must keep, and an old updated_at, so it fails if the migration
stops bumping it. Clients only save their settings cleanup while updated_at
is unchanged, so the bump keeps an older client from writing Opus back.
agent-stack.sh test now runs the script too, like npm run test:db.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 22, 2026

Copy link
Copy Markdown

Deployment failed for project debatelab-staging with the following error:

Resource is limited - try again in 24 hours (more than 100, code: "api-deployments-free-per-day").

Learn More: https://vercel.com/adam-gaborjanis-projects?upgradeToPro=build-rate-limit

@Programcode

Copy link
Copy Markdown
Owner Author

Review-only Opus pass on 220b367 found no high or medium defects. Four low findings, all checked against the code:

  1. scripts/agent-stack.sh test skipped the new test, although docs/development.md says it runs what npm run test:db runs. Fixed in 8ca1e46: it's now in the script list, and it passed on an agent stack.
  2. The test couldn't catch a rollback that drops the "was Opus before" check, because no seeded tool was on Sonnet before the migration. Fixed in 8ca1e46: user b now has brainstorm on Sonnet 5.5 beforehand. With that check removed, the test fails ("…a tool already on Sonnet before it are kept").
  3. Nothing tested the updated_at = now() bump, which is what makes persistMigratedFeatureConfigs (settings-store.ts, .eq('updated_at', …)) skip a stale client's cleanup save. Fixed in 8ca1e46: the seed sets an old updated_at. With the bump removed, the test fails ("migrated rows get a new updated_at…").
  4. The "Known limitation" section said only tabs left open across the migration are affected. hydrateFromServer also returns early when the settings read or the Pro check fails, and the client then keeps the settings it cached in the browser, which an unguarded updateSettings call can write back. Fixed in the PR description. It's a client-side gap, which this PR doesn't change.

@Programcode

Copy link
Copy Markdown
Owner Author

Checks on 8ca1e46: the GitHub Actions jobs stopped after 2 to 3 seconds because of the same billing block, so none of them ran the code. Vercel – debatelab-staging failed on Vercel's free-plan limit of 100 deployments a day (api-deployments-free-per-day), not on the build; the debate-navigator preview built fine. This PR changes no frontend code. CodeRabbit skipped the review because this PR targets cost/sonnet-pro-default rather than main. Local results at 8ca1e46 are in the PR description.

@Programcode
Programcode changed the base branch from cost/sonnet-pro-default to main October 22, 2026 13:47
# Conflicts:
#	.claude/skills/verify-debatelab/features/model-defaults.md
#	supabase/functions/_shared/model-catalog.ts
@vercel

vercel Bot commented Oct 22, 2026

Copy link
Copy Markdown

Deployment failed for project debate-navigator with the following error:

Resource is limited - try again in 24 hours (more than 100, code: "api-deployments-free-per-day").

Learn More: https://vercel.com/adam-gaborjanis-projects?upgradeToPro=build-rate-limit

@Programcode
Programcode merged commit bf4c34e into main Oct 22, 2026
3 of 14 checks passed
Programcode added a commit that referenced this pull request Oct 22, 2026
#180 added 20261022120000_opus_default_to_sonnet.sql, so the refund
migration's identical version failed to apply after the merge from main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Programcode added a commit that referenced this pull request Oct 22, 2026
Keep calculator v2 for the logged cost (BYOK fee plus the provider
charge) next to #178 cap figure, which still counts only OpenRouter fee.
Rename the backfill to 20261022160000: #180 took 20261022120000.
@Programcode

Copy link
Copy Markdown
Owner Author

Codex review (gpt-6.1-sol, xhigh) of 9565d73 compared the code with main at c27cace and found no SQL problems. It found two problems in the apply steps, both real, both fixed in the PR description:

  1. Medium: step 2 said to deploy ai and ai-stream "from Make Claude Sonnet 5.5 the Pro default and Opus opt-in only #164". Repair model JSON locally and enforce schemas before paying for a retry #176, Weight the daily AI quota by model and cap AI spend at $2 a week (BYOK Claude excluded) #178, Log the real cost of Anthropic (BYOK) calls and Exa research #179 and Cap each AI feature's reply length instead of 32,768 tokens #182 changed those functions or _shared after Make Claude Sonnet 5.5 the Pro default and Opus opt-in only #164, so deploying from Make Claude Sonnet 5.5 the Pro default and Opus opt-in only #164's commit would undo them, including the weighted quota and the weekly spending cap. The steps now say to deploy functions only from the newest reviewed main commit, as a separate step from this migration. The migration doesn't depend on any function.
  2. Low: step 4 said to run a plain db push from a checkout of the merge commit. bf4c34e lacks 20261022130000, 20261022130001 and 20261022160000. If production already has one of those, a push from that checkout fails, and a plain push also refuses to insert the older 20261022120000. The steps now say to run from the newest reviewed main, read the full pending list, and use --include-all for both the dry run and the push only when a newer migration is already applied.

Current status: staging-sync applied this migration to staging at bf4c34e. Production has not had it applied. Vercel rate-limited every production frontend deploy since #164, so production still serves the frontend from before #164, and the migration has to wait for it (step 1).

Programcode added a commit that referenced this pull request Oct 22, 2026
…185)

* Run Claude only on the Anthropic BYOK key, with no fallback provider.

Every anthropic/* call now asks OpenRouter for the anthropic provider only,
with fallbacks off, so Bedrock, Vertex and other providers are never tried.
Other models keep balanced routing. A refused Claude call returns a 503
outage message the client already translates, logs claude_unavailable, and
gives the reserved daily request back through a new refund_daily_quota RPC.
A Claude response whose usage shows is_byok false or a cost above zero logs
a claude_not_byok warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Refund a failed Claude request only when nothing was written yet.

A stream that fails after the model wrote text, or a repair retry after a
completed attempt, already spent tokens on the key, so the request stays
counted. The student still sees the outage message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Keep paid research out of Claude refunds and stop the Ask double attempt.

From the Opus review of #185:
- A request whose research steps already ran on Claude keeps its quota.
- Ask no longer resends a provider_unavailable stream failure through the
  plain endpoint, which tried Claude a second time on a failing key.
- ai-stream refunds before it sends the error, so an immediate retry sees it.
- Sparring shows the translated provider message instead of the raw one.
- Tests cover no refund on success, on a non-Claude failure, after a
  completed repair attempt, and after research.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Let a quota refund give back more than one unit.

refund_daily_quota takes p_units (default 1, the reservation every AI request
makes today) and never goes below zero. Once the weighted quota in #178
lands, the refund can return what consume_ai_quota reserved for the model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Give the refund migration its own version.

#180 added 20261022120000_opus_default_to_sonnet.sql, so the refund
migration's identical version failed to apply after the merge from main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Refund the weighted units consume_ai_quota reserved, and keep refunds out of the weekly cap.

#178 reserves each request's weight (Opus 5, Sonnet 2, others 1; Auto Pro
5) through consume_ai_quota. A refunded Claude failure now gives back the
same units checkRateLimit reserved, taken from its RateLimitResult, and
logs cap_cost_usd 0 so it does not count toward the weekly $2 either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Never refund a Claude failure OpenRouter already charged credits for.

Zeroing cap_cost_usd on such a row would hide real spend from the weekly
cap, so a request with a charge against the cap now keeps both its quota and
its cap cost. Only free failures are refunded and logged with cap cost 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Refund research tools in a Claude outage too.

Research steps run on Haiku, so in an outage they are refused along with the
main call. Such a refusal did no paid work and no longer blocks the refund;
a research step that finished or was charged still does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Bill usage on a stream's error frame, and never refund a cancelled request.

From Codex review round 2 of #185:
- Both stream readers threw on an error frame before reading its usage, so
  a charge reported only there was dropped, and the failure could then be
  refunded with cap cost 0. Usage is now kept before the error is thrown.
- ai/index.ts refunded a Claude refusal that raced a client abort.
  ai-stream already refused that; the plain handler now does too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@Programcode

Copy link
Copy Markdown
Owner Author

Status: the migration was applied to production on 2026-10-22 at 16:38:59 Budapest (14:38:59 UTC) and is live. Production's Edge Functions are already newer than this PR's commit. The apply steps in the description are now marked done and kept as a record. Don't re-apply the migration or redeploy anything for this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants