Skip to content

feat(pricing): check official display pricing drift - #136

Merged
pierreraby merged 5 commits into
patlux:mainfrom
sandexzx:feat/commandcode-pricing-check
Oct 6, 2026
Merged

pierreraby merged 5 commits into
patlux:mainfrom
sandexzx:feat/commandcode-pricing-check

Conversation

@sandexzx

@sandexzx sandexzx commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Add a read-only npm run check:commandcode-pricing command to detect pricing drift before newly advertised models silently display zero cost.

  • Compare the live Provider API model catalog and the official pricing page with MODEL_COSTS.
  • Parse and validate the page's structured Next.js Flight props.rows payload without executing JavaScript or adding dependencies.
  • Report missing prices, changed base rates/context tiers, time-pricing policy mismatches, ambiguous/missing page matches, and expired dated promotions.
  • Use exit codes 0 (matches), 1 (manual review required), and 2 (check could not complete).
  • Run alongside the existing catalog automation and publish Markdown reports to Actions summaries. Explicit shell: bash preserves checker failures through tee using pipefail.
  • Keep the pricing check after catalog PR creation so independent metadata synchronization is not blocked.
  • Add 41 offline tests and a focused real-page fixture; document the manual-review workflow.

This PR is based directly on main, independent of #135. No runtime files, manual prices, verification dates, or existing model/pricing fixtures are changed. The checker never writes prices or credentials and does not introduce runtime network requests.

Validation

  • npm test — passed, including TypeScript and real Pi mock-API integration. OMP compatibility skipped locally because omp is not installed.
  • npm run format:check — passed.
  • git diff --check — passed.
  • actionlint 1.7.7 on .github/workflows/model-metadata.yml — passed.
  • Live npm run check:commandcode-pricing — successfully parsed 85 API models and 89 page rows, exited 1 / REVIEW REQUIRED, as expected for current main.

Expected live-check failures

The new pricing step will currently fail intentionally; these are pre-existing data/runtime-policy discrepancies, not parser failures:

  • Missing MODEL_COSTS entries for claude-sonnet-5-5, gpt-6.1-sol, deepseek/deepseek-v4.1-flash-fast, and inclusionai/ling-3.1-flash:free (addressed separately by fix(pricing): add October catalog display prices #135).
  • Missing peak-pricing policy for DeepSeek V4.1 Flash Fast (also addressed by fix(pricing): add October catalog display prices #135).
  • Qwen 3.6 Plus: official > 256K context tier absent locally.
  • Step 3.5 Flash: official input rate 0.09, local rate 0.1.
  • Qwen 3.7 Max: the page still advertises a promotion with an expired 2026-06-22 date; this needs manual review of the upstream information.

The checker must not rewrite rates merely to make this job green. Actual prices remain manually reviewed, and the billing Usage page is authoritative.

Source-format limitation

The pricing page is not a public pricing API. This parser depends on its private Next.js Flight format and deliberately fails with exit code 2 if the required structure becomes unrecognizable, rather than reporting a false pass.

pierreraby
pierreraby previously approved these changes Oct 5, 2026

@pierreraby pierreraby left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified: 41/41 offline tests green, typecheck clean; ran the live checker locally and it correctly flags the four missing prices from #135 (expected — this branch predates it) plus real findings (Qwen3.6-Plus tier, Step-3.5-Flash 0.10→0.09, expired Qwen deal), so the tool bites. Read-only, no runtime or pricing changes. Leaving the merge to the maintainers — will merge tomorrow if no objections.

pierreraby
pierreraby previously approved these changes Oct 6, 2026

@pierreraby pierreraby left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-verified after the author's rebase onto 13acd52 (merge 1a4af44, changelog + 0.7.4 version only — no functional change vs 42c7cb5): typecheck + checker tests 41/41 green locally. Runs approved, merging as announced.

Comment thread .github/scripts/check-commandcode-pricing.ts Fixed
Comment thread .github/scripts/check-commandcode-pricing.ts Fixed
pierreraby
pierreraby previously approved these changes Oct 6, 2026

@pierreraby pierreraby left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-verified on cb38cf1 (merge of main + findings): typecheck, checker tests 42/42 green locally. Merging once CI is green.

@pierreraby

Copy link
Copy Markdown
Collaborator

Remaining drift findings after #135 (for follow-up, out of this PR's scope): Qwen/Qwen3.6-Plus long-context tier above 256K missing, stepfun/Step-3.5-Flash input 0.10 → 0.09, expired qwen-3.7-max-2x-usage deal. The checker correctly surfaces them — merging this tooling PR so it can keep reporting.

@pierreraby

Copy link
Copy Markdown
Collaborator

Heads-up, @sandexzx: #138 just merged into main, so this PR now conflicts on two files. Both resolutions are mechanical — keeping both sides each time:

  1. CHANGELOG.md (## Unreleased): keep both bullets — the reasoning-normalizer entry from fix: restore incremental Responses reasoning #138 first, then your checker entry.
  2. package.json (scripts.test): keep both additions in the chain — node --test tests/test-release.mjs tests/test-pi-child-cleanup.mjs (from fix: restore incremental Responses reasoning #138) and tsx tests/test-pricing-check.ts (yours, after test-pricing).

I verified the merged result locally (typecheck, release tests 46/46, pricing-check 42/42, transport 16/16, prettier clean), so a fresh git merge main resolving those two files the same way should go green. I can't push to your fork, hence this note instead of a commit. Thanks!

pierreraby
pierreraby previously approved these changes Oct 6, 2026

@pierreraby pierreraby left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final head b702076 verified locally: typecheck, checker 42/42, transport 16/16, release 46/46, pi-local PASS, prettier clean. CI + benchmark green; catalog-sync check reports only the 3 known residual drifts (already tracked in comments), no missing prices. Merging.

@pierreraby

Copy link
Copy Markdown
Collaborator

Almost there — but the branch policy requires the pricing check green, and it can't pass while 3 residual drifts remain. Could you fold these into the PR? I verified each against the official page just now:

  1. stepfun/Step-3.5-Flash: input: 0.1 → 0.09 in src/pricing.ts (page shows $0.09 in / $0.30 out / $0.02 cache).
  2. Qwen/Qwen3.6-Plus: add the long-context tier tiers: [{ inputTokensAbove: 256_000, input: 2, output: 6, cacheRead: 0.2, cacheWrite: 0 }] (page shows the 256K step-up; same shape as the Qwen3.7 tiers already in the file).
  3. Expired qwen-3.7-max-2x-usage deal (ended 2026-06-22): remove or mark resolved wherever the checker sources it, so the check no longer flags it.

Plus the matching fixture updates (commandcode-pricing.json, tests) — your checker tests will tell you exactly what's expected. Once the sync check is green I'll merge right away. Thanks in advance — and thanks again for the tool itself, it's already proving its worth.

@pierreraby pierreraby left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final head 90b8e00 verified: typecheck, checker 44/44, pricing 9/9, cost 9/9 green locally — and the live checker now reports PASS (no drift). The resolvedExpired mechanism for the stale Qwen badge is cleaner than my removal suggestion. Merging.

@pierreraby
pierreraby merged commit 35e7fd2 into patlux:main Oct 6, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants