Skip to content

ci: automate InvokeLLM model sync from apper and scoped SDK docs PRs - #284

Open
daniellekorn wants to merge 9 commits into
mainfrom
sdk-docs-automation
Open

daniellekorn wants to merge 9 commits into
mainfrom
sdk-docs-automation

Conversation

@daniellekorn

@daniellekorn daniellekorn commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

What

Automates drift check 1 (LLM model presets) end to end, plus the docs half of it, with no manual step other than merging:

apper RuntimeModel enum ──daily──▶ PR here (types union) ──merge──▶ PR in mintlify-docs (only the model lines)
File Role
.github/workflows/sync-runtime-models.yml Daily / manual. Fetches runtime_model_config.py from apper via the contents API, runs the sync script, opens or updates one PR on bot/sync-runtime-models. The PR closes itself if apper reverts.
scripts/sync-runtime-models.mjs Rewrites the InvokeLLMParams.model union and its Options: JSDoc line in apper's declaration order. Fails loudly if the enum class, the automatic member, either anchor line, or a sane model count is missing.
.github/workflows/publish-sdk-change-to-docs.yml Runs when a model-sync PR merges (or manually for any commit range). Regenerates the reference at the parent and merge commits in worktrees, applies only that diff to mintlify-docs, opens one PR per source commit on bot/sdk-docs/<sha>. Not model-specific: a future connectors sync only adds its branch to the trigger condition.
scripts/scoped-docs-patch.mjs The diff/apply step: English page + all locale mirrors. Refuses page additions/removals (needs docs.json) and patches that do not apply cleanly, with instructions to fall back to create-docs-local.
.github/actions/open-or-update-pr/ Local action that commits the listed paths, force-pushes the branch, and creates / updates / leaves alone / closes the PR. Replaces peter-evans/create-pull-request, which the org's Actions policy blocks (see below).
tests/unit/sync-runtime-models.test.ts, tests/unit/scoped-docs-patch.test.ts Vitest coverage for both scripts, run by the existing unit-tests workflow on this PR.

The docs PR is deliberately scoped: it never carries unrelated unpublished regeneration drift. Full regeneration stays a manual create-docs-local run.

How to review

  1. scripts/sync-runtime-models.mjs: header, then parseRuntimeModels and parseTarget.
  2. scripts/scoped-docs-patch.mjs: header, then main. The core is one git diff --no-index and one git apply -p2 --directory=... per locale.
  3. .github/actions/open-or-update-pr/open-or-update-pr.sh: ~50 lines of bash. The commit is built with write-tree/commit-tree so HEAD never moves.
  4. The two test files: each test(...) title states a guarantee.
  5. The workflows: trigger conditions and the two create-github-app-token blocks.

Verified on this PR (temporary pull_request triggers, see below)

  • Both workflows now start and run up to the first step that needs cross-org access. Before the local PR action they failed at startup with no message: the org's Actions allow-list does not include peter-evans/create-pull-request. Removing that step was the only change that made them start.
  • Sync Runtime Models: checkout, proxy, Node setup pass; fails at "Generate apper read token" with 404 because the App is not installed on base44-dev/apper. Expected until the install below.
  • Publish SDK Change to Docs: checkout, proxy, Node, npm ci, commit-range resolution and regeneration pass; then the same token failure for mintlify-docs.

Verified locally: both scripts against the real apper file and real mintlify-docs tree; the PR action's script against a scratch remote (create / unchanged / update / close / stale-branch cleanup / deletions / misspelled pathspec fails loudly); actionlint + shellcheck, eslint, check_wix_proxy_steps.py, full unit suite.

Setup required before the first real run

No new IDs or secrets. Both workflows reuse vars.BASE44_GITHUB_ACTIONS_APP_ID and secrets.BASE44_GITHUB_ACTIONS_APP_PRIVATE_KEY, i.e. the Base44 Github Actions App (base44-github-actions[bot]). An org admin needs to:

  1. Grant the App Pull requests: read & write (Contents already has write).
  2. The App is currently private, so it can only be installed on the org that owns it. Make it public under the App's Advanced settings (this only makes it installable on other orgs), then install it on base44-dev for apper and mintlify-docs.
  3. Re-run the two failed jobs on this PR from the Checks tab. Expected: Sync reports "already matches (11 presets)", no PR; Publish reports nothing to publish for this PR's range.

Before merging

Revert the commits marked TEMP (they add pull_request triggers for verification). Everything else stays.

How to test after merging

  1. Sync Runtime Models, ref main: expect "already matches", PR none.
  2. Sync Runtime Models, ref 6759819348a0f4588d48a5355b064f26ba3e2735 (apper still had gpt_5_5): expect a real PR on bot/sync-runtime-models with CI on it. Do not merge. Re-run with main: the PR closes itself.
  3. Publish SDK Change to Docs, head_sha = c26f065: expect a mintlify-docs PR touching exactly 8 files (createClient page, English + 7 locales). Genuinely unpublished, safe to merge.
  4. Publish SDK Change to Docs, head_sha = f03e949: expect a failed run at the apply step, no PR. Already published; red is correct.

Known behaviours

  • Commits are authored as base44-github-actions[bot] but are not signed (no "Verified" badge). If a target branch requires signed commits, the push will be rejected visibly.
  • The Claude review/drafter workflows will run on the bot's PRs; add the App's bot login to their allowed_bots lists if unwanted.
  • If a docs PR is left unmerged and a second model change merges, the second patch will not apply and the workflow fails with instructions. Merge the first, then dispatch for the second commit.
  • Unrelated: Check Wix Gateway Proxy fails at startup on every run on main (since at least 2026-09-07). Same policy family is the likely cause: it uses actions/checkout@v4 and actions/setup-python@v6 by tag, and the org policy appears to require SHA pins. Worth its own PR.

Two workflows chain apper -> SDK -> docs for LLM model presets with no
manual step other than merging.

sync-runtime-models.yml (daily / manual) reads apper's RuntimeModel enum
through the contents API and runs scripts/sync-runtime-models.mjs, which
rewrites the InvokeLLMParams.model union and its Options JSDoc line in
apper's declaration order. One PR on bot/sync-runtime-models is opened or
updated in place while drift exists and closed if apper reverts. The
script fails loudly on any anchor it cannot find rather than reporting a
false "in sync".

sdk-docs-scoped-pr.yml runs when that PR merges (or manually for any
commit range). It regenerates the reference at the parent and the merge
commit, and scripts/scoped-docs-patch.mjs applies only that diff to the
published English page and every locale mirror in mintlify-docs, so the
docs PR never carries unrelated unpublished regeneration drift. It
refuses page additions/removals (nav update needed) and patches that do
not apply cleanly, with instructions to run create-docs-local by hand.

Both reuse the existing GitHub App (BASE44_GITHUB_ACTIONS_APP_ID); it
must additionally be installed on base44-dev for apper (read) and
mintlify-docs (write).
@github-actions

github-actions Bot commented Sep 15, 2026

Copy link
Copy Markdown

🚀 Package Preview Available!


Install this PR's preview build with npm:

npm i @base44-preview/sdk@0.8.48-pr.284.344f0bc

Prefer not to change any import paths? Install using npm alias so your code still imports @base44/sdk:

npm i "@base44/sdk@npm:@base44-preview/sdk@0.8.48-pr.284.344f0bc"

Or add it to your package.json dependencies:

{
  "dependencies": {
    "@base44/sdk": "npm:@base44-preview/sdk@0.8.48-pr.284.344f0bc"
  }
}

Preview published to npm registry — try new features instantly!

The scoped docs workflow is not model-specific: any SDK commit range can
be published on its own, and a future connectors sync would only add its
branch to the trigger condition. Name it for what it does.

Add vitest coverage for both scripts so the PR's own CI proves them:
sync-runtime-models (enum parsing incl. docstring/comments/quotes,
anchor detection against the real integrations.types.ts, drift rewrite,
idempotence, every refusal path) and scoped-docs-patch (apply to English
+ locale mirrors, dry run, conflict refusal, missing locale, ignored
README, page add/remove refusal, bad target). Scripts now run main()
only when executed directly so tests can import them.
The scripts now always come from the branch the workflow runs on; only
the reference regeneration checks out the target commits.
The org's Actions policy blocks peter-evans/create-pull-request (the
workflow failed at startup until the step was removed). The local
action commits the listed paths without moving HEAD, force-pushes the
branch, and creates, updates, leaves alone, or closes the PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant