From 0ca67c6b4774a67699c68ef9f0d5826e7950bb5c Mon Sep 17 00:00:00 2001 From: MervinPraison Date: Wed, 5 Aug 2026 07:52:22 +0100 Subject: [PATCH] ci(release): harden the pipeline and add first-class single-package releases Multi-agent gap review of the release pipeline surfaced 17 findings across the gate, the workflow, and the bump script. This fixes the confirmed ones. Gate (release-gate.js + nightly-release-gate.yml): - A pypi-release run stuck in `waiting` (environment approval) blocked all auto-releases with no age limit -- GitHub only auto-fails unapproved deployments after 30 days, and exactly this stalled releases for 16h this week. Waiting runs older than 6h are now ignored (with a warning linking the run); in_progress/queued runs still block unconditionally, and the stuck run is never cancelled -- if later approved, the concurrency group serializes it and pypi_exists no-ops anything already published. - PACKAGE_PATHS covered 3 of 9 published packages, so a change landing only in e.g. praisonai-bot was never auto-released. Now lists all nine. - The 3-day dedupe counted any successful run -- including dry_run=true, which publishes nothing -- blocking real releases for 3 days after a dry run. Dedupe now keys on v* GitHub releases, which only real releases create. - The nightly path required last-green-Core-Tests SHA == HEAD exactly; a docs-only or [skip ci] commit at the tip (the release safety-net commit itself!) stalled the cron path indefinitely. A green ancestor with no package-path changes since is now accepted. - Replaced the dead 'Bump praisonai' workflow_run filter (those commits no longer exist) with 'chore(release):', and wired the previously never-run release-gate selftest into the gate as a step. Workflow (pypi-release.yml): - New `only` input: a dedicated single-package release is one dropdown pick instead of eight skip flags. A "Resolve effective skips" step is the single source of truth; the 19 publish/wait conditions are untouched (they gate via the seeded pypi_exists outputs). - PyPI existence probes now distinguish 404 (publish) from outages (fail the run before anything is published) instead of treating any error as "not published" and then dying on duplicate upload midway. - The token-refresh step is continue-on-error and its consumer only swaps tokens on success -- a failed re-mint previously overwrote a possibly-valid token with an empty string, guaranteeing the safety net could not push. - The safety net now (a) reverts uncommitted wrapper-side rewrites when the wrapper release didn't complete, so a wrapper version that never reached PyPI is not committed and then skipped past forever, and (b) retries rebase+push 3x preferring the version bumps on conflict -- losing that commit deadlocks future auto-releases ("already published" forever). - "Verify PyPI versions" actually probes PyPI now; it previously echoed checkmarks derived from input flags for publishes that may never have run. - Dedicated releases get per-package tags (praisonai-mcp-v0.0.12) since the wrapper's "Release v..." tag doesn't exist on that path; only packages confirmed on PyPI are tagged. - uv publish --check-url makes re-uploads idempotent (partial-upload recovery); dry-run summary now includes sandbox/deploy and the effective skip plan. Bump script (bump_and_release.py): - A conflicted `git pull --rebase` previously printed "trying to continue" and proceeded from a mid-rebase detached HEAD -- tagging the wrong commit and cascading conflict markers into the safety net. It now aborts, retries preferring the release commit, and exits before pushing if that fails. - Commit failures are no longer swallowed (only "nothing to commit" is tolerated), so the tag can't silently point at the previous HEAD. - `gh release create` is idempotent (view-then-create), unblocking reruns after a transient failure between push and wrapper publish. - Tag push is scoped to the release's own tag; `git push --tags -f` could force-move historical tags. Verified: node selftest (10 checks incl. new stale-waiting/dry-run-dedupe cases), py_compile, YAML parse, actionlint (same 3 pre-existing SC2129 style warnings as main, nothing new), and the rebase-conflict retry logic exercised against a scratch remote (bump wins, upstream commit preserved). Co-Authored-By: Claude Fable 5 (1M context) --- .github/scripts/release-gate-selftest.js | 113 ++++--- .github/scripts/release-gate.js | 93 +++++- .github/workflows/nightly-release-gate.yml | 6 +- .github/workflows/pypi-release.yml | 341 +++++++++++++++------ src/praisonai/scripts/bump_and_release.py | 63 +++- 5 files changed, 456 insertions(+), 160 deletions(-) diff --git a/.github/scripts/release-gate-selftest.js b/.github/scripts/release-gate-selftest.js index bccd9bd7fa..4aaa0bf121 100644 --- a/.github/scripts/release-gate-selftest.js +++ b/.github/scripts/release-gate-selftest.js @@ -11,71 +11,112 @@ assert.ok(versions.currentCode); assert.ok(versions.currentWrapper); assert.ok(versions.targetCode); assert.ok(rg.PACKAGE_PATHS.includes('src/praisonai-code')); +// Every released package must make the gate's path-change check eligible. +for (const p of [ + 'src/praisonai', 'src/praisonai-agents', 'src/praisonai-code', + 'src/praisonai-bot', 'src/praisonai-train', 'src/praisonai-browser', + 'src/praisonai-mcp', 'src/praisonai-sandbox', 'src/praisonai-deploy', +]) { + assert.ok(rg.PACKAGE_PATHS.includes(p), `PACKAGE_PATHS missing ${p}`); +} const noonUtc = new Date('2026-07-08T12:00:00Z'); const dayStart = rg.utcDayStart(noonUtc); assert.strictEqual(dayStart.toISOString(), '2026-07-08T00:00:00.000Z'); -(async () => { - assert.strictEqual(await rg.pypiVersionExists('praisonaiagents', '0.0.0'), false); - - const mockGithub = { +function mockGithub({ runs = [], releases = [] } = {}) { + return { rest: { actions: { - listWorkflowRuns: async () => ({ - data: { - workflow_runs: [ - { - conclusion: 'success', - created_at: '2026-07-08T09:00:00Z', - status: 'completed', - }, - ], - }, - }), + listWorkflowRuns: async () => ({ data: { workflow_runs: runs } }), + }, + repos: { + listReleases: async () => ({ data: releases }), }, }, }; +} + +(async () => { + assert.strictEqual(await rg.pypiVersionExists('praisonaiagents', '0.0.0'), false); + assert.strictEqual(rg.PATCH_RELEASE_INTERVAL_DAYS, 3); + // Dedupe is release-based: a v* release inside the window blocks… + const recentRelease = mockGithub({ + releases: [{ tag_name: 'v4.6.159', published_at: '2026-07-08T09:00:00Z' }], + }); assert.strictEqual( - await rg.hasSuccessfulReleaseWithinDays(mockGithub, 'o', 'r', noonUtc), + await rg.hasSuccessfulReleaseWithinDays(recentRelease, 'o', 'r', noonUtc), true ); const result = await rg.evaluateReleasePreflight( - mockGithub, 'o', 'r', + recentRelease, 'o', 'r', { headSha: 'abc', isCiTrigger: false, bump: 'patch', now: noonUtc }, null ); assert.ok(result.reasons.some((r) => r.includes('every 3 days'))); - const oldReleaseGithub = { - rest: { - actions: { - listWorkflowRuns: async () => ({ - data: { - workflow_runs: [ - { - conclusion: 'success', - created_at: '2026-07-04T09:00:00Z', - status: 'completed', - }, - ], - }, - }), - }, - }, - }; + // …an old release does not… + const oldRelease = mockGithub({ + releases: [{ tag_name: 'v4.6.150', published_at: '2026-07-04T09:00:00Z' }], + }); assert.strictEqual( - await rg.hasSuccessfulReleaseWithinDays(oldReleaseGithub, 'o', 'r', noonUtc), + await rg.hasSuccessfulReleaseWithinDays(oldRelease, 'o', 'r', noonUtc), false ); + // …and a dry-run "success" (workflow run, no release) does not consume the + // window: only real releases count. + const dryRunOnly = mockGithub({ + runs: [{ conclusion: 'success', created_at: '2026-07-08T09:00:00Z', status: 'completed' }], + releases: [], + }); + assert.strictEqual( + await rg.hasSuccessfulReleaseWithinDays(dryRunOnly, 'o', 'r', noonUtc), + false + ); + + // Active-run blocking: fresh waiting run blocks… + const freshWaiting = mockGithub({ + runs: [{ status: 'waiting', conclusion: null, created_at: '2026-07-08T11:00:00Z', html_url: 'x' }], + }); + assert.strictEqual( + await rg.hasActiveReleaseRun(freshWaiting, 'o', 'r', noonUtc, null), + true + ); + + // …a waiting run stuck longer than STALE_WAITING_HOURS no longer blocks… + const staleWaiting = mockGithub({ + runs: [{ status: 'waiting', conclusion: null, created_at: '2026-07-08T01:00:00Z', html_url: 'x' }], + }); + const warnings = []; + const fakeCore = { warning: (m) => warnings.push(m), info: () => {} }; + assert.strictEqual(rg.STALE_WAITING_HOURS, 6); + assert.strictEqual( + await rg.hasActiveReleaseRun(staleWaiting, 'o', 'r', noonUtc, fakeCore), + false + ); + assert.ok(warnings.length === 1 && warnings[0].includes('approve or cancel')); + + // …but an in_progress run always blocks regardless of age. + const oldInProgress = mockGithub({ + runs: [{ status: 'in_progress', conclusion: null, created_at: '2026-07-07T01:00:00Z', html_url: 'x' }], + }); + assert.strictEqual( + await rg.hasActiveReleaseRun(oldInProgress, 'o', 'r', noonUtc, null), + true + ); + console.log('ok: bumpPatch'); console.log('ok: readVersionsFromTree'); + console.log('ok: PACKAGE_PATHS covers all 9 released packages'); console.log('ok: pypiVersionExists missing version'); console.log('ok: utcDayStart'); - console.log('ok: hasSuccessfulReleaseWithinDays'); + console.log('ok: release-based dedupe (recent blocks, old does not)'); + console.log('ok: dry-run success does not consume the release window'); console.log('ok: 3-day dedupe blocks second release'); + console.log('ok: fresh waiting run blocks; stale waiting run ignored with warning'); + console.log('ok: in_progress always blocks'); })(); diff --git a/.github/scripts/release-gate.js b/.github/scripts/release-gate.js index be1d67847d..a34caa3885 100644 --- a/.github/scripts/release-gate.js +++ b/.github/scripts/release-gate.js @@ -4,12 +4,31 @@ const https = require('https'); -const PACKAGE_PATHS = ['src/praisonai', 'src/praisonai-agents', 'src/praisonai-code']; +/** Every directory whose changes should make a release eligible — all nine + * published packages (mirrors the skip_* inputs in pypi-release.yml). */ +const PACKAGE_PATHS = [ + 'src/praisonai', + 'src/praisonai-agents', + 'src/praisonai-code', + 'src/praisonai-bot', + 'src/praisonai-train', + 'src/praisonai-browser', + 'src/praisonai-mcp', + 'src/praisonai-sandbox', + 'src/praisonai-deploy', +]; /** Minimum days between successful patch auto-releases. */ const PATCH_RELEASE_INTERVAL_DAYS = 3; const ACTIVE_RELEASE_STATUSES = new Set([ 'queued', 'in_progress', 'waiting', 'pending', 'requested', ]); +/** A run stuck in `waiting` (environment approval) longer than this no longer + * blocks the gate. GitHub only auto-fails unapproved deployments after 30 + * days, so without a cutoff one forgotten manual dispatch stalls all + * auto-releases for up to a month. The run is warned about, never cancelled — + * if later approved, the pypi-release concurrency group serializes it and its + * pypi_exists checks no-op anything already published. */ +const STALE_WAITING_HOURS = 6; function bumpPatch(version) { const parts = version.split('.'); @@ -65,16 +84,27 @@ function pypiVersionExists(packageName, version) { }); } -async function hasActiveReleaseRun(github, owner, repo) { +async function hasActiveReleaseRun(github, owner, repo, now = new Date(), core = null) { const runs = await github.rest.actions.listWorkflowRuns({ owner, repo, workflow_id: 'pypi-release.yml', per_page: 20, }); - return runs.data.workflow_runs.some( - (r) => ACTIVE_RELEASE_STATUSES.has(r.status) && !r.conclusion - ); + const staleCutoff = now.getTime() - STALE_WAITING_HOURS * 60 * 60 * 1000; + return runs.data.workflow_runs.some((r) => { + if (!ACTIVE_RELEASE_STATUSES.has(r.status) || r.conclusion) return false; + if (r.status === 'waiting' && new Date(r.created_at).getTime() < staleCutoff) { + if (core) { + core.warning( + `Ignoring release run waiting >${STALE_WAITING_HOURS}h for environment approval: ` + + `${r.html_url} — approve or cancel it. It no longer blocks auto-releases.` + ); + } + return false; + } + return true; + }); } function utcDayStart(now = new Date()) { @@ -93,15 +123,18 @@ async function hasSuccessfulReleaseWithinDays( intervalDays = PATCH_RELEASE_INTERVAL_DAYS, ) { const windowStart = releaseIntervalStart(now, intervalDays); - const runs = await github.rest.actions.listWorkflowRuns({ + // Dedupe on GitHub releases (v* tags) rather than run conclusions: a + // dry_run=true dispatch concludes 'success' without publishing anything and + // must not consume the release window. Only a real full release creates a + // v* GitHub release (bump_and_release.py's `gh release create`). + const releases = await github.rest.repos.listReleases({ owner, repo, - workflow_id: 'pypi-release.yml', - status: 'completed', - per_page: 30, + per_page: 10, }); - return runs.data.workflow_runs.some( - (r) => r.conclusion === 'success' && new Date(r.created_at) >= windowStart + return releases.data.some( + (r) => r.tag_name && r.tag_name.startsWith('v') + && new Date(r.published_at || r.created_at) >= windowStart ); } @@ -146,12 +179,13 @@ async function evaluateReleasePreflight(github, owner, repo, options, core) { return out; } - if (await hasActiveReleaseRun(github, owner, repo)) { + const referenceTime = options.now instanceof Date ? options.now : new Date(); + + if (await hasActiveReleaseRun(github, owner, repo, referenceTime, core)) { reasons.push('PyPI Release already in progress or awaiting approval'); return out; } - const referenceTime = options.now instanceof Date ? options.now : new Date(); if (await hasSuccessfulReleaseWithinDays(github, owner, repo, referenceTime)) { reasons.push( `already released within last ${PATCH_RELEASE_INTERVAL_DAYS} days; ` @@ -226,8 +260,35 @@ async function evaluateReleasePreflight(github, owner, repo, options, core) { out.headSha = evalSha; const greenSha = await lastGreenCoreTestsSha(github, owner, repo); if (greenSha !== evalSha) { - reasons.push(`CI not green on HEAD (last green: ${greenSha ? greenSha.slice(0, 7) : 'none'})`); - return out; + // Core Tests has path filters and honors [skip ci] (the release + // safety-net commit uses it), so the tip of main may legitimately have + // no Core Tests run of its own — e.g. a docs-only commit, or the + // "chore(release): … [skip ci]" commit right after a release. Accept a + // green ancestor when nothing under the released package paths changed + // after it; otherwise the cron path stalls until the next src/** push. + let greenAncestorOk = false; + if (greenSha && /^[0-9a-f]{40}$/.test(greenSha)) { + try { + execSync(`git merge-base --is-ancestor ${greenSha} ${evalSha}`); + const delta = execSync( + `git diff --name-only ${greenSha} ${evalSha} -- ${PACKAGE_PATHS.join(' ')}`, + { encoding: 'utf8' } + ).trim(); + greenAncestorOk = delta === ''; + } catch { + greenAncestorOk = false; + } + } + if (!greenAncestorOk) { + reasons.push(`CI not green on HEAD (last green: ${greenSha ? greenSha.slice(0, 7) : 'none'})`); + return out; + } + if (core) { + core.info( + `HEAD ${evalSha.slice(0, 7)} has no Core Tests run; accepting green ancestor ` + + `${greenSha.slice(0, 7)} (no package-path changes since).` + ); + } } } @@ -242,7 +303,9 @@ module.exports = { readVersionsFromTree, pypiVersionExists, PATCH_RELEASE_INTERVAL_DAYS, + STALE_WAITING_HOURS, releaseIntervalStart, + hasActiveReleaseRun, hasSuccessfulReleaseWithinDays, hasSuccessfulReleaseToday, utcDayStart, diff --git a/.github/workflows/nightly-release-gate.yml b/.github/workflows/nightly-release-gate.yml index f7e1ed65a7..aed4956fd3 100644 --- a/.github/workflows/nightly-release-gate.yml +++ b/.github/workflows/nightly-release-gate.yml @@ -41,7 +41,7 @@ jobs: github.event.workflow_run.head_branch == 'main' && github.event.workflow_run.event == 'push' && !startsWith(github.event.workflow_run.head_commit.message, 'Release v') && - !startsWith(github.event.workflow_run.head_commit.message, 'Bump praisonai') + !startsWith(github.event.workflow_run.head_commit.message, 'chore(release):') ) runs-on: ubuntu-latest env: @@ -69,6 +69,10 @@ jobs: - name: Fetch main and tags run: git fetch --tags origin main + # Guards the gate logic itself — no workflow ran these selftests before. + - name: Selftest release-gate script + run: node .github/scripts/release-gate-selftest.js + - name: Preflight — path changes, CI SHA, dedupe id: preflight uses: actions/github-script@v7 diff --git a/.github/workflows/pypi-release.yml b/.github/workflows/pypi-release.yml index 12fcea3be8..6c7b53291e 100644 --- a/.github/workflows/pypi-release.yml +++ b/.github/workflows/pypi-release.yml @@ -66,6 +66,24 @@ on: required: false default: false type: boolean + # NOTE: GitHub caps workflow_dispatch at 25 top-level inputs; this + # workflow has 23. Adding a 10th package (skip_* + *_version) hits 25. + only: + description: 'Release ONLY this package (forces skip on all others); all = no restriction' + required: false + default: all + type: choice + options: + - all + - agents + - code + - bot + - train + - browser + - mcp + - sandbox + - deploy + - wrapper agents_version: description: 'Override agents version (empty = auto bump from pyproject.toml)' required: false @@ -205,6 +223,36 @@ jobs: echo "Dry run: ${{ inputs.dry_run }}" echo "Environment: ${{ (inputs.source == 'ci' || inputs.source == 'nightly') && inputs.bump == 'patch' && 'pypi-auto' || 'pypi' }}" + # Single source of truth for which packages this run releases. `only` + # inverts into the skip flags here; everything downstream (version + # computation, PyPI detection, publish conditions) consumes these + # outputs, so a dedicated one-package release is one input instead of + # eight skip flags. + - name: Resolve effective skips + id: flags + env: + ONLY: ${{ inputs.only }} + SKIP_AGENTS: ${{ inputs.skip_agents }} + SKIP_CODE: ${{ inputs.skip_code }} + SKIP_BOT: ${{ inputs.skip_bot }} + SKIP_TRAIN: ${{ inputs.skip_train }} + SKIP_BROWSER: ${{ inputs.skip_browser }} + SKIP_MCP: ${{ inputs.skip_mcp }} + SKIP_SANDBOX: ${{ inputs.skip_sandbox }} + SKIP_DEPLOY: ${{ inputs.skip_deploy }} + SKIP_WRAPPER: ${{ inputs.skip_wrapper }} + run: | + set -euo pipefail + for pkg in agents code bot train browser mcp sandbox deploy wrapper; do + var="SKIP_$(echo "$pkg" | tr '[:lower:]' '[:upper:]')" + skip="${!var}" + if [ -n "${ONLY}" ] && [ "${ONLY}" != "all" ] && [ "${ONLY}" != "$pkg" ]; then + skip=true + fi + echo "skip_${pkg}=${skip}" >> "$GITHUB_OUTPUT" + echo "skip_${pkg}=${skip}" + done + - name: Compute release versions id: versions env: @@ -218,14 +266,14 @@ jobs: MCP_OVERRIDE: ${{ inputs.mcp_version }} SANDBOX_OVERRIDE: ${{ inputs.sandbox_version }} DEPLOY_OVERRIDE: ${{ inputs.deploy_version }} - SKIP_AGENTS: ${{ inputs.skip_agents }} - SKIP_CODE: ${{ inputs.skip_code }} - SKIP_BOT: ${{ inputs.skip_bot }} - SKIP_TRAIN: ${{ inputs.skip_train }} - SKIP_BROWSER: ${{ inputs.skip_browser }} - SKIP_MCP: ${{ inputs.skip_mcp }} - SKIP_SANDBOX: ${{ inputs.skip_sandbox }} - SKIP_DEPLOY: ${{ inputs.skip_deploy }} + SKIP_AGENTS: ${{ steps.flags.outputs.skip_agents }} + SKIP_CODE: ${{ steps.flags.outputs.skip_code }} + SKIP_BOT: ${{ steps.flags.outputs.skip_bot }} + SKIP_TRAIN: ${{ steps.flags.outputs.skip_train }} + SKIP_BROWSER: ${{ steps.flags.outputs.skip_browser }} + SKIP_MCP: ${{ steps.flags.outputs.skip_mcp }} + SKIP_SANDBOX: ${{ steps.flags.outputs.skip_sandbox }} + SKIP_DEPLOY: ${{ steps.flags.outputs.skip_deploy }} run: | python <<'PY' import os @@ -475,15 +523,15 @@ jobs: SANDBOX_VERSION: ${{ steps.versions.outputs.sandbox_version }} DEPLOY_VERSION: ${{ steps.versions.outputs.deploy_version }} WRAPPER_VERSION: ${{ steps.versions.outputs.wrapper_version }} - INPUT_SKIP_AGENTS: ${{ inputs.skip_agents }} - INPUT_SKIP_CODE: ${{ inputs.skip_code }} - INPUT_SKIP_BOT: ${{ inputs.skip_bot }} - INPUT_SKIP_TRAIN: ${{ inputs.skip_train }} - INPUT_SKIP_BROWSER: ${{ inputs.skip_browser }} - INPUT_SKIP_MCP: ${{ inputs.skip_mcp }} - INPUT_SKIP_SANDBOX: ${{ inputs.skip_sandbox }} - INPUT_SKIP_DEPLOY: ${{ inputs.skip_deploy }} - INPUT_SKIP_WRAPPER: ${{ inputs.skip_wrapper }} + INPUT_SKIP_AGENTS: ${{ steps.flags.outputs.skip_agents }} + INPUT_SKIP_CODE: ${{ steps.flags.outputs.skip_code }} + INPUT_SKIP_BOT: ${{ steps.flags.outputs.skip_bot }} + INPUT_SKIP_TRAIN: ${{ steps.flags.outputs.skip_train }} + INPUT_SKIP_BROWSER: ${{ steps.flags.outputs.skip_browser }} + INPUT_SKIP_MCP: ${{ steps.flags.outputs.skip_mcp }} + INPUT_SKIP_SANDBOX: ${{ steps.flags.outputs.skip_sandbox }} + INPUT_SKIP_DEPLOY: ${{ steps.flags.outputs.skip_deploy }} + INPUT_SKIP_WRAPPER: ${{ steps.flags.outputs.skip_wrapper }} run: | set -euo pipefail skip_agents="${INPUT_SKIP_AGENTS}" @@ -495,32 +543,51 @@ jobs: skip_sandbox="${INPUT_SKIP_SANDBOX}" skip_deploy="${INPUT_SKIP_DEPLOY}" skip_wrapper="${INPUT_SKIP_WRAPPER}" + # Returns 0 if published, 1 if definitively absent (404). Any other + # HTTP status (PyPI outage, 5xx, rate limit) FAILS the run: guessing + # "not published" would lead to a duplicate-upload failure midway + # through the release, which is strictly worse than stopping here + # before anything has been published. + pypi_published() { # $1=package $2=version + local code attempt + for attempt in 1 2 3; do + code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 30 \ + "https://pypi.org/pypi/$1/$2/json" || echo "000") + case "$code" in + 200) return 0 ;; + 404) return 1 ;; + *) echo "PyPI check $1==$2 returned HTTP ${code} (attempt ${attempt}/3)"; sleep 5 ;; + esac + done + echo "❌ Cannot determine PyPI state for $1==$2; refusing to guess." >&2 + exit 2 + } if [ "$skip_agents" != "true" ]; then - curl -fsSL "https://pypi.org/pypi/praisonaiagents/${AGENTS_VERSION}/json" >/dev/null && skip_agents=true || true + pypi_published praisonaiagents "${AGENTS_VERSION}" && skip_agents=true || true fi if [ "$skip_code" != "true" ]; then - curl -fsSL "https://pypi.org/pypi/praisonai-code/${CODE_VERSION}/json" >/dev/null && skip_code=true || true + pypi_published praisonai-code "${CODE_VERSION}" && skip_code=true || true fi if [ "$skip_bot" != "true" ]; then - curl -fsSL "https://pypi.org/pypi/praisonai-bot/${BOT_VERSION}/json" >/dev/null && skip_bot=true || true + pypi_published praisonai-bot "${BOT_VERSION}" && skip_bot=true || true fi if [ "$skip_train" != "true" ]; then - curl -fsSL "https://pypi.org/pypi/praisonai-train/${TRAIN_VERSION}/json" >/dev/null && skip_train=true || true + pypi_published praisonai-train "${TRAIN_VERSION}" && skip_train=true || true fi if [ "$skip_browser" != "true" ]; then - curl -fsSL "https://pypi.org/pypi/praisonai-browser/${BROWSER_VERSION}/json" >/dev/null && skip_browser=true || true + pypi_published praisonai-browser "${BROWSER_VERSION}" && skip_browser=true || true fi if [ "$skip_mcp" != "true" ]; then - curl -fsSL "https://pypi.org/pypi/praisonai-mcp/${MCP_VERSION}/json" >/dev/null && skip_mcp=true || true + pypi_published praisonai-mcp "${MCP_VERSION}" && skip_mcp=true || true fi if [ "$skip_sandbox" != "true" ]; then - curl -fsSL "https://pypi.org/pypi/praisonai-sandbox/${SANDBOX_VERSION}/json" >/dev/null && skip_sandbox=true || true + pypi_published praisonai-sandbox "${SANDBOX_VERSION}" && skip_sandbox=true || true fi if [ "$skip_deploy" != "true" ]; then - curl -fsSL "https://pypi.org/pypi/praisonai-deploy/${DEPLOY_VERSION}/json" >/dev/null && skip_deploy=true || true + pypi_published praisonai-deploy "${DEPLOY_VERSION}" && skip_deploy=true || true fi if [ "$skip_wrapper" != "true" ]; then - curl -fsSL "https://pypi.org/pypi/praisonai/${WRAPPER_VERSION}/json" >/dev/null && skip_wrapper=true || true + pypi_published praisonai "${WRAPPER_VERSION}" && skip_wrapper=true || true fi echo "skip_agents=${skip_agents}" >> "$GITHUB_OUTPUT" echo "skip_code=${skip_code}" >> "$GITHUB_OUTPUT" @@ -553,34 +620,43 @@ jobs: if: inputs.dry_run run: | echo "Dry run — no publish, commit, or push." + echo "Only: ${{ inputs.only }}" echo "Agents version: ${{ steps.versions.outputs.agents_version }}" echo "Code version: ${{ steps.versions.outputs.code_version }}" echo "Bot version: ${{ steps.versions.outputs.bot_version }}" echo "Train version: ${{ steps.versions.outputs.train_version }}" echo "Browser version: ${{ steps.versions.outputs.browser_version }}" echo "MCP version: ${{ steps.versions.outputs.mcp_version }}" + echo "Sandbox version: ${{ steps.versions.outputs.sandbox_version }}" + echo "Deploy version: ${{ steps.versions.outputs.deploy_version }}" echo "Wrapper version: ${{ steps.versions.outputs.wrapper_version }}" echo "" - if [ "${{ inputs.skip_agents }}" != "true" ]; then + if [ "${{ steps.flags.outputs.skip_agents }}" != "true" ]; then echo "Would publish praisonaiagents ${{ steps.versions.outputs.agents_version }}" fi - if [ "${{ inputs.skip_code }}" != "true" ]; then + if [ "${{ steps.flags.outputs.skip_code }}" != "true" ]; then echo "Would publish praisonai-code ${{ steps.versions.outputs.code_version }}" fi - if [ "${{ inputs.skip_bot }}" != "true" ]; then + if [ "${{ steps.flags.outputs.skip_bot }}" != "true" ]; then echo "Would publish praisonai-bot ${{ steps.versions.outputs.bot_version }}" fi - if [ "${{ inputs.skip_train }}" != "true" ]; then + if [ "${{ steps.flags.outputs.skip_train }}" != "true" ]; then echo "Would publish praisonai-train ${{ steps.versions.outputs.train_version }}" fi - if [ "${{ inputs.skip_browser }}" != "true" ]; then + if [ "${{ steps.flags.outputs.skip_browser }}" != "true" ]; then echo "Would publish praisonai-browser ${{ steps.versions.outputs.browser_version }}" fi - if [ "${{ inputs.skip_mcp }}" != "true" ]; then + if [ "${{ steps.flags.outputs.skip_mcp }}" != "true" ]; then echo "Would publish praisonai-mcp ${{ steps.versions.outputs.mcp_version }}" fi - if [ "${{ inputs.skip_wrapper }}" != "true" ]; then - echo "Would run bump_and_release.py ${{ steps.versions.outputs.wrapper_version }} --agents ${{ steps.versions.outputs.agents_version }} --code-pin ${{ steps.versions.outputs.code_version }} --bot-pin ${{ steps.versions.outputs.bot_version }} --train-pin ${{ steps.versions.outputs.train_version }} --browser-pin ${{ steps.versions.outputs.browser_version }} --mcp-pin ${{ steps.versions.outputs.mcp_version }}" + if [ "${{ steps.flags.outputs.skip_sandbox }}" != "true" ]; then + echo "Would publish praisonai-sandbox ${{ steps.versions.outputs.sandbox_version }}" + fi + if [ "${{ steps.flags.outputs.skip_deploy }}" != "true" ]; then + echo "Would publish praisonai-deploy ${{ steps.versions.outputs.deploy_version }}" + fi + if [ "${{ steps.flags.outputs.skip_wrapper }}" != "true" ]; then + echo "Would run bump_and_release.py ${{ steps.versions.outputs.wrapper_version }} --agents ${{ steps.versions.outputs.agents_version }} --code-pin ${{ steps.versions.outputs.code_version }} --bot-pin ${{ steps.versions.outputs.bot_version }} --train-pin ${{ steps.versions.outputs.train_version }} --browser-pin ${{ steps.versions.outputs.browser_version }} --mcp-pin ${{ steps.versions.outputs.mcp_version }} --sandbox-pin ${{ steps.versions.outputs.sandbox_version }} --deploy-pin ${{ steps.versions.outputs.deploy_version }}" echo "Would uv publish praisonai ${{ steps.versions.outputs.wrapper_version }}" fi @@ -608,7 +684,9 @@ jobs: rm -rf dist uv lock uv build - uv publish + # --check-url makes re-uploads idempotent: files already on PyPI are + # skipped instead of failing the publish (partial-upload recovery). + uv publish --check-url https://pypi.org/simple/ - name: Wait for praisonaiagents on PyPI if: >- @@ -665,7 +743,9 @@ jobs: rm -rf dist uv lock uv build - uv publish + # --check-url makes re-uploads idempotent: files already on PyPI are + # skipped instead of failing the publish (partial-upload recovery). + uv publish --check-url https://pypi.org/simple/ - name: Wait for praisonai-code on PyPI if: >- @@ -722,7 +802,9 @@ jobs: rm -rf dist uv lock uv build - uv publish + # --check-url makes re-uploads idempotent: files already on PyPI are + # skipped instead of failing the publish (partial-upload recovery). + uv publish --check-url https://pypi.org/simple/ - name: Wait for praisonai-bot on PyPI if: >- @@ -779,7 +861,9 @@ jobs: rm -rf dist uv lock uv build - uv publish + # --check-url makes re-uploads idempotent: files already on PyPI are + # skipped instead of failing the publish (partial-upload recovery). + uv publish --check-url https://pypi.org/simple/ - name: Wait for praisonai-train on PyPI if: >- @@ -836,7 +920,9 @@ jobs: rm -rf dist uv lock uv build - uv publish + # --check-url makes re-uploads idempotent: files already on PyPI are + # skipped instead of failing the publish (partial-upload recovery). + uv publish --check-url https://pypi.org/simple/ - name: Wait for praisonai-browser on PyPI if: >- @@ -893,7 +979,9 @@ jobs: rm -rf dist uv lock uv build - uv publish + # --check-url makes re-uploads idempotent: files already on PyPI are + # skipped instead of failing the publish (partial-upload recovery). + uv publish --check-url https://pypi.org/simple/ - name: Wait for praisonai-mcp on PyPI if: >- @@ -950,7 +1038,9 @@ jobs: rm -rf dist uv lock uv build - uv publish + # --check-url makes re-uploads idempotent: files already on PyPI are + # skipped instead of failing the publish (partial-upload recovery). + uv publish --check-url https://pypi.org/simple/ - name: Wait for praisonai-sandbox on PyPI if: >- @@ -1007,7 +1097,9 @@ jobs: rm -rf dist uv lock uv build - uv publish + # --check-url makes re-uploads idempotent: files already on PyPI are + # skipped instead of failing the publish (partial-upload recovery). + uv publish --check-url https://pypi.org/simple/ - name: Wait for praisonai-deploy on PyPI if: >- @@ -1046,6 +1138,9 @@ jobs: - name: Refresh GitHub App token id: app-token-final if: ${{ !cancelled() && inputs.dry_run != true }} + # A failed re-mint must not fail the release: the original token may + # still be valid, and the next step only swaps tokens on success. + continue-on-error: true uses: actions/create-github-app-token@v1 with: app-id: ${{ secrets.CLAUDE_APP_ID }} @@ -1053,8 +1148,12 @@ jobs: owner: ${{ github.repository_owner }} repositories: ${{ github.event.repository.name }} + # Guarded on the refresh actually succeeding: if the re-mint failed + # (API blip), the ORIGINAL token — still wired into the remote URL and + # GH_TOKEN — may well be valid, and overwriting it with an empty string + # would turn a recoverable situation into a guaranteed auth failure. - name: Use refreshed GitHub App token - if: ${{ !cancelled() && inputs.dry_run != true }} + if: ${{ !cancelled() && inputs.dry_run != true && steps.app-token-final.outcome == 'success' }} env: APP_TOKEN: ${{ steps.app-token-final.outputs.token }} REPO: ${{ github.repository }} @@ -1064,6 +1163,7 @@ jobs: echo "GH_TOKEN=${APP_TOKEN}" >> "$GITHUB_ENV" - name: Bump and release wrapper + id: wrapper if: >- inputs.dry_run != true && inputs.skip_wrapper != true && @@ -1087,7 +1187,7 @@ jobs: inputs.skip_wrapper != true && steps.pypi_exists.outputs.skip_wrapper != 'true' working-directory: src/praisonai - run: uv publish + run: uv publish --check-url https://pypi.org/simple/ - name: Wait for praisonai on PyPI if: >- @@ -1116,55 +1216,33 @@ jobs: sleep "$INTERVAL" done + # Actually probes PyPI — previously this step only echoed checkmarks + # from the input flags, so it "verified" publishes that never happened. - name: Verify PyPI versions if: inputs.dry_run != true run: | set -euo pipefail - AGENTS="${{ steps.versions.outputs.agents_version }}" - CODE="${{ steps.versions.outputs.code_version }}" - BOT="${{ steps.versions.outputs.bot_version }}" - TRAIN="${{ steps.versions.outputs.train_version }}" - BROWSER="${{ steps.versions.outputs.browser_version }}" - MCP="${{ steps.versions.outputs.mcp_version }}" - SANDBOX="${{ steps.versions.outputs.sandbox_version }}" - DEPLOY="${{ steps.versions.outputs.deploy_version }}" - WRAPPER="${{ steps.versions.outputs.wrapper_version }}" - - if [ "${{ inputs.skip_agents }}" != "true" ]; then - echo "✅ praisonaiagents==${AGENTS}" - fi - - if [ "${{ inputs.skip_code }}" != "true" ]; then - echo "✅ praisonai-code==${CODE}" - fi - - if [ "${{ inputs.skip_bot }}" != "true" ]; then - echo "✅ praisonai-bot==${BOT}" - fi - - if [ "${{ inputs.skip_train }}" != "true" ]; then - echo "✅ praisonai-train==${TRAIN}" - fi - - if [ "${{ inputs.skip_browser }}" != "true" ]; then - echo "✅ praisonai-browser==${BROWSER}" - fi - - if [ "${{ inputs.skip_mcp }}" != "true" ]; then - echo "✅ praisonai-mcp==${MCP}" - fi - - if [ "${{ inputs.skip_sandbox }}" != "true" ]; then - echo "✅ praisonai-sandbox==${SANDBOX}" - fi - - if [ "${{ inputs.skip_deploy }}" != "true" ]; then - echo "✅ praisonai-deploy==${DEPLOY}" - fi - - if [ "${{ inputs.skip_wrapper }}" != "true" ]; then - echo "✅ praisonai==${WRAPPER}" - fi + verify() { # $1=pypi-name $2=version $3=effective-skip + if [ "$3" = "true" ]; then + echo "⏭️ $1 skipped" + return 0 + fi + if curl -fsSL --max-time 30 "https://pypi.org/pypi/$1/$2/json" >/dev/null; then + echo "✅ $1==$2" + else + echo "❌ $1==$2 NOT found on PyPI" + exit 1 + fi + } + verify praisonaiagents "${{ steps.versions.outputs.agents_version }}" "${{ steps.flags.outputs.skip_agents }}" + verify praisonai-code "${{ steps.versions.outputs.code_version }}" "${{ steps.flags.outputs.skip_code }}" + verify praisonai-bot "${{ steps.versions.outputs.bot_version }}" "${{ steps.flags.outputs.skip_bot }}" + verify praisonai-train "${{ steps.versions.outputs.train_version }}" "${{ steps.flags.outputs.skip_train }}" + verify praisonai-browser "${{ steps.versions.outputs.browser_version }}" "${{ steps.flags.outputs.skip_browser }}" + verify praisonai-mcp "${{ steps.versions.outputs.mcp_version }}" "${{ steps.flags.outputs.skip_mcp }}" + verify praisonai-sandbox "${{ steps.versions.outputs.sandbox_version }}" "${{ steps.flags.outputs.skip_sandbox }}" + verify praisonai-deploy "${{ steps.versions.outputs.deploy_version }}" "${{ steps.flags.outputs.skip_deploy }}" + verify praisonai "${{ steps.versions.outputs.wrapper_version }}" "${{ steps.flags.outputs.skip_wrapper }}" # Safety net. On a clean run the wrapper's "Release v…" commit already # carries every bump and this is a no-op. It only fires when the single @@ -1175,6 +1253,22 @@ jobs: if: ${{ !cancelled() && inputs.dry_run != true }} run: | set -euo pipefail + # If the wrapper release did not complete, drop any uncommitted + # wrapper-side rewrites (version.py, Dockerfiles, pins): committing + # them would put a wrapper version on main that never reached PyPI, + # and the next auto-bump would skip past it forever. Committed + # changes are untouched (checkout only reverts the working tree), + # and if the wrapper was skipped these files were never modified. + if [ "${{ steps.wrapper.outcome }}" != "success" ]; then + git checkout -- \ + src/praisonai/praisonai/version.py \ + src/praisonai/pyproject.toml \ + src/praisonai/uv.lock \ + src/praisonai/README.md \ + src/praisonai/praisonai.rb \ + src/praisonai-deploy/praisonai_deploy/docker.py \ + docker/ 2>/dev/null || true + fi # -u = tracked modifications only; uv build output under dist/ is # untracked and stays out of the commit. git add -u @@ -1191,5 +1285,70 @@ jobs: fi echo "Commit message: ${MSG}" git commit -m "$MSG" - git pull --rebase origin main - git push origin main + # This step fires exactly when the release went sideways — i.e. when + # main is most likely to have moved — so the rebase+push retries. + # Losing this commit deadlocks future auto-releases: the packages + # are on PyPI at N+1 while the repo says N, so every later run + # detects "already published" and skips both publish and bump. + for attempt in 1 2 3; do + git fetch origin main + if ! git rebase origin/main; then + git rebase --abort || true + # During rebase, "theirs" is the commit being replayed — the + # version bumps — so they win any conflict. + if ! git rebase -X theirs origin/main; then + git rebase --abort || true + echo "Rebase attempt ${attempt}/3 failed" + sleep 5 + continue + fi + fi + if git push origin main; then + echo "Pushed version bumps (attempt ${attempt}/3)" + exit 0 + fi + echo "Push attempt ${attempt}/3 rejected; retrying..." + sleep 5 + done + echo "❌ Could not push version bumps after 3 attempts." + echo " Repo is now BEHIND PyPI; commit the version files on main manually" + echo " or future auto-releases will no-op ('already published')." + exit 1 + + # Dedicated releases (skip_wrapper) create no "Release v…" tag — the + # wrapper drives that — so tag each published package individually for + # provenance. Full releases keep their single v… tag unchanged. + - name: Tag dedicated package releases + if: >- + ${{ !cancelled() && inputs.dry_run != true && + steps.flags.outputs.skip_wrapper == 'true' && + steps.versions.outcome == 'success' }} + run: | + set -euo pipefail + tag_pkg() { # $1=effective-skip $2=pypi-name $3=version + if [ "$1" = "true" ]; then + return 0 + fi + local tag="$2-v$3" + # Only tag what is confirmed on PyPI — a package whose publish + # failed must not get a tag claiming it shipped. + if ! curl -fsSL --max-time 30 "https://pypi.org/pypi/$2/$3/json" >/dev/null; then + echo "⏭️ $2==$3 not on PyPI; not tagging" + return 0 + fi + if git rev-parse -q --verify "refs/tags/${tag}" >/dev/null; then + echo "tag ${tag} already exists" + return 0 + fi + git tag "${tag}" + git push origin "refs/tags/${tag}" + echo "🏷️ ${tag}" + } + tag_pkg "${{ steps.flags.outputs.skip_agents }}" praisonaiagents "${{ steps.versions.outputs.agents_version }}" + tag_pkg "${{ steps.flags.outputs.skip_code }}" praisonai-code "${{ steps.versions.outputs.code_version }}" + tag_pkg "${{ steps.flags.outputs.skip_bot }}" praisonai-bot "${{ steps.versions.outputs.bot_version }}" + tag_pkg "${{ steps.flags.outputs.skip_train }}" praisonai-train "${{ steps.versions.outputs.train_version }}" + tag_pkg "${{ steps.flags.outputs.skip_browser }}" praisonai-browser "${{ steps.versions.outputs.browser_version }}" + tag_pkg "${{ steps.flags.outputs.skip_mcp }}" praisonai-mcp "${{ steps.versions.outputs.mcp_version }}" + tag_pkg "${{ steps.flags.outputs.skip_sandbox }}" praisonai-sandbox "${{ steps.versions.outputs.sandbox_version }}" + tag_pkg "${{ steps.flags.outputs.skip_deploy }}" praisonai-deploy "${{ steps.versions.outputs.deploy_version }}" diff --git a/src/praisonai/scripts/bump_and_release.py b/src/praisonai/scripts/bump_and_release.py index c1e4fd5eec..cca141d784 100644 --- a/src/praisonai/scripts/bump_and_release.py +++ b/src/praisonai/scripts/bump_and_release.py @@ -624,34 +624,63 @@ def release(version: str, use_frozen_lock: bool = False, no_add_all: bool = Fals else: run(["git", "add", "-A"], cwd=root) - run(["git", "commit", "-m", f"Release {tag}"], cwd=root, check=False) - + # Distinguish "nothing to commit" (legitimate rerun) from a real commit + # failure (hook, index lock): the latter must abort, or the tag below + # would point at the previous HEAD and the GitHub release would ship a + # tree that doesn't match the PyPI artifact. + staged = run(["git", "diff", "--cached", "--quiet"], cwd=root, check=False, silent=True) + if staged.returncode == 0: + print(" ℹ️ Nothing to commit (release files unchanged — likely a rerun).") + else: + run(["git", "commit", "-m", f"Release {tag}"], cwd=root) + # 5. Create git tag print(f"\n🏷️ Creating tag {tag}...") run(["git", "tag", "-f", tag], cwd=root) - + # 6. Pull rebase and push to GitHub print("\n⬆️ Pushing to GitHub...") # First fetch and rebase to handle any remote changes (e.g., auto-generated api.md) result = run(["git", "pull", "--rebase", "origin", "main"], cwd=root, check=False) if result.returncode != 0: - print(" ⚠️ Rebase failed, trying to continue...") - + # A conflicted rebase leaves the repo mid-rebase on a detached HEAD; + # "continuing" from there tags the wrong commit and the push fails. + # Abort and retry favoring the release commit ("theirs" during a + # rebase is the commit being replayed), then abort hard if that + # still fails. + print(" ⚠️ Rebase conflicted; retrying with release changes taking precedence...") + run(["git", "rebase", "--abort"], cwd=root, check=False) + result = run( + ["git", "pull", "--rebase", "-X", "theirs", "origin", "main"], + cwd=root, check=False, + ) + if result.returncode != 0: + run(["git", "rebase", "--abort"], cwd=root, check=False) + print(" ❌ Rebase failed even preferring release changes; aborting before push.") + sys.exit(1) + # Recreate tag after rebase (commit hash may have changed) run(["git", "tag", "-f", tag], cwd=root) - - # Push changes + + # Push changes (only this release's tag — a blanket `--tags -f` would + # force-move historical tags and corrupt old releases' provenance) run(["git", "push"], cwd=root) - run(["git", "push", "--tags", "-f"], cwd=root) - - # 7. Create GitHub release - print(f"\n🎉 Creating GitHub release {tag}...") - run([ - "gh", "release", "create", tag, - "--title", f"PraisonAI {tag}", - "--notes", f"Release {tag}", - "--latest" - ], cwd=root) + run(["git", "push", "origin", "-f", f"refs/tags/{tag}"], cwd=root) + + # 7. Create GitHub release (idempotent: a rerun after a transient failure + # must not die on "release already exists" — that would permanently block + # the recovery path, since the wrapper publish runs after this step) + existing = run(["gh", "release", "view", tag], cwd=root, check=False, silent=True) + if existing.returncode == 0: + print(f"\nℹ️ GitHub release {tag} already exists; skipping create.") + else: + print(f"\n🎉 Creating GitHub release {tag}...") + run([ + "gh", "release", "create", tag, + "--title", f"PraisonAI {tag}", + "--notes", f"Release {tag}", + "--latest" + ], cwd=root) print(f"\n✅ Released PraisonAI {tag}") print("\nNext step:")