diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 000000000..5f29f5167 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,4 @@ +# The plugin manifests are merged with the driver "fork-manifest", which the upstream sync job +# configures: the result is upstream's file without the version fields (.github/upstream-sync/README.md). +plugins/codex/.claude-plugin/plugin.json merge=fork-manifest +.claude-plugin/marketplace.json merge=fork-manifest diff --git a/.github/upstream-sync/README.md b/.github/upstream-sync/README.md new file mode 100644 index 000000000..5ee2b52b9 --- /dev/null +++ b/.github/upstream-sync/README.md @@ -0,0 +1,72 @@ +# Upstream sync + +This fork takes the changes of its upstream, [Crazytieguy/codex-plugin-cc](https://github.com/Crazytieguy/codex-plugin-cc), automatically, but only once they are at least 7 days old. The delay is a defense against supply-chain attacks: every machine that installs the plugin updates itself from this fork's `main`, so a malicious commit pushed to upstream should have a week to be noticed and removed before it gets here. The rule is the one package managers use for a minimum release age, such as npm's `min-release-age` or uv's `exclude-newer`: take the newest version that upstream has had for at least 7 days. + +## What the job does + +The workflow `.github/workflows/upstream-sync.yml` runs `sync.sh` once a day. Each run: + +1. Fetches upstream's `main`, this fork's `main`, and the branch `upstream-sync-state`. That branch holds `seen.tsv`, a log of upstream's head with one row per change (the full commit hash and the time the job saw it, in UTC), and `last_checked`, the time of the last run. The first run creates the branch. +2. Adds a row to `seen.tsv` when upstream's head differs from the last row's commit, even if that commit is in an earlier row. +3. Picks the newest row that is at least 7 days old and whose commit has stayed in upstream's history since: it is an ancestor of every later row's commit, the last of which is upstream's current head. Commit dates play no part, since anyone can backdate them. +4. Merges nothing if GitHub's activity log of upstream's `main` shows a force-push or a deletion of `main` after that row's time. +5. Merges nothing, and fails, if any upstream commit to merge touches `.github/` or `.gitattributes`, file modes and symbolic links included. The job's scripts, its workflow and its merge driver live there, so upstream can never change them. +6. Otherwise merges the commit into `main`, with a merge commit that names it and the time of its row. +7. Pushes `upstream-sync-state`, on every run, then `main` if it merged something. + +## How force-pushes are handled + +The job looks at upstream once a day, so `seen.tsv` shows upstream's head only at those times. A commit that a force-push removes from upstream is never merged, and a commit that comes back counts only from its new row. A removal and restore between two runs leaves no trace in `seen.tsv`; GitHub's activity log shows it, and step 4 then holds back every row logged before it. When the head after such a force-push is the same commit as the last row, the job logs it again, so it counts from that run. + +So after any force-push of upstream's `main`, nothing is merged until a head logged after it has been on upstream for 7 days. If an upstream commit waits more than 30 days, the job fails and says why. + +What the rule does not cover: a bad commit that upstream reverts instead of removing still reaches this fork's `main` 7 days after it appeared on upstream, and the revert follows 7 days after it appeared. Packages behave the same way: a bad version that is replaced by a newer one, but not withdrawn, is still installed once it is old enough. + +The job uses `git`, `jq` and `curl` (for GitHub's activity log), and no third-party actions. It never runs upstream code: step 5 keeps upstream from changing what the job runs. + +## Plugin versions + +Claude Code updates a plugin only when its version changes. Without a `version` field it uses the git commit, so each new commit of this fork counts as an update. The fork therefore keeps `version` deleted in `plugins/codex/.claude-plugin/plugin.json` and in the plugin entries of `.claude-plugin/marketplace.json`, while upstream changes the version in most commits. + +`.gitattributes` gives both files the merge driver `fork-manifest` (`fork-manifest.sh`), which `sync.sh` configures: where both sides changed the file, the result is upstream's file without the version fields. After each merge, `sync.sh` also deletes version fields that only upstream changed. Any other change this fork makes to these two files is replaced by upstream's next change to them. + +## When the job stops running + +GitHub turns off scheduled workflows in a public repository after 60 days without activity. The daily push to `upstream-sync-state` may count as activity; GitHub does not say. If GitHub turns the schedule off, it sends an email, the fork stays on the last upstream commit it merged, and this turns the schedule back on: + +```sh +gh workflow enable upstream-sync.yml -R TruthfulAI-research/codex-plugin-cc +``` + +## When the job fails + +GitHub emails the person who last changed the workflow's schedule. The run's log says why the run failed and names the upstream commit. Only the 30-day alert can come from a run that also merged something; every other failed run leaves `main` as it was. + +- **Upstream changed `.github/` or `.gitattributes`.** Read the change first: merged workflows run in this repository, and this fork's sync files should stay as they are unless you mean to change them. Then merge by hand, as below, from a machine whose token may push workflow files (Johannes's laptop). +- **The merge conflicts.** Merge by hand, as below. +- **Upstream removed commits that this fork merged.** Upstream may have removed them because they were bad. Check them, and revert them in `main` if needed. The job fails on every run until their rows are deleted from `seen.tsv` on the branch `upstream-sync-state`; delete them once the commits are checked. +- **An upstream commit has waited 30 days.** The log says what held it back, usually force-pushes on upstream. Check upstream; the job merges again once a head has been on upstream for 7 days without a force-push. +- **GitHub's activity log could not be read.** Nothing was merged; the next run tries again. + +To merge by hand, merge the commit that the log names, not upstream's newer head: it is the one that has been on upstream for 7 days. + +```sh +git clone https://github.com/TruthfulAI-research/codex-plugin-cc.git && cd codex-plugin-cc +git config merge.fork-manifest.driver "bash '$PWD/.github/upstream-sync/fork-manifest.sh' %B %A %P" +git fetch https://github.com/Crazytieguy/codex-plugin-cc.git main +git merge --no-ff --no-commit +# Resolve any conflicts, then delete the version fields that the merge driver did not: +for f in plugins/codex/.claude-plugin/plugin.json .claude-plugin/marketplace.json; do + bash .github/upstream-sync/fork-manifest.sh "$f" "$f.new" "$f" && mv "$f.new" "$f" && git add "$f" +done +git commit +git push origin main +``` + +The next run sees that the commit is in `main` and goes on from there. + +## Dry runs and tests + +- A pull request that changes the sync files runs `test.sh` and a dry run, which merges locally and pushes nothing. The dry run merges upstream's current candidate into the current `main`, so it can fail because of an upstream conflict that has nothing to do with the pull request. +- A dry run by hand: `gh workflow run upstream-sync.yml -R TruthfulAI-research/codex-plugin-cc -f dry_run=true`, or `SYNC_DRY_RUN=1 bash .github/upstream-sync/sync.sh` locally. +- `bash .github/upstream-sync/test.sh` tests `sync.sh` on scratch repos (a fake upstream and a fake fork, with the clock set by `SYNC_NOW`). It needs `git` and `jq`. diff --git a/.github/upstream-sync/fork-manifest.sh b/.github/upstream-sync/fork-manifest.sh new file mode 100755 index 000000000..b5c08b4e8 --- /dev/null +++ b/.github/upstream-sync/fork-manifest.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +# Writes a plugin manifest without its version fields: usage fork-manifest.sh . +# Git runs it as the merge driver "fork-manifest" (see .gitattributes) with upstream's file as the source, +# so the merge result is upstream's file without the version fields. sync.sh also runs it after each merge. +# Without a version, Claude Code updates the plugin on every new commit of the fork. +set -euo pipefail + +src=$1 dest=$2 path=$3 +if [ "$src" -ef "$dest" ]; then + echo "fork-manifest.sh: source and destination must be different files" >&2 + exit 1 +fi + +case $path in + plugins/codex/.claude-plugin/plugin.json) + has_version='has("version")' + strip='del(.version)' + ;; + .claude-plugin/marketplace.json) + has_version='any(.plugins[]?; has("version"))' + strip='del(.plugins[]?.version)' + ;; + *) + echo "fork-manifest.sh: not a plugin manifest: $path" >&2 + exit 1 + ;; +esac + +# Exits here on invalid JSON, which makes git report a conflict. +has=$(jq "$has_version" "$src") +if [ "$has" = true ]; then + jq "$strip" "$src" > "$dest" +else + # Keep the file byte for byte when there is nothing to delete. + cat "$src" > "$dest" +fi diff --git a/.github/upstream-sync/sync.sh b/.github/upstream-sync/sync.sh new file mode 100755 index 000000000..896536b22 --- /dev/null +++ b/.github/upstream-sync/sync.sh @@ -0,0 +1,285 @@ +#!/usr/bin/env bash +# Merges upstream's main into this fork's main, but only commits that upstream's main has had for at +# least 7 days, like a package manager's minimum release age. See README.md in this folder. +# +# Environment (all optional): +# SYNC_DRY_RUN=1 do everything locally, push nothing, print what would happen +# SYNC_TOKEN token for pushing to the fork over HTTPS (the job's GITHUB_TOKEN) +# SYNC_NOW the current time, as 2026-10-28T17:00:00Z (tests) +# SYNC_API_TOKEN token for reading GitHub's activity log of upstream (read only) +# SYNC_ORIGIN_URL the fork (default: this GitHub repository) +# SYNC_UPSTREAM_URL upstream (default: Crazytieguy/codex-plugin-cc) +# SYNC_ACTIVITY_URL GitHub's activity log of upstream (tests use a file:// URL) +set -euo pipefail +export LC_ALL=C + +here=$(cd "$(dirname "$0")" && pwd) +origin_url=${SYNC_ORIGIN_URL:-https://github.com/${GITHUB_REPOSITORY:-TruthfulAI-research/codex-plugin-cc}.git} +upstream_url=${SYNC_UPSTREAM_URL:-https://github.com/Crazytieguy/codex-plugin-cc.git} +activity_url=${SYNC_ACTIVITY_URL:-https://api.github.com/repos/Crazytieguy/codex-plugin-cc/activity} +state_branch=upstream-sync-state +min_age_days=7 +stall_days=30 +manifests="plugins/codex/.claude-plugin/plugin.json .claude-plugin/marketplace.json" +tab=$(printf '\t') +time_re='^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$' + +die() { + if [ "${GITHUB_ACTIONS:-}" = true ]; then echo "::error::$*"; else echo "error: $*" >&2; fi + exit 1 +} + +command -v jq > /dev/null || die "jq is not installed" + +case ${SYNC_DRY_RUN:-0} in + 1 | true) dry_run=1 ;; + 0 | false | '') dry_run=0 ;; + *) die "SYNC_DRY_RUN must be 0 or 1" ;; +esac + +now=${SYNC_NOW:-$(date -u +%Y-%m-%dT%H:%M:%SZ)} +[[ $now =~ $time_re ]] || die "SYNC_NOW must look like 2026-10-28T17:00:00Z, not $now" +# Times in this format compare correctly as strings. +days_before() { jq -nr --arg now "$now" --argjson days "$1" '$now | fromdateiso8601 - $days * 86400 | todate'; } +cutoff=$(days_before "$min_age_days") +stall_cutoff=$(days_before "$stall_days") + +work=$(mktemp -d "${RUNNER_TEMP:-${TMPDIR:-/tmp}}/upstream-sync.XXXXXX") +trap 'rm -rf "$work"' EXIT +git init -q -b sync "$work/repo" +cd "$work/repo" +if [ "${GITHUB_ACTIONS:-}" = true ]; then + git config user.name 'github-actions[bot]' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' +fi +git config merge.fork-manifest.name 'upstream manifest without version fields' +git config merge.fork-manifest.driver "bash '$here/fork-manifest.sh' %B %A %P" + +if [ -n "${SYNC_TOKEN:-}" ]; then + auth=$(printf 'x-access-token:%s' "$SYNC_TOKEN" | base64 | tr -d '\n') + if [ "${GITHUB_ACTIONS:-}" = true ]; then echo "::add-mask::$auth"; fi + # The header goes only to the fork's URL, and through the environment, so the token is in no file. + export GIT_CONFIG_COUNT=1 + export GIT_CONFIG_KEY_0="http.$origin_url.extraheader" + export GIT_CONFIG_VALUE_0="AUTHORIZATION: basic $auth" +fi + +git remote add origin "$origin_url" +git remote add upstream "$upstream_url" +git fetch -q --no-tags upstream +refs/heads/main:refs/remotes/upstream/main +git fetch -q --no-tags origin +refs/heads/main:refs/remotes/origin/main +status=0 +git ls-remote --exit-code origin "refs/heads/$state_branch" > /dev/null || status=$? +case $status in + 0) + have_state=1 + git fetch -q --no-tags origin "+refs/heads/$state_branch:refs/remotes/origin/$state_branch" + git show "refs/remotes/origin/$state_branch:seen.tsv" > "$work/seen.tsv" + ;; + 2) + have_state=0 + echo "The state branch $state_branch does not exist yet; this run creates it." + : > "$work/seen.tsv" + ;; + *) die "could not list the branches of $origin_url" ;; +esac + +# seen.tsv is a log of upstream's head: a row whenever the head differs from the last row's commit, +# even if that commit is in an earlier row. So the last row is upstream's current head. +head=$(git rev-parse refs/remotes/upstream/main) +last_row=$(tail -n 1 "$work/seen.tsv") +if [ "${last_row%%"$tab"*}" = "$head" ]; then + echo "Upstream main is at $head, its head since ${last_row#*"$tab"}." +else + printf '%s\t%s\n' "$head" "$now" >> "$work/seen.tsv" + echo "Upstream main is at $head, a new head as of now ($now)." +fi + +if [ -s "$work/seen.tsv" ] && [ -n "$(tail -c 1 "$work/seen.tsv")" ]; then + die "seen.tsv on $state_branch does not end with a newline" +fi +if grep -Evq "^[0-9a-f]{40}${tab}[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z\$" "$work/seen.tsv"; then + die "seen.tsv on $state_branch has a malformed line" +fi +if ! awk -F "$tab" 'NR > 1 && $2 < previous { exit 1 } { previous = $2 }' "$work/seen.tsv"; then + die "seen.tsv on $state_branch is not in time order, or the clock is behind its last row" +fi + +shas=() times=() +while IFS="$tab" read -r sha time; do + shas+=("$sha") + times+=("$time") +done < "$work/seen.tsv" + +# Commits that this fork merged and that upstream's main no longer has: upstream removed them after +# the 7 days, which may mean that they were bad. +removed=$(for sha in "${shas[@]}"; do + if git merge-base --is-ancestor "$sha" refs/remotes/origin/main 2> /dev/null && + ! git merge-base --is-ancestor "$sha" refs/remotes/upstream/main 2> /dev/null; then + echo "$sha" + fi +done | sort -u) + +# stayed : whether the row's commit is an ancestor of every later row's commit, that is, stayed +# in upstream's history without a break since that row. A later commit that is no longer on upstream +# main is not fetched; it counts as a break. +stayed() { + local j + for ((j = $1 + 1; j < ${#shas[@]}; j++)); do + git merge-base --is-ancestor "${shas[$1]}" "${shas[$j]}" 2> /dev/null || return 1 + done +} + +# The candidate: the newest row at least 7 days old whose commit stayed in upstream's history since. +# A commit that left upstream's history and came back counts only from its new row. Commit dates +# are never used: an attacker can backdate them. +candidate='' candidate_seen='' +for ((i = ${#shas[@]} - 1; i >= 0; i--)); do + if [[ ${times[$i]} > $cutoff ]]; then continue; fi + if stayed "$i"; then + candidate=${shas[$i]} candidate_seen=${times[$i]} + break + fi + echo "Skipping ${shas[$i]} (on upstream main at ${times[$i]}): not every later head is known to contain it." +done + +# rewrites_since