feat(git): git-backed stacks, core and command line (#138, part 1)

[grantemsley]

First of four PRs for git-backed stacks (#138): the core and a command-line tool, compose-git. Next come
credentials for private repositories, then the web UI, then scheduled checks. A git stack is an indirect stack
pointing at a plugin-managed clone, so the Compose page shows and runs it like any other stack.

Why so many checks: this runs as root on a box that often holds the only copy of people's appdata, so the
rule is that no typo or odd repository state loses data. Nothing is written unless the array is started and
the path is on a live mount, nothing of the user's is deleted (local changes are saved as a patch first,
untracked and ignored files are never touched), and no container changes until every check, pull and build has
passed. Each check is there for a failure we hit or reproduced.

What it adds

• compose.sh -c gitdeploy: fetch, check out, check, pull, build, then up, under the stack's usual lock. A
  failure before up puts the previous commit back; a failed up is recorded, not rolled back.
• Pre-flight checks built on docker compose config (no YAML parsing of our own): invalid compose, unset
  variables, .env.example keys missing from the .env, missing networks, volumes, bind sources, files and
  build folders, and container names or ports already taken.
• Recreate the stack when its folder changed (on by default, per stack), so a commit that only edits a mounted
  config file takes effect. With it off, such a commit deploys but the container silently keeps the old file.
  Flipping the default is one line if you prefer opt-in.
• compose-git add | convert | check | deploy | reclone | status, with the exit status as the result, and
  --help per command.
• docs/git-stacks.md, including moving an existing stack into git.

Ordinary stacks are unchanged: every StackInfo edit is behind isGitStack(), and compose.sh only gains the
gitdeploy case and its entries in the lock, credential-log and exit-status lists.

Reading it: the 14 commits are meant to be read in order, each one step with its tests: path guard, git
runner, settings, clone and checkout, saving local changes, StackInfo, pre-flight checks, deployed-commit
record, gitdeploy, recreate on folder change, compose-git, docs, missing networks (#167), override prune
(#169).

Testing: 303 new PHPUnit tests (real git against throwaway repositories) and 21 Bats tests; the full suite
and PHPStan pass at every commit. On Unraid 7.3.2 with real Docker and Compose, end-to-end scripts covered
deploys from the array and from GitHub, each kind of refusal, failed and interrupted deploys, local changes and
container data in the clone, reclone, convert, the #167 and #169 paths, and the web UI's Stop/Start on a git
stack.

Notes

• git and docker run through proc_open with an argument list and a fixed environment, not exec(): no shell,
  stdout and stderr kept apart, and a stray GIT_DIR cannot redirect git.
• PHP 8.1 features (readonly, array_is_list); Unraid 6.12 ships 8.2.
• compose-git is at scripts/compose-git; a symlink from the .plg would put it on the PATH. I left the
  .plg alone.
• GitCommand::run() takes extra environment variables no caller here uses yet; the credentials PR needs them.

[Nikosan73]

Sorry, I opened this can and disappeared.

Thanks, @grantemsley this is a thorough design. I'd want those safety checks on a server that holds real data. The command-line tool with exit codes matters most to me because our releases run via scripts and agents over SSH.

Our priorities differ a little, so a few points from my side.

Releases are deliberate. We only release when a person approves a specific version. For us, the schedule would stay on "notify" and never "deploy". Could new git stacks default to off or notify, so nothing deploys unattended unless someone chooses it?

We release versions, not branch heads. Each release is a numbered version. I see deploy --commit already covers a specific commit. Could it also accept a tag, for example compose-git deploy myapp --tag v1.4.0? Then we could deploy the exact version we approved without looking up the commit id.

Some releases need a step before the new containers start, such as a database migration. I would suggest the docs show the pattern of a one-off migration service that the app waits for (depends_on with condition: service_completed_successfully). That keeps it inside compose and needs no new code. Does that sit well with your pre-flight checks?

None of this should hold up part 1. Happy to see it land as planned.

[grantemsley]

@Nikosan73 No problem, I'm just happy that I'll soon be able to stop manually deploying stuff on unraid.

To address your concerns:

Unattended deploys are off by default. Right now there's no automatic updates, but the PR that includes scheduling will give each stack the option for off (default), notify, and deploy. My intention is to pick the safest defaults so nothing surprises anyone in an update.

Deploying a tag can't be done today. The clone is a single branch with no tags fetched, and it currently only takes a full commit id to prevent a typo deploying the wrong thing. Adding a --tag option to deploy that looks up the tag and resolves the commit for you could maybe be added later. Or if you deploy via scripts or CI, you can have that look up the commit id (git rev-parse v1.4.0^{commit}), as long as the tag is on the stack's branch.

Migrations as a one off service fit right in with the Folders and files your containers need section for other prep work your stack needs. The preflight checks don't have any issue with that, so I can just include an example in the docs later. I tested it with --wait: a migration that exits non-zero fails the deploy and records the commit as failed, but compose has already recreated the app containers, so the stack is down until the next deploy.

[Nikosan73]

Thanks, that all works for us. Off by default is what I hoped for, and git rev-parse covers tags until a --tag option exists.

One follow-up on migrations. If a failed migration leaves the stack down until the next deploy, that is the outcome we most want to avoid. Could there be an optional pre-deploy command per stack? It would run after fetch, checkout, pull and build, but before up. For example, docker compose run --rm migrate against the new commit.

A non-zero exit would then count like any other pre-flight failure. The previous commit goes back, and the running containers are never touched. That fits the rollback you already have before up. It cannot undo a migration that half-applied, but the old app stays up while someone looks.

Again, not something to hold part 1 for.

[mstrhakr]

I'll be merging this today after the dev->main PR is merged so that this is the main feature of the next beta. That gives us time to get it all put together before we push it to main.

[grantemsley]

@Copilot Thanks, all six are right and each is fixed as its own commit on top:

• Missing containers vs. inspect errors, fail closed on docker ps, reject failed or
  malformed bulk inspect: 0efe816. Only "No such container" now means a name is free. Any other
  failure of docker ps or docker inspect, and output that is not JSON, is reported as a problem and
  stops the deploy before up, like every other check. A container removed between ps and the bulk
  inspect is still accepted, since inspect prints the others and the removed one holds no port.
• Duplicate requested port bindings: 9cfcaf9. The stack's own published ports are now compared with
  each other too, with the same address and protocol rules.
• Retain the deployed SHA after a branch rewrite: 80ed389. reclone keeps it, so the next deploy
  takes the existing "commit no longer in the clone" path and recreates every container. Checked on
  the test box: rewrite, prune, reclone, deploy, and the containers were recreated.
• Caches when the repository override is deleted: a26b48a. When the override is missing, its
  folder's mtime is compared instead, which a removal updates.

ae4aee4 is a docs line next to reclone, since a bind mount keeps following the moved-aside folder
until the container is recreated.

Tests cover each case, and the suite is green.

@Nikosan73 A predeploy step could fit in pretty well. The one thing options built into the compose can't handle is the service failing, and a predeploy script could abort the deploy if needed. That would leave it up to your script to decide when to continue, when to roll back, and when to stop and wait for manual intervention. I'll add that to the list of things to look into after the features I've already got in progress are done.