feat(git): private repositories for git stacks (#138, part 2)

docs/git-stacks.md

@@ -22,6 +22,7 @@ to your shell profile.
 - [Moving an existing stack into git](#moving-an-existing-stack-into-git)
 - [Day to day](#day-to-day)
 - [How to lay out your repositories](#how-to-lay-out-your-repositories)
+- [Private repositories](#private-repositories)
 - [Which .env is used](#which-env-is-used)
 - [Folders and files your containers need](#folders-and-files-your-containers-need)
 - [Config files and recreating containers](#config-files-and-recreating-containers)
@@ -53,8 +54,9 @@ to your shell profile.
 
 ## Quick start
 
-You need a repository with a compose file in it, reachable over `https://` without a password (a public
-repository), or a repository on this server under `/mnt` (see [step 2](#2-create-the-repository)).
+You need a repository with a compose file in it: a public one over `https://`, a private one (see
+[Private repositories](#private-repositories)), or one on this server under `/mnt` (see
+[step 2](#2-create-the-repository)).
 
 ```bash
 compose-git add whoami --url https://github.com/you/stacks.git --path whoami/compose.yaml
@@ -91,9 +93,8 @@ stacks/
 
 Either:
 
-- **On GitHub, Gitea, Forgejo or similar.** It must be public for now (private repositories need credentials,
-  which are not supported yet; see [Limitations](#limitations)). Anything in a public repository is public,
-  so keep secrets out of it (step 4).
+- **On GitHub, Gitea, Forgejo or similar.** Public or private; a private one needs an access token or a
+  deploy key (see [Private repositories](#private-repositories)). Either way, keep secrets out of it (step 4).
 - **On this server.** A bare repository under `/mnt` works without any credentials:
 
   ```bash
@@ -218,6 +219,47 @@ big things:
 A change to another stack's folder in a shared repository is fetched by every stack, but only recreates the
 containers of the stack whose folder changed (see [below](#config-files-and-recreating-containers)).
 
+## Private repositories
+
+A private repository is reached in one of two ways. Either way, the secret is kept encrypted in the plugin's
+credential vault, given to git only while it runs, and never written into the clone.
+
+**An HTTPS access token.** Make a token on your git host that can only read the repository (on GitHub, a
+fine-grained token with read access to its contents; on Gitea or Forgejo, an access token with read access to
+repositories). Add it on the **Credentials** tab of the plugin settings, as **Git repository (HTTPS token, for
+git stacks)**, with the host only (`github.com`, or `git.example.com:3000`). Then name it when you add the
+stack:
+
+```bash
+compose-git add myapp --url https://github.com/you/stacks.git --path myapp/compose.yaml --credential "GitHub stacks"
+```
+
+Use the repository's address exactly as the host gives it for cloning, usually ending in `.git`. The token is
+offered only for that address, so a host that redirects a shorter address gets no token after the redirect.
+
+One token can serve every stack on the same host that it can read. **Test** on the Credentials tab tries it
+against a stack that already uses it, so add the stack first; until then the tab says no git stack uses it.
+
+**An SSH deploy key.** Use an ssh address, and the stack gets a key of its own:
+
+```bash
+compose-git add myapp --url git@github.com:you/stacks.git --path myapp/compose.yaml
+```
+
+The first time, the clone fails because the repository does not know the key yet. The command prints the
+public key: add it to the repository as a **read-only deploy key** (on GitHub: the repository's Settings >
+Deploy keys), then run the same command again. `compose-git deploy-key myapp` shows the key again later.
+Until then the key is listed on the Credentials tab as used by no stack: leave it there, or the second run
+makes a new key, and the one you added to the repository no longer works.
+
+When the stack is added, the server's ssh host keys are pinned and their fingerprints printed. Compare them with
+the ones your git host publishes (GitHub, GitLab and Codeberg list theirs). Every later connection must match
+them. If the server is rebuilt and its keys change, deploys stop until you run `compose-git trust-host myapp`,
+which shows the old and new fingerprints and pins the new ones.
+
+To change or remove a stack's HTTPS credential later: `compose-git credential myapp "Other token"` or
+`compose-git credential myapp --none`. The change is saved only if the repository can be reached with it.
+
 ## Which .env is used
 
 With no custom env file set for the stack, a git stack uses, in order:
@@ -348,17 +390,22 @@ otherwise make compose refuse the stack. The log names each one removed.
 
 | Command | What it does |
 |---|---|
-| `add <name> --url <url> --path <file> [--branch <b>] [--clones-root <folder>] [--description <text>]` | Clone a repository and make a new git stack. Not deployed yet. |
-| `convert <stack> --url <url> --path <file> [--branch <b>] [--clones-root <folder>]` | Turn an existing stack into a git stack (see above). |
+| `add <name> --url <url> --path <file> [--branch <b>] [--clones-root <folder>] [--description <text>] [--credential <name>]` | Clone a repository and make a new git stack. Not deployed yet. |
+| `convert <stack> --url <url> --path <file> [--branch <b>] [--clones-root <folder>] [--credential <name>]` | Turn an existing stack into a git stack (see above). |
+| `credential <stack> <name>` or `credential <stack> --none` | Change or remove the HTTPS credential a stack uses. |
+| `deploy-key <stack>` | Show an ssh stack's public deploy key. |
+| `trust-host <stack>` | Pin an ssh stack's server host keys again, after they changed. |
 | `check <stack>` or `check --all` | Compare the deployed commit with the branch on the remote. Changes nothing. |
 | `deploy <stack> [--commit <id>] [--save-local-changes] [--wait\|--no-wait] [--wait-timeout <s>] [--profile <p>]...` | Deploy the branch's latest commit, or the given one. Waits for healthy containers when the stack's wait-for-healthy setting says so, unless `--wait` or `--no-wait` overrides it. |
 | `reclone <stack>` | Move the clone aside and clone again at the deployed commit. |
 | `status [<stack>\|--all] [--json]` | Show what is deployed, failed and changed locally, without asking the remote. |
 
 `compose-git <command> --help` explains a command and each of its options.
 
-`<url>` is an `https://` address without a user name or password in it, or the path of a repository under
-`/mnt`. `<stack>` is the stack's exact folder name in the projects folder. `--clones-root` must be on a share,
+`<url>` is an `https://` address without a user name or password in it, an ssh address
+(`ssh://git@host[:port]/path` or `git@host:path`), or the path of a repository under `/mnt`. `--credential`
+is the name of a git credential (an HTTPS token) on the Credentials tab, for an `https://` address; an ssh
+stack always uses the deploy key made for it. `<stack>` is the stack's exact folder name in the projects folder. `--clones-root` must be on a share,
 disk or pool, and the share must exist.
 
 Exit status: `0` success (for `check`: up to date), `1` failed, `2` the command line was not understood, `3`
@@ -386,8 +433,8 @@ deployed commit: only `compose-git deploy` does that.
 
 ## Limitations
 
-- **Public `https` repositories and repositories on this server only.** Private repositories (access tokens,
-  SSH deploy keys) are planned.
+- **An `https` server needs a certificate this server trusts.** A self-hosted git server with a self-signed
+  certificate is refused over `https`; reach it over ssh instead.
 - **No web UI yet.** Git stacks are created and deployed from the command line; the Compose page shows and
   runs them like other stacks.
 - **No automatic deploys yet.** Use `check` and `deploy` from your own schedule or a git hook.

source/compose.manager/include/CredentialVault.php

@@ -51,9 +51,23 @@ public function saveCredential(array $input): array
             if ($name === '' || $registry === '' || $username === '' || $secret === '') {
                 throw new InvalidArgumentException('Name, registry, username, and token are required.');
             }
-            if (!in_array($provider, ['github', 'docker', 'gitlab', 'quay', 'aws', 'azure', 'gcr', 'generic'], true)) {
+            if (!in_array($provider, ['github', 'docker', 'gitlab', 'quay', 'aws', 'azure', 'gcr', 'generic', 'git', 'git-ssh'], true)) {
                 throw new InvalidArgumentException('Unsupported credential provider.');
             }
+            // A stack using the credential would break at its next run: a git stack needs a git
+            // credential, and a registry login is never a git one. Changes among registry kinds
+            // (generic to docker, say) stay allowed.
+            $existingProvider = (string) ($existing['provider'] ?? '');
+            $gitKinds = ['git', 'git-ssh'];
+            if ($existing !== [] && $existingProvider !== $provider
+                && (in_array($existingProvider, $gitKinds, true) || in_array($provider, $gitKinds, true))) {
+                throw new InvalidArgumentException('A credential cannot change between a registry login and a git credential. Add a new credential instead.');
+            }
+            // A git repository credential (an HTTPS token, or a git stack's ssh deploy key in
+            // "git-ssh") is for one host, such as github.com.
+            if (($provider === 'git' || $provider === 'git-ssh') && preg_match('/^[a-z0-9](?:[a-z0-9.-]*[a-z0-9])?(?::[0-9]{1,5})?$/', $registry) !== 1) {
+                throw new InvalidArgumentException('A git credential needs the repository host only, such as github.com.');
+            }
             if (!in_array($authMethod, ['manual', 'oauth_device'], true)) {
                 throw new InvalidArgumentException('Unsupported credential authentication method.');
             }
@@ -165,9 +179,31 @@ public function testCredential(string $id): array
         ];
     }
 
+    /**
+     * Hand one credential, secret included, to a callback, and return what it returns.
+     *
+     * The one place code reads a secret out of the vault, for code that has to pass it on
+     * to another program: git stacks write it to a short-lived file and remove it after
+     * the run. Whatever the callback returns, the secret included, goes back to the caller.
+     *
+     * @template T
+     * @param callable(array<string, string>): T $callback
+     * @return T
+     */
+    public function useCredential(string $id, callable $callback)
+    {
+        $credential = $this->withLock(LOCK_SH, fn(): array => $this->findCredential($id));
+        return $callback($credential);
+    }
+
     public function materializeDockerConfig(string $id): string
     {
         $credential = $this->withLock(LOCK_SH, fn(): array => $this->findCredential($id));
+        // A git token or deploy key is never a registry login. Stack settings refuse to assign one, but a
+        // hand-edited or restored credential_id could still name one: never write it out as a Docker login.
+        if (in_array($credential['provider'] ?? '', ['git', 'git-ssh'], true)) {
+            throw new RuntimeException("The credential '{$credential['name']}' is a git repository credential, not a registry login.");
+        }
         // Confirms which named credential a compose operation is using, without ever logging the secret.
         composeLogger(
             "Using registry credential '{$credential['name']}' ({$credential['registry']}) for this operation",

source/compose.manager/include/Defines.php

@@ -42,6 +42,8 @@ function locate_compose_root($name)
 defined('COMPOSE_MOUNTS_FILE') || define('COMPOSE_MOUNTS_FILE', '/proc/self/mounts');
 defined('COMPOSE_GIT_MNT_DIR') || define('COMPOSE_GIT_MNT_DIR', '/mnt');
 defined('COMPOSE_GIT_HOME_DIR') || define('COMPOSE_GIT_HOME_DIR', '/var/tmp/compose-manager-git-home');
+// Short-lived files that hand a git stack's credential to git for one run (in RAM, emptied at reboot).
+defined('COMPOSE_GIT_CREDENTIAL_DIR') || define('COMPOSE_GIT_CREDENTIAL_DIR', '/var/tmp/compose-manager-git-credentials');
 defined('COMPOSE_GIT_DEFAULT_CLONES_ROOT') || define('COMPOSE_GIT_DEFAULT_CLONES_ROOT', '/mnt/user/appdata/compose.manager/git');
 // The docker CLI (the git deploy checks run it; tests point this at a scripted stand-in).
 defined('COMPOSE_DOCKER_BIN') || define('COMPOSE_DOCKER_BIN', 'docker');

source/compose.manager/include/Exec.php

@@ -184,6 +184,7 @@ function composePurgeDeletedStackCaches(string $stackName): array
     /**
      * Single-pass scan of every stack's `credential_id` file, grouped by credential id.
      * Avoids re-globbing/re-reading the compose root once per credential.
+     * A git stack's repository credential (`credentialId` in its git.json) counts too.
      *
      * @return array<string, string[]> Credential id => assigned stack names
      */
@@ -197,6 +198,18 @@ function composeBuildCredentialStackMap(string $composeRoot): array
             }
             $map[$credentialId][] = basename(dirname($credentialFile));
         }
+        foreach (glob(rtrim($composeRoot, '/') . '/*/git.json') ?: [] as $gitSettingsFile) {
+            // Read as plain JSON: a git.json that fails validation still keeps its credential in use.
+            $gitSettings = json_decode((string) @file_get_contents($gitSettingsFile), true);
+            $credentialId = is_array($gitSettings) ? ($gitSettings['credentialId'] ?? null) : null;
+            if (!is_string($credentialId) || $credentialId === '') {
+                continue;
+            }
+            $stack = basename(dirname($gitSettingsFile));
+            if (!in_array($stack, $map[$credentialId] ?? [], true)) {
+                $map[$credentialId][] = $stack;
+            }
+        }
         return $map;
     }
 }
@@ -1019,6 +1032,10 @@ function composeResolveContainerIcon(string $containerName, string $service, arr
         break;
     case 'saveCredential':
         try {
+            // Deploy keys are made by compose-git for one stack, never on the Credentials tab.
+            if (strtolower(trim((string) ($_POST['provider'] ?? ''))) === 'git-ssh') {
+                throw new \InvalidArgumentException('A deploy key is made by compose-git for its stack, not added here.');
+            }
             $credential = (new CredentialVault())->saveCredential([
                 'id' => trim((string) ($_POST['id'] ?? '')),
                 'name' => trim((string) ($_POST['name'] ?? '')),
@@ -1068,9 +1085,20 @@ function composeResolveContainerIcon(string $containerName, string $service, arr
             break;
         }
         try {
-            $test = (new CredentialVault())->testCredential($credentialId);
+            if (in_array((new CredentialVault())->getCredentialSummary($credentialId)['provider'] ?? '', ['git', 'git-ssh'], true)) {
+                // A git credential is tested by reaching a repository that uses it, not with a registry handshake.
+                require_once("/usr/local/emhttp/plugins/compose.manager/include/GitStackManager.php");
+                $test = (new GitStackManager($compose_root))->testCredential($credentialId);
+                if ($test === null) {
+                    echo json_encode(['result' => 'error', 'message' => 'No git stack uses this credential yet. '
+                        . 'It is tested by reaching a repository: give it to a stack with compose-git, then test it again.']);
+                    break;
+                }
+            } else {
+                $test = (new CredentialVault())->testCredential($credentialId);
+            }
             composeLogger(
-                "Tested registry credential '{$test['name']}': " . ($test['valid'] ? 'valid' : 'invalid') . ' - ' . $test['message'],
+                "Tested credential '{$test['name']}': " . ($test['valid'] ? 'valid' : 'invalid') . ' - ' . $test['message'],
                 null,
                 'user',
                 $test['valid'] ? 'info' : 'warning',
@@ -1431,6 +1459,9 @@ function composeResolveContainerIcon(string $containerName, string $service, arr
                     if ($credentialId !== '' && !(new CredentialVault())->hasCredential($credentialId)) {
                         throw new RuntimeException('Selected credential no longer exists.');
                     }
+                    if ($credentialId !== '' && in_array((new CredentialVault())->getCredentialSummary($credentialId)['provider'] ?? '', ['git', 'git-ssh'], true)) {
+                        throw new RuntimeException('A git repository credential cannot be used to pull images.');
+                    }
                     if ($credentialId === '') {
                         if (is_file($credentialIdFile)) {
                             @unlink($credentialIdFile);