diff --git a/docs/git-stacks.md b/docs/git-stacks.md index 82c4cf7a..ec0082f4 100644 --- a/docs/git-stacks.md +++ b/docs/git-stacks.md @@ -22,6 +22,7 @@ to your shell profile. - [Moving an existing stack into git](#moving-an-existing-stack-into-git) - [Day to day](#day-to-day) - [How to lay out your repositories](#how-to-lay-out-your-repositories) +- [Private repositories](#private-repositories) - [Which .env is used](#which-env-is-used) - [Folders and files your containers need](#folders-and-files-your-containers-need) - [Config files and recreating containers](#config-files-and-recreating-containers) @@ -53,8 +54,9 @@ to your shell profile. ## Quick start -You need a repository with a compose file in it, reachable over `https://` without a password (a public -repository), or a repository on this server under `/mnt` (see [step 2](#2-create-the-repository)). +You need a repository with a compose file in it: a public one over `https://`, a private one (see +[Private repositories](#private-repositories)), or one on this server under `/mnt` (see +[step 2](#2-create-the-repository)). ```bash compose-git add whoami --url https://github.com/you/stacks.git --path whoami/compose.yaml @@ -91,9 +93,8 @@ stacks/ Either: -- **On GitHub, Gitea, Forgejo or similar.** It must be public for now (private repositories need credentials, - which are not supported yet; see [Limitations](#limitations)). Anything in a public repository is public, - so keep secrets out of it (step 4). +- **On GitHub, Gitea, Forgejo or similar.** Public or private; a private one needs an access token or a + deploy key (see [Private repositories](#private-repositories)). Either way, keep secrets out of it (step 4). - **On this server.** A bare repository under `/mnt` works without any credentials: ```bash @@ -218,6 +219,47 @@ big things: A change to another stack's folder in a shared repository is fetched by every stack, but only recreates the containers of the stack whose folder changed (see [below](#config-files-and-recreating-containers)). +## Private repositories + +A private repository is reached in one of two ways. Either way, the secret is kept encrypted in the plugin's +credential vault, given to git only while it runs, and never written into the clone. + +**An HTTPS access token.** Make a token on your git host that can only read the repository (on GitHub, a +fine-grained token with read access to its contents; on Gitea or Forgejo, an access token with read access to +repositories). Add it on the **Credentials** tab of the plugin settings, as **Git repository (HTTPS token, for +git stacks)**, with the host only (`github.com`, or `git.example.com:3000`). Then name it when you add the +stack: + +```bash +compose-git add myapp --url https://github.com/you/stacks.git --path myapp/compose.yaml --credential "GitHub stacks" +``` + +Use the repository's address exactly as the host gives it for cloning, usually ending in `.git`. The token is +offered only for that address, so a host that redirects a shorter address gets no token after the redirect. + +One token can serve every stack on the same host that it can read. **Test** on the Credentials tab tries it +against a stack that already uses it, so add the stack first; until then the tab says no git stack uses it. + +**An SSH deploy key.** Use an ssh address, and the stack gets a key of its own: + +```bash +compose-git add myapp --url git@github.com:you/stacks.git --path myapp/compose.yaml +``` + +The first time, the clone fails because the repository does not know the key yet. The command prints the +public key: add it to the repository as a **read-only deploy key** (on GitHub: the repository's Settings > +Deploy keys), then run the same command again. `compose-git deploy-key myapp` shows the key again later. +Until then the key is listed on the Credentials tab as used by no stack: leave it there, or the second run +makes a new key, and the one you added to the repository no longer works. + +When the stack is added, the server's ssh host keys are pinned and their fingerprints printed. Compare them with +the ones your git host publishes (GitHub, GitLab and Codeberg list theirs). Every later connection must match +them. If the server is rebuilt and its keys change, deploys stop until you run `compose-git trust-host myapp`, +which shows the old and new fingerprints and pins the new ones. + +To change or remove a stack's HTTPS credential later: `compose-git credential myapp "Other token"` or +`compose-git credential myapp --none`. The change is saved only if the repository can be reached with it. + ## Which .env is used With no custom env file set for the stack, a git stack uses, in order: @@ -348,8 +390,11 @@ otherwise make compose refuse the stack. The log names each one removed. | Command | What it does | |---|---| -| `add --url --path [--branch ] [--clones-root ] [--description ]` | Clone a repository and make a new git stack. Not deployed yet. | -| `convert --url --path [--branch ] [--clones-root ]` | Turn an existing stack into a git stack (see above). | +| `add --url --path [--branch ] [--clones-root ] [--description ] [--credential ]` | Clone a repository and make a new git stack. Not deployed yet. | +| `convert --url --path [--branch ] [--clones-root ] [--credential ]` | Turn an existing stack into a git stack (see above). | +| `credential ` or `credential --none` | Change or remove the HTTPS credential a stack uses. | +| `deploy-key ` | Show an ssh stack's public deploy key. | +| `trust-host ` | Pin an ssh stack's server host keys again, after they changed. | | `check ` or `check --all` | Compare the deployed commit with the branch on the remote. Changes nothing. | | `deploy [--commit ] [--save-local-changes] [--wait\|--no-wait] [--wait-timeout ] [--profile

]...` | Deploy the branch's latest commit, or the given one. Waits for healthy containers when the stack's wait-for-healthy setting says so, unless `--wait` or `--no-wait` overrides it. | | `reclone ` | Move the clone aside and clone again at the deployed commit. | @@ -357,8 +402,10 @@ otherwise make compose refuse the stack. The log names each one removed. `compose-git --help` explains a command and each of its options. -`` is an `https://` address without a user name or password in it, or the path of a repository under -`/mnt`. `` is the stack's exact folder name in the projects folder. `--clones-root` must be on a share, +`` is an `https://` address without a user name or password in it, an ssh address +(`ssh://git@host[:port]/path` or `git@host:path`), or the path of a repository under `/mnt`. `--credential` +is the name of a git credential (an HTTPS token) on the Credentials tab, for an `https://` address; an ssh +stack always uses the deploy key made for it. `` is the stack's exact folder name in the projects folder. `--clones-root` must be on a share, disk or pool, and the share must exist. Exit status: `0` success (for `check`: up to date), `1` failed, `2` the command line was not understood, `3` @@ -386,8 +433,8 @@ deployed commit: only `compose-git deploy` does that. ## Limitations -- **Public `https` repositories and repositories on this server only.** Private repositories (access tokens, - SSH deploy keys) are planned. +- **An `https` server needs a certificate this server trusts.** A self-hosted git server with a self-signed + certificate is refused over `https`; reach it over ssh instead. - **No web UI yet.** Git stacks are created and deployed from the command line; the Compose page shows and runs them like other stacks. - **No automatic deploys yet.** Use `check` and `deploy` from your own schedule or a git hook. diff --git a/source/compose.manager/include/CredentialVault.php b/source/compose.manager/include/CredentialVault.php index d0335014..803dbcdb 100644 --- a/source/compose.manager/include/CredentialVault.php +++ b/source/compose.manager/include/CredentialVault.php @@ -51,9 +51,23 @@ public function saveCredential(array $input): array if ($name === '' || $registry === '' || $username === '' || $secret === '') { throw new InvalidArgumentException('Name, registry, username, and token are required.'); } - if (!in_array($provider, ['github', 'docker', 'gitlab', 'quay', 'aws', 'azure', 'gcr', 'generic'], true)) { + if (!in_array($provider, ['github', 'docker', 'gitlab', 'quay', 'aws', 'azure', 'gcr', 'generic', 'git', 'git-ssh'], true)) { throw new InvalidArgumentException('Unsupported credential provider.'); } + // A stack using the credential would break at its next run: a git stack needs a git + // credential, and a registry login is never a git one. Changes among registry kinds + // (generic to docker, say) stay allowed. + $existingProvider = (string) ($existing['provider'] ?? ''); + $gitKinds = ['git', 'git-ssh']; + if ($existing !== [] && $existingProvider !== $provider + && (in_array($existingProvider, $gitKinds, true) || in_array($provider, $gitKinds, true))) { + throw new InvalidArgumentException('A credential cannot change between a registry login and a git credential. Add a new credential instead.'); + } + // A git repository credential (an HTTPS token, or a git stack's ssh deploy key in + // "git-ssh") is for one host, such as github.com. + if (($provider === 'git' || $provider === 'git-ssh') && preg_match('/^[a-z0-9](?:[a-z0-9.-]*[a-z0-9])?(?::[0-9]{1,5})?$/', $registry) !== 1) { + throw new InvalidArgumentException('A git credential needs the repository host only, such as github.com.'); + } if (!in_array($authMethod, ['manual', 'oauth_device'], true)) { throw new InvalidArgumentException('Unsupported credential authentication method.'); } @@ -165,9 +179,31 @@ public function testCredential(string $id): array ]; } + /** + * Hand one credential, secret included, to a callback, and return what it returns. + * + * The one place code reads a secret out of the vault, for code that has to pass it on + * to another program: git stacks write it to a short-lived file and remove it after + * the run. Whatever the callback returns, the secret included, goes back to the caller. + * + * @template T + * @param callable(array): T $callback + * @return T + */ + public function useCredential(string $id, callable $callback) + { + $credential = $this->withLock(LOCK_SH, fn(): array => $this->findCredential($id)); + return $callback($credential); + } + public function materializeDockerConfig(string $id): string { $credential = $this->withLock(LOCK_SH, fn(): array => $this->findCredential($id)); + // A git token or deploy key is never a registry login. Stack settings refuse to assign one, but a + // hand-edited or restored credential_id could still name one: never write it out as a Docker login. + if (in_array($credential['provider'] ?? '', ['git', 'git-ssh'], true)) { + throw new RuntimeException("The credential '{$credential['name']}' is a git repository credential, not a registry login."); + } // Confirms which named credential a compose operation is using, without ever logging the secret. composeLogger( "Using registry credential '{$credential['name']}' ({$credential['registry']}) for this operation", diff --git a/source/compose.manager/include/Defines.php b/source/compose.manager/include/Defines.php index 4a2ca601..90afe554 100644 --- a/source/compose.manager/include/Defines.php +++ b/source/compose.manager/include/Defines.php @@ -42,6 +42,8 @@ function locate_compose_root($name) defined('COMPOSE_MOUNTS_FILE') || define('COMPOSE_MOUNTS_FILE', '/proc/self/mounts'); defined('COMPOSE_GIT_MNT_DIR') || define('COMPOSE_GIT_MNT_DIR', '/mnt'); defined('COMPOSE_GIT_HOME_DIR') || define('COMPOSE_GIT_HOME_DIR', '/var/tmp/compose-manager-git-home'); +// Short-lived files that hand a git stack's credential to git for one run (in RAM, emptied at reboot). +defined('COMPOSE_GIT_CREDENTIAL_DIR') || define('COMPOSE_GIT_CREDENTIAL_DIR', '/var/tmp/compose-manager-git-credentials'); defined('COMPOSE_GIT_DEFAULT_CLONES_ROOT') || define('COMPOSE_GIT_DEFAULT_CLONES_ROOT', '/mnt/user/appdata/compose.manager/git'); // The docker CLI (the git deploy checks run it; tests point this at a scripted stand-in). defined('COMPOSE_DOCKER_BIN') || define('COMPOSE_DOCKER_BIN', 'docker'); diff --git a/source/compose.manager/include/Exec.php b/source/compose.manager/include/Exec.php index 13a3c646..98e05df0 100644 --- a/source/compose.manager/include/Exec.php +++ b/source/compose.manager/include/Exec.php @@ -184,6 +184,7 @@ function composePurgeDeletedStackCaches(string $stackName): array /** * Single-pass scan of every stack's `credential_id` file, grouped by credential id. * Avoids re-globbing/re-reading the compose root once per credential. + * A git stack's repository credential (`credentialId` in its git.json) counts too. * * @return array Credential id => assigned stack names */ @@ -197,6 +198,18 @@ function composeBuildCredentialStackMap(string $composeRoot): array } $map[$credentialId][] = basename(dirname($credentialFile)); } + foreach (glob(rtrim($composeRoot, '/') . '/*/git.json') ?: [] as $gitSettingsFile) { + // Read as plain JSON: a git.json that fails validation still keeps its credential in use. + $gitSettings = json_decode((string) @file_get_contents($gitSettingsFile), true); + $credentialId = is_array($gitSettings) ? ($gitSettings['credentialId'] ?? null) : null; + if (!is_string($credentialId) || $credentialId === '') { + continue; + } + $stack = basename(dirname($gitSettingsFile)); + if (!in_array($stack, $map[$credentialId] ?? [], true)) { + $map[$credentialId][] = $stack; + } + } return $map; } } @@ -1019,6 +1032,10 @@ function composeResolveContainerIcon(string $containerName, string $service, arr break; case 'saveCredential': try { + // Deploy keys are made by compose-git for one stack, never on the Credentials tab. + if (strtolower(trim((string) ($_POST['provider'] ?? ''))) === 'git-ssh') { + throw new \InvalidArgumentException('A deploy key is made by compose-git for its stack, not added here.'); + } $credential = (new CredentialVault())->saveCredential([ 'id' => trim((string) ($_POST['id'] ?? '')), 'name' => trim((string) ($_POST['name'] ?? '')), @@ -1068,9 +1085,20 @@ function composeResolveContainerIcon(string $containerName, string $service, arr break; } try { - $test = (new CredentialVault())->testCredential($credentialId); + if (in_array((new CredentialVault())->getCredentialSummary($credentialId)['provider'] ?? '', ['git', 'git-ssh'], true)) { + // A git credential is tested by reaching a repository that uses it, not with a registry handshake. + require_once("/usr/local/emhttp/plugins/compose.manager/include/GitStackManager.php"); + $test = (new GitStackManager($compose_root))->testCredential($credentialId); + if ($test === null) { + echo json_encode(['result' => 'error', 'message' => 'No git stack uses this credential yet. ' + . 'It is tested by reaching a repository: give it to a stack with compose-git, then test it again.']); + break; + } + } else { + $test = (new CredentialVault())->testCredential($credentialId); + } composeLogger( - "Tested registry credential '{$test['name']}': " . ($test['valid'] ? 'valid' : 'invalid') . ' - ' . $test['message'], + "Tested credential '{$test['name']}': " . ($test['valid'] ? 'valid' : 'invalid') . ' - ' . $test['message'], null, 'user', $test['valid'] ? 'info' : 'warning', @@ -1431,6 +1459,9 @@ function composeResolveContainerIcon(string $containerName, string $service, arr if ($credentialId !== '' && !(new CredentialVault())->hasCredential($credentialId)) { throw new RuntimeException('Selected credential no longer exists.'); } + if ($credentialId !== '' && in_array((new CredentialVault())->getCredentialSummary($credentialId)['provider'] ?? '', ['git', 'git-ssh'], true)) { + throw new RuntimeException('A git repository credential cannot be used to pull images.'); + } if ($credentialId === '') { if (is_file($credentialIdFile)) { @unlink($credentialIdFile); diff --git a/source/compose.manager/include/GitClone.php b/source/compose.manager/include/GitClone.php index d0c74771..ab409b1c 100644 --- a/source/compose.manager/include/GitClone.php +++ b/source/compose.manager/include/GitClone.php @@ -6,6 +6,8 @@ require_once '/usr/local/emhttp/plugins/compose.manager/include/GitPathGuard.php'; require_once '/usr/local/emhttp/plugins/compose.manager/include/GitCommand.php'; require_once '/usr/local/emhttp/plugins/compose.manager/include/GitStackSettings.php'; +require_once '/usr/local/emhttp/plugins/compose.manager/include/GitCredentials.php'; +require_once '/usr/local/emhttp/plugins/compose.manager/include/GitSsh.php'; /** * Thrown when a clone folder is not one this plugin can prove it made for this stack. @@ -668,12 +670,101 @@ private function isLocalRepository(): bool * it ("dubious ownership"). The stack's own repository path is trusted * for the run, and nothing else (see GitCommand). * + * A stack with a credential gets it for every run, not only clone and + * fetch: the clone is partial (--filter=blob:none), so a checkout or diff + * can fetch file contents from the repository too. An ssh stack gets its + * deploy key and pinned host keys the same way (see GitSsh). + * * @param string[] $args */ private function git(array $args, ?string $workingDirectory, int $timeoutSeconds = GitCommand::DEFAULT_TIMEOUT_SECONDS): ProcessResult { $trusted = $this->isLocalRepository() ? [$this->settings->url] : []; - return GitCommand::run($args, $workingDirectory, $timeoutSeconds, [], $trusted); + if ($this->settings->isSsh()) { + return $this->gitOverSsh($args, $workingDirectory, $timeoutSeconds); + } + if ($this->settings->credentialId === null) { + $result = GitCommand::run($args, $workingDirectory, $timeoutSeconds, [], $trusted); + } else { + $credentialFile = GitCredentials::writeFile($this->settings->credentialId, $this->settings->url); + try { + $result = GitCommand::run( + $args, + $workingDirectory, + $timeoutSeconds, + [], + $trusted, + GitCredentials::settingsFor($credentialFile) + ); + } finally { + GitCredentials::remove($credentialFile); + } + } + if (!$result->succeeded()) { + $hint = self::httpsFailureHint($result->stderr, $this->settings->credentialId !== null); + if ($hint !== null) { + throw new RuntimeException($result->errorSummary() . "\n" . $hint); + } + } + return $result; + } + + /** + * What to tell the person when git's own message for a failed https run + * points the wrong way, or null when it is clear enough. + */ + public static function httpsFailureHint(string $stderr, bool $hasCredential): ?string + { + // The token is offered only for the exact address the stack uses (credential.useHttpPath), + // so after a redirect git asks for the new address and gets nothing. + if ($hasCredential && preg_match('/^warning: redirecting to (\S+)/m', $stderr, $match) === 1) { + return "The server redirected to {$match[1]}, and the stack's credential is offered only for the address " + . 'the stack uses. Use the address the server redirects to (usually the one ending in .git).'; + } + if (!$hasCredential && str_contains($stderr, 'terminal prompts disabled')) { + return 'The repository asks for a user name and password, so it needs a credential: add a git token on the ' + . 'Credentials tab, then name it with --credential when adding the stack, or with compose-git credential ' + . ' for a stack that exists.'; + } + return null; + } + + /** + * Run git for an ssh stack, with its deploy key and its pinned host keys only. + * + * @param string[] $args + * @throws RuntimeException if the stack has no deploy key or no pinned host keys + */ + private function gitOverSsh(array $args, ?string $workingDirectory, int $timeoutSeconds): ProcessResult + { + if ($this->settings->credentialId === null || $this->settings->sshKnownHosts === null) { + throw new RuntimeException( + 'This ssh stack has no deploy key or no pinned host keys in its git.json. Set it up again with compose-git.' + ); + } + $files = GitSsh::writeRunFiles($this->settings->credentialId, $this->settings->sshKnownHosts); + try { + $result = GitCommand::run( + $args, + $workingDirectory, + $timeoutSeconds, + GitSsh::environmentFor($files), + [], + GitSsh::settings() + ); + // git's last line is only "Could not read from remote repository", so say what ssh said. + if (!$result->succeeded() && str_contains($result->stderr, 'Host key verification failed')) { + throw new RuntimeException( + 'The repository server\'s ssh host key does not match the one pinned for this stack, so nothing was fetched. ' + . 'If the server was rebuilt or its keys were changed on purpose, run compose-git trust-host and compare ' + . 'the new fingerprints with the ones the server publishes.' + ); + } + return $result; + } finally { + GitCredentials::remove($files['key']); + GitCredentials::remove($files['knownHosts']); + } } /** diff --git a/source/compose.manager/include/GitCommand.php b/source/compose.manager/include/GitCommand.php index 5ea5bddb..8c48b8dd 100644 --- a/source/compose.manager/include/GitCommand.php +++ b/source/compose.manager/include/GitCommand.php @@ -14,7 +14,8 @@ * up for other uses of git on the server), * - never asks for input (no password prompt that could hang a deploy), * - runs no repository hooks and no filesystem monitor, - * - allows only the https and local-path transports, + * - allows only the https and local-path transports (and ssh for a stack + * with a deploy key, through its own run settings: see GitSsh), * - starts from an empty environment, so stray GIT_DIR or GIT_WORK_TREE * variables cannot point it at a different repository, * - is stopped after a time limit, gets no input, and runs without a shell @@ -54,13 +55,18 @@ final class GitCommand * @param string|null $workingDirectory Folder to run in (the clone), or null for a neutral folder * @param array $extraEnvironment Extra environment variables for this run * @param string[] $trustedRepositories Repositories git may use even when another user owns them + * @param string[] $extraConfig Settings for this run only, passed with -c after the forced ones, such as + * a credential helper for one stack. Built by the plugin, never from user + * input, and never holding a secret: anyone on the server can read a + * command line. */ public static function run( array $args, ?string $workingDirectory = null, int $timeoutSeconds = self::DEFAULT_TIMEOUT_SECONDS, array $extraEnvironment = [], - array $trustedRepositories = [] + array $trustedRepositories = [], + array $extraConfig = [] ): ProcessResult { $homeDirectory = self::prepareHomeDirectory(); @@ -69,6 +75,10 @@ public static function run( $command[] = '-c'; $command[] = $setting; } + foreach ($extraConfig as $setting) { + $command[] = '-c'; + $command[] = $setting; + } foreach ($args as $arg) { $command[] = $arg; } diff --git a/source/compose.manager/include/GitCredentials.php b/source/compose.manager/include/GitCredentials.php new file mode 100644 index 00000000..1670a859 --- /dev/null +++ b/source/compose.manager/include/GitCredentials.php @@ -0,0 +1,164 @@ +useCredential( + $credentialId, + static function (array $credential) use ($host, $repository): string { + if (($credential['provider'] ?? '') !== 'git') { + throw new RuntimeException("The credential '{$credential['name']}' is not a git repository credential."); + } + if (($credential['registry'] ?? '') !== $host) { + throw new RuntimeException( + "The credential '{$credential['name']}' is for {$credential['registry']}, but the repository is on $host." + ); + } + return 'https://' . rawurlencode($credential['username']) . ':' . rawurlencode($credential['secret']) + . '@' . $host . $repository['path'] . "\n"; + } + ); + + return self::writeRunFile($line); + } + + /** + * Write a secret to a new file that only root can read, for one git run. + * + * Also used for a git stack's ssh deploy key. Remove it with remove() afterwards. + * + * @throws RuntimeException if the file cannot be written + */ + public static function writeRunFile(string $content): string + { + $directory = self::prepareDirectory(); + self::removeStaleFiles($directory); + + $file = $directory . '/' . bin2hex(random_bytes(16)); + // Created with root-only permissions (the umask), so the secret is never in a file others can read. + $oldUmask = umask(0077); + try { + $handle = @fopen($file, 'x'); + } finally { + umask($oldUmask); + } + if ($handle === false) { + throw new RuntimeException('Could not create a credential file for git.'); + } + chmod($file, 0600); + $written = fwrite($handle, $content); + fclose($handle); + if ($written !== strlen($content)) { + @unlink($file); + throw new RuntimeException('Could not write the credential file for git.'); + } + return $file; + } + + /** + * The settings that make git use a credential file (for GitCommand::run's $extraConfig). + * + * @return string[] + */ + public static function settingsFor(string $file): array + { + return [ + 'credential.helper=store --file=' . $file, + 'credential.useHttpPath=true', + ]; + } + + /** + * Remove a credential file written by writeFile(). + */ + public static function remove(string $file): void + { + $directory = realpath(COMPOSE_GIT_CREDENTIAL_DIR); + $target = realpath($file); + if ($directory === false || $target === false || dirname($target) !== $directory) { + return; + } + @unlink($target); + } + + /** + * The folder for run files, made if missing. It is in /var/tmp, where anyone can + * create things, so a folder someone else made first (or a symlink there) is refused: + * its owner could read or swap the files root writes into it. + */ + private static function prepareDirectory(): string + { + $directory = rtrim(COMPOSE_GIT_CREDENTIAL_DIR, '/'); + if (!file_exists($directory) && @readlink($directory) === false && !@mkdir($directory, 0700, true) && !is_dir($directory)) { + throw new RuntimeException('Could not create the folder for git credential files.'); + } + clearstatcache(true, $directory); + if (@readlink($directory) !== false || !is_dir($directory)) { + throw new RuntimeException("$directory is not a plain folder, so no credential was written there. Remove it."); + } + // Compared with whoever runs this (root on Unraid; the tests run as another user). + if (@fileowner($directory) !== (function_exists('posix_geteuid') ? posix_geteuid() : 0)) { + throw new RuntimeException("$directory belongs to another user, so no credential was written there. Remove it."); + } + if (!chmod($directory, 0700)) { + throw new RuntimeException("Could not make $directory private, so no credential was written there."); + } + return $directory; + } + + /** + * Remove files left behind by runs that were killed before they could clean up. + * + * Every run that uses one of these files is time-limited to well under an + * hour (GitCommand's and GitClone's timeouts), so a file older than that + * belongs to no run that is still going. + */ + private static function removeStaleFiles(string $directory): void + { + foreach (glob($directory . '/*') ?: [] as $file) { + $modified = @filemtime($file); + if ($modified === false || time() - $modified <= self::STALE_AFTER_SECONDS) { + continue; + } + self::remove($file); + } + } +} diff --git a/source/compose.manager/include/GitSsh.php b/source/compose.manager/include/GitSsh.php new file mode 100644 index 00000000..bd936865 --- /dev/null +++ b/source/compose.manager/include/GitSsh.php @@ -0,0 +1,202 @@ +succeeded() || !is_file($keyFile) || !is_file($keyFile . '.pub')) { + throw new RuntimeException('Could not make an ssh key: ' . $result->errorSummary()); + } + return [ + 'private' => (string) file_get_contents($keyFile), + 'public' => trim((string) file_get_contents($keyFile . '.pub')), + ]; + } finally { + @unlink($keyFile); + @unlink($keyFile . '.pub'); + GitCredentials::remove($file); + } + } + + /** + * The public key of a deploy key kept in the vault, to paste into the repository's settings. + * + * @throws RuntimeException if the credential is missing or not a deploy key + */ + public static function publicKey(string $credentialId): string + { + $keyFile = self::writeKeyFile($credentialId); + try { + $result = self::run(['ssh-keygen', '-y', '-f', $keyFile]); + if (!$result->succeeded()) { + throw new RuntimeException('Could not read the deploy key: ' . $result->errorSummary()); + } + return trim($result->stdout); + } finally { + GitCredentials::remove($keyFile); + } + } + + /** + * Ask a host for its keys, as known_hosts lines. + * + * @throws RuntimeException if the host does not answer + */ + public static function scanHostKeys(string $host, int $port): string + { + $result = self::run(['ssh-keyscan', '-T', '10', '-p', (string) $port, '-t', 'ed25519,ecdsa,rsa', $host]); + $lines = []; + foreach (explode("\n", $result->stdout) as $line) { + $line = trim($line); + if ($line !== '' && !str_starts_with($line, '#')) { + $lines[] = $line; + } + } + sort($lines); + if ($lines === []) { + throw new RuntimeException("Could not get the ssh host keys of $host (port $port). Is the address right?"); + } + return implode("\n", $lines) . "\n"; + } + + /** + * The SHA256 fingerprint of each pinned key, for a person to compare with the ones the host publishes. + * + * @return string[] Lines such as "SHA256:abc... (ED25519)" + */ + public static function fingerprints(string $knownHosts): array + { + $file = GitCredentials::writeRunFile($knownHosts); + try { + $result = self::run(['ssh-keygen', '-l', '-E', 'sha256', '-f', $file]); + $fingerprints = []; + foreach (explode("\n", trim($result->stdout)) as $line) { + // "256 SHA256:abc... host (ED25519)" + if (preg_match('/^\d+ (SHA256:\S+) .* (\([A-Z0-9-]+\))$/', trim($line), $match) === 1) { + $fingerprints[] = $match[1] . ' ' . $match[2]; + } + } + // Keys the person cannot compare must not be pinned: say so instead of showing nothing. + if ($fingerprints === []) { + throw new RuntimeException('Could not read the fingerprints of the ssh host keys, so nothing was pinned: ' . $result->errorSummary()); + } + return $fingerprints; + } finally { + GitCredentials::remove($file); + } + } + + /** + * Write a stack's deploy key and pinned host keys for one git run. + * + * @return array{key: string, knownHosts: string} The files; remove both with GitCredentials::remove() + * @throws RuntimeException if the key is missing or not a deploy key + */ + public static function writeRunFiles(string $credentialId, string $knownHosts): array + { + $keyFile = self::writeKeyFile($credentialId); + try { + $knownHostsFile = GitCredentials::writeRunFile($knownHosts); + } catch (Throwable $error) { + GitCredentials::remove($keyFile); + throw $error; + } + return ['key' => $keyFile, 'knownHosts' => $knownHostsFile]; + } + + /** + * The environment that makes git's ssh use only these files (for GitCommand::run). + * + * GIT_SSH_COMMAND goes through a shell, so it only ever holds the plugin's own + * file names, which are hex under a fixed folder. + * + * @param array{key: string, knownHosts: string} $files + * @return array + */ + public static function environmentFor(array $files): array + { + return [ + 'GIT_SSH_COMMAND' => implode(' ', [ + 'ssh', + '-F /dev/null', + '-i ' . escapeshellarg($files['key']), + '-o IdentitiesOnly=yes', + '-o BatchMode=yes', + '-o StrictHostKeyChecking=yes', + '-o UserKnownHostsFile=' . escapeshellarg($files['knownHosts']), + '-o GlobalKnownHostsFile=/dev/null', + '-o ConnectTimeout=30', + ]), + ]; + } + + /** + * The git settings for an ssh run: the ssh transport, which is otherwise not allowed. + * + * @return string[] + */ + public static function settings(): array + { + return ['protocol.ssh.allow=always']; + } + + private static function writeKeyFile(string $credentialId): string + { + $privateKey = (new CredentialVault())->useCredential( + $credentialId, + static function (array $credential): string { + if (($credential['provider'] ?? '') !== 'git-ssh') { + throw new RuntimeException("The credential '{$credential['name']}' is not an ssh deploy key."); + } + return $credential['secret']; + } + ); + // ssh needs the key file to end with a newline; the vault trims it off. + return GitCredentials::writeRunFile(rtrim($privateKey) . "\n"); + } + + /** + * @param string[] $command + */ + private static function run(array $command): ProcessResult + { + return ProcessRunner::run( + $command, + sys_get_temp_dir(), + ['PATH' => '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', 'HOME' => COMPOSE_GIT_HOME_DIR, 'LC_ALL' => 'C'], + self::TIMEOUT_SECONDS + ); + } +} diff --git a/source/compose.manager/include/GitStackManager.php b/source/compose.manager/include/GitStackManager.php index 847ff566..826283ac 100644 --- a/source/compose.manager/include/GitStackManager.php +++ b/source/compose.manager/include/GitStackManager.php @@ -8,6 +8,7 @@ require_once '/usr/local/emhttp/plugins/compose.manager/include/GitStackSettings.php'; require_once '/usr/local/emhttp/plugins/compose.manager/include/GitStackState.php'; require_once '/usr/local/emhttp/plugins/compose.manager/include/GitClone.php'; +require_once '/usr/local/emhttp/plugins/compose.manager/include/CredentialVault.php'; /** * Creates and looks after git stacks: what the compose-git command does, @@ -46,7 +47,8 @@ public function add( string $branch, string $composePath, ?string $clonesRoot, - string $description = '' + string $description = '', + ?string $credentialId = null ): string { $this->assertArrayStarted(); $folder = StackInfo::sanitizeProjectString($stackName); @@ -58,14 +60,14 @@ public function add( throw new RuntimeException("A stack folder named '$folder' already exists. Choose another name, or use convert for that stack."); } - $settings = GitStackSettings::createNew($url, $branch, $composePath, $clonesRoot ?? COMPOSE_GIT_DEFAULT_CLONES_ROOT, $folder); + $settings = $this->newSettings($url, $branch, $composePath, $clonesRoot, $folder, $credentialId); ($this->say)("Cloning $url ($branch) into {$settings->cloneDir}..."); - (new GitClone($settings))->create(); + $this->createClone($settings); try { $stack = StackInfo::createNew($this->composeRoot, $stackName, $description, $settings->composeFileInClone()); $stackDir = $this->composeRoot . '/' . $stack->projectFolder; - $settings->save($stackDir); + $settings->saveCheckingCredential($stackDir); (new GitStackState(null, null))->save($stackDir); } catch (Throwable $error) { // Nothing here deletes files, so name what was left: a retry makes a new clone, @@ -80,6 +82,9 @@ public function add( throw new RuntimeException($error->getMessage() . "\n" . $leftBehind, 0, $error); } ($this->say)("Created stack '{$stack->projectFolder}'. It is not deployed yet."); + if ($settings->credentialId !== null) { + $this->logCredentialChange($stack->projectFolder, $settings->credentialId); + } return $stack->projectFolder; } @@ -95,8 +100,14 @@ public function add( * @return string The backup folder * @throws RuntimeException|InvalidArgumentException naming why nothing was changed */ - public function convert(string $folder, string $url, string $branch, string $composePath, ?string $clonesRoot): string - { + public function convert( + string $folder, + string $url, + string $branch, + string $composePath, + ?string $clonesRoot, + ?string $credentialId = null + ): string { $this->assertArrayStarted(); $stack = $this->stack($folder); if ($stack->isGitStack()) { @@ -104,16 +115,16 @@ public function convert(string $folder, string $url, string $branch, string $com } $stackDir = $stack->path; - return $this->withStackLock($stack, function () use ($stack, $stackDir, $folder, $url, $branch, $composePath, $clonesRoot): string { + return $this->withStackLock($stack, function () use ($stack, $stackDir, $folder, $url, $branch, $composePath, $clonesRoot, $credentialId): string { $oldOverride = $stack->getOverridePath(); $oldEnv = $stack->getEffectiveEnvFilePath(); $hasExplicitEnvPath = trim((string) @file_get_contents($stackDir . '/envpath')) !== ''; $oldComposeFile = $stack->isIndirect ? null : $stack->composeFilePath; $foldersNextToOldComposeFile = $this->foldersIn($stack->composeSource); - $settings = GitStackSettings::createNew($url, $branch, $composePath, $clonesRoot ?? COMPOSE_GIT_DEFAULT_CLONES_ROOT, $folder); + $settings = $this->newSettings($url, $branch, $composePath, $clonesRoot, $folder, $credentialId); ($this->say)("Cloning $url ($branch) into {$settings->cloneDir}..."); - (new GitClone($settings))->create(); + $this->createClone($settings); // From here the stack folder changes; everything replaced goes into the backup. $backupDir = $stackDir . '/pre-git-' . date('Y-m-d_His'); @@ -159,7 +170,7 @@ public function convert(string $folder, string $url, string $branch, string $com ($this->say)("Copied $oldEnv to $stackEnv."); } - $settings->save($stackDir); + $settings->saveCheckingCredential($stackDir); (new GitStackState(null, null))->save($stackDir); // The managed override's name follows the compose file's name, so @@ -185,6 +196,9 @@ public function convert(string $folder, string $url, string $branch, string $com } ($this->say)("'$folder' is now a git stack. Its containers change at the next deploy."); + if ($settings->credentialId !== null) { + $this->logCredentialChange($folder, $settings->credentialId); + } if ($foldersNextToOldComposeFile !== []) { // The compose file is not parsed here; this is only a reminder. ($this->say)( @@ -242,7 +256,7 @@ public function check(string $folder): array /** * What is known about a git stack locally, without asking the remote. * - * @return array{stack: string, url: string, branch: string, composePath: string, cloneDir: string, recreateOnFolderChange: bool, deployedCommit: ?string, failedCommit: ?string, checkedOutCommit: ?string, localChanges: list, problem: ?string} + * @return array{stack: string, url: string, branch: string, composePath: string, cloneDir: string, recreateOnFolderChange: bool, credential: ?string, deployedCommit: ?string, failedCommit: ?string, checkedOutCommit: ?string, localChanges: list, problem: ?string} */ public function status(string $folder): array { @@ -266,6 +280,7 @@ public function status(string $folder): array 'composePath' => $settings->composePath, 'cloneDir' => $settings->cloneDir, 'recreateOnFolderChange' => $settings->recreateOnFolderChange, + 'credential' => $settings->credentialId === null ? null : $this->credentialName($settings->credentialId), 'deployedCommit' => $state->deployedCommit, 'failedCommit' => $state->failedCommit, 'checkedOutCommit' => $checkedOut, @@ -274,6 +289,262 @@ public function status(string $folder): array ]; } + /** + * Change the credential a git stack uses to reach its repository, or remove it. + * + * The repository is reached with the new setting first, and nothing is + * saved unless that works. + * + * @throws RuntimeException|InvalidArgumentException naming why nothing was changed + */ + public function setCredential(string $folder, ?string $credentialId): void + { + $this->assertArrayStarted(); + $stack = $this->stack($folder); + $settings = $this->settings($stack); + if ($settings->isSsh()) { + throw new InvalidArgumentException( + "An ssh stack always uses a deploy key of its own, so the credential of '$folder' cannot be changed. " + . 'compose-git deploy-key shows its public key.' + ); + } + + $this->withStackLock($stack, function () use ($stack, $settings, $credentialId): void { + $changed = $settings->withCredentialId($credentialId); + ($this->say)("Checking that {$changed->url} can be reached with the new setting..."); + (new GitClone($changed))->remoteBranchCommit(); + $changed->saveCheckingCredential($stack->path); + }); + $this->logCredentialChange($folder, $credentialId); + ($this->say)($credentialId === null + ? "'$folder' now reaches its repository without a credential." + : "'$folder' now uses the credential '" . $this->credentialName($credentialId) . "'."); + } + + /** + * The id of a git repository credential, given its exact name or its id. + * + * @throws RuntimeException if there is none, or several with that name + */ + public function findGitCredential(string $nameOrId): string + { + $gitCredentials = array_values(array_filter( + (new CredentialVault())->listCredentials(), + // HTTPS tokens only: a deploy key belongs to the one stack it was made for. + static fn(array $credential): bool => ($credential['provider'] ?? '') === 'git' + )); + $matches = array_values(array_filter( + $gitCredentials, + static fn(array $credential): bool => $credential['id'] === $nameOrId || $credential['name'] === $nameOrId + )); + if (count($matches) === 1) { + return $matches[0]['id']; + } + if (count($matches) > 1) { + throw new RuntimeException("Several git credentials are named '$nameOrId'. Use the credential's id instead."); + } + $names = array_map(static fn(array $credential): string => $credential['name'], $gitCredentials); + throw new RuntimeException( + "There is no git credential named '$nameOrId'. " + . ($names === [] ? 'Add one on the Credentials tab of the plugin settings first.' : 'Git credentials: ' . implode(', ', $names) . '.') + ); + } + + /** + * Test a git credential by reaching the repository of a git stack that uses it. + * + * @return array{id: string, name: string, registry: string, valid: bool, message: string}|null + * null when no git stack uses the credential, so there is no repository to try + */ + public function testCredential(string $credentialId): ?array + { + foreach ($this->listGitStacks() as $folder) { + try { + $settings = $this->settings($this->stack($folder)); + } catch (Throwable) { + continue; + } + if ($settings->credentialId !== $credentialId) { + continue; + } + $summary = (new CredentialVault())->getCredentialSummary($credentialId); + $result = ['id' => $credentialId, 'name' => $summary['name'], 'registry' => $summary['registry']]; + try { + (new GitClone($settings))->remoteBranchCommit(); + return $result + ['valid' => true, 'message' => "Reached {$settings->url} (used by '$folder')."]; + } catch (Throwable $error) { + return $result + ['valid' => false, 'message' => "Could not reach {$settings->url} (used by '$folder'): " . $error->getMessage()]; + } + } + return null; + } + + /** + * The public half of an ssh stack's deploy key, to add to the repository's settings. + * + * @throws RuntimeException if the stack is not an ssh stack + */ + public function deployKey(string $folder): string + { + $settings = $this->settings($this->stack($folder)); + if (!$settings->isSsh() || $settings->credentialId === null) { + throw new RuntimeException("'$folder' does not reach its repository over ssh, so it has no deploy key."); + } + return GitSsh::publicKey($settings->credentialId); + } + + /** + * Pin an ssh stack's repository host keys again, after the server's keys changed. + * + * The old and new fingerprints are shown, and the new keys are saved only + * after the repository has been reached with them. + * + * @throws RuntimeException naming why nothing was changed + */ + public function trustHost(string $folder): void + { + $this->assertArrayStarted(); + $stack = $this->stack($folder); + $settings = $this->settings($stack); + $address = GitStackSettings::sshAddress($settings->url); + if ($address === null) { + throw new RuntimeException("'$folder' does not reach its repository over ssh."); + } + + $this->withStackLock($stack, function () use ($stack, $settings, $address): void { + $knownHosts = GitSsh::scanHostKeys($address['host'], $address['port']); + ($this->say)('Pinned now: ' . implode(', ', GitSsh::fingerprints((string) $settings->sshKnownHosts))); + ($this->say)('Host offers: ' . implode(', ', GitSsh::fingerprints($knownHosts))); + if ($knownHosts === $settings->sshKnownHosts) { + ($this->say)('The host keys have not changed.'); + return; + } + ($this->say)('Compare the new fingerprints with the ones your git host publishes before you deploy.'); + $changed = $settings->withSshKnownHosts($knownHosts); + (new GitClone($changed))->remoteBranchCommit(); + $changed->save($stack->path); + ($this->say)('The new host keys are pinned.'); + }); + } + + /** + * Settings for a new git stack. For an ssh repository this also makes (or + * reuses) the stack's deploy key and pins the host's keys. + */ + private function newSettings( + string $url, + string $branch, + string $composePath, + ?string $clonesRoot, + string $folder, + ?string $credentialId + ): GitStackSettings { + $settings = GitStackSettings::createNew($url, $branch, $composePath, $clonesRoot ?? COMPOSE_GIT_DEFAULT_CLONES_ROOT, $folder); + if (!$settings->isSsh()) { + return $settings->withCredentialId($credentialId); + } + + $address = GitStackSettings::sshAddress($url); + if ($address === null) { + throw new InvalidArgumentException("The ssh repository address is not valid: $url"); + } + if ($credentialId !== null) { + throw new InvalidArgumentException('An ssh stack gets a deploy key of its own, so it cannot be given another credential.'); + } + // The host first: a mistyped host then fails before a deploy key is made for it. + $knownHosts = GitSsh::scanHostKeys($address['host'], $address['port']); + $credentialId = $this->deployKeyFor($folder, $address); + ($this->say)("Pinned the ssh host keys of {$address['host']}: " . implode(', ', GitSsh::fingerprints($knownHosts))); + ($this->say)('Compare them with the fingerprints your git host publishes.'); + return $settings->withCredentialId($credentialId)->withSshKnownHosts($knownHosts); + } + + /** + * The stack's deploy key: the one made for it by an earlier attempt, or a new one. + * + * @param array{user: string, host: string, port: int, path: string} $address + */ + private function deployKeyFor(string $folder, array $address): string + { + $name = "$folder deploy key"; + $host = $address['host'] . ($address['port'] === 22 ? '' : ':' . $address['port']); + foreach ((new CredentialVault())->listCredentials() as $credential) { + if ($credential['name'] === $name && $credential['provider'] === 'git-ssh' && $credential['registry'] === $host) { + ($this->say)("Using the deploy key made for '$folder' earlier."); + return $credential['id']; + } + } + $key = GitSsh::generateKey("compose-manager $folder"); + ($this->say)("Made a new deploy key for '$folder'."); + $id = (new CredentialVault())->saveCredential([ + 'name' => $name, + 'provider' => 'git-ssh', + 'registry' => $host, + 'username' => $address['user'], + 'secret' => $key['private'], + ])['id']; + composeLogger("Made a deploy key for git stack '$folder'", ['id' => $id, 'host' => $host], 'user', 'info', 'credentials'); + return $id; + } + + /** + * Clone a new stack's repository. When an ssh clone fails, show the deploy + * key: the usual reason is that it has not been added to the repository yet. + */ + private function createClone(GitStackSettings $settings): void + { + try { + (new GitClone($settings))->create(); + } catch (RuntimeException $error) { + if (!$settings->isSsh() || $settings->credentialId === null) { + throw $error; + } + try { + $publicKey = GitSsh::publicKey($settings->credentialId); + } catch (RuntimeException) { + // Not a deploy key at all (the wrong --credential): the clone's own message says so. + throw $error; + } + throw new RuntimeException( + $error->getMessage() . "\n\nIf the repository does not know this stack's deploy key yet, add it as a " + . "read-only deploy key in the repository's settings, then run the same command again:\n" + . $publicKey, + 0, + $error + ); + } + } + + /** + * A credential's name for messages, or a note that it is gone. + */ + private function credentialName(string $credentialId): string + { + try { + return (new CredentialVault())->getCredentialSummary($credentialId)['name']; + } catch (Throwable) { + return "$credentialId (no longer in the credential vault)"; + } + } + + /** + * Record in the syslog which credential a git stack now uses, as the web UI's credential actions do. + */ + private function logCredentialChange(string $folder, ?string $credentialId): void + { + if ($credentialId === null) { + composeLogger("Git stack '$folder' no longer uses a credential", null, 'user', 'info', 'credentials'); + return; + } + composeLogger( + "Git stack '$folder' now uses the credential '" . $this->credentialName($credentialId) . "'", + ['id' => $credentialId], + 'user', + 'info', + 'credentials' + ); + } + /** * Replace a stack's clone with a fresh one at the deployed commit. * diff --git a/source/compose.manager/include/GitStackSettings.php b/source/compose.manager/include/GitStackSettings.php index 2c79bf92..aa313cfd 100644 --- a/source/compose.manager/include/GitStackSettings.php +++ b/source/compose.manager/include/GitStackSettings.php @@ -5,6 +5,7 @@ require_once '/usr/local/emhttp/plugins/compose.manager/include/Defines.php'; require_once '/usr/local/emhttp/plugins/compose.manager/include/GitPathGuard.php'; require_once '/usr/local/emhttp/plugins/compose.manager/include/GitCommand.php'; +require_once '/usr/local/emhttp/plugins/compose.manager/include/CredentialVault.php'; /** * The git settings of one git-backed stack, stored as git.json in the stack folder. @@ -16,9 +17,11 @@ * Changing the format later. Every git.json any released version wrote must * keep loading, and must keep meaning what it meant: * - The keys of format 1 (KEYS) are always required. - * - A key added later may be missing from a file. It then takes a default - * that does exactly what the plugin did before the key existed, or less, - * never more: a stack written by an older version must behave as it did. + * - A key added later (OPTIONAL_KEYS) may be missing from a file. It then + * takes a default that does exactly what the plugin did before the key + * existed, or less, never more: a stack written by an older version must + * behave as it did. save() writes it only when it differs from that + * default, so a stack that never uses it still loads in an older version. * - A key this code does not know is refused, naming the key: it was * written by a newer version (after a downgrade), and ignoring a setting * could make the stack do something its owner turned off. @@ -33,13 +36,21 @@ final class GitStackSettings public const FILE_NAME = 'git.json'; public const FORMAT_VERSION = 1; - /** @var string[] The keys git.json must contain, no more and no fewer. */ + /** @var string[] The keys git.json must contain (format 1). */ private const KEYS = ['version', 'url', 'branch', 'composePath', 'cloneId', 'cloneDir', 'recreateOnFolderChange']; + /** @var string[] Keys added later, which a file may leave out. */ + private const OPTIONAL_KEYS = ['credentialId', 'sshKnownHosts']; + /** * @param bool $recreateOnFolderChange Recreate the stack's containers when a deploy changes * anything in the stack's folder, not only when the compose definition changes. A commit * that only edits a bind-mounted config file then takes effect. + * @param string|null $credentialId The credential vault entry used to reach the repository, or + * null for a repository that needs none (the default: format 1 had no credentials). For an + * ssh repository, the stack's deploy key. + * @param string|null $sshKnownHosts The ssh repository host's keys, pinned when the stack was + * set up, in known_hosts format; null for an https or local repository. */ private function __construct( public readonly string $url, @@ -47,7 +58,9 @@ private function __construct( public readonly string $composePath, public readonly string $cloneId, public readonly string $cloneDir, - public readonly bool $recreateOnFolderChange + public readonly bool $recreateOnFolderChange, + public readonly ?string $credentialId = null, + public readonly ?string $sshKnownHosts = null ) { } @@ -75,7 +88,62 @@ public static function createNew( */ public function withRecreateOnFolderChange(bool $recreateOnFolderChange): self { - return new self($this->url, $this->branch, $this->composePath, $this->cloneId, $this->cloneDir, $recreateOnFolderChange); + return new self( + $this->url, + $this->branch, + $this->composePath, + $this->cloneId, + $this->cloneDir, + $recreateOnFolderChange, + $this->credentialId, + $this->sshKnownHosts + ); + } + + /** + * The same settings with another credential, or none. + * + * @throws InvalidArgumentException if the id is not a credential vault id + */ + public function withCredentialId(?string $credentialId): self + { + return self::fromValues( + $this->url, + $this->branch, + $this->composePath, + $this->cloneId, + $this->cloneDir, + $this->recreateOnFolderChange, + $credentialId, + $this->sshKnownHosts + ); + } + + /** + * The same settings with other pinned ssh host keys. + * + * @throws InvalidArgumentException if they are not known_hosts lines, or the repository is not ssh + */ + public function withSshKnownHosts(?string $sshKnownHosts): self + { + return self::fromValues( + $this->url, + $this->branch, + $this->composePath, + $this->cloneId, + $this->cloneDir, + $this->recreateOnFolderChange, + $this->credentialId, + $sshKnownHosts + ); + } + + /** + * Whether the repository is reached over ssh (with a deploy key). + */ + public function isSsh(): bool + { + return self::isSshUrl($this->url); } /** @@ -107,7 +175,7 @@ public static function load(string $stackDir): ?self throw new RuntimeException("$file is not a valid git settings file (expected a JSON object)."); } - $unknown = array_values(array_diff(array_keys($data), self::KEYS)); + $unknown = array_values(array_diff(array_keys($data), self::KEYS, self::OPTIONAL_KEYS)); if ($unknown !== []) { throw new RuntimeException( "$file has settings this version of the plugin does not know: " . implode(', ', $unknown) @@ -132,6 +200,14 @@ public static function load(string $stackDir): ?self if (!is_bool($data['recreateOnFolderChange'])) { throw new RuntimeException("$file: 'recreateOnFolderChange' must be true or false."); } + $credentialId = $data['credentialId'] ?? null; + if ($credentialId !== null && !is_string($credentialId)) { + throw new RuntimeException("$file: 'credentialId' must be a string."); + } + $sshKnownHosts = $data['sshKnownHosts'] ?? null; + if ($sshKnownHosts !== null && !is_string($sshKnownHosts)) { + throw new RuntimeException("$file: 'sshKnownHosts' must be a string."); + } try { return self::fromValues( @@ -140,13 +216,36 @@ public static function load(string $stackDir): ?self $data['composePath'], $data['cloneId'], $data['cloneDir'], - $data['recreateOnFolderChange'] + $data['recreateOnFolderChange'], + $credentialId, + $sshKnownHosts ); } catch (InvalidArgumentException $error) { throw new RuntimeException("$file: " . $error->getMessage(), 0, $error); } } + /** + * Write git.json when it gives the stack a credential (a new stack, or a changed + * credential): check the credential still exists and write, both under the lock + * that deleting a credential takes. The repository was reached before this, which + * can take a while, and the credential may have been deleted meanwhile; it was not + * in use yet, so the delete was allowed. The lock is not held across the network. + * + * @throws RuntimeException if the credential is gone, or git.json cannot be written + */ + public function saveCheckingCredential(string $stackDir): void + { + CredentialVault::withCredentialAssignmentLock(function () use ($stackDir): void { + if ($this->credentialId !== null && !(new CredentialVault())->hasCredential($this->credentialId)) { + throw new RuntimeException( + 'The credential was deleted while the repository was being reached, so the stack settings were not saved.' + ); + } + $this->save($stackDir); + }); + } + /** * Write git.json to the stack folder, replacing it in one step. * @@ -163,7 +262,7 @@ public function save(string $stackDir): void throw new RuntimeException("Stack folder $stackDir does not exist."); } - $json = json_encode([ + $data = [ 'version' => self::FORMAT_VERSION, 'url' => $this->url, 'branch' => $this->branch, @@ -171,7 +270,15 @@ public function save(string $stackDir): void 'cloneId' => $this->cloneId, 'cloneDir' => $this->cloneDir, 'recreateOnFolderChange' => $this->recreateOnFolderChange, - ], JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES); + ]; + // Keys added later are written only when they differ from their default (see the class comment). + if ($this->credentialId !== null) { + $data['credentialId'] = $this->credentialId; + } + if ($this->sshKnownHosts !== null) { + $data['sshKnownHosts'] = $this->sshKnownHosts; + } + $json = json_encode($data, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES); if ($json === false) { throw new RuntimeException('Could not encode the git settings.'); } @@ -217,9 +324,10 @@ public static function cloneFolderName(string $stackFolderName, string $cloneId) /** * Check a repository address. * - * Accepted: an https URL without a user name or password in it, or the - * absolute path of a repository under /mnt (for a bare repository on this - * server that you push to). + * Accepted: an https URL without a user name or password in it, an ssh + * address (ssh://git@host[:port]/path or git@host:path, reached with the + * stack's deploy key), or the absolute path of a repository under /mnt (for + * a bare repository on this server that you push to). * * @throws InvalidArgumentException naming the problem */ @@ -235,6 +343,17 @@ public static function validateUrl(string $url): void throw new InvalidArgumentException('The repository address must not contain spaces or control characters.'); } + if (self::isSshUrl($url)) { + // Each part is held to plain characters: the host and user reach ssh's command line, + // where a value starting with "-" would be read as an option. + if (self::sshAddress($url) === null) { + throw new InvalidArgumentException( + "The ssh repository address is not valid. Use ssh://git@host[:port]/path or git@host:path: $url" + ); + } + return; + } + if (str_starts_with($url, '/')) { GitPathGuard::assertCleanAbsolutePath($url, 'The repository path'); if (!str_starts_with($url, rtrim(COMPOSE_GIT_MNT_DIR, '/') . '/')) { @@ -248,8 +367,8 @@ public static function validateUrl(string $url): void // Lower case only: git matches the transport name exactly, and refuses "HTTPS". if (!str_starts_with($url, 'https://')) { throw new InvalidArgumentException( - "The repository address must start with https://, or be a path under " . COMPOSE_GIT_MNT_DIR - . " for a repository on this server: $url" + "The repository address must start with https:// or ssh:// (or be git@host:path), or be a path under " + . COMPOSE_GIT_MNT_DIR . " for a repository on this server: $url" ); } @@ -260,7 +379,7 @@ public static function validateUrl(string $url): void if (isset($parts['user']) || isset($parts['pass'])) { throw new InvalidArgumentException( 'The repository address must not contain a user name, password or token ' - . '(it would be stored in plain text). Private repositories are not supported yet.' + . '(it would be stored in plain text). For a private repository, use a git credential instead.' ); } if (isset($parts['query']) || isset($parts['fragment'])) { @@ -275,6 +394,60 @@ public static function validateUrl(string $url): void } } + /** + * Whether an address is meant as ssh: ssh://..., or the user@host:path form. + */ + public static function isSshUrl(string $url): bool + { + return str_starts_with(strtolower($url), 'ssh://') + || preg_match('#^[^/:@]+@[^/:]+:#', $url) === 1; + } + + /** + * The parts of an ssh repository address, or null when it is not a valid one. + * + * The user, host and port go to ssh, so each is held to plain characters, and + * none can start with "-". The path is what the server is asked for. + * + * @return array{user: string, host: string, port: int, path: string}|null + */ + public static function sshAddress(string $url): ?array + { + $user = '[A-Za-z0-9_][A-Za-z0-9._-]*'; + $host = '(?:[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?\.)*[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?'; + $path = '[A-Za-z0-9._~+][A-Za-z0-9._~+/-]*'; + if (preg_match("#^ssh://($user)@($host)(?::([0-9]{1,5}))?/($path)$#i", $url, $match) === 1) { + $port = $match[3] === '' ? 22 : (int) $match[3]; + } elseif (preg_match("#^($user)@($host):(/?$path)$#", $url, $match) === 1) { + $port = 22; + $match[4] = $match[3]; + } else { + return null; + } + if ($port < 1 || $port > 65535 || str_contains($match[4], '..')) { + return null; + } + return ['user' => $match[1], 'host' => strtolower($match[2]), 'port' => $port, 'path' => $match[4]]; + } + + /** + * Check pinned ssh host keys: one or more known_hosts lines, "host type key". + * + * @throws InvalidArgumentException naming the problem + */ + public static function validateKnownHosts(string $knownHosts): void + { + $lines = array_filter(explode("\n", trim($knownHosts)), static fn(string $line): bool => $line !== ''); + if ($lines === []) { + throw new InvalidArgumentException('The pinned ssh host keys are empty.'); + } + foreach ($lines as $line) { + if (preg_match('#^[A-Za-z0-9.:\[\]_-]+ (?:ssh-ed25519|ssh-rsa|ecdsa-sha2-nistp(?:256|384|521)) [A-Za-z0-9+/]+={0,2}$#', $line) !== 1) { + throw new InvalidArgumentException("A pinned ssh host key is not a known_hosts line: $line"); + } + } + } + /** * Check a branch name, using git's own rules. * @@ -343,6 +516,18 @@ public static function validateComposePath(string $composePath): void } } + /** + * Check a credential vault id: null for none, or the 32 hex characters the vault gives every entry. + * + * @throws InvalidArgumentException naming the problem + */ + public static function validateCredentialId(?string $credentialId): void + { + if ($credentialId !== null && preg_match('/^[0-9a-f]{32}$/', $credentialId) !== 1) { + throw new InvalidArgumentException("The credential id is not valid: $credentialId"); + } + } + /** * Validate every value and build the settings. * @@ -354,7 +539,9 @@ private static function fromValues( string $composePath, string $cloneId, string $cloneDir, - bool $recreateOnFolderChange + bool $recreateOnFolderChange, + ?string $credentialId = null, + ?string $sshKnownHosts = null ): self { self::validateUrl($url); self::validateBranch($branch); @@ -371,6 +558,17 @@ private static function fromValues( throw new InvalidArgumentException("The clone folder $cloneDir does not belong to clone id $cloneId."); } - return new self($url, $branch, $composePath, $cloneId, $cloneDir, $recreateOnFolderChange); + self::validateCredentialId($credentialId); + if ($credentialId !== null && str_starts_with($url, '/')) { + throw new InvalidArgumentException('A repository on this server needs no credential.'); + } + if ($sshKnownHosts !== null) { + if (!self::isSshUrl($url)) { + throw new InvalidArgumentException('Pinned ssh host keys are only for an ssh repository.'); + } + self::validateKnownHosts($sshKnownHosts); + } + + return new self($url, $branch, $composePath, $cloneId, $cloneDir, $recreateOnFolderChange, $credentialId, $sshKnownHosts); } } diff --git a/source/compose.manager/javascript/composeManagerMain.js b/source/compose.manager/javascript/composeManagerMain.js index 9ec83c6b..bc9aa7d3 100644 --- a/source/compose.manager/javascript/composeManagerMain.js +++ b/source/compose.manager/javascript/composeManagerMain.js @@ -1303,7 +1303,7 @@ function initEditorModal() { ComposeCredentialManager.open(null, function(credential) { ComposeCredentialManager.populateSelect($('#settings-credential-id'), credential.id); $('#settings-credential-id').trigger('change'); - }); + }, { registryOnly: true }); }); // Additional compose files: combined change tracking for the candidate diff --git a/source/compose.manager/javascript/credentialManager.js b/source/compose.manager/javascript/credentialManager.js index 9e5ce221..4254a328 100644 --- a/source/compose.manager/javascript/credentialManager.js +++ b/source/compose.manager/javascript/credentialManager.js @@ -102,6 +102,31 @@ tokenUrlLabel: '', instructions: 'Enter your custom registry hostname (e.g. registry.example.com), username, and authentication token or password.', canOAuth: false + }, + git: { + label: 'Git repository (HTTPS token, for git stacks)', + defaultRegistry: '', + registryReadonly: false, + namePlaceholder: 'e.g. GitHub stacks repository', + usernamePlaceholder: 'Git user name (any value for a GitHub token)', + secretPlaceholder: 'Read-only access token', + tokenUrl: '', + tokenUrlLabel: '', + instructions: 'For a private repository that a git stack deploys from. Set the host to the git server only, such as github.com or git.example.com:3000. Use a token that can only read the repository, such as a GitHub fine-grained token with read access to its contents. Give it to a stack with compose-git (--credential).', + canOAuth: false + }, + 'git-ssh': { + // Made by compose-git for one git stack; never created or edited here. + label: 'SSH deploy key (git stack)', + defaultRegistry: '', + registryReadonly: true, + namePlaceholder: '', + usernamePlaceholder: '', + secretPlaceholder: '', + tokenUrl: '', + tokenUrlLabel: '', + instructions: '', + canOAuth: false } }; @@ -110,7 +135,7 @@ $('body').append( '