Implement manage destroy subcommand

docs/commands/manage.md

@@ -41,18 +41,67 @@ stack manage --dir DEPLOYMENT_DIR start [OPTIONS] [EXTRA_ARGS]...
 
 ### stop
 
-Stop the stack and remove the containers
+Stop the stack and remove the containers.
+
+`stop` is the symmetric opposite of `start`: it deletes only what `start` makes
+again from the deployment directory. Volumes, the namespace on Kubernetes, and
+the cluster on kind are all left in place, so a later `start` finds the data
+where it left it. Use [`destroy`](#destroy) when a deployment is finished.
+
+```bash
+stack manage --dir DEPLOYMENT_DIR stop [EXTRA_ARGS]...
+```
+
+`--delete-volumes` used to be how a finished deployment was cleaned up. It is
+rejected now rather than ignored, so that a script asking for deletion is told
+where deletion moved to instead of quietly leaking the volumes it meant to
+reclaim.
+
+### destroy
+
+Destroy the deployment: the signal that it is finished and will not be started
+again, which is what makes it safe to remove the things `stop` keeps precisely
+because `start` would want them back.
 
 ```bash
-stack manage --dir DEPLOYMENT_DIR stop [OPTIONS] [EXTRA_ARGS]...
+stack manage --dir DEPLOYMENT_DIR destroy [OPTIONS]
 ```
 
+On Kubernetes that is the deployment's PersistentVolumeClaims, the cluster-scoped
+PersistentVolumes a namespace delete does not reach, and the namespace itself; on
+kind, the cluster; on Docker, the compose project's volume objects. As everywhere
+else in stack, deleting a volume deletes the volume object and never the contents
+of a bind-mounted directory (see [volumes.md](../volumes.md)).
+
+Two things deliberately survive, and `destroy` says so rather than leaving you to
+wonder:
+
+- **Backups.** The restic/K8up repository is untouched: backups exist to outlive
+  the deployment that made them (see [backup.md](../backup.md)).
+- **The TLS certificate**, on a Gateway-provisioned cluster. Certificates are
+  keyed by hostname, so redeploying the same hostname reuses the one already
+  issued instead of asking Let's Encrypt for another — whose duplicate limit is
+  five a week for the same name. `--delete-certificate` overrides this for a
+  hostname you are retiring for good.
+
+  Certificates that no listener has referenced for a full certificate lifetime
+  are collected automatically as part of `destroy`: nothing renews an
+  unreferenced certificate, so one that has been idle that long is expired and
+  of no use to any future deployment.
+
+The deployment directory is left where it is — it is yours, and it is what a
+replacement deployment would be created from — but a `destroyed` marker is
+written into it, and the other `manage` subcommands refuse a directory carrying
+one rather than reporting on objects that no longer exist.
+
 #### Options
 
 | Option | Type | Description | Default |
 |--------|------|-------------|---------|
-| `--delete-volumes/--preserve-volumes` | FLAG | Delete data volumes | False |
-| `--skip-cluster-management/--perform-cluster-management` | FLAG | Skip cluster initialization/tear-down (kind-k8s only) | False |
+| `--yes` / `-y` | FLAG | Do not prompt for confirmation | False |
+| `--delete-volumes/--preserve-volumes` | FLAG | Delete the deployment's volumes (and, on k8s, its namespace) | True |
+| `--delete-certificate` | FLAG | Also delete the TLS certificate issued for this deployment's hostname | False |
+| `--skip-cluster-management/--perform-cluster-management` | FLAG | Skip cluster tear-down (kind-k8s only) | False |
 
 ### ps
 
@@ -292,11 +341,14 @@ stack manage --dir ~/deployments/my-stack start
 # Start and stay attached to see output
 stack manage --dir ~/deployments/my-stack start --stay-attached
 
-# Stop a stack (preserve volumes)
+# Stop a stack; its data stays where it is
 stack manage --dir ~/deployments/my-stack stop
 
-# Stop and delete volumes
-stack manage --dir ~/deployments/my-stack stop --delete-volumes
+# Finished with it: stop it for the last time and collect what it leaves
+stack manage --dir ~/deployments/my-stack destroy
+
+# The same without the prompt, keeping the volumes
+stack manage --dir ~/deployments/my-stack destroy --yes --preserve-volumes
 ```
 
 ### Monitoring and Debugging

docs/developing-applications.md

@@ -138,9 +138,9 @@ the pods. Use it to check the Kubernetes *shape* of a deployment — pods, volum
 without a real cluster; a change to the shape itself (ports, volumes, services) is the one
 thing `update` refuses, and there the loop is still `stop` then `start`.
 
-Note that `stop` deletes the kind cluster and `start` builds a new one, so nothing kept
-inside the cluster survives that loop. Your data does, because `init` maps each volume to a
-directory under the deployment:
+Note that `destroy` deletes the kind cluster and the next `start` builds a new one, so
+nothing kept inside the cluster survives that. Your data does, because `init` maps each
+volume to a directory under the deployment:
 
 ```yaml
 volumes:
@@ -149,7 +149,7 @@ volumes:
 
 which is bind mounted into the kind node, exactly as it is bind mounted into the container on
 compose. The database you were working against is still there after a `stop`/`start`. To
-start from an empty one, `stop --delete-volumes`, or delete the directory.
+start from an empty one, delete the directory.
 
 A remote cluster is the one target where this cannot work, since the data would have to live
 on the cluster's nodes: there `init` leaves the volume unmapped and it becomes a PVC from the

docs/gateway-api.md

@@ -48,6 +48,16 @@ For a spec with an `http-proxy` section, `stack manage start` creates:
 behind deliberately: redeploying the same hostname reuses the still-valid certificate instead of asking
 Let's Encrypt for a new one.
 
+`stack manage destroy` leaves it too, for the same reason — a destroyed deployment's hostname is often
+redeployed, and a certificate is worth more to the next deployment than the few kilobytes it occupies are worth
+to the cluster. `--delete-certificate` overrides that for a hostname being retired for good. What `destroy`
+does collect is certificates that are past being useful to anyone: a secret no listener has referenced for a
+full certificate lifetime holds an expired certificate, since cert-manager's Certificate object goes with the
+listener and nothing renews an unreferenced one. The first sweep to find such a secret marks it with
+`stack.bozemanpass.com/certificate-unreferenced-since` and a later one deletes it; serving the hostname again
+clears the mark, so the interval measured is always the current one. Measuring it, rather than reading
+`notAfter` out of the certificate, keeps an X.509 parser (and a dependency) out of stack for the sake of a date.
+
 Naming those objects after the hostname rather than after the deployment is what makes that reuse work. A
 deployment id changes whenever the stack is re-`init`ed, and a new secret name means cert-manager sees no
 certificate to reuse and places a fresh ACME order; Let's Encrypt issues five certificates per hostname per 168

docs/test-coverage.md

@@ -53,6 +53,7 @@ file listing is its own index.
 | `manage update`: data survives the in-place update | [`tests/app-deploy/run-test.sh`](../tests/app-deploy/run-test.sh) | `deploy update storage` | compose + kind per-PR; remote + remote-compose weekly |
 | `manage update`: rebuilt image content reaches the deployment | [`tests/app-deploy/run-test.sh`](../tests/app-deploy/run-test.sh) | `deploy update content` | compose + kind per-PR; remote + remote-compose weekly |
 | Spec-mapped volume path: pre-existing host data reaches the container | [`tests/volumes/run-test.sh`](../tests/volumes/run-test.sh) | `external data visible test`, `unmapped volume fresh test`, `volume write-back test` | compose + kind per-PR; remote weekly |
+| `manage destroy`: the deployment is finished; later `manage` commands refuse its directory | [`tests/smoke-test/run-smoke-test.sh`](../tests/smoke-test/run-smoke-test.sh) | `deploy destroy` | compose, per-PR |
 | `manage exec` against a running service | [`tests/database/run-backup-test.sh`](../tests/database/run-backup-test.sh) | `Replay dump test` | compose per-PR; remote weekly |
 
 ## Backup and restore

src/stack/constants.py

@@ -41,6 +41,10 @@
 # that the same stack produces a snapshot of the same name whichever engine took it.
 backup_default_file_extension = "dump"
 deployment_file_name = "deployment.yml"
+# Written by `manage destroy` to record that this deployment's cluster objects are
+# gone, so that the rest of `manage` refuses a directory that no longer describes
+# anything running.
+destroyed_file_name = "destroyed"
 host_name_key = "host-name"
 http_proxy_key = "http-proxy"
 http_proxy_prefix_key = "http-proxy-prefix"

src/stack/deploy/compose/deploy_docker.py

@@ -111,10 +111,25 @@ def up(self, detach, skip_cluster_management, services):
             except DockerException as e:
                 raise DeployerException(e)
 
-    def down(self, timeout, volumes, skip_cluster_management):
+    def down(self, timeout):
         if not opts.o.dry_run:
             try:
-                return self.docker.compose.down(timeout=timeout, volumes=volumes)
+                return self.docker.compose.down(timeout=timeout)
+            except DockerException as e:
+                raise DeployerException(e)
+
+    def destroy(self, timeout, delete_volumes, delete_certificate, skip_cluster_management):
+        """Stop the deployment and remove its volume objects.
+
+        There is no certificate of stack's own to collect here: TLS on this
+        target is the docker-ingress stack's business, and its certificates live
+        in its own volume (see docs/ingress.md).  Removing a named volume
+        removes the volume object, never a bind-mounted directory's contents --
+        the same thing it has always meant on this target.
+        """
+        if not opts.o.dry_run:
+            try:
+                return self.docker.compose.down(timeout=timeout, volumes=delete_volumes)
             except DockerException as e:
                 raise DeployerException(e)
 

src/stack/deploy/deploy.py

@@ -85,14 +85,22 @@ def up_operation(ctx, services_list, stay_attached=False, skip_cluster_managemen
     )
 
 
-def down_operation(ctx, delete_volumes, extra_args_list, skip_cluster_management=False):
+def down_operation(ctx, extra_args_list):
     timeout_arg = None
     if extra_args_list:
         timeout_arg = extra_args_list[0]
     # Specify shutdown timeout (default 10s) to give services enough time to shutdown gracefully
-    ctx.obj.deployer.down(
+    ctx.obj.deployer.down(timeout=timeout_arg)
+
+
+def destroy_operation(ctx, delete_volumes, delete_certificate, extra_args_list, skip_cluster_management=False):
+    timeout_arg = None
+    if extra_args_list:
+        timeout_arg = extra_args_list[0]
+    ctx.obj.deployer.destroy(
         timeout=timeout_arg,
-        volumes=delete_volumes,
+        delete_volumes=delete_volumes,
+        delete_certificate=delete_certificate,
         skip_cluster_management=skip_cluster_management,
     )
 

src/stack/deploy/deployer.py

@@ -23,7 +23,23 @@ def up(self, detach, skip_cluster_management, services):
         pass
 
     @abstractmethod
-    def down(self, timeout, volumes, skip_cluster_management):
+    def down(self, timeout):
+        """Stop the deployment.
+
+        Symmetric with up(): whatever this deletes, up() has to be able to make
+        again.  Nothing that holds data is touched -- that is destroy's job.
+        """
+        pass
+
+    @abstractmethod
+    def destroy(self, timeout, delete_volumes, delete_certificate, skip_cluster_management):
+        """Stop the deployment for the last time and collect what it leaves.
+
+        The signal that a deployment is finished, which is what makes it safe to
+        remove the things stop keeps precisely because start would want them
+        back.  Backups are not among them: they exist to outlive the deployment
+        that made them (see docs/backup.md).
+        """
         pass
 
     @abstractmethod
@@ -106,11 +122,11 @@ def __init__(self, *args: object) -> None:
 class ClusterNotRunningException(DeployerException):
     """There is no cluster to talk to, so nothing of the deployment is running.
 
-    Only a kind deployment reaches this state: stopping one deletes the whole
+    Only a kind deployment reaches this state: destroying one deletes the whole
     cluster, so afterwards there is no kube context left to connect to.  That is
-    the normal resting state of a stopped kind deployment rather than a fault,
-    which is why it is distinguishable -- the commands that report what is
-    running answer "nothing" instead of failing.
+    the resting state of a destroyed kind deployment rather than a fault, which
+    is why it is distinguishable -- the commands that report what is running
+    answer "nothing" instead of failing.
     """
 
 

src/stack/deploy/deployment.py

@@ -16,6 +16,7 @@
 
 import click
 
+from datetime import datetime, timezone
 from pathlib import Path
 
 from stack import constants
@@ -26,6 +27,7 @@
     ps_operation,
     port_operation,
     status_operation,
+    destroy_operation,
 )
 from stack.deploy.deploy import (
     exec_operation,
@@ -41,6 +43,7 @@
 )
 from stack.deploy.deploy_types import DeployCommandContext
 from stack.deploy.deployment_context import DeploymentContext
+from stack.deploy.backup import backup_settings
 from stack.deploy.explain import explain_op
 from stack.log import output_main
 from stack.util import error_exit, get_yaml
@@ -58,6 +61,16 @@ def command(ctx, dir):
         error_exit(f"Error: deployment directory {dir} does not exist")
     if not dir_path.is_dir():
         error_exit(f"Error: supplied deployment directory path {dir} exists but is a file not a directory")
+    # A destroyed deployment's directory still describes a deployment, but nothing
+    # it describes exists any more, so every command here but destroy itself would
+    # be answering about something that is gone.  Destroy stays available so that
+    # an interrupted one can be run again.
+    destroyed_marker = dir_path.joinpath(constants.destroyed_file_name)
+    if destroyed_marker.exists() and ctx.invoked_subcommand != "destroy":
+        error_exit(
+            f"Error: deployment {dir} was destroyed ({destroyed_marker.read_text().strip()}). "
+            "Create a new deployment with `stack deploy`."
+        )
     # Store the deployment context for subcommands
     deployment_context = DeploymentContext()
     deployment_context.init(dir_path)
@@ -114,19 +127,74 @@ def start(ctx, stay_attached, skip_cluster_management, extra_args):
 
 
 @command.command()
-@click.option("--delete-volumes/--preserve-volumes", default=False, help="delete data volumes")
+@click.option("--delete-volumes", is_flag=True, default=False, hidden=True)
+@click.argument("extra_args", nargs=-1)  # help: command: down <service1> <service2>
+@click.pass_context
+def stop(ctx, delete_volumes, extra_args):
+    """stop the deployment and remove the containers"""
+    # Stop is the symmetric opposite of start and deletes nothing that start
+    # cannot make again.  --delete-volumes used to be how a finished deployment
+    # was cleaned up; that is what destroy is for now.  It is still accepted so
+    # that a script asking for deletion is told, rather than quietly leaking the
+    # volumes it meant to reclaim.
+    if delete_volumes:
+        error_exit("Error: stop no longer deletes volumes. Use `stack manage --dir <dir> destroy` instead.")
+    # TODO: add cluster name and env file here
+    ctx.obj = make_deploy_context(ctx)
+    down_operation(ctx, extra_args)
+
+
+@command.command()
+@click.option("--yes", "-y", is_flag=True, default=False, help="do not prompt for confirmation")
+@click.option(
+    "--delete-volumes/--preserve-volumes",
+    default=True,
+    help="delete the deployment's volumes (and, on k8s, its namespace)",
+)
+@click.option(
+    "--delete-certificate",
+    is_flag=True,
+    default=False,
+    help="also delete the TLS certificate issued for this deployment's hostname",
+)
 @click.option(
     "--skip-cluster-management/--perform-cluster-management",
     default=False,
-    help="Skip cluster initialization/tear-down (only for kind-k8s deployments)",
+    help="Skip cluster tear-down (only for kind-k8s deployments)",
 )
-@click.argument("extra_args", nargs=-1)  # help: command: down <service1> <service2>
+@click.argument("extra_args", nargs=-1)  # help: command: destroy
 @click.pass_context
-def stop(ctx, delete_volumes, skip_cluster_management, extra_args):
-    """stop the deployment and remove the containers"""
-    # TODO: add cluster name and env file here
+def destroy(ctx, yes, delete_volumes, delete_certificate, skip_cluster_management, extra_args):
+    """destroy the deployment: it is finished and its resources can be collected"""
+    deployment_context: DeploymentContext = ctx.obj
+    if not yes:
+        volumes = "and its volumes " if delete_volumes else ""
+        click.confirm(
+            f"Destroy deployment {deployment_context.deployment_dir} {volumes}permanently?",
+            abort=True,
+        )
     ctx.obj = make_deploy_context(ctx)
-    down_operation(ctx, delete_volumes, extra_args, skip_cluster_management)
+    destroy_operation(ctx, delete_volumes, delete_certificate, extra_args, skip_cluster_management)
+    _report_backups_kept(deployment_context)
+    _mark_destroyed(deployment_context)
+
+
+def _report_backups_kept(deployment_context: DeploymentContext):
+    """Say out loud that the backups were not part of this.
+
+    Backups exist to outlive the deployment that made them (see docs/backup.md),
+    so destroy leaves the repository alone -- which is worth stating at the one
+    moment a user is being told everything else is gone.
+    """
+    settings = backup_settings()
+    if settings.enabled and settings.s3_bucket:
+        output_main(f"Backups kept: repository {settings.s3_bucket} at {settings.s3_endpoint} is untouched")
+
+
+def _mark_destroyed(deployment_context: DeploymentContext):
+    marker = deployment_context.deployment_dir.joinpath(constants.destroyed_file_name)
+    marker.write_text(f"destroyed {datetime.now(timezone.utc).isoformat()}\n")
+    output_main(f"Deployment destroyed. Its directory {deployment_context.deployment_dir} is left for you to remove.")
 
 
 @command.command()