diff --git a/docs/commands/manage.md b/docs/commands/manage.md index d6a9229..36fc864 100644 --- a/docs/commands/manage.md +++ b/docs/commands/manage.md @@ -41,18 +41,67 @@ stack manage --dir DEPLOYMENT_DIR start [OPTIONS] [EXTRA_ARGS]... ### stop -Stop the stack and remove the containers +Stop the stack and remove the containers. + +`stop` is the symmetric opposite of `start`: it deletes only what `start` makes +again from the deployment directory. Volumes, the namespace on Kubernetes, and +the cluster on kind are all left in place, so a later `start` finds the data +where it left it. Use [`destroy`](#destroy) when a deployment is finished. + +```bash +stack manage --dir DEPLOYMENT_DIR stop [EXTRA_ARGS]... +``` + +`--delete-volumes` used to be how a finished deployment was cleaned up. It is +rejected now rather than ignored, so that a script asking for deletion is told +where deletion moved to instead of quietly leaking the volumes it meant to +reclaim. + +### destroy + +Destroy the deployment: the signal that it is finished and will not be started +again, which is what makes it safe to remove the things `stop` keeps precisely +because `start` would want them back. ```bash -stack manage --dir DEPLOYMENT_DIR stop [OPTIONS] [EXTRA_ARGS]... +stack manage --dir DEPLOYMENT_DIR destroy [OPTIONS] ``` +On Kubernetes that is the deployment's PersistentVolumeClaims, the cluster-scoped +PersistentVolumes a namespace delete does not reach, and the namespace itself; on +kind, the cluster; on Docker, the compose project's volume objects. As everywhere +else in stack, deleting a volume deletes the volume object and never the contents +of a bind-mounted directory (see [volumes.md](../volumes.md)). + +Two things deliberately survive, and `destroy` says so rather than leaving you to +wonder: + +- **Backups.** The restic/K8up repository is untouched: backups exist to outlive + the deployment that made them (see [backup.md](../backup.md)). +- **The TLS certificate**, on a Gateway-provisioned cluster. Certificates are + keyed by hostname, so redeploying the same hostname reuses the one already + issued instead of asking Let's Encrypt for another — whose duplicate limit is + five a week for the same name. `--delete-certificate` overrides this for a + hostname you are retiring for good. + + Certificates that no listener has referenced for a full certificate lifetime + are collected automatically as part of `destroy`: nothing renews an + unreferenced certificate, so one that has been idle that long is expired and + of no use to any future deployment. + +The deployment directory is left where it is — it is yours, and it is what a +replacement deployment would be created from — but a `destroyed` marker is +written into it, and the other `manage` subcommands refuse a directory carrying +one rather than reporting on objects that no longer exist. + #### Options | Option | Type | Description | Default | |--------|------|-------------|---------| -| `--delete-volumes/--preserve-volumes` | FLAG | Delete data volumes | False | -| `--skip-cluster-management/--perform-cluster-management` | FLAG | Skip cluster initialization/tear-down (kind-k8s only) | False | +| `--yes` / `-y` | FLAG | Do not prompt for confirmation | False | +| `--delete-volumes/--preserve-volumes` | FLAG | Delete the deployment's volumes (and, on k8s, its namespace) | True | +| `--delete-certificate` | FLAG | Also delete the TLS certificate issued for this deployment's hostname | False | +| `--skip-cluster-management/--perform-cluster-management` | FLAG | Skip cluster tear-down (kind-k8s only) | False | ### ps @@ -292,11 +341,14 @@ stack manage --dir ~/deployments/my-stack start # Start and stay attached to see output stack manage --dir ~/deployments/my-stack start --stay-attached -# Stop a stack (preserve volumes) +# Stop a stack; its data stays where it is stack manage --dir ~/deployments/my-stack stop -# Stop and delete volumes -stack manage --dir ~/deployments/my-stack stop --delete-volumes +# Finished with it: stop it for the last time and collect what it leaves +stack manage --dir ~/deployments/my-stack destroy + +# The same without the prompt, keeping the volumes +stack manage --dir ~/deployments/my-stack destroy --yes --preserve-volumes ``` ### Monitoring and Debugging diff --git a/docs/developing-applications.md b/docs/developing-applications.md index 05df730..adf0254 100644 --- a/docs/developing-applications.md +++ b/docs/developing-applications.md @@ -138,9 +138,9 @@ the pods. Use it to check the Kubernetes *shape* of a deployment — pods, volum without a real cluster; a change to the shape itself (ports, volumes, services) is the one thing `update` refuses, and there the loop is still `stop` then `start`. -Note that `stop` deletes the kind cluster and `start` builds a new one, so nothing kept -inside the cluster survives that loop. Your data does, because `init` maps each volume to a -directory under the deployment: +Note that `destroy` deletes the kind cluster and the next `start` builds a new one, so +nothing kept inside the cluster survives that. Your data does, because `init` maps each +volume to a directory under the deployment: ```yaml volumes: @@ -149,7 +149,7 @@ volumes: which is bind mounted into the kind node, exactly as it is bind mounted into the container on compose. The database you were working against is still there after a `stop`/`start`. To -start from an empty one, `stop --delete-volumes`, or delete the directory. +start from an empty one, delete the directory. A remote cluster is the one target where this cannot work, since the data would have to live on the cluster's nodes: there `init` leaves the volume unmapped and it becomes a PVC from the diff --git a/docs/gateway-api.md b/docs/gateway-api.md index 440549b..b00d005 100644 --- a/docs/gateway-api.md +++ b/docs/gateway-api.md @@ -48,6 +48,16 @@ For a spec with an `http-proxy` section, `stack manage start` creates: behind deliberately: redeploying the same hostname reuses the still-valid certificate instead of asking Let's Encrypt for a new one. +`stack manage destroy` leaves it too, for the same reason — a destroyed deployment's hostname is often +redeployed, and a certificate is worth more to the next deployment than the few kilobytes it occupies are worth +to the cluster. `--delete-certificate` overrides that for a hostname being retired for good. What `destroy` +does collect is certificates that are past being useful to anyone: a secret no listener has referenced for a +full certificate lifetime holds an expired certificate, since cert-manager's Certificate object goes with the +listener and nothing renews an unreferenced one. The first sweep to find such a secret marks it with +`stack.bozemanpass.com/certificate-unreferenced-since` and a later one deletes it; serving the hostname again +clears the mark, so the interval measured is always the current one. Measuring it, rather than reading +`notAfter` out of the certificate, keeps an X.509 parser (and a dependency) out of stack for the sake of a date. + Naming those objects after the hostname rather than after the deployment is what makes that reuse work. A deployment id changes whenever the stack is re-`init`ed, and a new secret name means cert-manager sees no certificate to reuse and places a fresh ACME order; Let's Encrypt issues five certificates per hostname per 168 diff --git a/docs/test-coverage.md b/docs/test-coverage.md index 334dd2b..63aa8d2 100644 --- a/docs/test-coverage.md +++ b/docs/test-coverage.md @@ -53,6 +53,7 @@ file listing is its own index. | `manage update`: data survives the in-place update | [`tests/app-deploy/run-test.sh`](../tests/app-deploy/run-test.sh) | `deploy update storage` | compose + kind per-PR; remote + remote-compose weekly | | `manage update`: rebuilt image content reaches the deployment | [`tests/app-deploy/run-test.sh`](../tests/app-deploy/run-test.sh) | `deploy update content` | compose + kind per-PR; remote + remote-compose weekly | | Spec-mapped volume path: pre-existing host data reaches the container | [`tests/volumes/run-test.sh`](../tests/volumes/run-test.sh) | `external data visible test`, `unmapped volume fresh test`, `volume write-back test` | compose + kind per-PR; remote weekly | +| `manage destroy`: the deployment is finished; later `manage` commands refuse its directory | [`tests/smoke-test/run-smoke-test.sh`](../tests/smoke-test/run-smoke-test.sh) | `deploy destroy` | compose, per-PR | | `manage exec` against a running service | [`tests/database/run-backup-test.sh`](../tests/database/run-backup-test.sh) | `Replay dump test` | compose per-PR; remote weekly | ## Backup and restore diff --git a/src/stack/constants.py b/src/stack/constants.py index 2e0e5ea..c53594b 100644 --- a/src/stack/constants.py +++ b/src/stack/constants.py @@ -41,6 +41,10 @@ # that the same stack produces a snapshot of the same name whichever engine took it. backup_default_file_extension = "dump" deployment_file_name = "deployment.yml" +# Written by `manage destroy` to record that this deployment's cluster objects are +# gone, so that the rest of `manage` refuses a directory that no longer describes +# anything running. +destroyed_file_name = "destroyed" host_name_key = "host-name" http_proxy_key = "http-proxy" http_proxy_prefix_key = "http-proxy-prefix" diff --git a/src/stack/deploy/compose/deploy_docker.py b/src/stack/deploy/compose/deploy_docker.py index 2ab2e03..2ba533f 100644 --- a/src/stack/deploy/compose/deploy_docker.py +++ b/src/stack/deploy/compose/deploy_docker.py @@ -111,10 +111,25 @@ def up(self, detach, skip_cluster_management, services): except DockerException as e: raise DeployerException(e) - def down(self, timeout, volumes, skip_cluster_management): + def down(self, timeout): if not opts.o.dry_run: try: - return self.docker.compose.down(timeout=timeout, volumes=volumes) + return self.docker.compose.down(timeout=timeout) + except DockerException as e: + raise DeployerException(e) + + def destroy(self, timeout, delete_volumes, delete_certificate, skip_cluster_management): + """Stop the deployment and remove its volume objects. + + There is no certificate of stack's own to collect here: TLS on this + target is the docker-ingress stack's business, and its certificates live + in its own volume (see docs/ingress.md). Removing a named volume + removes the volume object, never a bind-mounted directory's contents -- + the same thing it has always meant on this target. + """ + if not opts.o.dry_run: + try: + return self.docker.compose.down(timeout=timeout, volumes=delete_volumes) except DockerException as e: raise DeployerException(e) diff --git a/src/stack/deploy/deploy.py b/src/stack/deploy/deploy.py index 561a22f..4b2dd4e 100644 --- a/src/stack/deploy/deploy.py +++ b/src/stack/deploy/deploy.py @@ -85,14 +85,22 @@ def up_operation(ctx, services_list, stay_attached=False, skip_cluster_managemen ) -def down_operation(ctx, delete_volumes, extra_args_list, skip_cluster_management=False): +def down_operation(ctx, extra_args_list): timeout_arg = None if extra_args_list: timeout_arg = extra_args_list[0] # Specify shutdown timeout (default 10s) to give services enough time to shutdown gracefully - ctx.obj.deployer.down( + ctx.obj.deployer.down(timeout=timeout_arg) + + +def destroy_operation(ctx, delete_volumes, delete_certificate, extra_args_list, skip_cluster_management=False): + timeout_arg = None + if extra_args_list: + timeout_arg = extra_args_list[0] + ctx.obj.deployer.destroy( timeout=timeout_arg, - volumes=delete_volumes, + delete_volumes=delete_volumes, + delete_certificate=delete_certificate, skip_cluster_management=skip_cluster_management, ) diff --git a/src/stack/deploy/deployer.py b/src/stack/deploy/deployer.py index 18a0753..d222b97 100644 --- a/src/stack/deploy/deployer.py +++ b/src/stack/deploy/deployer.py @@ -23,7 +23,23 @@ def up(self, detach, skip_cluster_management, services): pass @abstractmethod - def down(self, timeout, volumes, skip_cluster_management): + def down(self, timeout): + """Stop the deployment. + + Symmetric with up(): whatever this deletes, up() has to be able to make + again. Nothing that holds data is touched -- that is destroy's job. + """ + pass + + @abstractmethod + def destroy(self, timeout, delete_volumes, delete_certificate, skip_cluster_management): + """Stop the deployment for the last time and collect what it leaves. + + The signal that a deployment is finished, which is what makes it safe to + remove the things stop keeps precisely because start would want them + back. Backups are not among them: they exist to outlive the deployment + that made them (see docs/backup.md). + """ pass @abstractmethod @@ -106,11 +122,11 @@ def __init__(self, *args: object) -> None: class ClusterNotRunningException(DeployerException): """There is no cluster to talk to, so nothing of the deployment is running. - Only a kind deployment reaches this state: stopping one deletes the whole + Only a kind deployment reaches this state: destroying one deletes the whole cluster, so afterwards there is no kube context left to connect to. That is - the normal resting state of a stopped kind deployment rather than a fault, - which is why it is distinguishable -- the commands that report what is - running answer "nothing" instead of failing. + the resting state of a destroyed kind deployment rather than a fault, which + is why it is distinguishable -- the commands that report what is running + answer "nothing" instead of failing. """ diff --git a/src/stack/deploy/deployment.py b/src/stack/deploy/deployment.py index b945b5c..5855426 100644 --- a/src/stack/deploy/deployment.py +++ b/src/stack/deploy/deployment.py @@ -16,6 +16,7 @@ import click +from datetime import datetime, timezone from pathlib import Path from stack import constants @@ -26,6 +27,7 @@ ps_operation, port_operation, status_operation, + destroy_operation, ) from stack.deploy.deploy import ( exec_operation, @@ -41,6 +43,7 @@ ) from stack.deploy.deploy_types import DeployCommandContext from stack.deploy.deployment_context import DeploymentContext +from stack.deploy.backup import backup_settings from stack.deploy.explain import explain_op from stack.log import output_main from stack.util import error_exit, get_yaml @@ -58,6 +61,16 @@ def command(ctx, dir): error_exit(f"Error: deployment directory {dir} does not exist") if not dir_path.is_dir(): error_exit(f"Error: supplied deployment directory path {dir} exists but is a file not a directory") + # A destroyed deployment's directory still describes a deployment, but nothing + # it describes exists any more, so every command here but destroy itself would + # be answering about something that is gone. Destroy stays available so that + # an interrupted one can be run again. + destroyed_marker = dir_path.joinpath(constants.destroyed_file_name) + if destroyed_marker.exists() and ctx.invoked_subcommand != "destroy": + error_exit( + f"Error: deployment {dir} was destroyed ({destroyed_marker.read_text().strip()}). " + "Create a new deployment with `stack deploy`." + ) # Store the deployment context for subcommands deployment_context = DeploymentContext() deployment_context.init(dir_path) @@ -114,19 +127,74 @@ def start(ctx, stay_attached, skip_cluster_management, extra_args): @command.command() -@click.option("--delete-volumes/--preserve-volumes", default=False, help="delete data volumes") +@click.option("--delete-volumes", is_flag=True, default=False, hidden=True) +@click.argument("extra_args", nargs=-1) # help: command: down +@click.pass_context +def stop(ctx, delete_volumes, extra_args): + """stop the deployment and remove the containers""" + # Stop is the symmetric opposite of start and deletes nothing that start + # cannot make again. --delete-volumes used to be how a finished deployment + # was cleaned up; that is what destroy is for now. It is still accepted so + # that a script asking for deletion is told, rather than quietly leaking the + # volumes it meant to reclaim. + if delete_volumes: + error_exit("Error: stop no longer deletes volumes. Use `stack manage --dir destroy` instead.") + # TODO: add cluster name and env file here + ctx.obj = make_deploy_context(ctx) + down_operation(ctx, extra_args) + + +@command.command() +@click.option("--yes", "-y", is_flag=True, default=False, help="do not prompt for confirmation") +@click.option( + "--delete-volumes/--preserve-volumes", + default=True, + help="delete the deployment's volumes (and, on k8s, its namespace)", +) +@click.option( + "--delete-certificate", + is_flag=True, + default=False, + help="also delete the TLS certificate issued for this deployment's hostname", +) @click.option( "--skip-cluster-management/--perform-cluster-management", default=False, - help="Skip cluster initialization/tear-down (only for kind-k8s deployments)", + help="Skip cluster tear-down (only for kind-k8s deployments)", ) -@click.argument("extra_args", nargs=-1) # help: command: down +@click.argument("extra_args", nargs=-1) # help: command: destroy @click.pass_context -def stop(ctx, delete_volumes, skip_cluster_management, extra_args): - """stop the deployment and remove the containers""" - # TODO: add cluster name and env file here +def destroy(ctx, yes, delete_volumes, delete_certificate, skip_cluster_management, extra_args): + """destroy the deployment: it is finished and its resources can be collected""" + deployment_context: DeploymentContext = ctx.obj + if not yes: + volumes = "and its volumes " if delete_volumes else "" + click.confirm( + f"Destroy deployment {deployment_context.deployment_dir} {volumes}permanently?", + abort=True, + ) ctx.obj = make_deploy_context(ctx) - down_operation(ctx, delete_volumes, extra_args, skip_cluster_management) + destroy_operation(ctx, delete_volumes, delete_certificate, extra_args, skip_cluster_management) + _report_backups_kept(deployment_context) + _mark_destroyed(deployment_context) + + +def _report_backups_kept(deployment_context: DeploymentContext): + """Say out loud that the backups were not part of this. + + Backups exist to outlive the deployment that made them (see docs/backup.md), + so destroy leaves the repository alone -- which is worth stating at the one + moment a user is being told everything else is gone. + """ + settings = backup_settings() + if settings.enabled and settings.s3_bucket: + output_main(f"Backups kept: repository {settings.s3_bucket} at {settings.s3_endpoint} is untouched") + + +def _mark_destroyed(deployment_context: DeploymentContext): + marker = deployment_context.deployment_dir.joinpath(constants.destroyed_file_name) + marker.write_text(f"destroyed {datetime.now(timezone.utc).isoformat()}\n") + output_main(f"Deployment destroyed. Its directory {deployment_context.deployment_dir} is left for you to remove.") @command.command() diff --git a/src/stack/deploy/k8s/deploy_k8s.py b/src/stack/deploy/k8s/deploy_k8s.py index 699f56d..3b2f7da 100644 --- a/src/stack/deploy/k8s/deploy_k8s.py +++ b/src/stack/deploy/k8s/deploy_k8s.py @@ -251,8 +251,8 @@ def __init__( def connect_api(self): if self.is_kind(): - # Stopping a kind deployment deletes its cluster, and kind removes the - # context with it, so there is nothing to load once one is stopped. + # Destroying a kind deployment deletes its cluster, and kind removes + # the context with it, so there is nothing left to load afterwards. # Raised as a condition of its own rather than let the kubernetes # client's ConfigException out: to the caller this is "not running", # and asking a stopped deployment what it is running is a fair @@ -355,9 +355,9 @@ def _create_secrets(self): is typically baked into a data volume, so it has to live exactly as long as the data does. On a remote cluster both live in the cluster, so the existing Secret is the store. On kind the data lives under the - deployment directory and the cluster is destroyed on stop, so generated - values persist beside the data in secrets.env and the Secret is rebuilt - from them. Referenced values are resolved fresh on every up. + deployment directory and the cluster does not survive destroy, so + generated values persist beside the data in secrets.env and the Secret + is rebuilt from them. Referenced values are resolved fresh on every up. Returns True when the Secret was created or its data changed, so update knows the pods have to be restarted to see the new values. @@ -472,8 +472,12 @@ def _create_gateway_resources(self, http_proxy_info_list): log_debug(f"Host {host_name} already covered by Gateway listener {listener['name']}") else: # cert-manager sees the new listener on the annotated Gateway - # and obtains its certificate over HTTP-01. + # and obtains its certificate over HTTP-01 -- or reuses the + # certificate left behind by whoever served this hostname last. gateway.add_https_listener(self.custom_obj_api, gw, host_name) + # Either way the certificate is in use again, so the sweep's clock + # for it starts over from the next time it falls out of use. + gateway.clear_unreferenced_mark(self.core_api, host_name) http_route = self.cluster_info.get_http_route(gateway.GATEWAY_NAME, gateway.GATEWAY_NAMESPACE) log_debug(f"Sending this HTTPRoute: {http_route}") @@ -483,9 +487,10 @@ def up(self, detach, skip_cluster_management, services): try: self.skip_cluster_management = skip_cluster_management if not opts.o.dry_run: + cluster_created = False if self.is_kind() and not self.skip_cluster_management: # Create the kind cluster - create_cluster( + cluster_created = create_cluster( self.kind_cluster_name, self.deployment_dir.joinpath(constants.kind_config_filename), ) @@ -493,8 +498,10 @@ def up(self, detach, skip_cluster_management, services): load_images_into_kind(self.kind_cluster_name, self.cluster_info.image_set) self.connect_api() if self.is_kind() and not self.skip_cluster_management: - # Now configure an ingress controller (not installed by default in kind) - install_ingress_for_kind() + # Now configure an ingress controller (not installed by default in kind). + # A cluster that was already there still has the one it was given. + if cluster_created: + install_ingress_for_kind() # Wait for ingress to start (deployment provisioning will fail unless this is done) wait_for_ingress_in_kind() @@ -545,104 +552,166 @@ def up(self, detach, skip_cluster_management, services): except Exception as e: error_exit(f"Exception thrown bringing stack up: {e}") - def down(self, timeout, volumes, skip_cluster_management): # noqa: C901 - try: - self.skip_cluster_management = skip_cluster_management - self.connect_api() - # Delete the k8s objects - - if volumes: - # Create the host-path-mounted PVs for this deployment - pvs = self.cluster_info.get_pvs() - for pv in pvs: - log_debug(f"Deleting this pv: {pv}") - try: - pv_resp = self.core_api.delete_persistent_volume(name=pv.metadata.name) - log_debug("PV deleted:") - log_debug(f"{pv_resp}") - except client.exceptions.ApiException as e: - _check_delete_exception(e) - - # Figure out the PVCs for this deployment - pvcs = self.cluster_info.get_pvcs() - for pvc in pvcs: - log_debug(f"Deleting this pvc: {pvc}") - try: - pvc_resp = self.core_api.delete_namespaced_persistent_volume_claim( - name=pvc.metadata.name, namespace=self.k8s_namespace - ) - log_debug("PVCs deleted:") - log_debug(f"{pvc_resp}") - except client.exceptions.ApiException as e: - _check_delete_exception(e) + def _delete_deployment_objects(self): + """Delete the objects that up() creates from the deployment directory. - # Figure out the ConfigMaps for this deployment - cfg_maps = self.cluster_info.get_configmaps() - for cfg_map in cfg_maps: - log_debug(f"Deleting this ConfigMap: {cfg_map}") - try: - cfg_map_resp = self.core_api.delete_namespaced_config_map( - name=cfg_map.metadata.name, - namespace=self.k8s_namespace - ) - log_debug("ConfigMap deleted:") - log_debug(f"{cfg_map_resp}") - except client.exceptions.ApiException as e: - _check_delete_exception(e) + Everything here is derived from the deployment's own files, so a later + start recreates it identically. Nothing here holds data. + """ + # Figure out the ConfigMaps for this deployment + cfg_maps = self.cluster_info.get_configmaps() + for cfg_map in cfg_maps: + log_debug(f"Deleting this ConfigMap: {cfg_map}") + try: + cfg_map_resp = self.core_api.delete_namespaced_config_map( + name=cfg_map.metadata.name, + namespace=self.k8s_namespace + ) + log_debug("ConfigMap deleted:") + log_debug(f"{cfg_map_resp}") + except client.exceptions.ApiException as e: + _check_delete_exception(e) - deployments = self.cluster_info.get_deployments() - for deployment in deployments: - log_debug(f"Deleting this deployment: {deployment}") - try: - self.apps_api.delete_namespaced_deployment(name=deployment.metadata.name, namespace=self.k8s_namespace) - except client.exceptions.ApiException as e: - _check_delete_exception(e) + deployments = self.cluster_info.get_deployments() + for deployment in deployments: + log_debug(f"Deleting this deployment: {deployment}") + try: + self.apps_api.delete_namespaced_deployment(name=deployment.metadata.name, namespace=self.k8s_namespace) + except client.exceptions.ApiException as e: + _check_delete_exception(e) - services: client.V1Service = self.cluster_info.get_services() - for svc in services: - log_debug(f"Deleting service: {svc}") + services: client.V1Service = self.cluster_info.get_services() + for svc in services: + log_debug(f"Deleting service: {svc}") + try: + self.core_api.delete_namespaced_service(namespace=self.k8s_namespace, name=svc.metadata.name) + except client.exceptions.ApiException as e: + _check_delete_exception(e) + + # Only the scheduling stops; the repository the backups are in is + # deliberately left alone, since backups exist to outlive the + # deployment that made them. + if backup_settings().enabled and k8up.k8up_available(self.custom_obj_api): + k8up.delete_backup_configuration(self.core_api, self.custom_obj_api, self.k8s_namespace) + + http_proxy_info_list = self.cluster_info.spec.get_http_proxy() + if http_proxy_info_list and gateway.gateway_api_available(self.custom_obj_api): + gateway.delete_http_route(self.custom_obj_api, self.k8s_namespace) + # The certificate Secret survives so that a redeployment of the + # same hostname reuses it rather than asking for a new one. + host_name = http_proxy_info_list[0][constants.host_name_key] + gateway.remove_https_listener(self.custom_obj_api, host_name, self.k8s_namespace) + else: + ingress: client.V1Ingress = self.cluster_info.get_ingress(use_tls=not self.is_kind()) + if ingress: + log_debug(f"Deleting this ingress: {ingress}") try: - self.core_api.delete_namespaced_service(namespace=self.k8s_namespace, name=svc.metadata.name) + self.networking_api.delete_namespaced_ingress(name=ingress.metadata.name, namespace=self.k8s_namespace) except client.exceptions.ApiException as e: _check_delete_exception(e) + else: + log_debug("No ingress to delete") - # Only the scheduling stops; the repository the backups are in is - # deliberately left alone, since backups exist to outlive the - # deployment that made them. - if backup_settings().enabled and k8up.k8up_available(self.custom_obj_api): - k8up.delete_backup_configuration(self.core_api, self.custom_obj_api, self.k8s_namespace) + def _delete_volume_objects(self): + """Delete the deployment's PVCs and its cluster-scoped PVs. - http_proxy_info_list = self.cluster_info.spec.get_http_proxy() - if http_proxy_info_list and gateway.gateway_api_available(self.custom_obj_api): - gateway.delete_http_route(self.custom_obj_api, self.k8s_namespace) - # The certificate Secret survives so that a redeployment of the - # same hostname reuses it rather than asking for a new one. - host_name = http_proxy_info_list[0][constants.host_name_key] - gateway.remove_https_listener(self.custom_obj_api, host_name, self.k8s_namespace) - else: - ingress: client.V1Ingress = self.cluster_info.get_ingress(use_tls=not self.is_kind()) - if ingress: - log_debug(f"Deleting this ingress: {ingress}") - try: - self.networking_api.delete_namespaced_ingress(name=ingress.metadata.name, namespace=self.k8s_namespace) - except client.exceptions.ApiException as e: - _check_delete_exception(e) - else: - log_debug("No ingress to delete") + The PVs are cluster-scoped, so deleting the namespace does not reach + them. Neither deletion touches the data itself: a node-path volume's + directory is on the node, and stack has no way in from here. + """ + pvs = self.cluster_info.get_pvs() + for pv in pvs: + log_debug(f"Deleting this pv: {pv}") + try: + pv_resp = self.core_api.delete_persistent_volume(name=pv.metadata.name) + log_debug("PV deleted:") + log_debug(f"{pv_resp}") + except client.exceptions.ApiException as e: + _check_delete_exception(e) + + # Figure out the PVCs for this deployment + pvcs = self.cluster_info.get_pvcs() + for pvc in pvcs: + log_debug(f"Deleting this pvc: {pvc}") + try: + pvc_resp = self.core_api.delete_namespaced_persistent_volume_claim( + name=pvc.metadata.name, namespace=self.k8s_namespace + ) + log_debug("PVCs deleted:") + log_debug(f"{pvc_resp}") + except client.exceptions.ApiException as e: + _check_delete_exception(e) + + def down(self, timeout): + """Stop the deployment: delete what start recreates, and nothing else. + + The namespace, the volumes and (on kind) the cluster stay, so that a + later start finds the data where it left it. Getting rid of those is + destroy's business. + """ + try: + self.connect_api() + self._delete_deployment_objects() + except Exception as e: + error_exit(f"Exception thrown stopping stack: {e}") - if volumes: + def destroy(self, timeout, delete_volumes, delete_certificate, skip_cluster_management): # noqa: C901 + try: + self.skip_cluster_management = skip_cluster_management + try: + self.connect_api() + except ClusterNotRunningException as e: + # A kind deployment whose cluster is already gone has nothing + # left in a cluster to destroy. + log_debug(f"{e}: nothing left to destroy") + return + self._delete_deployment_objects() + + # A kind deployment's cluster goes below, taking every object in it, + # so deleting them one at a time here would be for nothing -- and + # promising to keep any of them would be a lie. Its volume data is + # not in the cluster anyway: it is bind-mounted from the deployment + # directory, where destroy leaves it. + cluster_goes = self.is_kind() and not self.skip_cluster_management + if delete_volumes and not cluster_goes: + self._delete_volume_objects() try: self.core_api.delete_namespace(name=self.k8s_namespace) log_debug(f"Namespace {self.k8s_namespace} deleted") except client.exceptions.ApiException as e: _check_delete_exception(e) + elif not delete_volumes and not cluster_goes: + # The PVCs live in the namespace, so keeping the volumes means + # keeping the namespace they are claimed in. + output_main(f"Volumes preserved: namespace {self.k8s_namespace} left in place") - if self.is_kind() and not self.skip_cluster_management: + http_proxy_info_list = self.cluster_info.spec.get_http_proxy() + if http_proxy_info_list and gateway.gateway_api_available(self.custom_obj_api): + host_name = http_proxy_info_list[0][constants.host_name_key] + secret_name = gateway.secret_name_for_host(host_name) + if delete_certificate: + if gateway.delete_certificate_secret(self.core_api, host_name): + output_main(f"Certificate for {host_name} deleted (Secret {secret_name})") + else: + output_main( + f"Certificate for {host_name} kept (Secret {secret_name} in {gateway.GATEWAY_NAMESPACE}): " + "redeploying this hostname reuses it rather than ordering a new one" + ) + # Whatever this deployment's hostname, this is a good moment to + # collect the certificates that are past being useful to anyone. + for name in gateway.sweep_certificate_secrets(self.core_api, self.custom_obj_api): + log_info(f"Deleted expired certificate Secret {name}") + + if cluster_goes: # Destroy the kind cluster destroy_cluster(self.kind_cluster_name) + output_main( + f"kind cluster {self.kind_cluster_name} deleted. Volume data under the deployment " + "directory is untouched." + ) except Exception as e: - error_exit(f"Exception thrown bringing stack up: {e}") + error_exit(f"Exception thrown destroying stack: {e}") def status(self): try: diff --git a/src/stack/deploy/k8s/gateway.py b/src/stack/deploy/k8s/gateway.py index e0ad1e2..1bafdea 100644 --- a/src/stack/deploy/k8s/gateway.py +++ b/src/stack/deploy/k8s/gateway.py @@ -35,6 +35,7 @@ no listener of their own -- only an HTTPRoute. """ +import datetime import hashlib import re @@ -63,8 +64,15 @@ # under a prefix that marks them as stack's among whatever else lives in the # Gateway's namespace. NAME_PREFIX = "stack-" +SECRET_SUFFIX = "-tls" MAX_OBJECT_NAME_LENGTH = 253 +# A certificate Secret is collectable only once it is certainly expired, which is +# a Let's Encrypt certificate's lifetime after the last listener stopped naming +# it. See sweep_certificate_secrets. +UNREFERENCED_SINCE_ANNOTATION = "stack.bozemanpass.com/certificate-unreferenced-since" +CERTIFICATE_LIFETIME = datetime.timedelta(days=90) + CLUSTER_ISSUER_ANNOTATION = "cert-manager.io/cluster-issuer" @@ -200,7 +208,7 @@ def listener_name_for_host(host_name: str) -> str: def secret_name_for_host(host_name: str) -> str: - return _name_for_host(host_name, "-tls") + return _name_for_host(host_name, SECRET_SUFFIX) def https_listener_for_host(host_name: str): @@ -289,3 +297,125 @@ def delete_http_route(custom_obj_api: client.CustomObjectsApi, namespace: str): if e.status != 404: raise log_debug("No HTTPRoute to delete") + + +def _certificate_secret_names_in_use(gateway) -> set: + """The certificate Secrets the Gateway's listeners currently reference.""" + names = set() + for listener in gateway.get("spec", {}).get("listeners", []): + for ref in (listener.get("tls") or {}).get("certificateRefs", []): + if ref.get("name"): + names.add(ref["name"]) + return names + + +def _stack_certificate_secrets(core_api: client.CoreV1Api): + """The TLS Secrets stack's listeners named, and only those. + + A machine-provisioned wildcard certificate lives in the same namespace and + is nobody's to collect, so the sweep goes by stack's own naming rather than + by "every TLS Secret here". + """ + secrets = core_api.list_namespaced_secret(namespace=GATEWAY_NAMESPACE, field_selector="type=kubernetes.io/tls") + return [ + secret + for secret in secrets.items + if secret.metadata.name.startswith(NAME_PREFIX) and secret.metadata.name.endswith(SECRET_SUFFIX) + ] + + +def _annotate_secret(core_api: client.CoreV1Api, name: str, value): + """Set (or, with value None, remove) the unreferenced mark on a Secret.""" + try: + core_api.patch_namespaced_secret( + name=name, + namespace=GATEWAY_NAMESPACE, + body={"metadata": {"annotations": {UNREFERENCED_SINCE_ANNOTATION: value}}}, + ) + except client.exceptions.ApiException as e: + if e.status != 404: + raise + + +def clear_unreferenced_mark(core_api: client.CoreV1Api, host_name: str): + """Note that a hostname's certificate is in use again. + + Called when a listener is added, so that the sweep below measures from the + last time the certificate fell out of use rather than from some earlier one. + """ + _annotate_secret(core_api, secret_name_for_host(host_name), None) + + +def delete_certificate_secret(core_api: client.CoreV1Api, host_name: str) -> bool: + """Delete a hostname's certificate Secret now. True if there was one.""" + try: + core_api.delete_namespaced_secret(name=secret_name_for_host(host_name), namespace=GATEWAY_NAMESPACE) + return True + except client.exceptions.ApiException as e: + if e.status != 404: + raise + return False + + +def _unreferenced_since(secret): + stamp = (secret.metadata.annotations or {}).get(UNREFERENCED_SINCE_ANNOTATION) + if not stamp: + return None + try: + return datetime.datetime.fromisoformat(stamp) + except ValueError: + # Something else wrote it, or an older stack wrote it differently. Treat + # it as unmarked and mark it again: worst case the clock restarts. + log_debug(f"Unparseable {UNREFERENCED_SINCE_ANNOTATION} on Secret: {stamp}") + return None + + +def sweep_certificate_secrets(core_api: client.CoreV1Api, custom_obj_api: client.CustomObjectsApi, now=None): + """Delete certificate Secrets that are certainly expired, and no others. + + A certificate Secret outlives the deployment that caused it on purpose: the + listener is removed whenever the deployment stops, and re-adding it later + reuses the still-valid certificate instead of asking Let's Encrypt for + another one -- whose duplicate-certificate limit is five a week for the same + name, which a redeploy loop reaches easily. So nothing here deletes a + certificate that anything could still use. + + What can be deleted is one that has been unreferenced by any listener for a + full certificate lifetime: nothing renews an unreferenced certificate, since + cert-manager's Certificate object goes with the listener, so such a Secret + holds an expired certificate and a redeployment of its hostname would order + a new one regardless. + + That is measured rather than read out of the certificate because reading it + would mean parsing X.509 -- a dependency for a date. The first sweep to see + a Secret unreferenced marks it and moves on; a later one deletes it. The + mark is cleared whenever the hostname is served again, so the interval is + always the current one. + + Returns the names deleted. + """ + gateway = get_gateway(custom_obj_api) + if not gateway: + return [] + in_use = _certificate_secret_names_in_use(gateway) + now = now or datetime.datetime.now(datetime.timezone.utc) + deleted = [] + for secret in _stack_certificate_secrets(core_api): + name = secret.metadata.name + if name in in_use: + continue + since = _unreferenced_since(secret) + if since is None: + log_debug(f"Marking unreferenced certificate Secret {name}") + _annotate_secret(core_api, name, now.isoformat()) + continue + if now - since < CERTIFICATE_LIFETIME: + continue + log_debug(f"Deleting certificate Secret {name}, unreferenced since {since.isoformat()}") + try: + core_api.delete_namespaced_secret(name=name, namespace=GATEWAY_NAMESPACE) + deleted.append(name) + except client.exceptions.ApiException as e: + if e.status != 404: + raise + return deleted diff --git a/src/stack/deploy/k8s/helpers.py b/src/stack/deploy/k8s/helpers.py index e1142b3..ccc30b1 100644 --- a/src/stack/deploy/k8s/helpers.py +++ b/src/stack/deploy/k8s/helpers.py @@ -17,6 +17,7 @@ import os import re import shlex +import subprocess from expandvars import expand from kubernetes import client, utils, watch @@ -35,10 +36,29 @@ DEFAULT_K8S_NAMESPACE = "default" -def create_cluster(name: str, config_file: str): +def cluster_exists(name: str) -> bool: + result = subprocess.run(["kind", "get", "clusters"], capture_output=True, text=True) + if result.returncode != 0: + raise DeployerException(f"kind get clusters failed: {result.returncode}") + return name in result.stdout.split() + + +def create_cluster(name: str, config_file: str) -> bool: + """Create the kind cluster if it is not already there. True if it was created. + + A stopped kind deployment keeps its cluster, since its volumes live inside + the node container and stop is not allowed to delete data. Starting it + again therefore finds the cluster already there, which kind treats as an + error rather than as nothing to do -- and what was installed into it the + first time is still installed, which is what the return value is for. + """ + if cluster_exists(name): + log_info(f"Using existing kind cluster {name}") + return False rc = run_shell_command(f"kind create cluster --name {name} --config {config_file}") if rc != 0: raise DeployerException(f"kind create cluster failed: {rc}") + return True def destroy_cluster(name: str): diff --git a/tests/backup/run-cross-target-test.sh b/tests/backup/run-cross-target-test.sh index 415f669..9cdcc52 100755 --- a/tests/backup/run-cross-target-test.sh +++ b/tests/backup/run-cross-target-test.sh @@ -115,7 +115,7 @@ extra_deployment_dirs=() stop_extra_deployments () { local dir for dir in "${extra_deployment_dirs[@]}"; do - $TEST_TARGET_STACK manage --dir "$dir" stop --delete-volumes + $TEST_TARGET_STACK manage --dir "$dir" destroy --yes done } TEST_EXTRA_CLEANUP=stop_extra_deployments diff --git a/tests/backup/run-test.sh b/tests/backup/run-test.sh index cd21deb..9ba0916 100755 --- a/tests/backup/run-test.sh +++ b/tests/backup/run-test.sh @@ -200,7 +200,7 @@ if [ -n "$TEST_BACKUP_CAN_SEED" ]; then # tears it down. first_deployment_dir=$test_deployment_dir stop_first_deployment () { - $TEST_TARGET_STACK manage --dir "$first_deployment_dir" stop --delete-volumes + $TEST_TARGET_STACK manage --dir "$first_deployment_dir" destroy --yes } TEST_EXTRA_CLEANUP=stop_first_deployment stop_deployment_on_exit "$seeded_deployment_dir" diff --git a/tests/lib/common.sh b/tests/lib/common.sh index 0c29292..dfae9cf 100644 --- a/tests/lib/common.sh +++ b/tests/lib/common.sh @@ -407,7 +407,7 @@ start_object_store_deployment () { } stop_object_store_deployment () { - $TEST_TARGET_STACK manage --dir "$TEST_OBJECT_STORE_DIR" stop --delete-volumes + $TEST_TARGET_STACK manage --dir "$TEST_OBJECT_STORE_DIR" destroy --yes } # Wait until the object store backups go to can serve back what it accepts. @@ -513,7 +513,7 @@ _test_exit_handler () { if [ $rc -ne 0 ]; then dump_diagnostics fi - $TEST_TARGET_STACK manage --dir "$TEST_DEPLOYMENT_DIR" stop --delete-volumes + $TEST_TARGET_STACK manage --dir "$TEST_DEPLOYMENT_DIR" destroy --yes fi if [ -n "$TEST_CONTAINER_IDS" ]; then docker stop $TEST_CONTAINER_IDS > /dev/null 2>&1 @@ -538,17 +538,18 @@ start_container () { trap _test_exit_handler EXIT } -# Destroy a deployment completely: stop it, delete its volumes, and remove the -# deployment directory, so that nothing of it is left for a later step to lean on -# by accident. For a test whose point is that something survives the deployment -# that produced it -- a backup taken from a deployment that is then destroyed -- -# a half-removed deployment is the way that test passes for the wrong reason. +# Destroy a deployment completely: `manage destroy`, which takes its volumes (and +# on k8s its namespace) with it, and then remove the deployment directory, so that +# nothing of it is left for a later step to lean on by accident. For a test whose +# point is that something survives the deployment that produced it -- a backup taken +# from a deployment that is then destroyed -- a half-removed deployment is the way +# that test passes for the wrong reason. # # The exit-time teardown is dropped along with it, so a test destroying the # deployment the helpers are pointed at should point them at its replacement with # stop_deployment_on_exit. destroy_deployment () { - $TEST_TARGET_STACK manage --dir "$1" stop --delete-volumes + $TEST_TARGET_STACK manage --dir "$1" destroy --yes force_rm "$1" if [ "$TEST_DEPLOYMENT_DIR" == "$1" ]; then TEST_DEPLOYMENT_DIR="" diff --git a/tests/smoke-test/run-smoke-test.sh b/tests/smoke-test/run-smoke-test.sh index 9e819d5..7ee6e32 100755 --- a/tests/smoke-test/run-smoke-test.sh +++ b/tests/smoke-test/run-smoke-test.sh @@ -37,6 +37,19 @@ assert_file_contains $test_deployment_dir/create-file "create-command-output-dat $TEST_TARGET_STACK manage --dir $test_deployment_dir start # Down $TEST_TARGET_STACK manage --dir $test_deployment_dir stop +# Destroy: the signal that the deployment is finished. On compose a stopped +# deployment and a destroyed one look much the same from outside, so what is +# asserted is the marker destroy leaves and the refusal that follows it -- a +# deployment directory whose objects are gone must not go on answering questions +# about them. +$TEST_TARGET_STACK manage --dir $test_deployment_dir destroy --yes +if [ ! -f "$test_deployment_dir/destroyed" ]; then + fail "deploy destroy: FAILED - destroyed marker not written" +fi +if $TEST_TARGET_STACK manage --dir $test_deployment_dir status > /dev/null 2>&1; then + fail "deploy destroy: FAILED - manage still operates on a destroyed deployment" +fi +echo "deploy destroy: PASSED" # Run same test but not using the stack definition # Test building the a stack container $TEST_TARGET_STACK --debug --verbose build containers --stack test --include bozemanpass/test-container diff --git a/tests/static-content-test/run-static-content-test.sh b/tests/static-content-test/run-static-content-test.sh index 6d018c4..11911c3 100755 --- a/tests/static-content-test/run-static-content-test.sh +++ b/tests/static-content-test/run-static-content-test.sh @@ -125,7 +125,7 @@ docker run -d --name stack-test-registry -p 5000:5000 registry:2 # The deployment holds a running container using the image, which would block # image removal below; it has served its purpose, so stop it now. -$TEST_TARGET_STACK manage --dir $test_deployment_dir stop --delete-volumes +$TEST_TARGET_STACK manage --dir $test_deployment_dir destroy --yes static_content_stack_dir=$STACK_REPO_BASE_DIR/github.com/bozemanpass/stack-test-stacks/stack-files/stacks/test-static-content-stack diff --git a/tests/unit/test_k8s_certificate_sweep.py b/tests/unit/test_k8s_certificate_sweep.py new file mode 100644 index 0000000..a2b3c0b --- /dev/null +++ b/tests/unit/test_k8s_certificate_sweep.py @@ -0,0 +1,198 @@ +# Copyright © 2026 Bozeman Pass, Inc. + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see . + +"""Tests for the certificate Secret lifecycle on a Gateway-provisioned cluster. + +A certificate outlives the deployment that caused it on purpose: deleting one +that a redeployment could have reused is how a hostname reaches Let's Encrypt's +duplicate-certificate limit (#283). So the property under test is mostly a +negative one -- what the sweep refuses to delete -- with deletion reserved for a +certificate that has been unreferenced long enough to have expired. + +No cluster is involved: the API objects are stand-ins that record what the sweep +asked for. +""" + +import datetime + +import pytest + +from types import SimpleNamespace + +from stack.deploy.k8s import gateway + + +HOST = "app.example.com" +SECRET = gateway.secret_name_for_host(HOST) +LISTENER = gateway.listener_name_for_host(HOST) + +NOW = datetime.datetime(2026, 8, 20, tzinfo=datetime.timezone.utc) + + +def stamp(days_ago): + return (NOW - datetime.timedelta(days=days_ago)).isoformat() + + +class FakeApiException(Exception): + def __init__(self, status): + self.status = status + super().__init__(f"status {status}") + + +@pytest.fixture(autouse=True) +def api_exception(monkeypatch): + """Let the stand-ins raise the exception type gateway.py catches.""" + monkeypatch.setattr(gateway.client.exceptions, "ApiException", FakeApiException) + + +def secret(name, unreferenced_since=None): + annotations = {gateway.UNREFERENCED_SINCE_ANNOTATION: unreferenced_since} if unreferenced_since else None + return SimpleNamespace(metadata=SimpleNamespace(name=name, annotations=annotations)) + + +class FakeCoreApi: + def __init__(self, secrets): + self.secrets = secrets + self.patched = {} + self.deleted = [] + + def list_namespaced_secret(self, namespace, field_selector=None): + assert namespace == gateway.GATEWAY_NAMESPACE + # The sweep asks only for TLS Secrets; anything else in kube-system is + # none of its business. + assert field_selector == "type=kubernetes.io/tls" + return SimpleNamespace(items=self.secrets) + + def patch_namespaced_secret(self, name, namespace, body): + self.patched[name] = body["metadata"]["annotations"][gateway.UNREFERENCED_SINCE_ANNOTATION] + + def delete_namespaced_secret(self, name, namespace): + if name not in [s.metadata.name for s in self.secrets]: + raise FakeApiException(404) + self.deleted.append(name) + + +class FakeCustomObjApi: + def __init__(self, listeners): + self.listeners = listeners + + def get_namespaced_custom_object(self, group, version, namespace, plural, name): + return {"spec": {"listeners": self.listeners}} + + +def https_listener(host): + return gateway.https_listener_for_host(host) + + +def sweep(secrets, listeners, now=NOW): + core_api = FakeCoreApi(secrets) + deleted = gateway.sweep_certificate_secrets(core_api, FakeCustomObjApi(listeners), now=now) + return core_api, deleted + + +def test_an_unreferenced_certificate_is_marked_before_it_is_deleted(): + # Nothing is deleted on first sight: the sweep does not know how long this + # one has been idle, so it starts the clock and comes back later. + core_api, deleted = sweep([secret(SECRET)], listeners=[]) + + assert deleted == [] + assert core_api.deleted == [] + assert core_api.patched[SECRET] == NOW.isoformat() + + +def test_a_certificate_unreferenced_past_its_lifetime_is_deleted(): + core_api, deleted = sweep([secret(SECRET, stamp(days_ago=91))], listeners=[]) + + assert deleted == [SECRET] + assert core_api.deleted == [SECRET] + + +def test_a_certificate_unreferenced_but_still_valid_is_kept(): + # Still inside a certificate's lifetime, so redeploying this hostname would + # reuse it -- which is the whole reason the Secret survives a destroy. + core_api, deleted = sweep([secret(SECRET, stamp(days_ago=30))], listeners=[]) + + assert deleted == [] + assert core_api.deleted == [] + # And the mark is left as it was, so the interval keeps running. + assert core_api.patched == {} + + +def test_a_referenced_certificate_is_never_deleted_however_old_its_mark(): + # A hostname served again keeps its certificate: a stale mark from an + # earlier idle spell must not condemn a certificate now in use. + core_api, deleted = sweep([secret(SECRET, stamp(days_ago=400))], listeners=[https_listener(HOST)]) + + assert deleted == [] + assert core_api.deleted == [] + + +def test_certificates_stack_did_not_name_are_left_alone(): + # A machine-provisioned wildcard certificate lives in the same namespace and + # is not stack's to collect. + core_api, deleted = sweep([secret("wildcard-example-com-tls", stamp(days_ago=400))], listeners=[]) + + assert deleted == [] + assert core_api.deleted == [] + assert core_api.patched == {} + + +def test_an_unreadable_mark_restarts_the_interval_rather_than_deleting(): + core_api, deleted = sweep([secret(SECRET, "last tuesday")], listeners=[]) + + assert deleted == [] + assert core_api.patched[SECRET] == NOW.isoformat() + + +def test_nothing_is_swept_without_a_gateway(): + class NoGateway: + def get_namespaced_custom_object(self, **kwargs): + raise FakeApiException(404) + + core_api = FakeCoreApi([secret(SECRET, stamp(days_ago=400))]) + assert gateway.sweep_certificate_secrets(core_api, NoGateway(), now=NOW) == [] + assert core_api.deleted == [] + + +def test_serving_a_hostname_again_clears_its_mark(): + core_api = FakeCoreApi([secret(SECRET, stamp(days_ago=30))]) + gateway.clear_unreferenced_mark(core_api, HOST) + + assert core_api.patched[SECRET] is None + + +def test_clearing_the_mark_tolerates_a_hostname_with_no_certificate_yet(): + # First deployment of a hostname, or one covered by a wildcard listener: + # there is no Secret of its own to annotate. + class Missing(FakeCoreApi): + def patch_namespaced_secret(self, name, namespace, body): + raise FakeApiException(404) + + gateway.clear_unreferenced_mark(Missing([]), HOST) + + +def test_deleting_a_certificate_reports_whether_there_was_one(): + core_api = FakeCoreApi([secret(SECRET)]) + assert gateway.delete_certificate_secret(core_api, HOST) is True + assert core_api.deleted == [SECRET] + + assert gateway.delete_certificate_secret(FakeCoreApi([]), HOST) is False + + +def test_the_listener_and_its_secret_agree_on_the_hostname(): + # The sweep's "is it referenced?" test compares the listener's certificateRef + # against the Secret name, so the two namings have to stay in step. + assert https_listener(HOST)["tls"]["certificateRefs"] == [{"name": SECRET}] + assert https_listener(HOST)["name"] == LISTENER diff --git a/tests/unit/test_manage_destroy.py b/tests/unit/test_manage_destroy.py new file mode 100644 index 0000000..8b236b6 --- /dev/null +++ b/tests/unit/test_manage_destroy.py @@ -0,0 +1,100 @@ +# Copyright © 2026 Bozeman Pass, Inc. + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. + +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see . + +"""Tests for `stack manage --dir destroy` and what stop no longer does. + +Stop is the symmetric opposite of start and deletes nothing that start cannot +make again; destroy is the signal that a deployment is finished (#287). These +cover the parts of that split which need no engine: the retired stop flag, the +confirmation, and the marker that keeps the rest of `manage` from operating on a +deployment whose objects are gone. +""" + +from conftest import make_stack_from_compose, run_stack + +from stack import constants + + +ONE_SERVICE_POD = """\ + services: + web: + image: nginx:latest + """ + + +def deploy(tmp_path, isolated_env): + stack_dir = make_stack_from_compose(tmp_path, ONE_SERVICE_POD) + spec_file = tmp_path / "spec.yml" + deployment_dir = tmp_path / "deployment" + + result = run_stack(["init", "--stack", str(stack_dir), "--output", str(spec_file)], isolated_env, cwd=tmp_path) + assert result.returncode == 0, f"init failed:\n{result.stdout}\n{result.stderr}" + result = run_stack( + ["deploy", "--spec-file", str(spec_file), "--deployment-dir", str(deployment_dir)], + isolated_env, + cwd=tmp_path, + ) + assert result.returncode == 0, f"deploy failed:\n{result.stdout}\n{result.stderr}" + return deployment_dir + + +def manage(deployment_dir, args, isolated_env, tmp_path): + return run_stack(["manage", "--dir", str(deployment_dir)] + args, isolated_env, cwd=tmp_path) + + +def mark_destroyed(deployment_dir): + """What destroy leaves behind, without needing an engine to reach it.""" + marker = deployment_dir.joinpath(constants.destroyed_file_name) + marker.write_text("destroyed 2026-08-20T00:00:00+00:00\n") + return marker + + +def test_stop_refuses_delete_volumes_rather_than_ignoring_it(tmp_path, isolated_env): + # Silently accepting the retired flag would leak exactly the volumes the + # caller asked to reclaim, so it fails loudly and names its replacement. + deployment_dir = deploy(tmp_path, isolated_env) + result = manage(deployment_dir, ["stop", "--delete-volumes"], isolated_env, tmp_path) + + assert result.returncode != 0 + assert "destroy" in result.stdout + result.stderr + + +def test_destroy_asks_before_it_destroys(tmp_path, isolated_env): + deployment_dir = deploy(tmp_path, isolated_env) + result = manage(deployment_dir, ["destroy"], isolated_env, tmp_path) + + # No answer on a closed stdin is not an answer, so nothing happened. + assert result.returncode != 0 + assert not deployment_dir.joinpath(constants.destroyed_file_name).exists() + + +def test_a_destroyed_deployment_is_refused_by_the_other_subcommands(tmp_path, isolated_env): + deployment_dir = deploy(tmp_path, isolated_env) + mark_destroyed(deployment_dir) + + result = manage(deployment_dir, ["services"], isolated_env, tmp_path) + assert result.returncode != 0 + assert "destroyed" in result.stdout + result.stderr + + +def test_destroy_still_runs_against_an_already_destroyed_deployment(tmp_path, isolated_env): + # An interrupted destroy has to be repeatable, so the marker does not lock + # the one command that could finish the job. + deployment_dir = deploy(tmp_path, isolated_env) + mark_destroyed(deployment_dir) + + result = manage(deployment_dir, ["destroy", "--help"], isolated_env, tmp_path) + assert result.returncode == 0, f"{result.stdout}\n{result.stderr}" + assert "--delete-certificate" in result.stdout