feat: automated s3 backup

content/features/backups.md

@@ -0,0 +1,189 @@
+---
+title: 'Backups'
+description: 'Protect Docker volumes and Arcane itself with encrypted local and S3-compatible backups.'
+---
+
+<script lang="ts">
+import { Link } from '#lib/components/ui/link/index.js';
+</script>
+
+Arcane uses short-lived [Rustic](https://rustic.cli.rs/) containers to create encrypted snapshots. Backups can stay local, be written directly to S3-compatible storage, or use both destinations.
+
+## Local backup storage
+
+The `/backups` mount is used only for local snapshots and local pre-restore safety backups. S3-only backups are written directly to the selected S3 destination and do not store a permanent local copy in `/backups`.
+
+Arcane maps this storage into each temporary Rustic container while it works. If the Arcane container does not have a mount at `/backups`, Arcane uses its fallback Docker volume and shows a warning in the backups UI.
+
+For local storage at a predictable host location, bind-mount a directory:
+
+```yaml
+services:
+  arcane:
+    volumes:
+      - /srv/arcane/backups:/backups
+```
+
+Alternatively, manage the local backup storage as a named volume in Compose:
+
+```yaml
+services:
+  arcane:
+    volumes:
+      - arcane-backups:/backups
+
+volumes:
+  arcane-backups:
+```
+
+A named volume remains inside Docker storage and may be lost if the Docker host or volume is removed. A bind mount makes the location of the local Rustic backup repository under `/backups` explicit, but it still needs separate protection from host or disk failure.
+
+### Rename the fallback backup volume
+
+Set this environment variable to avoid a name collision with another Docker volume:
+
+```env
+ARCANE_BACKUP_VOLUME_NAME=<your-name>
+```
+
+The default is `arcane-backups`. This setting only changes the fallback local Docker volume name. It does not change a host path mounted at `/backups` or the location of S3 backups.
+
+> [!IMPORTANT]
+> A local backup on the same host does not protect against disk or host failure. Use S3 or copy the local Rustic backup repository under `/backups` to another system for off-site recovery.
+
+## S3 destinations
+
+Open **Settings → S3 Destinations** to manage reusable storage configurations. Arcane supports AWS S3 and compatible services such as Backblaze B2, MinIO, Hetzner Object Storage, and similar providers.
+
+Each destination contains:
+
+- A display name
+- Endpoint URL
+- Bucket
+- Region
+- Access key and secret key
+- Optional object prefix
+- SSL and path-style access options
+
+The region is required for AWS S3. It can be left empty when a custom endpoint does not require one.
+
+### Test before saving
+
+The create and edit dialogs require a successful connection test before **Create** or **Save** becomes available. Changing a connection field invalidates the result and requires another test.
+
+The test writes a temporary object, downloads and verifies it, and then deletes it. Saving performs one final round-trip test on the backend so an unreachable destination cannot be persisted through the API.
+
+## Volume backups
+
+Open a volume and select its **Backups** tab. See <Link href="/docs/features/volumes">Volumes</Link> for the rest of the volume-management workflow.
+
+### Create an on-demand backup
+
+Click **Create Backup** for a local backup, or open its dropdown and choose:
+
+- **Local**
+- **S3**
+- **Local + S3**
+
+Choosing an option that includes S3 opens a dialog for selecting one of the saved S3 destinations.
+
+A completed row records the trigger, storage destination, destination name, size, creation time, and status. Failed runs remain visible with their error.
+
+### Schedule backups
+
+Click **Add Schedule** to create an independent backup policy for the volume. A volume can have multiple schedules, each with its own:
+
+- Enabled state
+- Six-field cron expression, including seconds
+- Retention count
+- Local, S3, or Local + S3 destination
+- S3 destination
+- **Stop containers during backup** option
+
+For example, `0 0 2 * * *` runs every day at 02:00. Set retention to `0` to keep every restore point.
+
+Scheduled runs and on-demand runs appear in the same backup table and Activity Center.
+
+### Container consistency
+
+Enable **Stop containers during backup** when applications may write to the volume while it is being copied. Arcane stops running containers that use the volume, creates the snapshot, and starts the containers again afterward.
+
+Leaving containers running avoids downtime, but applications with active writes may produce an inconsistent restore point.
+
+### Backup safety
+
+- Arcane waits for the Rustic container to finish and checks its exit code. A failed attempt remains in the backup table with a **Failed** status and its error; it is not treated as a usable restore point.
+- Before a whole-volume or selected-file restore, Arcane stops containers that use the volume and creates a local safety backup. If the safety backup fails, Arcane aborts without restoring any data.
+- Rustic restores directly into the target volume. A whole-volume restore uses Rustic's delete mode so files absent from the selected snapshot are removed.
+- If a restore fails after it starts writing, the volume may be partially changed. The local safety backup remains available for rollback, and Arcane attempts to restart every container it stopped.
+
+### Encryption
+
+Rustic encrypts every volume backup automatically. You do not need to configure a separate recovery key.
+
+Arcane derives the repository password from its internal `ENCRYPTION_KEY`. Keep the original key if you need to open the repository from another Arcane installation.
+
+> [!WARNING]
+> A fresh Arcane instance with a different `ENCRYPTION_KEY` cannot decrypt existing volume-backup repositories.
+
+### Upload an existing local backup
+
+A successful local backup can be uploaded later. Open its row actions, select **Upload to S3**, and choose a configured destination. The row then represents a Local + S3 backup.
+
+### Restore
+
+Arcane can restore the whole volume or selected files. Before changing any data, it creates a local safety backup.
+
+For volumes used by running containers, Arcane stops the affected containers, creates the safety backup, restores the data with Rustic, and starts the containers again.
+
+### Delete and retention
+
+Retention is applied independently per schedule. When a restore point expires, Arcane removes its local and remote Rustic snapshots where possible.
+
+Manual and bulk deletion also attempt to remove every stored copy:
+
+- If every copy is deleted, the row is removed.
+- If a local copy is deleted but S3 deletion fails, the row remains as S3-only and Arcane reports the error.
+- If a remaining remote copy cannot be deleted, Arcane keeps the row so the backup is not incorrectly reported as gone.
+
+## Arcane system backups
+
+System backups protect Arcane's persistent application data and runtime configuration so a replacement instance can be restored as a clone.
+
+Open **Settings → System Backups**. This feature requires Arcane to run in Docker with `/app/data` mounted and access to its local Docker daemon.
+
+### Recovery key
+
+System backups use a separate recovery key rather than Arcane's internal volume-backup key:
+
+1. Click **Set up recovery key**.
+2. Enter at least 16 characters.
+3. Store the key somewhere outside Arcane.
+
+The saved copy lets scheduled jobs run unattended. You still need an external copy to recover a lost installation.
+
+> [!WARNING]
+> Losing the recovery key makes the snapshots unrecoverable. Replacing it does not re-encrypt older backups, so retain previous keys for their existing restore points.
+
+### Manual and scheduled backups
+
+System backups support Local, S3, and Local + S3 destinations. You can create multiple schedules with independent cron expressions, destinations, S3 targets, and retention counts.
+
+For an on-demand backup, use a saved schedule's configuration or choose a custom destination. Existing local backups can also be uploaded to S3 later.
+
+The backup table shows each run's status, trigger, destination, size, and creation time. Schedule cards show the latest run's status and time. Use **Find S3 backups** with a destination and recovery key to discover restore points that are not present in the current database.
+
+### Restore Arcane
+
+Restoring replaces the running Arcane installation:
+
+1. Arcane creates and records a local safety backup.
+2. A detached recovery helper stops the Arcane container.
+3. Rustic restores the selected snapshot into `/app/data`.
+4. The helper recreates Arcane with the recovered runtime configuration.
+5. The recovered container starts, and the backup and Activity Center records are finalized.
+
+The page disconnects while Arcane restarts. Reload it after the container is available again.
+
+> [!CAUTION]
+> A system restore replaces Arcane's current database, users, settings, destinations, secrets, and other persistent application data with the selected restore point.

content/features/volumes.md

@@ -4,6 +4,7 @@ description: 'Manage Docker volumes in Arcane, including backups and restores.'
 ---
 
 <script lang="ts">
+import { Link } from '#lib/components/ui/link/index.js';
 import ScreenshotFrame from '#lib/components/screenshot-frame.svelte';
 </script>
 
@@ -38,34 +39,12 @@ Open **Volumes** in the sidebar. The table shows name, driver, and current usage
 
 ## Back up and restore
 
-Arcane runs a short-lived helper container to `tar` the volume contents into a backup, and reverses the process on restore.
+Open a volume and select **Backups** to create or schedule encrypted Rustic snapshots. Backups can use local storage, an S3-compatible destination, or both.
 
-### Backup storage
+Arcane supports multiple schedules per volume, retention policies, optional container shutdown for consistent snapshots, whole-volume restores, selected-file restores, and local safety backups before a restore.
 
-Backups are stored in a dedicated Docker volume mounted into the helper container at `/backups`. If the Arcane container itself doesn't have a host-backed mount at `/backups`, the backups UI shows a warning so you know backups only live inside Docker storage.
-
-To keep backups somewhere predictable, mount a host path or named volume to `/backups` in your `compose.yaml`:
-
-- Host path: `/srv/arcane/backups:/backups`
-- Named volume: `arcane-backups:/backups`
-
-If you use a named volume, declare it under the top-level `volumes:` section too.
-
-### Backup safety
-
-- Arcane waits for the backup container to finish and checks its exit code. If `tar` fails, the backup isn't recorded — you get an error instead of a silent failure.
-- Restore extracts the backup into a temporary directory first. Only after extraction succeeds does Arcane wipe the volume and move the data into place. If the restore container exits non-zero, you get an error noting that the volume may be partially changed.
-
-### Rename the backup volume
-
-Set this environment variable to avoid name collisions with your own volumes:
-
-```
-ARCANE_BACKUP_VOLUME_NAME=<your-name>
-```
-
-Default: `arcane-backups`.
+See <Link href="/docs/features/backups">Backups</Link> for storage setup, S3 destinations, encryption, scheduling, retention, and recovery instructions.
 
 ## Helper containers
 
-Arcane creates short-lived helper containers for backup and restore work. They carry the label `com.getarcaneapp.internal.resource=true` and are hidden from the Containers list by default. Toggle **Show Internal Containers** in the Containers view to see them.
+Arcane creates short-lived Rustic containers for backup and restore work. They carry the label `com.getarcaneapp.internal.resource=true` and are hidden from the Containers list by default. Toggle **Show Internal Containers** in the Containers view to see them.

src/lib/config/docs.ts

@@ -95,6 +95,7 @@ const FEATURES = group('Features', [
 	leaf('features/images'),
 	leaf('features/image-builds'),
 	leaf('features/volumes'),
+	leaf('features/backups'),
 	leaf('features/networks'),
 	leaf('features/vulnerability-scans'),
 	leaf('features/environments'),