Skip to content

feat(deploy): restore-on-install from a live deployment (spec 007) - #491

Merged
pofallon merged 5 commits into
mainfrom
007-restore-on-install
Sep 20, 2026
Merged

pofallon merged 5 commits into
mainfrom
007-restore-on-install

Conversation

@pofallon

Copy link
Copy Markdown
Contributor

Closes #429.

During an app install the operator may name an existing deployment of the same app on this host as a restore source. The server quiesces it with the backup contract's existing pre-hooks, captures its data root, and lays that data into the new install after images are pulled and before any container starts.

That timing is the whole reason the design is small. Restoring into a running app is hard — the data root is full, a database holds files open, a half-applied restore leaves a state nobody can reason about. At install time every one of those problems is absent by construction: createFromDraft mints a fresh id, the data root does not exist until the job creates it, and no container has ever started.

Why there is no backup provider in this

The source is a live deployment on the same host, so the server is both reader and writer — exactly as it already is for a data-aware rollback. No capability contract, no grant, no contract endpoint, no new JobType. packages/shared/src/contracts.ts is untouched.

Every subtle part of restore lives on the install side. Shipping those against a source that needs no contract negotiation means Sequence 6 (restore@1, the provider half) arrives with one new job — put the files in a staging directory — against machinery that has been in production for a release.

The two failures that govern the design

  1. A restore that cannot succeed refuses before any container starts. There is no partial-success state worth starting an app in. An operator who asked for their data and got a working-but-empty Gitea discovers it weeks later.
  2. A restore that carries data carries the configuration that data was written under — or names loudly every generated secret it could not carry. Data encrypted under a platform-generated key is unreadable beside a freshly generated one, and nothing in the running app reports that as an error.

Design summary

Almost entirely reuse: capturePreUpgradeSnapshot's hook-and-tar path, runPreHooksFailClosed's fail-closed policy, restoreTarGzInto, mergeUpgradeAppEnv's three-case rule, resolveContainedDir's containment, checkUpgradePath's skew rules, writeInstanceMarkers, and grants' consent shape. Genuinely new: a candidate resolver, the restore sequence in the lifecycle job, and composeUp's service-list/wait option.

The ten-step sequence in runLifecycleJob, between the cancellation check and composeUp:

assert target empty → re-resolve candidate → quiesce + capture → extract → assert payload present → apply discards → rewrite instance marker → write OIDC file → up -d --wait <hook services> → run restore hooks fail-closed

Three of those steps exist because of traps that are invisible until they bite:

  • writeOidcCredentialsFile moves to after extraction. Left where it was, a whole-root restore destroys the file it just wrote and the app boots with SSO silently disabled.
  • The instance marker is rewritten. The restored tree carries the source's record; left in place the new install claims to be the old one and every later capture compounds it.
  • lineageId is promoted onto the deployment record — the change spec 006 predicted, in the code it now changes. The ?? deployment.id fallback makes it zero-migration.

Review found three defects worth naming

Shell injection composeUp interpolated manifest-supplied service names into a string run through exec — in the same file whose backup path builds argv explicitly "so provisioned values can't be injected". Now argv via execFileAsync, plus a Compose-service-name pattern at coercion. New, not pre-existing; mutation-proved exploitable before the fix.
discard: ["."] wipes the payload resolveContainedDir treats the root itself as contained, so a manifest could rm -rf the restored tree after the post-condition passed and start the app empty reporting success. Now also requires isStrictlyInside.
Wizard judged no skew at all It requested candidates without a version, so every candidate returned skew: unknown — demanding an acknowledgement from every operator on every restore, and never surfacing a real version refusal until the draft create failed.

Four prompt claims corrected against the post-spec-006 tree

Recorded in the artifacts with their evidence, so they aren't re-inherited:

  • the job payload is { releaseId, action }deploymentId is a sibling Job field, never a payload key;
  • there is no CreateDeploymentRequest type; it is CreateDeploymentFromDraftRequest;
  • the "locate the subtree at <staging><candidate.path>" trap describes a provider archive tool — this codebase's helpers are root-relative, so FR-016 is a post-condition assertion rather than a subtree search (the trap is recorded for Sequence 6 in Sequence 6 (restore@1 provider) inherits the absolute-path subtree trap FR-016 generalises #486);
  • checkUpgradePath cannot express "refuse a newer source" — it returns ok for anything that is not a forward upgrade, so that rule is stated independently and evaluated first.

Test evidence

Suite Before After
server 1086 1130
web 360 367
cli 267 274

bun run typecheck && bun run lint && bun run typecheck && bun run test && bun run build — green (typecheck twice, per the CLAUDE.md note).

Three fixes are mutation-proved, not merely covered: the injection probe, the FR-014 end-to-end refusal, and the wizard rollback. The OIDC ordering test was separately verified to fail when the write is returned to its old position — a test that passes either way tests nothing.

Spec artifacts: 53 FRs, 13 SCs, 57 numbered verification scenarios, 88 tasks, 100% coverage both directions, Constitution Check clean with Complexity Tracking legitimately empty.

Gates that still apply before merge

  • Manual VM pass not yet run (task T079, quickstart scenarios 17, 24, 28, 33, 53, 56): source-untouched-after-capture, failing-hook-aborts, a real mealie restore, real Postgres discard-and-load, the Clone-with-data rehearsal: hola install <app> --channel <c> --from <deployment> #429 clone, and a restore with no backup provider installed. These need real containers and a real database.
  • The try-hola/apps catalog change is prepared but NOT submitted — schema + 5 manifests, all 17 catalog manifests validated with ajv. It targets a repo needing explicit per-instance permission. The platform half is useful without it: 12 of 17 acceptor apps restore by plain file copy today.

Follow-up issues filed

#484 (dead /api/backups/:id/restore stub, points at #160) · #485 (stale line-number comment) · #486 (Sequence 6 inherits the subtree trap) · #487 (composeUp's 5-minute default) · #488 (--carry-env arg parsing) · #489 (a restore failing after extraction leaves the install unstartable) · #490 (FR-035's subdomain default cannot succeed against a live candidate)

🤖 Generated with Claude Code

https://claude.ai/code/session_01Vck5KSX2CLxhohx14nb5Sh

Paul O'Fallon and others added 5 commits September 20, 2026 20:00
During an app install the operator may name an existing deployment of the
same app on this host as a restore source. The server quiesces it with the
backup contract's existing pre-hooks, captures its data root, and lays that
data into the new install AFTER images are pulled and BEFORE any container
starts — the one moment in an app's life when the data root is empty and no
process holds it open, so a restore cannot overwrite anything, cannot race a
live process, and needs no stop/start choreography.

This slice deliberately contains no backup provider: the source is a live
deployment on the same host, so the server is both reader and writer exactly
as it already is for a data-aware rollback. No capability contract, no grant,
no contract endpoint, no new job type. Sequence 6 (`restore@1`, the provider
half) is the consumer of this groundwork.

The design is almost entirely reuse — `capturePreUpgradeSnapshot`'s hook-and-tar
path, `runPreHooksFailClosed`'s fail-closed policy, `restoreTarGzInto`,
`mergeUpgradeAppEnv`'s three-case rule, `resolveContainedDir`'s containment,
`checkUpgradePath`'s skew rules, `writeInstanceMarkers`, and `grants`' consent
shape. Genuinely new: a candidate resolver, the restore sequence in the
lifecycle job, and `composeUp`'s service-list/wait option.

Two failures govern the whole design. A restore that cannot succeed refuses
before any container starts (there is no partial-success state worth starting
an app in), and a restore that carries data carries the configuration that data
was written under — or names loudly every generated secret it could not carry,
since data encrypted under a key the platform generated is unreadable beside a
freshly generated one and nothing in the running app reports that as an error.

Also promotes `lineageId` onto the deployment record, which spec 006 predicted
in the shipped code it now changes. The `?? deployment.id` fallback makes this
zero-migration: an older record reads undefined and yields the value it always
wrote, and captures taken since spec 006 already carry the field.

Review caught and fixed three defects worth naming. A shell-injection sink:
`composeUp` interpolated manifest-supplied service names into a string run
through `exec`, in the same file whose backup path builds argv explicitly "so
provisioned values can't be injected" — now argv via `execFileAsync`, plus a
Compose-service-name pattern at coercion. A `discard: ["."]` could wipe the
restored payload after the post-condition passed, because `resolveContainedDir`
treats the root as contained — now also requires `isStrictlyInside`. And the
wizard judged no version skew at all, demanding an acknowledgement from every
operator on every restore.

Four of the prompt's own claims were wrong against the post-spec-006 tree and
are corrected in the artifacts with evidence: the job payload's shape, the
absent `CreateDeploymentRequest` type, a "locate the subtree" trap that
describes a provider archive tool rather than this codebase's root-relative
helpers, and `checkUpgradePath`'s inability to express "refuse a newer source"
(it returns ok for anything that is not a forward upgrade).

53 FRs, 13 SCs, 57 verification scenarios, 88 tasks. Server 1086 -> 1130,
web 360 -> 367, CLI 267 -> 274. Three fixes are mutation-proved: the injection
probe, the FR-014 end-to-end refusal, and the wizard rollback.

The try-hola/apps catalog change (schema + 5 manifests) is prepared but NOT
submitted — it targets a repo requiring explicit per-instance permission.

Closes #429

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vck5KSX2CLxhohx14nb5Sh
… not line

The comment at install-markers.test.ts read "~:3410 (restore) before ~:3414
(materialize)". A single intervening feature merge (spec 006, 921c790) moved
both call sites by roughly 380 lines while the ordering claim itself stayed
true, so the citation was wrong within one release of being written.

Cite the symbols instead — `restoreAppDataSnapshot` runs before
`materializeCompose` in `runLifecycleJob`'s deploy/start/rollback branch —
and record why, because this is the second time in two specs that stale
line-number citations have cost verification work: spec 007's planning had to
re-verify roughly thirty anchors against the current tree because its prompt's
numbers predated the same merge, several of them wrong in shape rather than
merely in position.

A citation that rots faster than the invariant it documents is worse than no
citation: the next reader has to verify it before trusting the sentence built
around it.

Closes #485

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vck5KSX2CLxhohx14nb5Sh
Both defects are the same shape: restore-on-install refuses for the right
reason and tells the operator nothing they can act on. Neither refusal
changes — FR-022 forbids starting an app on a data root the platform cannot
vouch for, and an address is an operator decision — only what the refusal
says, and the structured `details` it carries (FR-037).

#489 — a restore that fails AFTER extraction left the install permanently
unstartable behind a generic `RESTORE_TARGET_NOT_EMPTY`, which reads as
"somebody else's data is in the way" when in fact the data is this install's
own half-landed payload. `restoreStartedAt` is now persisted (and flushed,
so it survives a hard kill mid-extraction) at the last read-only moment of
the sequence, and `classifyNonEmptyTarget` uses it to split the refusal into
`RESTORE_INCOMPLETE` — naming the state and the recovery, uninstall +
reinstall or clear the data root — and the unchanged generic code. The field
is a diagnostic only: the job-entry gate is untouched, so a retry still
enters the restore sequence and still refuses.

Two alternatives from the issue are rejected in code comments so they are
not re-proposed: persisting an attempted-marker so a retry takes the
no-restore path, and re-running only the post-extraction steps. Both can
start an app on a half-extracted tree — the silent-empty-app outcome the
spec's Executive Summary names as the worst in this feature.

#490 — FR-035 defaults a restored install's address to the candidate's own,
but a candidate is by definition a deployment that still exists on this host
and still owns that route, so the default could never succeed; it fell
through to `routingService.validateRule`'s bare host `CONFLICT`, which
carries no `details.code` and never mentions restore.
`resolveRestoreNameDefaults` now refuses with `RESTORE_ADDRESS_REQUIRED`,
naming the candidate and the address it holds, and only when the caller
supplied no name — an operator who names the install has made the address
decision and gets the routing layer's own conflict. It compares against the
candidate's LIVE subdomain (its deployment record), not its identity
record's, which can be stale. Deriving a distinct/suffixed slug is rejected
in a comment: that would put data full of the old address's absolute URLs at
a new address nobody chose.

Also: a CLI hint branch for `RESTORE_ADDRESS_REQUIRED` built from `details`;
`RESTORE_INCOMPLETE` gets none, because job-time codes reach `hola install`
as a failed job, not a `HolaApiError` — which is why its recovery is spelled
out in the server's message. data-model.md §4, contracts/api.md and
contracts/cli.md updated to agree on the code list, with the create-time vs
job-time split now nine/four.

Closes #489
Closes #490

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vck5KSX2CLxhohx14nb5Sh
…ssion (#487)

Spec 007 turned composeUp's hardcoded 300000ms into a parameter with a
default, which left every other call site on a five-minute cap nobody had
ever chosen for it. Audited all three sites in deployment.ts:

- the restore-hook `--wait` (research R12) already passes 15 minutes —
  unchanged;
- the deploy/start/rollback bring-up and the restart recreate now pass
  COMPOSE_UP_TIMEOUT_MS (15 minutes) explicitly, with the reason at the
  call site.

Five minutes is too thin for what these actually do. `up -d` is not just
container creation: Compose blocks on every `depends_on: service_healthy`
gate, so a first install of a large multi-service stack waits out an
`initdb` plus each dependency's healthcheck — the same cost R12 measured
for the restore hook, which is why this reuses that number rather than
inventing a third. A rollback that just replaced the data directory and a
restore-on-install whose database just received a large dump start equally
cold. The restart path is additionally the one composeUp with no
composePull in front of it, so a recreate that must fetch a pruned image
was running under a cap six times tighter than the one composePull was
given for exactly that reason.

The ceiling only delays reporting a wedged `up`; too low SIGKILLs a
slow-but-correct install. composeUp's own 300000ms stays as a fallback for
a caller with no opinion and is commented as such.

Closes #487

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vck5KSX2CLxhohx14nb5Sh
`hola install --carry-env gitea --restore-from latest` aborted with sade's
"Insufficient arguments!": mri classifies a flag as boolean only by
`typeof opts.default[key]`, and `--carry-env` is registered without a
default, so mri treated it as value-taking and ate `gitea`.

It cannot simply take a `false` default — `carryEnv` is tri-state
(contracts/cli.md): `undefined` means "default to the candidate's own
`carriesEnv`", `true`/`false` are explicit operator choices, and a `false`
default collapses the first into the second. Normalising after the fact,
the way `streamOpts` handles `--no-stream`, cannot help either: the token
is consumed during parsing, before any handler sees an opts bag.

So classify it at the parse layer instead. `parseOpts()` names `carry-env`
in mri's `boolean` list (sade forwards the object verbatim, overriding only
`alias`/`default`), which makes it boolean WITHOUT giving it a default —
and mri's boolean branch pushes the token it looked at back onto `_`, so
the positional is handed back rather than lost. Tri-state is preserved
exactly: omitted → undefined, `--carry-env` → true, `--no-carry-env` and
`--carry-env=false` → false. No documented behaviour changed, so
contracts/cli.md is untouched.

Tests parse the real flag shapes through sade rather than asserting on a
hand-built opts bag, since the bug lives in parsing: the issue's exact
repro, both flag positions, the `=`-form, the omitted case, and a guard
that index.ts actually hands `parseOpts()` to `prog.parse`.

Closes #488

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vck5KSX2CLxhohx14nb5Sh
@pofallon

Copy link
Copy Markdown
Contributor Author

Issue sweep folded in (5 commits, one CI run)

The sweep normally lands as its own PR off main, but #488/#489/#490 are defects in code that exists only on this branch — merging this PR with two documented dead-ends in its own new flows and fixing them afterwards would be worse than fixing them first. So they are folded in here.

Issue Fix
#485 Cite the restore-before-materialize ordering by symbol, not line number — the old citation was wrong within one release of being written
#487 Audit found three composeUp call sites; two inherited 5 min by omission. Both now state 15 min explicitly, reusing R12's number rather than inventing a third
#488 --carry-env classified as boolean via prog.parse(argv, { boolean: [...] }), preserving the tri-state that made the obvious fixes wrong
#489 New RESTORE_INCOMPLETE naming the state and the recovery
#490 New RESTORE_ADDRESS_REQUIRED naming the candidate, instead of a bare routing CONFLICT

The judgement call on #489 and #490

Both describe a restore that refuses correctly but illegibly. The refusal is right in each case and stays; what was broken is that the operator learned nothing actionable.

Two of #489's three suggested directions — persist an attempted-marker before extraction, or re-run only the post-extraction steps — would let a half-extracted tree start the app. Nothing can distinguish "extraction finished, a discard failed" from "extraction died halfway", so both trade a safe-but-terminal state for a possibly-silently-corrupt running one. That is exactly the trade FR-022 forbids. Rejected, with the reasoning written into the code so it is not re-proposed.

#490's auto-suffixed-slug option was rejected for a parallel reason: it would stand up an app at a new address holding data full of the old address's absolute URLs — manufacturing the divergence FR-035 exists to warn about, without the operator choosing it.

One correction to the brief, caught by the implementer

My specification for #489's trigger (restoreFrom set + restoredAt unset + root non-empty) is true at every firing of that check, so taken literally it would have made RESTORE_TARGET_NOT_EMPTY unreachable and broken scenario 16. restoreStartedAt is the actual discriminator — "this install's restore wrote here" vs "something else put data here".

It is persisted immediately before extraction, which superficially resembles the rejected attempted-marker, so to be explicit: it is a diagnostic, never a gate. Verified after integration — the job-entry condition is still restoreFrom && !restoredAt && !previousReleaseId, and restoreStartedAt appears nowhere in any conditional that decides whether to restore.

Gate, re-run once on the integrated branch

Suite At PR open Now
server 1130 1133
web 367 367
cli 274 282

typecheck → lint → typecheck → test → build all green. Both agent branches deleted.

Still outstanding, unchanged

  • Manual VM pass (T079) — scenarios 17, 24, 28, 33, 53, 56, needing real containers and a real database.
  • The try-hola/apps catalog change is prepared but not submitted — it targets a repo requiring explicit per-instance permission.

One out-of-scope finding worth its own issue later: --no-self-update and --no-backup are also registered without defaults. They cannot swallow a positional (mri's no- branch short-circuits), so this is not #488, but they land as selfUpdate: false/backup: false rather than the no-prefixed keys — worth confirming their handlers read the de-prefixed name.

@pofallon
pofallon merged commit 5588820 into main Sep 20, 2026
3 checks passed
@pofallon
pofallon deleted the 007-restore-on-install branch September 20, 2026 22:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Clone-with-data rehearsal: hola install <app> --channel <c> --from <deployment>

1 participant