Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 42 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,47 @@
# Changelog

## 3.0.0 (unreleased)
## 3.1.2

3.1.2 reverses the 3.0.0 notes below on `Aborted`, `markAborted()` and finalizing with `success()`.

### Breaking changes

- Requires `utopia-php/database` `^8.1.2` (3.1.1 requires `^8.0`), which moved the update models into the namespace
of the model they update.
- Provisioning ownership fencing is removed. The Appwrite destination tracks a database's lifecycle with `status`
alone again, but recovers incomplete databases differently from 2.x.
- `Destinations\Appwrite\ProvisioningOwner` is removed.
- `Destinations\Appwrite::__construct()` no longer takes `owner`, `getRecoverableOwner` or `provisioningLease`, and
`Destinations\Appwrite::DEFAULT_PROVISIONING_LEASE` is removed. `onDuplicate` and the arguments after it move up
two places for positional callers.
- The destination no longer reads or writes the `migrationId` and `migrationAttemptId` database attributes.
- A `provisioning` or `failed` database is recovered by the next run at once, without owner attestation or a lease
wait. It is always overwritten from the source, whatever `OnDuplicate` is and whatever the timestamps say. 2.1.0
forced the overwrite only for a `failed` database and never under `OnDuplicate::Fail`; it skipped a
`provisioning` database whose spec matched the source and only re-enrolled it for the `ready` flip.
- The standalone CLI no longer accepts `--migration-id`, `--migration-attempt-id`, `--recover-migration-id`,
`--recover-migration-attempt-id` or `-h`/`--help`. `MigrationCLI::__construct(array $arguments)` and
`MigrationCLI::getHelp()` are removed.
- `Destinations\Appwrite::run()` finalizes again, as in 2.x: once the transfer returns it flips the databases it
provisioned to `ready`, then runs the overwrite orphan sweep.
- `success()`, `error()` and `cleanUp()` are no-op lifecycle hooks again. Callers no longer call `success()` to
finalize, and `cleanUp()` no longer reports a skipped `success()`.
- A failed `ready` flip is recorded with `addError()`, and the sweep still runs.
- The orphan sweep stops at the first table that fails, and `run()` throws that table's database exception as is.
3.1.1's `success()` tried every table, recorded each failure with `addError()` and threw `Finalization`.
- `Utopia\Migration\Exception\Finalization` is removed.
- The abort mechanism is removed.
- `Utopia\Migration\Exception\Aborted` and `Destination::markAborted()` are removed.
- `Transfer::run()` no longer latches an exception thrown from its progress callback, and the library sources no
longer rethrow one: each source handles it as in 2.x.

### Concurrency

- The Appwrite destination treats every `provisioning` or `failed` database as abandoned by an earlier run. Callers
must serialise migrations per destination project. A concurrent run overwrites another run's incomplete databases
from its own source under any `OnDuplicate`, and may flip them to `ready` before the other run finishes.

## 3.0.0

3.0 is a major release. The breaking changes below all concern the Appwrite destination's lifecycle and the
database library it builds on; read the upgrade notes before you upgrade from 2.x.
Expand Down
12 changes: 0 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,18 +44,6 @@ $transfer->run(
);
```

## Appwrite database recovery

Appwrite database destinations use a `ProvisioningOwner` made from a stable logical migration identifier and a fresh attempt identifier for every execution. The required `getRecoverableOwner` callback is the recovery authority for an existing database whose status is `provisioning` or `failed`.

A database status is local to that resource. It does not prove that the migration attempt which owns it has stopped, because an import can continue with other resources after recording a database failure. The callback must therefore consult the caller's authoritative operation lifecycle and return the exact stored owner only after that attempt is terminal. Return `null` while it is active or unknown; recovery then fails closed. This rule also applies when the retry uses the same logical migration identifier.

Owners are written and enforced only when the destination's `databases` metadata declares both `migrationId` and `migrationAttemptId`; metadata with `status` alone gets status tracking without ownership. A `provisioning` or `failed` database that names no owner at all has nothing for the callback to judge, so its update timestamp stands in: it is recovered only once it has gone a whole `provisioningLease` without a write, since a fresher one may still belong to a migration that is provisioning it. A row still within the lease is refused with an error naming the database. Past the lease it is recovered as it was before ownership existed: a `failed` database is overwritten, a `provisioning` one is resolved like any existing database under the `OnDuplicate` policy (`fail` keeps it instead of colliding) and marked `ready` when the run succeeds, and either is claimed by the current attempt when the metadata can record it. A database naming only part of an owner is refused.

The `provisioningLease` constructor argument is the lease in seconds and defaults to `86400` (24 hours), the longest lease a live Appwrite migration attempt holds; `0` recovers an ownerless database at once, and a negative value is rejected with an `InvalidArgumentException`.

The standalone CLI requires `--migration-id` and a fresh `--migration-attempt-id`. Recovering an incomplete database that names an owner additionally requires both `--recover-migration-id` and `--recover-migration-attempt-id` for the exact terminal prior attempt.

## Supported Resources Chart

Sources:
Expand Down
106 changes: 3 additions & 103 deletions bin/MigrationCLI.php
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@
use Utopia\Database\Query;
use Utopia\Migration\Destination;
use Utopia\Migration\Destinations\Appwrite as DestinationsAppwrite;
use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner;
use Utopia\Migration\Destinations\Local;
use Utopia\Migration\Source;
use Utopia\Migration\Sources\Appwrite;
Expand All @@ -34,9 +33,6 @@ class MigrationCLI

protected mixed $destination;

/** @var list<string> */
private readonly array $arguments;

protected const STRUCTURE = [
'$collection' => 'databases',
'$id' => 'collections',
Expand Down Expand Up @@ -157,34 +153,6 @@ class MigrationCLI
],
];

/** @param list<string> $arguments */
public function __construct(array $arguments = [])
{
$this->arguments = $arguments;
}

public static function getHelp(): string
{
return <<<'HELP'
Usage: php bin/MigrationCLI.php [options]

Options:
-h, --help Show this help.
--migration-id=<identifier> Stable logical owner identifier for this migration.
Required for Appwrite; reuse it for retries.
--migration-attempt-id=<identifier>
Required for Appwrite; use a fresh attempt for every
execution or retry.
--recover-migration-id=<prior-migration-id>
--recover-migration-attempt-id=<prior-attempt-id>
Together attest that the exact prior migration attempt is terminal
and allow recovery of its provisioning or failed databases.
Recovery is refused by default.
Resource status alone never proves an attempt terminal.

HELP;
}

/**
* Prints the current status of migrations as a table after wiping the screen
*/
Expand Down Expand Up @@ -283,7 +251,6 @@ public function getDestination(): Destination
switch ($_ENV['DESTINATION_PROVIDER']) {
case 'appwrite':
$database = $this->getDatabase('destination');
$recoverableOwner = $this->getRecoverableOwner();

return new DestinationsAppwrite(
project: $_ENV['DESTINATION_APPWRITE_TEST_PROJECT'],
Expand All @@ -294,10 +261,6 @@ public function getDestination(): Destination
collectionStructure: self::STRUCTURE,
dbForPlatform: $database,
projectInternalId: $_ENV['DESTINATION_APPWRITE_TEST_PROJECT_INTERNAL_ID'],
owner: new ProvisioningOwner($this->getMigrationId(), $this->getMigrationAttemptId()),
// The standalone operator supplies the fixed pair only after confirming the prior
// lifecycle owner is terminal. The destination independently compares it with the row.
getRecoverableOwner: static fn (Document $document): ?ProvisioningOwner => $recoverableOwner,
);
case 'local':
return new Local('./localBackup');
Expand All @@ -306,62 +269,6 @@ public function getDestination(): Destination
}
}

private function getMigrationId(): string
{
foreach ($this->arguments as $argument) {
if (! \str_starts_with($argument, '--migration-id=')) {
continue;
}

$migrationId = \substr($argument, \strlen('--migration-id='));
if ($migrationId !== '') {
return $migrationId;
}
}

throw new \InvalidArgumentException('--migration-id is required for an Appwrite destination');
}

private function getMigrationAttemptId(): string
{
foreach ($this->arguments as $argument) {
if (! \str_starts_with($argument, '--migration-attempt-id=')) {
continue;
}

$attemptId = \substr($argument, \strlen('--migration-attempt-id='));
if ($attemptId !== '') {
return $attemptId;
}
}

throw new \InvalidArgumentException('--migration-attempt-id is required for an Appwrite destination');
}

private function getRecoverableOwner(): ?ProvisioningOwner
{
$migrationId = null;
$attemptId = null;

foreach ($this->arguments as $argument) {
if (\str_starts_with($argument, '--recover-migration-id=')) {
$value = \substr($argument, \strlen('--recover-migration-id='));
$migrationId = $value !== '' ? $value : null;
}

if (\str_starts_with($argument, '--recover-migration-attempt-id=')) {
$value = \substr($argument, \strlen('--recover-migration-attempt-id='));
$attemptId = $value !== '' ? $value : null;
}
}

if ($migrationId === null || $attemptId === null) {
return null;
}

return new ProvisioningOwner($migrationId, $attemptId);
}

public function getDatabase(string $type): Database
{
self::registerFilters();
Expand Down Expand Up @@ -534,8 +441,6 @@ function () {
$this->drawFrame();
}
);

$this->destination->success();
}

protected function loadEnvironment(): void
Expand All @@ -546,12 +451,7 @@ protected function loadEnvironment(): void
}

if (\realpath($_SERVER['SCRIPT_FILENAME'] ?? '') === __FILE__) {
$arguments = $_SERVER['argv'] ?? [];
if (\in_array('-h', $arguments, true) || \in_array('--help', $arguments, true)) {
echo MigrationCLI::getHelp();
} else {
$instance = new MigrationCLI($arguments);
$instance->start();
$instance->drawFrame();
}
$instance = new MigrationCLI();
$instance->start();
$instance->drawFrame();
}
11 changes: 2 additions & 9 deletions composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,7 @@
"migration"
],
"license": "MIT",
"minimum-stability": "dev",
"prefer-stable": true,
"minimum-stability": "stable",
"autoload": {
"psr-4": {
"Utopia\\Migration\\": "src/Migration"
Expand All @@ -33,7 +32,7 @@
"ext-curl": "*",
"ext-openssl": "*",
"appwrite/appwrite": "^30.2.2",
"utopia-php/database": "^8.0",
"utopia-php/database": "^8.1.2",
"utopia-php/storage": "4.*",
"utopia-php/dsn": "0.2.*",
"halaxa/json-machine": "^1.2",
Expand All @@ -53,11 +52,5 @@
"php-http/discovery": true,
"tbachert/spi": true
}
},
"extra": {
"branch-alias": {
"dev-feat-query-lib": "3.0.x-dev",
"dev-main": "3.0.x-dev"
}
}
}
18 changes: 9 additions & 9 deletions composer.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 0 additions & 10 deletions src/Migration/Destination.php
Original file line number Diff line number Diff line change
Expand Up @@ -96,16 +96,6 @@ public function runWithResourceSelector(
}
}

/**
* Report an abort that a source swallowed and Transfer latched, before Transfer rethrows it;
* an abort that propagates straight out of run() never reaches here. A destination that
* defers its terminal state to success() drops that state here, so a later success()
* finalizes nothing, exactly as after a run that threw.
*/
public function markAborted(): void
{
}

/**
* Import Resources
*
Expand Down
Loading
Loading