diff --git a/CHANGELOG.md b/CHANGELOG.md index d8f52143..2b906e08 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,47 @@ # Changelog -## 3.0.0 (unreleased) +## 3.1.2 + +3.1.2 reverses the 3.0.0 notes below on `Aborted`, `markAborted()` and finalizing with `success()`. + +### Breaking changes + +- Requires `utopia-php/database` `^8.1.2` (3.1.1 requires `^8.0`), which moved the update models into the namespace + of the model they update. +- Provisioning ownership fencing is removed. The Appwrite destination tracks a database's lifecycle with `status` + alone again, but recovers incomplete databases differently from 2.x. + - `Destinations\Appwrite\ProvisioningOwner` is removed. + - `Destinations\Appwrite::__construct()` no longer takes `owner`, `getRecoverableOwner` or `provisioningLease`, and + `Destinations\Appwrite::DEFAULT_PROVISIONING_LEASE` is removed. `onDuplicate` and the arguments after it move up + two places for positional callers. + - The destination no longer reads or writes the `migrationId` and `migrationAttemptId` database attributes. + - A `provisioning` or `failed` database is recovered by the next run at once, without owner attestation or a lease + wait. It is always overwritten from the source, whatever `OnDuplicate` is and whatever the timestamps say. 2.1.0 + forced the overwrite only for a `failed` database and never under `OnDuplicate::Fail`; it skipped a + `provisioning` database whose spec matched the source and only re-enrolled it for the `ready` flip. +- The standalone CLI no longer accepts `--migration-id`, `--migration-attempt-id`, `--recover-migration-id`, + `--recover-migration-attempt-id` or `-h`/`--help`. `MigrationCLI::__construct(array $arguments)` and + `MigrationCLI::getHelp()` are removed. +- `Destinations\Appwrite::run()` finalizes again, as in 2.x: once the transfer returns it flips the databases it + provisioned to `ready`, then runs the overwrite orphan sweep. + - `success()`, `error()` and `cleanUp()` are no-op lifecycle hooks again. Callers no longer call `success()` to + finalize, and `cleanUp()` no longer reports a skipped `success()`. + - A failed `ready` flip is recorded with `addError()`, and the sweep still runs. + - The orphan sweep stops at the first table that fails, and `run()` throws that table's database exception as is. + 3.1.1's `success()` tried every table, recorded each failure with `addError()` and threw `Finalization`. + - `Utopia\Migration\Exception\Finalization` is removed. +- The abort mechanism is removed. + - `Utopia\Migration\Exception\Aborted` and `Destination::markAborted()` are removed. + - `Transfer::run()` no longer latches an exception thrown from its progress callback, and the library sources no + longer rethrow one: each source handles it as in 2.x. + +### Concurrency + +- The Appwrite destination treats every `provisioning` or `failed` database as abandoned by an earlier run. Callers + must serialise migrations per destination project. A concurrent run overwrites another run's incomplete databases + from its own source under any `OnDuplicate`, and may flip them to `ready` before the other run finishes. + +## 3.0.0 3.0 is a major release. The breaking changes below all concern the Appwrite destination's lifecycle and the database library it builds on; read the upgrade notes before you upgrade from 2.x. diff --git a/README.md b/README.md index ee884d56..c021e60f 100644 --- a/README.md +++ b/README.md @@ -44,18 +44,6 @@ $transfer->run( ); ``` -## Appwrite database recovery - -Appwrite database destinations use a `ProvisioningOwner` made from a stable logical migration identifier and a fresh attempt identifier for every execution. The required `getRecoverableOwner` callback is the recovery authority for an existing database whose status is `provisioning` or `failed`. - -A database status is local to that resource. It does not prove that the migration attempt which owns it has stopped, because an import can continue with other resources after recording a database failure. The callback must therefore consult the caller's authoritative operation lifecycle and return the exact stored owner only after that attempt is terminal. Return `null` while it is active or unknown; recovery then fails closed. This rule also applies when the retry uses the same logical migration identifier. - -Owners are written and enforced only when the destination's `databases` metadata declares both `migrationId` and `migrationAttemptId`; metadata with `status` alone gets status tracking without ownership. A `provisioning` or `failed` database that names no owner at all has nothing for the callback to judge, so its update timestamp stands in: it is recovered only once it has gone a whole `provisioningLease` without a write, since a fresher one may still belong to a migration that is provisioning it. A row still within the lease is refused with an error naming the database. Past the lease it is recovered as it was before ownership existed: a `failed` database is overwritten, a `provisioning` one is resolved like any existing database under the `OnDuplicate` policy (`fail` keeps it instead of colliding) and marked `ready` when the run succeeds, and either is claimed by the current attempt when the metadata can record it. A database naming only part of an owner is refused. - -The `provisioningLease` constructor argument is the lease in seconds and defaults to `86400` (24 hours), the longest lease a live Appwrite migration attempt holds; `0` recovers an ownerless database at once, and a negative value is rejected with an `InvalidArgumentException`. - -The standalone CLI requires `--migration-id` and a fresh `--migration-attempt-id`. Recovering an incomplete database that names an owner additionally requires both `--recover-migration-id` and `--recover-migration-attempt-id` for the exact terminal prior attempt. - ## Supported Resources Chart Sources: diff --git a/bin/MigrationCLI.php b/bin/MigrationCLI.php index 422f21e0..a4f95df3 100644 --- a/bin/MigrationCLI.php +++ b/bin/MigrationCLI.php @@ -11,7 +11,6 @@ use Utopia\Database\Query; use Utopia\Migration\Destination; use Utopia\Migration\Destinations\Appwrite as DestinationsAppwrite; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; use Utopia\Migration\Destinations\Local; use Utopia\Migration\Source; use Utopia\Migration\Sources\Appwrite; @@ -34,9 +33,6 @@ class MigrationCLI protected mixed $destination; - /** @var list */ - private readonly array $arguments; - protected const STRUCTURE = [ '$collection' => 'databases', '$id' => 'collections', @@ -157,34 +153,6 @@ class MigrationCLI ], ]; - /** @param list $arguments */ - public function __construct(array $arguments = []) - { - $this->arguments = $arguments; - } - - public static function getHelp(): string - { - return <<<'HELP' -Usage: php bin/MigrationCLI.php [options] - -Options: - -h, --help Show this help. - --migration-id= Stable logical owner identifier for this migration. - Required for Appwrite; reuse it for retries. - --migration-attempt-id= - Required for Appwrite; use a fresh attempt for every - execution or retry. - --recover-migration-id= - --recover-migration-attempt-id= - Together attest that the exact prior migration attempt is terminal - and allow recovery of its provisioning or failed databases. - Recovery is refused by default. - Resource status alone never proves an attempt terminal. - -HELP; - } - /** * Prints the current status of migrations as a table after wiping the screen */ @@ -283,7 +251,6 @@ public function getDestination(): Destination switch ($_ENV['DESTINATION_PROVIDER']) { case 'appwrite': $database = $this->getDatabase('destination'); - $recoverableOwner = $this->getRecoverableOwner(); return new DestinationsAppwrite( project: $_ENV['DESTINATION_APPWRITE_TEST_PROJECT'], @@ -294,10 +261,6 @@ public function getDestination(): Destination collectionStructure: self::STRUCTURE, dbForPlatform: $database, projectInternalId: $_ENV['DESTINATION_APPWRITE_TEST_PROJECT_INTERNAL_ID'], - owner: new ProvisioningOwner($this->getMigrationId(), $this->getMigrationAttemptId()), - // The standalone operator supplies the fixed pair only after confirming the prior - // lifecycle owner is terminal. The destination independently compares it with the row. - getRecoverableOwner: static fn (Document $document): ?ProvisioningOwner => $recoverableOwner, ); case 'local': return new Local('./localBackup'); @@ -306,62 +269,6 @@ public function getDestination(): Destination } } - private function getMigrationId(): string - { - foreach ($this->arguments as $argument) { - if (! \str_starts_with($argument, '--migration-id=')) { - continue; - } - - $migrationId = \substr($argument, \strlen('--migration-id=')); - if ($migrationId !== '') { - return $migrationId; - } - } - - throw new \InvalidArgumentException('--migration-id is required for an Appwrite destination'); - } - - private function getMigrationAttemptId(): string - { - foreach ($this->arguments as $argument) { - if (! \str_starts_with($argument, '--migration-attempt-id=')) { - continue; - } - - $attemptId = \substr($argument, \strlen('--migration-attempt-id=')); - if ($attemptId !== '') { - return $attemptId; - } - } - - throw new \InvalidArgumentException('--migration-attempt-id is required for an Appwrite destination'); - } - - private function getRecoverableOwner(): ?ProvisioningOwner - { - $migrationId = null; - $attemptId = null; - - foreach ($this->arguments as $argument) { - if (\str_starts_with($argument, '--recover-migration-id=')) { - $value = \substr($argument, \strlen('--recover-migration-id=')); - $migrationId = $value !== '' ? $value : null; - } - - if (\str_starts_with($argument, '--recover-migration-attempt-id=')) { - $value = \substr($argument, \strlen('--recover-migration-attempt-id=')); - $attemptId = $value !== '' ? $value : null; - } - } - - if ($migrationId === null || $attemptId === null) { - return null; - } - - return new ProvisioningOwner($migrationId, $attemptId); - } - public function getDatabase(string $type): Database { self::registerFilters(); @@ -534,8 +441,6 @@ function () { $this->drawFrame(); } ); - - $this->destination->success(); } protected function loadEnvironment(): void @@ -546,12 +451,7 @@ protected function loadEnvironment(): void } if (\realpath($_SERVER['SCRIPT_FILENAME'] ?? '') === __FILE__) { - $arguments = $_SERVER['argv'] ?? []; - if (\in_array('-h', $arguments, true) || \in_array('--help', $arguments, true)) { - echo MigrationCLI::getHelp(); - } else { - $instance = new MigrationCLI($arguments); - $instance->start(); - $instance->drawFrame(); - } + $instance = new MigrationCLI(); + $instance->start(); + $instance->drawFrame(); } diff --git a/composer.json b/composer.json index c1f3be92..eb8e1ee8 100644 --- a/composer.json +++ b/composer.json @@ -10,8 +10,7 @@ "migration" ], "license": "MIT", - "minimum-stability": "dev", - "prefer-stable": true, + "minimum-stability": "stable", "autoload": { "psr-4": { "Utopia\\Migration\\": "src/Migration" @@ -33,7 +32,7 @@ "ext-curl": "*", "ext-openssl": "*", "appwrite/appwrite": "^30.2.2", - "utopia-php/database": "^8.0", + "utopia-php/database": "^8.1.2", "utopia-php/storage": "4.*", "utopia-php/dsn": "0.2.*", "halaxa/json-machine": "^1.2", @@ -53,11 +52,5 @@ "php-http/discovery": true, "tbachert/spi": true } - }, - "extra": { - "branch-alias": { - "dev-feat-query-lib": "3.0.x-dev", - "dev-main": "3.0.x-dev" - } } } diff --git a/composer.lock b/composer.lock index 79ad24c2..a3a776bf 100644 --- a/composer.lock +++ b/composer.lock @@ -4,7 +4,7 @@ "Read more about it at https://getcomposer.org/doc/01-basic-usage.md#installing-dependencies", "This file is @generated automatically" ], - "content-hash": "29449fc2362aa9af364bdf8af14c0fb5", + "content-hash": "4938aab4b4211b65ad5af5e402512d28", "packages": [ { "name": "adhocore/jwt", @@ -2396,16 +2396,16 @@ }, { "name": "utopia-php/database", - "version": "8.0.0", + "version": "8.1.2", "source": { "type": "git", "url": "https://github.com/utopia-php/database.git", - "reference": "e4c46dfe9b5fad60d8b47e09ecb0f134b90c47f7" + "reference": "79577115a047f8a0e24c7505eac8141513c4dfec" }, "dist": { "type": "zip", - "url": "https://api.github.com/repos/utopia-php/database/zipball/e4c46dfe9b5fad60d8b47e09ecb0f134b90c47f7", - "reference": "e4c46dfe9b5fad60d8b47e09ecb0f134b90c47f7", + "url": "https://api.github.com/repos/utopia-php/database/zipball/79577115a047f8a0e24c7505eac8141513c4dfec", + "reference": "79577115a047f8a0e24c7505eac8141513c4dfec", "shasum": "" }, "require": { @@ -2463,9 +2463,9 @@ ], "support": { "issues": "https://github.com/utopia-php/database/issues", - "source": "https://github.com/utopia-php/database/tree/8.0.0" + "source": "https://github.com/utopia-php/database/tree/8.1.2" }, - "time": "2026-10-09T08:45:12+00:00" + "time": "2026-10-14T07:00:00+00:00" }, { "name": "utopia-php/dsn", @@ -5192,9 +5192,9 @@ } ], "aliases": [], - "minimum-stability": "dev", + "minimum-stability": "stable", "stability-flags": {}, - "prefer-stable": true, + "prefer-stable": false, "prefer-lowest": false, "platform": { "php": ">=8.5", diff --git a/src/Migration/Destination.php b/src/Migration/Destination.php index 802bdd7f..14776727 100644 --- a/src/Migration/Destination.php +++ b/src/Migration/Destination.php @@ -96,16 +96,6 @@ public function runWithResourceSelector( } } - /** - * Report an abort that a source swallowed and Transfer latched, before Transfer rethrows it; - * an abort that propagates straight out of run() never reaches here. A destination that - * defers its terminal state to success() drops that state here, so a later success() - * finalizes nothing, exactly as after a run that threw. - */ - public function markAborted(): void - { - } - /** * Import Resources * diff --git a/src/Migration/Destinations/Appwrite.php b/src/Migration/Destinations/Appwrite.php index f436d7fb..2298c035 100644 --- a/src/Migration/Destinations/Appwrite.php +++ b/src/Migration/Destinations/Appwrite.php @@ -25,7 +25,7 @@ use Appwrite\Services\Users; use Override; use Utopia\Database\Attribute as UtopiaAttribute; -use Utopia\Database\AttributeUpdate; +use Utopia\Database\Attribute\Update as AttributeUpdate; use Utopia\Database\Capability; use Utopia\Database\Collection as UtopiaCollection; use Utopia\Database\Database as UtopiaDatabase; @@ -33,7 +33,6 @@ use Utopia\Database\Document as UtopiaDocument; use Utopia\Database\Exception as DatabaseException; use Utopia\Database\Exception\Authorization as AuthorizationException; -use Utopia\Database\Exception\Conflict as ConflictException; use Utopia\Database\Exception\Duplicate as DuplicateException; use Utopia\Database\Exception\Limit as LimitException; use Utopia\Database\Exception\Structure as StructureException; @@ -43,17 +42,15 @@ use Utopia\Database\Permission; use Utopia\Database\Query; use Utopia\Database\Relationship as UtopiaRelationship; +use Utopia\Database\Relationship\Update as RelationshipUpdate; use Utopia\Database\RelationshipDeleteAction; use Utopia\Database\RelationshipSide; -use Utopia\Database\RelationshipUpdate; use Utopia\Database\Role; use Utopia\Database\Validator\IndexDefinition as IndexValidator; use Utopia\Database\Validator\Structure; use Utopia\Database\Validator\UID; use Utopia\Migration\Destination; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; use Utopia\Migration\Exception; -use Utopia\Migration\Exception\Finalization; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Auth\AuthMethods; use Utopia\Migration\Resources\Auth\Hash; @@ -129,13 +126,6 @@ class Appwrite extends Destination private const DATABASE_STATUS_READY = 'ready'; private const DATABASE_STATUS_FAILED = 'failed'; - /** Attributes naming the migration attempt that provisions a database; a destination schema gains them after `status`. */ - private const OWNER_MIGRATION_ID = 'migrationId'; - private const OWNER_ATTEMPT_ID = 'migrationAttemptId'; - - /** 24 hours, the longest lease a live Appwrite attempt holds. */ - public const int DEFAULT_PROVISIONING_LEASE = 86_400; - /** * Attribute fields the SDK can't update in place (no per-type updateX endpoint exposes them); a change here * forces drop+recreate. The type is immutable too, but is compared through {@see self::typeMatches()}. @@ -188,19 +178,6 @@ class Appwrite extends Destination */ protected $getDatabaseDSN; - /** Immutable owner written with every provisioning transition initiated by this destination. */ - private readonly ProvisioningOwner $owner; - - /** - * Resolves the authoritative terminal owner of an incomplete database that - * names one. Database status is resource-local and never proves the overall - * migration attempt terminal. Callers must derive this from their authoritative - * operation lifecycle; null means active or unknown and fails closed. - * - * @var callable(UtopiaDocument $database): ?ProvisioningOwner - */ - private $getRecoverableOwner; - /** * @var array */ @@ -230,21 +207,16 @@ class Appwrite extends Destination private array $processedTwoWayPairs = []; /** - * Databases created this run, held in `provisioning` until success() flips - * each to `ready`. Keyed by database id. + * Databases created this run, held in `provisioning` until the run finishes + * successfully. The end-of-run sweep flips each to `ready`. Keyed by database id. * * @var array */ private array $provisioningDatabases = []; - private bool $finalizationPending = false; - /** Whether the destination project's database metadata supports lifecycle status. */ private ?bool $databaseStatusSupported = null; - /** Whether the destination project's database metadata can record which migration attempt provisions a database. */ - private ?bool $provisioningOwnerSupported = null; - /** * @param string $project * @param string $endpoint @@ -252,12 +224,9 @@ class Appwrite extends Destination * @param UtopiaDatabase $dbForProject * @param callable(UtopiaDocument $database):UtopiaDatabase $getDatabasesDB * @param CollectionStructure $collectionStructure - * @param ProvisioningOwner $owner Immutable logical migration and execution-attempt identifiers for databases provisioned by this destination. - * @param callable(UtopiaDocument $database): ?ProvisioningOwner $getRecoverableOwner Returns the exact authoritative terminal owner for an existing `provisioning` or `failed` database that names an owner. Resource status alone is not lifecycle proof; return null while the owning migration attempt is active or unknown. A database naming no owner predates ownership and is recovered without it. * @param OnDuplicate $onDuplicate Behavior when a row with an existing $id is encountered. * @param (callable(Database $resource): string)|null $getDatabaseDSN Resolver for the destination's `_databases.database` value. Pass when the destination project's DSN differs from the source's, so the destination row carries its own DSN instead of inheriting the source's. * @param array $collectionStructures Per-database-type metadata collection structures (e.g. `['vectorsdb' => ...]`), used instead of $collectionStructure when the imported database's type has an entry. Types with an entry also get type-specific metadata written (e.g. vectorsdb collection `dimension`). - * @param int $provisioningLease Seconds an existing `provisioning` or `failed` database that names no owner may go without an update before another migration may recover it. Such a row carries no owner for $getRecoverableOwner to judge, so its update timestamp stands in: one fresher than the lease may still belong to a migration that is provisioning it, and fails closed. Defaults to 24 hours, the longest lease a live Appwrite attempt holds. `0` recovers it at once; a negative value is rejected. */ public function __construct( string $project, @@ -268,17 +237,10 @@ public function __construct( protected array $collectionStructure, protected UtopiaDatabase $dbForPlatform, protected string $projectInternalId, - ProvisioningOwner $owner, - callable $getRecoverableOwner, protected OnDuplicate $onDuplicate = OnDuplicate::Fail, ?callable $getDatabaseDSN = null, protected array $collectionStructures = [], - protected int $provisioningLease = self::DEFAULT_PROVISIONING_LEASE, ) { - if ($provisioningLease < 0) { - throw new \InvalidArgumentException('Provisioning lease must not be negative'); - } - $this->projectId = $project; $this->endpoint = $endpoint; $this->key = $key; @@ -301,8 +263,6 @@ public function __construct( $this->getDatabasesDB = $getDatabasesDB; $this->getDatabaseDSN = $getDatabaseDSN; - $this->owner = $owner; - $this->getRecoverableOwner = $getRecoverableOwner; } /** @@ -329,36 +289,20 @@ private function getSupportForDatabaseStatus(): bool return false; } - return $this->databaseStatusSupported ??= $this->declaresDatabaseAttributes('status'); - } - - private function getSupportForProvisioningOwner(): bool - { - if (! $this->getSupportForDatabaseStatus()) { - return false; + if ($this->databaseStatusSupported !== null) { + return $this->databaseStatusSupported; } - return $this->provisioningOwnerSupported ??= $this->declaresDatabaseAttributes( - self::OWNER_MIGRATION_ID, - self::OWNER_ATTEMPT_ID, - ); - } - - private function declaresDatabaseAttributes(string ...$keys): bool - { - $declared = []; foreach ($this->dbForProject->findCollection(self::META_DATABASES)?->attributes() ?? [] as $attribute) { - $declared[] = $attribute->key; + if ($attribute->key === 'status') { + return $this->databaseStatusSupported = true; + } } - return \array_diff($keys, $declared) === []; + return $this->databaseStatusSupported = false; } - /** - * Transfer resources without committing terminal destination state. The - * caller must first persist its own finalization claim, then invoke - * success() while that generation is still authoritative. - */ + /** Orphan cleanup runs only after a successful migration — a mid-run throw preserves the destination as-is. */ #[Override] public function run( array $resources, @@ -366,58 +310,12 @@ public function run( string $rootResourceId = '', string $rootResourceType = '', ): void { - $this->reportSkippedFinalization(); $this->resetRunState(); parent::run($resources, $callback, $rootResourceId, $rootResourceType); - $this->finalizationPending = $this->provisioningDatabases !== [] || $this->orphansByTable !== []; - } - - /** - * Finalize destination state only after the caller has fenced terminal ownership. - * Only a run that returned is finalized: an interrupted one keeps its databases in - * `provisioning` for a later attempt to recover. - * - * @throws Finalization - */ - #[Override] - public function success(): void - { - if (! $this->finalizationPending) { - return; - } - $this->finalizationPending = false; - - // Flip status before the orphan sweep so a cleanup failure can't strand databases in `provisioning`. - $failures = $this->markProvisionedDatabasesReady(); - \array_push($failures, ...$this->cleanupOverwriteOrphans()); - - if ($failures !== []) { - throw new Finalization($failures); - } - } - - #[Override] - public function error(): void - { - $this->finalizationPending = false; - parent::error(); - } - - /** - * An aborted run is not finalized, whatever the source did with the abort: its databases stay - * in `provisioning` for a later attempt, as after an interrupted run. - */ - #[Override] - public function markAborted(): void - { - $this->finalizationPending = false; - } - - #[Override] - public function cleanUp(): void - { - $this->reportSkippedFinalization(); - parent::cleanUp(); + // parent::run() returning means every resource transferred, so the databases are usable. + // Flip status before the orphan sweep so a cleanup failure can't strand them in `provisioning`. + $this->markProvisionedDatabasesReady(); + $this->cleanupOverwriteOrphans(); } /** Per-run state must not leak across run() invocations on a reused instance. */ @@ -427,203 +325,37 @@ private function resetRunState(): void $this->orphansByTable = []; $this->processedTwoWayPairs = []; $this->provisioningDatabases = []; - $this->finalizationPending = false; - } - - /** A returned run that is neither finalized nor abandoned would otherwise leave its databases unusable without a trace. */ - private function reportSkippedFinalization(): void - { - if (! $this->finalizationPending) { - return; - } - $this->finalizationPending = false; - - $databaseIds = \array_map(\strval(...), \array_keys($this->provisioningDatabases)); - foreach ($this->orphansByTable as $tracked) { - $databaseIds[] = $tracked['database']->getId(); - } - - foreach (\array_unique($databaseIds) as $databaseId) { - $this->addError(new Exception( - resourceName: Resource::TYPE_DATABASE, - resourceGroup: Transfer::GROUP_DATABASES, - resourceId: $databaseId, - message: 'Database was not finalized: success() was not called after the transfer', - code: Exception::CODE_INTERNAL, - )); - } } - /** - * @return list - */ - private function markProvisionedDatabasesReady(): array + private function markProvisionedDatabasesReady(): void { - $failures = []; foreach (\array_keys($this->provisioningDatabases) as $databaseId) { try { - if (! $this->setDatabaseStatus($databaseId, self::DATABASE_STATUS_READY)) { - throw new DatabaseException('Database provisioning owner changed before finalization'); - } + $this->setDatabaseStatus($databaseId, self::DATABASE_STATUS_READY); } catch (\Throwable $error) { - $failures[] = $this->recordFinalizationFailure(Resource::TYPE_DATABASE, (string) $databaseId, $error); + $this->addError(new Exception( + resourceName: Resource::TYPE_DATABASE, + resourceGroup: Transfer::GROUP_DATABASES, + resourceId: (string) $databaseId, + message: $error->getMessage(), + code: $error->getCode(), + previous: $error, + )); } } - - return $failures; - } - - private function recordFinalizationFailure(string $resourceName, string $resourceId, \Throwable $error): Exception - { - $failure = new Exception( - resourceName: $resourceName, - resourceGroup: Transfer::GROUP_DATABASES, - resourceId: $resourceId, - message: $error->getMessage(), - code: $error->getCode(), - previous: $error, - ); - $this->addError($failure); - - return $failure; } - private function setDatabaseStatus(string $databaseId, string $status): bool + private function setDatabaseStatus(string $databaseId, string $status): void { if (! $this->getSupportForDatabaseStatus()) { - return true; - } - - try { - $database = $this->dbForProject->getDocument( - self::META_DATABASES, - $databaseId, - forUpdate: true, - ); - if ( - $database->isEmpty() - || $database->getAttribute('status') !== self::DATABASE_STATUS_PROVISIONING - || ($this->getSupportForProvisioningOwner() && ! $this->carriesOwner($database, $this->owner)) - ) { - return false; - } - - $updated = $this->updateOwned($database, new UtopiaDocument(['status' => $status])); - - return ! $updated->isEmpty(); - } catch (ConflictException) { - return false; - } - } - - /** - * A write pinned to the timestamp the row was read with is refused once another writer got - * there first, which holds only while every write moves `$updatedAt` strictly forward. - * Writes here never supply `$updatedAt`, and the stored value is checked after each one. - */ - private function updateOwned(UtopiaDocument $observed, UtopiaDocument $updates): UtopiaDocument - { - $readAt = $observed->getUpdatedAt(); - if ($readAt === null || $readAt === '') { - throw new DatabaseException('Database provisioning ownership requires an update timestamp'); + return; } - $updated = $this->dbForProject->withRequestTimestamp( - new \DateTime($readAt), - fn (): UtopiaDocument => $this->dbForProject->updateDocument(self::META_DATABASES, $observed->getId(), $updates), + $this->dbForProject->updateDocument( + self::META_DATABASES, + $databaseId, + new UtopiaDocument(['status' => $status]), ); - if ($updated->isEmpty()) { - return $updated; - } - - $stored = $this->dbForProject->getDocument(self::META_DATABASES, $observed->getId(), forUpdate: true); - if ($stored->isEmpty()) { - return $stored; - } - - if (new \DateTime((string) $stored->getUpdatedAt()) <= new \DateTime($readAt)) { - throw new DatabaseException('Database '.$observed->getId().' provisioning write did not move its update timestamp forward'); - } - - return $stored; - } - - private function getProvisioningOwner(UtopiaDocument $database): ?ProvisioningOwner - { - $migrationId = $database->getAttribute(self::OWNER_MIGRATION_ID); - $attemptId = $database->getAttribute(self::OWNER_ATTEMPT_ID); - if ( - ! \is_string($migrationId) - || $migrationId === '' - || ! \is_string($attemptId) - || $attemptId === '' - ) { - return null; - } - - return new ProvisioningOwner($migrationId, $attemptId); - } - - /** - * A row written before its schema could record an owner names none at all. A row naming - * only part of an owner is not unowned: it fails closed like any other unattested owner. - */ - private function isUnowned(UtopiaDocument $database): bool - { - if (! $this->getSupportForProvisioningOwner()) { - return true; - } - - foreach ([self::OWNER_MIGRATION_ID, self::OWNER_ATTEMPT_ID] as $key) { - if (! \in_array($database->getAttribute($key), [null, ''], true)) { - return false; - } - } - - return true; - } - - /** - * @throws \Exception - */ - private function isStale(UtopiaDocument $database): bool - { - if ($this->provisioningLease === 0) { - return true; - } - - $updatedAt = $database->getUpdatedAt(); - if ($updatedAt === null || $updatedAt === '') { - return false; - } - - return (new \DateTimeImmutable($updatedAt))->getTimestamp() + $this->provisioningLease - < (new \DateTimeImmutable())->getTimestamp(); - } - - /** Whether the row names exactly $owner, or no owner at all when $owner is null. */ - private function carriesOwner(UtopiaDocument $database, ?ProvisioningOwner $owner): bool - { - if ($owner === null) { - return $this->isUnowned($database); - } - - $current = $this->getProvisioningOwner($database); - - return $current !== null && $current->equals($owner); - } - - /** @return array */ - private function provisioningOwnerAttributes(): array - { - if (! $this->getSupportForProvisioningOwner()) { - return []; - } - - return [ - self::OWNER_MIGRATION_ID => $this->owner->migrationId, - self::OWNER_ATTEMPT_ID => $this->owner->attemptId, - ]; } /** Best-effort transition to `failed`; a secondary error here must not mask the caller's original throw. */ @@ -646,7 +378,6 @@ public static function getName(): string public static function getSupportedResources(): array { return [ - // Auth Resource::TYPE_USER, Resource::TYPE_TEAM, Resource::TYPE_MEMBERSHIP, @@ -654,7 +385,6 @@ public static function getSupportedResources(): array Resource::TYPE_POLICIES, Resource::TYPE_OAUTH2_PROVIDER, - // Database Resource::TYPE_DATABASE, Resource::TYPE_DATABASE_DOCUMENTSDB, Resource::TYPE_DATABASE_VECTORSDB, @@ -663,48 +393,39 @@ public static function getSupportedResources(): array Resource::TYPE_INDEX, Resource::TYPE_ROW, - // legacy Resource::TYPE_DOCUMENT, Resource::TYPE_ATTRIBUTE, Resource::TYPE_COLLECTION, - // Storage Resource::TYPE_BUCKET, Resource::TYPE_FILE, - // Functions Resource::TYPE_FUNCTION, Resource::TYPE_DEPLOYMENT, Resource::TYPE_ENVIRONMENT_VARIABLE, - // Messaging Resource::TYPE_PROVIDER, Resource::TYPE_TOPIC, Resource::TYPE_SUBSCRIBER, Resource::TYPE_MESSAGE, - // Sites Resource::TYPE_SITE, Resource::TYPE_SITE_DEPLOYMENT, Resource::TYPE_SITE_VARIABLE, - // Integrations Resource::TYPE_PLATFORM, Resource::TYPE_API_KEY, Resource::TYPE_WEBHOOK, Resource::TYPE_SMTP, - // Project Resource::TYPE_PROJECT_VARIABLE, Resource::TYPE_PROJECT_PROTOCOLS, Resource::TYPE_PROJECT_LABELS, Resource::TYPE_PROJECT_SERVICES, Resource::TYPE_PROJECT_EMAIL_TEMPLATE, - // Backups Resource::TYPE_BACKUP_POLICY, - // Domains Resource::TYPE_RULE, ]; } @@ -726,7 +447,6 @@ public function report(array $resources = [], array $resourceIds = []): array // Most of these API calls are purposely wrong. Appwrite will throw a 403 before a 400. // We want to make sure the API key has full read and write access to the project. try { - // Auth if (\in_array(Resource::TYPE_USER, $resources)) { $scope = 'users.read'; $this->users->list(); @@ -751,7 +471,6 @@ public function report(array $resources = [], array $resourceIds = []): array $this->teams->createMembership('', [], ''); } - // Storage if (\in_array(Resource::TYPE_BUCKET, $resources)) { $scope = 'storage.read'; $this->storage->listBuckets(); @@ -768,7 +487,6 @@ public function report(array $resources = [], array $resourceIds = []): array $this->storage->createFile('', '', new InputFile()); } - // Functions if (\in_array(Resource::TYPE_FUNCTION, $resources)) { $scope = 'functions.read'; $this->functions->list(); @@ -777,7 +495,6 @@ public function report(array $resources = [], array $resourceIds = []): array $this->functions->create('', '', Runtime::NODE180()); } - // Messaging if (\in_array(Resource::TYPE_PROVIDER, $resources)) { $scope = 'providers.read'; $this->messaging->listProviders(); @@ -810,7 +527,6 @@ public function report(array $resources = [], array $resourceIds = []): array $this->messaging->createEmail('', '', '', draft: true); } - // Sites if (\in_array(Resource::TYPE_SITE, $resources)) { $scope = 'sites.read'; $this->sites->list(); @@ -973,90 +689,37 @@ protected function createDatabase(Database $resource): bool $updatedAt = $this->normalizeDateTime($resource->getUpdatedAt(), $createdAt); $existing = $this->dbForProject->getDocument(self::META_DATABASES, $resource->getId()); - $supportsStatus = $this->getSupportForDatabaseStatus(); - $status = $supportsStatus && ! $existing->isEmpty() - ? $existing->getAttribute('status') - : null; - $isIncomplete = \in_array( - $status, - [self::DATABASE_STATUS_PROVISIONING, self::DATABASE_STATUS_FAILED], - true, - ); - $isUnowned = $isIncomplete && $this->isUnowned($existing); - $expectedOwner = null; - if ($isIncomplete && ! $isUnowned) { - $snapshotOwner = $this->getProvisioningOwner($existing); - $expectedOwner = ($this->getRecoverableOwner)($existing); - if ( - $snapshotOwner === null - || ! $expectedOwner instanceof ProvisioningOwner - || ! $snapshotOwner->equals($expectedOwner) - || $this->owner->attemptId === $expectedOwner->attemptId - ) { - throw new DatabaseException('Database '.$resource->getId().' recovery requires exact terminal-owner attestation and a fresh attempt'); - } - } elseif ($isUnowned && ! $this->isStale($existing)) { - throw new DatabaseException('Database '.$resource->getId().' names no owner and another migration may still be provisioning it; it becomes recoverable after '.$this->provisioningLease.' seconds without an update'); - } + $isIncomplete = ! $existing->isEmpty() + && $this->getSupportForDatabaseStatus() + && \in_array( + $existing->getAttribute('status'), + [self::DATABASE_STATUS_FAILED, self::DATABASE_STATUS_PROVISIONING], + true, + ); if ($this->onDuplicate !== OnDuplicate::Fail || $isIncomplete) { - /** @var array{action: SchemaAction, database: UtopiaDocument, incomplete: bool} $claim */ - $claim = $this->dbForProject->withTransaction(function () use ( - $resource, + $action = $this->onDuplicate->resolveSchemaAction( + !$existing->isEmpty(), $updatedAt, - $supportsStatus, - $status, - $isIncomplete, - $isUnowned, - $expectedOwner, - ): array { - $locked = $this->dbForProject->getDocument( - self::META_DATABASES, - $resource->getId(), - forUpdate: true, - ); - $lockedStatus = $supportsStatus && ! $locked->isEmpty() - ? $locked->getAttribute('status') - : null; - $lockedIncomplete = \in_array( - $lockedStatus, - [self::DATABASE_STATUS_PROVISIONING, self::DATABASE_STATUS_FAILED], - true, - ); - - if ($isIncomplete) { - if ($lockedStatus !== $status || ! $this->carriesOwner($locked, $expectedOwner)) { - throw new DatabaseException('Database '.$resource->getId().' recovery owner changed before it could be claimed'); - } - if ($isUnowned && ! $this->isStale($locked)) { - throw new DatabaseException('Database '.$resource->getId().' names no owner and was updated before it could be claimed'); - } - } elseif ($lockedIncomplete) { - throw new DatabaseException('Database '.$resource->getId().' requires terminal migration attestation before recovery'); - } - - $action = $this->onDuplicate->resolveSchemaAction( - ! $locked->isEmpty(), - $updatedAt, - $locked->getUpdatedAt(), - ); + $existing->getUpdatedAt(), + ); - if ($isUnowned && $status === self::DATABASE_STATUS_PROVISIONING) { - // Left provisioning before it could name an owner: recovered in place as it was then, under the - // OnDuplicate policy (Fail skips instead of colliding), and re-enrolled for the ready flip. - if ($action === SchemaAction::Create || $this->databaseSpecMatches($locked, $resource)) { - $action = SchemaAction::Skip; - } - } elseif ($isIncomplete) { - // A prior run created the metadata document but left the database unusable. Force an - // overwrite after the locked ownership check so retries can recreate its collection. - $action = SchemaAction::Overwrite; - } elseif ($action !== SchemaAction::Create && $this->databaseSpecMatches($locked, $resource)) { - $action = SchemaAction::Skip; - } + if ($isIncomplete) { + // A prior run created the metadata document but left the database unusable (its backing + // collection may be missing). Force Overwrite — regardless of timestamps, spec match, or + // OnDuplicate::Fail — so retries recreate the collection instead of hitting the existing ID. + $action = SchemaAction::Overwrite; + } elseif ($action !== SchemaAction::Create && $this->databaseSpecMatches($existing, $resource)) { + $action = SchemaAction::Skip; + } - $document = []; - if ($action === SchemaAction::Overwrite) { + $earlyReturn = match ($action) { + SchemaAction::Skip => (function () use ($resource, $existing): bool { + $resource->setSequence($existing->getSequence()); + $resource->setStatus(Resource::STATUS_SKIPPED, 'Already exists on destination'); + return false; + })(), + SchemaAction::Overwrite => (function () use ($resource, $existing, $updatedAt, $isIncomplete): bool { $document = [ 'name' => $resource->getDatabaseName(), 'search' => implode(' ', [$resource->getId(), $resource->getDatabaseName()]), @@ -1064,71 +727,44 @@ protected function createDatabase(Database $resource): bool 'type' => empty($resource->getType()) ? 'legacy' : $resource->getType(), 'originalId' => empty($resource->getOriginalId()) ? null : $resource->getOriginalId(), 'database' => $this->resolveDestinationDsn($resource), + '$updatedAt' => $updatedAt, ]; - if ($supportsStatus) { + if ($this->getSupportForDatabaseStatus()) { $document['status'] = self::DATABASE_STATUS_PROVISIONING; } - } - - if ($action === SchemaAction::Overwrite || $isIncomplete) { - $document = [...$document, ...$this->provisioningOwnerAttributes()]; - } - if ($document === [] && $isUnowned && $supportsStatus) { - $document = [ - 'status' => self::DATABASE_STATUS_PROVISIONING, - '$updatedAt' => DateTime::nowAfter($locked->getUpdatedAt()), - ]; - } + $this->dbForProject->updateDocument(self::META_DATABASES, $existing->getId(), new UtopiaDocument($document)); + $resource->setSequence($existing->getSequence()); - if ($document !== []) { - $updated = $this->updateOwned($locked, new UtopiaDocument($document)); - if ($updated->isEmpty()) { - throw new DatabaseException('Database '.$resource->getId().' provisioning owner changed before it could be claimed'); + // Only an incomplete database can be missing its backing collection (a prior run wrote the + // metadata document but threw before createCollection). Recreate it so we never flip a + // database to ready with no collection behind it. + if ($isIncomplete && $this->dbForProject->findCollection($this->databaseCollectionId($existing)) === null) { + try { + $structure = $this->collectionStructureFor($resource); + + $this->dbForProject->createCollection(UtopiaCollection::create( + id: $this->databaseCollectionId($existing), + attributes: $this->schemaAttributes($structure['attributes'] ?? []), + indexes: $this->schemaIndexes($structure['indexes'] ?? []), + )); + } catch (\Throwable $e) { + $this->markDatabaseFailed($resource->getId()); + throw $e; + } } - } - return [ - 'action' => $action, - 'database' => $locked, - 'incomplete' => $isIncomplete, - ]; - }); - - $action = $claim['action']; - $existing = $claim['database']; - $isIncomplete = $claim['incomplete']; - - if ($action !== SchemaAction::Create) { - $resource->setSequence($existing->getSequence()); - - // The claim transaction commits before inspecting or creating the backing collection. - if ($isIncomplete && $this->dbForProject->findCollection($this->databaseCollectionId($existing)) === null) { - try { - $structure = $this->collectionStructureFor($resource); - - $this->dbForProject->createCollection(UtopiaCollection::create( - id: $this->databaseCollectionId($existing), - attributes: $this->schemaAttributes($structure['attributes'] ?? []), - indexes: $this->schemaIndexes($structure['indexes'] ?? []), - )); - } catch (\Throwable $e) { - $this->markDatabaseFailed($resource->getId()); - throw $e; + if ($this->getSupportForDatabaseStatus()) { + $this->provisioningDatabases[$resource->getId()] = true; } - } - - if ($supportsStatus && ($action === SchemaAction::Overwrite || $isIncomplete)) { - $this->provisioningDatabases[$resource->getId()] = true; - } - if ($action === SchemaAction::Skip) { - $resource->setStatus(Resource::STATUS_SKIPPED, 'Already exists on destination'); - return false; - } - - return true; + return true; + })(), + SchemaAction::Create => null, + }; + if ($earlyReturn !== null) { + return $earlyReturn; } } @@ -1150,7 +786,6 @@ protected function createDatabase(Database $resource): bool // source leaves status untouched so the collection default applies. Never copy the source's state. if ($this->getSupportForDatabaseStatus()) { $document['status'] = self::DATABASE_STATUS_PROVISIONING; - $document = [...$document, ...$this->provisioningOwnerAttributes()]; } $database = $this->dbForProject->createDocument(self::META_DATABASES, new UtopiaDocument($document)); @@ -1312,8 +947,6 @@ protected function createField(Column|Attribute $resource): bool $resource->setStatus(Resource::STATUS_SKIPPED, 'Columns not supported for DocumentsDB'); return false; } - // column will be matching attribute as well - // column type will be matching attribute type as well $type = $this->schemaColumnType($resource); $database = $this->dbForProject->getDocument( @@ -1852,7 +1485,6 @@ protected function createIndex(Index $resource): bool return false; } - // Lengths hidden by default $lengths = []; if ($dbForDatabases->getAdapter()->supports(Capability::DefinedAttributes)) { @@ -1947,7 +1579,6 @@ protected function createRecord(Row $resource, bool $isLast): bool return false; } - // Check if document has already been created $exists = \array_key_exists( $resource->getId(), $this->cache->get($resource->getName()) @@ -2579,23 +2210,12 @@ private function trackOrphanCandidate( $this->orphansByTable[$tableId][$kind][] = $key; } - /** - * End-of-migration sweep — only visits tables that had no rows (rest were cleaned per-table in createRecord). - * - * @return list - */ - private function cleanupOverwriteOrphans(): array + /** End-of-migration sweep — only visits tables that had no rows (rest were cleaned per-table in createRecord). */ + private function cleanupOverwriteOrphans(): void { - $failures = []; - foreach ($this->orphansByTable as $tableId => $tracked) { - try { - $this->cleanupOverwriteOrphansForTable($tableId); - } catch (\Throwable $error) { - $failures[] = $this->recordFinalizationFailure(Resource::TYPE_TABLE, $tracked['table']->getId(), $error); - } + foreach (\array_keys($this->orphansByTable) as $tableId) { + $this->cleanupOverwriteOrphansForTable($tableId); } - - return $failures; } /** Called per-table from createRecord before rows land so Structure validator sees the post-cleanup schema. */ diff --git a/src/Migration/Destinations/Appwrite/ProvisioningOwner.php b/src/Migration/Destinations/Appwrite/ProvisioningOwner.php deleted file mode 100644 index 7a27879d..00000000 --- a/src/Migration/Destinations/Appwrite/ProvisioningOwner.php +++ /dev/null @@ -1,24 +0,0 @@ -migrationId === $owner->migrationId - && $this->attemptId === $owner->attemptId; - } -} diff --git a/src/Migration/Exception/Aborted.php b/src/Migration/Exception/Aborted.php deleted file mode 100644 index b13710dd..00000000 --- a/src/Migration/Exception/Aborted.php +++ /dev/null @@ -1,15 +0,0 @@ - $failures - */ - public function __construct(public readonly array $failures) - { - if ($failures === []) { - throw new \InvalidArgumentException('Finalization requires at least one failure'); - } - - $subjects = \array_map( - static fn (Exception $failure): string => \trim($failure->getResourceName().' '.$failure->getResourceId()), - $failures, - ); - - parent::__construct( - resourceName: '', - resourceGroup: '', - message: 'Finalization failed for '.\implode(', ', $subjects), - code: Exception::CODE_INTERNAL, - previous: $failures[0], - ); - } -} diff --git a/src/Migration/Sources/Appwrite.php b/src/Migration/Sources/Appwrite.php index b3a967bd..a014bf86 100644 --- a/src/Migration/Sources/Appwrite.php +++ b/src/Migration/Sources/Appwrite.php @@ -24,7 +24,6 @@ use Utopia\Database\Document as UtopiaDocument; use Utopia\Database\RelationshipSide; use Utopia\Migration\Exception; -use Utopia\Migration\Exception\Aborted; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Auth\AuthMethods; use Utopia\Migration\Resources\Auth\Hash; @@ -705,8 +704,6 @@ protected function exportGroupAuth(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_USER, $resources)) { $this->exportUsers($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_USER, @@ -721,8 +718,6 @@ protected function exportGroupAuth(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_TEAM, $resources)) { $this->exportTeams($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_TEAM, @@ -737,8 +732,6 @@ protected function exportGroupAuth(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_MEMBERSHIP, $resources)) { $this->exportMemberships($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_MEMBERSHIP, @@ -753,8 +746,6 @@ protected function exportGroupAuth(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_AUTH_METHODS, $resources)) { $this->exportAuthMethods(); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_AUTH_METHODS, @@ -769,8 +760,6 @@ protected function exportGroupAuth(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_OAUTH2_PROVIDER, $resources)) { $this->exportOAuth2Providers(); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_OAUTH2_PROVIDER, @@ -785,8 +774,6 @@ protected function exportGroupAuth(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_POLICIES, $resources)) { $this->exportPolicies(); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_POLICIES, @@ -1110,8 +1097,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (\in_array($entityResource[$resourceKey], $resources)) { $callback($entityKey, $entityResource); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -1133,8 +1118,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (Resource::isSupported(array_keys(Resource::DATABASE_TYPE_RESOURCE_MAP), $resources)) { $this->exportDatabases($batchSize, $resources); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -1154,8 +1137,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (\in_array($databaseResource['entity'], $resources)) { $this->exportEntities($databaseKey, $batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -1556,8 +1537,6 @@ protected function exportGroupStorage(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_BUCKET, $resources)) { $this->exportBuckets($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -1574,8 +1553,6 @@ protected function exportGroupStorage(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_FILE, $resources)) { $this->exportFiles($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -1664,8 +1641,6 @@ private function exportFiles(int $batchSize): void $file->sizeOriginal, folder: $file->folder, )); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( resourceName: Resource::TYPE_FILE, @@ -1734,8 +1709,6 @@ protected function exportGroupFunctions(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_FUNCTION, $resources)) { $this->exportFunctions($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_FUNCTION, @@ -1749,8 +1722,6 @@ protected function exportGroupFunctions(int $batchSize, array $resources): void try { $exportOnlyActive = !\in_array(Resource::TYPE_DEPLOYMENT, $resources); $this->exportDeployments($batchSize, $exportOnlyActive); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_DEPLOYMENT, @@ -1768,8 +1739,6 @@ protected function exportGroupSites(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_SITE, $resources)) { $this->exportSites($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_SITE, @@ -1783,8 +1752,6 @@ protected function exportGroupSites(int $batchSize, array $resources): void try { $exportOnlyActive = !\in_array(Resource::TYPE_SITE_DEPLOYMENT, $resources); $this->exportSiteDeployments($batchSize, $exportOnlyActive); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_SITE_DEPLOYMENT, @@ -1876,8 +1843,6 @@ protected function exportGroupProjects(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_PROJECT_VARIABLE, $resources)) { try { $this->exportProjectVariables($batchSize); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_PROJECT_VARIABLE, @@ -1893,8 +1858,6 @@ protected function exportGroupProjects(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_PROJECT_PROTOCOLS, $resources)) { $this->exportProtocols(); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_PROJECT_PROTOCOLS, @@ -1909,8 +1872,6 @@ protected function exportGroupProjects(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_PROJECT_LABELS, $resources)) { $this->exportLabels(); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_PROJECT_LABELS, @@ -1925,8 +1886,6 @@ protected function exportGroupProjects(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_PROJECT_SERVICES, $resources)) { $this->exportServices(); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_PROJECT_SERVICES, @@ -1940,8 +1899,6 @@ protected function exportGroupProjects(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_PROJECT_EMAIL_TEMPLATE, $resources)) { try { $this->exportEmailTemplates($batchSize); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_PROJECT_EMAIL_TEMPLATE, @@ -2035,8 +1992,6 @@ protected function exportGroupDomains(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_RULE, $resources)) { try { $this->exportRules($batchSize); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_RULE, @@ -2339,8 +2294,6 @@ private function exportDeployments(int $batchSize, bool $exportOnlyActive = fals try { $this->exportDeploymentData($func, $deployment); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $func->setStatus(Resource::STATUS_ERROR, $e->getMessage()); } @@ -2363,8 +2316,6 @@ private function exportDeployments(int $batchSize, bool $exportOnlyActive = fals foreach ($response->deployments as $deployment) { try { $this->exportDeploymentData($func, $deployment); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $func->setStatus(Resource::STATUS_ERROR, $e->getMessage()); } @@ -2579,8 +2530,6 @@ protected function exportGroupMessaging(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_PROVIDER, $resources)) { $this->exportProviders($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_PROVIDER, @@ -2595,8 +2544,6 @@ protected function exportGroupMessaging(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_TOPIC, $resources)) { $this->exportTopics($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_TOPIC, @@ -2611,8 +2558,6 @@ protected function exportGroupMessaging(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_SUBSCRIBER, $resources)) { $this->exportSubscribers($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_SUBSCRIBER, @@ -2627,8 +2572,6 @@ protected function exportGroupMessaging(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_MESSAGE, $resources)) { $this->exportMessages($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_MESSAGE, @@ -2941,8 +2884,6 @@ private function exportSiteDeployments(int $batchSize, bool $exportOnlyActive = try { $this->exportSiteDeploymentData($site, $deployment); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $site->setStatus(Resource::STATUS_ERROR, $e->getMessage()); } @@ -2965,8 +2906,6 @@ private function exportSiteDeployments(int $batchSize, bool $exportOnlyActive = foreach ($response->deployments as $deployment) { try { $this->exportSiteDeploymentData($site, $deployment); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $site->setStatus(Resource::STATUS_ERROR, $e->getMessage()); } @@ -3156,8 +3095,6 @@ protected function exportGroupIntegrations(int $batchSize, array $resources): vo if (\in_array(Resource::TYPE_PLATFORM, $resources)) { try { $this->exportPlatforms($batchSize); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_PLATFORM, @@ -3172,8 +3109,6 @@ protected function exportGroupIntegrations(int $batchSize, array $resources): vo if (\in_array(Resource::TYPE_API_KEY, $resources)) { try { $this->exportApiKeys($batchSize); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_API_KEY, @@ -3188,8 +3123,6 @@ protected function exportGroupIntegrations(int $batchSize, array $resources): vo if (\in_array(Resource::TYPE_WEBHOOK, $resources)) { try { $this->exportWebhooks($batchSize); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_WEBHOOK, @@ -3205,8 +3138,6 @@ protected function exportGroupIntegrations(int $batchSize, array $resources): vo if (\in_array(Resource::TYPE_SMTP, $resources)) { $this->exportSMTP(); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_SMTP, diff --git a/src/Migration/Sources/CSV.php b/src/Migration/Sources/CSV.php index 4f509450..643d3d5b 100644 --- a/src/Migration/Sources/CSV.php +++ b/src/Migration/Sources/CSV.php @@ -6,7 +6,6 @@ use Utopia\Database\RelationshipSide; use Utopia\Database\RelationshipType; use Utopia\Migration\Exception; -use Utopia\Migration\Exception\Aborted; use Utopia\Migration\Resource; use Utopia\Migration\Resource as UtopiaResource; use Utopia\Migration\Resources\Database\Column; @@ -132,8 +131,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (UtopiaResource::isSupported($this->getSupportedResources(), $resources)) { $this->exportRows($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( diff --git a/src/Migration/Sources/Firebase.php b/src/Migration/Sources/Firebase.php index e56ca10c..9f056e23 100644 --- a/src/Migration/Sources/Firebase.php +++ b/src/Migration/Sources/Firebase.php @@ -3,7 +3,6 @@ namespace Utopia\Migration\Sources; use Utopia\Migration\Exception; -use Utopia\Migration\Exception\Aborted; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Auth\Hash; use Utopia\Migration\Resources\Auth\User; @@ -190,8 +189,6 @@ protected function exportGroupAuth(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_USER, $resources)) { $this->exportUsers($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -290,8 +287,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_DATABASE, $resources)) { $this->callback([$database]); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -309,8 +304,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void $hasInResources = Resource::isSupported(Resource::TYPE_ROW, $resources); $this->exportDB($batchSize, $hasInResources, $database); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -651,8 +644,6 @@ protected function exportGroupStorage(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_BUCKET, $resources)) { $this->exportBuckets($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_BUCKET, @@ -667,8 +658,6 @@ protected function exportGroupStorage(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_FILE, $resources)) { $this->exportFiles($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_FILE, diff --git a/src/Migration/Sources/JSON.php b/src/Migration/Sources/JSON.php index 12425391..de41a138 100644 --- a/src/Migration/Sources/JSON.php +++ b/src/Migration/Sources/JSON.php @@ -8,7 +8,6 @@ use JsonMachine\JsonDecoder\PassThruDecoder; use Utopia\Database\Database as UtopiaDatabase; use Utopia\Migration\Exception; -use Utopia\Migration\Exception\Aborted; use Utopia\Migration\Resource as UtopiaResource; use Utopia\Migration\Resources\Database\Database; use Utopia\Migration\Resources\Database\Row; @@ -120,8 +119,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (UtopiaResource::isSupported(UtopiaResource::TYPE_ROW, $resources)) { $this->exportRows($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( diff --git a/src/Migration/Sources/NHost.php b/src/Migration/Sources/NHost.php index 545d1eff..7cd8b067 100644 --- a/src/Migration/Sources/NHost.php +++ b/src/Migration/Sources/NHost.php @@ -4,7 +4,6 @@ use PDO; use Utopia\Migration\Exception; -use Utopia\Migration\Exception\Aborted; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Auth\Hash; use Utopia\Migration\Resources\Auth\User; @@ -317,8 +316,6 @@ protected function exportGroupAuth(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_USER, $resources)) { $this->exportUsers($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_USER, @@ -384,8 +381,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_DATABASE, $resources)) { $this->exportDatabases($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -402,8 +397,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (Resource::isSupported(Resource::TYPE_TABLE, $resources)) { $this->exportTables($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -420,8 +413,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (Resource::isSupported(Resource::TYPE_COLUMN, $resources)) { $this->exportColumns($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -438,8 +429,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (Resource::isSupported(Resource::TYPE_ROW, $resources)) { $this->exportRows($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -456,8 +445,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_INDEX, $resources)) { $this->exportIndexes($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -806,8 +793,6 @@ protected function exportGroupStorage(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_BUCKET, $resources)) { $this->exportBuckets($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -824,8 +809,6 @@ protected function exportGroupStorage(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_FILE, $resources)) { $this->exportFiles($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( diff --git a/src/Migration/Sources/Supabase.php b/src/Migration/Sources/Supabase.php index 8ea3e217..8b512ab6 100644 --- a/src/Migration/Sources/Supabase.php +++ b/src/Migration/Sources/Supabase.php @@ -3,7 +3,6 @@ namespace Utopia\Migration\Sources; use Utopia\Migration\Exception; -use Utopia\Migration\Exception\Aborted; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Auth\Hash; use Utopia\Migration\Resources\Auth\User; @@ -377,8 +376,6 @@ protected function exportGroupAuth(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_USER, $resources)) { $this->exportUsers($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_BUCKET, @@ -453,8 +450,6 @@ protected function exportGroupStorage(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_BUCKET, $resources)) { $this->exportBuckets($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_BUCKET, @@ -469,8 +464,6 @@ protected function exportGroupStorage(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_FILE, $resources)) { $this->exportFiles($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_BUCKET, diff --git a/src/Migration/Target.php b/src/Migration/Target.php index a2ede6dc..5aa72e47 100644 --- a/src/Migration/Target.php +++ b/src/Migration/Target.php @@ -230,33 +230,21 @@ public function shutdown(): void } /** - * Commit the terminal state of the last run() that returned. - * - * run() may leave that state uncommitted — the Appwrite destination keeps the databases - * it provisions in `provisioning` and defers its overwrite cleanup — so once run() has - * returned, the caller persists whatever claim must precede those side effects and then - * calls success(). Every step is attempted; if any fails, each failure is recorded with - * addError() and success() throws Exception\Finalization carrying them, so a failed - * finalization cannot pass for a completed one. Call error() instead for a run that will - * not be finalized. - * - * @throws Exception\Finalization + * Success callback */ public function success(): void { } /** - * Called instead of success() when the run failed or will not be finalized. + * Error callback */ public function error(): void { } /** - * End the lifecycle. A target that defers work to success() records an error here for - * each resource a returned run left unfinalized when neither success() nor error() was - * called, so skipping success() is reported rather than passing silently. + * Clean up callback */ public function cleanUp(): void { diff --git a/src/Migration/Transfer.php b/src/Migration/Transfer.php index 1bb63cfd..fe83f2d6 100644 --- a/src/Migration/Transfer.php +++ b/src/Migration/Transfer.php @@ -2,8 +2,6 @@ namespace Utopia\Migration; -use Utopia\Migration\Exception\Aborted; - class Transfer { public const GROUP_GENERAL = 'general'; @@ -323,9 +321,8 @@ public function getStatusCounters(): array * Transfer Resources between adapters * * @param array $resources Resources to transfer - * @param callable $callback Callback to run after transfer; throw Aborted from it to stop the transfer + * @param callable $callback Callback to run after transfer * @param string|null $rootResourceId Root resource ID, If enabled you can only transfer a single root resource - * @throws Aborted The callback stopped the transfer * @throws \Exception */ public function run( @@ -373,27 +370,10 @@ public function run( $this->resources = $computedResources; - // A source can still catch the abort as an export failure and carry on; keep it so - // later batches rethrow it and the run ends with it. - $abort = null; - $progress = static function (array $resources) use ($callback, &$abort): void { - if ($abort !== null) { - throw $abort; - } - - try { - $callback($resources); - } catch (Aborted $aborted) { - $abort = $aborted; - - throw $aborted; - } - }; - if ($this->resourceSelector !== null) { $this->destination->runWithResourceSelector( $computedResources, - $progress, + $callback, $this->resourceSelector->resourceId, $this->resourceSelector->resourceInternalId, $this->resourceSelector->resourceType, @@ -401,23 +381,18 @@ public function run( $this->resourceSelector->parentResourceInternalId, $this->resourceSelector->parentResourceType, ); - } else { - $this->destination->run($computedResources, $progress, $rootResourceId, $rootResourceType); - } - if ($abort !== null) { - $this->destination->markAborted(); - - throw $abort; + return; } + + $this->destination->run($computedResources, $callback, $rootResourceId, $rootResourceType); } /** * Transfer resources using Appwrite's canonical resource relation fields. * * @param array> $resources Resources to transfer - * @param callable $callback Callback to run after transfer; throw Aborted from it to stop the transfer - * @throws Aborted The callback stopped the transfer + * @param callable $callback Callback to run after transfer * @throws \Exception */ public function runWithResourceSelector( diff --git a/tests/Migration/E2E/Destinations/AppwriteDatabaseConcurrencyTest.php b/tests/Migration/E2E/Destinations/AppwriteDatabaseConcurrencyTest.php deleted file mode 100644 index 51da650c..00000000 --- a/tests/Migration/E2E/Destinations/AppwriteDatabaseConcurrencyTest.php +++ /dev/null @@ -1,418 +0,0 @@ -> */ - public array $databaseWrites = []; - - public ?Closure $beforeBackingCollectionCreate = null; - - public bool $throwAfterBackingCollectionCallback = false; - - #[Override] - public function updateDocument(string $collection, string $id, UtopiaDocument $document): UtopiaDocument - { - if ($collection === 'databases') { - $this->databaseWrites[] = $document->getArrayCopy(); - } - - return parent::updateDocument($collection, $id, $document); - } - - #[Override] - public function createCollection(Collection $collection): Collection - { - if ( - $this->beforeBackingCollectionCreate !== null - && \str_starts_with($collection->getId(), 'database_') - ) { - $callback = $this->beforeBackingCollectionCreate; - $this->beforeBackingCollectionCreate = null; - $callback(); - - if ($this->throwAfterBackingCollectionCallback) { - throw new DatabaseException('backing collection creation failed'); - } - } - - return parent::createCollection($collection); - } -} - -final class AppwriteDatabaseConcurrencyTest extends TestCase -{ - /** - * @var array - */ - private const array PDO_ATTRIBUTES = [ - PDO::ATTR_TIMEOUT => 3, - PDO::ATTR_PERSISTENT => false, - PDO::ATTR_DEFAULT_FETCH_MODE => PDO::FETCH_ASSOC, - PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION, - PDO::ATTR_EMULATE_PREPARES => true, - PDO::ATTR_STRINGIFY_FETCHES => true, - ]; - - public function testStaleIncompleteClaimLosesAfterAnotherSuccessorCommits(): void - { - foreach (['provisioning', 'failed'] as $status) { - [$second, $third, $path] = $this->createSharedDatabases(); - - try { - $this->seedDatabase($second, $status, 'migration-first', 'attempt-first'); - $winner = null; - $loser = $this->createDestination( - $second, - 'migration-second', - 'attempt-second', - function (UtopiaDocument $snapshot) use ($third, &$winner): ProvisioningOwner { - $winner = $this->createDestination( - $third, - 'migration-third', - 'attempt-third', - static fn (UtopiaDocument $document): ProvisioningOwner => new ProvisioningOwner( - 'migration-first', - 'attempt-first', - ), - ); - $this->runTransfer($third, $winner); - - return new ProvisioningOwner('migration-first', 'attempt-first'); - }, - ); - $second->databaseWrites = []; - - $this->runTransfer($second, $loser); - - $database = $this->getDatabaseDocument($third); - $this->assertInstanceOf(AppwriteDestination::class, $winner); - $this->assertSame([], $this->errorMessages($winner)); - $this->assertNotSame([], $this->errorMessages($loser)); - $this->assertSame([], $second->databaseWrites); - $this->assertSame('ready', $database->getAttribute('status')); - $this->assertSame('migration-third', $database->getAttribute('migrationId')); - $this->assertSame('attempt-third', $database->getAttribute('migrationAttemptId')); - } finally { - $this->removeSQLiteFiles($path); - } - } - } - - public function testStaleOwnerCannotMarkDatabaseReadyAfterTakeover(): void - { - [$second, $third, $path] = $this->createSharedDatabases(); - - try { - $this->seedDatabase($second, 'provisioning', 'migration-shared', 'attempt-first'); - $successor = null; - $destination = $this->createDestination( - $second, - 'migration-shared', - 'attempt-second', - static fn (UtopiaDocument $snapshot): ProvisioningOwner => new ProvisioningOwner( - 'migration-shared', - 'attempt-first', - ), - ); - - $failure = null; - try { - $this->runTransfer( - $second, - $destination, - function () use ($third, &$successor): void { - $successor = $this->createDestination( - $third, - 'migration-shared', - 'attempt-third', - static fn (UtopiaDocument $snapshot): ProvisioningOwner => new ProvisioningOwner( - 'migration-shared', - 'attempt-second', - ), - ); - $this->claimWithoutTerminalTransition($third, $successor); - }, - ); - } catch (Finalization $error) { - $failure = $error; - } - - $database = $this->getDatabaseDocument($third); - $readyWrites = \array_values(\array_filter( - $second->databaseWrites, - static fn (array $document): bool => ($document['status'] ?? null) === 'ready', - )); - $this->assertInstanceOf(AppwriteDestination::class, $successor); - $this->assertInstanceOf(Finalization::class, $failure); - $this->assertSame($destination->getErrors(), $failure->failures); - $this->assertSame( - ['Database provisioning owner changed before finalization'], - $this->errorMessages($destination), - ); - $this->assertSame([], $this->errorMessages($successor)); - $this->assertSame([], $readyWrites); - $this->assertSame('provisioning', $database->getAttribute('status')); - $this->assertSame('migration-shared', $database->getAttribute('migrationId')); - $this->assertSame('attempt-third', $database->getAttribute('migrationAttemptId')); - } finally { - $this->removeSQLiteFiles($path); - } - } - - public function testStaleOwnerCannotMarkDatabaseFailedAfterTakeover(): void - { - [$second, $third, $path] = $this->createSharedDatabases(); - - try { - $this->seedDatabase($second, 'failed', 'migration-shared', 'attempt-first'); - $successor = null; - $second->throwAfterBackingCollectionCallback = true; - $second->beforeBackingCollectionCreate = function () use ($third, &$successor): void { - $successor = $this->createDestination( - $third, - 'migration-shared', - 'attempt-third', - static fn (UtopiaDocument $snapshot): ProvisioningOwner => new ProvisioningOwner( - 'migration-shared', - 'attempt-second', - ), - ); - $this->claimWithoutTerminalTransition($third, $successor); - }; - $destination = $this->createDestination( - $second, - 'migration-shared', - 'attempt-second', - static fn (UtopiaDocument $snapshot): ProvisioningOwner => new ProvisioningOwner( - 'migration-shared', - 'attempt-first', - ), - ); - - $this->runTransfer($second, $destination); - - $database = $this->getDatabaseDocument($third); - $failedWrites = \array_values(\array_filter( - $second->databaseWrites, - static fn (array $document): bool => ($document['status'] ?? null) === 'failed', - )); - $this->assertInstanceOf(AppwriteDestination::class, $successor); - $this->assertNotSame([], $this->errorMessages($destination)); - $this->assertSame([], $this->errorMessages($successor)); - $this->assertSame([], $failedWrites); - $this->assertSame('provisioning', $database->getAttribute('status')); - $this->assertSame('migration-shared', $database->getAttribute('migrationId')); - $this->assertSame('attempt-third', $database->getAttribute('migrationAttemptId')); - } finally { - $this->removeSQLiteFiles($path); - } - } - - private function createDestination( - UtopiaDatabase $database, - string $migrationId, - string $migrationAttemptId, - callable $getRecoverableOwner, - ): AppwriteDestination { - return new AppwriteDestination( - project: 'destination-project', - endpoint: 'http://example.test/v1', - key: 'test-key', - dbForProject: $database, - getDatabasesDB: static fn (UtopiaDocument $document): UtopiaDatabase => $database, - collectionStructure: ['attributes' => [], 'indexes' => []], - dbForPlatform: $database, - projectInternalId: '1', - owner: new ProvisioningOwner($migrationId, $migrationAttemptId), - getRecoverableOwner: $getRecoverableOwner, - onDuplicate: OnDuplicate::Fail, - ); - } - - private function runTransfer( - UtopiaDatabase $database, - AppwriteDestination $destination, - ?callable $callback = null, - ): void { - $source = new class () extends MockSource { - #[Override] - public function supportsDatabaseStatus(): bool - { - return true; - } - }; - $source->pushMockResource(new DatabaseResource( - id: 'database', - name: 'Database', - type: 'tablesdb', - database: 'source-dsn', - databaseStatus: 'ready', - )); - - $transfer = new Transfer($source, $destination); - $database->getAuthorization()->skip( - static function () use ($callback, $destination, $transfer): void { - $transfer->run( - [Resource::TYPE_DATABASE], - $callback ?? static function (): void { - }, - ); - $destination->success(); - }, - ); - } - - private function claimWithoutTerminalTransition( - UtopiaDatabase $database, - AppwriteDestination $destination, - ): void { - try { - $this->runTransfer( - $database, - $destination, - static function (): void { - throw new \RuntimeException('stop after provisioning claim'); - }, - ); - $this->fail('Expected transfer callback to stop before the terminal transition'); - } catch (\RuntimeException $error) { - $this->assertSame('stop after provisioning claim', $error->getMessage()); - } - } - - /** @return array{RecordingSQLiteProjectDatabase, RecordingSQLiteProjectDatabase, string} */ - private function createSharedDatabases(): array - { - $path = \tempnam(\sys_get_temp_dir(), 'migration-owner-'); - if ($path === false) { - throw new \RuntimeException('Failed to create SQLite test database'); - } - - $secondConnection = new SQLiteConnection('sqlite:'.$path, null, null, self::PDO_ATTRIBUTES); - $thirdConnection = new SQLiteConnection('sqlite:'.$path, null, null, self::PDO_ATTRIBUTES); - $secondConnection->exec('PRAGMA journal_mode = WAL'); - $secondConnection->exec('PRAGMA busy_timeout = 1000'); - $thirdConnection->exec('PRAGMA busy_timeout = 1000'); - - $second = new RecordingSQLiteProjectDatabase( - new SQLite($secondConnection), - new Cache(new None()), - ); - $third = new RecordingSQLiteProjectDatabase( - new SQLite($thirdConnection), - new Cache(new None()), - ); - $namespace = 'migration_owner_'.\uniqid(); - foreach ([$second, $third] as $database) { - $database - ->setDatabase('appwrite') - ->setNamespace($namespace); - } - - $second->create(); - $second->createCollection(Collection::create( - id: 'databases', - attributes: [ - $this->attribute('name', ColumnType::String, required: true, size: 256), - $this->attribute('enabled', ColumnType::Boolean, default: true), - $this->attribute('search', ColumnType::String, size: 16384), - $this->attribute('originalId', ColumnType::String, size: UtopiaDatabase::LENGTH_KEY), - $this->attribute('type', ColumnType::String, default: 'tablesdb', size: 128), - $this->attribute('database', ColumnType::String, size: 2000), - $this->attribute('status', ColumnType::String, size: 16), - $this->attribute('migrationId', ColumnType::String, size: UtopiaDatabase::LENGTH_KEY), - $this->attribute('migrationAttemptId', ColumnType::String, size: UtopiaDatabase::LENGTH_KEY), - ], - )); - - return [$second, $third, $path]; - } - - private function seedDatabase( - UtopiaDatabase $database, - string $status, - string $migrationId, - string $migrationAttemptId, - ): UtopiaDocument { - return $database->getAuthorization()->skip( - static fn (): UtopiaDocument => $database->createDocument('databases', new UtopiaDocument([ - '$id' => 'database', - 'name' => 'Database', - 'enabled' => true, - 'search' => 'database Database', - 'originalId' => null, - 'type' => 'tablesdb', - 'database' => '', - 'status' => $status, - 'migrationId' => $migrationId, - 'migrationAttemptId' => $migrationAttemptId, - ])), - ); - } - - private function attribute( - string $id, - ColumnType $type, - bool $required = false, - mixed $default = null, - int $size = 0, - ): UtopiaAttribute { - return UtopiaAttribute::fromArray([ - 'key' => $id, - 'type' => $type, - 'size' => $size, - 'required' => $required, - 'default' => $default, - ]); - } - - private function getDatabaseDocument(UtopiaDatabase $database): UtopiaDocument - { - return $database->getAuthorization()->skip( - static fn (): UtopiaDocument => $database->getDocument('databases', 'database'), - ); - } - - /** @return list */ - private function errorMessages(AppwriteDestination $destination): array - { - return \array_map( - static fn (\Throwable $error): string => $error->getMessage(), - $destination->getErrors(), - ); - } - - private function removeSQLiteFiles(string $path): void - { - foreach ([$path, $path.'-wal', $path.'-shm'] as $file) { - if (\is_file($file)) { - \unlink($file); - } - } - } -} diff --git a/tests/Migration/Unit/Destinations/AppwriteCheckAttributeTest.php b/tests/Migration/Unit/Destinations/AppwriteCheckAttributeTest.php index f2e0b398..a3ed5790 100644 --- a/tests/Migration/Unit/Destinations/AppwriteCheckAttributeTest.php +++ b/tests/Migration/Unit/Destinations/AppwriteCheckAttributeTest.php @@ -15,7 +15,6 @@ use Utopia\Database\Document as UtopiaDocument; use Utopia\Database\Query; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Database\Columns\Text; use Utopia\Migration\Resources\Database\Database as DatabaseResource; @@ -188,8 +187,6 @@ private function transferColumn(): array ], dbForPlatform: $database, projectInternalId: '1', - owner: new ProvisioningOwner('migration-test', 'attempt-test'), - getRecoverableOwner: static fn (UtopiaDocument $document): ?ProvisioningOwner => null, ); $transfer = new Transfer($source, $destination); diff --git a/tests/Migration/Unit/Destinations/AppwriteDatabaseStatusTest.php b/tests/Migration/Unit/Destinations/AppwriteDatabaseStatusTest.php index 13d8c79b..4b9789aa 100644 --- a/tests/Migration/Unit/Destinations/AppwriteDatabaseStatusTest.php +++ b/tests/Migration/Unit/Destinations/AppwriteDatabaseStatusTest.php @@ -9,43 +9,20 @@ use Utopia\Cache\Cache; use Utopia\Database\Adapter\Memory as MemoryAdapter; use Utopia\Database\Attribute as UtopiaAttribute; -use Utopia\Database\Capability; use Utopia\Database\Collection; use Utopia\Database\Database as UtopiaDatabase; use Utopia\Database\DateTime as UtopiaDateTime; use Utopia\Database\Document as UtopiaDocument; use Utopia\Database\Exception as DatabaseException; -use Utopia\Database\Exception\Conflict as ConflictException; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; use Utopia\Migration\Destinations\OnDuplicate; use Utopia\Migration\Exception as MigrationException; -use Utopia\Migration\Exception\Finalization; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Database\Database as DatabaseResource; use Utopia\Migration\Transfer; use Utopia\Query\Schema\ColumnType; use Utopia\Tests\Unit\Adapters\MockSource; -class ReplicaMemoryAdapter extends MemoryAdapter -{ - /** @return array */ - #[Override] - public function capabilities(): array - { - return [...parent::capabilities(), Capability::TransactionRetries]; - } -} - -final class StandaloneMemoryAdapter extends ReplicaMemoryAdapter -{ - #[Override] - public function withTransaction(callable $callback): mixed - { - return $callback(); - } -} - class CountingAppwriteDestination extends AppwriteDestination { public int $runCount = 0; @@ -115,127 +92,31 @@ public function updateDocument(string $collection, string $id, UtopiaDocument $d } } -final class InterleavingProjectDatabase extends UtopiaDatabase -{ - public bool $interceptNextDatabasesReload = false; - - /** @var (callable(): void)|null */ - public $onDatabasesReload = null; - - #[Override] - public function getDocument(string $collection, string $id, array $queries = [], bool $forUpdate = false): UtopiaDocument - { - $document = parent::getDocument($collection, $id, $queries, $forUpdate); - if ( - $this->interceptNextDatabasesReload - && $collection === 'databases' - && ! $document->isEmpty() - ) { - $this->interceptNextDatabasesReload = false; - ($this->onDatabasesReload)(); - } - - return $document; - } -} - -class RecordingProjectDatabase extends UtopiaDatabase +final class ReadyFailingProjectDatabase extends UtopiaDatabase { - /** @var list, guarded?: bool}> */ - public array $databaseWrites = []; - - public bool $failReadyWrite = false; - - public bool $disappearBeforeGuardedWrite = false; + /** @var list */ + public array $readyWrites = []; /** @var list */ public array $failReadyWrites = []; - protected ?\DateTime $guardedAt = null; - - #[Override] - public function withRequestTimestamp(?\DateTime $requestTimestamp, callable $callback): mixed - { - $previous = $this->guardedAt; - $this->guardedAt = $requestTimestamp; - try { - return parent::withRequestTimestamp($requestTimestamp, $callback); - } finally { - $this->guardedAt = $previous; - } - } - - #[Override] - public function createDocument(string $collection, UtopiaDocument $document): UtopiaDocument - { - if ($collection === 'databases') { - $this->databaseWrites[] = [ - 'operation' => 'create', - 'id' => $document->getId(), - 'document' => $document->getArrayCopy(), - ]; - } - - return parent::createDocument($collection, $document); - } - #[Override] public function updateDocument(string $collection, string $id, UtopiaDocument $document): UtopiaDocument { - if ($collection === 'databases') { - $this->databaseWrites[] = [ - 'operation' => 'update', - 'id' => $id, - 'document' => $document->getArrayCopy(), - 'guarded' => $this->guardedAt !== null, - ]; - } - - if ( - ($this->failReadyWrite || \in_array($id, $this->failReadyWrites, true)) - && $collection === 'databases' - && $document->getAttribute('status') === 'ready' - ) { - throw new DatabaseException('ready status unavailable'); - } - - if ( - $this->disappearBeforeGuardedWrite - && $collection === 'databases' - && $this->guardedAt !== null - ) { - $this->disappearBeforeGuardedWrite = false; - parent::deleteDocument($collection, $id); + if ($collection === 'databases' && $document->getAttribute('status') === 'ready') { + $this->readyWrites[] = $id; - return new UtopiaDocument(); + if (\in_array($id, $this->failReadyWrites, true)) { + throw new DatabaseException('ready status unavailable'); + } } return parent::updateDocument($collection, $id, $document); } } -final class FinalizerInterleavingProjectDatabase extends RecordingProjectDatabase -{ - public ?\Closure $beforeFinalizerRead = null; - - #[Override] - public function getDocument(string $collection, string $id, array $queries = [], bool $forUpdate = false): UtopiaDocument - { - $document = parent::getDocument($collection, $id, $queries, $forUpdate); - if ($forUpdate && $collection === 'databases' && $this->beforeFinalizerRead !== null) { - $callback = $this->beforeFinalizerRead; - $this->beforeFinalizerRead = null; - $callback(); - } - - return $document; - } -} - final class AppwriteDatabaseStatusTest extends TestCase { - private const string LAPSED_UPDATED_AT = '2020-01-01 00:00:00.000'; - public function testDatabaseCreationOmitsStatusThroughLegacyAndExplicitEntrypoints(): void { foreach ([false, true] as $explicit) { @@ -258,11 +139,7 @@ public function testDatabaseCreationOmitsStatusThroughLegacyAndExplicitEntrypoin public function testDatabaseCreationPreservesLifecycleThroughLegacyAndExplicitEntrypoints(): void { foreach ([false, true] as $explicit) { - $database = new UtopiaDatabase( - new StandaloneMemoryAdapter(), - new Cache(new MemoryCache()), - ); - $this->createProjectDatabase(withStatus: true, database: $database); + $database = $this->createProjectDatabase(withStatus: true); $destination = $this->runDatabaseTransfer($database, $explicit); @@ -278,523 +155,25 @@ public function testDatabaseCreationPreservesLifecycleThroughLegacyAndExplicitEn } } - public function testCreatePersistsProvisioningAndOwnerAtomically(): void - { - $database = new RecordingProjectDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-create', - migrationAttemptId: 'attempt-create', - ); - - $this->assertSame([], $this->errorMessages($destination)); - $this->assertSame('create', $database->databaseWrites[0]['operation']); - $this->assertSame('provisioning', $database->databaseWrites[0]['document']['status']); - $this->assertSame('migration-create', $database->databaseWrites[0]['document']['migrationId']); - $this->assertSame('attempt-create', $database->databaseWrites[0]['document']['migrationAttemptId']); - } - - public function testStandaloneUniqueCreateCannotFinalizeAfterOwnerMismatch(): void - { - $database = new FinalizerInterleavingProjectDatabase( - new StandaloneMemoryAdapter(), - new Cache(new MemoryCache()), - ); - $this->createProjectDatabase(withStatus: true, database: $database); - $database->beforeFinalizerRead = static function () use ($database): void { - $database->updateDocument( - 'databases', - 'database', - new UtopiaDocument([ - 'status' => 'provisioning', - 'migrationId' => 'migration-successor', - 'migrationAttemptId' => 'attempt-successor', - ]), - ); - }; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-create', - migrationAttemptId: 'attempt-create', - success: false, - ); - $failure = $this->finalizeExpectingFailure($database, $destination); - - $terminalWrites = \array_values(\array_filter( - $database->databaseWrites, - static fn (array $write): bool => \in_array( - $write['document']['status'] ?? null, - ['ready', 'failed'], - true, - ), - )); - $created = $this->getDatabaseDocument($database); - $this->assertSame(['Database provisioning owner changed before finalization'], $this->errorMessages($destination)); - $this->assertSame($destination->getErrors(), $failure->failures); - $this->assertCount(1, $terminalWrites); - $this->assertTrue($terminalWrites[0]['guarded'] ?? false, 'The terminal write must be pinned to the timestamp read under lock'); - $this->assertSame('provisioning', $created->getAttribute('status')); - $this->assertSame('migration-successor', $created->getAttribute('migrationId')); - $this->assertSame('attempt-successor', $created->getAttribute('migrationAttemptId')); - } - - public function testAuthorizedOverwritePersistsProvisioningAndNewOwnerAtomically(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'provisioning', - migrationId: 'migration-old', - migrationAttemptId: 'attempt-old', - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-new', - migrationAttemptId: 'attempt-new', - getRecoverableOwner: static fn (UtopiaDocument $database): ProvisioningOwner => new ProvisioningOwner( - 'migration-old', - 'attempt-old', - ), - ); - - $this->assertSame([], $this->errorMessages($destination)); - $write = $database->databaseWrites[0] ?? null; - $this->assertNotNull($write); - $this->assertSame('update', $write['operation']); - $this->assertSame('provisioning', $write['document']['status']); - $this->assertSame('migration-new', $write['document']['migrationId']); - $this->assertSame('attempt-new', $write['document']['migrationAttemptId']); - } - - public function testActiveProvisioningRefusalRetainsExistingOwner(): void - { - $database = new RecordingProjectDatabase(new StandaloneMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'provisioning', - migrationId: 'migration-active', - migrationAttemptId: 'attempt-active', - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-colliding', - migrationAttemptId: 'attempt-colliding', - getRecoverableOwner: static fn (UtopiaDocument $database): ?ProvisioningOwner => null, - ); - - $existing = $this->getDatabaseDocument($database); - $this->assertNotSame([], $this->errorMessages($destination)); - $this->assertSame([], $database->databaseWrites); - $this->assertSame('provisioning', $existing->getAttribute('status')); - $this->assertSame('migration-active', $existing->getAttribute('migrationId')); - $this->assertSame('attempt-active', $existing->getAttribute('migrationAttemptId')); - } - - public function testMissingOrMismatchedOwnerRefusesRecovery(): void - { - foreach ([null, 'migration-existing'] as $owner) { - $database = new RecordingProjectDatabase(new StandaloneMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'provisioning', - migrationId: $owner, - migrationAttemptId: 'attempt-existing', - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-new', - migrationAttemptId: 'attempt-new', - getRecoverableOwner: static fn (UtopiaDocument $database): ProvisioningOwner => new ProvisioningOwner( - 'migration-terminal', - 'attempt-existing', - ), - ); - - $existing = $this->getDatabaseDocument($database); - $this->assertNotSame([], $this->errorMessages($destination)); - $this->assertSame([], $database->databaseWrites); - $this->assertSame('provisioning', $existing->getAttribute('status')); - $this->assertSame($owner, $existing->getAttribute('migrationId')); - $this->assertSame('attempt-existing', $existing->getAttribute('migrationAttemptId')); - } - } - - public function testMissingOrMismatchedAttemptRefusesRecovery(): void - { - foreach ([null, 'attempt-existing'] as $attemptId) { - $database = new RecordingProjectDatabase(new StandaloneMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'provisioning', - migrationId: 'migration-existing', - migrationAttemptId: $attemptId, - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-new', - migrationAttemptId: 'attempt-new', - getRecoverableOwner: static fn (UtopiaDocument $database): ProvisioningOwner => new ProvisioningOwner( - 'migration-existing', - 'attempt-terminal', - ), - ); - - $existing = $this->getDatabaseDocument($database); - $this->assertNotSame([], $this->errorMessages($destination)); - $this->assertSame([], $database->databaseWrites); - $this->assertSame('provisioning', $existing->getAttribute('status')); - $this->assertSame('migration-existing', $existing->getAttribute('migrationId')); - $this->assertSame($attemptId, $existing->getAttribute('migrationAttemptId')); - } - } - - public function testRecoveryRefusesReusingThePriorAttempt(): void - { - $database = new RecordingProjectDatabase(new StandaloneMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'failed', - migrationId: 'migration-shared', - migrationAttemptId: 'attempt-shared', - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-shared', - migrationAttemptId: 'attempt-shared', - getRecoverableOwner: static fn (UtopiaDocument $database): ProvisioningOwner => new ProvisioningOwner( - 'migration-shared', - 'attempt-shared', - ), - ); - - $existing = $this->getDatabaseDocument($database); - $this->assertNotSame([], $this->errorMessages($destination)); - $this->assertSame([], $database->databaseWrites); - $this->assertSame('failed', $existing->getAttribute('status')); - $this->assertSame('migration-shared', $existing->getAttribute('migrationId')); - $this->assertSame('attempt-shared', $existing->getAttribute('migrationAttemptId')); - } - - public function testSameMigrationFailedRetryRequiresTerminalAttestation(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'failed', - migrationId: 'migration-shared', - migrationAttemptId: 'attempt-active', - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-shared', - migrationAttemptId: 'attempt-retry', - getRecoverableOwner: static fn (UtopiaDocument $database): ?ProvisioningOwner => null, - ); - - $existing = $this->getDatabaseDocument($database); - $this->assertNotSame([], $this->errorMessages($destination)); - $this->assertSame([], $database->databaseWrites); - $this->assertSame('failed', $existing->getAttribute('status')); - $this->assertSame('migration-shared', $existing->getAttribute('migrationId')); - $this->assertSame('attempt-active', $existing->getAttribute('migrationAttemptId')); - } - - public function testPreExistingRecoveryUsesDocumentVersionCompareAndSet(): void - { - foreach ([new MemoryAdapter(), new StandaloneMemoryAdapter()] as $adapter) { - $database = new RecordingProjectDatabase($adapter, new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'failed', - migrationId: 'migration-old', - migrationAttemptId: 'attempt-old', - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-old', - migrationAttemptId: 'attempt-new', - getRecoverableOwner: static fn (UtopiaDocument $database): ProvisioningOwner => new ProvisioningOwner( - 'migration-old', - 'attempt-old', - ), - ); - - $existing = $this->getDatabaseDocument($database); - $this->assertSame([], $this->errorMessages($destination)); - $this->assertNotSame([], $database->databaseWrites); - $this->assertSame('ready', $existing->getAttribute('status')); - $this->assertSame('migration-old', $existing->getAttribute('migrationId')); - $this->assertSame('attempt-new', $existing->getAttribute('migrationAttemptId')); - } - } - - public function testHealthyOverwriteUsesDocumentVersionCompareAndSet(): void - { - foreach ([new MemoryAdapter(), new StandaloneMemoryAdapter()] as $adapter) { - $database = new RecordingProjectDatabase($adapter, new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'ready', - migrationId: 'migration-old', - migrationAttemptId: 'attempt-old', - ); - $database->getAuthorization()->skip( - static fn (): UtopiaDocument => $database->updateDocument( - 'databases', - 'database', - new UtopiaDocument(['name' => 'Old database']), - ), - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-new', - migrationAttemptId: 'attempt-new', - onDuplicate: OnDuplicate::Overwrite, - resourceUpdatedAt: '2099-01-01T00:00:00.000+00:00', - ); - - $existing = $this->getDatabaseDocument($database); - $this->assertSame([], $this->errorMessages($destination)); - $this->assertNotSame([], $database->databaseWrites); - $this->assertSame('ready', $existing->getAttribute('status')); - $this->assertSame('migration-new', $existing->getAttribute('migrationId')); - $this->assertSame('attempt-new', $existing->getAttribute('migrationAttemptId')); - } - } - - public function testFailedDatabaseRecoveryRequiresExactTerminalOwner(): void - { - foreach ( - [ - 'no attestation' => ['migration-terminal', null], - 'missing owner' => [null, 'migration-terminal'], - 'mismatched owner' => ['migration-existing', 'migration-terminal'], - 'unknown terminal owner' => ['migration-existing', null], - ] as [$owner, $recoverableMigrationId] - ) { - $database = new RecordingProjectDatabase(new StandaloneMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'failed', - migrationId: $owner, - migrationAttemptId: 'attempt-existing', - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-new', - migrationAttemptId: 'attempt-new', - getRecoverableOwner: static fn (UtopiaDocument $database): ?ProvisioningOwner => $recoverableMigrationId === null - ? null - : new ProvisioningOwner($recoverableMigrationId, 'attempt-existing'), - ); - - $existing = $this->getDatabaseDocument($database); - $this->assertNotSame([], $this->errorMessages($destination)); - $this->assertSame([], $database->databaseWrites); - $this->assertSame('failed', $existing->getAttribute('status')); - $this->assertSame($owner, $existing->getAttribute('migrationId')); - $this->assertSame('attempt-existing', $existing->getAttribute('migrationAttemptId')); - } - } - - public function testReadyWriteFailureRetainsProvisioningOwner(): void - { - $database = new RecordingProjectDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $database->failReadyWrite = true; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-ready-failure', - migrationAttemptId: 'attempt-ready-failure', - success: false, - ); - $failure = $this->finalizeExpectingFailure($database, $destination); - - $created = $this->getDatabaseDocument($database); - $errors = $destination->getErrors(); - $this->assertCount(1, $errors); - $this->assertSame($errors, $failure->failures); - $this->assertSame(Resource::TYPE_DATABASE, $errors[0]->getResourceName()); - $this->assertSame(Transfer::GROUP_DATABASES, $errors[0]->getResourceGroup()); - $this->assertSame('database', $errors[0]->getResourceId()); - $this->assertSame('ready status unavailable', $errors[0]->getMessage()); - $this->assertInstanceOf(DatabaseException::class, $errors[0]->getPrevious()); - $this->assertSame('provisioning', $created->getAttribute('status')); - $this->assertSame('migration-ready-failure', $created->getAttribute('migrationId')); - $this->assertSame('attempt-ready-failure', $created->getAttribute('migrationAttemptId')); - } - - public function testDatabaseDisappearanceBeforeReadyWriteLosesFinalizationOwnership(): void + public function testReadyFlipAttemptsEveryDatabaseAndReportsEveryFailure(): void { - $database = new RecordingProjectDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $database->disappearBeforeGuardedWrite = true; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-disappearing-ready', - migrationAttemptId: 'attempt-disappearing-ready', - success: false, - ); - $failure = $this->finalizeExpectingFailure($database, $destination); - - $readyWrites = \array_values(\array_filter( - $database->databaseWrites, - static fn (array $write): bool => ($write['document']['status'] ?? null) === 'ready', - )); - - $this->assertSame(['Database provisioning owner changed before finalization'], $this->errorMessages($destination)); - $this->assertSame($destination->getErrors(), $failure->failures); - $this->assertCount(1, $readyWrites); - $this->assertTrue($readyWrites[0]['guarded'] ?? false, 'The ready write must be pinned to the timestamp read under lock'); - $this->assertTrue($this->getDatabaseDocument($database)->isEmpty()); - } - - public function testDatabaseDisappearanceBeforeRecoveryClaimLosesOwnership(): void - { - $database = new RecordingProjectDatabase(new StandaloneMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $seeded = $this->seedDatabase( - $database, - status: 'failed', - migrationId: 'migration-old', - migrationAttemptId: 'attempt-old', - ); - $database->databaseWrites = []; - $database->disappearBeforeGuardedWrite = true; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-new', - migrationAttemptId: 'attempt-new', - getRecoverableOwner: static fn (UtopiaDocument $database): ProvisioningOwner => new ProvisioningOwner( - 'migration-old', - 'attempt-old', - ), - success: false, - ); - - $claimWrites = \array_values(\array_filter( - $database->databaseWrites, - static fn (array $write): bool => ($write['document']['status'] ?? null) === 'provisioning', - )); - - $this->assertSame( - ['Database database provisioning owner changed before it could be claimed'], - $this->errorMessages($destination), - ); - $this->assertCount(1, $claimWrites); - $this->assertSame( - [true], - \array_map( - static fn (array $write): bool => $write['guarded'] ?? false, - $claimWrites, - ), - ); - $this->assertTrue($this->getDatabaseDocument($database)->isEmpty()); - $this->assertNull($database->findCollection('database_'.$seeded->getSequence())); - } - - public function testDatabaseFinalizersRunOnlyAfterSuccess(): void - { - $database = new RecordingProjectDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-finalizing', - migrationAttemptId: 'attempt-finalizing', - success: false, - ); - - $provisioning = $this->getDatabaseDocument($database); - $this->assertSame('provisioning', $provisioning->getAttribute('status')); - $this->assertSame('migration-finalizing', $provisioning->getAttribute('migrationId')); - $this->assertSame('attempt-finalizing', $provisioning->getAttribute('migrationAttemptId')); - - $database->getAuthorization()->skip($destination->success(...)); - - $this->assertSame('ready', $this->getDatabaseDocument($database)->getAttribute('status')); - } - - public function testReadyFinalizationAttemptsEveryDatabaseAndReportsEveryFailure(): void - { - $database = new RecordingProjectDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); + $database = new ReadyFailingProjectDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); $this->createProjectDatabase(withStatus: true, database: $database); $database->failReadyWrites = ['database-first', 'database-third']; $destination = $this->runDatabaseTransfer( $database, explicit: false, - migrationId: 'migration-finalizers', - migrationAttemptId: 'attempt-finalizers', databaseIds: ['database-first', 'database-second', 'database-third'], - success: false, ); - $failure = $this->finalizeExpectingFailure($database, $destination); - $readyAttempts = \array_values(\array_map( - static fn (array $write): string => $write['id'], - \array_filter( - $database->databaseWrites, - static fn (array $write): bool => ($write['document']['status'] ?? null) === 'ready', - ), - )); - $errors = $destination->getErrors(); $errorIds = \array_map( static fn (MigrationException $error): string => $error->getResourceId(), - $errors, + $destination->getErrors(), ); - $this->assertSame(['database-first', 'database-second', 'database-third'], $readyAttempts); + $this->assertSame(['database-first', 'database-second', 'database-third'], $database->readyWrites); $this->assertSame(['database-first', 'database-third'], $errorIds); - $this->assertSame($errors, $failure->failures); $this->assertSame('provisioning', $this->getDatabaseDocument($database, 'database-first')->getAttribute('status')); $this->assertSame('ready', $this->getDatabaseDocument($database, 'database-second')->getAttribute('status')); $this->assertSame('provisioning', $this->getDatabaseDocument($database, 'database-third')->getAttribute('status')); @@ -803,7 +182,7 @@ public function testReadyFinalizationAttemptsEveryDatabaseAndReportsEveryFailure public function testReloadFailureMarksTheDatabaseFailed(): void { $database = new ReloadFailingProjectDatabase( - new StandaloneMemoryAdapter(), + new MemoryAdapter(), new Cache(new MemoryCache()), ); $this->createProjectDatabase(withStatus: true, database: $database); @@ -815,8 +194,6 @@ public function testReloadFailureMarksTheDatabaseFailed(): void $this->assertNotSame([], $this->errorMessages($destination)); $this->assertStringContainsString('Failed to reload created database', $this->errorMessages($destination)[0]); $this->assertSame('failed', $created->getAttribute('status')); - $this->assertSame('migration-current', $created->getAttribute('migrationId')); - $this->assertSame('attempt-current', $created->getAttribute('migrationAttemptId')); $this->assertNull( $database->findCollection('database_'.$created->getSequence()), 'A reload failure must not leave a backing collection behind the metadata document', @@ -826,7 +203,7 @@ public function testReloadFailureMarksTheDatabaseFailed(): void public function testFailedDatabaseRetrySucceedsUnderOnDuplicateFail(): void { $database = new ReloadFailingProjectDatabase( - new ReplicaMemoryAdapter(), + new MemoryAdapter(), new Cache(new MemoryCache()), ); $this->createProjectDatabase(withStatus: true, database: $database); @@ -840,23 +217,12 @@ public function testFailedDatabaseRetrySucceedsUnderOnDuplicateFail(): void $database->findCollection('database_'.$failed->getSequence()), ); - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-current', - migrationAttemptId: 'attempt-recovery', - getRecoverableOwner: static fn (UtopiaDocument $existing): ProvisioningOwner => new ProvisioningOwner( - 'migration-current', - 'attempt-current', - ), - ); + $destination = $this->runDatabaseTransfer($database, explicit: false); $recovered = $this->getDatabaseDocument($database); $this->assertSame([], $this->errorMessages($destination)); $this->assertSame('ready', $recovered->getAttribute('status')); $this->assertSame($failed->getSequence(), $recovered->getSequence()); - $this->assertSame('migration-current', $recovered->getAttribute('migrationId')); - $this->assertSame('attempt-recovery', $recovered->getAttribute('migrationAttemptId')); $this->assertNotNull( $database->findCollection('database_'.$recovered->getSequence()), 'A Fail retry must recreate the backing collection for a previously failed database', @@ -866,7 +232,7 @@ public function testFailedDatabaseRetrySucceedsUnderOnDuplicateFail(): void public function testProvisioningDatabaseRetrySucceedsUnderOnDuplicateFail(): void { $database = new StrandedProvisioningProjectDatabase( - new ReplicaMemoryAdapter(), + new MemoryAdapter(), new Cache(new MemoryCache()), ); $this->createProjectDatabase(withStatus: true, database: $database); @@ -887,605 +253,54 @@ public function testProvisioningDatabaseRetrySucceedsUnderOnDuplicateFail(): voi ); $database->failDatabasesWrites = false; - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-recovery', - migrationAttemptId: 'attempt-recovery', - getRecoverableOwner: static fn (UtopiaDocument $existing): ProvisioningOwner => new ProvisioningOwner( - 'migration-current', - 'attempt-current', - ), - ); + $destination = $this->runDatabaseTransfer($database, explicit: false); $recovered = $this->getDatabaseDocument($database); $this->assertSame([], $this->errorMessages($destination)); $this->assertSame('ready', $recovered->getAttribute('status')); $this->assertSame($stranded->getSequence(), $recovered->getSequence()); - $this->assertSame('migration-recovery', $recovered->getAttribute('migrationId')); - $this->assertSame('attempt-recovery', $recovered->getAttribute('migrationAttemptId')); $this->assertNotNull( $database->findCollection('database_'.$recovered->getSequence()), 'A Fail retry must recover a database stranded in provisioning, not keep colliding with its metadata id', ); } - public function testConcurrentMigrationDoesNotRecoverAnActivelyProvisioningDatabase(): void - { - $database = new InterleavingProjectDatabase( - new MemoryAdapter(), - new Cache(new MemoryCache()), - ); - $this->createProjectDatabase(withStatus: true, database: $database); - - $second = null; - $statusDuringOverlap = null; - $collectionExistsDuringOverlap = null; - $database->onDatabasesReload = function () use ( - $database, - &$second, - &$statusDuringOverlap, - &$collectionExistsDuringOverlap, - ): void { - $second = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-second', - migrationAttemptId: 'attempt-second', - ); - $provisioning = $this->getDatabaseDocument($database); - $statusDuringOverlap = $provisioning->getAttribute('status'); - $collectionExistsDuringOverlap = $database - ->findCollection('database_'.$provisioning->getSequence()) !== null; - }; - $database->interceptNextDatabasesReload = true; - - $first = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-first', - migrationAttemptId: 'attempt-first', - ); - - $created = $this->getDatabaseDocument($database); - $this->assertSame([], $this->errorMessages($first)); - $this->assertInstanceOf(CountingAppwriteDestination::class, $second); - $this->assertNotSame([], $this->errorMessages($second)); - $this->assertStringContainsString('terminal-owner attestation', $this->errorMessages($second)[0]); - $this->assertSame('provisioning', $statusDuringOverlap); - $this->assertFalse($collectionExistsDuringOverlap); - $this->assertSame('ready', $created->getAttribute('status')); - $this->assertSame('migration-first', $created->getAttribute('migrationId')); - $this->assertSame('attempt-first', $created->getAttribute('migrationAttemptId')); - $this->assertNotNull($database->findCollection('database_'.$created->getSequence())); - } - - public function testAuthorizedOverwriteReplacesTerminalOwner(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'provisioning', - migrationId: 'migration-terminal', - migrationAttemptId: 'attempt-terminal', - ); - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-successor', - migrationAttemptId: 'attempt-successor', - getRecoverableOwner: static fn (UtopiaDocument $database): ProvisioningOwner => new ProvisioningOwner( - 'migration-terminal', - 'attempt-terminal', - ), - ); - - $recovered = $this->getDatabaseDocument($database); - $this->assertSame([], $this->errorMessages($destination)); - $this->assertSame('ready', $recovered->getAttribute('status')); - $this->assertSame('migration-successor', $recovered->getAttribute('migrationId')); - $this->assertSame('attempt-successor', $recovered->getAttribute('migrationAttemptId')); - } - - /** @return array */ - public static function unknownAttributePolicies(): array + /** @return array */ + public static function duplicatePolicies(): array { return [ - 'unknown attributes rejected' => [false], - 'unknown attributes dropped' => [true], + 'fail' => [OnDuplicate::Fail, ''], + 'skip' => [OnDuplicate::Skip, ''], + 'overwrite with an older source' => [OnDuplicate::Overwrite, '2000-01-01T00:00:00.000+00:00'], + 'overwrite with a newer source' => [OnDuplicate::Overwrite, '2099-01-01T00:00:00.000+00:00'], ]; } - #[DataProvider('unknownAttributePolicies')] - public function testStatusOnlySchemaImportsWithoutProvisioningOwner(bool $dropUnknownAttributes): void + #[DataProvider('duplicatePolicies')] + public function testIncompleteDatabasesAreRecoveredImmediately(OnDuplicate $onDuplicate, string $resourceUpdatedAt): void { - $database = new RecordingProjectDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); - $database->setDropUnknownAttributes($dropUnknownAttributes); - $this->createStatusOnlyProjectDatabase($database); - - $destination = $this->runDatabaseTransfer($database, explicit: false); - - $created = $this->getDatabaseDocument($database); - $this->assertSame([], $this->errorMessages($destination)); - $this->assertSame('create', $database->databaseWrites[0]['operation']); - $this->assertSame('provisioning', $database->databaseWrites[0]['document']['status']); - $this->assertArrayNotHasKey('migrationId', $database->databaseWrites[0]['document']); - $this->assertArrayNotHasKey('migrationAttemptId', $database->databaseWrites[0]['document']); - $this->assertSame('ready', $created->getAttribute('status')); - $this->assertNotNull($database->findCollection('database_'.$created->getSequence())); - } - - public function testStatusOnlySchemaRecoversIncompleteDatabasesWithoutOwner(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createStatusOnlyProjectDatabase($database); - $this->seedLegacyDatabase($database, 'database-provisioning', 'provisioning', withCollection: true); - $this->seedLegacyDatabase($database, 'database-failed', 'failed'); - $database->databaseWrites = []; + $database = $this->createProjectDatabase(withStatus: true); + $provisioning = $this->seedIncompleteDatabase($database, 'database-provisioning', 'provisioning', name: 'Incomplete database', withCollection: true, updatedAt: $this->secondsAgo(1)); + $failed = $this->seedIncompleteDatabase($database, 'database-failed', 'failed', name: 'Incomplete database', updatedAt: $this->secondsAgo(1)); $destination = $this->runDatabaseTransfer( $database, explicit: false, databaseIds: ['database-provisioning', 'database-failed'], - ); - - $this->assertSame([], $this->errorMessages($destination)); - foreach (['database-provisioning', 'database-failed'] as $databaseId) { - $recovered = $this->getDatabaseDocument($database, $databaseId); - $this->assertSame('ready', $recovered->getAttribute('status'), $databaseId); - $this->assertNotNull($database->findCollection('database_'.$recovered->getSequence()), $databaseId); - } - $this->assertNotSame([], $database->databaseWrites); - foreach ($database->databaseWrites as $write) { - $this->assertArrayNotHasKey('migrationId', $write['document']); - $this->assertArrayNotHasKey('migrationAttemptId', $write['document']); - } - } - - /** @return array */ - public static function legacyProvisioningResolutions(): array - { - return [ - 'fail keeps the destination' => [OnDuplicate::Fail, '', 'Legacy database'], - 'skip keeps the destination' => [OnDuplicate::Skip, '', 'Legacy database'], - 'overwrite with an older source keeps the destination' => [OnDuplicate::Overwrite, '2000-01-01T00:00:00.000+00:00', 'Legacy database'], - 'overwrite with a newer source replaces the destination' => [OnDuplicate::Overwrite, '2099-01-01T00:00:00.000+00:00', 'Database'], - ]; - } - - #[DataProvider('legacyProvisioningResolutions')] - public function testLegacyProvisioningDatabaseIsRecoveredAndClaimed( - OnDuplicate $onDuplicate, - string $resourceUpdatedAt, - string $expectedName, - ): void { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $seeded = $this->seedLegacyDatabase( - $database, - 'database', - 'provisioning', - name: 'Legacy database', - withCollection: true, - ); - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, onDuplicate: $onDuplicate, resourceUpdatedAt: $resourceUpdatedAt, ); - $recovered = $this->getDatabaseDocument($database); - $this->assertSame([], $this->errorMessages($destination)); - $this->assertSame('ready', $recovered->getAttribute('status')); - $this->assertSame($expectedName, $recovered->getAttribute('name')); - $this->assertSame($seeded->getSequence(), $recovered->getSequence()); - $this->assertSame('migration-current', $recovered->getAttribute('migrationId')); - $this->assertSame('attempt-current', $recovered->getAttribute('migrationAttemptId')); - $this->assertNotNull($database->findCollection('database_'.$recovered->getSequence())); - } - - public function testLegacyProvisioningDatabaseMissingItsCollectionIsRecreated(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $seeded = $this->seedLegacyDatabase($database, 'database', 'provisioning'); - - $destination = $this->runDatabaseTransfer($database, explicit: false); - - $recovered = $this->getDatabaseDocument($database); $this->assertSame([], $this->errorMessages($destination)); - $this->assertSame('ready', $recovered->getAttribute('status')); - $this->assertSame($seeded->getSequence(), $recovered->getSequence()); - $this->assertSame('migration-current', $recovered->getAttribute('migrationId')); - $this->assertSame('attempt-current', $recovered->getAttribute('migrationAttemptId')); - $this->assertNotNull( - $database->findCollection('database_'.$recovered->getSequence()), - 'A recovered database must never be marked ready without its backing collection', - ); - } - - public function testLegacyFailedDatabaseIsOverwrittenAndClaimed(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $seeded = $this->seedLegacyDatabase($database, 'database', 'failed', name: 'Legacy database'); - - $destination = $this->runDatabaseTransfer($database, explicit: false); - - $recovered = $this->getDatabaseDocument($database); - $this->assertSame([], $this->errorMessages($destination)); - $this->assertSame('ready', $recovered->getAttribute('status')); - $this->assertSame('Database', $recovered->getAttribute('name')); - $this->assertSame($seeded->getSequence(), $recovered->getSequence()); - $this->assertSame('migration-current', $recovered->getAttribute('migrationId')); - $this->assertSame('attempt-current', $recovered->getAttribute('migrationAttemptId')); - $this->assertNotNull($database->findCollection('database_'.$recovered->getSequence())); - } - - public function testOwnedDatabaseWithoutAttestationIsRefusedWhileLegacyDatabaseRecovers(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'provisioning', - migrationId: 'migration-active', - migrationAttemptId: 'attempt-active', - ); - $this->seedLegacyDatabase($database, 'database-legacy', 'failed'); - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - databaseIds: ['database', 'database-legacy'], - ); - - $owned = $this->getDatabaseDocument($database); - $legacy = $this->getDatabaseDocument($database, 'database-legacy'); - $this->assertSame( - ['Database database recovery requires exact terminal-owner attestation and a fresh attempt'], - $this->errorMessages($destination), - ); - $this->assertSame('provisioning', $owned->getAttribute('status')); - $this->assertSame('migration-active', $owned->getAttribute('migrationId')); - $this->assertSame('attempt-active', $owned->getAttribute('migrationAttemptId')); - $this->assertSame('ready', $legacy->getAttribute('status')); - $this->assertSame('migration-current', $legacy->getAttribute('migrationId')); - $this->assertSame('attempt-current', $legacy->getAttribute('migrationAttemptId')); - } - - public function testLegacyRecoveryYieldsToAClaimThatLandsFirst(): void - { - $database = new InterleavingProjectDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedLegacyDatabase($database, 'database', 'failed'); - $database->onDatabasesReload = static function () use ($database): void { - $database->updateDocument('databases', 'database', new UtopiaDocument([ - 'status' => 'provisioning', - 'migrationId' => 'migration-rival', - 'migrationAttemptId' => 'attempt-rival', - ])); - }; - $database->interceptNextDatabasesReload = true; - - $destination = $this->runDatabaseTransfer($database, explicit: false); - - $claimed = $this->getDatabaseDocument($database); - $this->assertSame( - ['Database database recovery owner changed before it could be claimed'], - $this->errorMessages($destination), - ); - $this->assertSame('provisioning', $claimed->getAttribute('status')); - $this->assertSame('migration-rival', $claimed->getAttribute('migrationId')); - $this->assertSame('attempt-rival', $claimed->getAttribute('migrationAttemptId')); - } - - public function testRecoveryClaimUnderPreservedDatesStillMovesTheTimestampForward(): void - { - $database = new class (new ReplicaMemoryAdapter(), new Cache(new MemoryCache())) extends RecordingProjectDatabase { - #[Override] - public function setPreserveDates(bool $preserve): static - { - return parent::setPreserveDates(true); - } - }; - $this->createProjectDatabase(withStatus: true, database: $database); - $seeded = $this->seedDatabase( - $database, - status: 'failed', - migrationId: 'migration-old', - migrationAttemptId: 'attempt-old', - ); - $readAt = (string) $seeded->getUpdatedAt(); - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-new', - migrationAttemptId: 'attempt-new', - getRecoverableOwner: static fn (UtopiaDocument $database): ProvisioningOwner => new ProvisioningOwner( - 'migration-old', - 'attempt-old', - ), - resourceUpdatedAt: '2000-01-01T00:00:00.000+00:00', - success: false, - ); - - $claimed = $this->getDatabaseDocument($database); - $this->assertSame([], $this->errorMessages($destination)); - $this->assertSame('attempt-new', $claimed->getAttribute('migrationAttemptId')); - $this->assertGreaterThan( - new \DateTime($readAt), - new \DateTime((string) $claimed->getUpdatedAt()), - 'The claim must move $updatedAt forward whatever the preserveDates setting', - ); - - $this->expectException(ConflictException::class); - $database->getAuthorization()->skip( - static fn (): UtopiaDocument => $database->withRequestTimestamp( - new \DateTime($readAt), - static fn (): UtopiaDocument => $database->updateDocument( - 'databases', - 'database', - new UtopiaDocument(['status' => 'ready']), - ), - ), - ); - } - - public function testGuardedWriteThatDoesNotAdvanceTheTimestampIsRolledBack(): void - { - $database = new class (new ReplicaMemoryAdapter(), new Cache(new MemoryCache())) extends RecordingProjectDatabase { - #[Override] - public function updateDocument(string $collection, string $id, UtopiaDocument $document): UtopiaDocument - { - if ($collection !== 'databases' || $this->guardedAt === null) { - return parent::updateDocument($collection, $id, $document); - } - - return $this->withPreserveDates( - true, - fn (): UtopiaDocument => parent::updateDocument( - $collection, - $id, - $document->setAttribute('$updatedAt', '2000-01-01T00:00:00.000+00:00'), - ), - ); - } - }; - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'failed', - migrationId: 'migration-old', - migrationAttemptId: 'attempt-old', - ); - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-new', - migrationAttemptId: 'attempt-new', - getRecoverableOwner: static fn (UtopiaDocument $database): ProvisioningOwner => new ProvisioningOwner( - 'migration-old', - 'attempt-old', - ), - ); - - $existing = $this->getDatabaseDocument($database); - $this->assertSame( - ['Database database provisioning write did not move its update timestamp forward'], - $this->errorMessages($destination), - ); - $this->assertSame('failed', $existing->getAttribute('status')); - $this->assertSame('migration-old', $existing->getAttribute('migrationId')); - $this->assertSame('attempt-old', $existing->getAttribute('migrationAttemptId')); - } - - - public function testOwnerlessProvisioningDatabaseWithinTheLeaseIsNotRecovered(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createStatusOnlyProjectDatabase($database); - $seeded = $this->seedLegacyDatabase( - $database, - 'database', - 'provisioning', - name: 'Provisioning database', - withCollection: true, - updatedAt: $this->secondsAgo(5), - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer($database, explicit: false); - - $untouched = $this->getDatabaseDocument($database); - $this->assertSame( - ['Database database names no owner and another migration may still be provisioning it; it becomes recoverable after 86400 seconds without an update'], - $this->errorMessages($destination), - ); - $this->assertSame('provisioning', $untouched->getAttribute('status')); - $this->assertSame('Provisioning database', $untouched->getAttribute('name')); - $this->assertSame($seeded->getUpdatedAt(), $untouched->getUpdatedAt()); - $this->assertSame([], $database->databaseWrites, 'A refused database must not be written, and success() must flip nothing'); - } - - public function testOwnerlessFailedDatabaseWithinTheLeaseIsNotOverwritten(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createStatusOnlyProjectDatabase($database); - $seeded = $this->seedLegacyDatabase( - $database, - 'database', - 'failed', - name: 'Failed database', - updatedAt: $this->secondsAgo(5), - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer($database, explicit: false); - - $untouched = $this->getDatabaseDocument($database); - $this->assertSame( - ['Database database names no owner and another migration may still be provisioning it; it becomes recoverable after 86400 seconds without an update'], - $this->errorMessages($destination), - ); - $this->assertSame('failed', $untouched->getAttribute('status')); - $this->assertSame('Failed database', $untouched->getAttribute('name')); - $this->assertSame($seeded->getUpdatedAt(), $untouched->getUpdatedAt()); - $this->assertSame([], $database->databaseWrites, 'A refused database must not be overwritten'); - } - - public function testOwnerlessDatabasesPastTheLeaseAreRecovered(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createStatusOnlyProjectDatabase($database); - $this->seedLegacyDatabase( - $database, - 'database-provisioning', - 'provisioning', - withCollection: true, - updatedAt: $this->secondsAgo(AppwriteDestination::DEFAULT_PROVISIONING_LEASE + 1), - ); - $this->seedLegacyDatabase( - $database, - 'database-failed', - 'failed', - updatedAt: $this->secondsAgo(AppwriteDestination::DEFAULT_PROVISIONING_LEASE + 1), - ); - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - databaseIds: ['database-provisioning', 'database-failed'], - ); - - $this->assertSame([], $this->errorMessages($destination)); - foreach (['database-provisioning', 'database-failed'] as $databaseId) { - $recovered = $this->getDatabaseDocument($database, $databaseId); - $this->assertSame('ready', $recovered->getAttribute('status'), $databaseId); - $this->assertNotNull($database->findCollection('database_'.$recovered->getSequence()), $databaseId); + foreach ([$provisioning, $failed] as $seeded) { + $recovered = $this->getDatabaseDocument($database, $seeded->getId()); + $this->assertSame('ready', $recovered->getAttribute('status'), $seeded->getId()); + $this->assertSame('Database', $recovered->getAttribute('name'), $seeded->getId().' must be overwritten from the source whatever the duplicate policy'); + $this->assertSame($seeded->getSequence(), $recovered->getSequence(), $seeded->getId()); + $this->assertNotNull($database->findCollection('database_'.$recovered->getSequence()), $seeded->getId()); } } - public function testTheOwnerlessProvisioningClaimRestartsTheLease(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createStatusOnlyProjectDatabase($database); - $seeded = $this->seedLegacyDatabase( - $database, - 'database', - 'provisioning', - withCollection: true, - updatedAt: $this->secondsAgo(AppwriteDestination::DEFAULT_PROVISIONING_LEASE + 1), - ); - - $claiming = $this->runDatabaseTransfer($database, explicit: false, success: false); - - $claimed = $this->getDatabaseDocument($database); - $this->assertSame([], $this->errorMessages($claiming)); - $this->assertSame('provisioning', $claimed->getAttribute('status')); - $this->assertGreaterThan( - new \DateTime((string) $seeded->getUpdatedAt()), - new \DateTime((string) $claimed->getUpdatedAt()), - 'The claim must restart the lease on the row it recovered', - ); - - $second = $this->runDatabaseTransfer($database, explicit: false, success: false); - - $refused = $this->getDatabaseDocument($database); - $this->assertSame( - ['Database database names no owner and another migration may still be provisioning it; it becomes recoverable after 86400 seconds without an update'], - $this->errorMessages($second), - 'A migration arriving inside the restarted lease must not recover the same database', - ); - $this->assertSame('provisioning', $refused->getAttribute('status')); - $this->assertSame((string) $claimed->getUpdatedAt(), (string) $refused->getUpdatedAt()); - } - - public function testProvisioningLeaseMovesTheOwnerlessRecoveryBoundary(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createStatusOnlyProjectDatabase($database); - $this->seedLegacyDatabase($database, 'database-lapsed', 'provisioning', withCollection: true, updatedAt: $this->secondsAgo(61)); - $this->seedLegacyDatabase($database, 'database-live', 'provisioning', withCollection: true, updatedAt: $this->secondsAgo(1)); - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - databaseIds: ['database-lapsed', 'database-live'], - provisioningLease: 60, - ); - - $this->assertSame( - ['Database database-live names no owner and another migration may still be provisioning it; it becomes recoverable after 60 seconds without an update'], - $this->errorMessages($destination), - ); - $this->assertSame('ready', $this->getDatabaseDocument($database, 'database-lapsed')->getAttribute('status')); - $this->assertSame('provisioning', $this->getDatabaseDocument($database, 'database-live')->getAttribute('status')); - } - - public function testZeroProvisioningLeaseRecoversAnOwnerlessDatabaseAtOnce(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createStatusOnlyProjectDatabase($database); - $this->seedLegacyDatabase($database, 'database', 'provisioning', withCollection: true, updatedAt: $this->secondsAgo(1)); - - $destination = $this->runDatabaseTransfer($database, explicit: false, provisioningLease: 0); - - $this->assertSame([], $this->errorMessages($destination)); - $this->assertSame('ready', $this->getDatabaseDocument($database)->getAttribute('status')); - } - - public function testNegativeProvisioningLeaseIsRejected(): void - { - $database = new UtopiaDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); - - $this->expectException(\InvalidArgumentException::class); - $this->expectExceptionMessage('Provisioning lease must not be negative'); - - new AppwriteDestination( - project: 'destination-project', - endpoint: 'http://example.test/v1', - key: 'test-key', - dbForProject: $database, - getDatabasesDB: static fn (UtopiaDocument $document): UtopiaDatabase => $database, - collectionStructure: ['attributes' => [], 'indexes' => []], - dbForPlatform: $database, - projectInternalId: '1', - owner: new ProvisioningOwner('migration-current', 'attempt-current'), - getRecoverableOwner: static fn (UtopiaDocument $document): ?ProvisioningOwner => null, - provisioningLease: -1, - ); - } - - public function testOwnerlessRecoveryYieldsToAnUpdateThatLandsFirst(): void - { - $database = new InterleavingProjectDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); - $this->createStatusOnlyProjectDatabase($database); - $this->seedLegacyDatabase($database, 'database', 'failed', updatedAt: $this->secondsAgo(AppwriteDestination::DEFAULT_PROVISIONING_LEASE + 1)); - $database->onDatabasesReload = static function () use ($database): void { - $database->updateDocument('databases', 'database', new UtopiaDocument([ - 'name' => 'Database claimed elsewhere', - ])); - }; - $database->interceptNextDatabasesReload = true; - - $destination = $this->runDatabaseTransfer($database, explicit: false); - - $claimed = $this->getDatabaseDocument($database); - $this->assertSame( - ['Database database names no owner and was updated before it could be claimed'], - $this->errorMessages($destination), - ); - $this->assertSame('failed', $claimed->getAttribute('status')); - $this->assertSame('Database claimed elsewhere', $claimed->getAttribute('name')); - } - private function createProjectDatabase(bool $withStatus, ?UtopiaDatabase $database = null): UtopiaDatabase { $database ??= new UtopiaDatabase( @@ -1508,8 +323,6 @@ private function createProjectDatabase(bool $withStatus, ?UtopiaDatabase $databa if ($withStatus) { $attributes[] = $this->attribute('status', ColumnType::String, size: 16); - $attributes[] = $this->attribute('migrationId', ColumnType::String, size: UtopiaDatabase::LENGTH_KEY); - $attributes[] = $this->attribute('migrationAttemptId', ColumnType::String, size: UtopiaDatabase::LENGTH_KEY); } $database->createCollection(Collection::create( @@ -1520,49 +333,13 @@ private function createProjectDatabase(bool $withStatus, ?UtopiaDatabase $databa return $database; } - private function seedDatabase( - UtopiaDatabase $database, - string $status, - ?string $migrationId, - ?string $migrationAttemptId, - ): UtopiaDocument { - $document = [ - '$id' => 'database', - 'name' => 'Database', - 'enabled' => true, - 'search' => 'database Database', - 'originalId' => null, - 'type' => 'tablesdb', - 'database' => '', - 'status' => $status, - ]; - if ($migrationId !== null) { - $document['migrationId'] = $migrationId; - } - if ($migrationAttemptId !== null) { - $document['migrationAttemptId'] = $migrationAttemptId; - } - - return $database->getAuthorization()->skip( - static fn (): UtopiaDocument => $database->createDocument('databases', new UtopiaDocument($document)), - ); - } - - private function createStatusOnlyProjectDatabase(UtopiaDatabase $database): UtopiaDatabase - { - $this->createProjectDatabase(withStatus: false, database: $database); - $database->createAttribute('databases', $this->attribute('status', ColumnType::String, size: 16)); - - return $database; - } - - private function seedLegacyDatabase( + private function seedIncompleteDatabase( UtopiaDatabase $database, string $databaseId, string $status, string $name = 'Database', bool $withCollection = false, - string $updatedAt = self::LAPSED_UPDATED_AT, + string $updatedAt = '', ): UtopiaDocument { $seeded = $database->getAuthorization()->skip( static fn (): UtopiaDocument => $database->withPreserveDates( @@ -1610,17 +387,15 @@ private function attribute( ]); } + /** + * @param list $databaseIds + */ private function runDatabaseTransfer( UtopiaDatabase $database, bool $explicit, - string $migrationId = 'migration-current', - string $migrationAttemptId = 'attempt-current', - ?callable $getRecoverableOwner = null, + array $databaseIds = ['database'], OnDuplicate $onDuplicate = OnDuplicate::Fail, string $resourceUpdatedAt = '', - array $databaseIds = ['database'], - bool $success = true, - int $provisioningLease = AppwriteDestination::DEFAULT_PROVISIONING_LEASE, ): CountingAppwriteDestination { $source = new class () extends MockSource { #[Override] @@ -1640,25 +415,11 @@ public function supportsDatabaseStatus(): bool )); } - $destination = new CountingAppwriteDestination( - project: 'destination-project', - endpoint: 'http://example.test/v1', - key: 'test-key', - dbForProject: $database, - getDatabasesDB: static fn (UtopiaDocument $document): UtopiaDatabase => $database, - collectionStructure: ['attributes' => [], 'indexes' => []], - dbForPlatform: $database, - projectInternalId: '1', - owner: new ProvisioningOwner($migrationId, $migrationAttemptId), - getRecoverableOwner: $getRecoverableOwner - ?? static fn (UtopiaDocument $document): ?ProvisioningOwner => null, - onDuplicate: $onDuplicate, - provisioningLease: $provisioningLease, - ); + $destination = $this->createDestination($database, $onDuplicate); $transfer = new Transfer($source, $destination); $database->getAuthorization()->skip( - static function () use ($destination, $explicit, $success, $transfer): void { + static function () use ($explicit, $transfer): void { if ($explicit) { $transfer->runWithResourceSelector( [Resource::TYPE_DATABASE], @@ -1671,7 +432,6 @@ static function (): void { parentResourceInternalId: '', parentResourceType: '', ); - } else { $transfer->run( [Resource::TYPE_DATABASE], @@ -1679,25 +439,25 @@ static function (): void { }, ); } - - if ($success) { - $destination->success(); - } }, ); return $destination; } - private function finalizeExpectingFailure(UtopiaDatabase $database, AppwriteDestination $destination): Finalization + private function createDestination(UtopiaDatabase $database, OnDuplicate $onDuplicate): CountingAppwriteDestination { - try { - $database->getAuthorization()->skip($destination->success(...)); - } catch (Finalization $failure) { - return $failure; - } - - $this->fail('A failed finalization must throw'); + return new CountingAppwriteDestination( + project: 'destination-project', + endpoint: 'http://example.test/v1', + key: 'test-key', + dbForProject: $database, + getDatabasesDB: static fn (UtopiaDocument $document): UtopiaDatabase => $database, + collectionStructure: ['attributes' => [], 'indexes' => []], + dbForPlatform: $database, + projectInternalId: '1', + onDuplicate: $onDuplicate, + ); } private function getDatabaseDocument(UtopiaDatabase $database, string $databaseId = 'database'): UtopiaDocument diff --git a/tests/Migration/Unit/Destinations/AppwriteDestinationDsnTest.php b/tests/Migration/Unit/Destinations/AppwriteDestinationDsnTest.php index 1b67b193..00673615 100644 --- a/tests/Migration/Unit/Destinations/AppwriteDestinationDsnTest.php +++ b/tests/Migration/Unit/Destinations/AppwriteDestinationDsnTest.php @@ -13,7 +13,6 @@ use Utopia\Database\Database as UtopiaDatabase; use Utopia\Database\Document as UtopiaDocument; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; use Utopia\Migration\Destinations\OnDuplicate; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Database\Database as DatabaseResource; @@ -97,8 +96,6 @@ private function transferDatabase(?callable $getDatabaseDSN): UtopiaDatabase collectionStructure: ['attributes' => [], 'indexes' => []], dbForPlatform: $database, projectInternalId: '1', - owner: new ProvisioningOwner('migration-test', 'attempt-test'), - getRecoverableOwner: static fn (UtopiaDocument $document): ?ProvisioningOwner => null, onDuplicate: OnDuplicate::Fail, getDatabaseDSN: $getDatabaseDSN, ); diff --git a/tests/Migration/Unit/Destinations/AppwriteIndexLengthsTest.php b/tests/Migration/Unit/Destinations/AppwriteIndexLengthsTest.php index 0f19fd3b..49433c06 100644 --- a/tests/Migration/Unit/Destinations/AppwriteIndexLengthsTest.php +++ b/tests/Migration/Unit/Destinations/AppwriteIndexLengthsTest.php @@ -13,7 +13,6 @@ use Utopia\Database\Database as UtopiaDatabase; use Utopia\Database\Document as UtopiaDocument; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; use Utopia\Migration\Destinations\OnDuplicate; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Database\Columns\Text; @@ -206,8 +205,6 @@ private function transferIndex( ], dbForPlatform: $database, projectInternalId: '1', - owner: new ProvisioningOwner('migration-test', 'attempt-test'), - getRecoverableOwner: static fn (UtopiaDocument $document): ?ProvisioningOwner => null, onDuplicate: $onDuplicate, ); diff --git a/tests/Migration/Unit/Destinations/AppwriteRelationshipOnDeleteTest.php b/tests/Migration/Unit/Destinations/AppwriteRelationshipOnDeleteTest.php index d8fb6c3c..9174b83c 100644 --- a/tests/Migration/Unit/Destinations/AppwriteRelationshipOnDeleteTest.php +++ b/tests/Migration/Unit/Destinations/AppwriteRelationshipOnDeleteTest.php @@ -17,7 +17,6 @@ use Utopia\Database\RelationshipDeleteAction; use Utopia\Database\RelationshipType; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; use Utopia\Migration\Destinations\OnDuplicate; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Database\Columns\Relationship; @@ -212,8 +211,6 @@ private function transfer(UtopiaDatabase $database, OnDuplicate $onDuplicate, Re ], dbForPlatform: $database, projectInternalId: '1', - owner: new ProvisioningOwner('migration-test', 'attempt-test'), - getRecoverableOwner: static fn (UtopiaDocument $document): ?ProvisioningOwner => null, onDuplicate: $onDuplicate, ); diff --git a/tests/Migration/Unit/Destinations/AppwriteRowImportTest.php b/tests/Migration/Unit/Destinations/AppwriteRowImportTest.php index 70801643..e5edb58a 100644 --- a/tests/Migration/Unit/Destinations/AppwriteRowImportTest.php +++ b/tests/Migration/Unit/Destinations/AppwriteRowImportTest.php @@ -13,7 +13,6 @@ use Utopia\Database\Document as UtopiaDocument; use Utopia\Database\Query; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Database\Columns\Text; use Utopia\Migration\Resources\Database\Database as DatabaseResource; @@ -206,8 +205,6 @@ private function destination(): AppwriteDestination ], dbForPlatform: $database, projectInternalId: '1', - owner: new ProvisioningOwner('migration-row-import', 'attempt-current'), - getRecoverableOwner: static fn (UtopiaDocument $document): ?ProvisioningOwner => null, ); } @@ -232,8 +229,6 @@ private function projectDatabase(): UtopiaDatabase $this->attribute('type', ColumnType::String, default: 'tablesdb', size: 128), $this->attribute('database', ColumnType::String, size: 2000), $this->attribute('status', ColumnType::String, size: 16), - $this->attribute('migrationId', ColumnType::String, size: UtopiaDatabase::LENGTH_KEY), - $this->attribute('migrationAttemptId', ColumnType::String, size: UtopiaDatabase::LENGTH_KEY), ], )); diff --git a/tests/Migration/Unit/Destinations/AppwriteFinalizationTest.php b/tests/Migration/Unit/Destinations/AppwriteRunCompletionTest.php similarity index 55% rename from tests/Migration/Unit/Destinations/AppwriteFinalizationTest.php rename to tests/Migration/Unit/Destinations/AppwriteRunCompletionTest.php index b771c044..c4e5eeb4 100644 --- a/tests/Migration/Unit/Destinations/AppwriteFinalizationTest.php +++ b/tests/Migration/Unit/Destinations/AppwriteRunCompletionTest.php @@ -4,7 +4,6 @@ use Override; use PHPUnit\Framework\TestCase; -use Throwable; use Utopia\Cache\Adapter\Memory as MemoryCache; use Utopia\Cache\Cache; use Utopia\Database\Adapter\Memory as MemoryAdapter; @@ -12,13 +11,11 @@ use Utopia\Database\Collection; use Utopia\Database\Database as UtopiaDatabase; use Utopia\Database\Document as UtopiaDocument; +use Utopia\Database\Exception as DatabaseException; use Utopia\Database\Query; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; use Utopia\Migration\Destinations\OnDuplicate; -use Utopia\Migration\Exception\Aborted; use Utopia\Migration\Exception as MigrationException; -use Utopia\Migration\Exception\Finalization; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Database\Columns\Text; use Utopia\Migration\Resources\Database\Database as DatabaseResource; @@ -27,13 +24,13 @@ use Utopia\Query\Schema\ColumnType; use Utopia\Tests\Unit\Adapters\MockSource; -final class AppwriteFinalizationTest extends TestCase +final class AppwriteRunCompletionTest extends TestCase { private const SOURCE_OLDER = '2000-01-01 00:00:00'; private const SOURCE_NEWER = '2999-01-01 00:00:00'; - public function testSkippedFinalizationIsReportedWhenTheLifecycleEnds(): void + public function testRunMarksItsDatabasesReady(): void { $database = $this->projectDatabase(); $destination = $this->destination($database); @@ -43,70 +40,12 @@ public function testSkippedFinalizationIsReportedWhenTheLifecycleEnds(): void $this->databaseResource('second'), ]); - $this->assertSame([], $destination->getErrors(), 'A run awaiting success() must not fail the caller\'s error gate'); - - $destination->cleanUp(); - - $this->assertSame( - [ - [Resource::TYPE_DATABASE, Transfer::GROUP_DATABASES, 'first'], - [Resource::TYPE_DATABASE, Transfer::GROUP_DATABASES, 'second'], - ], - $this->subjects($destination->getErrors()), - 'Ending the lifecycle without success() must report every database it left unfinalized', - ); - $this->assertStringContainsString('success()', $destination->getErrors()[0]->getMessage()); - $this->assertSame('provisioning', $this->databaseStatus($database, 'first')); - $this->assertSame('provisioning', $this->databaseStatus($database, 'second')); - } - - public function testFinalizedRunReportsNothingWhenTheLifecycleEnds(): void - { - $database = $this->projectDatabase(); - $destination = $this->destination($database); - $this->import($database, $destination, [$this->databaseResource('first')]); - - $this->assertNull($this->finalize($database, $destination)); - $destination->cleanUp(); - - $this->assertSame([], $destination->getErrors()); - $this->assertSame('ready', $this->databaseStatus($database, 'first')); - } - - public function testAbandonedRunReportsNothingWhenTheLifecycleEnds(): void - { - $database = $this->projectDatabase(); - $destination = $this->destination($database); - $this->import($database, $destination, [$this->databaseResource('first')]); - - $destination->error(); - $destination->cleanUp(); - $this->assertSame([], $destination->getErrors()); - $this->assertSame('provisioning', $this->databaseStatus($database, 'first')); + $this->assertSame('ready', $this->databaseStatus($database, 'first'), 'run() must flip every database it provisioned to ready'); + $this->assertSame('ready', $this->databaseStatus($database, 'second'), 'run() must flip every database it provisioned to ready'); } - public function testRerunWithoutFinalizationReportsTheDiscardedRun(): void - { - $database = $this->projectDatabase(); - $destination = $this->destination($database); - - $this->import($database, $destination, [$this->databaseResource('first')]); - $this->import($database, $destination, [$this->databaseResource('second')]); - - $this->assertSame( - [[Resource::TYPE_DATABASE, Transfer::GROUP_DATABASES, 'first']], - $this->subjects($destination->getErrors()), - 'A second run must not silently discard the first run\'s unfinalized databases', - ); - - $this->assertNull($this->finalize($database, $destination)); - - $this->assertSame('provisioning', $this->databaseStatus($database, 'first')); - $this->assertSame('ready', $this->databaseStatus($database, 'second')); - } - - public function testSuccessDoesNotFinalizeAnAbortedRun(): void + public function testInterruptedRunLeavesItsDatabasesProvisioning(): void { $database = $this->projectDatabase(); $destination = $this->destination($database); @@ -117,178 +56,60 @@ public function testSuccessDoesNotFinalizeAnAbortedRun(): void $destination, [$this->databaseResource('first')], static function (): void { - throw new \RuntimeException('transfer aborted'); + throw new TransferInterrupted('transfer interrupted'); }, ); - $this->fail('The aborted transfer must throw'); - } catch (\RuntimeException $error) { - $this->assertSame('transfer aborted', $error->getMessage()); + $this->fail('The interrupted transfer must throw'); + } catch (TransferInterrupted $error) { + $this->assertSame('transfer interrupted', $error->getMessage()); } - $this->assertNull($this->finalize($database, $destination)); - $destination->cleanUp(); - $this->assertSame('provisioning', $this->databaseStatus($database, 'first'), 'An interrupted run must never be marked ready'); $this->assertSame([], $destination->getErrors()); } - public function testSuccessFinalizesNothingWhenTheSourceSwallowedTheAbort(): void - { - $database = $this->projectDatabase(); - $this->seedTablesWithLegacyColumns($database); - - $abort = new Aborted('Migration attempt was superseded'); - $destination = $this->destination($database, 'attempt-overwrite', OnDuplicate::Overwrite); - - try { - $this->importSwallowingAbort($database, $destination, $this->newerSchemaWithoutLegacyColumns(), $abort); - $this->fail('The swallowed abort must still end the transfer'); - } catch (Aborted $caught) { - $this->assertSame($abort, $caught); - } - - $this->assertNull($this->finalize($database, $destination)); - - $this->assertSame('provisioning', $this->databaseStatus($database, 'first'), 'An aborted run must never be marked ready'); - $this->assertSame('provisioning', $this->databaseStatus($database, 'second'), 'An aborted run must never be marked ready'); - $this->assertSame(['legacy', 'name'], $this->columnKeys($database, 'first', 'first-items'), 'An aborted run must not sweep its overwrite orphans'); - $this->assertSame(['legacy', 'name'], $this->columnKeys($database, 'second', 'second-items'), 'An aborted run must not sweep its overwrite orphans'); - - $errors = $destination->getErrors(); - $destination->cleanUp(); - - $this->assertSame($errors, $destination->getErrors(), 'An aborted run is not reported as a skipped finalization'); - } - - public function testFailedFinalizationThrowsAfterAttemptingEveryDatabase(): void - { - $database = $this->projectDatabase(); - $database->failReadyWrites = ['first', 'third']; - $destination = $this->destination($database); - $this->import($database, $destination, [ - $this->databaseResource('first'), - $this->databaseResource('second'), - $this->databaseResource('third'), - ]); - - $failure = $this->finalize($database, $destination); - - $this->assertSame('provisioning', $this->databaseStatus($database, 'first')); - $this->assertSame('ready', $this->databaseStatus($database, 'second')); - $this->assertSame('provisioning', $this->databaseStatus($database, 'third')); - $this->assertNotNull($failure, 'A failed finalization must throw, not only append errors'); - $this->assertInstanceOf(Finalization::class, $failure); - $this->assertSame( - [ - [Resource::TYPE_DATABASE, Transfer::GROUP_DATABASES, 'first'], - [Resource::TYPE_DATABASE, Transfer::GROUP_DATABASES, 'third'], - ], - $this->subjects($failure->failures), - ); - $this->assertSame('ready status unavailable', $failure->failures[0]->getMessage()); - $this->assertSame($failure->failures, $destination->getErrors(), 'Each failure is also recorded for the report'); - $this->assertSame($failure->failures[0], $failure->getPrevious()); - $this->assertSame(MigrationException::CODE_INTERNAL, $failure->getCode()); - $this->assertStringContainsString('first', $failure->getMessage()); - $this->assertStringContainsString('third', $failure->getMessage()); - - $destination->cleanUp(); - - $this->assertCount(2, $destination->getErrors(), 'An attempted finalization is not reported again as skipped'); - } - public function testOneFailedReadyFlipStillSweepsEveryOverwrittenTable(): void { $database = $this->projectDatabase(); $this->seedTablesWithLegacyColumns($database); $database->failReadyWrites = ['first']; - $destination = $this->destination($database, 'attempt-overwrite', OnDuplicate::Overwrite); + $destination = $this->destination($database, OnDuplicate::Overwrite); $this->import($database, $destination, $this->newerSchemaWithoutLegacyColumns()); - $failure = $this->finalize($database, $destination); - $this->assertSame(['name'], $this->columnKeys($database, 'first', 'first-items'), 'The sweep must run for the database whose flip failed'); $this->assertSame(['name'], $this->columnKeys($database, 'second', 'second-items'), 'The sweep must run for every other database'); $this->assertSame('provisioning', $this->databaseStatus($database, 'first')); $this->assertSame('ready', $this->databaseStatus($database, 'second')); - $this->assertInstanceOf(Finalization::class, $failure); $this->assertSame( [[Resource::TYPE_DATABASE, Transfer::GROUP_DATABASES, 'first']], - $this->subjects($failure->failures), + $this->subjects($destination->getErrors()), ); + $this->assertSame('ready status unavailable', $destination->getErrors()[0]->getMessage()); } - public function testOrphanSweepFailureStillSweepsTheOtherTablesAndThrows(): void + public function testOrphanSweepFailureThrowsAfterEveryDatabaseIsReady(): void { $database = $this->projectDatabase(); $this->seedTablesWithLegacyColumns($database); - - $destination = $this->destination($database, 'attempt-overwrite', OnDuplicate::Overwrite); - $this->import($database, $destination, $this->newerSchemaWithoutLegacyColumns()); $database->failAttributeScanOf = $this->table($database, 'first', 'first-items'); - $failure = $this->finalize($database, $destination); + $destination = $this->destination($database, OnDuplicate::Overwrite); + + try { + $this->import($database, $destination, $this->newerSchemaWithoutLegacyColumns()); + $this->fail('A failed sweep must fail the run'); + } catch (DatabaseException $error) { + $this->assertSame('attribute scan unavailable', $error->getMessage()); + } $database->failAttributeScanOf = null; $this->assertSame(['legacy', 'name'], $this->columnKeys($database, 'first', 'first-items')); - $this->assertSame(['name'], $this->columnKeys($database, 'second', 'second-items'), 'One table\'s failed sweep must not skip the others'); $this->assertSame('ready', $this->databaseStatus($database, 'first'), 'Databases are flipped before any sweep can fail'); - $this->assertSame('ready', $this->databaseStatus($database, 'second')); - $this->assertInstanceOf(Finalization::class, $failure, 'A failed sweep must surface as the finalization exception'); - $this->assertSame( - [[Resource::TYPE_TABLE, Transfer::GROUP_DATABASES, 'first-items']], - $this->subjects($failure->failures), - ); - $this->assertSame('attribute scan unavailable', $failure->failures[0]->getMessage()); - $this->assertSame($failure->failures, $destination->getErrors()); - } - - /** - * A source that records the export failure and carries on, the way the library sources did - * before they rethrew aborts. - * - * @param list $resources - */ - private function importSwallowingAbort( - FailingFinalizationDatabase $database, - AppwriteDestination $destination, - array $resources, - Aborted $abort, - ): void { - $source = new class () extends MockSource { - #[Override] - public function supportsDatabaseStatus(): bool - { - return true; - } - - #[Override] - protected function exportGroupDatabases(int $batchSize, array $resources): void - { - foreach ($resources as $resource) { - try { - parent::exportGroupDatabases($batchSize, [$resource]); - } catch (Throwable $error) { - $this->addError(new MigrationException($resource, Transfer::GROUP_DATABASES, message: $error->getMessage(), previous: $error)); - } - } - } - }; - - $types = []; - foreach ($resources as $resource) { - $source->pushMockResource($resource); - $types[$resource->getName()] = true; - } - - $transfer = new Transfer($source, $destination); - $database->getAuthorization()->skip( - static fn () => $transfer->run(\array_keys($types), static fn () => throw $abort), - ); + $this->assertSame('ready', $this->databaseStatus($database, 'second'), 'Databases are flipped before any sweep can fail'); } - private function seedTablesWithLegacyColumns(FailingFinalizationDatabase $database): void + private function seedTablesWithLegacyColumns(FailingCompletionDatabase $database): void { $resources = []; foreach (['first', 'second'] as $databaseId) { @@ -303,10 +124,9 @@ private function seedTablesWithLegacyColumns(FailingFinalizationDatabase $databa ); } - $destination = $this->destination($database, 'attempt-seed'); + $destination = $this->destination($database); $this->import($database, $destination, $resources); - $this->assertNull($this->finalize($database, $destination)); $this->assertSame([], $destination->getErrors()); $this->assertSame(['legacy', 'name'], $this->columnKeys($database, 'first', 'first-items')); $this->assertSame(['legacy', 'name'], $this->columnKeys($database, 'second', 'second-items')); @@ -347,8 +167,7 @@ private function databaseResource(string $id, string $name = 'Database', string } private function destination( - FailingFinalizationDatabase $database, - string $attemptId = 'attempt-current', + FailingCompletionDatabase $database, OnDuplicate $onDuplicate = OnDuplicate::Fail, ): AppwriteDestination { return new AppwriteDestination( @@ -370,8 +189,6 @@ private function destination( ], dbForPlatform: $database, projectInternalId: '1', - owner: new ProvisioningOwner('migration-finalization', $attemptId), - getRecoverableOwner: static fn (UtopiaDocument $document): ?ProvisioningOwner => null, onDuplicate: $onDuplicate, ); } @@ -380,7 +197,7 @@ private function destination( * @param list $resources */ private function import( - FailingFinalizationDatabase $database, + FailingCompletionDatabase $database, AppwriteDestination $destination, array $resources, ?callable $callback = null, @@ -405,25 +222,14 @@ public function supportsDatabaseStatus(): bool ); } - private function finalize(FailingFinalizationDatabase $database, AppwriteDestination $destination): ?\Throwable - { - try { - $database->getAuthorization()->skip($destination->success(...)); - } catch (\Throwable $failure) { - return $failure; - } - - return null; - } - - private function databaseStatus(FailingFinalizationDatabase $database, string $databaseId): mixed + private function databaseStatus(FailingCompletionDatabase $database, string $databaseId): mixed { return $database->getAuthorization()->skip( static fn (): UtopiaDocument => $database->getDocument('databases', $databaseId), )->getAttribute('status'); } - private function table(FailingFinalizationDatabase $database, string $databaseId, string $tableId): UtopiaDocument + private function table(FailingCompletionDatabase $database, string $databaseId, string $tableId): UtopiaDocument { return $database->getAuthorization()->skip(static function () use ($database, $databaseId, $tableId): UtopiaDocument { $metadata = $database->getDocument('databases', $databaseId); @@ -435,7 +241,7 @@ private function table(FailingFinalizationDatabase $database, string $databaseId /** * @return list */ - private function columnKeys(FailingFinalizationDatabase $database, string $databaseId, string $tableId): array + private function columnKeys(FailingCompletionDatabase $database, string $databaseId, string $tableId): array { $table = $this->table($database, $databaseId, $tableId); $columns = $database->getAuthorization()->skip(static fn (): array => $database->find('attributes', [ @@ -464,9 +270,9 @@ private function subjects(array $errors): array )); } - private function projectDatabase(): FailingFinalizationDatabase + private function projectDatabase(): FailingCompletionDatabase { - $database = new FailingFinalizationDatabase( + $database = new FailingCompletionDatabase( new MemoryAdapter(), new Cache(new MemoryCache()), ); @@ -485,8 +291,6 @@ private function projectDatabase(): FailingFinalizationDatabase $this->attribute('type', ColumnType::String, default: 'tablesdb', size: 128), $this->attribute('database', ColumnType::String, size: 2000), $this->attribute('status', ColumnType::String, size: 16), - $this->attribute('migrationId', ColumnType::String, size: UtopiaDatabase::LENGTH_KEY), - $this->attribute('migrationAttemptId', ColumnType::String, size: UtopiaDatabase::LENGTH_KEY), ], )); diff --git a/tests/Migration/Unit/Destinations/AppwriteSpatialIndexTest.php b/tests/Migration/Unit/Destinations/AppwriteSpatialIndexTest.php index 143c0fb4..4d7cbca8 100644 --- a/tests/Migration/Unit/Destinations/AppwriteSpatialIndexTest.php +++ b/tests/Migration/Unit/Destinations/AppwriteSpatialIndexTest.php @@ -18,7 +18,6 @@ use Utopia\Database\Index as UtopiaIndex; use Utopia\Database\Query; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Database\Columns\Point; use Utopia\Migration\Resources\Database\Database as DatabaseResource; @@ -134,8 +133,6 @@ private function transferSpatialIndex(): array ], dbForPlatform: $project, projectInternalId: '1', - owner: new ProvisioningOwner('migration-test', 'attempt-test'), - getRecoverableOwner: static fn (UtopiaDocument $document): ?ProvisioningOwner => null, ); $transfer = new Transfer($source, $destination); diff --git a/tests/Migration/Unit/Destinations/FailingFinalizationDatabase.php b/tests/Migration/Unit/Destinations/FailingCompletionDatabase.php similarity index 94% rename from tests/Migration/Unit/Destinations/FailingFinalizationDatabase.php rename to tests/Migration/Unit/Destinations/FailingCompletionDatabase.php index 86b17ca8..345cd3c4 100644 --- a/tests/Migration/Unit/Destinations/FailingFinalizationDatabase.php +++ b/tests/Migration/Unit/Destinations/FailingCompletionDatabase.php @@ -10,11 +10,11 @@ use Utopia\Database\Query; /** - * A project database whose finalization steps can be made to fail: the + * A project database whose end-of-run steps can be made to fail: the * `ready` write of chosen databases, and the attribute scan the overwrite * sweep runs for one chosen table. */ -final class FailingFinalizationDatabase extends UtopiaDatabase +final class FailingCompletionDatabase extends UtopiaDatabase { /** @var list */ public array $failReadyWrites = []; diff --git a/tests/Migration/Unit/Destinations/ProvisioningOwnerTest.php b/tests/Migration/Unit/Destinations/ProvisioningOwnerTest.php deleted file mode 100644 index 54100d30..00000000 --- a/tests/Migration/Unit/Destinations/ProvisioningOwnerTest.php +++ /dev/null @@ -1,36 +0,0 @@ - */ - public static function emptyIdentifiers(): array - { - return [ - 'migration identifier' => ['', 'attempt'], - 'attempt identifier' => ['migration', ''], - ]; - } - - #[DataProvider('emptyIdentifiers')] - public function testIdentifiersMustBeNonEmpty(string $migrationId, string $attemptId): void - { - $this->expectException(\InvalidArgumentException::class); - - new ProvisioningOwner($migrationId, $attemptId); - } - - public function testEqualityRequiresTheExactPair(): void - { - $owner = new ProvisioningOwner('migration', 'attempt'); - - $this->assertTrue($owner->equals(new ProvisioningOwner('migration', 'attempt'))); - $this->assertFalse($owner->equals(new ProvisioningOwner('migration-other', 'attempt'))); - $this->assertFalse($owner->equals(new ProvisioningOwner('migration', 'attempt-other'))); - } -} diff --git a/tests/Migration/Unit/Destinations/TransferInterrupted.php b/tests/Migration/Unit/Destinations/TransferInterrupted.php new file mode 100644 index 00000000..169be520 --- /dev/null +++ b/tests/Migration/Unit/Destinations/TransferInterrupted.php @@ -0,0 +1,7 @@ +destination($database, $onDuplicate); $transfer = new Transfer($source, $destination); $database->getAuthorization()->skip( - static function () use ($transfer): void { - $noop = static function (): void { - }; - $transfer->run([Resource::TYPE_DATABASE, Resource::TYPE_TABLE, Resource::TYPE_COLUMN], $noop); - $transfer->run([Resource::TYPE_INDEX], $noop); - }, + static fn () => $transfer->run( + [Resource::TYPE_DATABASE, Resource::TYPE_TABLE, Resource::TYPE_COLUMN, Resource::TYPE_INDEX], + static function (): void { + }, + ), ); return [$destination, $resources]; @@ -163,8 +171,6 @@ private function destination(UtopiaDatabase $database, OnDuplicate $onDuplicate) ], dbForPlatform: $database, projectInternalId: '1', - owner: new ProvisioningOwner('migration-test', 'attempt-test'), - getRecoverableOwner: static fn (UtopiaDocument $document): ?ProvisioningOwner => null, onDuplicate: $onDuplicate, ); } diff --git a/tests/Migration/Unit/General/TransferAbortTest.php b/tests/Migration/Unit/General/TransferAbortTest.php deleted file mode 100644 index 97c65ea7..00000000 --- a/tests/Migration/Unit/General/TransferAbortTest.php +++ /dev/null @@ -1,163 +0,0 @@ -swallowingSource(); - $transfer = new Transfer($source, new MockDestination()); - $calls = 0; - - try { - $run($transfer, static function () use ($abort, &$calls): never { - $calls++; - - throw $abort; - }); - $this->fail('The abort did not stop the transfer.'); - } catch (Aborted $caught) { - $this->assertSame($abort, $caught); - } - - $this->assertSame(1, $calls, 'The callback must not run again once it aborted.'); - } - - /** - * @return array - */ - public static function runProvider(): array - { - return [ - 'run' => [ - static fn (Transfer $transfer, callable $callback) => $transfer->run( - [Resource::TYPE_USER, Resource::TYPE_DATABASE, Resource::TYPE_TABLE], - $callback, - ), - ], - 'resource selector' => [ - static fn (Transfer $transfer, callable $callback) => $transfer->runWithResourceSelector( - [Resource::TYPE_DATABASE, Resource::TYPE_TABLE], - $callback, - resourceId: 'table', - resourceInternalId: '2', - resourceType: Resource::TYPE_TABLE, - parentResourceId: 'database', - parentResourceInternalId: '1', - parentResourceType: Resource::TYPE_DATABASE, - ), - ], - ]; - } - - public function testAbortDoesNotCarryOverToTheNextRun(): void - { - $abort = new Aborted('Stopped'); - $source = $this->swallowingSource(); - $destination = new MockDestination(); - $transfer = new Transfer($source, $destination); - - try { - $transfer->run([Resource::TYPE_USER], static fn () => throw $abort); - $this->fail('The abort did not stop the transfer.'); - } catch (Aborted $caught) { - $this->assertSame($abort, $caught); - } - - $transfer->run([Resource::TYPE_DATABASE], static function (): void { - }); - - $this->assertSame(['database'], $destination->getResourceTypeData(Transfer::GROUP_DATABASES, Resource::TYPE_DATABASE)); - } - - public function testALatchedAbortTellsTheDestinationTheRunWasAborted(): void - { - $abort = new Aborted('Stopped'); - $destination = new class () extends MockDestination { - public int $aborts = 0; - - #[Override] - public function markAborted(): void - { - $this->aborts++; - } - }; - $transfer = new Transfer($this->swallowingSource(), $destination); - - try { - $transfer->run([Resource::TYPE_USER], static fn () => throw $abort); - $this->fail('The abort did not stop the transfer.'); - } catch (Aborted $caught) { - $this->assertSame($abort, $caught); - } - - $this->assertSame(1, $destination->aborts, 'A latched abort must tell the destination the run was aborted.'); - - $transfer->run([Resource::TYPE_DATABASE], static function (): void { - }); - - $this->assertSame(1, $destination->aborts, 'A run that completed must not be reported as aborted.'); - } - - /** - * A source that records every failure of a resource type and moves on, the way the - * library sources did before they rethrew aborts. - */ - private function swallowingSource(): MockSource - { - $source = new class () extends MockSource { - #[Override] - protected function exportGroupAuth(int $batchSize, array $resources): void - { - foreach ($resources as $resource) { - try { - parent::exportGroupAuth($batchSize, [$resource]); - } catch (Throwable $error) { - $this->addError(new Exception($resource, Transfer::GROUP_AUTH, message: $error->getMessage(), previous: $error)); - } - } - } - - #[Override] - protected function exportGroupDatabases(int $batchSize, array $resources): void - { - foreach ($resources as $resource) { - try { - parent::exportGroupDatabases($batchSize, [$resource]); - } catch (Throwable $error) { - $this->addError(new Exception($resource, Transfer::GROUP_DATABASES, message: $error->getMessage(), previous: $error)); - } - } - } - }; - - $database = new Database('database', 'Database'); - $source->pushMockResource(new User('user', 'user@example.com')); - $source->pushMockResource($database); - $source->pushMockResource(new Table($database, 'Table', 'table')); - - return $source; - } -} diff --git a/tests/Migration/Unit/MigrationCLITest.php b/tests/Migration/Unit/MigrationCLITest.php index 7b6aa9c1..62521d04 100644 --- a/tests/Migration/Unit/MigrationCLITest.php +++ b/tests/Migration/Unit/MigrationCLITest.php @@ -6,16 +6,15 @@ use Override; use PHPUnit\Framework\Attributes\BackupGlobals; +use PHPUnit\Framework\Attributes\RunInSeparateProcess; use PHPUnit\Framework\TestCase; use Utopia\Cache\Adapter\Memory as MemoryCache; use Utopia\Cache\Cache; use Utopia\Database\Adapter\Memory as MemoryAdapter; use Utopia\Database\Attribute; -use Utopia\Database\Capability; use Utopia\Database\Collection; use Utopia\Database\Database; use Utopia\Database\Document; -use Utopia\Migration\Destination; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Database\Database as DatabaseResource; use Utopia\Migration\Source; @@ -25,13 +24,10 @@ final class TestMigrationCLI extends \MigrationCLI { - /** @param list $arguments */ public function __construct( - array $arguments, private readonly Database $database, private readonly ?Source $injectedSource = null, ) { - parent::__construct($arguments); } #[Override] @@ -65,116 +61,12 @@ public function getErrors(): array } } -final class TransactionalMemoryAdapter extends MemoryAdapter -{ - /** @return array */ - #[Override] - public function capabilities(): array - { - return [...parent::capabilities(), Capability::TransactionRetries]; - } -} - #[BackupGlobals(true)] final class MigrationCLITest extends TestCase { - private const array FILTER_STATE = ['filters', 'defaultFiltersRegistered']; - - /** @var array */ - private array $filterState = []; - - public function testHelpDocumentsTheAcceptedRecoveryOptionsAndNotTheRetiredOne(): void - { - // The neighbouring test proves --recover-provisioning is refused; this is the - // separate property that it is not advertised, so nobody is invited to try it. - $this->assertStringContainsString('--recover-migration-id=', \MigrationCLI::getHelp()); - $this->assertStringContainsString('--recover-migration-attempt-id=', \MigrationCLI::getHelp()); - $this->assertStringNotContainsString('--recover-provisioning', \MigrationCLI::getHelp()); - $this->assertStringContainsString('--migration-id=', \MigrationCLI::getHelp()); - $this->assertStringContainsString('--migration-attempt-id=', \MigrationCLI::getHelp()); - } - - public function testIncompleteDatabaseRecoveryRequiresExactTerminalMigrationIdentifier(): void - { - $cases = [ - 'absent' => [[], false], - 'bare migration' => [['--recover-migration-id', '--recover-migration-attempt-id=attempt-terminal'], false], - 'bare attempt' => [['--recover-migration-id=migration-terminal', '--recover-migration-attempt-id'], false], - 'empty migration' => [['--recover-migration-id=', '--recover-migration-attempt-id=attempt-terminal'], false], - 'empty attempt' => [['--recover-migration-id=migration-terminal', '--recover-migration-attempt-id='], false], - 'migration only' => [['--recover-migration-id=migration-terminal'], false], - 'attempt only' => [['--recover-migration-attempt-id=attempt-terminal'], false], - 'migration mismatch' => [['--recover-migration-id=migration-other', '--recover-migration-attempt-id=attempt-terminal'], false], - 'attempt mismatch' => [['--recover-migration-id=migration-terminal', '--recover-migration-attempt-id=attempt-other'], false], - 'retired unsafe option' => [['--recover-provisioning'], false], - 'exact' => [[ - '--recover-migration-id=migration-terminal', - '--recover-migration-attempt-id=attempt-terminal', - ], true], - ]; - - foreach (['provisioning', 'failed'] as $status) { - foreach ($cases as [$recoveryArguments, $recover]) { - $database = $this->createProjectDatabase($status); - $arguments = [ - 'MigrationCLI.php', - '--migration-id=migration-current', - '--migration-attempt-id=attempt-current', - ...$recoveryArguments, - ]; - $cli = new TestMigrationCLI($arguments, $database); - - $destination = $cli->getDestination(); - $this->runTransfer($database, $destination); - - $created = $database->getAuthorization()->skip( - static fn (): Document => $database->getDocument('databases', 'database'), - ); - - if (! $recover) { - $this->assertNotSame([], $destination->getErrors()); - $this->assertSame($status, $created->getAttribute('status')); - $this->assertSame('migration-terminal', $created->getAttribute('migrationId')); - $this->assertSame('attempt-terminal', $created->getAttribute('migrationAttemptId')); - $this->assertNull($database->findCollection('database_'.$created->getSequence())); - continue; - } - - $this->assertSame([], $destination->getErrors()); - $this->assertSame('ready', $created->getAttribute('status')); - $this->assertSame('migration-current', $created->getAttribute('migrationId')); - $this->assertSame('attempt-current', $created->getAttribute('migrationAttemptId')); - $this->assertNotNull($database->findCollection('database_'.$created->getSequence())); - } - } - } - - public function testAppwriteDestinationRequiresMigrationIdentifier(): void - { - $cli = new TestMigrationCLI(['MigrationCLI.php'], $this->createProjectDatabase()); - - $this->expectException(\InvalidArgumentException::class); - $this->expectExceptionMessage('--migration-id is required for an Appwrite destination'); - - $cli->getDestination(); - } - - public function testAppwriteDestinationRequiresMigrationAttemptIdentifier(): void - { - $cli = new TestMigrationCLI( - ['MigrationCLI.php', '--migration-id=migration-current'], - $this->createProjectDatabase(), - ); - - $this->expectException(\InvalidArgumentException::class); - $this->expectExceptionMessage('--migration-attempt-id is required for an Appwrite destination'); - - $cli->getDestination(); - } - public function testStartFinalizesSuccessfulAppwriteDatabase(): void { - $database = $this->createProjectDatabase(status: null); + $database = $this->createProjectDatabase(); $source = $this->createSource(new DatabaseResource( id: 'database', name: 'Database', @@ -182,29 +74,19 @@ public function testStartFinalizesSuccessfulAppwriteDatabase(): void database: 'source-dsn', databaseStatus: 'ready', )); - $cli = new TestMigrationCLI( - [ - 'MigrationCLI.php', - '--migration-id=migration-current', - '--migration-attempt-id=attempt-current', - ], - $database, - $source, - ); + $cli = new TestMigrationCLI($database, $source); $cli->start(); $created = $database->getDocument('databases', 'database'); $this->assertSame([], $cli->getErrors()); $this->assertSame('ready', $created->getAttribute('status')); - $this->assertSame('migration-current', $created->getAttribute('migrationId')); - $this->assertSame('attempt-current', $created->getAttribute('migrationAttemptId')); $this->assertNotNull($database->findCollection('database_'.$created->getSequence())); } public function testStartFinalizesSuccessfulResourcesWhenDestinationHasErrors(): void { - $database = $this->createProjectDatabase(status: null); + $database = $this->createProjectDatabase(); $valid = new DatabaseResource( id: 'database', name: 'Database', @@ -219,15 +101,7 @@ public function testStartFinalizesSuccessfulResourcesWhenDestinationHasErrors(): database: 'source-dsn', databaseStatus: 'ready', ); - $cli = new TestMigrationCLI( - [ - 'MigrationCLI.php', - '--migration-id=migration-current', - '--migration-attempt-id=attempt-current', - ], - $database, - $this->createSource($valid, $invalid), - ); + $cli = new TestMigrationCLI($database, $this->createSource($valid, $invalid)); $cli->start(); @@ -239,11 +113,10 @@ public function testStartFinalizesSuccessfulResourcesWhenDestinationHasErrors(): $this->assertSame(Resource::STATUS_SUCCESS, $valid->getStatus()); $this->assertSame(Resource::STATUS_ERROR, $invalid->getStatus()); $this->assertSame('ready', $created->getAttribute('status')); - $this->assertSame('migration-current', $created->getAttribute('migrationId')); - $this->assertSame('attempt-current', $created->getAttribute('migrationAttemptId')); $this->assertNotNull($database->findCollection('database_'.$created->getSequence())); } + #[RunInSeparateProcess] public function testTheMetadataSubqueryFiltersLoadTheTableColumnsAndIndexes(): void { \MigrationCLI::registerFilters(); @@ -329,9 +202,9 @@ private function createMetadataDatabase(): Database return $database; } - private function createProjectDatabase(?string $status = 'provisioning'): Database + private function createProjectDatabase(): Database { - $database = new Database(new TransactionalMemoryAdapter(), new Cache(new MemoryCache())); + $database = new Database(new MemoryAdapter(), new Cache(new MemoryCache())); $database ->setDatabase('appwrite') ->setNamespace('_project'); @@ -346,26 +219,8 @@ private function createProjectDatabase(?string $status = 'provisioning'): Databa Attribute::string(key: 'type', size: 128, default: 'tablesdb'), Attribute::string(key: 'database', size: 2000), Attribute::string(key: 'status', size: 16), - Attribute::string(key: 'migrationId', size: Database::LENGTH_KEY), - Attribute::string(key: 'migrationAttemptId', size: Database::LENGTH_KEY), ], )); - if ($status !== null) { - $database->getAuthorization()->skip( - static fn (): Document => $database->createDocument('databases', new Document([ - '$id' => 'database', - 'name' => 'Database', - 'enabled' => true, - 'search' => 'database Database', - 'originalId' => null, - 'type' => 'tablesdb', - 'database' => '', - 'status' => $status, - 'migrationId' => 'migration-terminal', - 'migrationAttemptId' => 'attempt-terminal', - ])), - ); - } return $database; } @@ -393,53 +248,14 @@ public function supportsDatabaseStatus(): bool return $source; } - private function runTransfer(Database $database, Destination $destination): void - { - $source = new class () extends MockSource { - #[Override] - public function supportsDatabaseStatus(): bool - { - return true; - } - }; - $source->pushMockResource(new DatabaseResource( - id: 'database', - name: 'Database', - type: 'tablesdb', - database: 'source-dsn', - databaseStatus: 'ready', - )); - - $transfer = new Transfer($source, $destination); - $database->getAuthorization()->skip( - static function () use ($destination, $transfer): void { - $transfer->run([Resource::TYPE_DATABASE], static function (): void { - }); - $destination->success(); - }, - ); - } - #[Override] protected function setUp(): void { parent::setUp(); - foreach (self::FILTER_STATE as $property) { - $this->filterState[$property] = (new \ReflectionProperty(Database::class, $property))->getValue(); - } $_ENV['DESTINATION_PROVIDER'] = 'appwrite'; $_ENV['DESTINATION_APPWRITE_TEST_PROJECT'] = 'destination-project'; $_ENV['DESTINATION_APPWRITE_TEST_ENDPOINT'] = 'http://example.test/v1'; $_ENV['DESTINATION_APPWRITE_TEST_KEY'] = 'test-key'; $_ENV['DESTINATION_APPWRITE_TEST_PROJECT_INTERNAL_ID'] = '1'; } - - #[Override] - protected function tearDown(): void - { - foreach ($this->filterState as $property => $value) { - (new \ReflectionProperty(Database::class, $property))->setValue(null, $value); - } - parent::tearDown(); - } } diff --git a/tests/Migration/Unit/Sources/AppwriteAbortTest.php b/tests/Migration/Unit/Sources/AppwriteAbortTest.php deleted file mode 100644 index e8747591..00000000 --- a/tests/Migration/Unit/Sources/AppwriteAbortTest.php +++ /dev/null @@ -1,413 +0,0 @@ -superseded(); - $log = new ArrayObject(); - $source = $this->source($log, [ - '/teams' => ['total' => 1, 'teams' => [self::team('team')]], - '/storage/buckets' => ['total' => 1, 'buckets' => [self::bucket('bucket')]], - '/messaging/topics' => ['total' => 1, 'topics' => [self::topic('topic')]], - ]); - $destination = new MockDestination(); - $transfer = new Transfer($source, $destination); - - $this->assertAborted($abort, static fn () => $transfer->run( - [Resource::TYPE_TEAM, Resource::TYPE_BUCKET, Resource::TYPE_TOPIC], - static fn () => throw $abort, - )); - - $this->assertSame(['/teams'], $log->getArrayCopy()); - $this->assertSame([Transfer::GROUP_AUTH], \array_keys($destination->data)); - $this->assertSame([], $source->getErrors()); - } - - /** - * @param array $resources - * @param array $cached - * @param array> $responses - * @param array $expected - */ - #[DataProvider('itemProvider')] - public function testAbortFromTheProgressCallbackStopsLaterItems(array $resources, array $cached, array $responses, array $expected): void - { - $abort = $this->superseded(); - $log = new ArrayObject(); - $source = $this->source($log, $responses); - $transfer = new Transfer($source, new MockDestination()); - - foreach ($cached as $resource) { - $transfer->getCache()->add($resource); - } - - $this->assertAborted($abort, static fn () => $transfer->run($resources, static fn () => throw $abort)); - - $this->assertSame($expected, $log->getArrayCopy()); - $this->assertSame([], $source->getErrors()); - } - - /** - * @return array, array, array>, array}> - */ - public static function itemProvider(): array - { - $deployments = ['total' => 2, 'deployments' => [self::deployment('first'), self::deployment('second')]]; - - return [ - 'files' => [ - [Resource::TYPE_FILE], - [new Bucket('bucket', 'Bucket')], - ['/storage/buckets/bucket/files' => ['total' => 2, 'files' => [self::file('first'), self::file('second')]]], - ['/storage/buckets/bucket/files', 'GET /storage/buckets/bucket/files/first/download'], - ], - 'function deployments' => [ - [Resource::TYPE_DEPLOYMENT], - [new Func('function', 'Function', 'node-22')], - ['/functions/function/deployments' => $deployments], - [ - '/functions/function/deployments', - 'HEAD /functions/function/deployments/first/download', - 'GET /functions/function/deployments/first/download', - ], - ], - 'active function deployments' => [ - [Resource::TYPE_ENVIRONMENT_VARIABLE], - [ - new Func('first', 'First', 'node-22', activeDeployment: 'active'), - new Func('second', 'Second', 'node-22', activeDeployment: 'active'), - ], - [ - '/functions/first/deployments/active' => self::deployment('active'), - '/functions/second/deployments/active' => self::deployment('active'), - ], - [ - '/functions/first/deployments/active', - 'HEAD /functions/first/deployments/active/download', - 'GET /functions/first/deployments/active/download', - ], - ], - 'site deployments' => [ - [Resource::TYPE_SITE_DEPLOYMENT], - [new Site('site', 'Site', 'nextjs', 'node-22')], - ['/sites/site/deployments' => $deployments], - [ - '/sites/site/deployments', - 'HEAD /sites/site/deployments/first/download', - 'GET /sites/site/deployments/first/download', - ], - ], - 'active site deployments' => [ - [Resource::TYPE_SITE_VARIABLE], - [ - new Site('first', 'First', 'nextjs', 'node-22', activeDeployment: 'active'), - new Site('second', 'Second', 'nextjs', 'node-22', activeDeployment: 'active'), - ], - [ - '/sites/first/deployments/active' => self::deployment('active'), - '/sites/second/deployments/active' => self::deployment('active'), - ], - [ - '/sites/first/deployments/active', - 'HEAD /sites/first/deployments/active/download', - 'GET /sites/first/deployments/active/download', - ], - ], - ]; - } - - /** - * @param array $cached - */ - #[DataProvider('typeProvider')] - public function testAbortDuringAnExportStopsTheTransfer(string $type, array $cached): void - { - $abort = $this->superseded(); - $log = new ArrayObject(); - $source = $this->source($log, [], $abort); - $transfer = new Transfer($source, new MockDestination()); - $later = \in_array($type, Transfer::GROUP_INTEGRATIONS_RESOURCES, true) ? Resource::TYPE_RULE : Resource::TYPE_WEBHOOK; - - foreach ($cached as $resource) { - $transfer->getCache()->add($resource); - } - - $this->assertAborted($abort, static fn () => $transfer->run([$type, $later], static function (): void { - })); - - $this->assertCount(1, $log, 'Nothing may be requested after the abort.'); - $this->assertSame([], $source->getErrors()); - } - - /** - * @return array}> - */ - public static function typeProvider(): array - { - $database = new Database('database', 'Database'); - $table = new Table($database, 'Table', 'table'); - - return [ - 'users' => [Resource::TYPE_USER, []], - 'teams' => [Resource::TYPE_TEAM, []], - 'memberships' => [Resource::TYPE_MEMBERSHIP, [new Team('team', 'Team')]], - 'auth methods' => [Resource::TYPE_AUTH_METHODS, []], - 'OAuth2 providers' => [Resource::TYPE_OAUTH2_PROVIDER, []], - 'policies' => [Resource::TYPE_POLICIES, []], - 'databases' => [Resource::TYPE_DATABASE, []], - 'tables' => [Resource::TYPE_TABLE, [$database]], - 'columns' => [Resource::TYPE_COLUMN, [$table]], - 'indexes' => [Resource::TYPE_INDEX, [$table]], - 'rows' => [Resource::TYPE_ROW, [$table]], - 'buckets' => [Resource::TYPE_BUCKET, []], - 'files' => [Resource::TYPE_FILE, [new Bucket('bucket', 'Bucket')]], - 'functions' => [Resource::TYPE_FUNCTION, []], - 'deployments' => [Resource::TYPE_DEPLOYMENT, [new Func('function', 'Function', 'node-22')]], - 'sites' => [Resource::TYPE_SITE, []], - 'site deployments' => [Resource::TYPE_SITE_DEPLOYMENT, [new Site('site', 'Site', 'nextjs', 'node-22')]], - 'project variables' => [Resource::TYPE_PROJECT_VARIABLE, []], - 'protocols' => [Resource::TYPE_PROJECT_PROTOCOLS, []], - 'labels' => [Resource::TYPE_PROJECT_LABELS, []], - 'services' => [Resource::TYPE_PROJECT_SERVICES, []], - 'email templates' => [Resource::TYPE_PROJECT_EMAIL_TEMPLATE, []], - 'rules' => [Resource::TYPE_RULE, []], - 'providers' => [Resource::TYPE_PROVIDER, []], - 'topics' => [Resource::TYPE_TOPIC, []], - 'subscribers' => [Resource::TYPE_SUBSCRIBER, [new Topic('topic', 'Topic')]], - 'messages' => [Resource::TYPE_MESSAGE, []], - 'platforms' => [Resource::TYPE_PLATFORM, []], - 'API keys' => [Resource::TYPE_API_KEY, []], - 'webhooks' => [Resource::TYPE_WEBHOOK, []], - 'SMTP' => [Resource::TYPE_SMTP, []], - ]; - } - - private function superseded(): Aborted - { - return new class ('Migration attempt was superseded') extends Aborted { - }; - } - - private function assertAborted(Aborted $abort, callable $run): void - { - try { - $run(); - } catch (Aborted $caught) { - $this->assertSame($abort, $caught); - - return; - } - - $this->fail('The abort did not stop the transfer.'); - } - - /** - * A real Appwrite source whose SDK requests and downloads are answered from the script - * and recorded in the log, in order. - * - * @param ArrayObject $log - * @param array> $responses - */ - private function source(ArrayObject $log, array $responses, ?Throwable $unscripted = null): Appwrite - { - $source = new class ($log) extends Appwrite { - /** - * @param ArrayObject $log - */ - public function __construct(private readonly ArrayObject $log) - { - parent::__construct('project', 'http://localhost', 'key', static fn (): null => null); - } - - #[Override] - protected function call(string $method, string $path = '', array $headers = [], array $params = [], array &$responseHeaders = []): array|string - { - $this->log->append($method . ' ' . $path); - - return 'data'; - } - }; - - $client = new class ($log, $responses, $unscripted ?? new LogicException('Unscripted request')) extends Client { - /** - * @param ArrayObject $log - * @param array> $responses - */ - public function __construct( - private readonly ArrayObject $log, - private readonly array $responses, - private readonly Throwable $unscripted, - ) { - parent::__construct(); - } - - #[Override] - public function call(string $method, string $path = '', array $headers = [], array $params = [], ?string $responseType = null): array - { - $this->log->append($path); - - return $this->responses[$path] ?? throw $this->unscripted; - } - }; - - $class = new ReflectionClass(Appwrite::class); - - foreach ($class->getProperties() as $property) { - $type = $property->getType(); - - if ($type instanceof ReflectionNamedType && \is_subclass_of($type->getName(), Service::class)) { - $service = $type->getName(); - $property->setValue($source, new $service($client)); - } - } - - $class->getProperty('client')->setValue($source, $client); - $class->getProperty('reader')->setValue($source, new API(new TablesDB($client))); - - return $source; - } - - /** - * @return array - */ - private static function team(string $id): array - { - return [ - '$id' => $id, - '$createdAt' => self::TIMESTAMP, - '$updatedAt' => self::TIMESTAMP, - 'name' => $id, - 'total' => 0, - 'prefs' => [], - ]; - } - - /** - * @return array - */ - private static function bucket(string $id): array - { - return [ - '$id' => $id, - '$createdAt' => self::TIMESTAMP, - '$updatedAt' => self::TIMESTAMP, - '$permissions' => [], - 'fileSecurity' => false, - 'name' => $id, - 'enabled' => true, - 'maximumFileSize' => 1024, - 'allowedFileExtensions' => [], - 'compression' => 'none', - 'encryption' => false, - 'antivirus' => false, - 'transformations' => false, - 'totalSize' => 0, - ]; - } - - /** - * @return array - */ - private static function topic(string $id): array - { - return [ - '$id' => $id, - '$createdAt' => self::TIMESTAMP, - '$updatedAt' => self::TIMESTAMP, - 'name' => $id, - 'emailTotal' => 0, - 'smsTotal' => 0, - 'pushTotal' => 0, - 'subscribe' => [], - ]; - } - - /** - * @return array - */ - private static function file(string $id): array - { - return [ - '$id' => $id, - 'bucketId' => 'bucket', - '$createdAt' => self::TIMESTAMP, - '$updatedAt' => self::TIMESTAMP, - '$permissions' => [], - 'name' => $id, - 'signature' => '', - 'mimeType' => 'text/plain', - 'sizeOriginal' => 4, - 'sizeActual' => 4, - 'chunksTotal' => 1, - 'chunksUploaded' => 1, - 'encryption' => false, - 'compression' => 'none', - ]; - } - - /** - * @return array - */ - private static function deployment(string $id): array - { - return [ - '$id' => $id, - '$createdAt' => self::TIMESTAMP, - '$updatedAt' => self::TIMESTAMP, - 'type' => 'manual', - 'resourceId' => '', - 'resourceType' => '', - 'entrypoint' => 'index.js', - 'sourceSize' => 4, - 'buildSize' => 0, - 'totalSize' => 4, - 'buildId' => '', - 'activate' => true, - 'screenshotLight' => '', - 'screenshotDark' => '', - 'status' => 'ready', - 'buildLogs' => '', - 'buildDuration' => 0, - 'providerRepositoryName' => '', - 'providerRepositoryOwner' => '', - 'providerRepositoryUrl' => '', - 'providerCommitHash' => '', - 'providerCommitAuthorUrl' => '', - 'providerCommitAuthor' => '', - 'providerCommitMessage' => '', - 'providerCommitUrl' => '', - 'providerBranch' => '', - 'providerBranchUrl' => '', - ]; - } -} diff --git a/tests/Migration/Unit/Sources/SourceAbortTest.php b/tests/Migration/Unit/Sources/SourceAbortTest.php deleted file mode 100644 index 031eb56a..00000000 --- a/tests/Migration/Unit/Sources/SourceAbortTest.php +++ /dev/null @@ -1,328 +0,0 @@ - $resources - * @param array $cached - */ - #[DataProvider('firebaseProvider')] - public function testFirebaseStopsAtTheAbort(array $resources, array $cached): void - { - $abort = $this->superseded(); - $log = new ArrayObject(); - $source = new class (['project_id' => 'project'], $log) extends Firebase { - /** - * @param array $serviceAccount - * @param ArrayObject $log - */ - public function __construct(array $serviceAccount, private readonly ArrayObject $log) - { - parent::__construct($serviceAccount); - } - - #[Override] - protected function call(string $method, string $path = '', array $headers = [], array $params = [], array &$responseHeaders = []): array|string - { - $this->log->append($method . ' ' . $path); - - return match (true) { - \str_ends_with($path, '/config') => ['signIn' => ['hashConfig' => []]], - \str_ends_with($path, '/downloadAccount') => ['users' => [['localId' => 'user']]], - \str_ends_with($path, ':listCollectionIds') => ['collectionIds' => ['collection']], - \str_ends_with($path, '/storage/v1/b') => ['items' => [['id' => 'bucket', 'name' => 'bucket']]], - \str_ends_with($path, '/o') => ['items' => [['name' => 'file']]], - \str_ends_with($path, '?alt=media') => 'data', - default => [], - }; - } - }; - $transfer = new Transfer($source, new MockDestination()); - - foreach ($cached as $resource) { - $transfer->getCache()->add($resource); - } - - $this->assertAborted($abort, fn () => $transfer->run($resources, $this->abortingCallback($log, $abort))); - - $events = $log->getArrayCopy(); - $this->assertSame(\count($events) - 1, \array_search(self::CALLBACK, $events, true), 'Nothing may be requested after the abort.'); - $this->assertSame([], $source->getErrors()); - } - - /** - * @return array, array}> - */ - public static function firebaseProvider(): array - { - return [ - 'users' => [[Resource::TYPE_USER, Resource::TYPE_BUCKET], []], - 'databases' => [[Resource::TYPE_DATABASE, Resource::TYPE_BUCKET], []], - 'tables' => [[Resource::TYPE_TABLE, Resource::TYPE_BUCKET], []], - 'buckets' => [[Resource::TYPE_BUCKET, Resource::TYPE_USER], []], - 'files' => [[Resource::TYPE_FILE, Resource::TYPE_USER], [(new Bucket('bucket', 'bucket'))->setOriginalId('bucket')]], - ]; - } - - /** - * @param array $cached - */ - #[DataProvider('nhostProvider')] - public function testNHostStopsAtTheAbort(string $type, array $cached): void - { - $abort = $this->superseded(); - $log = new ArrayObject(); - $source = new NHost('subdomain', 'region', 'secret', 'database', 'username', 'password'); - $source->pdo = $this->pdo($log, $abort); - - $this->assertStopsAtTheFirstEvent($source, $type, $cached, $log, $abort); - } - - /** - * @return array}> - */ - public static function nhostProvider(): array - { - $database = new Database('public', 'public'); - $table = new Table($database, 'table', 'table'); - - return [ - 'users' => [Resource::TYPE_USER, []], - 'databases' => [Resource::TYPE_DATABASE, []], - 'tables' => [Resource::TYPE_TABLE, [$database]], - 'columns' => [Resource::TYPE_COLUMN, [$table]], - 'rows' => [Resource::TYPE_ROW, [$database, $table]], - 'indexes' => [Resource::TYPE_INDEX, [$table]], - 'buckets' => [Resource::TYPE_BUCKET, []], - 'files' => [Resource::TYPE_FILE, [new Bucket('bucket', 'bucket')]], - ]; - } - - /** - * @param array $cached - */ - #[DataProvider('supabaseProvider')] - public function testSupabaseStopsAtTheAbort(string $type, array $cached): void - { - $abort = $this->superseded(); - $log = new ArrayObject(); - $source = new class () extends Supabase { - public function __construct() - { - // The real constructor connects to Postgres; the test supplies the database instead. - } - }; - $source->pdo = $this->pdo($log, $abort); - - $this->assertStopsAtTheFirstEvent($source, $type, $cached, $log, $abort); - } - - /** - * @return array}> - */ - public static function supabaseProvider(): array - { - return [ - 'users' => [Resource::TYPE_USER, []], - 'buckets' => [Resource::TYPE_BUCKET, []], - 'files' => [Resource::TYPE_FILE, [new Bucket('bucket', 'bucket')]], - ]; - } - - public function testCsvStopsAtTheAbort(): void - { - $abort = $this->superseded(); - $directory = \sys_get_temp_dir() . '/csv_abort_' . \uniqid(); - \mkdir($directory); - $device = new Local($directory); - $filePath = $device->getPath('rows.csv'); - \file_put_contents($filePath, "\$id,name\nfirst,First\nsecond,Second\n"); - - $source = CSV::fromResourceIds('database', 'table', $filePath, $device, $this->projectDatabase()); - $transfer = new Transfer($source, new MockDestination()); - - try { - $this->assertAborted($abort, static fn () => $transfer->run([Resource::TYPE_ROW], static fn () => throw $abort)); - $this->assertSame([], $source->getErrors()); - } finally { - \array_map(\unlink(...), \glob($directory . '/*') ?: []); - \rmdir($directory); - } - } - - public function testJsonStopsAtTheAbort(): void - { - $abort = $this->superseded(); - $directory = \sys_get_temp_dir() . '/json_abort_' . \uniqid(); - \mkdir($directory); - $device = new Local($directory); - $filePath = $device->getPath('rows.json'); - \file_put_contents($filePath, \json_encode([['$id' => 'first'], ['$id' => 'second']]) ?: '[]'); - - $source = JSON::fromResourceIds('database', 'table', $filePath, $device, null); - $transfer = new Transfer($source, new MockDestination()); - - try { - $this->assertAborted($abort, static fn () => $transfer->run([Resource::TYPE_ROW], static fn () => throw $abort)); - $this->assertSame([], $source->getErrors()); - } finally { - \array_map(\unlink(...), \glob($directory . '/*') ?: []); - \rmdir($directory); - } - } - - /** - * Runs the type plus a type from a later group; the database answers every statement - * with the abort and the callback aborts too, so the first event must also be the last. - * - * @param array $cached - * @param ArrayObject $log - */ - private function assertStopsAtTheFirstEvent(NHost $source, string $type, array $cached, ArrayObject $log, Aborted $abort): void - { - $transfer = new Transfer($source, new MockDestination()); - $later = \in_array($type, Transfer::GROUP_STORAGE_RESOURCES, true) ? Resource::TYPE_USER : Resource::TYPE_BUCKET; - - foreach ($cached as $resource) { - $transfer->getCache()->add($resource); - } - - $this->assertAborted($abort, fn () => $transfer->run([$type, $later], $this->abortingCallback($log, $abort))); - - $this->assertCount(1, $log, 'Nothing may happen after the abort.'); - $this->assertSame([], $source->getErrors()); - } - - private function superseded(): Aborted - { - return new class ('Migration attempt was superseded') extends Aborted { - }; - } - - private function assertAborted(Aborted $abort, callable $run): void - { - try { - $run(); - } catch (Aborted $caught) { - $this->assertSame($abort, $caught); - - return; - } - - $this->fail('The abort did not stop the transfer.'); - } - - /** - * @param ArrayObject $log - */ - private function abortingCallback(ArrayObject $log, Aborted $abort): callable - { - return static function () use ($log, $abort): never { - $log->append(self::CALLBACK); - - throw $abort; - }; - } - - /** - * @param ArrayObject $log - */ - private function pdo(ArrayObject $log, Aborted $abort): PDO - { - return new class ($log, $abort) extends PDO { - /** - * @param ArrayObject $log - */ - public function __construct(private readonly ArrayObject $log, private readonly Aborted $abort) - { - parent::__construct('sqlite::memory:'); - } - - #[Override] - public function prepare(string $query, array $options = []): never - { - $this->log->append($query); - - throw $this->abort; - } - - #[Override] - public function query(string $query, ?int $fetchMode = null, mixed ...$fetchModeArgs): never - { - $this->log->append($query); - - throw $this->abort; - } - }; - } - - private function projectDatabase(): UtopiaDatabase - { - $database = new UtopiaDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); - $database - ->setDatabase('appwrite') - ->setNamespace('_project'); - $database->create(); - $database->getAuthorization()->disable(); - - $database->createCollection(Collection::create( - id: 'databases', - attributes: [ - Attribute::string(key: 'name', size: 256), - Attribute::string(key: 'type', size: 128), - ], - )); - $metadata = $database->createDocument('databases', new Document([ - '$id' => 'database', - 'name' => 'Database', - 'type' => 'tablesdb', - ])); - - $tables = 'database_' . $metadata->getSequence(); - $database->createCollection(Collection::create( - id: $tables, - attributes: [Attribute::string(key: 'name', size: 256)], - )); - $database->createDocument($tables, new Document(['$id' => 'table', 'name' => 'Table'])); - - $database->createCollection(Collection::create( - id: 'attributes', - attributes: [ - Attribute::string(key: 'databaseInternalId', size: 64), - Attribute::string(key: 'collectionInternalId', size: 64), - ], - )); - - return $database; - } -}