From 6ac6d4df39162abeab86dd70f15f6804a35a7b68 Mon Sep 17 00:00:00 2001 From: Jake Barnby Date: Wed, 14 Oct 2026 17:55:47 +1300 Subject: [PATCH 1/8] refactor(destination)!: remove provisioning ownership fencing The ownership fence existed only to serve Appwrite's V26 migration claim protocol, which is being removed everywhere. Without that caller nothing can attest a terminal owner, so the fence only refused work: an ownerless `provisioning` or `failed` database waited out a 24-hour lease before any retry could recover it. The destination returns to the 2.x status-only lifecycle on the 8.x API: an incomplete database is overwritten and its backing collection recreated on the next run, and status flips are plain updates again. The non-fencing fixes made since 2.x stay (recovery under OnDuplicate::Fail, marking a database failed when its reload misses), as does the finalization in success(), which is removed separately. Breaking: ProvisioningOwner, the owner, getRecoverableOwner and provisioningLease constructor arguments, DEFAULT_PROVISIONING_LEASE and the CLI's --migration-id, --migration-attempt-id, --recover-* and --help options are gone. ProvisioningOwnerTest and the e2e AppwriteDatabaseConcurrencyTest covered only the fence and are deleted. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 16 + README.md | 12 - bin/MigrationCLI.php | 104 +- src/Migration/Destinations/Appwrite.php | 391 +---- .../Appwrite/ProvisioningOwner.php | 24 - .../AppwriteDatabaseConcurrencyTest.php | 418 ------ .../AppwriteCheckAttributeTest.php | 3 - .../AppwriteDatabaseStatusTest.php | 1303 +---------------- .../AppwriteDestinationDsnTest.php | 3 - .../Destinations/AppwriteFinalizationTest.php | 14 +- .../Destinations/AppwriteIndexLengthsTest.php | 3 - .../AppwriteRelationshipOnDeleteTest.php | 3 - .../Destinations/AppwriteRowImportTest.php | 5 - .../Destinations/AppwriteSpatialIndexTest.php | 3 - .../Destinations/ProvisioningOwnerTest.php | 36 - .../Unit/Destinations/TransfersColumns.php | 3 - tests/Migration/Unit/MigrationCLITest.php | 181 +-- 17 files changed, 131 insertions(+), 2391 deletions(-) delete mode 100644 src/Migration/Destinations/Appwrite/ProvisioningOwner.php delete mode 100644 tests/Migration/E2E/Destinations/AppwriteDatabaseConcurrencyTest.php delete mode 100644 tests/Migration/Unit/Destinations/ProvisioningOwnerTest.php diff --git a/CHANGELOG.md b/CHANGELOG.md index d8f52143..f5cb58dc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,21 @@ # Changelog +## 3.1.2 + +### Breaking changes + +- Provisioning ownership fencing is removed. The Appwrite destination is back on the 2.x status-only database + lifecycle. + - `Destinations\Appwrite\ProvisioningOwner` is removed. + - `Destinations\Appwrite::__construct()` no longer takes `owner`, `getRecoverableOwner` or `provisioningLease`, and + `Destinations\Appwrite::DEFAULT_PROVISIONING_LEASE` is removed. `onDuplicate` and the arguments after it move up + two places for positional callers. + - The destination no longer reads or writes the `migrationId` and `migrationAttemptId` database attributes. + - A `provisioning` or `failed` database is recovered by the next run at once, without owner attestation or a lease + wait. +- The standalone CLI no longer accepts `--migration-id`, `--migration-attempt-id`, `--recover-migration-id`, + `--recover-migration-attempt-id` or `-h`/`--help`. + ## 3.0.0 (unreleased) 3.0 is a major release. The breaking changes below all concern the Appwrite destination's lifecycle and the diff --git a/README.md b/README.md index ee884d56..c021e60f 100644 --- a/README.md +++ b/README.md @@ -44,18 +44,6 @@ $transfer->run( ); ``` -## Appwrite database recovery - -Appwrite database destinations use a `ProvisioningOwner` made from a stable logical migration identifier and a fresh attempt identifier for every execution. The required `getRecoverableOwner` callback is the recovery authority for an existing database whose status is `provisioning` or `failed`. - -A database status is local to that resource. It does not prove that the migration attempt which owns it has stopped, because an import can continue with other resources after recording a database failure. The callback must therefore consult the caller's authoritative operation lifecycle and return the exact stored owner only after that attempt is terminal. Return `null` while it is active or unknown; recovery then fails closed. This rule also applies when the retry uses the same logical migration identifier. - -Owners are written and enforced only when the destination's `databases` metadata declares both `migrationId` and `migrationAttemptId`; metadata with `status` alone gets status tracking without ownership. A `provisioning` or `failed` database that names no owner at all has nothing for the callback to judge, so its update timestamp stands in: it is recovered only once it has gone a whole `provisioningLease` without a write, since a fresher one may still belong to a migration that is provisioning it. A row still within the lease is refused with an error naming the database. Past the lease it is recovered as it was before ownership existed: a `failed` database is overwritten, a `provisioning` one is resolved like any existing database under the `OnDuplicate` policy (`fail` keeps it instead of colliding) and marked `ready` when the run succeeds, and either is claimed by the current attempt when the metadata can record it. A database naming only part of an owner is refused. - -The `provisioningLease` constructor argument is the lease in seconds and defaults to `86400` (24 hours), the longest lease a live Appwrite migration attempt holds; `0` recovers an ownerless database at once, and a negative value is rejected with an `InvalidArgumentException`. - -The standalone CLI requires `--migration-id` and a fresh `--migration-attempt-id`. Recovering an incomplete database that names an owner additionally requires both `--recover-migration-id` and `--recover-migration-attempt-id` for the exact terminal prior attempt. - ## Supported Resources Chart Sources: diff --git a/bin/MigrationCLI.php b/bin/MigrationCLI.php index 422f21e0..e105bf04 100644 --- a/bin/MigrationCLI.php +++ b/bin/MigrationCLI.php @@ -11,7 +11,6 @@ use Utopia\Database\Query; use Utopia\Migration\Destination; use Utopia\Migration\Destinations\Appwrite as DestinationsAppwrite; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; use Utopia\Migration\Destinations\Local; use Utopia\Migration\Source; use Utopia\Migration\Sources\Appwrite; @@ -34,9 +33,6 @@ class MigrationCLI protected mixed $destination; - /** @var list */ - private readonly array $arguments; - protected const STRUCTURE = [ '$collection' => 'databases', '$id' => 'collections', @@ -157,34 +153,6 @@ class MigrationCLI ], ]; - /** @param list $arguments */ - public function __construct(array $arguments = []) - { - $this->arguments = $arguments; - } - - public static function getHelp(): string - { - return <<<'HELP' -Usage: php bin/MigrationCLI.php [options] - -Options: - -h, --help Show this help. - --migration-id= Stable logical owner identifier for this migration. - Required for Appwrite; reuse it for retries. - --migration-attempt-id= - Required for Appwrite; use a fresh attempt for every - execution or retry. - --recover-migration-id= - --recover-migration-attempt-id= - Together attest that the exact prior migration attempt is terminal - and allow recovery of its provisioning or failed databases. - Recovery is refused by default. - Resource status alone never proves an attempt terminal. - -HELP; - } - /** * Prints the current status of migrations as a table after wiping the screen */ @@ -283,7 +251,6 @@ public function getDestination(): Destination switch ($_ENV['DESTINATION_PROVIDER']) { case 'appwrite': $database = $this->getDatabase('destination'); - $recoverableOwner = $this->getRecoverableOwner(); return new DestinationsAppwrite( project: $_ENV['DESTINATION_APPWRITE_TEST_PROJECT'], @@ -294,10 +261,6 @@ public function getDestination(): Destination collectionStructure: self::STRUCTURE, dbForPlatform: $database, projectInternalId: $_ENV['DESTINATION_APPWRITE_TEST_PROJECT_INTERNAL_ID'], - owner: new ProvisioningOwner($this->getMigrationId(), $this->getMigrationAttemptId()), - // The standalone operator supplies the fixed pair only after confirming the prior - // lifecycle owner is terminal. The destination independently compares it with the row. - getRecoverableOwner: static fn (Document $document): ?ProvisioningOwner => $recoverableOwner, ); case 'local': return new Local('./localBackup'); @@ -306,62 +269,6 @@ public function getDestination(): Destination } } - private function getMigrationId(): string - { - foreach ($this->arguments as $argument) { - if (! \str_starts_with($argument, '--migration-id=')) { - continue; - } - - $migrationId = \substr($argument, \strlen('--migration-id=')); - if ($migrationId !== '') { - return $migrationId; - } - } - - throw new \InvalidArgumentException('--migration-id is required for an Appwrite destination'); - } - - private function getMigrationAttemptId(): string - { - foreach ($this->arguments as $argument) { - if (! \str_starts_with($argument, '--migration-attempt-id=')) { - continue; - } - - $attemptId = \substr($argument, \strlen('--migration-attempt-id=')); - if ($attemptId !== '') { - return $attemptId; - } - } - - throw new \InvalidArgumentException('--migration-attempt-id is required for an Appwrite destination'); - } - - private function getRecoverableOwner(): ?ProvisioningOwner - { - $migrationId = null; - $attemptId = null; - - foreach ($this->arguments as $argument) { - if (\str_starts_with($argument, '--recover-migration-id=')) { - $value = \substr($argument, \strlen('--recover-migration-id=')); - $migrationId = $value !== '' ? $value : null; - } - - if (\str_starts_with($argument, '--recover-migration-attempt-id=')) { - $value = \substr($argument, \strlen('--recover-migration-attempt-id=')); - $attemptId = $value !== '' ? $value : null; - } - } - - if ($migrationId === null || $attemptId === null) { - return null; - } - - return new ProvisioningOwner($migrationId, $attemptId); - } - public function getDatabase(string $type): Database { self::registerFilters(); @@ -546,12 +453,7 @@ protected function loadEnvironment(): void } if (\realpath($_SERVER['SCRIPT_FILENAME'] ?? '') === __FILE__) { - $arguments = $_SERVER['argv'] ?? []; - if (\in_array('-h', $arguments, true) || \in_array('--help', $arguments, true)) { - echo MigrationCLI::getHelp(); - } else { - $instance = new MigrationCLI($arguments); - $instance->start(); - $instance->drawFrame(); - } + $instance = new MigrationCLI(); + $instance->start(); + $instance->drawFrame(); } diff --git a/src/Migration/Destinations/Appwrite.php b/src/Migration/Destinations/Appwrite.php index f436d7fb..00a95472 100644 --- a/src/Migration/Destinations/Appwrite.php +++ b/src/Migration/Destinations/Appwrite.php @@ -33,7 +33,6 @@ use Utopia\Database\Document as UtopiaDocument; use Utopia\Database\Exception as DatabaseException; use Utopia\Database\Exception\Authorization as AuthorizationException; -use Utopia\Database\Exception\Conflict as ConflictException; use Utopia\Database\Exception\Duplicate as DuplicateException; use Utopia\Database\Exception\Limit as LimitException; use Utopia\Database\Exception\Structure as StructureException; @@ -51,7 +50,6 @@ use Utopia\Database\Validator\Structure; use Utopia\Database\Validator\UID; use Utopia\Migration\Destination; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; use Utopia\Migration\Exception; use Utopia\Migration\Exception\Finalization; use Utopia\Migration\Resource; @@ -129,13 +127,6 @@ class Appwrite extends Destination private const DATABASE_STATUS_READY = 'ready'; private const DATABASE_STATUS_FAILED = 'failed'; - /** Attributes naming the migration attempt that provisions a database; a destination schema gains them after `status`. */ - private const OWNER_MIGRATION_ID = 'migrationId'; - private const OWNER_ATTEMPT_ID = 'migrationAttemptId'; - - /** 24 hours, the longest lease a live Appwrite attempt holds. */ - public const int DEFAULT_PROVISIONING_LEASE = 86_400; - /** * Attribute fields the SDK can't update in place (no per-type updateX endpoint exposes them); a change here * forces drop+recreate. The type is immutable too, but is compared through {@see self::typeMatches()}. @@ -188,19 +179,6 @@ class Appwrite extends Destination */ protected $getDatabaseDSN; - /** Immutable owner written with every provisioning transition initiated by this destination. */ - private readonly ProvisioningOwner $owner; - - /** - * Resolves the authoritative terminal owner of an incomplete database that - * names one. Database status is resource-local and never proves the overall - * migration attempt terminal. Callers must derive this from their authoritative - * operation lifecycle; null means active or unknown and fails closed. - * - * @var callable(UtopiaDocument $database): ?ProvisioningOwner - */ - private $getRecoverableOwner; - /** * @var array */ @@ -242,9 +220,6 @@ class Appwrite extends Destination /** Whether the destination project's database metadata supports lifecycle status. */ private ?bool $databaseStatusSupported = null; - /** Whether the destination project's database metadata can record which migration attempt provisions a database. */ - private ?bool $provisioningOwnerSupported = null; - /** * @param string $project * @param string $endpoint @@ -252,12 +227,9 @@ class Appwrite extends Destination * @param UtopiaDatabase $dbForProject * @param callable(UtopiaDocument $database):UtopiaDatabase $getDatabasesDB * @param CollectionStructure $collectionStructure - * @param ProvisioningOwner $owner Immutable logical migration and execution-attempt identifiers for databases provisioned by this destination. - * @param callable(UtopiaDocument $database): ?ProvisioningOwner $getRecoverableOwner Returns the exact authoritative terminal owner for an existing `provisioning` or `failed` database that names an owner. Resource status alone is not lifecycle proof; return null while the owning migration attempt is active or unknown. A database naming no owner predates ownership and is recovered without it. * @param OnDuplicate $onDuplicate Behavior when a row with an existing $id is encountered. * @param (callable(Database $resource): string)|null $getDatabaseDSN Resolver for the destination's `_databases.database` value. Pass when the destination project's DSN differs from the source's, so the destination row carries its own DSN instead of inheriting the source's. * @param array $collectionStructures Per-database-type metadata collection structures (e.g. `['vectorsdb' => ...]`), used instead of $collectionStructure when the imported database's type has an entry. Types with an entry also get type-specific metadata written (e.g. vectorsdb collection `dimension`). - * @param int $provisioningLease Seconds an existing `provisioning` or `failed` database that names no owner may go without an update before another migration may recover it. Such a row carries no owner for $getRecoverableOwner to judge, so its update timestamp stands in: one fresher than the lease may still belong to a migration that is provisioning it, and fails closed. Defaults to 24 hours, the longest lease a live Appwrite attempt holds. `0` recovers it at once; a negative value is rejected. */ public function __construct( string $project, @@ -268,17 +240,10 @@ public function __construct( protected array $collectionStructure, protected UtopiaDatabase $dbForPlatform, protected string $projectInternalId, - ProvisioningOwner $owner, - callable $getRecoverableOwner, protected OnDuplicate $onDuplicate = OnDuplicate::Fail, ?callable $getDatabaseDSN = null, protected array $collectionStructures = [], - protected int $provisioningLease = self::DEFAULT_PROVISIONING_LEASE, ) { - if ($provisioningLease < 0) { - throw new \InvalidArgumentException('Provisioning lease must not be negative'); - } - $this->projectId = $project; $this->endpoint = $endpoint; $this->key = $key; @@ -301,8 +266,6 @@ public function __construct( $this->getDatabasesDB = $getDatabasesDB; $this->getDatabaseDSN = $getDatabaseDSN; - $this->owner = $owner; - $this->getRecoverableOwner = $getRecoverableOwner; } /** @@ -329,29 +292,17 @@ private function getSupportForDatabaseStatus(): bool return false; } - return $this->databaseStatusSupported ??= $this->declaresDatabaseAttributes('status'); - } - - private function getSupportForProvisioningOwner(): bool - { - if (! $this->getSupportForDatabaseStatus()) { - return false; + if ($this->databaseStatusSupported !== null) { + return $this->databaseStatusSupported; } - return $this->provisioningOwnerSupported ??= $this->declaresDatabaseAttributes( - self::OWNER_MIGRATION_ID, - self::OWNER_ATTEMPT_ID, - ); - } - - private function declaresDatabaseAttributes(string ...$keys): bool - { - $declared = []; foreach ($this->dbForProject->findCollection(self::META_DATABASES)?->attributes() ?? [] as $attribute) { - $declared[] = $attribute->key; + if ($attribute->key === 'status') { + return $this->databaseStatusSupported = true; + } } - return \array_diff($keys, $declared) === []; + return $this->databaseStatusSupported = false; } /** @@ -462,9 +413,7 @@ private function markProvisionedDatabasesReady(): array $failures = []; foreach (\array_keys($this->provisioningDatabases) as $databaseId) { try { - if (! $this->setDatabaseStatus($databaseId, self::DATABASE_STATUS_READY)) { - throw new DatabaseException('Database provisioning owner changed before finalization'); - } + $this->setDatabaseStatus($databaseId, self::DATABASE_STATUS_READY); } catch (\Throwable $error) { $failures[] = $this->recordFinalizationFailure(Resource::TYPE_DATABASE, (string) $databaseId, $error); } @@ -488,142 +437,17 @@ private function recordFinalizationFailure(string $resourceName, string $resourc return $failure; } - private function setDatabaseStatus(string $databaseId, string $status): bool + private function setDatabaseStatus(string $databaseId, string $status): void { if (! $this->getSupportForDatabaseStatus()) { - return true; - } - - try { - $database = $this->dbForProject->getDocument( - self::META_DATABASES, - $databaseId, - forUpdate: true, - ); - if ( - $database->isEmpty() - || $database->getAttribute('status') !== self::DATABASE_STATUS_PROVISIONING - || ($this->getSupportForProvisioningOwner() && ! $this->carriesOwner($database, $this->owner)) - ) { - return false; - } - - $updated = $this->updateOwned($database, new UtopiaDocument(['status' => $status])); - - return ! $updated->isEmpty(); - } catch (ConflictException) { - return false; - } - } - - /** - * A write pinned to the timestamp the row was read with is refused once another writer got - * there first, which holds only while every write moves `$updatedAt` strictly forward. - * Writes here never supply `$updatedAt`, and the stored value is checked after each one. - */ - private function updateOwned(UtopiaDocument $observed, UtopiaDocument $updates): UtopiaDocument - { - $readAt = $observed->getUpdatedAt(); - if ($readAt === null || $readAt === '') { - throw new DatabaseException('Database provisioning ownership requires an update timestamp'); + return; } - $updated = $this->dbForProject->withRequestTimestamp( - new \DateTime($readAt), - fn (): UtopiaDocument => $this->dbForProject->updateDocument(self::META_DATABASES, $observed->getId(), $updates), + $this->dbForProject->updateDocument( + self::META_DATABASES, + $databaseId, + new UtopiaDocument(['status' => $status]), ); - if ($updated->isEmpty()) { - return $updated; - } - - $stored = $this->dbForProject->getDocument(self::META_DATABASES, $observed->getId(), forUpdate: true); - if ($stored->isEmpty()) { - return $stored; - } - - if (new \DateTime((string) $stored->getUpdatedAt()) <= new \DateTime($readAt)) { - throw new DatabaseException('Database '.$observed->getId().' provisioning write did not move its update timestamp forward'); - } - - return $stored; - } - - private function getProvisioningOwner(UtopiaDocument $database): ?ProvisioningOwner - { - $migrationId = $database->getAttribute(self::OWNER_MIGRATION_ID); - $attemptId = $database->getAttribute(self::OWNER_ATTEMPT_ID); - if ( - ! \is_string($migrationId) - || $migrationId === '' - || ! \is_string($attemptId) - || $attemptId === '' - ) { - return null; - } - - return new ProvisioningOwner($migrationId, $attemptId); - } - - /** - * A row written before its schema could record an owner names none at all. A row naming - * only part of an owner is not unowned: it fails closed like any other unattested owner. - */ - private function isUnowned(UtopiaDocument $database): bool - { - if (! $this->getSupportForProvisioningOwner()) { - return true; - } - - foreach ([self::OWNER_MIGRATION_ID, self::OWNER_ATTEMPT_ID] as $key) { - if (! \in_array($database->getAttribute($key), [null, ''], true)) { - return false; - } - } - - return true; - } - - /** - * @throws \Exception - */ - private function isStale(UtopiaDocument $database): bool - { - if ($this->provisioningLease === 0) { - return true; - } - - $updatedAt = $database->getUpdatedAt(); - if ($updatedAt === null || $updatedAt === '') { - return false; - } - - return (new \DateTimeImmutable($updatedAt))->getTimestamp() + $this->provisioningLease - < (new \DateTimeImmutable())->getTimestamp(); - } - - /** Whether the row names exactly $owner, or no owner at all when $owner is null. */ - private function carriesOwner(UtopiaDocument $database, ?ProvisioningOwner $owner): bool - { - if ($owner === null) { - return $this->isUnowned($database); - } - - $current = $this->getProvisioningOwner($database); - - return $current !== null && $current->equals($owner); - } - - /** @return array */ - private function provisioningOwnerAttributes(): array - { - if (! $this->getSupportForProvisioningOwner()) { - return []; - } - - return [ - self::OWNER_MIGRATION_ID => $this->owner->migrationId, - self::OWNER_ATTEMPT_ID => $this->owner->attemptId, - ]; } /** Best-effort transition to `failed`; a secondary error here must not mask the caller's original throw. */ @@ -973,90 +797,37 @@ protected function createDatabase(Database $resource): bool $updatedAt = $this->normalizeDateTime($resource->getUpdatedAt(), $createdAt); $existing = $this->dbForProject->getDocument(self::META_DATABASES, $resource->getId()); - $supportsStatus = $this->getSupportForDatabaseStatus(); - $status = $supportsStatus && ! $existing->isEmpty() - ? $existing->getAttribute('status') - : null; - $isIncomplete = \in_array( - $status, - [self::DATABASE_STATUS_PROVISIONING, self::DATABASE_STATUS_FAILED], - true, - ); - $isUnowned = $isIncomplete && $this->isUnowned($existing); - $expectedOwner = null; - if ($isIncomplete && ! $isUnowned) { - $snapshotOwner = $this->getProvisioningOwner($existing); - $expectedOwner = ($this->getRecoverableOwner)($existing); - if ( - $snapshotOwner === null - || ! $expectedOwner instanceof ProvisioningOwner - || ! $snapshotOwner->equals($expectedOwner) - || $this->owner->attemptId === $expectedOwner->attemptId - ) { - throw new DatabaseException('Database '.$resource->getId().' recovery requires exact terminal-owner attestation and a fresh attempt'); - } - } elseif ($isUnowned && ! $this->isStale($existing)) { - throw new DatabaseException('Database '.$resource->getId().' names no owner and another migration may still be provisioning it; it becomes recoverable after '.$this->provisioningLease.' seconds without an update'); - } + $isIncomplete = ! $existing->isEmpty() + && $this->getSupportForDatabaseStatus() + && \in_array( + $existing->getAttribute('status'), + [self::DATABASE_STATUS_FAILED, self::DATABASE_STATUS_PROVISIONING], + true, + ); if ($this->onDuplicate !== OnDuplicate::Fail || $isIncomplete) { - /** @var array{action: SchemaAction, database: UtopiaDocument, incomplete: bool} $claim */ - $claim = $this->dbForProject->withTransaction(function () use ( - $resource, + $action = $this->onDuplicate->resolveSchemaAction( + !$existing->isEmpty(), $updatedAt, - $supportsStatus, - $status, - $isIncomplete, - $isUnowned, - $expectedOwner, - ): array { - $locked = $this->dbForProject->getDocument( - self::META_DATABASES, - $resource->getId(), - forUpdate: true, - ); - $lockedStatus = $supportsStatus && ! $locked->isEmpty() - ? $locked->getAttribute('status') - : null; - $lockedIncomplete = \in_array( - $lockedStatus, - [self::DATABASE_STATUS_PROVISIONING, self::DATABASE_STATUS_FAILED], - true, - ); - - if ($isIncomplete) { - if ($lockedStatus !== $status || ! $this->carriesOwner($locked, $expectedOwner)) { - throw new DatabaseException('Database '.$resource->getId().' recovery owner changed before it could be claimed'); - } - if ($isUnowned && ! $this->isStale($locked)) { - throw new DatabaseException('Database '.$resource->getId().' names no owner and was updated before it could be claimed'); - } - } elseif ($lockedIncomplete) { - throw new DatabaseException('Database '.$resource->getId().' requires terminal migration attestation before recovery'); - } - - $action = $this->onDuplicate->resolveSchemaAction( - ! $locked->isEmpty(), - $updatedAt, - $locked->getUpdatedAt(), - ); + $existing->getUpdatedAt(), + ); - if ($isUnowned && $status === self::DATABASE_STATUS_PROVISIONING) { - // Left provisioning before it could name an owner: recovered in place as it was then, under the - // OnDuplicate policy (Fail skips instead of colliding), and re-enrolled for the ready flip. - if ($action === SchemaAction::Create || $this->databaseSpecMatches($locked, $resource)) { - $action = SchemaAction::Skip; - } - } elseif ($isIncomplete) { - // A prior run created the metadata document but left the database unusable. Force an - // overwrite after the locked ownership check so retries can recreate its collection. - $action = SchemaAction::Overwrite; - } elseif ($action !== SchemaAction::Create && $this->databaseSpecMatches($locked, $resource)) { - $action = SchemaAction::Skip; - } + if ($isIncomplete) { + // A prior run created the metadata document but left the database unusable (its backing + // collection may be missing). Force Overwrite — regardless of timestamps, spec match, or + // OnDuplicate::Fail — so retries recreate the collection instead of hitting the existing ID. + $action = SchemaAction::Overwrite; + } elseif ($action !== SchemaAction::Create && $this->databaseSpecMatches($existing, $resource)) { + $action = SchemaAction::Skip; + } - $document = []; - if ($action === SchemaAction::Overwrite) { + $earlyReturn = match ($action) { + SchemaAction::Skip => (function () use ($resource, $existing): bool { + $resource->setSequence($existing->getSequence()); + $resource->setStatus(Resource::STATUS_SKIPPED, 'Already exists on destination'); + return false; + })(), + SchemaAction::Overwrite => (function () use ($resource, $existing, $updatedAt, $isIncomplete): bool { $document = [ 'name' => $resource->getDatabaseName(), 'search' => implode(' ', [$resource->getId(), $resource->getDatabaseName()]), @@ -1064,71 +835,44 @@ protected function createDatabase(Database $resource): bool 'type' => empty($resource->getType()) ? 'legacy' : $resource->getType(), 'originalId' => empty($resource->getOriginalId()) ? null : $resource->getOriginalId(), 'database' => $this->resolveDestinationDsn($resource), + '$updatedAt' => $updatedAt, ]; - if ($supportsStatus) { + if ($this->getSupportForDatabaseStatus()) { $document['status'] = self::DATABASE_STATUS_PROVISIONING; } - } - if ($action === SchemaAction::Overwrite || $isIncomplete) { - $document = [...$document, ...$this->provisioningOwnerAttributes()]; - } - - if ($document === [] && $isUnowned && $supportsStatus) { - $document = [ - 'status' => self::DATABASE_STATUS_PROVISIONING, - '$updatedAt' => DateTime::nowAfter($locked->getUpdatedAt()), - ]; - } + $this->dbForProject->updateDocument(self::META_DATABASES, $existing->getId(), new UtopiaDocument($document)); + $resource->setSequence($existing->getSequence()); - if ($document !== []) { - $updated = $this->updateOwned($locked, new UtopiaDocument($document)); - if ($updated->isEmpty()) { - throw new DatabaseException('Database '.$resource->getId().' provisioning owner changed before it could be claimed'); + // Only an incomplete database can be missing its backing collection (a prior run wrote the + // metadata document but threw before createCollection). Recreate it so we never flip a + // database to ready with no collection behind it. + if ($isIncomplete && $this->dbForProject->findCollection($this->databaseCollectionId($existing)) === null) { + try { + $structure = $this->collectionStructureFor($resource); + + $this->dbForProject->createCollection(UtopiaCollection::create( + id: $this->databaseCollectionId($existing), + attributes: $this->schemaAttributes($structure['attributes'] ?? []), + indexes: $this->schemaIndexes($structure['indexes'] ?? []), + )); + } catch (\Throwable $e) { + $this->markDatabaseFailed($resource->getId()); + throw $e; + } } - } - return [ - 'action' => $action, - 'database' => $locked, - 'incomplete' => $isIncomplete, - ]; - }); - - $action = $claim['action']; - $existing = $claim['database']; - $isIncomplete = $claim['incomplete']; - - if ($action !== SchemaAction::Create) { - $resource->setSequence($existing->getSequence()); - - // The claim transaction commits before inspecting or creating the backing collection. - if ($isIncomplete && $this->dbForProject->findCollection($this->databaseCollectionId($existing)) === null) { - try { - $structure = $this->collectionStructureFor($resource); - - $this->dbForProject->createCollection(UtopiaCollection::create( - id: $this->databaseCollectionId($existing), - attributes: $this->schemaAttributes($structure['attributes'] ?? []), - indexes: $this->schemaIndexes($structure['indexes'] ?? []), - )); - } catch (\Throwable $e) { - $this->markDatabaseFailed($resource->getId()); - throw $e; + if ($this->getSupportForDatabaseStatus()) { + $this->provisioningDatabases[$resource->getId()] = true; } - } - - if ($supportsStatus && ($action === SchemaAction::Overwrite || $isIncomplete)) { - $this->provisioningDatabases[$resource->getId()] = true; - } - - if ($action === SchemaAction::Skip) { - $resource->setStatus(Resource::STATUS_SKIPPED, 'Already exists on destination'); - return false; - } - return true; + return true; + })(), + SchemaAction::Create => null, + }; + if ($earlyReturn !== null) { + return $earlyReturn; } } @@ -1150,7 +894,6 @@ protected function createDatabase(Database $resource): bool // source leaves status untouched so the collection default applies. Never copy the source's state. if ($this->getSupportForDatabaseStatus()) { $document['status'] = self::DATABASE_STATUS_PROVISIONING; - $document = [...$document, ...$this->provisioningOwnerAttributes()]; } $database = $this->dbForProject->createDocument(self::META_DATABASES, new UtopiaDocument($document)); diff --git a/src/Migration/Destinations/Appwrite/ProvisioningOwner.php b/src/Migration/Destinations/Appwrite/ProvisioningOwner.php deleted file mode 100644 index 7a27879d..00000000 --- a/src/Migration/Destinations/Appwrite/ProvisioningOwner.php +++ /dev/null @@ -1,24 +0,0 @@ -migrationId === $owner->migrationId - && $this->attemptId === $owner->attemptId; - } -} diff --git a/tests/Migration/E2E/Destinations/AppwriteDatabaseConcurrencyTest.php b/tests/Migration/E2E/Destinations/AppwriteDatabaseConcurrencyTest.php deleted file mode 100644 index 51da650c..00000000 --- a/tests/Migration/E2E/Destinations/AppwriteDatabaseConcurrencyTest.php +++ /dev/null @@ -1,418 +0,0 @@ -> */ - public array $databaseWrites = []; - - public ?Closure $beforeBackingCollectionCreate = null; - - public bool $throwAfterBackingCollectionCallback = false; - - #[Override] - public function updateDocument(string $collection, string $id, UtopiaDocument $document): UtopiaDocument - { - if ($collection === 'databases') { - $this->databaseWrites[] = $document->getArrayCopy(); - } - - return parent::updateDocument($collection, $id, $document); - } - - #[Override] - public function createCollection(Collection $collection): Collection - { - if ( - $this->beforeBackingCollectionCreate !== null - && \str_starts_with($collection->getId(), 'database_') - ) { - $callback = $this->beforeBackingCollectionCreate; - $this->beforeBackingCollectionCreate = null; - $callback(); - - if ($this->throwAfterBackingCollectionCallback) { - throw new DatabaseException('backing collection creation failed'); - } - } - - return parent::createCollection($collection); - } -} - -final class AppwriteDatabaseConcurrencyTest extends TestCase -{ - /** - * @var array - */ - private const array PDO_ATTRIBUTES = [ - PDO::ATTR_TIMEOUT => 3, - PDO::ATTR_PERSISTENT => false, - PDO::ATTR_DEFAULT_FETCH_MODE => PDO::FETCH_ASSOC, - PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION, - PDO::ATTR_EMULATE_PREPARES => true, - PDO::ATTR_STRINGIFY_FETCHES => true, - ]; - - public function testStaleIncompleteClaimLosesAfterAnotherSuccessorCommits(): void - { - foreach (['provisioning', 'failed'] as $status) { - [$second, $third, $path] = $this->createSharedDatabases(); - - try { - $this->seedDatabase($second, $status, 'migration-first', 'attempt-first'); - $winner = null; - $loser = $this->createDestination( - $second, - 'migration-second', - 'attempt-second', - function (UtopiaDocument $snapshot) use ($third, &$winner): ProvisioningOwner { - $winner = $this->createDestination( - $third, - 'migration-third', - 'attempt-third', - static fn (UtopiaDocument $document): ProvisioningOwner => new ProvisioningOwner( - 'migration-first', - 'attempt-first', - ), - ); - $this->runTransfer($third, $winner); - - return new ProvisioningOwner('migration-first', 'attempt-first'); - }, - ); - $second->databaseWrites = []; - - $this->runTransfer($second, $loser); - - $database = $this->getDatabaseDocument($third); - $this->assertInstanceOf(AppwriteDestination::class, $winner); - $this->assertSame([], $this->errorMessages($winner)); - $this->assertNotSame([], $this->errorMessages($loser)); - $this->assertSame([], $second->databaseWrites); - $this->assertSame('ready', $database->getAttribute('status')); - $this->assertSame('migration-third', $database->getAttribute('migrationId')); - $this->assertSame('attempt-third', $database->getAttribute('migrationAttemptId')); - } finally { - $this->removeSQLiteFiles($path); - } - } - } - - public function testStaleOwnerCannotMarkDatabaseReadyAfterTakeover(): void - { - [$second, $third, $path] = $this->createSharedDatabases(); - - try { - $this->seedDatabase($second, 'provisioning', 'migration-shared', 'attempt-first'); - $successor = null; - $destination = $this->createDestination( - $second, - 'migration-shared', - 'attempt-second', - static fn (UtopiaDocument $snapshot): ProvisioningOwner => new ProvisioningOwner( - 'migration-shared', - 'attempt-first', - ), - ); - - $failure = null; - try { - $this->runTransfer( - $second, - $destination, - function () use ($third, &$successor): void { - $successor = $this->createDestination( - $third, - 'migration-shared', - 'attempt-third', - static fn (UtopiaDocument $snapshot): ProvisioningOwner => new ProvisioningOwner( - 'migration-shared', - 'attempt-second', - ), - ); - $this->claimWithoutTerminalTransition($third, $successor); - }, - ); - } catch (Finalization $error) { - $failure = $error; - } - - $database = $this->getDatabaseDocument($third); - $readyWrites = \array_values(\array_filter( - $second->databaseWrites, - static fn (array $document): bool => ($document['status'] ?? null) === 'ready', - )); - $this->assertInstanceOf(AppwriteDestination::class, $successor); - $this->assertInstanceOf(Finalization::class, $failure); - $this->assertSame($destination->getErrors(), $failure->failures); - $this->assertSame( - ['Database provisioning owner changed before finalization'], - $this->errorMessages($destination), - ); - $this->assertSame([], $this->errorMessages($successor)); - $this->assertSame([], $readyWrites); - $this->assertSame('provisioning', $database->getAttribute('status')); - $this->assertSame('migration-shared', $database->getAttribute('migrationId')); - $this->assertSame('attempt-third', $database->getAttribute('migrationAttemptId')); - } finally { - $this->removeSQLiteFiles($path); - } - } - - public function testStaleOwnerCannotMarkDatabaseFailedAfterTakeover(): void - { - [$second, $third, $path] = $this->createSharedDatabases(); - - try { - $this->seedDatabase($second, 'failed', 'migration-shared', 'attempt-first'); - $successor = null; - $second->throwAfterBackingCollectionCallback = true; - $second->beforeBackingCollectionCreate = function () use ($third, &$successor): void { - $successor = $this->createDestination( - $third, - 'migration-shared', - 'attempt-third', - static fn (UtopiaDocument $snapshot): ProvisioningOwner => new ProvisioningOwner( - 'migration-shared', - 'attempt-second', - ), - ); - $this->claimWithoutTerminalTransition($third, $successor); - }; - $destination = $this->createDestination( - $second, - 'migration-shared', - 'attempt-second', - static fn (UtopiaDocument $snapshot): ProvisioningOwner => new ProvisioningOwner( - 'migration-shared', - 'attempt-first', - ), - ); - - $this->runTransfer($second, $destination); - - $database = $this->getDatabaseDocument($third); - $failedWrites = \array_values(\array_filter( - $second->databaseWrites, - static fn (array $document): bool => ($document['status'] ?? null) === 'failed', - )); - $this->assertInstanceOf(AppwriteDestination::class, $successor); - $this->assertNotSame([], $this->errorMessages($destination)); - $this->assertSame([], $this->errorMessages($successor)); - $this->assertSame([], $failedWrites); - $this->assertSame('provisioning', $database->getAttribute('status')); - $this->assertSame('migration-shared', $database->getAttribute('migrationId')); - $this->assertSame('attempt-third', $database->getAttribute('migrationAttemptId')); - } finally { - $this->removeSQLiteFiles($path); - } - } - - private function createDestination( - UtopiaDatabase $database, - string $migrationId, - string $migrationAttemptId, - callable $getRecoverableOwner, - ): AppwriteDestination { - return new AppwriteDestination( - project: 'destination-project', - endpoint: 'http://example.test/v1', - key: 'test-key', - dbForProject: $database, - getDatabasesDB: static fn (UtopiaDocument $document): UtopiaDatabase => $database, - collectionStructure: ['attributes' => [], 'indexes' => []], - dbForPlatform: $database, - projectInternalId: '1', - owner: new ProvisioningOwner($migrationId, $migrationAttemptId), - getRecoverableOwner: $getRecoverableOwner, - onDuplicate: OnDuplicate::Fail, - ); - } - - private function runTransfer( - UtopiaDatabase $database, - AppwriteDestination $destination, - ?callable $callback = null, - ): void { - $source = new class () extends MockSource { - #[Override] - public function supportsDatabaseStatus(): bool - { - return true; - } - }; - $source->pushMockResource(new DatabaseResource( - id: 'database', - name: 'Database', - type: 'tablesdb', - database: 'source-dsn', - databaseStatus: 'ready', - )); - - $transfer = new Transfer($source, $destination); - $database->getAuthorization()->skip( - static function () use ($callback, $destination, $transfer): void { - $transfer->run( - [Resource::TYPE_DATABASE], - $callback ?? static function (): void { - }, - ); - $destination->success(); - }, - ); - } - - private function claimWithoutTerminalTransition( - UtopiaDatabase $database, - AppwriteDestination $destination, - ): void { - try { - $this->runTransfer( - $database, - $destination, - static function (): void { - throw new \RuntimeException('stop after provisioning claim'); - }, - ); - $this->fail('Expected transfer callback to stop before the terminal transition'); - } catch (\RuntimeException $error) { - $this->assertSame('stop after provisioning claim', $error->getMessage()); - } - } - - /** @return array{RecordingSQLiteProjectDatabase, RecordingSQLiteProjectDatabase, string} */ - private function createSharedDatabases(): array - { - $path = \tempnam(\sys_get_temp_dir(), 'migration-owner-'); - if ($path === false) { - throw new \RuntimeException('Failed to create SQLite test database'); - } - - $secondConnection = new SQLiteConnection('sqlite:'.$path, null, null, self::PDO_ATTRIBUTES); - $thirdConnection = new SQLiteConnection('sqlite:'.$path, null, null, self::PDO_ATTRIBUTES); - $secondConnection->exec('PRAGMA journal_mode = WAL'); - $secondConnection->exec('PRAGMA busy_timeout = 1000'); - $thirdConnection->exec('PRAGMA busy_timeout = 1000'); - - $second = new RecordingSQLiteProjectDatabase( - new SQLite($secondConnection), - new Cache(new None()), - ); - $third = new RecordingSQLiteProjectDatabase( - new SQLite($thirdConnection), - new Cache(new None()), - ); - $namespace = 'migration_owner_'.\uniqid(); - foreach ([$second, $third] as $database) { - $database - ->setDatabase('appwrite') - ->setNamespace($namespace); - } - - $second->create(); - $second->createCollection(Collection::create( - id: 'databases', - attributes: [ - $this->attribute('name', ColumnType::String, required: true, size: 256), - $this->attribute('enabled', ColumnType::Boolean, default: true), - $this->attribute('search', ColumnType::String, size: 16384), - $this->attribute('originalId', ColumnType::String, size: UtopiaDatabase::LENGTH_KEY), - $this->attribute('type', ColumnType::String, default: 'tablesdb', size: 128), - $this->attribute('database', ColumnType::String, size: 2000), - $this->attribute('status', ColumnType::String, size: 16), - $this->attribute('migrationId', ColumnType::String, size: UtopiaDatabase::LENGTH_KEY), - $this->attribute('migrationAttemptId', ColumnType::String, size: UtopiaDatabase::LENGTH_KEY), - ], - )); - - return [$second, $third, $path]; - } - - private function seedDatabase( - UtopiaDatabase $database, - string $status, - string $migrationId, - string $migrationAttemptId, - ): UtopiaDocument { - return $database->getAuthorization()->skip( - static fn (): UtopiaDocument => $database->createDocument('databases', new UtopiaDocument([ - '$id' => 'database', - 'name' => 'Database', - 'enabled' => true, - 'search' => 'database Database', - 'originalId' => null, - 'type' => 'tablesdb', - 'database' => '', - 'status' => $status, - 'migrationId' => $migrationId, - 'migrationAttemptId' => $migrationAttemptId, - ])), - ); - } - - private function attribute( - string $id, - ColumnType $type, - bool $required = false, - mixed $default = null, - int $size = 0, - ): UtopiaAttribute { - return UtopiaAttribute::fromArray([ - 'key' => $id, - 'type' => $type, - 'size' => $size, - 'required' => $required, - 'default' => $default, - ]); - } - - private function getDatabaseDocument(UtopiaDatabase $database): UtopiaDocument - { - return $database->getAuthorization()->skip( - static fn (): UtopiaDocument => $database->getDocument('databases', 'database'), - ); - } - - /** @return list */ - private function errorMessages(AppwriteDestination $destination): array - { - return \array_map( - static fn (\Throwable $error): string => $error->getMessage(), - $destination->getErrors(), - ); - } - - private function removeSQLiteFiles(string $path): void - { - foreach ([$path, $path.'-wal', $path.'-shm'] as $file) { - if (\is_file($file)) { - \unlink($file); - } - } - } -} diff --git a/tests/Migration/Unit/Destinations/AppwriteCheckAttributeTest.php b/tests/Migration/Unit/Destinations/AppwriteCheckAttributeTest.php index f2e0b398..a3ed5790 100644 --- a/tests/Migration/Unit/Destinations/AppwriteCheckAttributeTest.php +++ b/tests/Migration/Unit/Destinations/AppwriteCheckAttributeTest.php @@ -15,7 +15,6 @@ use Utopia\Database\Document as UtopiaDocument; use Utopia\Database\Query; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Database\Columns\Text; use Utopia\Migration\Resources\Database\Database as DatabaseResource; @@ -188,8 +187,6 @@ private function transferColumn(): array ], dbForPlatform: $database, projectInternalId: '1', - owner: new ProvisioningOwner('migration-test', 'attempt-test'), - getRecoverableOwner: static fn (UtopiaDocument $document): ?ProvisioningOwner => null, ); $transfer = new Transfer($source, $destination); diff --git a/tests/Migration/Unit/Destinations/AppwriteDatabaseStatusTest.php b/tests/Migration/Unit/Destinations/AppwriteDatabaseStatusTest.php index 13d8c79b..026c1a08 100644 --- a/tests/Migration/Unit/Destinations/AppwriteDatabaseStatusTest.php +++ b/tests/Migration/Unit/Destinations/AppwriteDatabaseStatusTest.php @@ -3,22 +3,17 @@ namespace Utopia\Tests\Unit\Destinations; use Override; -use PHPUnit\Framework\Attributes\DataProvider; use PHPUnit\Framework\TestCase; use Utopia\Cache\Adapter\Memory as MemoryCache; use Utopia\Cache\Cache; use Utopia\Database\Adapter\Memory as MemoryAdapter; use Utopia\Database\Attribute as UtopiaAttribute; -use Utopia\Database\Capability; use Utopia\Database\Collection; use Utopia\Database\Database as UtopiaDatabase; use Utopia\Database\DateTime as UtopiaDateTime; use Utopia\Database\Document as UtopiaDocument; use Utopia\Database\Exception as DatabaseException; -use Utopia\Database\Exception\Conflict as ConflictException; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; -use Utopia\Migration\Destinations\OnDuplicate; use Utopia\Migration\Exception as MigrationException; use Utopia\Migration\Exception\Finalization; use Utopia\Migration\Resource; @@ -27,25 +22,6 @@ use Utopia\Query\Schema\ColumnType; use Utopia\Tests\Unit\Adapters\MockSource; -class ReplicaMemoryAdapter extends MemoryAdapter -{ - /** @return array */ - #[Override] - public function capabilities(): array - { - return [...parent::capabilities(), Capability::TransactionRetries]; - } -} - -final class StandaloneMemoryAdapter extends ReplicaMemoryAdapter -{ - #[Override] - public function withTransaction(callable $callback): mixed - { - return $callback(); - } -} - class CountingAppwriteDestination extends AppwriteDestination { public int $runCount = 0; @@ -115,127 +91,31 @@ public function updateDocument(string $collection, string $id, UtopiaDocument $d } } -final class InterleavingProjectDatabase extends UtopiaDatabase -{ - public bool $interceptNextDatabasesReload = false; - - /** @var (callable(): void)|null */ - public $onDatabasesReload = null; - - #[Override] - public function getDocument(string $collection, string $id, array $queries = [], bool $forUpdate = false): UtopiaDocument - { - $document = parent::getDocument($collection, $id, $queries, $forUpdate); - if ( - $this->interceptNextDatabasesReload - && $collection === 'databases' - && ! $document->isEmpty() - ) { - $this->interceptNextDatabasesReload = false; - ($this->onDatabasesReload)(); - } - - return $document; - } -} - -class RecordingProjectDatabase extends UtopiaDatabase +final class ReadyFailingProjectDatabase extends UtopiaDatabase { - /** @var list, guarded?: bool}> */ - public array $databaseWrites = []; - - public bool $failReadyWrite = false; - - public bool $disappearBeforeGuardedWrite = false; + /** @var list */ + public array $readyWrites = []; /** @var list */ public array $failReadyWrites = []; - protected ?\DateTime $guardedAt = null; - - #[Override] - public function withRequestTimestamp(?\DateTime $requestTimestamp, callable $callback): mixed - { - $previous = $this->guardedAt; - $this->guardedAt = $requestTimestamp; - try { - return parent::withRequestTimestamp($requestTimestamp, $callback); - } finally { - $this->guardedAt = $previous; - } - } - - #[Override] - public function createDocument(string $collection, UtopiaDocument $document): UtopiaDocument - { - if ($collection === 'databases') { - $this->databaseWrites[] = [ - 'operation' => 'create', - 'id' => $document->getId(), - 'document' => $document->getArrayCopy(), - ]; - } - - return parent::createDocument($collection, $document); - } - #[Override] public function updateDocument(string $collection, string $id, UtopiaDocument $document): UtopiaDocument { - if ($collection === 'databases') { - $this->databaseWrites[] = [ - 'operation' => 'update', - 'id' => $id, - 'document' => $document->getArrayCopy(), - 'guarded' => $this->guardedAt !== null, - ]; - } - - if ( - ($this->failReadyWrite || \in_array($id, $this->failReadyWrites, true)) - && $collection === 'databases' - && $document->getAttribute('status') === 'ready' - ) { - throw new DatabaseException('ready status unavailable'); - } + if ($collection === 'databases' && $document->getAttribute('status') === 'ready') { + $this->readyWrites[] = $id; - if ( - $this->disappearBeforeGuardedWrite - && $collection === 'databases' - && $this->guardedAt !== null - ) { - $this->disappearBeforeGuardedWrite = false; - parent::deleteDocument($collection, $id); - - return new UtopiaDocument(); + if (\in_array($id, $this->failReadyWrites, true)) { + throw new DatabaseException('ready status unavailable'); + } } return parent::updateDocument($collection, $id, $document); } } -final class FinalizerInterleavingProjectDatabase extends RecordingProjectDatabase -{ - public ?\Closure $beforeFinalizerRead = null; - - #[Override] - public function getDocument(string $collection, string $id, array $queries = [], bool $forUpdate = false): UtopiaDocument - { - $document = parent::getDocument($collection, $id, $queries, $forUpdate); - if ($forUpdate && $collection === 'databases' && $this->beforeFinalizerRead !== null) { - $callback = $this->beforeFinalizerRead; - $this->beforeFinalizerRead = null; - $callback(); - } - - return $document; - } -} - final class AppwriteDatabaseStatusTest extends TestCase { - private const string LAPSED_UPDATED_AT = '2020-01-01 00:00:00.000'; - public function testDatabaseCreationOmitsStatusThroughLegacyAndExplicitEntrypoints(): void { foreach ([false, true] as $explicit) { @@ -258,11 +138,7 @@ public function testDatabaseCreationOmitsStatusThroughLegacyAndExplicitEntrypoin public function testDatabaseCreationPreservesLifecycleThroughLegacyAndExplicitEntrypoints(): void { foreach ([false, true] as $explicit) { - $database = new UtopiaDatabase( - new StandaloneMemoryAdapter(), - new Cache(new MemoryCache()), - ); - $this->createProjectDatabase(withStatus: true, database: $database); + $database = $this->createProjectDatabase(withStatus: true); $destination = $this->runDatabaseTransfer($database, $explicit); @@ -278,485 +154,13 @@ public function testDatabaseCreationPreservesLifecycleThroughLegacyAndExplicitEn } } - public function testCreatePersistsProvisioningAndOwnerAtomically(): void - { - $database = new RecordingProjectDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-create', - migrationAttemptId: 'attempt-create', - ); - - $this->assertSame([], $this->errorMessages($destination)); - $this->assertSame('create', $database->databaseWrites[0]['operation']); - $this->assertSame('provisioning', $database->databaseWrites[0]['document']['status']); - $this->assertSame('migration-create', $database->databaseWrites[0]['document']['migrationId']); - $this->assertSame('attempt-create', $database->databaseWrites[0]['document']['migrationAttemptId']); - } - - public function testStandaloneUniqueCreateCannotFinalizeAfterOwnerMismatch(): void - { - $database = new FinalizerInterleavingProjectDatabase( - new StandaloneMemoryAdapter(), - new Cache(new MemoryCache()), - ); - $this->createProjectDatabase(withStatus: true, database: $database); - $database->beforeFinalizerRead = static function () use ($database): void { - $database->updateDocument( - 'databases', - 'database', - new UtopiaDocument([ - 'status' => 'provisioning', - 'migrationId' => 'migration-successor', - 'migrationAttemptId' => 'attempt-successor', - ]), - ); - }; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-create', - migrationAttemptId: 'attempt-create', - success: false, - ); - $failure = $this->finalizeExpectingFailure($database, $destination); - - $terminalWrites = \array_values(\array_filter( - $database->databaseWrites, - static fn (array $write): bool => \in_array( - $write['document']['status'] ?? null, - ['ready', 'failed'], - true, - ), - )); - $created = $this->getDatabaseDocument($database); - $this->assertSame(['Database provisioning owner changed before finalization'], $this->errorMessages($destination)); - $this->assertSame($destination->getErrors(), $failure->failures); - $this->assertCount(1, $terminalWrites); - $this->assertTrue($terminalWrites[0]['guarded'] ?? false, 'The terminal write must be pinned to the timestamp read under lock'); - $this->assertSame('provisioning', $created->getAttribute('status')); - $this->assertSame('migration-successor', $created->getAttribute('migrationId')); - $this->assertSame('attempt-successor', $created->getAttribute('migrationAttemptId')); - } - - public function testAuthorizedOverwritePersistsProvisioningAndNewOwnerAtomically(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'provisioning', - migrationId: 'migration-old', - migrationAttemptId: 'attempt-old', - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-new', - migrationAttemptId: 'attempt-new', - getRecoverableOwner: static fn (UtopiaDocument $database): ProvisioningOwner => new ProvisioningOwner( - 'migration-old', - 'attempt-old', - ), - ); - - $this->assertSame([], $this->errorMessages($destination)); - $write = $database->databaseWrites[0] ?? null; - $this->assertNotNull($write); - $this->assertSame('update', $write['operation']); - $this->assertSame('provisioning', $write['document']['status']); - $this->assertSame('migration-new', $write['document']['migrationId']); - $this->assertSame('attempt-new', $write['document']['migrationAttemptId']); - } - - public function testActiveProvisioningRefusalRetainsExistingOwner(): void - { - $database = new RecordingProjectDatabase(new StandaloneMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'provisioning', - migrationId: 'migration-active', - migrationAttemptId: 'attempt-active', - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-colliding', - migrationAttemptId: 'attempt-colliding', - getRecoverableOwner: static fn (UtopiaDocument $database): ?ProvisioningOwner => null, - ); - - $existing = $this->getDatabaseDocument($database); - $this->assertNotSame([], $this->errorMessages($destination)); - $this->assertSame([], $database->databaseWrites); - $this->assertSame('provisioning', $existing->getAttribute('status')); - $this->assertSame('migration-active', $existing->getAttribute('migrationId')); - $this->assertSame('attempt-active', $existing->getAttribute('migrationAttemptId')); - } - - public function testMissingOrMismatchedOwnerRefusesRecovery(): void - { - foreach ([null, 'migration-existing'] as $owner) { - $database = new RecordingProjectDatabase(new StandaloneMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'provisioning', - migrationId: $owner, - migrationAttemptId: 'attempt-existing', - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-new', - migrationAttemptId: 'attempt-new', - getRecoverableOwner: static fn (UtopiaDocument $database): ProvisioningOwner => new ProvisioningOwner( - 'migration-terminal', - 'attempt-existing', - ), - ); - - $existing = $this->getDatabaseDocument($database); - $this->assertNotSame([], $this->errorMessages($destination)); - $this->assertSame([], $database->databaseWrites); - $this->assertSame('provisioning', $existing->getAttribute('status')); - $this->assertSame($owner, $existing->getAttribute('migrationId')); - $this->assertSame('attempt-existing', $existing->getAttribute('migrationAttemptId')); - } - } - - public function testMissingOrMismatchedAttemptRefusesRecovery(): void - { - foreach ([null, 'attempt-existing'] as $attemptId) { - $database = new RecordingProjectDatabase(new StandaloneMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'provisioning', - migrationId: 'migration-existing', - migrationAttemptId: $attemptId, - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-new', - migrationAttemptId: 'attempt-new', - getRecoverableOwner: static fn (UtopiaDocument $database): ProvisioningOwner => new ProvisioningOwner( - 'migration-existing', - 'attempt-terminal', - ), - ); - - $existing = $this->getDatabaseDocument($database); - $this->assertNotSame([], $this->errorMessages($destination)); - $this->assertSame([], $database->databaseWrites); - $this->assertSame('provisioning', $existing->getAttribute('status')); - $this->assertSame('migration-existing', $existing->getAttribute('migrationId')); - $this->assertSame($attemptId, $existing->getAttribute('migrationAttemptId')); - } - } - - public function testRecoveryRefusesReusingThePriorAttempt(): void - { - $database = new RecordingProjectDatabase(new StandaloneMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'failed', - migrationId: 'migration-shared', - migrationAttemptId: 'attempt-shared', - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-shared', - migrationAttemptId: 'attempt-shared', - getRecoverableOwner: static fn (UtopiaDocument $database): ProvisioningOwner => new ProvisioningOwner( - 'migration-shared', - 'attempt-shared', - ), - ); - - $existing = $this->getDatabaseDocument($database); - $this->assertNotSame([], $this->errorMessages($destination)); - $this->assertSame([], $database->databaseWrites); - $this->assertSame('failed', $existing->getAttribute('status')); - $this->assertSame('migration-shared', $existing->getAttribute('migrationId')); - $this->assertSame('attempt-shared', $existing->getAttribute('migrationAttemptId')); - } - - public function testSameMigrationFailedRetryRequiresTerminalAttestation(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'failed', - migrationId: 'migration-shared', - migrationAttemptId: 'attempt-active', - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-shared', - migrationAttemptId: 'attempt-retry', - getRecoverableOwner: static fn (UtopiaDocument $database): ?ProvisioningOwner => null, - ); - - $existing = $this->getDatabaseDocument($database); - $this->assertNotSame([], $this->errorMessages($destination)); - $this->assertSame([], $database->databaseWrites); - $this->assertSame('failed', $existing->getAttribute('status')); - $this->assertSame('migration-shared', $existing->getAttribute('migrationId')); - $this->assertSame('attempt-active', $existing->getAttribute('migrationAttemptId')); - } - - public function testPreExistingRecoveryUsesDocumentVersionCompareAndSet(): void - { - foreach ([new MemoryAdapter(), new StandaloneMemoryAdapter()] as $adapter) { - $database = new RecordingProjectDatabase($adapter, new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'failed', - migrationId: 'migration-old', - migrationAttemptId: 'attempt-old', - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-old', - migrationAttemptId: 'attempt-new', - getRecoverableOwner: static fn (UtopiaDocument $database): ProvisioningOwner => new ProvisioningOwner( - 'migration-old', - 'attempt-old', - ), - ); - - $existing = $this->getDatabaseDocument($database); - $this->assertSame([], $this->errorMessages($destination)); - $this->assertNotSame([], $database->databaseWrites); - $this->assertSame('ready', $existing->getAttribute('status')); - $this->assertSame('migration-old', $existing->getAttribute('migrationId')); - $this->assertSame('attempt-new', $existing->getAttribute('migrationAttemptId')); - } - } - - public function testHealthyOverwriteUsesDocumentVersionCompareAndSet(): void - { - foreach ([new MemoryAdapter(), new StandaloneMemoryAdapter()] as $adapter) { - $database = new RecordingProjectDatabase($adapter, new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'ready', - migrationId: 'migration-old', - migrationAttemptId: 'attempt-old', - ); - $database->getAuthorization()->skip( - static fn (): UtopiaDocument => $database->updateDocument( - 'databases', - 'database', - new UtopiaDocument(['name' => 'Old database']), - ), - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-new', - migrationAttemptId: 'attempt-new', - onDuplicate: OnDuplicate::Overwrite, - resourceUpdatedAt: '2099-01-01T00:00:00.000+00:00', - ); - - $existing = $this->getDatabaseDocument($database); - $this->assertSame([], $this->errorMessages($destination)); - $this->assertNotSame([], $database->databaseWrites); - $this->assertSame('ready', $existing->getAttribute('status')); - $this->assertSame('migration-new', $existing->getAttribute('migrationId')); - $this->assertSame('attempt-new', $existing->getAttribute('migrationAttemptId')); - } - } - - public function testFailedDatabaseRecoveryRequiresExactTerminalOwner(): void - { - foreach ( - [ - 'no attestation' => ['migration-terminal', null], - 'missing owner' => [null, 'migration-terminal'], - 'mismatched owner' => ['migration-existing', 'migration-terminal'], - 'unknown terminal owner' => ['migration-existing', null], - ] as [$owner, $recoverableMigrationId] - ) { - $database = new RecordingProjectDatabase(new StandaloneMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'failed', - migrationId: $owner, - migrationAttemptId: 'attempt-existing', - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-new', - migrationAttemptId: 'attempt-new', - getRecoverableOwner: static fn (UtopiaDocument $database): ?ProvisioningOwner => $recoverableMigrationId === null - ? null - : new ProvisioningOwner($recoverableMigrationId, 'attempt-existing'), - ); - - $existing = $this->getDatabaseDocument($database); - $this->assertNotSame([], $this->errorMessages($destination)); - $this->assertSame([], $database->databaseWrites); - $this->assertSame('failed', $existing->getAttribute('status')); - $this->assertSame($owner, $existing->getAttribute('migrationId')); - $this->assertSame('attempt-existing', $existing->getAttribute('migrationAttemptId')); - } - } - - public function testReadyWriteFailureRetainsProvisioningOwner(): void - { - $database = new RecordingProjectDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $database->failReadyWrite = true; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-ready-failure', - migrationAttemptId: 'attempt-ready-failure', - success: false, - ); - $failure = $this->finalizeExpectingFailure($database, $destination); - - $created = $this->getDatabaseDocument($database); - $errors = $destination->getErrors(); - $this->assertCount(1, $errors); - $this->assertSame($errors, $failure->failures); - $this->assertSame(Resource::TYPE_DATABASE, $errors[0]->getResourceName()); - $this->assertSame(Transfer::GROUP_DATABASES, $errors[0]->getResourceGroup()); - $this->assertSame('database', $errors[0]->getResourceId()); - $this->assertSame('ready status unavailable', $errors[0]->getMessage()); - $this->assertInstanceOf(DatabaseException::class, $errors[0]->getPrevious()); - $this->assertSame('provisioning', $created->getAttribute('status')); - $this->assertSame('migration-ready-failure', $created->getAttribute('migrationId')); - $this->assertSame('attempt-ready-failure', $created->getAttribute('migrationAttemptId')); - } - - public function testDatabaseDisappearanceBeforeReadyWriteLosesFinalizationOwnership(): void - { - $database = new RecordingProjectDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $database->disappearBeforeGuardedWrite = true; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-disappearing-ready', - migrationAttemptId: 'attempt-disappearing-ready', - success: false, - ); - $failure = $this->finalizeExpectingFailure($database, $destination); - - $readyWrites = \array_values(\array_filter( - $database->databaseWrites, - static fn (array $write): bool => ($write['document']['status'] ?? null) === 'ready', - )); - - $this->assertSame(['Database provisioning owner changed before finalization'], $this->errorMessages($destination)); - $this->assertSame($destination->getErrors(), $failure->failures); - $this->assertCount(1, $readyWrites); - $this->assertTrue($readyWrites[0]['guarded'] ?? false, 'The ready write must be pinned to the timestamp read under lock'); - $this->assertTrue($this->getDatabaseDocument($database)->isEmpty()); - } - - public function testDatabaseDisappearanceBeforeRecoveryClaimLosesOwnership(): void - { - $database = new RecordingProjectDatabase(new StandaloneMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $seeded = $this->seedDatabase( - $database, - status: 'failed', - migrationId: 'migration-old', - migrationAttemptId: 'attempt-old', - ); - $database->databaseWrites = []; - $database->disappearBeforeGuardedWrite = true; - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-new', - migrationAttemptId: 'attempt-new', - getRecoverableOwner: static fn (UtopiaDocument $database): ProvisioningOwner => new ProvisioningOwner( - 'migration-old', - 'attempt-old', - ), - success: false, - ); - - $claimWrites = \array_values(\array_filter( - $database->databaseWrites, - static fn (array $write): bool => ($write['document']['status'] ?? null) === 'provisioning', - )); - - $this->assertSame( - ['Database database provisioning owner changed before it could be claimed'], - $this->errorMessages($destination), - ); - $this->assertCount(1, $claimWrites); - $this->assertSame( - [true], - \array_map( - static fn (array $write): bool => $write['guarded'] ?? false, - $claimWrites, - ), - ); - $this->assertTrue($this->getDatabaseDocument($database)->isEmpty()); - $this->assertNull($database->findCollection('database_'.$seeded->getSequence())); - } - public function testDatabaseFinalizersRunOnlyAfterSuccess(): void { - $database = new RecordingProjectDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); + $database = $this->createProjectDatabase(withStatus: true); - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-finalizing', - migrationAttemptId: 'attempt-finalizing', - success: false, - ); + $destination = $this->runDatabaseTransfer($database, explicit: false, success: false); - $provisioning = $this->getDatabaseDocument($database); - $this->assertSame('provisioning', $provisioning->getAttribute('status')); - $this->assertSame('migration-finalizing', $provisioning->getAttribute('migrationId')); - $this->assertSame('attempt-finalizing', $provisioning->getAttribute('migrationAttemptId')); + $this->assertSame('provisioning', $this->getDatabaseDocument($database)->getAttribute('status')); $database->getAuthorization()->skip($destination->success(...)); @@ -765,34 +169,25 @@ public function testDatabaseFinalizersRunOnlyAfterSuccess(): void public function testReadyFinalizationAttemptsEveryDatabaseAndReportsEveryFailure(): void { - $database = new RecordingProjectDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); + $database = new ReadyFailingProjectDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); $this->createProjectDatabase(withStatus: true, database: $database); $database->failReadyWrites = ['database-first', 'database-third']; $destination = $this->runDatabaseTransfer( $database, explicit: false, - migrationId: 'migration-finalizers', - migrationAttemptId: 'attempt-finalizers', databaseIds: ['database-first', 'database-second', 'database-third'], success: false, ); $failure = $this->finalizeExpectingFailure($database, $destination); - $readyAttempts = \array_values(\array_map( - static fn (array $write): string => $write['id'], - \array_filter( - $database->databaseWrites, - static fn (array $write): bool => ($write['document']['status'] ?? null) === 'ready', - ), - )); $errors = $destination->getErrors(); $errorIds = \array_map( static fn (MigrationException $error): string => $error->getResourceId(), $errors, ); - $this->assertSame(['database-first', 'database-second', 'database-third'], $readyAttempts); + $this->assertSame(['database-first', 'database-second', 'database-third'], $database->readyWrites); $this->assertSame(['database-first', 'database-third'], $errorIds); $this->assertSame($errors, $failure->failures); $this->assertSame('provisioning', $this->getDatabaseDocument($database, 'database-first')->getAttribute('status')); @@ -803,7 +198,7 @@ public function testReadyFinalizationAttemptsEveryDatabaseAndReportsEveryFailure public function testReloadFailureMarksTheDatabaseFailed(): void { $database = new ReloadFailingProjectDatabase( - new StandaloneMemoryAdapter(), + new MemoryAdapter(), new Cache(new MemoryCache()), ); $this->createProjectDatabase(withStatus: true, database: $database); @@ -815,8 +210,6 @@ public function testReloadFailureMarksTheDatabaseFailed(): void $this->assertNotSame([], $this->errorMessages($destination)); $this->assertStringContainsString('Failed to reload created database', $this->errorMessages($destination)[0]); $this->assertSame('failed', $created->getAttribute('status')); - $this->assertSame('migration-current', $created->getAttribute('migrationId')); - $this->assertSame('attempt-current', $created->getAttribute('migrationAttemptId')); $this->assertNull( $database->findCollection('database_'.$created->getSequence()), 'A reload failure must not leave a backing collection behind the metadata document', @@ -826,7 +219,7 @@ public function testReloadFailureMarksTheDatabaseFailed(): void public function testFailedDatabaseRetrySucceedsUnderOnDuplicateFail(): void { $database = new ReloadFailingProjectDatabase( - new ReplicaMemoryAdapter(), + new MemoryAdapter(), new Cache(new MemoryCache()), ); $this->createProjectDatabase(withStatus: true, database: $database); @@ -840,23 +233,12 @@ public function testFailedDatabaseRetrySucceedsUnderOnDuplicateFail(): void $database->findCollection('database_'.$failed->getSequence()), ); - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-current', - migrationAttemptId: 'attempt-recovery', - getRecoverableOwner: static fn (UtopiaDocument $existing): ProvisioningOwner => new ProvisioningOwner( - 'migration-current', - 'attempt-current', - ), - ); + $destination = $this->runDatabaseTransfer($database, explicit: false); $recovered = $this->getDatabaseDocument($database); $this->assertSame([], $this->errorMessages($destination)); $this->assertSame('ready', $recovered->getAttribute('status')); $this->assertSame($failed->getSequence(), $recovered->getSequence()); - $this->assertSame('migration-current', $recovered->getAttribute('migrationId')); - $this->assertSame('attempt-recovery', $recovered->getAttribute('migrationAttemptId')); $this->assertNotNull( $database->findCollection('database_'.$recovered->getSequence()), 'A Fail retry must recreate the backing collection for a previously failed database', @@ -866,7 +248,7 @@ public function testFailedDatabaseRetrySucceedsUnderOnDuplicateFail(): void public function testProvisioningDatabaseRetrySucceedsUnderOnDuplicateFail(): void { $database = new StrandedProvisioningProjectDatabase( - new ReplicaMemoryAdapter(), + new MemoryAdapter(), new Cache(new MemoryCache()), ); $this->createProjectDatabase(withStatus: true, database: $database); @@ -887,143 +269,23 @@ public function testProvisioningDatabaseRetrySucceedsUnderOnDuplicateFail(): voi ); $database->failDatabasesWrites = false; - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-recovery', - migrationAttemptId: 'attempt-recovery', - getRecoverableOwner: static fn (UtopiaDocument $existing): ProvisioningOwner => new ProvisioningOwner( - 'migration-current', - 'attempt-current', - ), - ); + $destination = $this->runDatabaseTransfer($database, explicit: false); $recovered = $this->getDatabaseDocument($database); $this->assertSame([], $this->errorMessages($destination)); $this->assertSame('ready', $recovered->getAttribute('status')); $this->assertSame($stranded->getSequence(), $recovered->getSequence()); - $this->assertSame('migration-recovery', $recovered->getAttribute('migrationId')); - $this->assertSame('attempt-recovery', $recovered->getAttribute('migrationAttemptId')); $this->assertNotNull( $database->findCollection('database_'.$recovered->getSequence()), 'A Fail retry must recover a database stranded in provisioning, not keep colliding with its metadata id', ); } - public function testConcurrentMigrationDoesNotRecoverAnActivelyProvisioningDatabase(): void - { - $database = new InterleavingProjectDatabase( - new MemoryAdapter(), - new Cache(new MemoryCache()), - ); - $this->createProjectDatabase(withStatus: true, database: $database); - - $second = null; - $statusDuringOverlap = null; - $collectionExistsDuringOverlap = null; - $database->onDatabasesReload = function () use ( - $database, - &$second, - &$statusDuringOverlap, - &$collectionExistsDuringOverlap, - ): void { - $second = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-second', - migrationAttemptId: 'attempt-second', - ); - $provisioning = $this->getDatabaseDocument($database); - $statusDuringOverlap = $provisioning->getAttribute('status'); - $collectionExistsDuringOverlap = $database - ->findCollection('database_'.$provisioning->getSequence()) !== null; - }; - $database->interceptNextDatabasesReload = true; - - $first = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-first', - migrationAttemptId: 'attempt-first', - ); - - $created = $this->getDatabaseDocument($database); - $this->assertSame([], $this->errorMessages($first)); - $this->assertInstanceOf(CountingAppwriteDestination::class, $second); - $this->assertNotSame([], $this->errorMessages($second)); - $this->assertStringContainsString('terminal-owner attestation', $this->errorMessages($second)[0]); - $this->assertSame('provisioning', $statusDuringOverlap); - $this->assertFalse($collectionExistsDuringOverlap); - $this->assertSame('ready', $created->getAttribute('status')); - $this->assertSame('migration-first', $created->getAttribute('migrationId')); - $this->assertSame('attempt-first', $created->getAttribute('migrationAttemptId')); - $this->assertNotNull($database->findCollection('database_'.$created->getSequence())); - } - - public function testAuthorizedOverwriteReplacesTerminalOwner(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'provisioning', - migrationId: 'migration-terminal', - migrationAttemptId: 'attempt-terminal', - ); - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-successor', - migrationAttemptId: 'attempt-successor', - getRecoverableOwner: static fn (UtopiaDocument $database): ProvisioningOwner => new ProvisioningOwner( - 'migration-terminal', - 'attempt-terminal', - ), - ); - - $recovered = $this->getDatabaseDocument($database); - $this->assertSame([], $this->errorMessages($destination)); - $this->assertSame('ready', $recovered->getAttribute('status')); - $this->assertSame('migration-successor', $recovered->getAttribute('migrationId')); - $this->assertSame('attempt-successor', $recovered->getAttribute('migrationAttemptId')); - } - - /** @return array */ - public static function unknownAttributePolicies(): array + public function testIncompleteDatabasesAreRecoveredImmediately(): void { - return [ - 'unknown attributes rejected' => [false], - 'unknown attributes dropped' => [true], - ]; - } - - #[DataProvider('unknownAttributePolicies')] - public function testStatusOnlySchemaImportsWithoutProvisioningOwner(bool $dropUnknownAttributes): void - { - $database = new RecordingProjectDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); - $database->setDropUnknownAttributes($dropUnknownAttributes); - $this->createStatusOnlyProjectDatabase($database); - - $destination = $this->runDatabaseTransfer($database, explicit: false); - - $created = $this->getDatabaseDocument($database); - $this->assertSame([], $this->errorMessages($destination)); - $this->assertSame('create', $database->databaseWrites[0]['operation']); - $this->assertSame('provisioning', $database->databaseWrites[0]['document']['status']); - $this->assertArrayNotHasKey('migrationId', $database->databaseWrites[0]['document']); - $this->assertArrayNotHasKey('migrationAttemptId', $database->databaseWrites[0]['document']); - $this->assertSame('ready', $created->getAttribute('status')); - $this->assertNotNull($database->findCollection('database_'.$created->getSequence())); - } - - public function testStatusOnlySchemaRecoversIncompleteDatabasesWithoutOwner(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createStatusOnlyProjectDatabase($database); - $this->seedLegacyDatabase($database, 'database-provisioning', 'provisioning', withCollection: true); - $this->seedLegacyDatabase($database, 'database-failed', 'failed'); - $database->databaseWrites = []; + $database = $this->createProjectDatabase(withStatus: true); + $provisioning = $this->seedIncompleteDatabase($database, 'database-provisioning', 'provisioning', withCollection: true, updatedAt: $this->secondsAgo(1)); + $failed = $this->seedIncompleteDatabase($database, 'database-failed', 'failed', updatedAt: $this->secondsAgo(1)); $destination = $this->runDatabaseTransfer( $database, @@ -1032,460 +294,14 @@ public function testStatusOnlySchemaRecoversIncompleteDatabasesWithoutOwner(): v ); $this->assertSame([], $this->errorMessages($destination)); - foreach (['database-provisioning', 'database-failed'] as $databaseId) { - $recovered = $this->getDatabaseDocument($database, $databaseId); - $this->assertSame('ready', $recovered->getAttribute('status'), $databaseId); - $this->assertNotNull($database->findCollection('database_'.$recovered->getSequence()), $databaseId); - } - $this->assertNotSame([], $database->databaseWrites); - foreach ($database->databaseWrites as $write) { - $this->assertArrayNotHasKey('migrationId', $write['document']); - $this->assertArrayNotHasKey('migrationAttemptId', $write['document']); + foreach ([$provisioning, $failed] as $seeded) { + $recovered = $this->getDatabaseDocument($database, $seeded->getId()); + $this->assertSame('ready', $recovered->getAttribute('status'), $seeded->getId()); + $this->assertSame($seeded->getSequence(), $recovered->getSequence(), $seeded->getId()); + $this->assertNotNull($database->findCollection('database_'.$recovered->getSequence()), $seeded->getId()); } } - /** @return array */ - public static function legacyProvisioningResolutions(): array - { - return [ - 'fail keeps the destination' => [OnDuplicate::Fail, '', 'Legacy database'], - 'skip keeps the destination' => [OnDuplicate::Skip, '', 'Legacy database'], - 'overwrite with an older source keeps the destination' => [OnDuplicate::Overwrite, '2000-01-01T00:00:00.000+00:00', 'Legacy database'], - 'overwrite with a newer source replaces the destination' => [OnDuplicate::Overwrite, '2099-01-01T00:00:00.000+00:00', 'Database'], - ]; - } - - #[DataProvider('legacyProvisioningResolutions')] - public function testLegacyProvisioningDatabaseIsRecoveredAndClaimed( - OnDuplicate $onDuplicate, - string $resourceUpdatedAt, - string $expectedName, - ): void { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $seeded = $this->seedLegacyDatabase( - $database, - 'database', - 'provisioning', - name: 'Legacy database', - withCollection: true, - ); - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - onDuplicate: $onDuplicate, - resourceUpdatedAt: $resourceUpdatedAt, - ); - - $recovered = $this->getDatabaseDocument($database); - $this->assertSame([], $this->errorMessages($destination)); - $this->assertSame('ready', $recovered->getAttribute('status')); - $this->assertSame($expectedName, $recovered->getAttribute('name')); - $this->assertSame($seeded->getSequence(), $recovered->getSequence()); - $this->assertSame('migration-current', $recovered->getAttribute('migrationId')); - $this->assertSame('attempt-current', $recovered->getAttribute('migrationAttemptId')); - $this->assertNotNull($database->findCollection('database_'.$recovered->getSequence())); - } - - public function testLegacyProvisioningDatabaseMissingItsCollectionIsRecreated(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $seeded = $this->seedLegacyDatabase($database, 'database', 'provisioning'); - - $destination = $this->runDatabaseTransfer($database, explicit: false); - - $recovered = $this->getDatabaseDocument($database); - $this->assertSame([], $this->errorMessages($destination)); - $this->assertSame('ready', $recovered->getAttribute('status')); - $this->assertSame($seeded->getSequence(), $recovered->getSequence()); - $this->assertSame('migration-current', $recovered->getAttribute('migrationId')); - $this->assertSame('attempt-current', $recovered->getAttribute('migrationAttemptId')); - $this->assertNotNull( - $database->findCollection('database_'.$recovered->getSequence()), - 'A recovered database must never be marked ready without its backing collection', - ); - } - - public function testLegacyFailedDatabaseIsOverwrittenAndClaimed(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $seeded = $this->seedLegacyDatabase($database, 'database', 'failed', name: 'Legacy database'); - - $destination = $this->runDatabaseTransfer($database, explicit: false); - - $recovered = $this->getDatabaseDocument($database); - $this->assertSame([], $this->errorMessages($destination)); - $this->assertSame('ready', $recovered->getAttribute('status')); - $this->assertSame('Database', $recovered->getAttribute('name')); - $this->assertSame($seeded->getSequence(), $recovered->getSequence()); - $this->assertSame('migration-current', $recovered->getAttribute('migrationId')); - $this->assertSame('attempt-current', $recovered->getAttribute('migrationAttemptId')); - $this->assertNotNull($database->findCollection('database_'.$recovered->getSequence())); - } - - public function testOwnedDatabaseWithoutAttestationIsRefusedWhileLegacyDatabaseRecovers(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'provisioning', - migrationId: 'migration-active', - migrationAttemptId: 'attempt-active', - ); - $this->seedLegacyDatabase($database, 'database-legacy', 'failed'); - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - databaseIds: ['database', 'database-legacy'], - ); - - $owned = $this->getDatabaseDocument($database); - $legacy = $this->getDatabaseDocument($database, 'database-legacy'); - $this->assertSame( - ['Database database recovery requires exact terminal-owner attestation and a fresh attempt'], - $this->errorMessages($destination), - ); - $this->assertSame('provisioning', $owned->getAttribute('status')); - $this->assertSame('migration-active', $owned->getAttribute('migrationId')); - $this->assertSame('attempt-active', $owned->getAttribute('migrationAttemptId')); - $this->assertSame('ready', $legacy->getAttribute('status')); - $this->assertSame('migration-current', $legacy->getAttribute('migrationId')); - $this->assertSame('attempt-current', $legacy->getAttribute('migrationAttemptId')); - } - - public function testLegacyRecoveryYieldsToAClaimThatLandsFirst(): void - { - $database = new InterleavingProjectDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedLegacyDatabase($database, 'database', 'failed'); - $database->onDatabasesReload = static function () use ($database): void { - $database->updateDocument('databases', 'database', new UtopiaDocument([ - 'status' => 'provisioning', - 'migrationId' => 'migration-rival', - 'migrationAttemptId' => 'attempt-rival', - ])); - }; - $database->interceptNextDatabasesReload = true; - - $destination = $this->runDatabaseTransfer($database, explicit: false); - - $claimed = $this->getDatabaseDocument($database); - $this->assertSame( - ['Database database recovery owner changed before it could be claimed'], - $this->errorMessages($destination), - ); - $this->assertSame('provisioning', $claimed->getAttribute('status')); - $this->assertSame('migration-rival', $claimed->getAttribute('migrationId')); - $this->assertSame('attempt-rival', $claimed->getAttribute('migrationAttemptId')); - } - - public function testRecoveryClaimUnderPreservedDatesStillMovesTheTimestampForward(): void - { - $database = new class (new ReplicaMemoryAdapter(), new Cache(new MemoryCache())) extends RecordingProjectDatabase { - #[Override] - public function setPreserveDates(bool $preserve): static - { - return parent::setPreserveDates(true); - } - }; - $this->createProjectDatabase(withStatus: true, database: $database); - $seeded = $this->seedDatabase( - $database, - status: 'failed', - migrationId: 'migration-old', - migrationAttemptId: 'attempt-old', - ); - $readAt = (string) $seeded->getUpdatedAt(); - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-new', - migrationAttemptId: 'attempt-new', - getRecoverableOwner: static fn (UtopiaDocument $database): ProvisioningOwner => new ProvisioningOwner( - 'migration-old', - 'attempt-old', - ), - resourceUpdatedAt: '2000-01-01T00:00:00.000+00:00', - success: false, - ); - - $claimed = $this->getDatabaseDocument($database); - $this->assertSame([], $this->errorMessages($destination)); - $this->assertSame('attempt-new', $claimed->getAttribute('migrationAttemptId')); - $this->assertGreaterThan( - new \DateTime($readAt), - new \DateTime((string) $claimed->getUpdatedAt()), - 'The claim must move $updatedAt forward whatever the preserveDates setting', - ); - - $this->expectException(ConflictException::class); - $database->getAuthorization()->skip( - static fn (): UtopiaDocument => $database->withRequestTimestamp( - new \DateTime($readAt), - static fn (): UtopiaDocument => $database->updateDocument( - 'databases', - 'database', - new UtopiaDocument(['status' => 'ready']), - ), - ), - ); - } - - public function testGuardedWriteThatDoesNotAdvanceTheTimestampIsRolledBack(): void - { - $database = new class (new ReplicaMemoryAdapter(), new Cache(new MemoryCache())) extends RecordingProjectDatabase { - #[Override] - public function updateDocument(string $collection, string $id, UtopiaDocument $document): UtopiaDocument - { - if ($collection !== 'databases' || $this->guardedAt === null) { - return parent::updateDocument($collection, $id, $document); - } - - return $this->withPreserveDates( - true, - fn (): UtopiaDocument => parent::updateDocument( - $collection, - $id, - $document->setAttribute('$updatedAt', '2000-01-01T00:00:00.000+00:00'), - ), - ); - } - }; - $this->createProjectDatabase(withStatus: true, database: $database); - $this->seedDatabase( - $database, - status: 'failed', - migrationId: 'migration-old', - migrationAttemptId: 'attempt-old', - ); - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - migrationId: 'migration-new', - migrationAttemptId: 'attempt-new', - getRecoverableOwner: static fn (UtopiaDocument $database): ProvisioningOwner => new ProvisioningOwner( - 'migration-old', - 'attempt-old', - ), - ); - - $existing = $this->getDatabaseDocument($database); - $this->assertSame( - ['Database database provisioning write did not move its update timestamp forward'], - $this->errorMessages($destination), - ); - $this->assertSame('failed', $existing->getAttribute('status')); - $this->assertSame('migration-old', $existing->getAttribute('migrationId')); - $this->assertSame('attempt-old', $existing->getAttribute('migrationAttemptId')); - } - - - public function testOwnerlessProvisioningDatabaseWithinTheLeaseIsNotRecovered(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createStatusOnlyProjectDatabase($database); - $seeded = $this->seedLegacyDatabase( - $database, - 'database', - 'provisioning', - name: 'Provisioning database', - withCollection: true, - updatedAt: $this->secondsAgo(5), - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer($database, explicit: false); - - $untouched = $this->getDatabaseDocument($database); - $this->assertSame( - ['Database database names no owner and another migration may still be provisioning it; it becomes recoverable after 86400 seconds without an update'], - $this->errorMessages($destination), - ); - $this->assertSame('provisioning', $untouched->getAttribute('status')); - $this->assertSame('Provisioning database', $untouched->getAttribute('name')); - $this->assertSame($seeded->getUpdatedAt(), $untouched->getUpdatedAt()); - $this->assertSame([], $database->databaseWrites, 'A refused database must not be written, and success() must flip nothing'); - } - - public function testOwnerlessFailedDatabaseWithinTheLeaseIsNotOverwritten(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createStatusOnlyProjectDatabase($database); - $seeded = $this->seedLegacyDatabase( - $database, - 'database', - 'failed', - name: 'Failed database', - updatedAt: $this->secondsAgo(5), - ); - $database->databaseWrites = []; - - $destination = $this->runDatabaseTransfer($database, explicit: false); - - $untouched = $this->getDatabaseDocument($database); - $this->assertSame( - ['Database database names no owner and another migration may still be provisioning it; it becomes recoverable after 86400 seconds without an update'], - $this->errorMessages($destination), - ); - $this->assertSame('failed', $untouched->getAttribute('status')); - $this->assertSame('Failed database', $untouched->getAttribute('name')); - $this->assertSame($seeded->getUpdatedAt(), $untouched->getUpdatedAt()); - $this->assertSame([], $database->databaseWrites, 'A refused database must not be overwritten'); - } - - public function testOwnerlessDatabasesPastTheLeaseAreRecovered(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createStatusOnlyProjectDatabase($database); - $this->seedLegacyDatabase( - $database, - 'database-provisioning', - 'provisioning', - withCollection: true, - updatedAt: $this->secondsAgo(AppwriteDestination::DEFAULT_PROVISIONING_LEASE + 1), - ); - $this->seedLegacyDatabase( - $database, - 'database-failed', - 'failed', - updatedAt: $this->secondsAgo(AppwriteDestination::DEFAULT_PROVISIONING_LEASE + 1), - ); - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - databaseIds: ['database-provisioning', 'database-failed'], - ); - - $this->assertSame([], $this->errorMessages($destination)); - foreach (['database-provisioning', 'database-failed'] as $databaseId) { - $recovered = $this->getDatabaseDocument($database, $databaseId); - $this->assertSame('ready', $recovered->getAttribute('status'), $databaseId); - $this->assertNotNull($database->findCollection('database_'.$recovered->getSequence()), $databaseId); - } - } - - public function testTheOwnerlessProvisioningClaimRestartsTheLease(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createStatusOnlyProjectDatabase($database); - $seeded = $this->seedLegacyDatabase( - $database, - 'database', - 'provisioning', - withCollection: true, - updatedAt: $this->secondsAgo(AppwriteDestination::DEFAULT_PROVISIONING_LEASE + 1), - ); - - $claiming = $this->runDatabaseTransfer($database, explicit: false, success: false); - - $claimed = $this->getDatabaseDocument($database); - $this->assertSame([], $this->errorMessages($claiming)); - $this->assertSame('provisioning', $claimed->getAttribute('status')); - $this->assertGreaterThan( - new \DateTime((string) $seeded->getUpdatedAt()), - new \DateTime((string) $claimed->getUpdatedAt()), - 'The claim must restart the lease on the row it recovered', - ); - - $second = $this->runDatabaseTransfer($database, explicit: false, success: false); - - $refused = $this->getDatabaseDocument($database); - $this->assertSame( - ['Database database names no owner and another migration may still be provisioning it; it becomes recoverable after 86400 seconds without an update'], - $this->errorMessages($second), - 'A migration arriving inside the restarted lease must not recover the same database', - ); - $this->assertSame('provisioning', $refused->getAttribute('status')); - $this->assertSame((string) $claimed->getUpdatedAt(), (string) $refused->getUpdatedAt()); - } - - public function testProvisioningLeaseMovesTheOwnerlessRecoveryBoundary(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createStatusOnlyProjectDatabase($database); - $this->seedLegacyDatabase($database, 'database-lapsed', 'provisioning', withCollection: true, updatedAt: $this->secondsAgo(61)); - $this->seedLegacyDatabase($database, 'database-live', 'provisioning', withCollection: true, updatedAt: $this->secondsAgo(1)); - - $destination = $this->runDatabaseTransfer( - $database, - explicit: false, - databaseIds: ['database-lapsed', 'database-live'], - provisioningLease: 60, - ); - - $this->assertSame( - ['Database database-live names no owner and another migration may still be provisioning it; it becomes recoverable after 60 seconds without an update'], - $this->errorMessages($destination), - ); - $this->assertSame('ready', $this->getDatabaseDocument($database, 'database-lapsed')->getAttribute('status')); - $this->assertSame('provisioning', $this->getDatabaseDocument($database, 'database-live')->getAttribute('status')); - } - - public function testZeroProvisioningLeaseRecoversAnOwnerlessDatabaseAtOnce(): void - { - $database = new RecordingProjectDatabase(new ReplicaMemoryAdapter(), new Cache(new MemoryCache())); - $this->createStatusOnlyProjectDatabase($database); - $this->seedLegacyDatabase($database, 'database', 'provisioning', withCollection: true, updatedAt: $this->secondsAgo(1)); - - $destination = $this->runDatabaseTransfer($database, explicit: false, provisioningLease: 0); - - $this->assertSame([], $this->errorMessages($destination)); - $this->assertSame('ready', $this->getDatabaseDocument($database)->getAttribute('status')); - } - - public function testNegativeProvisioningLeaseIsRejected(): void - { - $database = new UtopiaDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); - - $this->expectException(\InvalidArgumentException::class); - $this->expectExceptionMessage('Provisioning lease must not be negative'); - - new AppwriteDestination( - project: 'destination-project', - endpoint: 'http://example.test/v1', - key: 'test-key', - dbForProject: $database, - getDatabasesDB: static fn (UtopiaDocument $document): UtopiaDatabase => $database, - collectionStructure: ['attributes' => [], 'indexes' => []], - dbForPlatform: $database, - projectInternalId: '1', - owner: new ProvisioningOwner('migration-current', 'attempt-current'), - getRecoverableOwner: static fn (UtopiaDocument $document): ?ProvisioningOwner => null, - provisioningLease: -1, - ); - } - - public function testOwnerlessRecoveryYieldsToAnUpdateThatLandsFirst(): void - { - $database = new InterleavingProjectDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); - $this->createStatusOnlyProjectDatabase($database); - $this->seedLegacyDatabase($database, 'database', 'failed', updatedAt: $this->secondsAgo(AppwriteDestination::DEFAULT_PROVISIONING_LEASE + 1)); - $database->onDatabasesReload = static function () use ($database): void { - $database->updateDocument('databases', 'database', new UtopiaDocument([ - 'name' => 'Database claimed elsewhere', - ])); - }; - $database->interceptNextDatabasesReload = true; - - $destination = $this->runDatabaseTransfer($database, explicit: false); - - $claimed = $this->getDatabaseDocument($database); - $this->assertSame( - ['Database database names no owner and was updated before it could be claimed'], - $this->errorMessages($destination), - ); - $this->assertSame('failed', $claimed->getAttribute('status')); - $this->assertSame('Database claimed elsewhere', $claimed->getAttribute('name')); - } - private function createProjectDatabase(bool $withStatus, ?UtopiaDatabase $database = null): UtopiaDatabase { $database ??= new UtopiaDatabase( @@ -1508,8 +324,6 @@ private function createProjectDatabase(bool $withStatus, ?UtopiaDatabase $databa if ($withStatus) { $attributes[] = $this->attribute('status', ColumnType::String, size: 16); - $attributes[] = $this->attribute('migrationId', ColumnType::String, size: UtopiaDatabase::LENGTH_KEY); - $attributes[] = $this->attribute('migrationAttemptId', ColumnType::String, size: UtopiaDatabase::LENGTH_KEY); } $database->createCollection(Collection::create( @@ -1520,58 +334,21 @@ private function createProjectDatabase(bool $withStatus, ?UtopiaDatabase $databa return $database; } - private function seedDatabase( - UtopiaDatabase $database, - string $status, - ?string $migrationId, - ?string $migrationAttemptId, - ): UtopiaDocument { - $document = [ - '$id' => 'database', - 'name' => 'Database', - 'enabled' => true, - 'search' => 'database Database', - 'originalId' => null, - 'type' => 'tablesdb', - 'database' => '', - 'status' => $status, - ]; - if ($migrationId !== null) { - $document['migrationId'] = $migrationId; - } - if ($migrationAttemptId !== null) { - $document['migrationAttemptId'] = $migrationAttemptId; - } - - return $database->getAuthorization()->skip( - static fn (): UtopiaDocument => $database->createDocument('databases', new UtopiaDocument($document)), - ); - } - - private function createStatusOnlyProjectDatabase(UtopiaDatabase $database): UtopiaDatabase - { - $this->createProjectDatabase(withStatus: false, database: $database); - $database->createAttribute('databases', $this->attribute('status', ColumnType::String, size: 16)); - - return $database; - } - - private function seedLegacyDatabase( + private function seedIncompleteDatabase( UtopiaDatabase $database, string $databaseId, string $status, - string $name = 'Database', bool $withCollection = false, - string $updatedAt = self::LAPSED_UPDATED_AT, + string $updatedAt = '', ): UtopiaDocument { $seeded = $database->getAuthorization()->skip( static fn (): UtopiaDocument => $database->withPreserveDates( true, static fn (): UtopiaDocument => $database->createDocument('databases', new UtopiaDocument([ '$id' => $databaseId, - 'name' => $name, + 'name' => 'Database', 'enabled' => true, - 'search' => $databaseId.' '.$name, + 'search' => $databaseId.' Database', 'originalId' => null, 'type' => 'tablesdb', 'database' => '', @@ -1610,17 +387,14 @@ private function attribute( ]); } + /** + * @param list $databaseIds + */ private function runDatabaseTransfer( UtopiaDatabase $database, bool $explicit, - string $migrationId = 'migration-current', - string $migrationAttemptId = 'attempt-current', - ?callable $getRecoverableOwner = null, - OnDuplicate $onDuplicate = OnDuplicate::Fail, - string $resourceUpdatedAt = '', array $databaseIds = ['database'], bool $success = true, - int $provisioningLease = AppwriteDestination::DEFAULT_PROVISIONING_LEASE, ): CountingAppwriteDestination { $source = new class () extends MockSource { #[Override] @@ -1633,7 +407,6 @@ public function supportsDatabaseStatus(): bool $source->pushMockResource(new DatabaseResource( id: $databaseId, name: 'Database', - updatedAt: $resourceUpdatedAt, type: 'tablesdb', database: 'source-dsn', databaseStatus: 'ready', @@ -1649,11 +422,6 @@ public function supportsDatabaseStatus(): bool collectionStructure: ['attributes' => [], 'indexes' => []], dbForPlatform: $database, projectInternalId: '1', - owner: new ProvisioningOwner($migrationId, $migrationAttemptId), - getRecoverableOwner: $getRecoverableOwner - ?? static fn (UtopiaDocument $document): ?ProvisioningOwner => null, - onDuplicate: $onDuplicate, - provisioningLease: $provisioningLease, ); $transfer = new Transfer($source, $destination); @@ -1671,7 +439,6 @@ static function (): void { parentResourceInternalId: '', parentResourceType: '', ); - } else { $transfer->run( [Resource::TYPE_DATABASE], diff --git a/tests/Migration/Unit/Destinations/AppwriteDestinationDsnTest.php b/tests/Migration/Unit/Destinations/AppwriteDestinationDsnTest.php index 1b67b193..00673615 100644 --- a/tests/Migration/Unit/Destinations/AppwriteDestinationDsnTest.php +++ b/tests/Migration/Unit/Destinations/AppwriteDestinationDsnTest.php @@ -13,7 +13,6 @@ use Utopia\Database\Database as UtopiaDatabase; use Utopia\Database\Document as UtopiaDocument; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; use Utopia\Migration\Destinations\OnDuplicate; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Database\Database as DatabaseResource; @@ -97,8 +96,6 @@ private function transferDatabase(?callable $getDatabaseDSN): UtopiaDatabase collectionStructure: ['attributes' => [], 'indexes' => []], dbForPlatform: $database, projectInternalId: '1', - owner: new ProvisioningOwner('migration-test', 'attempt-test'), - getRecoverableOwner: static fn (UtopiaDocument $document): ?ProvisioningOwner => null, onDuplicate: OnDuplicate::Fail, getDatabaseDSN: $getDatabaseDSN, ); diff --git a/tests/Migration/Unit/Destinations/AppwriteFinalizationTest.php b/tests/Migration/Unit/Destinations/AppwriteFinalizationTest.php index b771c044..57ac904c 100644 --- a/tests/Migration/Unit/Destinations/AppwriteFinalizationTest.php +++ b/tests/Migration/Unit/Destinations/AppwriteFinalizationTest.php @@ -14,7 +14,6 @@ use Utopia\Database\Document as UtopiaDocument; use Utopia\Database\Query; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; use Utopia\Migration\Destinations\OnDuplicate; use Utopia\Migration\Exception\Aborted; use Utopia\Migration\Exception as MigrationException; @@ -138,7 +137,7 @@ public function testSuccessFinalizesNothingWhenTheSourceSwallowedTheAbort(): voi $this->seedTablesWithLegacyColumns($database); $abort = new Aborted('Migration attempt was superseded'); - $destination = $this->destination($database, 'attempt-overwrite', OnDuplicate::Overwrite); + $destination = $this->destination($database, OnDuplicate::Overwrite); try { $this->importSwallowingAbort($database, $destination, $this->newerSchemaWithoutLegacyColumns(), $abort); @@ -203,7 +202,7 @@ public function testOneFailedReadyFlipStillSweepsEveryOverwrittenTable(): void $this->seedTablesWithLegacyColumns($database); $database->failReadyWrites = ['first']; - $destination = $this->destination($database, 'attempt-overwrite', OnDuplicate::Overwrite); + $destination = $this->destination($database, OnDuplicate::Overwrite); $this->import($database, $destination, $this->newerSchemaWithoutLegacyColumns()); $failure = $this->finalize($database, $destination); @@ -224,7 +223,7 @@ public function testOrphanSweepFailureStillSweepsTheOtherTablesAndThrows(): void $database = $this->projectDatabase(); $this->seedTablesWithLegacyColumns($database); - $destination = $this->destination($database, 'attempt-overwrite', OnDuplicate::Overwrite); + $destination = $this->destination($database, OnDuplicate::Overwrite); $this->import($database, $destination, $this->newerSchemaWithoutLegacyColumns()); $database->failAttributeScanOf = $this->table($database, 'first', 'first-items'); @@ -303,7 +302,7 @@ private function seedTablesWithLegacyColumns(FailingFinalizationDatabase $databa ); } - $destination = $this->destination($database, 'attempt-seed'); + $destination = $this->destination($database); $this->import($database, $destination, $resources); $this->assertNull($this->finalize($database, $destination)); @@ -348,7 +347,6 @@ private function databaseResource(string $id, string $name = 'Database', string private function destination( FailingFinalizationDatabase $database, - string $attemptId = 'attempt-current', OnDuplicate $onDuplicate = OnDuplicate::Fail, ): AppwriteDestination { return new AppwriteDestination( @@ -370,8 +368,6 @@ private function destination( ], dbForPlatform: $database, projectInternalId: '1', - owner: new ProvisioningOwner('migration-finalization', $attemptId), - getRecoverableOwner: static fn (UtopiaDocument $document): ?ProvisioningOwner => null, onDuplicate: $onDuplicate, ); } @@ -485,8 +481,6 @@ private function projectDatabase(): FailingFinalizationDatabase $this->attribute('type', ColumnType::String, default: 'tablesdb', size: 128), $this->attribute('database', ColumnType::String, size: 2000), $this->attribute('status', ColumnType::String, size: 16), - $this->attribute('migrationId', ColumnType::String, size: UtopiaDatabase::LENGTH_KEY), - $this->attribute('migrationAttemptId', ColumnType::String, size: UtopiaDatabase::LENGTH_KEY), ], )); diff --git a/tests/Migration/Unit/Destinations/AppwriteIndexLengthsTest.php b/tests/Migration/Unit/Destinations/AppwriteIndexLengthsTest.php index 0f19fd3b..49433c06 100644 --- a/tests/Migration/Unit/Destinations/AppwriteIndexLengthsTest.php +++ b/tests/Migration/Unit/Destinations/AppwriteIndexLengthsTest.php @@ -13,7 +13,6 @@ use Utopia\Database\Database as UtopiaDatabase; use Utopia\Database\Document as UtopiaDocument; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; use Utopia\Migration\Destinations\OnDuplicate; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Database\Columns\Text; @@ -206,8 +205,6 @@ private function transferIndex( ], dbForPlatform: $database, projectInternalId: '1', - owner: new ProvisioningOwner('migration-test', 'attempt-test'), - getRecoverableOwner: static fn (UtopiaDocument $document): ?ProvisioningOwner => null, onDuplicate: $onDuplicate, ); diff --git a/tests/Migration/Unit/Destinations/AppwriteRelationshipOnDeleteTest.php b/tests/Migration/Unit/Destinations/AppwriteRelationshipOnDeleteTest.php index d8fb6c3c..9174b83c 100644 --- a/tests/Migration/Unit/Destinations/AppwriteRelationshipOnDeleteTest.php +++ b/tests/Migration/Unit/Destinations/AppwriteRelationshipOnDeleteTest.php @@ -17,7 +17,6 @@ use Utopia\Database\RelationshipDeleteAction; use Utopia\Database\RelationshipType; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; use Utopia\Migration\Destinations\OnDuplicate; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Database\Columns\Relationship; @@ -212,8 +211,6 @@ private function transfer(UtopiaDatabase $database, OnDuplicate $onDuplicate, Re ], dbForPlatform: $database, projectInternalId: '1', - owner: new ProvisioningOwner('migration-test', 'attempt-test'), - getRecoverableOwner: static fn (UtopiaDocument $document): ?ProvisioningOwner => null, onDuplicate: $onDuplicate, ); diff --git a/tests/Migration/Unit/Destinations/AppwriteRowImportTest.php b/tests/Migration/Unit/Destinations/AppwriteRowImportTest.php index 70801643..e5edb58a 100644 --- a/tests/Migration/Unit/Destinations/AppwriteRowImportTest.php +++ b/tests/Migration/Unit/Destinations/AppwriteRowImportTest.php @@ -13,7 +13,6 @@ use Utopia\Database\Document as UtopiaDocument; use Utopia\Database\Query; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Database\Columns\Text; use Utopia\Migration\Resources\Database\Database as DatabaseResource; @@ -206,8 +205,6 @@ private function destination(): AppwriteDestination ], dbForPlatform: $database, projectInternalId: '1', - owner: new ProvisioningOwner('migration-row-import', 'attempt-current'), - getRecoverableOwner: static fn (UtopiaDocument $document): ?ProvisioningOwner => null, ); } @@ -232,8 +229,6 @@ private function projectDatabase(): UtopiaDatabase $this->attribute('type', ColumnType::String, default: 'tablesdb', size: 128), $this->attribute('database', ColumnType::String, size: 2000), $this->attribute('status', ColumnType::String, size: 16), - $this->attribute('migrationId', ColumnType::String, size: UtopiaDatabase::LENGTH_KEY), - $this->attribute('migrationAttemptId', ColumnType::String, size: UtopiaDatabase::LENGTH_KEY), ], )); diff --git a/tests/Migration/Unit/Destinations/AppwriteSpatialIndexTest.php b/tests/Migration/Unit/Destinations/AppwriteSpatialIndexTest.php index 143c0fb4..4d7cbca8 100644 --- a/tests/Migration/Unit/Destinations/AppwriteSpatialIndexTest.php +++ b/tests/Migration/Unit/Destinations/AppwriteSpatialIndexTest.php @@ -18,7 +18,6 @@ use Utopia\Database\Index as UtopiaIndex; use Utopia\Database\Query; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Database\Columns\Point; use Utopia\Migration\Resources\Database\Database as DatabaseResource; @@ -134,8 +133,6 @@ private function transferSpatialIndex(): array ], dbForPlatform: $project, projectInternalId: '1', - owner: new ProvisioningOwner('migration-test', 'attempt-test'), - getRecoverableOwner: static fn (UtopiaDocument $document): ?ProvisioningOwner => null, ); $transfer = new Transfer($source, $destination); diff --git a/tests/Migration/Unit/Destinations/ProvisioningOwnerTest.php b/tests/Migration/Unit/Destinations/ProvisioningOwnerTest.php deleted file mode 100644 index 54100d30..00000000 --- a/tests/Migration/Unit/Destinations/ProvisioningOwnerTest.php +++ /dev/null @@ -1,36 +0,0 @@ - */ - public static function emptyIdentifiers(): array - { - return [ - 'migration identifier' => ['', 'attempt'], - 'attempt identifier' => ['migration', ''], - ]; - } - - #[DataProvider('emptyIdentifiers')] - public function testIdentifiersMustBeNonEmpty(string $migrationId, string $attemptId): void - { - $this->expectException(\InvalidArgumentException::class); - - new ProvisioningOwner($migrationId, $attemptId); - } - - public function testEqualityRequiresTheExactPair(): void - { - $owner = new ProvisioningOwner('migration', 'attempt'); - - $this->assertTrue($owner->equals(new ProvisioningOwner('migration', 'attempt'))); - $this->assertFalse($owner->equals(new ProvisioningOwner('migration-other', 'attempt'))); - $this->assertFalse($owner->equals(new ProvisioningOwner('migration', 'attempt-other'))); - } -} diff --git a/tests/Migration/Unit/Destinations/TransfersColumns.php b/tests/Migration/Unit/Destinations/TransfersColumns.php index 3b219170..1e00ae23 100644 --- a/tests/Migration/Unit/Destinations/TransfersColumns.php +++ b/tests/Migration/Unit/Destinations/TransfersColumns.php @@ -13,7 +13,6 @@ use Utopia\Database\Document as UtopiaDocument; use Utopia\Database\Query; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; -use Utopia\Migration\Destinations\Appwrite\ProvisioningOwner; use Utopia\Migration\Destinations\OnDuplicate; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Database\Column; @@ -163,8 +162,6 @@ private function destination(UtopiaDatabase $database, OnDuplicate $onDuplicate) ], dbForPlatform: $database, projectInternalId: '1', - owner: new ProvisioningOwner('migration-test', 'attempt-test'), - getRecoverableOwner: static fn (UtopiaDocument $document): ?ProvisioningOwner => null, onDuplicate: $onDuplicate, ); } diff --git a/tests/Migration/Unit/MigrationCLITest.php b/tests/Migration/Unit/MigrationCLITest.php index 7b6aa9c1..43455435 100644 --- a/tests/Migration/Unit/MigrationCLITest.php +++ b/tests/Migration/Unit/MigrationCLITest.php @@ -11,11 +11,9 @@ use Utopia\Cache\Cache; use Utopia\Database\Adapter\Memory as MemoryAdapter; use Utopia\Database\Attribute; -use Utopia\Database\Capability; use Utopia\Database\Collection; use Utopia\Database\Database; use Utopia\Database\Document; -use Utopia\Migration\Destination; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Database\Database as DatabaseResource; use Utopia\Migration\Source; @@ -25,13 +23,10 @@ final class TestMigrationCLI extends \MigrationCLI { - /** @param list $arguments */ public function __construct( - array $arguments, private readonly Database $database, private readonly ?Source $injectedSource = null, ) { - parent::__construct($arguments); } #[Override] @@ -65,16 +60,6 @@ public function getErrors(): array } } -final class TransactionalMemoryAdapter extends MemoryAdapter -{ - /** @return array */ - #[Override] - public function capabilities(): array - { - return [...parent::capabilities(), Capability::TransactionRetries]; - } -} - #[BackupGlobals(true)] final class MigrationCLITest extends TestCase { @@ -83,98 +68,9 @@ final class MigrationCLITest extends TestCase /** @var array */ private array $filterState = []; - public function testHelpDocumentsTheAcceptedRecoveryOptionsAndNotTheRetiredOne(): void - { - // The neighbouring test proves --recover-provisioning is refused; this is the - // separate property that it is not advertised, so nobody is invited to try it. - $this->assertStringContainsString('--recover-migration-id=', \MigrationCLI::getHelp()); - $this->assertStringContainsString('--recover-migration-attempt-id=', \MigrationCLI::getHelp()); - $this->assertStringNotContainsString('--recover-provisioning', \MigrationCLI::getHelp()); - $this->assertStringContainsString('--migration-id=', \MigrationCLI::getHelp()); - $this->assertStringContainsString('--migration-attempt-id=', \MigrationCLI::getHelp()); - } - - public function testIncompleteDatabaseRecoveryRequiresExactTerminalMigrationIdentifier(): void - { - $cases = [ - 'absent' => [[], false], - 'bare migration' => [['--recover-migration-id', '--recover-migration-attempt-id=attempt-terminal'], false], - 'bare attempt' => [['--recover-migration-id=migration-terminal', '--recover-migration-attempt-id'], false], - 'empty migration' => [['--recover-migration-id=', '--recover-migration-attempt-id=attempt-terminal'], false], - 'empty attempt' => [['--recover-migration-id=migration-terminal', '--recover-migration-attempt-id='], false], - 'migration only' => [['--recover-migration-id=migration-terminal'], false], - 'attempt only' => [['--recover-migration-attempt-id=attempt-terminal'], false], - 'migration mismatch' => [['--recover-migration-id=migration-other', '--recover-migration-attempt-id=attempt-terminal'], false], - 'attempt mismatch' => [['--recover-migration-id=migration-terminal', '--recover-migration-attempt-id=attempt-other'], false], - 'retired unsafe option' => [['--recover-provisioning'], false], - 'exact' => [[ - '--recover-migration-id=migration-terminal', - '--recover-migration-attempt-id=attempt-terminal', - ], true], - ]; - - foreach (['provisioning', 'failed'] as $status) { - foreach ($cases as [$recoveryArguments, $recover]) { - $database = $this->createProjectDatabase($status); - $arguments = [ - 'MigrationCLI.php', - '--migration-id=migration-current', - '--migration-attempt-id=attempt-current', - ...$recoveryArguments, - ]; - $cli = new TestMigrationCLI($arguments, $database); - - $destination = $cli->getDestination(); - $this->runTransfer($database, $destination); - - $created = $database->getAuthorization()->skip( - static fn (): Document => $database->getDocument('databases', 'database'), - ); - - if (! $recover) { - $this->assertNotSame([], $destination->getErrors()); - $this->assertSame($status, $created->getAttribute('status')); - $this->assertSame('migration-terminal', $created->getAttribute('migrationId')); - $this->assertSame('attempt-terminal', $created->getAttribute('migrationAttemptId')); - $this->assertNull($database->findCollection('database_'.$created->getSequence())); - continue; - } - - $this->assertSame([], $destination->getErrors()); - $this->assertSame('ready', $created->getAttribute('status')); - $this->assertSame('migration-current', $created->getAttribute('migrationId')); - $this->assertSame('attempt-current', $created->getAttribute('migrationAttemptId')); - $this->assertNotNull($database->findCollection('database_'.$created->getSequence())); - } - } - } - - public function testAppwriteDestinationRequiresMigrationIdentifier(): void - { - $cli = new TestMigrationCLI(['MigrationCLI.php'], $this->createProjectDatabase()); - - $this->expectException(\InvalidArgumentException::class); - $this->expectExceptionMessage('--migration-id is required for an Appwrite destination'); - - $cli->getDestination(); - } - - public function testAppwriteDestinationRequiresMigrationAttemptIdentifier(): void - { - $cli = new TestMigrationCLI( - ['MigrationCLI.php', '--migration-id=migration-current'], - $this->createProjectDatabase(), - ); - - $this->expectException(\InvalidArgumentException::class); - $this->expectExceptionMessage('--migration-attempt-id is required for an Appwrite destination'); - - $cli->getDestination(); - } - public function testStartFinalizesSuccessfulAppwriteDatabase(): void { - $database = $this->createProjectDatabase(status: null); + $database = $this->createProjectDatabase(); $source = $this->createSource(new DatabaseResource( id: 'database', name: 'Database', @@ -182,29 +78,19 @@ public function testStartFinalizesSuccessfulAppwriteDatabase(): void database: 'source-dsn', databaseStatus: 'ready', )); - $cli = new TestMigrationCLI( - [ - 'MigrationCLI.php', - '--migration-id=migration-current', - '--migration-attempt-id=attempt-current', - ], - $database, - $source, - ); + $cli = new TestMigrationCLI($database, $source); $cli->start(); $created = $database->getDocument('databases', 'database'); $this->assertSame([], $cli->getErrors()); $this->assertSame('ready', $created->getAttribute('status')); - $this->assertSame('migration-current', $created->getAttribute('migrationId')); - $this->assertSame('attempt-current', $created->getAttribute('migrationAttemptId')); $this->assertNotNull($database->findCollection('database_'.$created->getSequence())); } public function testStartFinalizesSuccessfulResourcesWhenDestinationHasErrors(): void { - $database = $this->createProjectDatabase(status: null); + $database = $this->createProjectDatabase(); $valid = new DatabaseResource( id: 'database', name: 'Database', @@ -219,15 +105,7 @@ public function testStartFinalizesSuccessfulResourcesWhenDestinationHasErrors(): database: 'source-dsn', databaseStatus: 'ready', ); - $cli = new TestMigrationCLI( - [ - 'MigrationCLI.php', - '--migration-id=migration-current', - '--migration-attempt-id=attempt-current', - ], - $database, - $this->createSource($valid, $invalid), - ); + $cli = new TestMigrationCLI($database, $this->createSource($valid, $invalid)); $cli->start(); @@ -239,8 +117,6 @@ public function testStartFinalizesSuccessfulResourcesWhenDestinationHasErrors(): $this->assertSame(Resource::STATUS_SUCCESS, $valid->getStatus()); $this->assertSame(Resource::STATUS_ERROR, $invalid->getStatus()); $this->assertSame('ready', $created->getAttribute('status')); - $this->assertSame('migration-current', $created->getAttribute('migrationId')); - $this->assertSame('attempt-current', $created->getAttribute('migrationAttemptId')); $this->assertNotNull($database->findCollection('database_'.$created->getSequence())); } @@ -329,9 +205,9 @@ private function createMetadataDatabase(): Database return $database; } - private function createProjectDatabase(?string $status = 'provisioning'): Database + private function createProjectDatabase(): Database { - $database = new Database(new TransactionalMemoryAdapter(), new Cache(new MemoryCache())); + $database = new Database(new MemoryAdapter(), new Cache(new MemoryCache())); $database ->setDatabase('appwrite') ->setNamespace('_project'); @@ -346,26 +222,8 @@ private function createProjectDatabase(?string $status = 'provisioning'): Databa Attribute::string(key: 'type', size: 128, default: 'tablesdb'), Attribute::string(key: 'database', size: 2000), Attribute::string(key: 'status', size: 16), - Attribute::string(key: 'migrationId', size: Database::LENGTH_KEY), - Attribute::string(key: 'migrationAttemptId', size: Database::LENGTH_KEY), ], )); - if ($status !== null) { - $database->getAuthorization()->skip( - static fn (): Document => $database->createDocument('databases', new Document([ - '$id' => 'database', - 'name' => 'Database', - 'enabled' => true, - 'search' => 'database Database', - 'originalId' => null, - 'type' => 'tablesdb', - 'database' => '', - 'status' => $status, - 'migrationId' => 'migration-terminal', - 'migrationAttemptId' => 'attempt-terminal', - ])), - ); - } return $database; } @@ -393,33 +251,6 @@ public function supportsDatabaseStatus(): bool return $source; } - private function runTransfer(Database $database, Destination $destination): void - { - $source = new class () extends MockSource { - #[Override] - public function supportsDatabaseStatus(): bool - { - return true; - } - }; - $source->pushMockResource(new DatabaseResource( - id: 'database', - name: 'Database', - type: 'tablesdb', - database: 'source-dsn', - databaseStatus: 'ready', - )); - - $transfer = new Transfer($source, $destination); - $database->getAuthorization()->skip( - static function () use ($destination, $transfer): void { - $transfer->run([Resource::TYPE_DATABASE], static function (): void { - }); - $destination->success(); - }, - ); - } - #[Override] protected function setUp(): void { From c9a06ac00da59f712667d4a37f1f32a34a6943dd Mon Sep 17 00:00:00 2001 From: Jake Barnby Date: Wed, 14 Oct 2026 18:04:53 +1300 Subject: [PATCH 2/8] refactor(destination)!: remove the abort mechanism and finalize in run() again The claim protocol is gone, so nothing needs a caller to persist a claim between the transfer and its terminal side effects. Deferring finalization to success() only left every consumer that kept run() as its last call with databases stuck in provisioning, and the reporting added to catch that (cleanUp() and rerun errors, Exception\Finalization) existed only to paper over the split. run() flips provisioned databases to ready and then sweeps overwrite orphans, exactly as 2.x did, and success(), error() and cleanUp() are plain hooks again. Exception\Aborted served only appwrite's Superseded, which is being deleted; the library never threw it. The Transfer latch, Destination::markAborted() and the 52 pasted catch-and-rethrow blocks in the sources go with it, so a callback exception is handled by each source as in 2.x. Kept from the 3.x line: a failed ready flip is recorded with addError() and the sweep still runs after it. The column/index test helper now runs one transfer with columns before indexes, since a split run would sweep the index between the two halves. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 10 + bin/MigrationCLI.php | 2 - src/Migration/Destination.php | 10 - src/Migration/Destinations/Appwrite.php | 143 +----- src/Migration/Exception/Aborted.php | 15 - src/Migration/Exception/Finalization.php | 36 -- src/Migration/Sources/Appwrite.php | 69 --- src/Migration/Sources/CSV.php | 3 - src/Migration/Sources/Firebase.php | 11 - src/Migration/Sources/JSON.php | 3 - src/Migration/Sources/NHost.php | 17 - src/Migration/Sources/Supabase.php | 7 - src/Migration/Target.php | 18 +- src/Migration/Transfer.php | 37 +- .../AppwriteDatabaseStatusTest.php | 77 ++-- ...Test.php => AppwriteRunCompletionTest.php} | 241 ++-------- ...base.php => FailingCompletionDatabase.php} | 4 +- .../Unit/Destinations/TransfersColumns.php | 23 +- .../Unit/General/TransferAbortTest.php | 163 ------- .../Unit/Sources/AppwriteAbortTest.php | 413 ------------------ .../Unit/Sources/SourceAbortTest.php | 328 -------------- 21 files changed, 121 insertions(+), 1509 deletions(-) delete mode 100644 src/Migration/Exception/Aborted.php delete mode 100644 src/Migration/Exception/Finalization.php rename tests/Migration/Unit/Destinations/{AppwriteFinalizationTest.php => AppwriteRunCompletionTest.php} (57%) rename tests/Migration/Unit/Destinations/{FailingFinalizationDatabase.php => FailingCompletionDatabase.php} (94%) delete mode 100644 tests/Migration/Unit/General/TransferAbortTest.php delete mode 100644 tests/Migration/Unit/Sources/AppwriteAbortTest.php delete mode 100644 tests/Migration/Unit/Sources/SourceAbortTest.php diff --git a/CHANGELOG.md b/CHANGELOG.md index f5cb58dc..d9821859 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,16 @@ wait. - The standalone CLI no longer accepts `--migration-id`, `--migration-attempt-id`, `--recover-migration-id`, `--recover-migration-attempt-id` or `-h`/`--help`. +- `Destinations\Appwrite::run()` finalizes again, as in 2.x: once the transfer returns it flips the databases it + provisioned to `ready`, then runs the overwrite orphan sweep. + - `success()`, `error()` and `cleanUp()` are no-op lifecycle hooks again. Callers no longer call `success()` to + finalize, and `cleanUp()` no longer reports a skipped `success()`. + - A failed `ready` flip is recorded with `addError()`; a failed orphan sweep throws from `run()`. + - `Utopia\Migration\Exception\Finalization` is removed. +- The abort mechanism is removed. + - `Utopia\Migration\Exception\Aborted` and `Destination::markAborted()` are removed. + - `Transfer::run()` no longer latches an exception thrown from its progress callback, and the library sources no + longer rethrow one: each source handles it as in 2.x. ## 3.0.0 (unreleased) diff --git a/bin/MigrationCLI.php b/bin/MigrationCLI.php index e105bf04..a4f95df3 100644 --- a/bin/MigrationCLI.php +++ b/bin/MigrationCLI.php @@ -441,8 +441,6 @@ function () { $this->drawFrame(); } ); - - $this->destination->success(); } protected function loadEnvironment(): void diff --git a/src/Migration/Destination.php b/src/Migration/Destination.php index 802bdd7f..14776727 100644 --- a/src/Migration/Destination.php +++ b/src/Migration/Destination.php @@ -96,16 +96,6 @@ public function runWithResourceSelector( } } - /** - * Report an abort that a source swallowed and Transfer latched, before Transfer rethrows it; - * an abort that propagates straight out of run() never reaches here. A destination that - * defers its terminal state to success() drops that state here, so a later success() - * finalizes nothing, exactly as after a run that threw. - */ - public function markAborted(): void - { - } - /** * Import Resources * diff --git a/src/Migration/Destinations/Appwrite.php b/src/Migration/Destinations/Appwrite.php index 00a95472..da9ae990 100644 --- a/src/Migration/Destinations/Appwrite.php +++ b/src/Migration/Destinations/Appwrite.php @@ -51,7 +51,6 @@ use Utopia\Database\Validator\UID; use Utopia\Migration\Destination; use Utopia\Migration\Exception; -use Utopia\Migration\Exception\Finalization; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Auth\AuthMethods; use Utopia\Migration\Resources\Auth\Hash; @@ -208,15 +207,13 @@ class Appwrite extends Destination private array $processedTwoWayPairs = []; /** - * Databases created this run, held in `provisioning` until success() flips - * each to `ready`. Keyed by database id. + * Databases created this run, held in `provisioning` until the run finishes + * successfully. The end-of-run sweep flips each to `ready`. Keyed by database id. * * @var array */ private array $provisioningDatabases = []; - private bool $finalizationPending = false; - /** Whether the destination project's database metadata supports lifecycle status. */ private ?bool $databaseStatusSupported = null; @@ -305,11 +302,7 @@ private function getSupportForDatabaseStatus(): bool return $this->databaseStatusSupported = false; } - /** - * Transfer resources without committing terminal destination state. The - * caller must first persist its own finalization claim, then invoke - * success() while that generation is still authoritative. - */ + /** Orphan cleanup runs only after a successful migration — a mid-run throw preserves the destination as-is. */ #[Override] public function run( array $resources, @@ -317,58 +310,12 @@ public function run( string $rootResourceId = '', string $rootResourceType = '', ): void { - $this->reportSkippedFinalization(); $this->resetRunState(); parent::run($resources, $callback, $rootResourceId, $rootResourceType); - $this->finalizationPending = $this->provisioningDatabases !== [] || $this->orphansByTable !== []; - } - - /** - * Finalize destination state only after the caller has fenced terminal ownership. - * Only a run that returned is finalized: an interrupted one keeps its databases in - * `provisioning` for a later attempt to recover. - * - * @throws Finalization - */ - #[Override] - public function success(): void - { - if (! $this->finalizationPending) { - return; - } - $this->finalizationPending = false; - - // Flip status before the orphan sweep so a cleanup failure can't strand databases in `provisioning`. - $failures = $this->markProvisionedDatabasesReady(); - \array_push($failures, ...$this->cleanupOverwriteOrphans()); - - if ($failures !== []) { - throw new Finalization($failures); - } - } - - #[Override] - public function error(): void - { - $this->finalizationPending = false; - parent::error(); - } - - /** - * An aborted run is not finalized, whatever the source did with the abort: its databases stay - * in `provisioning` for a later attempt, as after an interrupted run. - */ - #[Override] - public function markAborted(): void - { - $this->finalizationPending = false; - } - - #[Override] - public function cleanUp(): void - { - $this->reportSkippedFinalization(); - parent::cleanUp(); + // parent::run() returning means every resource transferred, so the databases are usable. + // Flip status before the orphan sweep so a cleanup failure can't strand them in `provisioning`. + $this->markProvisionedDatabasesReady(); + $this->cleanupOverwriteOrphans(); } /** Per-run state must not leak across run() invocations on a reused instance. */ @@ -378,63 +325,24 @@ private function resetRunState(): void $this->orphansByTable = []; $this->processedTwoWayPairs = []; $this->provisioningDatabases = []; - $this->finalizationPending = false; } - /** A returned run that is neither finalized nor abandoned would otherwise leave its databases unusable without a trace. */ - private function reportSkippedFinalization(): void + private function markProvisionedDatabasesReady(): void { - if (! $this->finalizationPending) { - return; - } - $this->finalizationPending = false; - - $databaseIds = \array_map(\strval(...), \array_keys($this->provisioningDatabases)); - foreach ($this->orphansByTable as $tracked) { - $databaseIds[] = $tracked['database']->getId(); - } - - foreach (\array_unique($databaseIds) as $databaseId) { - $this->addError(new Exception( - resourceName: Resource::TYPE_DATABASE, - resourceGroup: Transfer::GROUP_DATABASES, - resourceId: $databaseId, - message: 'Database was not finalized: success() was not called after the transfer', - code: Exception::CODE_INTERNAL, - )); - } - } - - /** - * @return list - */ - private function markProvisionedDatabasesReady(): array - { - $failures = []; foreach (\array_keys($this->provisioningDatabases) as $databaseId) { try { $this->setDatabaseStatus($databaseId, self::DATABASE_STATUS_READY); } catch (\Throwable $error) { - $failures[] = $this->recordFinalizationFailure(Resource::TYPE_DATABASE, (string) $databaseId, $error); + $this->addError(new Exception( + resourceName: Resource::TYPE_DATABASE, + resourceGroup: Transfer::GROUP_DATABASES, + resourceId: (string) $databaseId, + message: $error->getMessage(), + code: $error->getCode(), + previous: $error, + )); } } - - return $failures; - } - - private function recordFinalizationFailure(string $resourceName, string $resourceId, \Throwable $error): Exception - { - $failure = new Exception( - resourceName: $resourceName, - resourceGroup: Transfer::GROUP_DATABASES, - resourceId: $resourceId, - message: $error->getMessage(), - code: $error->getCode(), - previous: $error, - ); - $this->addError($failure); - - return $failure; } private function setDatabaseStatus(string $databaseId, string $status): void @@ -2322,23 +2230,12 @@ private function trackOrphanCandidate( $this->orphansByTable[$tableId][$kind][] = $key; } - /** - * End-of-migration sweep — only visits tables that had no rows (rest were cleaned per-table in createRecord). - * - * @return list - */ - private function cleanupOverwriteOrphans(): array + /** End-of-migration sweep — only visits tables that had no rows (rest were cleaned per-table in createRecord). */ + private function cleanupOverwriteOrphans(): void { - $failures = []; - foreach ($this->orphansByTable as $tableId => $tracked) { - try { - $this->cleanupOverwriteOrphansForTable($tableId); - } catch (\Throwable $error) { - $failures[] = $this->recordFinalizationFailure(Resource::TYPE_TABLE, $tracked['table']->getId(), $error); - } + foreach (\array_keys($this->orphansByTable) as $tableId) { + $this->cleanupOverwriteOrphansForTable($tableId); } - - return $failures; } /** Called per-table from createRecord before rows land so Structure validator sees the post-cleanup schema. */ diff --git a/src/Migration/Exception/Aborted.php b/src/Migration/Exception/Aborted.php deleted file mode 100644 index b13710dd..00000000 --- a/src/Migration/Exception/Aborted.php +++ /dev/null @@ -1,15 +0,0 @@ - $failures - */ - public function __construct(public readonly array $failures) - { - if ($failures === []) { - throw new \InvalidArgumentException('Finalization requires at least one failure'); - } - - $subjects = \array_map( - static fn (Exception $failure): string => \trim($failure->getResourceName().' '.$failure->getResourceId()), - $failures, - ); - - parent::__construct( - resourceName: '', - resourceGroup: '', - message: 'Finalization failed for '.\implode(', ', $subjects), - code: Exception::CODE_INTERNAL, - previous: $failures[0], - ); - } -} diff --git a/src/Migration/Sources/Appwrite.php b/src/Migration/Sources/Appwrite.php index b3a967bd..a014bf86 100644 --- a/src/Migration/Sources/Appwrite.php +++ b/src/Migration/Sources/Appwrite.php @@ -24,7 +24,6 @@ use Utopia\Database\Document as UtopiaDocument; use Utopia\Database\RelationshipSide; use Utopia\Migration\Exception; -use Utopia\Migration\Exception\Aborted; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Auth\AuthMethods; use Utopia\Migration\Resources\Auth\Hash; @@ -705,8 +704,6 @@ protected function exportGroupAuth(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_USER, $resources)) { $this->exportUsers($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_USER, @@ -721,8 +718,6 @@ protected function exportGroupAuth(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_TEAM, $resources)) { $this->exportTeams($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_TEAM, @@ -737,8 +732,6 @@ protected function exportGroupAuth(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_MEMBERSHIP, $resources)) { $this->exportMemberships($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_MEMBERSHIP, @@ -753,8 +746,6 @@ protected function exportGroupAuth(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_AUTH_METHODS, $resources)) { $this->exportAuthMethods(); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_AUTH_METHODS, @@ -769,8 +760,6 @@ protected function exportGroupAuth(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_OAUTH2_PROVIDER, $resources)) { $this->exportOAuth2Providers(); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_OAUTH2_PROVIDER, @@ -785,8 +774,6 @@ protected function exportGroupAuth(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_POLICIES, $resources)) { $this->exportPolicies(); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_POLICIES, @@ -1110,8 +1097,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (\in_array($entityResource[$resourceKey], $resources)) { $callback($entityKey, $entityResource); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -1133,8 +1118,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (Resource::isSupported(array_keys(Resource::DATABASE_TYPE_RESOURCE_MAP), $resources)) { $this->exportDatabases($batchSize, $resources); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -1154,8 +1137,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (\in_array($databaseResource['entity'], $resources)) { $this->exportEntities($databaseKey, $batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -1556,8 +1537,6 @@ protected function exportGroupStorage(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_BUCKET, $resources)) { $this->exportBuckets($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -1574,8 +1553,6 @@ protected function exportGroupStorage(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_FILE, $resources)) { $this->exportFiles($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -1664,8 +1641,6 @@ private function exportFiles(int $batchSize): void $file->sizeOriginal, folder: $file->folder, )); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( resourceName: Resource::TYPE_FILE, @@ -1734,8 +1709,6 @@ protected function exportGroupFunctions(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_FUNCTION, $resources)) { $this->exportFunctions($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_FUNCTION, @@ -1749,8 +1722,6 @@ protected function exportGroupFunctions(int $batchSize, array $resources): void try { $exportOnlyActive = !\in_array(Resource::TYPE_DEPLOYMENT, $resources); $this->exportDeployments($batchSize, $exportOnlyActive); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_DEPLOYMENT, @@ -1768,8 +1739,6 @@ protected function exportGroupSites(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_SITE, $resources)) { $this->exportSites($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_SITE, @@ -1783,8 +1752,6 @@ protected function exportGroupSites(int $batchSize, array $resources): void try { $exportOnlyActive = !\in_array(Resource::TYPE_SITE_DEPLOYMENT, $resources); $this->exportSiteDeployments($batchSize, $exportOnlyActive); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_SITE_DEPLOYMENT, @@ -1876,8 +1843,6 @@ protected function exportGroupProjects(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_PROJECT_VARIABLE, $resources)) { try { $this->exportProjectVariables($batchSize); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_PROJECT_VARIABLE, @@ -1893,8 +1858,6 @@ protected function exportGroupProjects(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_PROJECT_PROTOCOLS, $resources)) { $this->exportProtocols(); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_PROJECT_PROTOCOLS, @@ -1909,8 +1872,6 @@ protected function exportGroupProjects(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_PROJECT_LABELS, $resources)) { $this->exportLabels(); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_PROJECT_LABELS, @@ -1925,8 +1886,6 @@ protected function exportGroupProjects(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_PROJECT_SERVICES, $resources)) { $this->exportServices(); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_PROJECT_SERVICES, @@ -1940,8 +1899,6 @@ protected function exportGroupProjects(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_PROJECT_EMAIL_TEMPLATE, $resources)) { try { $this->exportEmailTemplates($batchSize); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_PROJECT_EMAIL_TEMPLATE, @@ -2035,8 +1992,6 @@ protected function exportGroupDomains(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_RULE, $resources)) { try { $this->exportRules($batchSize); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_RULE, @@ -2339,8 +2294,6 @@ private function exportDeployments(int $batchSize, bool $exportOnlyActive = fals try { $this->exportDeploymentData($func, $deployment); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $func->setStatus(Resource::STATUS_ERROR, $e->getMessage()); } @@ -2363,8 +2316,6 @@ private function exportDeployments(int $batchSize, bool $exportOnlyActive = fals foreach ($response->deployments as $deployment) { try { $this->exportDeploymentData($func, $deployment); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $func->setStatus(Resource::STATUS_ERROR, $e->getMessage()); } @@ -2579,8 +2530,6 @@ protected function exportGroupMessaging(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_PROVIDER, $resources)) { $this->exportProviders($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_PROVIDER, @@ -2595,8 +2544,6 @@ protected function exportGroupMessaging(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_TOPIC, $resources)) { $this->exportTopics($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_TOPIC, @@ -2611,8 +2558,6 @@ protected function exportGroupMessaging(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_SUBSCRIBER, $resources)) { $this->exportSubscribers($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_SUBSCRIBER, @@ -2627,8 +2572,6 @@ protected function exportGroupMessaging(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_MESSAGE, $resources)) { $this->exportMessages($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_MESSAGE, @@ -2941,8 +2884,6 @@ private function exportSiteDeployments(int $batchSize, bool $exportOnlyActive = try { $this->exportSiteDeploymentData($site, $deployment); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $site->setStatus(Resource::STATUS_ERROR, $e->getMessage()); } @@ -2965,8 +2906,6 @@ private function exportSiteDeployments(int $batchSize, bool $exportOnlyActive = foreach ($response->deployments as $deployment) { try { $this->exportSiteDeploymentData($site, $deployment); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $site->setStatus(Resource::STATUS_ERROR, $e->getMessage()); } @@ -3156,8 +3095,6 @@ protected function exportGroupIntegrations(int $batchSize, array $resources): vo if (\in_array(Resource::TYPE_PLATFORM, $resources)) { try { $this->exportPlatforms($batchSize); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_PLATFORM, @@ -3172,8 +3109,6 @@ protected function exportGroupIntegrations(int $batchSize, array $resources): vo if (\in_array(Resource::TYPE_API_KEY, $resources)) { try { $this->exportApiKeys($batchSize); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_API_KEY, @@ -3188,8 +3123,6 @@ protected function exportGroupIntegrations(int $batchSize, array $resources): vo if (\in_array(Resource::TYPE_WEBHOOK, $resources)) { try { $this->exportWebhooks($batchSize); - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_WEBHOOK, @@ -3205,8 +3138,6 @@ protected function exportGroupIntegrations(int $batchSize, array $resources): vo if (\in_array(Resource::TYPE_SMTP, $resources)) { $this->exportSMTP(); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_SMTP, diff --git a/src/Migration/Sources/CSV.php b/src/Migration/Sources/CSV.php index 4f509450..643d3d5b 100644 --- a/src/Migration/Sources/CSV.php +++ b/src/Migration/Sources/CSV.php @@ -6,7 +6,6 @@ use Utopia\Database\RelationshipSide; use Utopia\Database\RelationshipType; use Utopia\Migration\Exception; -use Utopia\Migration\Exception\Aborted; use Utopia\Migration\Resource; use Utopia\Migration\Resource as UtopiaResource; use Utopia\Migration\Resources\Database\Column; @@ -132,8 +131,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (UtopiaResource::isSupported($this->getSupportedResources(), $resources)) { $this->exportRows($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( diff --git a/src/Migration/Sources/Firebase.php b/src/Migration/Sources/Firebase.php index e56ca10c..9f056e23 100644 --- a/src/Migration/Sources/Firebase.php +++ b/src/Migration/Sources/Firebase.php @@ -3,7 +3,6 @@ namespace Utopia\Migration\Sources; use Utopia\Migration\Exception; -use Utopia\Migration\Exception\Aborted; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Auth\Hash; use Utopia\Migration\Resources\Auth\User; @@ -190,8 +189,6 @@ protected function exportGroupAuth(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_USER, $resources)) { $this->exportUsers($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -290,8 +287,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_DATABASE, $resources)) { $this->callback([$database]); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -309,8 +304,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void $hasInResources = Resource::isSupported(Resource::TYPE_ROW, $resources); $this->exportDB($batchSize, $hasInResources, $database); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -651,8 +644,6 @@ protected function exportGroupStorage(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_BUCKET, $resources)) { $this->exportBuckets($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_BUCKET, @@ -667,8 +658,6 @@ protected function exportGroupStorage(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_FILE, $resources)) { $this->exportFiles($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_FILE, diff --git a/src/Migration/Sources/JSON.php b/src/Migration/Sources/JSON.php index 12425391..de41a138 100644 --- a/src/Migration/Sources/JSON.php +++ b/src/Migration/Sources/JSON.php @@ -8,7 +8,6 @@ use JsonMachine\JsonDecoder\PassThruDecoder; use Utopia\Database\Database as UtopiaDatabase; use Utopia\Migration\Exception; -use Utopia\Migration\Exception\Aborted; use Utopia\Migration\Resource as UtopiaResource; use Utopia\Migration\Resources\Database\Database; use Utopia\Migration\Resources\Database\Row; @@ -120,8 +119,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (UtopiaResource::isSupported(UtopiaResource::TYPE_ROW, $resources)) { $this->exportRows($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( diff --git a/src/Migration/Sources/NHost.php b/src/Migration/Sources/NHost.php index 545d1eff..7cd8b067 100644 --- a/src/Migration/Sources/NHost.php +++ b/src/Migration/Sources/NHost.php @@ -4,7 +4,6 @@ use PDO; use Utopia\Migration\Exception; -use Utopia\Migration\Exception\Aborted; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Auth\Hash; use Utopia\Migration\Resources\Auth\User; @@ -317,8 +316,6 @@ protected function exportGroupAuth(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_USER, $resources)) { $this->exportUsers($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_USER, @@ -384,8 +381,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_DATABASE, $resources)) { $this->exportDatabases($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -402,8 +397,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (Resource::isSupported(Resource::TYPE_TABLE, $resources)) { $this->exportTables($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -420,8 +413,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (Resource::isSupported(Resource::TYPE_COLUMN, $resources)) { $this->exportColumns($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -438,8 +429,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (Resource::isSupported(Resource::TYPE_ROW, $resources)) { $this->exportRows($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -456,8 +445,6 @@ protected function exportGroupDatabases(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_INDEX, $resources)) { $this->exportIndexes($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -806,8 +793,6 @@ protected function exportGroupStorage(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_BUCKET, $resources)) { $this->exportBuckets($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( @@ -824,8 +809,6 @@ protected function exportGroupStorage(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_FILE, $resources)) { $this->exportFiles($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError( new Exception( diff --git a/src/Migration/Sources/Supabase.php b/src/Migration/Sources/Supabase.php index 8ea3e217..8b512ab6 100644 --- a/src/Migration/Sources/Supabase.php +++ b/src/Migration/Sources/Supabase.php @@ -3,7 +3,6 @@ namespace Utopia\Migration\Sources; use Utopia\Migration\Exception; -use Utopia\Migration\Exception\Aborted; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Auth\Hash; use Utopia\Migration\Resources\Auth\User; @@ -377,8 +376,6 @@ protected function exportGroupAuth(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_USER, $resources)) { $this->exportUsers($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_BUCKET, @@ -453,8 +450,6 @@ protected function exportGroupStorage(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_BUCKET, $resources)) { $this->exportBuckets($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_BUCKET, @@ -469,8 +464,6 @@ protected function exportGroupStorage(int $batchSize, array $resources): void if (\in_array(Resource::TYPE_FILE, $resources)) { $this->exportFiles($batchSize); } - } catch (Aborted $abort) { - throw $abort; } catch (\Throwable $e) { $this->addError(new Exception( Resource::TYPE_BUCKET, diff --git a/src/Migration/Target.php b/src/Migration/Target.php index a2ede6dc..5aa72e47 100644 --- a/src/Migration/Target.php +++ b/src/Migration/Target.php @@ -230,33 +230,21 @@ public function shutdown(): void } /** - * Commit the terminal state of the last run() that returned. - * - * run() may leave that state uncommitted — the Appwrite destination keeps the databases - * it provisions in `provisioning` and defers its overwrite cleanup — so once run() has - * returned, the caller persists whatever claim must precede those side effects and then - * calls success(). Every step is attempted; if any fails, each failure is recorded with - * addError() and success() throws Exception\Finalization carrying them, so a failed - * finalization cannot pass for a completed one. Call error() instead for a run that will - * not be finalized. - * - * @throws Exception\Finalization + * Success callback */ public function success(): void { } /** - * Called instead of success() when the run failed or will not be finalized. + * Error callback */ public function error(): void { } /** - * End the lifecycle. A target that defers work to success() records an error here for - * each resource a returned run left unfinalized when neither success() nor error() was - * called, so skipping success() is reported rather than passing silently. + * Clean up callback */ public function cleanUp(): void { diff --git a/src/Migration/Transfer.php b/src/Migration/Transfer.php index 1bb63cfd..fe83f2d6 100644 --- a/src/Migration/Transfer.php +++ b/src/Migration/Transfer.php @@ -2,8 +2,6 @@ namespace Utopia\Migration; -use Utopia\Migration\Exception\Aborted; - class Transfer { public const GROUP_GENERAL = 'general'; @@ -323,9 +321,8 @@ public function getStatusCounters(): array * Transfer Resources between adapters * * @param array $resources Resources to transfer - * @param callable $callback Callback to run after transfer; throw Aborted from it to stop the transfer + * @param callable $callback Callback to run after transfer * @param string|null $rootResourceId Root resource ID, If enabled you can only transfer a single root resource - * @throws Aborted The callback stopped the transfer * @throws \Exception */ public function run( @@ -373,27 +370,10 @@ public function run( $this->resources = $computedResources; - // A source can still catch the abort as an export failure and carry on; keep it so - // later batches rethrow it and the run ends with it. - $abort = null; - $progress = static function (array $resources) use ($callback, &$abort): void { - if ($abort !== null) { - throw $abort; - } - - try { - $callback($resources); - } catch (Aborted $aborted) { - $abort = $aborted; - - throw $aborted; - } - }; - if ($this->resourceSelector !== null) { $this->destination->runWithResourceSelector( $computedResources, - $progress, + $callback, $this->resourceSelector->resourceId, $this->resourceSelector->resourceInternalId, $this->resourceSelector->resourceType, @@ -401,23 +381,18 @@ public function run( $this->resourceSelector->parentResourceInternalId, $this->resourceSelector->parentResourceType, ); - } else { - $this->destination->run($computedResources, $progress, $rootResourceId, $rootResourceType); - } - if ($abort !== null) { - $this->destination->markAborted(); - - throw $abort; + return; } + + $this->destination->run($computedResources, $callback, $rootResourceId, $rootResourceType); } /** * Transfer resources using Appwrite's canonical resource relation fields. * * @param array> $resources Resources to transfer - * @param callable $callback Callback to run after transfer; throw Aborted from it to stop the transfer - * @throws Aborted The callback stopped the transfer + * @param callable $callback Callback to run after transfer * @throws \Exception */ public function runWithResourceSelector( diff --git a/tests/Migration/Unit/Destinations/AppwriteDatabaseStatusTest.php b/tests/Migration/Unit/Destinations/AppwriteDatabaseStatusTest.php index 026c1a08..9d6332cf 100644 --- a/tests/Migration/Unit/Destinations/AppwriteDatabaseStatusTest.php +++ b/tests/Migration/Unit/Destinations/AppwriteDatabaseStatusTest.php @@ -15,7 +15,6 @@ use Utopia\Database\Exception as DatabaseException; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; use Utopia\Migration\Exception as MigrationException; -use Utopia\Migration\Exception\Finalization; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Database\Database as DatabaseResource; use Utopia\Migration\Transfer; @@ -154,20 +153,37 @@ public function testDatabaseCreationPreservesLifecycleThroughLegacyAndExplicitEn } } - public function testDatabaseFinalizersRunOnlyAfterSuccess(): void + public function testRunAloneMarksProvisionedDatabasesReady(): void { $database = $this->createProjectDatabase(withStatus: true); + $source = new class () extends MockSource { + #[Override] + public function supportsDatabaseStatus(): bool + { + return true; + } + }; + foreach (['database-first', 'database-second'] as $databaseId) { + $source->pushMockResource(new DatabaseResource( + id: $databaseId, + name: 'Database', + type: 'tablesdb', + database: 'source-dsn', + databaseStatus: 'ready', + )); + } + $transfer = new Transfer($source, $this->createDestination($database)); - $destination = $this->runDatabaseTransfer($database, explicit: false, success: false); - - $this->assertSame('provisioning', $this->getDatabaseDocument($database)->getAttribute('status')); - - $database->getAuthorization()->skip($destination->success(...)); + $database->getAuthorization()->skip( + static fn () => $transfer->run([Resource::TYPE_DATABASE], static function (): void { + }), + ); - $this->assertSame('ready', $this->getDatabaseDocument($database)->getAttribute('status')); + $this->assertSame('ready', $this->getDatabaseDocument($database, 'database-first')->getAttribute('status'), 'run() must flip every database it provisioned to ready'); + $this->assertSame('ready', $this->getDatabaseDocument($database, 'database-second')->getAttribute('status'), 'run() must flip every database it provisioned to ready'); } - public function testReadyFinalizationAttemptsEveryDatabaseAndReportsEveryFailure(): void + public function testReadyFlipAttemptsEveryDatabaseAndReportsEveryFailure(): void { $database = new ReadyFailingProjectDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); $this->createProjectDatabase(withStatus: true, database: $database); @@ -177,19 +193,15 @@ public function testReadyFinalizationAttemptsEveryDatabaseAndReportsEveryFailure $database, explicit: false, databaseIds: ['database-first', 'database-second', 'database-third'], - success: false, ); - $failure = $this->finalizeExpectingFailure($database, $destination); - $errors = $destination->getErrors(); $errorIds = \array_map( static fn (MigrationException $error): string => $error->getResourceId(), - $errors, + $destination->getErrors(), ); $this->assertSame(['database-first', 'database-second', 'database-third'], $database->readyWrites); $this->assertSame(['database-first', 'database-third'], $errorIds); - $this->assertSame($errors, $failure->failures); $this->assertSame('provisioning', $this->getDatabaseDocument($database, 'database-first')->getAttribute('status')); $this->assertSame('ready', $this->getDatabaseDocument($database, 'database-second')->getAttribute('status')); $this->assertSame('provisioning', $this->getDatabaseDocument($database, 'database-third')->getAttribute('status')); @@ -394,7 +406,6 @@ private function runDatabaseTransfer( UtopiaDatabase $database, bool $explicit, array $databaseIds = ['database'], - bool $success = true, ): CountingAppwriteDestination { $source = new class () extends MockSource { #[Override] @@ -413,20 +424,11 @@ public function supportsDatabaseStatus(): bool )); } - $destination = new CountingAppwriteDestination( - project: 'destination-project', - endpoint: 'http://example.test/v1', - key: 'test-key', - dbForProject: $database, - getDatabasesDB: static fn (UtopiaDocument $document): UtopiaDatabase => $database, - collectionStructure: ['attributes' => [], 'indexes' => []], - dbForPlatform: $database, - projectInternalId: '1', - ); + $destination = $this->createDestination($database); $transfer = new Transfer($source, $destination); $database->getAuthorization()->skip( - static function () use ($destination, $explicit, $success, $transfer): void { + static function () use ($explicit, $transfer): void { if ($explicit) { $transfer->runWithResourceSelector( [Resource::TYPE_DATABASE], @@ -446,25 +448,24 @@ static function (): void { }, ); } - - if ($success) { - $destination->success(); - } }, ); return $destination; } - private function finalizeExpectingFailure(UtopiaDatabase $database, AppwriteDestination $destination): Finalization + private function createDestination(UtopiaDatabase $database): CountingAppwriteDestination { - try { - $database->getAuthorization()->skip($destination->success(...)); - } catch (Finalization $failure) { - return $failure; - } - - $this->fail('A failed finalization must throw'); + return new CountingAppwriteDestination( + project: 'destination-project', + endpoint: 'http://example.test/v1', + key: 'test-key', + dbForProject: $database, + getDatabasesDB: static fn (UtopiaDocument $document): UtopiaDatabase => $database, + collectionStructure: ['attributes' => [], 'indexes' => []], + dbForPlatform: $database, + projectInternalId: '1', + ); } private function getDatabaseDocument(UtopiaDatabase $database, string $databaseId = 'database'): UtopiaDocument diff --git a/tests/Migration/Unit/Destinations/AppwriteFinalizationTest.php b/tests/Migration/Unit/Destinations/AppwriteRunCompletionTest.php similarity index 57% rename from tests/Migration/Unit/Destinations/AppwriteFinalizationTest.php rename to tests/Migration/Unit/Destinations/AppwriteRunCompletionTest.php index 57ac904c..a8853ad8 100644 --- a/tests/Migration/Unit/Destinations/AppwriteFinalizationTest.php +++ b/tests/Migration/Unit/Destinations/AppwriteRunCompletionTest.php @@ -4,7 +4,6 @@ use Override; use PHPUnit\Framework\TestCase; -use Throwable; use Utopia\Cache\Adapter\Memory as MemoryCache; use Utopia\Cache\Cache; use Utopia\Database\Adapter\Memory as MemoryAdapter; @@ -15,9 +14,7 @@ use Utopia\Database\Query; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; use Utopia\Migration\Destinations\OnDuplicate; -use Utopia\Migration\Exception\Aborted; use Utopia\Migration\Exception as MigrationException; -use Utopia\Migration\Exception\Finalization; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Database\Columns\Text; use Utopia\Migration\Resources\Database\Database as DatabaseResource; @@ -26,13 +23,13 @@ use Utopia\Query\Schema\ColumnType; use Utopia\Tests\Unit\Adapters\MockSource; -final class AppwriteFinalizationTest extends TestCase +final class AppwriteRunCompletionTest extends TestCase { private const SOURCE_OLDER = '2000-01-01 00:00:00'; private const SOURCE_NEWER = '2999-01-01 00:00:00'; - public function testSkippedFinalizationIsReportedWhenTheLifecycleEnds(): void + public function testRunMarksItsDatabasesReady(): void { $database = $this->projectDatabase(); $destination = $this->destination($database); @@ -42,70 +39,12 @@ public function testSkippedFinalizationIsReportedWhenTheLifecycleEnds(): void $this->databaseResource('second'), ]); - $this->assertSame([], $destination->getErrors(), 'A run awaiting success() must not fail the caller\'s error gate'); - - $destination->cleanUp(); - - $this->assertSame( - [ - [Resource::TYPE_DATABASE, Transfer::GROUP_DATABASES, 'first'], - [Resource::TYPE_DATABASE, Transfer::GROUP_DATABASES, 'second'], - ], - $this->subjects($destination->getErrors()), - 'Ending the lifecycle without success() must report every database it left unfinalized', - ); - $this->assertStringContainsString('success()', $destination->getErrors()[0]->getMessage()); - $this->assertSame('provisioning', $this->databaseStatus($database, 'first')); - $this->assertSame('provisioning', $this->databaseStatus($database, 'second')); - } - - public function testFinalizedRunReportsNothingWhenTheLifecycleEnds(): void - { - $database = $this->projectDatabase(); - $destination = $this->destination($database); - $this->import($database, $destination, [$this->databaseResource('first')]); - - $this->assertNull($this->finalize($database, $destination)); - $destination->cleanUp(); - $this->assertSame([], $destination->getErrors()); $this->assertSame('ready', $this->databaseStatus($database, 'first')); - } - - public function testAbandonedRunReportsNothingWhenTheLifecycleEnds(): void - { - $database = $this->projectDatabase(); - $destination = $this->destination($database); - $this->import($database, $destination, [$this->databaseResource('first')]); - - $destination->error(); - $destination->cleanUp(); - - $this->assertSame([], $destination->getErrors()); - $this->assertSame('provisioning', $this->databaseStatus($database, 'first')); - } - - public function testRerunWithoutFinalizationReportsTheDiscardedRun(): void - { - $database = $this->projectDatabase(); - $destination = $this->destination($database); - - $this->import($database, $destination, [$this->databaseResource('first')]); - $this->import($database, $destination, [$this->databaseResource('second')]); - - $this->assertSame( - [[Resource::TYPE_DATABASE, Transfer::GROUP_DATABASES, 'first']], - $this->subjects($destination->getErrors()), - 'A second run must not silently discard the first run\'s unfinalized databases', - ); - - $this->assertNull($this->finalize($database, $destination)); - - $this->assertSame('provisioning', $this->databaseStatus($database, 'first')); $this->assertSame('ready', $this->databaseStatus($database, 'second')); } - public function testSuccessDoesNotFinalizeAnAbortedRun(): void + public function testInterruptedRunLeavesItsDatabasesProvisioning(): void { $database = $this->projectDatabase(); $destination = $this->destination($database); @@ -116,86 +55,18 @@ public function testSuccessDoesNotFinalizeAnAbortedRun(): void $destination, [$this->databaseResource('first')], static function (): void { - throw new \RuntimeException('transfer aborted'); + throw new \RuntimeException('transfer interrupted'); }, ); - $this->fail('The aborted transfer must throw'); + $this->fail('The interrupted transfer must throw'); } catch (\RuntimeException $error) { - $this->assertSame('transfer aborted', $error->getMessage()); + $this->assertSame('transfer interrupted', $error->getMessage()); } - $this->assertNull($this->finalize($database, $destination)); - $destination->cleanUp(); - $this->assertSame('provisioning', $this->databaseStatus($database, 'first'), 'An interrupted run must never be marked ready'); $this->assertSame([], $destination->getErrors()); } - public function testSuccessFinalizesNothingWhenTheSourceSwallowedTheAbort(): void - { - $database = $this->projectDatabase(); - $this->seedTablesWithLegacyColumns($database); - - $abort = new Aborted('Migration attempt was superseded'); - $destination = $this->destination($database, OnDuplicate::Overwrite); - - try { - $this->importSwallowingAbort($database, $destination, $this->newerSchemaWithoutLegacyColumns(), $abort); - $this->fail('The swallowed abort must still end the transfer'); - } catch (Aborted $caught) { - $this->assertSame($abort, $caught); - } - - $this->assertNull($this->finalize($database, $destination)); - - $this->assertSame('provisioning', $this->databaseStatus($database, 'first'), 'An aborted run must never be marked ready'); - $this->assertSame('provisioning', $this->databaseStatus($database, 'second'), 'An aborted run must never be marked ready'); - $this->assertSame(['legacy', 'name'], $this->columnKeys($database, 'first', 'first-items'), 'An aborted run must not sweep its overwrite orphans'); - $this->assertSame(['legacy', 'name'], $this->columnKeys($database, 'second', 'second-items'), 'An aborted run must not sweep its overwrite orphans'); - - $errors = $destination->getErrors(); - $destination->cleanUp(); - - $this->assertSame($errors, $destination->getErrors(), 'An aborted run is not reported as a skipped finalization'); - } - - public function testFailedFinalizationThrowsAfterAttemptingEveryDatabase(): void - { - $database = $this->projectDatabase(); - $database->failReadyWrites = ['first', 'third']; - $destination = $this->destination($database); - $this->import($database, $destination, [ - $this->databaseResource('first'), - $this->databaseResource('second'), - $this->databaseResource('third'), - ]); - - $failure = $this->finalize($database, $destination); - - $this->assertSame('provisioning', $this->databaseStatus($database, 'first')); - $this->assertSame('ready', $this->databaseStatus($database, 'second')); - $this->assertSame('provisioning', $this->databaseStatus($database, 'third')); - $this->assertNotNull($failure, 'A failed finalization must throw, not only append errors'); - $this->assertInstanceOf(Finalization::class, $failure); - $this->assertSame( - [ - [Resource::TYPE_DATABASE, Transfer::GROUP_DATABASES, 'first'], - [Resource::TYPE_DATABASE, Transfer::GROUP_DATABASES, 'third'], - ], - $this->subjects($failure->failures), - ); - $this->assertSame('ready status unavailable', $failure->failures[0]->getMessage()); - $this->assertSame($failure->failures, $destination->getErrors(), 'Each failure is also recorded for the report'); - $this->assertSame($failure->failures[0], $failure->getPrevious()); - $this->assertSame(MigrationException::CODE_INTERNAL, $failure->getCode()); - $this->assertStringContainsString('first', $failure->getMessage()); - $this->assertStringContainsString('third', $failure->getMessage()); - - $destination->cleanUp(); - - $this->assertCount(2, $destination->getErrors(), 'An attempted finalization is not reported again as skipped'); - } - public function testOneFailedReadyFlipStillSweepsEveryOverwrittenTable(): void { $database = $this->projectDatabase(); @@ -205,89 +76,39 @@ public function testOneFailedReadyFlipStillSweepsEveryOverwrittenTable(): void $destination = $this->destination($database, OnDuplicate::Overwrite); $this->import($database, $destination, $this->newerSchemaWithoutLegacyColumns()); - $failure = $this->finalize($database, $destination); - $this->assertSame(['name'], $this->columnKeys($database, 'first', 'first-items'), 'The sweep must run for the database whose flip failed'); $this->assertSame(['name'], $this->columnKeys($database, 'second', 'second-items'), 'The sweep must run for every other database'); $this->assertSame('provisioning', $this->databaseStatus($database, 'first')); $this->assertSame('ready', $this->databaseStatus($database, 'second')); - $this->assertInstanceOf(Finalization::class, $failure); $this->assertSame( [[Resource::TYPE_DATABASE, Transfer::GROUP_DATABASES, 'first']], - $this->subjects($failure->failures), + $this->subjects($destination->getErrors()), ); + $this->assertSame('ready status unavailable', $destination->getErrors()[0]->getMessage()); } - public function testOrphanSweepFailureStillSweepsTheOtherTablesAndThrows(): void + public function testOrphanSweepFailureThrowsAfterEveryDatabaseIsReady(): void { $database = $this->projectDatabase(); $this->seedTablesWithLegacyColumns($database); + $database->failAttributeScanOf = $this->table($database, 'first', 'first-items'); $destination = $this->destination($database, OnDuplicate::Overwrite); - $this->import($database, $destination, $this->newerSchemaWithoutLegacyColumns()); - $database->failAttributeScanOf = $this->table($database, 'first', 'first-items'); - $failure = $this->finalize($database, $destination); + try { + $this->import($database, $destination, $this->newerSchemaWithoutLegacyColumns()); + $this->fail('A failed sweep must fail the run'); + } catch (\Throwable $error) { + $this->assertSame('attribute scan unavailable', $error->getMessage()); + } $database->failAttributeScanOf = null; $this->assertSame(['legacy', 'name'], $this->columnKeys($database, 'first', 'first-items')); - $this->assertSame(['name'], $this->columnKeys($database, 'second', 'second-items'), 'One table\'s failed sweep must not skip the others'); $this->assertSame('ready', $this->databaseStatus($database, 'first'), 'Databases are flipped before any sweep can fail'); - $this->assertSame('ready', $this->databaseStatus($database, 'second')); - $this->assertInstanceOf(Finalization::class, $failure, 'A failed sweep must surface as the finalization exception'); - $this->assertSame( - [[Resource::TYPE_TABLE, Transfer::GROUP_DATABASES, 'first-items']], - $this->subjects($failure->failures), - ); - $this->assertSame('attribute scan unavailable', $failure->failures[0]->getMessage()); - $this->assertSame($failure->failures, $destination->getErrors()); - } - - /** - * A source that records the export failure and carries on, the way the library sources did - * before they rethrew aborts. - * - * @param list $resources - */ - private function importSwallowingAbort( - FailingFinalizationDatabase $database, - AppwriteDestination $destination, - array $resources, - Aborted $abort, - ): void { - $source = new class () extends MockSource { - #[Override] - public function supportsDatabaseStatus(): bool - { - return true; - } - - #[Override] - protected function exportGroupDatabases(int $batchSize, array $resources): void - { - foreach ($resources as $resource) { - try { - parent::exportGroupDatabases($batchSize, [$resource]); - } catch (Throwable $error) { - $this->addError(new MigrationException($resource, Transfer::GROUP_DATABASES, message: $error->getMessage(), previous: $error)); - } - } - } - }; - - $types = []; - foreach ($resources as $resource) { - $source->pushMockResource($resource); - $types[$resource->getName()] = true; - } - - $transfer = new Transfer($source, $destination); - $database->getAuthorization()->skip( - static fn () => $transfer->run(\array_keys($types), static fn () => throw $abort), - ); + $this->assertSame('ready', $this->databaseStatus($database, 'second'), 'Databases are flipped before any sweep can fail'); } - private function seedTablesWithLegacyColumns(FailingFinalizationDatabase $database): void + private function seedTablesWithLegacyColumns(FailingCompletionDatabase $database): void { $resources = []; foreach (['first', 'second'] as $databaseId) { @@ -305,7 +126,6 @@ private function seedTablesWithLegacyColumns(FailingFinalizationDatabase $databa $destination = $this->destination($database); $this->import($database, $destination, $resources); - $this->assertNull($this->finalize($database, $destination)); $this->assertSame([], $destination->getErrors()); $this->assertSame(['legacy', 'name'], $this->columnKeys($database, 'first', 'first-items')); $this->assertSame(['legacy', 'name'], $this->columnKeys($database, 'second', 'second-items')); @@ -346,7 +166,7 @@ private function databaseResource(string $id, string $name = 'Database', string } private function destination( - FailingFinalizationDatabase $database, + FailingCompletionDatabase $database, OnDuplicate $onDuplicate = OnDuplicate::Fail, ): AppwriteDestination { return new AppwriteDestination( @@ -376,7 +196,7 @@ private function destination( * @param list $resources */ private function import( - FailingFinalizationDatabase $database, + FailingCompletionDatabase $database, AppwriteDestination $destination, array $resources, ?callable $callback = null, @@ -401,25 +221,14 @@ public function supportsDatabaseStatus(): bool ); } - private function finalize(FailingFinalizationDatabase $database, AppwriteDestination $destination): ?\Throwable - { - try { - $database->getAuthorization()->skip($destination->success(...)); - } catch (\Throwable $failure) { - return $failure; - } - - return null; - } - - private function databaseStatus(FailingFinalizationDatabase $database, string $databaseId): mixed + private function databaseStatus(FailingCompletionDatabase $database, string $databaseId): mixed { return $database->getAuthorization()->skip( static fn (): UtopiaDocument => $database->getDocument('databases', $databaseId), )->getAttribute('status'); } - private function table(FailingFinalizationDatabase $database, string $databaseId, string $tableId): UtopiaDocument + private function table(FailingCompletionDatabase $database, string $databaseId, string $tableId): UtopiaDocument { return $database->getAuthorization()->skip(static function () use ($database, $databaseId, $tableId): UtopiaDocument { $metadata = $database->getDocument('databases', $databaseId); @@ -431,7 +240,7 @@ private function table(FailingFinalizationDatabase $database, string $databaseId /** * @return list */ - private function columnKeys(FailingFinalizationDatabase $database, string $databaseId, string $tableId): array + private function columnKeys(FailingCompletionDatabase $database, string $databaseId, string $tableId): array { $table = $this->table($database, $databaseId, $tableId); $columns = $database->getAuthorization()->skip(static fn (): array => $database->find('attributes', [ @@ -460,9 +269,9 @@ private function subjects(array $errors): array )); } - private function projectDatabase(): FailingFinalizationDatabase + private function projectDatabase(): FailingCompletionDatabase { - $database = new FailingFinalizationDatabase( + $database = new FailingCompletionDatabase( new MemoryAdapter(), new Cache(new MemoryCache()), ); diff --git a/tests/Migration/Unit/Destinations/FailingFinalizationDatabase.php b/tests/Migration/Unit/Destinations/FailingCompletionDatabase.php similarity index 94% rename from tests/Migration/Unit/Destinations/FailingFinalizationDatabase.php rename to tests/Migration/Unit/Destinations/FailingCompletionDatabase.php index 86b17ca8..345cd3c4 100644 --- a/tests/Migration/Unit/Destinations/FailingFinalizationDatabase.php +++ b/tests/Migration/Unit/Destinations/FailingCompletionDatabase.php @@ -10,11 +10,11 @@ use Utopia\Database\Query; /** - * A project database whose finalization steps can be made to fail: the + * A project database whose end-of-run steps can be made to fail: the * `ready` write of chosen databases, and the attribute scan the overwrite * sweep runs for one chosen table. */ -final class FailingFinalizationDatabase extends UtopiaDatabase +final class FailingCompletionDatabase extends UtopiaDatabase { /** @var list */ public array $failReadyWrites = []; diff --git a/tests/Migration/Unit/Destinations/TransfersColumns.php b/tests/Migration/Unit/Destinations/TransfersColumns.php index 1e00ae23..b7c57be1 100644 --- a/tests/Migration/Unit/Destinations/TransfersColumns.php +++ b/tests/Migration/Unit/Destinations/TransfersColumns.php @@ -107,7 +107,17 @@ static function (): void { */ private function transferSchema(callable $makeResources, UtopiaDatabase $database, OnDuplicate $onDuplicate, string $updatedAt): array { - $source = new MockSource(); + $source = new class () extends MockSource { + #[\Override] + protected function exportGroupDatabases(int $batchSize, array $resources): void + { + foreach ([Resource::TYPE_DATABASE, Resource::TYPE_TABLE, Resource::TYPE_COLUMN, Resource::TYPE_INDEX] as $type) { + if (Resource::isSupported($type, $resources)) { + parent::exportGroupDatabases($batchSize, [$type]); + } + } + } + }; $databaseResource = new DatabaseResource( id: 'shop', name: 'Shop', @@ -128,12 +138,11 @@ private function transferSchema(callable $makeResources, UtopiaDatabase $databas $destination = $this->destination($database, $onDuplicate); $transfer = new Transfer($source, $destination); $database->getAuthorization()->skip( - static function () use ($transfer): void { - $noop = static function (): void { - }; - $transfer->run([Resource::TYPE_DATABASE, Resource::TYPE_TABLE, Resource::TYPE_COLUMN], $noop); - $transfer->run([Resource::TYPE_INDEX], $noop); - }, + static fn () => $transfer->run( + [Resource::TYPE_DATABASE, Resource::TYPE_TABLE, Resource::TYPE_COLUMN, Resource::TYPE_INDEX], + static function (): void { + }, + ), ); return [$destination, $resources]; diff --git a/tests/Migration/Unit/General/TransferAbortTest.php b/tests/Migration/Unit/General/TransferAbortTest.php deleted file mode 100644 index 97c65ea7..00000000 --- a/tests/Migration/Unit/General/TransferAbortTest.php +++ /dev/null @@ -1,163 +0,0 @@ -swallowingSource(); - $transfer = new Transfer($source, new MockDestination()); - $calls = 0; - - try { - $run($transfer, static function () use ($abort, &$calls): never { - $calls++; - - throw $abort; - }); - $this->fail('The abort did not stop the transfer.'); - } catch (Aborted $caught) { - $this->assertSame($abort, $caught); - } - - $this->assertSame(1, $calls, 'The callback must not run again once it aborted.'); - } - - /** - * @return array - */ - public static function runProvider(): array - { - return [ - 'run' => [ - static fn (Transfer $transfer, callable $callback) => $transfer->run( - [Resource::TYPE_USER, Resource::TYPE_DATABASE, Resource::TYPE_TABLE], - $callback, - ), - ], - 'resource selector' => [ - static fn (Transfer $transfer, callable $callback) => $transfer->runWithResourceSelector( - [Resource::TYPE_DATABASE, Resource::TYPE_TABLE], - $callback, - resourceId: 'table', - resourceInternalId: '2', - resourceType: Resource::TYPE_TABLE, - parentResourceId: 'database', - parentResourceInternalId: '1', - parentResourceType: Resource::TYPE_DATABASE, - ), - ], - ]; - } - - public function testAbortDoesNotCarryOverToTheNextRun(): void - { - $abort = new Aborted('Stopped'); - $source = $this->swallowingSource(); - $destination = new MockDestination(); - $transfer = new Transfer($source, $destination); - - try { - $transfer->run([Resource::TYPE_USER], static fn () => throw $abort); - $this->fail('The abort did not stop the transfer.'); - } catch (Aborted $caught) { - $this->assertSame($abort, $caught); - } - - $transfer->run([Resource::TYPE_DATABASE], static function (): void { - }); - - $this->assertSame(['database'], $destination->getResourceTypeData(Transfer::GROUP_DATABASES, Resource::TYPE_DATABASE)); - } - - public function testALatchedAbortTellsTheDestinationTheRunWasAborted(): void - { - $abort = new Aborted('Stopped'); - $destination = new class () extends MockDestination { - public int $aborts = 0; - - #[Override] - public function markAborted(): void - { - $this->aborts++; - } - }; - $transfer = new Transfer($this->swallowingSource(), $destination); - - try { - $transfer->run([Resource::TYPE_USER], static fn () => throw $abort); - $this->fail('The abort did not stop the transfer.'); - } catch (Aborted $caught) { - $this->assertSame($abort, $caught); - } - - $this->assertSame(1, $destination->aborts, 'A latched abort must tell the destination the run was aborted.'); - - $transfer->run([Resource::TYPE_DATABASE], static function (): void { - }); - - $this->assertSame(1, $destination->aborts, 'A run that completed must not be reported as aborted.'); - } - - /** - * A source that records every failure of a resource type and moves on, the way the - * library sources did before they rethrew aborts. - */ - private function swallowingSource(): MockSource - { - $source = new class () extends MockSource { - #[Override] - protected function exportGroupAuth(int $batchSize, array $resources): void - { - foreach ($resources as $resource) { - try { - parent::exportGroupAuth($batchSize, [$resource]); - } catch (Throwable $error) { - $this->addError(new Exception($resource, Transfer::GROUP_AUTH, message: $error->getMessage(), previous: $error)); - } - } - } - - #[Override] - protected function exportGroupDatabases(int $batchSize, array $resources): void - { - foreach ($resources as $resource) { - try { - parent::exportGroupDatabases($batchSize, [$resource]); - } catch (Throwable $error) { - $this->addError(new Exception($resource, Transfer::GROUP_DATABASES, message: $error->getMessage(), previous: $error)); - } - } - } - }; - - $database = new Database('database', 'Database'); - $source->pushMockResource(new User('user', 'user@example.com')); - $source->pushMockResource($database); - $source->pushMockResource(new Table($database, 'Table', 'table')); - - return $source; - } -} diff --git a/tests/Migration/Unit/Sources/AppwriteAbortTest.php b/tests/Migration/Unit/Sources/AppwriteAbortTest.php deleted file mode 100644 index e8747591..00000000 --- a/tests/Migration/Unit/Sources/AppwriteAbortTest.php +++ /dev/null @@ -1,413 +0,0 @@ -superseded(); - $log = new ArrayObject(); - $source = $this->source($log, [ - '/teams' => ['total' => 1, 'teams' => [self::team('team')]], - '/storage/buckets' => ['total' => 1, 'buckets' => [self::bucket('bucket')]], - '/messaging/topics' => ['total' => 1, 'topics' => [self::topic('topic')]], - ]); - $destination = new MockDestination(); - $transfer = new Transfer($source, $destination); - - $this->assertAborted($abort, static fn () => $transfer->run( - [Resource::TYPE_TEAM, Resource::TYPE_BUCKET, Resource::TYPE_TOPIC], - static fn () => throw $abort, - )); - - $this->assertSame(['/teams'], $log->getArrayCopy()); - $this->assertSame([Transfer::GROUP_AUTH], \array_keys($destination->data)); - $this->assertSame([], $source->getErrors()); - } - - /** - * @param array $resources - * @param array $cached - * @param array> $responses - * @param array $expected - */ - #[DataProvider('itemProvider')] - public function testAbortFromTheProgressCallbackStopsLaterItems(array $resources, array $cached, array $responses, array $expected): void - { - $abort = $this->superseded(); - $log = new ArrayObject(); - $source = $this->source($log, $responses); - $transfer = new Transfer($source, new MockDestination()); - - foreach ($cached as $resource) { - $transfer->getCache()->add($resource); - } - - $this->assertAborted($abort, static fn () => $transfer->run($resources, static fn () => throw $abort)); - - $this->assertSame($expected, $log->getArrayCopy()); - $this->assertSame([], $source->getErrors()); - } - - /** - * @return array, array, array>, array}> - */ - public static function itemProvider(): array - { - $deployments = ['total' => 2, 'deployments' => [self::deployment('first'), self::deployment('second')]]; - - return [ - 'files' => [ - [Resource::TYPE_FILE], - [new Bucket('bucket', 'Bucket')], - ['/storage/buckets/bucket/files' => ['total' => 2, 'files' => [self::file('first'), self::file('second')]]], - ['/storage/buckets/bucket/files', 'GET /storage/buckets/bucket/files/first/download'], - ], - 'function deployments' => [ - [Resource::TYPE_DEPLOYMENT], - [new Func('function', 'Function', 'node-22')], - ['/functions/function/deployments' => $deployments], - [ - '/functions/function/deployments', - 'HEAD /functions/function/deployments/first/download', - 'GET /functions/function/deployments/first/download', - ], - ], - 'active function deployments' => [ - [Resource::TYPE_ENVIRONMENT_VARIABLE], - [ - new Func('first', 'First', 'node-22', activeDeployment: 'active'), - new Func('second', 'Second', 'node-22', activeDeployment: 'active'), - ], - [ - '/functions/first/deployments/active' => self::deployment('active'), - '/functions/second/deployments/active' => self::deployment('active'), - ], - [ - '/functions/first/deployments/active', - 'HEAD /functions/first/deployments/active/download', - 'GET /functions/first/deployments/active/download', - ], - ], - 'site deployments' => [ - [Resource::TYPE_SITE_DEPLOYMENT], - [new Site('site', 'Site', 'nextjs', 'node-22')], - ['/sites/site/deployments' => $deployments], - [ - '/sites/site/deployments', - 'HEAD /sites/site/deployments/first/download', - 'GET /sites/site/deployments/first/download', - ], - ], - 'active site deployments' => [ - [Resource::TYPE_SITE_VARIABLE], - [ - new Site('first', 'First', 'nextjs', 'node-22', activeDeployment: 'active'), - new Site('second', 'Second', 'nextjs', 'node-22', activeDeployment: 'active'), - ], - [ - '/sites/first/deployments/active' => self::deployment('active'), - '/sites/second/deployments/active' => self::deployment('active'), - ], - [ - '/sites/first/deployments/active', - 'HEAD /sites/first/deployments/active/download', - 'GET /sites/first/deployments/active/download', - ], - ], - ]; - } - - /** - * @param array $cached - */ - #[DataProvider('typeProvider')] - public function testAbortDuringAnExportStopsTheTransfer(string $type, array $cached): void - { - $abort = $this->superseded(); - $log = new ArrayObject(); - $source = $this->source($log, [], $abort); - $transfer = new Transfer($source, new MockDestination()); - $later = \in_array($type, Transfer::GROUP_INTEGRATIONS_RESOURCES, true) ? Resource::TYPE_RULE : Resource::TYPE_WEBHOOK; - - foreach ($cached as $resource) { - $transfer->getCache()->add($resource); - } - - $this->assertAborted($abort, static fn () => $transfer->run([$type, $later], static function (): void { - })); - - $this->assertCount(1, $log, 'Nothing may be requested after the abort.'); - $this->assertSame([], $source->getErrors()); - } - - /** - * @return array}> - */ - public static function typeProvider(): array - { - $database = new Database('database', 'Database'); - $table = new Table($database, 'Table', 'table'); - - return [ - 'users' => [Resource::TYPE_USER, []], - 'teams' => [Resource::TYPE_TEAM, []], - 'memberships' => [Resource::TYPE_MEMBERSHIP, [new Team('team', 'Team')]], - 'auth methods' => [Resource::TYPE_AUTH_METHODS, []], - 'OAuth2 providers' => [Resource::TYPE_OAUTH2_PROVIDER, []], - 'policies' => [Resource::TYPE_POLICIES, []], - 'databases' => [Resource::TYPE_DATABASE, []], - 'tables' => [Resource::TYPE_TABLE, [$database]], - 'columns' => [Resource::TYPE_COLUMN, [$table]], - 'indexes' => [Resource::TYPE_INDEX, [$table]], - 'rows' => [Resource::TYPE_ROW, [$table]], - 'buckets' => [Resource::TYPE_BUCKET, []], - 'files' => [Resource::TYPE_FILE, [new Bucket('bucket', 'Bucket')]], - 'functions' => [Resource::TYPE_FUNCTION, []], - 'deployments' => [Resource::TYPE_DEPLOYMENT, [new Func('function', 'Function', 'node-22')]], - 'sites' => [Resource::TYPE_SITE, []], - 'site deployments' => [Resource::TYPE_SITE_DEPLOYMENT, [new Site('site', 'Site', 'nextjs', 'node-22')]], - 'project variables' => [Resource::TYPE_PROJECT_VARIABLE, []], - 'protocols' => [Resource::TYPE_PROJECT_PROTOCOLS, []], - 'labels' => [Resource::TYPE_PROJECT_LABELS, []], - 'services' => [Resource::TYPE_PROJECT_SERVICES, []], - 'email templates' => [Resource::TYPE_PROJECT_EMAIL_TEMPLATE, []], - 'rules' => [Resource::TYPE_RULE, []], - 'providers' => [Resource::TYPE_PROVIDER, []], - 'topics' => [Resource::TYPE_TOPIC, []], - 'subscribers' => [Resource::TYPE_SUBSCRIBER, [new Topic('topic', 'Topic')]], - 'messages' => [Resource::TYPE_MESSAGE, []], - 'platforms' => [Resource::TYPE_PLATFORM, []], - 'API keys' => [Resource::TYPE_API_KEY, []], - 'webhooks' => [Resource::TYPE_WEBHOOK, []], - 'SMTP' => [Resource::TYPE_SMTP, []], - ]; - } - - private function superseded(): Aborted - { - return new class ('Migration attempt was superseded') extends Aborted { - }; - } - - private function assertAborted(Aborted $abort, callable $run): void - { - try { - $run(); - } catch (Aborted $caught) { - $this->assertSame($abort, $caught); - - return; - } - - $this->fail('The abort did not stop the transfer.'); - } - - /** - * A real Appwrite source whose SDK requests and downloads are answered from the script - * and recorded in the log, in order. - * - * @param ArrayObject $log - * @param array> $responses - */ - private function source(ArrayObject $log, array $responses, ?Throwable $unscripted = null): Appwrite - { - $source = new class ($log) extends Appwrite { - /** - * @param ArrayObject $log - */ - public function __construct(private readonly ArrayObject $log) - { - parent::__construct('project', 'http://localhost', 'key', static fn (): null => null); - } - - #[Override] - protected function call(string $method, string $path = '', array $headers = [], array $params = [], array &$responseHeaders = []): array|string - { - $this->log->append($method . ' ' . $path); - - return 'data'; - } - }; - - $client = new class ($log, $responses, $unscripted ?? new LogicException('Unscripted request')) extends Client { - /** - * @param ArrayObject $log - * @param array> $responses - */ - public function __construct( - private readonly ArrayObject $log, - private readonly array $responses, - private readonly Throwable $unscripted, - ) { - parent::__construct(); - } - - #[Override] - public function call(string $method, string $path = '', array $headers = [], array $params = [], ?string $responseType = null): array - { - $this->log->append($path); - - return $this->responses[$path] ?? throw $this->unscripted; - } - }; - - $class = new ReflectionClass(Appwrite::class); - - foreach ($class->getProperties() as $property) { - $type = $property->getType(); - - if ($type instanceof ReflectionNamedType && \is_subclass_of($type->getName(), Service::class)) { - $service = $type->getName(); - $property->setValue($source, new $service($client)); - } - } - - $class->getProperty('client')->setValue($source, $client); - $class->getProperty('reader')->setValue($source, new API(new TablesDB($client))); - - return $source; - } - - /** - * @return array - */ - private static function team(string $id): array - { - return [ - '$id' => $id, - '$createdAt' => self::TIMESTAMP, - '$updatedAt' => self::TIMESTAMP, - 'name' => $id, - 'total' => 0, - 'prefs' => [], - ]; - } - - /** - * @return array - */ - private static function bucket(string $id): array - { - return [ - '$id' => $id, - '$createdAt' => self::TIMESTAMP, - '$updatedAt' => self::TIMESTAMP, - '$permissions' => [], - 'fileSecurity' => false, - 'name' => $id, - 'enabled' => true, - 'maximumFileSize' => 1024, - 'allowedFileExtensions' => [], - 'compression' => 'none', - 'encryption' => false, - 'antivirus' => false, - 'transformations' => false, - 'totalSize' => 0, - ]; - } - - /** - * @return array - */ - private static function topic(string $id): array - { - return [ - '$id' => $id, - '$createdAt' => self::TIMESTAMP, - '$updatedAt' => self::TIMESTAMP, - 'name' => $id, - 'emailTotal' => 0, - 'smsTotal' => 0, - 'pushTotal' => 0, - 'subscribe' => [], - ]; - } - - /** - * @return array - */ - private static function file(string $id): array - { - return [ - '$id' => $id, - 'bucketId' => 'bucket', - '$createdAt' => self::TIMESTAMP, - '$updatedAt' => self::TIMESTAMP, - '$permissions' => [], - 'name' => $id, - 'signature' => '', - 'mimeType' => 'text/plain', - 'sizeOriginal' => 4, - 'sizeActual' => 4, - 'chunksTotal' => 1, - 'chunksUploaded' => 1, - 'encryption' => false, - 'compression' => 'none', - ]; - } - - /** - * @return array - */ - private static function deployment(string $id): array - { - return [ - '$id' => $id, - '$createdAt' => self::TIMESTAMP, - '$updatedAt' => self::TIMESTAMP, - 'type' => 'manual', - 'resourceId' => '', - 'resourceType' => '', - 'entrypoint' => 'index.js', - 'sourceSize' => 4, - 'buildSize' => 0, - 'totalSize' => 4, - 'buildId' => '', - 'activate' => true, - 'screenshotLight' => '', - 'screenshotDark' => '', - 'status' => 'ready', - 'buildLogs' => '', - 'buildDuration' => 0, - 'providerRepositoryName' => '', - 'providerRepositoryOwner' => '', - 'providerRepositoryUrl' => '', - 'providerCommitHash' => '', - 'providerCommitAuthorUrl' => '', - 'providerCommitAuthor' => '', - 'providerCommitMessage' => '', - 'providerCommitUrl' => '', - 'providerBranch' => '', - 'providerBranchUrl' => '', - ]; - } -} diff --git a/tests/Migration/Unit/Sources/SourceAbortTest.php b/tests/Migration/Unit/Sources/SourceAbortTest.php deleted file mode 100644 index 031eb56a..00000000 --- a/tests/Migration/Unit/Sources/SourceAbortTest.php +++ /dev/null @@ -1,328 +0,0 @@ - $resources - * @param array $cached - */ - #[DataProvider('firebaseProvider')] - public function testFirebaseStopsAtTheAbort(array $resources, array $cached): void - { - $abort = $this->superseded(); - $log = new ArrayObject(); - $source = new class (['project_id' => 'project'], $log) extends Firebase { - /** - * @param array $serviceAccount - * @param ArrayObject $log - */ - public function __construct(array $serviceAccount, private readonly ArrayObject $log) - { - parent::__construct($serviceAccount); - } - - #[Override] - protected function call(string $method, string $path = '', array $headers = [], array $params = [], array &$responseHeaders = []): array|string - { - $this->log->append($method . ' ' . $path); - - return match (true) { - \str_ends_with($path, '/config') => ['signIn' => ['hashConfig' => []]], - \str_ends_with($path, '/downloadAccount') => ['users' => [['localId' => 'user']]], - \str_ends_with($path, ':listCollectionIds') => ['collectionIds' => ['collection']], - \str_ends_with($path, '/storage/v1/b') => ['items' => [['id' => 'bucket', 'name' => 'bucket']]], - \str_ends_with($path, '/o') => ['items' => [['name' => 'file']]], - \str_ends_with($path, '?alt=media') => 'data', - default => [], - }; - } - }; - $transfer = new Transfer($source, new MockDestination()); - - foreach ($cached as $resource) { - $transfer->getCache()->add($resource); - } - - $this->assertAborted($abort, fn () => $transfer->run($resources, $this->abortingCallback($log, $abort))); - - $events = $log->getArrayCopy(); - $this->assertSame(\count($events) - 1, \array_search(self::CALLBACK, $events, true), 'Nothing may be requested after the abort.'); - $this->assertSame([], $source->getErrors()); - } - - /** - * @return array, array}> - */ - public static function firebaseProvider(): array - { - return [ - 'users' => [[Resource::TYPE_USER, Resource::TYPE_BUCKET], []], - 'databases' => [[Resource::TYPE_DATABASE, Resource::TYPE_BUCKET], []], - 'tables' => [[Resource::TYPE_TABLE, Resource::TYPE_BUCKET], []], - 'buckets' => [[Resource::TYPE_BUCKET, Resource::TYPE_USER], []], - 'files' => [[Resource::TYPE_FILE, Resource::TYPE_USER], [(new Bucket('bucket', 'bucket'))->setOriginalId('bucket')]], - ]; - } - - /** - * @param array $cached - */ - #[DataProvider('nhostProvider')] - public function testNHostStopsAtTheAbort(string $type, array $cached): void - { - $abort = $this->superseded(); - $log = new ArrayObject(); - $source = new NHost('subdomain', 'region', 'secret', 'database', 'username', 'password'); - $source->pdo = $this->pdo($log, $abort); - - $this->assertStopsAtTheFirstEvent($source, $type, $cached, $log, $abort); - } - - /** - * @return array}> - */ - public static function nhostProvider(): array - { - $database = new Database('public', 'public'); - $table = new Table($database, 'table', 'table'); - - return [ - 'users' => [Resource::TYPE_USER, []], - 'databases' => [Resource::TYPE_DATABASE, []], - 'tables' => [Resource::TYPE_TABLE, [$database]], - 'columns' => [Resource::TYPE_COLUMN, [$table]], - 'rows' => [Resource::TYPE_ROW, [$database, $table]], - 'indexes' => [Resource::TYPE_INDEX, [$table]], - 'buckets' => [Resource::TYPE_BUCKET, []], - 'files' => [Resource::TYPE_FILE, [new Bucket('bucket', 'bucket')]], - ]; - } - - /** - * @param array $cached - */ - #[DataProvider('supabaseProvider')] - public function testSupabaseStopsAtTheAbort(string $type, array $cached): void - { - $abort = $this->superseded(); - $log = new ArrayObject(); - $source = new class () extends Supabase { - public function __construct() - { - // The real constructor connects to Postgres; the test supplies the database instead. - } - }; - $source->pdo = $this->pdo($log, $abort); - - $this->assertStopsAtTheFirstEvent($source, $type, $cached, $log, $abort); - } - - /** - * @return array}> - */ - public static function supabaseProvider(): array - { - return [ - 'users' => [Resource::TYPE_USER, []], - 'buckets' => [Resource::TYPE_BUCKET, []], - 'files' => [Resource::TYPE_FILE, [new Bucket('bucket', 'bucket')]], - ]; - } - - public function testCsvStopsAtTheAbort(): void - { - $abort = $this->superseded(); - $directory = \sys_get_temp_dir() . '/csv_abort_' . \uniqid(); - \mkdir($directory); - $device = new Local($directory); - $filePath = $device->getPath('rows.csv'); - \file_put_contents($filePath, "\$id,name\nfirst,First\nsecond,Second\n"); - - $source = CSV::fromResourceIds('database', 'table', $filePath, $device, $this->projectDatabase()); - $transfer = new Transfer($source, new MockDestination()); - - try { - $this->assertAborted($abort, static fn () => $transfer->run([Resource::TYPE_ROW], static fn () => throw $abort)); - $this->assertSame([], $source->getErrors()); - } finally { - \array_map(\unlink(...), \glob($directory . '/*') ?: []); - \rmdir($directory); - } - } - - public function testJsonStopsAtTheAbort(): void - { - $abort = $this->superseded(); - $directory = \sys_get_temp_dir() . '/json_abort_' . \uniqid(); - \mkdir($directory); - $device = new Local($directory); - $filePath = $device->getPath('rows.json'); - \file_put_contents($filePath, \json_encode([['$id' => 'first'], ['$id' => 'second']]) ?: '[]'); - - $source = JSON::fromResourceIds('database', 'table', $filePath, $device, null); - $transfer = new Transfer($source, new MockDestination()); - - try { - $this->assertAborted($abort, static fn () => $transfer->run([Resource::TYPE_ROW], static fn () => throw $abort)); - $this->assertSame([], $source->getErrors()); - } finally { - \array_map(\unlink(...), \glob($directory . '/*') ?: []); - \rmdir($directory); - } - } - - /** - * Runs the type plus a type from a later group; the database answers every statement - * with the abort and the callback aborts too, so the first event must also be the last. - * - * @param array $cached - * @param ArrayObject $log - */ - private function assertStopsAtTheFirstEvent(NHost $source, string $type, array $cached, ArrayObject $log, Aborted $abort): void - { - $transfer = new Transfer($source, new MockDestination()); - $later = \in_array($type, Transfer::GROUP_STORAGE_RESOURCES, true) ? Resource::TYPE_USER : Resource::TYPE_BUCKET; - - foreach ($cached as $resource) { - $transfer->getCache()->add($resource); - } - - $this->assertAborted($abort, fn () => $transfer->run([$type, $later], $this->abortingCallback($log, $abort))); - - $this->assertCount(1, $log, 'Nothing may happen after the abort.'); - $this->assertSame([], $source->getErrors()); - } - - private function superseded(): Aborted - { - return new class ('Migration attempt was superseded') extends Aborted { - }; - } - - private function assertAborted(Aborted $abort, callable $run): void - { - try { - $run(); - } catch (Aborted $caught) { - $this->assertSame($abort, $caught); - - return; - } - - $this->fail('The abort did not stop the transfer.'); - } - - /** - * @param ArrayObject $log - */ - private function abortingCallback(ArrayObject $log, Aborted $abort): callable - { - return static function () use ($log, $abort): never { - $log->append(self::CALLBACK); - - throw $abort; - }; - } - - /** - * @param ArrayObject $log - */ - private function pdo(ArrayObject $log, Aborted $abort): PDO - { - return new class ($log, $abort) extends PDO { - /** - * @param ArrayObject $log - */ - public function __construct(private readonly ArrayObject $log, private readonly Aborted $abort) - { - parent::__construct('sqlite::memory:'); - } - - #[Override] - public function prepare(string $query, array $options = []): never - { - $this->log->append($query); - - throw $this->abort; - } - - #[Override] - public function query(string $query, ?int $fetchMode = null, mixed ...$fetchModeArgs): never - { - $this->log->append($query); - - throw $this->abort; - } - }; - } - - private function projectDatabase(): UtopiaDatabase - { - $database = new UtopiaDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); - $database - ->setDatabase('appwrite') - ->setNamespace('_project'); - $database->create(); - $database->getAuthorization()->disable(); - - $database->createCollection(Collection::create( - id: 'databases', - attributes: [ - Attribute::string(key: 'name', size: 256), - Attribute::string(key: 'type', size: 128), - ], - )); - $metadata = $database->createDocument('databases', new Document([ - '$id' => 'database', - 'name' => 'Database', - 'type' => 'tablesdb', - ])); - - $tables = 'database_' . $metadata->getSequence(); - $database->createCollection(Collection::create( - id: $tables, - attributes: [Attribute::string(key: 'name', size: 256)], - )); - $database->createDocument($tables, new Document(['$id' => 'table', 'name' => 'Table'])); - - $database->createCollection(Collection::create( - id: 'attributes', - attributes: [ - Attribute::string(key: 'databaseInternalId', size: 64), - Attribute::string(key: 'collectionInternalId', size: 64), - ], - )); - - return $database; - } -} From 6f2f3e71374d5f2abfc26188c1e82c3d111edc94 Mon Sep 17 00:00:00 2001 From: Jake Barnby Date: Wed, 14 Oct 2026 19:21:23 +1300 Subject: [PATCH 3/8] test(destination): cover incomplete-database recovery under every duplicate policy 3.1.2 forces an overwrite of any provisioning or failed database whatever OnDuplicate or the timestamps say, but the recovery test ran only under Fail. It now runs under Fail, Skip and Overwrite with an older and a newer source, and checks the source name lands, so a policy that skipped an incomplete database would fail. The run-completion tests caught their own fail() because AssertionFailedError is a RuntimeException, so a run that did not throw surfaced as a confusing message mismatch. They now catch only the interruption the callback throws and the database exception the sweep raises. testRunAloneMarksProvisionedDatabasesReady duplicated AppwriteRunCompletionTest::testRunMarksItsDatabasesReady, which also asserts the run recorded no errors; the latter keeps its failure messages. Co-Authored-By: Claude Opus 5.5 --- .../AppwriteDatabaseStatusTest.php | 66 ++++++++----------- .../AppwriteRunCompletionTest.php | 11 ++-- .../Unit/Destinations/TransferInterrupted.php | 7 ++ 3 files changed, 42 insertions(+), 42 deletions(-) create mode 100644 tests/Migration/Unit/Destinations/TransferInterrupted.php diff --git a/tests/Migration/Unit/Destinations/AppwriteDatabaseStatusTest.php b/tests/Migration/Unit/Destinations/AppwriteDatabaseStatusTest.php index 9d6332cf..4b9789aa 100644 --- a/tests/Migration/Unit/Destinations/AppwriteDatabaseStatusTest.php +++ b/tests/Migration/Unit/Destinations/AppwriteDatabaseStatusTest.php @@ -3,6 +3,7 @@ namespace Utopia\Tests\Unit\Destinations; use Override; +use PHPUnit\Framework\Attributes\DataProvider; use PHPUnit\Framework\TestCase; use Utopia\Cache\Adapter\Memory as MemoryCache; use Utopia\Cache\Cache; @@ -14,6 +15,7 @@ use Utopia\Database\Document as UtopiaDocument; use Utopia\Database\Exception as DatabaseException; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; +use Utopia\Migration\Destinations\OnDuplicate; use Utopia\Migration\Exception as MigrationException; use Utopia\Migration\Resource; use Utopia\Migration\Resources\Database\Database as DatabaseResource; @@ -153,36 +155,6 @@ public function testDatabaseCreationPreservesLifecycleThroughLegacyAndExplicitEn } } - public function testRunAloneMarksProvisionedDatabasesReady(): void - { - $database = $this->createProjectDatabase(withStatus: true); - $source = new class () extends MockSource { - #[Override] - public function supportsDatabaseStatus(): bool - { - return true; - } - }; - foreach (['database-first', 'database-second'] as $databaseId) { - $source->pushMockResource(new DatabaseResource( - id: $databaseId, - name: 'Database', - type: 'tablesdb', - database: 'source-dsn', - databaseStatus: 'ready', - )); - } - $transfer = new Transfer($source, $this->createDestination($database)); - - $database->getAuthorization()->skip( - static fn () => $transfer->run([Resource::TYPE_DATABASE], static function (): void { - }), - ); - - $this->assertSame('ready', $this->getDatabaseDocument($database, 'database-first')->getAttribute('status'), 'run() must flip every database it provisioned to ready'); - $this->assertSame('ready', $this->getDatabaseDocument($database, 'database-second')->getAttribute('status'), 'run() must flip every database it provisioned to ready'); - } - public function testReadyFlipAttemptsEveryDatabaseAndReportsEveryFailure(): void { $database = new ReadyFailingProjectDatabase(new MemoryAdapter(), new Cache(new MemoryCache())); @@ -293,22 +265,37 @@ public function testProvisioningDatabaseRetrySucceedsUnderOnDuplicateFail(): voi ); } - public function testIncompleteDatabasesAreRecoveredImmediately(): void + /** @return array */ + public static function duplicatePolicies(): array + { + return [ + 'fail' => [OnDuplicate::Fail, ''], + 'skip' => [OnDuplicate::Skip, ''], + 'overwrite with an older source' => [OnDuplicate::Overwrite, '2000-01-01T00:00:00.000+00:00'], + 'overwrite with a newer source' => [OnDuplicate::Overwrite, '2099-01-01T00:00:00.000+00:00'], + ]; + } + + #[DataProvider('duplicatePolicies')] + public function testIncompleteDatabasesAreRecoveredImmediately(OnDuplicate $onDuplicate, string $resourceUpdatedAt): void { $database = $this->createProjectDatabase(withStatus: true); - $provisioning = $this->seedIncompleteDatabase($database, 'database-provisioning', 'provisioning', withCollection: true, updatedAt: $this->secondsAgo(1)); - $failed = $this->seedIncompleteDatabase($database, 'database-failed', 'failed', updatedAt: $this->secondsAgo(1)); + $provisioning = $this->seedIncompleteDatabase($database, 'database-provisioning', 'provisioning', name: 'Incomplete database', withCollection: true, updatedAt: $this->secondsAgo(1)); + $failed = $this->seedIncompleteDatabase($database, 'database-failed', 'failed', name: 'Incomplete database', updatedAt: $this->secondsAgo(1)); $destination = $this->runDatabaseTransfer( $database, explicit: false, databaseIds: ['database-provisioning', 'database-failed'], + onDuplicate: $onDuplicate, + resourceUpdatedAt: $resourceUpdatedAt, ); $this->assertSame([], $this->errorMessages($destination)); foreach ([$provisioning, $failed] as $seeded) { $recovered = $this->getDatabaseDocument($database, $seeded->getId()); $this->assertSame('ready', $recovered->getAttribute('status'), $seeded->getId()); + $this->assertSame('Database', $recovered->getAttribute('name'), $seeded->getId().' must be overwritten from the source whatever the duplicate policy'); $this->assertSame($seeded->getSequence(), $recovered->getSequence(), $seeded->getId()); $this->assertNotNull($database->findCollection('database_'.$recovered->getSequence()), $seeded->getId()); } @@ -350,6 +337,7 @@ private function seedIncompleteDatabase( UtopiaDatabase $database, string $databaseId, string $status, + string $name = 'Database', bool $withCollection = false, string $updatedAt = '', ): UtopiaDocument { @@ -358,9 +346,9 @@ private function seedIncompleteDatabase( true, static fn (): UtopiaDocument => $database->createDocument('databases', new UtopiaDocument([ '$id' => $databaseId, - 'name' => 'Database', + 'name' => $name, 'enabled' => true, - 'search' => $databaseId.' Database', + 'search' => $databaseId.' '.$name, 'originalId' => null, 'type' => 'tablesdb', 'database' => '', @@ -406,6 +394,8 @@ private function runDatabaseTransfer( UtopiaDatabase $database, bool $explicit, array $databaseIds = ['database'], + OnDuplicate $onDuplicate = OnDuplicate::Fail, + string $resourceUpdatedAt = '', ): CountingAppwriteDestination { $source = new class () extends MockSource { #[Override] @@ -418,13 +408,14 @@ public function supportsDatabaseStatus(): bool $source->pushMockResource(new DatabaseResource( id: $databaseId, name: 'Database', + updatedAt: $resourceUpdatedAt, type: 'tablesdb', database: 'source-dsn', databaseStatus: 'ready', )); } - $destination = $this->createDestination($database); + $destination = $this->createDestination($database, $onDuplicate); $transfer = new Transfer($source, $destination); $database->getAuthorization()->skip( @@ -454,7 +445,7 @@ static function (): void { return $destination; } - private function createDestination(UtopiaDatabase $database): CountingAppwriteDestination + private function createDestination(UtopiaDatabase $database, OnDuplicate $onDuplicate): CountingAppwriteDestination { return new CountingAppwriteDestination( project: 'destination-project', @@ -465,6 +456,7 @@ private function createDestination(UtopiaDatabase $database): CountingAppwriteDe collectionStructure: ['attributes' => [], 'indexes' => []], dbForPlatform: $database, projectInternalId: '1', + onDuplicate: $onDuplicate, ); } diff --git a/tests/Migration/Unit/Destinations/AppwriteRunCompletionTest.php b/tests/Migration/Unit/Destinations/AppwriteRunCompletionTest.php index a8853ad8..c4e5eeb4 100644 --- a/tests/Migration/Unit/Destinations/AppwriteRunCompletionTest.php +++ b/tests/Migration/Unit/Destinations/AppwriteRunCompletionTest.php @@ -11,6 +11,7 @@ use Utopia\Database\Collection; use Utopia\Database\Database as UtopiaDatabase; use Utopia\Database\Document as UtopiaDocument; +use Utopia\Database\Exception as DatabaseException; use Utopia\Database\Query; use Utopia\Migration\Destinations\Appwrite as AppwriteDestination; use Utopia\Migration\Destinations\OnDuplicate; @@ -40,8 +41,8 @@ public function testRunMarksItsDatabasesReady(): void ]); $this->assertSame([], $destination->getErrors()); - $this->assertSame('ready', $this->databaseStatus($database, 'first')); - $this->assertSame('ready', $this->databaseStatus($database, 'second')); + $this->assertSame('ready', $this->databaseStatus($database, 'first'), 'run() must flip every database it provisioned to ready'); + $this->assertSame('ready', $this->databaseStatus($database, 'second'), 'run() must flip every database it provisioned to ready'); } public function testInterruptedRunLeavesItsDatabasesProvisioning(): void @@ -55,11 +56,11 @@ public function testInterruptedRunLeavesItsDatabasesProvisioning(): void $destination, [$this->databaseResource('first')], static function (): void { - throw new \RuntimeException('transfer interrupted'); + throw new TransferInterrupted('transfer interrupted'); }, ); $this->fail('The interrupted transfer must throw'); - } catch (\RuntimeException $error) { + } catch (TransferInterrupted $error) { $this->assertSame('transfer interrupted', $error->getMessage()); } @@ -98,7 +99,7 @@ public function testOrphanSweepFailureThrowsAfterEveryDatabaseIsReady(): void try { $this->import($database, $destination, $this->newerSchemaWithoutLegacyColumns()); $this->fail('A failed sweep must fail the run'); - } catch (\Throwable $error) { + } catch (DatabaseException $error) { $this->assertSame('attribute scan unavailable', $error->getMessage()); } diff --git a/tests/Migration/Unit/Destinations/TransferInterrupted.php b/tests/Migration/Unit/Destinations/TransferInterrupted.php new file mode 100644 index 00000000..169be520 --- /dev/null +++ b/tests/Migration/Unit/Destinations/TransferInterrupted.php @@ -0,0 +1,7 @@ + Date: Wed, 14 Oct 2026 19:21:23 +1300 Subject: [PATCH 4/8] test(cli): isolate the subquery filter test in its own process registerFilters() writes the global Database filter registry, and utopia-php/database 8 has no public way to read or restore it: addFilter() only adds, and the per-handle Registry codecs never see the document or database the subquery filters need. Running the test in a separate process keeps the CLI filters out of the other tests without reflecting on the library's static state. Co-Authored-By: Claude Opus 5.5 --- tests/Migration/Unit/MigrationCLITest.php | 19 ++----------------- 1 file changed, 2 insertions(+), 17 deletions(-) diff --git a/tests/Migration/Unit/MigrationCLITest.php b/tests/Migration/Unit/MigrationCLITest.php index 43455435..62521d04 100644 --- a/tests/Migration/Unit/MigrationCLITest.php +++ b/tests/Migration/Unit/MigrationCLITest.php @@ -6,6 +6,7 @@ use Override; use PHPUnit\Framework\Attributes\BackupGlobals; +use PHPUnit\Framework\Attributes\RunInSeparateProcess; use PHPUnit\Framework\TestCase; use Utopia\Cache\Adapter\Memory as MemoryCache; use Utopia\Cache\Cache; @@ -63,11 +64,6 @@ public function getErrors(): array #[BackupGlobals(true)] final class MigrationCLITest extends TestCase { - private const array FILTER_STATE = ['filters', 'defaultFiltersRegistered']; - - /** @var array */ - private array $filterState = []; - public function testStartFinalizesSuccessfulAppwriteDatabase(): void { $database = $this->createProjectDatabase(); @@ -120,6 +116,7 @@ public function testStartFinalizesSuccessfulResourcesWhenDestinationHasErrors(): $this->assertNotNull($database->findCollection('database_'.$created->getSequence())); } + #[RunInSeparateProcess] public function testTheMetadataSubqueryFiltersLoadTheTableColumnsAndIndexes(): void { \MigrationCLI::registerFilters(); @@ -255,22 +252,10 @@ public function supportsDatabaseStatus(): bool protected function setUp(): void { parent::setUp(); - foreach (self::FILTER_STATE as $property) { - $this->filterState[$property] = (new \ReflectionProperty(Database::class, $property))->getValue(); - } $_ENV['DESTINATION_PROVIDER'] = 'appwrite'; $_ENV['DESTINATION_APPWRITE_TEST_PROJECT'] = 'destination-project'; $_ENV['DESTINATION_APPWRITE_TEST_ENDPOINT'] = 'http://example.test/v1'; $_ENV['DESTINATION_APPWRITE_TEST_KEY'] = 'test-key'; $_ENV['DESTINATION_APPWRITE_TEST_PROJECT_INTERNAL_ID'] = '1'; } - - #[Override] - protected function tearDown(): void - { - foreach ($this->filterState as $property => $value) { - (new \ReflectionProperty(Database::class, $property))->setValue(null, $value); - } - parent::tearDown(); - } } From 8ea7f19eec288e87c77df4201080127ad62fbfec Mon Sep 17 00:00:00 2001 From: Jake Barnby Date: Wed, 14 Oct 2026 19:22:00 +1300 Subject: [PATCH 5/8] docs(changelog): state the 3.1.2 concurrency contract and recovery semantics Without the ownership fence the destination cannot tell an abandoned incomplete database from one another run is still provisioning, so it treats both as abandoned and forces an overwrite under any OnDuplicate. Callers have to serialise migrations per destination project, and the notes have to say so. "Back on the 2.x lifecycle" was wrong for recovery: 2.1.0 skipped a matching provisioning database and never recovered under Fail. The sweep also now throws on the first failing table instead of collecting every failure into Finalization, the removed CLI constructor and getHelp() were unlisted, and the 3.0.0 section still read as unreleased although 3.0.0 through 3.1.1 are tagged. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d9821859..73b262dd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,31 +2,44 @@ ## 3.1.2 +3.1.2 reverses the 3.0.0 notes below on `Aborted`, `markAborted()` and finalizing with `success()`. + ### Breaking changes -- Provisioning ownership fencing is removed. The Appwrite destination is back on the 2.x status-only database - lifecycle. +- Provisioning ownership fencing is removed. The Appwrite destination tracks a database's lifecycle with `status` + alone again, but recovers incomplete databases differently from 2.x. - `Destinations\Appwrite\ProvisioningOwner` is removed. - `Destinations\Appwrite::__construct()` no longer takes `owner`, `getRecoverableOwner` or `provisioningLease`, and `Destinations\Appwrite::DEFAULT_PROVISIONING_LEASE` is removed. `onDuplicate` and the arguments after it move up two places for positional callers. - The destination no longer reads or writes the `migrationId` and `migrationAttemptId` database attributes. - A `provisioning` or `failed` database is recovered by the next run at once, without owner attestation or a lease - wait. + wait. It is always overwritten from the source, whatever `OnDuplicate` is and whatever the timestamps say. 2.1.0 + forced the overwrite only for a `failed` database and never under `OnDuplicate::Fail`; it skipped a + `provisioning` database whose spec matched the source and only re-enrolled it for the `ready` flip. - The standalone CLI no longer accepts `--migration-id`, `--migration-attempt-id`, `--recover-migration-id`, - `--recover-migration-attempt-id` or `-h`/`--help`. + `--recover-migration-attempt-id` or `-h`/`--help`. `MigrationCLI::__construct(array $arguments)` and + `MigrationCLI::getHelp()` are removed. - `Destinations\Appwrite::run()` finalizes again, as in 2.x: once the transfer returns it flips the databases it provisioned to `ready`, then runs the overwrite orphan sweep. - `success()`, `error()` and `cleanUp()` are no-op lifecycle hooks again. Callers no longer call `success()` to finalize, and `cleanUp()` no longer reports a skipped `success()`. - - A failed `ready` flip is recorded with `addError()`; a failed orphan sweep throws from `run()`. + - A failed `ready` flip is recorded with `addError()`, and the sweep still runs. + - The orphan sweep stops at the first table that fails, and `run()` throws that table's database exception as is. + 3.1.1's `success()` tried every table, recorded each failure with `addError()` and threw `Finalization`. - `Utopia\Migration\Exception\Finalization` is removed. - The abort mechanism is removed. - `Utopia\Migration\Exception\Aborted` and `Destination::markAborted()` are removed. - `Transfer::run()` no longer latches an exception thrown from its progress callback, and the library sources no longer rethrow one: each source handles it as in 2.x. -## 3.0.0 (unreleased) +### Concurrency + +- The Appwrite destination treats every `provisioning` or `failed` database as abandoned by an earlier run. Callers + must serialise migrations per destination project. A concurrent run overwrites another run's incomplete databases + from its own source under any `OnDuplicate`, and may flip them to `ready` before the other run finishes. + +## 3.0.0 3.0 is a major release. The breaking changes below all concern the Appwrite destination's lifecycle and the database library it builds on; read the upgrade notes before you upgrade from 2.x. From d82fb20d05fe3d897ec25733ac6fda9d5e012c6d Mon Sep 17 00:00:00 2001 From: Jake Barnby Date: Wed, 14 Oct 2026 20:05:30 +1300 Subject: [PATCH 6/8] chore(deps): require utopia-php/database ^8.1.2 8.1.2 is released and moves AttributeUpdate and RelationshipUpdate into the Attribute and Relationship namespaces with no alias, so the destination's imports follow them. Aliased as AttributeUpdate and RelationshipUpdate because both classes are now named Update. The constraint is stable now, so minimum-stability goes back to stable and prefer-stable and the stale 3.0.x branch aliases are dropped. Only utopia-php/database moves in the lock; its requirements are unchanged. Co-Authored-By: Claude Opus 5.5 --- composer.json | 11 ++--------- composer.lock | 18 +++++++++--------- src/Migration/Destinations/Appwrite.php | 4 ++-- 3 files changed, 13 insertions(+), 20 deletions(-) diff --git a/composer.json b/composer.json index c1f3be92..eb8e1ee8 100644 --- a/composer.json +++ b/composer.json @@ -10,8 +10,7 @@ "migration" ], "license": "MIT", - "minimum-stability": "dev", - "prefer-stable": true, + "minimum-stability": "stable", "autoload": { "psr-4": { "Utopia\\Migration\\": "src/Migration" @@ -33,7 +32,7 @@ "ext-curl": "*", "ext-openssl": "*", "appwrite/appwrite": "^30.2.2", - "utopia-php/database": "^8.0", + "utopia-php/database": "^8.1.2", "utopia-php/storage": "4.*", "utopia-php/dsn": "0.2.*", "halaxa/json-machine": "^1.2", @@ -53,11 +52,5 @@ "php-http/discovery": true, "tbachert/spi": true } - }, - "extra": { - "branch-alias": { - "dev-feat-query-lib": "3.0.x-dev", - "dev-main": "3.0.x-dev" - } } } diff --git a/composer.lock b/composer.lock index 79ad24c2..a3a776bf 100644 --- a/composer.lock +++ b/composer.lock @@ -4,7 +4,7 @@ "Read more about it at https://getcomposer.org/doc/01-basic-usage.md#installing-dependencies", "This file is @generated automatically" ], - "content-hash": "29449fc2362aa9af364bdf8af14c0fb5", + "content-hash": "4938aab4b4211b65ad5af5e402512d28", "packages": [ { "name": "adhocore/jwt", @@ -2396,16 +2396,16 @@ }, { "name": "utopia-php/database", - "version": "8.0.0", + "version": "8.1.2", "source": { "type": "git", "url": "https://github.com/utopia-php/database.git", - "reference": "e4c46dfe9b5fad60d8b47e09ecb0f134b90c47f7" + "reference": "79577115a047f8a0e24c7505eac8141513c4dfec" }, "dist": { "type": "zip", - "url": "https://api.github.com/repos/utopia-php/database/zipball/e4c46dfe9b5fad60d8b47e09ecb0f134b90c47f7", - "reference": "e4c46dfe9b5fad60d8b47e09ecb0f134b90c47f7", + "url": "https://api.github.com/repos/utopia-php/database/zipball/79577115a047f8a0e24c7505eac8141513c4dfec", + "reference": "79577115a047f8a0e24c7505eac8141513c4dfec", "shasum": "" }, "require": { @@ -2463,9 +2463,9 @@ ], "support": { "issues": "https://github.com/utopia-php/database/issues", - "source": "https://github.com/utopia-php/database/tree/8.0.0" + "source": "https://github.com/utopia-php/database/tree/8.1.2" }, - "time": "2026-10-09T08:45:12+00:00" + "time": "2026-10-14T07:00:00+00:00" }, { "name": "utopia-php/dsn", @@ -5192,9 +5192,9 @@ } ], "aliases": [], - "minimum-stability": "dev", + "minimum-stability": "stable", "stability-flags": {}, - "prefer-stable": true, + "prefer-stable": false, "prefer-lowest": false, "platform": { "php": ">=8.5", diff --git a/src/Migration/Destinations/Appwrite.php b/src/Migration/Destinations/Appwrite.php index da9ae990..d1d75833 100644 --- a/src/Migration/Destinations/Appwrite.php +++ b/src/Migration/Destinations/Appwrite.php @@ -25,7 +25,7 @@ use Appwrite\Services\Users; use Override; use Utopia\Database\Attribute as UtopiaAttribute; -use Utopia\Database\AttributeUpdate; +use Utopia\Database\Attribute\Update as AttributeUpdate; use Utopia\Database\Capability; use Utopia\Database\Collection as UtopiaCollection; use Utopia\Database\Database as UtopiaDatabase; @@ -42,9 +42,9 @@ use Utopia\Database\Permission; use Utopia\Database\Query; use Utopia\Database\Relationship as UtopiaRelationship; +use Utopia\Database\Relationship\Update as RelationshipUpdate; use Utopia\Database\RelationshipDeleteAction; use Utopia\Database\RelationshipSide; -use Utopia\Database\RelationshipUpdate; use Utopia\Database\Role; use Utopia\Database\Validator\IndexDefinition as IndexValidator; use Utopia\Database\Validator\Structure; From 18307bc1f0acb04f54a5a210b8966b8f5e74a969 Mon Sep 17 00:00:00 2001 From: Jake Barnby Date: Wed, 14 Oct 2026 20:05:31 +1300 Subject: [PATCH 7/8] refactor(destination): drop narrating comments from appwrite destination The resource group labels and the comments that restate the next line add nothing the constants and names do not already say. Co-Authored-By: Claude Opus 5.5 --- src/Migration/Destinations/Appwrite.php | 20 -------------------- 1 file changed, 20 deletions(-) diff --git a/src/Migration/Destinations/Appwrite.php b/src/Migration/Destinations/Appwrite.php index d1d75833..2298c035 100644 --- a/src/Migration/Destinations/Appwrite.php +++ b/src/Migration/Destinations/Appwrite.php @@ -378,7 +378,6 @@ public static function getName(): string public static function getSupportedResources(): array { return [ - // Auth Resource::TYPE_USER, Resource::TYPE_TEAM, Resource::TYPE_MEMBERSHIP, @@ -386,7 +385,6 @@ public static function getSupportedResources(): array Resource::TYPE_POLICIES, Resource::TYPE_OAUTH2_PROVIDER, - // Database Resource::TYPE_DATABASE, Resource::TYPE_DATABASE_DOCUMENTSDB, Resource::TYPE_DATABASE_VECTORSDB, @@ -395,48 +393,39 @@ public static function getSupportedResources(): array Resource::TYPE_INDEX, Resource::TYPE_ROW, - // legacy Resource::TYPE_DOCUMENT, Resource::TYPE_ATTRIBUTE, Resource::TYPE_COLLECTION, - // Storage Resource::TYPE_BUCKET, Resource::TYPE_FILE, - // Functions Resource::TYPE_FUNCTION, Resource::TYPE_DEPLOYMENT, Resource::TYPE_ENVIRONMENT_VARIABLE, - // Messaging Resource::TYPE_PROVIDER, Resource::TYPE_TOPIC, Resource::TYPE_SUBSCRIBER, Resource::TYPE_MESSAGE, - // Sites Resource::TYPE_SITE, Resource::TYPE_SITE_DEPLOYMENT, Resource::TYPE_SITE_VARIABLE, - // Integrations Resource::TYPE_PLATFORM, Resource::TYPE_API_KEY, Resource::TYPE_WEBHOOK, Resource::TYPE_SMTP, - // Project Resource::TYPE_PROJECT_VARIABLE, Resource::TYPE_PROJECT_PROTOCOLS, Resource::TYPE_PROJECT_LABELS, Resource::TYPE_PROJECT_SERVICES, Resource::TYPE_PROJECT_EMAIL_TEMPLATE, - // Backups Resource::TYPE_BACKUP_POLICY, - // Domains Resource::TYPE_RULE, ]; } @@ -458,7 +447,6 @@ public function report(array $resources = [], array $resourceIds = []): array // Most of these API calls are purposely wrong. Appwrite will throw a 403 before a 400. // We want to make sure the API key has full read and write access to the project. try { - // Auth if (\in_array(Resource::TYPE_USER, $resources)) { $scope = 'users.read'; $this->users->list(); @@ -483,7 +471,6 @@ public function report(array $resources = [], array $resourceIds = []): array $this->teams->createMembership('', [], ''); } - // Storage if (\in_array(Resource::TYPE_BUCKET, $resources)) { $scope = 'storage.read'; $this->storage->listBuckets(); @@ -500,7 +487,6 @@ public function report(array $resources = [], array $resourceIds = []): array $this->storage->createFile('', '', new InputFile()); } - // Functions if (\in_array(Resource::TYPE_FUNCTION, $resources)) { $scope = 'functions.read'; $this->functions->list(); @@ -509,7 +495,6 @@ public function report(array $resources = [], array $resourceIds = []): array $this->functions->create('', '', Runtime::NODE180()); } - // Messaging if (\in_array(Resource::TYPE_PROVIDER, $resources)) { $scope = 'providers.read'; $this->messaging->listProviders(); @@ -542,7 +527,6 @@ public function report(array $resources = [], array $resourceIds = []): array $this->messaging->createEmail('', '', '', draft: true); } - // Sites if (\in_array(Resource::TYPE_SITE, $resources)) { $scope = 'sites.read'; $this->sites->list(); @@ -963,8 +947,6 @@ protected function createField(Column|Attribute $resource): bool $resource->setStatus(Resource::STATUS_SKIPPED, 'Columns not supported for DocumentsDB'); return false; } - // column will be matching attribute as well - // column type will be matching attribute type as well $type = $this->schemaColumnType($resource); $database = $this->dbForProject->getDocument( @@ -1503,7 +1485,6 @@ protected function createIndex(Index $resource): bool return false; } - // Lengths hidden by default $lengths = []; if ($dbForDatabases->getAdapter()->supports(Capability::DefinedAttributes)) { @@ -1598,7 +1579,6 @@ protected function createRecord(Row $resource, bool $isLast): bool return false; } - // Check if document has already been created $exists = \array_key_exists( $resource->getId(), $this->cache->get($resource->getName()) From 84787a911ac0bf6ed0fbab34dda46c88c57cd3b8 Mon Sep 17 00:00:00 2001 From: Jake Barnby Date: Wed, 14 Oct 2026 20:05:31 +1300 Subject: [PATCH 8/8] docs(changelog): note the utopia-php/database ^8.1.2 requirement Consumers on 8.0 or 8.1.1 must upgrade the database library alongside 3.1.2, so the requirement leads the breaking changes as it did for 3.0.0. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 73b262dd..2b906e08 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,8 @@ ### Breaking changes +- Requires `utopia-php/database` `^8.1.2` (3.1.1 requires `^8.0`), which moved the update models into the namespace + of the model they update. - Provisioning ownership fencing is removed. The Appwrite destination tracks a database's lifecycle with `status` alone again, but recovers incomplete databases differently from 2.x. - `Destinations\Appwrite\ProvisioningOwner` is removed.