test(datastore): check that migrating produces the schema SpiceDB expects

[vroldanbet]

What

Adds TestSchemaDrift to the shared datastore test suite, so every engine gets it: migrate an empty database to head, record the schema, build the datastore exactly as the server does, record the schema again, and require the two to be identical.

It found one difference, on CockroachDB, which this PR also fixes: the expiration migration now sets ttl_disable_changefeed_replication on the two relationship tables, instead of leaving it to be set by an ALTER TABLE on every datastore construction.

Why

We check that migrations run. We never check what they produce.

TestMigration applies each migration in order, confirms the recorded version advances, and confirms data written before a migration survives it. It never compares the resulting schema against anything. So a migration that is simply incomplete passes.

When that happens, the usual repair is to fix the schema up at startup: the datastore issues the missing statement itself, every time it is constructed. Nothing then fails, because by the time any other test looks, the schema is correct. That is exactly how the CockroachDB TTL parameter ended up where it is - a human noticed it much later by reasoning about Watch, not because a test said so.

TestSchemaDrift turns that from something you have to notice into something that fails. It asserts a rule rather than a snapshot - migrations own the schema, startup does not change it - so there is no golden file, nothing to update as migrations are added, and nothing that breaks when a CI image moves to a new engine version.

How

SchemaDriftTest lives in pkg/datastore/test under the existing Migration category, alongside MigrationTest. memdb is the only engine excluded, and only because it has no migrations at all.

Each engine registers how to read its own schema, via test.RegisterSchemaSnapshotter. Both snapshots come from the same database on the same server seconds apart, so the text needs no normalising across engine versions:

• CockroachDB: SHOW CREATE ALL TABLES, which carries the table storage parameters where the drift actually was.
• MySQL: SHOW CREATE TABLE per table. The auto-increment counter is stripped, because it moves when a row is written rather than when the schema changes.
• Spanner: the database's own DDL, from the admin API.
• PostgreSQL: it has no SHOW CREATE, so columns, indexes, constraints and storage parameters are read out of the system catalogs.

Run against real containers, the only difference found anywhere was on CockroachDB, on both tested versions:

- ) WITH (ttl = 'on', ttl_expiration_expression = 'expires_at', ttl_job_cron = '@daily');
- ) WITH (ttl = 'on', ttl_expiration_expression = 'expires_at', ttl_job_cron = '@daily');
+ ) WITH (ttl = 'on', ttl_expiration_expression = 'expires_at', ttl_job_cron = '@daily', ttl_disable_changefeed_replication = true);
+ ) WITH (ttl = 'on', ttl_expiration_expression = 'expires_at', ttl_job_cron = '@daily', ttl_disable_changefeed_replication = true);

The second commit closes it: the expiration migration sets the parameter as part of the TTL policy, on clusters at v24.1 or later, which is the same gate the startup check already uses. Older clusters get the policy without it, as before.

The startup check stays, permanently. A database migrated before this change has that migration recorded as applied and will never run it again, so the check is the only thing that ever sets the parameter there. It also leaves alone an operator who has deliberately configured it. The comment in ttl_changefeed.go now says so.

References

Verified against containers for every engine: CockroachDB 25.2.0 and 26.2.5, PostgreSQL 14 and 18, MySQL 8, the Spanner emulator, and memdb (skipped, no migrations). Shown failing on CockroachDB before the migration change and passing after; TestTTLChangefeedSuppressionParam and TestTTLChangefeedSuppressionWatch still pass.

Considered and skipped: a committed per-engine schema snapshot. CI covers CockroachDB 25.2 and 26.2, PostgreSQL 14 through 18, MySQL and Spanner, and the catalogs differ between those versions in ways that have nothing to do with SpiceDB - so a golden file would need hand-editing on every image bump, and would go stale rather than catch anything. The invariant test needs no maintenance and finds the same class of bug.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!