3.0.2: fix Core Data legacy migration + stop destroying user data

[kern]

Root cause

3.0.0's LegacyDataMigrator (commit e8704d7) destroyed v2.x users' on-disk dittos because of two coupled bugs:

1. The Core Data model wasn't in the main app's bundle. e8704d7 added LegacyDataMigrator.swift to the Sources build phase but did not add Ditto.xcdatamodeld to the main app target's Resources build phase — the xcdatamodel was (and still is on master) only in the keyboard extension's Resources phase. So at runtime:

   Bundle.main.url(forResource: "Ditto", withExtension: "momd")

   returned nil in the main app, the model-load guard fell through, and readLegacyStore returned [] without throwing.

2. The "no data found" branch deleted the SQLite. Once readLegacyStore returned empty, migrateIfNeeded ran:

   if legacyData.isEmpty {
       markComplete()
       cleanupLegacyFiles(at: storeURL)   // removes Ditto.sqlite, -shm, -wal
       return false
   }

   The cleanup used try? FileManager.removeItem(at:), which silently swallows errors. For some users it succeeded and they lost the SQLite outright; for others (e.g. keyboard extension holding an open handle, file protection, transient I/O error) it silently failed and the data is still physically present.

3.0.1's NSUserDefaults migrator didn't make this worse (it never deletes anything) but it was reading the wrong source — the 2.0.1 ship (commit 60f395d) confirms data lives in containerURL(forSecurityApplicationGroupIdentifier: "group.io.kern.ditto").appendingPathComponent("Ditto.sqlite") as Core Data, not NSUserDefaults.

Field signal

A user wrote in reporting 578 KB of Documents & Data still attributed to Ditto under Settings → General → iPhone Storage → Ditto. A fresh-install 3.0.x SwiftData container with just the preset categories sits in the ~50–150 KB range, so ~580 KB is well above baseline and strongly suggests the v2 Ditto.sqlite (or its WAL/SHM siblings) is still on disk on at least some affected devices. The try?-swallowed-errors theory matches — likely keyboard-extension-held file handles. That informed the hardening pass in 3b3e35d and the telemetry tag in d7248ce. Whether this is rare or common won't be known until we have field data from a TestFlight rollout (see test plan below).

What 3.0.2 changes

Migrator (data path)

• Bundle Ditto.xcdatamodeld into the main app's Resources (new PBXBuildFile pointing at the existing fileRef, added to FA70EAA919F5A49D00960EE2). Bundle.main can now load it.
• Rewrite LegacyDataMigrator to open the legacy SQLite read-only, walk Profile → ordered categories → ordered dittos, and map use_count → useCount on DittoItem. Schema and store path verified against commit 60f395d (the 2.0.1 ship).
• Never call removeItem on the legacy files, on any branch. Even successful migration leaves the SQLite where it is. A few KB of disk is cheap; losing user data is not.
• Bump the completion-flag key to legacyCoreDataMigrationComplete_v302, so any device that hit 3.0.0 or 3.0.1 retries the corrected migration on first 3.0.2 launch — no menu tap required if the data is still on disk.

Wider probe + observability (hardening for the 578 KB user)

• legacyStoreURL now probes case-variant filenames (Ditto.sqlite, ditto.sqlite, Ditto.SQLite) and additional in-container subdirectories (Library/Application Support, Library/Application Support/Ditto, Documents). When multiple candidates exist, the largest matching file is preferred — which handles the partial-cleanup case where 3.0.0 zero-truncated one path while the real data survived at another.
• Whenever legacyStoreURL runs, it logs a full inventory of every file under the App Group container (plus the app's Application Support / Documents / Library) with name and size at .info level. Affected users can capture this in a sysdiagnose so we can see exactly what survived 3.0.0's cleanup on their device.

Field telemetry (d7248ce)

• Every terminal exit of an auto- or manual-migration attempt emits a single-line, no-PII, all-.public tag:

  migration_outcome source=<auto|manual> outcome=<code> categories=N dittos=N inserted=N
  

  Outcomes: success | no_new_data | empty_store | found_unreadable | nothing_on_disk.
• Designed to be aggregated across a TestFlight cohort:

  log show --predicate 'subsystem == "io.kern.ditto"' --info | grep migration_outcome
  

  Counting the distribution of outcomes across sysdiagnoses will tell us whether the 578 KB user is representative (mostly success / no_new_data = cleanup silently failed for most users, data is recoverable) or an outlier (mostly nothing_on_disk = cleanup succeeded for most users, data is gone). Fields are .public so they survive sysdiagnose redaction on non-developer devices.

UI

• Wire the "Recover Old Dittos" menu item to the same Core Data path. It ignores the completion flag, so users can re-trigger after restoring an iCloud backup.

• The menu item is now visible whenever any legacy SQLite file is present (using hasRecoverableLegacyData), even if the file turns out to be unreadable. Tapping it runs recoverNow, which returns a structured RecoveryResult so the alert can surface why nothing was recovered:

  Result	User-facing message
  .nothingOnDisk	"No legacy dittos were found on this device. If you had dittos in an older version, they may have been removed by an earlier 3.0 update."
  .foundButUnreadable(detail)	"Found old data on this device, but couldn't read it. Please send a sysdiagnose so we can investigate." (with the underlying error)
  .emptyStore	"Found an old data file on this device, but it had no dittos in it."
  .inserted(0)	"Your old dittos were already in your current library — nothing new to recover."
  .inserted(N)	"Recovered N dittos from your previous version."

• CFBundleShortVersionString → 3.0.2 in Ditto/Info.plist and DittoKeyboard/Info.plist.

Docs

• Add docs/RECOVERING_LOST_DITTOS.md with step-by-step user-facing recovery instructions and a Console.app filter for verification.
• Includes a TestFlight-install callout: install the TestFlight build over the existing App Store version. Do not delete the App Store version first — iOS would wipe the per-app container and recovery would be impossible.

Recovery story per upgrade path

Path	Status after 3.0.2
2.0.1 → 3.0.1 (skipped 3.0.0)	SQLite intact in <groupURL>/Ditto.sqlite. 3.0.2 auto-migrates on first launch.
2.0.1 → 3.0.0 → 3.0.1, cleanup actually succeeded	SQLite was deleted by 3.0.0. Recoverable only by restoring an iCloud device backup made before the 3.0.0 launch, then installing 3.0.2 directly (don't open older versions).
2.0.1 → 3.0.0 → 3.0.1, cleanup silently failed (likely 578 KB user)	SQLite still on disk. 3.0.2 finds it, opens it read-only, imports everything. No user action needed beyond updating.
Fresh install	No-op. Migrator sees no legacy store, marks complete, never runs again.

How to verify on an affected device

With the iPhone connected to a Mac, open Console.app, select the device, filter by subsystem:io.kern.ditto category:LegacyDataMigrator, then launch 3.0.2. The inventory dump shows exactly what's in the App Group container, e.g.:

inventory[appgroup] 524288B /Ditto.sqlite
inventory[appgroup]  32768B /Ditto.sqlite-shm
inventory[appgroup]  16384B /Ditto.sqlite-wal
legacyStoreURL: matched <path>/Ditto.sqlite (524288 bytes)
runMigration(auto): importing 4 categories / 27 dittos
writeMigratedData: inserted 4 new categories, 27 new dittos (skipped 0 duplicates)
migration_outcome source=auto outcome=success categories=4 dittos=27 inserted=27

If the inventory shows a non-trivial Ditto.sqlite, the data is still there and recovery will succeed. If it shows only Ditto.store* (the SwiftData files) and no .sqlite, the cleanup did succeed on that device and iCloud restore is the only path.

Tester onboarding (for the 578 KB user and any other beta testers)

Open the TestFlight invite link and tap Install (or Update) on the Ditto entry. Do not delete the App Store version first. TestFlight replaces the App Store build in place, which keeps your existing App Group container — including any legacy data 3.0.2 needs to recover — intact. Deleting the App Store build first would wipe the per-app container.

Same App Group ID (group.io.kern.ditto) and same Team ID across App Store and TestFlight distribution are required for the new build to see the existing container. If you've migrated developer accounts at some point, double-check both distributions are still on the same Team ID in App Store Connect.

Test plan

• 2.0.1 → 3.0.2 directly: open 3.0.2 fresh on top of an in-place 2.0.1 install (or a synthetic <groupURL>/Ditto.sqlite built from the v2 schema) and confirm SwiftData contains the imported categories/dittos with order preserved.
• 3.0.1 → 3.0.2 with stale flag: pre-set legacyUserDefaultsMigrationComplete = true (the 3.0.1 key) on a device that still has Ditto.sqlite and confirm 3.0.2 auto-migrates because it uses a new flag key.
• 578 KB user simulation: place Ditto.sqlite (v2 schema, populated) at the App Group container root and also at Library/Application Support/Ditto.sqlite, with different sizes. Confirm the migrator picks the larger one.
• File present but unreadable: place a zero-byte or garbage Ditto.sqlite in the container. Confirm "Recover Old Dittos" menu item is still visible, the alert text is .foundButUnreadable(...) with a usable error string, and the outcome tag is migration_outcome source=manual outcome=found_unreadable.
• Inventory log: confirm the inventory[appgroup] <size>B <relative-path> lines appear in Console at .info level under io.kern.ditto / LegacyDataMigrator.
• Telemetry tag: confirm a migration_outcome ... line is emitted at every terminal path (nothing_on_disk / found_unreadable / empty_store / no_new_data / success) and survives sysdiagnose collection on a non-developer device.
• Idempotency: launch 3.0.2 twice, confirm second launch doesn't double-insert (completion flag short-circuits) and <groupURL>/Ditto.sqlite is still on disk.
• Manual recovery: pre-set legacyCoreDataMigrationComplete_v302 = true to simulate a previously-migrated install, then place a fresh SQLite at <groupURL>/Ditto.sqlite, confirm "Recover Old Dittos" inserts only new dittos (duplicates skipped).
• Data preservation: after every success and every failure path above, confirm Ditto.sqlite / -shm / -wal are still on disk.
• Fresh install: confirm migrateIfNeeded marks complete and returns false, no menu item appears, no inventory contains a legacy SQLite, outcome tag is nothing_on_disk.
• Unit tests pass (DittoTests/LegacyDataMigratorTests.swift exercises the v2 schema read on a temp-dir fixture).
• Beta with the 578 KB user: ship a TestFlight build to that customer first, confirm recovery succeeds (outcome tag = success with non-zero inserted) before App Store submission.
• Cohort aggregation: after a few days on TestFlight, log collect from a handful of tester devices and grep migration_outcome to see the rough distribution of outcomes across the field. Use that distribution to decide between "ship 3.0.2 widely with a hopeful tone" vs "ship with iCloud-restore as the headline recovery path".

Closes the 3.0.0 data-loss regression. Companion user-facing instructions at docs/RECOVERING_LOST_DITTOS.md.