Skip to content

fix(datastore): merge disjoint aw-watcher-android-test events on name collision - #661

Open
TimeToBuildBob wants to merge 3 commits into
ActivityWatch:masterfrom
TimeToBuildBob:fix/merge-legacy-android-bucket
Open

fix(datastore): merge disjoint aw-watcher-android-test events on name collision#661
TimeToBuildBob wants to merge 3 commits into
ActivityWatch:masterfrom
TimeToBuildBob:fix/merge-legacy-android-bucket

Conversation

@TimeToBuildBob

Copy link
Copy Markdown
Contributor

Summary

migrate_test_bucket_names() (from #628) renamed aw-watcher-android-test* buckets with UPDATE OR IGNORE. That no-ops when the canonical aw-watcher-android_* bucket already exists — exactly the post-upgrade state in ActivityWatch/aw-android#243, where years of history sit in the old bucket and the Activity view queries the new empty-ish one.

This replaces the rename-only path with:

  1. Rename when the destination does not exist.
  2. Merge disjoint events when both buckets exist: move legacy events that do not overlap any destination event.
  3. Leave overlapping events in the legacy bucket so we never create duplicate activity records. Delete the legacy bucket only once it is empty.

A single overlapping cutover heartbeat no longer strands the rest of the history.

Also rebuilds the in-memory bucket cache from SQLite after the rewrite so deleted buckets disappear (the previous insert-only refresh kept stale IDs).

Why not skip the whole bucket on any overlap?

Erik's constraint in ActivityWatch/aw-android#243 is merge when it is safe/non-overlapping. Skipping the entire legacy bucket because one event overlaps at the upgrade boundary would leave years of disjoint history invisible. Partial merge matches that constraint without throwing away the rest.

Tests

  • test_migrate_test_bucket_names_renames_bucket_and_preserves_events
  • test_migrate_test_bucket_names_merges_non_overlapping_buckets
  • test_migrate_test_bucket_names_keeps_overlapping_buckets_separate
  • test_migrate_test_bucket_names_merges_interleaved_non_overlapping_events
  • test_migrate_test_bucket_names_moves_disjoint_events_when_some_overlap (new)

cargo test -p aw-datastore --test datastore — 16 passed.

Follow-up

ActivityWatch/aw-android still never calls this JNI. Companion PR will wire migrateWatcherAndroidBucketNames() from BackgroundService startup.

Refs: ActivityWatch/aw-android#149, ActivityWatch/aw-android#150, ActivityWatch/aw-android#243

UPDATE OR IGNORE skipped the collision case where both the legacy
aw-watcher-android-test_* bucket and the canonical aw-watcher-android_*
bucket exist, leaving years of history stranded after upgrades
(ActivityWatch/aw-android#243).

Move non-overlapping legacy events into the canonical bucket, leave
overlapping events in the legacy bucket, and delete the legacy bucket
only once it is empty. Replace the in-memory bucket cache from SQLite
after the rewrite so deleted buckets disappear.

Refs: ActivityWatch/aw-android#149, ActivityWatch/aw-android#150
@greptile-apps

greptile-apps Bot commented Aug 27, 2026

Copy link
Copy Markdown

Greptile Summary

The PR safely completes the Android legacy-bucket migration fix by retaining mutually overlapping legacy events while moving disjoint history into the canonical bucket.

  • Checks candidate legacy events against both destination and legacy bucket events before moving them.
  • Deletes a legacy bucket only after all its events have been moved.
  • Rebuilds the bucket cache after migration so removed and renamed buckets are represented correctly.
  • Adds migration coverage for overlapping legacy events and partial merges.
  • Configures a SQLite busy timeout for concurrent access.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the updated predicate keeps mutually overlapping legacy events together, resolving the previously reported migration defect.

Important Files Changed

Filename Overview
aw-datastore/src/datastore.rs The migration now prevents mutually overlapping legacy events from moving together, preserves conflicting events in the legacy bucket, and rebuilds the cache from SQLite.
aw-datastore/src/worker.rs Adds a five-second SQLite busy timeout before configuring journal and synchronization behavior.
aw-datastore/tests/datastore.rs Adds focused coverage for rename, full merge, partial merge, destination overlap, legacy overlap, and interleaved disjoint-event scenarios.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Legacy Android bucket] --> B{Canonical bucket exists?}
  B -- No --> C[Rename legacy bucket]
  B -- Yes --> D[Inspect each legacy event]
  D --> E{Overlaps destination or another legacy event?}
  E -- No --> F[Move event to canonical bucket]
  E -- Yes --> G[Keep event in legacy bucket]
  F --> H{Legacy bucket empty?}
  G --> H
  H -- Yes --> I[Delete legacy bucket]
  H -- No --> J[Retain partially merged bucket]
  C --> K[Rebuild bucket cache]
  I --> K
  J --> K
Loading

Reviews (2): Last reviewed commit: "fix(datastore): retain overlapping legac..." | Re-trigger Greptile

Comment thread aw-datastore/src/datastore.rs
@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

Windows sync_roundtrip::test_own_data_does_not_return_via_peer failed with database is locked in the datastore worker (journal_mode query). Unrelated to this change — same flake as #655. Ubuntu, macOS, Android, clippy, and format are green; datastore tests including the new merge cases passed.

@codecov

codecov Bot commented Aug 27, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.73171% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.40%. Comparing base (656f3c9) to head (edbf0c9).
⚠️ Report is 93 commits behind head on master.

Files with missing lines Patch % Lines
aw-datastore/src/datastore.rs 89.74% 4 Missing ⚠️
aw-datastore/tests/datastore.rs 97.58% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #661      +/-   ##
==========================================
+ Coverage   70.81%   79.40%   +8.58%     
==========================================
  Files          51       66      +15     
  Lines        2916     5603    +2687     
==========================================
+ Hits         2065     4449    +2384     
- Misses        851     1154     +303     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Set a 5-second busy_timeout after opening SQLite connections to allow
the database to wait for locks rather than immediately failing with
DatabaseBusy errors. This fixes test failures on Windows where multiple
connections to the same database file can create lock contention.

Fixes windows-latest CI failure in sync_roundtrip tests.
@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

Fixed Greptile's P1 in edbf0c9: the move predicate now rejects legacy events that overlap either the destination or another legacy event. Added a regression test that failed before the fix and confirms the overlapping pair remains in the legacy bucket while a disjoint event still moves. The full datastore integration test passes (17/17).

@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

@greptileai review

@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

CI-green and mergeable (Greptile 5/5) — waiting only on a maintainer click.

This PR is ready to merge, but the bot has pull-only access to this repo and can't self-merge — surfacing it here so it isn't lost. The monitoring loop will stop re-flagging it now that this note is posted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant