Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,20 @@ guidance. Refer to the corresponding local guide when working on that component:
- **Report the version:** Explicitly state the newly bumped version in the final
response to the user.

## Task Tracking & Documentation Lifecycle (TODO vs. DONE)

- **Never mark items as `[COMPLETED]` inline within `docs/TODO/`:** When a feature,
milestone, or roadmap task is completed and verified, do NOT retain it in
`docs/TODO/` with a `[COMPLETED]` or `[DONE]` tag.
- **Migrate to `docs/DONE/`:** Completely remove finished tasks from
`docs/TODO/roadmap-backlog.md` (and other backlog files in `docs/TODO/`) and
transfer their implementation reports, architectural notes, and verification audits
into `docs/DONE/` (such as dedicated milestone files and updating
`docs/DONE/implementation-status.md`).
- **Comprehensive Documentation Updates:** Fully update all affected documentation,
status tables, component versions, and cross-references across `docs/` in lockstep
whenever an initiative lands.

## Editing Rules

- Preserve unrelated user changes: check `git status --short` inside the
Expand Down
1 change: 1 addition & 0 deletions docs/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ This directory (`docs/`) contains the architectural blueprints, technical specif
## Operating Principles for Agents

- **Sync with Code**: Ensure architectural documents accurately reflect current implementations in `passly_api`, `passly_browser_extension`, `passly_android`, and `passly_styleguide`.
- **Task Lifecycle & Backlog Discipline (TODO vs. DONE)**: Never mark items as `[COMPLETED]` or `[DONE]` inline inside `docs/TODO/`. Fully remove finished items from `docs/TODO/` and transfer their completion reports, verification records, and architecture summaries into `docs/DONE/`. Fully update all overview tables and cross-references across `docs/` in lockstep.
- **Localization Parity**: When updating foundational concepts in `architecture/`, maintain parity with `UA/architecture/`.
- **Naming & Identifiers**: Do not invent `PB-` or Passbolt ticket prefixes. Use plain descriptions or verified Passly issue IDs.
- **Security & Hygiene**: Never insert private keys, tokens, session data, or production credentials into documentation.
2 changes: 1 addition & 1 deletion docs/DONE/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ This directory (`docs/DONE/`) tracks delivered milestones, implemented architect

## Guidelines for Agents

- **Moving Items from TODO**: When a task meets its Definition of Done and passes verification, move its documentation here.
- **Moving Items from TODO (No `[COMPLETED]` tags in TODO)**: Never leave completed items marked as `[COMPLETED]` in `docs/TODO/`. When a task meets its Definition of Done and passes verification, completely remove it from `docs/TODO/` and move its documentation and verification reports here, updating `docs/DONE/implementation-status.md` and related references.
- **Traceability**: Reference affected subprojects, key commits, or PRs where applicable.
- **Conventions**: Keep records concise, factual, and free of fictitious `PB-` ticket prefixes.
- **Security**: Never log real credentials, cryptographic keys, or private tokens in completion reports.
124 changes: 124 additions & 0 deletions docs/DONE/android-delta-sync-m3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
# Milestone M3: Android Client Incremental Delta Sync (`sync/1`)

**Release Version:** `6.0.12` (Build code `69`, initial release `6.0.11`)
**Target Subproject:** `passly_android`
**Reference Specification:** [`passly_android/docs/sync-protocol.md`](../../passly_android/docs/sync-protocol.md)
**Status:** Completed, Verified, and Merged

---

## 1. Overview

Milestone M3 delivers the client-side delta synchronization engine for the Android application (`passly_android`). Prior to this milestone, the Android client relied on full background data refreshing (`core/fulldatarefresh`). M3 fully operationalizes the client-side implementation of the `sync/1` wire protocol, consuming the server change journal (`sync_log`) and endpoints delivered in Milestone M2 (`/sync/changes.json`, `/sync/bootstrap.json`, `/sync/manifest.json`, and `/sync/meta.json`).

The implementation strictly satisfies all core invariants of the Passly offline architecture:
- **R0 (Autonomous Store):** The local Room database is a self-sufficient replica. Sync is an enhancement for freshness, never availability.
- **R3 (Monotonic Delta & Atomic Application):** Sync cursor is a monotonic 64-bit integer, never a timestamp. Delta pages and cursor updates commit in a single database transaction.
- **R4 (Transparent Recovery):** Server retention purges (`410 Gone`) or epoch changes transparently trigger a bootstrap on top of existing local data without data loss.
- **Non-blocking Execution:** Synchronization operates in the background via WorkManager or as user-initiated actions, never blocking local reads or writes.

---

## 2. Architecture & Delivered Components

```
passly_android/
├── core/database/
│ ├── schemas/net.svaroh.passly.database.ResourceDatabase/31.json
│ └── src/main/java/
│ ├── net/svaroh/passly/database/impl/sync/SyncStateDao.kt
│ └── com/passbolt/mobile/android/database/
│ ├── migrations/Migration30to31.kt
│ └── ResourceDatabase.kt (version 31)
├── model/entity/
│ └── src/main/java/net/svaroh/passly/entity/sync/SyncState.kt
└── core/sync/
└── src/main/java/net/svaroh/passly/core/sync/
├── api/SyncApi.kt
├── dto/SyncDtos.kt
├── ChangesApplier.kt
├── DeltaSyncStrategy.kt
├── BootstrapStrategy.kt
├── ManifestReconciler.kt
├── SyncEngine.kt
├── SyncResult.kt
├── SyncTrigger.kt
├── SyncStatus.kt
├── SyncStatusRepository.kt
├── SyncBackoff.kt
├── SyncModule.kt
└── work/
├── SyncWorker.kt
└── SyncScheduler.kt
```

### 2.1. Persistent Sync State & Room Database Migration 30 -> 31
- **`SyncState` Entity:** Stores `cursor: Long?`, `epoch: String?`, `lastSyncTimestamp: Long?`, `lastReconcileTimestamp: Long?`, and `failureCount: Int`.
- **`SyncStateDao`:** Provides queries for retrieving sync state, updating the cursor, recording reconciliation timestamps, and managing retry failure counters.
- **`Migration30to31`:** Automates database schema evolution, creating the `SyncState` table with zero downtime or data loss across app updates.
- **Batch Deletions:** Added `deleteAll(ids: List<String>)` in `ResourcesDao`, `FoldersDao`, `UsersDao`, and `GroupsDao` to support atomic batch deletions.

### 2.2. Atomic Batch Application: `ChangesApplier`
- Evaluates additions, modifications (`upserts`), and deletions (`deletions`) inside a single Room transaction (`DatabaseTransactionRunner`).
- Idempotently updates metadata keys, resource types, users, groups, folders, resources, secrets, tags, and cross-references.
- Applies deletions cascaded across resources, folders, users, and groups.
- Persists the new cursor and epoch atomically with the data, preventing the cursor from advancing ahead of uncommitted local state.

### 2.3. Delta Engine: `DeltaSyncStrategy`
- Queries `GET /sync/changes.json?since={cursor}&limit=500`.
- Executes iterative pagination while `has_more == true`, accumulating applied changes.
- Intercepts HTTP `410 Gone` (purged cursor past server 90-day retention or epoch reset) and returns `DeltaSyncResult.Gone` for transparent recovery.

### 2.4. Initial & Recovery Bootstrap: `BootstrapStrategy`
- Queries `GET /sync/bootstrap.json?page=N&limit=500&cursor={pinnedCursor}`.
- Pins the cursor on page 1 and passes it through subsequent pages to guarantee a consistent snapshot even if new mutations occur concurrently on the server.
- Safely overlays data on top of existing local records without purging user data.

### 2.5. Access Revocation & State Reconciliation: `ManifestReconciler`
- Queries the compact manifest `GET /sync/manifest.json?type=resources`.
- Diffing algorithm compares server manifest against local `Resource` and `Secret` entries.
- Purges orphaned local resources whose visibility was revoked on the server.
- Updates `lastReconcileTimestamp` upon completion.

### 2.6. Central Orchestrator: `SyncEngine`
- Single entry point (`sync(trigger: SyncTrigger): SyncResult`) for all sync invocations:
- `MANUAL`: Pull-to-refresh or force-sync button (forces manifest reconciliation).
- `UNLOCK`: Post-authentication refresh.
- `PENDING_WRITE`: Triggered when offline writes are queued.
- `BACKGROUND`: Scheduled background runs.
- Guarded by coroutine `Mutex` to prevent concurrent sync operations.
- Checks connectivity via `NetworkStateProvider`. Returns `Skipped(NO_NETWORK)` when offline without throwing errors or blocking the UI.
- Schedules retries using `SyncBackoff` (30s -> 1m -> 5m -> 15m -> 1h -> 6h) and records failure telemetry in `SyncStatusRepository`.

### 2.7. Background WorkManager Scheduler: `SyncWorker` & `SyncScheduler`
- **`SyncWorker`:** `CoroutineWorker` integrated with Koin dependency injection, invoking `SyncEngine.sync(SyncTrigger.BACKGROUND)`.
- **`SyncScheduler`:** Enqueues periodic WorkManager jobs with `NetworkType.CONNECTED` constraints and exponential backoff, plus support for immediate one-time triggers.

### 2.8. UI Refresh Integration & Data Absence Policy: `FullDataRefreshExecutor` (v6.0.12)
- Integrates `SyncEngine` directly into `FullDataRefreshExecutor` used by `HomeScreen` and `OtpScreen` pull-to-refresh and initial screen loading.
- **Strict Rule: Full synchronization is executed strictly only when local replica data is absent.**
- When local replica data exists (`hasLocalReplica == true`), the system runs lightweight incremental delta sync (`SyncEngine.sync(SyncTrigger.MANUAL)`), eliminating prolonged UI blocking progress spinners.
- When local replica data is completely absent (first app run on a fresh install), the system executes full initial data refresh / bootstrap.

---

## 3. Verification & Test Suite

The implementation is verified by 31 automated tests across 9 test suites in `core/sync/src/test`:

| Test Suite | Tests | Verification Focus |
|---|:---:|---|
| **`SyncIntegrationE2ETest`** | 3 | End-to-end integration lifecycle: cold-start bootstrap, incremental delta sync, 410 Gone recovery, and access revocation via manifest reconciliation. |
| **`SyncEngineTest`** | 7 | Orchestration, mutex locks, offline skipping, transparent 410 fallback, session failure handling, and retry scheduling. |
| **`DeltaSyncStrategyTest`** | 4 | Multi-page delta pagination (`hasMore`), cursor advancement, 410 Gone interception, and network failure propagation. |
| **`BootstrapStrategyTest`** | 3 | Multi-page bootstrap with pinned cursor consistency and error recovery. |
| **`ManifestReconcilerTest`** | 2 | Detection and deletion of revoked local resources; error handling. |
| **`ChangesApplierTest`** | 1 | Atomic database transactions, batch deletions, and cursor advancement. |
| **`SyncWorkerTest`** | 3 | WorkManager `doWork()` execution, mapping `SyncResult` to `Result.success()` and `Result.retry()`. |
| **`SyncStatusRepositoryTest`** | 5 | In-memory status publisher state transitions (`Idle`, `Running`, `Offline`, `Failed`). |
| **`SyncBackoffTest`** | 3 | Adherence to backoff ladder; immediate zero-delay retry for user-initiated triggers. |

### Build & Migration Verification
- Room migration test compiled: `:database:compileDebugAndroidTestKotlin` — **BUILD SUCCESSFUL**.
- Full test suite passed: `:sync:testDebugUnitTest` — **31 tests passed, 0 failures, 0 errors**.
- Full application package build: `:app:assembleDebug` — **BUILD SUCCESSFUL** (output artifact: `app-debug.apk`, version `6.0.11`, build code `68`).
32 changes: 26 additions & 6 deletions docs/DONE/implementation-status.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,24 @@
# Completed Implementation Milestones: Plan (`PLAN.md`) vs. Reality

**Audit Date:** October 16, 2026
**Audit Date:** November 2, 2026
**Reference Document:** `PLAN.md` (Dated 2026-08-07)
**Status:** Landed & Verified Deliverables

---

## 1. Executive Summary

An audit of the codebase across all four subprojects confirms that the **foundational offline architecture has been implemented in production**, with the browser extension, styleguide, and WebAuthn / Passkeys progressing beyond initial targets to release **v6.1.0**.
An audit of the codebase across all subprojects confirms that the **foundational offline architecture has been implemented in production**, with the backend protocol `sync/1` landed in API **v6.1.0**, the web extension and styleguide progressing to **v6.1.0**, and the Android client reaching release **v6.0.11** with full incremental delta sync and Room schema 31.

| Component | Target in PLAN.md | Actual Status in Codebase | Completed Deliverables |
|-----------|-------------------|---------------------------|------------------------|
| **`passly_android`** | 3.1.x | **6.0.8** (commit `daebc1379`) | **M0** and **M1 (Autonomy)**: Local secret persistence, non-blocking unlock, hardware Keystore decoupled from biometrics, `core/sync` foundation. |
| **`passly_android`** | 3.1.x | **6.0.12** (versionCode `69`) | **M0, M1 (Autonomy), M3 (Incremental Delta Sync)**: Local secret persistence, non-blocking unlock, hardware Keystore decoupled from biometrics, `core/sync` engine (`SyncEngine`, `DeltaSyncStrategy`, `ChangesApplier`, `BootstrapStrategy`, `ManifestReconciler`), Room schema 31 (`SyncState`), WorkManager scheduler (`SyncWorker`, `SyncScheduler`), fast delta refresh integrated into `FullDataRefreshExecutor` (full sync restricted strictly to empty local replica). |
| **`passly_api`** | 5.12.x | **6.1.0** (commits up to `2c021d5`) | **M2 (Full Protocol `sync/1`)**: `PassboltCe/Sync` plugin (`sync_log`, `/sync/meta`, `/sync/changes`, `/sync/bootstrap`, `/sync/manifest`, sync audit logging), adapted `OfflineMode` plugin. |
| **`passly_browser_extension`** | 6.0.13 | **6.1.0** (tag `v6.1.0`) | **E-A, E-B, E-B1, E-C, E-V, E-D**: Containerized workspace, offline session unlock, local-first reads, Seamless Offline Routing, offline-first passkeys, biometric unlock. |
| **`passly_styleguide`** | v5.12.1 (fork) | **6.1.0** (tag `v6.1.0`) | Fully adapted for autonomous execution, offline states, and biometric auth dialogs. |

> [!NOTE]
> All pending milestones and unfulfilled requirements have been transitioned to the roadmap backlog in **[docs/TODO/roadmap-backlog.md](../TODO/roadmap-backlog.md)**.
> All pending milestones and unfulfilled requirements are tracked in the roadmap backlog in **[docs/TODO/roadmap-backlog.md](../TODO/roadmap-backlog.md)**.

---

Expand Down Expand Up @@ -57,6 +57,27 @@ An audit of the codebase across all four subprojects confirms that the **foundat

---

### Milestone M3: Android Client Incremental Delta Sync (Requirement R3)
- **Reality:** ✅ **Fully Landed in v6.0.12.**
- **Detailed Audit & Verification:** See dedicated report in **[docs/DONE/android-delta-sync-m3.md](android-delta-sync-m3.md)**.
- **Delta Synchronization Strategies & UI Integration:**
- `DeltaSyncStrategy`: Pulls `/sync/changes.json?since={localSeq}`, loops through paginated changes up to target watermark, and applies updates to Room DB.
- `ChangesApplier`: Atomically applies entity additions, updates, and deletions for `resources`, `secrets`, and `permissions` inside a Room transaction while advancing `localSeq`.
- `BootstrapStrategy`: Handles initial synchronization and transparent recovery when receiving `410 Gone` (cursor expired), fetching full encrypted snapshots from `/sync/bootstrap.json`.
- `ManifestReconciler`: Lightweight periodic sanity check verifying local entity counts against `/sync/manifest.json`, triggering automatic recovery upon discrepancy.
- `SyncEngine`: Master coordinator orchestrating delta sync, 410 Gone recovery fallback, manifest reconciliation, and telemetry.
- `FullDataRefreshExecutor` Integration: Fully wired into `SyncEngine`. Enforces strict offline-first policy where full synchronization is executed **strictly only when local replica data is completely absent**. Whenever local data exists, rapid incremental delta sync is executed, eliminating UI blocking spinners on `HomeScreen` and `OtpScreen`.
- **Database Migration (Room Schema 31):**
- Created `SyncState` entity and `SyncStateDao` to persist `localSeq`, `lastSyncTimestamp`, and `lastManifestCheck`.
- Implemented `Migration30to31.kt` with auto-migration and automated Room migration verification tests.
- **Background WorkManager Integration:**
- `SyncWorker`: Periodic and on-demand `CoroutineWorker` integrated with network connectivity constraints and exponential backoff.
- `SyncScheduler`: Schedules periodic background sync (15-min intervals) and dispatches immediate manual sync requests.
- **Koin DI Wiring:** Complete module registration in `SyncModule.kt` loaded into `KoinInitializer.kt`.
- **Verification & Test Coverage:** 31 automated unit and E2E tests passing with 0 failures (`./gradlew :sync:testDebugUnitTest`), production debug APK built (`app-debug.apk` v6.0.11).

---

### Milestone M4: Status Messaging & Neutral Phrasing (Requirement R4)
- **Reality:** ✅ **Landed.**
- Offline status messaging and neutral phrasing have been integrated across Android screens.
Expand Down Expand Up @@ -86,8 +107,7 @@ All planned read-only and autonomous workspace objectives are fully implemented

## 4. Pending Initiatives

The following unfulfilled roadmap items have been transferred to **[docs/TODO/roadmap-backlog.md](../TODO/roadmap-backlog.md)**:
- **Milestone M3**: Incremental Delta Sync client in `passly_android`.
The following unfulfilled roadmap items are tracked in **[docs/TODO/roadmap-backlog.md](../TODO/roadmap-backlog.md)**:
- **Milestone M4**: User-configurable sync preferences screen.
- **Milestone M5 & Step E-E**: Offline write queue (`pending_ops`) and conflict resolution (`sync_conflicts`).
- **Milestone M6**: Standalone emergency vault export (KDBX 4 / PGP-JSON).
2 changes: 1 addition & 1 deletion docs/TODO/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,5 @@ This directory (`docs/TODO/`) tracks pending architectural requirements, upcomin

- **Defining Tasks**: Structure items with clear objectives, affected subprojects (`passly_api`, `passly_browser_extension`, `passly_android`, `passly_styleguide`), and verifiable acceptance criteria.
- **Architectural Alignment**: Ensure all planned designs adhere to Passly's core requirement (R0: full offline autonomy) and cryptographic guidelines.
- **Transitioning**: Once an initiative is completed, verified, and merged, move its documentation from `TODO/` to `DONE/`.
- **Task Migration (No `[COMPLETED]` tags)**: Never mark completed tasks or milestones with `[COMPLETED]` or `[DONE]` inline inside `docs/TODO/`. Once an initiative is completed and verified, completely remove it from `docs/TODO/` and transfer its documentation and verification records into `docs/DONE/`, fully updating `docs/DONE/implementation-status.md` and all related documentation.
- **Naming**: Use clear, descriptive titles without invented `PB-` prefixes.
Loading