From cc0f03beb0fe93bbfb45847518d9453a7bba8f5b Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 12 Nov 2026 18:56:47 +0000 Subject: [PATCH 01/11] Keep the sync position inside the history it describes (6.11.1) AdGuard hands each tab the storage it had when the tab loaded, so a tab left open writes back an older history carrying the history id it was loaded with. After any history loss, old and new tabs kept swapping two ids, and every new page took the mismatch for a lost history: a full download and a "Restored N visited URLs" note on page after page. With matching ids, the configuration's cursor instead claimed rows the older copy did not hold, and they were never downloaded again. The history document now records how far into the worker's history it is complete (the worker named by a hash of its endpoint), saved in the same write as the rows. A sync resumes from there, so an older copy quietly catches up on the difference; only a history with no position at all is treated as lost. The session marker now also requires the history to reach the cursor it recorded. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VCuhK4TQqcEbLXdmxLjggy --- CHANGELOG.md | 8 +++ README.md | 2 +- javstore-full-layout-cleanup.user.js | 82 +++++++++++++++++++++++++--- tests/harness.mjs | 50 ++++++++++++++--- tests/sync-session.test.mjs | 45 ++++++++++++++- worker/src/worker.mjs | 2 +- 6 files changed, 168 insertions(+), 21 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index df0d4f6..8ddd065 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,13 @@ # Changelog +## 6.11.1 + +- Fix: on AdGuard, "Restored N visited URLs from the worker" could appear on page after page while other JavStore tabs were open, and each time the device downloaded its whole history again. AdGuard gives every tab the storage it had when that tab loaded, and a tab never sees what other tabs save after that. So a tab left open for a while saves an older copy of the history, carrying the history id it was loaded with. After any loss of history (an AdGuard update wiping storage, for example), those old tabs and the newer ones kept swapping two ids. Every new page then took the mismatch for a lost history. +- Fix: in the same situation with matching ids, an older copy written back by a long-open tab could lose visits for good. The sync bookmark still claimed the device held everything up to the newest row, so what the old copy lacked was never downloaded again. +- Change: the history now records how far into the worker's history it is complete, in the same save as the visits themselves. The record names the worker by a hash, never by its address. A page that finds an older copy catches up from where that copy reaches, quietly and only for the difference. The note and the full download are left for a history that has really been lost. Histories saved by 6.11.0 or earlier keep using the bookmark from the sync settings until their next sync. +- Tabs that were already open before updating keep running the code they loaded with, so the note can still appear until they are reloaded or closed. +- The worker version moves to 6.11.1 with the userscript; the worker is unchanged. + ## 6.11.0 - Share initial cloud sync across a browsing session: page navigation and new tabs reuse locally stored history instead of starting another download. Sessions expire after 30 minutes without startup/sync activity; a new session resumes from the durable cursor. diff --git a/README.md b/README.md index 24663e9..d252bc8 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ Once it is deployed, open the `JVS` panel → **Cloud sync**, enter the worker U - The Cloud sync panel shows the worker version from the worker's last answer. When the worker is behind the userscript, a **⚠ Redeploy worker** button appears on screen next to the JavStore controls; tap it for the versions. The worker redeploys through Cloudflare whenever `main` changes, so a lasting warning means its Cloudflare build did not go through. When the worker is ahead instead, an **⚠ Update script** button appears on screen and links to the latest release, because the userscript manager has not installed it yet. - AdGuard for Android clears a userscript's storage whenever it installs an update. After setting sync up, press **Copy sync link** and keep the link somewhere private, such as a password manager. After a wipe, open it on JavStore, or paste it into the endpoint box and save, to set everything up again. The link carries your token, so treat it like a password. - The endpoint and token live only on the device they were entered on. They are never written into the synced document, never appear in an exported backup, and are not left in the page: the token box stays empty once a token is stored and only reports that one exists. -- A worker that is unreachable does not affect anything locally: history is still saved to the userscript manager, and the next sync picks it up. Failed or incomplete restoration is not marked complete. If local history is actually lost, recovery bypasses the session marker and downloads it again; **Restored … visited URLs** reports that recovery, not ordinary page navigation. +- A worker that is unreachable does not affect anything locally: history is still saved to the userscript manager, and the next sync picks it up. Failed or incomplete restoration is not marked complete. If local history is actually lost, recovery bypasses the session marker and downloads it again; **Restored … visited URLs** reports that recovery, not ordinary page navigation. The history records how far into the worker's history it is complete, so an older copy written back by a tab that was opened earlier (AdGuard hands each tab the storage it had when it loaded) only catches up on the difference, without the note. - Current browsers coordinate cloud work with Web Locks. Browsers without that API use a userscript-storage lease with expiry and ownership checks, so a tab that disappears cannot hold sync indefinitely. Session/progress metadata stays in userscript-manager storage; credentials and history are not placed in page `localStorage`. - **Sync now** reports both directions—what went up and what came down—so a device that is failing to push is not mistaken for one with nothing to push. - Restoring a backup keeps the backup's own timestamps, so the restored entries reach the worker and the other devices instead of being discarded by them. Entries the backup leaves out are dropped on the importing device only. Entries a history clear on this device already covers cannot be restored, and the panel says how many were skipped. diff --git a/javstore-full-layout-cleanup.user.js b/javstore-full-layout-cleanup.user.js index a28451c..d90296a 100644 --- a/javstore-full-layout-cleanup.user.js +++ b/javstore-full-layout-cleanup.user.js @@ -1,7 +1,7 @@ // ==UserScript== // @name JavStore Full Layout Cleanup - No Sidebars + Mosaic Overlay // @namespace http://tampermonkey.net/ -// @version 6.11.0 +// @version 6.11.1 // @description Clean up JavStore's layout, filter keyword-matched thumbnails, and track visited items with private, configurable controls. // @homepageURL https://github.com/jitdor/userscript-javstore // @supportURL https://github.com/jitdor/userscript-javstore/issues @@ -29,7 +29,7 @@ (function () { 'use strict'; - const SCRIPT_VERSION = '6.11.0'; + const SCRIPT_VERSION = '6.11.1'; // The same address as @downloadURL: opening it hands the newest release to the // userscript manager, which offers to install it. const INSTALL_URL = 'https://github.com/jitdor/userscript-javstore/releases/latest/download/javstore-full-layout-cleanup.user.js'; @@ -128,6 +128,14 @@ // configuration records the id the cursor belongs to, so a history that no longer // carries it—wiped, unreadable, or written before ids existed—is downloaded in full. let historyId = ''; + // How far into the worker's history this document is complete: every row up to + // `cursor` on the worker `remote` names (by a hash of its endpoint, which itself stays + // out of the history) is already merged into it. It is saved in the same + // write as the rows it describes, so the two cannot disagree. A cursor kept only in the + // sync configuration could: an engine that hands each tab the storage it had when the + // tab loaded (AdGuard does) lets a long-open tab write back an older history while the + // configuration still claims everything up to a later row is on the device. + let syncedThrough = { remote: '', cursor: 0 }; let settingsUpdatedAt = 0; // When each setting was last changed on purpose, keyed by setting name. Settings merge // field by field on these, so changing one setting on one device does not carry that @@ -650,6 +658,7 @@ resetAt, prunedBefore, ...(historyId ? { historyId } : {}), + ...(syncedThrough.remote ? { syncedThrough: { ...syncedThrough } } : {}), settings: { ...settings }, visited: Object.fromEntries(visited), overrides: Object.fromEntries(overrides), @@ -822,6 +831,7 @@ const snapshot = await readStorageSnapshot(); const combined = combineSnapshot(snapshot); adoptHistoryId(combined); + adoptSyncedThrough(combined); mergeStoredState(combined); notePushBacklog(); pruneVisited(); @@ -1252,6 +1262,41 @@ if (id) historyId = id; } + function remoteTag(endpoint) { + if (!endpoint) return ''; + let hash = 0x811c9dc5; + for (let index = 0; index < endpoint.length; index += 1) { + hash ^= endpoint.charCodeAt(index); + hash = Math.imul(hash, 0x01000193) >>> 0; + } + return hash.toString(16).padStart(8, '0'); + } + + function parseSyncedThrough(value) { + const remote = typeof value?.remote === 'string' ? value.remote.slice(0, 40) : ''; + const cursor = clamp(value?.cursor, 0, Number.MAX_SAFE_INTEGER, 0); + return remote && cursor ? { remote, cursor } : { remote: '', cursor: 0 }; + } + + // Like the history id, only ever taken from this device's own storage, and only at the + // points where a stored document is merged into this tab's: the merged history holds + // everything both copies held, so it is complete as far as the further of the two. + function adoptSyncedThrough(stored) { + const theirs = parseSyncedThrough(stored?.syncedThrough); + if (!theirs.remote) return; + if (theirs.remote === syncedThrough.remote) { + if (theirs.cursor > syncedThrough.cursor) syncedThrough = theirs; + } else if (syncedThrough.remote !== remoteTag(syncConfig.endpoint)) { + syncedThrough = theirs; + } + } + + // Exports and the legacy worker get the history without this device's place in it. + function portableState() { + const { syncedThrough: _local, ...state } = serializeState(); + return state; + } + function syncConfigured() { return syncConfig.enabled && Boolean(syncConfig.endpoint); } @@ -1590,6 +1635,7 @@ const needsCheckpoint = journalsAhead(snapshot); if (!storageAvailable) throw new Error('local history could not be read'); adoptHistoryId(stored); + adoptSyncedThrough(stored); applyStoredState(stored); pruneVisited(); @@ -1600,14 +1646,25 @@ // A history that has lost it pulls everything again; the worker settles each row // on its timestamp, so rows the device still holds cost bandwidth and nothing else. const sameHistory = Boolean(historyId) && syncConfig.historyId === historyId; - const recovering = sameRemote && syncConfig.cursor > 0 && !sameHistory; + // Where this history is complete up to, as the history itself records it. A + // document saved before it carried that falls back to the configuration's + // cursor when the two name the same history. + const stamped = syncedThrough.remote === remoteTag(syncConfig.endpoint) ? syncedThrough.cursor : 0; + const held = stamped || (sameRemote && sameHistory && !syncedThrough.remote ? syncConfig.cursor : 0); + // Only a history with no place in the sync at all, on a device that has synced + // before, has actually been lost. One that is merely behind—an older copy written + // back by a tab that had been open a long time—picks up from where it is, quietly. + const recovering = sameRemote && syncConfig.cursor > 0 && !held; const heldBefore = visited.size; - let cursor = sameRemote && sameHistory ? syncConfig.cursor : 0; + let cursor = held; const pushedAt = sameRemote ? syncConfig.pushedAt : 0; const session = await loadValue(SYNC_SESSION_KEY); const now = Date.now(); + // The session only vouches for a history that still reaches as far as the + // download it records; one written back from an older copy has to catch up. + // Sessions recorded before they carried a cursor vouch for nothing. const currentSession = session?.endpoint === syncConfig.endpoint - && session?.historyId === historyId && sameHistory && sameRemote + && sameRemote && Number.isFinite(session?.cursor) && held >= session.cursor && session?.completedAt > 0 && now - session.activeAt < SYNC_SESSION_IDLE_MS; if (!manual && currentSession) { const pending = localChangesSince(pushedAt).length @@ -1655,7 +1712,7 @@ if (answer.state && typeof answer.state === 'object' && answer.cursor === undefined) { legacy = true; downloadComplete = true; - const echoed = await requestRemote({ client: SCRIPT_VERSION, state: serializeState() }); + const echoed = await requestRemote({ client: SCRIPT_VERSION, state: portableState() }); const document = echoed?.state && typeof echoed.state === 'object' ? echoed.state : answer.state; if (applyStoredState(document)) changed = true; if (lastMergeAdopted) absorbed = true; @@ -1690,8 +1747,12 @@ // cursor is recorded against it, for the same reason as the rows above. const naming = !legacy && !historyId; if (naming) historyId = makeHistoryId(); + // The rows and the place they reach go into storage in the same write. + const reached = legacy ? { remote: '', cursor: 0 } : parseSyncedThrough({ remote: remoteTag(syncConfig.endpoint), cursor }); + const advanced = reached.remote !== syncedThrough.remote || reached.cursor !== syncedThrough.cursor; + syncedThrough = reached; applyingRemoteState = true; - const saved = (changed || absorbed || naming || needsCheckpoint) ? await persistState() : true; + const saved = (changed || absorbed || naming || needsCheckpoint || advanced) ? await persistState() : true; applyingRemoteState = false; if (!saved) throw new Error('the merged history could not be saved'); @@ -1721,7 +1782,7 @@ // page budget was exhausted, the next page must continue from the saved cursor. if (downloadComplete) { await storeValue(SYNC_SESSION_KEY, { - endpoint: syncConfig.endpoint, historyId, uploadedMeta, workerVersion: remoteWorkerVersion, + endpoint: syncConfig.endpoint, historyId, cursor, uploadedMeta, workerVersion: remoteWorkerVersion, completedAt: Date.now(), activeAt: Date.now(), }); } @@ -2493,7 +2554,7 @@ } function exportState() { - const payload = JSON.stringify(serializeState(), null, 2); + const payload = JSON.stringify(portableState(), null, 2); const blob = new Blob([payload], { type: 'application/json' }); const url = URL.createObjectURL(blob); const anchor = document.createElement('a'); @@ -2863,6 +2924,7 @@ function reloadRemoteState(newValue) { adoptHistoryId(newValue); + adoptSyncedThrough(newValue); if (applyStoredState(newValue)) showToast('Settings synchronized from another tab.'); } @@ -2873,6 +2935,7 @@ if (document.visibilityState === 'hidden') return; const stored = await readStoredState(); adoptHistoryId(stored); + adoptSyncedThrough(stored); applyStoredState(stored); } @@ -2962,6 +3025,7 @@ resetAt = parseTimestamp(initialState?.resetAt); prunedBefore = parseTimestamp(initialState?.prunedBefore); historyId = parseHistoryId(initialState?.historyId); + syncedThrough = parseSyncedThrough(initialState?.syncedThrough); lastKnownUpdatedAt = parseTimestamp(initialState?.updatedAt); settingsUpdatedAt = parseTimestamp(initialState?.settingsUpdatedAt) || lastKnownUpdatedAt; settingTimes = documentSettingTimes(initialState); diff --git a/tests/harness.mjs b/tests/harness.mjs index 7c8dbc1..618af66 100644 --- a/tests/harness.mjs +++ b/tests/harness.mjs @@ -27,13 +27,20 @@ const openWindows = new Set(); // // `dottedOnly` is the Userscripts app for Safari, which provides only the dotted GM4 API // (GM.getValue, GM.setValue, ...) and no GM_* functions at all. -export function makeStore({ latency = 0, journaling = true, primitivesOnly = false, dottedOnly = false } = {}) { +// +// `snapshotPerTab` is AdGuard's behaviour across tabs: each tab reads the values storage +// held when the tab loaded, plus its own writes. Writes still reach storage, so a tab +// opened later sees them, but a tab that is already open never does. +export function makeStore({ + latency = 0, journaling = true, primitivesOnly = false, dottedOnly = false, snapshotPerTab = false, +} = {}) { const data = new Map(); const delay = () => (latency ? new Promise(resolve => setTimeout(resolve, Math.random() * latency)) : null); return { data, journaling, dottedOnly, + snapshotPerTab, dropWrites: false, // Storage that rejects the history document while still taking the small // bookkeeping values: that is the shape of a quota refusal, and the case where a @@ -73,6 +80,30 @@ export function makeStore({ latency = 0, journaling = true, primitivesOnly = fal state() { return this.read('javstore_cleanup_state_v2') ?? null; }, + // What one tab's GM calls see; see `snapshotPerTab`. + view() { + if (!snapshotPerTab) return this; + const store = this; + const seen = new Map(data); + return { + async get(key, fallback) { + await delay(); + return seen.has(key) ? JSON.parse(seen.get(key)) : fallback; + }, + async set(key, value) { + await store.set(key, value); + if (data.has(key)) seen.set(key, data.get(key)); + }, + async list() { + await delay(); + return [...seen.keys()]; + }, + async delete(key) { + await store.delete(key); + seen.delete(key); + }, + }; + }, }; } @@ -229,22 +260,23 @@ export async function openTab(store, { removeItem: key => session.delete(key), }, }); + const gm = store.view ? store.view() : store; if (store.dottedOnly) { window.GM = { - getValue: (key, fallback) => store.get(key, fallback), - setValue: (key, value) => store.set(key, value), + getValue: (key, fallback) => gm.get(key, fallback), + setValue: (key, value) => gm.set(key, value), addStyle: async () => {}, }; if (store.journaling !== false) { - window.GM.listValues = () => store.list(); - window.GM.deleteValue = key => store.delete(key); + window.GM.listValues = () => gm.list(); + window.GM.deleteValue = key => gm.delete(key); } } else { - window.GM_getValue = (key, fallback) => store.get(key, fallback); - window.GM_setValue = (key, value) => store.set(key, value); + window.GM_getValue = (key, fallback) => gm.get(key, fallback); + window.GM_setValue = (key, value) => gm.set(key, value); if (store.journaling !== false) { - window.GM_listValues = () => store.list(); - window.GM_deleteValue = key => store.delete(key); + window.GM_listValues = () => gm.list(); + window.GM_deleteValue = key => gm.delete(key); } window.GM_addStyle = () => {}; } diff --git a/tests/sync-session.test.mjs b/tests/sync-session.test.mjs index 12c9647..0b99918 100644 --- a/tests/sync-session.test.mjs +++ b/tests/sync-session.test.mjs @@ -2,7 +2,7 @@ import test, { afterEach } from 'node:test'; import assert from 'node:assert/strict'; import { makeStore, makeRemote, enableSync, openTab, closeTabs, settle, waitFor, - listingHtml, detailHtml, card, visitedUrls, + listingHtml, detailHtml, card, pressOnCard, visitedUrls, } from './harness.mjs'; const SESSION_KEY = 'javstore_sync_session_v1'; @@ -194,3 +194,46 @@ test('foreground tabs share periodic refresh and hidden tabs do not poll', async assert.equal(remote.requests.length, 2, 'only one foreground refresh'); assert.equal(remote.responses.at(-1).changes.length, 0); }); + +// AdGuard hands each tab the storage it had when the tab loaded. A tab left open that long +// writes back an older history: one that lacks what other tabs and devices have since +// added, and that carries whatever history id it was loaded with. The cursor used to live +// only in the sync configuration, so the next page either took that older copy for a lost +// history and downloaded everything with a "Restored" note—on every page, for as long as +// the old tab kept writing—or, with the ids agreeing, trusted a cursor the copy did not +// reach and lost the difference for good. +test('an older history written back by a long-open tab catches up quietly from its own place', async () => { + const remote = seededRemote(300); + const store = makeStore({ snapshotPerTab: true }); + await restored(remote, store); + const old = await openTab(store, { html: listing(), remote, url: 'https://javstore.net/page/1' }); + const reachedByOld = store.read(CONFIG_KEY).cursor; + + // Another device adds visits, then this device loses its history while the old tab + // stays open. The next page is a genuine recovery, and says so once. + const at = Date.now() - 30000; + remote.store.sync({ cursor: 0, changes: Array.from({ length: 30 }, (_, i) => ({ + kind: 'v', key: `https://javstore.net/elsewhere-${i}.html`, at: at + i, + })) }); + store.data.delete(STATE_KEY); + store.journals().forEach(key => store.data.delete(key)); + let previous = store.read(SESSION_KEY).completedAt; + const first = await openTab(store, { html: listing(), remote, url: 'https://javstore.net/page/2' }); + await waitFor(() => store.read(SESSION_KEY).completedAt > previous); + assert.match(first.shadow().textContent, /Restored 330 visited URLs/); + assert.equal(visitedUrls(store).length, 330); + + // The old tab records a visit: its write puts back the history it loaded with. + pressOnCard(old.window, '/new.html', 'v'); + await waitFor(() => visitedUrls(store).length === 301); + + previous = store.read(SESSION_KEY).completedAt; + const requests = remote.requests.length; + const next = await openTab(store, { html: listing(), remote, url: 'https://javstore.net/page/3' }); + await waitFor(() => store.read(SESSION_KEY).completedAt > previous); + assert.equal(remote.requests[requests].body.cursor, reachedByOld, 'it resumes where the older copy reaches'); + assert.doesNotMatch(next.shadow().textContent, /Restored \d+ visited URLs/); + assert.equal(visitedUrls(store).length, 331); + assert.ok(visitedUrls(store).includes('https://javstore.net/elsewhere-0.html')); + assert.ok(visitedUrls(store).includes('https://javstore.net/new.html')); +}); diff --git a/worker/src/worker.mjs b/worker/src/worker.mjs index f3cf17e..f9ecc96 100644 --- a/worker/src/worker.mjs +++ b/worker/src/worker.mjs @@ -23,7 +23,7 @@ const STORAGE_VERSION = 3; // Reported in every answer so the userscript can show which worker it is talking to. It // moves in step with the userscript's own version; a test holds the two together. -export const WORKER_VERSION = '6.11.0'; +export const WORKER_VERSION = '6.11.1'; const OBJECT_NAME = 'default'; const MAX_BODY_BYTES = 8 * 1024 * 1024; const MAX_PAGE_ROWS = 2000; From c73b95109e6182504cd6a1dc5dc01b6b03b2fd83 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 12 Nov 2026 19:15:39 +0000 Subject: [PATCH 02/11] Apply removals when booting from journals boot() initialized state from combineSnapshot(), which carries resetAt and tombstones along without applying them. A visit still held in another page's journal was painted as visited after a V-unmark or a clear, and the page's first save wrote it back next to its own tombstone, where it then loaded from the main document for good. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VCuhK4TQqcEbLXdmxLjggy --- javstore-full-layout-cleanup.user.js | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/javstore-full-layout-cleanup.user.js b/javstore-full-layout-cleanup.user.js index d90296a..636ba70 100644 --- a/javstore-full-layout-cleanup.user.js +++ b/javstore-full-layout-cleanup.user.js @@ -3029,6 +3029,20 @@ lastKnownUpdatedAt = parseTimestamp(initialState?.updatedAt); settingsUpdatedAt = parseTimestamp(initialState?.settingsUpdatedAt) || lastKnownUpdatedAt; settingTimes = documentSettingTimes(initialState); + // combineSnapshot carries removals along without applying them, and a journal can + // still hold a visit that was unmarked or cleared after it was written. Apply them + // here by the same rules as mergeStoredState, or the page paints those visits and + // its first save writes them back next to their own tombstones. + for (const [url, at] of visited) { + if (at <= resetAt || at <= prunedBefore || at <= tombstoneTime('v', url)) visited.delete(url); + } + for (const url of [...overrides.keys()]) { + const removedAt = tombstoneTime('o', url); + if (removedAt && (overrideTimes.get(url) || 0) <= removedAt) { + overrides.delete(url); + overrideTimes.delete(url); + } + } } async function boot() { From 8a0bbf61b2aba39e80fa0c713b09549c9f11cac0 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 12 Nov 2026 19:09:34 +0000 Subject: [PATCH 03/11] Add review tests for local history under per-tab storage snapshots Six of eight fail against 6.11.1: a long-open tab's save, once journals pass their 10-minute TTL, drops visits, overrides, unmarks and clears made in tabs opened later; retireJournals deletes a journal its owner rewrote after the retiring tab loaded; and an import cannot restore a visit over a newer stored tombstone. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VCuhK4TQqcEbLXdmxLjggy --- tests/review-persistence.test.mjs | 227 ++++++++++++++++++++++++++++++ 1 file changed, 227 insertions(+) create mode 100644 tests/review-persistence.test.mjs diff --git a/tests/review-persistence.test.mjs b/tests/review-persistence.test.mjs new file mode 100644 index 0000000..f37daf7 --- /dev/null +++ b/tests/review-persistence.test.mjs @@ -0,0 +1,227 @@ +// Review: local history under AdGuard's per-tab storage snapshots. +// +// AdGuard hands every tab the GM storage it had when the tab loaded, plus that tab's own +// writes; a tab never sees what another tab writes after it loaded. These tests open a +// long-lived "old" tab, make changes in tabs opened later, let ten minutes pass (the +// journal TTL), and then let the old tab save once. A fresh tab must still see every +// change the newer tabs made. +import test, { afterEach } from 'node:test'; +import assert from 'node:assert/strict'; +import { + makeStore, openTab, closeTabs, settle, listingHtml, detailHtml, card, clickCard, + pressOnCard, visitedUrls, +} from './harness.mjs'; + +const CARDS = [['/a.html', 'Alpha'], ['/b.html', 'Beta'], ['/c.html', 'Gamma'], ['/d.html', 'Delta']]; +const listing = () => listingHtml(CARDS); +const u = href => `https://javstore.net${href}`; +const TTL_PLUS = 11 * 60000; + +afterEach(closeTabs); + +// One clock for every tab of the device, which a test can move forward. +function device(options = {}) { + const store = makeStore({ snapshotPerTab: true, ...options }); + const clock = { offset: 0 }; + const open = (extra = {}) => openTab(store, { + html: listing(), + ...extra, + beforeRun: window => { + const realNow = window.Date.now.bind(window.Date); + window.Date.now = () => realNow() + clock.offset; + }, + }); + return { store, clock, open }; +} + +const isVisited = (tab, href) => card(tab.window, href).classList.contains('jvs-visited'); + +test('review: a visit recorded after a long-open tab loaded survives that tab saving once journals age out', async () => { + const { store, clock, open } = device(); + const old = await open(); + await open({ html: detailHtml, url: u('/b.html') }); + await settle(); + assert.deepEqual(visitedUrls(store), [u('/b.html')]); + + clock.offset = TTL_PLUS; + const newer = await open(); + clickCard(newer.window, '/c.html'); + await settle(); + assert.deepEqual(visitedUrls(store), [u('/b.html'), u('/c.html')]); + + clickCard(old.window, '/a.html'); + await settle(); + + const fresh = await open(); + await settle(); + assert.ok(isVisited(fresh, '/b.html'), 'b was recorded by a tab opened after the old one'); + assert.ok(isVisited(fresh, '/c.html')); + assert.ok(isVisited(fresh, '/a.html')); +}); + +test('review: a long-open tab does not delete a journal its owner rewrote after the tab loaded', async () => { + const { store, clock, open } = device(); + const owner = await open(); + clickCard(owner.window, '/a.html'); + await settle(); + const [ownerJournal] = store.journals(); + assert.ok(ownerJournal); + + // Loaded after the owner's first save, so its snapshot holds that copy of the journal. + const stale = await open(); + clock.offset = TTL_PLUS; + // Only the owner records b (a manual mark; a click whose item tab never loaded). + pressOnCard(owner.window, '/b.html', 'v'); + await settle(); + assert.deepEqual(visitedUrls(store), [u('/a.html'), u('/b.html')]); + + clickCard(stale.window, '/c.html'); + await settle(); + assert.ok(store.journals().includes(ownerJournal), + 'the owner journal was rewritten after the stale tab loaded, so it is not stale'); + + const fresh = await open(); + await settle(); + assert.ok(isVisited(fresh, '/b.html')); +}); + +test('review: "Clear visited history" in a newer tab is not undone by a long-open tab', async () => { + const { store, clock, open } = device(); + const seeded = await open(); + clickCard(seeded.window, '/a.html'); + clickCard(seeded.window, '/b.html'); + await settle(); + + const old = await open(); + const clearing = await open(); + clearing.shadow().querySelector('.clear').click(); + await settle(); + assert.deepEqual(visitedUrls(store), []); + + clock.offset = TTL_PLUS; + const later = await open(); + clickCard(later.window, '/d.html'); + await settle(); + clickCard(old.window, '/c.html'); + await settle(); + + const fresh = await open(); + await settle(); + assert.ok(!isVisited(fresh, '/a.html'), 'a was cleared'); + assert.ok(!isVisited(fresh, '/b.html'), 'b was cleared'); +}); + +test('review: an item unmarked in a newer tab is not resurrected by a long-open tab', async () => { + const { clock, open } = device(); + const seeded = await open(); + clickCard(seeded.window, '/a.html'); + await settle(); + + const old = await open(); + const unmarking = await open(); + pressOnCard(unmarking.window, '/a.html', 'v'); + await settle(); + + clock.offset = TTL_PLUS; + const later = await open(); + clickCard(later.window, '/d.html'); + await settle(); + clickCard(old.window, '/c.html'); + await settle(); + + const fresh = await open(); + await settle(); + assert.ok(!isVisited(fresh, '/a.html')); +}); + +test('review: a setting changed in a newer tab is not reverted by a long-open tab', async () => { + const { clock, open } = device(); + const old = await open(); + const changing = await open(); + const form = changing.shadow().querySelector('form.settings-form'); + form.elements.mode.value = 'blur'; + form.dispatchEvent(new changing.window.Event('submit', { bubbles: true, cancelable: true })); + await settle(); + + clock.offset = TTL_PLUS; + const later = await open(); + clickCard(later.window, '/d.html'); + await settle(); + clickCard(old.window, '/c.html'); + await settle(); + + const fresh = await open(); + await settle(); + assert.equal(fresh.shadow().querySelector('form.settings-form').elements.mode.value, 'blur'); +}); + +test('review: a per-card override set in a newer tab is not dropped by a long-open tab', async () => { + const { store, clock, open } = device(); + const old = await open(); + const overriding = await open(); + pressOnCard(overriding.window, '/a.html', 'o'); + await settle(); + assert.equal(store.state().overrides[u('/a.html')], 'allow'); + + clock.offset = TTL_PLUS; + const later = await open(); + clickCard(later.window, '/d.html'); + await settle(); + clickCard(old.window, '/c.html'); + await settle(); + + const fresh = await open(); + await settle(); + assert.ok(card(fresh.window, '/a.html'), 'fresh tab loaded'); + assert.equal(store.state().overrides[u('/a.html')], 'allow'); +}); + +// Not snapshot-specific: the import deletes this device's tombstone for a restored URL in +// memory, then its own save merges the stored tombstone straight back over the restored +// (older) visit. +test('review: importing a backup restores a visit that was unmarked after the backup was taken', async () => { + const store = makeStore(); + const backupAt = Date.now() - 86400000; + const tab = await openTab(store, { html: listing() }); + clickCard(tab.window, '/a.html'); + await settle(); + pressOnCard(tab.window, '/a.html', 'v'); + await settle(); + assert.deepEqual(visitedUrls(store), []); + + const input = tab.shadow().querySelector('.import-file'); + Object.defineProperty(input, 'files', { + configurable: true, + value: [{ + text: async () => JSON.stringify({ + version: 3, updatedAt: backupAt, visited: { [u('/a.html')]: backupAt }, + overrides: {}, overrideTimes: {}, tombstones: {}, + }), + }], + }); + input.dispatchEvent(new tab.window.Event('change')); + await settle(); + assert.ok(isVisited(tab, '/a.html'), 'the importing tab shows it restored'); + assert.deepEqual(visitedUrls(store), [u('/a.html')], 'and storage keeps it'); +}); + +// The 6.5.0 scenario (Ctrl-click bursts on slow storage) under the per-tab snapshot model. +test('review: a burst of Ctrl-clicks on slow snapshot-per-tab storage loses nothing', async () => { + const TILES = Array.from({ length: 8 }, (_, index) => [`/t-${index}.html`, `Tile ${index}`]); + const store = makeStore({ latency: 150, snapshotPerTab: true }); + const opening = []; + const listingTab = await openTab(store, { html: listingHtml(TILES) }); + for (const [href] of TILES) { + clickCard(listingTab.window, href); + opening.push(openTab(store, { html: detailHtml, url: u(href) })); + await new Promise(resolve => setTimeout(resolve, 30)); + } + await Promise.all(opening); + await new Promise(resolve => setTimeout(resolve, 3000)); + const reloaded = await openTab(store, { html: listingHtml(TILES) }); + await new Promise(resolve => setTimeout(resolve, 2500)); + const unmarked = TILES.map(([href]) => href) + .filter(href => !card(reloaded.window, href).classList.contains('jvs-visited')); + assert.deepEqual(unmarked, []); + await new Promise(resolve => setTimeout(resolve, 2000)); +}); From 9e01d95acfdb732b49dbf4ef25410ce77bbd07c4 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 12 Nov 2026 19:14:36 +0000 Subject: [PATCH 04/11] Add sync review tests for per-tab storage snapshots Eight failing tests documenting sync defects under AdGuard-style per-tab storage snapshots: push horizon stranding, stale sync config write-back, full re-upload at the visit cap, endpoint-change re-download, session freshness, lease exclusion, burst cost and worker reset. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VCuhK4TQqcEbLXdmxLjggy --- tests/review-sync.test.mjs | 230 +++++++++++++++++++++++++++++++++++++ 1 file changed, 230 insertions(+) create mode 100644 tests/review-sync.test.mjs diff --git a/tests/review-sync.test.mjs b/tests/review-sync.test.mjs new file mode 100644 index 0000000..c63e44d --- /dev/null +++ b/tests/review-sync.test.mjs @@ -0,0 +1,230 @@ +// Review of the cloud sync protocol under AdGuard's per-tab storage snapshots. +// +// Every test here models AdGuard: each tab reads GM storage as it was when the tab loaded, +// plus its own writes (makeStore({ snapshotPerTab: true })). Tests that fail on the +// current script document a defect; the report that accompanies this file ranks them. +import test, { afterEach } from 'node:test'; +import assert from 'node:assert/strict'; +import { + makeStore, makeRemote, enableSync, openTab, closeTabs, settle, waitFor, + listingHtml, detailHtml, visitedUrls, pressOnCard, +} from './harness.mjs'; + +const SESSION_KEY = 'javstore_sync_session_v1'; +const CONFIG_KEY = 'javstore_sync_config_v1'; +const STATE_KEY = 'javstore_cleanup_state_v2'; +const listing = () => listingHtml([['/old.html', 'Old'], ['/new.html', 'New']]); +afterEach(closeTabs); + +function seededRemote(count = 1, at = Date.now() - 60000) { + const remote = makeRemote(); + remote.store.sync({ cursor: 0, changes: Array.from({ length: count }, (_, i) => ({ + kind: 'v', key: `https://javstore.net/${i ? `old-${i}` : 'old'}.html`, at: at + i, + })) }); + return remote; +} + +function closeTab(tab) { + tab.window.dispatchEvent(new tab.window.Event('pagehide')); + tab.window.close(); +} + +async function syncNow(tab, remote) { + const before = remote.requests.length; + tab.shadow().querySelector('.sync-now').click(); + await waitFor(() => remote.requests.length > before); + await settle(100); +} + +async function firstTab(remote, store, options = {}) { + enableSync(store, remote, options.config); + const tab = await openTab(store, { html: listing(), remote, ...options }); + await waitFor(() => store.read(SESSION_KEY)?.completedAt); + await settle(); + return tab; +} + +// F1. The push horizon is a device-wide claim ("everything stamped before pushedAt is on +// the worker") written by whichever tab syncs, from that tab's view of storage. A tab that +// loaded before a visit was recorded cannot see it, yet still moves the horizon past it. +test('F1 a long-open tab syncing does not strand a visit another tab recorded but had not pushed', async () => { + const remote = seededRemote(); + const store = makeStore({ snapshotPerTab: true }); + await firstTab(remote, store); + const old = await openTab(store, { html: listing(), remote, url: 'https://javstore.net/page/2' }); + + // An item tab records its visit and is closed before its 4 s upload debounce fires. + const item = await openTab(store, { html: detailHtml, remote, url: 'https://javstore.net/x.html' }); + await waitFor(() => visitedUrls(store).includes('https://javstore.net/x.html')); + closeTab(item); + + // The long-open listing tab syncs (focus/poll/Sync now) and records its horizon. + await syncNow(old, remote); + assert.ok(!remote.visited().includes('https://javstore.net/x.html')); + + // Every later page on this device, even a manual sync, should still send it. + const next = await openTab(store, { html: listing(), remote, url: 'https://javstore.net/page/3' }); + await new Promise(resolve => setTimeout(resolve, 4500)); + await syncNow(next, remote); + assert.ok(visitedUrls(store).includes('https://javstore.net/x.html'), 'still held locally'); + assert.ok(remote.visited().includes('https://javstore.net/x.html'), 'visit stranded on this device'); +}); + +// F2. runSync writes `{ ...syncConfig, cursor, pushedAt, ... }` where syncConfig was read +// from the syncing tab's snapshot. Endpoint, token and the enabled flag go back with it. +test('F2 a long-open tab syncing does not revert sync settings changed in a newer tab', async () => { + const remote = seededRemote(); + const store = makeStore({ snapshotPerTab: true }); + await firstTab(remote, store); + const old = await openTab(store, { html: listing(), remote, url: 'https://javstore.net/page/2' }); + + // The user turns sync off in a new tab (same for a rotated token or new endpoint). + const settingsTab = await openTab(store, { html: listing(), remote, url: 'https://javstore.net/page/3' }); + const shadow = settingsTab.shadow(); + const enabled = shadow.querySelector('.sync-form input[type="checkbox"]'); + enabled.checked = false; + shadow.querySelector('.sync-form').dispatchEvent(new settingsTab.window.Event('submit', { cancelable: true })); + await waitFor(() => store.read(CONFIG_KEY).enabled === false); + + // The old tab regains focus after a minute and syncs. + await syncNow(old, remote); + assert.equal(store.read(CONFIG_KEY).enabled, false, 'sync was switched back on by a stale tab'); +}); + +// F3. At the 5,000-visit cap, a stale tab still holds (and writes back) the oldest visits +// that newer tabs have pruned. The next merge in a fresh tab adopts them as "backdated", +// lowers the push horizon to the oldest of them, and the next sync re-uploads the entire +// history, every time a stale tab writes. The same happens without snapshots: another +// page's journal (kept for 10 minutes) carries the pruned visits back just the same. +test('F3 at the visit cap, a stale tab writing back pruned visits does not trigger a full re-upload', async () => { + const base = Date.now() - 86400000; + const remote = makeRemote(); + const store = makeStore({ snapshotPerTab: true }); + // 5,000 visits already on the device and the worker, all pushed. + const visited = {}; + for (let i = 0; i < 5000; i++) visited[`https://javstore.net/v-${i}.html`] = base + i; + remote.store.sync({ cursor: 0, changes: Object.entries(visited).map(([key, at]) => ({ kind: 'v', key, at })) }); + store.write(STATE_KEY, { version: 3, updatedAt: base + 5000, visited, settings: {} }); + await firstTab(remote, store, { config: {} }); + const stale = await openTab(store, { html: listing(), remote, url: 'https://javstore.net/page/2' }); + + // New visits in fresh tabs push the oldest out of the capped history. + for (let i = 0; i < 3; i++) { + const tab = await openTab(store, { html: detailHtml, remote, url: `https://javstore.net/new-${i}.html` }); + await waitFor(() => remote.visited().includes(`https://javstore.net/new-${i}.html`), { timeout: 15000 }); + closeTab(tab); + } + // The stale tab records a visit of its own: its write carries the pruned visits back. + pressOnCard(stale.window, '/new.html', 'v'); + await waitFor(() => visitedUrls(store).includes('https://javstore.net/new.html')); + await new Promise(resolve => setTimeout(resolve, 4500)); + + const fresh = await openTab(store, { html: listing(), remote, url: 'https://javstore.net/page/3' }); + pressOnCard(fresh.window, '/old.html', 'v'); + const before = remote.requests.length; + await new Promise(resolve => setTimeout(resolve, 5000)); + await settle(200); + const pushed = remote.requests.slice(before).flatMap(request => request.body.changes || []); + assert.ok(pushed.length < 50, `re-uploaded ${pushed.length} rows for one new visit`); +}); + +// F4. After the endpoint changes, a tab opened before the change keeps stamping the history +// with the old worker's position (its own syncConfig still names the old endpoint, so +// adoptSyncedThrough keeps it). Every new page then finds no position for the new worker +// while the config has a cursor, takes that for lost history, and downloads everything. +test('F4 after an endpoint change, a stale tab does not make every page re-download the whole history', async () => { + const first = seededRemote(50); + const second = makeRemote({ origin: 'https://sync2.test' }); + second.store.sync({ cursor: 0, changes: Array.from({ length: 50 }, (_, i) => ({ + kind: 'v', key: `https://javstore.net/second-${i}.html`, at: Date.now() - 50000 + i, + })) }); + const store = makeStore({ snapshotPerTab: true }); + await firstTab(first, store); + const stale = await openTab(store, { html: listing(), remote: [first, second], url: 'https://javstore.net/page/2' }); + + // Repoint this device at the second worker in a newer tab. + const config = store.read(CONFIG_KEY); + store.write(CONFIG_KEY, { ...config, endpoint: second.endpoint }); + const repointed = await openTab(store, { html: listing(), remote: [first, second], url: 'https://javstore.net/page/3' }); + await waitFor(() => second.requests.length >= 1); + await settle(200); + + // The stale tab saves a visit: its document carries the first worker's position. + pressOnCard(stale.window, '/new.html', 'v'); + await waitFor(() => visitedUrls(store).includes('https://javstore.net/new.html')); + await settle(200); + closeTab(repointed); + + const fullPulls = () => second.requests.filter(request => request.body.cursor === 0).length; + const pullsBefore = fullPulls(); + const next = await openTab(store, { html: listing(), remote: [first, second], url: 'https://javstore.net/page/4' }); + await settle(300); + assert.doesNotMatch(next.shadow().textContent, /Restored \d+ visited URLs/); + assert.equal(fullPulls(), pullsBefore, 'a new page downloaded the whole history again'); +}); + +// F5. A startup inside a live session never downloads. activeAt is refreshed by every page +// load, so a device browsing listing pages continuously never sees visits made on another +// device until it records a visit of its own, regains focus, or a foreground tab survives +// a whole poll interval without navigating. +test('F5 continuous navigation still picks up another device\'s visits within the poll interval', async () => { + const remote = seededRemote(); + const store = makeStore({ snapshotPerTab: true }); + await firstTab(remote, store); + // The iPhone records a visit. + remote.store.sync({ cursor: 0, changes: [{ kind: 'v', key: 'https://javstore.net/new.html', at: Date.now() }] }); + // Ten minutes of browsing listing pages later (session last completed 10 min ago, + // last active a moment ago), a new listing page opens. + const session = store.read(SESSION_KEY); + store.write(SESSION_KEY, { ...session, completedAt: Date.now() - 10 * 60000, activeAt: Date.now() - 1000 }); + const before = remote.requests.length; + await openTab(store, { html: listing(), remote, url: 'https://javstore.net/page/9' }); + await settle(200); + assert.ok(remote.requests.length > before, 'no download for 10 minutes of navigation'); +}); + +// F6. Under per-tab snapshots the GM-storage lease cannot exclude anything: each tab only +// ever reads back its own claim. Two tabs syncing at once both download and both push. +test('F6 the storage lease serializes two tabs under per-tab snapshots', async () => { + const remote = seededRemote(300); + const store = makeStore({ snapshotPerTab: true, latency: 5 }); + enableSync(store, remote); + await Promise.all([0, 1].map(() => openTab(store, { html: listing(), remote, webLocks: false }))); + await waitFor(() => store.read(SESSION_KEY)?.completedAt); + await settle(300); + assert.equal(remote.requests.filter(request => request.body.cursor === 0).length, 1); +}); + +// F7. Cost of a 50-tab context-menu burst on AdGuard. The batching in 6.11.0 relies on each +// tab seeing the horizon the first upload recorded; under per-tab snapshots no item tab +// sees another's upload, so every tab uploads (and downloads) on its own. +test('F7 a 50-tab burst under per-tab snapshots batches its uploads', async () => { + const remote = seededRemote(300); + const store = makeStore({ snapshotPerTab: true }); + await firstTab(remote, store); + const before = remote.requests.length; + await Promise.all(Array.from({ length: 50 }, (_, i) => openTab(store, { + remote, html: detailHtml, url: `https://javstore.net/burst-${i}.html`, + }))); + await waitFor(() => remote.visited().length === 350, { timeout: 20000 }); + await new Promise(resolve => setTimeout(resolve, 4500)); + const requests = remote.requests.length - before; + const rows = remote.requests.slice(before).reduce((sum, request) => sum + (request.body.changes || []).length, 0); + assert.ok(requests <= 5, `50 visits cost ${requests} requests carrying ${rows} rows`); +}); + +// F8. A worker that comes back empty at the same URL (Durable Object recreated, storage +// reset) is never re-seeded: the device's cursor is past everything the new worker holds, +// and its push horizon says every local visit is already upstream. +test('F8 a worker reset at the same endpoint is re-seeded from the device', async () => { + const remote = seededRemote(20); + const store = makeStore(); + const tab = await firstTab(remote, store); + const fresh = makeRemote(); + remote.store = fresh.store; + remote.handle = fresh.handle; + remote.requests = fresh.requests; + remote.visited = fresh.visited; + await syncNow(tab, fresh); + assert.equal(fresh.visited().length, 20, 'the reset worker stays empty'); +}); From b12edc62281a11fc10a9239cd16b4e1573261444 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 12 Nov 2026 19:15:39 +0000 Subject: [PATCH 05/11] Add lifecycle and multi-tab review tests and measurements Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VCuhK4TQqcEbLXdmxLjggy --- tests/review-lifecycle.test.mjs | 457 ++++++++++++++++++++++++++++++++ 1 file changed, 457 insertions(+) create mode 100644 tests/review-lifecycle.test.mjs diff --git a/tests/review-lifecycle.test.mjs b/tests/review-lifecycle.test.mjs new file mode 100644 index 0000000..c9f1178 --- /dev/null +++ b/tests/review-lifecycle.test.mjs @@ -0,0 +1,457 @@ +// Design review: page lifecycle, many long-open tabs, and cost per page on a storage +// engine that behaves like AdGuard's (a snapshot per tab, asynchronous GM calls). +// +// Tests marked `todo` demonstrate a problem: they assert the behaviour the script should +// have and currently fail, without failing the suite. The `measure:` tests report numbers +// through diagnostics (visible with --test-reporter=spec) and only assert loose bounds. +import test, { afterEach } from 'node:test'; +import assert from 'node:assert/strict'; +import { + makeStore, makeRemote, enableSync, openTab, closeTabs, settle, waitFor, + listingHtml, detailHtml, card, clickCard, pressOnCard, visitedUrls, +} from './harness.mjs'; + +const STATE_KEY = 'javstore_cleanup_state_v2'; +const SESSION_KEY = 'javstore_sync_session_v1'; +const CONFIG_KEY = 'javstore_sync_config_v1'; +const CARDS = [['/a.html', 'Alpha'], ['/b.html', 'Beta'], ['/c.html', 'Gamma']]; +const listing = () => listingHtml(CARDS); +const abs = href => `https://javstore.net${href}`; +const TTL = 10 * 60000; + +afterEach(closeTabs); + +function closeTab(tab) { + tab.window.dispatchEvent(new tab.window.Event('pagehide')); + tab.window.close(); +} + +// Pushes a journal's last write into the past, the way it looks once JOURNAL_TTL_MS has +// gone by without its page writing again. +function age(store, key, ms = TTL + 60000) { + const journal = store.read(key); + journal.writtenAt -= ms; + store.write(key, journal); +} + +// A history the size the cap allows: 5,000 visits and 2,000 tombstones, with item URLs of +// the length the site uses. +function bigHistory(now = Date.now()) { + const visited = {}; + const tombstones = {}; + for (let i = 0; i < 5000; i++) { + visited[`https://javstore.net/${600000 + i}-FC2-PPV-${3000000 + i}-Some-Typical-Title-Words-pn.html`] = now - 86400000 + i; + } + for (let i = 0; i < 2000; i++) { + tombstones[`v|https://javstore.net/${500000 + i}-FC2-PPV-${2000000 + i}-Some-Typical-Title-Words-pn.html`] = now - 3600000 + i; + } + return { + version: 3, updatedAt: now - 1000, settingsUpdatedAt: 0, settingTimes: {}, + resetAt: 0, prunedBefore: 0, settings: {}, visited, overrides: {}, overrideTimes: {}, tombstones, + }; +} + +// Counts GM calls and the bytes they move. Only valid for a store without snapshotPerTab +// (a snapshot view reads from its own copy). +function meter(store) { + const counts = { get: 0, set: 0, list: 0, delete: 0, readBytes: 0, writeBytes: 0, mainReads: 0, mainWrites: 0, sessionWrites: 0 }; + const original = { get: store.get, set: store.set, list: store.list, delete: store.delete }; + store.get = async function (key, fallback) { + counts.get++; + if (key === STATE_KEY) counts.mainReads++; + const value = await original.get.call(this, key, fallback); + if (typeof value === 'string') counts.readBytes += value.length; + return value; + }; + store.set = async function (key, value) { + counts.set++; + if (key === STATE_KEY) counts.mainWrites++; + if (key === SESSION_KEY) counts.sessionWrites++; + counts.writeBytes += typeof value === 'string' ? value.length : JSON.stringify(value).length; + return original.set.call(this, key, value); + }; + store.list = async function () { counts.list++; return original.list.call(this); }; + store.delete = async function (key) { counts.delete++; return original.delete.call(this, key); }; + const snapshot = () => ({ ...counts }); + const since = before => Object.fromEntries(Object.keys(counts).map(key => [key, counts[key] - before[key]])); + return { counts, snapshot, since }; +} + +const kb = bytes => `${(bytes / 1024).toFixed(0)} KB`; +const describe = c => `GM get ${c.get} (main ${c.mainReads}), set ${c.set} (main ${c.mainWrites}), list ${c.list}, delete ${c.delete}; read ${kb(c.readBytes)}, wrote ${kb(c.writeBytes)}`; + +// --------------------------------------------------------------------------------------- +// 1. What each page load, click and poll costs with a full history. +// --------------------------------------------------------------------------------------- + +test('measure: GM traffic per page load, click and poll with a 5,000-visit history (sync off)', async t => { + const store = makeStore(); + store.write(STATE_KEY, bigHistory()); + const docBytes = store.data.get(STATE_KEY).length; + t.diagnostic(`main document: ${kb(docBytes)}`); + const m = meter(store); + const polls = []; + const hookPoll = window => { + const setInterval = window.setInterval.bind(window); + window.setInterval = (callback, ms) => { + if (ms === 45000) polls.push(callback); + return setInterval(callback, ms); + }; + }; + + let before = m.snapshot(); + const listingTab = await openTab(store, { html: listing(), url: 'https://javstore.net/416-av-uncensored-page-19-cn.html', beforeRun: hookPoll }); + await settle(); + const listingLoad = m.since(before); + t.diagnostic(`listing page load: ${describe(listingLoad)}`); + + before = m.snapshot(); + await openTab(store, { html: detailHtml, url: 'https://javstore.net/700001-Item-pn.html' }); + await waitFor(() => store.state().visited['https://javstore.net/700001-Item-pn.html']); + await settle(); + const itemLoad = m.since(before); + t.diagnostic(`item page load (records the visit): ${describe(itemLoad)}`); + + before = m.snapshot(); + clickCard(listingTab.window, '/a.html'); + await waitFor(() => store.state().visited[abs('/a.html')]); + await settle(); + const click = m.since(before); + t.diagnostic(`one click on a tile: ${describe(click)}`); + + before = m.snapshot(); + const started = performance.now(); + await polls[0](); + const pollMs = performance.now() - started; + const poll = m.since(before); + t.diagnostic(`one 45 s poll in a visible tab: ${describe(poll)}, ${pollMs.toFixed(1)} ms in jsdom`); + + before = m.snapshot(); + listingTab.window.document.dispatchEvent(new listingTab.window.Event('visibilitychange')); + listingTab.window.dispatchEvent(new listingTab.window.Event('focus')); + await settle(); + const activation = m.since(before); + t.diagnostic(`tab activation (visibilitychange + focus): ${describe(activation)}`); + + // Every save reads the document, writes it, and reads it back. + assert.ok(itemLoad.mainWrites >= 1 && itemLoad.writeBytes >= docBytes * 0.9); + assert.ok(click.mainReads >= 2, 'a save reads the whole document and reads it back'); + assert.equal(activation.mainReads, 2, 'activation re-reads the whole document twice (once per event)'); +}); + +test('measure: GM traffic for an item page load and its push with sync on (session current)', async t => { + const store = makeStore(); + store.write(STATE_KEY, bigHistory()); + const remote = makeRemote(); + enableSync(store, remote); + // Establish a session the way a first page does. + await openTab(store, { html: listing(), remote }); + await waitFor(() => store.read(SESSION_KEY)?.completedAt); + await settle(); + const m = meter(store); + const requests = remote.requests.length; + + const before = m.snapshot(); + await openTab(store, { html: detailHtml, url: 'https://javstore.net/700002-Item-pn.html', remote }); + await settle(); + const load = m.since(before); + t.diagnostic(`item page load, sync on: ${describe(load)}; worker requests ${remote.requests.length - requests}`); + + const beforePush = m.snapshot(); + await waitFor(() => remote.requests.length > requests, { timeout: 9000 }); + await settle(); + const push = m.since(beforePush); + t.diagnostic(`debounced push 4 s later: ${describe(push)}; worker requests ${remote.requests.length - requests}`); + assert.ok(load.sessionWrites >= 1, 'every page load writes the session key'); + assert.ok(load.mainReads + push.mainReads >= 4, 'the document is read in full several times per page'); +}); + +// --------------------------------------------------------------------------------------- +// 2. boot(): journals folded in without their removals. +// +// boot() calls initializeState(combineSnapshot(...)). combineSnapshot takes the newest +// visit from every journal but only carries resetAt and tombstones along ("mergeStoredState +// does that for whoever reads the result"), and initializeState applies none of them. An +// item page journals its own visit, so for the journal's lifetime every new page boots with +// visits that were unmarked or cleared since, paints them visited, and its first save +// writes them back into the main document next to their tombstone, after which they load +// from the main document for good. Happens on any engine, not only snapshot ones. +// Both tests below failed on cc0f03b; initializeState now applies the removals. +// --------------------------------------------------------------------------------------- + +test('a visit unmarked with V stays unmarked in tabs opened afterwards', async () => { + const store = makeStore(); + await openTab(store, { html: detailHtml, url: abs('/a.html') }); // journals its own visit + await waitFor(() => visitedUrls(store).length === 1); + const listingTab = await openTab(store, { html: listing() }); + pressOnCard(listingTab.window, '/a.html', 'v'); + await waitFor(() => visitedUrls(store).length === 0); + await settle(); + + const next = await openTab(store, { html: listing() }); + assert.ok(!card(next.window, '/a.html').classList.contains('jvs-visited'), 'painted unvisited'); + clickCard(next.window, '/b.html'); + await settle(); + assert.deepEqual(visitedUrls(store), [abs('/b.html')], 'the next save does not write it back'); +}); + +test('"Clear visited history" is not undone by journals of other pages', async () => { + const store = makeStore(); + await openTab(store, { html: detailHtml, url: abs('/a.html') }); + await waitFor(() => visitedUrls(store).length === 1); + const listingTab = await openTab(store, { html: listing() }); + listingTab.shadow().querySelector('.clear').click(); + await waitFor(() => visitedUrls(store).length === 0); + await settle(); + + const next = await openTab(store, { html: listing() }); + assert.ok(!card(next.window, '/a.html').classList.contains('jvs-visited'), 'painted unvisited'); + clickCard(next.window, '/b.html'); + await settle(); + assert.deepEqual(visitedUrls(store), [abs('/b.html')]); +}); + +// --------------------------------------------------------------------------------------- +// 3. Long-open tabs on a snapshot engine. +// --------------------------------------------------------------------------------------- + +// Tab A has been open for hours. It loaded when tab B's journal was already more than a +// TTL old. B is still open and keeps recording: each save rewrites B's journal. A's first +// save then "retires" B's journal: it checks writtenAt against its own snapshot, which +// still holds B's old journal, so it deletes the live key. A's main-document write has +// already dropped what B recorded after A loaded, so nothing in storage holds it. +test('a long-open tab does not delete journals other live tabs have rewritten since it loaded', { todo: 'retireJournals trusts a stale snapshot (data loss)' }, async () => { + const store = makeStore({ snapshotPerTab: true }); + const b = await openTab(store, { html: listing() }); + clickCard(b.window, '/a.html'); + await waitFor(() => store.journals().length === 1); + const [bJournal] = store.journals(); + age(store, bJournal); + + const a = await openTab(store, { html: listing() }); // the long-open tab + clickCard(b.window, '/b.html'); // B records more; its journal is rewritten now + await waitFor(() => store.read(bJournal)?.visited?.[abs('/b.html')]); + await settle(); + + clickCard(a.window, '/c.html'); // A's first save, hours later + await settle(80); + + const fresh = await openTab(store, { html: listing() }); + assert.ok(card(fresh.window, '/b.html').classList.contains('jvs-visited'), + 'B\'s visit survives A\'s save'); + assert.ok(store.journals().includes(bJournal), 'B\'s live journal is still there'); +}); + +// The common case with 40 tabs: tab C is opened after A, records a visit and is closed. +// A fresh tab later merges C's journal into the main document and, a TTL on, retires it. +// Then A, which loaded before C existed, saves: its write is built on the snapshot it took +// at load and drops C's visit. No journal is left to bring it back. +test('a long-open tab\'s save does not drop visits other tabs recorded after it loaded', { todo: 'stale tabs write the whole document (data loss when sync is off or the push failed)' }, async () => { + const store = makeStore({ snapshotPerTab: true }); + const a = await openTab(store, { html: listing() }); // open for hours + const c = await openTab(store, { html: detailHtml, url: 'https://javstore.net/700003-Item-pn.html' }); + await waitFor(() => visitedUrls(store).includes('https://javstore.net/700003-Item-pn.html')); + await settle(); + closeTab(c); + store.journals().forEach(key => age(store, key)); + + const d = await openTab(store, { html: listing() }); + clickCard(d.window, '/b.html'); // a fresh save that retires C's aged journal + await waitFor(() => store.journals().length === 1); + await settle(); + assert.ok(visitedUrls(store).includes('https://javstore.net/700003-Item-pn.html')); + + clickCard(a.window, '/a.html'); + await settle(80); + const e = await openTab(store, { html: listing() }); + await settle(); + assert.ok(visitedUrls(store).includes('https://javstore.net/700003-Item-pn.html'), + 'the visit C recorded is still stored after A saved'); + assert.ok(card(e.window, '/a.html')); +}); + +// A V-unmark is a per-URL tombstone, and a journal carries it only while it is newer than +// the journal's horizon (resetAt, by contrast, rides in every journal, so a clear +// survives in whichever fresh journal is newest). Once the unmarking tab's journal has +// been retired, a tab that loaded before the unmark writes the visit back without the +// tombstone, and every later page shows it as visited again. +test('a long-open tab cannot undo a V-unmark once the unmarking journal is retired', { todo: 'stale tabs write the whole document (unmarked visits come back)' }, async () => { + const store = makeStore({ snapshotPerTab: true }); + const seed = await openTab(store, { html: listing() }); + clickCard(seed.window, '/a.html'); + await waitFor(() => visitedUrls(store).length === 1); + closeTab(seed); + + const a = await openTab(store, { html: listing() }); // open for hours + const unmarker = await openTab(store, { html: listing() }); + pressOnCard(unmarker.window, '/a.html', 'v'); + await waitFor(() => visitedUrls(store).length === 0); + await settle(); + closeTab(unmarker); + store.journals().forEach(key => age(store, key)); + const d = await openTab(store, { html: listing() }); + clickCard(d.window, '/c.html'); // a fresh save that retires the aged journals + await waitFor(() => visitedUrls(store).includes(abs('/c.html')) && store.journals().length === 1); + await settle(); + + clickCard(a.window, '/b.html'); // any save by the long-open tab + await settle(80); + const e = await openTab(store, { html: listing() }); + assert.ok(!card(e.window, '/a.html').classList.contains('jvs-visited'), 'the unmarked visit stays unmarked'); +}); + +// The sync lock is held for the whole exchange. GM_xmlhttpRequest has a 20 s timeout, but +// the fetch fallback (used whenever the manager refuses the GM request, as AdGuard can) +// has none: one tab whose request never settles holds the Web Lock until it is closed, and +// every other tab's sync queues behind it. Timers in the hung tab are sped up 100x, so a +// page-side timeout of REMOTE_TIMEOUT_MS would release the lock within a fraction of a +// second here. +test('a request that never settles in one tab does not block every other tab\'s sync', { todo: 'fetchRemote has no timeout; the lock is held indefinitely (sync stalls on all tabs)' }, async () => { + const store = makeStore(); + const remote = makeRemote(); + enableSync(store, remote); + await openTab(store, { html: listing(), remote, beforeRun(window) { + window.GM_xmlhttpRequest = ({ onerror }) => onerror({ error: 'Forbidden' }); + window.fetch = () => new Promise(() => {}); + const setTimeout = window.setTimeout.bind(window); + window.setTimeout = (callback, ms, ...rest) => setTimeout(callback, ms >= 1000 ? ms / 100 : ms, ...rest); + } }); + await settle(); + await openTab(store, { html: listing(), remote }); + await waitFor(() => remote.requests.length > 0, { timeout: 3000 }).catch(() => {}); + assert.ok(remote.requests.length > 0, 'the second tab reached the worker'); +}); + +// With sync on, a tab that has been open past REMOTE_REFRESH_MIN_GAP_MS syncs every time +// it is activated. On a snapshot engine it reads the session and cursor it loaded with, +// so the shared-session shortcut never applies to it: each activation is a worker request, +// and when its old cursor brings rows down, a full read-merge-write of the document. +test('measure: activating long-open tabs on a snapshot engine with sync on', async t => { + const remote = makeRemote(); + const store = makeStore({ snapshotPerTab: true }); + enableSync(store, remote); + const first = await openTab(store, { html: listing(), remote }); + await waitFor(() => store.read(SESSION_KEY)?.completedAt); + await settle(); + let skew = 0; + const tabs = []; + for (let i = 0; i < 10; i++) { + tabs.push(await openTab(store, { html: listing(), remote, beforeRun(window) { + const now = window.Date.now.bind(window.Date); + window.Date.now = () => now() + skew; + } })); + } + await settle(); + // Other tabs and devices keep adding visits; the worker has them. + remote.store.sync({ cursor: 0, changes: Array.from({ length: 20 }, (_, i) => ({ kind: 'v', key: `https://javstore.net/other-${i}.html`, at: Date.now() - 1000 + i })) }); + clickCard(first.window, '/a.html'); + await waitFor(() => remote.requests.length >= 2, { timeout: 9000 }); + await settle(); + + const requests = remote.requests.length; + let writes = 0; + const set = store.set.bind(store); + store.set = async (key, value) => { if (key === STATE_KEY) writes++; return set(key, value); }; + skew = 2 * 60000; // two minutes later the user flips through the tabs + for (const tab of tabs) { + tab.window.document.dispatchEvent(new tab.window.Event('visibilitychange')); + tab.window.dispatchEvent(new tab.window.Event('focus')); + } + await waitFor(() => remote.requests.length >= requests + tabs.length, { timeout: 9000 }).catch(() => {}); + await settle(80); + t.diagnostic(`10 long-open tabs activated once each: ${remote.requests.length - requests} worker requests, ${writes} full-document writes`); + assert.ok(remote.requests.length - requests >= tabs.length, 'each stale tab syncs on activation'); +}); + +// --------------------------------------------------------------------------------------- +// 4. Page lifecycle. +// --------------------------------------------------------------------------------------- + +// visibilitychange and focus both fire when a tab is activated, and both start a sync: +// lastRemoteAttemptAt is set only once runSync gets the lock, after both handlers ran. +// The second one finds the session fresh and only reads storage and rewrites the session, +// but that is still two full-document reads on top of the two refreshFromStorage reads. +test('measure: one tab activation runs two syncs (visibilitychange and focus)', async t => { + const store = makeStore(); + const remote = makeRemote(); + enableSync(store, remote); + let skew = 0; + const tab = await openTab(store, { html: listing(), remote, beforeRun(window) { + const now = window.Date.now.bind(window.Date); + window.Date.now = () => now() + skew; + } }); + await waitFor(() => store.read(SESSION_KEY)?.completedAt); + await settle(); + const m = meter(store); + skew = 2 * 60000; + const before = m.snapshot(); + tab.window.document.dispatchEvent(new tab.window.Event('visibilitychange')); + tab.window.dispatchEvent(new tab.window.Event('focus')); + await settle(80); + const cost = m.since(before); + t.diagnostic(`activation with sync on: ${describe(cost)}, session writes ${cost.sessionWrites}, requests ${remote.requests.length - 1}`); + assert.equal(cost.sessionWrites, 2, 'both handlers queue a sync before either records its attempt'); + assert.ok(cost.mainReads >= 4, 'two refreshes plus two sync snapshots read the document'); +}); + +// After the page comes back from the back/forward cache its timers are restarted and the +// tab syncs; a click recorded before it was hidden is still pushed. +test('a page restored from the back/forward cache syncs again and keeps its poll timers', async () => { + const store = makeStore(); + const remote = makeRemote(); + enableSync(store, remote); + const intervals = new Set(); + const tab = await openTab(store, { html: listing(), remote, beforeRun(window) { + const setInterval = window.setInterval.bind(window); + const clearInterval = window.clearInterval.bind(window); + window.setInterval = (callback, ms) => { const id = setInterval(callback, ms); intervals.add(id); return id; }; + window.clearInterval = id => { intervals.delete(id); clearInterval(id); }; + } }); + await waitFor(() => store.read(SESSION_KEY)?.completedAt); + assert.equal(intervals.size, 2, 'storage poll and remote poll'); + tab.window.dispatchEvent(new tab.window.Event('pagehide')); + assert.equal(intervals.size, 0, 'both timers stop on pagehide'); + const restored = new tab.window.Event('pageshow'); + Object.defineProperty(restored, 'persisted', { value: true }); + tab.window.dispatchEvent(restored); + assert.equal(intervals.size, 2, 'both timers come back'); + clickCard(tab.window, '/a.html'); + await waitFor(() => remote.visited().includes(abs('/a.html')), { timeout: 9000 }); +}); + +// --------------------------------------------------------------------------------------- +// 5. Early paint and the observer. +// --------------------------------------------------------------------------------------- + +test('measure: tiles are processed a bounded number of times while a 600-tile listing streams in', async t => { + const store = makeStore(); + store.write(STATE_KEY, bigHistory()); + const hrefs = Array.from({ length: 600 }, (_, i) => `https://javstore.net/${600000 + i}-FC2-PPV-${3000000 + i}-Some-Typical-Title-Words-pn.html`); + let processed = 0; + const tab = await openTab(store, { html: '
', beforeRun(window) { + Object.defineProperty(window.document, 'readyState', { configurable: true, get: () => 'loading' }); + const setAttribute = window.Element.prototype.setAttribute; + window.Element.prototype.setAttribute = function (name, value) { + if (name === 'aria-description') processed++; + return setAttribute.call(this, name, value); + }; + } }); + const grid = tab.window.document.querySelector('main .grid'); + const started = performance.now(); + // The parser appends one element at a time; tasks in between let the observer run. + for (let i = 0; i < hrefs.length; i += 20) { + for (const href of hrefs.slice(i, i + 20)) { + const a = tab.window.document.createElement('a'); + a.href = href; + a.innerHTML = '

Title

'; + grid.append(a); + } + await new Promise(resolve => setTimeout(resolve, 0)); + } + await settle(); + const ms = performance.now() - started; + const visited = tab.window.document.querySelectorAll('a.jvs-visited').length; + t.diagnostic(`600 tiles: ${processed} processCard passes (${(processed / 600).toFixed(2)} per tile), ${visited} marked visited, ${ms.toFixed(0)} ms in jsdom`); + assert.equal(visited, 600); + assert.ok(processed / 600 <= 3); +}); From 3b73ccb3c7a159a62d7c5e58c1b0f07033030307 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 12 Nov 2026 19:13:28 +0000 Subject: [PATCH 06/11] Add failing review tests for the 6.11.1 sync-position fix Four cases the fix leaves open: stamp-less documents from tabs still running 6.11.0 (repeated Restored toast with a full download; silent loss when the ids match), a second history save and a follow-up sync pass for every push-only sync, and a stale stamp surviving a document-only exchange. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VCuhK4TQqcEbLXdmxLjggy --- tests/review-fix.test.mjs | 190 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 190 insertions(+) create mode 100644 tests/review-fix.test.mjs diff --git a/tests/review-fix.test.mjs b/tests/review-fix.test.mjs new file mode 100644 index 0000000..4ff9053 --- /dev/null +++ b/tests/review-fix.test.mjs @@ -0,0 +1,190 @@ +import test, { afterEach } from 'node:test'; +import assert from 'node:assert/strict'; +import { + makeStore, makeRemote, enableSync, openTab, closeTabs, settle, waitFor, + listingHtml, pressOnCard, visitedUrls, +} from './harness.mjs'; + +const SESSION_KEY = 'javstore_sync_session_v1'; +const CONFIG_KEY = 'javstore_sync_config_v1'; +const STATE_KEY = 'javstore_cleanup_state_v2'; +const listing = () => listingHtml([['/old.html', 'Old'], ['/new.html', 'New']]); +afterEach(closeTabs); + +function seededRemote(count) { + const remote = makeRemote(); + const at = Date.now() - 60000; + remote.store.sync({ cursor: 0, changes: Array.from({ length: count }, (_, i) => ({ + kind: 'v', key: `https://javstore.net/${i ? `old-${i}` : 'old'}.html`, at, + })) }); + remote.responses = []; + const handle = remote.handle.bind(remote); + remote.handle = async request => { + const response = await handle(request); + remote.responses.push(JSON.parse(response.text)); + return response; + }; + return remote; +} + +async function restored(remote, store) { + enableSync(store, remote); + const tab = await openTab(store, { html: listing(), remote }); + await waitFor(() => store.read(SESSION_KEY)?.completedAt); + await settle(); + return tab; +} + +// What a tab still running 6.11.0 writes: it never heard of `syncedThrough`, so its +// serializeState drops the field, and under AdGuard it carries the history id of the +// storage it loaded with. +function writtenBy6110Tab(doc, historyId = doc.historyId) { + const { syncedThrough: _dropped, ...rest } = doc; + return { ...rest, historyId, updatedAt: Date.now() }; +} + +function dropJournals(store) { + // Journals retire JOURNAL_TTL_MS (10 min) after a verified save; past that the main + // document is the only local copy. + store.journals().forEach(key => store.data.delete(key)); +} + +// Updating the script does not reload the tabs AdGuard already injected 6.11.0 into. The +// reported device had ~40 of them open, swapping two history ids. Until every one of them +// is reloaded, each of their writes leaves a document with no stamp and the other id, +// and 6.11.1 still reads that as a lost history: a full download and the toast, on every +// page that follows such a write: the exact symptom the release is meant to end. +test('a stamp-less document written by a still-open 6.11.0 tab does not trigger a full download and "Restored" on every page', async () => { + const remote = seededRemote(300); + const store = makeStore({ snapshotPerTab: true }); + await restored(remote, store); + const loadedByOldTab = store.state(); + // Since the old tab loaded, another device added visits and a page here pulled them. + remote.store.sync({ cursor: 0, changes: Array.from({ length: 20 }, (_, i) => ({ + kind: 'v', key: `https://javstore.net/elsewhere-${i}.html`, at: Date.now() - 1000 + i, + })) }); + store.write(SESSION_KEY, { ...store.read(SESSION_KEY), activeAt: 0 }); + const before = store.read(SESSION_KEY).completedAt; + await openTab(store, { html: listing(), remote, url: 'https://javstore.net/page/1' }); + await waitFor(() => store.read(SESSION_KEY).completedAt > before); + await settle(); + for (let round = 0; round < 2; round += 1) { + // The old tab records a visit: its whole document goes back with its own id. + store.write(STATE_KEY, writtenBy6110Tab(loadedByOldTab, 'id-from-before-the-loss')); + dropJournals(store); + const previous = store.read(SESSION_KEY).completedAt; + const requests = remote.requests.length; + const page = await openTab(store, { html: listing(), remote, url: `https://javstore.net/page/${round + 2}` }); + await waitFor(() => store.read(SESSION_KEY).completedAt > previous); + await settle(); + assert.doesNotMatch(page.shadow().textContent, /Restored \d+ visited URLs/, `round ${round}`); + assert.notEqual(remote.requests[requests].body.cursor, 0, `round ${round}: full download`); + } +}); + +// The other half of the same transition: when the old 6.11.0 tab and the configuration +// agree on the id, the stamp-less fallback trusts the configuration's cursor, which the old +// copy does not reach. The rows in between are never downloaded, and the save that follows +// writes a stamp claiming they are held, so nothing after the old tabs close repairs it. +test('a stamp-less older copy with the matching id still catches up instead of trusting the configuration cursor', async () => { + const remote = seededRemote(50); + const store = makeStore({ snapshotPerTab: true }); + await restored(remote, store); + const loadedByOldTab = store.state(); + + // Another device adds visits; a page on this device pulls them. + remote.store.sync({ cursor: 0, changes: Array.from({ length: 20 }, (_, i) => ({ + kind: 'v', key: `https://javstore.net/elsewhere-${i}.html`, at: Date.now() - 1000 + i, + })) }); + store.write(SESSION_KEY, { ...store.read(SESSION_KEY), activeAt: 0 }); + let previous = store.read(SESSION_KEY).completedAt; + await openTab(store, { html: listing(), remote, url: 'https://javstore.net/page/2' }); + await waitFor(() => store.read(SESSION_KEY).completedAt > previous); + await settle(); + assert.equal(visitedUrls(store).length, 70); + + // The 6.11.0 tab, loaded before that, writes back its copy with the same id. + store.write(STATE_KEY, writtenBy6110Tab(loadedByOldTab)); + dropJournals(store); + store.write(SESSION_KEY, { ...store.read(SESSION_KEY), activeAt: 0 }); + previous = store.read(SESSION_KEY).completedAt; + await openTab(store, { html: listing(), remote, url: 'https://javstore.net/page/3' }); + await waitFor(() => store.read(SESSION_KEY).completedAt > previous); + await settle(); + assert.ok(visitedUrls(store).includes('https://javstore.net/elsewhere-0.html'), 'pulled rows lost for good'); + assert.equal(visitedUrls(store).length, 70); +}); + +// A push-only sync moves the cursor (the worker numbers this device's own rows), and 6.11.1 +// now rewrites and re-reads the whole history document for that alone. On AdGuard that is +// ~500 KB serialized, written, and read back again, for rows the document already holds. +// One visit should cost one save of the history, not two. +test('a visit that is only pushed up saves the history document once', async () => { + const remote = seededRemote(1); + const store = makeStore(); + const tab = await restored(remote, store); + let saves = 0; + const set = store.set.bind(store); + store.set = async (key, value) => { + if (key === STATE_KEY) saves += 1; + return set(key, value); + }; + const requests = remote.requests.length; + pressOnCard(tab.window, '/new.html', 'v'); + await waitFor(() => remote.requests.length > requests, { timeout: 9000 }); + await waitFor(() => store.read(CONFIG_KEY).cursor === 2); + await settle(80); + assert.equal(saves, 1, `history document saved ${saves} times for one pushed visit`); +}); + +function remoteTag(endpoint) { + let hash = 0x811c9dc5; + for (let index = 0; index < endpoint.length; index += 1) { + hash ^= endpoint.charCodeAt(index); + hash = Math.imul(hash, 0x01000193) >>> 0; + } + return hash.toString(16).padStart(8, '0'); +} + +// runSync means to drop the stamp after a document-only exchange (`reached` is empty for +// `legacy`), but the save that follows re-adopts the old stamp from storage, because an +// empty in-memory stamp never outranks a stored one. The configuration is reset to +// cursor 0 / remoteKey '' as before, yet the next exchange with a cursor-speaking worker +// at that address resumes from the stale stamp instead of 0, which 6.11.0 never did +// (its cursor required `sameRemote`). +test('a document-only exchange forgets the stamp, so a worker that speaks cursors again starts from zero', async () => { + const v3 = makeRemote(); + v3.store.sync({ cursor: 0, changes: [{ kind: 'v', key: 'https://javstore.net/new.html', at: Date.now() - 5000 }] }); + const legacy = { + async handle({ body }) { + const parsed = JSON.parse(body); + if (parsed.state) { + legacy.document = { ...legacy.document, visited: { ...legacy.document.visited, ...parsed.state.visited } }; + } + return { status: 200, statusText: '', text: JSON.stringify({ state: legacy.document }) }; + }, + document: { version: 3, updatedAt: Date.now(), visited: {} }, + endpoint: v3.endpoint, + token: v3.token, + requests: [], + }; + const store = makeStore(); + enableSync(store, legacy); + // This device last spoke to a cursor worker at the same address, up to row 500. + store.write(STATE_KEY, { + version: 3, updatedAt: Date.now() - 10000, historyId: 'h', + syncedThrough: { remote: remoteTag(v3.endpoint), cursor: 500 }, + visited: { 'https://javstore.net/old.html': Date.now() - 20000 }, + }); + await openTab(store, { html: listing(), remote: legacy }); + await waitFor(() => store.read(SESSION_KEY)?.completedAt); + await settle(); + assert.equal(store.read(CONFIG_KEY).remoteKey, ''); + + const previous = store.read(SESSION_KEY).completedAt; + await openTab(store, { html: listing(), remote: v3 }); + await waitFor(() => store.read(SESSION_KEY).completedAt > previous); + await settle(); + assert.equal(v3.requests[0].body.cursor, 0); + assert.ok(visitedUrls(store).includes('https://javstore.net/new.html')); +}); From 353006a533ae303c079c2ec658388317ededb2fb Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 12 Nov 2026 19:41:57 +0000 Subject: [PATCH 07/11] Design history and sync around AdGuard's per-tab storage (6.12.0) AdGuard gives each tab the stored values as they were when it loaded and never shows it other tabs' writes. A review of the whole design against that found data loss and sync faults wherever a long-open tab wrote what it had loaded; each is now reproduced by a test. - Without change notifications, a tab open more than eight minutes saves only to its own journal; newer tabs fold journals in. - Journal keys are written for five minutes at most, then final, and are removed only when the main document a newer tab loaded covers them and no tab lacking them may still write the main document. - The single push horizon is replaced by an outbox of unconfirmed changes, merged by union across the document and journals. - The download position is kept per worker, with the worker's new store id (epoch) and a start generation, so a reset worker is re-seeded and a stale copy can never claim rows it lacks. Journals carry it. - Sync never writes the user's sync settings. - A push-only sync updates the journal, not the main document; startup syncs again after the poll interval; requests and the sync lock are bounded; one sync per tab activation; no storage lease where tabs cannot see each other's writes; imports restore entries unmarked after the backup was taken; boot applies removals from journals. Adds the reviewers' acceptance suites (review-*.test.mjs) and adapts existing tests to the position living in the history. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VCuhK4TQqcEbLXdmxLjggy --- CHANGELOG.md | 26 +- README.md | 14 +- javstore-full-layout-cleanup.user.js | 814 ++++++++++++++++++--------- tests/harness.mjs | 14 + tests/persistence.test.mjs | 22 +- tests/review-fix.test.mjs | 12 +- tests/review-lifecycle.test.mjs | 46 +- tests/review-persistence.test.mjs | 10 +- tests/review-sync.test.mjs | 14 +- tests/sync-session.test.mjs | 33 +- tests/sync.test.mjs | 24 +- worker/src/worker.mjs | 15 +- 12 files changed, 691 insertions(+), 353 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8ddd065..3f99c07 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,12 +1,24 @@ # Changelog -## 6.11.1 - -- Fix: on AdGuard, "Restored N visited URLs from the worker" could appear on page after page while other JavStore tabs were open, and each time the device downloaded its whole history again. AdGuard gives every tab the storage it had when that tab loaded, and a tab never sees what other tabs save after that. So a tab left open for a while saves an older copy of the history, carrying the history id it was loaded with. After any loss of history (an AdGuard update wiping storage, for example), those old tabs and the newer ones kept swapping two ids. Every new page then took the mismatch for a lost history. -- Fix: in the same situation with matching ids, an older copy written back by a long-open tab could lose visits for good. The sync bookmark still claimed the device held everything up to the newest row, so what the old copy lacked was never downloaded again. -- Change: the history now records how far into the worker's history it is complete, in the same save as the visits themselves. The record names the worker by a hash, never by its address. A page that finds an older copy catches up from where that copy reaches, quietly and only for the difference. The note and the full download are left for a history that has really been lost. Histories saved by 6.11.0 or earlier keep using the bookmark from the sync settings until their next sync. -- Tabs that were already open before updating keep running the code they loaded with, so the note can still appear until they are reloaded or closed. -- The worker version moves to 6.11.1 with the userscript; the worker is unchanged. +## 6.12.0 + +A review of the whole history and sync design against how AdGuard actually stores userscript data. AdGuard for Mac, Windows and Android give each tab the stored values as they were when the tab loaded, and a tab never sees what other tabs save after that. The script had been designed as if every tab read the same, current storage. With many tabs left open, that broke in several ways, each now covered by a test that reproduces it. + +- Fix (data loss): a tab left open for a while wrote its old copy of the history over the newer one. Visits other tabs recorded after it loaded were lost once their journals were removed ten minutes later. Unmarked items and per-card overrides came back, and with sync off nothing could recover them. Now, on managers that don't report other tabs' writes, a tab open for more than eight minutes saves only to its own journal, and newer tabs fold that journal into the shared history. +- Fix (data loss): a long-open tab could remove another page's journal after that page had written to it again, because its check read the tab's own stale copy. Each journal key is now written for five minutes at most and then left unchanged. It is removed only once it has been unchanged for longer than any tab is allowed to keep writing the shared history, and only after a newer tab has found it already in the saved history. +- Fix (data loss): a visit could be stranded on the device for good. A long-open tab that synced recorded "everything up to now is uploaded", including visits other tabs had recorded that it could not see. The single upload mark is replaced by a list of unconfirmed changes, kept in the history and in each journal and combined across all copies. A change leaves the list only when the worker has confirmed it; an older copy can at worst offer a confirmed change again, which the worker ignores. +- Fix: a long-open tab that synced wrote back its whole copy of the sync settings, which could turn sync back on, restore an old token or endpoint, or undo a pasted sync link. Sync no longer writes the sync settings at all. +- Fix: with other JavStore tabs open, "Restored N visited URLs from the worker" could appear on page after page, and each time the device downloaded its whole history again. Old tabs saved their outdated copy of the history, with the history id they had loaded, so the stored history and the sync settings kept naming different histories, and every new page took that for a lost history. The download position now lives inside the history, in the same save as the visits it covers. It is kept per worker (the worker is named by a hash, never by its address), and journals carry it along with their rows. A page that finds an older copy catches up from where that copy reaches, quietly and only for the difference. In the same situation, an older copy could also lose visits for good, because the position claimed rows the copy did not hold; that can no longer happen either. Histories saved by 6.11.0 or earlier are downloaded again once, without the note. The note is reserved for a history that has lost most of what the worker holds. +- Fix: at the 5,000-visit limit, each new visit could upload the whole history again. +- Fix: a sync that only uploaded changes rewrote the whole history document a second time. It now updates the page's journal only. +- Fix: a device browsing listing pages without pause never downloaded other devices' visits. Page loads inside a session now check the worker again once the sync interval has passed. +- Fix: importing a backup could not restore a visit unmarked after the backup was taken: the newer unmark won the next merge. Such entries are now restored just after the unmark. +- Fix: a sync request that never finished held up syncing in every other tab. Requests now time out on the page's side as well, and no tab holds the sync lock for more than five minutes. Bringing a tab to the front runs one sync instead of two. +- Fix: on AdGuard, the storage-based lock used when Web Locks are missing could not keep tabs apart, and a lease left behind could hold a tab up for 95 seconds. It is no longer used on managers that don't report other tabs' writes; tabs sync side by side, at the cost of at most a duplicate request. +- Fix: opening a new tab could show a visit that had just been unmarked, or cleared, in another tab, and save it back. +- Change: the worker now gives its store a random id and returns it with every answer. A device that finds a different store at the same address, such as one recreated empty, starts over and sends it everything it holds; until now such a worker stayed empty. The worker needs redeploying; the Cloudflare build does that when this reaches `main`. Until it does, an older worker still works, and a reset is noticed from its sequence numbers going backwards. +- After updating, reload or close JavStore tabs that were open before the update. Until then they keep running the old code. +- The worker version moves to 6.12.0 with the userscript. ## 6.11.0 diff --git a/README.md b/README.md index d252bc8..4206e1e 100644 --- a/README.md +++ b/README.md @@ -32,11 +32,12 @@ Userscript-manager storage is local to one device and AdGuard has been known to Once it is deployed, open the `JVS` panel → **Cloud sync**, enter the worker URL and the access token you generated, tick the box, and save. Repeat on every device with the same URL and token. -- The first page in a browsing session checks the worker; later pages and new tabs immediately use the shared local history. A session is shared across tabs and expires after 30 minutes without startup/sync activity. Starting another session checks for new changes using the saved cursor; it does not download the entire history again. +- The first page in a browsing session checks the worker; later pages and new tabs use the shared local history, and check again once the sync interval has passed so that visits from other devices keep arriving while you browse. A session is shared across tabs and expires after 30 minutes without startup/sync activity. Starting another session checks for new changes from the saved position; it does not download the entire history again. - Opening 30–50 item tabs from the context menu still records every visit when its item page loads. Uploads wait a few seconds and collect visits from all tabs; tabs whose changes were already uploaded skip their redundant requests. Advancing the listing to page 2 does not restart restoration. - **Sync now** always checks the worker. Focus refreshes and the configured periodic refresh use shared progress, so tabs do not independently download the same changes. Hidden tabs skip periodic refreshes, but their new visits can still upload. The panel's status line reports the last sync or the reason the last one failed. -- Devices exchange only what changed: each remembers the sequence number it last saw and the moment it last pushed, so incremental checks exchange only new rows, and an ordinary page load during an active session needs no cloud request. The first sync on a device transfers everything, and so does the next sync on a device whose userscript manager lost its history but kept the sync settings: the history carries an id that the saved position is tied to, so a history without it is downloaded again rather than left behind. -- Tabs on one device share history, a sync cursor, and session progress. Before deciding what to send, a sync merges what the device has on disk, so a visit recorded in one tab is not passed over by another tab's sync; and an entry that turns up stamped before the last push—a click replayed from the previous page, a restored backup—pulls the mark back to itself rather than being left behind it. +- Devices exchange only what changed. The stored history records, in the same save as the visits, how far into the worker's history it is complete, and keeps a list of the changes made on the device that the worker has not confirmed yet. Incremental checks exchange only new rows, and an ordinary page load during an active session needs no cloud request. The first sync with a worker transfers everything, and so does the next sync on a device whose userscript manager lost its history but kept the sync settings: a history with no position for the worker is downloaded again rather than left behind. +- Each worker store has a random id. A device that finds a different store at the same address (for example one recreated empty) starts over against it and sends it everything it holds. +- Sync never writes the sync settings themselves (endpoint, token, on/off, interval); only the panel and sync links do, so a tab that was opened before a change cannot undo it. - The worker applies the same rules as the local merge—newest timestamp wins per URL—so a device that has been offline for a week cannot overwrite what the others recorded. A device that pushes a stale entry is handed the winning one back. - No device can delete visited history from the worker. **Clear visited history**, retention pruning, unmarking a card and a backup import that leaves entries out all apply to the device they happen on; the worker and the other devices keep their copies. The worker drops only its oldest visits beyond 50,000, to bound its storage. Per-card overrides are choices rather than history, so removing one still reaches every device. - Because a Durable Object is single-threaded, each sync's read-merge-write is serialized and strongly consistent: two devices syncing in the same second queue behind one another rather than racing. @@ -44,8 +45,8 @@ Once it is deployed, open the `JVS` panel → **Cloud sync**, enter the worker U - The Cloud sync panel shows the worker version from the worker's last answer. When the worker is behind the userscript, a **⚠ Redeploy worker** button appears on screen next to the JavStore controls; tap it for the versions. The worker redeploys through Cloudflare whenever `main` changes, so a lasting warning means its Cloudflare build did not go through. When the worker is ahead instead, an **⚠ Update script** button appears on screen and links to the latest release, because the userscript manager has not installed it yet. - AdGuard for Android clears a userscript's storage whenever it installs an update. After setting sync up, press **Copy sync link** and keep the link somewhere private, such as a password manager. After a wipe, open it on JavStore, or paste it into the endpoint box and save, to set everything up again. The link carries your token, so treat it like a password. - The endpoint and token live only on the device they were entered on. They are never written into the synced document, never appear in an exported backup, and are not left in the page: the token box stays empty once a token is stored and only reports that one exists. -- A worker that is unreachable does not affect anything locally: history is still saved to the userscript manager, and the next sync picks it up. Failed or incomplete restoration is not marked complete. If local history is actually lost, recovery bypasses the session marker and downloads it again; **Restored … visited URLs** reports that recovery, not ordinary page navigation. The history records how far into the worker's history it is complete, so an older copy written back by a tab that was opened earlier (AdGuard hands each tab the storage it had when it loaded) only catches up on the difference, without the note. -- Current browsers coordinate cloud work with Web Locks. Browsers without that API use a userscript-storage lease with expiry and ownership checks, so a tab that disappears cannot hold sync indefinitely. Session/progress metadata stays in userscript-manager storage; credentials and history are not placed in page `localStorage`. +- A worker that is unreachable does not affect anything locally: history is still saved to the userscript manager, and the next sync picks it up. Failed or incomplete restoration is not marked complete. If local history is actually lost, recovery bypasses the session marker and downloads it again; **Restored … visited URLs** reports that recovery, not ordinary page navigation. A history that is merely behind catches up on the difference without the note. +- Current browsers coordinate cloud work with Web Locks, and no tab holds them for more than five minutes. Managers that notify tabs of each other's writes fall back to a userscript-storage lease where Web Locks are missing; elsewhere tabs sync side by side, which costs at most a duplicate request. Session metadata stays in userscript-manager storage; credentials and history are not placed in page `localStorage`. - **Sync now** reports both directions—what went up and what came down—so a device that is failing to push is not mistaken for one with nothing to push. - Restoring a backup keeps the backup's own timestamps, so the restored entries reach the worker and the other devices instead of being discarded by them. Entries the backup leaves out are dropped on the importing device only. Entries a history clear on this device already covers cannot be restored, and the panel says how many were skipped. @@ -66,6 +67,7 @@ Once it is deployed, open the `JVS` panel → **Cloud sync**, enter the worker U - Optional userscript APIs: `GM_addValueChangeListener` and `GM_registerMenuCommand`. AdGuard does not implement either; the script polls for changes instead of being notified, and its own on-page control replaces the manager menu. - `GM_getValue`/`GM_setValue` are supported in both the synchronous Tampermonkey style and the asynchronous GM4 style AdGuard uses. Values are stored as JSON strings, because GM4 only promises to keep strings, numbers and booleans, and AdGuard for Android drops stored objects when the page reloads. - `GM_listValues` and `GM_deleteValue` let each page keep its own journal of what it recorded, so tabs saving at the same moment cannot overwrite each other's visits. AdGuard and Tampermonkey provide both; an engine without them falls back to a single stored document. +- AdGuard (for Mac, Windows and Android) gives each tab the stored values as they were when the tab loaded, and never shows it what other tabs save afterwards. On managers without `GM_addValueChangeListener`, a tab open for more than eight minutes therefore saves only to its own journal and leaves the shared history to newer tabs, which fold its journal in. Journal keys are each written for five minutes at most and removed only once a newer tab has found them in the saved history, so a long-open tab can neither write an older history over a newer one nor remove another tab's journal. Long-open tabs do not show what other tabs recorded after they loaded until they sync or are reloaded. The script depends on JavStore's current page structure. If the site changes, please [open an issue](https://github.com/jitdor/userscript-javstore/issues). @@ -75,6 +77,6 @@ The distributable is [`javstore-full-layout-cleanup.user.js`](./javstore-full-la Releases publish themselves: a push to `main` whose `@version` header names a version with no release yet runs the tests and publishes that release, which is what `@updateURL`/`@downloadURL` point at. A push that does not bump the header is a no-op, so releasing a change means bumping `@version` and `SCRIPT_VERSION` and adding the matching `## ` section to `CHANGELOG.md` in the same pull request. The workflow can still be run manually to release from a branch. -`npm install && npm test` runs the link, startup, storage-persistence, and cloud-sync suites (a current Node.js 22 or newer with `node:sqlite`; CI uses Node.js 22). The suites load the userscript into jsdom windows that share one asynchronous value store with no change notifications—an AdGuard-shaped engine—and assert that history survives reloads, concurrent tabs, interrupted saves, and explicit removals. The store can be given random per-call latency, which is how the suite reproduces several tabs' saves overlapping on a slow engine. The sync suite additionally runs the real worker from [`worker/src/worker.mjs`](./worker/src/worker.mjs): its Durable Object class executes against a `node:sqlite` database standing in for `ctx.storage.sql`, with each jsdom window acting as a separate device, so both halves of a sync are covered against real SQL. +`npm install && npm test` runs the link, startup, storage-persistence, and cloud-sync suites (a current Node.js 22 or newer with `node:sqlite`; CI uses Node.js 22). The suites load the userscript into jsdom windows that share one asynchronous value store with no change notifications, or (with `snapshotPerTab`) give each window its own copy of the store taken when it opened, as AdGuard does, and assert that history survives reloads, concurrent tabs, interrupted saves, and explicit removals. The store can be given random per-call latency, which is how the suite reproduces several tabs' saves overlapping on a slow engine. The sync suite additionally runs the real worker from [`worker/src/worker.mjs`](./worker/src/worker.mjs): its Durable Object class executes against a `node:sqlite` database standing in for `ctx.storage.sql`, with each jsdom window acting as a separate device, so both halves of a sync are covered against real SQL. Session regression tests assert request counts and downloaded rows across navigation, concurrent first tabs, a 50-tab context-menu burst, session expiry, stale tab cursors, failed progress writes, and actual history loss. diff --git a/javstore-full-layout-cleanup.user.js b/javstore-full-layout-cleanup.user.js index 636ba70..60ccb28 100644 --- a/javstore-full-layout-cleanup.user.js +++ b/javstore-full-layout-cleanup.user.js @@ -1,7 +1,7 @@ // ==UserScript== // @name JavStore Full Layout Cleanup - No Sidebars + Mosaic Overlay // @namespace http://tampermonkey.net/ -// @version 6.11.1 +// @version 6.12.0 // @description Clean up JavStore's layout, filter keyword-matched thumbnails, and track visited items with private, configurable controls. // @homepageURL https://github.com/jitdor/userscript-javstore // @supportURL https://github.com/jitdor/userscript-javstore/issues @@ -29,7 +29,7 @@ (function () { 'use strict'; - const SCRIPT_VERSION = '6.11.1'; + const SCRIPT_VERSION = '6.12.0'; // The same address as @downloadURL: opening it hands the newest release to the // userscript manager, which offers to install it. const INSTALL_URL = 'https://github.com/jitdor/userscript-javstore/releases/latest/download/javstore-full-layout-cleanup.user.js'; @@ -38,11 +38,29 @@ const LEGACY_STORAGE_KEY = 'javstore_seen_links'; const PENDING_VISITS_KEY = 'javstore_pending_visits'; // Each page keeps its own journal under this prefix; see "Per-page journals" below. - const JOURNAL_KEY_PREFIX = 'javstore_journal_v1_'; - // How long a journal outlives the last write that could have depended on it. A - // read-merge-write cycle takes well under a second even on a slow engine, so this is - // generous; it only bounds how many stale journal keys can pile up. + // Journals written since 6.12.0 live under this prefix and are each written for a + // limited time only (JOURNAL_ROTATE_MS); those under the older prefix were rewritten + // for as long as their page stayed open. + const JOURNAL_KEY_PREFIX = 'javstore_journal_v2_'; + const LEGACY_JOURNAL_KEY_PREFIX = 'javstore_journal_v1_'; + // AdGuard hands each tab the stored values as they were when the tab loaded, and never + // shows it what other tabs write afterwards. A tab is trusted to write the shared + // history only while what it loaded is younger than FRESH_TAB_MS; past that it writes + // its own journal and nothing else. See "Per-page journals" below. + const FRESH_TAB_MS = 8 * 60000; + // How long a page writes one journal key before starting the next. A key that is no + // longer written is final, so any tab that loaded after that holds all of it. + const JOURNAL_ROTATE_MS = 5 * 60000; + // How long after its last possible write a journal may be removed: by then every tab + // that loaded before that write is too old to write the shared history. const JOURNAL_TTL_MS = 10 * 60000; + // Slack for a write that is still on its way into storage when a tab loads. + const STORAGE_SETTLE_MS = 60000; + // Journals from before 6.12.0 can still be rewritten by tabs running the older code. + const LEGACY_JOURNAL_TTL_MS = 24 * 3600000; + // Changes the worker has confirmed stay listed this long, so an older copy that still + // lists them as unsent does not offer them again straight away. + const OUTBOX_KEEP_MS = 3600000; // The sync endpoint and its token are deliberately kept outside the synchronized // document: they are per-device credentials, so they neither travel to the worker nor // end up in an exported backup. @@ -51,6 +69,9 @@ const SYNC_LEASE_KEY = 'javstore_sync_lease_v1'; const SYNC_SESSION_IDLE_MS = 30 * 60000; const SYNC_LEASE_MS = 90000; + // The longest one exchange may hold the sync lock. Its requests time out long before; + // this is for whatever else could hang, so one stuck tab cannot stop every tab syncing. + const SYNC_HOLD_MS = 5 * 60000; const MAX_VISITED_ITEMS = 5000; const MAX_PENDING_VISITS = 50; const MAX_TOMBSTONES = 2000; @@ -128,14 +149,24 @@ // configuration records the id the cursor belongs to, so a history that no longer // carries it—wiped, unreadable, or written before ids existed—is downloaded in full. let historyId = ''; - // How far into the worker's history this document is complete: every row up to - // `cursor` on the worker `remote` names (by a hash of its endpoint, which itself stays - // out of the history) is already merged into it. It is saved in the same - // write as the rows it describes, so the two cannot disagree. A cursor kept only in the - // sync configuration could: an engine that hands each tab the storage it had when the - // tab loaded (AdGuard does) lets a long-open tab write back an older history while the - // configuration still claims everything up to a later row is on the device. - let syncedThrough = { remote: '', cursor: 0 }; + // How far into each worker's history this document is complete, keyed by a hash of the + // worker's endpoint (the endpoint itself stays out of the history): every row up to + // `cursor` is already merged into it. It is saved in the same write as the rows it + // describes, so the two cannot disagree. A cursor kept only in the sync configuration + // could: an engine that hands each tab the storage it had when the tab loaded (AdGuard + // does) lets a long-open tab write back an older history while the configuration still + // claims everything up to a later row is on the device. + // + // `epoch` is the worker's own id for its store, so a worker that comes back empty at + // the same address is noticed; `gen` is when this device started counting against + // that store, so a position from an earlier start never outranks a later one. + let syncedThrough = {}; + // Changes made on this device that the worker has to hear about, keyed `kind|key`: + // `at` is the change, `sent` the newest change the worker has confirmed, `t` when the + // entry last changed. A change is outstanding while `at > sent`. Both only ever grow, + // so copies merge by taking the larger of each, and an older copy can at worst offer a + // confirmed change once more—which the worker takes as already known. + let outbox = new Map(); let settingsUpdatedAt = 0; // When each setting was last changed on purpose, keyed by setting name. Settings merge // field by field on these, so changing one setting on one device does not carry that @@ -148,7 +179,6 @@ let writeQueue = Promise.resolve(false); let syncConfig = { ...DEFAULT_SYNC_CONFIG }; let remoteQueue = Promise.resolve(false); - let pendingHorizonWrite = Promise.resolve(); let pageClosed = false; let remotePushTimer = 0; let remotePollTimer = 0; @@ -166,28 +196,34 @@ // different. That still has to reach storage, so the last merge also records whether it // adopted anything at all. let lastMergeAdopted = false; - // The oldest entry the last merge took on that the push horizon has already moved - // past. Such an entry would never be offered to the worker again, so the horizon is - // pulled back to it; `mergeStoredState` only reports it, because it also runs while - // remote rows are being folded in, where the horizon must not move at all. - let lastMergeBackdatedTo = 0; - // The oldest timestamp among the entries the last merge took on, whatever the push - // horizon: a merge of rows that came from outside local storage has to be carried by - // this page's journal back to there. + // The oldest timestamp among the entries the last merge took on: a merge of rows that + // came from outside local storage has to be carried by this page's journal back to + // there. let lastMergeOldest = 0; // A local change made while a sync was writing could not arm the push timer, because // arming it there would have the sync push its own merge straight back. The change // still has to go up, so the sync re-arms it once it is done. let deferredRemotePush = false; + let legacyOutboxSeeded = false; let syncTimer = 0; let storageAvailable = true; - // This page's journal: its key, the oldest timestamp it still has to carry, and the - // verified saves that may later let it carry less. See "Per-page journals" below. - const journalKey = `${JOURNAL_KEY_PREFIX}${Date.now().toString(36)}_${Math.random().toString(36).slice(2, 10)}`; + // When this page loaded, which is when its view of AdGuard's storage was taken. + const pageLoadedAt = Date.now(); + // Whether the userscript manager tells a tab about other tabs' writes. Without that, + // a tab's reads are assumed to be the snapshot it loaded with; see FRESH_TAB_MS. + const liveStorage = typeof GM_addValueChangeListener === 'function' + || (typeof GM === 'object' && GM && typeof GM.addValueChangeListener === 'function'); + // This page's journal: its current key, when that key was started, and the oldest + // timestamp it has to carry. See "Per-page journals" below. + const journalToken = Math.random().toString(36).slice(2, 10); + let journalCreatedAt = Date.now(); + let journalKey = makeJournalKey(journalCreatedAt); let journalSince = Date.now(); - let journalCheckpoints = []; - let journalLowerings = 0; let journalWritten = false; + // What this page loaded: the main document and the journals beside it. Only journals + // that this already-stored document covers may be removed. + let loadedSnapshot = { main: null, journals: [] }; + const retiredJournals = new Set(); let observer = null; let ui = null; let selectedCard = null; @@ -354,28 +390,56 @@ return combineSnapshot(await readStorageSnapshot()); } - // Whether any journal carries something the main document does not: the trace a - // racing write leaves behind, and what the next page load puts back. - function journalsAhead({ main, journals }) { + // Whether a journal carries something the main document does not: the trace a racing + // write, or a tab writing from an older view, leaves behind, and what the next page + // load puts back. A journal with nothing ahead of a stored main document is covered by + // it, which is what allows the journal to be removed. + function journalAhead(doc, main) { + if (!doc || typeof doc !== 'object') return false; const base = main || {}; const ahead = (source, target) => Object.entries(source && typeof source === 'object' ? source : {}) .some(([key, value]) => parseTimestamp(value) > parseTimestamp(target?.[key])); + // A full main document dropped its oldest visits to stay under the cap, and a + // journal still holding one of those is not ahead of it. + const baseVisits = base.visited && typeof base.visited === 'object' ? base.visited : {}; + let capFloor = 0; + if (Object.keys(baseVisits).length >= MAX_VISITED_ITEMS) { + capFloor = Math.min(...Object.values(baseVisits).map(parseTimestamp).filter(Boolean)); + } // A visit the main document has since removed, cleared or pruned is not missing. const visitAhead = visits => Object.entries(visits && typeof visits === 'object' ? visits : {}) .some(([url, value]) => { const at = parseTimestamp(value); - return at > parseTimestamp(base.visited?.[url]) + return at > parseTimestamp(baseVisits[url]) && at > parseTimestamp(base.tombstones?.[tombstoneKey('v', url)]) && at > parseTimestamp(base.resetAt) - && at > parseTimestamp(base.prunedBefore); + && at > parseTimestamp(base.prunedBefore) + && at > capFloor; }); - return journals.some(({ doc }) => doc && ( - visitAhead(doc.visited) + const baseOutbox = parseOutbox(base.outbox); + const outboxAhead = [...parseOutbox(doc.outbox)].some(([id, entry]) => { + const held = baseOutbox.get(id); + // A change already confirmed may have been dropped from the main list. + if (!held) return entry.at > entry.sent; + return entry.at > held.at || entry.sent > held.sent; + }); + const basePositions = parseSyncPositions(base.syncedThrough); + const positionAhead = Object.entries(parseSyncPositions(doc.syncedThrough)) + .some(([tag, position]) => newerSyncPosition(position, basePositions[tag]) === position + && !sameSyncPosition(position, basePositions[tag])); + return visitAhead(doc.visited) || ahead(doc.tombstones, base.tombstones) || ahead(doc.overrideTimes, base.overrideTimes) || parseTimestamp(doc.resetAt) > parseTimestamp(base.resetAt) + || parseTimestamp(doc.prunedBefore) > parseTimestamp(base.prunedBefore) + || ahead(doc.settingTimes, main ? documentSettingTimes(base) : {}) || parseTimestamp(doc.settingsUpdatedAt) > (main ? documentSettingsAt(base) : 0) - )); + || outboxAhead + || positionAhead; + } + + function journalsAhead({ main, journals }) { + return journals.some(({ doc }) => journalAhead(doc, main)); } async function readMainState() { @@ -400,24 +464,48 @@ // Per-page journals // // The history is one stored value, and every save is a read-merge-write of all of it. - // Nothing makes that atomic across tabs: when a burst of Ctrl-clicks opens several - // tabs at once, one tab can read the document, another save a visit, and the first - // then write back a document without it. The merge cannot help, because the visit was - // never in anything the first tab read—and the check after the write passes too, - // because the write that clobbered it really did land. + // Nothing makes that atomic across tabs, and on AdGuard it is worse than a race: each + // tab reads the values storage held when the tab loaded and never sees what other + // tabs write afterwards, so a tab that has been open a while writes back the history + // as it was then, without whatever every other tab has added since. // - // So each page also writes what it has recorded to a key of its own, which no other - // page ever writes. Every read folds all the journals into the main document, so a - // visit that a racing write dropped from the main document comes straight back on the - // next read by any tab, and the next save puts it back for good. A journal is removed - // by whichever page next saves once it has sat unchanged for JOURNAL_TTL_MS and that - // page's own save—which merged it—is verified in place: by then no write that could - // have been built without it can still be in flight. + // So each page also writes what it has recorded—its own visits and changes, rows it + // pulled from the worker, and its place in the sync—to a journal key no other page + // writes. Every read folds all the journals into the main document, so whatever a + // stale write dropped from the main document comes straight back. Three rules keep + // the main document and the journals together holding everything: + // + // - Without notice of other tabs' writes, only a tab that loaded less than + // FRESH_TAB_MS ago writes the main document. An older one writes its journal only. + // - A page writes each journal key for JOURNAL_ROTATE_MS at most and then moves on to + // a new one that carries everything the old one did, so a key that is old enough is + // final: any tab that loaded later holds all of it. + // - A journal is removed only once it is final, the main document this tab loaded + // from storage already covers it, and JOURNAL_TTL_MS have passed since its last + // write, which leaves no tab that might lack it still allowed to write the main + // document. // // Engines without GM_listValues or GM_deleteValue cannot find or retire journals, so // they keep the single-document behaviour. // ------------------------------------------------------------------ + function makeJournalKey(createdAt) { + return `${JOURNAL_KEY_PREFIX}${createdAt.toString(36)}_${journalToken}`; + } + + // When a journal key was started, from its name, for keys that rotate. + function journalCreatedAtOf(key) { + if (!key.startsWith(JOURNAL_KEY_PREFIX)) return 0; + const created = parseInt(key.slice(JOURNAL_KEY_PREFIX.length).split('_')[0], 36); + return Number.isFinite(created) && created > 0 ? created : 0; + } + + // Whether this tab is still allowed to write the main document and the sync session: + // what it loaded is recent enough that every journal written since is still in storage. + function sharedWritesAllowed() { + return liveStorage || Date.now() - pageLoadedAt < FRESH_TAB_MS; + } + function journalingAvailable() { return Boolean(gm.listValues && gm.deleteValue); } @@ -434,7 +522,8 @@ // Read side by side: on an engine where each call is a round trip, one after // another would hold up every page load by the number of open journals. const journals = await Promise.all(keys - .filter(key => typeof key === 'string' && key.startsWith(JOURNAL_KEY_PREFIX)) + .filter(key => typeof key === 'string' + && (key.startsWith(JOURNAL_KEY_PREFIX) || key.startsWith(LEGACY_JOURNAL_KEY_PREFIX))) .map(async key => { try { return { key, doc: await loadValue(key) }; @@ -513,6 +602,8 @@ overrides: { ...(base.overrides && typeof base.overrides === 'object' ? base.overrides : {}) }, overrideTimes: { ...(base.overrideTimes && typeof base.overrideTimes === 'object' ? base.overrideTimes : {}) }, }; + const outboxes = parseOutbox(base.outbox); + const positions = parseSyncPositions(base.syncedThrough); const newest = (target, source) => { if (!source || typeof source !== 'object') return; Object.entries(source).forEach(([key, value]) => { @@ -524,6 +615,17 @@ newest(combined.visited, doc.visited); newest(combined.tombstones, doc.tombstones); combined.resetAt = Math.max(combined.resetAt, parseTimestamp(doc.resetAt)); + for (const [id, entry] of parseOutbox(doc.outbox)) { + const held = outboxes.get(id); + outboxes.set(id, held ? { + at: Math.max(held.at, entry.at), + sent: Math.max(held.sent, entry.sent), + t: Math.max(held.t, entry.t), + } : entry); + } + Object.entries(parseSyncPositions(doc.syncedThrough)).forEach(([tag, position]) => { + positions[tag] = newerSyncPosition(positions[tag], position); + }); combined.prunedBefore = Math.max(combined.prunedBefore, parseTimestamp(doc.prunedBefore)); const docOverrides = doc.overrides && typeof doc.overrides === 'object' ? doc.overrides : {}; const docOverrideTimes = doc.overrideTimes && typeof doc.overrideTimes === 'object' ? doc.overrideTimes : {}; @@ -546,29 +648,19 @@ } } }); + combined.outbox = Object.fromEntries([...outboxes].map(([id, entry]) => [id, [entry.at, entry.sent, entry.t]])); + combined.syncedThrough = positions; return combined; } // Entries this page adopted with a timestamp from the past—a replayed click, rows // pulled from the worker, a restored backup—exist nowhere else in local storage yet, - // so the journal has to reach back far enough to carry them. + // so the journal has to reach back far enough to carry them. It never stops carrying + // them while the page is open: on AdGuard a page cannot tell whether its own write of + // the main document survived another tab writing an older one. function lowerJournalSince(at) { const time = parseTimestamp(at); - if (!time || time >= journalSince) return; - journalSince = time; - // A checkpoint taken before this entry arrived says nothing about it, and neither - // does one from a save that was already under way. - journalCheckpoints = []; - journalLowerings += 1; - } - - // Once a save that covered everything up to some moment has been verified in place for - // longer than JOURNAL_TTL_MS, the journal no longer needs to reach back past it. - function advanceJournalSince(now) { - while (journalCheckpoints.length && now - journalCheckpoints[0].verifiedAt > JOURNAL_TTL_MS) { - const { covers } = journalCheckpoints.shift(); - if (covers >= journalSince) journalSince = covers + 1; - } + if (time && time < journalSince) journalSince = time; } function serializeJournal(writtenAt) { @@ -588,6 +680,10 @@ journalOverrideTimes[url] = at; } } + const journalOutbox = {}; + for (const [id, entry] of outbox) { + if (entry.t >= journalSince) journalOutbox[id] = [entry.at, entry.sent, entry.t]; + } const journal = { version: STORAGE_VERSION, writtenAt, @@ -597,6 +693,9 @@ tombstones: pick(tombstones), overrides: journalOverrides, overrideTimes: journalOverrideTimes, + outbox: journalOutbox, + // Small, and it only ever describes rows this page holds. + syncedThrough: { ...syncedThrough }, }; // Settings are one small object, so they always ride along rather than being // tracked by where they came from; the newest copy wins when journals are folded. @@ -609,41 +708,52 @@ } async function writeJournal(writtenAt) { - if (!journalingAvailable()) return; - advanceJournalSince(writtenAt); + if (!journalingAvailable()) return null; const journal = serializeJournal(writtenAt); const carries = Object.keys(journal.visited).length || Object.keys(journal.tombstones).length || Object.keys(journal.overrides).length + || Object.keys(journal.outbox).length + || Object.keys(journal.syncedThrough).length || resetAt >= journalSince || settingsUpdatedAt >= journalSince; - if (carries) { - await storeValue(journalKey, journal); - journalWritten = true; - } else if (journalWritten) { - // Everything it held has been safely in the main document for a TTL. - await gmCall('deleteValue', journalKey); - journalWritten = false; - } - } - - // Called after this page's save is verified in place: that save merged every journal - // in `snapshot`, so any of them that has sat unchanged past the TTL is no longer the - // only copy of anything a racing write could still drop. - async function retireJournals(snapshot, now) { - if (!journalingAvailable()) return; - for (const { key, doc } of snapshot.journals) { - if (key === journalKey) continue; - const writtenAt = parseTimestamp(doc?.writtenAt); - if (doc && now - writtenAt <= JOURNAL_TTL_MS) continue; + if (!carries) return null; + // A key that has been written long enough is left as it is, final, and the page + // moves to a new one that carries everything the old one did. + const previous = journalWritten && writtenAt - journalCreatedAt > JOURNAL_ROTATE_MS ? journalKey : ''; + if (previous) { + journalCreatedAt = writtenAt; + journalKey = makeJournalKey(journalCreatedAt); + } + await storeValue(journalKey, journal); + journalWritten = true; + if (previous) { + // The new key holds everything the old one did, so the old one is only a copy. + await gmCall('deleteValue', previous).catch(() => {}); + } + return journal; + } + + // Removes the journals the main document this tab loaded already covers, once nothing + // that lacks them can still be written over the main document. + async function retireJournals(now) { + if (!journalingAvailable() || !loadedSnapshot.main) return; + for (const { key, doc } of loadedSnapshot.journals) { + if (key === journalKey || retiredJournals.has(key)) continue; + const created = journalCreatedAtOf(key); + // The last moment its page could have written it. + const closedAt = created + ? created + JOURNAL_ROTATE_MS + : parseTimestamp(doc?.writtenAt) + LEGACY_JOURNAL_TTL_MS; + // Final before this tab loaded, so what it loaded is all of it. + if (closedAt + STORAGE_SETTLE_MS > pageLoadedAt) continue; + if (now < closedAt + JOURNAL_TTL_MS + STORAGE_SETTLE_MS) continue; + if (journalAhead(doc, loadedSnapshot.main)) continue; + retiredJournals.add(key); try { - // Its page may have written to it since this save read it; if so it is - // left for a later save to merge. - const current = await loadValue(key); - if (current && parseTimestamp(current.writtenAt) !== writtenAt) continue; await gmCall('deleteValue', key); } catch (error) { - // Left in place; the next save tries again. + // Left in place; a later page tries again. } } } @@ -658,7 +768,8 @@ resetAt, prunedBefore, ...(historyId ? { historyId } : {}), - ...(syncedThrough.remote ? { syncedThrough: { ...syncedThrough } } : {}), + syncedThrough: { ...syncedThrough }, + outbox: serializeOutbox(), settings: { ...settings }, visited: Object.fromEntries(visited), overrides: Object.fromEntries(overrides), @@ -678,12 +789,14 @@ function forgetVisited(url, at = Date.now()) { visited.delete(url); tombstones.set(tombstoneKey('v', url), at); + noteLocalChange('v', url, at); } function forgetOverride(url, at = Date.now()) { overrides.delete(url); overrideTimes.delete(url); tombstones.set(tombstoneKey('o', url), at); + noteLocalChange('o', url, at); } // Engines without GM_addValueChangeListener—AdGuard is one—leave every tab holding the @@ -694,17 +807,12 @@ // tombstone, or `resetAt` for a full clear) so that merging cannot resurrect them. function mergeStoredState(stored, { adoptSettings = false } = {}) { lastMergeAdopted = false; - lastMergeBackdatedTo = 0; lastMergeOldest = 0; if (!stored || typeof stored !== 'object') return false; - // An entry is "backdated" when it is stamped before this device's push horizon: - // it was recorded somewhere this device had not looked—another tab, a replayed - // click, a restored backup—so the horizon has already swept past it. + // The oldest entry this merge takes on, for the journal to reach back to. const backdated = at => { if (at) lastMergeOldest = lastMergeOldest ? Math.min(lastMergeOldest, at) : at; - if (!at || !pushHorizon() || at >= pushHorizon()) return; - lastMergeBackdatedTo = lastMergeBackdatedTo ? Math.min(lastMergeBackdatedTo, at) : at; }; const remoteVisited = parseVisited(stored.visited); @@ -810,6 +918,9 @@ } } + // What other tabs still have to send, and what they have sent. + if (mergeOutbox(parseOutbox(stored.outbox))) adopted = true; + lastKnownUpdatedAt = Math.max(lastKnownUpdatedAt, remoteUpdatedAt); lastMergeAdopted = adopted || changed; return changed; @@ -817,31 +928,46 @@ // Writes are queued so two read-merge-write cycles can never interleave and drop each // other's changes on engines where GM_setValue resolves asynchronously. - function persistState() { - writeQueue = writeQueue.then(() => writeStoredState(), () => writeStoredState()); - scheduleRemotePush(); + function persistState({ journalOnly = false } = {}) { + const write = () => writeStoredState({ journalOnly }); + writeQueue = writeQueue.then(write, write); + if (!journalOnly) scheduleRemotePush(); return writeQueue; } - async function writeStoredState() { + // `journalOnly` keeps the main document as it is: for a save that only records what + // this page has sent, which no other copy depends on. + async function writeStoredState({ journalOnly = false } = {}) { const startedAt = Date.now(); for (let attempt = 0; attempt < 2; attempt += 1) { try { - const lowerings = journalLowerings; const snapshot = await readStorageSnapshot(); const combined = combineSnapshot(snapshot); adoptHistoryId(combined); adoptSyncedThrough(combined); mergeStoredState(combined); - notePushBacklog(); pruneVisited(); + const now = Date.now(); + // A tab whose view of storage may be missing what other tabs have written + // since does not write over the shared document; its journal carries its + // changes until a newer tab folds them in. Without journals there is + // nothing else to write to. + const shared = !journalingAvailable() || (!journalOnly && sharedWritesAllowed()); + const journal = await writeJournal(now); + if (!shared) { + lastSavedAt = Date.now(); + lastSaveFailed = false; + storageAvailable = true; + if (journal) clearPendingVisits(startedAt, journal); + scheduleCountUpdate(); + return true; + } const payload = serializeState(); - // The journal goes first: if the page unloads between the two writes, the - // journal is the copy that no other tab can overwrite. - await writeJournal(payload.updatedAt); await storeValue(STORAGE_KEY, payload); - // Read back rather than trusting the write: a value that never landed is - // exactly the failure that used to go unnoticed until the history was gone. + // Read back rather than trusting the write: on engines that read storage + // itself, a value that never landed is exactly the failure that used to go + // unnoticed until the history was gone. (On AdGuard the read comes from the + // tab's own copy and proves less, which is why journals carry the rest.) const verified = await loadValue(STORAGE_KEY); if (!verified || typeof verified !== 'object' || parseTimestamp(verified.updatedAt) < payload.updatedAt @@ -853,11 +979,7 @@ lastSaveFailed = false; storageAvailable = true; clearPendingVisits(startedAt, verified); - // Only a save still in place when read back vouches for what it merged. - if (parseTimestamp(verified.updatedAt) === payload.updatedAt) { - if (lowerings === journalLowerings) journalCheckpoints.push({ verifiedAt: Date.now(), covers: payload.updatedAt }); - await retireJournals(snapshot, Date.now()); - } + await retireJournals(Date.now()); scheduleCountUpdate(); return true; } catch (error) { @@ -959,9 +1081,8 @@ if (at <= resetAt || at <= prunedBefore || at <= tombstoneTime('v', url)) return; if (at <= (visited.get(url) || 0)) return; visited.set(url, at); - // The click happened before this page loaded, and possibly before the last - // push: without pulling the horizon back it would never be offered upstream. - lowerPushHorizon(at); + noteLocalChange('v', url, at); + // The click happened before this page loaded, so the journal has to reach back. lowerJournalSince(at); changed = true; }); @@ -984,6 +1105,7 @@ if ((visited.get(url) || 0) >= now) return false; visited.set(url, now); tombstones.delete(tombstoneKey('v', url)); + noteLocalChange('v', url, now); return true; } @@ -1031,7 +1153,10 @@ const entries = JSON.parse(raw); if (Array.isArray(entries)) { const migratedAt = Date.now(); - entries.forEach(url => visited.set(normalizeUrl(url), migratedAt)); + entries.forEach(url => { + visited.set(normalizeUrl(url), migratedAt); + noteLocalChange('v', normalizeUrl(url), migratedAt); + }); pruneVisited(); if (!(await persistState())) return; } @@ -1066,6 +1191,8 @@ .forEach(([url]) => visited.delete(url)); } + pruneOutbox(now); + // Tombstones only need to outlive the stale snapshots they protect against. for (const [key, at] of tombstones) { if (now - at > TOMBSTONE_TTL_MS) tombstones.delete(key); @@ -1272,28 +1399,170 @@ return hash.toString(16).padStart(8, '0'); } - function parseSyncedThrough(value) { - const remote = typeof value?.remote === 'string' ? value.remote.slice(0, 40) : ''; - const cursor = clamp(value?.cursor, 0, Number.MAX_SAFE_INTEGER, 0); - return remote && cursor ? { remote, cursor } : { remote: '', cursor: 0 }; + function parseSyncPosition(value) { + if (!value || typeof value !== 'object') return null; + const cursor = clamp(value.cursor, 0, Number.MAX_SAFE_INTEGER, 0); + const gen = parseTimestamp(value.gen); + const epoch = typeof value.epoch === 'string' ? value.epoch.slice(0, 64) : ''; + return cursor || gen || epoch ? { epoch, gen, cursor } : null; + } + + function parseSyncPositions(value) { + const result = {}; + if (!value || typeof value !== 'object' || Array.isArray(value)) return result; + // 6.11.1 kept a single position: `{ remote, cursor }`. + if (typeof value.remote === 'string') { + const cursor = clamp(value.cursor, 0, Number.MAX_SAFE_INTEGER, 0); + if (value.remote && cursor) result[value.remote.slice(0, 40)] = { epoch: '', gen: 0, cursor }; + return result; + } + Object.entries(value).slice(0, 20).forEach(([tag, position]) => { + const parsed = parseSyncPosition(position); + if (tag && parsed) result[tag.slice(0, 40)] = parsed; + }); + return result; + } + + // A later start against a worker's store outranks an earlier one; within one start, the + // further position does. + function newerSyncPosition(a, b) { + if (!a) return b || null; + if (!b) return a; + if (a.gen !== b.gen) return a.gen > b.gen ? a : b; + return a.cursor >= b.cursor ? a : b; + } + + function sameSyncPosition(a, b) { + return Boolean(a && b) && a.gen === b.gen && a.cursor === b.cursor && a.epoch === b.epoch; } // Like the history id, only ever taken from this device's own storage, and only at the // points where a stored document is merged into this tab's: the merged history holds // everything both copies held, so it is complete as far as the further of the two. function adoptSyncedThrough(stored) { - const theirs = parseSyncedThrough(stored?.syncedThrough); - if (!theirs.remote) return; - if (theirs.remote === syncedThrough.remote) { - if (theirs.cursor > syncedThrough.cursor) syncedThrough = theirs; - } else if (syncedThrough.remote !== remoteTag(syncConfig.endpoint)) { - syncedThrough = theirs; + Object.entries(parseSyncPositions(stored?.syncedThrough)).forEach(([tag, position]) => { + syncedThrough[tag] = newerSyncPosition(syncedThrough[tag], position); + }); + } + + function parseOutbox(value) { + const result = new Map(); + if (!value || typeof value !== 'object' || Array.isArray(value)) return result; + Object.entries(value).forEach(([id, entry]) => { + if (!/^[vo]\|./.test(id) || !Array.isArray(entry)) return; + const at = parseTimestamp(entry[0]); + if (!at) return; + result.set(id, { at, sent: parseTimestamp(entry[1]), t: parseTimestamp(entry[2]) || at }); + }); + return result; + } + + function serializeOutbox() { + const result = {}; + for (const [id, entry] of outbox) result[id] = [entry.at, entry.sent, entry.t]; + return result; + } + + function mergeOutbox(source) { + let grew = false; + for (const [id, entry] of source) { + const held = outbox.get(id); + if (!held) { + outbox.set(id, { ...entry }); + grew = true; + continue; + } + if (entry.at > held.at || entry.sent > held.sent) { + outbox.set(id, { + at: Math.max(held.at, entry.at), + sent: Math.max(held.sent, entry.sent), + t: Math.max(held.t, entry.t), + }); + grew = true; + } + } + return grew; + } + + // Every change made on this device goes through here, so the worker hears about it + // whichever tab next syncs, however stale the view that tab has of storage. + function noteLocalChange(kind, key, at) { + if (!key || !at) return; + const id = `${kind}|${key}`; + const held = outbox.get(id); + if (held && held.at >= at) return; + outbox.set(id, { at, sent: held?.sent || 0, t: Date.now() }); + } + + // Everything this device holds, as changes the worker has not heard of yet: for a + // worker this device has never synced with, or one that has come back empty. + function noteEverythingUnsent() { + for (const [url, at] of visited) noteLocalChange('v', url, at); + for (const [url] of overrides) noteLocalChange('o', url, overrideTimes.get(url) || 0); + for (const [key, at] of tombstones) noteLocalChange(key.slice(0, 1), key.slice(2), at); + } + + // The change the worker should get for an outbox entry, from what this device holds + // now. Nothing, when the entry has since been pruned or cleared away here. + function outgoingChange(id) { + const kind = id.slice(0, 1); + const key = id.slice(2); + if (kind === 'v') { + const at = visited.get(key); + if (at) return { kind, key, at }; + } else if (overrides.has(key)) { + return { kind, key, at: overrideTimes.get(key) || 0, value: overrides.get(key) }; + } + const removed = tombstones.get(tombstoneKey(kind, key)); + return removed ? { kind, key, at: removed, deleted: 1 } : null; + } + + function outstandingChanges() { + const changes = []; + for (const [id, entry] of outbox) { + if (entry.at <= entry.sent) continue; + const change = outgoingChange(id); + if (change) { + changes.push({ id, mark: entry.at, change }); + } else { + // Gone here since: nothing to tell the worker. + outbox.set(id, { ...entry, sent: entry.at, t: Date.now() }); + } + } + return changes.sort((a, b) => a.change.at - b.change.at); + } + + function outstandingCount() { + let count = 0; + for (const entry of outbox.values()) if (entry.at > entry.sent) count += 1; + return count; + } + + function everythingHeld() { + return localChangesSince(0).map(change => ({ + id: `${change.kind}|${change.key}`, + mark: Math.max(change.at, outbox.get(`${change.kind}|${change.key}`)?.at || 0), + change, + })); + } + + function markSent(batch) { + const now = Date.now(); + batch.forEach(({ id, mark }) => { + const held = outbox.get(id); + if (held && mark > held.sent) outbox.set(id, { ...held, sent: mark, t: now }); + }); + } + + function pruneOutbox(now) { + for (const [id, entry] of outbox) { + if (entry.at <= entry.sent && now - entry.t > OUTBOX_KEEP_MS) outbox.delete(id); } } // Exports and the legacy worker get the history without this device's place in it. function portableState() { - const { syncedThrough: _local, ...state } = serializeState(); + const { syncedThrough: _position, outbox: _unsent, ...state } = serializeState(); return state; } @@ -1407,7 +1676,21 @@ }); } + function withTimeout(promise, ms, message) { + let timer = 0; + const expiry = new Promise((_resolve, reject) => { + timer = window.setTimeout(() => reject(remoteFailure(message)), ms); + }); + return Promise.race([promise, expiry]).finally(() => window.clearTimeout(timer)); + } + + // Each request is bounded on the page's side as well: a fetch has no timeout of its + // own, and a manager's request can be dropped without either callback firing. function requestRemote(payload) { + return withTimeout(sendRemote(payload), REMOTE_TIMEOUT_MS + 5000, 'the worker timed out'); + } + + function sendRemote(payload) { const url = syncConfig.endpoint; const headers = { 'Content-Type': 'application/json', @@ -1431,54 +1714,9 @@ }); } - // How far this device's pushes have got. Only meaningful against the worker it was - // recorded from, so pointing the panel elsewhere reads it as zero. - function pushHorizon() { - return syncConfig.remoteKey === syncConfig.endpoint ? syncConfig.pushedAt : 0; - } - - // The horizon is a wall-clock high-water mark: a sync declares "everything stamped - // before now is upstream". That claim is only true of entries the syncing tab could - // see, and an entry can reach this device stamped in the past—another tab recorded it - // while this one was not looking, a click was replayed from the previous page, a - // backup was restored. Such an entry is behind the horizon the moment it arrives and - // would never be offered again, which is how a visit ends up on one device for good. - // So the horizon is pulled back to it, and the next sync rescans from there. The cost - // is re-offering a few entries the worker already has, which it discards on arrival. - function lowerPushHorizon(at) { - const time = parseTimestamp(at); - if (!syncConfigured() || !time) return; - const horizon = pushHorizon(); - if (!horizon || time >= horizon) return; - syncConfig = { ...syncConfig, pushedAt: time }; - // Persisted through a read-modify-write that can only ever lower the stored - // value: another tab may have moved it on legitimately in the meantime, and this - // must not become a way to undo that. - pendingHorizonWrite = pendingHorizonWrite.then(() => withSyncLock(async () => { - // A sync that ran in the meantime has pushed everything from here anyway, and - // its horizon is then the authority: dragging it back would only cost a round - // trip. Anything still holding this value has not been superseded. - if (syncConfig.pushedAt !== time) return; - const stored = await readSyncConfig(); - if (stored.remoteKey !== syncConfig.endpoint || !stored.pushedAt) return; - if (stored.pushedAt <= time) return; - await writeSyncConfig({ ...stored, pushedAt: time }); - })).catch(() => {}); - } - - // Called after every merge of a document this device stores itself, so that whatever - // the merge took on from another tab is pushed rather than silently swallowed. Remote - // rows are excluded: they came from the worker, so re-offering them is pure waste. - function notePushBacklog() { - const at = lastMergeBackdatedTo; - lastMergeBackdatedTo = 0; - if (at && !applyingRemoteState) lowerPushHorizon(at); - } - - // The entries this device has touched since its last successful push. Everything - // recorded locally is stamped with the current time, so a single high-water mark is - // enough to find them—and entries that arrived from the worker are always older than - // it, so they are never sent straight back. + // The entries stamped at or after `since`. Before 6.12.0 a device tracked what it had + // pushed as one such moment, kept in the sync configuration; a history saved by then + // seeds its outbox from it once. function localChangesSince(since) { const changes = []; for (const [url, at] of visited) { @@ -1547,10 +1785,7 @@ // Syncs are queued for the same reason writes are: two overlapping merge cycles would // each build on a snapshot the other has already moved past. function queueSync(options = {}) { - const run = async () => { - await pendingHorizonWrite; - return withSyncLock(() => runSync(options)); - }; + const run = () => withSyncLock(() => runSync(options)); remoteQueue = remoteQueue.then(run, run).catch(error => { lastRemoteError = String(error?.message || 'sync failed'); if (options.manual) showToast(`Sync failed: ${lastRemoteError}`, true); @@ -1568,12 +1803,20 @@ let checkSyncOwnership = checkPageOpen; async function withSyncLock(action) { if (pageClosed) return false; + // However an exchange gets stuck, it gives the lock up in the end, so one hung tab + // cannot stop every other tab from syncing. Its requests time out well before. + const bounded = () => withTimeout(action(), SYNC_HOLD_MS, 'the sync took too long'); if (navigator.locks?.request) { return navigator.locks.request('javstore-cloud-sync-v1', async () => { if (pageClosed) return false; - return action(); + return bounded(); }); } + // A lease kept in storage only excludes anything where tabs read each other's + // writes. Where each tab reads its own copy (AdGuard), every tab would see its own + // claim and win, and a lease taken before a tab loaded would hold it up for the + // whole lease; two tabs syncing at once cost a duplicate request and nothing more. + if (!liveStorage) return bounded(); const owner = makeHistoryId(); const pause = ms => new Promise(resolve => window.setTimeout(resolve, ms)); const deadline = Date.now() + SYNC_LEASE_MS + 5000; @@ -1598,7 +1841,7 @@ await storeValue(SYNC_LEASE_KEY, { owner, expiresAt: Date.now() + SYNC_LEASE_MS }); }; try { - return await action(); + return await bounded(); } finally { checkSyncOwnership = checkPageOpen; if ((await loadValue(SYNC_LEASE_KEY))?.owner === owner) { @@ -1611,7 +1854,7 @@ } async function runSync({ manual = false, reason = 'push' } = {}) { - // Every entry point uses current shared progress, not the tab's boot snapshot. + // Every entry point uses the current sync settings, not the tab's boot snapshot. syncConfig = await readSyncConfig(); if (!syncConfigured() || pageClosed) return false; lastRemoteAttemptAt = Date.now(); @@ -1621,15 +1864,8 @@ // Let any local write finish first so what gets pushed is the current state. await writeQueue.catch(() => false); - // Read before the horizon is taken, so nothing recorded between the two can - // fall into the gap. - const horizon = Date.now(); - // Sibling tabs keep their own copy of the document and only write it to - // storage; nothing tells this one that they have. Pushing from memory alone is - // therefore pushing a stale document, and since the horizon afterwards claims - // everything older than now is upstream, a visit another tab recorded is - // stranded on this device for good. So the stored document is merged in first: - // what this sync is about to declare pushed, it has now actually seen. + // Whatever the stored document and the journals hold is merged in first: what + // other tabs recorded and still have to send, and how far they have got. const snapshot = await readStorageSnapshot(); const stored = combineSnapshot(snapshot); const needsCheckpoint = journalsAhead(snapshot); @@ -1639,43 +1875,50 @@ applyStoredState(stored); pruneVisited(); - // The cursor and the high-water mark only mean anything against the worker they - // were recorded from, so pointing the panel at a different one starts over. - const sameRemote = syncConfig.remoteKey === syncConfig.endpoint; - // And the cursor only means anything against the history it was recorded into. - // A history that has lost it pulls everything again; the worker settles each row - // on its timestamp, so rows the device still holds cost bandwidth and nothing else. - const sameHistory = Boolean(historyId) && syncConfig.historyId === historyId; - // Where this history is complete up to, as the history itself records it. A - // document saved before it carried that falls back to the configuration's - // cursor when the two name the same history. - const stamped = syncedThrough.remote === remoteTag(syncConfig.endpoint) ? syncedThrough.cursor : 0; - const held = stamped || (sameRemote && sameHistory && !syncedThrough.remote ? syncConfig.cursor : 0); - // Only a history with no place in the sync at all, on a device that has synced - // before, has actually been lost. One that is merely behind—an older copy written - // back by a tab that had been open a long time—picks up from where it is, quietly. - const recovering = sameRemote && syncConfig.cursor > 0 && !held; - const heldBefore = visited.size; - let cursor = held; - const pushedAt = sameRemote ? syncConfig.pushedAt : 0; + // Only a tab whose view of storage is recent writes the main document and the + // sync session; an older one still syncs, and its journal carries the result. + const shared = sharedWritesAllowed(); + const tag = remoteTag(syncConfig.endpoint); + // Where this history is complete up to on this worker, as the history itself + // records it. None: never synced with this worker, or the history was lost. + let position = syncedThrough[tag] || null; + const hadPosition = Boolean(position); const session = await loadValue(SYNC_SESSION_KEY); + // Whether this device has synced with this worker before; a history with no + // place in it then is one that was lost, which is worth saying. + const syncedBefore = session?.endpoint === syncConfig.endpoint + || (syncConfig.remoteKey === syncConfig.endpoint && syncConfig.cursor > 0); + // A history saved before 6.12.0 lists nothing as unsent: what it had not + // pushed is what the configuration's old horizon had not passed. + if (position && !legacyOutboxSeeded && snapshot.main && snapshot.main.outbox === undefined) { + legacyOutboxSeeded = true; + const since = syncConfig.remoteKey === syncConfig.endpoint ? syncConfig.pushedAt : 0; + localChangesSince(since).forEach(change => noteLocalChange(change.kind, change.key, change.at)); + } + const heldBefore = visited.size; + let cursor = position?.cursor || 0; const now = Date.now(); // The session only vouches for a history that still reaches as far as the // download it records; one written back from an older copy has to catch up. - // Sessions recorded before they carried a cursor vouch for nothing. - const currentSession = session?.endpoint === syncConfig.endpoint - && sameRemote && Number.isFinite(session?.cursor) && held >= session.cursor + // A worker that only exchanges whole documents keeps no place to resume from. + const currentSession = position && position.epoch !== 'document' + && session?.endpoint === syncConfig.endpoint + && Number.isFinite(session?.cursor) && session.gen === position.gen && cursor >= session.cursor && session?.completedAt > 0 && now - session.activeAt < SYNC_SESSION_IDLE_MS; if (!manual && currentSession) { - const pending = localChangesSince(pushedAt).length - || session.uploadedMeta !== metadataSignature(); + const pending = outstandingCount() || session.uploadedMeta !== metadataSignature(); if (reason === 'startup' && pending) scheduleRemotePush(); - const fresh = reason === 'startup' + const sinceDownload = now - session.completedAt; + const interval = syncConfig.intervalMinutes * 60000; + // Navigation inside a session reuses its download, but not for longer than + // the polling interval, or a device that keeps browsing never sees what + // other devices recorded. + const fresh = (reason === 'startup' && sinceDownload < interval) || (reason === 'push' && !pending) - || (reason === 'focus' && now - session.completedAt < REMOTE_REFRESH_MIN_GAP_MS) - || (reason === 'poll' && now - session.completedAt < syncConfig.intervalMinutes * 60000); + || (reason === 'focus' && sinceDownload < REMOTE_REFRESH_MIN_GAP_MS) + || (reason === 'poll' && sinceDownload < interval); if (fresh) { - await storeValue(SYNC_SESSION_KEY, { ...session, activeAt: now }); + if (shared) await storeValue(SYNC_SESSION_KEY, { ...session, activeAt: now }); lastRemoteSyncAt = session.completedAt; remoteWorkerVersion = session.workerVersion || ''; lastRemoteError = ''; @@ -1683,24 +1926,28 @@ } } - let outgoing = localChangesSince(pushedAt); + // Everything this device holds goes up to a worker it has no place with yet; + // otherwise what is still outstanding. + let outgoing = position ? outstandingChanges() : everythingHeld(); let sent = 0; let changed = false; let absorbed = false; let legacy = false; let downloadComplete = false; let uploadedMeta = ''; + let restarted = false; for (let page = 0; page < MAX_SYNC_PAGES; page += 1) { const batch = outgoing.slice(0, REMOTE_PAGE_ROWS); await checkSyncOwnership(); uploadedMeta = metadataSignature(); + const sentCursor = cursor; const answer = await requestRemote({ client: SCRIPT_VERSION, cursor, limit: REMOTE_PAGE_ROWS, meta: localMeta(), - changes: batch, + changes: batch.map(item => item.change), }); if (!answer || typeof answer !== 'object') { throw new Error('the worker returned nothing'); @@ -1717,79 +1964,77 @@ if (applyStoredState(document)) changed = true; if (lastMergeAdopted) absorbed = true; lowerJournalSince(lastMergeOldest); + markSent(outgoing); + // Nothing about this store counts towards a worker that speaks cursors, + // should one answer at this address again. + position = { epoch: 'document', gen: Date.now(), cursor: 0 }; + cursor = 0; break; } + const epoch = typeof answer.epoch === 'string' ? answer.epoch.slice(0, 64) : ''; + const answered = Math.max(0, Number(answer.cursor) || 0); + // Not the store this device counted against: the worker came back empty, or a + // different store answers at this address. Start over against it, and send it + // everything, since it may hold none of it. + const reset = Boolean(position) + && ((position.epoch && position.epoch !== epoch) || answered < sentCursor); + if (reset && !restarted) { + restarted = true; + position = { epoch, gen: Date.now(), cursor: 0 }; + cursor = 0; + outgoing = everythingHeld(); + continue; + } + if (!position) position = { epoch, gen: Date.now(), cursor: 0 }; + else if (!position.epoch && epoch) position = { ...position, epoch }; + + markSent(batch); outgoing = outgoing.slice(batch.length); sent += batch.length; - cursor = Math.max(0, Number(answer.cursor) || 0); + cursor = answered; applyingRemoteState = true; if (applyStoredState(documentFromRows(answer.changes, answer.meta))) changed = true; - // Pulled rows exist nowhere else on this device until they are saved. + // Pulled rows go into the journal as well: they are what this page's place + // in the sync vouches for, and the journal carries that place. lowerJournalSince(lastMergeOldest); // A batch can carry nothing but protective metadata—a tombstone, a newer // `resetAt`, `prunedBefore` or `settingsUpdatedAt`—which removes nothing on - // screen and so leaves `changed` false. It still has to be saved: the cursor - // moves past it either way, and losing it lets a stale sibling tab merge an - // already-deleted visit back in. + // screen and so leaves `changed` false. It still has to be saved. if (lastMergeAdopted) absorbed = true; applyingRemoteState = false; downloadComplete = !answer.more; if (!outgoing.length && downloadComplete) break; } - // What was pulled has to be in storage before the cursor is allowed past it. - // The cursor is the durable record of what this device has already seen, so if - // it advanced first and the state write were then interrupted by a navigation or - // rejected outright, the next sync would ask only for rows after that cursor and - // the merged ones would never be offered again. - // A history without an id gets one now, and it has to be in storage before the - // cursor is recorded against it, for the same reason as the rows above. + // The rows and the place they reach go into storage in the same write. const naming = !legacy && !historyId; if (naming) historyId = makeHistoryId(); - // The rows and the place they reach go into storage in the same write. - const reached = legacy ? { remote: '', cursor: 0 } : parseSyncedThrough({ remote: remoteTag(syncConfig.endpoint), cursor }); - const advanced = reached.remote !== syncedThrough.remote || reached.cursor !== syncedThrough.cursor; - syncedThrough = reached; + syncedThrough[tag] = { ...position, cursor }; + // A sync that only sent changes moved nothing but this page's place and its + // list of what is sent; its journal records that, and the main document, which + // is far larger, is left as it is. + const rewrite = changed || absorbed || naming || needsCheckpoint; applyingRemoteState = true; - const saved = (changed || absorbed || naming || needsCheckpoint || advanced) ? await persistState() : true; + const saved = await persistState({ journalOnly: !rewrite }); applyingRemoteState = false; if (!saved) throw new Error('the merged history could not be saved'); - // Only recorded after the exchange succeeded: a failed sync has to send the same - // entries again rather than assume the worker took them. - // The horizon may only cover what actually went up. `outgoing` is still - // holding entries when the page budget ran out before they could be sent, and - // moving the horizon past them would retire them unsent. - const unsent = outgoing.reduce((oldest, change) => ( - oldest ? Math.min(oldest, change.at) : change.at - ), 0); - // And something backdated may have arrived after this sync chose what to send, - // which pulled the horizon below where the exchange started. That entry was - // never offered, so the sync cannot claim it. - const arrived = syncConfig.pushedAt < pushedAt ? syncConfig.pushedAt : horizon; - await checkSyncOwnership(); - const progressSaved = await writeSyncConfig({ - ...syncConfig, - cursor: legacy ? 0 : cursor, - pushedAt: legacy ? 0 : Math.min(horizon, unsent || horizon, arrived), - remoteKey: legacy ? '' : syncConfig.endpoint, - historyId: legacy ? '' : historyId, - }); - - if (!progressSaved) throw new Error('sync progress could not be saved'); // Only completed downloads suppress subsequent page-load requests. If the // page budget was exhausted, the next page must continue from the saved cursor. - if (downloadComplete) { + if (downloadComplete && shared) { + await checkSyncOwnership(); await storeValue(SYNC_SESSION_KEY, { - endpoint: syncConfig.endpoint, historyId, cursor, uploadedMeta, workerVersion: remoteWorkerVersion, - completedAt: Date.now(), activeAt: Date.now(), + endpoint: syncConfig.endpoint, gen: position.gen, cursor, uploadedMeta, + workerVersion: remoteWorkerVersion, completedAt: Date.now(), activeAt: Date.now(), }); } lastRemoteSyncAt = Date.now(); lastRemoteError = ''; const restored = visited.size - heldBefore; - if (recovering && restored > 0) { + // Only a history that had lost most of what the worker holds says so; one that + // was merely behind catches up without a word. + if (syncedBefore && !hadPosition && !legacy && restored > 0 && restored > heldBefore) { console.info(`[JVS] Local history had lost its place in the sync; ${restored} visits were downloaded again.`); showToast(`Restored ${restored} visited URLs from the worker.`); } else if (manual) { @@ -1833,6 +2078,9 @@ function syncOnFocus() { if (!syncConfigured()) return; if (Date.now() - lastRemoteAttemptAt < REMOTE_REFRESH_MIN_GAP_MS) return; + // Taken now rather than when the sync starts, so `focus` and `visibilitychange` + // arriving together for one activation queue one sync, not two. + lastRemoteAttemptAt = Date.now(); queueSync({ reason: 'focus' }); } @@ -2217,6 +2465,7 @@ if (shouldVisit) { visited.set(url, now); tombstones.delete(tombstoneKey('v', url)); + noteLocalChange('v', url, now); rememberPendingVisit(url, now); } else { forgetVisited(url, now); @@ -2242,6 +2491,7 @@ overrides.set(url, current ? 'block' : 'allow'); overrideTimes.set(url, now); tombstones.delete(tombstoneKey('o', url)); + noteLocalChange('o', url, now); } else { forgetOverride(url, now); } @@ -2586,14 +2836,25 @@ // entry by entry instead, which travels without taking the restored history // with it and leaves its timestamps intact. for (const url of visited.keys()) { - if (!restored.has(url)) tombstones.set(tombstoneKey('v', url), now); + if (!restored.has(url)) forgetVisited(url, now); } for (const url of overrides.keys()) { - if (!restoredOverrides.has(url)) tombstones.set(tombstoneKey('o', url), now); + if (!restoredOverrides.has(url)) forgetOverride(url, now); } // A restored entry outranks any removal this device was still carrying for it. - for (const url of restored.keys()) tombstones.delete(tombstoneKey('v', url)); - for (const url of restoredOverrides.keys()) tombstones.delete(tombstoneKey('o', url)); + // A removal newer than the backup's own timestamp would win every later merge + // and take the entry straight back out, so such an entry is stamped just past + // it; every other entry keeps the backup's timestamp. + for (const [url, at] of restored) { + const removedAt = tombstoneTime('v', url); + if (removedAt >= at) restored.set(url, removedAt + 1); + tombstones.delete(tombstoneKey('v', url)); + } + for (const url of restoredOverrides.keys()) { + const removedAt = tombstoneTime('o', url); + if (removedAt >= (restoredOverrideTimes.get(url) || 0)) restoredOverrideTimes.set(url, removedAt + 1); + tombstones.delete(tombstoneKey('o', url)); + } stampSettings(sanitizeSettings(parsed.settings), { all: true }); visited = restored; @@ -2612,11 +2873,12 @@ } } pruneVisited(); - // Everything restored is stamped in the past, so without this the push horizon - // would already be sitting in front of it and none of it would ever go up. + // Everything restored goes up, and the journal carries it until the main + // document does. + for (const [url, at] of visited) noteLocalChange('v', url, at); + for (const [url] of overrides) noteLocalChange('o', url, overrideTimes.get(url) || 0); const oldest = [...visited.values(), ...overrideTimes.values()] .reduce((least, at) => (least ? Math.min(least, at) : at), 0); - lowerPushHorizon(oldest); lowerJournalSince(oldest); persistState(); setRootState(); @@ -2912,7 +3174,6 @@ function applyStoredState(stored) { const merged = mergeStoredState(stored, { adoptSettings: true }); - notePushBacklog(); if (!merged) return false; pruneVisited(); setRootState(); @@ -3025,7 +3286,8 @@ resetAt = parseTimestamp(initialState?.resetAt); prunedBefore = parseTimestamp(initialState?.prunedBefore); historyId = parseHistoryId(initialState?.historyId); - syncedThrough = parseSyncedThrough(initialState?.syncedThrough); + syncedThrough = parseSyncPositions(initialState?.syncedThrough); + outbox = parseOutbox(initialState?.outbox); lastKnownUpdatedAt = parseTimestamp(initialState?.updatedAt); settingsUpdatedAt = parseTimestamp(initialState?.settingsUpdatedAt) || lastKnownUpdatedAt; settingTimes = documentSettingTimes(initialState); @@ -3061,6 +3323,7 @@ const [config, journals] = await Promise.all([configPromise, journalsPromise]); syncConfig = config; const snapshot = { main, journals }; + loadedSnapshot = snapshot; initializeState(combineSnapshot(snapshot)); const replayed = replayPendingVisits(); pruneVisited(); @@ -3068,6 +3331,7 @@ processAllCards(); await migrateLegacyHistory(); if (replayed || journalsAhead(snapshot)) persistState(); + retireJournals(Date.now()); setRootState(); if (document.readyState === 'loading') { diff --git a/tests/harness.mjs b/tests/harness.mjs index 618af66..bdf7b2b 100644 --- a/tests/harness.mjs +++ b/tests/harness.mjs @@ -365,3 +365,17 @@ export async function waitFor(check, { timeout = 9000, interval = 20 } = {}) { export function visitedUrls(store) { return Object.keys(store.state()?.visited || {}).sort(); } + +// How far into the worker's history this device's stored history reaches: the furthest +// position recorded in the main document or any journal. Since 6.12.0 the position lives +// with the rows it describes, not in the sync configuration. +export function syncPosition(store) { + const docs = [store.state(), ...store.journals().map(key => store.read(key))].filter(Boolean); + let furthest = 0; + for (const doc of docs) { + const positions = doc.syncedThrough && typeof doc.syncedThrough === 'object' ? doc.syncedThrough : {}; + const values = typeof positions.remote === 'string' ? [positions] : Object.values(positions); + for (const position of values) furthest = Math.max(furthest, Number(position?.cursor) || 0); + } + return furthest; +} diff --git a/tests/persistence.test.mjs b/tests/persistence.test.mjs index 09a82eb..fa2836a 100644 --- a/tests/persistence.test.mjs +++ b/tests/persistence.test.mjs @@ -336,7 +336,10 @@ test('a visit a racing write dropped from the main document comes back from the assert.deepEqual(visitedUrls(store), ['https://javstore.net/a.html']); }); -test('a journal is retired once it has sat past its TTL, and not before', async () => { +// A journal key is written for a few minutes and then left final. It is removed once a tab +// that loaded after it was final finds it covered by the main document it loaded, and +// enough time has passed that no tab lacking it may still write the main document. +test('a journal is retired once it has been final past its TTL, and not before', async () => { const store = makeStore(); const first = await openTab(store, { html: listing() }); clickCard(first.window, '/a.html'); @@ -349,12 +352,17 @@ test('a journal is retired once it has sat past its TTL, and not before', async await settle(); assert.ok(store.journals().includes(firstJournal), 'a recent journal is kept'); - const aged = store.read(firstJournal); - aged.writtenAt -= 11 * 60000; - store.write(firstJournal, aged); - clickCard(second.window, '/c.html'); - await settle(); - assert.ok(!store.journals().includes(firstJournal), 'the aged journal is retired'); + // The same journal, started 20 minutes ago: final for 15, and covered by the main + // document a tab loading now finds. + const createdAt = Date.now() - 20 * 60000; + const aged = firstJournal.replace(/_v2_[0-9a-z]+_/, `_v2_${createdAt.toString(36)}_`); + store.data.set(aged, store.data.get(firstJournal)); + store.data.delete(firstJournal); + const third = await openTab(store, { html: listing() }); + clickCard(third.window, '/c.html'); + await settle(); + assert.ok(!store.journals().includes(aged), 'the aged journal is retired'); + assert.ok(store.journals().length > 0, 'recent journals stay'); assert.deepEqual(visitedUrls(store), CARDS.map(([href]) => `https://javstore.net${href}`)); }); diff --git a/tests/review-fix.test.mjs b/tests/review-fix.test.mjs index 4ff9053..c215de2 100644 --- a/tests/review-fix.test.mjs +++ b/tests/review-fix.test.mjs @@ -2,7 +2,7 @@ import test, { afterEach } from 'node:test'; import assert from 'node:assert/strict'; import { makeStore, makeRemote, enableSync, openTab, closeTabs, settle, waitFor, - listingHtml, pressOnCard, visitedUrls, + listingHtml, pressOnCard, visitedUrls, syncPosition, } from './harness.mjs'; const SESSION_KEY = 'javstore_sync_session_v1'; @@ -54,7 +54,7 @@ function dropJournals(store) { // is reloaded, each of their writes leaves a document with no stamp and the other id, // and 6.11.1 still reads that as a lost history: a full download and the toast, on every // page that follows such a write: the exact symptom the release is meant to end. -test('a stamp-less document written by a still-open 6.11.0 tab does not trigger a full download and "Restored" on every page', async () => { +test('a stamp-less document written by a still-open 6.11.0 tab catches up without "Restored" on every page', async () => { const remote = seededRemote(300); const store = makeStore({ snapshotPerTab: true }); await restored(remote, store); @@ -78,7 +78,10 @@ test('a stamp-less document written by a still-open 6.11.0 tab does not trigger await waitFor(() => store.read(SESSION_KEY).completedAt > previous); await settle(); assert.doesNotMatch(page.shadow().textContent, /Restored \d+ visited URLs/, `round ${round}`); - assert.notEqual(remote.requests[requests].body.cursor, 0, `round ${round}: full download`); + // A copy written without a position says nothing about how far it reaches, so it + // is downloaded again in full, quietly—the price while tabs from before the update + // stay open—and nothing the copy lacks is left out. + assert.ok(visitedUrls(store).includes('https://javstore.net/elsewhere-0.html'), `round ${round}`); } }); @@ -132,7 +135,7 @@ test('a visit that is only pushed up saves the history document once', async () const requests = remote.requests.length; pressOnCard(tab.window, '/new.html', 'v'); await waitFor(() => remote.requests.length > requests, { timeout: 9000 }); - await waitFor(() => store.read(CONFIG_KEY).cursor === 2); + await waitFor(() => syncPosition(store) === 2); await settle(80); assert.equal(saves, 1, `history document saved ${saves} times for one pushed visit`); }); @@ -179,7 +182,6 @@ test('a document-only exchange forgets the stamp, so a worker that speaks cursor await openTab(store, { html: listing(), remote: legacy }); await waitFor(() => store.read(SESSION_KEY)?.completedAt); await settle(); - assert.equal(store.read(CONFIG_KEY).remoteKey, ''); const previous = store.read(SESSION_KEY).completedAt; await openTab(store, { html: listing(), remote: v3 }); diff --git a/tests/review-lifecycle.test.mjs b/tests/review-lifecycle.test.mjs index c9f1178..a21ea28 100644 --- a/tests/review-lifecycle.test.mjs +++ b/tests/review-lifecycle.test.mjs @@ -220,7 +220,7 @@ test('"Clear visited history" is not undone by journals of other pages', async ( // save then "retires" B's journal: it checks writtenAt against its own snapshot, which // still holds B's old journal, so it deletes the live key. A's main-document write has // already dropped what B recorded after A loaded, so nothing in storage holds it. -test('a long-open tab does not delete journals other live tabs have rewritten since it loaded', { todo: 'retireJournals trusts a stale snapshot (data loss)' }, async () => { +test('a long-open tab does not delete journals other live tabs have rewritten since it loaded', async () => { const store = makeStore({ snapshotPerTab: true }); const b = await openTab(store, { html: listing() }); clickCard(b.window, '/a.html'); @@ -246,7 +246,7 @@ test('a long-open tab does not delete journals other live tabs have rewritten si // A fresh tab later merges C's journal into the main document and, a TTL on, retires it. // Then A, which loaded before C existed, saves: its write is built on the snapshot it took // at load and drops C's visit. No journal is left to bring it back. -test('a long-open tab\'s save does not drop visits other tabs recorded after it loaded', { todo: 'stale tabs write the whole document (data loss when sync is off or the push failed)' }, async () => { +test('a long-open tab\'s save does not drop visits other tabs recorded after it loaded', async () => { const store = makeStore({ snapshotPerTab: true }); const a = await openTab(store, { html: listing() }); // open for hours const c = await openTab(store, { html: detailHtml, url: 'https://javstore.net/700003-Item-pn.html' }); @@ -271,33 +271,42 @@ test('a long-open tab\'s save does not drop visits other tabs recorded after it }); // A V-unmark is a per-URL tombstone, and a journal carries it only while it is newer than -// the journal's horizon (resetAt, by contrast, rides in every journal, so a clear -// survives in whichever fresh journal is newest). Once the unmarking tab's journal has -// been retired, a tab that loaded before the unmark writes the visit back without the -// tombstone, and every later page shows it as visited again. -test('a long-open tab cannot undo a V-unmark once the unmarking journal is retired', { todo: 'stale tabs write the whole document (unmarked visits come back)' }, async () => { +// the journal's horizon. Once the unmarking tab's journal has been retired, a tab that +// loaded before the unmark must not write the visit back over the main document. One +// clock for every tab, moved forward, stands in for the hours the old tab stays open. +test('a long-open tab cannot undo a V-unmark once the unmarking journal is retired', async () => { const store = makeStore({ snapshotPerTab: true }); - const seed = await openTab(store, { html: listing() }); + const clock = { offset: 0 }; + const open = () => openTab(store, { html: listing(), beforeRun(window) { + const realNow = window.Date.now.bind(window.Date); + window.Date.now = () => realNow() + clock.offset; + } }); + const seed = await open(); clickCard(seed.window, '/a.html'); await waitFor(() => visitedUrls(store).length === 1); closeTab(seed); - const a = await openTab(store, { html: listing() }); // open for hours - const unmarker = await openTab(store, { html: listing() }); + const a = await open(); // open for hours + const unmarker = await open(); pressOnCard(unmarker.window, '/a.html', 'v'); await waitFor(() => visitedUrls(store).length === 0); await settle(); closeTab(unmarker); - store.journals().forEach(key => age(store, key)); - const d = await openTab(store, { html: listing() }); - clickCard(d.window, '/c.html'); // a fresh save that retires the aged journals - await waitFor(() => visitedUrls(store).includes(abs('/c.html')) && store.journals().length === 1); + const unmarkerJournals = store.journals(); + + clock.offset = 17 * 60000; + const d = await open(); + clickCard(d.window, '/c.html'); // a fresh save; the unmarking journal is retired + await waitFor(() => visitedUrls(store).includes(abs('/c.html')) + && unmarkerJournals.every(key => !store.journals().includes(key))); await settle(); clickCard(a.window, '/b.html'); // any save by the long-open tab await settle(80); - const e = await openTab(store, { html: listing() }); + const e = await open(); + await settle(); assert.ok(!card(e.window, '/a.html').classList.contains('jvs-visited'), 'the unmarked visit stays unmarked'); + assert.ok(card(e.window, '/b.html').classList.contains('jvs-visited'), 'the old tab\'s own visit is kept'); }); // The sync lock is held for the whole exchange. GM_xmlhttpRequest has a 20 s timeout, but @@ -306,7 +315,7 @@ test('a long-open tab cannot undo a V-unmark once the unmarking journal is retir // every other tab's sync queues behind it. Timers in the hung tab are sped up 100x, so a // page-side timeout of REMOTE_TIMEOUT_MS would release the lock within a fraction of a // second here. -test('a request that never settles in one tab does not block every other tab\'s sync', { todo: 'fetchRemote has no timeout; the lock is held indefinitely (sync stalls on all tabs)' }, async () => { +test('a request that never settles in one tab does not block every other tab\'s sync', async () => { const store = makeStore(); const remote = makeRemote(); enableSync(store, remote); @@ -371,7 +380,7 @@ test('measure: activating long-open tabs on a snapshot engine with sync on', asy // lastRemoteAttemptAt is set only once runSync gets the lock, after both handlers ran. // The second one finds the session fresh and only reads storage and rewrites the session, // but that is still two full-document reads on top of the two refreshFromStorage reads. -test('measure: one tab activation runs two syncs (visibilitychange and focus)', async t => { +test('measure: one tab activation runs one sync (visibilitychange and focus)', async t => { const store = makeStore(); const remote = makeRemote(); enableSync(store, remote); @@ -390,8 +399,7 @@ test('measure: one tab activation runs two syncs (visibilitychange and focus)', await settle(80); const cost = m.since(before); t.diagnostic(`activation with sync on: ${describe(cost)}, session writes ${cost.sessionWrites}, requests ${remote.requests.length - 1}`); - assert.equal(cost.sessionWrites, 2, 'both handlers queue a sync before either records its attempt'); - assert.ok(cost.mainReads >= 4, 'two refreshes plus two sync snapshots read the document'); + assert.equal(cost.sessionWrites, 1, 'the second handler finds the first one\'s attempt'); }); // After the page comes back from the back/forward cache its timers are restarted and the diff --git a/tests/review-persistence.test.mjs b/tests/review-persistence.test.mjs index f37daf7..e4488dc 100644 --- a/tests/review-persistence.test.mjs +++ b/tests/review-persistence.test.mjs @@ -70,15 +70,17 @@ test('review: a long-open tab does not delete a journal its owner rewrote after // Loaded after the owner's first save, so its snapshot holds that copy of the journal. const stale = await open(); clock.offset = TTL_PLUS; - // Only the owner records b (a manual mark; a click whose item tab never loaded). + // Only the owner records b (a manual mark; a click whose item tab never loaded). The + // owner is itself long-open by now, so b goes to its journal, which has moved on to a + // new key that carries everything the old one did. pressOnCard(owner.window, '/b.html', 'v'); await settle(); - assert.deepEqual(visitedUrls(store), [u('/a.html'), u('/b.html')]); + const holdsB = () => store.journals().some(key => store.read(key)?.visited?.[u('/b.html')]); + assert.ok(holdsB(), 'the owner journal carries b'); clickCard(stale.window, '/c.html'); await settle(); - assert.ok(store.journals().includes(ownerJournal), - 'the owner journal was rewritten after the stale tab loaded, so it is not stale'); + assert.ok(holdsB(), 'the stale tab left the owner journal alone'); const fresh = await open(); await settle(); diff --git a/tests/review-sync.test.mjs b/tests/review-sync.test.mjs index c63e44d..6068a54 100644 --- a/tests/review-sync.test.mjs +++ b/tests/review-sync.test.mjs @@ -185,20 +185,23 @@ test('F5 continuous navigation still picks up another device\'s visits within th // F6. Under per-tab snapshots the GM-storage lease cannot exclude anything: each tab only // ever reads back its own claim. Two tabs syncing at once both download and both push. -test('F6 the storage lease serializes two tabs under per-tab snapshots', async () => { +test('F6 two tabs without Web Locks under per-tab snapshots sync without waiting on each other', async () => { const remote = seededRemote(300); const store = makeStore({ snapshotPerTab: true, latency: 5 }); enableSync(store, remote); await Promise.all([0, 1].map(() => openTab(store, { html: listing(), remote, webLocks: false }))); await waitFor(() => store.read(SESSION_KEY)?.completedAt); await settle(300); - assert.equal(remote.requests.filter(request => request.body.cursor === 0).length, 1); + // Nothing in storage can keep the two apart when each reads its own copy; the cost is + // one duplicate download, never a wait on a lease neither can see released. + assert.ok(remote.requests.filter(request => request.body.cursor === 0).length <= 2); + assert.equal(visitedUrls(store).length, 300); }); // F7. Cost of a 50-tab context-menu burst on AdGuard. The batching in 6.11.0 relies on each // tab seeing the horizon the first upload recorded; under per-tab snapshots no item tab // sees another's upload, so every tab uploads (and downloads) on its own. -test('F7 a 50-tab burst under per-tab snapshots batches its uploads', async () => { +test('F7 a 50-tab burst under per-tab snapshots costs about one small request per visit', async () => { const remote = seededRemote(300); const store = makeStore({ snapshotPerTab: true }); await firstTab(remote, store); @@ -210,7 +213,10 @@ test('F7 a 50-tab burst under per-tab snapshots batches its uploads', async () = await new Promise(resolve => setTimeout(resolve, 4500)); const requests = remote.requests.length - before; const rows = remote.requests.slice(before).reduce((sum, request) => sum + (request.body.changes || []).length, 0); - assert.ok(requests <= 5, `50 visits cost ${requests} requests carrying ${rows} rows`); + // No item tab can see another's upload, so each sends its own visit: about one small + // request per visit, and never the whole history. + assert.ok(requests <= 55, `50 visits cost ${requests} requests carrying ${rows} rows`); + assert.ok(rows <= 150, `50 visits sent ${rows} rows`); }); // F8. A worker that comes back empty at the same URL (Durable Object recreated, storage diff --git a/tests/sync-session.test.mjs b/tests/sync-session.test.mjs index 0b99918..dc8881e 100644 --- a/tests/sync-session.test.mjs +++ b/tests/sync-session.test.mjs @@ -2,7 +2,7 @@ import test, { afterEach } from 'node:test'; import assert from 'node:assert/strict'; import { makeStore, makeRemote, enableSync, openTab, closeTabs, settle, waitFor, - listingHtml, detailHtml, card, pressOnCard, visitedUrls, + listingHtml, detailHtml, card, pressOnCard, visitedUrls, syncPosition, } from './harness.mjs'; const SESSION_KEY = 'javstore_sync_session_v1'; @@ -81,7 +81,7 @@ test('an old tab reads the cursor advanced by another tab before syncing', async const newer = await openTab(store, { html: listing(), remote }); remote.store.sync({ cursor: 0, changes: [{ kind: 'v', key: 'https://javstore.net/new.html', at: Date.now() }] }); await manualSync(newer, store); - const cursor = store.read(CONFIG_KEY).cursor; + const cursor = syncPosition(store); await manualSync(old, store); assert.equal(remote.requests.at(-1).body.cursor, cursor); assert.equal(remote.responses.at(-1).changes.length, 0); @@ -102,7 +102,7 @@ test('concurrent first tabs share one restoration', async () => { test('idle session expiry checks for new rows without discarding the cursor', async () => { const remote = seededRemote(1); const { store } = await restored(remote); - const cursor = store.read(CONFIG_KEY).cursor; + const cursor = syncPosition(store); const session = store.read(SESSION_KEY); store.write(SESSION_KEY, { ...session, activeAt: Date.now() - 31 * 60000 }); await openTab(store, { html: listing(), remote }); @@ -127,30 +127,39 @@ test('lost history bypasses the session marker, then the next page reuses the re assert.equal(remote.requests.length, 2); }); -test('a rejected progress write does not mark restoration complete', async () => { +test('a rejected history write does not mark restoration complete', async () => { const remote = seededRemote(1); const store = makeStore(); enableSync(store, remote); const set = store.set.bind(store); - store.set = async (key, value) => { if (key !== CONFIG_KEY) await set(key, value); }; + const history = key => key === STATE_KEY || key.startsWith('javstore_journal_'); + store.set = async (key, value) => { if (!history(key)) await set(key, value); }; const tab = await openTab(store, { html: listing(), remote }); - await waitFor(() => tab.shadow().textContent.includes('sync progress could not be saved')); + await waitFor(() => tab.shadow().textContent.includes('Could not save')); + await settle(200); assert.equal(store.read(SESSION_KEY), undefined); + assert.equal(syncPosition(store), 0); store.set = set; await manualSync(tab, store); - assert.ok(store.read(CONFIG_KEY).cursor > 0); + assert.ok(syncPosition(store) > 0); assert.ok(store.read(SESSION_KEY).completedAt); }); -test('fallback lease coordinates first tabs without Web Locks and can recover an expired owner', async () => { +// Without Web Locks, and without notice of other tabs' writes, a lease kept in storage +// cannot exclude anything (each tab reads back its own claim), and one left behind by a +// closed tab would hold the next ones up. Tabs then sync side by side: a few duplicate +// requests, and the same history. +test('without Web Locks, first tabs sync side by side without waiting on a stale lease', async () => { const remote = seededRemote(1); const store = makeStore({ latency: 2 }); enableSync(store, remote); - store.write('javstore_sync_lease_v1', { owner: 'closed-tab', expiresAt: Date.now() - 1 }); + store.write('javstore_sync_lease_v1', { owner: 'closed-tab', expiresAt: Date.now() + 90000 }); + const started = Date.now(); await Promise.all(Array.from({ length: 5 }, () => openTab(store, { html: listing(), remote, webLocks: false }))); await waitFor(() => store.read(SESSION_KEY)?.completedAt); await settle(600); - assert.equal(remote.requests.length, 1); + assert.ok(Date.now() - started < 8000, 'held up by a lease'); + assert.ok(remote.requests.length <= 5); assert.equal(visitedUrls(store).length, 1); }); @@ -161,7 +170,7 @@ test('an incomplete paged download resumes on the next page instead of marking t const store = makeStore(); enableSync(store, remote); await openTab(store, { html: listing(), remote }); - await waitFor(() => store.read(CONFIG_KEY).cursor === 12); + await waitFor(() => syncPosition(store) === 12); assert.equal(store.read(SESSION_KEY), undefined); await openTab(store, { html: listing(), remote }); await waitFor(() => store.read(SESSION_KEY)?.completedAt); @@ -207,7 +216,7 @@ test('an older history written back by a long-open tab catches up quietly from i const store = makeStore({ snapshotPerTab: true }); await restored(remote, store); const old = await openTab(store, { html: listing(), remote, url: 'https://javstore.net/page/1' }); - const reachedByOld = store.read(CONFIG_KEY).cursor; + const reachedByOld = syncPosition(store); // Another device adds visits, then this device loses its history while the old tab // stays open. The next page is a genuine recovery, and says so once. diff --git a/tests/sync.test.mjs b/tests/sync.test.mjs index 9f16195..4497ab0 100644 --- a/tests/sync.test.mjs +++ b/tests/sync.test.mjs @@ -4,7 +4,7 @@ import worker, { WORKER_VERSION } from '../worker/src/worker.mjs'; import fs from 'node:fs'; import { makeStore, makeRemote, enableSync, openTab, closeTabs, settle, waitFor, - listingHtml, card, clickCard, pressOnCard, visitedUrls, + listingHtml, card, clickCard, pressOnCard, visitedUrls, syncPosition, } from './harness.mjs'; const CARDS = [['/a.html', 'Alpha'], ['/b.html', 'Beta'], ['/c.html', 'Gamma']]; @@ -400,9 +400,12 @@ test('the cursor does not move past history that could not be saved', async () = // Long enough for the write to be retried and given up on. await settle(600); assert.equal(secondStore.state(), null, 'precondition: the history never landed'); + // A position may only be stored beside the rows it vouches for: here that is the + // journal, which the failed write of the main document did not take with it. + const journaled = secondStore.journals().map(key => secondStore.read(key)); assert.ok( - !secondStore.read('javstore_sync_config_v1').cursor, - 'a cursor recorded here would skip the rows that were lost', + !syncPosition(secondStore) || journaled.some(doc => doc.visited?.['https://javstore.net/a.html']), + 'a position recorded without its rows would skip the rows that were lost', ); // Reopening the device is the reload after the failed write: the rows have to come @@ -429,7 +432,7 @@ test('a merge that only brings back a tombstone is still saved', async () => { await waitFor(() => secondStore.state()); assert.deepEqual(secondStore.state().overrides, {}); assert.ok(secondStore.state().tombstones[`o|${url}`], 'the tombstone reached storage'); - assert.ok(secondStore.read('javstore_sync_config_v1').cursor > 0); + assert.ok(syncPosition(secondStore) > 0); }); test('pointing the panel at a different worker starts over', async () => { @@ -445,7 +448,7 @@ test('pointing the panel at a different worker starts over', async () => { clickCard(tab.window, '/a.html'); await settle(); await syncNow(tab); - assert.ok(store.read('javstore_sync_config_v1').cursor > 0); + assert.ok(syncPosition(store) > 0); const form = tab.shadow().querySelector('form.sync-form'); form.elements.syncEndpoint.value = second.endpoint; @@ -471,18 +474,14 @@ test('a device that lost its history but kept its cursor downloads it again', as await settle(); await syncNow(tab); assert.equal(remote.visited().length, 3); - assert.ok(store.read('javstore_sync_config_v1').cursor > 0); + assert.ok(syncPosition(store) > 0); closeTabs(); loseHistory(store); const reopened = await openTab(store, { html: listing(), remote }); await waitFor(() => visitedUrls(store).length === 3); assert.ok(card(reopened.window, '/b.html').classList.contains('jvs-visited')); - assert.equal( - store.read('javstore_sync_config_v1').historyId, - store.state().historyId, - 'the cursor is recorded against the history that now holds the rows', - ); + await waitFor(() => syncPosition(store) > 0); }); test('a visit recorded after the history was lost does not hide what is missing', async () => { @@ -859,9 +858,8 @@ test('a click replayed from the previous page reaches the worker', async () => { // What a dropped GM write leaves behind: a click stamped before the last push, which // only the next page load in that tab will find. - const config = store.read('javstore_sync_config_v1'); const session = new Map([['javstore_pending_visits', JSON.stringify([ - { url: 'https://javstore.net/c.html', at: config.pushedAt - 5000 }, + { url: 'https://javstore.net/c.html', at: Date.now() - 5000 }, ])]]); first.window.close(); diff --git a/worker/src/worker.mjs b/worker/src/worker.mjs index f9ecc96..3f5c58a 100644 --- a/worker/src/worker.mjs +++ b/worker/src/worker.mjs @@ -23,7 +23,7 @@ const STORAGE_VERSION = 3; // Reported in every answer so the userscript can show which worker it is talking to. It // moves in step with the userscript's own version; a test holds the two together. -export const WORKER_VERSION = '6.11.1'; +export const WORKER_VERSION = '6.12.0'; const OBJECT_NAME = 'default'; const MAX_BODY_BYTES = 8 * 1024 * 1024; const MAX_PAGE_ROWS = 2000; @@ -117,6 +117,18 @@ export class SyncStore { ); } + // A random id for this store, made once. A device that finds a different one answering + // at the same address knows the store it counted against is gone—recreated empty, or + // replaced—and starts over, sending everything it holds. + #epoch() { + let epoch = this.#meta('epoch', ''); + if (typeof epoch !== 'string' || !epoch) { + epoch = crypto.randomUUID(); + this.#setMeta('epoch', epoch); + } + return epoch; + } + #nextSeq() { const next = timestamp(this.#meta('seq', 0)) + 1; this.#setMeta('seq', next); @@ -264,6 +276,7 @@ export class SyncStore { return { worker: WORKER_VERSION, + epoch: this.#epoch(), cursor: more ? changes[limit - 1].seq : timestamp(this.#meta('seq', 0)), more, meta: this.#readMeta(), From 501e9167a0b9de337c5868a099f37be5e7a60f01 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 12 Nov 2026 19:52:17 +0000 Subject: [PATCH 08/11] Add failing review tests for 6.12.0 history and sync design Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VCuhK4TQqcEbLXdmxLjggy --- tests/review-final.test.mjs | 211 ++++++++++++++++++++++++++++++++++++ 1 file changed, 211 insertions(+) create mode 100644 tests/review-final.test.mjs diff --git a/tests/review-final.test.mjs b/tests/review-final.test.mjs new file mode 100644 index 0000000..fb61b12 --- /dev/null +++ b/tests/review-final.test.mjs @@ -0,0 +1,211 @@ +// Final adversarial review of 6.12.0's history and sync design under AdGuard's per-tab +// storage snapshots. Each test here reproduces a defect found in that review; the +// accompanying report ranks them. +import test, { afterEach } from 'node:test'; +import assert from 'node:assert/strict'; +import { + makeStore, makeRemote, enableSync, openTab, closeTabs, settle, waitFor, + listingHtml, detailHtml, card, pressOnCard, visitedUrls, +} from './harness.mjs'; + +const CARDS = [['/a.html', 'Alpha'], ['/b.html', 'Beta'], ['/c.html', 'Gamma'], ['/d.html', 'Delta']]; +const listing = () => listingHtml(CARDS); +const u = href => `https://javstore.net${href}`; +const MIN = 60000; +const CONFIG_KEY = 'javstore_sync_config_v1'; +const SESSION_KEY = 'javstore_sync_session_v1'; + +afterEach(closeTabs); + +// One clock for every tab of the device, which a test can move forward. +function device(options = {}) { + const store = makeStore({ snapshotPerTab: true, ...options }); + const clock = { offset: 0 }; + const open = (extra = {}) => openTab(store, { + html: listing(), + ...extra, + beforeRun: window => { + const realNow = window.Date.now.bind(window.Date); + window.Date.now = () => realNow() + clock.offset; + }, + }); + return { store, clock, open }; +} + +const isVisited = (tab, href) => card(tab.window, href).classList.contains('jvs-visited'); + +// R1. A page's journal key is named after the moment the page loaded, not the moment it is +// first written. A tab that sits idle for longer than JOURNAL_ROTATE_MS and then records +// something writes a key whose name already says "final since minutes ago". A newer tab +// folds it into the main document, the next one retires it at once, and a third tab that +// loaded before the write—still young enough to write the main document—writes its older +// view over the top. Nothing is left holding the visit. +test('R1 a long-idle tab\'s first journal write is not treated as final the moment it lands', async () => { + const { store, clock, open } = device(); + // A main document exists, so retirement has something to check coverage against. + await open({ html: detailHtml, url: u('/d.html') }); + await settle(); + assert.deepEqual(visitedUrls(store), [u('/d.html')]); + + const old = await open(); // loads at 0 and records nothing for 20 minutes + clock.offset = 15 * MIN; + const young = await open(); // loads at 15; may write the main document until 23 + + clock.offset = 20 * MIN; + pressOnCard(old.window, '/a.html', 'v'); // its first save: journal only, key named "0" + await settle(); + assert.ok(store.journals().some(key => store.read(key)?.visited?.[u('/a.html')]), + 'the old tab journaled a'); + + clock.offset = 20.5 * MIN; + await open(); // folds the journal into the main document + await settle(); + assert.ok(visitedUrls(store).includes(u('/a.html'))); + + clock.offset = 21 * MIN; + await open(); // finds it covered and, by the key's name, long final + await settle(); + + clock.offset = 22 * MIN; + pressOnCard(young.window, '/c.html', 'v'); // a young tab, loaded before a was recorded + await settle(); + + clock.offset = 23 * MIN; + const fresh = await open(); + await settle(); + assert.ok(isVisited(fresh, '/c.html')); + assert.ok(isVisited(fresh, '/a.html'), 'the visit the long-idle tab recorded survives'); +}); + +// R2. The outbox records one "sent" mark per change, whichever worker confirmed it, while +// sync positions are kept per worker. A change made while the device synced with worker B +// is marked sent; when the device is pointed back at worker A, whose position it still +// holds, only outstanding changes go up, so A never gets it—and nor does any other device +// syncing with A. +test('R2 a change confirmed by one worker still reaches another worker this device returns to', async () => { + const first = makeRemote({ origin: 'https://one.test' }); + const second = makeRemote({ origin: 'https://two.test' }); + const store = makeStore(); + const remotes = [first, second]; + + enableSync(store, first); + await openTab(store, { html: detailHtml, url: u('/a.html'), remote: remotes }); + await waitFor(() => first.visited().includes(u('/a.html'))); + await settle(100); + closeTabs(); + + enableSync(store, second); + await openTab(store, { html: detailHtml, url: u('/b.html'), remote: remotes }); + await waitFor(() => second.visited().includes(u('/b.html'))); + await settle(100); + closeTabs(); + + enableSync(store, first); + const before = first.requests.length; + await openTab(store, { html: listing(), remote: remotes }); + await waitFor(() => first.requests.length > before); + await settle(100); + assert.deepEqual(first.visited(), [u('/a.html'), u('/b.html')], + 'b, recorded while syncing with the other worker, reaches the first one'); +}); + +// R3. journalSince never moves forward, and every pull lowers it to the oldest row it took +// on. One row another device recorded days ago (a phone that was offline, a visit synced +// late) is enough: from then on, every save by that tab writes a journal holding every +// visit and tombstone stamped since—here the whole 4,900-visit history—for as long as the +// tab stays open. Old tabs on AdGuard write nothing else, so they each carry it. +test('R3 pulling one old row does not make every later save write the whole history to the journal', async () => { + const now = Date.now(); + const visited = {}; + for (let i = 0; i < 4900; i++) { + visited[`https://javstore.net/${600000 + i}-FC2-PPV-${3000000 + i}-Some-Typical-Title-Words-pn.html`] = now - 86400000 + i; + } + const store = makeStore({ snapshotPerTab: true }); + store.write('javstore_cleanup_state_v2', { + version: 3, updatedAt: now - 1000, resetAt: 0, prunedBefore: 0, settings: {}, settingTimes: {}, + visited, overrides: {}, overrideTimes: {}, tombstones: {}, + }); + const remote = makeRemote(); + // Another device's visit, pushed late, stamped two days ago. + remote.store.sync({ cursor: 0, changes: [{ kind: 'v', key: u('/b.html'), at: now - 2 * 86400000 }] }); + enableSync(store, remote); + + const tab = await openTab(store, { html: listing(), remote }); + await waitFor(() => store.read(SESSION_KEY)?.completedAt, { timeout: 20000 }); + await settle(100); + assert.ok(isVisited(tab, '/b.html'), 'the old row was pulled'); + + pressOnCard(tab.window, '/a.html', 'v'); + await settle(100); + const sizes = store.journals().map(key => store.data.get(key).length); + const largest = Math.max(0, ...sizes); + assert.ok(largest < 20000, + `a one-visit save writes a journal of ${(largest / 1024).toFixed(0)} KB`); +}); + +// R4. After the update, the user reloads the JavStore tabs that were open (or restarts the +// browser, which restores them). The history was saved by 6.11.0 and has no position, so +// each tab uploads everything it holds and downloads everything the worker holds. Web +// Locks put the tabs one after another, but each reads its own snapshot, taken before the +// first finished, so every one of them repeats the whole exchange. +test('R4 tabs reloaded together after the update re-exchange the whole history once, not once per tab', async () => { + const now = Date.now(); + const remote = makeRemote(); + const visited = {}; + const changes = []; + for (let i = 0; i < 2000; i++) { + const key = `https://javstore.net/${600000 + i}-FC2-PPV-${3000000 + i}-Some-Typical-Title-Words-pn.html`; + visited[key] = now - 86400000 + i; + changes.push({ kind: 'v', key, at: visited[key] }); + } + remote.store.sync({ cursor: 0, changes }); + const store = makeStore({ snapshotPerTab: true }); + // What 6.11.0 left: no position and no outbox in the history, the cursor in the config. + store.write('javstore_cleanup_state_v2', { + version: 3, scriptVersion: '6.11.0', updatedAt: now - 1000, resetAt: 0, prunedBefore: 0, + settings: {}, settingTimes: {}, historyId: 'h1', visited, overrides: {}, overrideTimes: {}, tombstones: {}, + }); + enableSync(store, remote, { remoteKey: remote.endpoint, cursor: 2000, pushedAt: now - 500, historyId: 'h1' }); + store.write(SESSION_KEY, { + endpoint: remote.endpoint, historyId: 'h1', completedAt: now - 3600000, activeAt: now - 3600000, + }); + + const tabs = await Promise.all(Array.from({ length: 6 }, (_, i) => + openTab(store, { html: listing(), remote, url: `https://javstore.net/page/${i + 1}` }))); + await waitFor(() => remote.requests.length >= 6, { timeout: 30000 }); + await settle(400); + const uploaded = remote.requests.reduce((sum, request) => sum + (request.body?.changes?.length || 0), 0); + assert.equal(tabs.length, 6); + assert.ok(uploaded <= 2 * 2000, + `${uploaded} changes uploaded for a 2,000-visit history over ${remote.requests.length} requests`); +}); + +// R5. Rotation writes the new journal key and then deletes the previous one. AdGuard does +// not report a write that fails (a full quota, say, which R3's journals make likelier), so +// when the new key does not land the delete still does, and what the long-open tab had +// safely stored—and nowhere else, since it may not write the main document—is gone. +test('R5 a journal rotation whose new key never lands does not delete the key it replaces', async () => { + const { store, clock, open } = device(); + await open({ html: detailHtml, url: u('/d.html') }); + await settle(); + const old = await open(); + + clock.offset = 9 * MIN; // too old to write the main document + pressOnCard(old.window, '/a.html', 'v'); + await settle(); + assert.ok(store.journals().some(key => store.read(key)?.visited?.[u('/a.html')]), 'a is journaled'); + + clock.offset = 15 * MIN; // past JOURNAL_ROTATE_MS: the next save moves to a new key + const set = store.set; + store.set = async function (key, value) { + if (key.startsWith('javstore_journal_')) return; // dropped without an error + return set.call(this, key, value); + }; + pressOnCard(old.window, '/b.html', 'v'); + await settle(); + store.set = set; + + const fresh = await open(); + await settle(); + assert.ok(isVisited(fresh, '/a.html'), 'the visit stored before the failed write is still stored'); +}); From da7cdebf008bc23ae57fca723d3b0b5d31b71c89 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 12 Nov 2026 19:58:00 +0000 Subject: [PATCH 09/11] Close the gaps a final review found in the 6.12.0 design - Name a journal key for its first write, not the page load, so a tab that first saves long after loading cannot have its journal taken for final and removed while a stale write can still drop it. - Never delete the previous journal key on rotation: AdGuard does not report failed writes, so it may be the only copy. Retirement removes it once a newer tab finds it covered. - Keep the outbox's confirmations per worker, and send everything once when returning to a worker after syncing with another. - Journal pulled and imported rows by key instead of reaching the journal back in time, so one old row no longer makes every save write the whole history; journals carry only changes still to send. - Against a store with no position, download first and upload only what it lacks; entries listed just for that are dropped once confirmed. - Prefer the copy that knows the store's id when two positions tie. Adds the final review's tests (review-final.test.mjs). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VCuhK4TQqcEbLXdmxLjggy --- CHANGELOG.md | 1 + README.md | 2 +- javstore-full-layout-cleanup.user.js | 225 ++++++++++++++++++--------- 3 files changed, 157 insertions(+), 71 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3f99c07..b9dc4b2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ A review of the whole history and sync design against how AdGuard actually store - Fix: on AdGuard, the storage-based lock used when Web Locks are missing could not keep tabs apart, and a lease left behind could hold a tab up for 95 seconds. It is no longer used on managers that don't report other tabs' writes; tabs sync side by side, at the cost of at most a duplicate request. - Fix: opening a new tab could show a visit that had just been unmarked, or cleared, in another tab, and save it back. - Change: the worker now gives its store a random id and returns it with every answer. A device that finds a different store at the same address, such as one recreated empty, starts over and sends it everything it holds; until now such a worker stayed empty. The worker needs redeploying; the Cloudflare build does that when this reaches `main`. Until it does, an older worker still works, and a reset is noticed from its sequence numbers going backwards. +- Change: a first sync with a worker, or after its store was reset, downloads first and then uploads only what the worker lacks or holds an older version of, rather than everything the device holds. After the update, each JavStore tab reloaded at once on AdGuard starts this way from its own copy, so this keeps that to one download per tab and almost no uploads. Each worker's confirmations are kept apart, so moving between two workers sends each one everything it is missing. - After updating, reload or close JavStore tabs that were open before the update. Until then they keep running the old code. - The worker version moves to 6.12.0 with the userscript. diff --git a/README.md b/README.md index 4206e1e..13933f9 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,7 @@ Once it is deployed, open the `JVS` panel → **Cloud sync**, enter the worker U - The first page in a browsing session checks the worker; later pages and new tabs use the shared local history, and check again once the sync interval has passed so that visits from other devices keep arriving while you browse. A session is shared across tabs and expires after 30 minutes without startup/sync activity. Starting another session checks for new changes from the saved position; it does not download the entire history again. - Opening 30–50 item tabs from the context menu still records every visit when its item page loads. Uploads wait a few seconds and collect visits from all tabs; tabs whose changes were already uploaded skip their redundant requests. Advancing the listing to page 2 does not restart restoration. - **Sync now** always checks the worker. Focus refreshes and the configured periodic refresh use shared progress, so tabs do not independently download the same changes. Hidden tabs skip periodic refreshes, but their new visits can still upload. The panel's status line reports the last sync or the reason the last one failed. -- Devices exchange only what changed. The stored history records, in the same save as the visits, how far into the worker's history it is complete, and keeps a list of the changes made on the device that the worker has not confirmed yet. Incremental checks exchange only new rows, and an ordinary page load during an active session needs no cloud request. The first sync with a worker transfers everything, and so does the next sync on a device whose userscript manager lost its history but kept the sync settings: a history with no position for the worker is downloaded again rather than left behind. +- Devices exchange only what changed. The stored history records, in the same save as the visits, how far into the worker's history it is complete, and keeps a list of the changes made on the device that the worker has not confirmed yet. Incremental checks exchange only new rows, and an ordinary page load during an active session needs no cloud request. The first sync with a worker downloads everything and uploads only what the worker lacks, and so does the next sync on a device whose userscript manager lost its history but kept the sync settings: a history with no position for the worker is downloaded again rather than left behind. - Each worker store has a random id. A device that finds a different store at the same address (for example one recreated empty) starts over against it and sends it everything it holds. - Sync never writes the sync settings themselves (endpoint, token, on/off, interval); only the panel and sync links do, so a tab that was opened before a change cannot undo it. - The worker applies the same rules as the local merge—newest timestamp wins per URL—so a device that has been offline for a week cannot overwrite what the others recorded. A device that pushes a stale entry is handed the winning one back. diff --git a/javstore-full-layout-cleanup.user.js b/javstore-full-layout-cleanup.user.js index 60ccb28..2cb50d3 100644 --- a/javstore-full-layout-cleanup.user.js +++ b/javstore-full-layout-cleanup.user.js @@ -161,12 +161,17 @@ // the same address is noticed; `gen` is when this device started counting against // that store, so a position from an earlier start never outranks a later one. let syncedThrough = {}; - // Changes made on this device that the worker has to hear about, keyed `kind|key`: - // `at` is the change, `sent` the newest change the worker has confirmed, `t` when the - // entry last changed. A change is outstanding while `at > sent`. Both only ever grow, - // so copies merge by taking the larger of each, and an older copy can at worst offer a - // confirmed change once more—which the worker takes as already known. + // Changes made on this device that workers have to hear about, keyed `kind|key`: `at` + // is the change, `sent` the newest change each worker (by tag) has confirmed, `t` when + // the entry last changed. A change is outstanding for a worker while `at` is past what + // that worker confirmed. Everything only ever grows, so copies merge by taking the + // larger of each, and an older copy can at worst offer a confirmed change once more— + // which the worker takes as already known. let outbox = new Map(); + // Entries this page took on from outside local storage—rows pulled from the worker, a + // restored backup—which its journal carries whatever their age. A journal reaching + // back in time instead would carry everything stamped since the oldest of them. + const journalExtras = new Set(); let settingsUpdatedAt = 0; // When each setting was last changed on purpose, keyed by setting name. Settings merge // field by field on these, so changing one setting on one device does not carry that @@ -420,8 +425,8 @@ const outboxAhead = [...parseOutbox(doc.outbox)].some(([id, entry]) => { const held = baseOutbox.get(id); // A change already confirmed may have been dropped from the main list. - if (!held) return entry.at > entry.sent; - return entry.at > held.at || entry.sent > held.sent; + if (!held) return !Object.values(entry.sent).some(mark => mark >= entry.at); + return mergedOutboxEntry(held, entry) !== null; }); const basePositions = parseSyncPositions(base.syncedThrough); const positionAhead = Object.entries(parseSyncPositions(doc.syncedThrough)) @@ -616,12 +621,8 @@ newest(combined.tombstones, doc.tombstones); combined.resetAt = Math.max(combined.resetAt, parseTimestamp(doc.resetAt)); for (const [id, entry] of parseOutbox(doc.outbox)) { - const held = outboxes.get(id); - outboxes.set(id, held ? { - at: Math.max(held.at, entry.at), - sent: Math.max(held.sent, entry.sent), - t: Math.max(held.t, entry.t), - } : entry); + const merged = mergedOutboxEntry(outboxes.get(id), entry); + if (merged) outboxes.set(id, merged); } Object.entries(parseSyncPositions(doc.syncedThrough)).forEach(([tag, position]) => { positions[tag] = newerSyncPosition(positions[tag], position); @@ -648,7 +649,7 @@ } } }); - combined.outbox = Object.fromEntries([...outboxes].map(([id, entry]) => [id, [entry.at, entry.sent, entry.t]])); + combined.outbox = Object.fromEntries([...outboxes].map(([id, entry]) => [id, serializeOutboxEntry(entry)])); combined.syncedThrough = positions; return combined; } @@ -664,10 +665,10 @@ } function serializeJournal(writtenAt) { - const pick = source => { + const pick = (source, kind = '') => { const result = {}; for (const [key, at] of source) { - if (at >= journalSince) result[key] = at; + if (at >= journalSince || journalExtras.has(kind ? `${kind}|${key}` : key)) result[key] = at; } return result; }; @@ -675,21 +676,26 @@ const journalOverrideTimes = {}; for (const [url, value] of overrides) { const at = overrideTimes.get(url) || 0; - if (at >= journalSince) { + if (at >= journalSince || journalExtras.has(`o|${url}`)) { journalOverrides[url] = value; journalOverrideTimes[url] = at; } } + // Only what is still to be sent: a confirmation that never reaches the main + // document costs one repeated upload, which the worker takes as already known. + const tag = syncConfigured() ? remoteTag(syncConfig.endpoint) : ''; const journalOutbox = {}; for (const [id, entry] of outbox) { - if (entry.t >= journalSince) journalOutbox[id] = [entry.at, entry.sent, entry.t]; + if (entry.t >= journalSince && (!tag || isOutstanding(entry, tag))) { + journalOutbox[id] = serializeOutboxEntry(entry); + } } const journal = { version: STORAGE_VERSION, writtenAt, resetAt, prunedBefore, - visited: pick(visited), + visited: pick(visited, 'v'), tombstones: pick(tombstones), overrides: journalOverrides, overrideTimes: journalOverrideTimes, @@ -718,19 +724,17 @@ || resetAt >= journalSince || settingsUpdatedAt >= journalSince; if (!carries) return null; - // A key that has been written long enough is left as it is, final, and the page - // moves to a new one that carries everything the old one did. - const previous = journalWritten && writtenAt - journalCreatedAt > JOURNAL_ROTATE_MS ? journalKey : ''; - if (previous) { + // A key is named for the moment it is first written, so its name says when it can + // last have been written. One written long enough is left as it is, final, and the + // page moves to a new key that carries everything the old one did. The old one is + // not removed here: if the new write never lands (AdGuard does not say), it is the + // only copy. It goes once a newer tab finds it in the main document. + if (!journalWritten || writtenAt - journalCreatedAt > JOURNAL_ROTATE_MS) { journalCreatedAt = writtenAt; journalKey = makeJournalKey(journalCreatedAt); } await storeValue(journalKey, journal); journalWritten = true; - if (previous) { - // The new key holds everything the old one did, so the old one is only a copy. - await gmCall('deleteValue', previous).catch(() => {}); - } return journal; } @@ -1429,7 +1433,9 @@ if (!a) return b || null; if (!b) return a; if (a.gen !== b.gen) return a.gen > b.gen ? a : b; - return a.cursor >= b.cursor ? a : b; + if (a.cursor !== b.cursor) return a.cursor > b.cursor ? a : b; + // The same place, one copy having learned the store's id since. + return a.epoch || !b.epoch ? a : b; } function sameSyncPosition(a, b) { @@ -1445,6 +1451,16 @@ }); } + function parseSentMarks(value) { + const result = {}; + if (!value || typeof value !== 'object' || Array.isArray(value)) return result; + Object.entries(value).slice(0, 20).forEach(([tag, mark]) => { + const at = parseTimestamp(mark); + if (tag && at) result[tag.slice(0, 40)] = at; + }); + return result; + } + function parseOutbox(value) { const result = new Map(); if (!value || typeof value !== 'object' || Array.isArray(value)) return result; @@ -1452,32 +1468,41 @@ if (!/^[vo]\|./.test(id) || !Array.isArray(entry)) return; const at = parseTimestamp(entry[0]); if (!at) return; - result.set(id, { at, sent: parseTimestamp(entry[1]), t: parseTimestamp(entry[2]) || at }); + result.set(id, { at, sent: parseSentMarks(entry[1]), t: parseTimestamp(entry[2]) || at }); }); return result; } + function serializeOutboxEntry(entry) { + return [entry.at, { ...entry.sent }, entry.t]; + } + function serializeOutbox() { const result = {}; - for (const [id, entry] of outbox) result[id] = [entry.at, entry.sent, entry.t]; + for (const [id, entry] of outbox) result[id] = serializeOutboxEntry(entry); return result; } + // The larger of each part of two copies of one entry; `null` when `entry` adds nothing. + function mergedOutboxEntry(held, entry) { + if (!held) return { at: entry.at, sent: { ...entry.sent }, t: entry.t }; + let grew = entry.at > held.at; + const sent = { ...held.sent }; + Object.entries(entry.sent).forEach(([tag, mark]) => { + if (mark > (sent[tag] || 0)) { + sent[tag] = mark; + grew = true; + } + }); + return grew ? { at: Math.max(held.at, entry.at), sent, t: Math.max(held.t, entry.t) } : null; + } + function mergeOutbox(source) { let grew = false; for (const [id, entry] of source) { - const held = outbox.get(id); - if (!held) { - outbox.set(id, { ...entry }); - grew = true; - continue; - } - if (entry.at > held.at || entry.sent > held.sent) { - outbox.set(id, { - at: Math.max(held.at, entry.at), - sent: Math.max(held.sent, entry.sent), - t: Math.max(held.t, entry.t), - }); + const merged = mergedOutboxEntry(outbox.get(id), entry); + if (merged) { + outbox.set(id, merged); grew = true; } } @@ -1491,15 +1516,11 @@ const id = `${kind}|${key}`; const held = outbox.get(id); if (held && held.at >= at) return; - outbox.set(id, { at, sent: held?.sent || 0, t: Date.now() }); + outbox.set(id, { at, sent: { ...(held?.sent || {}) }, t: Date.now() }); } - // Everything this device holds, as changes the worker has not heard of yet: for a - // worker this device has never synced with, or one that has come back empty. - function noteEverythingUnsent() { - for (const [url, at] of visited) noteLocalChange('v', url, at); - for (const [url] of overrides) noteLocalChange('o', url, overrideTimes.get(url) || 0); - for (const [key, at] of tombstones) noteLocalChange(key.slice(0, 1), key.slice(2), at); + function isOutstanding(entry, tag) { + return entry.at > (entry.sent[tag] || 0); } // The change the worker should get for an outbox entry, from what this device holds @@ -1517,27 +1538,55 @@ return removed ? { kind, key, at: removed, deleted: 1 } : null; } - function outstandingChanges() { + function outstandingChanges(tag) { const changes = []; for (const [id, entry] of outbox) { - if (entry.at <= entry.sent) continue; + if (!isOutstanding(entry, tag)) continue; const change = outgoingChange(id); if (change) { changes.push({ id, mark: entry.at, change }); } else { // Gone here since: nothing to tell the worker. - outbox.set(id, { ...entry, sent: entry.at, t: Date.now() }); + outbox.set(id, { ...entry, sent: { ...entry.sent, [tag]: entry.at }, t: Date.now() }); } } return changes.sort((a, b) => a.change.at - b.change.at); } - function outstandingCount() { + function outstandingCount(tag) { let count = 0; - for (const entry of outbox.values()) if (entry.at > entry.sent) count += 1; + for (const entry of outbox.values()) if (isOutstanding(entry, tag)) count += 1; return count; } + // After downloading a whole store: what it already holds is marked as sent to it, and + // what it lacks, or holds an older version of, is listed and returned to send. + // Entries listed only so a first upload survives an interrupted sync; once the worker + // has confirmed them, no other copy ever listed them, so they go straight away. + const reconcileListed = new Set(); + + function reconcileWith(pulledAt, tag) { + localChangesSince(0).forEach(change => { + const id = `${change.kind}|${change.key}`; + if ((pulledAt.get(id) || 0) >= change.at) { + const held = outbox.get(id); + if (held && held.at <= change.at) markSent([{ id, mark: held.at }], tag); + } else { + if (!outbox.has(id)) reconcileListed.add(id); + noteLocalChange(change.kind, change.key, change.at); + } + }); + return outstandingChanges(tag); + } + + function forgetReconciled(tag) { + for (const id of reconcileListed) { + const entry = outbox.get(id); + if (entry && !isOutstanding(entry, tag)) outbox.delete(id); + } + reconcileListed.clear(); + } + function everythingHeld() { return localChangesSince(0).map(change => ({ id: `${change.kind}|${change.key}`, @@ -1546,17 +1595,25 @@ })); } - function markSent(batch) { + function markSent(batch, tag) { const now = Date.now(); batch.forEach(({ id, mark }) => { const held = outbox.get(id); - if (held && mark > held.sent) outbox.set(id, { ...held, sent: mark, t: now }); + if (held && mark > (held.sent[tag] || 0)) { + outbox.set(id, { ...held, sent: { ...held.sent, [tag]: mark }, t: now }); + } }); } + // A confirmed entry stays listed a while for older copies' sake. With sync off nothing + // is dropped: the list is bounded by the history itself, and it is what goes up once + // sync is back on. function pruneOutbox(now) { + const tag = syncConfigured() ? remoteTag(syncConfig.endpoint) : ''; + if (!tag) return; for (const [id, entry] of outbox) { - if (entry.at <= entry.sent && now - entry.t > OUTBOX_KEEP_MS) outbox.delete(id); + if (now - entry.t <= OUTBOX_KEEP_MS) continue; + if (!isOutstanding(entry, tag) || !outgoingChange(id)) outbox.delete(id); } } @@ -1906,7 +1963,7 @@ && Number.isFinite(session?.cursor) && session.gen === position.gen && cursor >= session.cursor && session?.completedAt > 0 && now - session.activeAt < SYNC_SESSION_IDLE_MS; if (!manual && currentSession) { - const pending = outstandingCount() || session.uploadedMeta !== metadataSignature(); + const pending = outstandingCount(tag) || session.uploadedMeta !== metadataSignature(); if (reason === 'startup' && pending) scheduleRemotePush(); const sinceDownload = now - session.completedAt; const interval = syncConfig.intervalMinutes * 60000; @@ -1926,9 +1983,16 @@ } } - // Everything this device holds goes up to a worker it has no place with yet; - // otherwise what is still outstanding. - let outgoing = position ? outstandingChanges() : everythingHeld(); + // Against a store this device has no place with yet, download first and then send + // only what the store turns out to lack, rather than everything held: after an + // update every reloaded tab on AdGuard starts out this way, each from its own copy. + let reconciling = !position; + const pulledAt = new Map(); + // Back on a worker after syncing with another one: what that one confirmed may + // have been dropped from the list since, so everything goes up once. + const switchedBack = Boolean(position) && Boolean(session?.endpoint) + && session.endpoint !== syncConfig.endpoint; + let outgoing = reconciling ? [] : (switchedBack ? everythingHeld() : outstandingChanges(tag)); let sent = 0; let changed = false; let absorbed = false; @@ -1964,7 +2028,8 @@ if (applyStoredState(document)) changed = true; if (lastMergeAdopted) absorbed = true; lowerJournalSince(lastMergeOldest); - markSent(outgoing); + markSent(outgoing, tag); + reconciling = false; // Nothing about this store counts towards a worker that speaks cursors, // should one answer at this address again. position = { epoch: 'document', gen: Date.now(), cursor: 0 }; @@ -1983,13 +2048,15 @@ restarted = true; position = { epoch, gen: Date.now(), cursor: 0 }; cursor = 0; - outgoing = everythingHeld(); + reconciling = true; + pulledAt.clear(); + outgoing = []; continue; } if (!position) position = { epoch, gen: Date.now(), cursor: 0 }; else if (!position.epoch && epoch) position = { ...position, epoch }; - markSent(batch); + markSent(batch, tag); outgoing = outgoing.slice(batch.length); sent += batch.length; cursor = answered; @@ -1997,16 +2064,31 @@ if (applyStoredState(documentFromRows(answer.changes, answer.meta))) changed = true; // Pulled rows go into the journal as well: they are what this page's place // in the sync vouches for, and the journal carries that place. - lowerJournalSince(lastMergeOldest); + (Array.isArray(answer.changes) ? answer.changes : []).forEach(row => { + if (!row || typeof row !== 'object' || !row.key) return; + const id = `${row.kind === 'o' ? 'o' : 'v'}|${row.key}`; + journalExtras.add(id); + if (reconciling) pulledAt.set(id, Math.max(pulledAt.get(id) || 0, parseTimestamp(row.at))); + }); // A batch can carry nothing but protective metadata—a tombstone, a newer // `resetAt`, `prunedBefore` or `settingsUpdatedAt`—which removes nothing on // screen and so leaves `changed` false. It still has to be saved. if (lastMergeAdopted) absorbed = true; applyingRemoteState = false; downloadComplete = !answer.more; + if (reconciling && downloadComplete) { + reconciling = false; + outgoing = reconcileWith(pulledAt, tag); + } if (!outgoing.length && downloadComplete) break; } + // The page budget ran out before the store was fully downloaded, so what it + // lacks is not known yet: everything held is listed as unsent for it. + if (reconciling) { + localChangesSince(0).forEach(change => noteLocalChange(change.kind, change.key, change.at)); + } + forgetReconciled(tag); // The rows and the place they reach go into storage in the same write. const naming = !legacy && !historyId; if (naming) historyId = makeHistoryId(); @@ -2875,11 +2957,14 @@ pruneVisited(); // Everything restored goes up, and the journal carries it until the main // document does. - for (const [url, at] of visited) noteLocalChange('v', url, at); - for (const [url] of overrides) noteLocalChange('o', url, overrideTimes.get(url) || 0); - const oldest = [...visited.values(), ...overrideTimes.values()] - .reduce((least, at) => (least ? Math.min(least, at) : at), 0); - lowerJournalSince(oldest); + for (const [url, at] of visited) { + noteLocalChange('v', url, at); + journalExtras.add(`v|${url}`); + } + for (const [url] of overrides) { + noteLocalChange('o', url, overrideTimes.get(url) || 0); + journalExtras.add(`o|${url}`); + } persistState(); setRootState(); processAllCards(); From f46fe6965e7d0cf749a7e2f0cfb53a7c437d862e Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 12 Nov 2026 20:07:21 +0000 Subject: [PATCH 10/11] Let the test suites run against the Userscripts app's API shape JVS_DOTTED=1 makes every store provide only the dotted GM.* API, as the Userscripts app for Safari does. All 132 tests pass that way. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VCuhK4TQqcEbLXdmxLjggy --- tests/harness.mjs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/harness.mjs b/tests/harness.mjs index bdf7b2b..b13654c 100644 --- a/tests/harness.mjs +++ b/tests/harness.mjs @@ -32,7 +32,9 @@ const openWindows = new Set(); // held when the tab loaded, plus its own writes. Writes still reach storage, so a tab // opened later sees them, but a tab that is already open never does. export function makeStore({ - latency = 0, journaling = true, primitivesOnly = false, dottedOnly = false, snapshotPerTab = false, + latency = 0, journaling = true, primitivesOnly = false, + // JVS_DOTTED=1 runs every suite against the Userscripts app's API shape. + dottedOnly = process.env.JVS_DOTTED === '1', snapshotPerTab = false, } = {}) { const data = new Map(); const delay = () => (latency ? new Promise(resolve => setTimeout(resolve, Math.random() * latency)) : null); From 693f38494275f3bd61706caed31d722657047448 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 12 Nov 2026 20:21:39 +0000 Subject: [PATCH 11/11] Keep a long-open tab's journal keys small after a large download Rows pulled from the worker stayed in every later journal write for the life of the tab, and since a page no longer removes its own keys, a long-open tab left a full-size key behind every five minutes. A new key now overlaps only the key it replaces; older pulled rows and entries stay in the final key that first carried them until a newer tab finds them in the main document and retires it. Reported by the Codex review on the pull request. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VCuhK4TQqcEbLXdmxLjggy --- javstore-full-layout-cleanup.user.js | 47 ++++++++++++++++++---------- tests/review-final.test.mjs | 34 ++++++++++++++++++++ 2 files changed, 64 insertions(+), 17 deletions(-) diff --git a/javstore-full-layout-cleanup.user.js b/javstore-full-layout-cleanup.user.js index 2cb50d3..aefc20d 100644 --- a/javstore-full-layout-cleanup.user.js +++ b/javstore-full-layout-cleanup.user.js @@ -169,9 +169,11 @@ // which the worker takes as already known. let outbox = new Map(); // Entries this page took on from outside local storage—rows pulled from the worker, a - // restored backup—which its journal carries whatever their age. A journal reaching - // back in time instead would carry everything stamped since the oldest of them. - const journalExtras = new Set(); + // restored backup—which its journal carries whatever their age, keyed to the journal + // key that first carried them. A journal reaching back in time instead would carry + // everything stamped since the oldest of them. + const journalExtras = new Map(); + let journalGeneration = 0; let settingsUpdatedAt = 0; // When each setting was last changed on purpose, keyed by setting name. Settings merge // field by field on these, so changing one setting on one device does not carry that @@ -483,8 +485,9 @@ // - Without notice of other tabs' writes, only a tab that loaded less than // FRESH_TAB_MS ago writes the main document. An older one writes its journal only. // - A page writes each journal key for JOURNAL_ROTATE_MS at most and then moves on to - // a new one that carries everything the old one did, so a key that is old enough is - // final: any tab that loaded later holds all of it. + // a new one, so a key that is old enough is final: any tab that loaded later holds + // all of it. A page never removes its own keys; a final key stays in storage until + // the rule below removes it, so the next key only has to overlap it. // - A journal is removed only once it is final, the main document this tab loaded // from storage already covers it, and JOURNAL_TTL_MS have passed since its last // write, which leaves no tab that might lack it still allowed to write the main @@ -656,9 +659,9 @@ // Entries this page adopted with a timestamp from the past—a replayed click, rows // pulled from the worker, a restored backup—exist nowhere else in local storage yet, - // so the journal has to reach back far enough to carry them. It never stops carrying - // them while the page is open: on AdGuard a page cannot tell whether its own write of - // the main document survived another tab writing an older one. + // so the journal has to reach back far enough to carry them. A journal key that has + // carried them stays in storage until a newer tab finds them in the main document, + // which is what lets later keys leave them out (see writeJournal). function lowerJournalSince(at) { const time = parseTimestamp(at); if (time && time < journalSince) journalSince = time; @@ -715,6 +718,21 @@ async function writeJournal(writtenAt) { if (!journalingAvailable()) return null; + // A key is named for the moment it is first written, so its name says when it can + // last have been written. One written long enough is left as it is, final, and the + // page moves to a new key. The old one is not removed here: it stays until a newer + // tab finds it in the main document. The new key overlaps it by the span of the + // key it replaces—so a last write of the old key that never landed (AdGuard does + // not say) is still covered—and leaves anything older to the keys that hold it, so + // a long-open page's journal stays as small as its last few minutes. + const rotating = journalWritten && writtenAt - journalCreatedAt > JOURNAL_ROTATE_MS; + if (rotating) { + journalSince = Math.max(journalSince, journalCreatedAt); + for (const [id, generation] of journalExtras) { + if (generation < journalGeneration) journalExtras.delete(id); + } + journalGeneration += 1; + } const journal = serializeJournal(writtenAt); const carries = Object.keys(journal.visited).length || Object.keys(journal.tombstones).length @@ -724,12 +742,7 @@ || resetAt >= journalSince || settingsUpdatedAt >= journalSince; if (!carries) return null; - // A key is named for the moment it is first written, so its name says when it can - // last have been written. One written long enough is left as it is, final, and the - // page moves to a new key that carries everything the old one did. The old one is - // not removed here: if the new write never lands (AdGuard does not say), it is the - // only copy. It goes once a newer tab finds it in the main document. - if (!journalWritten || writtenAt - journalCreatedAt > JOURNAL_ROTATE_MS) { + if (!journalWritten || rotating) { journalCreatedAt = writtenAt; journalKey = makeJournalKey(journalCreatedAt); } @@ -2067,7 +2080,7 @@ (Array.isArray(answer.changes) ? answer.changes : []).forEach(row => { if (!row || typeof row !== 'object' || !row.key) return; const id = `${row.kind === 'o' ? 'o' : 'v'}|${row.key}`; - journalExtras.add(id); + journalExtras.set(id, journalGeneration); if (reconciling) pulledAt.set(id, Math.max(pulledAt.get(id) || 0, parseTimestamp(row.at))); }); // A batch can carry nothing but protective metadata—a tombstone, a newer @@ -2959,11 +2972,11 @@ // document does. for (const [url, at] of visited) { noteLocalChange('v', url, at); - journalExtras.add(`v|${url}`); + journalExtras.set(`v|${url}`, journalGeneration); } for (const [url] of overrides) { noteLocalChange('o', url, overrideTimes.get(url) || 0); - journalExtras.add(`o|${url}`); + journalExtras.set(`o|${url}`, journalGeneration); } persistState(); setRootState(); diff --git a/tests/review-final.test.mjs b/tests/review-final.test.mjs index fb61b12..e73515d 100644 --- a/tests/review-final.test.mjs +++ b/tests/review-final.test.mjs @@ -209,3 +209,37 @@ test('R5 a journal rotation whose new key never lands does not delete the key it await settle(); assert.ok(isVisited(fresh, '/a.html'), 'the visit stored before the failed write is still stored'); }); + +// Codex review on #23: a tab that pulled a large history carried every pulled row in every +// later journal write, and since a page never removes its own keys, a long-open tab left a +// full-size key behind every few minutes. Each key now overlaps only the one it replaces; +// the rows stay in the final key that first carried them until a newer tab retires it. +test('a long-open tab that pulled a large history writes small journals a few rotations later', async () => { + const now = Date.now(); + const remote = makeRemote(); + const changes = []; + for (let i = 0; i < 3000; i++) { + changes.push({ kind: 'v', key: `https://javstore.net/${600000 + i}-FC2-PPV-${3000000 + i}-pn.html`, at: now - 86400000 + i }); + } + remote.store.sync({ cursor: 0, changes }); + const { store, clock, open } = device(); + enableSync(store, remote); + const tab = await open({ remote }); + await waitFor(() => store.read(SESSION_KEY)?.completedAt, { timeout: 20000 }); + await settle(100); + const largest = () => Math.max(0, ...store.journals().map(key => store.data.get(key).length)); + assert.ok(largest() > 100000, 'the pulled rows are journaled at first'); + const firstKeys = new Set(store.journals()); + + for (const minutes of [6, 12]) { + clock.offset = minutes * 60000; + pressOnCard(tab.window, minutes === 6 ? '/a.html' : '/b.html', 'v'); + await settle(100); + } + const latest = store.journals().filter(key => !firstKeys.has(key)) + .sort((a, b) => parseInt(b.split('_')[3], 36) - parseInt(a.split('_')[3], 36))[0]; + assert.ok(latest, 'the page moved on to a new key'); + assert.ok(store.data.get(latest).length < 20000, + `the latest journal is ${(store.data.get(latest).length / 1024).toFixed(0)} KB`); + assert.ok([...firstKeys].every(key => store.journals().includes(key)), 'the key holding the rows stays'); +});