diff --git a/CHANGELOG.md b/CHANGELOG.md index 969b190..397c58a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,12 @@ # Changelog +## 6.10.0 + +- Change: no device can delete visited history from the sync worker any more. The worker used to apply every removal a device sent, and delete the rows outright: one **Clear visited history**, one retention setting above 0, or one import of an older backup on any device wiped those visits from the worker and from every other device, with no way back short of an exported backup. Now the worker only ever adds visits. Clearing history, retention pruning, unmarking a card with `V`, and the entries an import leaves out all apply to the device they happen on, and the worker and the other devices keep their copies. The clear confirmation says so when sync is on. +- The worker still drops its oldest visits beyond 50,000, which bounds how much storage it uses. Per-card overrides are settings rather than history, so removing one still reaches every device. +- Visits a device has cleared or unmarked stay hidden on that device even though the worker still holds them. A card visited again later, on any device, shows as visited everywhere again. +- The worker needs redeploying; the Cloudflare build does that when this reaches `main`. A clear or retention cutoff that an older worker already stored is no longer handed to devices. Visits an older worker already deleted are not restored by this; importing a backup that has them puts them back. + ## 6.9.1 - Fix: a device whose userscript manager lost its visited history but kept its sync settings stayed short of history for good. The panel showed a handful of visits and "synced just now" while the worker still held hundreds. A device remembers how far it has downloaded from the worker, and that bookmark is stored apart from the history. When only the history went, the device kept asking for what was new since the bookmark, so nothing it had lost was ever sent back. The history now carries an id, and the bookmark records which history it belongs to. A history that has lost it (wiped, unreadable, or saved by an older version) is downloaded in full on the next sync, with a "Restored N visited URLs" note, and syncs pick up from the bookmark again after that. Devices updating from 6.9.0 download everything once. Visits recorded since the loss are kept and still go up. diff --git a/README.md b/README.md index ffba530..6b9f449 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,8 @@ Once it is deployed, open the `JVS` panel → **Cloud sync**, enter the worker U - Each device syncs on page load, when its tab regains focus, a few seconds after a visit, on the interval you choose, and whenever you press **Sync now**. The panel's status line reports the last sync or the reason the last one failed. - Devices exchange only what changed: each remembers the sequence number it last saw and the moment it last pushed, so an idle page load costs about 150 bytes each way rather than the whole history. The first sync on a device transfers everything, and so does the next sync on a device whose userscript manager lost its history but kept the sync settings: the history carries an id that the saved position is tied to, so a history without it is downloaded again rather than left behind. - Tabs on one device share a history but not a sync. Before deciding what to send, a sync merges what the device has on disk, so a visit recorded in one tab is not passed over by another tab's sync; and an entry that turns up stamped before the last push—a click replayed from the previous page, a restored backup—pulls the mark back to itself rather than being left behind it. -- The worker applies the same rules as the local merge—newest timestamp wins per URL, tombstones and the clear/retention horizons outrank stale entries—so a device that has been offline for a week cannot overwrite what the others recorded, and clearing history on one device clears it everywhere instead of being undone. A device that pushes a stale entry is handed the winning one back. +- The worker applies the same rules as the local merge—newest timestamp wins per URL—so a device that has been offline for a week cannot overwrite what the others recorded. A device that pushes a stale entry is handed the winning one back. +- No device can delete visited history from the worker. **Clear visited history**, retention pruning, unmarking a card and a backup import that leaves entries out all apply to the device they happen on; the worker and the other devices keep their copies. The worker drops only its oldest visits beyond 50,000, to bound its storage. Per-card overrides are choices rather than history, so removing one still reaches every device. - Because a Durable Object is single-threaded, each sync's read-merge-write is serialized and strongly consistent: two devices syncing in the same second queue behind one another rather than racing. - Settings sync too, including the match and excluded terms, and they merge setting by setting: each one goes to the device that changed it last. A setting a device has never changed carries no timestamp and cannot win, so a new install picks up your keywords from the worker instead of pushing its defaults over them, even if you changed something else on it first. **Restore defaults** and a backup import set every setting on purpose, so they win everywhere. - The Cloud sync panel shows the worker version from the worker's last answer. When the worker is behind the userscript, a **⚠ Redeploy worker** button appears on screen next to the JavStore controls; tap it for the versions. The worker redeploys through Cloudflare whenever `main` changes, so a lasting warning means its Cloudflare build did not go through. When the worker is ahead instead, an **⚠ Update script** button appears on screen and links to the latest release, because the userscript manager has not installed it yet. @@ -43,7 +44,7 @@ Once it is deployed, open the `JVS` panel → **Cloud sync**, enter the worker U - The endpoint and token live only on the device they were entered on. They are never written into the synced document, never appear in an exported backup, and are not left in the page: the token box stays empty once a token is stored and only reports that one exists. - A worker that is unreachable does not affect anything locally: history is still saved to the userscript manager, and the next sync picks it up. - **Sync now** reports both directions—what went up and what came down—so a device that is failing to push is not mistaken for one with nothing to push. -- Restoring a backup keeps the backup's own timestamps and records the entries it drops as removals, so the restore reaches the other devices instead of being discarded by them. Entries a history clear already covers cannot be restored, and the panel says how many were skipped. +- Restoring a backup keeps the backup's own timestamps, so the restored entries reach the worker and the other devices instead of being discarded by them. Entries the backup leaves out are dropped on the importing device only. Entries a history clear on this device already covers cannot be restored, and the panel says how many were skipped. ## Controls diff --git a/javstore-full-layout-cleanup.user.js b/javstore-full-layout-cleanup.user.js index 4feaaea..f216d10 100644 --- a/javstore-full-layout-cleanup.user.js +++ b/javstore-full-layout-cleanup.user.js @@ -1,7 +1,7 @@ // ==UserScript== // @name JavStore Full Layout Cleanup - No Sidebars + Mosaic Overlay // @namespace http://tampermonkey.net/ -// @version 6.9.1 +// @version 6.10.0 // @description Clean up JavStore's layout, filter keyword-matched thumbnails, and track visited items with private, configurable controls. // @homepageURL https://github.com/jitdor/userscript-javstore // @supportURL https://github.com/jitdor/userscript-javstore/issues @@ -29,7 +29,7 @@ (function () { 'use strict'; - const SCRIPT_VERSION = '6.9.1'; + const SCRIPT_VERSION = '6.10.0'; // The same address as @downloadURL: opening it hands the newest release to the // userscript manager, which offers to install it. const INSTALL_URL = 'https://github.com/jitdor/userscript-javstore/releases/latest/download/javstore-full-layout-cleanup.user.js'; @@ -2630,7 +2630,11 @@ importFile.value = ''; }); shadow.querySelector('.clear').addEventListener('click', () => { - if (!window.confirm(`Clear all ${visited.size} visited URLs? This cannot be undone unless you exported a backup.`)) return; + // The worker never deletes visited history, so a clear is this device's alone. + const kept = syncConfigured() + ? 'Only this device is cleared: your sync worker and other devices keep their copies.' + : 'This cannot be undone unless you exported a backup.'; + if (!window.confirm(`Clear all ${visited.size} visited URLs? ${kept}`)) return; resetAt = Date.now(); visited.clear(); for (const key of tombstones.keys()) { diff --git a/tests/sync.test.mjs b/tests/sync.test.mjs index 8f67a13..9f16195 100644 --- a/tests/sync.test.mjs +++ b/tests/sync.test.mjs @@ -57,7 +57,7 @@ test('a visit is pushed without being asked to sync', async () => { await waitFor(() => remote.visited().includes('https://javstore.net/b.html')); }); -test('an item unmarked on one device does not come back from the cloud', async () => { +test('an item unmarked on one device stays unmarked there and nowhere else', async () => { const remote = makeRemote(); const first = await openDevice(remote); clickCard(first.window, '/a.html'); @@ -71,14 +71,15 @@ test('an item unmarked on one device does not come back from the cloud', async ( pressOnCard(first.window, '/a.html', 'v'); await settle(); await syncNow(first); - assert.deepEqual(remote.visited(), []); + assert.deepEqual(remote.visited(), ['https://javstore.net/a.html'], 'the worker keeps the visit'); + assert.deepEqual(visitedUrls(first.store), [], 'the worker does not hand it back'); await syncNow(second); - assert.deepEqual(visitedUrls(secondStore), []); - assert.ok(!card(second.window, '/a.html').classList.contains('jvs-visited')); + assert.deepEqual(visitedUrls(secondStore), ['https://javstore.net/a.html']); + assert.ok(card(second.window, '/a.html').classList.contains('jvs-visited')); }); -test('clearing history on one device clears it everywhere', async () => { +test('clearing history on one device leaves the worker and the other devices alone', async () => { const remote = makeRemote(); const first = await openDevice(remote); clickCard(first.window, '/a.html'); @@ -90,13 +91,32 @@ test('clearing history on one device clears it everywhere', async () => { const second = await openDevice(remote, { store: secondStore }); assert.equal(visitedUrls(secondStore).length, 2); + let asked = ''; + first.window.confirm = message => { asked = message; return true; }; first.shadow().querySelector('.clear').click(); await settle(); + assert.match(asked, /Only this device is cleared/); await syncNow(first); - assert.deepEqual(remote.visited(), []); + assert.equal(remote.visited().length, 2); + assert.deepEqual(visitedUrls(first.store), []); await syncNow(second); - assert.deepEqual(visitedUrls(secondStore), []); + assert.equal(visitedUrls(secondStore).length, 2); + + // Neither a reload nor a full download brings the cleared visits back to this device. + closeTabs(); + const store = first.store; + store.write('javstore_sync_config_v1', { ...store.read('javstore_sync_config_v1'), cursor: 0 }); + await openTab(store, { html: listing(), remote }); + await settle(); + assert.deepEqual(visitedUrls(store), []); + + // A card visited again afterwards counts, here and everywhere. + const again = await openTab(store, { html: listing(), remote }); + clickCard(again.window, '/a.html'); + await settle(); + await syncNow(again); + assert.deepEqual(visitedUrls(store), ['https://javstore.net/a.html']); }); test('a device that has been offline cannot overwrite what the others recorded', async () => { @@ -396,25 +416,19 @@ test('the cursor does not move past history that could not be saved', async () = test('a merge that only brings back a tombstone is still saved', async () => { const remote = makeRemote(); - const first = await openDevice(remote); - clickCard(first.window, '/a.html'); - await settle(); - await syncNow(first); - pressOnCard(first.window, '/a.html', 'v'); - await settle(); - await syncNow(first); + const url = 'https://javstore.net/a.html'; + const at = Date.now() - 1000; + remote.store.sync({ cursor: 0, changes: [{ kind: 'o', key: url, at, value: 'allow' }] }); + remote.store.sync({ cursor: 0, changes: [{ kind: 'o', key: url, at: at + 1, deleted: 1 }] }); - // This device has never seen /a.html, so the visit and the deletion that follows it - // cancel out: nothing on screen changes, but the tombstone that keeps the visit from + // This device has never seen the override, so it and the removal that follows it + // cancel out: nothing on screen changes, but the tombstone that keeps the override from // coming back has to survive the reload the cursor was recorded for. const secondStore = makeStore(); await openDevice(remote, { store: secondStore }); await waitFor(() => secondStore.state()); - assert.deepEqual(visitedUrls(secondStore), []); - assert.ok( - secondStore.state().tombstones['v|https://javstore.net/a.html'], - 'the tombstone reached storage', - ); + assert.deepEqual(secondStore.state().overrides, {}); + assert.ok(secondStore.state().tombstones[`o|${url}`], 'the tombstone reached storage'); assert.ok(secondStore.read('javstore_sync_config_v1').cursor > 0); }); @@ -660,7 +674,7 @@ test('the worker refuses a request without the right token', async () => { assert.deepEqual(remote.visited(), ['https://javstore.net/a.html']); }); -test('the worker keeps the newest timestamp and honours tombstones', async () => { +test('the worker keeps the newest timestamp and ignores visit deletions', async () => { const remote = makeRemote(); const sync = payload => remote.store.sync(payload, 1000); @@ -671,13 +685,44 @@ test('the worker keeps the newest timestamp and honours tombstones', async () => { kind: 'v', key: '/a', at: 400 }, { kind: 'v', key: '/b', at: 200 }, { kind: 'v', key: '/c', at: 300 }, - { kind: 'v', key: '/c', at: 300, deleted: 1 }, + { kind: 'v', key: '/c', at: 350, deleted: 1 }, + { kind: 'o', key: '/d', at: 300, value: 'allow' }, + { kind: 'o', key: '/d', at: 350, deleted: 1 }, ], }); const document = remote.state(); - assert.deepEqual(document.visited, { '/a': 400, '/b': 500 }); - assert.equal(document.tombstones['v|/c'], 300); + assert.deepEqual(document.visited, { '/a': 400, '/b': 500, '/c': 300 }); + assert.equal(document.tombstones['v|/c'], undefined); + assert.deepEqual(document.overrides, {}, 'removing an override still travels'); + assert.equal(document.tombstones['o|/d'], 350); +}); + +test('no device can clear or prune the worker', async () => { + const remote = makeRemote(); + remote.store.sync({ cursor: 0, changes: [{ kind: 'v', key: '/a', at: 100 }, { kind: 'v', key: '/b', at: 200 }] }); + const answer = remote.store.sync({ cursor: 0, meta: { resetAt: 5000, prunedBefore: 4000 }, changes: [] }); + + assert.deepEqual(remote.state().visited, { '/a': 100, '/b': 200 }); + assert.equal(answer.meta.resetAt, 0); + assert.equal(answer.meta.prunedBefore, 0); + assert.equal(remote.state().resetAt, 0); + + // The whole-document exchange of a device on 6.2.0 is held to the same rule. + remote.store.applyDocument({ resetAt: 5000, visited: {}, tombstones: { 'v|/a': 6000 } }, 7000); + assert.deepEqual(remote.state().visited, { '/a': 100, '/b': 200 }); +}); + +test('what an older worker stored to delete visits no longer holds them back', async () => { + const remote = makeRemote(); + // A clear, and a tombstone for /b, as a worker before 6.10.0 recorded them. + remote.store.sql.exec("INSERT INTO meta (name, value) VALUES ('resetAt', '1000')"); + remote.store.sql.exec("INSERT INTO meta (name, value) VALUES ('seq', '1')"); + remote.store.sql.exec("INSERT INTO entries (kind, key, at, deleted, value, seq) VALUES ('v', '/b', 2000, 1, NULL, 1)"); + + const answer = remote.store.sync({ cursor: 0, changes: [{ kind: 'v', key: '/a', at: 500 }, { kind: 'v', key: '/b', at: 1500 }] }); + assert.equal(answer.meta.resetAt, 0, 'the stored clear is not handed to devices'); + assert.deepEqual(remote.state().visited, { '/a': 500, '/b': 1500 }); }); test('a device that is behind is told the winning entry it missed', async () => { @@ -862,7 +907,7 @@ test('an imported backup reaches the other devices', async () => { assert.deepEqual(visitedUrls(elsewhere), ['https://javstore.net/a.html']); }); -test('an import records what it drops so the removal travels too', async () => { +test('an import drops what it leaves out on the importing device only', async () => { const remote = makeRemote(); const first = await openDevice(remote); clickCard(first.window, '/a.html'); @@ -882,7 +927,7 @@ test('an import records what it drops so the removal travels too', async () => { text: async () => JSON.stringify({ version: 3, updatedAt: at, - visited: { 'https://javstore.net/a.html': at }, + visited: { 'https://javstore.net/a.html': at, 'https://javstore.net/c.html': at }, overrides: {}, overrideTimes: {}, tombstones: {}, @@ -893,9 +938,14 @@ test('an import records what it drops so the removal travels too', async () => { await settle(); await syncNow(second); - assert.deepEqual(remote.visited(), ['https://javstore.net/a.html']); + assert.deepEqual(visitedUrls(secondStore), ['https://javstore.net/a.html', 'https://javstore.net/c.html']); + assert.deepEqual(remote.visited(), [ + 'https://javstore.net/a.html', 'https://javstore.net/b.html', 'https://javstore.net/c.html', + ]); await syncNow(first); - assert.deepEqual(visitedUrls(first.store), ['https://javstore.net/a.html']); + assert.deepEqual(visitedUrls(first.store), [ + 'https://javstore.net/a.html', 'https://javstore.net/b.html', 'https://javstore.net/c.html', + ]); }); test('a visit recorded while a sync is in flight is not declared pushed', async () => { diff --git a/worker/README.md b/worker/README.md index 5999e70..13667dd 100644 --- a/worker/README.md +++ b/worker/README.md @@ -103,12 +103,16 @@ panel and flags a worker that is behind it. A delta `POST` hands over the entries the device has touched since its last push and asks for everything recorded after `cursor`. Entries are last-writer-wins on their own event -timestamp, with a deletion winning a tie; `resetAt` and `prunedBefore` in `meta` are horizons -below which nothing survives. A device that pushes an entry older than the stored one is -handed the winner back, so it cannot stay wrong about it. +timestamp, with a deletion winning a tie. A device that pushes an entry older than the +stored one is handed the winner back, so it cannot stay wrong about it. -The object keeps up to 50,000 visited entries (a device keeps its newest 5,000) and expires -tombstones after 90 days. +Visited history is append-only: the worker ignores a device's visit deletions and its +`resetAt` and `prunedBefore` horizons (from 6.10.0), and always reports both horizons as 0. +Clearing history, retention pruning, unmarking a card and a backup import therefore change +only the device they happen on. Per-card override removals still apply. + +The object keeps up to 50,000 visited entries, dropping the oldest beyond that (a device +keeps its newest 5,000), and expires override tombstones after 90 days. ## Notes diff --git a/worker/src/worker.mjs b/worker/src/worker.mjs index 630325f..6b92754 100644 --- a/worker/src/worker.mjs +++ b/worker/src/worker.mjs @@ -13,11 +13,17 @@ // // Every entry is last-writer-wins on its own event timestamp, with a deletion winning a tie // — the same rule the userscript applies locally, so both sides converge on the same state. +// +// Visited history is append-only here: no device can delete it. A cleared history, a +// retention cutoff, an unmarked card or a backup import that drops entries all stay on the +// device that did it, and the worker keeps its copy. The only visits that ever leave are +// the oldest ones beyond MAX_ARCHIVED_VISITS, which bounds the storage the object uses. +// Per-card overrides are choices rather than history, so removing one still travels. const STORAGE_VERSION = 3; // Reported in every answer so the userscript can show which worker it is talking to. It // moves in step with the userscript's own version; a test holds the two together. -export const WORKER_VERSION = '6.9.1'; +export const WORKER_VERSION = '6.10.0'; const OBJECT_NAME = 'default'; const MAX_BODY_BYTES = 8 * 1024 * 1024; const MAX_PAGE_ROWS = 2000; @@ -117,41 +123,25 @@ export class SyncStore { return next; } - #horizons() { - return { - resetAt: timestamp(this.#meta('resetAt', 0)), - prunedBefore: timestamp(this.#meta('prunedBefore', 0)), - }; - } - + // A device merges any horizon it is handed and drops every visit stamped before it, so + // the worker reports none. One a worker from before 6.10.0 stored is left where it is, + // unread, rather than going on clearing each device that syncs. #readMeta() { return { - resetAt: timestamp(this.#meta('resetAt', 0)), - prunedBefore: timestamp(this.#meta('prunedBefore', 0)), + resetAt: 0, + prunedBefore: 0, settings: this.#meta('settings', null), settingsUpdatedAt: timestamp(this.#meta('settingsUpdatedAt', 0)), settingTimes: this.#meta('settingTimes', null), }; } - // The horizons only ever move forward, and dropping what falls below them is what stops - // a device that has been offline from resurrecting a cleared history. + // A device's `resetAt` and `prunedBefore` are ignored: clearing history and retention + // pruning apply to that device alone, and the archive keeps everything. #applyMeta(meta) { if (!meta || typeof meta !== 'object') return; const current = this.#readMeta(); - const resetAt = timestamp(meta.resetAt); - if (resetAt > current.resetAt) { - this.#setMeta('resetAt', resetAt); - this.sql.exec('DELETE FROM entries WHERE kind = ? AND deleted = 0 AND at <= ?', 'v', resetAt); - } - - const prunedBefore = timestamp(meta.prunedBefore); - if (prunedBefore > current.prunedBefore) { - this.#setMeta('prunedBefore', prunedBefore); - this.sql.exec('DELETE FROM entries WHERE kind = ? AND deleted = 0 AND at <= ?', 'v', prunedBefore); - } - // Each setting carries its own timestamp, so a device that has never changed one // sends a zero for it and can never push its default over another device's choice, // and a device that changed only its tint does not bring its keywords along. @@ -192,13 +182,15 @@ export class SyncStore { const value = kind === 'o' && !deleted ? overrideValue(change.value) : null; if (!key || !at) return null; if (kind === 'o' && !deleted && !value) return null; - - const { resetAt, prunedBefore } = this.#horizons(); - if (kind === 'v' && !deleted && (at <= resetAt || at <= prunedBefore)) return null; + // A device removing a visit—unmarking a card, or an import that leaves it out— + // removes it from that device only. + if (kind === 'v' && deleted) return null; const existing = this.#existing(kind, key); if (existing) { - const wins = at > existing.at || (at === existing.at && deleted && !existing.deleted); + // A visit tombstone a worker from before 6.10.0 stored never outranks a visit. + const buried = kind === 'v' && existing.deleted; + const wins = buried || at > existing.at || (at === existing.at && deleted && !existing.deleted); if (!wins) { const agreed = existing.at === at && Boolean(existing.deleted) === Boolean(deleted)