Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
# Changelog

## 6.10.0

- Change: no device can delete visited history from the sync worker any more. The worker used to apply every removal a device sent, and delete the rows outright: one **Clear visited history**, one retention setting above 0, or one import of an older backup on any device wiped those visits from the worker and from every other device, with no way back short of an exported backup. Now the worker only ever adds visits. Clearing history, retention pruning, unmarking a card with `V`, and the entries an import leaves out all apply to the device they happen on, and the worker and the other devices keep their copies. The clear confirmation says so when sync is on.
- The worker still drops its oldest visits beyond 50,000, which bounds how much storage it uses. Per-card overrides are settings rather than history, so removing one still reaches every device.
- Visits a device has cleared or unmarked stay hidden on that device even though the worker still holds them. A card visited again later, on any device, shows as visited everywhere again.
- The worker needs redeploying; the Cloudflare build does that when this reaches `main`. A clear or retention cutoff that an older worker already stored is no longer handed to devices. Visits an older worker already deleted are not restored by this; importing a backup that has them puts them back.

## 6.9.1

- Fix: a device whose userscript manager lost its visited history but kept its sync settings stayed short of history for good. The panel showed a handful of visits and "synced just now" while the worker still held hundreds. A device remembers how far it has downloaded from the worker, and that bookmark is stored apart from the history. When only the history went, the device kept asking for what was new since the bookmark, so nothing it had lost was ever sent back. The history now carries an id, and the bookmark records which history it belongs to. A history that has lost it (wiped, unreadable, or saved by an older version) is downloaded in full on the next sync, with a "Restored N visited URLs" note, and syncs pick up from the bookmark again after that. Devices updating from 6.9.0 download everything once. Visits recorded since the loss are kept and still go up.
Expand Down
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,15 +35,16 @@ Once it is deployed, open the `JVS` panel → **Cloud sync**, enter the worker U
- Each device syncs on page load, when its tab regains focus, a few seconds after a visit, on the interval you choose, and whenever you press **Sync now**. The panel's status line reports the last sync or the reason the last one failed.
- Devices exchange only what changed: each remembers the sequence number it last saw and the moment it last pushed, so an idle page load costs about 150 bytes each way rather than the whole history. The first sync on a device transfers everything, and so does the next sync on a device whose userscript manager lost its history but kept the sync settings: the history carries an id that the saved position is tied to, so a history without it is downloaded again rather than left behind.
- Tabs on one device share a history but not a sync. Before deciding what to send, a sync merges what the device has on disk, so a visit recorded in one tab is not passed over by another tab's sync; and an entry that turns up stamped before the last push—a click replayed from the previous page, a restored backup—pulls the mark back to itself rather than being left behind it.
- The worker applies the same rules as the local merge—newest timestamp wins per URL, tombstones and the clear/retention horizons outrank stale entries—so a device that has been offline for a week cannot overwrite what the others recorded, and clearing history on one device clears it everywhere instead of being undone. A device that pushes a stale entry is handed the winning one back.
- The worker applies the same rules as the local merge—newest timestamp wins per URL—so a device that has been offline for a week cannot overwrite what the others recorded. A device that pushes a stale entry is handed the winning one back.
- No device can delete visited history from the worker. **Clear visited history**, retention pruning, unmarking a card and a backup import that leaves entries out all apply to the device they happen on; the worker and the other devices keep their copies. The worker drops only its oldest visits beyond 50,000, to bound its storage. Per-card overrides are choices rather than history, so removing one still reaches every device.
- Because a Durable Object is single-threaded, each sync's read-merge-write is serialized and strongly consistent: two devices syncing in the same second queue behind one another rather than racing.
- Settings sync too, including the match and excluded terms, and they merge setting by setting: each one goes to the device that changed it last. A setting a device has never changed carries no timestamp and cannot win, so a new install picks up your keywords from the worker instead of pushing its defaults over them, even if you changed something else on it first. **Restore defaults** and a backup import set every setting on purpose, so they win everywhere.
- The Cloud sync panel shows the worker version from the worker's last answer. When the worker is behind the userscript, a **⚠ Redeploy worker** button appears on screen next to the JavStore controls; tap it for the versions. The worker redeploys through Cloudflare whenever `main` changes, so a lasting warning means its Cloudflare build did not go through. When the worker is ahead instead, an **⚠ Update script** button appears on screen and links to the latest release, because the userscript manager has not installed it yet.
- AdGuard for Android clears a userscript's storage whenever it installs an update. After setting sync up, press **Copy sync link** and keep the link somewhere private, such as a password manager. After a wipe, open it on JavStore, or paste it into the endpoint box and save, to set everything up again. The link carries your token, so treat it like a password.
- The endpoint and token live only on the device they were entered on. They are never written into the synced document, never appear in an exported backup, and are not left in the page: the token box stays empty once a token is stored and only reports that one exists.
- A worker that is unreachable does not affect anything locally: history is still saved to the userscript manager, and the next sync picks it up.
- **Sync now** reports both directions—what went up and what came down—so a device that is failing to push is not mistaken for one with nothing to push.
- Restoring a backup keeps the backup's own timestamps and records the entries it drops as removals, so the restore reaches the other devices instead of being discarded by them. Entries a history clear already covers cannot be restored, and the panel says how many were skipped.
- Restoring a backup keeps the backup's own timestamps, so the restored entries reach the worker and the other devices instead of being discarded by them. Entries the backup leaves out are dropped on the importing device only. Entries a history clear on this device already covers cannot be restored, and the panel says how many were skipped.

## Controls

Expand Down
10 changes: 7 additions & 3 deletions javstore-full-layout-cleanup.user.js
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
// ==UserScript==
// @name JavStore Full Layout Cleanup - No Sidebars + Mosaic Overlay
// @namespace http://tampermonkey.net/
// @version 6.9.1
// @version 6.10.0
// @description Clean up JavStore's layout, filter keyword-matched thumbnails, and track visited items with private, configurable controls.
// @homepageURL https://github.com/jitdor/userscript-javstore
// @supportURL https://github.com/jitdor/userscript-javstore/issues
Expand Down Expand Up @@ -29,7 +29,7 @@
(function () {
'use strict';

const SCRIPT_VERSION = '6.9.1';
const SCRIPT_VERSION = '6.10.0';
// The same address as @downloadURL: opening it hands the newest release to the
// userscript manager, which offers to install it.
const INSTALL_URL = 'https://github.com/jitdor/userscript-javstore/releases/latest/download/javstore-full-layout-cleanup.user.js';
Expand Down Expand Up @@ -2630,7 +2630,11 @@
importFile.value = '';
});
shadow.querySelector('.clear').addEventListener('click', () => {
if (!window.confirm(`Clear all ${visited.size} visited URLs? This cannot be undone unless you exported a backup.`)) return;
// The worker never deletes visited history, so a clear is this device's alone.
const kept = syncConfigured()
? 'Only this device is cleared: your sync worker and other devices keep their copies.'
: 'This cannot be undone unless you exported a backup.';
Comment on lines +2634 to +2636

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Warn before clearing through an older worker

When sync is configured but the worker has not yet been redeployed to 6.10.0, this confirmation falsely promises that the worker and other devices will keep their copies. The clear still sends the new resetAt through persistState(), and a pre-6.10 worker applies that horizon by deleting its visits and propagating the clear globally. Only show this reassurance after confirming a worker version with append-only behavior; otherwise warn that clearing may affect every device or require redeployment first.

Useful? React with 👍 / 👎.

if (!window.confirm(`Clear all ${visited.size} visited URLs? ${kept}`)) return;
resetAt = Date.now();
visited.clear();
for (const key of tombstones.keys()) {
Expand Down
108 changes: 79 additions & 29 deletions tests/sync.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ test('a visit is pushed without being asked to sync', async () => {
await waitFor(() => remote.visited().includes('https://javstore.net/b.html'));
});

test('an item unmarked on one device does not come back from the cloud', async () => {
test('an item unmarked on one device stays unmarked there and nowhere else', async () => {
const remote = makeRemote();
const first = await openDevice(remote);
clickCard(first.window, '/a.html');
Expand All @@ -71,14 +71,15 @@ test('an item unmarked on one device does not come back from the cloud', async (
pressOnCard(first.window, '/a.html', 'v');
await settle();
await syncNow(first);
assert.deepEqual(remote.visited(), []);
assert.deepEqual(remote.visited(), ['https://javstore.net/a.html'], 'the worker keeps the visit');
assert.deepEqual(visitedUrls(first.store), [], 'the worker does not hand it back');

await syncNow(second);
assert.deepEqual(visitedUrls(secondStore), []);
assert.ok(!card(second.window, '/a.html').classList.contains('jvs-visited'));
assert.deepEqual(visitedUrls(secondStore), ['https://javstore.net/a.html']);
assert.ok(card(second.window, '/a.html').classList.contains('jvs-visited'));
});

test('clearing history on one device clears it everywhere', async () => {
test('clearing history on one device leaves the worker and the other devices alone', async () => {
const remote = makeRemote();
const first = await openDevice(remote);
clickCard(first.window, '/a.html');
Expand All @@ -90,13 +91,32 @@ test('clearing history on one device clears it everywhere', async () => {
const second = await openDevice(remote, { store: secondStore });
assert.equal(visitedUrls(secondStore).length, 2);

let asked = '';
first.window.confirm = message => { asked = message; return true; };
first.shadow().querySelector('.clear').click();
await settle();
assert.match(asked, /Only this device is cleared/);
await syncNow(first);
assert.deepEqual(remote.visited(), []);
assert.equal(remote.visited().length, 2);
assert.deepEqual(visitedUrls(first.store), []);

await syncNow(second);
assert.deepEqual(visitedUrls(secondStore), []);
assert.equal(visitedUrls(secondStore).length, 2);

// Neither a reload nor a full download brings the cleared visits back to this device.
closeTabs();
const store = first.store;
store.write('javstore_sync_config_v1', { ...store.read('javstore_sync_config_v1'), cursor: 0 });
await openTab(store, { html: listing(), remote });
await settle();
assert.deepEqual(visitedUrls(store), []);

// A card visited again afterwards counts, here and everywhere.
const again = await openTab(store, { html: listing(), remote });
clickCard(again.window, '/a.html');
await settle();
await syncNow(again);
assert.deepEqual(visitedUrls(store), ['https://javstore.net/a.html']);
});

test('a device that has been offline cannot overwrite what the others recorded', async () => {
Expand Down Expand Up @@ -396,25 +416,19 @@ test('the cursor does not move past history that could not be saved', async () =

test('a merge that only brings back a tombstone is still saved', async () => {
const remote = makeRemote();
const first = await openDevice(remote);
clickCard(first.window, '/a.html');
await settle();
await syncNow(first);
pressOnCard(first.window, '/a.html', 'v');
await settle();
await syncNow(first);
const url = 'https://javstore.net/a.html';
const at = Date.now() - 1000;
remote.store.sync({ cursor: 0, changes: [{ kind: 'o', key: url, at, value: 'allow' }] });
remote.store.sync({ cursor: 0, changes: [{ kind: 'o', key: url, at: at + 1, deleted: 1 }] });

// This device has never seen /a.html, so the visit and the deletion that follows it
// cancel out: nothing on screen changes, but the tombstone that keeps the visit from
// This device has never seen the override, so it and the removal that follows it
// cancel out: nothing on screen changes, but the tombstone that keeps the override from
// coming back has to survive the reload the cursor was recorded for.
const secondStore = makeStore();
await openDevice(remote, { store: secondStore });
await waitFor(() => secondStore.state());
assert.deepEqual(visitedUrls(secondStore), []);
assert.ok(
secondStore.state().tombstones['v|https://javstore.net/a.html'],
'the tombstone reached storage',
);
assert.deepEqual(secondStore.state().overrides, {});
assert.ok(secondStore.state().tombstones[`o|${url}`], 'the tombstone reached storage');
assert.ok(secondStore.read('javstore_sync_config_v1').cursor > 0);
});

Expand Down Expand Up @@ -660,7 +674,7 @@ test('the worker refuses a request without the right token', async () => {
assert.deepEqual(remote.visited(), ['https://javstore.net/a.html']);
});

test('the worker keeps the newest timestamp and honours tombstones', async () => {
test('the worker keeps the newest timestamp and ignores visit deletions', async () => {
const remote = makeRemote();
const sync = payload => remote.store.sync(payload, 1000);

Expand All @@ -671,13 +685,44 @@ test('the worker keeps the newest timestamp and honours tombstones', async () =>
{ kind: 'v', key: '/a', at: 400 },
{ kind: 'v', key: '/b', at: 200 },
{ kind: 'v', key: '/c', at: 300 },
{ kind: 'v', key: '/c', at: 300, deleted: 1 },
{ kind: 'v', key: '/c', at: 350, deleted: 1 },
{ kind: 'o', key: '/d', at: 300, value: 'allow' },
{ kind: 'o', key: '/d', at: 350, deleted: 1 },
],
});

const document = remote.state();
assert.deepEqual(document.visited, { '/a': 400, '/b': 500 });
assert.equal(document.tombstones['v|/c'], 300);
assert.deepEqual(document.visited, { '/a': 400, '/b': 500, '/c': 300 });
assert.equal(document.tombstones['v|/c'], undefined);
assert.deepEqual(document.overrides, {}, 'removing an override still travels');
assert.equal(document.tombstones['o|/d'], 350);
});

test('no device can clear or prune the worker', async () => {
const remote = makeRemote();
remote.store.sync({ cursor: 0, changes: [{ kind: 'v', key: '/a', at: 100 }, { kind: 'v', key: '/b', at: 200 }] });
const answer = remote.store.sync({ cursor: 0, meta: { resetAt: 5000, prunedBefore: 4000 }, changes: [] });

assert.deepEqual(remote.state().visited, { '/a': 100, '/b': 200 });
assert.equal(answer.meta.resetAt, 0);
assert.equal(answer.meta.prunedBefore, 0);
assert.equal(remote.state().resetAt, 0);

// The whole-document exchange of a device on 6.2.0 is held to the same rule.
remote.store.applyDocument({ resetAt: 5000, visited: {}, tombstones: { 'v|/a': 6000 } }, 7000);
assert.deepEqual(remote.state().visited, { '/a': 100, '/b': 200 });
});

test('what an older worker stored to delete visits no longer holds them back', async () => {
const remote = makeRemote();
// A clear, and a tombstone for /b, as a worker before 6.10.0 recorded them.
remote.store.sql.exec("INSERT INTO meta (name, value) VALUES ('resetAt', '1000')");
remote.store.sql.exec("INSERT INTO meta (name, value) VALUES ('seq', '1')");
remote.store.sql.exec("INSERT INTO entries (kind, key, at, deleted, value, seq) VALUES ('v', '/b', 2000, 1, NULL, 1)");

const answer = remote.store.sync({ cursor: 0, changes: [{ kind: 'v', key: '/a', at: 500 }, { kind: 'v', key: '/b', at: 1500 }] });
assert.equal(answer.meta.resetAt, 0, 'the stored clear is not handed to devices');
assert.deepEqual(remote.state().visited, { '/a': 500, '/b': 1500 });
});

test('a device that is behind is told the winning entry it missed', async () => {
Expand Down Expand Up @@ -862,7 +907,7 @@ test('an imported backup reaches the other devices', async () => {
assert.deepEqual(visitedUrls(elsewhere), ['https://javstore.net/a.html']);
});

test('an import records what it drops so the removal travels too', async () => {
test('an import drops what it leaves out on the importing device only', async () => {
const remote = makeRemote();
const first = await openDevice(remote);
clickCard(first.window, '/a.html');
Expand All @@ -882,7 +927,7 @@ test('an import records what it drops so the removal travels too', async () => {
text: async () => JSON.stringify({
version: 3,
updatedAt: at,
visited: { 'https://javstore.net/a.html': at },
visited: { 'https://javstore.net/a.html': at, 'https://javstore.net/c.html': at },
overrides: {},
overrideTimes: {},
tombstones: {},
Expand All @@ -893,9 +938,14 @@ test('an import records what it drops so the removal travels too', async () => {
await settle();
await syncNow(second);

assert.deepEqual(remote.visited(), ['https://javstore.net/a.html']);
assert.deepEqual(visitedUrls(secondStore), ['https://javstore.net/a.html', 'https://javstore.net/c.html']);
assert.deepEqual(remote.visited(), [
'https://javstore.net/a.html', 'https://javstore.net/b.html', 'https://javstore.net/c.html',
]);
await syncNow(first);
assert.deepEqual(visitedUrls(first.store), ['https://javstore.net/a.html']);
assert.deepEqual(visitedUrls(first.store), [
'https://javstore.net/a.html', 'https://javstore.net/b.html', 'https://javstore.net/c.html',
]);
});

test('a visit recorded while a sync is in flight is not declared pushed', async () => {
Expand Down
14 changes: 9 additions & 5 deletions worker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,12 +103,16 @@ panel and flags a worker that is behind it.

A delta `POST` hands over the entries the device has touched since its last push and asks
for everything recorded after `cursor`. Entries are last-writer-wins on their own event
timestamp, with a deletion winning a tie; `resetAt` and `prunedBefore` in `meta` are horizons
below which nothing survives. A device that pushes an entry older than the stored one is
handed the winner back, so it cannot stay wrong about it.
timestamp, with a deletion winning a tie. A device that pushes an entry older than the
stored one is handed the winner back, so it cannot stay wrong about it.

The object keeps up to 50,000 visited entries (a device keeps its newest 5,000) and expires
tombstones after 90 days.
Visited history is append-only: the worker ignores a device's visit deletions and its
`resetAt` and `prunedBefore` horizons (from 6.10.0), and always reports both horizons as 0.
Clearing history, retention pruning, unmarking a card and a backup import therefore change
only the device they happen on. Per-card override removals still apply.

The object keeps up to 50,000 visited entries, dropping the oldest beyond that (a device
keeps its newest 5,000), and expires override tombstones after 90 days.

## Notes

Expand Down
Loading
Loading