From 6d02cf6b457844c733eaf820f897ed7b900d0576 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 14 Nov 2026 21:02:44 +0000 Subject: [PATCH 1/4] Store large values in parts that AdGuard keeps (6.13.1) AdGuard sends each GM_setValue to its app as one request with the value URL-encoded in it, and drops a value too large for that request without telling the script, while the writing page still reads it back. Past a few hundred visits the history survived no page load at all, so every page downloaded it again with "Restored N visited URLs". Values larger than the device keeps are stored as parts under that size, plus a pointer written last; a value with a missing part reads as missing. Each tab trims the part sets it replaced (keeping two), and parts nothing refers to are removed 30 minutes after they were written. The device's limit is measured with probe values (2/8/32/128 KB) judged on the next page load, and shown in the panel; until known, parts are about 6 KB. The harness gains maxEncodedValue (JVS_VALUE_LIMIT) to model this; under an 8 KB limit 6.13.0 fails 12 tests and this passes all 144. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VCuhK4TQqcEbLXdmxLjggy --- CHANGELOG.md | 8 + README.md | 1 + javstore-full-layout-cleanup.user.js | 275 ++++++++++++++++++++++++++- tests/harness.mjs | 25 ++- tests/review-final.test.mjs | 8 +- tests/storage-size.test.mjs | 141 ++++++++++++++ worker/src/worker.mjs | 2 +- 7 files changed, 445 insertions(+), 15 deletions(-) create mode 100644 tests/storage-size.test.mjs diff --git a/CHANGELOG.md b/CHANGELOG.md index 24085c1..c44cb17 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,13 @@ # Changelog +## 6.13.1 + +- Fix: on AdGuard, "Restored N visited URLs from the worker" appeared on every page load, and every page downloaded the whole history again. 6.12.0 removed one cause; this fixes the one that remained. AdGuard sends each stored value to its app as a single request, with the value URL-encoded inside it. A value too large for that request is not kept, and the script is not told: the page that saved it still reads it back from its own copy. Once the history grew past a few hundred visits it no longer fit, so it survived no page load at all. Small values, such as the sync settings, were kept, which is why sync kept working and the problem looked like something else. +- Change: a value larger than the device keeps is now stored as parts, each small enough, plus a short pointer to them, written last. A value with a part missing is read as missing, never as a shorter history. Each tab removes the part sets it replaced, keeping the latest two in case the newest pointer never landed. Parts that nothing refers to are removed half an hour after they were written. +- Add: the script measures how large a value the device keeps. It stores test values of 2, 8, 32 and 128 KB, checks on the next page load which came back, uses the largest, and measures again weekly. Until it knows, it uses parts of about 6 KB. The settings panel shows the result, and warns if a stored value was found with parts missing. Managers that report other tabs' writes, such as Tampermonkey, are not measured: they have no such limit. +- The test suites can now run against a manager that drops oversized values (`JVS_VALUE_LIMIT=8000 npm test`). Under that limit, 6.13.0 fails 12 tests; this release passes all of them. +- The worker version moves to 6.13.1 with the userscript; the worker is unchanged. + ## 6.13.0 - Add: a live sync status next to the JavStore controls, so you can tell when a sync is still running and wait before opening new tabs. A tab opened during a long sync starts from the history as it was before the sync, and on AdGuard it then downloads the whole history again itself. diff --git a/README.md b/README.md index d15795f..ade87cb 100644 --- a/README.md +++ b/README.md @@ -67,6 +67,7 @@ Once it is deployed, open the `JVS` panel → **Cloud sync**, enter the worker U - Cloud sync additionally uses `GM_xmlhttpRequest` with `@connect *`, because the worker URL is yours to choose. It is only called once sync is switched on; engines without it fall back to `fetch`, which the worker's CORS headers allow. - Optional userscript APIs: `GM_addValueChangeListener` and `GM_registerMenuCommand`. AdGuard does not implement either; the script polls for changes instead of being notified, and its own on-page control replaces the manager menu. - `GM_getValue`/`GM_setValue` are supported in both the synchronous Tampermonkey style and the asynchronous GM4 style AdGuard uses. Values are stored as JSON strings, because GM4 only promises to keep strings, numbers and booleans, and AdGuard for Android drops stored objects when the page reloads. +- AdGuard does not keep a stored value that is too large to fit in the single request it sends to its app, and does not say so. The script measures how large a value the device keeps (the settings panel shows it) and stores anything larger as parts behind a short pointer. - `GM_listValues` and `GM_deleteValue` let each page keep its own journal of what it recorded, so tabs saving at the same moment cannot overwrite each other's visits. AdGuard and Tampermonkey provide both; an engine without them falls back to a single stored document. - AdGuard (for Mac, Windows and Android) gives each tab the stored values as they were when the tab loaded, and never shows it what other tabs save afterwards. On managers without `GM_addValueChangeListener`, a tab open for more than eight minutes therefore saves only to its own journal and leaves the shared history to newer tabs, which fold its journal in. Journal keys are each written for five minutes at most and removed only once a newer tab has found them in the saved history, so a long-open tab can neither write an older history over a newer one nor remove another tab's journal. Long-open tabs do not show what other tabs recorded after they loaded until they sync or are reloaded. diff --git a/javstore-full-layout-cleanup.user.js b/javstore-full-layout-cleanup.user.js index dbb426f..fcb253b 100644 --- a/javstore-full-layout-cleanup.user.js +++ b/javstore-full-layout-cleanup.user.js @@ -1,7 +1,7 @@ // ==UserScript== // @name JavStore Full Layout Cleanup - No Sidebars + Mosaic Overlay // @namespace http://tampermonkey.net/ -// @version 6.13.0 +// @version 6.13.1 // @description Clean up JavStore's layout, filter keyword-matched thumbnails, and track visited items with private, configurable controls. // @homepageURL https://github.com/jitdor/userscript-javstore // @supportURL https://github.com/jitdor/userscript-javstore/issues @@ -29,7 +29,7 @@ (function () { 'use strict'; - const SCRIPT_VERSION = '6.13.0'; + const SCRIPT_VERSION = '6.13.1'; // The same address as @downloadURL: opening it hands the newest release to the // userscript manager, which offers to install it. const INSTALL_URL = 'https://github.com/jitdor/userscript-javstore/releases/latest/download/javstore-full-layout-cleanup.user.js'; @@ -401,12 +401,266 @@ } } + // ------------------------------------------------------------------ + // Values too large to store in one piece + // + // AdGuard sends each GM_setValue to its app as one request, with the value URL-encoded + // in it, and a value too large for that request is not kept—without the script being + // told, and while the page that wrote it still reads it back from its own copy. Past a + // few hundred visits the history no longer fit, so on AdGuard it survived no page load + // at all, and every page downloaded it again with a "Restored N" note. + // + // So a value larger than the device keeps is stored as parts, each small enough, and + // then a short pointer to them under its own key. The pointer is written last: until it + // lands, whatever it replaces is still whole. A value whose parts are not all there is + // read as missing. How large a value this device keeps is measured on the device (see + // the storage probe below); until then parts stay small. + // ------------------------------------------------------------------ + + const PARTS_MARK = '__jvsParts'; + const PART_KEY_PREFIX = 'javstore_part_'; + // In URL-encoded characters: what any engine is assumed to keep until measured. + const DEFAULT_VALUE_LIMIT = 6000; + // Parts nothing refers to are removed once they are this much older than the page + // that finds them, so no write still on its way can be referring to them. + const PART_RETIRE_MS = 30 * 60000; + let valueLimit = DEFAULT_VALUE_LIMIT; + let partSequence = 0; + // Part sets this page has seen referred to, by their id. + const referencedParts = new Set(); + // Keys whose parts were not all there when read. + const incompleteValues = new Set(); + // For each key, the part sets this page has read or written for it, oldest first. + const partSets = new Map(); + + function notePartSet(key, set) { + const sets = partSets.get(key) || []; + if (!set || !sets.some(known => known.id === set.id)) sets.push(set); + partSets.set(key, sets); + } + + // Once a key has been written again, the part sets it used before are removed, all but + // the latest two: if the newest pointer never landed (AdGuard does not say), the one + // before still finds its parts. Every stored value is handed to each page AdGuard + // loads, so sets left behind would weigh on every page until the age rule caught them. + async function trimPartSets(key) { + const sets = partSets.get(key) || []; + while (sets.length > 2) { + const set = sets.shift(); + if (!set) continue; + referencedParts.delete(set.id); + for (let index = 0; index < set.count; index += 1) { + await gmCall('deleteValue', partKey(set.id, index)).catch(() => {}); + } + } + } + + function encodedLength(text) { + try { + return encodeURIComponent(text).length; + } catch (error) { + return text.length * 9; + } + } + + // Pieces of `text` whose URL-encoded size is at most `limit`, cut between characters + // (never inside a surrogate pair, which encodeURIComponent rejects). + function splitEncoded(text, limit) { + const pieces = []; + let start = 0; + while (start < text.length) { + let length = Math.min(text.length - start, limit); + for (;;) { + let end = start + length; + const code = text.charCodeAt(end - 1); + if (end < text.length && code >= 0xd800 && code <= 0xdbff) end -= 1; + const size = encodedLength(text.slice(start, end)); + if (size <= limit || end - start <= 1) { + length = end - start; + break; + } + length = Math.max(1, Math.floor((end - start) * (limit / size) * 0.95)); + } + pieces.push(text.slice(start, start + length)); + start += length; + } + return pieces; + } + + function partKey(id, index) { + return `${PART_KEY_PREFIX}${id}_${index}`; + } + + // When a part set was written, from its id. + function partWrittenAt(id) { + const written = parseInt(String(id).split('-')[0], 36); + return Number.isFinite(written) ? written : 0; + } + async function loadValue(key) { - return decodeValue(await gmCall('getValue', key, null), key); + const value = decodeValue(await gmCall('getValue', key, null), key); + if (!value || typeof value[PARTS_MARK] !== 'number') return value; + return assembleValue(key, value); + } + + async function assembleValue(key, pointer) { + const count = clamp(pointer[PARTS_MARK], 0, 100000, 0); + const id = typeof pointer.id === 'string' ? pointer.id : ''; + if (!count || !id) return null; + referencedParts.add(id); + notePartSet(key, { id, count }); + const pieces = await Promise.all(Array.from({ length: count }, (_, index) => gmCall('getValue', partKey(id, index), null))); + const text = pieces.every(piece => typeof piece === 'string') ? pieces.join('') : null; + if (text === null || text.length !== pointer.length) { + incompleteValues.add(key); + console.warn(`[JVS] Stored value ${key} is missing some of its parts; it is read as missing.`); + return null; + } + incompleteValues.delete(key); + return decodeValue(text, key); + } + + async function storeValue(key, value) { + const text = JSON.stringify(value); + if (encodedLength(text) <= valueLimit) { + await gmCall('setValue', key, text); + if (partSets.has(key)) { + notePartSet(key, null); + await trimPartSets(key); + } + return; + } + partSequence += 1; + const id = `${Date.now().toString(36)}-${journalToken}${partSequence.toString(36)}`; + const pieces = splitEncoded(text, valueLimit); + referencedParts.add(id); + await Promise.all(pieces.map((piece, index) => gmCall('setValue', partKey(id, index), piece))); + await gmCall('setValue', key, JSON.stringify({ [PARTS_MARK]: pieces.length, id, length: text.length })); + notePartSet(key, { id, count: pieces.length }); + await trimPartSets(key); + } + + // Removes part sets no stored value refers to. Only those written well before this page + // loaded: one written since may belong to a value another tab is still storing, which + // this page cannot see (on AdGuard it never sees other tabs' writes). + async function retireParts() { + if (!journalingAvailable()) return; + let keys; + try { + keys = await gmCall('listValues'); + } catch (error) { + return; + } + if (!Array.isArray(keys)) return; + const referenced = new Set(referencedParts); + await Promise.all(keys.filter(key => typeof key === 'string' && !key.startsWith(PART_KEY_PREFIX)) + .map(async key => { + try { + const value = decodeValue(await gmCall('getValue', key, null), key); + if (value && typeof value[PARTS_MARK] === 'number' && typeof value.id === 'string') referenced.add(value.id); + } catch (error) { + // An unreadable value refers to nothing this page can tell. + } + })); + const stale = keys.filter(key => { + if (typeof key !== 'string' || !key.startsWith(PART_KEY_PREFIX)) return false; + const id = key.slice(PART_KEY_PREFIX.length).replace(/_\d+$/, ''); + const written = partWrittenAt(id); + return !referenced.has(id) && written && written + PART_RETIRE_MS < pageLoadedAt; + }); + for (const key of stale) { + try { + await gmCall('deleteValue', key); + } catch (error) { + // Left for a later page. + } + } + } + + // ------------------------------------------------------------------ + // Storage probe + // + // How large a value this device's userscript manager keeps is not documented for + // AdGuard, so it is measured: values of a few sizes are written, and the next page, + // which reads what storage actually kept, records the largest that came back. Managers + // that report other tabs' writes (Tampermonkey and the like) read storage itself and + // are not limited this way. + // ------------------------------------------------------------------ + + const PROBE_KEY = 'javstore_storage_probe_v1'; + const PROBE_VALUE_PREFIX = 'javstore_probe_'; + const PROBE_SIZES = [2000, 8000, 32000, 128000]; + const PROBE_MAX_AGE_MS = 7 * 86400000; + let measuredValueLimit = 0; + + // Reads what earlier pages measured, and decides what this page's probe step is. Writes + // nothing: startup writes wait until recovery has been read. + let probeStep = null; + + async function measureValueLimit() { + if (liveStorage) { + valueLimit = Infinity; + return; + } + let probe = null; + try { + probe = decodeValue(await gmCall('getValue', PROBE_KEY, null), PROBE_KEY); + } catch (error) { + return; + } + const now = Date.now(); + if (probe?.measuredAt && now - probe.measuredAt < PROBE_MAX_AGE_MS) { + measuredValueLimit = clamp(probe.limit, 0, Number.MAX_SAFE_INTEGER, 0); + if (measuredValueLimit) valueLimit = measuredValueLimit; + return; + } + const token = typeof probe?.token === 'string' ? probe.token : ''; + const startedAt = parseTimestamp(probe?.startedAt); + // A probe started a moment ago may still be on its way into storage; the next page + // judges it. + if (token && startedAt > pageLoadedAt - 5000) return; + // Written by an earlier page, so what comes back now is what storage kept. + if (token && startedAt) { + const kept = await Promise.all(PROBE_SIZES.map(async size => { + const value = await gmCall('getValue', `${PROBE_VALUE_PREFIX}${token}_${size}`, null).catch(() => null); + return typeof value === 'string' && value.length === size ? size : 0; + })); + measuredValueLimit = Math.max(0, ...kept); + if (measuredValueLimit) valueLimit = measuredValueLimit; + console.info(`[JVS] Userscript storage keeps values up to ${measuredValueLimit || 'none of the tested'} encoded characters.`); + probeStep = { record: { limit: measuredValueLimit, measuredAt: now }, token }; + return; + } + probeStep = { start: true }; + } + + // Records a finished measurement, or starts one for the next page to judge. + async function advanceStorageProbe() { + const step = probeStep; + probeStep = null; + if (!step) return; + try { + if (step.record) { + await gmCall('setValue', PROBE_KEY, JSON.stringify(step.record)); + for (const size of PROBE_SIZES) { + await gmCall('deleteValue', `${PROBE_VALUE_PREFIX}${step.token}_${size}`).catch(() => {}); + } + return; + } + const token = Math.random().toString(36).slice(2, 10); + for (const size of PROBE_SIZES) { + await gmCall('setValue', `${PROBE_VALUE_PREFIX}${token}_${size}`, 'a'.repeat(size)); + } + await gmCall('setValue', PROBE_KEY, JSON.stringify({ token, startedAt: Date.now() })); + } catch (error) { + // Measured on a later page. + } } - function storeValue(key, value) { - return gmCall('setValue', key, JSON.stringify(value)); + function describeValueLimit() { + if (liveStorage) return ''; + if (!measuredValueLimit) return 'storage size limit not measured yet'; + return `storage keeps values up to ${Math.round(measuredValueLimit / 1000)} KB`; } async function readStoredState() { @@ -2860,7 +3114,9 @@ const counts = getCounts(); const capped = counts.stored >= MAX_VISITED_ITEMS ? ' (cap reached—oldest are dropped)' : ''; ui.summary.textContent = `${counts.matched} matched · ${counts.visited} visited`; - ui.counts.textContent = `${counts.total} cards on this page · ${counts.stored} visited URLs stored${capped} · ${describeSaveState()} · ${describeSyncState()}`; + const limit = describeValueLimit(); + const incomplete = incompleteValues.size ? ' · some stored values were missing parts' : ''; + ui.counts.textContent = `${counts.total} cards on this page · ${counts.stored} visited URLs stored${capped} · ${describeSaveState()} · ${describeSyncState()}${limit ? ` · ${limit}` : ''}${incomplete}`; const workerVersion = describeWorkerVersion(); ui.workerVersion.textContent = workerVersion; ui.workerVersion.hidden = !workerVersion; @@ -3551,6 +3807,9 @@ // Start independent reads together. Only the local working copy is needed // to paint tiles; sync configuration and recovery must not hold it up. const configPromise = readSyncConfig(); + // How large a value this device keeps decides how anything large is written, so it + // is known before the first write; the reads below do not depend on it. + const limitPromise = measureValueLimit().catch(() => {}); const journalsPromise = readJournals(); const main = await readMainState(); initializeState(main); @@ -3560,7 +3819,7 @@ // No writes or controls are enabled during the preview. Reconcile journals // before replay/migration so stale local data cannot overwrite recovery. - const [config, journals] = await Promise.all([configPromise, journalsPromise]); + const [config, journals] = await Promise.all([configPromise, journalsPromise, limitPromise]); syncConfig = config; const snapshot = { main, journals }; loadedSnapshot = snapshot; @@ -3571,7 +3830,7 @@ processAllCards(); await migrateLegacyHistory(); if (replayed || journalsAhead(snapshot)) persistState(); - retireJournals(Date.now()); + retireJournals(Date.now()).then(retireParts).then(advanceStorageProbe).catch(() => {}); setRootState(); if (document.readyState === 'loading') { diff --git a/tests/harness.mjs b/tests/harness.mjs index f44222a..7e129d6 100644 --- a/tests/harness.mjs +++ b/tests/harness.mjs @@ -31,10 +31,18 @@ const openWindows = new Set(); // `snapshotPerTab` is AdGuard's behaviour across tabs: each tab reads the values storage // held when the tab loaded, plus its own writes. Writes still reach storage, so a tab // opened later sees them, but a tab that is already open never does. +// +// `maxEncodedValue` is the size limit AdGuard appears to put on one stored value: each +// GM_setValue travels to the AdGuard app as one request with the value URL-encoded in it, +// and a value too large for that request is not kept, without the script being told. The +// tab that wrote it still reads it back (AdGuard keeps the page's own copy); no tab that +// loads later ever sees it. export function makeStore({ latency = 0, journaling = true, primitivesOnly = false, // JVS_DOTTED=1 runs every suite against the Userscripts app's API shape. dottedOnly = process.env.JVS_DOTTED === '1', snapshotPerTab = false, + // JVS_VALUE_LIMIT=8000 runs every suite against a manager that drops larger values. + maxEncodedValue = Number(process.env.JVS_VALUE_LIMIT) || Infinity, } = {}) { const data = new Map(); const delay = () => (latency ? new Promise(resolve => setTimeout(resolve, Math.random() * latency)) : null); @@ -56,6 +64,7 @@ export function makeStore({ await delay(); if (this.dropWrites) return; if (this.dropStateWrites && key === 'javstore_cleanup_state_v2') return; + if (encodeURIComponent(typeof value === 'string' ? value : JSON.stringify(value)).length > maxEncodedValue) return; const kept = primitivesOnly && value && typeof value === 'object' ? String(value) : value; data.set(key, JSON.stringify(kept)); }, @@ -68,10 +77,19 @@ export function makeStore({ if (this.dropWrites) return; data.delete(key); }, + // A value the script stored as parts comes back whole, as the script reads it; one + // whose parts are not all there reads as missing. read(key) { if (!data.has(key)) return undefined; - const value = JSON.parse(data.get(key)); - return typeof value === 'string' ? JSON.parse(value) : value; + const raw = JSON.parse(data.get(key)); + const value = typeof raw === 'string' ? JSON.parse(raw) : raw; + if (!value || typeof value.__jvsParts !== 'number') return value; + const pieces = Array.from({ length: value.__jvsParts }, (_, index) => { + const part = `javstore_part_${value.id}_${index}`; + return data.has(part) ? JSON.parse(data.get(part)) : null; + }); + if (pieces.some(piece => typeof piece !== 'string')) return undefined; + return JSON.parse(pieces.join('')); }, write(key, value) { data.set(key, JSON.stringify(JSON.stringify(value))); @@ -94,7 +112,8 @@ export function makeStore({ }, async set(key, value) { await store.set(key, value); - if (data.has(key)) seen.set(key, data.get(key)); + // The page's own copy, whether or not storage kept it. + seen.set(key, JSON.stringify(value)); }, async list() { await delay(); diff --git a/tests/review-final.test.mjs b/tests/review-final.test.mjs index e73515d..97fd687 100644 --- a/tests/review-final.test.mjs +++ b/tests/review-final.test.mjs @@ -227,7 +227,9 @@ test('a long-open tab that pulled a large history writes small journals a few ro const tab = await open({ remote }); await waitFor(() => store.read(SESSION_KEY)?.completedAt, { timeout: 20000 }); await settle(100); - const largest = () => Math.max(0, ...store.journals().map(key => store.data.get(key).length)); + // Measured whole: a large journal is stored as parts behind a short pointer. + const size = key => JSON.stringify(store.read(key) ?? '').length; + const largest = () => Math.max(0, ...store.journals().map(size)); assert.ok(largest() > 100000, 'the pulled rows are journaled at first'); const firstKeys = new Set(store.journals()); @@ -239,7 +241,7 @@ test('a long-open tab that pulled a large history writes small journals a few ro const latest = store.journals().filter(key => !firstKeys.has(key)) .sort((a, b) => parseInt(b.split('_')[3], 36) - parseInt(a.split('_')[3], 36))[0]; assert.ok(latest, 'the page moved on to a new key'); - assert.ok(store.data.get(latest).length < 20000, - `the latest journal is ${(store.data.get(latest).length / 1024).toFixed(0)} KB`); + assert.ok(size(latest) < 20000, + `the latest journal is ${(size(latest) / 1024).toFixed(0)} KB`); assert.ok([...firstKeys].every(key => store.journals().includes(key)), 'the key holding the rows stays'); }); diff --git a/tests/storage-size.test.mjs b/tests/storage-size.test.mjs new file mode 100644 index 0000000..6124f30 --- /dev/null +++ b/tests/storage-size.test.mjs @@ -0,0 +1,141 @@ +// AdGuard sends each stored value to its app as one request with the value URL-encoded in +// it, and a value too large for that request is dropped without the script being told; the +// page that wrote it still reads it back. A history of a thousand visits is far past that, +// so it never survived a page load, and every page downloaded it again with "Restored N". +import test, { afterEach } from 'node:test'; +import assert from 'node:assert/strict'; +import { + makeStore, makeRemote, enableSync, openTab, closeTabs, settle, waitFor, listingHtml, visitedUrls, +} from './harness.mjs'; + +const SESSION_KEY = 'javstore_sync_session_v1'; +const listing = () => listingHtml([['/a.html', 'Alpha']]); +afterEach(closeTabs); + +function seeded(count) { + const remote = makeRemote(); + const at = Date.now() - 86400000; + remote.store.sync({ cursor: 0, changes: Array.from({ length: count }, (_, i) => ({ + kind: 'v', key: `https://javstore.net/${600000 + i}-FC2-PPV-${3000000 + i}-Some-Title-pn.html`, at: at + i, + })) }); + return remote; +} + +const restoredNote = tab => /Restored \d+ visited URLs/.test(tab.shadow().textContent); + +test('a large history survives page loads on a manager that drops oversized values', async () => { + const remote = seeded(1034); + const store = makeStore({ snapshotPerTab: true, maxEncodedValue: 8000 }); + enableSync(store, remote); + const first = await openTab(store, { html: listing(), remote, url: 'https://javstore.net/page/1' }); + await waitFor(() => store.read(SESSION_KEY)?.completedAt, { timeout: 20000 }); + await settle(200); + for (let page = 2; page <= 4; page += 1) { + const requests = remote.requests.length; + const tab = await openTab(store, { html: listing(), remote, url: `https://javstore.net/page/${page}` }); + await settle(300); + assert.ok(!restoredNote(tab), `page ${page} restored the history again`); + assert.ok(remote.requests.slice(requests).every(request => request.body?.cursor !== 0), + `page ${page} downloaded the whole history again`); + } + assert.ok(first); +}); + +const PROBE_KEY = 'javstore_storage_probe_v1'; +const partKeys = store => [...store.data.keys()].filter(key => key.startsWith('javstore_part_')); + +test('the device\'s size limit is measured across two page loads and then used', async () => { + const store = makeStore({ maxEncodedValue: 30000 }); + await openTab(store, { html: listing() }); + await settle(100); + assert.ok(store.read(PROBE_KEY)?.token, 'the first page starts a measurement'); + // A later page judges it; one opened within seconds would wait. + const started = store.read(PROBE_KEY); + store.write(PROBE_KEY, { ...started, startedAt: started.startedAt - 10000 }); + const second = await openTab(store, { html: listing() }); + await settle(100); + assert.equal(store.read(PROBE_KEY).limit, 8000, '2 KB and 8 KB came back, 32 KB did not'); + assert.ok(![...store.data.keys()].some(key => key.startsWith('javstore_probe_')), 'probe values are removed'); + second.shadow().querySelector('.summary').click(); + assert.match(second.shadow().textContent, /storage keeps values up to 8 KB/); +}); + +test('with the limit measured, a large history is stored in parts that fit and survives', async () => { + const store = makeStore({ snapshotPerTab: true, maxEncodedValue: 30000 }); + store.write(PROBE_KEY, { limit: 8000, measuredAt: Date.now() }); + const remote = seeded(800); + enableSync(store, remote); + await openTab(store, { html: listing(), remote }); + await waitFor(() => store.read(SESSION_KEY)?.completedAt, { timeout: 20000 }); + await settle(200); + assert.equal(visitedUrls(store).length, 800); + const parts = partKeys(store); + assert.ok(parts.length > 1, 'the history is in parts'); + assert.ok(parts.every(key => encodeURIComponent(JSON.parse(store.data.get(key))).length <= 8000)); +}); + +test('a value with a part missing is read as missing, not as a shorter history', async () => { + const store = makeStore({ maxEncodedValue: 8000 }); + store.write(PROBE_KEY, { limit: 6000, measuredAt: Date.now() }); + const visited = {}; + for (let i = 0; i < 300; i++) visited[`https://javstore.net/v-${i}.html`] = Date.now() - 1000000 + i; + // A main document in three parts, one of which never landed. + const text = JSON.stringify({ version: 3, updatedAt: Date.now(), visited, settings: {} }); + const id = `${(Date.now() - 3600000).toString(36)}-test1`; + const third = Math.ceil(text.length / 3); + store.data.set(`javstore_part_${id}_0`, JSON.stringify(text.slice(0, third))); + store.data.set(`javstore_part_${id}_2`, JSON.stringify(text.slice(2 * third))); + store.data.set('javstore_cleanup_state_v2', JSON.stringify(JSON.stringify({ __jvsParts: 3, id, length: text.length }))); + const reader = await openTab(store, { html: listing() }); + await settle(50); + reader.shadow().querySelector('.summary').click(); + assert.match(reader.shadow().textContent, /0 visited URLs stored/); + assert.match(reader.shadow().textContent, /some stored values were missing parts/); +}); + +test('parts nothing refers to are removed once old, and parts in use are kept', async () => { + const store = makeStore({ snapshotPerTab: true, maxEncodedValue: 30000 }); + store.write(PROBE_KEY, { limit: 8000, measuredAt: Date.now() }); + const old = (Date.now() - 2 * 3600000).toString(36); + const recent = (Date.now() - 60000).toString(36); + store.data.set(`javstore_part_${old}-orphan1_0`, JSON.stringify('x')); + store.data.set(`javstore_part_${recent}-fresh1_0`, JSON.stringify('y')); + const remote = seeded(600); + enableSync(store, remote); + await openTab(store, { html: listing(), remote }); + await waitFor(() => store.read(SESSION_KEY)?.completedAt, { timeout: 20000 }); + await settle(200); + // The next page loads after the parts the history uses were written. + const inUse = partKeys(store).filter(key => !key.includes('orphan') && !key.includes('fresh')); + assert.ok(inUse.length > 0); + await openTab(store, { html: listing(), remote }); + await settle(200); + assert.ok(!store.data.has(`javstore_part_${old}-orphan1_0`), 'an old orphan is removed'); + assert.ok(store.data.has(`javstore_part_${recent}-fresh1_0`), 'a recent one may still be in use elsewhere'); + assert.ok(inUse.every(key => store.data.has(key)), 'parts the history uses stay'); + assert.equal(visitedUrls(store).length, 600); +}); + +// Every stored value is handed to each page AdGuard loads, so the part sets a key used +// before must not pile up while a tab keeps saving. +test('saving again and again keeps at most two part sets per stored value', async () => { + const store = makeStore({ maxEncodedValue: 8000 }); + store.write(PROBE_KEY, { limit: 6000, measuredAt: Date.now() }); + const remote = seeded(600); + enableSync(store, remote); + const tab = await openTab(store, { html: listingHtml(Array.from({ length: 12 }, (_, i) => [`/c-${i}.html`, `Card ${i}`])), remote }); + await waitFor(() => store.read(SESSION_KEY)?.completedAt, { timeout: 20000 }); + for (let i = 0; i < 12; i++) { + tab.window.document.querySelector(`main .grid a[href="/c-${i}.html"]`) + .dispatchEvent(new tab.window.MouseEvent('click', { bubbles: true, cancelable: true, button: 0 })); + await settle(30); + } + await settle(200); + // Two per value stored in parts (the history, and this page's journal of what it pulled). + const sets = new Set(partKeys(store).map(key => key.replace(/_\d+$/, ''))); + const parted = [...store.data.keys()].filter(key => !key.startsWith('javstore_part_') + && JSON.parse(store.data.get(key)).includes?.('__jvsParts')).length; + assert.ok(parted >= 1); + assert.ok(sets.size <= 2 * parted, `${sets.size} part sets for ${parted} values`); + assert.equal(visitedUrls(store).length, 612); +}); diff --git a/worker/src/worker.mjs b/worker/src/worker.mjs index f8ced5e..0c85b7e 100644 --- a/worker/src/worker.mjs +++ b/worker/src/worker.mjs @@ -23,7 +23,7 @@ const STORAGE_VERSION = 3; // Reported in every answer so the userscript can show which worker it is talking to. It // moves in step with the userscript's own version; a test holds the two together. -export const WORKER_VERSION = '6.13.0'; +export const WORKER_VERSION = '6.13.1'; const OBJECT_NAME = 'default'; const MAX_BODY_BYTES = 8 * 1024 * 1024; const MAX_PAGE_ROWS = 2000; From ef464203a2d6cacff5b8b3f0ea19b674e6bd3859 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 14 Nov 2026 21:12:36 +0000 Subject: [PATCH 2/4] Add failing review tests for 6.13.1 part storage and probe P1 orphaned history copies per page load, P2 leaked probe values, P3 missing part wipes history despite kept previous set, P4 click-navigate lands pointer before parts, P5 JSON-encoded size exceeds measured limit. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VCuhK4TQqcEbLXdmxLjggy --- tests/review-parts.test.mjs | 212 ++++++++++++++++++++++++++++++++++++ 1 file changed, 212 insertions(+) create mode 100644 tests/review-parts.test.mjs diff --git a/tests/review-parts.test.mjs b/tests/review-parts.test.mjs new file mode 100644 index 0000000..e9f3ca5 --- /dev/null +++ b/tests/review-parts.test.mjs @@ -0,0 +1,212 @@ +// Adversarial review of 6.13.1's part storage (values too large for AdGuard stored as parts +// behind a pointer) and its storage probe. Each test reproduces a defect; the report that +// came with this file ranks them. Tests P3 and P4 model AdGuard behaviour the platform +// notes leave open (unordered delivery of in-flight writes; the value JSON-encoded before +// it is URL-encoded); each says which assumption it makes. +import test, { afterEach } from 'node:test'; +import assert from 'node:assert/strict'; +import { + makeStore, openTab, closeTabs, settle, waitFor, listingHtml, clickCard, visitedUrls, +} from './harness.mjs'; + +const STATE_KEY = 'javstore_cleanup_state_v2'; +const PROBE_KEY = 'javstore_storage_probe_v1'; +const PART_PREFIX = 'javstore_part_'; +const CARDS = Array.from({ length: 20 }, (_, i) => [`/c-${i}.html`, `Card ${i}`]); +const listing = () => listingHtml(CARDS); + +afterEach(closeTabs); + +function history(count, at = Date.now() - 86400000) { + const visited = {}; + for (let i = 0; i < count; i++) { + visited[`https://javstore.net/${600000 + i}-FC2-PPV-${3000000 + i}-Some-Title-pn.html`] = at + i; + } + return { version: 3, updatedAt: at + count, visited, settings: {} }; +} + +const partSets = store => new Set([...store.data.keys()] + .filter(key => key.startsWith(PART_PREFIX)).map(key => key.replace(/_\d+$/, ''))); +const storedBytes = store => [...store.data.values()] + .reduce((sum, raw) => sum + encodeURIComponent(JSON.parse(raw)).length, 0); +const visitedCount = tab => Number(/(\d+) visited URLs stored/.exec(tab.shadow().textContent)?.[1] ?? -1); +const openPanel = tab => tab.shadow().querySelector('.summary').click(); + +// One clock for every tab of the device, which a test can move forward. +function device(store) { + const clock = { offset: 0 }; + const open = (extra = {}) => openTab(store, { + html: listing(), + ...extra, + beforeRun: window => { + const realNow = window.Date.now.bind(window.Date); + window.Date.now = () => realNow() + clock.offset; + }, + }); + return { clock, open }; +} + +// P1 (perf). A page trims only part sets it replaced itself, and only once it has written +// the same key twice more. An ordinary page writes the history once (the click that leaves +// it), so every page load leaves a full copy of the history behind for PART_RETIRE_MS (30 +// minutes). AdGuard embeds every stored value into each page it injects into, so browsing +// at a page or two a minute makes each page load carry dozens of copies of the history. +test('P1 browsing page after page does not leave a full copy of the history per page load', async () => { + const store = makeStore({ snapshotPerTab: true, maxEncodedValue: 8006 }); + store.write(PROBE_KEY, { limit: 8000, measuredAt: Date.now() }); + store.write(STATE_KEY, history(1000)); + const docBytes = storedBytes(store); + const session = new Map(); + for (let page = 0; page < 12; page++) { + const tab = await openTab(store, { html: listing(), session }); + await settle(40); + clickCard(tab.window, `/c-${page}.html`); + await settle(60); + tab.window.dispatchEvent(new tab.window.Event('pagehide')); + } + assert.equal(visitedUrls(store).length, 1012); + const sets = partSets(store).size; + assert.ok(sets <= 3, `${sets} part sets of the history in storage after 12 page loads`); + assert.ok(storedBytes(store) < 4 * docBytes, + `${Math.round(storedBytes(store) / 1000)} KB stored for a ${Math.round(docBytes / 1000)} KB history`); +}); + +// P2 (perf). Every page that loads before the probe pointer has landed starts a probe of its +// own (40 tabs restored at browser start all do), and the judge deletes only the values of +// the token the pointer names. The other tokens' 2 KB and 8 KB values stay in storage for +// good: nothing else ever removes javstore_probe_* keys, and they ride along in every page. +test('P2 probes started by tabs loading together leave no probe values behind', async () => { + const store = makeStore({ snapshotPerTab: true, maxEncodedValue: 30000 }); + const { clock, open } = device(store); + await Promise.all([open(), open(), open(), open()]); + await settle(100); + clock.offset = 60000; + await open(); // judges the probe the pointer names + await settle(100); + clock.offset = 120000; + await open(); + await settle(100); + assert.equal(store.read(PROBE_KEY)?.limit, 8000, 'the limit was measured'); + const left = [...store.data.keys()].filter(key => key.startsWith('javstore_probe_')); + assert.deepEqual(left, [], `${left.length} probe values left behind for good`); +}); + +// P3 (data loss). A value with one part missing reads as missing, and the page then writes +// the history it has—empty, or only what its journals hold—over the pointer. The set the +// key used before is kept in storage on purpose ("if the newest pointer never landed, the +// one before still finds its parts"), but the pointer does not name it, so nothing reads +// it, and the next save replaces the whole history even though a complete copy is right +// there. Before parts, a value either landed whole or left the previous one in place. +test('P3 a pointer with a missing part falls back to the complete set kept before it', async () => { + const store = makeStore({ snapshotPerTab: true, maxEncodedValue: 8006 }); + store.write(PROBE_KEY, { limit: 6000, measuredAt: Date.now() }); + const old = history(400); + const text = JSON.stringify(old); + const cut = (id, missing) => { + const third = Math.ceil(text.length / 3); + for (let i = 0; i < 3; i++) { + if (i !== missing) store.data.set(`${PART_PREFIX}${id}_${i}`, JSON.stringify(text.slice(i * third, (i + 1) * third))); + } + }; + // The set a page wrote a minute ago (complete), then its next one, which lost a part. + const before = `${(Date.now() - 60000).toString(36)}-prev0001`; + const latest = `${(Date.now() - 30000).toString(36)}-prev0002`; + cut(before, -1); + cut(latest, 1); + store.data.set(STATE_KEY, JSON.stringify(JSON.stringify({ __jvsParts: 3, id: latest, length: text.length }))); + + const tab = await openTab(store, { html: listing() }); + await settle(50); + clickCard(tab.window, '/c-1.html'); + await settle(100); + const later = await openTab(store, { html: listing() }); + await settle(50); + openPanel(later); + assert.ok(visitedCount(later) >= 400, `${visitedCount(later)} visits stored after one click`); +}); + +// P4 (data loss; assumes AdGuard's GM_setValue resolves once the page's own copy is updated +// and the requests to the app are in flight side by side). A click saves and navigates. +// storeValue sends every part at once and the pointer right after; the pointer is a tiny +// request and lands first, the page unloads, and the larger part requests still uploading +// are lost. The next page reads the history as missing and, replaying the click, writes a +// one-visit history over it. With one value per key, a cancelled save left the old one. +test('P4 a click that navigates while parts are uploading does not lose the history', async () => { + const store = makeStore({ snapshotPerTab: true }); + const views = []; + store.view = () => { + const seen = new Map(store.data); + const view = { + alive: true, + async get(key, fallback) { return seen.has(key) ? JSON.parse(seen.get(key)) : fallback; }, + async set(key, value) { + seen.set(key, JSON.stringify(value)); + // Upload time grows with the request: a 6 KB part takes a few ms, a pointer none. + const ms = Math.floor(encodeURIComponent(value).length / 2000); + setTimeout(() => { + if (!view.alive) return; + store.data.set(key, JSON.stringify(value)); + // The click's navigation completes as soon as the pointer is in. + if (key === STATE_KEY && String(value).includes('__jvsParts')) view.alive = false; + }, ms); + }, + async list() { return [...seen.keys()]; }, + async delete(key) { + seen.delete(key); + setTimeout(() => { if (view.alive) store.data.delete(key); }, 0); + }, + }; + views.push(view); + return view; + }; + store.write(PROBE_KEY, { limit: 6000, measuredAt: Date.now() }); + store.write(STATE_KEY, history(400)); + const session = new Map(); + const first = await openTab(store, { html: listing(), session }); + await settle(50); + clickCard(first.window, '/c-1.html'); + await waitFor(() => !views[0].alive, { timeout: 3000 }); + const next = await openTab(store, { html: listing(), session }); + await settle(100); + const third = await openTab(store, { html: listing() }); + await settle(100); + openPanel(third); + assert.ok(visitedCount(third) >= 400, `${visitedCount(third)} visits stored after the click`); + assert.ok(next); +}); + +// P5 (data loss; assumes AdGuard JSON-encodes the value it is handed before URL-encoding +// it, as it must to carry non-string values). storeValue already hands GM_setValue a JSON +// string and parts are slices of JSON text, so every `"` in them costs `%5C%22`, not the +// `%22` encodedLength counts—a history slice runs ~6% larger than measured. The probe's +// 'a'.repeat values have no quotes, so they measure the limit itself exactly. Parts are cut +// with only the 5% margin splitEncoded happens to leave, and whole values with none, so on +// a device whose limit lies just above a probe size, parts are dropped while the pointer +// lands, and the history reads as missing. +test('P5 parts survive when the manager counts the value JSON-encoded', async () => { + const store = makeStore({ snapshotPerTab: true }); + const limit = 8200; // keeps the 8 KB probe ('a' x 8000 is 8006 JSON+URL-encoded) + const set = store.set.bind(store); + store.set = async (key, value) => { + if (encodeURIComponent(JSON.stringify(value)).length > limit) return; + return set(key, value); + }; + const { clock, open } = device(store); + await open(); + await settle(100); + clock.offset = 60000; + await open(); + await settle(100); + assert.equal(store.read(PROBE_KEY)?.limit, 8000, 'the probe measured 8 KB'); + + store.write(STATE_KEY, history(400)); + const writer = await open(); + await settle(50); + clickCard(writer.window, '/c-1.html'); + await settle(100); + const reader = await open(); + await settle(50); + openPanel(reader); + assert.doesNotMatch(reader.shadow().textContent, /some stored values were missing parts/); + assert.ok(visitedCount(reader) >= 401, `${visitedCount(reader)} visits read back`); +}); From ce896bd5e3b14273d70bff3798d5751555f337e5 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 14 Nov 2026 22:11:13 +0000 Subject: [PATCH 3/4] Harden parted storage after review: fallback, settle, cleanup, cost Review found that a page usually saved the history once, so replaced part sets lingered 30 minutes (a full copy per page load); that one missing part made the next save write a near-empty history over a complete previous set; that an unordered or interrupted upload could land the pointer without its parts; that encoded size was measured more loosely than AdGuard may count it; and that tabs probing together leaked probe values. - Pointers name the previous complete set; readers fall back to it, and a history found incomplete but recent is not written over. - Parts are cut at 80% of the measured limit, counted JSON-then-URL encoded; the pointer waits for parts to arrive; saves not made by a sync pause before writing parts, so a page being left writes none. - Tabs trim the set before the one they replaced; unreferenced parts go after two minutes; retirement also runs after main saves. - Queued saves coalesce, so a burst of clicks costs one or two saves. - Fixed probe keys, cleared by the judging page; last limit kept while re-measuring; a floor when nothing was kept. - A save stopped by leaving the page is not reported as a failure. Adds the review's tests (review-parts.test.mjs) and a test for leaving mid-save; 150/150 pass with and without an 8 KB value limit and with the dotted GM API. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VCuhK4TQqcEbLXdmxLjggy --- CHANGELOG.md | 11 +- javstore-full-layout-cleanup.user.js | 215 ++++++++++++++++++++------- tests/review-final.test.mjs | 3 +- tests/review-lifecycle.test.mjs | 2 +- tests/review-parts.test.mjs | 23 ++- tests/storage-size.test.mjs | 25 +++- 6 files changed, 215 insertions(+), 64 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c44cb17..4035bac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,9 +3,14 @@ ## 6.13.1 - Fix: on AdGuard, "Restored N visited URLs from the worker" appeared on every page load, and every page downloaded the whole history again. 6.12.0 removed one cause; this fixes the one that remained. AdGuard sends each stored value to its app as a single request, with the value URL-encoded inside it. A value too large for that request is not kept, and the script is not told: the page that saved it still reads it back from its own copy. Once the history grew past a few hundred visits it no longer fit, so it survived no page load at all. Small values, such as the sync settings, were kept, which is why sync kept working and the problem looked like something else. -- Change: a value larger than the device keeps is now stored as parts, each small enough, plus a short pointer to them, written last. A value with a part missing is read as missing, never as a shorter history. Each tab removes the part sets it replaced, keeping the latest two in case the newest pointer never landed. Parts that nothing refers to are removed half an hour after they were written. -- Add: the script measures how large a value the device keeps. It stores test values of 2, 8, 32 and 128 KB, checks on the next page load which came back, uses the largest, and measures again weekly. Until it knows, it uses parts of about 6 KB. The settings panel shows the result, and warns if a stored value was found with parts missing. Managers that report other tabs' writes, such as Tampermonkey, are not measured: they have no such limit. -- The test suites can now run against a manager that drops oversized values (`JVS_VALUE_LIMIT=8000 npm test`). Under that limit, 6.13.0 fails 12 tests; this release passes all of them. +- Change: a value larger than the device keeps is now stored as parts, each well under the limit, plus a short pointer to them. Parts are cut with 20% headroom, counted the strict way (as if quotes were escaped twice). + - **The pointer is written last, after a short pause.** A page left before then never writes it, and the value before stays whole. + - **Each pointer also names the previous complete version.** If a part of the newest is missing, the previous one is read rather than nothing, and journals and the worker bring it up to date. A history found with parts still arriving is not written over until it settles. + - **A save made as the page is left writes no parts**, so it leaves no copies behind; the journal and the replay note carry the click to the next page. + - **Cleanup:** each tab removes the part set it replaced, keeping the current and previous ones. Parts nothing refers to are removed two minutes after they were written. + - **Fewer saves:** save requests made while a save is waiting are folded into it, so a burst of clicks costs one or two saves, not one each. +- Add: the script measures how large a value the device keeps. It stores test values of 2, 8, 32 and 128 KB, checks on the next page load which came back, uses the largest, and measures again weekly, keeping the last result meanwhile. Until it knows, it uses parts of about 6 KB. Tabs that start a measurement together share the same keys, and the page that judges it removes them all. The settings panel shows the result, and warns if a stored value was found with parts missing. Managers that report other tabs' writes, such as Tampermonkey, are not measured: they have no such limit. +- The test suites can now run against a manager that drops oversized values (`JVS_VALUE_LIMIT=8000 npm test`). Under that limit, 6.13.0 fails 12 tests; this release passes all 150, in every configuration. - The worker version moves to 6.13.1 with the userscript; the worker is unchanged. ## 6.13.0 diff --git a/javstore-full-layout-cleanup.user.js b/javstore-full-layout-cleanup.user.js index fcb253b..18d8d4c 100644 --- a/javstore-full-layout-cleanup.user.js +++ b/javstore-full-layout-cleanup.user.js @@ -421,28 +421,34 @@ const PART_KEY_PREFIX = 'javstore_part_'; // In URL-encoded characters: what any engine is assumed to keep until measured. const DEFAULT_VALUE_LIMIT = 6000; - // Parts nothing refers to are removed once they are this much older than the page - // that finds them, so no write still on its way can be referring to them. - const PART_RETIRE_MS = 30 * 60000; + // What is assumed kept when the probe found nothing kept at all. + const MIN_VALUE_LIMIT = 1000; + // Values are cut to this share of the measured limit: the probe's values are plain + // letters, and the request that carries a value carries more than the value itself. + const VALUE_LIMIT_HEADROOM = 0.8; + // A part set nothing refers to is removed once it is this much older than the page + // that finds it: long enough for any pointer written after its parts to have landed. + const PART_RETIRE_MS = 2 * 60000; let valueLimit = DEFAULT_VALUE_LIMIT; let partSequence = 0; + let lastPartRetireAt = 0; // Part sets this page has seen referred to, by their id. const referencedParts = new Set(); - // Keys whose parts were not all there when read. - const incompleteValues = new Set(); + // Keys whose parts were not all there when read, with when that pointer was written. + const incompleteValues = new Map(); // For each key, the part sets this page has read or written for it, oldest first. const partSets = new Map(); function notePartSet(key, set) { const sets = partSets.get(key) || []; - if (!set || !sets.some(known => known.id === set.id)) sets.push(set); + if (!set || !sets.some(known => known && known.id === set.id)) sets.push(set); partSets.set(key, sets); } // Once a key has been written again, the part sets it used before are removed, all but - // the latest two: if the newest pointer never landed (AdGuard does not say), the one - // before still finds its parts. Every stored value is handed to each page AdGuard - // loads, so sets left behind would weigh on every page until the age rule caught them. + // the latest two: the new pointer names the one before it, which a reader falls back to + // if a part of the newest never landed. Every stored value is handed to each page + // AdGuard loads, so sets left behind would weigh on every page. async function trimPartSets(key) { const sets = partSets.get(key) || []; while (sets.length > 2) { @@ -463,8 +469,18 @@ } } - // Pieces of `text` whose URL-encoded size is at most `limit`, cut between characters - // (never inside a surrogate pair, which encodeURIComponent rejects). + // What a value costs in the request that stores it, counted the stricter way: as if the + // manager encoded it as JSON before URL-encoding it, which turns each quote into `%5C%22`. + function wireLength(text) { + return encodedLength(JSON.stringify(text)); + } + + function valueBudget() { + return Number.isFinite(valueLimit) ? Math.max(200, Math.floor(valueLimit * VALUE_LIMIT_HEADROOM)) : Infinity; + } + + // Pieces of `text` that each cost at most `limit`, cut between characters (never inside a + // surrogate pair, which encodeURIComponent rejects). function splitEncoded(text, limit) { const pieces = []; let start = 0; @@ -474,7 +490,7 @@ let end = start + length; const code = text.charCodeAt(end - 1); if (end < text.length && code >= 0xd800 && code <= 0xdbff) end -= 1; - const size = encodedLength(text.slice(start, end)); + const size = wireLength(text.slice(start, end)); if (size <= limit || end - start <= 1) { length = end - start; break; @@ -497,22 +513,45 @@ return Number.isFinite(written) ? written : 0; } + function parseSetRef(value) { + if (!value || typeof value !== 'object') return null; + const id = typeof value.id === 'string' ? value.id : ''; + const count = clamp(value.count, 0, 100000, 0); + const length = clamp(value.length, 0, Number.MAX_SAFE_INTEGER, 0); + return id && count ? { id, count, length } : null; + } + async function loadValue(key) { const value = decodeValue(await gmCall('getValue', key, null), key); if (!value || typeof value[PARTS_MARK] !== 'number') return value; return assembleValue(key, value); } - async function assembleValue(key, pointer) { - const count = clamp(pointer[PARTS_MARK], 0, 100000, 0); - const id = typeof pointer.id === 'string' ? pointer.id : ''; - if (!count || !id) return null; - referencedParts.add(id); - notePartSet(key, { id, count }); - const pieces = await Promise.all(Array.from({ length: count }, (_, index) => gmCall('getValue', partKey(id, index), null))); + async function readPartSet(set) { + const pieces = await Promise.all(Array.from({ length: set.count }, (_, index) => gmCall('getValue', partKey(set.id, index), null))); const text = pieces.every(piece => typeof piece === 'string') ? pieces.join('') : null; - if (text === null || text.length !== pointer.length) { - incompleteValues.add(key); + return text !== null && text.length === set.length ? text : null; + } + + // A pointer names its own part set and the complete one it replaced. If a part of the + // newest never landed, the one before is read instead: an older history, which the + // journals and the worker bring up to date, rather than none at all. + async function assembleValue(key, pointer) { + const current = parseSetRef({ id: pointer.id, count: pointer[PARTS_MARK], length: pointer.length }); + const previous = parseSetRef(pointer.prev); + if (!current) return null; + // Oldest first, as writes record them. + [previous, current].filter(Boolean).forEach(set => { + referencedParts.add(set.id); + notePartSet(key, set); + }); + let text = await readPartSet(current); + if (text === null && previous) { + text = await readPartSet(previous); + if (text !== null) console.warn(`[JVS] Stored value ${key} is missing parts of its latest version; the one before is read.`); + } + if (text === null) { + incompleteValues.set(key, partWrittenAt(current.id)); console.warn(`[JVS] Stored value ${key} is missing some of its parts; it is read as missing.`); return null; } @@ -520,9 +559,19 @@ return decodeValue(text, key); } - async function storeValue(key, value) { + // Whether the stored history was found with parts missing that may still be on their + // way. Until that settles, this tab does not write over it. + function historyStillArriving() { + const writtenAt = incompleteValues.get(STORAGE_KEY); + return Boolean(writtenAt) && writtenAt > pageLoadedAt - 2 * STORAGE_SETTLE_MS; + } + + // `prompt`: a save that is not made as the page is being left (a sync's), which needs no + // pause before its parts and should not hold the sync lock for one. + async function storeValue(key, value, { prompt = false } = {}) { const text = JSON.stringify(value); - if (encodedLength(text) <= valueLimit) { + const budget = valueBudget(); + if (wireLength(text) <= budget) { await gmCall('setValue', key, text); if (partSets.has(key)) { notePartSet(key, null); @@ -530,13 +579,30 @@ } return; } + // A save made as the page is being left—a click on a card is one—would be cut off + // partway, leaving parts behind that nothing refers to. Parts wait a moment, and a + // page that is gone by then never writes them; the journal, written before this, and + // the replay note carry the change to the next page. (A page only frozen in the + // back/forward cache finishes the save if it is brought back.) + if (!prompt) await new Promise(resolve => window.setTimeout(resolve, 300)); partSequence += 1; const id = `${Date.now().toString(36)}-${journalToken}${partSequence.toString(36)}`; - const pieces = splitEncoded(text, valueLimit); + const pieces = splitEncoded(text, budget); referencedParts.add(id); await Promise.all(pieces.map((piece, index) => gmCall('setValue', partKey(id, index), piece))); - await gmCall('setValue', key, JSON.stringify({ [PARTS_MARK]: pieces.length, id, length: text.length })); - notePartSet(key, { id, count: pieces.length }); + // A manager may deliver writes in any order, and a page that is left drops what has + // not gone yet. The pointer waits until the parts have had time to arrive: if the page + // is gone before then, the pointer is never written and the value before stays whole. + await new Promise(resolve => window.setTimeout(resolve, 100 + Math.ceil(wireLength(text) / 2000))); + const known = (partSets.get(key) || []).filter(Boolean); + const previous = known[known.length - 1]; + await gmCall('setValue', key, JSON.stringify({ + [PARTS_MARK]: pieces.length, + id, + length: text.length, + ...(previous?.length ? { prev: previous } : {}), + })); + notePartSet(key, { id, count: pieces.length, length: text.length }); await trimPartSets(key); } @@ -545,6 +611,7 @@ // this page cannot see (on AdGuard it never sees other tabs' writes). async function retireParts() { if (!journalingAvailable()) return; + lastPartRetireAt = Date.now(); let keys; try { keys = await gmCall('listValues'); @@ -557,7 +624,10 @@ .map(async key => { try { const value = decodeValue(await gmCall('getValue', key, null), key); - if (value && typeof value[PARTS_MARK] === 'number' && typeof value.id === 'string') referenced.add(value.id); + if (value && typeof value[PARTS_MARK] === 'number') { + if (typeof value.id === 'string') referenced.add(value.id); + if (typeof value.prev?.id === 'string') referenced.add(value.prev.id); + } } catch (error) { // An unreadable value refers to nothing this page can tell. } @@ -592,11 +662,21 @@ const PROBE_SIZES = [2000, 8000, 32000, 128000]; const PROBE_MAX_AGE_MS = 7 * 86400000; let measuredValueLimit = 0; - + let valueLimitMeasured = false; // Reads what earlier pages measured, and decides what this page's probe step is. Writes // nothing: startup writes wait until recovery has been read. let probeStep = null; + function probeValue(token, size) { + return `${token}${'a'.repeat(Math.max(0, size - token.length))}`; + } + + function useMeasuredLimit(limit) { + valueLimitMeasured = true; + measuredValueLimit = limit; + valueLimit = limit || MIN_VALUE_LIMIT; + } + async function measureValueLimit() { if (liveStorage) { valueLimit = Infinity; @@ -609,29 +689,30 @@ return; } const now = Date.now(); - if (probe?.measuredAt && now - probe.measuredAt < PROBE_MAX_AGE_MS) { - measuredValueLimit = clamp(probe.limit, 0, Number.MAX_SAFE_INTEGER, 0); - if (measuredValueLimit) valueLimit = measuredValueLimit; - return; - } + // Until a new measurement is in, the last one stands. + const last = clamp(probe?.limit ?? probe?.lastLimit, 0, Number.MAX_SAFE_INTEGER, 0); + if (last || probe?.measuredAt) useMeasuredLimit(last); + if (probe?.measuredAt && now - probe.measuredAt < PROBE_MAX_AGE_MS) return; const token = typeof probe?.token === 'string' ? probe.token : ''; const startedAt = parseTimestamp(probe?.startedAt); // A probe started a moment ago may still be on its way into storage; the next page // judges it. if (token && startedAt > pageLoadedAt - 5000) return; - // Written by an earlier page, so what comes back now is what storage kept. + // Written by an earlier page, so what comes back now is what storage kept. Tabs that + // load together each start a probe under the same keys; values that do not match the + // token that won only make the measurement lower, never higher. if (token && startedAt) { const kept = await Promise.all(PROBE_SIZES.map(async size => { - const value = await gmCall('getValue', `${PROBE_VALUE_PREFIX}${token}_${size}`, null).catch(() => null); - return typeof value === 'string' && value.length === size ? size : 0; + const value = await gmCall('getValue', `${PROBE_VALUE_PREFIX}${size}`, null).catch(() => null); + return value === probeValue(token, size) ? size : 0; })); - measuredValueLimit = Math.max(0, ...kept); - if (measuredValueLimit) valueLimit = measuredValueLimit; - console.info(`[JVS] Userscript storage keeps values up to ${measuredValueLimit || 'none of the tested'} encoded characters.`); - probeStep = { record: { limit: measuredValueLimit, measuredAt: now }, token }; + const limit = Math.max(0, ...kept); + useMeasuredLimit(limit); + console.info(`[JVS] Userscript storage keeps values up to ${limit || 'none of the tested'} encoded characters.`); + probeStep = { record: { limit, measuredAt: now } }; return; } - probeStep = { start: true }; + probeStep = { start: true, lastLimit: last }; } // Records a finished measurement, or starts one for the next page to judge. @@ -642,16 +723,19 @@ try { if (step.record) { await gmCall('setValue', PROBE_KEY, JSON.stringify(step.record)); - for (const size of PROBE_SIZES) { - await gmCall('deleteValue', `${PROBE_VALUE_PREFIX}${step.token}_${size}`).catch(() => {}); + const keys = await gmCall('listValues').catch(() => []); + for (const key of Array.isArray(keys) ? keys : []) { + if (typeof key === 'string' && key.startsWith(PROBE_VALUE_PREFIX)) { + await gmCall('deleteValue', key).catch(() => {}); + } } return; } const token = Math.random().toString(36).slice(2, 10); for (const size of PROBE_SIZES) { - await gmCall('setValue', `${PROBE_VALUE_PREFIX}${token}_${size}`, 'a'.repeat(size)); + await gmCall('setValue', `${PROBE_VALUE_PREFIX}${size}`, probeValue(token, size)); } - await gmCall('setValue', PROBE_KEY, JSON.stringify({ token, startedAt: Date.now() })); + await gmCall('setValue', PROBE_KEY, JSON.stringify({ token, startedAt: Date.now(), lastLimit: step.lastLimit || 0 })); } catch (error) { // Measured on a later page. } @@ -659,7 +743,8 @@ function describeValueLimit() { if (liveStorage) return ''; - if (!measuredValueLimit) return 'storage size limit not measured yet'; + if (!valueLimitMeasured) return 'storage size limit not measured yet'; + if (!measuredValueLimit) return 'storage kept none of the tested sizes (2 KB and up)'; return `storage keeps values up to ${Math.round(measuredValueLimit / 1000)} KB`; } @@ -1016,7 +1101,8 @@ journalCreatedAt = writtenAt; journalKey = makeJournalKey(journalCreatedAt); } - await storeValue(journalKey, journal); + // The journal is what carries a click to the next page, so it does not wait. + await storeValue(journalKey, journal, { prompt: true }); journalWritten = true; return journal; } @@ -1215,16 +1301,31 @@ // Writes are queued so two read-merge-write cycles can never interleave and drop each // other's changes on engines where GM_setValue resolves asynchronously. - function persistState({ journalOnly = false } = {}) { - const write = () => writeStoredState({ journalOnly }); - writeQueue = writeQueue.then(write, write); + // A save that has not started yet takes in any asked for meanwhile: each writes all this + // page holds, so a burst of clicks needs one save after the one under way, not one each. + let queuedWrite = null; + + function persistState({ journalOnly = false, prompt = false } = {}) { if (!journalOnly) scheduleRemotePush(); + if (queuedWrite) { + queuedWrite.options.journalOnly = queuedWrite.options.journalOnly && journalOnly; + queuedWrite.options.prompt = queuedWrite.options.prompt || prompt; + return queuedWrite.promise; + } + const entry = { options: { journalOnly, prompt } }; + const write = () => { + if (queuedWrite === entry) queuedWrite = null; + return writeStoredState(entry.options); + }; + writeQueue = writeQueue.then(write, write); + entry.promise = writeQueue; + queuedWrite = entry; return writeQueue; } // `journalOnly` keeps the main document as it is: for a save that only records what // this page has sent, which no other copy depends on. - async function writeStoredState({ journalOnly = false } = {}) { + async function writeStoredState({ journalOnly = false, prompt = false } = {}) { const startedAt = Date.now(); for (let attempt = 0; attempt < 2; attempt += 1) { try { @@ -1239,7 +1340,9 @@ // since does not write over the shared document; its journal carries its // changes until a newer tab folds them in. Without journals there is // nothing else to write to. - const shared = !journalingAvailable() || (!journalOnly && sharedWritesAllowed()); + // Nor does one that found the stored history with parts still on their way. + const shared = !journalingAvailable() + || (!journalOnly && sharedWritesAllowed() && !historyStillArriving()); const journal = await writeJournal(now); if (!shared) { lastSavedAt = Date.now(); @@ -1250,7 +1353,7 @@ return true; } const payload = serializeState(); - await storeValue(STORAGE_KEY, payload); + await storeValue(STORAGE_KEY, payload, { prompt }); // Read back rather than trusting the write: on engines that read storage // itself, a value that never landed is exactly the failure that used to go // unnoticed until the history was gone. (On AdGuard the read comes from the @@ -1267,9 +1370,13 @@ storageAvailable = true; clearPendingVisits(startedAt, verified); await retireJournals(Date.now()); + if (Date.now() - lastPartRetireAt > 60000) await retireParts(); scheduleCountUpdate(); return true; } catch (error) { + // A save stopped because the page is being left is not a storage failure: the + // journal and the replay note carry it to the next page. + if (pageClosed) return false; if (attempt === 0) { await new Promise(resolve => window.setTimeout(resolve, 300)); continue; @@ -2399,7 +2506,7 @@ // is far larger, is left as it is. const rewrite = changed || absorbed || naming || needsCheckpoint; applyingRemoteState = true; - const saved = await persistState({ journalOnly: !rewrite }); + const saved = await persistState({ journalOnly: !rewrite, prompt: true }); applyingRemoteState = false; if (!saved) throw new Error('the merged history could not be saved'); diff --git a/tests/review-final.test.mjs b/tests/review-final.test.mjs index 97fd687..37b9d7c 100644 --- a/tests/review-final.test.mjs +++ b/tests/review-final.test.mjs @@ -236,7 +236,8 @@ test('a long-open tab that pulled a large history writes small journals a few ro for (const minutes of [6, 12]) { clock.offset = minutes * 60000; pressOnCard(tab.window, minutes === 6 ? '/a.html' : '/b.html', 'v'); - await settle(100); + // Let the save finish: a history stored in parts takes a moment. + await new Promise(resolve => setTimeout(resolve, 1500)); } const latest = store.journals().filter(key => !firstKeys.has(key)) .sort((a, b) => parseInt(b.split('_')[3], 36) - parseInt(a.split('_')[3], 36))[0]; diff --git a/tests/review-lifecycle.test.mjs b/tests/review-lifecycle.test.mjs index 51c3c3a..63c8e9d 100644 --- a/tests/review-lifecycle.test.mjs +++ b/tests/review-lifecycle.test.mjs @@ -153,7 +153,7 @@ test('measure: GM traffic for an item page load and its push with sync on (sessi const before = m.snapshot(); await openTab(store, { html: detailHtml, url: 'https://javstore.net/700002-Item-pn.html', remote }); - await settle(); + await waitFor(() => m.since(before).sessionWrites >= 1); const load = m.since(before); t.diagnostic(`item page load, sync on: ${describe(load)}; worker requests ${remote.requests.length - requests}`); diff --git a/tests/review-parts.test.mjs b/tests/review-parts.test.mjs index e9f3ca5..5053d00 100644 --- a/tests/review-parts.test.mjs +++ b/tests/review-parts.test.mjs @@ -59,12 +59,22 @@ test('P1 browsing page after page does not leave a full copy of the history per const session = new Map(); for (let page = 0; page < 12; page++) { const tab = await openTab(store, { html: listing(), session }); - await settle(40); + // A page someone reads for a moment before clicking on. + await new Promise(resolve => setTimeout(resolve, 1000)); clickCard(tab.window, `/c-${page}.html`); await settle(60); + // The click navigates: the page is gone, and its timers with it. tab.window.dispatchEvent(new tab.window.Event('pagehide')); + tab.window.close(); } - assert.equal(visitedUrls(store).length, 1012); + // The next page in the tab replays the last click, which left before it was saved. + await openTab(store, { html: listing(), session }); + await settle(100); + // Every visit is held: in the history, or (for a page left before its history save + // landed) in that page's journal, which the next page folds in. + const held = new Set(visitedUrls(store)); + store.journals().forEach(key => Object.keys(store.read(key)?.visited || {}).forEach(url => held.add(url))); + assert.equal(held.size, 1012); const sets = partSets(store).size; assert.ok(sets <= 3, `${sets} part sets of the history in storage after 12 page loads`); assert.ok(storedBytes(store) < 4 * docBytes, @@ -113,7 +123,10 @@ test('P3 a pointer with a missing part falls back to the complete set kept befor const latest = `${(Date.now() - 30000).toString(36)}-prev0002`; cut(before, -1); cut(latest, 1); - store.data.set(STATE_KEY, JSON.stringify(JSON.stringify({ __jvsParts: 3, id: latest, length: text.length }))); + // The pointer names the set it replaced, as every pointer written with a set before it does. + store.data.set(STATE_KEY, JSON.stringify(JSON.stringify({ + __jvsParts: 3, id: latest, length: text.length, prev: { id: before, count: 3, length: text.length }, + }))); const tab = await openTab(store, { html: listing() }); await settle(50); @@ -164,7 +177,9 @@ test('P4 a click that navigates while parts are uploading does not lose the hist const session = new Map(); const first = await openTab(store, { html: listing(), session }); await settle(50); - clickCard(first.window, '/c-1.html'); + // A click saves the history; the page is then left the moment that save's pointer is in. + first.window.document.querySelector('main .grid a[href="/c-1.html"]') + .dispatchEvent(new first.window.MouseEvent('click', { bubbles: true, cancelable: true, button: 0 })); await waitFor(() => !views[0].alive, { timeout: 3000 }); const next = await openTab(store, { html: listing(), session }); await settle(100); diff --git a/tests/storage-size.test.mjs b/tests/storage-size.test.mjs index 6124f30..7780392 100644 --- a/tests/storage-size.test.mjs +++ b/tests/storage-size.test.mjs @@ -130,7 +130,8 @@ test('saving again and again keeps at most two part sets per stored value', asyn .dispatchEvent(new tab.window.MouseEvent('click', { bubbles: true, cancelable: true, button: 0 })); await settle(30); } - await settle(200); + // Let the last save finish: its pointer waits a moment after its parts. + await new Promise(resolve => setTimeout(resolve, 1500)); // Two per value stored in parts (the history, and this page's journal of what it pulled). const sets = new Set(partKeys(store).map(key => key.replace(/_\d+$/, ''))); const parted = [...store.data.keys()].filter(key => !key.startsWith('javstore_part_') @@ -139,3 +140,25 @@ test('saving again and again keeps at most two part sets per stored value', asyn assert.ok(sets.size <= 2 * parted, `${sets.size} part sets for ${parted} values`); assert.equal(visitedUrls(store).length, 612); }); + +// A large save that is stopped because the page is being left is not a storage failure, and +// a page brought back from the back/forward cache must not say "Could not save". +test('leaving the page during a large save does not report a failed save', async () => { + const store = makeStore({ maxEncodedValue: 8000 }); + store.write(PROBE_KEY, { limit: 6000, measuredAt: Date.now() }); + const remote = seeded(600); + enableSync(store, remote); + const tab = await openTab(store, { html: listing(), remote }); + await waitFor(() => store.read(SESSION_KEY)?.completedAt, { timeout: 20000 }); + await new Promise(resolve => setTimeout(resolve, 800)); + tab.window.document.querySelector('main .grid a[href="/a.html"]') + .dispatchEvent(new tab.window.MouseEvent('click', { bubbles: true, cancelable: true, button: 0 })); + await settle(5); + tab.window.dispatchEvent(new tab.window.Event('pagehide')); + await new Promise(resolve => setTimeout(resolve, 1500)); + assert.doesNotMatch(tab.shadow().textContent, /Could not save/); + // The click is carried by the journal to the next page. + const next = await openTab(store, { html: listing(), remote }); + await settle(100); + assert.ok(next.window.document.querySelector('main .grid a[href="/a.html"]').classList.contains('jvs-visited')); +}); From 8c51c71f755e93baa5ae9330e91ffa66f43e165b Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 14 Nov 2026 22:37:46 +0000 Subject: [PATCH 4/4] Hold saves while the newest part set may still arrive A page that read the previous part set because the newest one was missing a part did not count the value as incomplete, so its next save could replace the pointer before the late part landed and drop the visits only the newest set held. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VCuhK4TQqcEbLXdmxLjggy --- CHANGELOG.md | 2 +- javstore-full-layout-cleanup.user.js | 8 ++++++- tests/review-parts.test.mjs | 36 ++++++++++++++++++++++++++++ 3 files changed, 44 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4035bac..a408ba2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,7 +5,7 @@ - Fix: on AdGuard, "Restored N visited URLs from the worker" appeared on every page load, and every page downloaded the whole history again. 6.12.0 removed one cause; this fixes the one that remained. AdGuard sends each stored value to its app as a single request, with the value URL-encoded inside it. A value too large for that request is not kept, and the script is not told: the page that saved it still reads it back from its own copy. Once the history grew past a few hundred visits it no longer fit, so it survived no page load at all. Small values, such as the sync settings, were kept, which is why sync kept working and the problem looked like something else. - Change: a value larger than the device keeps is now stored as parts, each well under the limit, plus a short pointer to them. Parts are cut with 20% headroom, counted the strict way (as if quotes were escaped twice). - **The pointer is written last, after a short pause.** A page left before then never writes it, and the value before stays whole. - - **Each pointer also names the previous complete version.** If a part of the newest is missing, the previous one is read rather than nothing, and journals and the worker bring it up to date. A history found with parts still arriving is not written over until it settles. + - **Each pointer also names the previous complete version.** If a part of the newest is missing, the previous one is read rather than nothing, and journals and the worker bring it up to date. A history found with parts still arriving, including one read from that previous version, is not written over until it settles. - **A save made as the page is left writes no parts**, so it leaves no copies behind; the journal and the replay note carry the click to the next page. - **Cleanup:** each tab removes the part set it replaced, keeping the current and previous ones. Parts nothing refers to are removed two minutes after they were written. - **Fewer saves:** save requests made while a save is waiting are folded into it, so a burst of clicks costs one or two saves, not one each. diff --git a/javstore-full-layout-cleanup.user.js b/javstore-full-layout-cleanup.user.js index 18d8d4c..b39c202 100644 --- a/javstore-full-layout-cleanup.user.js +++ b/javstore-full-layout-cleanup.user.js @@ -548,7 +548,13 @@ let text = await readPartSet(current); if (text === null && previous) { text = await readPartSet(previous); - if (text !== null) console.warn(`[JVS] Stored value ${key} is missing parts of its latest version; the one before is read.`); + if (text !== null) { + // Still counted as incomplete: the newest parts may be on their way, and a + // save made from the older copy would replace what only they hold. + incompleteValues.set(key, partWrittenAt(current.id)); + console.warn(`[JVS] Stored value ${key} is missing parts of its latest version; the one before is read.`); + return decodeValue(text, key); + } } if (text === null) { incompleteValues.set(key, partWrittenAt(current.id)); diff --git a/tests/review-parts.test.mjs b/tests/review-parts.test.mjs index 5053d00..f0659ee 100644 --- a/tests/review-parts.test.mjs +++ b/tests/review-parts.test.mjs @@ -138,6 +138,42 @@ test('P3 a pointer with a missing part falls back to the complete set kept befor assert.ok(visitedCount(later) >= 400, `${visitedCount(later)} visits stored after one click`); }); +// P3b (data loss). The newest set is still missing a part when a page loads, so the page +// reads the set before it. That part then lands. A save from the older copy must not +// replace the pointer while the newest set may still complete, or the visits only it holds +// are gone. +test('P3b a page that read the previous set does not write over a newest set still arriving', async () => { + const store = makeStore({ snapshotPerTab: true, maxEncodedValue: 8006 }); + store.write(PROBE_KEY, { limit: 6000, measuredAt: Date.now() }); + const older = history(400); + const newer = history(400); + const extra = 'https://javstore.net/999999-FC2-PPV-9999999-Only-In-Newest-pn.html'; + newer.visited[extra] = Date.now() - 20000; + const parts = (id, value) => { + const text = JSON.stringify(value); + const third = Math.ceil(text.length / 3); + return { text, pieces: [0, 1, 2].map(i => [`${PART_PREFIX}${id}_${i}`, JSON.stringify(text.slice(i * third, (i + 1) * third))]) }; + }; + const before = `${(Date.now() - 60000).toString(36)}-prev0001`; + const latest = `${(Date.now() - 30000).toString(36)}-prev0002`; + const a = parts(before, older); + const b = parts(latest, newer); + a.pieces.forEach(([key, raw]) => store.data.set(key, raw)); + store.data.set(b.pieces[0][0], b.pieces[0][1]); + store.data.set(b.pieces[2][0], b.pieces[2][1]); + store.data.set(STATE_KEY, JSON.stringify(JSON.stringify({ + __jvsParts: 3, id: latest, length: b.text.length, prev: { id: before, count: 3, length: a.text.length }, + }))); + + const tab = await openTab(store, { html: listing() }); + await settle(50); + // The late part lands after the page took its snapshot. + store.data.set(b.pieces[1][0], b.pieces[1][1]); + clickCard(tab.window, '/c-1.html'); + await settle(800); + assert.ok(visitedUrls(store).includes(extra), 'the visit only the newest set held was lost'); +}); + // P4 (data loss; assumes AdGuard's GM_setValue resolves once the page's own copy is updated // and the requests to the app are in flight side by side). A click saves and navigates. // storeValue sends every part at once and the pointer right after; the pointer is a tiny