From 47ff269e252887ca57986c0d93c58cf021ab2481 Mon Sep 17 00:00:00 2001 From: JOY <5027251+JOY@users.noreply.github.com> Date: Wed, 4 Nov 2026 12:32:52 +0700 Subject: [PATCH 1/7] feat(extension): onboarding consent screen for the opt-in (#1605) * feat(extension): onboarding consent screen for the opt-in On a first install the extension opens a short consent page (vi, en, cs): what the lookup sends, what the opt-in would add (the same field list as the side panel), that nothing more is sent until the user says yes, and that the side panel can change it. "No" has the focus and the primary style; closing the page keeps the default, off. The choice goes through RATE_UPDATE_SETTINGS to the same setting as the side panel toggle, which now follows changes made elsewhere. The sender guard accepts a tab sender only when it is this extension's consent page URL, exactly; content scripts stay refused. Updates and reloads do not open the page. Refs #1603, #1175 * fix(extension): consent page sets only the opt-in, defaults first - isOnboardingSender accepts the exact consent page URL and lets it set the opt-in alone; isPrivilegedExtensionSender refuses tab senders again. - onInstalled writes each missing default before opening the page, so a quick choice is never overwritten; a stored choice is kept. - The page status follows the stored choice. Refs #1603 * fix(extension): consent copy, choice state and docs after review - The consent page has its own opt-in text (onboardingOptInBody): the same fields as the side panel, tested, plus the server re-check and "no account, name, email or cookies". The lookup line lists every field lookup sends. - "Off by default" and how to reach the side panel toggle replace the vague line; vi, en and cs say the same. - Keyboard focus stays on the clicked button, the status is announced again on a repeat, and aria-pressed marks the stored choice. - The privacy policy and CHROMEWEBSTORE.md name the consent page; RELEASE.md has a test step and changelog line for it. Refs #1603 Co-Authored-By: Claude Opus 5.5 --- apps/extension/CHROMEWEBSTORE.md | 2 +- apps/extension/RELEASE.md | 5 +- apps/extension/scripts/build.mjs | 19 ++- apps/extension/src/_locales/cs/messages.json | 33 +++++ apps/extension/src/_locales/en/messages.json | 33 +++++ apps/extension/src/_locales/vi/messages.json | 33 +++++ .../src/background/service-worker.ts | 34 ++++- apps/extension/src/onboarding/onboarding.html | 138 ++++++++++++++++++ apps/extension/src/onboarding/onboarding.ts | 73 +++++++++ apps/extension/src/security-guards.ts | 32 +++- apps/extension/src/sidepanel/sidepanel.ts | 7 + apps/extension/src/test-extension-pipeline.ts | 63 ++++++++ apps/web/locales/cs.json | 2 +- apps/web/locales/en.json | 2 +- apps/web/locales/vi.json | 2 +- 15 files changed, 461 insertions(+), 17 deletions(-) create mode 100644 apps/extension/src/onboarding/onboarding.html create mode 100644 apps/extension/src/onboarding/onboarding.ts diff --git a/apps/extension/CHROMEWEBSTORE.md b/apps/extension/CHROMEWEBSTORE.md index 5f79ca19b..56bc201b9 100644 --- a/apps/extension/CHROMEWEBSTORE.md +++ b/apps/extension/CHROMEWEBSTORE.md @@ -45,7 +45,7 @@ This table lists exactly what `manifest.json` requests; `test-extension-pipeline 1. **Where it runs:** the content script loads on the 10 supported stores listed above (`shopee.vn`, `lazada.vn`, `tiki.vn`, TikTok Shop product pages on `www.tiktok.com` and `shop.tiktok.com`, `cellphones.com.vn`, `thegioididong.com`, `fptshop.com.vn`, `hoanghamobile.com`, `didongviet.vn`, `phongvu.vn`). It acts only on pages it recognises as a product page. The extension itself never opens or fetches pages the user did not open, never works around a store's bot checks and never solves captchas. 2. **Lookup (on by default, no opt-in):** on a recognised product page the extension sends `GET /api/v1/rate/extension/lookup` with the store, the product URL with query parameters and tracking tokens (`utm_*`, `sp_atk`, `xptdk`, `fbclid`, `gclid`) stripped locally, the product title, the store's item id and shop id, the selected variant, the listed price and the original price. The request is read-only and carries no account, name, email or cookies. -3. **Opt-in crowd signal (off by default):** only when the user turns on "Đóng góp phát hiện sản phẩm mới (chỉ khi bạn bật)" (English: "Help find new products (only when you turn it on)") in the Side Panel, the extension also sends `POST /api/v1/rate/extension/ingest` with the store, item id, shop id, cleaned URL, title, selected variant, price, original price, rating and rating count. The toggle's label lists these same fields (DOS/Rate#1230). Submissions go to a moderation queue, stored with the time the server received them; Rate.vn's servers may re-check the listing on the store. They never change public prices directly. +3. **Opt-in crowd signal (off by default):** only when the user chooses "Có" / "Yes" on the consent page opened once at first install, or turns on "Đóng góp phát hiện sản phẩm mới (chỉ khi bạn bật)" (English: "Help find new products (only when you turn it on)") in the Side Panel, the extension also sends `POST /api/v1/rate/extension/ingest` with the store, item id, shop id, cleaned URL, title, selected variant, price, original price, rating and rating count. The toggle's label lists these same fields (DOS/Rate#1230). Submissions go to a moderation queue, stored with the time the server received them; Rate.vn's servers may re-check the listing on the store. They never change public prices directly. Closing the consent page keeps the opt-in off, and updates or reloads do not open it again (DOS/Rate#1603). 4. **Stored in the browser only:** the server address and the opt-in choice (`chrome.storage.local`), and a session-only cache of the products viewed in each tab, the latest side panel search, their lookup results and a lookup wait time after a rate limit (`chrome.storage.session`). 5. **IP address:** like any web request, the Rate.vn server receives the user's IP address. The lookup and opt-in routes use it only for a per-minute request limit held in memory, and Rate.vn's code does not store it; the hosting provider keeps standard access logs for a short time. 6. **Side panel requests:** text the user types into the side panel search is sent to the same lookup endpoint; the side panel also reads public deals and merchant lists from the Rate.vn API. diff --git a/apps/extension/RELEASE.md b/apps/extension/RELEASE.md index 3d41739bd..2c44ff297 100644 --- a/apps/extension/RELEASE.md +++ b/apps/extension/RELEASE.md @@ -26,6 +26,7 @@ Release-readiness check of the extension built from `dev` at `fc9b820d` on 2026- - **Product pages:** on one product page each of Shopee, Tiki and CellphoneS, the pill appears and the side panel opens. - **Language:** set Chrome to English, then Czech. The panel and the verdict follow the browser language, and en/cs never show the Vietnamese review summary (#1385). - **Opt-in:** the crowd opt-in is off by default, and its label names the fields it sends (#1230, #1241). + - **Consent page:** a fresh install opens the consent page once, with "No" focused. Closing it leaves the opt-in off; "Yes" turns on the side panel toggle; reloading or updating the extension does not open it again (#1603). 5. Update the store listing: - Description and screenshots match the panel as it is now (the price list and the states from #1319). - The changelog below. @@ -44,7 +45,7 @@ Release-readiness check of the extension built from `dev` at `fc9b820d` on 2026- - Biểu tượng trên thanh công cụ cho biết nhanh giá đang xem có tốt không. Tin bán đã hết hàng không còn được tính vào giá so sánh. - Nhận diện sản phẩm chính xác hơn: tra "iPhone 16" không còn ra iPhone 16e. Trang danh mục và trang thương hiệu không bị nhầm là trang sản phẩm. - Gian hàng chưa có đánh giá được phân tích sẽ ghi "Chưa có dữ liệu", không hiện con số rủi ro. -- Lựa chọn "Đóng góp phát hiện sản phẩm mới" ghi rõ những gì được gửi đi và vẫn tắt mặc định. TikTok chỉ còn chạy trên trang sản phẩm. +- Lựa chọn "Đóng góp phát hiện sản phẩm mới" ghi rõ những gì được gửi đi và vẫn tắt mặc định. Lần cài đầu tiên, tiện ích hỏi bạn một lần. TikTok chỉ còn chạy trên trang sản phẩm. ### English @@ -55,4 +56,4 @@ Release-readiness check of the extension built from `dev` at `fc9b820d` on 2026- - The toolbar badge shows at a glance whether the price you see is a good one. Sold-out listings no longer count towards the price comparison. - More accurate product matching: "iPhone 16" no longer matches the iPhone 16e. Category and brand pages are no longer mistaken for product pages. - Sellers with no analysed reviews show "No data yet" instead of a risk number. -- The "Help find new products" option names exactly what it sends and is still off by default. On TikTok, the extension runs only on product pages. +- The "Help find new products" option names exactly what it sends and is still off by default. On first install, the extension asks you once. On TikTok, the extension runs only on product pages. diff --git a/apps/extension/scripts/build.mjs b/apps/extension/scripts/build.mjs index f8b0892fd..7486dd105 100644 --- a/apps/extension/scripts/build.mjs +++ b/apps/extension/scripts/build.mjs @@ -16,6 +16,7 @@ async function main() { fs.rmSync(distDir, { recursive: true, force: true }); ensureDir(path.join(distDir, 'icons')); ensureDir(path.join(distDir, 'sidepanel')); + ensureDir(path.join(distDir, 'onboarding')); // 1. Copy manifest.json fs.copyFileSync( @@ -53,8 +54,13 @@ async function main() { ...common, format: 'esm', entryPoints: { - 'background/service-worker': path.join(srcDir, 'background', 'service-worker.ts'), + 'background/service-worker': path.join( + srcDir, + 'background', + 'service-worker.ts' + ), 'sidepanel/sidepanel': path.join(srcDir, 'sidepanel', 'sidepanel.ts'), + 'onboarding/onboarding': path.join(srcDir, 'onboarding', 'onboarding.ts'), }, outdir: distDir, }); @@ -62,7 +68,11 @@ async function main() { ...common, format: 'iife', entryPoints: { - 'content/content-script': path.join(srcDir, 'content', 'content-script.ts'), + 'content/content-script': path.join( + srcDir, + 'content', + 'content-script.ts' + ), }, outdir: distDir, }); @@ -70,6 +80,11 @@ async function main() { path.join(srcDir, 'sidepanel', 'sidepanel.html'), path.join(distDir, 'sidepanel', 'sidepanel.html') ); + // The install consent page (DOS/Rate#1603). + fs.copyFileSync( + path.join(srcDir, 'onboarding', 'onboarding.html'), + path.join(distDir, 'onboarding', 'onboarding.html') + ); console.log( '✅ Built Rate Browser Extension (Manifest V3) into apps/extension/dist' diff --git a/apps/extension/src/_locales/cs/messages.json b/apps/extension/src/_locales/cs/messages.json index 6d2564ce0..f61b6bbf7 100644 --- a/apps/extension/src/_locales/cs/messages.json +++ b/apps/extension/src/_locales/cs/messages.json @@ -652,5 +652,38 @@ "content": "$2" } } + }, + "onboardingTitle": { + "message": "Rate.vn je připraveno" + }, + "onboardingLookup": { + "message": "Na stránkách produktů 10 podporovaných obchodů rozšíření posílá Rate.vn obchod, URL bez parametrů, název, ID produktu a obchodu, variantu, cenu a původní cenu, aby vyhledalo historii cen. Účet není potřeba." + }, + "onboardingQuestion": { + "message": "Chcete přispívat nabídkami, které si prohlížíte?" + }, + "onboardingNothingYet": { + "message": "Ve výchozím stavu vypnuto. Změnit to můžete kdykoli: klikněte na ikonu Rate.vn, otevře se boční panel, volba „Pomoc s hledáním nových produktů“." + }, + "onboardingNo": { + "message": "Ne, děkuji" + }, + "onboardingYes": { + "message": "Ano, chci přispívat" + }, + "onboardingSavedNo": { + "message": "Uloženo: nepřispívat. Tuto stránku můžete zavřít." + }, + "onboardingSavedYes": { + "message": "Uloženo: přispívat. Vypnout to můžete v bočním panelu." + }, + "onboardingSaveFailed": { + "message": "Neuloženo. Zkuste to znovu nebo zvolte v bočním panelu." + }, + "onboardingPrivacyLink": { + "message": "Zásady ochrany soukromí" + }, + "onboardingOptInBody": { + "message": "Když zvolíte „Ano“, každá nabídka, kterou otevřete, se také odešle do fronty ke kontrole Rate.vn s těmito údaji: obchod, ID produktu, ID obchodu, URL bez sledovacích parametrů, název produktu, zvolená varianta, cena, původní cena, počet hvězdiček a počet recenzí. Servery Rate.vn mohou nabídku v obchodě znovu ověřit. Bez účtu, jména, e-mailu a cookies." } } diff --git a/apps/extension/src/_locales/en/messages.json b/apps/extension/src/_locales/en/messages.json index f69af4fcd..2843358d2 100644 --- a/apps/extension/src/_locales/en/messages.json +++ b/apps/extension/src/_locales/en/messages.json @@ -652,5 +652,38 @@ "content": "$2" } } + }, + "onboardingTitle": { + "message": "Rate.vn is ready" + }, + "onboardingLookup": { + "message": "On product pages of the 10 supported stores, the extension sends Rate.vn the store, the URL without parameters, the title, the item and shop ids, the variant, the price and the original price to look up price history. No account needed." + }, + "onboardingQuestion": { + "message": "Would you like to contribute the listings you view?" + }, + "onboardingNothingYet": { + "message": "Off by default. Change it any time: click the Rate.vn icon to open the side panel, \"Help find new products\"." + }, + "onboardingNo": { + "message": "No, thanks" + }, + "onboardingYes": { + "message": "Yes, I want to contribute" + }, + "onboardingSavedNo": { + "message": "Saved: not contributing. You can close this page." + }, + "onboardingSavedYes": { + "message": "Saved: contributing. You can turn it off in the side panel." + }, + "onboardingSaveFailed": { + "message": "Not saved. Try again or choose in the side panel." + }, + "onboardingPrivacyLink": { + "message": "Privacy policy" + }, + "onboardingOptInBody": { + "message": "If you choose \"Yes\", every listing you open is also sent to the Rate.vn review queue with: store, product ID, shop ID, URL with tracking parameters removed, product name, selected variant, price, original price, star rating and number of reviews. Rate.vn's servers may re-check that listing on the store. No account, name, email or cookies are included." } } diff --git a/apps/extension/src/_locales/vi/messages.json b/apps/extension/src/_locales/vi/messages.json index 188394535..355d9682a 100644 --- a/apps/extension/src/_locales/vi/messages.json +++ b/apps/extension/src/_locales/vi/messages.json @@ -652,5 +652,38 @@ "content": "$2" } } + }, + "onboardingTitle": { + "message": "Rate.vn đã sẵn sàng" + }, + "onboardingLookup": { + "message": "Trên trang sản phẩm của 10 sàn được hỗ trợ, tiện ích gửi cho Rate.vn sàn, URL đã bỏ tham số, tên, mã sản phẩm và mã gian hàng, phiên bản, giá bán và giá gốc để tra lịch sử giá. Không cần tài khoản." + }, + "onboardingQuestion": { + "message": "Bạn có muốn đóng góp các tin bán bạn xem không?" + }, + "onboardingNothingYet": { + "message": "Mặc định tắt. Bạn đổi được bất cứ lúc nào: bấm biểu tượng Rate.vn để mở bảng bên cạnh, mục \"Đóng góp phát hiện sản phẩm mới\"." + }, + "onboardingNo": { + "message": "Không, cảm ơn" + }, + "onboardingYes": { + "message": "Có, tôi muốn đóng góp" + }, + "onboardingSavedNo": { + "message": "Đã lưu: không đóng góp. Bạn có thể đóng trang này." + }, + "onboardingSavedYes": { + "message": "Đã lưu: bật đóng góp. Bạn tắt được trong bảng bên cạnh." + }, + "onboardingSaveFailed": { + "message": "Chưa lưu được. Hãy thử lại hoặc chọn trong bảng bên cạnh." + }, + "onboardingPrivacyLink": { + "message": "Chính sách bảo mật" + }, + "onboardingOptInBody": { + "message": "Nếu bạn chọn \"Có\", mỗi tin bán bạn mở còn được gửi vào hàng đợi kiểm duyệt của Rate.vn với: sàn, mã sản phẩm, mã gian hàng, URL đã cắt tham số theo dõi, tên sản phẩm, phiên bản đang chọn, giá bán, giá gốc, số sao và số lượt đánh giá. Máy chủ Rate.vn có thể kiểm tra lại tin bán đó trên sàn. Không kèm tài khoản, tên, email hay cookie." } } diff --git a/apps/extension/src/background/service-worker.ts b/apps/extension/src/background/service-worker.ts index 0ba4c584a..268352c08 100644 --- a/apps/extension/src/background/service-worker.ts +++ b/apps/extension/src/background/service-worker.ts @@ -12,8 +12,10 @@ import { import { buildCrowdIngestPayload, DEFAULT_API_BASE_URL, + isOnboardingSender, isPrivilegedExtensionSender, normalizeAllowedApiBaseUrl, + ONBOARDING_PAGE_PATH, sanitizeExtensionProductSignal, selectFreshestSessionSignal, } from '../security-guards.js'; @@ -209,22 +211,39 @@ async function updateActionBadge( } } -chrome.runtime.onInstalled.addListener(async () => { +chrome.runtime.onInstalled.addListener(async (details) => { try { await chrome.sidePanel.setPanelBehavior({ openPanelOnActionClick: true }); + // Each missing setting gets its default; a stored choice is kept. const existing = await chrome.storage.local.get([ 'apiBaseUrl', 'crowdContributeOptIn', ]); + const defaults: Partial = {}; if (typeof existing.apiBaseUrl !== 'string') { - await chrome.storage.local.set({ - apiBaseUrl: DEFAULT_SETTINGS.apiBaseUrl, - crowdContributeOptIn: DEFAULT_SETTINGS.crowdContributeOptIn, - }); + defaults.apiBaseUrl = DEFAULT_SETTINGS.apiBaseUrl; + } + if (typeof existing.crowdContributeOptIn !== 'boolean') { + defaults.crowdContributeOptIn = DEFAULT_SETTINGS.crowdContributeOptIn; + } + if (Object.keys(defaults).length > 0) { + await chrome.storage.local.set(defaults); } } catch { // Ignore sidePanel or storage init errors during extension reload } + // On a first install only, after the defaults, ask about the opt-in + // (DOS/Rate#1603). The default stays off, so closing the page sends + // nothing extra. + if (details?.reason === 'install') { + try { + await chrome.tabs.create({ + url: chrome.runtime.getURL(ONBOARDING_PAGE_PATH), + }); + } catch { + // The side panel toggle still offers the choice. + } + } }); chrome.runtime.onMessage.addListener((message, sender, sendResponse) => { @@ -566,7 +585,9 @@ chrome.runtime.onMessage.addListener((message, sender, sendResponse) => { } if (msgType === 'RATE_UPDATE_SETTINGS') { - if (!isPrivilegedExtensionSender(sender)) { + // The install consent page may set the opt-in and nothing else. + const fromOnboarding = isOnboardingSender(sender); + if (!fromOnboarding && !isPrivilegedExtensionSender(sender)) { sendResponse({ ok: false, error: 'Settings can only be updated from the Rate extension sidepanel', @@ -577,6 +598,7 @@ chrome.runtime.onMessage.addListener((message, sender, sendResponse) => { try { const nextSettings: Partial = {}; if ( + !fromOnboarding && typeof message.apiBaseUrl === 'string' && message.apiBaseUrl.trim() ) { diff --git a/apps/extension/src/onboarding/onboarding.html b/apps/extension/src/onboarding/onboarding.html new file mode 100644 index 000000000..0f393d878 --- /dev/null +++ b/apps/extension/src/onboarding/onboarding.html @@ -0,0 +1,138 @@ + + + + + + + + Rate.vn + + + +
+
+

+

+

+

+

+
+ + +
+

+

+ +

+
+
+ + + diff --git a/apps/extension/src/onboarding/onboarding.ts b/apps/extension/src/onboarding/onboarding.ts new file mode 100644 index 000000000..bf8ba484f --- /dev/null +++ b/apps/extension/src/onboarding/onboarding.ts @@ -0,0 +1,73 @@ +/** + * The consent screen shown once at install (DOS/Rate#1603, JOY 2026-11-04: + * ask at install and keep the side panel toggle, default off). The choice + * goes to the same setting as the side panel toggle; closing the page leaves + * the default, off, so nothing extra is sent. + */ +import { applyDocumentMessages, t } from '../i18n.js'; + +let busy = false; + +/** Marks the stored choice on its button. */ +function markChoice(optIn: boolean | null): void { + for (const [id, value] of [ + ['btn-no', false], + ['btn-yes', true], + ] as const) { + document + .getElementById(id) + ?.setAttribute('aria-pressed', String(optIn === value)); + } +} + +async function choose(optIn: boolean): Promise { + // A flag, not disabled buttons, so keyboard focus stays where it was. + if (busy) return; + busy = true; + const status = document.getElementById('status'); + // Cleared first, so the same choice made twice is announced again. + if (status) status.textContent = ''; + try { + const res = (await chrome.runtime.sendMessage({ + type: 'RATE_UPDATE_SETTINGS', + crowdContributeOptIn: optIn, + })) as { ok?: boolean; settings?: { crowdContributeOptIn?: boolean } }; + if (!res?.ok || res.settings?.crowdContributeOptIn !== optIn) { + throw new Error('not saved'); + } + markChoice(optIn); + if (status) { + status.textContent = t( + optIn ? 'onboardingSavedYes' : 'onboardingSavedNo' + ); + } + } catch { + if (status) status.textContent = t('onboardingSaveFailed'); + } finally { + busy = false; + } +} + +document.addEventListener('DOMContentLoaded', () => { + applyDocumentMessages(document); + document.title = t('onboardingTitle'); + document + .getElementById('btn-no') + ?.addEventListener('click', () => void choose(false)); + document + .getElementById('btn-yes') + ?.addEventListener('click', () => void choose(true)); + // The status always shows the stored choice, also one changed in the side + // panel after this page saved it. + if (typeof chrome === 'undefined') return; + chrome.storage.onChanged?.addListener((changes, areaName) => { + const change = changes.crowdContributeOptIn; + const status = document.getElementById('status'); + if (areaName === 'local' && change && status) { + markChoice(change.newValue === true); + status.textContent = t( + change.newValue === true ? 'onboardingSavedYes' : 'onboardingSavedNo' + ); + } + }); +}); diff --git a/apps/extension/src/security-guards.ts b/apps/extension/src/security-guards.ts index 869032254..d25026002 100644 --- a/apps/extension/src/security-guards.ts +++ b/apps/extension/src/security-guards.ts @@ -65,6 +65,31 @@ export function normalizeAllowedApiBaseUrl( } } +/** The consent page opened once at install (DOS/Rate#1603). */ +export const ONBOARDING_PAGE_PATH = 'onboarding/onboarding.html'; + +/** + * The install consent page, an extension page in a tab: exactly its URL in + * this extension, never a content script (whose url is the web page). It may + * only set the opt-in (DOS/Rate#1603). + */ +export function isOnboardingSender( + sender: { id?: string; tab?: unknown; url?: string } | null | undefined, + expectedRuntimeId?: string +): boolean { + if (!sender || typeof sender !== 'object' || Array.isArray(sender)) { + return false; + } + const runtimeId = + expectedRuntimeId ?? + (typeof chrome !== 'undefined' ? chrome.runtime?.id : undefined); + if (!runtimeId) return false; + if (typeof sender.id === 'string' && sender.id !== runtimeId) return false; + return ( + sender.url === `chrome-extension://${runtimeId}/${ONBOARDING_PAGE_PATH}` + ); +} + export function isPrivilegedExtensionSender( sender: { id?: string; @@ -76,9 +101,6 @@ export function isPrivilegedExtensionSender( if (!sender || typeof sender !== 'object' || Array.isArray(sender)) { return false; } - if (sender.tab != null) { - return false; - } const runtimeId = expectedRuntimeId ?? (typeof globalThis !== 'undefined' && @@ -90,6 +112,10 @@ export function isPrivilegedExtensionSender( .runtime!.id : undefined); + if (sender.tab != null) { + return false; + } + if ( runtimeId && typeof sender.id === 'string' && diff --git a/apps/extension/src/sidepanel/sidepanel.ts b/apps/extension/src/sidepanel/sidepanel.ts index 314aa2424..bb016e453 100644 --- a/apps/extension/src/sidepanel/sidepanel.ts +++ b/apps/extension/src/sidepanel/sidepanel.ts @@ -820,6 +820,13 @@ document.addEventListener('DOMContentLoaded', () => { } }); + chrome.storage.onChanged?.addListener((changes, areaName) => { + const change = changes.crowdContributeOptIn; + if (areaName === 'local' && change && chkOptIn) { + chkOptIn.checked = change.newValue === true; + } + }); + chkOptIn?.addEventListener('change', async () => { await chrome.runtime.sendMessage({ type: 'RATE_UPDATE_SETTINGS', diff --git a/apps/extension/src/test-extension-pipeline.ts b/apps/extension/src/test-extension-pipeline.ts index c641cfe40..a7ffce1d3 100644 --- a/apps/extension/src/test-extension-pipeline.ts +++ b/apps/extension/src/test-extension-pipeline.ts @@ -19,6 +19,7 @@ import { DEFAULT_API_BASE_URL, MAX_SPARKLINE_POINTS, escapeSafeExternalHref, + isOnboardingSender, isPrivilegedExtensionSender, normalizeAllowedApiBaseUrl, normalizeFiveStarRating, @@ -467,6 +468,52 @@ async function runTests() { }), true ); + // The install consent page is an extension page in a tab: it is never a + // privileged sender, and only isOnboardingSender, which lets it set the + // opt-in alone, accepts it by its exact URL (DOS/Rate#1603). + const onboardingUrl = + 'chrome-extension://abcdefghijklmnop/onboarding/onboarding.html'; + assert.equal( + isPrivilegedExtensionSender( + { tab: { id: 7 }, url: onboardingUrl }, + 'abcdefghijklmnop' + ), + false + ); + assert.equal( + isOnboardingSender( + { tab: { id: 7 }, url: onboardingUrl }, + 'abcdefghijklmnop' + ), + true + ); + for (const sender of [ + { tab: { id: 7 }, url: 'https://shopee.vn/onboarding/onboarding.html' }, + { + tab: { id: 7 }, + url: 'chrome-extension://otherextensionid/onboarding/onboarding.html', + }, + { + tab: undefined, + url: 'chrome-extension://abcdefghijklmnop/sidepanel/sidepanel.html', + }, + { tab: { id: 7 }, url: `${onboardingUrl}?x=1` }, + { tab: { id: 7 }, url: `${onboardingUrl}#x` }, + { tab: { id: 7 }, url: undefined }, + { id: 'otherextensionid', tab: { id: 7 }, url: onboardingUrl }, + null, + ]) { + assert.equal( + isOnboardingSender(sender as never, 'abcdefghijklmnop'), + false, + JSON.stringify(sender) + ); + } + assert.equal( + isOnboardingSender({ tab: { id: 7 }, url: onboardingUrl }), + false, + 'without a known runtime id nothing is the consent page' + ); assert.equal( escapeSafeExternalHref('javascript:alert(1)'), 'https://rate.vn', @@ -605,7 +652,23 @@ async function runTests() { ); } assert.ok(!/Chỉ gửi URL/.test(optInText)); + // The install consent page names the same fields (DOS/Rate#1603). + const consentText = catalogue.onboardingOptInBody?.message ?? ''; + for (const key of Object.values(CROWD_INGEST_FIELD_LABELS)) { + assert.ok( + consentText.includes(catalogue[key]?.message ?? '\u0000'), + `${locale} consent text lacks ${key}` + ); + } } + assert.ok( + fs + .readFileSync( + path.resolve(__dirname, 'onboarding/onboarding.html'), + 'utf8' + ) + .includes('data-i18n="onboardingOptInBody"') + ); // The store listing names the extension as the vi catalogue does. const viName = readLocaleCatalogues().vi.extName?.message; assert.equal( diff --git a/apps/web/locales/cs.json b/apps/web/locales/cs.json index bd829dc46..b9c405417 100644 --- a/apps/web/locales/cs.json +++ b/apps/web/locales/cs.json @@ -1534,7 +1534,7 @@ "privacyS4Title": "4. Soukromé údaje prodejců", "privacyS4Desc": "Soukromé údaje prodejce (telefonní číslo, bankovní účet) jsou v polích, která veřejnost nemůže číst. Web i API mohou číst jen veřejná pole.", "privacyS5Title": "5. Rozšíření prohlížeče Rate.vn", - "privacyS5Desc": "Na stránkách produktů 10 podporovaných obchodů rozšíření posílá Rate.vn obchod, URL produktu bez parametrů, název produktu, ID položky a obchodu, zvolenou variantu, cenu a původní cenu, aby vyhledalo historii cen; toto vyhledávání je zapnuté ve výchozím stavu, jen čte a nepotřebuje účet. Text, který napíšete do vyhledávacího pole rozšíření, se také odešle k vyhledání. Jen když zapnete \"Pomoc s hledáním nových produktů (jen když ji zapnete)\" (ve výchozím stavu vypnuto), posílá také danou nabídku s hodnocením a počtem hodnocení do fronty ke kontrole, uloženou s časem, kdy ji server přijal; servery Rate.vn mohou nabídku v obchodě znovu ověřit a tato data sama veřejné ceny nemění. Rozšíření neposílá účet, jméno, e-mail ani cookies, neotevírá stránky, které jste neotevřeli, a ve vašem prohlížeči ukládá jen adresu serveru, volbu opt-in a mezipaměť platnou po dobu relace s produkty, které jste si prohlédli, vaším posledním hledáním, výsledky vyhledávání a dobou čekání na další vyhledání. Jako u každého webového požadavku server obdrží vaši IP adresu: kód Rate.vn ji používá jen v paměti pro limit požadavků za minutu a náš poskytovatel hostingu uchovává běžné přístupové záznamy po krátkou dobu.", + "privacyS5Desc": "Na stránkách produktů 10 podporovaných obchodů rozšíření posílá Rate.vn obchod, URL produktu bez parametrů, název produktu, ID položky a obchodu, zvolenou variantu, cenu a původní cenu, aby vyhledalo historii cen; toto vyhledávání je zapnuté ve výchozím stavu, jen čte a nepotřebuje účet. Text, který napíšete do vyhledávacího pole rozšíření, se také odešle k vyhledání. Jen když zvolíte \"Ano\" na stránce zobrazené při instalaci nebo zapnete \"Pomoc s hledáním nových produktů (jen když ji zapnete)\" v bočním panelu (ve výchozím stavu vypnuto), posílá také danou nabídku s hodnocením a počtem hodnocení do fronty ke kontrole, uloženou s časem, kdy ji server přijal; servery Rate.vn mohou nabídku v obchodě znovu ověřit a tato data sama veřejné ceny nemění. Rozšíření neposílá účet, jméno, e-mail ani cookies, neotevírá stránky, které jste neotevřeli, a ve vašem prohlížeči ukládá jen adresu serveru, volbu opt-in a mezipaměť platnou po dobu relace s produkty, které jste si prohlédli, vaším posledním hledáním, výsledky vyhledávání a dobou čekání na další vyhledání. Jako u každého webového požadavku server obdrží vaši IP adresu: kód Rate.vn ji používá jen v paměti pro limit požadavků za minutu a náš poskytovatel hostingu uchovává běžné přístupové záznamy po krátkou dobu.", "privacySide1Title": "Ověřte si otevřený kód", "privacySide1Desc": "Pravidla přístupu k datům a způsob ukládání hlasů jsou veřejné v repozitáři na GitHubu.", "privacySide1Btn": "Technická a API dokumentace", diff --git a/apps/web/locales/en.json b/apps/web/locales/en.json index b7d1b41df..fc5960b1f 100644 --- a/apps/web/locales/en.json +++ b/apps/web/locales/en.json @@ -1534,7 +1534,7 @@ "privacyS4Title": "4. Sellers' private details", "privacyS4Desc": "A seller's private details (phone number, bank account) are kept in fields the public cannot read. The website and the API can only read the public fields.", "privacyS5Title": "5. The Rate.vn browser extension", - "privacyS5Desc": "On product pages of the 10 supported stores, the extension sends Rate.vn the store, the product URL without its parameters, the product title, the item and shop ids, the selected variant, the price and the original price to look up price history; this lookup is on by default, read-only and needs no account. Text you type into the extension's search box is also sent to look it up. Only if you turn on \"Help find new products (only when you turn it on)\" (off by default) does it also send that listing, with its rating and rating count, to a moderation queue, stored with the time the server received it; Rate.vn's servers may re-check that listing on the store, and this data does not change public prices by itself. The extension sends no account, name, email or cookies, opens no page you did not open, and keeps only the server address, the opt-in choice and a session-only cache of the products you viewed, your latest search, their lookup results and a lookup wait time in your browser. Like any web request, the server receives your IP address: Rate.vn's code uses it only in memory for a per-minute request limit, and our hosting provider keeps standard access logs for a short time.", + "privacyS5Desc": "On product pages of the 10 supported stores, the extension sends Rate.vn the store, the product URL without its parameters, the product title, the item and shop ids, the selected variant, the price and the original price to look up price history; this lookup is on by default, read-only and needs no account. Text you type into the extension's search box is also sent to look it up. Only if you choose \"Yes\" on the page shown at install or turn on \"Help find new products (only when you turn it on)\" in the side panel (off by default) does it also send that listing, with its rating and rating count, to a moderation queue, stored with the time the server received it; Rate.vn's servers may re-check that listing on the store, and this data does not change public prices by itself. The extension sends no account, name, email or cookies, opens no page you did not open, and keeps only the server address, the opt-in choice and a session-only cache of the products you viewed, your latest search, their lookup results and a lookup wait time in your browser. Like any web request, the server receives your IP address: Rate.vn's code uses it only in memory for a per-minute request limit, and our hosting provider keeps standard access logs for a short time.", "privacySide1Title": "Check the open code", "privacySide1Desc": "The data access rules and how votes are stored are public in the GitHub repository.", "privacySide1Btn": "Technical and API docs", diff --git a/apps/web/locales/vi.json b/apps/web/locales/vi.json index a7009f971..c4564139d 100644 --- a/apps/web/locales/vi.json +++ b/apps/web/locales/vi.json @@ -1534,7 +1534,7 @@ "privacyS4Title": "4. Thông tin riêng của gian hàng", "privacyS4Desc": "Thông tin riêng của gian hàng (số điện thoại, tài khoản ngân hàng) nằm ở các trường mà công chúng không đọc được. Trang web và API chỉ đọc được các trường công khai.", "privacyS5Title": "5. Tiện ích trình duyệt Rate.vn", - "privacyS5Desc": "Trên trang sản phẩm của 10 sàn được hỗ trợ, tiện ích gửi cho Rate.vn sàn, URL sản phẩm đã bỏ tham số, tên sản phẩm, mã sản phẩm và mã gian hàng, phiên bản đang chọn, giá bán và giá gốc để tra lịch sử giá; việc tra cứu này bật sẵn, chỉ đọc và không cần tài khoản. Nội dung bạn gõ vào ô tìm kiếm của tiện ích cũng được gửi để tra cứu. Chỉ khi bạn bật \"Đóng góp phát hiện sản phẩm mới (chỉ khi bạn bật)\" (mặc định tắt), tiện ích còn gửi tin bán đó kèm số sao và số lượt đánh giá vào hàng đợi kiểm duyệt, lưu kèm thời điểm máy chủ nhận; máy chủ Rate.vn có thể kiểm tra lại tin bán đó trên sàn, và dữ liệu này không tự đổi giá công khai. Tiện ích không gửi tài khoản, tên, email hay cookie, không mở trang bạn không mở, và chỉ lưu trên trình duyệt của bạn địa chỉ máy chủ, lựa chọn có đóng góp hay không, cùng bộ nhớ tạm theo phiên gồm các sản phẩm bạn vừa xem, lần tìm kiếm gần nhất, kết quả tra cứu và thời gian chờ khi bị giới hạn tra cứu. Như mọi yêu cầu web, máy chủ nhận địa chỉ IP của bạn: mã của Rate.vn chỉ dùng nó trong bộ nhớ để giới hạn số yêu cầu mỗi phút, còn nhà cung cấp hosting giữ nhật ký truy cập thông thường trong thời gian ngắn.", + "privacyS5Desc": "Trên trang sản phẩm của 10 sàn được hỗ trợ, tiện ích gửi cho Rate.vn sàn, URL sản phẩm đã bỏ tham số, tên sản phẩm, mã sản phẩm và mã gian hàng, phiên bản đang chọn, giá bán và giá gốc để tra lịch sử giá; việc tra cứu này bật sẵn, chỉ đọc và không cần tài khoản. Nội dung bạn gõ vào ô tìm kiếm của tiện ích cũng được gửi để tra cứu. Chỉ khi bạn chọn \"Có\" ở trang hiện ra lúc cài đặt hoặc bật \"Đóng góp phát hiện sản phẩm mới (chỉ khi bạn bật)\" trong bảng bên cạnh (mặc định tắt), tiện ích còn gửi tin bán đó kèm số sao và số lượt đánh giá vào hàng đợi kiểm duyệt, lưu kèm thời điểm máy chủ nhận; máy chủ Rate.vn có thể kiểm tra lại tin bán đó trên sàn, và dữ liệu này không tự đổi giá công khai. Tiện ích không gửi tài khoản, tên, email hay cookie, không mở trang bạn không mở, và chỉ lưu trên trình duyệt của bạn địa chỉ máy chủ, lựa chọn có đóng góp hay không, cùng bộ nhớ tạm theo phiên gồm các sản phẩm bạn vừa xem, lần tìm kiếm gần nhất, kết quả tra cứu và thời gian chờ khi bị giới hạn tra cứu. Như mọi yêu cầu web, máy chủ nhận địa chỉ IP của bạn: mã của Rate.vn chỉ dùng nó trong bộ nhớ để giới hạn số yêu cầu mỗi phút, còn nhà cung cấp hosting giữ nhật ký truy cập thông thường trong thời gian ngắn.", "privacySide1Title": "Kiểm tra mã nguồn công khai", "privacySide1Desc": "Quy tắc phân quyền dữ liệu và cách lưu bình chọn đều công khai trong repository trên GitHub.", "privacySide1Btn": "Xem tài liệu kỹ thuật và API", From d50de0db601103c93ea7eb20f5261fc6ae198523 Mon Sep 17 00:00:00 2001 From: JOY <5027251+JOY@users.noreply.github.com> Date: Wed, 4 Nov 2026 12:40:57 +0700 Subject: [PATCH 2/7] feat(importer): review ai points job calls models through the dos.ai gateway (#1607) * feat(importer): review ai points job calls models through the dos.ai gateway JOY decided on 2026-11-04 that the job uses dos.ai, not the Anthropic API (D2 of the AI review points design). The job now sends its batch through the gateway's OpenAI-compatible Files and Batch API with DOS_AI_API_KEY (DOS_AI_BASE_URL overrides the URL). That Batch API serves dos, qwen3.8 and deepseek models only, so the default model is dos (Qwen3.8 27B on DOS GPUs) and qwen3.8-max is the fallback, with the same monthly caps. Caps, kill switch, process-once, evidence checks and quality stops are unchanged. A cut-off or filtered answer (finish_reason length or content_filter) and an errored line are stored as failed. Cached tokens count at the full input price, an upper bound. @anthropic-ai/sdk is removed. The proposal notes the D2 change. Refs #1531 * fix(importer): dos.ai batch client never creates a batch twice Review findings on the dos.ai client (#1531): - POST /batches is retried only on 429; a lost answer throws and says a batch may exist, instead of creating and billing a second one. - A batch failed as a whole, one with no output, or one whose output file is gone (404) closes its row and stores nothing, so its reviews are due again without spending an attempt; a line not run for want of credits is unbilled, not failed. - Each call's timeout covers reading the body. - Tests: create-once on a network error and a 502, retry on 429, failed and purged batches, 402 lines, the request body; batch ids renamed. - The proposal notes the dos.ai batch path in sections 4 and 5. Refs #1531 Co-Authored-By: Claude Opus 5.5 --- .../2026-11-03-ai-review-points.html | 16 +- apps/importer/.env.example | 7 +- apps/importer/package.json | 1 - .../shopee/__tests__/review-ai-points.test.ts | 300 +++++++++++++--- .../src/shopee/jobs/review-ai-points.ts | 322 +++++++++++++++--- apps/importer/src/shopee/review-ai-prompt.ts | 33 +- pnpm-lock.yaml | 52 --- 7 files changed, 583 insertions(+), 148 deletions(-) diff --git a/Docs/Proposals/2026-11-03-ai-review-points.html b/Docs/Proposals/2026-11-03-ai-review-points.html index cca943450..b5580e0db 100644 --- a/Docs/Proposals/2026-11-03-ai-review-points.html +++ b/Docs/Proposals/2026-11-03-ai-review-points.html @@ -122,6 +122,15 @@

2.3 Dạng kết quả (structured output, JSON có schema)

criterion_key trong schema là một danh sách liệt kê đúng các khoá của ngành hàng, nên AI không thể trả một khoá lạ.

3. Chọn model

+
Cập nhật 04/11/2026: JOY đổi D2, gọi qua dos.ai. Job gọi model qua cổng dos.ai (Batch API kiểu OpenAI: POST /v1/files, POST /v1/batches), không gọi thẳng Anthropic. Batch API của dos.ai chỉ nhận model của chính DOS và Alibaba (dos, qwen3.8-*, deepseek-*), không nhận Claude, và tính nửa giá. Vì vậy: +
    +
  • Model mặc định là dos (Qwen3.8 27B chạy trên GPU của DOS), giá niêm yết 0,25 / 1,50 USD mỗi 1 triệu token, còn một nửa khi đi qua Batch API.
    • +
  • Model dự phòng là qwen3.8-max (2 / 6 USD), chỉ dùng nếu dos không đạt ngưỡng ở mục 7.1. Cách chọn giống D1: rẻ trước, mạnh hơn khi bộ mẫu bảo cần.
    • +
  • Giới hạn tiền giữ nguyên mức: 5 USD mỗi tháng cho dos, 15 USD cho qwen3.8-max. Token đọc lại từ cache được tính đủ giá đọc vào, nên con số job ghi là mức trần; hoá đơn thật nằm trong sổ của dos.ai.
    • +
  • Kết quả vẫn ép theo schema (response_format json_schema), và mọi kiểm tra ở mục 7.2 giữ nguyên.
    • +
  • Khoá: DOS_AI_API_KEY (khoá dos_sk_ tạo ở dev.dos.me), thay cho ANTHROPIC_API_KEY.
    • +
+ Bảng dưới đây là phương án Claude ban đầu, giữ lại để tham khảo.
@@ -132,6 +141,7 @@

3. Chọn model

Gọi qua đâu. Importer hiện gọi AI qua chuẩn OpenAI (biến AI_PROVIDER, cổng dos-ai). Đề xuất bước này gọi thẳng Claude API bằng SDK chính thức của Anthropic, vì cần structured outputs, Batch API và prompt caching. Khoá API chỉ nằm trong secret của Rate-Services. Câu hỏi này để JOY quyết ở D2.

4. Gửi theo lô và chỉ xử lý một lần

+

Từ 04/11/2026 lô được gửi qua Batch API của dos.ai, theo chuẩn OpenAI (file JSONL rồi tạo batch). Dấu đã xử lý, mã lô lưu ngay khi tạo và các luật thử lại bên dưới giữ nguyên. Phần nói về cache và giá Claude chỉ còn để tham khảo.

  • Gom lô: mỗi yêu cầu chứa tối đa 20 review cùng ngành hàng. Phần hướng dẫn của ngành hàng (khoảng 1.500 token) dùng chung, nên được cache: các yêu cầu sau chỉ trả 10% cho phần này. Tối thiểu để cache trên Haiku 5.5 là 512 token, phần hướng dẫn vượt mức này.
  • Batch API: mỗi lần chạy gửi một batch, chờ kết quả, rồi mới ghi. Việc này không cần trả lời ngay.
    • @@ -156,6 +166,7 @@

    4. Gửi theo lô và chỉ xử lý một lần

5. Ước tính chi phí

+

Từ 04/11/2026: với dos qua Batch API của dos.ai (0,125 / 0,75 USD mỗi 1 triệu token sau khi giảm 50%), 1.000 review tốn khoảng 0,12 USD (200.000 token đọc vào, 120.000 token viết ra); kho ~250 review dưới 0,05 USD. Bảng dưới là ước tính cũ cho Claude.

Giả định:

  • Một review sàn qua điều kiện ở mục 1 dài trung bình khoảng 200 token. Tiếng Việt có dấu tốn token hơn tiếng Anh nên ước rộng tay.
    • @@ -187,7 +198,7 @@

    6. Nơi chạy và ranh giới open-core

  • test chạy bằng dữ liệu giả và một client AI giả. Không commit chữ review thật từ sàn vào repo.
-
  • DOS/Rate-Services (riêng tư): chỉ thêm một lịch chạy, giữ khoá ANTHROPIC_API_KEY và tài khoản rate_worker. Thứ tự mỗi đêm: crawl xong, rồi review-ai-points, rồi product-points (để gom luôn các dòng mới), rồi review-values. Không chạy trùng giờ crawl. Không có gì của Rate-Services (proxy, scraper, API nội bộ của sàn) đi vào DOS/Rate.
    • +
  • DOS/Rate-Services (riêng tư): chỉ thêm một lịch chạy, giữ khoá DOS_AI_API_KEY của cổng dos.ai (trước là ANTHROPIC_API_KEY, đổi theo D2 ngày 04/11) và tài khoản rate_worker. Thứ tự mỗi đêm: crawl xong, rồi review-ai-points, rồi product-points (để gom luôn các dòng mới), rồi review-values. Không chạy trùng giờ crawl. Không có gì của Rate-Services (proxy, scraper, API nội bộ của sàn) đi vào DOS/Rate.
  • Database:
    • Job đọc bằng rate_worker.
      • @@ -258,7 +269,8 @@

      10. Các quyết định cần JOY

      D1. Model. Haiku 5.5 làm mặc định, chuyển lên Sonnet 5.5 nếu không đạt ngưỡng ở 7.1.

      Đề xuất: duyệt. Chi phí cả hai đều nhỏ; chất lượng quyết định bằng bộ mẫu, không bằng đoán.

      D2. Gọi AI qua đâu. (a) Thẳng Claude API bằng SDK Anthropic, khoá trong Rate-Services; (b) qua cổng dos-ai kiểu OpenAI như importer đang làm. -

      Đề xuất: (a). Cần structured outputs, Batch API và prompt caching. Qua cổng kiểu OpenAI thì mất hoặc phải tự làm lại các tính năng này.

      +

      Đề xuất: (a). Cần structured outputs, Batch API và prompt caching. Qua cổng kiểu OpenAI thì mất hoặc phải tự làm lại các tính năng này.

      +

      Quyết định 04/11/2026: JOY chọn (b), gọi qua dos.ai ("Tao có nguyên 1 cái dos.ai cơ mà?"). Cổng có Batch API và nhận json_schema, nên job giữ được gửi theo lô, chỉ xử lý một lần và ép schema. Model đổi thành dos / qwen3.8-max vì Batch API của cổng không nhận Claude (xem mục 3).

      D3. Điều kiện chọn review. Các ngưỡng ở mục 1: từ 40 ký tự và 8 từ, tiếng Việt hoặc tiếng Anh, qua cổng chặn, chưa có dòng của người viết.

      Đề xuất: duyệt. Chỉnh lại sau lần chạy thử đầu nếu bỏ sót quá nhiều.

      D4. Trọng số dòng AI trong D9 (mục 9). diff --git a/apps/importer/.env.example b/apps/importer/.env.example index 1c33ceca6..01927701e 100644 --- a/apps/importer/.env.example +++ b/apps/importer/.env.example @@ -19,8 +19,13 @@ AI_BASE_URL= AI_MODEL= AI_TIMEOUT_MS=25000 -# Provider-specific keys +# Provider-specific keys. The review AI points job (jobs/review-ai-points.ts) uses only +# DOS_AI_API_KEY (a dos_sk_ key) and DOS_AI_BASE_URL (default https://api.dos.me/v1), +# with REVIEW_AI_POINTS_ENABLED=true and REVIEW_AI_MODEL (dos or qwen3.8-max). DOS_AI_API_KEY= +DOS_AI_BASE_URL= +REVIEW_AI_POINTS_ENABLED= +REVIEW_AI_MODEL= OPENAI_API_KEY= GEMINI_API_KEY= DEEPSEEK_API_KEY= diff --git a/apps/importer/package.json b/apps/importer/package.json index fe8fa907a..143cd899f 100644 --- a/apps/importer/package.json +++ b/apps/importer/package.json @@ -58,7 +58,6 @@ "node": ">=22.0.0" }, "dependencies": { - "@anthropic-ai/sdk": "1.3.0", "@repo/shared-data": "workspace:*", "@supabase/supabase-js": "^2.117.2", "dotenv": "^18.0.8" diff --git a/apps/importer/src/shopee/__tests__/review-ai-points.test.ts b/apps/importer/src/shopee/__tests__/review-ai-points.test.ts index 04c896aa7..8669eb436 100644 --- a/apps/importer/src/shopee/__tests__/review-ai-points.test.ts +++ b/apps/importer/src/shopee/__tests__/review-ai-points.test.ts @@ -16,7 +16,10 @@ import { import { aiCandidates, buildRequests, + BatchGoneError, collectBatch, + dosAiReviewAiClient, + fromDosAiOutputLine, qualityStop, runReviewAiPointsJob, type AiMarkerRow, @@ -169,24 +172,25 @@ function testPrompt() { too_many: 1, }); - // Prices: Haiku 5.5 at batch prices, cache reads and writes included. + // Prices: dos.ai list prices at the Batch API's half, cached tokens at the full input + // price (an upper bound). assert.equal( - batchCostUsd('claude-haiku-5-5', { + batchCostUsd('dos', { input_tokens: 1_000_000, output_tokens: 1_000_000, cache_read_input_tokens: 1_000_000, cache_creation_input_tokens: 1_000_000, }).toFixed(4), - ((0.1 + 0.5 + 0.01 + 0.125) * 0.5).toFixed(4) + ((0.25 + 1.5 + 0.25 + 0.25) * 0.5).toFixed(4) ); assert.equal( - batchCostUsd('claude-sonnet-5-5', { output_tokens: 1_000_000 }), - 20 * batchCostUsd('claude-haiku-5-5', { output_tokens: 1_000_000 }) + batchCostUsd('qwen3.8-max', { output_tokens: 1_000_000 }), + 4 * batchCostUsd('dos', { output_tokens: 1_000_000 }) ); - const ceiling = reserveUsd('claude-haiku-5-5', [ + const ceiling = reserveUsd('dos', [ { instructions: 'x'.repeat(3000), message: 'y'.repeat(8000) }, ]); - assert.ok(ceiling > 0.0015 && ceiling < 0.003, `one request's ceiling ${ceiling}`); + assert.ok(ceiling > 0.004 && ceiling < 0.007, `one request's ceiling ${ceiling}`); console.log(' ✅ hash, qualifying text, schema, evidence, line checks and prices verified'); } @@ -252,16 +256,23 @@ function testCandidates() { textHash: hash, category: i < 41 ? 'smartphones' : 'laptops', })); - const built = buildRequests(many, 'claude-haiku-5-5'); + const built = buildRequests(many, 'dos'); assert.deepEqual( built.manifest.map(r => `${r.custom_id}:${r.reviews.length}`), ['laptops-0:4', 'smartphones-1:20', 'smartphones-2:20', 'smartphones-3:1'] ); + // One line of an OpenAI-format batch file, as dos.ai's Batch API takes it. const request = built.requests[1]!; - assert.equal(request.params.model, 'claude-haiku-5-5'); - assert.equal(request.params.output_config.effort, 'low'); - assert.equal(request.params.system[0]!.cache_control.type, 'ephemeral'); - const sent = JSON.parse(request.params.messages[0]!.content.split('\n').slice(1).join('\n')) as { + assert.deepEqual([request.method, request.url], ['POST', '/v1/chat/completions']); + assert.equal(request.body.model, 'dos'); + assert.equal(request.body.stream, false); + assert.equal(request.body.messages[0].role, 'system'); + assert.equal(request.body.messages[0].content, reviewAiInstructions('smartphones')); + assert.deepEqual( + [request.body.response_format.type, request.body.response_format.json_schema.strict], + ['json_schema', true] + ); + const sent = JSON.parse(request.body.messages[1].content.split('\n').slice(1).join('\n')) as { ref: number; text: string; }[]; @@ -269,7 +280,7 @@ function testCandidates() { sent.map(r => r.ref), Array.from({ length: 20 }, (_, i) => i + 1) ); - assert.ok(!request.params.messages[0]!.content.includes(id(100)), 'no review id is sent'); + assert.ok(!JSON.stringify(request).includes(id(100)), 'no review id is sent'); console.log(' ✅ the gate, the process-once rule and the requests verified'); } @@ -351,7 +362,7 @@ function testCollect() { ]; const texts = new Map([1, 2, 3, 4, 5, 6, 7].map(n => [id(n), LONG])); const collected = collectBatch( - { id: 'msgbatch_test0001', model: 'claude-haiku-5-5', prompt_version: 'v1', manifest }, + { id: 'batch_test0001', model: 'dos', prompt_version: 'v1', manifest }, results, texts ); @@ -366,16 +377,16 @@ function testCollect() { for (const n of [3, 4, 5, 6, 7]) assert.equal(rows.get(id(n))!.failed, true, `review ${n} failed`); assert.equal(collected.failedReviews, 5); - assert.equal(rows.get(id(1))!.batch_id, 'msgbatch_test0001'); + assert.equal(rows.get(id(1))!.batch_id, 'batch_test0001'); assert.equal(collected.usage.input_tokens, 16000, 'cut-off and refused requests are paid for'); assert.equal( collected.costUsd, - batchCostUsd('claude-haiku-5-5', { input_tokens: 16000, output_tokens: 3600 }) + batchCostUsd('dos', { input_tokens: 16000, output_tokens: 3600 }) ); const errored = collectBatch( { - id: 'msgbatch_test0002', - model: 'claude-haiku-5-5', + id: 'batch_test0002', + model: 'dos', prompt_version: 'v1', manifest: manifest.slice(1, 2), }, @@ -476,7 +487,7 @@ async function testJob() { assert.equal(batch.status, 'stored'); assert.equal( Number(batch.cost_usd), - Number(batchCostUsd('claude-haiku-5-5', { input_tokens: 4000, output_tokens: 900 }).toFixed(4)) + Number(batchCostUsd('dos', { input_tokens: 4000, output_tokens: 900 }).toFixed(4)) ); assert.equal(db.stores.flat().length, 3); assert.ok( @@ -499,8 +510,8 @@ async function testJob() { // A crash after the batch was created: the next run collects it without sending again. const crashed = fakeSupabase(seed()); crashed.tables.review_ai_batches!.push({ - id: 'msgbatch_crashed01', - model: 'claude-haiku-5-5', + id: 'batch_crashed01', + model: 'dos', prompt_version: REVIEW_AI_PROMPT_VERSION, status: 'submitted', manifest: [ @@ -517,7 +528,7 @@ async function testJob() { stored_at: null, }); ai.calls.length = 0; - ai.batches.set('msgbatch_crashed01', [ + ai.batches.set('batch_crashed01', [ { custom_id: 'smartphones-0', ...answer([1, 2, 3].map(ref => ({ ref, points: GOOD_POINTS }))) }, ]); // A dry run while it is open does not count its reviews as due. @@ -532,7 +543,7 @@ async function testJob() { category_slug: 'smartphones', outcome: 'stored', attempts: 1, - batch_id: 'msgbatch_crashed01', + batch_id: 'batch_crashed01', }); const resumed = await runReviewAiPointsJob(crashed.client, { apply: true, @@ -540,7 +551,7 @@ async function testJob() { ai: ai.client, now, }); - assert.deepEqual(resumed.collected, ['msgbatch_crashed01']); + assert.deepEqual(resumed.collected, ['batch_crashed01']); assert.equal(resumed.sent, 0, 'nothing sent twice'); assert.ok(!ai.calls.includes('create')); assert.deepEqual( @@ -553,7 +564,7 @@ async function testJob() { const running = fakeSupabase(seed()); running.tables.review_ai_batches!.push({ ...crashed.tables.review_ai_batches![0]!, - id: 'msgbatch_running01', + id: 'batch_running01', status: 'submitted', cost_usd: null, }); @@ -567,7 +578,7 @@ async function testJob() { waitMs: 0, wait: async () => {}, }); - assert.deepEqual(waited.waiting, ['msgbatch_running01']); + assert.deepEqual(waited.waiting, ['batch_running01']); assert.equal(waited.candidates, 0); assert.ok(!ai.calls.includes('create')); // ...and the kill switch cancels it. @@ -577,7 +588,7 @@ async function testJob() { ai: ai.client, now, }); - assert.deepEqual(stopped.canceled, ['msgbatch_running01']); + assert.deepEqual(stopped.canceled, ['batch_running01']); assert.ok(ai.calls.includes('cancel')); assert.equal( running.tables.review_ai_batches![0]!.status, @@ -587,9 +598,7 @@ async function testJob() { // Switched on again: the canceled batch ends; its finished request is stored, the // canceled one comes back as failed rows, and nothing is sent twice. ai.status = 'ended'; - ai.batches.set('msgbatch_running01', [ - { custom_id: 'smartphones-0', result: { type: 'canceled' } }, - ]); + ai.batches.set('batch_running01', [{ custom_id: 'smartphones-0', result: { type: 'canceled' } }]); ai.calls.length = 0; const back = await runReviewAiPointsJob(running.client, { apply: true, @@ -597,7 +606,7 @@ async function testJob() { ai: ai.client, now, }); - assert.equal(back.collected[0], 'msgbatch_running01', 'the old batch first, then the new one'); + assert.equal(back.collected[0], 'batch_running01', 'the old batch first, then the new one'); assert.equal(back.stored.failed, 3); // The fake writer has no attempts; the failed reviews are due again, in a new batch. assert.equal(back.sent, 3); @@ -605,8 +614,8 @@ async function testJob() { // The monthly cap: this month's spend plus the ceiling over $5 sends nothing. const capped = fakeSupabase(seed()); capped.tables.review_ai_batches!.push({ - id: 'msgbatch_spent0001', - model: 'claude-haiku-5-5', + id: 'batch_spent0001', + model: 'dos', prompt_version: 'v0', status: 'stored', manifest: [], @@ -616,8 +625,8 @@ async function testJob() { stored_at: '2026-11-02T01:00:00Z', }); capped.tables.review_ai_batches!.push({ - id: 'msgbatch_lastmonth', - model: 'claude-haiku-5-5', + id: 'batch_lastmonth', + model: 'dos', prompt_version: 'v0', status: 'stored', manifest: [], @@ -636,12 +645,12 @@ async function testJob() { assert.equal(over.capped, true); assert.equal(over.monthSpendUsd, 4.999, 'last month does not count'); assert.equal(ai.calls.length, 0); - const sonnet = await runReviewAiPointsJob(capped.client, { + const qwenMax = await runReviewAiPointsJob(capped.client, { enabled: true, - model: 'claude-sonnet-5-5', + model: 'qwen3.8-max', now, }); - assert.equal(sonnet.capped, false, 'each model has its own cap'); + assert.equal(qwenMax.capped, false, 'each model has its own cap'); // A quality stop: most quotes made up; nothing stored, the batch left ended. const bad = fakeSupabase(seed()); @@ -679,6 +688,37 @@ async function testJob() { /--wait-minutes/ ); + // A batch the gateway failed as a whole: the row is closed, nothing is stored, and the + // reviews are due again with no attempt spent. + const goneDb = fakeSupabase(seed()); + goneDb.tables.review_ai_batches!.push({ + ...crashed.tables.review_ai_batches![0]!, + id: 'batch_failed01', + status: 'submitted', + cost_usd: null, + stored_at: null, + }); + const goneAi: ReviewAiClient = { + ...ai.client, + async createBatch() { + throw new Error('not in this test'); + }, + // eslint-disable-next-line require-yield + async *results() { + throw new BatchGoneError('dos.ai batch ended failed with no output', 'canceled'); + }, + }; + const afterGone = await runReviewAiPointsJob(goneDb.client, { + enabled: true, + ai: goneAi, + now, + apply: true, + maxReviews: 1, + }).catch((err: unknown) => err); + assert.match(String(afterGone), /not in this test/, 'its reviews were due again and sent'); + assert.equal(goneDb.tables.review_ai_batches![0]!.status, 'canceled'); + assert.equal(goneDb.stores.length, 0); + // The purge: a dry run says so; --apply calls the writer; a bad version is refused. const purge = fakeSupabase(seed()); const purgeDry = await runReviewAiPointsJob(purge.client, { purge: 'v1' }); @@ -697,8 +737,187 @@ async function testJob() { ); } +async function testDosAiClient() { + // Output lines: a normal answer, a cut-off one, a filtered one, an error, a 400. + const ok = (finish: string) => ({ + status_code: 200, + body: { + choices: [{ message: { content: '{"reviews":[]}' }, finish_reason: finish }], + usage: { + prompt_tokens: 1000, + completion_tokens: 50, + prompt_tokens_details: { cached_tokens: 600 }, + }, + }, + }); + const read = (line: object) => fromDosAiOutputLine(line); + const stopped = read({ custom_id: 'a', response: ok('stop') }); + assert.equal(stopped?.result.type, 'succeeded'); + if (stopped?.result.type === 'succeeded') { + assert.equal(stopped.result.message.stop_reason, 'end_turn'); + assert.deepEqual(stopped.result.message.usage, { + input_tokens: 400, + output_tokens: 50, + cache_read_input_tokens: 600, + cache_creation_input_tokens: 0, + }); + } + const reason = (line: object) => { + const r = read(line); + return r?.result.type === 'succeeded' ? r.result.message.stop_reason : r?.result.type; + }; + assert.equal(reason({ custom_id: 'b', response: ok('length') }), 'max_tokens'); + assert.equal(reason({ custom_id: 'c', response: ok('content_filter') }), 'refusal'); + assert.equal( + reason({ custom_id: 'd', error: { code: 'model_sunset', message: 'x' } }), + 'errored' + ); + assert.equal(reason({ custom_id: 'e', response: { status_code: 400, body: {} } }), 'errored'); + assert.equal(read({ response: ok('stop') }), null, 'a line with no custom_id is skipped'); + + // The client against a fake gateway: upload, create, status, results, cancel, auth, retry, 402. + const calls: string[] = []; + let busy = 1; + const gateway = (async (url: string | URL | Request, init?: RequestInit) => { + const path = String(url).replace('https://beta-api.dos.ai/v1', ''); + const method = init?.method ?? 'GET'; + calls.push(`${method} ${path}`); + assert.equal( + (init?.headers as Record).Authorization, + 'Bearer dos_sk_test', + 'the key as a Bearer token' + ); + const reply = (status: number, body: unknown) => + new Response(typeof body === 'string' ? body : JSON.stringify(body), { status }); + if (path === '/files' && method === 'POST') { + const form = init!.body as FormData; + assert.equal(form.get('purpose'), 'batch'); + const lines = (await (form.get('file') as Blob).text()).trim().split('\n'); + assert.equal(JSON.parse(lines[0]!).url, '/v1/chat/completions'); + return reply(200, { id: 'file_in' }); + } + if (path === '/batches' && method === 'POST') { + const body = JSON.parse(String(init!.body)) as Record; + assert.deepEqual(body, { + input_file_id: 'file_in', + endpoint: '/v1/chat/completions', + completion_window: '24h', + }); + return reply(200, { id: 'batch_abc123' }); + } + if (path === '/batches/batch_abc123') { + if (busy-- > 0) return reply(503, 'busy'); + return reply(200, { status: 'completed', output_file_id: 'file_out' }); + } + if (path === '/files/file_out/content') { + return reply( + 200, + [ + JSON.stringify({ custom_id: 'smartphones-0', response: ok('stop') }), + 'not json', + JSON.stringify({ custom_id: 'smartphones-1', error: { message: 'x' } }), + ].join('\n') + ); + } + if (path === '/batches/batch_abc123/cancel') return reply(200, { status: 'cancelling' }); + if (path === '/batches/batch_broke') return reply(402, { error: 'insufficient credits' }); + if (path === '/batches/batch_failed') { + return reply(200, { status: 'failed', output_file_id: null, errmsg: 'zdr org' }); + } + if (path === '/batches/batch_purged') { + return reply(200, { status: 'completed', output_file_id: 'file_gone' }); + } + return reply(404, 'no route'); + }) as typeof fetch; + const noWait = async () => {}; + const client = dosAiReviewAiClient('dos_sk_test', 'https://beta-api.dos.ai/v1/', gateway, noWait); + const built = buildRequests( + [{ id: id(1), text: LONG, textHash: reviewTextHash(LONG), category: 'smartphones' }], + 'dos' + ); + assert.deepEqual(await client.createBatch(built.requests), { id: 'batch_abc123' }); + assert.equal(await client.batchStatus('batch_abc123'), 'ended', 'a 503 is retried'); + const out: BatchResult[] = []; + for await (const r of client.results('batch_abc123')) out.push(r); + assert.deepEqual( + out.map(r => `${r.custom_id}:${r.result.type}`), + ['smartphones-0:succeeded', 'smartphones-1:errored'] + ); + await client.cancelBatch('batch_abc123'); + await assert.rejects(client.batchStatus('batch_broke'), /no credits left/); + await assert.rejects(client.batchStatus('batch_missing'), /404/); + assert.deepEqual(calls.slice(0, 3), [ + 'POST /files', + 'POST /batches', + 'GET /batches/batch_abc123', + ]); + // A batch failed as a whole, or one whose output file is gone: no results, ever. + const drain = async (batchId: string) => { + for await (const r of client.results(batchId)) void r; + }; + await assert.rejects(drain('batch_failed'), (err: unknown) => { + assert.ok(err instanceof BatchGoneError); + assert.equal(err.status, 'canceled'); + assert.match(err.message, /ended failed with no output: zdr org/); + return true; + }); + await assert.rejects(drain('batch_purged'), (err: unknown) => { + assert.ok(err instanceof BatchGoneError); + assert.equal(err.status, 'expired'); + return true; + }); + // A line the gateway did not run for want of credits is unbilled, not failed. + assert.equal(reason({ custom_id: 'f', response: { status_code: 402, body: {} } }), 'unbilled'); + assert.equal( + reason({ custom_id: 'g', error: { code: 'insufficient_credits', message: 'x' } }), + 'unbilled' + ); + + // POST /batches is never retried on an answer that may hide a created batch. + for (const lost of ['network', '502'] as const) { + let creates = 0; + const flaky = (async (url: string | URL | Request, init?: RequestInit) => { + const path = String(url).replace('https://api.dos.me/v1', ''); + if (path === '/files') return new Response(JSON.stringify({ id: 'file_in' })); + creates += 1; + if (init?.method === 'POST' && lost === 'network') throw new Error('socket hang up'); + return new Response('bad gateway', { status: 502 }); + }) as typeof fetch; + const once = dosAiReviewAiClient('dos_sk_test', undefined, flaky, noWait); + await assert.rejects( + once.createBatch(built.requests), + lost === 'network' ? /A batch may have been created/ : /502/ + ); + assert.equal(creates, 1, `POST /batches sent once (${lost})`); + } + // ...but a 429, refused before anything exists, is retried. + let throttled = 0; + const busyGateway = (async (url: string | URL | Request) => { + const path = String(url).replace('https://api.dos.me/v1', ''); + if (path === '/files') return new Response(JSON.stringify({ id: 'file_in' })); + throttled += 1; + return throttled === 1 + ? new Response('slow down', { status: 429 }) + : new Response(JSON.stringify({ id: 'batch_after_429' })); + }) as typeof fetch; + assert.deepEqual( + await dosAiReviewAiClient('dos_sk_test', undefined, busyGateway, noWait).createBatch( + built.requests + ), + { id: 'batch_after_429' } + ); + // The request body: deterministic and bounded. + assert.deepEqual( + [built.requests[0]!.body.temperature, built.requests[0]!.body.max_tokens], + [0, 6000] + ); + console.log( + ' ✅ dos.ai batch client: upload, create once, status, results, gone batches, cancel, retry and 402' + ); +} + function refsOf(request: BatchRequest): number[] { - const content = request.params.messages[0]!.content; + const content = request.body.messages[1].content; return (JSON.parse(content.slice(content.indexOf('\n') + 1)) as { ref: number }[]).map( r => r.ref ); @@ -717,7 +936,7 @@ function fakeAi(answerFor: (requests: BatchRequest[]) => BatchResult[]) { async createBatch(requests) { state.calls.push('create'); state.onCreate?.(); - const batchId = `msgbatch_fake${String((n += 1)).padStart(4, '0')}`; + const batchId = `batch_fake${String((n += 1)).padStart(4, '0')}`; state.batches.set(batchId, answerFor(requests)); return { id: batchId }; }, @@ -855,6 +1074,7 @@ async function run() { testCandidates(); testCollect(); await testJob(); + await testDosAiClient(); } run().catch(err => { diff --git a/apps/importer/src/shopee/jobs/review-ai-points.ts b/apps/importer/src/shopee/jobs/review-ai-points.ts index a04677923..a7eaf57dc 100644 --- a/apps/importer/src/shopee/jobs/review-ai-points.ts +++ b/apps/importer/src/shopee/jobs/review-ai-points.ts @@ -6,15 +6,17 @@ * [--apply] [--purge ] [--max ] [--wait-minutes ] * * Reads crawled user reviews, picks those that qualify (design section 1, D3), - * sends them through the Message Batches API with a structured output schema - * per category (section 2), checks every line it gets back (section 7.2) and + * sends them through the dos.ai gateway's Batch API (OpenAI format, D2 as JOY + * decided on 2026-11-04) with a structured output schema per category + * (section 2), checks every line it gets back (section 7.2) and * stores what passes as rate.review_points method 'ai' through * rate.store_review_ai_points, which writes the lines and the process-once * marker (rate.review_ai_extractions) in one transaction. * * Dry run by default: it reads, plans and logs counts and the cost ceiling, - * and never calls a model (no API key needed). `--apply` sends one batch, - * waits for it (at most --wait-minutes, default 60) and stores the results. + * and never calls a model (no key needed). `--apply` sends one batch with the + * key in DOS_AI_API_KEY (DOS_AI_BASE_URL overrides the gateway URL), waits for + * it (at most --wait-minutes, default 60) and stores the results. * It signs in as the rate_worker user (RATE_WORKER_EMAIL and * RATE_WORKER_PASSWORD with the publishable key), which reads the review * tables, writes the batch record and writes lines only through the writer. @@ -22,9 +24,10 @@ * Switches and limits (section 8, D5): * - REVIEW_AI_POINTS_ENABLED must be 'true', or the job exits before any * model call; with --apply it also cancels a batch still running. - * - REVIEW_AI_MODEL: claude-haiku-5-5 (default) or claude-sonnet-5-5 (D1). + * - REVIEW_AI_MODEL: dos (default, Qwen3.8 27B on DOS's GPUs) or + * qwen3.8-max, the models the gateway's Batch API serves (D1's shape). * - At most 2000 reviews a run (--max lowers it). - * - A monthly cap per model ($5 Haiku, $15 Sonnet). The month's spend is the + * - A monthly cap per model ($5 dos, $15 qwen3.8-max). The month's spend is the * reported cost of its batches, or their reserved ceiling while running, * read from rate.review_ai_batches. A batch whose ceiling would pass the * cap is not sent, and the CLI exits with code 3. @@ -97,22 +100,26 @@ export const SHARE_JUMP_STOP = 0.3; /** ...when both batches have at least this many reviews. */ export const SHARE_JUMP_MIN_REVIEWS = 50; const DEADLOCK_ATTEMPTS = 4; -/** The Batch API keeps results 29 days; past 28 a batch is treated as gone. */ +/** A batch whose results cannot be read after 28 days is treated as gone. */ const RESULTS_KEPT_MS = 28 * 24 * 60 * 60 * 1000; const PROMPT_VERSION_RE = /^[a-z0-9][a-z0-9._-]{0,31}$/; // --- The model client: the Batch API as the job uses it, so tests pass a fake. ------------ +/** One line of a dos.ai (OpenAI-format) batch input file. */ export interface BatchRequest { custom_id: string; - params: { + method: 'POST'; + url: '/v1/chat/completions'; + body: { model: ReviewAiModel; max_tokens: number; - system: { type: 'text'; text: string; cache_control: { type: 'ephemeral' } }[]; - messages: { role: 'user'; content: string }[]; - output_config: { - effort: 'low'; - format: { type: 'json_schema'; schema: Record }; + temperature: number; + stream: false; + messages: [{ role: 'system'; content: string }, { role: 'user'; content: string }]; + response_format: { + type: 'json_schema'; + json_schema: { name: string; strict: true; schema: Record }; }; }; } @@ -136,7 +143,7 @@ export type BatchResult = }; }; } - | { custom_id: string; result: { type: 'errored' | 'expired' | 'canceled' } }; + | { custom_id: string; result: { type: 'errored' | 'expired' | 'canceled' | 'unbilled' } }; export interface ReviewAiClient { createBatch(requests: BatchRequest[]): Promise<{ id: string }>; @@ -145,29 +152,226 @@ export interface ReviewAiClient { cancelBatch(id: string): Promise; } -/** The real client: the official Anthropic SDK, the key from ANTHROPIC_API_KEY. */ -export async function anthropicReviewAiClient(apiKey: string): Promise { - const { default: Anthropic } = await import('@anthropic-ai/sdk'); - const client = new Anthropic({ apiKey }); +/** The dos.ai gateway's default base URL, as the importer's other AI clients use it. */ +export const DOS_AI_DEFAULT_BASE_URL = 'https://api.dos.me/v1'; +const DOS_AI_TIMEOUT_MS = 60_000; +const DOS_AI_ATTEMPTS = 3; + +/** One line of a dos.ai batch output file (OpenAI batch format). */ +interface DosAiOutputLine { + custom_id?: string; + response?: { + status_code?: number; + body?: { + choices?: { message?: { content?: string | null }; finish_reason?: string | null }[]; + usage?: { + prompt_tokens?: number; + completion_tokens?: number; + prompt_tokens_details?: { cached_tokens?: number; cache_write_tokens?: number } | null; + }; + }; + } | null; + error?: { code?: string; message?: string } | null; +} + +/** + * A dos.ai output line as the job reads results: a 200 response is a + * message whose stop reason is the finish reason ('stop' ends normally, + * 'length' is cut off, 'content_filter' is refused); a line the gateway did + * not run for want of credits (402, insufficient_credits) is 'unbilled', so + * the review stays due without spending an attempt; anything else errored. + * Cached prompt tokens are split out of prompt_tokens. + */ +export function fromDosAiOutputLine(line: DosAiOutputLine): BatchResult | null { + if (typeof line.custom_id !== 'string') return null; + const body = line.response?.body; + if ( + line.response?.status_code === 402 || + /insufficient|credit|balance/i.test(line.error?.code ?? '') + ) { + return { custom_id: line.custom_id, result: { type: 'unbilled' } }; + } + if (line.error || line.response?.status_code !== 200 || !body) { + return { custom_id: line.custom_id, result: { type: 'errored' } }; + } + const choice = body.choices?.[0]; + const finish = choice?.finish_reason ?? null; + const n = (v: unknown) => (typeof v === 'number' && Number.isFinite(v) && v > 0 ? v : 0); + const cachedRead = n(body.usage?.prompt_tokens_details?.cached_tokens); + const cachedWrite = n(body.usage?.prompt_tokens_details?.cache_write_tokens); + return { + custom_id: line.custom_id, + result: { + type: 'succeeded', + message: { + stop_reason: + finish === 'stop' + ? 'end_turn' + : finish === 'length' + ? 'max_tokens' + : finish === 'content_filter' + ? 'refusal' + : finish, + content: [{ type: 'text', text: choice?.message?.content ?? '' }], + usage: { + input_tokens: Math.max(0, n(body.usage?.prompt_tokens) - cachedRead - cachedWrite), + output_tokens: n(body.usage?.completion_tokens), + cache_read_input_tokens: cachedRead, + cache_creation_input_tokens: cachedWrite, + }, + }, + }, + }; +} + +/** + * A batch whose results will never come: the gateway failed it as a whole + * (a refused file, a Zero Data Retention organization, no credits at start), + * it ended with no output, or its output file is gone. The job closes its row + * and stores nothing, so its reviews are due again without spending an attempt. + */ +export class BatchGoneError extends Error { + constructor( + message: string, + readonly status: 'canceled' | 'expired' + ) { + super(message); + this.name = 'BatchGoneError'; + } +} + +/** + * The real client: the dos.ai gateway's OpenAI-compatible Files and Batch API + * (POST /files, POST /batches, GET /batches/:id, POST /batches/:id/cancel, + * GET /files/:id/content), with the key in DOS_AI_API_KEY as a Bearer token. + * Each call, body included, has a timeout. A 429 or 5xx, or a network error, + * is retried, except on POST /batches: a batch may have been created behind a + * lost answer, and a retry would run and bill a second one, so that call is + * retried only on 429 (refused before anything was created). A 402 (no + * credits left) stops the run. + */ +export function dosAiReviewAiClient( + apiKey: string, + baseUrl: string = DOS_AI_DEFAULT_BASE_URL, + fetchImpl: typeof fetch = fetch, + wait: (ms: number) => Promise = ms => new Promise(r => setTimeout(r, ms)) +): ReviewAiClient { + const base = baseUrl.replace(/\/+$/, ''); + const call = async ( + path: string, + init: RequestInit = {}, + retry: 'all' | 'throttled' = 'all' + ): Promise<{ status: number; text: string }> => { + for (let attempt = 1; ; attempt++) { + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), DOS_AI_TIMEOUT_MS); + let status = 0; + let text = ''; + let failure: unknown = null; + try { + const response = await fetchImpl(`${base}${path}`, { + ...init, + headers: { Authorization: `Bearer ${apiKey}`, ...(init.headers ?? {}) }, + signal: controller.signal, + }); + status = response.status; + text = await response.text(); + } catch (err) { + failure = err; + } finally { + clearTimeout(timer); + } + if (failure === null && status >= 200 && status < 300) return { status, text }; + if (status === 402) { + throw new Error('dos.ai refused the request: no credits left on the key (402).'); + } + const retryable = + retry === 'all' ? failure !== null || status === 429 || status >= 500 : status === 429; + if (!retryable || attempt >= DOS_AI_ATTEMPTS) { + if (failure !== null && retry === 'throttled') { + throw new Error( + `dos.ai ${init.method ?? 'GET'} ${path}: no answer (${String(failure)}). A batch may have been created; check the key's batches on dos.ai before the next run.` + ); + } + if (failure !== null) throw failure; + // The body names the reason; it holds no review text of ours. + throw new Error(`dos.ai ${init.method ?? 'GET'} ${path}: ${status} ${text.slice(0, 300)}`); + } + await wait(1000 * 2 ** (attempt - 1)); + } + }; + const json = async (path: string, init?: RequestInit, retry?: 'all' | 'throttled') => + JSON.parse((await call(path, init, retry)).text) as T; return { async createBatch(requests) { - const batch = await client.messages.batches.create({ - requests: requests as unknown as Parameters< - typeof client.messages.batches.create - >[0]['requests'], - }); + const form = new FormData(); + form.append('purpose', 'batch'); + form.append( + 'file', + new Blob([requests.map(r => JSON.stringify(r)).join('\n') + '\n'], { + type: 'application/jsonl', + }), + 'review-ai-points.jsonl' + ); + const file = await json<{ id: string }>('/files', { method: 'POST', body: form }); + const batch = await json<{ id: string }>( + '/batches', + { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + input_file_id: file.id, + endpoint: '/v1/chat/completions', + completion_window: '24h', + }), + }, + 'throttled' + ); return { id: batch.id }; }, async batchStatus(id) { - return (await client.messages.batches.retrieve(id)).processing_status; + const batch = await json<{ status: string }>(`/batches/${encodeURIComponent(id)}`); + if (batch.status === 'cancelling') return 'canceling'; + return ['completed', 'failed', 'cancelled', 'expired'].includes(batch.status) + ? 'ended' + : 'in_progress'; }, async *results(id) { - for await (const entry of await client.messages.batches.results(id)) { - yield entry as unknown as BatchResult; + const batch = await json<{ + status?: string; + output_file_id?: string | null; + errmsg?: string | null; + }>(`/batches/${encodeURIComponent(id)}`); + if (!batch.output_file_id) { + // Failed as a whole, or ended with no line run: nothing to store. + throw new BatchGoneError( + `dos.ai batch ${id} ended ${batch.status ?? 'without a status'} with no output${batch.errmsg ? `: ${batch.errmsg.slice(0, 200)}` : ''}`, + batch.status === 'expired' ? 'expired' : 'canceled' + ); + } + let text: string; + try { + text = (await call(`/files/${encodeURIComponent(batch.output_file_id)}/content`)).text; + } catch (err) { + if (/ 404 /.test(String(err))) { + throw new BatchGoneError(`dos.ai batch ${id}: its output file is gone (404)`, 'expired'); + } + throw err; + } + for (const raw of text.split('\n')) { + if (!raw.trim()) continue; + let line: DosAiOutputLine; + try { + line = JSON.parse(raw) as DosAiOutputLine; + } catch { + continue; + } + const result = fromDosAiOutputLine(line); + if (result) yield result; } }, async cancelBatch(id) { - await client.messages.batches.cancel(id); + await call(`/batches/${encodeURIComponent(id)}/cancel`, { method: 'POST' }); }, }; } @@ -384,17 +588,26 @@ export function buildRequests( const refs = part.map((c, j) => ({ ref: j + 1, candidate: c })); requests.push({ custom_id, - params: { + method: 'POST', + url: '/v1/chat/completions', + body: { model, max_tokens: REQUEST_MAX_TOKENS, - system: [{ type: 'text', text: system, cache_control: { type: 'ephemeral' } }], + temperature: 0, + stream: false, + // The instructions first and the same for a category: a shared prefix the + // gateway's backends can cache on their own. messages: [ + { role: 'system', content: system }, { role: 'user', content: reviewAiUserMessage(refs.map(r => ({ ref: r.ref, text: r.candidate.text }))), }, ], - output_config: { effort: 'low', format: { type: 'json_schema', schema } }, + response_format: { + type: 'json_schema', + json_schema: { name: 'review_points', strict: true, schema }, + }, }, }); instructions.push(system); @@ -423,6 +636,8 @@ export interface CollectedBatch { rejects: Record; unaccented: number; failedReviews: number; + /** Reviews in requests the gateway did not run for want of credits: still due. */ + unbilledReviews: number; reviewsWithLines: number; reviewsAnswered: number; } @@ -475,6 +690,7 @@ export function collectBatch( }, unaccented: 0, failedReviews: 0, + unbilledReviews: 0, reviewsWithLines: 0, reviewsAnswered: 0, }; @@ -499,6 +715,11 @@ export function collectBatch( } } } + // A request the gateway never ran for want of credits: its reviews stay due, no attempt spent. + if (result?.result.type === 'unbilled') { + out.unbilledReviews += request.reviews.length; + continue; + } for (const review of request.reviews) { const base = { id: review.id, @@ -673,7 +894,7 @@ export async function runReviewAiPointsJob( purged: null, }; const needAi = (): ReviewAiClient => { - if (!options.ai) throw new Error('ANTHROPIC_API_KEY is needed to send or collect a batch.'); + if (!options.ai) throw new Error('DOS_AI_API_KEY is needed to send or collect a batch.'); return options.ai; }; @@ -738,16 +959,26 @@ export async function runReviewAiPointsJob( try { for await (const entry of ai.results(batch.id)) results.push(entry); } catch (err) { - // The API keeps results 29 days: an older batch is gone, so mark it and move on. - if (now.getTime() - new Date(batch.submitted_at).getTime() < RESULTS_KEPT_MS) throw err; + // Results that will never come (the client says so, or the batch is past 28 days): + // close the row and store nothing, so the reviews are due again with no attempt spent. + const gone = err instanceof BatchGoneError; + if (!gone && now.getTime() - new Date(batch.submitted_at).getTime() < RESULTS_KEPT_MS) { + throw err; + } + const status = gone ? err.status : 'expired'; const { error } = await supabase .from('review_ai_batches') - .update({ status: 'expired', ended_at: now.toISOString() }) + .update({ status, ended_at: now.toISOString() }) .eq('id', batch.id); - if (error) throw new Error(`review_ai_batches expire: ${error.message}`); - logger.warn('review_ai.expired', `Batch ${batch.id} results are gone; marked expired.`, { - batch: batch.id, - }); + if (error) throw new Error(`review_ai_batches close: ${error.message}`); + logger.warn( + 'review_ai.gone', + `Batch ${batch.id} has no results to store; marked ${status}.`, + { + batch: batch.id, + reason: err instanceof Error ? err.message : String(err), + } + ); return false; } const ids = batch.manifest.flatMap(r => r.reviews.map(v => v.id)); @@ -808,6 +1039,7 @@ export async function runReviewAiPointsJob( rejects: collected.rejects, unaccented: collected.unaccented, failed_reviews: collected.failedReviews, + unbilled_reviews: collected.unbilledReviews, cost_usd: Number(collected.costUsd.toFixed(4)), ...counts, }); @@ -879,7 +1111,7 @@ export async function runReviewAiPointsJob( model, requests.map((r, i) => ({ instructions: instructions[i]!, - message: r.params.messages[0]!.content, + message: r.body.messages[1].content, })) ); const refreshed = await fetchByKey( @@ -1055,9 +1287,15 @@ if (isDirectExecution) { throw new Error(`The signed-in user has role '${role}', not rate_worker; refusing to run.`); } const supabase = scopeSupabaseToRate(client); - // The model key is read only for a run that may call the API. - const apiKey = (process.env.ANTHROPIC_API_KEY || '').trim(); - const ai = apply && apiKey ? await anthropicReviewAiClient(apiKey) : null; + // The gateway key is read only for a run that may call it. DOS_AI_BASE_URL alone + // overrides the gateway (beta-api.dos.ai, say); AI_BASE_URL is the other + // importer jobs' and may name another provider. + const apiKey = (process.env.DOS_AI_API_KEY || '').trim(); + const baseUrl = (process.env.DOS_AI_BASE_URL || DOS_AI_DEFAULT_BASE_URL).trim(); + if (!/^https:\/\/[^/\s]+(\/[^\s]*)?$/.test(baseUrl)) { + throw new Error('DOS_AI_BASE_URL must be an https URL.'); + } + const ai = apply && apiKey ? dosAiReviewAiClient(apiKey, baseUrl) : null; logger.info( 'review_ai.target', `Target: ${host} (${apply ? 'apply' : 'dry run'}, ${enabled ? 'enabled' : 'disabled'}, ${modelName})`, diff --git a/apps/importer/src/shopee/review-ai-prompt.ts b/apps/importer/src/shopee/review-ai-prompt.ts index 48ab48551..36991a646 100644 --- a/apps/importer/src/shopee/review-ai-prompt.ts +++ b/apps/importer/src/shopee/review-ai-prompt.ts @@ -29,11 +29,11 @@ import { export const REVIEW_AI_PROMPT_VERSION = 'v1'; export { RATING_CRITERIA_VERSION }; -/** Reviews per request: one category, one cached instruction block. */ +/** Reviews per request: one category, one instruction block (a shared prefix). */ export const REVIEWS_PER_REQUEST = 20; /** Lines per side per review (design 2.2). */ export const AI_POINTS_PER_KIND = 3; -/** Output tokens per request: 20 reviews x ~120 tokens, with room for low-effort thinking. */ +/** Output tokens per request: 20 reviews x ~120 tokens, with room for any reasoning. */ export const REQUEST_MAX_TOKENS = 6000; /** Evidence length, in characters as stored (the writer takes 2 to 300). */ export const EVIDENCE_MAX = 300; @@ -42,18 +42,31 @@ export const EVIDENCE_MAX = 300; export const REVIEW_AI_MIN_CHARS = 40; export const REVIEW_AI_MIN_WORDS = 8; -/** Models the job may use (D1) and their list prices in USD per million tokens. */ +/** + * Models the job may use through the dos.ai gateway's Batch API (D2 as JOY + * decided on 2026-11-04: the job calls dos.ai, not a model vendor), and their + * dos.ai list prices in USD per million tokens (dosai.model_pricing, cents + * per million: dos 25/150, qwen3.8-max 200/600). The gateway's Batch API + * serves only its own and Alibaba models (dos, qwen3.8-*, deepseek-*); dos is + * Qwen3.8 27B on DOS's own GPUs. D1's shape stays: the cheap model by + * default, the stronger one only if the default misses the gold set (7.1). + */ export const REVIEW_AI_MODELS = { - 'claude-haiku-5-5': { input: 0.1, output: 0.5, monthlyCapUsd: 5 }, - 'claude-sonnet-5-5': { input: 2, output: 10, monthlyCapUsd: 15 }, + dos: { input: 0.25, output: 1.5, monthlyCapUsd: 5 }, + 'qwen3.8-max': { input: 2, output: 6, monthlyCapUsd: 15 }, } as const; export type ReviewAiModel = keyof typeof REVIEW_AI_MODELS; -export const DEFAULT_REVIEW_AI_MODEL: ReviewAiModel = 'claude-haiku-5-5'; -/** The Batch API halves every token price. */ +export const DEFAULT_REVIEW_AI_MODEL: ReviewAiModel = 'dos'; +/** The gateway bills Batch API lines at half the synchronous price. */ export const BATCH_DISCOUNT = 0.5; -/** Cache reads cost 0.1x the input price; cache writes 1.25x. */ -export const CACHE_READ_FACTOR = 0.1; -export const CACHE_WRITE_FACTOR = 1.25; +/** + * Cached prompt tokens, read or written, are counted at the full input price: + * the gateway reports them (prompt_tokens_details) but its cached rates per + * model are not published, so the job's figure is an upper bound. The + * gateway's own ledger is the bill. + */ +export const CACHE_READ_FACTOR = 1; +export const CACHE_WRITE_FACTOR = 1; export function isReviewAiModel(value: unknown): value is ReviewAiModel { return typeof value === 'string' && Object.prototype.hasOwnProperty.call(REVIEW_AI_MODELS, value); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 98f965971..f91e3db04 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -130,9 +130,6 @@ importers: apps/importer: dependencies: - '@anthropic-ai/sdk': - specifier: 1.3.0 - version: 1.3.0(zod@4.6.7) '@repo/shared-data': specifier: workspace:* version: link:../../packages/shared-data @@ -350,15 +347,6 @@ packages: resolution: {integrity: sha512-U4+70Pc5ZS9osnCBCE5Jha/ciHM+Yp+CNMNC/7HvYbNRk1Ldd+f7qO65W5qfhu/TCv+/ozljlXXe9Nj8419DMA==} engines: {node: '>=10'} - '@anthropic-ai/sdk@1.3.0': - resolution: {integrity: sha512-k7z9BlLLPxfZH+ov7dS6EnaN/wiZeV8YNT1LZfrpAHIy7oY7pHr4UzirF9f2OlHtFEnRBOX+AZQ91Y5fK+TPTw==} - hasBin: true - peerDependencies: - zod: ^3.25.0 || ^4.0.0 - peerDependenciesMeta: - zod: - optional: true - '@axe-core/playwright@4.13.0': resolution: {integrity: sha512-6YLx+kxXu5GJceG4ozFg+33a2EMTdjYwWGloJ3sb9Kta5pp+ZNS53uxGVog5JetIY8s++P5UrtX+cri+u0VAVg==} peerDependencies: @@ -1651,9 +1639,6 @@ packages: resolution: {integrity: sha512-fCTuZK4QBa+39Oz9l4OGfJfz+GpwCp3AqO7Zch3to99xHPgstVsRFpeQ8LNd2o1Gv8raL2mCFwiaHh7bFSp5DQ==} engines: {node: '>=22'} - '@stablelib/base64@1.0.1': - resolution: {integrity: sha512-1bnPQqSxSuc3Ii6MhBysoWCg58j97aUjuCSZrGSmDxNqtytIi0k8utUenAwTZN4V5mXXYGsVUI9zeBqy+jBOSQ==} - '@supabase/auth-js@2.117.2': resolution: {integrity: sha512-Z3WnGwrphYZubrLbxp5Iv0tLA9A5GhvKzJ/ZXglxqwd2QEH2R4dXRltjVJ8sIn4xEW2BSFIGT0yALVJIgZeDYw==} engines: {node: '>=22.0.0'} @@ -2725,9 +2710,6 @@ packages: fast-levenshtein@2.0.6: resolution: {integrity: sha512-DCXu6Ifhqcks7TZKY3Hxp3y6qphY5SJZmrWMDrKcERSOXWQdMhU9Ig/PYrzyw/ul9jOIyh0N4M0tbC5hodg8dw==} - fast-sha256@1.3.0: - resolution: {integrity: sha512-n11RGP/lrWEFI/bWdygLxhI+pVeo1ZYIVwvvPkW7azl/rOy+F3HYRZ2K5zeE9mmkhQppyv9sQFx0JM9UabnpPQ==} - fast-uri@3.1.9: resolution: {integrity: sha512-hS/0Wt3Ya043oKXs0sIxrMpKQITXhMLIrVAKKXi4gozGC9v7YRWVnXIZzjNPTO02M9BdxLCWqK1yRXgNziVyJQ==} @@ -3157,10 +3139,6 @@ packages: json-parse-even-better-errors@2.3.1: resolution: {integrity: sha512-xyFwyhro/JEof6Ghe2iz2NcXoj2sloNsWr/XsERDK/oiPCfaNhl5ONfp+jQdAZRQQ0IJWNzH9zIZF7li91kh2w==} - json-schema-to-ts@3.1.1: - resolution: {integrity: sha512-+DWg8jCJG2TEnpy7kOm/7/AxaYoaRbjVB4LFZLySZlWn8exGs3A4OLJR966cVvU26N7X9TWxl+Jsw7dzAqKT6g==} - engines: {node: '>=16'} - json-schema-traverse@0.4.1: resolution: {integrity: sha512-xbbCH5dCYU5T8LcEhhuh7HJ88HXuW3qsI3Y0zOZFKfZEHcpWiHU/Jxzk629Brsab/mMiHQti9wMP+845RPe3Vg==} @@ -3860,9 +3838,6 @@ packages: resolution: {integrity: sha512-WjlahMgHmCJpqzU8bIBy4qtsZdU9lRlcZE3Lvyej6t4tuOuv1vk57OW3MBrj6hXBFx/nNoC9MPMTcr5YA7NQbg==} engines: {node: '>=6'} - standardwebhooks@1.1.1: - resolution: {integrity: sha512-bCbX9ZEyFkWPsRz7Bl3NuQUJohmwGSev/yhr7vhaGPlc4AfIrspIRa6cPTBuI1ItmrTDJ4d/S2hCsfe4+vQGnQ==} - stop-iteration-iterator@1.1.0: resolution: {integrity: sha512-eLoXW/DHyl62zxY4SCaIgnRhuMr6ri4juEYARS8E6sCEqzKpOiE521Ucofdx+KnDZl5xmvGYaaKCk5FEOxJCoQ==} engines: {node: '>= 0.4'} @@ -3981,9 +3956,6 @@ packages: resolution: {integrity: sha512-65P7iz6X5yEr1cwcgvQxbbIw7Uk3gOy5dIdtZ4rDveLqhrdJP+Li/Hx6tyK0NEb+2GCyneCMJiGqrADCSNk8sQ==} engines: {node: '>=8.0'} - ts-algebra@2.0.0: - resolution: {integrity: sha512-FPAhNPFMrkwz76P7cdjdmiShwMynZYN6SgOujD1urY4oNm80Ou9oMdmbR45LotcKOXoy7wSmHkRFE6Mxbrhefw==} - ts-api-utils@2.5.0: resolution: {integrity: sha512-OJ/ibxhPlqrMM0UiNHJ/0CKQkoKF243/AEmplt3qpRgkW8VG7IfOS41h7V8TjITqdByHzrjcS/2si+y4lIh8NA==} engines: {node: '>=18.12'} @@ -4186,14 +4158,6 @@ snapshots: '@alloc/quick-lru@5.3.0': {} - '@anthropic-ai/sdk@1.3.0(zod@4.6.7)': - dependencies: - '@opentelemetry/api': 1.9.1 - json-schema-to-ts: 3.1.1 - standardwebhooks: 1.1.1 - optionalDependencies: - zod: 4.6.7 - '@axe-core/playwright@4.13.0(playwright-core@1.64.1)': dependencies: axe-core: 4.13.0 @@ -5261,8 +5225,6 @@ snapshots: '@simple-libs/stream-utils@2.0.0': {} - '@stablelib/base64@1.0.1': {} - '@supabase/auth-js@2.117.2': dependencies: tslib: 2.8.1 @@ -6461,8 +6423,6 @@ snapshots: fast-levenshtein@2.0.6: {} - fast-sha256@1.3.0: {} - fast-uri@3.1.9: {} fastq@1.20.3: @@ -6900,11 +6860,6 @@ snapshots: json-parse-even-better-errors@2.3.1: {} - json-schema-to-ts@3.1.1: - dependencies: - '@babel/runtime': 7.29.7 - ts-algebra: 2.0.0 - json-schema-traverse@0.4.1: {} json-schema-traverse@1.0.0: {} @@ -7635,11 +7590,6 @@ snapshots: dependencies: type-fest: 0.7.1 - standardwebhooks@1.1.1: - dependencies: - '@stablelib/base64': 1.0.1 - fast-sha256: 1.3.0 - stop-iteration-iterator@1.1.0: dependencies: es-errors: 1.3.0 @@ -7777,8 +7727,6 @@ snapshots: dependencies: is-number: 7.0.0 - ts-algebra@2.0.0: {} - ts-api-utils@2.5.0: dependencies: typescript: 6.0.3 From d62763494e79b866c127fe95899878d2444a558e Mon Sep 17 00:00:00 2001 From: JOY <5027251+JOY@users.noreply.github.com> Date: Wed, 4 Nov 2026 14:16:52 +0700 Subject: [PATCH 3/7] feat(extension,web): extension observation v1, stock and sku, flagged ingest (#1609) * feat(extension,web): extension observation v1, stock and sku, flagged ingest - shared-data: ExtensionObservation v1 in rules, plain functions with no zod. The parser names the first broken field the way the DOS-Me RPC does in DETAIL. No ratings, all keys required, listing host must match the marketplace, and observed_at must be within the last 24 hours (up to 10 minutes ahead). - Extension: the extractor reads stock state and quantity (JSON-LD availability and inventoryLevel, product meta) and the SKU (Lazada and Tiki URLs, JSON-LD sku). buildExtensionObservationV1 maps a signal to v1. Nothing new is sent: the lookup and the opt-in payload are unchanged, and a test pins that. - Web: POST and DELETE /api/v1/rate/extension/submitter and POST /api/v1/rate/extension/observation call the narrow M2 RPCs as anon. They answer 404 NOT_ENABLED unless RATE_EXTENSION_OBSERVATIONS=1, and observations also need RATE_NETWORK_HASH_SALT. Only an HMAC of the /24 or /48 reaches the database, never the raw IP. Refs #1606, #1175 * fix(extension,web): m2 review fixes, ingest key and variant-true stock Web: - The RPCs now take RATE_EXTENSION_INGEST_KEY (checked by sha256 in the database), so nobody can call them around these routes with the anon key. Writes need the key and the salt, else 503 before the database. - Register sends the network hash too. An address with no range hashes to one shared "unknown" bucket instead of skipping the limit. IPv4-mapped IPv6 maps to its /24. - Web page origins are refused (403); CORS goes only to extension origins; bodies must be application/json (415). OPTIONS is 404 while off. RPC refusals log their code; 54000 answers Retry-After 3600; 42501 is a 503. Shared rules: listing_url and observed_at follow the RPC exactly (raw https prefix, lowercase host, no port, user, ';' or space; ISO 8601 with a zone, offsets as the RPC takes them); lengths count code points; unpaired surrogates are refused. Extension: - Stock and SKU come from the selected variant's own offer: the only offer, or the one whose sku or name matches the selected variant; never another variant's or the parent's when several exist. The meta fallback reads head meta only. - Availability: schema: prefixes, arrays, Reserved and pre-order forms. A quantity is kept only when it agrees with the state. - extractorSource follows price, list price and stock. A long variant label is cut to 120 for v1. skuId uses the external id sanitizer. - buildLookupParams carries the lookup query; tests pin the exact lookup query and opt-in payload, and cover a page end to end. Refs #1606 * fix(web): observed_at bounds match postgres, brake retry-after - observed_at refuses seconds of 60 and offsets beyond +-15:59, which Postgres refuses, so the validator and the RPC agree. - A 54000 from the global register brake answers Retry-After 60; the hourly limits keep 3600. Refs #1606 * fix(shared-data,web): host check without a built regex, test key at run time - isExtensionMarketplaceHost checks subdomains by suffix and a fixed label pattern instead of a RegExp built from the host table (CodeQL incomplete escaping). - The ingest key test value is built at run time; the already-pushed fixture line in d19083ac gets a .gitleaksignore fingerprint. Refs #1606 Co-Authored-By: Claude Opus 5.5 --- .gitleaksignore | 3 + .../src/background/service-worker.ts | 19 +- .../src/extractors/ecommerce-detector.ts | 205 +++++++++++ apps/extension/src/security-guards.ts | 100 ++++++ apps/extension/src/test-extension-pipeline.ts | 291 +++++++++++++++- apps/extension/src/types.ts | 12 + .../api/v1/rate/__tests__/write-paths.test.ts | 323 ++++++++++++++++++ .../v1/rate/extension/observation/route.ts | 111 ++++++ .../api/v1/rate/extension/submitter/route.ts | 126 +++++++ .../src/server/rate/extension-observations.ts | 260 ++++++++++++++ .../src/__tests__/shared-data.test.ts | 145 ++++++++ .../src/rules/extension-observation.ts | 317 +++++++++++++++++ packages/shared-data/src/rules/index.ts | 1 + 13 files changed, 1893 insertions(+), 20 deletions(-) create mode 100644 apps/web/src/app/api/v1/rate/extension/observation/route.ts create mode 100644 apps/web/src/app/api/v1/rate/extension/submitter/route.ts create mode 100644 apps/web/src/server/rate/extension-observations.ts create mode 100644 packages/shared-data/src/rules/extension-observation.ts diff --git a/.gitleaksignore b/.gitleaksignore index a74366ea9..c293aac92 100644 --- a/.gitleaksignore +++ b/.gitleaksignore @@ -11,3 +11,6 @@ 2f624a1c620f68ecb658eed02aba62b05f05b04b:apps/web/src/lib/review-form-v2.ts:generic-api-key:193 4c0506053bbbc6668df016cdebfd3ee37fc9cea8:apps/web/src/lib/review-form-v2.ts:generic-api-key:75 4c0506053bbbc6668df016cdebfd3ee37fc9cea8:apps/web/src/lib/review-form-v2.ts:generic-api-key:76 +# Test fixture of the extension ingest key in write-paths.test.ts (#1606), a fake value read +# as a generic API key in an already-pushed commit. Later commits build it at run time. +d19083ac00645e6296031283025b5dd751eaab78:apps/web/src/app/api/v1/rate/__tests__/write-paths.test.ts:generic-api-key:894 diff --git a/apps/extension/src/background/service-worker.ts b/apps/extension/src/background/service-worker.ts index 268352c08..50f560419 100644 --- a/apps/extension/src/background/service-worker.ts +++ b/apps/extension/src/background/service-worker.ts @@ -11,6 +11,7 @@ import { } from '../extractors/ecommerce-detector.js'; import { buildCrowdIngestPayload, + buildLookupParams, DEFAULT_API_BASE_URL, isOnboardingSender, isPrivilegedExtensionSender, @@ -118,24 +119,8 @@ async function fetchExtensionLookup( rawSignal: ExtractedPageProductSignal, mode: 'pill' | 'full' ): Promise { - const signal = sanitizeExtensionProductSignal(rawSignal); const settings = await getSettings(); - const params = new URLSearchParams(); - params.set('platform', signal.platform); - params.set('url', signal.canonicalUrl); - params.set('title', signal.title); - params.set('mode', mode); - if (signal.externalItemId) params.set('externalId', signal.externalItemId); - if (signal.externalShopId) params.set('shopId', signal.externalShopId); - if (signal.selectedVariant) { - params.set('selectedVariant', signal.selectedVariant); - } - if (signal.price && signal.price > 0) { - params.set('price', String(signal.price)); - } - if (signal.originalPrice && signal.originalPrice > 0) { - params.set('originalPrice', String(signal.originalPrice)); - } + const params = buildLookupParams(rawSignal, mode); return requestLookup( `${settings.apiBaseUrl}${RATE_API_V1_ENDPOINTS.EXTENSION_LOOKUP}?${params.toString()}` diff --git a/apps/extension/src/extractors/ecommerce-detector.ts b/apps/extension/src/extractors/ecommerce-detector.ts index 46c9e6221..c0a8f890e 100644 --- a/apps/extension/src/extractors/ecommerce-detector.ts +++ b/apps/extension/src/extractors/ecommerce-detector.ts @@ -1,5 +1,6 @@ import type { ExtractedPageProductSignal, + ExtractedStockState, SupportedEcommercePlatform, } from '../types.js'; @@ -473,6 +474,167 @@ export function extractPriceFromJsonLdOffers(offersRaw: unknown): { return { price: resolvedPrice, originalPrice: validOriginal }; } +/** + * A schema.org availability (`https://schema.org/InStock`, `schema:InStock`, + * `InStock`, or a page's own "in stock" text) as a stock state; null when it + * says nothing (DOS/Rate#1606). Pre-orders and back-orders are unknown, not + * in stock. An array gives its first string. + */ +export function stockStateFromAvailability( + raw: unknown +): ExtractedStockState | null { + const first = Array.isArray(raw) + ? raw.find((v): v is string => typeof v === 'string') + : raw; + if (typeof first !== 'string' || first.length > 200) return null; + const v = first + .toLowerCase() + .replace(/^(?:https?:\/\/(?:www\.)?schema\.org\/|schema:)/, '') + .replace(/[\s_-]+/g, ''); + if (!v) return null; + if (/^(?:instock|instoreonly|onlineonly|available)$/.test(v)) { + return 'in_stock'; + } + if (/^(?:limitedavailability|lowstock)$/.test(v)) return 'low_stock'; + if (/^(?:outofstock|soldout|discontinued|reserved|oos)$/.test(v)) { + return 'out_of_stock'; + } + if ( + /^(?:preorder|presale|backorder|madetoorder|availablefororder)$/.test(v) + ) { + return 'unknown'; + } + return null; +} + +function stockQtyOf(raw: unknown): number | null { + const value = + raw && typeof raw === 'object' && !Array.isArray(raw) + ? (raw as Record).value + : raw; + const n = + typeof value === 'number' + ? value + : typeof value === 'string' && /^\d{1,7}$/.test(value.trim()) + ? Number(value.trim()) + : Number.NaN; + return Number.isInteger(n) && n >= 0 && n <= 1_000_000 ? n : null; +} + +/** The offers of a JSON-LD `offers` value, nested AggregateOffer ones too. */ +export function jsonLdOfferList(offersRaw: unknown): Record[] { + const offers: Record[] = []; + const push = (entry: unknown) => { + if ( + entry && + typeof entry === 'object' && + !Array.isArray(entry) && + offers.length < MAX_JSON_LD_OFFERS + ) { + offers.push(entry as Record); + } + }; + for (const entry of Array.isArray(offersRaw) ? offersRaw : [offersRaw]) { + const nested = (entry as { offers?: unknown } | null)?.offers; + if (nested && typeof nested === 'object') { + (Array.isArray(nested) ? nested : [nested]).forEach(push); + if (!(entry as { availability?: unknown }).availability) continue; + } + push(entry); + } + return offers; +} + +/** + * The stock of one offer. A quantity is kept only when it agrees with the + * state: none for an unknown state, 0 for out of stock, and none for an + * "in stock" offer that says 0. + */ +export function stockOfOffer(offer: Record | null): { + stockState: ExtractedStockState | null; + stockQty: number | null; +} { + const stockState = offer + ? stockStateFromAvailability(offer.availability) + : null; + if (!stockState || stockState === 'unknown') { + return { stockState, stockQty: null }; + } + if (stockState === 'out_of_stock') return { stockState, stockQty: 0 }; + const qty = stockQtyOf(offer?.inventoryLevel); + return { stockState, stockQty: qty === 0 ? null : qty }; +} + +/** Stock state and quantity from JSON-LD offers: the first offer that says. */ +export function extractStockFromJsonLdOffers(offersRaw: unknown): { + stockState: ExtractedStockState | null; + stockQty: number | null; +} { + for (const offer of jsonLdOfferList(offersRaw)) { + const stock = stockOfOffer(offer); + if (stock.stockState) return stock; + } + return { stockState: null, stockQty: null }; +} + +/** + * The offer of the selected variant: the only offer, or the one whose sku or + * name matches the selected variant; null when several offers and none can be + * told apart, so stock and SKU are not read off another variant. + */ +export function pickVariantOffer( + offers: readonly Record[], + selectedVariant: string | null +): Record | null { + if (offers.length === 1) return offers[0] ?? null; + if (!selectedVariant) return null; + const wanted = selectedVariant.toLowerCase(); + const matches = offers.filter((offer) => + [offer.sku, offer.name].some( + (v) => + typeof v === 'string' && + v.trim().length > 0 && + wanted.includes(v.trim().toLowerCase()) + ) + ); + return matches.length === 1 ? (matches[0] ?? null) : null; +} + +const SKU_ID_RE = /^[A-Za-z0-9_-][A-Za-z0-9_.-]{0,63}$/; + +/** + * The id of the selected variant: from the URL where the store puts it there + * (Lazada `-s.html`, Tiki `?spid=`), else the given structured-data sku + * (DOS/Rate#1606). `rawUrl` is the page URL before cleaning, since Tiki keeps + * it in the query. + */ +export function extractSkuId( + platform: SupportedEcommercePlatform, + rawUrl: string, + jsonLdSku: unknown +): string | null { + try { + const url = new URL(rawUrl); + if (platform === 'lazada') { + const m = url.pathname.match(/-s(\d{1,20})\.html$/i); + if (m?.[1]) return m[1]; + } + if (platform === 'tiki') { + const spid = url.searchParams.get('spid'); + if (spid && /^\d{1,20}$/.test(spid)) return spid; + } + } catch { + // Fall through to the structured data. + } + const sku = + typeof jsonLdSku === 'number' && Number.isInteger(jsonLdSku) + ? String(jsonLdSku) + : typeof jsonLdSku === 'string' + ? jsonLdSku.trim() + : ''; + return SKU_ID_RE.test(sku) ? sku : null; +} + export function extractRatingFromJsonLd(aggRaw: unknown): { ratingStar: number | null; ratingCount: number | null; @@ -640,6 +802,11 @@ export function extractProductSignalFromDom( let originalPrice: number | null = null; let ratingStar: number | null = null; let ratingCount: number | null = null; + // The first JSON-LD product's offers and sku (DOS/Rate#1606). + let productOffers: Record[] | null = null; + let productSku: unknown = null; + let priceFromJsonLd = false; + let originalFromDom = false; // 1. Try JSON-LD Product structured data first (cleanest & resilient across SPA updates) const ldScripts = doc.querySelectorAll('script[type="application/ld+json"]'); @@ -652,10 +819,15 @@ export function extractProductSignalFromDom( if (!title && typeof item.name === 'string') { title = cleanExtractedProductTitle(item.name); } + if (productOffers === null) { + productOffers = jsonLdOfferList(item.offers); + productSku = item.sku ?? null; + } if (!price || !originalPrice) { const offerPrices = extractPriceFromJsonLdOffers(item.offers); if (!price && offerPrices.price) { price = offerPrices.price; + priceFromJsonLd = true; } if ( !originalPrice && @@ -726,12 +898,36 @@ export function extractProductSignalFromDom( const parsed = parseVndPriceText(el?.textContent); if (parsed && (!price || parsed > price)) { originalPrice = parsed; + originalFromDom = true; break; } } const selectedVariant = extractSelectedVariantFromDom(doc); + // 4. Stock and SKU from the selected variant's offer only, so they never + // describe another variant; then the page's product meta for the stock. + const offers = productOffers ?? []; + const offer = pickVariantOffer(offers, selectedVariant); + let { stockState, stockQty } = stockOfOffer(offer); + const stockFromJsonLd = stockState !== null; + if (!stockState && offers.length <= 1) { + const meta = doc.querySelector( + 'meta[property="product:availability"], meta[property="og:availability"]' + ); + ({ stockState, stockQty } = stockOfOffer({ + availability: meta?.getAttribute('content'), + })); + } + const jsonLdSku = + offer?.sku ?? (offers.length <= 1 ? productSku : null) ?? null; + + // Where the fields v1 carries came from: price, list price and stock. + const sources = [ + price !== null ? priceFromJsonLd : null, + originalPrice !== null ? !originalFromDom : null, + stockState !== null ? stockFromJsonLd : null, + ].filter((v): v is boolean => v !== null); return { platform, canonicalUrl, @@ -743,5 +939,14 @@ export function extractProductSignalFromDom( originalPrice, ratingStar, ratingCount, + skuId: extractSkuId(platform, currentUrl, jsonLdSku), + stockState: stockState ?? 'unknown', + stockQty: stockState ? stockQty : null, + extractorSource: + sources.length > 0 && sources.every(Boolean) + ? 'json_ld' + : sources.some(Boolean) + ? 'mixed' + : 'dom', }; } diff --git a/apps/extension/src/security-guards.ts b/apps/extension/src/security-guards.ts index d25026002..9c248025a 100644 --- a/apps/extension/src/security-guards.ts +++ b/apps/extension/src/security-guards.ts @@ -1,3 +1,7 @@ +import { + parseExtensionObservationV1, + type ExtensionObservationV1, +} from '@repo/shared-data/rules'; import type { ExtractedPageProductSignal, SupportedEcommercePlatform, @@ -312,9 +316,105 @@ export function sanitizeExtensionProductSignal( originalPrice: validOriginalPrice, ratingStar: validRatingStar, ratingCount: validRatingCount, + // The v1 observation fields (DOS/Rate#1606), checked like the rest. + skuId: (() => { + const sku = sanitizeExternalIdentifier(safeSignal.skuId); + return sku && sku.length <= 64 ? sku : null; + })(), + stockState: ( + ['in_stock', 'low_stock', 'out_of_stock', 'unknown'] as const + ).includes(safeSignal.stockState as never) + ? safeSignal.stockState + : 'unknown', + // A quantity only with a known state (DOS/Rate#1606). + stockQty: + ['in_stock', 'low_stock', 'out_of_stock'].includes( + safeSignal.stockState as string + ) && + typeof safeSignal.stockQty === 'number' && + Number.isInteger(safeSignal.stockQty) && + safeSignal.stockQty >= 0 && + safeSignal.stockQty <= 1_000_000 + ? safeSignal.stockQty + : null, + extractorSource: (['json_ld', 'dom', 'mixed'] as const).includes( + safeSignal.extractorSource as never + ) + ? safeSignal.extractorSource + : 'dom', }; } +/** + * The query of the product lookup: exactly the fields the disclosures name + * (store, cleaned URL, title, item and shop ids, variant, price, original + * price) and the mode. A test pins its output (DOS/Rate#1606). + */ +export function buildLookupParams( + rawSignal: ExtractedPageProductSignal, + mode: 'pill' | 'full' +): URLSearchParams { + const signal = sanitizeExtensionProductSignal(rawSignal); + const params = new URLSearchParams(); + params.set('platform', signal.platform); + params.set('url', signal.canonicalUrl); + params.set('title', signal.title); + params.set('mode', mode); + if (signal.externalItemId) params.set('externalId', signal.externalItemId); + if (signal.externalShopId) params.set('shopId', signal.externalShopId); + if (signal.selectedVariant) { + params.set('selectedVariant', signal.selectedVariant); + } + if (signal.price && signal.price > 0) { + params.set('price', String(signal.price)); + } + if (signal.originalPrice && signal.originalPrice > 0) { + params.set('originalPrice', String(signal.originalPrice)); + } + return params; +} + +/** + * The ExtensionObservation v1 an opted-in install would send for `signal` + * (DOS/Rate#1606), or null when the page lacks what v1 needs (an item id, a + * price). Not sent yet: the opt-in signal still uses buildCrowdIngestPayload + * until the quarantine table exists and the disclosures name v1. + */ +export function buildExtensionObservationV1( + rawSignal: ExtractedPageProductSignal, + observedAt: Date, + extensionVersion: string +): ExtensionObservationV1 | null { + const signal = sanitizeExtensionProductSignal(rawSignal); + const parsed = parseExtensionObservationV1( + { + schema_version: 1, + marketplace: signal.platform, + listing_url: signal.canonicalUrl, + external_item_id: signal.externalItemId, + external_shop_id: signal.externalShopId, + sku_id: signal.skuId ?? null, + // v1 allows 120 characters; a longer label is cut, not a reason to + // drop the observation. + variant_label: signal.selectedVariant + ? Array.from(signal.selectedVariant).slice(0, 120).join('').trim() || + null + : null, + price_vnd: signal.price, + list_price_vnd: signal.originalPrice, + stock_state: signal.stockState ?? 'unknown', + stock_qty: signal.stockQty ?? null, + observed_at: observedAt.toISOString(), + extractor: { + source: signal.extractorSource ?? 'dom', + extension_version: extensionVersion, + }, + }, + observedAt.getTime() + ); + return parsed.ok ? parsed.value : null; +} + export function escapeSafeExternalHref( rawUrl: string | null | undefined, fallbackUrl = 'https://rate.vn' diff --git a/apps/extension/src/test-extension-pipeline.ts b/apps/extension/src/test-extension-pipeline.ts index a7ffce1d3..33318d466 100644 --- a/apps/extension/src/test-extension-pipeline.ts +++ b/apps/extension/src/test-extension-pipeline.ts @@ -5058,11 +5058,18 @@ async function runTests() { path.resolve(__dirname, 'background/service-worker.ts'), 'utf8' ); + // The lookup query is built in security-guards (DOS/Rate#1606). + const guardsSource = fs.readFileSync( + path.resolve(__dirname, 'security-guards.ts'), + 'utf8' + ); const sentParams = [ ...new Set( - [...workerSource.matchAll(/params\.set\(\s*'([^']+)'/g)].map( - (m) => m[1]! - ) + [ + ...(workerSource + guardsSource).matchAll( + /params\.set\(\s*'([^']+)'/g + ), + ].map((m) => m[1]!) ), ].sort(); assert.deepEqual(sentParams, [ @@ -5089,6 +5096,284 @@ async function runTests() { ); } + // --- DOS/Rate#1606: stock, SKU and the v1 observation (read, not sent) --- + { + const det = await import('./extractors/ecommerce-detector.js'); + const guards = await import('./security-guards.js'); + assert.equal( + det.stockStateFromAvailability('https://schema.org/InStock'), + 'in_stock' + ); + assert.equal( + det.stockStateFromAvailability('http://schema.org/OutOfStock'), + 'out_of_stock' + ); + assert.equal( + det.stockStateFromAvailability('LimitedAvailability'), + 'low_stock' + ); + assert.equal(det.stockStateFromAvailability('PreOrder'), 'unknown'); + assert.equal(det.stockStateFromAvailability('in stock'), 'in_stock'); + assert.equal(det.stockStateFromAvailability('blue'), null); + assert.equal(det.stockStateFromAvailability(42), null); + assert.deepEqual( + det.extractStockFromJsonLdOffers([ + { price: 1 }, + { + availability: 'https://schema.org/InStock', + inventoryLevel: { value: 7 }, + }, + ]), + { stockState: 'in_stock', stockQty: 7 } + ); + assert.deepEqual( + det.extractStockFromJsonLdOffers({ + availability: 'OutOfStock', + inventoryLevel: 9, + }), + { stockState: 'out_of_stock', stockQty: 0 } + ); + assert.deepEqual( + det.extractStockFromJsonLdOffers({ + availability: 'InStock', + inventoryLevel: -3, + }), + { stockState: 'in_stock', stockQty: null } + ); + assert.equal( + det.extractSkuId( + 'lazada', + 'https://www.lazada.vn/products/iphone-i123-s456.html', + null + ), + '456' + ); + assert.equal( + det.extractSkuId( + 'tiki', + 'https://tiki.vn/iphone-p123.html?spid=789', + 'X1' + ), + '789' + ); + assert.equal( + det.extractSkuId('shopee', 'https://shopee.vn/x-i.1.2', 'MDL-9'), + 'MDL-9' + ); + assert.equal( + det.extractSkuId('shopee', 'https://shopee.vn/x-i.1.2', 'a b'), + null + ); + assert.equal( + det.extractSkuId('tiki', 'https://tiki.vn/x-p1.html?spid=abc', null), + null + ); + + const signal = { + platform: 'shopee' as const, + canonicalUrl: 'https://shopee.vn/iphone-16-i.123.456', + externalItemId: '456', + externalShopId: '123', + title: 'iPhone 16 128GB', + selectedVariant: 'Đen', + price: 19_990_000, + originalPrice: 22_990_000, + ratingStar: 4.9, + ratingCount: 1000, + skuId: '789', + stockState: 'in_stock' as const, + stockQty: 5, + extractorSource: 'json_ld' as const, + }; + const at = new Date('2026-11-04T10:00:00Z'); + const obs = guards.buildExtensionObservationV1(signal, at, '2.1.0'); + assert.deepEqual(obs, { + schema_version: 1, + marketplace: 'shopee', + listing_url: 'https://shopee.vn/iphone-16-i.123.456', + external_item_id: '456', + external_shop_id: '123', + sku_id: '789', + variant_label: 'Đen', + price_vnd: 19_990_000, + list_price_vnd: 22_990_000, + stock_state: 'in_stock', + stock_qty: 5, + observed_at: '2026-11-04T10:00:00.000Z', + extractor: { source: 'json_ld', extension_version: '2.1.0' }, + }); + // No ratings in v1 (JOY approved dropping them). + assert.ok(obs && !('rating_star' in obs) && !('ratingStar' in obs)); + assert.equal( + guards.buildExtensionObservationV1( + { ...signal, price: null }, + at, + '2.1.0' + ), + null + ); + assert.equal( + guards.buildExtensionObservationV1( + { ...signal, externalItemId: null }, + at, + '2.1.0' + ), + null + ); + // The sanitizer keeps only valid v1 fields. + const dirty = guards.sanitizeExtensionProductSignal({ + ...signal, + skuId: 'a b', + stockState: 'lots' as never, + stockQty: -1, + extractorSource: 'api' as never, + }); + assert.equal(dirty.skuId, null); + assert.equal(dirty.stockState, 'unknown'); + assert.equal(dirty.stockQty, null); + assert.equal(dirty.extractorSource, 'dom'); + // Neither the opt-in signal nor the lookup carries the new fields yet. + const crowd = guards.buildCrowdIngestPayload(signal as never); + for (const key of ['skuId', 'stockState', 'stockQty', 'extractorSource']) { + assert.ok(!(key in crowd), key); + } + // The exact lookup query and opt-in payload, pinned: they are what the + // disclosures describe. + assert.equal( + guards.buildLookupParams(signal, 'pill').toString(), + 'platform=shopee&url=https%3A%2F%2Fshopee.vn%2Fiphone-16-i.123.456&title=iPhone+16+128GB&mode=pill&externalId=456&shopId=123&selectedVariant=%C4%90en&price=19990000&originalPrice=22990000' + ); + assert.equal( + JSON.stringify(crowd), + JSON.stringify({ + platform: 'shopee', + externalItemId: '456', + externalShopId: '123', + url: 'https://shopee.vn/iphone-16-i.123.456', + title: 'iPhone 16 128GB', + selectedVariant: 'Đen', + price: 19_990_000, + originalPrice: 22_990_000, + ratingStar: 4.9, + ratingCount: 1000, + userConsentOptIn: true, + }) + ); + const workerSource = fs.readFileSync( + path.resolve(__dirname, 'background/service-worker.ts'), + 'utf8' + ); + assert.ok(!/stockState|skuId|stockQty|extractorSource/.test(workerSource)); + assert.ok(workerSource.includes('buildLookupParams(rawSignal, mode)')); + + // A long variant label is cut to 120 characters, not a lost observation. + const longObs = guards.buildExtensionObservationV1( + { ...signal, selectedVariant: 'Màu '.repeat(50) }, + at, + '2.1.0' + ); + assert.ok(longObs && longObs.variant_label!.length <= 120); + + // More availability forms; quantities only agree with the state. + assert.equal(det.stockStateFromAvailability('schema:InStock'), 'in_stock'); + assert.equal( + det.stockStateFromAvailability('https://www.schema.org/SoldOut'), + 'out_of_stock' + ); + assert.equal(det.stockStateFromAvailability('Reserved'), 'out_of_stock'); + assert.equal(det.stockStateFromAvailability(['InStock']), 'in_stock'); + assert.equal( + det.stockStateFromAvailability('available for order'), + 'unknown' + ); + assert.deepEqual( + det.stockOfOffer({ availability: 'InStock', inventoryLevel: 0 }), + { + stockState: 'in_stock', + stockQty: null, + } + ); + assert.deepEqual( + det.stockOfOffer({ availability: 'PreOrder', inventoryLevel: 50 }), + { + stockState: 'unknown', + stockQty: null, + } + ); + assert.deepEqual( + det.extractStockFromJsonLdOffers({ + '@type': 'AggregateOffer', + offers: { availability: 'LimitedAvailability', inventoryLevel: '3' }, + }), + { stockState: 'low_stock', stockQty: 3 } + ); + assert.equal( + det.extractSkuId('shopee', 'https://shopee.vn/x-i.1.2', '.'), + null + ); + + // End to end on a page: the selected variant's own offer gives stock and + // SKU, never another variant's or the parent's. + const variantPage = (selected: string | null, offers: unknown) => + ({ + title: 'iPhone 16', + querySelectorAll(sel: string) { + if (sel === 'script[type="application/ld+json"]') { + return [ + { + textContent: JSON.stringify({ + '@type': 'Product', + name: 'iPhone 16 128GB', + sku: 'BASE-1', + offers, + }), + }, + ]; + } + if (selected && sel.includes('aria-checked')) { + return [{ textContent: selected }]; + } + return []; + }, + querySelector() { + return null; + }, + }) as unknown as Document; + const twoOffers = [ + { sku: 'V-RED', name: 'Đỏ', price: 19990000, availability: 'OutOfStock' }, + { + sku: 'V-BLUE', + name: 'Xanh', + price: 19990000, + availability: 'InStock', + inventoryLevel: 4, + }, + ]; + const url = 'https://shopee.vn/iphone-16-i.123.456'; + const none = det.extractProductSignalFromDom( + variantPage(null, twoOffers), + url + ); + assert.ok(none); + assert.equal(none.stockState, 'unknown'); + assert.equal(none.skuId, null); + assert.equal(none.extractorSource, 'json_ld'); + const single = det.extractProductSignalFromDom( + variantPage(null, { + price: 19990000, + availability: 'InStock', + inventoryLevel: 9, + }), + url + ); + assert.ok(single); + assert.equal(single.stockState, 'in_stock'); + assert.equal(single.stockQty, 9); + assert.equal(single.skuId, 'BASE-1'); + console.log( + ' ✅ [Stock, SKU and the v1 observation, not sent (#1606)] Verified' + ); + } console.log('🎉 All Rate Browser Extension MV3 v2 tests passed!'); } diff --git a/apps/extension/src/types.ts b/apps/extension/src/types.ts index 0328f57fb..e40ab530a 100644 --- a/apps/extension/src/types.ts +++ b/apps/extension/src/types.ts @@ -202,6 +202,9 @@ export interface ExtensionLookupData { } | null; } +export type ExtractedStockState = + 'in_stock' | 'low_stock' | 'out_of_stock' | 'unknown'; + export interface ExtractedPageProductSignal { platform: SupportedEcommercePlatform; canonicalUrl: string; @@ -213,6 +216,15 @@ export interface ExtractedPageProductSignal { originalPrice: number | null; ratingStar: number | null; ratingCount: number | null; + /** + * Read for ExtensionObservation v1 (DOS/Rate#1606) and not sent yet: + * neither the lookup nor the opt-in signal carries them. + */ + skuId?: string | null; + stockState?: ExtractedStockState; + stockQty?: number | null; + /** Where the title and price came from. */ + extractorSource?: 'json_ld' | 'dom' | 'mixed'; } export interface ExtensionSettings { diff --git a/apps/web/src/app/api/v1/rate/__tests__/write-paths.test.ts b/apps/web/src/app/api/v1/rate/__tests__/write-paths.test.ts index 8524c91de..39f2e294d 100644 --- a/apps/web/src/app/api/v1/rate/__tests__/write-paths.test.ts +++ b/apps/web/src/app/api/v1/rate/__tests__/write-paths.test.ts @@ -829,6 +829,328 @@ async function testMyReviewValues() { assert.equal(res.status, 502); } +async function testExtensionObservations() { + const submitterRoute = await import('../extension/submitter/route'); + const observationRoute = await import('../extension/observation/route'); + const obsMod = await import('@/server/rate/extension-observations'); + const SUBMITTER = 'cccccccc-1111-4222-8333-444444444444'; + const observation = { + schema_version: 1, + marketplace: 'shopee', + listing_url: 'https://shopee.vn/iphone-16-i.123.456', + external_item_id: '456', + external_shop_id: '123', + sku_id: '789', + variant_label: 'Đen', + price_vnd: 19_990_000, + list_price_vnd: 22_990_000, + stock_state: 'in_stock', + stock_qty: 5, + observed_at: new Date(Date.now() - 60_000).toISOString(), + extractor: { source: 'json_ld', extension_version: '2.1.0' }, + }; + const withIp = (req: Request, ip: string) => { + req.headers.set('x-forwarded-for', ip); + return req; + }; + const del = (body: unknown) => + new Request('http://localhost/api/v1/rate/extension/submitter', { + method: 'DELETE', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(body), + }); + + // Off by default: 404 and no database call on any route. + delete process.env.RATE_EXTENSION_OBSERVATIONS; + calls = []; + for (const res of [ + await submitterRoute.POST(post('/api/v1/rate/extension/submitter', {})), + await submitterRoute.DELETE(del({ submitterId: SUBMITTER })), + await observationRoute.POST( + post('/api/v1/rate/extension/observation', { + submitterId: SUBMITTER, + observation, + }) + ), + ]) { + assert.equal(res.status, 404); + assert.equal((await res.json()).code, 'NOT_ENABLED'); + } + assert.equal(calls.length, 0, 'a disabled route touches nothing'); + + // A preflight is a 404 too while off. + assert.equal( + observationRoute.OPTIONS( + new Request('http://localhost/api/v1/rate/extension/observation', { + method: 'OPTIONS', + }) + ).status, + 404 + ); + + process.env.RATE_EXTENSION_OBSERVATIONS = '1'; + const savedSalt = process.env.RATE_NETWORK_HASH_SALT; + const savedKey = process.env.RATE_EXTENSION_INGEST_KEY; + // A test value built at run time, so it never reads as a secret. + const INGEST_KEY = ['ingest', 'test', 'key'].join('-').padEnd(40, 'x'); + try { + // On, but without the salt or the ingest key: writes are refused before + // the database, since its limits and key check would not hold. + for (const [salt, key] of [ + [undefined, INGEST_KEY], + ['test-salt-0123456789', undefined], + ['test-salt-0123456789', 'too-short'], + ] as const) { + if (salt) process.env.RATE_NETWORK_HASH_SALT = salt; + else delete process.env.RATE_NETWORK_HASH_SALT; + if (key) process.env.RATE_EXTENSION_INGEST_KEY = key; + else delete process.env.RATE_EXTENSION_INGEST_KEY; + calls = []; + for (const res of [ + await submitterRoute.POST(post('/api/v1/rate/extension/submitter', {})), + await observationRoute.POST( + post('/api/v1/rate/extension/observation', { + submitterId: SUBMITTER, + observation, + }) + ), + ]) { + assert.equal(res.status, 503); + } + assert.equal(calls.length, 0); + } + process.env.RATE_NETWORK_HASH_SALT = 'test-salt-0123456789'; + process.env.RATE_EXTENSION_INGEST_KEY = INGEST_KEY; + + // A web page cannot call the routes; an extension origin gets CORS; a + // body that is not JSON is refused. + calls = []; + const fromSite = post('/api/v1/rate/extension/submitter', {}); + fromSite.headers.set('origin', 'https://evil.example'); + assert.equal((await submitterRoute.POST(fromSite)).status, 403); + const asText = new Request( + 'http://localhost/api/v1/rate/extension/observation', + { + method: 'POST', + headers: { 'Content-Type': 'text/plain' }, + body: JSON.stringify({ submitterId: SUBMITTER, observation }), + } + ); + assert.equal((await observationRoute.POST(asText)).status, 415); + assert.equal(calls.length, 0); + const preflight = observationRoute.OPTIONS( + new Request('http://localhost/api/v1/rate/extension/observation', { + method: 'OPTIONS', + headers: { origin: 'chrome-extension://abcdefghijklmnop' }, + }) + ); + assert.equal(preflight.status, 204); + assert.equal( + preflight.headers.get('access-control-allow-origin'), + 'chrome-extension://abcdefghijklmnop' + ); + + // A submitter id comes from the RPC, as anon, never the service key. + calls = []; + rpcResponder = (name) => + name === 'register_extension_submitter' + ? { status: 200, json: SUBMITTER } + : { status: 500, json: { message: 'unexpected rpc' } }; + let res = await submitterRoute.POST( + post('/api/v1/rate/extension/submitter', {}) + ); + assert.equal(res.status, 200); + assert.deepEqual(await res.json(), { ok: true, submitterId: SUBMITTER }); + assert.deepEqual(rpcCalls('register_extension_submitter')[0]?.body, { + p_ingest_key: INGEST_KEY, + p_network_hash: obsMod.networkHashFor('anon', 'test-salt-0123456789'), + }); + assert.equal(res.headers.get('access-control-allow-origin'), null); + assert.match(res.headers.get('cache-control') ?? '', /no-store/); + assert.ok( + calls.every((c) => !JSON.stringify(c.headers).includes(SERVICE_KEY)) + ); + + // A valid observation goes to the RPC with a hash of the /24, not the IP. + calls = []; + rpcResponder = (name) => + name === 'submit_extension_observation' + ? { + status: 200, + json: { id: 1, status: 'quarantined', received_at: 'now' }, + } + : { status: 500, json: { message: 'unexpected rpc' } }; + res = await observationRoute.POST( + withIp( + post('/api/v1/rate/extension/observation', { + submitterId: SUBMITTER.toUpperCase(), + observation, + }), + '203.0.113.77' + ) + ); + assert.equal(res.status, 200); + assert.deepEqual(await res.json(), { ok: true, status: 'quarantined' }); + const [call] = rpcCalls('submit_extension_observation'); + assert.ok(call, 'submit_extension_observation must be called'); + const args = call.body as Record; + assert.equal(args.p_ingest_key, INGEST_KEY); + assert.equal(args.p_submitter_id, SUBMITTER); + assert.deepEqual(args.p_observation, observation); + assert.equal( + args.p_network_hash, + obsMod.networkHashFor('203.0.113.1', 'test-salt-0123456789') + ); + assert.match(String(args.p_network_hash), /^[0-9a-f]{64}$/); + assert.ok( + !JSON.stringify(calls).includes('203.0.113.77'), + 'the raw IP is never sent on' + ); + + // Bad input is refused before the database. + for (const [body, code, field] of [ + [{ submitterId: 'x', observation }, 'INVALID_SUBMITTER', undefined], + [ + { + submitterId: SUBMITTER, + observation: { ...observation, rating_star: 5 }, + }, + 'INVALID_OBSERVATION', + 'observation', + ], + [ + { + submitterId: SUBMITTER, + observation: { ...observation, listing_url: 'https://evil.com/x' }, + }, + 'INVALID_OBSERVATION', + 'listing_url', + ], + ] as const) { + calls = []; + res = await observationRoute.POST( + post('/api/v1/rate/extension/observation', body) + ); + assert.equal(res.status, 400); + const json = await res.json(); + assert.equal(json.code, code); + if (field) assert.equal(json.field, field); + assert.equal(calls.length, 0); + } + + // Database answers map to fixed codes, never a faked success. + for (const [error, status, code] of [ + [{ code: '22023', details: 'price_vnd' }, 400, 'INVALID_OBSERVATION'], + [{ code: 'P0002', details: 'submitter_id' }, 404, 'UNKNOWN_SUBMITTER'], + [{ code: '54000' }, 429, 'RATE_LIMIT_EXCEEDED'], + [ + { code: '42501', details: 'ingest_key' }, + 503, + 'OBSERVATIONS_UNAVAILABLE', + ], + [ + { code: 'PGRST202', message: 'Could not find' }, + 503, + 'OBSERVATIONS_UNAVAILABLE', + ], + [{ code: 'XX000', message: 'boom' }, 502, 'PERSIST_FAILED'], + ] as const) { + rpcResponder = () => ({ + status: error.code === 'PGRST202' ? 404 : 400, + json: error, + }); + res = await observationRoute.POST( + post('/api/v1/rate/extension/observation', { + submitterId: SUBMITTER, + observation, + }) + ); + assert.equal(res.status, status, error.code); + const json = await res.json(); + assert.equal(json.code, code, error.code); + if (error.code === '22023') assert.equal(json.field, 'price_vnd'); + if (error.code === '54000') { + assert.equal(res.headers.get('retry-after'), '3600'); + } + } + // The global register brake is per minute. + rpcResponder = () => ({ + status: 400, + json: { code: '54000', details: 'submitter' }, + }); + const brake = await submitterRoute.POST( + post('/api/v1/rate/extension/submitter', {}) + ); + assert.equal(brake.status, 429); + assert.equal(brake.headers.get('retry-after'), '60'); + + // Deleting answers the same for any id and calls the delete RPC. + calls = []; + rpcResponder = (name) => + name === 'delete_extension_submitter' + ? { status: 200, json: 0 } + : { status: 500, json: { message: 'unexpected rpc' } }; + res = await submitterRoute.DELETE(del({ submitterId: SUBMITTER })); + assert.equal(res.status, 200); + assert.deepEqual(await res.json(), { ok: true }); + assert.deepEqual(rpcCalls('delete_extension_submitter')[0]?.body, { + p_submitter_id: SUBMITTER, + }); + res = await submitterRoute.DELETE(del({ submitterId: 'nope' })); + assert.equal(res.status, 400); + + // Network ranges: /24 for IPv4, /48 for IPv6, nothing otherwise. + assert.equal(obsMod.networkPrefixOf('203.0.113.77'), '203.0.113.0/24'); + assert.equal( + obsMod.networkPrefixOf('2001:db8:abcd:12::1'), + '2001:db8:abcd::/48' + ); + assert.equal( + obsMod.networkPrefixOf('2001:0db8:0000::1'), + '2001:db8:0::/48' + ); + assert.equal(obsMod.networkPrefixOf('::1'), '0:0:0::/48'); + assert.equal( + obsMod.networkPrefixOf('::ffff:203.0.113.9'), + '203.0.113.0/24' + ); + for (const bad of [ + 'anon', + '999.1.1.1', + '1.2.3', + 'a::b::c', + 'fe80::1%eth0', + '', + ]) { + assert.equal(obsMod.networkPrefixOf(bad), null, bad); + } + assert.equal(obsMod.networkHashFor('203.0.113.77', 'short'), null); + // An unusable address still lands in one shared bucket. + assert.match( + String(obsMod.networkHashFor('anon', 'test-salt-0123456789')), + /^[0-9a-f]{64}$/ + ); + assert.equal( + obsMod.networkHashFor('anon', 'test-salt-0123456789'), + obsMod.networkHashFor('1.2.3', 'test-salt-0123456789') + ); + assert.equal( + obsMod.networkHashFor('203.0.113.5', 'test-salt-0123456789'), + obsMod.networkHashFor('203.0.113.200', 'test-salt-0123456789') + ); + assert.notEqual( + obsMod.networkHashFor('203.0.113.5', 'test-salt-0123456789'), + obsMod.networkHashFor('203.0.114.5', 'test-salt-0123456789') + ); + } finally { + delete process.env.RATE_EXTENSION_OBSERVATIONS; + if (savedSalt === undefined) delete process.env.RATE_NETWORK_HASH_SALT; + else process.env.RATE_NETWORK_HASH_SALT = savedSalt; + if (savedKey === undefined) delete process.env.RATE_EXTENSION_INGEST_KEY; + else process.env.RATE_EXTENSION_INGEST_KEY = savedKey; + } +} + async function testWatchlist() { const { POST } = await import('../watchlist/route'); type Req = Parameters[0]; @@ -2120,6 +2442,7 @@ async function main() { await testReviewsV2(); await testReviewsByIdPublishedOnly(); await testMyReviewValues(); + await testExtensionObservations(); await testWatchlist(); await testCookieSessions(); await testFilterBoundsAndWatchlistPrices(); diff --git a/apps/web/src/app/api/v1/rate/extension/observation/route.ts b/apps/web/src/app/api/v1/rate/extension/observation/route.ts new file mode 100644 index 000000000..17bafadfa --- /dev/null +++ b/apps/web/src/app/api/v1/rate/extension/observation/route.ts @@ -0,0 +1,111 @@ +import { parseExtensionObservationV1 } from '@repo/shared-data/rules'; +import { + BoundedSlidingWindowRateLimiter, + extractClientRateLimitKey, + payloadTooLargeResponse, + rateLimitExceededResponse, + readBoundedJsonBody, +} from '@/server/rate/api-handler'; +import { + callObservationRpc, + hasNetworkHashSalt, + ingestKey, + isExtensionObservationsEnabled, + isJsonRequest, + isWebPageOrigin, + networkHashFor, + notEnabledResponse, + notJsonResponse, + observationHeaders, + observationOptions, + observationResponse, + parseSubmitterId, + rpcFailureResponse, + unavailableResponse, + webPageRefusedResponse, +} from '@/server/rate/extension-observations'; +import { createLogger, serializeError } from '@/lib/logger'; + +const log = createLogger('api.extension.observation'); + +// Best-effort per-instance brake; the RPC holds the per-submitter and +// per-network limits. +const observationLimiter = new BoundedSlidingWindowRateLimiter({ + windowMs: 60_000, + maxRequests: 60, + maxBuckets: 2_000, +}); + +export function OPTIONS(request: Request) { + return observationOptions(request); +} + +/** + * POST /api/v1/rate/extension/observation { submitterId, observation } + * (DOS/Rate#1606). The observation must be an ExtensionObservation v1; it is + * stored in the quarantine, never as a public price. The raw IP is not sent + * on: only a salted hash of its network range goes to the RPC. + */ +export async function POST(request: Request) { + if (!isExtensionObservationsEnabled()) return notEnabledResponse(request); + if (isWebPageOrigin(request)) return webPageRefusedResponse(request); + if (!isJsonRequest(request)) return notJsonResponse(request); + const key = ingestKey(); + if (!key || !hasNetworkHashSalt()) return unavailableResponse(request); + const clientIp = extractClientRateLimitKey(request.headers); + if (!observationLimiter.allow(clientIp)) { + return rateLimitExceededResponse( + 'Too many observations; try again in a minute.', + observationHeaders(request) + ); + } + const body = await readBoundedJsonBody(request, 8 * 1024); + if (body.tooLarge) + return payloadTooLargeResponse(observationHeaders(request)); + const raw = + body.value && typeof body.value === 'object' && !Array.isArray(body.value) + ? (body.value as { submitterId?: unknown; observation?: unknown }) + : {}; + const submitterId = parseSubmitterId(raw.submitterId); + if (!submitterId) { + return observationResponse(request, 400, { + ok: false, + code: 'INVALID_SUBMITTER', + error: 'Provide submitterId as a uuid.', + }); + } + const parsed = parseExtensionObservationV1(raw.observation); + if (!parsed.ok) { + return observationResponse(request, 400, { + ok: false, + code: 'INVALID_OBSERVATION', + error: 'The observation was refused.', + field: parsed.field, + }); + } + try { + const { data, error } = await callObservationRpc( + 'submit_extension_observation', + { + p_ingest_key: key, + p_submitter_id: submitterId, + p_observation: parsed.value, + p_network_hash: networkHashFor(clientIp), + } + ); + if (error) return rpcFailureResponse(request, error); + const stored = + data && typeof data === 'object' && !Array.isArray(data) + ? (data as { status?: unknown }) + : null; + return observationResponse(request, 200, { + ok: true, + status: stored?.status === 'quarantined' ? 'quarantined' : 'received', + }); + } catch (err) { + log.error('submit_extension_observation failed', { + error: serializeError(err), + }); + return rpcFailureResponse(request, {}); + } +} diff --git a/apps/web/src/app/api/v1/rate/extension/submitter/route.ts b/apps/web/src/app/api/v1/rate/extension/submitter/route.ts new file mode 100644 index 000000000..1c3ccb2ed --- /dev/null +++ b/apps/web/src/app/api/v1/rate/extension/submitter/route.ts @@ -0,0 +1,126 @@ +import { + BoundedSlidingWindowRateLimiter, + extractClientRateLimitKey, + payloadTooLargeResponse, + rateLimitExceededResponse, + readBoundedJsonBody, +} from '@/server/rate/api-handler'; +import { + callObservationRpc, + hasNetworkHashSalt, + ingestKey, + isExtensionObservationsEnabled, + isJsonRequest, + isWebPageOrigin, + networkHashFor, + notEnabledResponse, + notJsonResponse, + observationHeaders, + observationOptions, + observationResponse, + parseSubmitterId, + rpcFailureResponse, + unavailableResponse, + webPageRefusedResponse, +} from '@/server/rate/extension-observations'; +import { createLogger, serializeError } from '@/lib/logger'; + +const log = createLogger('api.extension.submitter'); + +// Best-effort per-instance brakes; the RPCs hold the real limits. +const registerLimiter = new BoundedSlidingWindowRateLimiter({ + windowMs: 60_000, + maxRequests: 10, + maxBuckets: 2_000, +}); +const deleteLimiter = new BoundedSlidingWindowRateLimiter({ + windowMs: 60_000, + maxRequests: 10, + maxBuckets: 2_000, +}); + +export function OPTIONS(request: Request) { + return observationOptions(request); +} + +/** + * POST /api/v1/rate/extension/submitter: a new pseudonymous submitter id for + * an install that opted in (DOS/Rate#1606). No body; the server creates the + * id and records when, and the RPC limits new ids per network. + */ +export async function POST(request: Request) { + if (!isExtensionObservationsEnabled()) return notEnabledResponse(request); + if (isWebPageOrigin(request)) return webPageRefusedResponse(request); + const key = ingestKey(); + if (!key || !hasNetworkHashSalt()) return unavailableResponse(request); + const clientIp = extractClientRateLimitKey(request.headers); + if (!registerLimiter.allow(clientIp)) { + return rateLimitExceededResponse( + 'Too many requests; try again in a minute.', + observationHeaders(request) + ); + } + try { + const { data, error } = await callObservationRpc( + 'register_extension_submitter', + { p_ingest_key: key, p_network_hash: networkHashFor(clientIp) } + ); + if (error) return rpcFailureResponse(request, error); + const submitterId = parseSubmitterId(data); + if (!submitterId) { + return observationResponse(request, 502, { + ok: false, + code: 'PERSIST_FAILED', + error: 'Could not create a submitter id.', + }); + } + return observationResponse(request, 200, { ok: true, submitterId }); + } catch (err) { + log.error('register_extension_submitter failed', { + error: serializeError(err), + }); + return rpcFailureResponse(request, {}); + } +} + +/** + * DELETE /api/v1/rate/extension/submitter { submitterId }: deletes the + * submitter and every observation it sent (the data subject's right to + * erasure). The answer is the same whether or not the id existed. + */ +export async function DELETE(request: Request) { + if (!isExtensionObservationsEnabled()) return notEnabledResponse(request); + if (isWebPageOrigin(request)) return webPageRefusedResponse(request); + if (!isJsonRequest(request)) return notJsonResponse(request); + if (!deleteLimiter.allow(extractClientRateLimitKey(request.headers))) { + return rateLimitExceededResponse( + 'Too many requests; try again in a minute.', + observationHeaders(request) + ); + } + const body = await readBoundedJsonBody(request, 1024); + if (body.tooLarge) + return payloadTooLargeResponse(observationHeaders(request)); + const submitterId = parseSubmitterId( + (body.value as { submitterId?: unknown } | null)?.submitterId + ); + if (!submitterId) { + return observationResponse(request, 400, { + ok: false, + code: 'INVALID_SUBMITTER', + error: 'Provide submitterId as a uuid.', + }); + } + try { + const { error } = await callObservationRpc('delete_extension_submitter', { + p_submitter_id: submitterId, + }); + if (error) return rpcFailureResponse(request, error); + return observationResponse(request, 200, { ok: true }); + } catch (err) { + log.error('delete_extension_submitter failed', { + error: serializeError(err), + }); + return rpcFailureResponse(request, {}); + } +} diff --git a/apps/web/src/server/rate/extension-observations.ts b/apps/web/src/server/rate/extension-observations.ts new file mode 100644 index 000000000..8259c90cc --- /dev/null +++ b/apps/web/src/server/rate/extension-observations.ts @@ -0,0 +1,260 @@ +import { createHmac } from 'node:crypto'; +import { NextResponse } from 'next/server'; +import { getRateClient } from '@/lib/rate-api'; +import { isMissingRpcError } from '@/server/rate/request-auth'; +import { rateLimitExceededResponse } from '@/server/rate/api-handler'; +import { createLogger } from '@/lib/logger'; + +const log = createLogger('server.extension-observations'); + +/** + * ExtensionObservation v1 ingest (DOS/Rate#1606, M2 of + * Docs/Proposals/2026-10-28-extension-data-source.md). An opted-in install + * gets a pseudonymous submitter id and sends observations into the + * quarantine table, through the narrow RPCs of + * Docs/Proposals/2026-11-04-dos-me-extension-observations.sql. Nothing + * reaches public prices; Rate-Services verifies (M3). + * + * Off until the table exists: every route answers 404 unless + * RATE_EXTENSION_OBSERVATIONS is "1", and then makes no database call. + * Writes also need two server secrets: RATE_EXTENSION_INGEST_KEY, which the + * RPCs check so nobody can call them around this route, and + * RATE_NETWORK_HASH_SALT for the per-network limits. + */ + +const BASE_HEADERS: Record = { + 'Access-Control-Allow-Methods': 'POST, DELETE, OPTIONS', + 'Access-Control-Allow-Headers': 'Content-Type, X-Rate-Extension-Version', + 'Cache-Control': 'no-store', + Vary: 'Origin', +}; + +/** Browser-extension origins: the only browser callers these routes serve. */ +const EXTENSION_ORIGIN_RE = + /^(?:chrome-extension|moz-extension|extension):\/\/[a-z0-9-]{1,64}$/i; + +/** + * The response headers for a request: CORS only for an extension origin. + * Callers without an Origin (the extension's service worker sends one; + * server tools send none) get no CORS header at all. + */ +export function observationHeaders( + request?: Request | null +): Record { + const origin = request?.headers.get('origin') ?? ''; + return EXTENSION_ORIGIN_RE.test(origin) + ? { ...BASE_HEADERS, 'Access-Control-Allow-Origin': origin } + : { ...BASE_HEADERS }; +} + +/** + * A web page's request: any http(s) Origin. Refused, so a site the user + * visits cannot register ids or send observations from the user's network. + */ +export function isWebPageOrigin(request: Request): boolean { + return /^https?:\/\//i.test(request.headers.get('origin') ?? ''); +} + +export function isJsonRequest(request: Request): boolean { + return /^application\/json\b/i.test( + request.headers.get('content-type') ?? '' + ); +} + +export function isExtensionObservationsEnabled(): boolean { + return process.env.RATE_EXTENSION_OBSERVATIONS === '1'; +} + +/** + * The per-network limits need the salt, and the RPCs need the ingest key; + * writes are refused until both are set. + */ +export function hasNetworkHashSalt( + salt = process.env.RATE_NETWORK_HASH_SALT +): boolean { + return typeof salt === 'string' && salt.length >= 16; +} + +export function ingestKey( + key = process.env.RATE_EXTENSION_INGEST_KEY +): string | null { + return typeof key === 'string' && key.length >= 32 ? key : null; +} + +export function observationResponse( + request: Request | null, + status: number, + body: Record +): NextResponse { + return NextResponse.json(body, { + status, + headers: observationHeaders(request), + }); +} + +export const notEnabledResponse = (request: Request | null) => + observationResponse(request, 404, { + ok: false, + code: 'NOT_ENABLED', + error: 'Extension observations are not enabled.', + }); + +export const unavailableResponse = (request: Request | null) => + observationResponse(request, 503, { + ok: false, + code: 'OBSERVATIONS_UNAVAILABLE', + error: 'Extension observations are not available yet.', + }); + +export const webPageRefusedResponse = (request: Request | null) => + observationResponse(request, 403, { + ok: false, + code: 'CROSS_SITE', + error: 'Web pages cannot call this endpoint.', + }); + +export const notJsonResponse = (request: Request | null) => + observationResponse(request, 415, { + ok: false, + code: 'UNSUPPORTED_MEDIA_TYPE', + error: 'Send Content-Type: application/json.', + }); + +const UUID_RE = + /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; + +export function parseSubmitterId(raw: unknown): string | null { + return typeof raw === 'string' && UUID_RE.test(raw) + ? raw.toLowerCase() + : null; +} + +/** + * The network range of an IP: the /24 of IPv4 (also IPv4-mapped IPv6), the + * /48 of IPv6; null for anything else (no address, a placeholder, a + * malformed value). + */ +export function networkPrefixOf(ip: string): string | null { + const mapped = /^::ffff:(\d{1,3}(?:\.\d{1,3}){3})$/i.exec(ip); + const v4 = /^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$/.exec( + mapped?.[1] ?? ip + ); + if (v4) { + const parts = v4.slice(1, 5).map(Number); + if (parts.some((p) => p > 255)) return null; + return `${parts[0]}.${parts[1]}.${parts[2]}.0/24`; + } + if (!/^[0-9a-f:]{2,39}$/i.test(ip) || !ip.includes(':')) return null; + const halves = ip.toLowerCase().split('::'); + if (halves.length > 2) return null; + const head = halves[0] ? halves[0].split(':') : []; + const tail = halves.length === 2 && halves[1] ? halves[1].split(':') : []; + const missing = 8 - head.length - tail.length; + if (halves.length === 1 ? missing !== 0 : missing < 1) return null; + const groups = [...head, ...Array(missing).fill('0'), ...tail]; + if (groups.some((g) => !/^[0-9a-f]{1,4}$/.test(g))) return null; + return `${groups + .slice(0, 3) + .map((g) => g.replace(/^0+(?=.)/, '')) + .join(':')}::/48`; +} + +/** + * HMAC-SHA256 of the network range with RATE_NETWORK_HASH_SALT, as lowercase + * hex: lets Rate-Services tell independent submitters apart without the raw + * IP ever being stored. An address that gives no range shares one "unknown" + * bucket, so the per-network limits still hold. Null only without a salt of + * at least 16 characters. + */ +export function networkHashFor( + ip: string, + salt = process.env.RATE_NETWORK_HASH_SALT +): string | null { + if (!hasNetworkHashSalt(salt)) return null; + const prefix = networkPrefixOf(ip) ?? 'unknown'; + return createHmac('sha256', salt as string) + .update(prefix) + .digest('hex'); +} + +interface LooseRpcClient { + rpc: ( + fn: string, + args?: Record + ) => PromiseLike<{ + data: unknown; + error: { code?: string; message?: string; details?: string } | null; + }>; +} + +/** Calls one of the M2 RPCs as anon; their types land with the migration. */ +export function callObservationRpc( + fn: + | 'register_extension_submitter' + | 'submit_extension_observation' + | 'delete_extension_submitter', + args?: Record +) { + return (getRateClient() as unknown as LooseRpcClient).rpc(fn, args); +} + +/** + * An RPC failure as an answer: never a success, never the raw message. The + * code is logged (not the message, which can carry input). + */ +export function rpcFailureResponse( + request: Request | null, + error: { code?: string; details?: string } +): NextResponse { + log.warn('extension observation rpc refused', { + code: typeof error.code === 'string' ? error.code.slice(0, 16) : null, + }); + if (isMissingRpcError(error)) return unavailableResponse(request); + if (error.code === '54000') { + // The per-submitter and per-network limits are hourly; the global + // register brake (DETAIL "submitter") is per minute. + return rateLimitExceededResponse( + 'Too many requests; try again later.', + observationHeaders(request), + error.details === 'submitter' ? 60 : 3600 + ); + } + if (error.code === 'P0002') { + return observationResponse(request, 404, { + ok: false, + code: 'UNKNOWN_SUBMITTER', + error: 'Unknown submitter.', + }); + } + if (error.code === '42501') { + // The ingest key was refused: a server configuration problem. + return unavailableResponse(request); + } + if (error.code === '22023') { + const field = + typeof error.details === 'string' && + /^[a-z_]{1,32}(?:\.[a-z_]{1,32})?$/.test(error.details) + ? error.details + : null; + return observationResponse(request, 400, { + ok: false, + code: 'INVALID_OBSERVATION', + error: 'The observation was refused.', + field, + }); + } + return observationResponse(request, 502, { + ok: false, + code: 'PERSIST_FAILED', + error: 'Could not store the observation.', + }); +} + +/** The answer to a preflight: 404 while the routes are off. */ +export function observationOptions(request: Request): Response { + if (!isExtensionObservationsEnabled()) return notEnabledResponse(request); + return new Response(null, { + status: 204, + headers: observationHeaders(request), + }); +} diff --git a/packages/shared-data/src/__tests__/shared-data.test.ts b/packages/shared-data/src/__tests__/shared-data.test.ts index 34fe87ffc..f008e4d4b 100644 --- a/packages/shared-data/src/__tests__/shared-data.test.ts +++ b/packages/shared-data/src/__tests__/shared-data.test.ts @@ -2322,6 +2322,149 @@ function testReviewDepth(): void { console.log(' ✅ review depth (axis C) over the whole review text (#1581)'); } +function testExtensionObservation(): void { + const now = Date.parse('2026-11-04T10:00:00Z'); + const valid = { + schema_version: 1, + marketplace: 'shopee', + listing_url: 'https://shopee.vn/iphone-16-i.123.456', + external_item_id: '456', + external_shop_id: '123', + sku_id: '789', + variant_label: 'Đen, 128GB', + price_vnd: 19_990_000, + list_price_vnd: 22_990_000, + stock_state: 'in_stock', + stock_qty: 12, + observed_at: '2026-11-04T09:59:00Z', + extractor: { source: 'json_ld', extension_version: '2.1.0' }, + }; + const parse = (o: unknown) => rules.parseExtensionObservationV1(o, now); + const ok = parse(valid); + assert.ok(ok.ok); + assert.deepEqual(ok.ok && ok.value, valid); + // Nullable fields may be null, but must be present. + assert.ok( + parse({ + ...valid, + external_shop_id: null, + sku_id: null, + variant_label: null, + list_price_vnd: null, + stock_qty: null, + stock_state: 'unknown', + }).ok + ); + const { sku_id: _dropped, ...missing } = valid; + void _dropped; + const cases: Array<[Record | unknown, string]> = [ + [null, 'observation'], + [[], 'observation'], + [{ ...valid, rating_star: 5 }, 'observation'], + [missing, 'sku_id'], + [{ ...valid, schema_version: 2 }, 'schema_version'], + [{ ...valid, marketplace: 'amazon' }, 'marketplace'], + [ + { ...valid, listing_url: 'https://shopee.vn/x?utm_source=a' }, + 'listing_url', + ], + [{ ...valid, listing_url: 'https://shopee.vn/x#a' }, 'listing_url'], + [{ ...valid, listing_url: 'http://shopee.vn/x' }, 'listing_url'], + [{ ...valid, listing_url: 'https://evil.com/shopee.vn' }, 'listing_url'], + [{ ...valid, listing_url: 'https://shopee.vn.evil.com/x' }, 'listing_url'], + [{ ...valid, listing_url: 'https://u:p@shopee.vn/x' }, 'listing_url'], + [ + { ...valid, listing_url: `https://shopee.vn/${'a'.repeat(2048)}` }, + 'listing_url', + ], + // The RPC's URL rules: lowercase host, no port, user, path parameter + // or space; no empty subdomain label. + [{ ...valid, listing_url: 'https://Shopee.vn/x' }, 'listing_url'], + [{ ...valid, listing_url: 'https://shopee.vn:443/x' }, 'listing_url'], + [{ ...valid, listing_url: 'https://@shopee.vn/x' }, 'listing_url'], + [ + { ...valid, listing_url: 'https://shopee.vn/x;jsessionid=1' }, + 'listing_url', + ], + [{ ...valid, listing_url: 'https://shopee.vn/a b' }, 'listing_url'], + [{ ...valid, listing_url: 'https:shopee.vn/x' }, 'listing_url'], + [ + { + ...valid, + marketplace: 'lazada', + listing_url: 'https://a..lazada.vn/x', + }, + 'listing_url', + ], + [{ ...valid, variant_label: 'a\ud800b' }, 'variant_label'], + [{ ...valid, observed_at: '2026-11-04T10:00:00+16:00' }, 'observed_at'], + [{ ...valid, observed_at: '2026-11-04T09:59:60Z' }, 'observed_at'], + [{ ...valid, observed_at: '2026-11-04' }, 'observed_at'], + [{ ...valid, observed_at: '2026-11-04T09:59' }, 'observed_at'], + [{ ...valid, observed_at: '2026-11-04 09:59Z' }, 'observed_at'], + [{ ...valid, external_item_id: '' }, 'external_item_id'], + [{ ...valid, external_item_id: 'a b' }, 'external_item_id'], + [{ ...valid, external_shop_id: 'x'.repeat(65) }, 'external_shop_id'], + [{ ...valid, sku_id: 7 }, 'sku_id'], + [{ ...valid, variant_label: '' }, 'variant_label'], + [{ ...valid, variant_label: 'a\u0000b' }, 'variant_label'], + [{ ...valid, price_vnd: 999 }, 'price_vnd'], + [{ ...valid, price_vnd: 1.5e3 + 0.5 }, 'price_vnd'], + [{ ...valid, price_vnd: '19990000' }, 'price_vnd'], + [{ ...valid, list_price_vnd: 19_000_000 }, 'list_price_vnd'], + [{ ...valid, stock_state: 'many' }, 'stock_state'], + [{ ...valid, stock_qty: -1 }, 'stock_qty'], + [{ ...valid, stock_qty: 1_000_001 }, 'stock_qty'], + [{ ...valid, observed_at: '2026-11-04T10:11:00Z' }, 'observed_at'], + [{ ...valid, observed_at: '2026-11-03T09:59:00Z' }, 'observed_at'], + [{ ...valid, observed_at: 'yesterday' }, 'observed_at'], + [{ ...valid, extractor: 'json_ld' }, 'extractor'], + [{ ...valid, extractor: { ...valid.extractor, user: 'x' } }, 'extractor'], + [ + { ...valid, extractor: { ...valid.extractor, source: 'api' } }, + 'extractor.source', + ], + [ + { ...valid, extractor: { ...valid.extractor, extension_version: '2 1' } }, + 'extractor.extension_version', + ], + ]; + for (const [input, field] of cases) { + const result = parse(input); + assert.equal(result.ok, false, JSON.stringify(input).slice(0, 80)); + assert.equal( + !result.ok && result.field, + field, + JSON.stringify(input).slice(0, 80) + ); + } + // Offsets the RPC accepts are accepted here too. + for (const at of [ + '2026-11-04T16:59+07', + '2026-11-04T16:59+0700', + '2026-11-04T16:59:00.123456+07:00', + ]) { + assert.ok(parse({ ...valid, observed_at: at }).ok, at); + } + assert.ok( + parse({ + ...valid, + marketplace: 'lazada', + listing_url: 'https://www.lazada.vn/products/x-i1-s2.html', + }).ok + ); + // Every marketplace accepts its own hosts and refuses another's. + assert.ok(rules.isExtensionMarketplaceHost('lazada', 'www.lazada.vn')); + assert.ok(rules.isExtensionMarketplaceHost('tgdd', 'www.thegioididong.com')); + assert.ok(rules.isExtensionMarketplaceHost('tiktok', 'shop.tiktok.com')); + assert.ok(!rules.isExtensionMarketplaceHost('tiktok', 'm.tiktok.com')); + assert.ok(!rules.isExtensionMarketplaceHost('lazada', 'notlazada.vn')); + assert.ok(!rules.isExtensionMarketplaceHost('shopee', 'tiki.vn')); + for (const m of rules.EXTENSION_MARKETPLACES) { + assert.ok(rules.EXTENSION_MARKETPLACE_HOSTS[m].length > 0, m); + } +} + function testReviewValue(): void { const phone = rules.getRatingCriteria('smartphones'); const allRated: Record = Object.fromEntries( @@ -2686,6 +2829,8 @@ function main(): void { console.log( ' ✅ [13/13] Review value score and weight factor (DOS/Rate#1547) verified' ); + testExtensionObservation(); + console.log(' ✅ [14/14] ExtensionObservation v1 (DOS/Rate#1606) verified'); console.log('🎉 All @repo/shared-data tests passed!'); } diff --git a/packages/shared-data/src/rules/extension-observation.ts b/packages/shared-data/src/rules/extension-observation.ts new file mode 100644 index 000000000..714a5ae14 --- /dev/null +++ b/packages/shared-data/src/rules/extension-observation.ts @@ -0,0 +1,317 @@ +/** + * ExtensionObservation v1 (DOS/Rate#1606, milestone M2 of + * Docs/Proposals/2026-10-28-extension-data-source.md): the minimal price, + * stock and SKU observation an opted-in browser extension sends. It goes to + * a quarantine table and never to public prices directly. + * + * The rules here are repeated by rate.submit_extension_observation in the + * DOS-Me migration (Docs/Proposals/2026-11-04-dos-me-extension-observations.sql); + * a change to one needs the other. Plain functions only, no zod, so the + * extension and client code can import it. + */ + +export const EXTENSION_OBSERVATION_VERSION = 1; + +const EXTENSION_MARKETPLACES = [ + 'shopee', + 'lazada', + 'tiki', + 'tiktok', + 'cellphones', + 'tgdd', + 'fptshop', + 'hoangha', + 'didongviet', + 'phongvu', +] as const; + +export type ExtensionMarketplace = (typeof EXTENSION_MARKETPLACES)[number]; + +const EXTENSION_STOCK_STATES = [ + 'in_stock', + 'low_stock', + 'out_of_stock', + 'unknown', +] as const; + +export type ExtensionStockState = (typeof EXTENSION_STOCK_STATES)[number]; + +const EXTENSION_EXTRACTOR_SOURCES = ['json_ld', 'dom', 'mixed'] as const; + +/** The hosts each marketplace's listings live on; `.x` also allows subdomains. */ +const EXTENSION_MARKETPLACE_HOSTS: Readonly< + Record +> = { + shopee: ['shopee.vn'], + lazada: ['lazada.vn', '.lazada.vn'], + tiki: ['tiki.vn'], + tiktok: ['www.tiktok.com', 'shop.tiktok.com'], + cellphones: ['cellphones.com.vn'], + tgdd: ['thegioididong.com', '.thegioididong.com'], + fptshop: ['fptshop.com.vn'], + hoangha: ['hoanghamobile.com'], + didongviet: ['didongviet.vn'], + phongvu: ['phongvu.vn'], +}; + +export const EXTENSION_OBSERVATION_LIMITS = { + urlMax: 2048, + idMax: 64, + variantMax: 120, + priceMin: 1_000, + priceMax: 2_000_000_000, + stockQtyMax: 1_000_000, + extensionVersionMax: 20, + /** observed_at may run ahead of the server clock by this much. */ + futureSkewMs: 10 * 60 * 1000, + /** observed_at may be at most this old. */ + maxAgeMs: 24 * 60 * 60 * 1000, +} as const; + +export interface ExtensionObservationV1 { + schema_version: 1; + marketplace: ExtensionMarketplace; + listing_url: string; + external_item_id: string; + external_shop_id: string | null; + sku_id: string | null; + variant_label: string | null; + price_vnd: number; + list_price_vnd: number | null; + stock_state: ExtensionStockState; + stock_qty: number | null; + observed_at: string; + extractor: { + source: (typeof EXTENSION_EXTRACTOR_SOURCES)[number]; + extension_version: string; + }; +} + +export type ExtensionObservationField = + | keyof ExtensionObservationV1 + | 'observation' + | 'extractor.source' + | 'extractor.extension_version'; + +export type ExtensionObservationResult = + | { ok: true; value: ExtensionObservationV1 } + | { ok: false; field: ExtensionObservationField }; + +const KEYS: readonly (keyof ExtensionObservationV1)[] = [ + 'schema_version', + 'marketplace', + 'listing_url', + 'external_item_id', + 'external_shop_id', + 'sku_id', + 'variant_label', + 'price_vnd', + 'list_price_vnd', + 'stock_state', + 'stock_qty', + 'observed_at', + 'extractor', +]; + +const ID_RE = /^[A-Za-z0-9_.-]+$/; +const VERSION_RE = /^[0-9A-Za-z.+-]+$/; +// eslint-disable-next-line no-control-regex +const CONTROL_RE = /[\u0000-\u001f\u007f-\u009f]/; + +const isRecord = (v: unknown): v is Record => + typeof v === 'object' && v !== null && !Array.isArray(v); + +const has = (list: readonly T[], v: unknown): v is T => + typeof v === 'string' && (list as readonly string[]).includes(v); + +function idOk(v: unknown): v is string { + return ( + typeof v === 'string' && + v.length >= 1 && + v.length <= EXTENSION_OBSERVATION_LIMITS.idMax && + ID_RE.test(v) + ); +} + +const nullableId = (v: unknown) => v === null || idOk(v); + +const priceOk = (v: unknown): v is number => + typeof v === 'number' && + Number.isInteger(v) && + v >= EXTENSION_OBSERVATION_LIMITS.priceMin && + v <= EXTENSION_OBSERVATION_LIMITS.priceMax; + +/** True when `host` is one of the marketplace's listing hosts. */ +export function isExtensionMarketplaceHost( + marketplace: ExtensionMarketplace, + host: string +): boolean { + const h = host.toLowerCase(); + return EXTENSION_MARKETPLACE_HOSTS[marketplace].some((allowed) => { + if (!allowed.startsWith('.')) return h === allowed; + // A subdomain: non-empty labels of [a-z0-9-] before the base domain, + // as the RPC's ^([a-z0-9-]+\.)*base$ requires. No pattern is built + // from data. + if (h === allowed.slice(1)) return true; + if (!h.endsWith(allowed)) return false; + return /^(?:[a-z0-9-]+\.)+$/.test( + h.slice(0, h.length - allowed.length + 1) + ); + }); +} + +/** Code points, as the database's char_length counts them. */ +const codePoints = (v: string) => Array.from(v).length; + +/** An unpaired UTF-16 surrogate, which Postgres jsonb cannot store. */ +const LONE_SURROGATE_RE = + /[\uD800-\uDBFF](?![\uDC00-\uDFFF])|(? EXTENSION_OBSERVATION_LIMITS.urlMax || + URL_FORBIDDEN_RE.test(v) || + LONE_SURROGATE_RE.test(v) + ) { + return false; + } + const host = /^https:\/\/([^/]*)/.exec(v)?.[1]; + return ( + typeof host === 'string' && + /^[a-z0-9.-]+$/.test(host) && + isExtensionMarketplaceHost(marketplace, host) + ); +} + +/** ISO 8601 with a zone, as the RPC requires it. */ +// Seconds stop at 59 and offsets at ±15:59, the range Postgres accepts. +const OBSERVED_AT_RE = + /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(?::[0-5]\d(?:\.\d{1,9})?)?(?:Z|[+-](?:0\d|1[0-5])(?::?[0-5]\d)?)$/; + +function observedAtMs(v: unknown): number { + if (typeof v !== 'string' || !OBSERVED_AT_RE.test(v)) return Number.NaN; + // Date.parse wants the offset as ±hh:mm. + const normalized = v + .replace(/([+-]\d{2})$/, '$1:00') + .replace(/([+-]\d{2})(\d{2})$/, '$1:$2'); + return Date.parse(normalized); +} + +/** + * Checks an observation against v1 at server time `nowMs`. It refuses what + * the RPC refuses; the field it names is the first rule broken in this order, + * which can differ from the RPC's when several rules break (an unknown key is + * `observation` here and its own name there). + */ +export function parseExtensionObservationV1( + raw: unknown, + nowMs: number = Date.now() +): ExtensionObservationResult { + const fail = (field: ExtensionObservationField) => + ({ ok: false, field }) as const; + if (!isRecord(raw)) return fail('observation'); + for (const key of Object.keys(raw)) { + if (!(KEYS as readonly string[]).includes(key)) return fail('observation'); + } + if (raw.schema_version !== EXTENSION_OBSERVATION_VERSION) { + return fail('schema_version'); + } + const marketplace = raw.marketplace; + if (!has(EXTENSION_MARKETPLACES, marketplace)) return fail('marketplace'); + if (!listingUrlOk(raw.listing_url, marketplace)) return fail('listing_url'); + if (!idOk(raw.external_item_id)) return fail('external_item_id'); + if (!nullableId(raw.external_shop_id)) return fail('external_shop_id'); + if (!nullableId(raw.sku_id)) return fail('sku_id'); + const variant = raw.variant_label; + if ( + variant !== null && + (typeof variant !== 'string' || + codePoints(variant) < 1 || + LONE_SURROGATE_RE.test(variant) || + codePoints(variant) > EXTENSION_OBSERVATION_LIMITS.variantMax || + CONTROL_RE.test(variant)) + ) { + return fail('variant_label'); + } + if (!priceOk(raw.price_vnd)) return fail('price_vnd'); + const listPrice = raw.list_price_vnd; + if ( + listPrice !== null && + (!priceOk(listPrice) || listPrice < raw.price_vnd) + ) { + return fail('list_price_vnd'); + } + if (!has(EXTENSION_STOCK_STATES, raw.stock_state)) return fail('stock_state'); + const qty = raw.stock_qty; + if ( + qty !== null && + (typeof qty !== 'number' || + !Number.isInteger(qty) || + qty < 0 || + qty > EXTENSION_OBSERVATION_LIMITS.stockQtyMax) + ) { + return fail('stock_qty'); + } + const observedMs = observedAtMs(raw.observed_at); + if ( + !Number.isFinite(observedMs) || + observedMs > nowMs + EXTENSION_OBSERVATION_LIMITS.futureSkewMs || + observedMs < nowMs - EXTENSION_OBSERVATION_LIMITS.maxAgeMs + ) { + return fail('observed_at'); + } + const extractor = raw.extractor; + if (!isRecord(extractor)) return fail('extractor'); + for (const key of Object.keys(extractor)) { + if (key !== 'source' && key !== 'extension_version') + return fail('extractor'); + } + if (!has(EXTENSION_EXTRACTOR_SOURCES, extractor.source)) { + return fail('extractor.source'); + } + const version = extractor.extension_version; + if ( + typeof version !== 'string' || + version.length < 1 || + version.length > EXTENSION_OBSERVATION_LIMITS.extensionVersionMax || + !VERSION_RE.test(version) + ) { + return fail('extractor.extension_version'); + } + return { + ok: true, + value: { + schema_version: 1, + marketplace, + listing_url: raw.listing_url as string, + external_item_id: raw.external_item_id as string, + external_shop_id: raw.external_shop_id as string | null, + sku_id: raw.sku_id as string | null, + variant_label: variant as string | null, + price_vnd: raw.price_vnd as number, + list_price_vnd: listPrice as number | null, + stock_state: raw.stock_state as ExtensionStockState, + stock_qty: qty as number | null, + observed_at: raw.observed_at as string, + extractor: { + source: extractor.source, + extension_version: version, + }, + }, + }; +} + +// Exported here: an `export const` holding strings would read as an import +// to the rules bundle check in shared-data.test.ts. +export { + EXTENSION_MARKETPLACES, + EXTENSION_STOCK_STATES, + EXTENSION_EXTRACTOR_SOURCES, + EXTENSION_MARKETPLACE_HOSTS, +}; diff --git a/packages/shared-data/src/rules/index.ts b/packages/shared-data/src/rules/index.ts index b99154c9e..909c2a3a7 100644 --- a/packages/shared-data/src/rules/index.ts +++ b/packages/shared-data/src/rules/index.ts @@ -16,3 +16,4 @@ export * from './review-form.js'; export * from './product-points.js'; export * from './review-depth.js'; export * from './review-value.js'; +export * from './extension-observation.js'; From 59676e6bd90c5d970ee96adeafe6467f5effd840 Mon Sep 17 00:00:00 2001 From: JOY <5027251+JOY@users.noreply.github.com> Date: Wed, 4 Nov 2026 14:26:33 +0700 Subject: [PATCH 4/7] docs(proposals): dos-me extension observation quarantine (m2) (#1608) * docs(proposals): dos-me extension observation quarantine (m2) Draft DOS-Me migration for ExtensionObservation v1 (DOS/Rate#1606). Nothing is applied. - rate.extension_submitters and rate.extension_observations, RLS on, no anon or authenticated table access, rate_worker reads and updates status only. - register, submit, delete and purge RPCs, security definer with an empty search_path, with the contract's SQLSTATEs and DETAILs, limits and 28/180 day retention. - Verify and rollback files, and a Docker replay that exercises every RPC, the grants, the rollback and a re-apply. Refs #1606 * docs(proposals): extension observations revision 2, ingest key and limits Applies Revision 2 of the M2 contract (DOS/Rate#1606). Nothing is applied. - rate.extension_ingest_secret: register and submit check the sha256 of an ingest key (42501, DETAIL ingest_key); the network hash is required in both. - Register: 20 an hour per network hash, then the global brake under an advisory lock; submitters keep their network hash. - Purge: empty submitters after 7 days, locked first and deleted under a fresh check. - DETAIL observation for an unknown key; presence checked per field in the validator order; semicolons and JavaScript spaces refused in listing_url. - Verify: owner, no forced RLS, column list, limits in the bodies, no grants on the key. - Replay: parity with parseExtensionObservationV1, exact thresholds, ingest key, the purge race; wired into CI after the shared-data build. Refs #1606 * fix(proposals): extension observations purge lock order and key checks Re-review fixes for the M2 quarantine proposal (DOS/Rate#1606). Nothing is applied. - Purge clears submitter hashes before observation hashes, the lock order of delete, so a delete during a purge waits instead of deadlocking. The replay stalls a purge, runs a delete in the middle, and fails with the old order. - The key row cannot hold the sha256 of the empty string, and register and submit refuse a missing key or one under 32 characters (42501, DETAIL ingest_key). - observed_at: seconds 00 to 59 and offsets up to 15:59, as Postgres parses them. - The .md runbook is written for the applying session: CSPRNG key, printf hash, Vercel secrets, apply, row, verify, then the flag; rotation is a hard cut. Refs #1606 * Merge remote-tracking branch 'origin/dev' into docs/1606-dos-me-extension-observations Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 3 + .gitignore | 3 + ...dos-me-extension-observations-rollback.sql | 22 + ...4-dos-me-extension-observations-verify.sql | 263 ++++ ...026-11-04-dos-me-extension-observations.md | 224 ++++ ...26-11-04-dos-me-extension-observations.sql | 659 +++++++++ .../test-extension-observations-proposal.mjs | 1177 +++++++++++++++++ 7 files changed, 2351 insertions(+) create mode 100644 Docs/Proposals/2026-11-04-dos-me-extension-observations-rollback.sql create mode 100644 Docs/Proposals/2026-11-04-dos-me-extension-observations-verify.sql create mode 100644 Docs/Proposals/2026-11-04-dos-me-extension-observations.md create mode 100644 Docs/Proposals/2026-11-04-dos-me-extension-observations.sql create mode 100644 scripts/test-extension-observations-proposal.mjs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0b4f6967c..d67dd412a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -88,6 +88,9 @@ jobs: - name: Replay the DOS-Me review AI points proposal (Docker) run: node scripts/test-review-ai-points-proposal.mjs + - name: Replay the DOS-Me extension observations proposal (Docker) + run: pnpm --filter @repo/shared-data build && node scripts/test-extension-observations-proposal.mjs + - name: Check generated rate schema types match the migrations run: pnpm check:db-types diff --git a/.gitignore b/.gitignore index be909f563..3e5f451ed 100644 --- a/.gitignore +++ b/.gitignore @@ -164,6 +164,9 @@ playwright-report-item-model/ !Docs/Proposals/2026-11-03-dos-me-review-ai-points.sql !Docs/Proposals/2026-11-03-dos-me-review-ai-points-rollback.sql !Docs/Proposals/2026-11-03-dos-me-review-ai-points-verify.sql +!Docs/Proposals/2026-11-04-dos-me-extension-observations.sql +!Docs/Proposals/2026-11-04-dos-me-extension-observations-rollback.sql +!Docs/Proposals/2026-11-04-dos-me-extension-observations-verify.sql * - Copy* * - Bản sao* *.dump diff --git a/Docs/Proposals/2026-11-04-dos-me-extension-observations-rollback.sql b/Docs/Proposals/2026-11-04-dos-me-extension-observations-rollback.sql new file mode 100644 index 000000000..57820b2ca --- /dev/null +++ b/Docs/Proposals/2026-11-04-dos-me-extension-observations-rollback.sql @@ -0,0 +1,22 @@ +-- ROLLBACK of the PROPOSAL 2026-11-04-dos-me-extension-observations.sql (DOS/Rate#1606). +-- One transaction: drops the four functions, then rate.extension_observations, +-- rate.extension_submitters and rate.extension_ingest_secret with their policies and indexes. +-- Every quarantined observation and submitter id is lost, and so is the stored ingest key hash +-- (set it again after a re-apply). Turn RATE_EXTENSION_OBSERVATIONS off on the web first; the +-- routes would otherwise answer errors. Nothing else reads these tables. +-- Safe to run twice. + +BEGIN; + +SET LOCAL lock_timeout = '5s'; + +DROP FUNCTION IF EXISTS rate.purge_extension_observations(integer, integer, integer); +DROP FUNCTION IF EXISTS rate.delete_extension_submitter(uuid); +DROP FUNCTION IF EXISTS rate.submit_extension_observation(text, uuid, jsonb, text); +DROP FUNCTION IF EXISTS rate.register_extension_submitter(text, text); + +DROP TABLE IF EXISTS rate.extension_observations; +DROP TABLE IF EXISTS rate.extension_submitters; +DROP TABLE IF EXISTS rate.extension_ingest_secret; + +COMMIT; diff --git a/Docs/Proposals/2026-11-04-dos-me-extension-observations-verify.sql b/Docs/Proposals/2026-11-04-dos-me-extension-observations-verify.sql new file mode 100644 index 000000000..ee762c574 --- /dev/null +++ b/Docs/Proposals/2026-11-04-dos-me-extension-observations-verify.sql @@ -0,0 +1,263 @@ +-- VERIFY the PROPOSAL 2026-11-04-dos-me-extension-observations.sql (DOS/Rate#1606). Read-only. +-- Run after the migration and after the ingest key row is set, on preview, beta and production, +-- as postgres or service_role. V1 to V12 return 0. V13 is information: rows the purge should +-- already have handled. + +-- V1 the three tables exist with row level security on and not forced (the functions' owner +-- must pass it); expect 0 +SELECT 3 - count(*) FROM pg_class +WHERE oid IN (to_regclass('rate.extension_ingest_secret'), to_regclass('rate.extension_submitters'), + to_regclass('rate.extension_observations')) + AND relrowsecurity AND NOT relforcerowsecurity; + +-- V2 anon and authenticated hold no privilege on the three tables or the identity sequence, +-- and PUBLIC holds none either; expect 0 +SELECT count(*) FROM ( + SELECT r.role, t.tbl, p.priv + FROM unnest(ARRAY['anon', 'authenticated']) AS r(role) + CROSS JOIN unnest(ARRAY['rate.extension_ingest_secret', 'rate.extension_submitters', + 'rate.extension_observations']) AS t(tbl) + CROSS JOIN unnest(ARRAY['SELECT', 'INSERT', 'UPDATE', 'DELETE', 'TRUNCATE', 'REFERENCES', + 'TRIGGER']) AS p(priv) + WHERE has_table_privilege(r.role, t.tbl, p.priv) + OR has_any_column_privilege(r.role, t.tbl, 'SELECT') + OR has_any_column_privilege(r.role, t.tbl, 'INSERT') + OR has_any_column_privilege(r.role, t.tbl, 'UPDATE') + UNION ALL + SELECT r.role, 'sequence', p.priv + FROM unnest(ARRAY['anon', 'authenticated']) AS r(role) + CROSS JOIN unnest(ARRAY['USAGE', 'SELECT', 'UPDATE']) AS p(priv) + WHERE has_sequence_privilege(r.role, + pg_get_serial_sequence('rate.extension_observations', 'id'), p.priv) + UNION ALL + SELECT 'PUBLIC', c.relname::text, a.privilege_type + FROM pg_class c + CROSS JOIN LATERAL aclexplode(c.relacl) a + WHERE c.oid IN (to_regclass('rate.extension_ingest_secret'), + to_regclass('rate.extension_submitters'), + to_regclass('rate.extension_observations')) + AND a.grantee = 0 +) x; + +-- V3 service_role has full access; rate_worker reads submitters and observations, updates only +-- extension_observations.status, cannot insert, delete or truncate, and has nothing on the +-- ingest key; expect 0 +SELECT count(*) FROM ( + SELECT t.tbl, p.priv FROM unnest(ARRAY['rate.extension_ingest_secret', + 'rate.extension_submitters', + 'rate.extension_observations']) AS t(tbl) + CROSS JOIN unnest(ARRAY['SELECT', 'INSERT', 'UPDATE', 'DELETE']) AS p(priv) + WHERE NOT has_table_privilege('service_role', t.tbl, p.priv) + UNION ALL + SELECT t.tbl, 'rate_worker select' FROM unnest(ARRAY['rate.extension_submitters', + 'rate.extension_observations']) AS t(tbl) + WHERE NOT has_table_privilege('rate_worker', t.tbl, 'SELECT') + OR has_table_privilege('rate_worker', t.tbl, 'INSERT') + OR has_table_privilege('rate_worker', t.tbl, 'UPDATE') + OR has_table_privilege('rate_worker', t.tbl, 'DELETE') + OR has_table_privilege('rate_worker', t.tbl, 'TRUNCATE') + OR has_any_column_privilege('rate_worker', t.tbl, 'INSERT') + UNION ALL + SELECT 'rate.extension_submitters', 'rate_worker column update' + WHERE has_any_column_privilege('rate_worker', 'rate.extension_submitters', 'UPDATE') + UNION ALL + SELECT 'rate.extension_ingest_secret', 'rate_worker' + WHERE has_any_column_privilege('rate_worker', 'rate.extension_ingest_secret', 'SELECT') + OR has_any_column_privilege('rate_worker', 'rate.extension_ingest_secret', 'INSERT') + OR has_any_column_privilege('rate_worker', 'rate.extension_ingest_secret', 'UPDATE') + OR has_table_privilege('rate_worker', 'rate.extension_ingest_secret', 'DELETE') + OR has_table_privilege('rate_worker', 'rate.extension_ingest_secret', 'TRUNCATE') + UNION ALL + SELECT 'rate.extension_observations', a.attname::text + FROM pg_attribute a + WHERE a.attrelid = to_regclass('rate.extension_observations') AND a.attnum > 0 + AND NOT a.attisdropped + AND has_column_privilege('rate_worker', a.attrelid, a.attnum, 'UPDATE') <> (a.attname = 'status') +) x; + +-- V4 the six policies exist, and none applies to PUBLIC, anon or authenticated; expect 0 +SELECT (6 - ( + SELECT count(*) FROM pg_policies + WHERE schemaname = 'rate' AND (tablename, policyname, cmd, roles) IN ( + ('extension_ingest_secret', 'Service role access extension_ingest_secret', 'ALL', + '{service_role}'::name[]), + ('extension_submitters', 'Service role access extension_submitters', 'ALL', + '{service_role}'::name[]), + ('extension_submitters', 'Rate worker read extension_submitters', 'SELECT', + '{rate_worker}'::name[]), + ('extension_observations', 'Service role access extension_observations', 'ALL', + '{service_role}'::name[]), + ('extension_observations', 'Rate worker read extension_observations', 'SELECT', + '{rate_worker}'::name[]), + ('extension_observations', 'Rate worker status extension_observations', 'UPDATE', + '{rate_worker}'::name[])) +)) + ( + SELECT count(*) FROM pg_policies + WHERE schemaname = 'rate' + AND tablename IN ('extension_ingest_secret', 'extension_submitters', 'extension_observations') + AND roles && '{public,anon,authenticated}'::name[] +); + +-- V5 the four functions are SECURITY DEFINER with an empty search_path, and owned by the +-- tables' owner (who passes their row level security); expect 0 +SELECT 4 - count(*) FROM pg_proc p +WHERE p.oid IN ( + to_regprocedure('rate.register_extension_submitter(text, text)'), + to_regprocedure('rate.submit_extension_observation(text, uuid, jsonb, text)'), + to_regprocedure('rate.delete_extension_submitter(uuid)'), + to_regprocedure('rate.purge_extension_observations(integer, integer, integer)')) + AND p.prosecdef AND p.proconfig @> ARRAY['search_path=""'] + AND p.proowner = (SELECT relowner FROM pg_class + WHERE oid = to_regclass('rate.extension_observations')) + AND p.proowner = ALL (SELECT relowner FROM pg_class + WHERE oid IN (to_regclass('rate.extension_ingest_secret'), + to_regclass('rate.extension_submitters'))); + +-- V6 EXECUTE: PUBLIC has none; anon, authenticated and service_role call the first three; +-- only service_role calls the purge; rate_worker calls none; expect 0 +SELECT count(*) FROM ( + SELECT f.sig, a.privilege_type + FROM unnest(ARRAY['rate.register_extension_submitter(text, text)', + 'rate.submit_extension_observation(text, uuid, jsonb, text)', + 'rate.delete_extension_submitter(uuid)', + 'rate.purge_extension_observations(integer, integer, integer)']) AS f(sig) + JOIN pg_proc p ON p.oid = to_regprocedure(f.sig) + CROSS JOIN LATERAL aclexplode(COALESCE(p.proacl, acldefault('f', p.proowner))) a + WHERE a.grantee = 0 + UNION ALL + SELECT f.sig, r.role + FROM unnest(ARRAY['rate.register_extension_submitter(text, text)', + 'rate.submit_extension_observation(text, uuid, jsonb, text)', + 'rate.delete_extension_submitter(uuid)']) AS f(sig) + CROSS JOIN unnest(ARRAY['anon', 'authenticated', 'service_role']) AS r(role) + WHERE to_regprocedure(f.sig) IS NULL OR NOT has_function_privilege(r.role, f.sig, 'EXECUTE') + UNION ALL + SELECT 'purge', r.role FROM unnest(ARRAY['anon', 'authenticated']) AS r(role) + WHERE has_function_privilege(r.role, + 'rate.purge_extension_observations(integer, integer, integer)', 'EXECUTE') + UNION ALL + SELECT 'purge', 'service_role' + WHERE NOT has_function_privilege('service_role', + 'rate.purge_extension_observations(integer, integer, integer)', 'EXECUTE') + UNION ALL + SELECT f.sig, 'rate_worker' + FROM unnest(ARRAY['rate.register_extension_submitter(text, text)', + 'rate.submit_extension_observation(text, uuid, jsonb, text)', + 'rate.delete_extension_submitter(uuid)', + 'rate.purge_extension_observations(integer, integer, integer)']) AS f(sig) + WHERE has_function_privilege('rate_worker', f.sig, 'EXECUTE') +) x; + +-- V7 the CHECKs exist and are validated, and the foreign key cascades on delete; expect 0 +SELECT (20 - ( + SELECT count(*) FROM pg_constraint + WHERE contype = 'c' AND convalidated AND ( + (conrelid = to_regclass('rate.extension_observations') AND conname IN ( + 'extension_observations_schema_version_check', 'extension_observations_marketplace_check', + 'extension_observations_listing_url_check', 'extension_observations_external_item_id_check', + 'extension_observations_external_shop_id_check', 'extension_observations_sku_id_check', + 'extension_observations_variant_label_check', 'extension_observations_price_vnd_check', + 'extension_observations_list_price_vnd_check', 'extension_observations_stock_state_check', + 'extension_observations_stock_qty_check', 'extension_observations_observed_at_check', + 'extension_observations_extractor_check', 'extension_observations_network_hash_check', + 'extension_observations_status_check')) + OR (conrelid = to_regclass('rate.extension_submitters') + AND conname = 'extension_submitters_network_hash_check') + OR (conrelid = to_regclass('rate.extension_ingest_secret') + AND conname IN ('extension_ingest_secret_id_check', 'extension_ingest_secret_key_not_empty_check', + 'extension_ingest_secret_key_sha256_check'))) +)) - ( + SELECT count(*) FROM pg_constraint + WHERE conrelid = to_regclass('rate.extension_observations') AND contype = 'f' + AND confrelid = to_regclass('rate.extension_submitters') AND confdeltype = 'c' +); + +-- V8 the five indexes exist; expect 0 +SELECT 5 - count(*) FROM pg_indexes +WHERE schemaname = 'rate' AND indexname IN ( + 'extension_submitters_created_idx', 'extension_submitters_network_created_idx', + 'extension_observations_submitter_received_idx', 'extension_observations_network_received_idx', + 'extension_observations_item_received_idx'); + +-- V9 every column has the type, nullability and default of the proposal, and no other column +-- exists; expect 0 +WITH expected (tbl, col, typ, not_null, def) AS (VALUES + ('extension_ingest_secret', 'id', 'boolean', true, 'true'), + ('extension_ingest_secret', 'key_sha256', 'text', true, NULL), + ('extension_ingest_secret', 'updated_at', 'timestamp with time zone', true, 'now()'), + ('extension_submitters', 'id', 'uuid', true, 'gen_random_uuid()'), + ('extension_submitters', 'created_at', 'timestamp with time zone', true, 'now()'), + ('extension_submitters', 'network_hash', 'text', false, NULL), + ('extension_observations', 'id', 'bigint', true, 'identity always'), + ('extension_observations', 'submitter_id', 'uuid', true, NULL), + ('extension_observations', 'schema_version', 'smallint', true, NULL), + ('extension_observations', 'marketplace', 'text', true, NULL), + ('extension_observations', 'listing_url', 'text', true, NULL), + ('extension_observations', 'external_item_id', 'text', true, NULL), + ('extension_observations', 'external_shop_id', 'text', false, NULL), + ('extension_observations', 'sku_id', 'text', false, NULL), + ('extension_observations', 'variant_label', 'text', false, NULL), + ('extension_observations', 'price_vnd', 'bigint', true, NULL), + ('extension_observations', 'list_price_vnd', 'bigint', false, NULL), + ('extension_observations', 'stock_state', 'text', true, NULL), + ('extension_observations', 'stock_qty', 'integer', false, NULL), + ('extension_observations', 'observed_at', 'timestamp with time zone', true, NULL), + ('extension_observations', 'extractor', 'jsonb', true, NULL), + ('extension_observations', 'received_at', 'timestamp with time zone', true, 'now()'), + ('extension_observations', 'network_hash', 'text', false, NULL), + ('extension_observations', 'status', 'text', true, '''quarantined''::text') +), actual AS ( + SELECT c.relname::text AS tbl, a.attname::text AS col, + format_type(a.atttypid, a.atttypmod) AS typ, a.attnotnull AS not_null, + COALESCE(pg_get_expr(d.adbin, d.adrelid), + CASE a.attidentity WHEN 'a' THEN 'identity always' + WHEN 'd' THEN 'identity by default' END) AS def + FROM pg_attribute a + JOIN pg_class c ON c.oid = a.attrelid + LEFT JOIN pg_attrdef d ON d.adrelid = a.attrelid AND d.adnum = a.attnum + WHERE c.oid IN (to_regclass('rate.extension_ingest_secret'), + to_regclass('rate.extension_submitters'), + to_regclass('rate.extension_observations')) + AND a.attnum > 0 AND NOT a.attisdropped +) +SELECT count(*) FROM ( + (SELECT * FROM expected EXCEPT SELECT * FROM actual) + UNION ALL + (SELECT * FROM actual EXCEPT SELECT * FROM expected) +) x; + +-- V10 the limits are in the function bodies: 120 and 600 an hour and the 16 KB cap in submit, +-- 20 an hour and 600 a minute in register; expect 0 +SELECT count(*) FROM (VALUES + ('rate.submit_extension_observation(text, uuid, jsonb, text)', '%) >= 120 THEN%'), + ('rate.submit_extension_observation(text, uuid, jsonb, text)', '%) >= 600 THEN%'), + ('rate.submit_extension_observation(text, uuid, jsonb, text)', '%octet_length(o::text) > 16384%'), + ('rate.register_extension_submitter(text, text)', '%) >= 20 THEN%'), + ('rate.register_extension_submitter(text, text)', '%) >= 600 THEN%') +) AS t(sig, pattern) +WHERE NOT EXISTS (SELECT 1 FROM pg_proc p + WHERE p.oid = to_regprocedure(t.sig) AND p.prosrc LIKE t.pattern); + +-- V11 the ingest key table is granted to nobody but its owner and service_role; expect 0 +SELECT count(*) FROM pg_class c +CROSS JOIN LATERAL aclexplode(COALESCE(c.relacl, acldefault('r', c.relowner))) a +WHERE c.oid = to_regclass('rate.extension_ingest_secret') + AND a.grantee <> c.relowner + AND a.grantee <> (SELECT oid FROM pg_roles WHERE rolname = 'service_role'); + +-- V12 the ingest key row is set (without it, register and submit refuse every caller); expect 0 +SELECT 1 - count(*) FROM rate.extension_ingest_secret WHERE id; + +-- V13 information: rows past the default retention (non-zero means the daily purge is not +-- running) +SELECT + (SELECT count(*) FROM rate.extension_observations + WHERE network_hash IS NOT NULL AND received_at < now() - interval '29 days') + + (SELECT count(*) FROM rate.extension_submitters + WHERE network_hash IS NOT NULL AND created_at < now() - interval '29 days') AS hashes_due, + (SELECT count(*) FROM rate.extension_observations + WHERE received_at < now() - interval '181 days') AS observations_due, + (SELECT count(*) FROM rate.extension_submitters s + WHERE s.created_at < now() - interval '8 days' + AND NOT EXISTS (SELECT 1 FROM rate.extension_observations o + WHERE o.submitter_id = s.id)) AS empty_submitters_due; diff --git a/Docs/Proposals/2026-11-04-dos-me-extension-observations.md b/Docs/Proposals/2026-11-04-dos-me-extension-observations.md new file mode 100644 index 000000000..56fe41dc2 --- /dev/null +++ b/Docs/Proposals/2026-11-04-dos-me-extension-observations.md @@ -0,0 +1,224 @@ +# DOS-Me proposal: extension observation quarantine (M2) + +DOS/Rate#1606, milestone M2 of `Docs/Proposals/2026-10-28-extension-data-source.md`, with the +contract's Revision 2 (after four reviews). Status: **proposal, nothing applied.** It goes +through DOS-Me after JOY approves it. + +| File | What | +| ------------------------------------------------------- | --------------------------------------------------------- | +| `2026-11-04-dos-me-extension-observations.sql` | the migration, one transaction | +| `2026-11-04-dos-me-extension-observations-verify.sql` | read-only checks, V1 to V12 return 0 (V13 is information) | +| `2026-11-04-dos-me-extension-observations-rollback.sql` | drops the functions and the three tables | +| `scripts/test-extension-observations-proposal.mjs` | replays it all on PostgreSQL in Docker | + +## What it does + +The extension sends what a shopper's browser shows on a listing: price, list price, stock and +variant. This migration stores those observations in a quarantine. Nothing reads them into +prices, products or ratings. Corroboration and promotion are M3/M4, in DOS/Rate-Services. + +- **`rate.extension_ingest_secret`** (new): one private row, the sha256 of the web's secret + `RATE_EXTENSION_INGEST_KEY`. + - The migration inserts no row. DOS-Me sets it at apply time. The key itself is never in git. + - Only `service_role` has access. +- **`rate.extension_submitters`** (new): one row per install that opted in. It holds a random + id, its creation time and the network hash it registered from. +- **`rate.extension_observations`** (new): the quarantine. + - One column per key of the `ExtensionObservation v1` payload, each with a CHECK that repeats + the payload rule. + - `received_at`, set by the server. The server trusts it more than `observed_at`. + - `network_hash`: the web's HMAC-SHA256 of the sender's /24 or /48. + - `status`: `quarantined`, `corroborated` or `rejected`. + - It cascades from its submitter. +- **Access.** + - Row level security is on for the three tables, and not forced. + - `anon` and `authenticated` have no table privilege; they use the RPCs only. + - `service_role` has full access. + - `rate_worker` reads submitters and observations and may update `status` only (a column + grant). It has nothing on the ingest key. +- **RPCs.** All four are `SECURITY DEFINER` with `search_path = ''`, owned by the tables' owner. + EXECUTE is revoked from PUBLIC. + 1. `rate.register_extension_submitter(p_ingest_key, p_network_hash)`, for anon, + authenticated and service_role. + - A key that is missing, shorter than 32 characters, or whose sha256 is not the stored row + raises 42501, DETAIL `ingest_key`. Without a row, every call is refused. The row can + never hold the sha256 of the empty string (a CHECK). + - A network hash that is not 64 lowercase hex characters raises 22023, DETAIL + `network_hash`. + - Over 20 new submitters per network hash in the last hour raises 54000, DETAIL + `network_hash`. + - Over 600 new submitters in the last minute, in all, raises 54000, DETAIL `submitter`. + - Each count runs under an advisory lock, so it is exact. + 2. `rate.submit_extension_observation(...)`, for anon, authenticated and service_role. Its + arguments: `p_ingest_key`, `p_submitter_id`, `p_observation`, `p_network_hash`. + - The ingest key and the network hash are checked as in register. + - Then every payload rule, in the order of the shared validator. A refusal raises 22023, + and DETAIL names the field: + - `observation` for a payload that is not an object, is over 16 KB, or has another key. + A caller's key name is never echoed. + - then each key in turn, its presence checked with its rule; a nullable key must be + present as null; + - then `extractor.source` and `extractor.extension_version`. + - An unknown submitter raises P0002 with DETAIL `submitter_id`. + - Over 120 observations per submitter, or 600 per network hash, in the last hour, it + raises 54000. DETAIL is `submitter_id` or `network_hash`. + - It returns `{"id", "status": "quarantined", "received_at"}`. + 3. `rate.delete_extension_submitter(p_submitter_id)`, for anon, authenticated and + service_role. + - It takes no ingest key, so erasure always works. + - It deletes the submitter, and its observations by cascade. + - It returns how many observations went. An unknown id returns 0. + 4. `rate.purge_extension_observations(...)`, for service_role only. Its arguments: + `p_network_hash_days` (28), `p_keep_days` (180), `p_empty_submitter_days` (7). + - It clears `network_hash` on older submitters, then on older observations. That order + matches delete (the submitter, then its observations), so a delete during a purge waits + instead of deadlocking. + - It deletes older observations. + - It deletes submitters without observations that are older than + `p_empty_submitter_days`. + - It locks them first, skipping any a submit holds, then deletes them under a fresh + check. + - So an observation committed during the purge is never cascaded away. + - It returns the four counts. + +Marketplace ids are the platform ids of the extension and the crawl: `shopee`, `lazada`, `tiki`, +`tiktok`, `cellphones`, `tgdd`, `fptshop`, `hoangha`, `didongviet`, `phongvu`. The host table: + +| marketplace | hosts | +| ----------- | -------------------------------- | +| shopee | shopee.vn | +| lazada | lazada.vn and subdomains | +| tiki | tiki.vn | +| tiktok | www.tiktok.com, shop.tiktok.com | +| cellphones | cellphones.com.vn | +| tgdd | thegioididong.com and subdomains | +| fptshop | fptshop.com.vn | +| hoangha | hoanghamobile.com | +| didongviet | didongviet.vn | +| phongvu | phongvu.vn | + +A listing URL is https, at most 2048 characters. It has no `?`, `#` or `;`, no space and no +control character. Its host is lowercase, with no user and no port. + +## The ingest key + +- Without the key, a caller holding only the public anon key can no longer call register or + submit. So nobody can bypass the web: its flag, its validator, its salt and the network + limit. +- The network hash is therefore always the web's. It is evidence of independence only for rows + written through the web, and with the key in place that means all rows. +- The key is a server secret of the web. It must never be committed. Its sha256 may be. +- **Rotation is a hard cut.** Once the new row is set, register and submit answer 42501 to the + old key, so the web's writes fail (503) until it redeploys with the new key. Deletes keep + working. + +## Privacy + +- **The submitter id is pseudonymous personal data.** It links every observation of one + install. It holds no account, email or device data. +- **The network hash is pseudonymous personal data too.** It is keyed by a web secret, so the + database alone cannot reverse it. It is kept for the limits and for abuse review. +- **Retention:** + - `network_hash` is cleared after 28 days, on observations and on submitters; + - observations are deleted after 180 days; + - submitters without observations are deleted after 7 days. + - All three are done by `rate.purge_extension_observations`. It still needs a daily job as + `service_role`: the DOS-Me pg_cron or an RS worker. V13 of the verify file shows rows it + should already have handled. +- **Delete on request:** the extension calls the web's `DELETE` route, which calls + `rate.delete_extension_submitter`. Everything that install sent goes at once. +- **No other personal data:** the payload refuses any other key. It has no ratings, account + data, cookies, query strings or referrers. + +## Order and dependencies + +- **Needs:** the `rate` schema, the API roles and `rate_worker` + (`20261018183000_rate_worker_role.sql`). All are in production. +- **Shared validator:** the replay compares the RPC with `parseExtensionObservationV1` in + `packages/shared-data/src/rules/extension-observation.ts`. That code PR merges first. +- **Web:** the routes behind `RATE_EXTENSION_OBSERVATIONS=1` (off by default) call RPCs 1 to 3 + with the ingest key. Turn the flag on only after the apply, the key row and the verify. +- **After this:** M3/M4 in DOS/Rate-Services read the quarantine as `rate_worker` and set + `status`. + +**Rating impact: none.** Nothing reads these tables into prices, products or ratings. + +## Decisions and limits + +1. **The limits are coarse.** + - The register brake of 600 a minute is global. A flood of registrations from many networks + blocks new installs for a minute, but not installs that already have an id. + - Per submitter, per network and in all, calls run one at a time (a row lock and advisory + locks), so the counts are exact. +2. **The submitter id is the only key to its data.** Whoever holds an id can delete it. The id + is random (122 bits) and stays in the extension's storage. +3. **The CHECKs repeat the payload rules** so that `service_role`'s direct writes keep to them. + One difference: a CHECK cannot read the clock, so `observed_at` is held to `received_at` + (10 minutes ahead, 24 hours behind), not to `now()`. +4. **A change to the payload needs a migration.** The SQL repeats the rules of the shared + validator, which is plain functions. A drift fails closed, and the replay compares both + over 167 payloads. + - `observed_at` takes seconds 00 to 59 (no leap second) and an offset up to 15:59, which is + what Postgres parses. +5. **Where the RPC and the validator differ on purpose:** + - the 16 KB cap is the RPC's alone (the web caps the request body); + - a lone UTF-16 surrogate: the validator names the field, and jsonb refuses the text + itself (22P02). +6. **Two indexes are not in the contract:** `extension_submitters_created_idx` and + `extension_submitters_network_created_idx`, for the register limits and the purge. + +## Apply + +These steps are for the session or agent that applies DOS-Me. JOY runs no command. + +1. **Generate the key** with a CSPRNG, for example `openssl rand -hex 32`. Keep it out of git, + logs and chat. +2. **Hash the exact bytes** with `printf %s "$KEY" | sha256sum`. + - Never `echo`, which adds a newline. + - Never in SQL, where the key would reach the database logs. + - Check that `$KEY` is set: the sha256 of an empty key is refused by a CHECK. +3. **Set the web's secrets** on every Vercel environment that points at that database: + - `RATE_EXTENSION_INGEST_KEY`: the key; + - `RATE_NETWORK_HASH_SALT`: random, at least 16 characters. + - Preview, beta and production share the prod database, so they share one key. +4. **Apply the migration.** Copy the `.sql` into DOS-Me `supabase/migrations/` with a new + timestamp, and the verify and rollback files into `scripts/ops/`. +5. **Insert the row**, as `postgres` or `service_role`, with only the hash: + + ```sql + INSERT INTO rate.extension_ingest_secret (key_sha256) VALUES ('') + ON CONFLICT (id) DO UPDATE SET key_sha256 = EXCLUDED.key_sha256, updated_at = now(); + ``` + +6. **Run the verify file** read-only. V1 to V12 must return 0; V12 proves the row is set. +7. **Only then turn the web flag on** (`RATE_EXTENSION_OBSERVATIONS=1`). +8. Schedule the daily purge as `service_role`. + +**Rollback:** the rollback file, safe to run twice. Turn the web flag off first. Every +observation and submitter id is lost, and the key row with them. + +## Tests (local, Docker) + +`pnpm --filter @repo/shared-data build && node scripts/test-extension-observations-proposal.mjs`: + +- the proposal applied twice, and V1 to V12; V1, V2 and V11 fail on a forced RLS or a stray + grant; +- the ingest key: no row, a wrong, empty or null key, and the hash itself are refused (42501); + a 31-character key is refused even when its hash is stored; the empty key's hash cannot be + stored; the right key passes; the table holds one row; +- a required network hash in register and submit; +- 39 valid and 128 refused payloads, each with the same verdict and field from the RPC and from + `parseExtensionObservationV1`, URL and time edges included; +- an unknown or null submitter (P0002), and nothing written by a refused call; +- the limits at their exact thresholds: the 120th per submitter, the 600th per network (with + 100 older rows), the 20th register per network and the 600th register in a minute pass; the + next one is refused; +- delete: the count, the cascade, 0 for an unknown or null id, no key needed; +- purge: hashes cleared on both tables, rows and empty submitters deleted, service_role only; + a submit held open during the purge keeps its submitter and its row; and a delete in the + middle of a stalled purge waits for it instead of deadlocking (the old lock order fails this + test with "deadlock detected"); +- grants: anon and authenticated reach none of the three tables; rate_worker reads and updates + `status` only and has nothing on the key; the CHECKs hold `service_role`; +- the rollback twice, then the proposal again (refusing until the key row is set), and again + over data. diff --git a/Docs/Proposals/2026-11-04-dos-me-extension-observations.sql b/Docs/Proposals/2026-11-04-dos-me-extension-observations.sql new file mode 100644 index 000000000..7a214a38a --- /dev/null +++ b/Docs/Proposals/2026-11-04-dos-me-extension-observations.sql @@ -0,0 +1,659 @@ +-- ============================================================================= +-- PROPOSAL (DOS/Rate#1606): milestone M2 of Docs/Proposals/2026-10-28-extension-data-source.md, +-- the quarantine for ExtensionObservation v1. A DOS-Me migration to review and apply; nothing +-- here is applied by DOS/Rate. Verify: 2026-11-04-dos-me-extension-observations-verify.sql +-- (V1 to V12 return 0). Rollback: 2026-11-04-dos-me-extension-observations-rollback.sql. +-- Contract: the M2 contract with its Revision 2 (ingest key, required network hash, register +-- limits, purge race fix). +-- +-- What it does: +-- rate.extension_ingest_secret one private row: the sha256 of the web's ingest key. This file +-- inserts no row; DOS-Me sets it at apply time. Without it, or +-- with a wrong key, register and submit refuse (42501). +-- rate.extension_submitters one row per extension install that opted in: a random id, its +-- creation time and the network hash it registered from. +-- Pseudonymous personal data. +-- rate.extension_observations the quarantine: one row per observation the extension sent, +-- one column per payload key, checked like the payload, plus +-- received_at, a salted network hash and a status +-- (quarantined, corroborated or rejected). +-- All three tables: row level security on; no anon or authenticated access; service_role +-- full; rate_worker reads submitters and observations and may update status only. +-- rate.register_extension_submitter(key, h) anon with the ingest key: a new submitter +-- id; 20 an hour per network hash, 600 a minute in all (54000). +-- rate.submit_extension_observation(key, id, obs, h) anon with the ingest key: one +-- observation, validated field by field (22023 with the field in +-- DETAIL), 120 an hour per submitter and 600 an hour per network +-- hash (54000). +-- rate.delete_extension_submitter(id) anon, no key: deletes the submitter and its +-- observations; returns how many observations went (0 for an +-- unknown id). +-- rate.purge_extension_observations(d, k, e) service_role: clears network hashes after d +-- days (28), deletes observations after k days (180) and +-- submitters without observations after e days (7). +-- +-- Nothing here writes price_history, products or any public table. Promotion out of the +-- quarantine is M3/M4, in DOS/Rate-Services. +-- +-- Depends on: the rate schema, the API roles (anon, authenticated, service_role) and rate_worker +-- (20261018183000_rate_worker_role.sql); all are in production. +-- +-- Rating impact: none. No rating, price or product reads these tables. +-- +-- Idempotent: IF NOT EXISTS for the tables and indexes, CREATE OR REPLACE for functions, +-- DROP POLICY IF EXISTS before each policy, grants restated every run. +-- ============================================================================= + +BEGIN; + +-- New tables only; the FK takes a lock on its own new parent. Give up rather than queue. +SET LOCAL lock_timeout = '5s'; + +-- 1. The ingest key ---------------------------------------------------------------------------- +-- One row at most (id is always true). Only the key's sha256 is stored; the key is the web's +-- secret RATE_EXTENSION_INGEST_KEY and is never in git. DOS-Me inserts the row at apply time. + +CREATE TABLE IF NOT EXISTS rate.extension_ingest_secret ( + id boolean PRIMARY KEY DEFAULT true CHECK (id), + key_sha256 text NOT NULL CHECK (key_sha256 ~ '^[0-9a-f]{64}$'), + updated_at timestamptz NOT NULL DEFAULT now(), + -- Not the sha256 of the empty string: a runbook that hashed an unset variable would + -- otherwise store a key that an empty caller matches. + CONSTRAINT extension_ingest_secret_key_not_empty_check + CHECK (key_sha256 <> 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855') +); + +COMMENT ON TABLE rate.extension_ingest_secret IS + 'sha256 of the web''s extension ingest key (DOS/Rate#1606). register_extension_submitter and submit_extension_observation refuse a caller without the key. service_role only.'; + +-- 2. Submitters -------------------------------------------------------------------------------- + +CREATE TABLE IF NOT EXISTS rate.extension_submitters ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + created_at timestamptz NOT NULL DEFAULT now(), + network_hash text, + CONSTRAINT extension_submitters_network_hash_check CHECK (network_hash ~ '^[0-9a-f]{64}$') +); + +COMMENT ON TABLE rate.extension_submitters IS + 'One row per extension install that opted in to sending observations (DOS/Rate#1606). Pseudonymous personal data: delete on request through rate.delete_extension_submitter; network_hash cleared after 28 days; deleted 7 days after creation when it has no observations, else with its last observation.'; + +-- The global register brake and the purge read created_at; the per-network register limit +-- reads network_hash. +CREATE INDEX IF NOT EXISTS extension_submitters_created_idx + ON rate.extension_submitters (created_at); +CREATE INDEX IF NOT EXISTS extension_submitters_network_created_idx + ON rate.extension_submitters (network_hash, created_at) WHERE network_hash IS NOT NULL; + +-- 3. The quarantine ---------------------------------------------------------------------------- +-- Each CHECK repeats a payload rule of rate.submit_extension_observation, which refuses first +-- with the field's name; the CHECKs hold service_role's direct writes to the same rules. + +CREATE TABLE IF NOT EXISTS rate.extension_observations ( + id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + submitter_id uuid NOT NULL REFERENCES rate.extension_submitters(id) ON DELETE CASCADE, + schema_version smallint NOT NULL, + marketplace text NOT NULL, + listing_url text NOT NULL, + external_item_id text NOT NULL, + external_shop_id text, + sku_id text, + variant_label text, + price_vnd bigint NOT NULL, + list_price_vnd bigint, + stock_state text NOT NULL, + stock_qty integer, + observed_at timestamptz NOT NULL, + extractor jsonb NOT NULL, + received_at timestamptz NOT NULL DEFAULT now(), + network_hash text, + status text NOT NULL DEFAULT 'quarantined', + CONSTRAINT extension_observations_schema_version_check CHECK (schema_version = 1), + CONSTRAINT extension_observations_marketplace_check CHECK (marketplace IN ( + 'shopee', 'lazada', 'tiki', 'tiktok', 'cellphones', 'tgdd', 'fptshop', + 'hoangha', 'didongviet', 'phongvu')), + -- https, at most 2048 characters, no query, fragment or path parameter, no space (JavaScript's + -- \s set) or control character, and a lowercase host without user or port that belongs to the + -- marketplace (the host table). + CONSTRAINT extension_observations_listing_url_check CHECK ( + char_length(listing_url) <= 2048 + AND listing_url !~ '[?#;[:space:]\x01-\x1f\x7f-\x9f\xa0\x1680\x2000-\x200a\x2028\x2029\x202f\x205f\x3000\xfeff]' + AND CASE marketplace + WHEN 'shopee' THEN listing_url ~ '^https://shopee\.vn(/|$)' + WHEN 'lazada' THEN listing_url ~ '^https://([a-z0-9-]+\.)*lazada\.vn(/|$)' + WHEN 'tiki' THEN listing_url ~ '^https://tiki\.vn(/|$)' + WHEN 'tiktok' THEN listing_url ~ '^https://(www|shop)\.tiktok\.com(/|$)' + WHEN 'cellphones' THEN listing_url ~ '^https://cellphones\.com\.vn(/|$)' + WHEN 'tgdd' THEN listing_url ~ '^https://([a-z0-9-]+\.)*thegioididong\.com(/|$)' + WHEN 'fptshop' THEN listing_url ~ '^https://fptshop\.com\.vn(/|$)' + WHEN 'hoangha' THEN listing_url ~ '^https://hoanghamobile\.com(/|$)' + WHEN 'didongviet' THEN listing_url ~ '^https://didongviet\.vn(/|$)' + WHEN 'phongvu' THEN listing_url ~ '^https://phongvu\.vn(/|$)' + ELSE false END), + CONSTRAINT extension_observations_external_item_id_check + CHECK (external_item_id ~ '^[A-Za-z0-9_.-]{1,64}$'), + CONSTRAINT extension_observations_external_shop_id_check + CHECK (external_shop_id ~ '^[A-Za-z0-9_.-]{1,64}$'), + CONSTRAINT extension_observations_sku_id_check CHECK (sku_id ~ '^[A-Za-z0-9_.-]{1,64}$'), + CONSTRAINT extension_observations_variant_label_check CHECK ( + char_length(variant_label) BETWEEN 1 AND 120 AND variant_label !~ '[\x01-\x1f\x7f-\x9f]'), + CONSTRAINT extension_observations_price_vnd_check + CHECK (price_vnd BETWEEN 1000 AND 2000000000), + CONSTRAINT extension_observations_list_price_vnd_check + CHECK (list_price_vnd >= price_vnd AND list_price_vnd <= 2000000000), + CONSTRAINT extension_observations_stock_state_check + CHECK (stock_state IN ('in_stock', 'low_stock', 'out_of_stock', 'unknown')), + CONSTRAINT extension_observations_stock_qty_check CHECK (stock_qty BETWEEN 0 AND 1000000), + -- The RPC checks observed_at against now(); a CHECK can only hold it to received_at. + CONSTRAINT extension_observations_observed_at_check CHECK ( + observed_at <= received_at + interval '10 minutes' + AND observed_at >= received_at - interval '24 hours'), + CONSTRAINT extension_observations_extractor_check CHECK ( + jsonb_typeof(extractor) = 'object' + AND extractor - 'source' - 'extension_version' = '{}'::jsonb + AND extractor ->> 'source' IN ('json_ld', 'dom', 'mixed') + AND jsonb_typeof(extractor -> 'source') = 'string' + AND jsonb_typeof(extractor -> 'extension_version') = 'string' + AND (extractor ->> 'extension_version') ~ '^[0-9A-Za-z.+-]{1,20}$'), + CONSTRAINT extension_observations_network_hash_check CHECK (network_hash ~ '^[0-9a-f]{64}$'), + CONSTRAINT extension_observations_status_check + CHECK (status IN ('quarantined', 'corroborated', 'rejected')) +); + +COMMENT ON TABLE rate.extension_observations IS + 'Quarantine of ExtensionObservation v1 (DOS/Rate#1606). Unverified crowd data: nothing reads it into prices or ratings until M3/M4 corroborates it. network_hash is cleared after 28 days and rows are deleted after 180 days by rate.purge_extension_observations.'; +COMMENT ON COLUMN rate.extension_observations.network_hash IS + 'HMAC-SHA256 of the sender''s /24 (IPv4) or /48 (IPv6), keyed by a web secret; pseudonymous personal data, cleared after 28 days.'; + +CREATE INDEX IF NOT EXISTS extension_observations_submitter_received_idx + ON rate.extension_observations (submitter_id, received_at); +CREATE INDEX IF NOT EXISTS extension_observations_network_received_idx + ON rate.extension_observations (network_hash, received_at) WHERE network_hash IS NOT NULL; +CREATE INDEX IF NOT EXISTS extension_observations_item_received_idx + ON rate.extension_observations (marketplace, external_item_id, received_at); + +-- 4. Grants and row level security ------------------------------------------------------------- +-- anon and authenticated reach the tables only through the RPCs below. + +ALTER TABLE rate.extension_ingest_secret ENABLE ROW LEVEL SECURITY; +ALTER TABLE rate.extension_submitters ENABLE ROW LEVEL SECURITY; +ALTER TABLE rate.extension_observations ENABLE ROW LEVEL SECURITY; + +-- The ingest key: service_role only (the functions read it as their owner). +REVOKE ALL ON rate.extension_ingest_secret FROM PUBLIC, anon, authenticated, rate_worker; +GRANT ALL ON rate.extension_ingest_secret TO service_role; +DROP POLICY IF EXISTS "Service role access extension_ingest_secret" ON rate.extension_ingest_secret; +CREATE POLICY "Service role access extension_ingest_secret" ON rate.extension_ingest_secret + FOR ALL TO service_role USING (true) WITH CHECK (true); + +REVOKE ALL ON rate.extension_submitters, rate.extension_observations + FROM PUBLIC, anon, authenticated, rate_worker; +GRANT ALL ON rate.extension_submitters, rate.extension_observations TO service_role; +GRANT SELECT ON rate.extension_submitters, rate.extension_observations TO rate_worker; +GRANT UPDATE (status) ON rate.extension_observations TO rate_worker; + +-- The identity sequence: only service_role (a project's default privileges may grant it wider). +DO $$ +DECLARE + v_seq text := pg_get_serial_sequence('rate.extension_observations', 'id'); +BEGIN + EXECUTE format('REVOKE ALL ON SEQUENCE %s FROM PUBLIC, anon, authenticated, rate_worker', v_seq); + EXECUTE format('GRANT ALL ON SEQUENCE %s TO service_role', v_seq); +END $$; + +DROP POLICY IF EXISTS "Service role access extension_submitters" ON rate.extension_submitters; +CREATE POLICY "Service role access extension_submitters" ON rate.extension_submitters + FOR ALL TO service_role USING (true) WITH CHECK (true); +DROP POLICY IF EXISTS "Rate worker read extension_submitters" ON rate.extension_submitters; +CREATE POLICY "Rate worker read extension_submitters" ON rate.extension_submitters + FOR SELECT TO rate_worker USING (true); + +DROP POLICY IF EXISTS "Service role access extension_observations" ON rate.extension_observations; +CREATE POLICY "Service role access extension_observations" ON rate.extension_observations + FOR ALL TO service_role USING (true) WITH CHECK (true); +DROP POLICY IF EXISTS "Rate worker read extension_observations" ON rate.extension_observations; +CREATE POLICY "Rate worker read extension_observations" ON rate.extension_observations + FOR SELECT TO rate_worker USING (true); +-- The column grant above limits the update to status; the CHECK limits its values. +DROP POLICY IF EXISTS "Rate worker status extension_observations" ON rate.extension_observations; +CREATE POLICY "Rate worker status extension_observations" ON rate.extension_observations + FOR UPDATE TO rate_worker USING (true) WITH CHECK (true); + +-- 5. register_extension_submitter -------------------------------------------------------------- +-- The ingest key (42501, DETAIL ingest_key: its sha256 must equal the stored row, and no row +-- refuses every caller), a network hash (22023), then the limits (54000): at most 20 new +-- submitters per network hash in the last hour (DETAIL network_hash), then a global brake of +-- 600 in the last minute (DETAIL submitter). Each count runs under its advisory lock, so it is +-- exact: per network first, then the global one, always in that order. + +CREATE OR REPLACE FUNCTION rate.register_extension_submitter( + p_ingest_key text, + p_network_hash text +) +RETURNS uuid +LANGUAGE plpgsql +VOLATILE +SECURITY DEFINER +SET search_path = '' +AS $$ +DECLARE + v_id uuid; +BEGIN + -- The web's key has at least 32 characters; a shorter or missing one never reaches the hash. + IF p_ingest_key IS NULL OR char_length(p_ingest_key) < 32 THEN + RAISE EXCEPTION 'Extension ingest key refused' USING ERRCODE = '42501', + DETAIL = 'ingest_key'; + END IF; + IF (SELECT key_sha256 FROM rate.extension_ingest_secret WHERE id) IS DISTINCT FROM + encode(sha256(convert_to(coalesce(p_ingest_key, ''), 'UTF8')), 'hex') THEN + RAISE EXCEPTION 'Extension ingest key refused' USING ERRCODE = '42501', + DETAIL = 'ingest_key'; + END IF; + + IF p_network_hash IS NULL OR p_network_hash !~ '^[0-9a-f]{64}$' THEN + RAISE EXCEPTION 'Invalid network hash' USING ERRCODE = '22023', DETAIL = 'network_hash'; + END IF; + + PERFORM pg_catalog.pg_advisory_xact_lock( + pg_catalog.hashtextextended('rate.extension_submitters:' || p_network_hash, 0)); + IF (SELECT count(*) FROM rate.extension_submitters + WHERE network_hash = p_network_hash + AND created_at > now() - interval '1 hour') >= 20 THEN + RAISE EXCEPTION 'Too many extension submitters from this network, try again later' + USING ERRCODE = '54000', DETAIL = 'network_hash'; + END IF; + + PERFORM pg_catalog.pg_advisory_xact_lock( + pg_catalog.hashtextextended('rate.extension_submitters:register', 0)); + IF (SELECT count(*) FROM rate.extension_submitters + WHERE created_at > now() - interval '1 minute') >= 600 THEN + RAISE EXCEPTION 'Too many extension submitters registered, try again later' + USING ERRCODE = '54000', DETAIL = 'submitter'; + END IF; + + INSERT INTO rate.extension_submitters (network_hash) VALUES (p_network_hash) + RETURNING id INTO v_id; + RETURN v_id; +END; +$$; + +-- 6. submit_extension_observation -------------------------------------------------------------- +-- The ingest key (42501, DETAIL ingest_key), a network hash (22023, DETAIL network_hash), then +-- every rule of ExtensionObservation v1 (22023, DETAIL the field) in the order of +-- parseExtensionObservationV1 (packages/shared-data/src/rules/extension-observation.ts): +-- 'observation' for a payload that is not an object, is over 16 KB or has another key (its name +-- is never echoed), then each key in turn, its presence checked with its rule (a nullable key +-- must be present as null), then extractor.source and extractor.extension_version. Then the +-- submitter (P0002, DETAIL submitter_id) and the limits (54000): at most 120 per submitter and +-- 600 per network hash in the last hour. + +CREATE OR REPLACE FUNCTION rate.submit_extension_observation( + p_ingest_key text, + p_submitter_id uuid, + p_observation jsonb, + p_network_hash text +) +RETURNS jsonb +LANGUAGE plpgsql +VOLATILE +SECURITY DEFINER +SET search_path = '' +AS $$ +DECLARE + c_keys constant text[] := ARRAY['schema_version', 'marketplace', 'listing_url', + 'external_item_id', 'external_shop_id', 'sku_id', 'variant_label', 'price_vnd', + 'list_price_vnd', 'stock_state', 'stock_qty', 'observed_at', 'extractor']; + c_id_pattern constant text := '^[A-Za-z0-9_.-]{1,64}$'; + o jsonb := p_observation; + v_marketplace text; + v_url text; + v_host text; + v_host_ok boolean; + v_item text; + v_shop text; + v_sku text; + v_variant text; + v_price bigint; + v_list_price bigint; + v_stock_state text; + v_stock_qty integer; + v_observed_text text; + v_observed timestamptz; + v_extractor jsonb; + v_num numeric; + v_now timestamptz := now(); + v_id bigint; +BEGIN + -- The web's key has at least 32 characters; a shorter or missing one never reaches the hash. + IF p_ingest_key IS NULL OR char_length(p_ingest_key) < 32 THEN + RAISE EXCEPTION 'Extension ingest key refused' USING ERRCODE = '42501', + DETAIL = 'ingest_key'; + END IF; + IF (SELECT key_sha256 FROM rate.extension_ingest_secret WHERE id) IS DISTINCT FROM + encode(sha256(convert_to(coalesce(p_ingest_key, ''), 'UTF8')), 'hex') THEN + RAISE EXCEPTION 'Extension ingest key refused' USING ERRCODE = '42501', + DETAIL = 'ingest_key'; + END IF; + + IF p_network_hash IS NULL OR p_network_hash !~ '^[0-9a-f]{64}$' THEN + RAISE EXCEPTION 'Invalid network hash' USING ERRCODE = '22023', DETAIL = 'network_hash'; + END IF; + + IF o IS NULL OR jsonb_typeof(o) <> 'object' OR octet_length(o::text) > 16384 THEN + RAISE EXCEPTION 'Observation must be a JSON object' USING ERRCODE = '22023', + DETAIL = 'observation'; + END IF; + + -- Any other key is refused as 'observation': a caller's key name is never echoed. + IF EXISTS (SELECT 1 FROM jsonb_object_keys(o) AS k WHERE k <> ALL (c_keys)) THEN + RAISE EXCEPTION 'Unknown observation key' USING ERRCODE = '22023', DETAIL = 'observation'; + END IF; + + -- schema_version: exactly 1. + -- A number is read only once its type is known, so a string never reaches the cast. + v_num := CASE WHEN jsonb_typeof(o -> 'schema_version') = 'number' + THEN (o ->> 'schema_version')::numeric END; + IF v_num IS DISTINCT FROM 1 THEN + RAISE EXCEPTION 'Unsupported schema_version' USING ERRCODE = '22023', + DETAIL = 'schema_version'; + END IF; + + -- marketplace. + v_marketplace := CASE WHEN jsonb_typeof(o -> 'marketplace') = 'string' + THEN o ->> 'marketplace' END; + IF v_marketplace IS NULL OR v_marketplace NOT IN ('shopee', 'lazada', 'tiki', 'tiktok', + 'cellphones', 'tgdd', 'fptshop', 'hoangha', 'didongviet', 'phongvu') THEN + RAISE EXCEPTION 'Unsupported marketplace' USING ERRCODE = '22023', DETAIL = 'marketplace'; + END IF; + + -- listing_url: https, at most 2048 characters, no query, fragment, path parameter (;), space + -- (JavaScript's \s set) or control character; a lowercase host without user or port, from the marketplace's host table. + v_url := CASE WHEN jsonb_typeof(o -> 'listing_url') = 'string' THEN o ->> 'listing_url' END; + v_host := substring(v_url FROM '^https://([^/]*)'); + -- The host table (a CASE inside an IF condition would end the condition at its first THEN). + v_host_ok := CASE v_marketplace + WHEN 'shopee' THEN v_host = 'shopee.vn' + WHEN 'lazada' THEN v_host ~ '^([a-z0-9-]+\.)*lazada\.vn$' + WHEN 'tiki' THEN v_host = 'tiki.vn' + WHEN 'tiktok' THEN v_host IN ('www.tiktok.com', 'shop.tiktok.com') + WHEN 'cellphones' THEN v_host = 'cellphones.com.vn' + WHEN 'tgdd' THEN v_host ~ '^([a-z0-9-]+\.)*thegioididong\.com$' + WHEN 'fptshop' THEN v_host = 'fptshop.com.vn' + WHEN 'hoangha' THEN v_host = 'hoanghamobile.com' + WHEN 'didongviet' THEN v_host = 'didongviet.vn' + WHEN 'phongvu' THEN v_host = 'phongvu.vn' + ELSE false END; + IF v_url IS NULL OR char_length(v_url) > 2048 + OR v_url ~ '[?#;[:space:]\x01-\x1f\x7f-\x9f\xa0\x1680\x2000-\x200a\x2028\x2029\x202f\x205f\x3000\xfeff]' + OR v_host IS NULL OR v_host !~ '^[a-z0-9.-]+$' + OR v_host_ok IS NOT TRUE THEN + RAISE EXCEPTION 'Invalid listing_url' USING ERRCODE = '22023', DETAIL = 'listing_url'; + END IF; + + -- The ids: 1 to 64 of [A-Za-z0-9_.-]; the shop and SKU may be null. + v_item := CASE WHEN jsonb_typeof(o -> 'external_item_id') = 'string' + THEN o ->> 'external_item_id' END; + IF v_item IS NULL OR v_item !~ c_id_pattern THEN + RAISE EXCEPTION 'Invalid external_item_id' USING ERRCODE = '22023', + DETAIL = 'external_item_id'; + END IF; + IF NOT o ? 'external_shop_id' + OR COALESCE(jsonb_typeof(o -> 'external_shop_id'), 'null') NOT IN ('null', 'string') + OR (o ->> 'external_shop_id') !~ c_id_pattern THEN + RAISE EXCEPTION 'Invalid external_shop_id' USING ERRCODE = '22023', + DETAIL = 'external_shop_id'; + END IF; + v_shop := o ->> 'external_shop_id'; + IF NOT o ? 'sku_id' + OR COALESCE(jsonb_typeof(o -> 'sku_id'), 'null') NOT IN ('null', 'string') + OR (o ->> 'sku_id') !~ c_id_pattern THEN + RAISE EXCEPTION 'Invalid sku_id' USING ERRCODE = '22023', DETAIL = 'sku_id'; + END IF; + v_sku := o ->> 'sku_id'; + + -- variant_label: null, or 1 to 120 characters without a control character. + IF NOT o ? 'variant_label' + OR COALESCE(jsonb_typeof(o -> 'variant_label'), 'null') NOT IN ('null', 'string') + OR char_length(o ->> 'variant_label') NOT BETWEEN 1 AND 120 + OR (o ->> 'variant_label') ~ '[\x01-\x1f\x7f-\x9f]' THEN + RAISE EXCEPTION 'Invalid variant_label' USING ERRCODE = '22023', DETAIL = 'variant_label'; + END IF; + v_variant := o ->> 'variant_label'; + + -- price_vnd: an integer from 1,000 to 2,000,000,000. + v_num := CASE WHEN jsonb_typeof(o -> 'price_vnd') = 'number' + THEN (o ->> 'price_vnd')::numeric END; + IF v_num IS NULL OR v_num NOT BETWEEN 1000 AND 2000000000 OR v_num <> trunc(v_num) THEN + RAISE EXCEPTION 'Invalid price_vnd' USING ERRCODE = '22023', DETAIL = 'price_vnd'; + END IF; + v_price := v_num; + + -- list_price_vnd: null, or an integer from price_vnd to 2,000,000,000. + v_num := CASE WHEN jsonb_typeof(o -> 'list_price_vnd') = 'number' + THEN (o ->> 'list_price_vnd')::numeric END; + IF NOT o ? 'list_price_vnd' + OR COALESCE(jsonb_typeof(o -> 'list_price_vnd'), 'null') NOT IN ('null', 'number') + OR v_num NOT BETWEEN v_price AND 2000000000 OR v_num <> trunc(v_num) THEN + RAISE EXCEPTION 'Invalid list_price_vnd' USING ERRCODE = '22023', DETAIL = 'list_price_vnd'; + END IF; + v_list_price := v_num; + + -- stock_state and stock_qty. + v_stock_state := CASE WHEN jsonb_typeof(o -> 'stock_state') = 'string' + THEN o ->> 'stock_state' END; + IF v_stock_state IS NULL + OR v_stock_state NOT IN ('in_stock', 'low_stock', 'out_of_stock', 'unknown') THEN + RAISE EXCEPTION 'Invalid stock_state' USING ERRCODE = '22023', DETAIL = 'stock_state'; + END IF; + v_num := CASE WHEN jsonb_typeof(o -> 'stock_qty') = 'number' + THEN (o ->> 'stock_qty')::numeric END; + IF NOT o ? 'stock_qty' + OR COALESCE(jsonb_typeof(o -> 'stock_qty'), 'null') NOT IN ('null', 'number') + OR v_num NOT BETWEEN 0 AND 1000000 OR v_num <> trunc(v_num) THEN + RAISE EXCEPTION 'Invalid stock_qty' USING ERRCODE = '22023', DETAIL = 'stock_qty'; + END IF; + v_stock_qty := v_num; + + -- observed_at: ISO 8601 with a zone (seconds 00 to 59, so no leap second; an offset up to + -- 15:59, as Postgres parses it), at most 10 minutes ahead and 24 hours behind now(). + v_observed_text := CASE WHEN jsonb_typeof(o -> 'observed_at') = 'string' + THEN o ->> 'observed_at' END; + IF v_observed_text IS NULL OR v_observed_text !~ + '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(:[0-5]\d(\.\d{1,9})?)?(Z|[+-](0\d|1[0-5])(:?[0-5]\d)?)$' + THEN + RAISE EXCEPTION 'Invalid observed_at' USING ERRCODE = '22023', DETAIL = 'observed_at'; + END IF; + BEGIN + v_observed := v_observed_text::timestamptz; + EXCEPTION WHEN others THEN + RAISE EXCEPTION 'Invalid observed_at' USING ERRCODE = '22023', DETAIL = 'observed_at'; + END; + IF v_observed > v_now + interval '10 minutes' OR v_observed < v_now - interval '24 hours' THEN + RAISE EXCEPTION 'observed_at is out of range' USING ERRCODE = '22023', + DETAIL = 'observed_at'; + END IF; + + -- extractor: exactly {source, extension_version}. + v_extractor := o -> 'extractor'; + IF jsonb_typeof(v_extractor) IS DISTINCT FROM 'object' + OR v_extractor - 'source' - 'extension_version' <> '{}'::jsonb THEN + RAISE EXCEPTION 'Invalid extractor' USING ERRCODE = '22023', DETAIL = 'extractor'; + END IF; + IF jsonb_typeof(v_extractor -> 'source') IS DISTINCT FROM 'string' + OR (v_extractor ->> 'source') NOT IN ('json_ld', 'dom', 'mixed') THEN + RAISE EXCEPTION 'Invalid extractor.source' USING ERRCODE = '22023', + DETAIL = 'extractor.source'; + END IF; + IF jsonb_typeof(v_extractor -> 'extension_version') IS DISTINCT FROM 'string' + OR (v_extractor ->> 'extension_version') !~ '^[0-9A-Za-z.+-]{1,20}$' THEN + RAISE EXCEPTION 'Invalid extractor.extension_version' USING ERRCODE = '22023', + DETAIL = 'extractor.extension_version'; + END IF; + + -- The submitter, locked: one submitter's calls (and its deletion) run one at a time, so its + -- count below is exact. + PERFORM 1 FROM rate.extension_submitters WHERE id = p_submitter_id FOR UPDATE; + IF NOT FOUND THEN + RAISE EXCEPTION 'Unknown extension submitter' USING ERRCODE = 'P0002', + DETAIL = 'submitter_id'; + END IF; + + IF (SELECT count(*) FROM rate.extension_observations + WHERE submitter_id = p_submitter_id + AND received_at > v_now - interval '1 hour') >= 120 THEN + RAISE EXCEPTION 'Too many observations from this submitter, try again later' + USING ERRCODE = '54000', DETAIL = 'submitter_id'; + END IF; + + -- One network's calls run one at a time too (a transaction-scoped advisory lock). + PERFORM pg_catalog.pg_advisory_xact_lock( + pg_catalog.hashtextextended('rate.extension_observations:' || p_network_hash, 0)); + IF (SELECT count(*) FROM rate.extension_observations + WHERE network_hash = p_network_hash + AND received_at > v_now - interval '1 hour') >= 600 THEN + RAISE EXCEPTION 'Too many observations from this network, try again later' + USING ERRCODE = '54000', DETAIL = 'network_hash'; + END IF; + + INSERT INTO rate.extension_observations ( + submitter_id, schema_version, marketplace, listing_url, external_item_id, external_shop_id, + sku_id, variant_label, price_vnd, list_price_vnd, stock_state, stock_qty, observed_at, + extractor, received_at, network_hash, status) + VALUES ( + p_submitter_id, 1, v_marketplace, v_url, v_item, v_shop, + v_sku, v_variant, v_price, v_list_price, v_stock_state, v_stock_qty, v_observed, + v_extractor, v_now, p_network_hash, 'quarantined') + RETURNING id INTO v_id; + + RETURN jsonb_build_object('id', v_id, 'status', 'quarantined', 'received_at', v_now); +END; +$$; + +-- 7. delete_extension_submitter ---------------------------------------------------------------- +-- Deletes the submitter; its observations go by the cascade. Returns how many observations went; +-- an unknown or null id returns 0, so a call does not reveal whether an id exists. + +CREATE OR REPLACE FUNCTION rate.delete_extension_submitter(p_submitter_id uuid) +RETURNS integer +LANGUAGE plpgsql +VOLATILE +SECURITY DEFINER +SET search_path = '' +AS $$ +DECLARE + v_count integer; +BEGIN + -- Locked first: no observation of this submitter can be added between the count and the + -- delete. + PERFORM 1 FROM rate.extension_submitters WHERE id = p_submitter_id FOR UPDATE; + IF NOT FOUND THEN + RETURN 0; + END IF; + + SELECT count(*) INTO v_count FROM rate.extension_observations + WHERE submitter_id = p_submitter_id; + + DELETE FROM rate.extension_submitters WHERE id = p_submitter_id; + RETURN v_count; +END; +$$; + +-- 8. purge_extension_observations -------------------------------------------------------------- +-- Retention, ages from received_at and created_at. Meant to run daily as service_role. +-- network_hash cleared after p_network_hash_days (28), on observations and on submitters; +-- observations deleted after p_keep_days (180); +-- submitters without observations deleted after p_empty_submitter_days (7). +-- The empty submitters are locked first (SKIP LOCKED: one a submit holds waits for the next +-- run), then deleted under a fresh NOT EXISTS, so an observation committed in between keeps its +-- submitter instead of going with it by the cascade. + +CREATE OR REPLACE FUNCTION rate.purge_extension_observations( + p_network_hash_days integer DEFAULT 28, + p_keep_days integer DEFAULT 180, + p_empty_submitter_days integer DEFAULT 7 +) +RETURNS jsonb +LANGUAGE plpgsql +VOLATILE +SECURITY DEFINER +SET search_path = '' +AS $$ +DECLARE + v_hashes integer; + v_submitter_hashes integer; + v_observations integer; + v_submitters integer; + v_candidates uuid[]; +BEGIN + IF p_network_hash_days IS NULL OR p_network_hash_days < 1 THEN + RAISE EXCEPTION 'p_network_hash_days must be at least 1' USING ERRCODE = '22023', + DETAIL = 'p_network_hash_days'; + END IF; + IF p_keep_days IS NULL OR p_keep_days < 1 THEN + RAISE EXCEPTION 'p_keep_days must be at least 1' USING ERRCODE = '22023', + DETAIL = 'p_keep_days'; + END IF; + IF p_empty_submitter_days IS NULL OR p_empty_submitter_days < 1 THEN + RAISE EXCEPTION 'p_empty_submitter_days must be at least 1' USING ERRCODE = '22023', + DETAIL = 'p_empty_submitter_days'; + END IF; + + -- Lock order: submitters before observations, as delete_extension_submitter does (the + -- submitter row, then its observations through the cascade). The other order deadlocks with a + -- delete that runs in the middle of a purge. + UPDATE rate.extension_submitters SET network_hash = NULL + WHERE network_hash IS NOT NULL + AND created_at < now() - make_interval(days => p_network_hash_days); + GET DIAGNOSTICS v_submitter_hashes = ROW_COUNT; + + UPDATE rate.extension_observations SET network_hash = NULL + WHERE network_hash IS NOT NULL + AND received_at < now() - make_interval(days => p_network_hash_days); + GET DIAGNOSTICS v_hashes = ROW_COUNT; + + DELETE FROM rate.extension_observations + WHERE received_at < now() - make_interval(days => p_keep_days); + GET DIAGNOSTICS v_observations = ROW_COUNT; + + SELECT array_agg(c.id) INTO v_candidates FROM ( + SELECT s.id FROM rate.extension_submitters s + WHERE s.created_at < now() - make_interval(days => p_empty_submitter_days) + AND NOT EXISTS (SELECT 1 FROM rate.extension_observations o WHERE o.submitter_id = s.id) + FOR UPDATE OF s SKIP LOCKED + ) c; + + DELETE FROM rate.extension_submitters s + WHERE s.id = ANY (COALESCE(v_candidates, '{}')) + AND NOT EXISTS (SELECT 1 FROM rate.extension_observations o WHERE o.submitter_id = s.id); + GET DIAGNOSTICS v_submitters = ROW_COUNT; + + RETURN jsonb_build_object('network_hashes_cleared', v_hashes, + 'submitter_network_hashes_cleared', v_submitter_hashes, + 'observations_deleted', v_observations, + 'submitters_deleted', v_submitters); +END; +$$; + +-- 9. Execute grants ---------------------------------------------------------------------------- + +REVOKE EXECUTE ON FUNCTION rate.register_extension_submitter(text, text) + FROM PUBLIC, anon, authenticated, service_role, rate_worker; +REVOKE EXECUTE ON FUNCTION rate.submit_extension_observation(text, uuid, jsonb, text) + FROM PUBLIC, anon, authenticated, service_role, rate_worker; +REVOKE EXECUTE ON FUNCTION rate.delete_extension_submitter(uuid) + FROM PUBLIC, anon, authenticated, service_role, rate_worker; +REVOKE EXECUTE ON FUNCTION rate.purge_extension_observations(integer, integer, integer) + FROM PUBLIC, anon, authenticated, service_role, rate_worker; + +-- register and submit stay open to anon: the web calls them with the anon key plus the ingest +-- key. delete takes no ingest key, so erasure always works. +GRANT EXECUTE ON FUNCTION rate.register_extension_submitter(text, text) + TO anon, authenticated, service_role; +GRANT EXECUTE ON FUNCTION rate.submit_extension_observation(text, uuid, jsonb, text) + TO anon, authenticated, service_role; +GRANT EXECUTE ON FUNCTION rate.delete_extension_submitter(uuid) + TO anon, authenticated, service_role; +GRANT EXECUTE ON FUNCTION rate.purge_extension_observations(integer, integer, integer) + TO service_role; + +COMMIT; diff --git a/scripts/test-extension-observations-proposal.mjs b/scripts/test-extension-observations-proposal.mjs new file mode 100644 index 000000000..94aa3fe8a --- /dev/null +++ b/scripts/test-extension-observations-proposal.mjs @@ -0,0 +1,1177 @@ +#!/usr/bin/env node +// Replays the DOS-Me extension observations PROPOSAL (DOS/Rate#1606, M2 of +// Docs/Proposals/2026-10-28-extension-data-source.md, contract Revision 2) on the migration +// replay stack: the proposal twice; the ingest key (no row, a wrong key, the right key); every +// verification query; each payload, valid or refused, against the shared validator +// (parseExtensionObservationV1 in @repo/shared-data) with the same verdict and field; the +// unknown submitter; the limits at their exact thresholds; the delete cascade; the purge and its +// race with a submit; the table grants; the rollback twice and the proposal again. Nothing +// touches a real database. +// +// Usage: pnpm --filter @repo/shared-data build && node scripts/test-extension-observations-proposal.mjs +// (needs Docker) + +import assert from 'node:assert/strict'; +import { createHash } from 'node:crypto'; +import { readFileSync } from 'node:fs'; +import path from 'node:path'; +import { spawn, spawnSync } from 'node:child_process'; +import { pathToFileURL } from 'node:url'; + +import { repoRoot, startRateDb } from './lib/rate-db.mjs'; + +const rules = await import( + pathToFileURL( + path.join(repoRoot, 'packages', 'shared-data', 'dist', 'rules', 'index.js') + ).href +); +const parse = rules.parseExtensionObservationV1; +assert.equal( + typeof parse, + 'function', + 'parseExtensionObservationV1 is missing from @repo/shared-data: build it first' +); + +const read = (name) => + readFileSync(path.join(repoRoot, 'Docs', 'Proposals', name), 'utf8'); +const proposal = read('2026-11-04-dos-me-extension-observations.sql'); +const rollback = read('2026-11-04-dos-me-extension-observations-rollback.sql'); +const verify = read('2026-11-04-dos-me-extension-observations-verify.sql'); + +const container = `rate-extension-observations-proposal-${process.pid}`; +const removeContainer = () => + spawnSync('docker', ['rm', '--force', container], { timeout: 30_000 }); +for (const signal of ['SIGINT', 'SIGTERM']) { + process.on(signal, () => { + removeContainer(); + process.exit(130); + }); +} + +const checks = [ + ...verify.matchAll( + /^-- (V\d+) [^\n]*(?:\n--[^\n]*)*expect 0\n((?:SELECT|WITH)[\s\S]*?;)/gm + ), +].map(([, id, query]) => ({ id, query })); +assert.deepEqual( + checks.map((c) => c.id), + ['V1', 'V2', 'V3', 'V4', 'V5', 'V6', 'V7', 'V8', 'V9', 'V10', 'V11', 'V12'], + 'the verification lists its expect-0 queries' +); +const check = (id) => checks.find((c) => c.id === id).query; + +const q = (value) => `'${String(value).replaceAll("'", "''")}'`; +const j = (value) => `${q(JSON.stringify(value))}::jsonb`; +const sha = (text) => createHash('sha256').update(text, 'utf8').digest('hex'); +const ch = (code) => String.fromCodePoint(code); +// A test value made up here, not a secret: the replay sets its hash as the ingest key row. +const KEY = `local-replay-ingest-key-not-a-secret-${process.pid}`; +const NET_A = sha('net-a'); +const UNKNOWN = '00000000-0000-4000-8000-000000000000'; +let nets = 0; +const freshNet = () => sha(`net-${(nets += 1)}`); + +const MIN = 60_000; +const HOUR = 60 * MIN; +const pad = (n) => String(n).padStart(2, '0'); +// ISO 8601 at `ms`, written in the offset `off` minutes: sep ':' gives +07:00, '' gives +0700, +// null gives +07; offset 0 is Z. +const iso = (ms, { off = 0, sec = true, frac = '', sep = ':' } = {}) => { + const d = new Date(ms + off * MIN); + let s = `${d.getUTCFullYear()}-${pad(d.getUTCMonth() + 1)}-${pad(d.getUTCDate())}T${pad(d.getUTCHours())}:${pad(d.getUTCMinutes())}`; + if (sec) s += `:${pad(d.getUTCSeconds())}${frac}`; + if (off === 0) return `${s}Z`; + const a = Math.abs(off); + return `${s}${off > 0 ? '+' : '-'}${pad(Math.floor(a / 60))}${sep === null ? '' : `${sep}${pad(a % 60)}`}`; +}; +const ago = (ms) => iso(Date.now() - ms, { frac: '.000' }); + +const good = (over = {}) => ({ + schema_version: 1, + marketplace: 'shopee', + listing_url: 'https://shopee.vn/product/123456/7890123', + external_item_id: '7890123', + external_shop_id: '123456', + sku_id: null, + variant_label: 'Den, 128GB', + price_vnd: 10990000, + list_price_vnd: 12990000, + stock_state: 'in_stock', + stock_qty: null, + observed_at: new Date().toISOString(), + extractor: { source: 'json_ld', extension_version: '0.2.0' }, + ...over, +}); +const without = (key) => { + const o = good(); + delete o[key]; + return o; +}; + +try { + const { sql } = startRateDb({ name: container }); + // One psql session per call; the role and its errors stay in it. VERBOSITY verbose prints the + // SQLSTATE in front of the message. + const as = (role, query) => + sql(`\\set VERBOSITY verbose\nSET ROLE ${role};\n${query};\nRESET ROLE;`); + const refused = (role, query, code, detail, what) => + assert.throws( + () => as(role, query), + (err) => { + assert.match(err.message, new RegExp(`ERROR:\\s+${code}:`), what); + if (detail !== undefined) + assert.match( + err.message, + new RegExp(`DETAIL:\\s+${detail.replaceAll('.', '\\.')}\\r?\\n`), + what + ); + return true; + }, + what + ); + const checkAll = (when) => { + for (const { id, query } of checks) + assert.equal(sql(query), '0', `${id} ${when}`); + }; + const lit = (v) => (v === null ? 'NULL' : q(v)); + const registerSql = (net, key = KEY) => + `SELECT rate.register_extension_submitter(${lit(key)}, ${lit(net)})`; + const register = (net = freshNet(), role = 'anon') => + as(role, registerSql(net)); + const submitSql = (id, obs, net = NET_A, key = KEY) => + `SELECT rate.submit_extension_observation(${lit(key)}, ${lit(id)}, ${ + obs === undefined ? 'NULL' : j(obs) + }, ${lit(net)})::text`; + const submit = (id, obs, net = NET_A, role = 'anon') => + JSON.parse(as(role, submitSql(id, obs, net))); + const count = (where = 'true', table = 'rate.extension_observations') => + Number(sql(`SELECT count(*) FROM ${table} WHERE ${where}`)); + // Rows written straight as postgres, received (and observed) `age` ago. + const backfill = (id, n, age, net = null) => + sql(`INSERT INTO rate.extension_observations (submitter_id, schema_version, marketplace, + listing_url, external_item_id, price_vnd, stock_state, observed_at, extractor, + received_at, network_hash) + SELECT ${q(id)}, 1, 'shopee', 'https://shopee.vn/p', 'i' || g, 100000, 'unknown', + now() - ${q(age)}::interval, '{"source": "dom", "extension_version": "1"}', + now() - ${q(age)}::interval, ${lit(net)} + FROM generate_series(1, ${n}) g`); + + // 1. The proposal twice. + sql(proposal); + sql(proposal); + + // 2. The ingest key: no row refuses every caller; then a wrong, empty or null key refuses and + // the right one passes. delete takes no key. + refused( + 'anon', + registerSql(NET_A), + '42501', + 'ingest_key', + 'register without a key row' + ); + refused( + 'anon', + submitSql(UNKNOWN, good()), + '42501', + 'ingest_key', + 'submit without a key row' + ); + assert.equal( + count('true', 'rate.extension_ingest_secret'), + 0, + 'the migration sets no key' + ); + sql( + `INSERT INTO rate.extension_ingest_secret (key_sha256) VALUES (${q(sha(KEY))})` + ); + refused( + 'postgres', + `INSERT INTO rate.extension_ingest_secret (key_sha256) VALUES (${q(sha('x'))})`, + '23505', + undefined, + 'one row only' + ); + refused( + 'postgres', + `INSERT INTO rate.extension_ingest_secret (id, key_sha256) + VALUES (false, ${q(sha('x'))})`, + '23514', + undefined, + 'id is always true' + ); + refused( + 'postgres', + `UPDATE rate.extension_ingest_secret SET key_sha256 = 'ABC'`, + '23514', + undefined, + 'the hash format is checked' + ); + for (const key of ['wrong', '', null, KEY.toUpperCase(), sha(KEY)]) { + refused( + 'anon', + registerSql(NET_A, key), + '42501', + 'ingest_key', + `register with ${key}` + ); + refused( + 'anon', + submitSql(UNKNOWN, good(), NET_A, key), + '42501', + 'ingest_key', + `submit with ${key}` + ); + } + // The empty key's hash cannot be stored; a key under 32 characters is refused even when its + // hash is the stored one. + refused( + 'postgres', + `UPDATE rate.extension_ingest_secret SET key_sha256 = ${q(sha(''))}`, + '23514', + undefined, + 'the empty key hash is refused' + ); + const shortKey = 'k'.repeat(31); + sql( + `UPDATE rate.extension_ingest_secret SET key_sha256 = ${q(sha(shortKey))}` + ); + refused( + 'anon', + registerSql(NET_A, shortKey), + '42501', + 'ingest_key', + 'a 31-character key' + ); + refused( + 'anon', + submitSql(UNKNOWN, good(), NET_A, shortKey), + '42501', + 'ingest_key', + 'a 31-character key on submit' + ); + sql(`UPDATE rate.extension_ingest_secret SET key_sha256 = ${q(sha(KEY))}`); + assert.ok(KEY.length >= 32, 'the replay key is long enough'); + + // 3. Every check, and a check fails when its rule breaks. + checkAll('after the apply'); + sql(verify); + sql('GRANT SELECT ON rate.extension_observations TO anon'); + assert.notEqual(sql(check('V2')), '0', 'V2 sees a grant to anon'); + sql('REVOKE SELECT ON rate.extension_observations FROM anon'); + sql('GRANT SELECT ON rate.extension_ingest_secret TO rate_worker'); + assert.notEqual(sql(check('V11')), '0', 'V11 sees a grant on the key'); + sql('REVOKE SELECT ON rate.extension_ingest_secret FROM rate_worker'); + sql('ALTER TABLE rate.extension_observations FORCE ROW LEVEL SECURITY'); + assert.notEqual(sql(check('V1')), '0', 'V1 sees forced row level security'); + sql('ALTER TABLE rate.extension_observations NO FORCE ROW LEVEL SECURITY'); + checkAll('after the probes'); + + // 4. Register: the right key and a network hash; anon and authenticated; not rate_worker. + const s1 = register(); + assert.match(s1, /^[0-9a-f-]{36}$/); + const sAuth = register(freshNet(), 'authenticated'); + assert.notEqual(s1, sAuth); + assert.equal( + sql( + `SELECT network_hash FROM rate.extension_submitters WHERE id = ${q(s1)}` + ), + sha(`net-${nets - 1}`), + 'the submitter keeps its network hash' + ); + refused( + 'rate_worker', + registerSql(NET_A), + '42501', + undefined, + 'rate_worker cannot register' + ); + for (const net of [ + null, + 'abc', + NET_A.toUpperCase(), + NET_A.slice(1), + `${NET_A}0`, + '', + ]) { + refused( + 'anon', + registerSql(net), + '22023', + 'network_hash', + `register network_hash ${net}` + ); + refused( + 'anon', + submitSql(s1, good(), net), + '22023', + 'network_hash', + `submit network_hash ${net}` + ); + } + + // 5. A valid observation: the row, the answer. + const r1 = submit(s1, good()); + assert.deepEqual(Object.keys(r1).sort(), ['id', 'received_at', 'status']); + assert.equal(r1.status, 'quarantined'); + assert.equal(typeof r1.id, 'number'); + assert.equal( + sql(`SELECT submitter_id || '|' || marketplace || '|' || price_vnd || '|' || list_price_vnd + || '|' || coalesce(sku_id, '-') || '|' || status || '|' || network_hash || '|' + || (received_at = ${q(r1.received_at)}::timestamptz) || '|' || extractor::text + FROM rate.extension_observations WHERE id = ${r1.id}`), + `${s1}|shopee|10990000|12990000|-|quarantined|${NET_A}|true|{"source": "json_ld", "extension_version": "0.2.0"}` + ); + assert.equal( + submit(sAuth, good(), NET_A, 'authenticated').status, + 'quarantined' + ); + assert.equal(submit(s1, good(), NET_A, 'service_role').status, 'quarantined'); + + // 6. Each payload against the shared validator: the same verdict and the same field. The RPC + // writes the valid ones and nothing for the others. + const now = Date.now(); + const emoji = ch(0x1f600); + const valid = [ + {}, + { + marketplace: 'lazada', + listing_url: 'https://www.lazada.vn/products/x-i1.html', + }, + { marketplace: 'lazada', listing_url: 'https://lazada.vn/products/x' }, + { marketplace: 'lazada', listing_url: 'https://a.b-c.lazada.vn/x' }, + { marketplace: 'tiki', listing_url: 'https://tiki.vn/x-p1.html' }, + { + marketplace: 'tiktok', + listing_url: 'https://www.tiktok.com/view/product/1', + }, + { + marketplace: 'tiktok', + listing_url: 'https://shop.tiktok.com/view/product/1', + }, + { + marketplace: 'cellphones', + listing_url: 'https://cellphones.com.vn/x.html', + }, + { + marketplace: 'tgdd', + listing_url: 'https://www.thegioididong.com/dtdd/x', + }, + { marketplace: 'tgdd', listing_url: 'https://thegioididong.com/dtdd/x' }, + { + marketplace: 'fptshop', + listing_url: 'https://fptshop.com.vn/dien-thoai/x', + }, + { marketplace: 'hoangha', listing_url: 'https://hoanghamobile.com/x' }, + { marketplace: 'didongviet', listing_url: 'https://didongviet.vn/x.html' }, + { marketplace: 'phongvu', listing_url: 'https://phongvu.vn/x' }, + { listing_url: 'https://shopee.vn' }, + { + listing_url: `https://shopee.vn/${'a'.repeat(2048 - 'https://shopee.vn/'.length)}`, + }, + { + listing_url: `https://shopee.vn/${emoji.repeat(2048 - 'https://shopee.vn/'.length)}`, + }, + { listing_url: `https://shopee.vn/${ch(0x200b)}x` }, + { external_shop_id: null, variant_label: null, list_price_vnd: null }, + { + price_vnd: 1000, + list_price_vnd: 1000, + stock_qty: 0, + stock_state: 'low_stock', + }, + { price_vnd: 2000000000, list_price_vnd: 2000000000, stock_qty: 1000000 }, + { price_vnd: 1000.0, schema_version: 1.0 }, + { + external_item_id: 'A'.repeat(64), + sku_id: 'sku_1.2-3', + variant_label: 'x'.repeat(120), + }, + { variant_label: emoji.repeat(120) }, + { variant_label: `Mau ${ch(0x111)}en ${ch(0x200b)}` }, + { + observed_at: iso(now - 23 * HOUR - 59 * MIN), + stock_state: 'out_of_stock', + }, + { observed_at: iso(now + 9 * MIN + 30_000), stock_state: 'unknown' }, + { observed_at: iso(now - HOUR, { off: 420 }) }, + { observed_at: iso(now - HOUR, { off: 420, sep: '' }) }, + { observed_at: iso(now - HOUR, { off: 420, sep: null }) }, + { observed_at: iso(now - HOUR, { off: -300 }) }, + { observed_at: iso(now - HOUR, { off: 345 }) }, + { observed_at: iso(now - HOUR, { off: 959 }) }, + { observed_at: iso(now - HOUR, { off: -900 }) }, + { observed_at: iso(now - HOUR, { sec: false }) }, + { observed_at: iso(now - HOUR, { frac: '.123456789' }) }, + { observed_at: iso(now - HOUR, { frac: '.5' }) }, + { extractor: { source: 'mixed', extension_version: '1.0.0-beta+2' } }, + { extractor: { source: 'dom', extension_version: 'v'.repeat(20) } }, + ].map((over) => ['ok', good(over)]); + const hourAgo = iso(now - HOUR); + const bad = [ + ['observation', []], + ['observation', 'x'], + ['observation', null], + ['observation', 5], + ['observation', { ...good(), rating: 5 }], + ['observation', { ...good(), cookies: 'a=b' }], + ['observation', { ...good(), ["x'; DROP TABLE x; --"]: 1 }], + ['observation', { ...without('schema_version'), referrer: 'https://x' }], + ['schema_version', good({ schema_version: 2 })], + ['schema_version', good({ schema_version: '1' })], + ['schema_version', good({ schema_version: null })], + ['marketplace', good({ marketplace: 'amazon' })], + ['marketplace', good({ marketplace: 'Shopee' })], + ['marketplace', good({ marketplace: 'thegioididong' })], + ['marketplace', good({ marketplace: 'hoanghamobile' })], + ['marketplace', good({ marketplace: null })], + ['listing_url', good({ listing_url: 'http://shopee.vn/x' })], + ['listing_url', good({ listing_url: 'https://shopee.vn/x?sp_atk=1' })], + ['listing_url', good({ listing_url: 'https://shopee.vn/x#reviews' })], + ['listing_url', good({ listing_url: 'https://shopee.vn/x;jsessionid=1' })], + ['listing_url', good({ listing_url: 'https://shopee.vn;x' })], + ['listing_url', good({ listing_url: 'https://shopee.vn/x y' })], + ['listing_url', good({ listing_url: 'https://shopee.vn/x\ny' })], + ['listing_url', good({ listing_url: 'https://shopee.vn/x\ty' })], + ...[ + 0xa0, 0x85, 0x1680, 0x2000, 0x200a, 0x2028, 0x2029, 0x202f, 0x205f, + 0x3000, 0xfeff, + ].map((c) => [ + 'listing_url', + good({ listing_url: `https://shopee.vn/x${ch(c)}y` }), + ]), + ['listing_url', good({ listing_url: 'https://Shopee.vn/x' })], + ['listing_url', good({ listing_url: 'https://shopee.vn:443/x' })], + ['listing_url', good({ listing_url: 'https://user@shopee.vn/x' })], + ['listing_url', good({ listing_url: 'https://shopee.vn./x' })], + ['listing_url', good({ listing_url: 'https:///shopee.vn/x' })], + ['listing_url', good({ listing_url: 'https://shopee.vn.evil.com/x' })], + ['listing_url', good({ listing_url: 'https://evil.com/shopee.vn' })], + ['listing_url', good({ listing_url: 'https://www.shopee.vn/x' })], + ['listing_url', good({ listing_url: 'https://lazada.vn/x' })], + [ + 'listing_url', + good({ marketplace: 'lazada', listing_url: 'https://xlazada.vn/x' }), + ], + [ + 'listing_url', + good({ marketplace: 'tiktok', listing_url: 'https://tiktok.com/x' }), + ], + [ + 'listing_url', + good({ marketplace: 'tiktok', listing_url: 'https://m.tiktok.com/x' }), + ], + [ + 'listing_url', + good({ listing_url: `https://shopee.vn/${'a'.repeat(2031)}` }), + ], + [ + 'listing_url', + good({ listing_url: `https://shopee.vn/${emoji.repeat(2031)}` }), + ], + ['listing_url', good({ listing_url: 'https://shopee.vn\\@evil.com/x' })], + ['listing_url', good({ listing_url: 5 })], + ['listing_url', good({ listing_url: null })], + ['external_item_id', good({ external_item_id: '' })], + ['external_item_id', good({ external_item_id: 'a'.repeat(65) })], + ['external_item_id', good({ external_item_id: 'a b' })], + ['external_item_id', good({ external_item_id: 'a/b' })], + ['external_item_id', good({ external_item_id: 'a\n' })], + ['external_item_id', good({ external_item_id: 123 })], + ['external_shop_id', good({ external_shop_id: '' })], + ['external_shop_id', good({ external_shop_id: 'x y' })], + ['external_shop_id', good({ external_shop_id: 5 })], + ['sku_id', good({ sku_id: 'a'.repeat(65) })], + ['sku_id', good({ sku_id: ch(0xe1) })], + ['variant_label', good({ variant_label: '' })], + ['variant_label', good({ variant_label: 'x'.repeat(121) })], + ['variant_label', good({ variant_label: emoji.repeat(121) })], + ['variant_label', good({ variant_label: 'a\nb' })], + ['variant_label', good({ variant_label: `a${ch(0x85)}b` })], + ['variant_label', good({ variant_label: `a${ch(0x7f)}b` })], + ['variant_label', good({ variant_label: 5 })], + ['price_vnd', good({ price_vnd: 999 })], + ['price_vnd', good({ price_vnd: 2000000001 })], + ['price_vnd', good({ price_vnd: 1000.5 })], + ['price_vnd', good({ price_vnd: '10990000' })], + ['price_vnd', good({ price_vnd: null })], + ['list_price_vnd', good({ list_price_vnd: 10989999 })], + ['list_price_vnd', good({ list_price_vnd: 2000000001 })], + ['list_price_vnd', good({ list_price_vnd: 11000000.5 })], + ['list_price_vnd', good({ list_price_vnd: '12990000' })], + ['stock_state', good({ stock_state: 'sold_out' })], + ['stock_state', good({ stock_state: null })], + ['stock_qty', good({ stock_qty: -1 })], + ['stock_qty', good({ stock_qty: 1000001 })], + ['stock_qty', good({ stock_qty: 1.5 })], + ['stock_qty', good({ stock_qty: '3' })], + ['observed_at', good({ observed_at: iso(now - 24 * HOUR - 60_000) })], + ['observed_at', good({ observed_at: iso(now + 10 * MIN + 60_000) })], + ['observed_at', good({ observed_at: 'yesterday' })], + ['observed_at', good({ observed_at: 'now' })], + ['observed_at', good({ observed_at: '2026-13-01T00:00:00Z' })], + ['observed_at', good({ observed_at: hourAgo.slice(0, 19) })], + ['observed_at', good({ observed_at: hourAgo.replace('T', ' ') })], + ['observed_at', good({ observed_at: hourAgo.replace('Z', 'z') })], + ['observed_at', good({ observed_at: hourAgo.replace(/:\d\dZ$/, ':61Z') })], + ['observed_at', good({ observed_at: hourAgo.replace(/T\d\d/, 'T25') })], + ['observed_at', good({ observed_at: hourAgo.replace('Z', '+24:00') })], + ['observed_at', good({ observed_at: hourAgo.replace(/:\d\dZ$/, ':60Z') })], + ['observed_at', good({ observed_at: hourAgo.replace('Z', '+16:00') })], + ['observed_at', good({ observed_at: hourAgo.replace('Z', '-16:00') })], + ['observed_at', good({ observed_at: hourAgo.replace('Z', '+1560') })], + ['observed_at', good({ observed_at: hourAgo.replace('Z', '+05:60') })], + [ + 'observed_at', + good({ observed_at: `${hourAgo.slice(0, 19)}.1234567890Z` }), + ], + ['observed_at', good({ observed_at: now })], + ['observed_at', good({ observed_at: null })], + ['extractor', good({ extractor: 'dom' })], + ['extractor', good({ extractor: null })], + ['extractor', good({ extractor: [] })], + [ + 'extractor', + good({ extractor: { source: 'dom', extension_version: '1', x: 1 } }), + ], + [ + 'extractor.source', + good({ extractor: { source: 'ai', extension_version: '1' } }), + ], + ['extractor.source', good({ extractor: { extension_version: '1' } })], + ['extractor.extension_version', good({ extractor: { source: 'dom' } })], + [ + 'extractor.extension_version', + good({ extractor: { source: 'dom', extension_version: '' } }), + ], + [ + 'extractor.extension_version', + good({ extractor: { source: 'dom', extension_version: 'v'.repeat(21) } }), + ], + [ + 'extractor.extension_version', + good({ extractor: { source: 'dom', extension_version: 'v 1' } }), + ], + [ + 'extractor.extension_version', + good({ extractor: { source: 'dom', extension_version: 1 } }), + ], + // Every key must be present, a nullable one as null; each is named by its own check. + ...Object.keys(good()).map((key) => [key, without(key)]), + ]; + // The RPC's verdict: 'ok', or the field of its 22023. + const sParity = register(); + const rpcVerdict = (obs) => { + try { + submit(sParity, obs); + return 'ok'; + } catch (err) { + const m = /ERROR:\s+(\w+):[\s\S]*?DETAIL:\s+(\S+)\r?\n/.exec(err.message); + assert.ok(m, err.message); + assert.equal(m[1], '22023', err.message); + return m[2]; + } + }; + const before = count(); + const disagree = []; + for (const [expected, obs] of [...valid, ...bad]) { + const ts = parse(obs, Date.now()); + const tsVerdict = ts.ok ? 'ok' : ts.field; + const db = rpcVerdict(obs); + const what = JSON.stringify(obs).slice(0, 160); + if (tsVerdict !== db) + disagree.push(`${what}: shared-data ${tsVerdict}, RPC ${db}`); + assert.equal(db, expected, `RPC on ${what}`); + } + assert.deepEqual( + disagree, + [], + 'the RPC and parseExtensionObservationV1 agree' + ); + assert.equal( + count() - before, + valid.length, + 'only the valid payloads are written' + ); + // Where they differ on purpose: the 16 KB cap is the RPC's alone (the web caps the body). + const big = good({ variant_label: 'x'.repeat(17000) }); + assert.equal(parse(big, Date.now()).field, 'variant_label'); + assert.equal(rpcVerdict(big), 'observation'); + // A lone surrogate: shared-data names the field; jsonb refuses the text itself. + const lone = good({ + listing_url: `https://shopee.vn/x${String.fromCharCode(0xd800)}`, + }); + assert.equal(parse(lone, Date.now()).field, 'listing_url'); + refused( + 'anon', + submitSql(sParity, lone), + '22P02', + undefined, + 'jsonb refuses a lone surrogate' + ); + // A SQL NULL payload, an unknown submitter and a null one. + refused( + 'anon', + submitSql(s1, undefined), + '22023', + 'observation', + 'a NULL payload' + ); + refused( + 'anon', + submitSql(UNKNOWN, good()), + 'P0002', + 'submitter_id', + 'unknown submitter' + ); + refused( + 'anon', + submitSql(null, good()), + 'P0002', + 'submitter_id', + 'null submitter' + ); + assert.equal(count() - before, valid.length, 'refused calls write nothing'); + + // 7. The limits at their exact thresholds (54000). + // 120 an hour per submitter: 119 in the hour and 50 older, then the 120th passes. + const sLimit = register(); + backfill(sLimit, 119, '30 minutes'); + backfill(sLimit, 50, '61 minutes'); + submit(sLimit, good()); + refused( + 'anon', + submitSql(sLimit, good()), + '54000', + 'submitter_id', + '121st in an hour' + ); + assert.equal(count(`submitter_id = ${q(sLimit)}`), 170); + // 600 an hour per network: 599 in the hour and 100 older, then the 600th passes. + const NET_C = freshNet(); + const sBulk = register(); + backfill(sBulk, 599, '30 minutes', NET_C); + backfill(sBulk, 100, '61 minutes', NET_C); + const sNet = register(); + assert.equal( + submit(sNet, good(), NET_C).status, + 'quarantined', + '600th from a network' + ); + refused( + 'anon', + submitSql(sNet, good(), NET_C), + '54000', + 'network_hash', + '601st from a network' + ); + assert.equal( + submit(sNet, good(), NET_A).status, + 'quarantined', + 'another network passes' + ); + // 20 registers an hour per network: 19 in the hour and 5 older, then the 20th passes. + const NET_D = freshNet(); + sql(`INSERT INTO rate.extension_submitters (created_at, network_hash) + SELECT now() - interval '30 minutes', ${q(NET_D)} FROM generate_series(1, 19); + INSERT INTO rate.extension_submitters (created_at, network_hash) + SELECT now() - interval '61 minutes', ${q(NET_D)} FROM generate_series(1, 5)`); + assert.match( + register(NET_D), + /^[0-9a-f-]{36}$/, + '20th register from a network' + ); + refused( + 'anon', + registerSql(NET_D), + '54000', + 'network_hash', + '21st register from a network' + ); + // 600 registers a minute in all: fill to 599, then the 600th passes and the 601st is refused. + // The test's own registers age out of the minute as it runs, so they are moved out first. + sql(`UPDATE rate.extension_submitters SET created_at = now() - interval '2 minutes' + WHERE created_at > now() - interval '2 minutes'; + INSERT INTO rate.extension_submitters (created_at) + SELECT now() - interval '20 seconds' FROM generate_series(1, 599)`); + assert.match(register(), /^[0-9a-f-]{36}$/, '600th register in a minute'); + refused( + 'anon', + registerSql(freshNet()), + '54000', + 'submitter', + '601st register in a minute' + ); + sql(`UPDATE rate.extension_submitters SET created_at = now() - interval '2 minutes' + WHERE created_at > now() - interval '1 minute'`); + assert.match(register(), /^[0-9a-f-]{36}$/, 'the brake lifts after a minute'); + + // 8. Delete: no key; the submitter and its observations; the count; 0 for an unknown or null id. + const sDel = register(); + for (let i = 0; i < 3; i += 1) submit(sDel, good()); + assert.equal( + as('anon', `SELECT rate.delete_extension_submitter(${q(sDel)})`), + '3' + ); + assert.equal(count(`submitter_id = ${q(sDel)}`), 0); + assert.equal(count(`id = ${q(sDel)}`, 'rate.extension_submitters'), 0); + assert.equal( + as('anon', `SELECT rate.delete_extension_submitter(${q(sDel)})`), + '0' + ); + assert.equal( + as('anon', `SELECT rate.delete_extension_submitter(${q(UNKNOWN)})`), + '0' + ); + assert.equal( + as('authenticated', 'SELECT rate.delete_extension_submitter(NULL)'), + '0' + ); + const sEmpty = register(); + assert.equal( + as('authenticated', `SELECT rate.delete_extension_submitter(${q(sEmpty)})`), + '0' + ); + assert.equal(count(`id = ${q(sEmpty)}`, 'rate.extension_submitters'), 0); + refused( + 'anon', + submitSql(sDel, good()), + 'P0002', + 'submitter_id', + 'a deleted submitter' + ); + const sCascade = register(); + submit(sCascade, good()); + sql(`DELETE FROM rate.extension_submitters WHERE id = ${q(sCascade)}`); + assert.equal( + count(`submitter_id = ${q(sCascade)}`), + 0, + 'the cascade holds for service_role' + ); + + // 9. Purge: hashes after 28 days (observations and submitters), rows after 180, submitters + // without observations after 7. + const [p1, p2, p3, p4, p5, p6] = Array.from({ length: 6 }, () => register()); + const age = (ids, days) => + sql(`UPDATE rate.extension_submitters SET created_at = now() - interval '${days} days' + WHERE id IN (${ids.map(q).join(', ')})`); + age([p1, p2, p3], 200); + age([p4], 10); + age([p5], 3); + age([p6], 30); + backfill(p1, 1, '200 days', NET_A); + backfill(p1, 1, '30 days', NET_A); + backfill(p1, 1, '10 days', NET_A); + backfill(p3, 1, '190 days'); + backfill(p6, 1, '1 day', NET_A); + const total = count(); + for (const role of ['anon', 'authenticated', 'rate_worker']) + refused( + role, + 'SELECT rate.purge_extension_observations()', + '42501', + undefined, + `${role} purge` + ); + refused( + 'service_role', + 'SELECT rate.purge_extension_observations(0)', + '22023', + 'p_network_hash_days', + 'purge hash days' + ); + refused( + 'service_role', + 'SELECT rate.purge_extension_observations(28, NULL)', + '22023', + 'p_keep_days', + 'purge keep days' + ); + refused( + 'service_role', + 'SELECT rate.purge_extension_observations(28, 180, 0)', + '22023', + 'p_empty_submitter_days', + 'purge empty days' + ); + assert.deepEqual( + JSON.parse( + as('service_role', 'SELECT rate.purge_extension_observations()::text') + ), + { + network_hashes_cleared: 2, + submitter_network_hashes_cleared: 4, + observations_deleted: 2, + submitters_deleted: 3, + } + ); + assert.equal(count(), total - 2); + assert.equal( + sql(`SELECT string_agg(to_char(now() - received_at, 'DD') || ':' || (network_hash IS NOT NULL), + ',' ORDER BY received_at) FROM rate.extension_observations WHERE submitter_id = ${q(p1)}`), + '30:false,10:true' + ); + assert.equal( + sql(`SELECT string_agg(left(id::text, 8) || ':' || (network_hash IS NOT NULL), ',' + ORDER BY created_at) FROM rate.extension_submitters + WHERE id IN (${[p1, p2, p3, p4, p5, p6].map(q).join(', ')})`), + `${p1.slice(0, 8)}:false,${p6.slice(0, 8)}:false,${p5.slice(0, 8)}:true`, + 'p2 (never sent), p3 (emptied) and p4 (empty for 10 days) go; p1 and p6 keep rows, p5 is young' + ); + assert.deepEqual( + JSON.parse( + as('service_role', 'SELECT rate.purge_extension_observations()::text') + ), + { + network_hashes_cleared: 0, + submitter_network_hashes_cleared: 0, + observations_deleted: 0, + submitters_deleted: 0, + }, + 'a second purge finds nothing' + ); + // The race: a submit for an empty, purgeable submitter is still open when the purge runs. The + // purge skips the submitter it holds; the observation and its submitter stay. + const p7 = register(); + age([p7], 10); + const racing = spawn( + 'docker', + [ + 'exec', + '-i', + container, + 'psql', + '-XqAt', + '-v', + 'ON_ERROR_STOP=1', + '-U', + 'postgres', + ], + { stdio: ['pipe', 'pipe', 'pipe'] } + ); + let racingOut = ''; + racing.stdout.on('data', (d) => (racingOut += d)); + racing.stderr.on('data', (d) => (racingOut += d)); + const racingDone = new Promise((resolve) => racing.on('close', resolve)); + racing.stdin.end(`BEGIN; SET LOCAL ROLE anon; + SELECT 'submitted:' || (${submitSql(p7, good()).slice(7)} IS NOT NULL); + SELECT pg_sleep(3); COMMIT;`); + const deadline = Date.now() + 10_000; + while ( + count(`submitter_id = ${q(p7)}`) === 0 && + sql(`SELECT count(*) FROM pg_locks l JOIN pg_class c ON c.oid = l.relation + WHERE c.relname = 'extension_submitters' AND l.mode = 'RowShareLock' + AND l.pid <> pg_backend_pid()`) === '0' + ) { + assert.ok(Date.now() < deadline, 'the racing submit started'); + await new Promise((resolve) => setTimeout(resolve, 200)); + } + const raced = JSON.parse( + as('service_role', 'SELECT rate.purge_extension_observations()::text') + ); + assert.equal(await racingDone, 0, racingOut); + assert.match(racingOut, /submitted:true/); + assert.equal( + raced.submitters_deleted, + 0, + 'the purge skipped the held submitter' + ); + assert.equal( + count(`id = ${q(p7)}`, 'rate.extension_submitters'), + 1, + 'p7 stays' + ); + assert.equal(count(`submitter_id = ${q(p7)}`), 1, 'its observation stays'); + + // The lock order: a delete in the middle of a purge must not deadlock. A third session holds + // an old observation X and an old submitter Y, so the purge stops partway; then the user + // deletes S, whose old observation O the purge also clears. Purge and delete both take the + // submitter before its observations, so the delete waits for the purge and both finish. + const background = (input) => { + const child = spawn( + 'docker', + [ + 'exec', + '-i', + container, + 'psql', + '-XqAt', + '-v', + 'ON_ERROR_STOP=1', + '-U', + 'postgres', + ], + { stdio: ['pipe', 'pipe', 'pipe'] } + ); + let out = ''; + child.stdout.on('data', (d) => (out += d)); + child.stderr.on('data', (d) => (out += d)); + child.stdin.end(input); + return new Promise((resolve) => + child.on('close', (code) => resolve({ code, out })) + ); + }; + // Async between polls, so the background sessions' pipes keep flowing. + const waitFor = async (query, what) => { + const until = Date.now() + 15_000; + while (sql(query) === '0') { + assert.ok(Date.now() < until, what); + await new Promise((resolve) => setTimeout(resolve, 200)); + } + }; + const [dS, dY, dZ] = [register(), register(), register()]; + age([dS, dY, dZ], 30); + backfill(dS, 1, '30 days', NET_A); // O + backfill(dZ, 1, '30 days', NET_A); // X + const holder = background(`BEGIN; + SELECT 1 FROM rate.extension_observations WHERE submitter_id = ${q(dZ)} FOR UPDATE; + SELECT 1 FROM rate.extension_submitters WHERE id = ${q(dY)} FOR UPDATE; + SELECT pg_sleep(4); COMMIT;`); + await waitFor( + `SELECT count(*) FROM pg_stat_activity WHERE query LIKE '%pg_sleep(4)%' + AND pid <> pg_backend_pid() AND wait_event = 'PgSleep'`, + 'the holder took its locks' + ); + const purging = background( + `SET ROLE service_role; SELECT rate.purge_extension_observations()::text;` + ); + await waitFor( + `SELECT count(*) FROM pg_stat_activity WHERE query LIKE '%purge_extension_observations%' + AND pid <> pg_backend_pid() AND wait_event_type = 'Lock'`, + 'the purge waits on the holder' + ); + const deleting = background( + `SET ROLE anon; SELECT rate.delete_extension_submitter(${q(dS)});` + ); + const [held, purged, deleted] = await Promise.all([ + holder, + purging, + deleting, + ]); + assert.equal(held.code, 0, held.out); + assert.equal(purged.code, 0, `the purge finished: ${purged.out}`); + assert.equal(deleted.code, 0, `the delete finished: ${deleted.out}`); + assert.doesNotMatch(`${purged.out}${deleted.out}`, /deadlock/); + assert.equal(deleted.out.trim(), '1', 'the delete took O with S'); + assert.equal(count(`id = ${q(dS)}`, 'rate.extension_submitters'), 0); + + // 10. The tables: anon and authenticated reach nothing; rate_worker reads and sets status only + // and has nothing on the key; service_role writes directly within the CHECKs. + const tables = [ + 'rate.extension_ingest_secret', + 'rate.extension_submitters', + 'rate.extension_observations', + ]; + for (const role of ['anon', 'authenticated']) { + for (const table of tables) { + refused( + role, + `SELECT count(*) FROM ${table}`, + '42501', + undefined, + `${role} select ${table}` + ); + refused( + role, + `DELETE FROM ${table}`, + '42501', + undefined, + `${role} delete ${table}` + ); + refused( + role, + `TRUNCATE ${table} CASCADE`, + '42501', + undefined, + `${role} truncate ${table}` + ); + } + refused( + role, + `UPDATE rate.extension_submitters SET created_at = now()`, + '42501', + undefined, + `${role} update submitters` + ); + refused( + role, + `UPDATE rate.extension_observations SET status = 'rejected'`, + '42501', + undefined, + `${role} update observations` + ); + refused( + role, + `UPDATE rate.extension_ingest_secret SET updated_at = now()`, + '42501', + undefined, + `${role} update the key` + ); + refused( + role, + 'INSERT INTO rate.extension_submitters DEFAULT VALUES', + '42501', + undefined, + `${role} insert submitter` + ); + refused( + role, + `INSERT INTO rate.extension_ingest_secret (key_sha256) VALUES (${q(sha('x'))})`, + '42501', + undefined, + `${role} insert key` + ); + refused( + role, + `INSERT INTO rate.extension_observations (submitter_id, schema_version, + marketplace, listing_url, external_item_id, price_vnd, stock_state, observed_at, extractor) + VALUES (${q(s1)}, 1, 'shopee', 'https://shopee.vn/x', 'x', 1000, 'unknown', now(), + '{"source": "dom", "extension_version": "1"}')`, + '42501', + undefined, + `${role} insert observation` + ); + } + refused( + 'rate_worker', + 'SELECT count(*) FROM rate.extension_ingest_secret', + '42501', + undefined, + 'rate_worker reads no key' + ); + assert.ok( + Number( + as('rate_worker', 'SELECT count(*) FROM rate.extension_observations') + ) > 0 + ); + assert.ok( + Number( + as('rate_worker', 'SELECT count(*) FROM rate.extension_submitters') + ) > 0 + ); + assert.equal( + as( + 'rate_worker', + `UPDATE rate.extension_observations SET status = 'corroborated' + WHERE id = ${r1.id} RETURNING status` + ), + 'corroborated' + ); + refused( + 'rate_worker', + `UPDATE rate.extension_observations SET status = 'promoted' + WHERE id = ${r1.id}`, + '23514', + undefined, + 'status is checked' + ); + refused( + 'rate_worker', + `UPDATE rate.extension_observations SET price_vnd = 1000 + WHERE id = ${r1.id}`, + '42501', + undefined, + 'rate_worker price' + ); + refused( + 'rate_worker', + `UPDATE rate.extension_observations SET network_hash = NULL + WHERE id = ${r1.id}`, + '42501', + undefined, + 'rate_worker hash' + ); + refused( + 'rate_worker', + `UPDATE rate.extension_submitters SET created_at = now()`, + '42501', + undefined, + 'rate_worker submitter' + ); + refused( + 'rate_worker', + `DELETE FROM rate.extension_observations WHERE id = ${r1.id}`, + '42501', + undefined, + 'rate_worker delete' + ); + refused( + 'rate_worker', + 'INSERT INTO rate.extension_submitters DEFAULT VALUES', + '42501', + undefined, + 'rate_worker insert' + ); + const direct = `INSERT INTO rate.extension_observations (submitter_id, schema_version, + marketplace, listing_url, external_item_id, price_vnd, stock_state, observed_at, extractor) + VALUES (${q(s1)}, 1, 'shopee', URL, 'x', 1000, 'unknown', now(), + '{"source": "dom", "extension_version": "1"}') RETURNING status`; + assert.equal( + as('service_role', direct.replace('URL', q('https://shopee.vn/x'))), + 'quarantined', + 'service_role inserts directly' + ); + for (const url of [ + 'https://lazada.vn/x', + 'https://shopee.vn/x;a=1', + `https://shopee.vn/x${ch(0xa0)}`, + ]) + refused( + 'service_role', + direct.replace('URL', q(url)), + '23514', + undefined, + `the URL CHECK holds service_role: ${url}` + ); + refused( + 'service_role', + `INSERT INTO rate.extension_submitters (network_hash) VALUES ('x')`, + '23514', + undefined, + 'the submitter hash CHECK' + ); + checkAll('after the exercise'); + + // 11. The rollback twice; everything goes; the proposal again needs the key row again. + sql(rollback); + sql(rollback); + assert.equal( + sql(`SELECT count(*) FROM pg_class WHERE relnamespace = 'rate'::regnamespace + AND relname LIKE 'extension\\_%'`), + '0' + ); + assert.equal( + sql(`SELECT count(*) FROM pg_proc WHERE pronamespace = 'rate'::regnamespace + AND proname IN ('register_extension_submitter', 'submit_extension_observation', + 'delete_extension_submitter', 'purge_extension_observations')`), + '0' + ); + sql(proposal); + assert.equal(sql(check('V12')), '1', 'V12 sees the missing key row'); + refused( + 'anon', + registerSql(NET_A), + '42501', + 'ingest_key', + 'no key row after a re-apply' + ); + sql( + `INSERT INTO rate.extension_ingest_secret (key_sha256) VALUES (${q(sha(KEY))})` + ); + checkAll('after the rollback and a re-apply'); + const again = register(); + assert.equal(submit(again, good()).status, 'quarantined'); + sql(proposal); + checkAll('after a re-apply over data'); + assert.equal(count(), 1, 'a re-apply keeps rows'); + assert.equal( + count('true', 'rate.extension_ingest_secret'), + 1, + 'a re-apply keeps the key' + ); + + console.log( + `extension observations proposal: ${valid.length} valid and ${bad.length} refused payloads agree with parseExtensionObservationV1; ingest key, exact limits, delete, purge and its race, grants, verify and rollback replayed` + ); +} finally { + removeContainer(); +} From d2445e2a618c65dc792e89a918dcdb091efaac3b Mon Sep 17 00:00:00 2001 From: JOY <5027251+JOY@users.noreply.github.com> Date: Wed, 4 Nov 2026 20:48:18 +0700 Subject: [PATCH 5/7] feat(extension): opted-in installs send observation v1, with disclosures (#1610) * feat(extension): opted-in installs send observation v1, with disclosures The opt-in now sends ExtensionObservation v1 to the observation route with a random submitter id from the submitter route, instead of the legacy crowd ingest (no title, no ratings). Turning the opt-in off deletes the id and every observation sent with it. The web routes stay off (RATE_EXTENSION_OBSERVATIONS), so nothing is stored until JOY turns them on. The opt-in text, the consent page, the privacy policy (vi, en, cs), CHROMEWEBSTORE.md and the changelog name the v1 fields, the submitter id and the salted network-range hash with its retention; a test ties the texts to the v1 keys. Refs #1606 * fix(extension): observation client with kept erasure, holds and consent v2 - observation-client.ts runs register, send and erase on one chain: two product pages register one id, and an opt-out waits for a send in flight; an id issued after the opt-out is erased at once. - Erasures stay queued until the server confirms (200) and are retried at start-up and on the next send; the stored id carries the server that issued it, and an id from another server is erased there. - 429 holds for its Retry-After; routes off (404 NOT_ENABLED) or 503 hold an hour, so opted-in installs stop asking. - An opt-in given to the older crowd signal is not consent to v1: it reads as off, and an update reopens the consent page. - Disclosures: stock quantity, the server-derived network hash, the IP sentence, one name for the contributor ID, the uninstall case, and a release test step for erasure. - Behavioural tests for the client with stubbed storage and fetch. Refs #1606 Co-Authored-By: Claude Opus 5.5 --- apps/extension/CHROMEWEBSTORE.md | 6 +- apps/extension/RELEASE.md | 5 +- apps/extension/src/_locales/cs/messages.json | 22 +- apps/extension/src/_locales/en/messages.json | 22 +- apps/extension/src/_locales/vi/messages.json | 22 +- .../src/background/observation-client.ts | 274 ++++++++++++++ .../src/background/service-worker.ts | 92 ++++- apps/extension/src/security-guards.ts | 38 +- apps/extension/src/test-extension-pipeline.ts | 335 +++++++++++++++++- apps/extension/src/types.ts | 4 + apps/web/locales/cs.json | 2 +- apps/web/locales/en.json | 2 +- apps/web/locales/vi.json | 2 +- packages/shared-data/src/constants/routes.ts | 2 + 14 files changed, 776 insertions(+), 52 deletions(-) create mode 100644 apps/extension/src/background/observation-client.ts diff --git a/apps/extension/CHROMEWEBSTORE.md b/apps/extension/CHROMEWEBSTORE.md index 56bc201b9..01a030a9c 100644 --- a/apps/extension/CHROMEWEBSTORE.md +++ b/apps/extension/CHROMEWEBSTORE.md @@ -45,9 +45,9 @@ This table lists exactly what `manifest.json` requests; `test-extension-pipeline 1. **Where it runs:** the content script loads on the 10 supported stores listed above (`shopee.vn`, `lazada.vn`, `tiki.vn`, TikTok Shop product pages on `www.tiktok.com` and `shop.tiktok.com`, `cellphones.com.vn`, `thegioididong.com`, `fptshop.com.vn`, `hoanghamobile.com`, `didongviet.vn`, `phongvu.vn`). It acts only on pages it recognises as a product page. The extension itself never opens or fetches pages the user did not open, never works around a store's bot checks and never solves captchas. 2. **Lookup (on by default, no opt-in):** on a recognised product page the extension sends `GET /api/v1/rate/extension/lookup` with the store, the product URL with query parameters and tracking tokens (`utm_*`, `sp_atk`, `xptdk`, `fbclid`, `gclid`) stripped locally, the product title, the store's item id and shop id, the selected variant, the listed price and the original price. The request is read-only and carries no account, name, email or cookies. -3. **Opt-in crowd signal (off by default):** only when the user chooses "Có" / "Yes" on the consent page opened once at first install, or turns on "Đóng góp phát hiện sản phẩm mới (chỉ khi bạn bật)" (English: "Help find new products (only when you turn it on)") in the Side Panel, the extension also sends `POST /api/v1/rate/extension/ingest` with the store, item id, shop id, cleaned URL, title, selected variant, price, original price, rating and rating count. The toggle's label lists these same fields (DOS/Rate#1230). Submissions go to a moderation queue, stored with the time the server received them; Rate.vn's servers may re-check the listing on the store. They never change public prices directly. Closing the consent page keeps the opt-in off, and updates or reloads do not open it again (DOS/Rate#1603). -4. **Stored in the browser only:** the server address and the opt-in choice (`chrome.storage.local`), and a session-only cache of the products viewed in each tab, the latest side panel search, their lookup results and a lookup wait time after a rate limit (`chrome.storage.session`). -5. **IP address:** like any web request, the Rate.vn server receives the user's IP address. The lookup and opt-in routes use it only for a per-minute request limit held in memory, and Rate.vn's code does not store it; the hosting provider keeps standard access logs for a short time. +3. **Opt-in observation (off by default):** only when the user chooses "Có" / "Yes" on the consent page opened once at first install, or turns on "Đóng góp phát hiện sản phẩm mới (chỉ khi bạn bật)" (English: "Help find new products (only when you turn it on)") in the Side Panel, the extension sends an ExtensionObservation v1 to `POST /api/v1/rate/extension/observation`: the store, the cleaned URL (no query), the item and shop ids, the variant id (SKU), the selected variant, the price, the original price, the stock status and quantity, the time the user saw the price and the extension version, with how the page was read (DOS/Rate#1606). It carries a random submitter id for the install, obtained once from `POST /api/v1/rate/extension/submitter`. No title, no ratings, no account data. The toggle's label and the consent page list these same fields; a test ties them to the payload. Observations go to a quarantine, stored with the time received and a salted hash of the network range (never the IP) to tell independent senders apart; the hash is deleted after 28 days and observations after 180 days. Rate.vn's servers may re-check the listing on the store. They never change public prices directly. Turning the opt-in off calls `DELETE /api/v1/rate/extension/submitter`, which deletes the id and every observation sent with it; the extension keeps retrying that erasure until the server confirms it. Uninstalling without turning it off leaves the observations to the 180-day purge. An opt-in given to the older crowd signal is not taken as consent to this: after the update it is off and the consent page asks again. Closing the consent page keeps the opt-in off, and updates or reloads do not open it again (DOS/Rate#1603). +4. **Stored in the browser only:** the server address, the opt-in choice and, while the user contributes, the submitter id (`chrome.storage.local`), and a session-only cache of the products viewed in each tab, the latest side panel search, their lookup results and a lookup wait time after a rate limit (`chrome.storage.session`). +5. **IP address:** like any web request, the Rate.vn server receives the user's IP address. The lookup and observation routes use it in memory for a per-minute request limit; the observation routes also derive the salted network-range hash above from it. Rate.vn's code never stores the IP itself; the hosting provider keeps standard access logs for a short time. 6. **Side panel requests:** text the user types into the side panel search is sent to the same lookup endpoint; the side panel also reads public deals and merchant lists from the Rate.vn API. 7. **Privacy policy:** https://rate.vn/privacy-policy has a section on the extension. diff --git a/apps/extension/RELEASE.md b/apps/extension/RELEASE.md index 2c44ff297..83a56311d 100644 --- a/apps/extension/RELEASE.md +++ b/apps/extension/RELEASE.md @@ -26,6 +26,7 @@ Release-readiness check of the extension built from `dev` at `fc9b820d` on 2026- - **Product pages:** on one product page each of Shopee, Tiki and CellphoneS, the pill appears and the side panel opens. - **Language:** set Chrome to English, then Czech. The panel and the verdict follow the browser language, and en/cs never show the Vietnamese review summary (#1385). - **Opt-in:** the crowd opt-in is off by default, and its label names the fields it sends (#1230, #1241). + - **Contribution erasure:** with the observation routes on, opt in, open a product page, then turn the opt-in off: the server holds no observation for the install's id afterwards, and with the network off the erasure is retried at the next start (#1606). - **Consent page:** a fresh install opens the consent page once, with "No" focused. Closing it leaves the opt-in off; "Yes" turns on the side panel toggle; reloading or updating the extension does not open it again (#1603). 5. Update the store listing: - Description and screenshots match the panel as it is now (the price list and the states from #1319). @@ -45,7 +46,7 @@ Release-readiness check of the extension built from `dev` at `fc9b820d` on 2026- - Biểu tượng trên thanh công cụ cho biết nhanh giá đang xem có tốt không. Tin bán đã hết hàng không còn được tính vào giá so sánh. - Nhận diện sản phẩm chính xác hơn: tra "iPhone 16" không còn ra iPhone 16e. Trang danh mục và trang thương hiệu không bị nhầm là trang sản phẩm. - Gian hàng chưa có đánh giá được phân tích sẽ ghi "Chưa có dữ liệu", không hiện con số rủi ro. -- Lựa chọn "Đóng góp phát hiện sản phẩm mới" ghi rõ những gì được gửi đi và vẫn tắt mặc định. Lần cài đầu tiên, tiện ích hỏi bạn một lần. TikTok chỉ còn chạy trên trang sản phẩm. +- Lựa chọn "Đóng góp phát hiện sản phẩm mới" ghi rõ những gì được gửi đi và vẫn tắt mặc định. Lần cài đầu tiên, tiện ích hỏi bạn một lần. Khi bật, tiện ích gửi một quan sát gọn (giá, tồn kho, SKU) thay cho số sao và lượt đánh giá; tắt là xóa mọi thứ đã gửi. TikTok chỉ còn chạy trên trang sản phẩm. ### English @@ -56,4 +57,4 @@ Release-readiness check of the extension built from `dev` at `fc9b820d` on 2026- - The toolbar badge shows at a glance whether the price you see is a good one. Sold-out listings no longer count towards the price comparison. - More accurate product matching: "iPhone 16" no longer matches the iPhone 16e. Category and brand pages are no longer mistaken for product pages. - Sellers with no analysed reviews show "No data yet" instead of a risk number. -- The "Help find new products" option names exactly what it sends and is still off by default. On first install, the extension asks you once. On TikTok, the extension runs only on product pages. +- The "Help find new products" option names exactly what it sends and is still off by default. On first install, the extension asks you once. When on, it sends a minimal observation (price, stock, SKU) instead of ratings; turning it off deletes everything it sent. On TikTok, the extension runs only on product pages. diff --git a/apps/extension/src/_locales/cs/messages.json b/apps/extension/src/_locales/cs/messages.json index f61b6bbf7..0b0a958ea 100644 --- a/apps/extension/src/_locales/cs/messages.json +++ b/apps/extension/src/_locales/cs/messages.json @@ -53,7 +53,7 @@ "message": "Pomoc s hledáním nových produktů (jen když ji zapnete):" }, "optInBody": { - "message": "Když je zapnuto, každá nabídka, kterou otevřete, se také odešle do nezávislé fronty ke kontrole Rate.vn s těmito údaji: obchod, ID produktu, ID obchodu, URL bez sledovacích parametrů, název produktu, zvolená varianta, cena, původní cena, počet hvězdiček a počet recenzí. Bez účtu, jména, e-mailu a cookies." + "message": "Když je zapnuto, každá nabídka, kterou otevřete, se také odešle do nezávislé fronty ke kontrole Rate.vn s těmito údaji: obchod, ID produktu, ID obchodu, URL bez sledovacích parametrů, ID varianty (SKU), zvolená varianta, cena, původní cena, stav a počet kusů skladem, čas, kdy jste cenu viděli, verze rozšíření, a navíc náhodné ID přispěvatele této instalace. Server odvodí a 28 dní uchovává otisk rozsahu vaší sítě (ne IP adresa), odeslané záznamy uchovává 180 dní a může nabídku v obchodě znovu ověřit. Vypnutím se toto ID i vše odeslané smaže. Bez účtu, jména, e-mailu a cookies." }, "crowdFieldPlatform": { "message": "obchod" @@ -684,6 +684,24 @@ "message": "Zásady ochrany soukromí" }, "onboardingOptInBody": { - "message": "Když zvolíte „Ano“, každá nabídka, kterou otevřete, se také odešle do fronty ke kontrole Rate.vn s těmito údaji: obchod, ID produktu, ID obchodu, URL bez sledovacích parametrů, název produktu, zvolená varianta, cena, původní cena, počet hvězdiček a počet recenzí. Servery Rate.vn mohou nabídku v obchodě znovu ověřit. Bez účtu, jména, e-mailu a cookies." + "message": "Když zvolíte „Ano“, každá nabídka, kterou otevřete, se také odešle do fronty ke kontrole Rate.vn s těmito údaji: obchod, ID produktu, ID obchodu, URL bez sledovacích parametrů, ID varianty (SKU), zvolená varianta, cena, původní cena, stav a počet kusů skladem, čas, kdy jste cenu viděli, verze rozšíření, a navíc náhodné ID přispěvatele této instalace. Server odvodí a 28 dní uchovává otisk rozsahu vaší sítě (ne IP adresa), odeslané záznamy uchovává 180 dní a může nabídku v obchodě znovu ověřit. Vypnutím se toto ID i vše odeslané smaže. Bez účtu, jména, e-mailu a cookies." + }, + "observationFieldSku": { + "message": "ID varianty (SKU)" + }, + "observationFieldStock": { + "message": "stav a počet kusů skladem" + }, + "observationFieldObservedAt": { + "message": "čas, kdy jste cenu viděli" + }, + "observationFieldExtractor": { + "message": "verze rozšíření" + }, + "observationFieldSubmitter": { + "message": "náhodné ID přispěvatele této instalace" + }, + "observationFieldNetwork": { + "message": "otisk rozsahu vaší sítě (ne IP adresa)" } } diff --git a/apps/extension/src/_locales/en/messages.json b/apps/extension/src/_locales/en/messages.json index 2843358d2..1328a9807 100644 --- a/apps/extension/src/_locales/en/messages.json +++ b/apps/extension/src/_locales/en/messages.json @@ -53,7 +53,7 @@ "message": "Help find new products (only when you turn it on):" }, "optInBody": { - "message": "When on, every listing you open is also sent to the independent Rate.vn review queue with: store, product ID, shop ID, URL with tracking parameters removed, product name, selected variant, price, original price, star rating and number of reviews. No account, name, email or cookies are included." + "message": "When on, every listing you open is also sent to the independent Rate.vn review queue with: store, product ID, shop ID, URL with tracking parameters removed, variant ID (SKU), selected variant, price, original price, stock status and quantity, the time you saw the price, the extension version, plus a random contributor ID for this install. The server derives and keeps a hash of your network range (not your IP) for 28 days, keeps what you sent for 180 days, and may re-check that listing on the store. Turning it off deletes that ID and everything sent with it. No account, name, email or cookies are included." }, "crowdFieldPlatform": { "message": "store" @@ -684,6 +684,24 @@ "message": "Privacy policy" }, "onboardingOptInBody": { - "message": "If you choose \"Yes\", every listing you open is also sent to the Rate.vn review queue with: store, product ID, shop ID, URL with tracking parameters removed, product name, selected variant, price, original price, star rating and number of reviews. Rate.vn's servers may re-check that listing on the store. No account, name, email or cookies are included." + "message": "If you choose \"Yes\", every listing you open is also sent to the Rate.vn review queue with: store, product ID, shop ID, URL with tracking parameters removed, variant ID (SKU), selected variant, price, original price, stock status and quantity, the time you saw the price, the extension version, plus a random contributor ID for this install. The server derives and keeps a hash of your network range (not your IP) for 28 days, keeps what you sent for 180 days, and may re-check that listing on the store. Turning it off deletes that ID and everything sent with it. No account, name, email or cookies are included." + }, + "observationFieldSku": { + "message": "variant ID (SKU)" + }, + "observationFieldStock": { + "message": "stock status and quantity" + }, + "observationFieldObservedAt": { + "message": "the time you saw the price" + }, + "observationFieldExtractor": { + "message": "the extension version" + }, + "observationFieldSubmitter": { + "message": "a random contributor ID for this install" + }, + "observationFieldNetwork": { + "message": "a hash of your network range (not your IP)" } } diff --git a/apps/extension/src/_locales/vi/messages.json b/apps/extension/src/_locales/vi/messages.json index 355d9682a..70db62958 100644 --- a/apps/extension/src/_locales/vi/messages.json +++ b/apps/extension/src/_locales/vi/messages.json @@ -53,7 +53,7 @@ "message": "Đóng góp phát hiện sản phẩm mới (chỉ khi bạn bật):" }, "optInBody": { - "message": "Khi bật, mỗi tin bán bạn mở còn được gửi vào hàng đợi kiểm duyệt độc lập của Rate.vn với: sàn, mã sản phẩm, mã gian hàng, URL đã cắt tham số theo dõi, tên sản phẩm, phiên bản đang chọn, giá bán, giá gốc, số sao và số lượt đánh giá. Không kèm tài khoản, tên, email hay cookie." + "message": "Khi bật, mỗi tin bán bạn mở còn được gửi vào hàng đợi kiểm duyệt độc lập của Rate.vn với: sàn, mã sản phẩm, mã gian hàng, URL đã cắt tham số theo dõi, mã SKU của phân loại, phiên bản đang chọn, giá bán, giá gốc, tình trạng và số lượng còn hàng, thời điểm bạn thấy giá, phiên bản tiện ích, kèm một mã người đóng góp ngẫu nhiên của lượt cài này. Máy chủ tính và giữ mã băm dải mạng của bạn (không phải IP) trong 28 ngày, giữ tin đã gửi 180 ngày, và có thể kiểm tra lại tin bán đó trên sàn. Tắt đóng góp là xóa mã đó và mọi tin đã gửi. Không kèm tài khoản, tên, email hay cookie." }, "crowdFieldPlatform": { "message": "sàn" @@ -684,6 +684,24 @@ "message": "Chính sách bảo mật" }, "onboardingOptInBody": { - "message": "Nếu bạn chọn \"Có\", mỗi tin bán bạn mở còn được gửi vào hàng đợi kiểm duyệt của Rate.vn với: sàn, mã sản phẩm, mã gian hàng, URL đã cắt tham số theo dõi, tên sản phẩm, phiên bản đang chọn, giá bán, giá gốc, số sao và số lượt đánh giá. Máy chủ Rate.vn có thể kiểm tra lại tin bán đó trên sàn. Không kèm tài khoản, tên, email hay cookie." + "message": "Nếu bạn chọn \"Có\", mỗi tin bán bạn mở còn được gửi vào hàng đợi kiểm duyệt của Rate.vn với: sàn, mã sản phẩm, mã gian hàng, URL đã cắt tham số theo dõi, mã SKU của phân loại, phiên bản đang chọn, giá bán, giá gốc, tình trạng và số lượng còn hàng, thời điểm bạn thấy giá, phiên bản tiện ích, kèm một mã người đóng góp ngẫu nhiên của lượt cài này. Máy chủ tính và giữ mã băm dải mạng của bạn (không phải IP) trong 28 ngày, giữ tin đã gửi 180 ngày, và có thể kiểm tra lại tin bán đó trên sàn. Tắt đóng góp là xóa mã đó và mọi tin đã gửi. Không kèm tài khoản, tên, email hay cookie." + }, + "observationFieldSku": { + "message": "mã SKU của phân loại" + }, + "observationFieldStock": { + "message": "tình trạng và số lượng còn hàng" + }, + "observationFieldObservedAt": { + "message": "thời điểm bạn thấy giá" + }, + "observationFieldExtractor": { + "message": "phiên bản tiện ích" + }, + "observationFieldSubmitter": { + "message": "một mã người đóng góp ngẫu nhiên của lượt cài này" + }, + "observationFieldNetwork": { + "message": "mã băm dải mạng của bạn (không phải IP)" } } diff --git a/apps/extension/src/background/observation-client.ts b/apps/extension/src/background/observation-client.ts new file mode 100644 index 000000000..42ebef87a --- /dev/null +++ b/apps/extension/src/background/observation-client.ts @@ -0,0 +1,274 @@ +/** + * The opt-in observation client (DOS/Rate#1606): this install's + * pseudonymous submitter id, ExtensionObservation v1 sends, and the erasure + * that turning the opt-in off promises. Storage and fetch are injected so + * the behaviour is tested without a browser. + * + * Every operation runs one after another on one chain, so an opt-out waits + * for a send in flight, and two product pages never register two ids. + */ +import type { ExtensionObservationV1 } from '@repo/shared-data/rules'; + +export interface ObservationStorage { + get(key: string): Promise>; + set(items: Record): Promise; + remove(key: string): Promise; +} + +export interface ObservationClientDeps { + storage: ObservationStorage; + fetch: (url: string, init: RequestInit) => Promise; + now: () => number; + extensionVersion: string; + /** Read at send time and again after a registration. */ + optedIn: () => Promise; + submitterPath: string; + observationPath: string; +} + +/** The stored id, with the server that issued it. */ +export const SUBMITTER_KEY = 'observationSubmitter'; +/** Ids whose erasure the server has not confirmed yet. */ +export const PENDING_ERASURE_KEY = 'observationPendingErasure'; +/** No register or observation before this time (ms). */ +export const HOLD_UNTIL_KEY = 'observationHoldUntil'; + +/** How long to wait after a 429 without Retry-After, or while the routes are off. */ +export const OBSERVATION_DEFAULT_HOLD_SECONDS = 60; +export const OBSERVATION_OFF_HOLD_SECONDS = 3600; +const MAX_HOLD_SECONDS = 3600; +const MAX_PENDING_ERASURES = 20; +const TIMEOUT_MS = 8_000; + +const ID_RE = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/; + +interface SubmitterRecord { + id: string; + apiBaseUrl: string; +} + +function asRecord(raw: unknown): SubmitterRecord | null { + const r = raw as Partial | null; + return r && + typeof r === 'object' && + typeof r.id === 'string' && + ID_RE.test(r.id) && + typeof r.apiBaseUrl === 'string' && + r.apiBaseUrl.length > 0 + ? { id: r.id, apiBaseUrl: r.apiBaseUrl } + : null; +} + +export class ObservationClient { + private chain: Promise = Promise.resolve(); + + constructor(private readonly deps: ObservationClientDeps) {} + + private run(task: () => Promise): Promise { + const result = this.chain.then(task, task); + this.chain = result.catch(() => undefined); + return result; + } + + private async read(key: string, parse: (raw: unknown) => T): Promise { + const stored = await this.deps.storage.get(key); + return parse(stored[key]); + } + + private headers(json: boolean): Record { + return json + ? { + 'Content-Type': 'application/json', + 'X-Rate-Extension-Version': this.deps.extensionVersion, + } + : { 'X-Rate-Extension-Version': this.deps.extensionVersion }; + } + + private async hold(seconds: number): Promise { + const ms = Math.min(Math.max(seconds, 1), MAX_HOLD_SECONDS) * 1000; + await this.deps.storage.set({ [HOLD_UNTIL_KEY]: this.deps.now() + ms }); + } + + private async holdFor(res: Response): Promise { + if (res.status === 429) { + const header = res.headers.get('Retry-After'); + const seconds = + header && /^\d{1,6}$/.test(header.trim()) + ? Number(header.trim()) + : OBSERVATION_DEFAULT_HOLD_SECONDS; + await this.hold(seconds); + } else if (res.status === 503) { + await this.hold(OBSERVATION_OFF_HOLD_SECONDS); + } + } + + private async held(): Promise { + const until = await this.read(HOLD_UNTIL_KEY, (v) => + typeof v === 'number' && Number.isFinite(v) ? v : 0 + ); + return until > this.deps.now(); + } + + private async code(res: Response): Promise { + const body = (await res.json().catch(() => null)) as { + code?: unknown; + } | null; + return typeof body?.code === 'string' ? body.code : null; + } + + private async pending(): Promise { + return this.read(PENDING_ERASURE_KEY, (v) => + Array.isArray(v) + ? v.map(asRecord).filter((r): r is SubmitterRecord => r !== null) + : [] + ); + } + + private async queueErasure(record: SubmitterRecord): Promise { + const list = (await this.pending()).filter((r) => r.id !== record.id); + list.push(record); + await this.deps.storage.set({ + [PENDING_ERASURE_KEY]: list.slice(-MAX_PENDING_ERASURES), + }); + } + + /** + * Sends every queued erasure. One the server confirmed (200), or refused + * as malformed (400), leaves the queue; any other answer or a network + * error keeps it for the next try. + */ + private async flushErasures(): Promise { + const list = await this.pending(); + if (list.length === 0) return; + const left: SubmitterRecord[] = []; + for (const record of list) { + try { + const res = await this.deps.fetch( + `${record.apiBaseUrl}${this.deps.submitterPath}`, + { + method: 'DELETE', + headers: this.headers(true), + signal: AbortSignal.timeout(TIMEOUT_MS), + body: JSON.stringify({ submitterId: record.id }), + } + ); + if (res.status !== 200 && res.status !== 400) left.push(record); + } catch { + left.push(record); + } + } + if (left.length > 0) { + await this.deps.storage.set({ [PENDING_ERASURE_KEY]: left }); + } else { + await this.deps.storage.remove(PENDING_ERASURE_KEY); + } + } + + /** The stored id for this server; an id from another server is erased there. */ + private async currentId(apiBaseUrl: string): Promise { + const record = await this.read(SUBMITTER_KEY, asRecord); + if (!record) return null; + if (record.apiBaseUrl === apiBaseUrl) return record.id; + await this.deps.storage.remove(SUBMITTER_KEY); + await this.queueErasure(record); + await this.flushErasures(); + return null; + } + + private async register(apiBaseUrl: string): Promise { + const res = await this.deps.fetch( + `${apiBaseUrl}${this.deps.submitterPath}`, + { + method: 'POST', + headers: this.headers(false), + signal: AbortSignal.timeout(TIMEOUT_MS), + } + ); + if (!res.ok) { + if (res.status === 404) { + // NOT_ENABLED: the routes are off; ask again in an hour. + await this.hold(OBSERVATION_OFF_HOLD_SECONDS); + } else { + await this.holdFor(res); + } + return null; + } + const body = (await res.json().catch(() => null)) as { + submitterId?: unknown; + } | null; + const id = body?.submitterId; + return typeof id === 'string' && ID_RE.test(id) ? id : null; + } + + /** Sends one observation for an opted-in install; never throws. */ + send(apiBaseUrl: string, observation: ExtensionObservationV1): Promise { + return this.run(async () => { + try { + await this.flushErasures(); + if (!(await this.deps.optedIn()) || (await this.held())) return; + let id = await this.currentId(apiBaseUrl); + if (!id) { + id = await this.register(apiBaseUrl); + if (!id) return; + // Turned off while the id was being issued: erase it at once. + if (!(await this.deps.optedIn())) { + await this.queueErasure({ id, apiBaseUrl }); + await this.flushErasures(); + return; + } + await this.deps.storage.set({ [SUBMITTER_KEY]: { id, apiBaseUrl } }); + } + const res = await this.deps.fetch( + `${apiBaseUrl}${this.deps.observationPath}`, + { + method: 'POST', + headers: this.headers(true), + signal: AbortSignal.timeout(TIMEOUT_MS), + body: JSON.stringify({ submitterId: id, observation }), + } + ); + if (res.status === 404) { + const code = await this.code(res); + if (code === 'UNKNOWN_SUBMITTER') { + // Deleted or purged on the server: get a new id next time. + await this.deps.storage.remove(SUBMITTER_KEY); + } else if (code === 'NOT_ENABLED') { + await this.hold(OBSERVATION_OFF_HOLD_SECONDS); + } + } else { + await this.holdFor(res); + } + } catch { + // Non-blocking: a lost observation is fine, a lost erasure is queued. + } + }); + } + + /** + * The opt-in was turned off: forget the id here and erase it on the server + * that issued it, retrying until the server confirms. + */ + forget(): Promise { + return this.run(async () => { + try { + const record = await this.read(SUBMITTER_KEY, asRecord); + await this.deps.storage.remove(SUBMITTER_KEY); + if (record) await this.queueErasure(record); + await this.flushErasures(); + } catch { + // The queued erasure is retried on the next start or send. + } + }); + } + + /** Retries queued erasures (at start-up). */ + retryErasures(): Promise { + return this.run(async () => { + try { + await this.flushErasures(); + } catch { + // Kept for the next try. + } + }); + } +} diff --git a/apps/extension/src/background/service-worker.ts b/apps/extension/src/background/service-worker.ts index 50f560419..7e2509345 100644 --- a/apps/extension/src/background/service-worker.ts +++ b/apps/extension/src/background/service-worker.ts @@ -10,7 +10,7 @@ import { sanitizeProductUrl, } from '../extractors/ecommerce-detector.js'; import { - buildCrowdIngestPayload, + buildExtensionObservationV1, buildLookupParams, DEFAULT_API_BASE_URL, isOnboardingSender, @@ -21,6 +21,7 @@ import { selectFreshestSessionSignal, } from '../security-guards.js'; import { LookupCooldown } from '../rate-limit.js'; +import { ObservationClient } from './observation-client.js'; import { t } from '../i18n.js'; import { resolveActionBadge } from '../verdict-display.js'; @@ -29,17 +30,27 @@ const DEFAULT_SETTINGS: ExtensionSettings = { crowdContributeOptIn: false, }; +/** + * The consent the opt-in was given under. Version 2 is ExtensionObservation + * v1 with its submitter id (DOS/Rate#1606); an opt-in given to the older + * crowd signal counts as off until the user says yes again. + */ +const OPT_IN_CONSENT_VERSION = 2; + async function getSettings(): Promise { try { const stored = await chrome.storage.local.get([ 'apiBaseUrl', 'crowdContributeOptIn', + 'crowdConsentVersion', ]); return { apiBaseUrl: normalizeAllowedApiBaseUrl( typeof stored.apiBaseUrl === 'string' ? stored.apiBaseUrl : null ), - crowdContributeOptIn: Boolean(stored.crowdContributeOptIn ?? false), + crowdContributeOptIn: + stored.crowdContributeOptIn === true && + stored.crowdConsentVersion === OPT_IN_CONSENT_VERSION, }; } catch { return DEFAULT_SETTINGS; @@ -159,24 +170,31 @@ async function maybeSendOptInCrowdSignal( // A 429 may have come back while the settings were read. if ((await lookupCooldown.secondsLeft()) > 0) return; - try { - await fetch( - `${settings.apiBaseUrl}${RATE_API_V1_ENDPOINTS.EXTENSION_INGEST}`, - { - method: 'POST', - headers: { - 'Content-Type': 'application/json', - 'X-Rate-Extension-Version': EXTENSION_VERSION, - }, - signal: AbortSignal.timeout(8_000), - body: JSON.stringify(buildCrowdIngestPayload(signal)), - } - ); - } catch { - // Non-blocking background quarantine submission - } + // ExtensionObservation v1 into the quarantine (DOS/Rate#1606): only the + // fields v1 names, under this install's random submitter id. + const observation = buildExtensionObservationV1( + signal, + new Date(), + EXTENSION_VERSION + ); + if (!observation) return; + await observationClient.send(settings.apiBaseUrl, observation); } +const observationClient = new ObservationClient({ + storage: { + get: (key) => chrome.storage.local.get(key), + set: (items) => chrome.storage.local.set(items), + remove: (key) => chrome.storage.local.remove(key), + }, + fetch: (url, init) => fetch(url, init), + now: () => Date.now(), + extensionVersion: EXTENSION_VERSION, + optedIn: async () => (await getSettings()).crowdContributeOptIn, + submitterPath: RATE_API_V1_ENDPOINTS.EXTENSION_SUBMITTER, + observationPath: RATE_API_V1_ENDPOINTS.EXTENSION_OBSERVATION, +}); + async function updateActionBadge( tabId: number | undefined, lookup: ExtensionLookupData | null @@ -196,7 +214,33 @@ async function updateActionBadge( } } +chrome.runtime.onStartup?.addListener(() => { + void observationClient.retryErasures(); +}); + chrome.runtime.onInstalled.addListener(async (details) => { + void observationClient.retryErasures(); + // An opt-in given to the older crowd signal is not consent to v1: it is + // off, and the consent page asks again (DOS/Rate#1606). + if (details?.reason === 'update') { + try { + const stored = await chrome.storage.local.get([ + 'crowdContributeOptIn', + 'crowdConsentVersion', + ]); + if ( + stored.crowdContributeOptIn === true && + stored.crowdConsentVersion !== OPT_IN_CONSENT_VERSION + ) { + await chrome.storage.local.set({ crowdContributeOptIn: false }); + await chrome.tabs.create({ + url: chrome.runtime.getURL(ONBOARDING_PAGE_PATH), + }); + } + } catch { + // The side panel toggle still offers the choice. + } + } try { await chrome.sidePanel.setPanelBehavior({ openPanelOnActionClick: true }); // Each missing setting gets its default; a stored choice is kept. @@ -595,10 +639,20 @@ chrome.runtime.onMessage.addListener((message, sender, sendResponse) => { await lookupCooldown.clear(); } } + const consent: Record = {}; if (typeof message.crowdContributeOptIn === 'boolean') { nextSettings.crowdContributeOptIn = message.crowdContributeOptIn; + consent.crowdConsentVersion = message.crowdContributeOptIn + ? OPT_IN_CONSENT_VERSION + : null; + } + await chrome.storage.local.set({ ...nextSettings, ...consent }); + // Turning it off erases this install's contributions (DOS/Rate#1606); + // a new server address leaves the old id to be erased where it came + // from on the next send. + if (nextSettings.crowdContributeOptIn === false) { + void observationClient.forget(); } - await chrome.storage.local.set(nextSettings); const updated = await getSettings(); sendResponse({ ok: true, settings: updated }); } catch (err) { diff --git a/apps/extension/src/security-guards.ts b/apps/extension/src/security-guards.ts index 9c248025a..cff80b0e8 100644 --- a/apps/extension/src/security-guards.ts +++ b/apps/extension/src/security-guards.ts @@ -375,10 +375,9 @@ export function buildLookupParams( } /** - * The ExtensionObservation v1 an opted-in install would send for `signal` + * The ExtensionObservation v1 an opted-in install sends for `signal` * (DOS/Rate#1606), or null when the page lacks what v1 needs (an item id, a - * price). Not sent yet: the opt-in signal still uses buildCrowdIngestPayload - * until the quarantine table exists and the disclosures name v1. + * price). */ export function buildExtensionObservationV1( rawSignal: ExtractedPageProductSignal, @@ -627,7 +626,38 @@ export function shouldRenderTrustPill( } /** - * What the opt-in crowd signal sends, with the message key of the words the + * What an opted-in install sends (ExtensionObservation v1, DOS/Rate#1606), + * with the message key of the words the opt-in texts use for each field; + * every locale's opt-in and consent texts name each one, plus the submitter + * id and the network hash the server adds. + */ +export const OBSERVATION_FIELD_LABELS = { + marketplace: 'crowdFieldPlatform', + listing_url: 'crowdFieldUrl', + external_item_id: 'crowdFieldExternalItemId', + external_shop_id: 'crowdFieldExternalShopId', + sku_id: 'observationFieldSku', + variant_label: 'crowdFieldSelectedVariant', + price_vnd: 'crowdFieldPrice', + list_price_vnd: 'crowdFieldOriginalPrice', + stock_state: 'observationFieldStock', + stock_qty: 'observationFieldStock', + observed_at: 'observationFieldObservedAt', + extractor: 'observationFieldExtractor', +} as const satisfies Record< + Exclude, + MessageKey +>; + +/** What else an observation carries: the submitter id and the network hash. */ +export const OBSERVATION_EXTRA_LABELS = [ + 'observationFieldSubmitter', + 'observationFieldNetwork', +] as const satisfies readonly MessageKey[]; + +/** + * The legacy opt-in crowd signal (still accepted by the ingest route for the + * contact form); the extension now sends v1. What it sends, with the message key of the words the * side panel uses for each field. The opt-in text in every locale names * each one, so a field added here is disclosed too (DOS/Rate#1230, #1304). */ diff --git a/apps/extension/src/test-extension-pipeline.ts b/apps/extension/src/test-extension-pipeline.ts index 33318d466..f988013e6 100644 --- a/apps/extension/src/test-extension-pipeline.ts +++ b/apps/extension/src/test-extension-pipeline.ts @@ -618,18 +618,34 @@ async function runTests() { hosts.filter((h) => !/rate\.vn/.test(h)).sort() ); - // The opt-in label names every field the crowd signal sends - // (DOS/Rate#1230). - const { buildCrowdIngestPayload, CROWD_INGEST_FIELD_LABELS } = - await import('./security-guards.js'); - const payload = buildCrowdIngestPayload({ - platform: 'shopee', - canonicalUrl: 'https://shopee.vn/x-i.1.2', - title: 'iPhone 16', - } as never); + // The opt-in label names every field an opted-in install sends: the v1 + // observation's keys, plus the submitter id and the network hash + // (DOS/Rate#1230, #1606). + const { + buildExtensionObservationV1, + OBSERVATION_FIELD_LABELS, + OBSERVATION_EXTRA_LABELS, + } = await import('./security-guards.js'); + const sample = buildExtensionObservationV1( + { + platform: 'shopee', + canonicalUrl: 'https://shopee.vn/x-i.1.2', + externalItemId: '2', + externalShopId: '1', + title: 'iPhone 16', + selectedVariant: null, + price: 19_990_000, + originalPrice: null, + ratingStar: 5, + ratingCount: 3, + }, + new Date(), + '2.1.0' + ); + assert.ok(sample); assert.deepEqual( - Object.keys(payload).sort(), - [...Object.keys(CROWD_INGEST_FIELD_LABELS), 'userConsentOptIn'].sort() + Object.keys(sample).sort(), + [...Object.keys(OBSERVATION_FIELD_LABELS), 'schema_version'].sort() ); const panelHtml = fs.readFileSync( path.resolve(__dirname, 'sidepanel/sidepanel.html'), @@ -640,10 +656,16 @@ async function runTests() { panelHtml.indexOf('', panelHtml.indexOf('id="chk-crowd-optin"')) ); assert.ok(optInLabel.includes('data-i18n="optInBody"')); + const disclosedKeys = [ + ...new Set([ + ...Object.values(OBSERVATION_FIELD_LABELS), + ...OBSERVATION_EXTRA_LABELS, + ]), + ]; // Every locale's opt-in text names every field (DOS/Rate#1304). for (const [locale, catalogue] of Object.entries(readLocaleCatalogues())) { const optInText = catalogue.optInBody?.message ?? ''; - for (const key of Object.values(CROWD_INGEST_FIELD_LABELS)) { + for (const key of disclosedKeys) { const label = catalogue[key]?.message ?? ''; assert.ok(label, `${locale} lacks ${key}`); assert.ok( @@ -651,10 +673,21 @@ async function runTests() { `${locale} opt-in text lacks "${label}"` ); } + // Ratings are no longer sent (JOY approved dropping them). + for (const gone of [ + 'crowdFieldRatingStar', + 'crowdFieldRatingCount', + 'crowdFieldTitle', + ]) { + assert.ok( + !optInText.includes(catalogue[gone]?.message ?? '\u0000'), + `${locale} opt-in text still names ${gone}` + ); + } assert.ok(!/Chỉ gửi URL/.test(optInText)); // The install consent page names the same fields (DOS/Rate#1603). const consentText = catalogue.onboardingOptInBody?.message ?? ''; - for (const key of Object.values(CROWD_INGEST_FIELD_LABELS)) { + for (const key of disclosedKeys) { assert.ok( consentText.includes(catalogue[key]?.message ?? '\u0000'), `${locale} consent text lacks ${key}` @@ -4296,8 +4329,25 @@ async function runTests() { workerSource.match(/'X-Rate-Extension-Version': EXTENSION_VERSION/g) ?? [] ).length, - 2 + 1 ); + // The observation client's requests carry it too (DOS/Rate#1606): each + // of its fetches takes this.headers(...), which sets the version. + { + const clientSource = fs.readFileSync( + path.resolve(__dirname, 'background/observation-client.ts'), + 'utf8' + ); + assert.equal( + (clientSource.match(/this\.deps\.fetch\(/g) ?? []).length, + (clientSource.match(/headers: this\.headers\(/g) ?? []).length + ); + assert.ok( + clientSource.includes( + "'X-Rate-Extension-Version': this.deps.extensionVersion" + ) + ); + } assert.ok(workerSource.includes('chrome.runtime.getManifest().version')); // The cooldown is checked before any fetch, and a 429 sets it from Retry-After. const requestLookupBody = workerSource.slice( @@ -5265,6 +5315,24 @@ async function runTests() { ); assert.ok(!/stockState|skuId|stockQty|extractorSource/.test(workerSource)); assert.ok(workerSource.includes('buildLookupParams(rawSignal, mode)')); + // The opt-in sends v1 to the observation route, never the legacy crowd + // ingest, and turning it off erases the install's id (DOS/Rate#1606). + assert.ok( + workerSource.includes('RATE_API_V1_ENDPOINTS.EXTENSION_OBSERVATION') + ); + assert.ok(!workerSource.includes('EXTENSION_INGEST')); + assert.ok(!workerSource.includes('buildCrowdIngestPayload')); + assert.ok( + /crowdContributeOptIn === false\)\s*\{\s*void observationClient\.forget\(\)/.test( + workerSource + ) + ); + // An opt-in given before v1 is not consent to it. + assert.ok( + workerSource.includes( + 'stored.crowdConsentVersion === OPT_IN_CONSENT_VERSION' + ) + ); // A long variant label is cut to 120 characters, not a lost observation. const longObs = guards.buildExtensionObservationV1( @@ -5370,8 +5438,245 @@ async function runTests() { assert.equal(single.stockState, 'in_stock'); assert.equal(single.stockQty, 9); assert.equal(single.skuId, 'BASE-1'); + console.log(' ✅ [Stock, SKU and the v1 observation (#1606)] Verified'); + } + // --- DOS/Rate#1606: the observation client (register, send, erase) --- + { + const { + ObservationClient, + SUBMITTER_KEY, + PENDING_ERASURE_KEY, + HOLD_UNTIL_KEY, + } = await import('./background/observation-client.js'); + const ID1 = '11111111-2222-4333-8444-555555555555'; + const ID2 = '66666666-7777-4888-8999-aaaaaaaaaaaa'; + const BASE = 'https://rate.vn'; + const obs = { + schema_version: 1 as const, + marketplace: 'shopee' as const, + listing_url: 'https://shopee.vn/x-i.1.2', + external_item_id: '2', + external_shop_id: '1', + sku_id: null, + variant_label: null, + price_vnd: 19_990_000, + list_price_vnd: null, + stock_state: 'in_stock' as const, + stock_qty: null, + observed_at: '2026-11-04T10:00:00.000Z', + extractor: { source: 'json_ld' as const, extension_version: '2.1.0' }, + }; + type Call = { + method: string; + url: string; + headers: Record; + body: unknown; + }; + const json = ( + status: number, + body: unknown, + headers: Record = {} + ) => + new Response(JSON.stringify(body), { + status, + headers: { 'Content-Type': 'application/json', ...headers }, + }); + const make = (opts: { + optedIn?: () => boolean; + respond: (call: Call) => Response | Promise; + now?: number; + }) => { + const store = new Map(); + const calls: Call[] = []; + const client = new ObservationClient({ + storage: { + get: async (key) => (store.has(key) ? { [key]: store.get(key) } : {}), + set: async (items) => { + for (const [k, v] of Object.entries(items)) store.set(k, v); + }, + remove: async (key) => { + store.delete(key); + }, + }, + fetch: async (url, init) => { + const call: Call = { + method: String(init.method), + url, + headers: (init.headers ?? {}) as Record, + body: init.body ? JSON.parse(String(init.body)) : undefined, + }; + calls.push(call); + return opts.respond(call); + }, + now: () => opts.now ?? 1_000_000, + extensionVersion: '2.1.0', + optedIn: async () => (opts.optedIn ? opts.optedIn() : true), + submitterPath: '/api/v1/rate/extension/submitter', + observationPath: '/api/v1/rate/extension/observation', + }); + return { client, store, calls }; + }; + const ok = (call: Call) => + call.method === 'POST' && call.url.endsWith('/submitter') + ? json(200, { ok: true, submitterId: ID1 }) + : json(200, { ok: true, status: 'quarantined' }); + + // (a) Register once (no body, no Content-Type), then send with the id. + { + const { client, store, calls } = make({ respond: ok }); + await client.send(BASE, obs); + assert.deepEqual( + calls.map((c) => `${c.method} ${c.url}`), + [ + `POST ${BASE}/api/v1/rate/extension/submitter`, + `POST ${BASE}/api/v1/rate/extension/observation`, + ] + ); + assert.equal(calls[0]!.body, undefined); + assert.equal(calls[0]!.headers['Content-Type'], undefined); + assert.equal(calls[1]!.headers['Content-Type'], 'application/json'); + assert.deepEqual(calls[1]!.body, { submitterId: ID1, observation: obs }); + assert.deepEqual(store.get(SUBMITTER_KEY), { id: ID1, apiBaseUrl: BASE }); + // (b) The next product reuses the id. + await client.send(BASE, obs); + assert.equal(calls.filter((c) => c.url.endsWith('/submitter')).length, 1); + } + // (c) Two products at once register one id. + { + const { client, calls } = make({ respond: ok }); + await Promise.all([client.send(BASE, obs), client.send(BASE, obs)]); + assert.equal(calls.filter((c) => c.url.endsWith('/submitter')).length, 1); + } + // (d) Routes off: no observation, no id, and a hold, so the next product + // asks nothing. + { + const { client, store, calls } = make({ + respond: () => json(404, { ok: false, code: 'NOT_ENABLED' }), + }); + await client.send(BASE, obs); + await client.send(BASE, obs); + assert.equal(calls.length, 1); + assert.equal(store.has(SUBMITTER_KEY), false); + assert.ok(Number(store.get(HOLD_UNTIL_KEY)) > 1_000_000); + } + // (e) UNKNOWN_SUBMITTER drops the id; (f) a 429 holds for its Retry-After. + { + let observationAnswer: Response = json(404, { + ok: false, + code: 'UNKNOWN_SUBMITTER', + }); + const { client, store, calls } = make({ + respond: (c) => + c.url.endsWith('/submitter') ? ok(c) : observationAnswer, + }); + await client.send(BASE, obs); + assert.equal(store.has(SUBMITTER_KEY), false); + observationAnswer = json(429, { ok: false }, { 'Retry-After': '3600' }); + await client.send(BASE, obs); + assert.equal(store.get(HOLD_UNTIL_KEY), 1_000_000 + 3_600_000); + const before = calls.length; + await client.send(BASE, obs); + assert.equal(calls.length, before, 'held: nothing is sent'); + } + // (g) Opt-out erases on the issuing server; (h) a failed erasure is kept + // and retried until the server confirms. + { + let deleteOk = false; + const { client, store, calls } = make({ + respond: (c) => + c.method === 'DELETE' + ? deleteOk + ? json(200, { ok: true }) + : json(503, { ok: false }) + : ok(c), + }); + await client.send(BASE, obs); + await client.forget(); + const del = calls.filter((c) => c.method === 'DELETE'); + assert.equal(del.length, 1); + assert.equal(del[0]!.headers['Content-Type'], 'application/json'); + assert.deepEqual(del[0]!.body, { submitterId: ID1 }); + assert.equal(store.has(SUBMITTER_KEY), false); + assert.deepEqual(store.get(PENDING_ERASURE_KEY), [ + { id: ID1, apiBaseUrl: BASE }, + ]); + deleteOk = true; + await client.retryErasures(); + assert.equal(store.has(PENDING_ERASURE_KEY), false); + } + // (i) Opt-out while the id is being issued: it is erased, nothing sent. + { + let opted = true; + let release!: () => void; + const gate = new Promise((r) => (release = r)); + let registering!: () => void; + const registerStarted = new Promise((r) => (registering = r)); + const { client, store, calls } = make({ + optedIn: () => opted, + respond: async (c) => { + if (c.method === 'POST' && c.url.endsWith('/submitter')) { + registering(); + await gate; + return json(200, { ok: true, submitterId: ID2 }); + } + return json(200, { ok: true }); + }, + }); + const sending = client.send(BASE, obs); + await registerStarted; + // The user turns it off while the id is being issued. + opted = false; + const forgetting = client.forget(); + release(); + await Promise.all([sending, forgetting]); + assert.equal( + calls.filter((c) => c.url.endsWith('/observation')).length, + 0 + ); + assert.equal(store.has(SUBMITTER_KEY), false); + assert.deepEqual( + calls.filter((c) => c.method === 'DELETE').map((c) => c.body), + [{ submitterId: ID2 }] + ); + } + // (j) Opted out: nothing at all. + { + const { client, calls } = make({ optedIn: () => false, respond: ok }); + await client.send(BASE, obs); + assert.equal(calls.length, 0); + } + // An id from another server is erased there, not sent here. + { + const { client, store, calls } = make({ respond: ok }); + store.set(SUBMITTER_KEY, { + id: ID2, + apiBaseUrl: 'http://localhost:3000', + }); + await client.send(BASE, obs); + const del = calls.find((c) => c.method === 'DELETE'); + assert.ok(del && del.url.startsWith('http://localhost:3000')); + assert.deepEqual(del.body, { submitterId: ID2 }); + assert.deepEqual( + calls.find((c) => c.url.endsWith('/observation'))?.body, + { + submitterId: ID1, + observation: obs, + } + ); + } + // (k) A body that is not JSON keeps the id and throws nothing. + { + const { client, store } = make({ + respond: (c) => + c.url.endsWith('/submitter') + ? ok(c) + : new Response('nope', { status: 404 }), + }); + await client.send(BASE, obs); + assert.deepEqual(store.get(SUBMITTER_KEY), { id: ID1, apiBaseUrl: BASE }); + } console.log( - ' ✅ [Stock, SKU and the v1 observation, not sent (#1606)] Verified' + ' ✅ [Observation client: one id, erasure kept until confirmed, holds (#1606)] Verified' ); } console.log('🎉 All Rate Browser Extension MV3 v2 tests passed!'); diff --git a/apps/extension/src/types.ts b/apps/extension/src/types.ts index e40ab530a..c067d7ee2 100644 --- a/apps/extension/src/types.ts +++ b/apps/extension/src/types.ts @@ -262,6 +262,9 @@ declare global { ) => boolean | void ) => void; }; + onStartup?: { + addListener: (callback: () => void) => void; + }; onInstalled: { addListener: ( callback: (details: { reason: string }) => void | Promise @@ -274,6 +277,7 @@ declare global { keys?: string | string[] | Record ) => Promise>; set: (items: Record) => Promise; + remove: (keys: string | string[]) => Promise; }; session: { get: ( diff --git a/apps/web/locales/cs.json b/apps/web/locales/cs.json index b9c405417..843d29f16 100644 --- a/apps/web/locales/cs.json +++ b/apps/web/locales/cs.json @@ -1534,7 +1534,7 @@ "privacyS4Title": "4. Soukromé údaje prodejců", "privacyS4Desc": "Soukromé údaje prodejce (telefonní číslo, bankovní účet) jsou v polích, která veřejnost nemůže číst. Web i API mohou číst jen veřejná pole.", "privacyS5Title": "5. Rozšíření prohlížeče Rate.vn", - "privacyS5Desc": "Na stránkách produktů 10 podporovaných obchodů rozšíření posílá Rate.vn obchod, URL produktu bez parametrů, název produktu, ID položky a obchodu, zvolenou variantu, cenu a původní cenu, aby vyhledalo historii cen; toto vyhledávání je zapnuté ve výchozím stavu, jen čte a nepotřebuje účet. Text, který napíšete do vyhledávacího pole rozšíření, se také odešle k vyhledání. Jen když zvolíte \"Ano\" na stránce zobrazené při instalaci nebo zapnete \"Pomoc s hledáním nových produktů (jen když ji zapnete)\" v bočním panelu (ve výchozím stavu vypnuto), posílá také danou nabídku s hodnocením a počtem hodnocení do fronty ke kontrole, uloženou s časem, kdy ji server přijal; servery Rate.vn mohou nabídku v obchodě znovu ověřit a tato data sama veřejné ceny nemění. Rozšíření neposílá účet, jméno, e-mail ani cookies, neotevírá stránky, které jste neotevřeli, a ve vašem prohlížeči ukládá jen adresu serveru, volbu opt-in a mezipaměť platnou po dobu relace s produkty, které jste si prohlédli, vaším posledním hledáním, výsledky vyhledávání a dobou čekání na další vyhledání. Jako u každého webového požadavku server obdrží vaši IP adresu: kód Rate.vn ji používá jen v paměti pro limit požadavků za minutu a náš poskytovatel hostingu uchovává běžné přístupové záznamy po krátkou dobu.", + "privacyS5Desc": "Na stránkách produktů 10 podporovaných obchodů rozšíření posílá Rate.vn obchod, URL produktu bez parametrů, název produktu, ID položky a obchodu, zvolenou variantu, cenu a původní cenu, aby vyhledalo historii cen; toto vyhledávání je zapnuté ve výchozím stavu, jen čte a nepotřebuje účet. Text, který napíšete do vyhledávacího pole rozšíření, se také odešle k vyhledání. Jen když zvolíte \"Ano\" na stránce zobrazené při instalaci nebo zapnete \"Pomoc s hledáním nových produktů (jen když ji zapnete)\" v bočním panelu (ve výchozím stavu vypnuto), posílá také do fronty ke kontrole záznam o dané nabídce: obchod, URL bez parametrů, ID položky a obchodu, ID varianty (SKU), zvolenou variantu, cenu, původní cenu, stav a počet kusů skladem, čas, kdy jste cenu viděli, a verzi rozšíření, spolu s náhodným ID přispěvatele této instalace. Server jej ukládá s časem přijetí a s osoleným otiskem rozsahu vaší sítě (ne IP adresou), aby rozlišil nezávislé odesílatele; otisk se maže po 28 dnech a záznam po 180 dnech. Servery Rate.vn mohou nabídku v obchodě znovu ověřit a tato data sama veřejné ceny nemění. Vypnutím přispívání se ID přispěvatele i všechny odeslané záznamy smažou; pokud rozšíření odinstalujete bez vypnutí, smažou se po 180 dnech. Rozšíření neposílá účet, jméno, e-mail ani cookies, neotevírá stránky, které jste neotevřeli, a ve vašem prohlížeči ukládá jen adresu serveru, volbu opt-in, ID přispěvatele této instalace, dokud přispíváte, a mezipaměť platnou po dobu relace s produkty, které jste si prohlédli, vaším posledním hledáním, výsledky vyhledávání a dobou čekání na další vyhledání. Jako u každého webového požadavku server obdrží vaši IP adresu: kód Rate.vn ji používá v paměti pro limit požadavků za minutu a při přispívání k odvození výše uvedeného otisku rozsahu sítě; samotná IP adresa se nikdy neukládá a náš poskytovatel hostingu uchovává běžné přístupové záznamy po krátkou dobu.", "privacySide1Title": "Ověřte si otevřený kód", "privacySide1Desc": "Pravidla přístupu k datům a způsob ukládání hlasů jsou veřejné v repozitáři na GitHubu.", "privacySide1Btn": "Technická a API dokumentace", diff --git a/apps/web/locales/en.json b/apps/web/locales/en.json index fc5960b1f..91c1ddbb2 100644 --- a/apps/web/locales/en.json +++ b/apps/web/locales/en.json @@ -1534,7 +1534,7 @@ "privacyS4Title": "4. Sellers' private details", "privacyS4Desc": "A seller's private details (phone number, bank account) are kept in fields the public cannot read. The website and the API can only read the public fields.", "privacyS5Title": "5. The Rate.vn browser extension", - "privacyS5Desc": "On product pages of the 10 supported stores, the extension sends Rate.vn the store, the product URL without its parameters, the product title, the item and shop ids, the selected variant, the price and the original price to look up price history; this lookup is on by default, read-only and needs no account. Text you type into the extension's search box is also sent to look it up. Only if you choose \"Yes\" on the page shown at install or turn on \"Help find new products (only when you turn it on)\" in the side panel (off by default) does it also send that listing, with its rating and rating count, to a moderation queue, stored with the time the server received it; Rate.vn's servers may re-check that listing on the store, and this data does not change public prices by itself. The extension sends no account, name, email or cookies, opens no page you did not open, and keeps only the server address, the opt-in choice and a session-only cache of the products you viewed, your latest search, their lookup results and a lookup wait time in your browser. Like any web request, the server receives your IP address: Rate.vn's code uses it only in memory for a per-minute request limit, and our hosting provider keeps standard access logs for a short time.", + "privacyS5Desc": "On product pages of the 10 supported stores, the extension sends Rate.vn the store, the product URL without its parameters, the product title, the item and shop ids, the selected variant, the price and the original price to look up price history; this lookup is on by default, read-only and needs no account. Text you type into the extension's search box is also sent to look it up. Only if you choose \"Yes\" on the page shown at install or turn on \"Help find new products (only when you turn it on)\" in the side panel (off by default) does it also send an observation of that listing to a moderation queue: the store, the URL without its parameters, the item and shop ids, the variant ID (SKU), the selected variant, the price, the original price, the stock status and quantity, the time you saw the price and the extension version, with a random contributor ID for this install. The server stores it with the time it was received and a salted hash of your network range (not your IP) to tell independent senders apart; the hash is deleted after 28 days and the observation after 180 days. Rate.vn's servers may re-check that listing on the store, and this data does not change public prices by itself. Turning contributions off deletes the contributor ID and every observation sent with it; if you uninstall without turning it off, they are deleted after 180 days. The extension sends no account, name, email or cookies, opens no page you did not open, and keeps only the server address, the opt-in choice, the contributor ID while you contribute, and a session-only cache of the products you viewed, your latest search, their lookup results and a lookup wait time in your browser. Like any web request, the server receives your IP address: Rate.vn's code uses it in memory for a per-minute request limit and, when you contribute, to derive the network-range hash above; the IP itself is never stored, and our hosting provider keeps standard access logs for a short time.", "privacySide1Title": "Check the open code", "privacySide1Desc": "The data access rules and how votes are stored are public in the GitHub repository.", "privacySide1Btn": "Technical and API docs", diff --git a/apps/web/locales/vi.json b/apps/web/locales/vi.json index c4564139d..f1ce32e47 100644 --- a/apps/web/locales/vi.json +++ b/apps/web/locales/vi.json @@ -1534,7 +1534,7 @@ "privacyS4Title": "4. Thông tin riêng của gian hàng", "privacyS4Desc": "Thông tin riêng của gian hàng (số điện thoại, tài khoản ngân hàng) nằm ở các trường mà công chúng không đọc được. Trang web và API chỉ đọc được các trường công khai.", "privacyS5Title": "5. Tiện ích trình duyệt Rate.vn", - "privacyS5Desc": "Trên trang sản phẩm của 10 sàn được hỗ trợ, tiện ích gửi cho Rate.vn sàn, URL sản phẩm đã bỏ tham số, tên sản phẩm, mã sản phẩm và mã gian hàng, phiên bản đang chọn, giá bán và giá gốc để tra lịch sử giá; việc tra cứu này bật sẵn, chỉ đọc và không cần tài khoản. Nội dung bạn gõ vào ô tìm kiếm của tiện ích cũng được gửi để tra cứu. Chỉ khi bạn chọn \"Có\" ở trang hiện ra lúc cài đặt hoặc bật \"Đóng góp phát hiện sản phẩm mới (chỉ khi bạn bật)\" trong bảng bên cạnh (mặc định tắt), tiện ích còn gửi tin bán đó kèm số sao và số lượt đánh giá vào hàng đợi kiểm duyệt, lưu kèm thời điểm máy chủ nhận; máy chủ Rate.vn có thể kiểm tra lại tin bán đó trên sàn, và dữ liệu này không tự đổi giá công khai. Tiện ích không gửi tài khoản, tên, email hay cookie, không mở trang bạn không mở, và chỉ lưu trên trình duyệt của bạn địa chỉ máy chủ, lựa chọn có đóng góp hay không, cùng bộ nhớ tạm theo phiên gồm các sản phẩm bạn vừa xem, lần tìm kiếm gần nhất, kết quả tra cứu và thời gian chờ khi bị giới hạn tra cứu. Như mọi yêu cầu web, máy chủ nhận địa chỉ IP của bạn: mã của Rate.vn chỉ dùng nó trong bộ nhớ để giới hạn số yêu cầu mỗi phút, còn nhà cung cấp hosting giữ nhật ký truy cập thông thường trong thời gian ngắn.", + "privacyS5Desc": "Trên trang sản phẩm của 10 sàn được hỗ trợ, tiện ích gửi cho Rate.vn sàn, URL sản phẩm đã bỏ tham số, tên sản phẩm, mã sản phẩm và mã gian hàng, phiên bản đang chọn, giá bán và giá gốc để tra lịch sử giá; việc tra cứu này bật sẵn, chỉ đọc và không cần tài khoản. Nội dung bạn gõ vào ô tìm kiếm của tiện ích cũng được gửi để tra cứu. Chỉ khi bạn chọn \"Có\" ở trang hiện ra lúc cài đặt hoặc bật \"Đóng góp phát hiện sản phẩm mới (chỉ khi bạn bật)\" trong bảng bên cạnh (mặc định tắt), tiện ích còn gửi vào hàng đợi kiểm duyệt một quan sát về tin bán đó: sàn, URL đã bỏ tham số, mã sản phẩm, mã gian hàng, mã SKU của phân loại, phiên bản đang chọn, giá bán, giá gốc, tình trạng và số lượng còn hàng, thời điểm bạn thấy giá và phiên bản tiện ích, kèm một mã người đóng góp ngẫu nhiên của lượt cài này. Máy chủ lưu kèm thời điểm nhận và một mã băm có muối của dải mạng (không phải IP) để phân biệt người gửi độc lập; mã băm được xóa sau 28 ngày, quan sát sau 180 ngày. Máy chủ Rate.vn có thể kiểm tra lại tin bán đó trên sàn, và dữ liệu này không tự đổi giá công khai. Tắt đóng góp là xóa mã người đóng góp cùng mọi quan sát đã gửi; nếu gỡ tiện ích mà chưa tắt, chúng tự xóa sau 180 ngày. Tiện ích không gửi tài khoản, tên, email hay cookie, không mở trang bạn không mở, và chỉ lưu trên trình duyệt của bạn địa chỉ máy chủ, lựa chọn có đóng góp hay không, mã người đóng góp khi bạn đóng góp, cùng bộ nhớ tạm theo phiên gồm các sản phẩm bạn vừa xem, lần tìm kiếm gần nhất, kết quả tra cứu và thời gian chờ khi bị giới hạn tra cứu. Như mọi yêu cầu web, máy chủ nhận địa chỉ IP của bạn: mã của Rate.vn dùng nó trong bộ nhớ để giới hạn số yêu cầu mỗi phút và, khi bạn đóng góp, để tính mã băm dải mạng nói trên; bản thân IP không bao giờ được lưu, còn nhà cung cấp hosting giữ nhật ký truy cập thông thường trong thời gian ngắn.", "privacySide1Title": "Kiểm tra mã nguồn công khai", "privacySide1Desc": "Quy tắc phân quyền dữ liệu và cách lưu bình chọn đều công khai trong repository trên GitHub.", "privacySide1Btn": "Xem tài liệu kỹ thuật và API", diff --git a/packages/shared-data/src/constants/routes.ts b/packages/shared-data/src/constants/routes.ts index 21270c4c7..2ebd98074 100644 --- a/packages/shared-data/src/constants/routes.ts +++ b/packages/shared-data/src/constants/routes.ts @@ -56,6 +56,8 @@ export const RATE_API_V1_ENDPOINTS = { REVALIDATE: '/api/v1/rate/revalidate', EXTENSION_LOOKUP: '/api/v1/rate/extension/lookup', EXTENSION_INGEST: '/api/v1/rate/extension/ingest', + EXTENSION_SUBMITTER: '/api/v1/rate/extension/submitter', + EXTENSION_OBSERVATION: '/api/v1/rate/extension/observation', REVIEW_VOTE: (reviewId: string) => `/api/reviews/${encodeURIComponent(sanitizePathSegment(reviewId, 128))}/vote`, HEALTH: '/api/health', From 5538ea3710e65ea291fa0d2a2e47a15b426032d1 Mon Sep 17 00:00:00 2001 From: JOY <5027251+JOY@users.noreply.github.com> Date: Thu, 5 Nov 2026 23:53:54 +0700 Subject: [PATCH 6/7] chore(db): mirror the m2 extension observation migrations from dos-me (#1611) * chore(db): mirror the m2 extension observation migrations from dos-me DOS/DOS.Me#1560 (quarantine and key row), #1564 (daily purge schedule) and #1569 (key row rotation) carry DOS/Rate#1606. The quarantine and the purge schedule are on production through #1561 and #1568; #1569 is merged on dev. Mirror the four migrations byte for byte into supabase/migrations and regenerate packages/shared-data/src/database.types.ts: the three extension tables and the register, submit, delete and purge functions. The extension observations proposal replay now rolls the mirrored quarantine back first, so it still replays from the stack before it. Nothing is applied by this change. Refs #1606 * chore(db): follow the ingest key rotation to version 20261109090000 DOS/DOS.Me#1578 moves the rotation off 20261107100000, which two other DOS.Me migrations also used. Mirror the renamed file byte for byte. Refs #1606 Co-Authored-By: Claude Sonnet 5.5 --- packages/shared-data/src/database.types.ts | 131 ++++ .../test-extension-observations-proposal.mjs | 3 + ...1106160000_rate_extension_observations.sql | 676 ++++++++++++++++++ ...261106160100_rate_extension_ingest_key.sql | 15 + ..._extension_observations_purge_schedule.sql | 35 + ...000_rate_extension_ingest_key_rotation.sql | 16 + 6 files changed, 876 insertions(+) create mode 100644 supabase/migrations/20261106160000_rate_extension_observations.sql create mode 100644 supabase/migrations/20261106160100_rate_extension_ingest_key.sql create mode 100644 supabase/migrations/20261106160200_rate_extension_observations_purge_schedule.sql create mode 100644 supabase/migrations/20261109090000_rate_extension_ingest_key_rotation.sql diff --git a/packages/shared-data/src/database.types.ts b/packages/shared-data/src/database.types.ts index f54114ed0..8b40c7fc3 100644 --- a/packages/shared-data/src/database.types.ts +++ b/packages/shared-data/src/database.types.ts @@ -161,6 +161,112 @@ export type Database = { }; Relationships: []; }; + extension_ingest_secret: { + Row: { + id: boolean; + key_sha256: string; + updated_at: string; + }; + Insert: { + id?: boolean; + key_sha256: string; + updated_at?: string; + }; + Update: { + id?: boolean; + key_sha256?: string; + updated_at?: string; + }; + Relationships: []; + }; + extension_observations: { + Row: { + external_item_id: string; + external_shop_id: string | null; + extractor: Json; + id: number; + list_price_vnd: number | null; + listing_url: string; + marketplace: string; + network_hash: string | null; + observed_at: string; + price_vnd: number; + received_at: string; + schema_version: number; + sku_id: string | null; + status: string; + stock_qty: number | null; + stock_state: string; + submitter_id: string; + variant_label: string | null; + }; + Insert: { + external_item_id: string; + external_shop_id?: string | null; + extractor: Json; + id?: never; + list_price_vnd?: number | null; + listing_url: string; + marketplace: string; + network_hash?: string | null; + observed_at: string; + price_vnd: number; + received_at?: string; + schema_version: number; + sku_id?: string | null; + status?: string; + stock_qty?: number | null; + stock_state: string; + submitter_id: string; + variant_label?: string | null; + }; + Update: { + external_item_id?: string; + external_shop_id?: string | null; + extractor?: Json; + id?: never; + list_price_vnd?: number | null; + listing_url?: string; + marketplace?: string; + network_hash?: string | null; + observed_at?: string; + price_vnd?: number; + received_at?: string; + schema_version?: number; + sku_id?: string | null; + status?: string; + stock_qty?: number | null; + stock_state?: string; + submitter_id?: string; + variant_label?: string | null; + }; + Relationships: [ + { + foreignKeyName: 'extension_observations_submitter_id_fkey'; + columns: ['submitter_id']; + referencedRelation: 'extension_submitters'; + referencedColumns: ['id']; + }, + ]; + }; + extension_submitters: { + Row: { + created_at: string; + id: string; + network_hash: string | null; + }; + Insert: { + created_at?: string; + id?: string; + network_hash?: string | null; + }; + Update: { + created_at?: string; + id?: string; + network_hash?: string | null; + }; + Relationships: []; + }; master_products: { Row: { authentic_review_count: number | null; @@ -1366,6 +1472,10 @@ export type Database = { }[]; }; current_user_can_vote_reviews: { Args: never; Returns: boolean }; + delete_extension_submitter: { + Args: { p_submitter_id: string }; + Returns: number; + }; dequeue_crawl_job: { Args: { p_lease?: string; p_platform: string; p_worker_id: string }; Returns: { @@ -1525,6 +1635,14 @@ export type Database = { Args: { p_retention_days?: number }; Returns: number; }; + purge_extension_observations: { + Args: { + p_empty_submitter_days?: number; + p_keep_days?: number; + p_network_hash_days?: number; + }; + Returns: Json; + }; purge_review_ai_points: { Args: { p_prompt_version: string }; Returns: Json; @@ -1560,6 +1678,10 @@ export type Database = { Args: { p_review_id: string }; Returns: undefined; }; + register_extension_submitter: { + Args: { p_ingest_key: string; p_network_hash: string }; + Returns: string; + }; release_crawl_job: { Args: { p_claim_token: string; p_id: string; p_reason?: string }; Returns: boolean; @@ -1721,6 +1843,15 @@ export type Database = { }; Returns: Json; }; + submit_extension_observation: { + Args: { + p_ingest_key: string; + p_network_hash: string; + p_observation: Json; + p_submitter_id: string; + }; + Returns: Json; + }; upsert_price_alert: { Args: { p_client_fingerprint: string; diff --git a/scripts/test-extension-observations-proposal.mjs b/scripts/test-extension-observations-proposal.mjs index 94aa3fe8a..387129fe7 100644 --- a/scripts/test-extension-observations-proposal.mjs +++ b/scripts/test-extension-observations-proposal.mjs @@ -110,6 +110,9 @@ const without = (key) => { try { const { sql } = startRateDb({ name: container }); + // The reference stack has the quarantine (20261106160000, with its key row): roll it back, so + // the proposal replays on the stack before it, with no key row, as this script expects. + sql(rollback); // One psql session per call; the role and its errors stay in it. VERBOSITY verbose prints the // SQLSTATE in front of the message. const as = (role, query) => diff --git a/supabase/migrations/20261106160000_rate_extension_observations.sql b/supabase/migrations/20261106160000_rate_extension_observations.sql new file mode 100644 index 000000000..354274e55 --- /dev/null +++ b/supabase/migrations/20261106160000_rate_extension_observations.sql @@ -0,0 +1,676 @@ +-- ============================================================================= +-- Migration: 20261106160000_rate_extension_observations.sql +-- Purpose (DOS/Rate#1606): milestone M2 of the DOS/Rate extension data source plan +-- (Docs/Proposals/2026-10-28-extension-data-source.md, option B, chosen by JOY on 2026-11-04), +-- the quarantine for ExtensionObservation v1. JOY approved the apply on 2026-11-04 in the +-- Rate Core PO chat: "Duyệt M2". Copied unchanged from the reviewed DOS/Rate proposal +-- Docs/Proposals/2026-11-04-dos-me-extension-observations.sql (DOS/Rate#1608) apart from this +-- header. One transaction. +-- Verify: scripts/ops/rate-extension-observations-verify.sql (V1 to V12 return 0 once +-- 20261106160100_rate_extension_ingest_key.sql has set the key row). +-- Rollback: scripts/ops/rate-extension-observations-rollback.sql. +-- +-- Three new private tables (ingest key hash, submitters, observations) and four narrow RPCs. +-- Nothing writes public prices; DOS/Rate-Services verifies observations later (M3). +-- Details in the proposal's header, below. +-- ============================================================================= +-- +-- ============================================================================= +-- PROPOSAL (DOS/Rate#1606): milestone M2 of Docs/Proposals/2026-10-28-extension-data-source.md, +-- the quarantine for ExtensionObservation v1. A DOS-Me migration to review and apply; nothing +-- here is applied by DOS/Rate. Verify: 2026-11-04-dos-me-extension-observations-verify.sql +-- (V1 to V12 return 0). Rollback: 2026-11-04-dos-me-extension-observations-rollback.sql. +-- Contract: the M2 contract with its Revision 2 (ingest key, required network hash, register +-- limits, purge race fix). +-- +-- What it does: +-- rate.extension_ingest_secret one private row: the sha256 of the web's ingest key. This file +-- inserts no row; DOS-Me sets it at apply time. Without it, or +-- with a wrong key, register and submit refuse (42501). +-- rate.extension_submitters one row per extension install that opted in: a random id, its +-- creation time and the network hash it registered from. +-- Pseudonymous personal data. +-- rate.extension_observations the quarantine: one row per observation the extension sent, +-- one column per payload key, checked like the payload, plus +-- received_at, a salted network hash and a status +-- (quarantined, corroborated or rejected). +-- All three tables: row level security on; no anon or authenticated access; service_role +-- full; rate_worker reads submitters and observations and may update status only. +-- rate.register_extension_submitter(key, h) anon with the ingest key: a new submitter +-- id; 20 an hour per network hash, 600 a minute in all (54000). +-- rate.submit_extension_observation(key, id, obs, h) anon with the ingest key: one +-- observation, validated field by field (22023 with the field in +-- DETAIL), 120 an hour per submitter and 600 an hour per network +-- hash (54000). +-- rate.delete_extension_submitter(id) anon, no key: deletes the submitter and its +-- observations; returns how many observations went (0 for an +-- unknown id). +-- rate.purge_extension_observations(d, k, e) service_role: clears network hashes after d +-- days (28), deletes observations after k days (180) and +-- submitters without observations after e days (7). +-- +-- Nothing here writes price_history, products or any public table. Promotion out of the +-- quarantine is M3/M4, in DOS/Rate-Services. +-- +-- Depends on: the rate schema, the API roles (anon, authenticated, service_role) and rate_worker +-- (20261018183000_rate_worker_role.sql); all are in production. +-- +-- Rating impact: none. No rating, price or product reads these tables. +-- +-- Idempotent: IF NOT EXISTS for the tables and indexes, CREATE OR REPLACE for functions, +-- DROP POLICY IF EXISTS before each policy, grants restated every run. +-- ============================================================================= + +BEGIN; + +-- New tables only; the FK takes a lock on its own new parent. Give up rather than queue. +SET LOCAL lock_timeout = '5s'; + +-- 1. The ingest key ---------------------------------------------------------------------------- +-- One row at most (id is always true). Only the key's sha256 is stored; the key is the web's +-- secret RATE_EXTENSION_INGEST_KEY and is never in git. DOS-Me inserts the row at apply time. + +CREATE TABLE IF NOT EXISTS rate.extension_ingest_secret ( + id boolean PRIMARY KEY DEFAULT true CHECK (id), + key_sha256 text NOT NULL CHECK (key_sha256 ~ '^[0-9a-f]{64}$'), + updated_at timestamptz NOT NULL DEFAULT now(), + -- Not the sha256 of the empty string: a runbook that hashed an unset variable would + -- otherwise store a key that an empty caller matches. + CONSTRAINT extension_ingest_secret_key_not_empty_check + CHECK (key_sha256 <> 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855') +); + +COMMENT ON TABLE rate.extension_ingest_secret IS + 'sha256 of the web''s extension ingest key (DOS/Rate#1606). register_extension_submitter and submit_extension_observation refuse a caller without the key. service_role only.'; + +-- 2. Submitters -------------------------------------------------------------------------------- + +CREATE TABLE IF NOT EXISTS rate.extension_submitters ( + id uuid PRIMARY KEY DEFAULT gen_random_uuid(), + created_at timestamptz NOT NULL DEFAULT now(), + network_hash text, + CONSTRAINT extension_submitters_network_hash_check CHECK (network_hash ~ '^[0-9a-f]{64}$') +); + +COMMENT ON TABLE rate.extension_submitters IS + 'One row per extension install that opted in to sending observations (DOS/Rate#1606). Pseudonymous personal data: delete on request through rate.delete_extension_submitter; network_hash cleared after 28 days; deleted 7 days after creation when it has no observations, else with its last observation.'; + +-- The global register brake and the purge read created_at; the per-network register limit +-- reads network_hash. +CREATE INDEX IF NOT EXISTS extension_submitters_created_idx + ON rate.extension_submitters (created_at); +CREATE INDEX IF NOT EXISTS extension_submitters_network_created_idx + ON rate.extension_submitters (network_hash, created_at) WHERE network_hash IS NOT NULL; + +-- 3. The quarantine ---------------------------------------------------------------------------- +-- Each CHECK repeats a payload rule of rate.submit_extension_observation, which refuses first +-- with the field's name; the CHECKs hold service_role's direct writes to the same rules. + +CREATE TABLE IF NOT EXISTS rate.extension_observations ( + id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + submitter_id uuid NOT NULL REFERENCES rate.extension_submitters(id) ON DELETE CASCADE, + schema_version smallint NOT NULL, + marketplace text NOT NULL, + listing_url text NOT NULL, + external_item_id text NOT NULL, + external_shop_id text, + sku_id text, + variant_label text, + price_vnd bigint NOT NULL, + list_price_vnd bigint, + stock_state text NOT NULL, + stock_qty integer, + observed_at timestamptz NOT NULL, + extractor jsonb NOT NULL, + received_at timestamptz NOT NULL DEFAULT now(), + network_hash text, + status text NOT NULL DEFAULT 'quarantined', + CONSTRAINT extension_observations_schema_version_check CHECK (schema_version = 1), + CONSTRAINT extension_observations_marketplace_check CHECK (marketplace IN ( + 'shopee', 'lazada', 'tiki', 'tiktok', 'cellphones', 'tgdd', 'fptshop', + 'hoangha', 'didongviet', 'phongvu')), + -- https, at most 2048 characters, no query, fragment or path parameter, no space (JavaScript's + -- \s set) or control character, and a lowercase host without user or port that belongs to the + -- marketplace (the host table). + CONSTRAINT extension_observations_listing_url_check CHECK ( + char_length(listing_url) <= 2048 + AND listing_url !~ '[?#;[:space:]\x01-\x1f\x7f-\x9f\xa0\x1680\x2000-\x200a\x2028\x2029\x202f\x205f\x3000\xfeff]' + AND CASE marketplace + WHEN 'shopee' THEN listing_url ~ '^https://shopee\.vn(/|$)' + WHEN 'lazada' THEN listing_url ~ '^https://([a-z0-9-]+\.)*lazada\.vn(/|$)' + WHEN 'tiki' THEN listing_url ~ '^https://tiki\.vn(/|$)' + WHEN 'tiktok' THEN listing_url ~ '^https://(www|shop)\.tiktok\.com(/|$)' + WHEN 'cellphones' THEN listing_url ~ '^https://cellphones\.com\.vn(/|$)' + WHEN 'tgdd' THEN listing_url ~ '^https://([a-z0-9-]+\.)*thegioididong\.com(/|$)' + WHEN 'fptshop' THEN listing_url ~ '^https://fptshop\.com\.vn(/|$)' + WHEN 'hoangha' THEN listing_url ~ '^https://hoanghamobile\.com(/|$)' + WHEN 'didongviet' THEN listing_url ~ '^https://didongviet\.vn(/|$)' + WHEN 'phongvu' THEN listing_url ~ '^https://phongvu\.vn(/|$)' + ELSE false END), + CONSTRAINT extension_observations_external_item_id_check + CHECK (external_item_id ~ '^[A-Za-z0-9_.-]{1,64}$'), + CONSTRAINT extension_observations_external_shop_id_check + CHECK (external_shop_id ~ '^[A-Za-z0-9_.-]{1,64}$'), + CONSTRAINT extension_observations_sku_id_check CHECK (sku_id ~ '^[A-Za-z0-9_.-]{1,64}$'), + CONSTRAINT extension_observations_variant_label_check CHECK ( + char_length(variant_label) BETWEEN 1 AND 120 AND variant_label !~ '[\x01-\x1f\x7f-\x9f]'), + CONSTRAINT extension_observations_price_vnd_check + CHECK (price_vnd BETWEEN 1000 AND 2000000000), + CONSTRAINT extension_observations_list_price_vnd_check + CHECK (list_price_vnd >= price_vnd AND list_price_vnd <= 2000000000), + CONSTRAINT extension_observations_stock_state_check + CHECK (stock_state IN ('in_stock', 'low_stock', 'out_of_stock', 'unknown')), + CONSTRAINT extension_observations_stock_qty_check CHECK (stock_qty BETWEEN 0 AND 1000000), + -- The RPC checks observed_at against now(); a CHECK can only hold it to received_at. + CONSTRAINT extension_observations_observed_at_check CHECK ( + observed_at <= received_at + interval '10 minutes' + AND observed_at >= received_at - interval '24 hours'), + CONSTRAINT extension_observations_extractor_check CHECK ( + jsonb_typeof(extractor) = 'object' + AND extractor - 'source' - 'extension_version' = '{}'::jsonb + AND extractor ->> 'source' IN ('json_ld', 'dom', 'mixed') + AND jsonb_typeof(extractor -> 'source') = 'string' + AND jsonb_typeof(extractor -> 'extension_version') = 'string' + AND (extractor ->> 'extension_version') ~ '^[0-9A-Za-z.+-]{1,20}$'), + CONSTRAINT extension_observations_network_hash_check CHECK (network_hash ~ '^[0-9a-f]{64}$'), + CONSTRAINT extension_observations_status_check + CHECK (status IN ('quarantined', 'corroborated', 'rejected')) +); + +COMMENT ON TABLE rate.extension_observations IS + 'Quarantine of ExtensionObservation v1 (DOS/Rate#1606). Unverified crowd data: nothing reads it into prices or ratings until M3/M4 corroborates it. network_hash is cleared after 28 days and rows are deleted after 180 days by rate.purge_extension_observations.'; +COMMENT ON COLUMN rate.extension_observations.network_hash IS + 'HMAC-SHA256 of the sender''s /24 (IPv4) or /48 (IPv6), keyed by a web secret; pseudonymous personal data, cleared after 28 days.'; + +CREATE INDEX IF NOT EXISTS extension_observations_submitter_received_idx + ON rate.extension_observations (submitter_id, received_at); +CREATE INDEX IF NOT EXISTS extension_observations_network_received_idx + ON rate.extension_observations (network_hash, received_at) WHERE network_hash IS NOT NULL; +CREATE INDEX IF NOT EXISTS extension_observations_item_received_idx + ON rate.extension_observations (marketplace, external_item_id, received_at); + +-- 4. Grants and row level security ------------------------------------------------------------- +-- anon and authenticated reach the tables only through the RPCs below. + +ALTER TABLE rate.extension_ingest_secret ENABLE ROW LEVEL SECURITY; +ALTER TABLE rate.extension_submitters ENABLE ROW LEVEL SECURITY; +ALTER TABLE rate.extension_observations ENABLE ROW LEVEL SECURITY; + +-- The ingest key: service_role only (the functions read it as their owner). +REVOKE ALL ON rate.extension_ingest_secret FROM PUBLIC, anon, authenticated, rate_worker; +GRANT ALL ON rate.extension_ingest_secret TO service_role; +DROP POLICY IF EXISTS "Service role access extension_ingest_secret" ON rate.extension_ingest_secret; +CREATE POLICY "Service role access extension_ingest_secret" ON rate.extension_ingest_secret + FOR ALL TO service_role USING (true) WITH CHECK (true); + +REVOKE ALL ON rate.extension_submitters, rate.extension_observations + FROM PUBLIC, anon, authenticated, rate_worker; +GRANT ALL ON rate.extension_submitters, rate.extension_observations TO service_role; +GRANT SELECT ON rate.extension_submitters, rate.extension_observations TO rate_worker; +GRANT UPDATE (status) ON rate.extension_observations TO rate_worker; + +-- The identity sequence: only service_role (a project's default privileges may grant it wider). +DO $$ +DECLARE + v_seq text := pg_get_serial_sequence('rate.extension_observations', 'id'); +BEGIN + EXECUTE format('REVOKE ALL ON SEQUENCE %s FROM PUBLIC, anon, authenticated, rate_worker', v_seq); + EXECUTE format('GRANT ALL ON SEQUENCE %s TO service_role', v_seq); +END $$; + +DROP POLICY IF EXISTS "Service role access extension_submitters" ON rate.extension_submitters; +CREATE POLICY "Service role access extension_submitters" ON rate.extension_submitters + FOR ALL TO service_role USING (true) WITH CHECK (true); +DROP POLICY IF EXISTS "Rate worker read extension_submitters" ON rate.extension_submitters; +CREATE POLICY "Rate worker read extension_submitters" ON rate.extension_submitters + FOR SELECT TO rate_worker USING (true); + +DROP POLICY IF EXISTS "Service role access extension_observations" ON rate.extension_observations; +CREATE POLICY "Service role access extension_observations" ON rate.extension_observations + FOR ALL TO service_role USING (true) WITH CHECK (true); +DROP POLICY IF EXISTS "Rate worker read extension_observations" ON rate.extension_observations; +CREATE POLICY "Rate worker read extension_observations" ON rate.extension_observations + FOR SELECT TO rate_worker USING (true); +-- The column grant above limits the update to status; the CHECK limits its values. +DROP POLICY IF EXISTS "Rate worker status extension_observations" ON rate.extension_observations; +CREATE POLICY "Rate worker status extension_observations" ON rate.extension_observations + FOR UPDATE TO rate_worker USING (true) WITH CHECK (true); + +-- 5. register_extension_submitter -------------------------------------------------------------- +-- The ingest key (42501, DETAIL ingest_key: its sha256 must equal the stored row, and no row +-- refuses every caller), a network hash (22023), then the limits (54000): at most 20 new +-- submitters per network hash in the last hour (DETAIL network_hash), then a global brake of +-- 600 in the last minute (DETAIL submitter). Each count runs under its advisory lock, so it is +-- exact: per network first, then the global one, always in that order. + +CREATE OR REPLACE FUNCTION rate.register_extension_submitter( + p_ingest_key text, + p_network_hash text +) +RETURNS uuid +LANGUAGE plpgsql +VOLATILE +SECURITY DEFINER +SET search_path = '' +AS $$ +DECLARE + v_id uuid; +BEGIN + -- The web's key has at least 32 characters; a shorter or missing one never reaches the hash. + IF p_ingest_key IS NULL OR char_length(p_ingest_key) < 32 THEN + RAISE EXCEPTION 'Extension ingest key refused' USING ERRCODE = '42501', + DETAIL = 'ingest_key'; + END IF; + IF (SELECT key_sha256 FROM rate.extension_ingest_secret WHERE id) IS DISTINCT FROM + encode(sha256(convert_to(coalesce(p_ingest_key, ''), 'UTF8')), 'hex') THEN + RAISE EXCEPTION 'Extension ingest key refused' USING ERRCODE = '42501', + DETAIL = 'ingest_key'; + END IF; + + IF p_network_hash IS NULL OR p_network_hash !~ '^[0-9a-f]{64}$' THEN + RAISE EXCEPTION 'Invalid network hash' USING ERRCODE = '22023', DETAIL = 'network_hash'; + END IF; + + PERFORM pg_catalog.pg_advisory_xact_lock( + pg_catalog.hashtextextended('rate.extension_submitters:' || p_network_hash, 0)); + IF (SELECT count(*) FROM rate.extension_submitters + WHERE network_hash = p_network_hash + AND created_at > now() - interval '1 hour') >= 20 THEN + RAISE EXCEPTION 'Too many extension submitters from this network, try again later' + USING ERRCODE = '54000', DETAIL = 'network_hash'; + END IF; + + PERFORM pg_catalog.pg_advisory_xact_lock( + pg_catalog.hashtextextended('rate.extension_submitters:register', 0)); + IF (SELECT count(*) FROM rate.extension_submitters + WHERE created_at > now() - interval '1 minute') >= 600 THEN + RAISE EXCEPTION 'Too many extension submitters registered, try again later' + USING ERRCODE = '54000', DETAIL = 'submitter'; + END IF; + + INSERT INTO rate.extension_submitters (network_hash) VALUES (p_network_hash) + RETURNING id INTO v_id; + RETURN v_id; +END; +$$; + +-- 6. submit_extension_observation -------------------------------------------------------------- +-- The ingest key (42501, DETAIL ingest_key), a network hash (22023, DETAIL network_hash), then +-- every rule of ExtensionObservation v1 (22023, DETAIL the field) in the order of +-- parseExtensionObservationV1 (packages/shared-data/src/rules/extension-observation.ts): +-- 'observation' for a payload that is not an object, is over 16 KB or has another key (its name +-- is never echoed), then each key in turn, its presence checked with its rule (a nullable key +-- must be present as null), then extractor.source and extractor.extension_version. Then the +-- submitter (P0002, DETAIL submitter_id) and the limits (54000): at most 120 per submitter and +-- 600 per network hash in the last hour. + +CREATE OR REPLACE FUNCTION rate.submit_extension_observation( + p_ingest_key text, + p_submitter_id uuid, + p_observation jsonb, + p_network_hash text +) +RETURNS jsonb +LANGUAGE plpgsql +VOLATILE +SECURITY DEFINER +SET search_path = '' +AS $$ +DECLARE + c_keys constant text[] := ARRAY['schema_version', 'marketplace', 'listing_url', + 'external_item_id', 'external_shop_id', 'sku_id', 'variant_label', 'price_vnd', + 'list_price_vnd', 'stock_state', 'stock_qty', 'observed_at', 'extractor']; + c_id_pattern constant text := '^[A-Za-z0-9_.-]{1,64}$'; + o jsonb := p_observation; + v_marketplace text; + v_url text; + v_host text; + v_host_ok boolean; + v_item text; + v_shop text; + v_sku text; + v_variant text; + v_price bigint; + v_list_price bigint; + v_stock_state text; + v_stock_qty integer; + v_observed_text text; + v_observed timestamptz; + v_extractor jsonb; + v_num numeric; + v_now timestamptz := now(); + v_id bigint; +BEGIN + -- The web's key has at least 32 characters; a shorter or missing one never reaches the hash. + IF p_ingest_key IS NULL OR char_length(p_ingest_key) < 32 THEN + RAISE EXCEPTION 'Extension ingest key refused' USING ERRCODE = '42501', + DETAIL = 'ingest_key'; + END IF; + IF (SELECT key_sha256 FROM rate.extension_ingest_secret WHERE id) IS DISTINCT FROM + encode(sha256(convert_to(coalesce(p_ingest_key, ''), 'UTF8')), 'hex') THEN + RAISE EXCEPTION 'Extension ingest key refused' USING ERRCODE = '42501', + DETAIL = 'ingest_key'; + END IF; + + IF p_network_hash IS NULL OR p_network_hash !~ '^[0-9a-f]{64}$' THEN + RAISE EXCEPTION 'Invalid network hash' USING ERRCODE = '22023', DETAIL = 'network_hash'; + END IF; + + IF o IS NULL OR jsonb_typeof(o) <> 'object' OR octet_length(o::text) > 16384 THEN + RAISE EXCEPTION 'Observation must be a JSON object' USING ERRCODE = '22023', + DETAIL = 'observation'; + END IF; + + -- Any other key is refused as 'observation': a caller's key name is never echoed. + IF EXISTS (SELECT 1 FROM jsonb_object_keys(o) AS k WHERE k <> ALL (c_keys)) THEN + RAISE EXCEPTION 'Unknown observation key' USING ERRCODE = '22023', DETAIL = 'observation'; + END IF; + + -- schema_version: exactly 1. + -- A number is read only once its type is known, so a string never reaches the cast. + v_num := CASE WHEN jsonb_typeof(o -> 'schema_version') = 'number' + THEN (o ->> 'schema_version')::numeric END; + IF v_num IS DISTINCT FROM 1 THEN + RAISE EXCEPTION 'Unsupported schema_version' USING ERRCODE = '22023', + DETAIL = 'schema_version'; + END IF; + + -- marketplace. + v_marketplace := CASE WHEN jsonb_typeof(o -> 'marketplace') = 'string' + THEN o ->> 'marketplace' END; + IF v_marketplace IS NULL OR v_marketplace NOT IN ('shopee', 'lazada', 'tiki', 'tiktok', + 'cellphones', 'tgdd', 'fptshop', 'hoangha', 'didongviet', 'phongvu') THEN + RAISE EXCEPTION 'Unsupported marketplace' USING ERRCODE = '22023', DETAIL = 'marketplace'; + END IF; + + -- listing_url: https, at most 2048 characters, no query, fragment, path parameter (;), space + -- (JavaScript's \s set) or control character; a lowercase host without user or port, from the marketplace's host table. + v_url := CASE WHEN jsonb_typeof(o -> 'listing_url') = 'string' THEN o ->> 'listing_url' END; + v_host := substring(v_url FROM '^https://([^/]*)'); + -- The host table (a CASE inside an IF condition would end the condition at its first THEN). + v_host_ok := CASE v_marketplace + WHEN 'shopee' THEN v_host = 'shopee.vn' + WHEN 'lazada' THEN v_host ~ '^([a-z0-9-]+\.)*lazada\.vn$' + WHEN 'tiki' THEN v_host = 'tiki.vn' + WHEN 'tiktok' THEN v_host IN ('www.tiktok.com', 'shop.tiktok.com') + WHEN 'cellphones' THEN v_host = 'cellphones.com.vn' + WHEN 'tgdd' THEN v_host ~ '^([a-z0-9-]+\.)*thegioididong\.com$' + WHEN 'fptshop' THEN v_host = 'fptshop.com.vn' + WHEN 'hoangha' THEN v_host = 'hoanghamobile.com' + WHEN 'didongviet' THEN v_host = 'didongviet.vn' + WHEN 'phongvu' THEN v_host = 'phongvu.vn' + ELSE false END; + IF v_url IS NULL OR char_length(v_url) > 2048 + OR v_url ~ '[?#;[:space:]\x01-\x1f\x7f-\x9f\xa0\x1680\x2000-\x200a\x2028\x2029\x202f\x205f\x3000\xfeff]' + OR v_host IS NULL OR v_host !~ '^[a-z0-9.-]+$' + OR v_host_ok IS NOT TRUE THEN + RAISE EXCEPTION 'Invalid listing_url' USING ERRCODE = '22023', DETAIL = 'listing_url'; + END IF; + + -- The ids: 1 to 64 of [A-Za-z0-9_.-]; the shop and SKU may be null. + v_item := CASE WHEN jsonb_typeof(o -> 'external_item_id') = 'string' + THEN o ->> 'external_item_id' END; + IF v_item IS NULL OR v_item !~ c_id_pattern THEN + RAISE EXCEPTION 'Invalid external_item_id' USING ERRCODE = '22023', + DETAIL = 'external_item_id'; + END IF; + IF NOT o ? 'external_shop_id' + OR COALESCE(jsonb_typeof(o -> 'external_shop_id'), 'null') NOT IN ('null', 'string') + OR (o ->> 'external_shop_id') !~ c_id_pattern THEN + RAISE EXCEPTION 'Invalid external_shop_id' USING ERRCODE = '22023', + DETAIL = 'external_shop_id'; + END IF; + v_shop := o ->> 'external_shop_id'; + IF NOT o ? 'sku_id' + OR COALESCE(jsonb_typeof(o -> 'sku_id'), 'null') NOT IN ('null', 'string') + OR (o ->> 'sku_id') !~ c_id_pattern THEN + RAISE EXCEPTION 'Invalid sku_id' USING ERRCODE = '22023', DETAIL = 'sku_id'; + END IF; + v_sku := o ->> 'sku_id'; + + -- variant_label: null, or 1 to 120 characters without a control character. + IF NOT o ? 'variant_label' + OR COALESCE(jsonb_typeof(o -> 'variant_label'), 'null') NOT IN ('null', 'string') + OR char_length(o ->> 'variant_label') NOT BETWEEN 1 AND 120 + OR (o ->> 'variant_label') ~ '[\x01-\x1f\x7f-\x9f]' THEN + RAISE EXCEPTION 'Invalid variant_label' USING ERRCODE = '22023', DETAIL = 'variant_label'; + END IF; + v_variant := o ->> 'variant_label'; + + -- price_vnd: an integer from 1,000 to 2,000,000,000. + v_num := CASE WHEN jsonb_typeof(o -> 'price_vnd') = 'number' + THEN (o ->> 'price_vnd')::numeric END; + IF v_num IS NULL OR v_num NOT BETWEEN 1000 AND 2000000000 OR v_num <> trunc(v_num) THEN + RAISE EXCEPTION 'Invalid price_vnd' USING ERRCODE = '22023', DETAIL = 'price_vnd'; + END IF; + v_price := v_num; + + -- list_price_vnd: null, or an integer from price_vnd to 2,000,000,000. + v_num := CASE WHEN jsonb_typeof(o -> 'list_price_vnd') = 'number' + THEN (o ->> 'list_price_vnd')::numeric END; + IF NOT o ? 'list_price_vnd' + OR COALESCE(jsonb_typeof(o -> 'list_price_vnd'), 'null') NOT IN ('null', 'number') + OR v_num NOT BETWEEN v_price AND 2000000000 OR v_num <> trunc(v_num) THEN + RAISE EXCEPTION 'Invalid list_price_vnd' USING ERRCODE = '22023', DETAIL = 'list_price_vnd'; + END IF; + v_list_price := v_num; + + -- stock_state and stock_qty. + v_stock_state := CASE WHEN jsonb_typeof(o -> 'stock_state') = 'string' + THEN o ->> 'stock_state' END; + IF v_stock_state IS NULL + OR v_stock_state NOT IN ('in_stock', 'low_stock', 'out_of_stock', 'unknown') THEN + RAISE EXCEPTION 'Invalid stock_state' USING ERRCODE = '22023', DETAIL = 'stock_state'; + END IF; + v_num := CASE WHEN jsonb_typeof(o -> 'stock_qty') = 'number' + THEN (o ->> 'stock_qty')::numeric END; + IF NOT o ? 'stock_qty' + OR COALESCE(jsonb_typeof(o -> 'stock_qty'), 'null') NOT IN ('null', 'number') + OR v_num NOT BETWEEN 0 AND 1000000 OR v_num <> trunc(v_num) THEN + RAISE EXCEPTION 'Invalid stock_qty' USING ERRCODE = '22023', DETAIL = 'stock_qty'; + END IF; + v_stock_qty := v_num; + + -- observed_at: ISO 8601 with a zone (seconds 00 to 59, so no leap second; an offset up to + -- 15:59, as Postgres parses it), at most 10 minutes ahead and 24 hours behind now(). + v_observed_text := CASE WHEN jsonb_typeof(o -> 'observed_at') = 'string' + THEN o ->> 'observed_at' END; + IF v_observed_text IS NULL OR v_observed_text !~ + '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(:[0-5]\d(\.\d{1,9})?)?(Z|[+-](0\d|1[0-5])(:?[0-5]\d)?)$' + THEN + RAISE EXCEPTION 'Invalid observed_at' USING ERRCODE = '22023', DETAIL = 'observed_at'; + END IF; + BEGIN + v_observed := v_observed_text::timestamptz; + EXCEPTION WHEN others THEN + RAISE EXCEPTION 'Invalid observed_at' USING ERRCODE = '22023', DETAIL = 'observed_at'; + END; + IF v_observed > v_now + interval '10 minutes' OR v_observed < v_now - interval '24 hours' THEN + RAISE EXCEPTION 'observed_at is out of range' USING ERRCODE = '22023', + DETAIL = 'observed_at'; + END IF; + + -- extractor: exactly {source, extension_version}. + v_extractor := o -> 'extractor'; + IF jsonb_typeof(v_extractor) IS DISTINCT FROM 'object' + OR v_extractor - 'source' - 'extension_version' <> '{}'::jsonb THEN + RAISE EXCEPTION 'Invalid extractor' USING ERRCODE = '22023', DETAIL = 'extractor'; + END IF; + IF jsonb_typeof(v_extractor -> 'source') IS DISTINCT FROM 'string' + OR (v_extractor ->> 'source') NOT IN ('json_ld', 'dom', 'mixed') THEN + RAISE EXCEPTION 'Invalid extractor.source' USING ERRCODE = '22023', + DETAIL = 'extractor.source'; + END IF; + IF jsonb_typeof(v_extractor -> 'extension_version') IS DISTINCT FROM 'string' + OR (v_extractor ->> 'extension_version') !~ '^[0-9A-Za-z.+-]{1,20}$' THEN + RAISE EXCEPTION 'Invalid extractor.extension_version' USING ERRCODE = '22023', + DETAIL = 'extractor.extension_version'; + END IF; + + -- The submitter, locked: one submitter's calls (and its deletion) run one at a time, so its + -- count below is exact. + PERFORM 1 FROM rate.extension_submitters WHERE id = p_submitter_id FOR UPDATE; + IF NOT FOUND THEN + RAISE EXCEPTION 'Unknown extension submitter' USING ERRCODE = 'P0002', + DETAIL = 'submitter_id'; + END IF; + + IF (SELECT count(*) FROM rate.extension_observations + WHERE submitter_id = p_submitter_id + AND received_at > v_now - interval '1 hour') >= 120 THEN + RAISE EXCEPTION 'Too many observations from this submitter, try again later' + USING ERRCODE = '54000', DETAIL = 'submitter_id'; + END IF; + + -- One network's calls run one at a time too (a transaction-scoped advisory lock). + PERFORM pg_catalog.pg_advisory_xact_lock( + pg_catalog.hashtextextended('rate.extension_observations:' || p_network_hash, 0)); + IF (SELECT count(*) FROM rate.extension_observations + WHERE network_hash = p_network_hash + AND received_at > v_now - interval '1 hour') >= 600 THEN + RAISE EXCEPTION 'Too many observations from this network, try again later' + USING ERRCODE = '54000', DETAIL = 'network_hash'; + END IF; + + INSERT INTO rate.extension_observations ( + submitter_id, schema_version, marketplace, listing_url, external_item_id, external_shop_id, + sku_id, variant_label, price_vnd, list_price_vnd, stock_state, stock_qty, observed_at, + extractor, received_at, network_hash, status) + VALUES ( + p_submitter_id, 1, v_marketplace, v_url, v_item, v_shop, + v_sku, v_variant, v_price, v_list_price, v_stock_state, v_stock_qty, v_observed, + v_extractor, v_now, p_network_hash, 'quarantined') + RETURNING id INTO v_id; + + RETURN jsonb_build_object('id', v_id, 'status', 'quarantined', 'received_at', v_now); +END; +$$; + +-- 7. delete_extension_submitter ---------------------------------------------------------------- +-- Deletes the submitter; its observations go by the cascade. Returns how many observations went; +-- an unknown or null id returns 0, so a call does not reveal whether an id exists. + +CREATE OR REPLACE FUNCTION rate.delete_extension_submitter(p_submitter_id uuid) +RETURNS integer +LANGUAGE plpgsql +VOLATILE +SECURITY DEFINER +SET search_path = '' +AS $$ +DECLARE + v_count integer; +BEGIN + -- Locked first: no observation of this submitter can be added between the count and the + -- delete. + PERFORM 1 FROM rate.extension_submitters WHERE id = p_submitter_id FOR UPDATE; + IF NOT FOUND THEN + RETURN 0; + END IF; + + SELECT count(*) INTO v_count FROM rate.extension_observations + WHERE submitter_id = p_submitter_id; + + DELETE FROM rate.extension_submitters WHERE id = p_submitter_id; + RETURN v_count; +END; +$$; + +-- 8. purge_extension_observations -------------------------------------------------------------- +-- Retention, ages from received_at and created_at. Meant to run daily as service_role. +-- network_hash cleared after p_network_hash_days (28), on observations and on submitters; +-- observations deleted after p_keep_days (180); +-- submitters without observations deleted after p_empty_submitter_days (7). +-- The empty submitters are locked first (SKIP LOCKED: one a submit holds waits for the next +-- run), then deleted under a fresh NOT EXISTS, so an observation committed in between keeps its +-- submitter instead of going with it by the cascade. + +CREATE OR REPLACE FUNCTION rate.purge_extension_observations( + p_network_hash_days integer DEFAULT 28, + p_keep_days integer DEFAULT 180, + p_empty_submitter_days integer DEFAULT 7 +) +RETURNS jsonb +LANGUAGE plpgsql +VOLATILE +SECURITY DEFINER +SET search_path = '' +AS $$ +DECLARE + v_hashes integer; + v_submitter_hashes integer; + v_observations integer; + v_submitters integer; + v_candidates uuid[]; +BEGIN + IF p_network_hash_days IS NULL OR p_network_hash_days < 1 THEN + RAISE EXCEPTION 'p_network_hash_days must be at least 1' USING ERRCODE = '22023', + DETAIL = 'p_network_hash_days'; + END IF; + IF p_keep_days IS NULL OR p_keep_days < 1 THEN + RAISE EXCEPTION 'p_keep_days must be at least 1' USING ERRCODE = '22023', + DETAIL = 'p_keep_days'; + END IF; + IF p_empty_submitter_days IS NULL OR p_empty_submitter_days < 1 THEN + RAISE EXCEPTION 'p_empty_submitter_days must be at least 1' USING ERRCODE = '22023', + DETAIL = 'p_empty_submitter_days'; + END IF; + + -- Lock order: submitters before observations, as delete_extension_submitter does (the + -- submitter row, then its observations through the cascade). The other order deadlocks with a + -- delete that runs in the middle of a purge. + UPDATE rate.extension_submitters SET network_hash = NULL + WHERE network_hash IS NOT NULL + AND created_at < now() - make_interval(days => p_network_hash_days); + GET DIAGNOSTICS v_submitter_hashes = ROW_COUNT; + + UPDATE rate.extension_observations SET network_hash = NULL + WHERE network_hash IS NOT NULL + AND received_at < now() - make_interval(days => p_network_hash_days); + GET DIAGNOSTICS v_hashes = ROW_COUNT; + + DELETE FROM rate.extension_observations + WHERE received_at < now() - make_interval(days => p_keep_days); + GET DIAGNOSTICS v_observations = ROW_COUNT; + + SELECT array_agg(c.id) INTO v_candidates FROM ( + SELECT s.id FROM rate.extension_submitters s + WHERE s.created_at < now() - make_interval(days => p_empty_submitter_days) + AND NOT EXISTS (SELECT 1 FROM rate.extension_observations o WHERE o.submitter_id = s.id) + FOR UPDATE OF s SKIP LOCKED + ) c; + + DELETE FROM rate.extension_submitters s + WHERE s.id = ANY (COALESCE(v_candidates, '{}')) + AND NOT EXISTS (SELECT 1 FROM rate.extension_observations o WHERE o.submitter_id = s.id); + GET DIAGNOSTICS v_submitters = ROW_COUNT; + + RETURN jsonb_build_object('network_hashes_cleared', v_hashes, + 'submitter_network_hashes_cleared', v_submitter_hashes, + 'observations_deleted', v_observations, + 'submitters_deleted', v_submitters); +END; +$$; + +-- 9. Execute grants ---------------------------------------------------------------------------- + +REVOKE EXECUTE ON FUNCTION rate.register_extension_submitter(text, text) + FROM PUBLIC, anon, authenticated, service_role, rate_worker; +REVOKE EXECUTE ON FUNCTION rate.submit_extension_observation(text, uuid, jsonb, text) + FROM PUBLIC, anon, authenticated, service_role, rate_worker; +REVOKE EXECUTE ON FUNCTION rate.delete_extension_submitter(uuid) + FROM PUBLIC, anon, authenticated, service_role, rate_worker; +REVOKE EXECUTE ON FUNCTION rate.purge_extension_observations(integer, integer, integer) + FROM PUBLIC, anon, authenticated, service_role, rate_worker; + +-- register and submit stay open to anon: the web calls them with the anon key plus the ingest +-- key. delete takes no ingest key, so erasure always works. +GRANT EXECUTE ON FUNCTION rate.register_extension_submitter(text, text) + TO anon, authenticated, service_role; +GRANT EXECUTE ON FUNCTION rate.submit_extension_observation(text, uuid, jsonb, text) + TO anon, authenticated, service_role; +GRANT EXECUTE ON FUNCTION rate.delete_extension_submitter(uuid) + TO anon, authenticated, service_role; +GRANT EXECUTE ON FUNCTION rate.purge_extension_observations(integer, integer, integer) + TO service_role; + +COMMIT; diff --git a/supabase/migrations/20261106160100_rate_extension_ingest_key.sql b/supabase/migrations/20261106160100_rate_extension_ingest_key.sql new file mode 100644 index 000000000..a5ff20ec1 --- /dev/null +++ b/supabase/migrations/20261106160100_rate_extension_ingest_key.sql @@ -0,0 +1,15 @@ +-- ============================================================================= +-- Migration: 20261106160100_rate_extension_ingest_key.sql +-- Purpose (DOS/Rate#1606): sets the ingest key row of 20261106160000_rate_extension_observations.sql, +-- step 5 of the proposal's apply runbook (Docs/Proposals/2026-11-04-dos-me-extension-observations.md). +-- JOY approved on 2026-11-04: "Duyệt M2". Only the sha256 of the web's RATE_EXTENSION_INGEST_KEY +-- is stored; the key itself is never in git. The key was generated with a CSPRNG (32 random +-- bytes, hex) and hashed over its exact bytes; it is set on the Vercel environments of the +-- DOS/Rate web that use this database. +-- Rotation: a new migration with the new hash, then the web redeploys with the new key (writes +-- answer 503 in between; deletes keep working). +-- ============================================================================= + +INSERT INTO rate.extension_ingest_secret (key_sha256) +VALUES ('4e6e92c9d3114d98ad3ee2a13de33be70865adc1723cbf8f84b7db613217d55b') +ON CONFLICT (id) DO UPDATE SET key_sha256 = EXCLUDED.key_sha256, updated_at = now(); diff --git a/supabase/migrations/20261106160200_rate_extension_observations_purge_schedule.sql b/supabase/migrations/20261106160200_rate_extension_observations_purge_schedule.sql new file mode 100644 index 000000000..4e5b60aea --- /dev/null +++ b/supabase/migrations/20261106160200_rate_extension_observations_purge_schedule.sql @@ -0,0 +1,35 @@ +-- ============================================================================= +-- Migration: 20261106160200_rate_extension_observations_purge_schedule.sql +-- Purpose (DOS/Rate#1606): step 8 of the apply runbook of +-- 20261106160000_rate_extension_observations.sql: run the retention purge daily. JOY approved +-- the M2 apply on 2026-11-04 in the Rate Core PO chat: "Duyệt M2". +-- +-- rate.purge_extension_observations() with its defaults clears the network hashes after 28 +-- days, deletes observations after 180 days and empty submitters after 7 days, as the +-- privacy policy and the extension's disclosures say. The job runs as its owner (postgres), +-- like dosai_audit_log_retention. Daily at 03:30 UTC (10:30 Vietnam time). +-- +-- Idempotent: an existing job of the same name is unscheduled first. Where pg_cron is not +-- installed (the disposable replay database) it does nothing and says so; Beta and +-- Production have pg_cron. +-- Verify: SELECT jobname, schedule, command FROM cron.job +-- WHERE jobname = 'rate_extension_observations_purge'; -- one row +-- Rollback: SELECT cron.unschedule('rate_extension_observations_purge'); +-- ============================================================================= + +DO $$ +BEGIN + IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pg_cron') THEN + RAISE NOTICE 'pg_cron is not installed: rate_extension_observations_purge not scheduled'; + RETURN; + END IF; + IF EXISTS (SELECT 1 FROM cron.job WHERE jobname = 'rate_extension_observations_purge') THEN + PERFORM cron.unschedule('rate_extension_observations_purge'); + END IF; + PERFORM cron.schedule( + 'rate_extension_observations_purge', + '30 3 * * *', + $cmd$ SELECT rate.purge_extension_observations(); $cmd$ + ); +END +$$; diff --git a/supabase/migrations/20261109090000_rate_extension_ingest_key_rotation.sql b/supabase/migrations/20261109090000_rate_extension_ingest_key_rotation.sql new file mode 100644 index 000000000..f50569ed8 --- /dev/null +++ b/supabase/migrations/20261109090000_rate_extension_ingest_key_rotation.sql @@ -0,0 +1,16 @@ +-- ============================================================================= +-- Migration: 20261109090000_rate_extension_ingest_key_rotation.sql +-- Version moved from 20261107100000, which two other DOS.Me migrations also used +-- (nano_banana_2_1_zdr_data_retention, dosafe_close_anon_reads); the body is unchanged. +-- Purpose (DOS/Rate#1606): rotates the ingest key row set by +-- 20261106160100_rate_extension_ingest_key.sql, as the apply runbook says ("a new migration +-- with the new hash, then the web redeploys with the new key"). The first key was lost before +-- it was set on Vercel, so nothing ever used it; a new key, 32 CSPRNG bytes in hex, is now set +-- on the DOS/Rate web's Vercel environments (JOY, 2026-11-05: "Cho đặt lên Vercel"). Only its +-- sha256 is stored; the key itself is never in git. The web flag RATE_EXTENSION_OBSERVATIONS is +-- still off, so no write depended on the old row. +-- ============================================================================= + +INSERT INTO rate.extension_ingest_secret (key_sha256) +VALUES ('aff995fa98271f0b1bb45f92567da3d425287726ccf759aa44e7cd3234aa5979') +ON CONFLICT (id) DO UPDATE SET key_sha256 = EXCLUDED.key_sha256, updated_at = now(); From 1ba73251a0d80c415f34ac86c0d1611594e15559 Mon Sep 17 00:00:00 2001 From: JOY <5027251+JOY@users.noreply.github.com> Date: Thu, 5 Nov 2026 23:54:47 +0700 Subject: [PATCH 7/7] chore(db): mirror the four older drift migrations from dos-me (#1612) * chore(db): mirror the four older drift migrations from dos-me DOS/DOS.Me is the only deployable ledger. Four rate migrations differed between it and supabase/migrations: - 20261013192302_jreviews_rating_and_review_flags.sql and 20261014011054_review_votes.sql: the DOS.Me version is the one on production (no explanatory comments; review_flags policies created only when missing). Replace the reference copies with it. - 20261025090000_rate_product_images_bucket.sql: only in DOS.Me. It needs the Supabase storage schema, so the replay skips it, as DOS.Me's own replay does (NOT_REPLAYED in scripts/lib/rate-db.mjs). - 20261106150000_rate_listing_price_context_peer_max_history_first.sql: only in DOS.Me (Rate-Services price context). Mirrored, and database.types.ts regenerated: history_first_at and peer_reference_max. Nothing is applied by this change. Refs #1606 Co-Authored-By: Claude Sonnet 5.5 --- packages/shared-data/src/database.types.ts | 2 + scripts/lib/rate-db.mjs | 11 +- ...92302_jreviews_rating_and_review_flags.sql | 27 +-- .../20261014011054_review_votes.sql | 117 +++++------ ...61025090000_rate_product_images_bucket.sql | 63 ++++++ ...g_price_context_peer_max_history_first.sql | 182 ++++++++++++++++++ 6 files changed, 325 insertions(+), 77 deletions(-) create mode 100644 supabase/migrations/20261025090000_rate_product_images_bucket.sql create mode 100644 supabase/migrations/20261106150000_rate_listing_price_context_peer_max_history_first.sql diff --git a/packages/shared-data/src/database.types.ts b/packages/shared-data/src/database.types.ts index 8b40c7fc3..1ae3fb041 100644 --- a/packages/shared-data/src/database.types.ts +++ b/packages/shared-data/src/database.types.ts @@ -1569,11 +1569,13 @@ export type Database = { cluster_official_min: number; history_avg: number; history_count: number; + history_first_at: string; history_max: number; history_median: number; history_min: number; merchant_id: string; merchant_is_official: boolean; + peer_reference_max: number; peer_reference_min: number; platform: string; price_current: number; diff --git a/scripts/lib/rate-db.mjs b/scripts/lib/rate-db.mjs index 01fd11c7f..57e1616b3 100644 --- a/scripts/lib/rate-db.mjs +++ b/scripts/lib/rate-db.mjs @@ -13,10 +13,19 @@ export const repoRoot = path.resolve( '..' ); +// Migrations the throwaway database cannot apply, and the test that covers each +// instead (the same list DOS/DOS.Me keeps in scripts/migration/test-rate-schema-replay.mjs). +export const NOT_REPLAYED = new Map([ + [ + '20261025090000_rate_product_images_bucket.sql', + 'needs the Supabase storage schema; covered by DOS/DOS.Me scripts/ci/rate-product-images-bucket.integration.test.mjs', + ], +]); + export function readMigrations() { const migDir = path.join(repoRoot, 'supabase', 'migrations'); return readdirSync(migDir) - .filter((f) => f.endsWith('.sql')) + .filter((f) => f.endsWith('.sql') && !NOT_REPLAYED.has(f)) .sort() .map((f) => ({ name: f, sql: readFileSync(path.join(migDir, f), 'utf8') })); } diff --git a/supabase/migrations/20261013192302_jreviews_rating_and_review_flags.sql b/supabase/migrations/20261013192302_jreviews_rating_and_review_flags.sql index 0ad23c5f0..51501aa4a 100644 --- a/supabase/migrations/20261013192302_jreviews_rating_and_review_flags.sql +++ b/supabase/migrations/20261013192302_jreviews_rating_and_review_flags.sql @@ -1,8 +1,3 @@ --- ============================================================================== --- Migration: JReviews-Style Dual Rating (Expert vs Weighted User), --- Review Upvote/Downvote Weighting, 5-Category Anti-Scam Flags & Multi-Entity Support --- ============================================================================== - -- 1. Extend rate.review_flags with 5 distinct Vietnamese e-commerce & service flag categories ALTER TABLE rate.review_flags ADD COLUMN IF NOT EXISTS flag_category TEXT DEFAULT 'authentic' @@ -54,7 +49,6 @@ DECLARE v_net INTEGER := COALESCE(p_upvotes, 0) - COALESCE(p_downvotes, 0); v_weight NUMERIC; BEGIN - -- Skip/Ignore coin spam, seeding fake, competitor sabotage, and review bombing completely IF p_flag_category IN ('low_value_coin_spam', 'seeding_fake', 'competitor_sabotage', 'review_bombing') OR p_verdict = 'fake' THEN RETURN 0.00; @@ -72,18 +66,27 @@ BEGIN v_weight := v_base; END IF; - -- Penalty for user abuse reports v_weight := v_weight - (COALESCE(p_reports, 0) * 0.35); RETURN ROUND(GREATEST(0.00, LEAST(3.00, v_weight)), 2); END; $$; --- Public read access for review_flags so UI can display transparency badges ALTER TABLE rate.review_flags ENABLE ROW LEVEL SECURITY; -DROP POLICY IF EXISTS "Public can view review_flags" ON rate.review_flags; -CREATE POLICY "Public can view review_flags" ON rate.review_flags FOR SELECT USING (true); -DROP POLICY IF EXISTS "Service role write review_flags" ON rate.review_flags; -CREATE POLICY "Service role write review_flags" ON rate.review_flags FOR ALL TO service_role USING (true) WITH CHECK (true); + +DO $$ +BEGIN + IF NOT EXISTS ( + SELECT 1 FROM pg_policies WHERE schemaname = 'rate' AND tablename = 'review_flags' AND policyname = 'Public can view review_flags' + ) THEN + CREATE POLICY "Public can view review_flags" ON rate.review_flags FOR SELECT USING (true); + END IF; + IF NOT EXISTS ( + SELECT 1 FROM pg_policies WHERE schemaname = 'rate' AND tablename = 'review_flags' AND policyname = 'Service role write review_flags' + ) THEN + CREATE POLICY "Service role write review_flags" ON rate.review_flags FOR ALL TO service_role USING (true) WITH CHECK (true); + END IF; +END $$; + GRANT SELECT ON rate.review_flags TO anon, authenticated; GRANT ALL ON rate.review_flags TO service_role; diff --git a/supabase/migrations/20261014011054_review_votes.sql b/supabase/migrations/20261014011054_review_votes.sql index 6790db606..61855ea12 100644 --- a/supabase/migrations/20261014011054_review_votes.sql +++ b/supabase/migrations/20261014011054_review_votes.sql @@ -1,20 +1,3 @@ --- ============================================================================== --- Migration: per-user review votes / reports (rate.review_votes) --- --- Design notes --- * One row per (review, voter): a user has ONE up/down vote per review and may --- additionally REPORT it. Changing a vote is an UPDATE, not a second row. --- * Authenticated users only (auth.users, shared Supabase Auth). Anonymous voting is --- the easiest brigading vector (review_bombing / competitor_sabotage), so it is --- deliberately not supported in v1. --- * Counters on rate.reviews (upvotes/downvotes/report_count/helpfulness_weight) are --- ALWAYS recomputed from rate.review_votes (set-based, idempotent -> no drift), and --- helpfulness_weight reuses rate.compute_review_weight so SQL and TS agree. --- * rate.master_products.user_rating_weighted is recomputed across ALL listings of a --- master product. --- * Flood guard: max 60 new votes / hour / user. --- ============================================================================== - CREATE TABLE IF NOT EXISTS rate.review_votes ( review_id UUID NOT NULL REFERENCES rate.reviews(id) ON DELETE CASCADE, voter_id UUID NOT NULL REFERENCES auth.users(id) ON DELETE CASCADE, @@ -34,10 +17,6 @@ CREATE TABLE IF NOT EXISTS rate.review_votes ( CREATE INDEX IF NOT EXISTS idx_rate_review_votes_voter_time ON rate.review_votes (voter_id, created_at DESC); --- ------------------------------------------------------------------------------ --- Aggregate: weighted user rating of a master product across ALL its listings. --- Only analyzed reviews (a review_flags row exists) with weight > 0 count. --- ------------------------------------------------------------------------------ CREATE OR REPLACE FUNCTION rate.refresh_master_product_rating(p_master_id UUID) RETURNS void LANGUAGE sql @@ -64,9 +43,6 @@ AS $$ WHERE m.id = p_master_id; $$; --- ------------------------------------------------------------------------------ --- Recompute counters + weight of one review from review_votes, then its master rating. --- ------------------------------------------------------------------------------ CREATE OR REPLACE FUNCTION rate.refresh_review_vote_counts(p_review_id UUID) RETURNS void LANGUAGE plpgsql @@ -115,9 +91,6 @@ BEGIN END; $$; --- ------------------------------------------------------------------------------ --- Triggers --- ------------------------------------------------------------------------------ CREATE OR REPLACE FUNCTION rate.review_votes_guard() RETURNS trigger LANGUAGE plpgsql @@ -161,52 +134,68 @@ BEGIN END; $$; -DROP TRIGGER IF EXISTS trg_review_votes_guard ON rate.review_votes; -CREATE TRIGGER trg_review_votes_guard - BEFORE INSERT OR UPDATE ON rate.review_votes - FOR EACH ROW EXECUTE FUNCTION rate.review_votes_guard(); +DO $$ +BEGIN + IF NOT EXISTS ( + SELECT 1 FROM pg_trigger WHERE tgname = 'trg_review_votes_guard' + ) THEN + CREATE TRIGGER trg_review_votes_guard + BEFORE INSERT OR UPDATE ON rate.review_votes + FOR EACH ROW EXECUTE FUNCTION rate.review_votes_guard(); + END IF; -DROP TRIGGER IF EXISTS trg_review_votes_after_change ON rate.review_votes; -CREATE TRIGGER trg_review_votes_after_change - AFTER INSERT OR UPDATE OR DELETE ON rate.review_votes - FOR EACH ROW EXECUTE FUNCTION rate.review_votes_after_change(); + IF NOT EXISTS ( + SELECT 1 FROM pg_trigger WHERE tgname = 'trg_review_votes_after_change' + ) THEN + CREATE TRIGGER trg_review_votes_after_change + AFTER INSERT OR UPDATE OR DELETE ON rate.review_votes + FOR EACH ROW EXECUTE FUNCTION rate.review_votes_after_change(); + END IF; +END $$; --- ------------------------------------------------------------------------------ --- RLS: users read/write ONLY their own votes. Public sees only the counters that --- live on rate.reviews (never who voted). --- ------------------------------------------------------------------------------ ALTER TABLE rate.review_votes ENABLE ROW LEVEL SECURITY; -DROP POLICY IF EXISTS "Users read own review votes" ON rate.review_votes; -CREATE POLICY "Users read own review votes" ON rate.review_votes - FOR SELECT TO authenticated USING (voter_id = (SELECT auth.uid())); +DO $$ +BEGIN + IF NOT EXISTS ( + SELECT 1 FROM pg_policies WHERE schemaname = 'rate' AND tablename = 'review_votes' AND policyname = 'Users read own review votes' + ) THEN + CREATE POLICY "Users read own review votes" ON rate.review_votes + FOR SELECT TO authenticated USING (voter_id = (SELECT auth.uid())); + END IF; -DROP POLICY IF EXISTS "Users insert own review votes" ON rate.review_votes; -CREATE POLICY "Users insert own review votes" ON rate.review_votes - FOR INSERT TO authenticated WITH CHECK (voter_id = (SELECT auth.uid())); + IF NOT EXISTS ( + SELECT 1 FROM pg_policies WHERE schemaname = 'rate' AND tablename = 'review_votes' AND policyname = 'Users insert own review votes' + ) THEN + CREATE POLICY "Users insert own review votes" ON rate.review_votes + FOR INSERT TO authenticated WITH CHECK (voter_id = (SELECT auth.uid())); + END IF; -DROP POLICY IF EXISTS "Users update own review votes" ON rate.review_votes; -CREATE POLICY "Users update own review votes" ON rate.review_votes - FOR UPDATE TO authenticated - USING (voter_id = (SELECT auth.uid())) - WITH CHECK (voter_id = (SELECT auth.uid())); + IF NOT EXISTS ( + SELECT 1 FROM pg_policies WHERE schemaname = 'rate' AND tablename = 'review_votes' AND policyname = 'Users update own review votes' + ) THEN + CREATE POLICY "Users update own review votes" ON rate.review_votes + FOR UPDATE TO authenticated + USING (voter_id = (SELECT auth.uid())) + WITH CHECK (voter_id = (SELECT auth.uid())); + END IF; -DROP POLICY IF EXISTS "Users delete own review votes" ON rate.review_votes; -CREATE POLICY "Users delete own review votes" ON rate.review_votes - FOR DELETE TO authenticated USING (voter_id = (SELECT auth.uid())); + IF NOT EXISTS ( + SELECT 1 FROM pg_policies WHERE schemaname = 'rate' AND tablename = 'review_votes' AND policyname = 'Users delete own review votes' + ) THEN + CREATE POLICY "Users delete own review votes" ON rate.review_votes + FOR DELETE TO authenticated USING (voter_id = (SELECT auth.uid())); + END IF; -DROP POLICY IF EXISTS "Service role full access review_votes" ON rate.review_votes; -CREATE POLICY "Service role full access review_votes" ON rate.review_votes - FOR ALL TO service_role USING (true) WITH CHECK (true); + IF NOT EXISTS ( + SELECT 1 FROM pg_policies WHERE schemaname = 'rate' AND tablename = 'review_votes' AND policyname = 'Service role full access review_votes' + ) THEN + CREATE POLICY "Service role full access review_votes" ON rate.review_votes + FOR ALL TO service_role USING (true) WITH CHECK (true); + END IF; +END $$; --- ------------------------------------------------------------------------------ --- Grants. Internal functions are NOT callable through the public API. --- ------------------------------------------------------------------------------ -REVOKE ALL ON rate.review_votes FROM anon; GRANT SELECT, INSERT, UPDATE, DELETE ON rate.review_votes TO authenticated; GRANT ALL ON rate.review_votes TO service_role; - -REVOKE EXECUTE ON FUNCTION rate.refresh_master_product_rating(UUID) FROM PUBLIC, anon, authenticated; -REVOKE EXECUTE ON FUNCTION rate.refresh_review_vote_counts(UUID) FROM PUBLIC, anon, authenticated; GRANT EXECUTE ON FUNCTION rate.refresh_master_product_rating(UUID) TO service_role; -GRANT EXECUTE ON FUNCTION rate.refresh_review_vote_counts(UUID) TO service_role; +GRANT EXECUTE ON FUNCTION rate.refresh_review_vote_counts(UUID) TO service_role; \ No newline at end of file diff --git a/supabase/migrations/20261025090000_rate_product_images_bucket.sql b/supabase/migrations/20261025090000_rate_product_images_bucket.sql new file mode 100644 index 000000000..375693f95 --- /dev/null +++ b/supabase/migrations/20261025090000_rate_product_images_bucket.sql @@ -0,0 +1,63 @@ +-- Rate.vn product image mirror (DOS/Rate#887). +-- +-- Some marketplace CDNs refuse images to pages on rate.vn (hotlink +-- protection), and listings lose their pictures when a marketplace removes +-- them. Rate-Services copies a listing's first images into this bucket at +-- crawl time and stores the public URLs on rate.products.images. +-- +-- Public bucket: rate.vn serves the objects through the Storage public URL, +-- which does not consult RLS. Only service_role (Rate-Services) writes. The +-- restrictive fences keep anon and authenticated from listing, writing or +-- deleting objects here even when a broad Storage policy exists. +-- Idempotent: replays are no-ops, and a bucket or fence that differs from this +-- definition fails the deployment instead of being silently widened. +-- CREATE POLICY locks storage.objects, which every product on the shared +-- project uses: give up after 5s rather than queue Storage behind a blocker. +BEGIN; +SET LOCAL lock_timeout = '5s'; + +INSERT INTO storage.buckets(id, name, public, file_size_limit, allowed_mime_types) +VALUES ('rate-product-images', 'rate-product-images', true, 2097152, + ARRAY['image/jpeg', 'image/png', 'image/webp']) ON CONFLICT (id) DO NOTHING; + +DO $bucket$ +BEGIN + IF NOT EXISTS (SELECT 1 FROM storage.buckets WHERE id = 'rate-product-images' AND name = 'rate-product-images' + AND public = true AND file_size_limit = 2097152 + AND allowed_mime_types = ARRAY['image/jpeg', 'image/png', 'image/webp']) THEN + RAISE EXCEPTION 'RATE_IMAGE_BUCKET_CONFIGURATION_MISMATCH'; + END IF; +END; +$bucket$; + +DO $storage_policies$ +DECLARE v_expected record; v_policy record; v_relation regclass; v_roles oid[]; +BEGIN + SELECT array_agg(oid ORDER BY oid) INTO v_roles FROM pg_catalog.pg_roles WHERE rolname IN ('anon', 'authenticated'); + FOR v_expected IN SELECT * FROM (VALUES + ('objects', 'rate_product_images_server_only', 'bucket_id'), + ('buckets', 'rate_product_image_bucket_server_only', 'id')) AS x(table_name, policy_name, column_name) + LOOP + v_relation := to_regclass('storage.' || v_expected.table_name); + IF NOT EXISTS (SELECT 1 FROM pg_catalog.pg_class WHERE oid = v_relation AND relrowsecurity) THEN + RAISE EXCEPTION 'RATE_IMAGE_STORAGE_RLS_REQUIRED'; + END IF; + SELECT * INTO v_policy FROM pg_catalog.pg_policy WHERE polrelid = v_relation AND polname = v_expected.policy_name; + IF NOT FOUND THEN + EXECUTE format('CREATE POLICY %I ON storage.%I AS RESTRICTIVE FOR ALL TO anon, authenticated + USING (%I <> ''rate-product-images'') WITH CHECK (%I <> ''rate-product-images'')', + v_expected.policy_name, v_expected.table_name, v_expected.column_name, v_expected.column_name); + ELSE + IF v_policy.polpermissive OR v_policy.polcmd <> '*' OR cardinality(v_policy.polroles) <> 2 + OR NOT v_policy.polroles @> v_roles + OR regexp_replace(pg_catalog.pg_get_expr(v_policy.polqual, v_relation), '[[:space:]()]', '', 'g') + IS DISTINCT FROM v_expected.column_name || '<>''rate-product-images''::text' + OR regexp_replace(pg_catalog.pg_get_expr(v_policy.polwithcheck, v_relation), '[[:space:]()]', '', 'g') + IS DISTINCT FROM v_expected.column_name || '<>''rate-product-images''::text' THEN + RAISE EXCEPTION 'RATE_IMAGE_STORAGE_POLICY_MISMATCH'; + END IF; + END IF; + END LOOP; +END; +$storage_policies$; +COMMIT; diff --git a/supabase/migrations/20261106150000_rate_listing_price_context_peer_max_history_first.sql b/supabase/migrations/20261106150000_rate_listing_price_context_peer_max_history_first.sql new file mode 100644 index 000000000..fecf6b9f5 --- /dev/null +++ b/supabase/migrations/20261106150000_rate_listing_price_context_peer_max_history_first.sql @@ -0,0 +1,182 @@ +-- Rate: market reference and history age for the inflated_fake_sale signal +-- (DOS/Rate-Services#582, decided by JOY 2026-11-04). +-- +-- Purpose: the Rate-Services auditors flag `inflated_fake_sale` when a listing's +-- struck-through price is far above the median of its own 60-day price points. +-- A retailer that always sells below the manufacturer's list price (MSRP) and +-- strikes the list price through therefore trips the signal after one week at a +-- flat price, although the struck price is what the official stores sell at. +-- The auditors need two more facts per listing, both computed here because +-- PostgREST cuts max_rows (the reason 20261018160000 moved the aggregates to SQL): +-- +-- peer_reference_max the highest current price among the OTHER priced, +-- in-stock listings of the same cluster, official +-- merchants first, any merchant when the cluster has no +-- other official listing. The struck price is no inflated +-- original price when it does not exceed this by more than +-- the auditor's margin (3%). +-- history_first_at the oldest price point inside the history window. The +-- auditor only concludes once it is at least 14 days old. +-- +-- rate.listing_price_context(...) gains the two columns at the END of its +-- result. A function cannot change its result type with CREATE OR REPLACE, so +-- this drops and recreates it with the same signature and re-grants EXECUTE +-- (DROP removes the grants): service_role as in 20261018160000, rate_worker as +-- in 20261018183000. Both steps run in the migration's transaction, so no caller +-- sees the function missing. The body is that of 20261020230000 plus the two +-- columns, with the aggregates of the peers and median expressions schema-qualified +-- like those of the history expression (pg_catalog is searched anyway, so the +-- result is the same); nothing else changes. Nothing depends on the function (no view, +-- trigger or other function), so the DROP needs no CASCADE and fails loudly if +-- one appears. +-- +-- Idempotent: DROP FUNCTION IF EXISTS, then the full definition and the grants +-- again; a second run leaves the same function. Touches the `rate` schema only. + +DROP FUNCTION IF EXISTS rate.listing_price_context(UUID, INTEGER, INTEGER, UUID[]); + +CREATE FUNCTION rate.listing_price_context( + p_after UUID DEFAULT NULL, + p_limit INTEGER DEFAULT 500, + p_history_days INTEGER DEFAULT 60, + p_merchant_ids UUID[] DEFAULT NULL +) +RETURNS TABLE ( + product_id UUID, + cluster_id UUID, + merchant_id UUID, + merchant_is_official BOOLEAN, + platform TEXT, + title TEXT, + price_current NUMERIC, + price_original NUMERIC, + -- Minimum price among the OTHER priced, in-stock listings of the cluster, + -- official merchants first, any merchant when the cluster has no other + -- official listing (MerchantRiskAuditor.computePeerReferenceMinPrice). + peer_reference_min NUMERIC, + -- Minimum official-merchant price of the cluster's in-stock listings, this + -- listing included while it is in stock (PriceAnomalyAuditor.computeOfficialMinByCluster). + cluster_official_min NUMERIC, + history_count INTEGER, + history_avg NUMERIC, + history_median NUMERIC, + history_min NUMERIC, + history_max NUMERIC, + -- Maximum price among the OTHER priced, in-stock listings of the cluster, + -- chosen like peer_reference_min: official merchants first, any merchant when + -- the cluster has no other official listing. NULL when no other listing of + -- the cluster is in stock. + peer_reference_max NUMERIC, + -- recorded_at of the oldest crawled price point inside the history window + -- (the same points history_count counts); NULL when there is none. + history_first_at TIMESTAMPTZ +) +LANGUAGE plpgsql +STABLE +SET search_path = '' +AS $$ +DECLARE + -- Index-friendly predicates: no "param IS NULL OR ..." (which forces a + -- scan in a generic plan); the nil uuid sorts before every real id. + v_after UUID := COALESCE(p_after, '00000000-0000-0000-0000-000000000000'::uuid); + v_limit INTEGER := LEAST(GREATEST(COALESCE(p_limit, 500), 1), 1000); + v_since TIMESTAMPTZ := pg_catalog.now() - pg_catalog.make_interval(days => GREATEST(COALESCE(p_history_days, 60), 1)); + v_page UUID[]; +BEGIN + IF p_merchant_ids IS NULL THEN + SELECT pg_catalog.array_agg(x.id ORDER BY x.id) INTO v_page + FROM ( + SELECT p.id + FROM rate.products p + WHERE p.id > v_after + AND p.origin = 'crawl' + AND p.price_current > 0 + ORDER BY p.id + LIMIT v_limit + ) x; + ELSE + SELECT pg_catalog.array_agg(x.id ORDER BY x.id) INTO v_page + FROM ( + SELECT p.id + FROM rate.products p + WHERE p.merchant_id = ANY (p_merchant_ids) + AND p.id > v_after + AND p.origin = 'crawl' + AND p.price_current > 0 + ORDER BY p.id + LIMIT v_limit + ) x; + END IF; + + IF v_page IS NULL THEN + RETURN; + END IF; + + RETURN QUERY + SELECT + pg.id, + pg.cluster_id, + pg.merchant_id, + COALESCE(m.is_official, false), + pg.platform, + pg.title, + pg.price_current, + pg.price_original, + peers.peer_reference_min, + peers.cluster_official_min, + COALESCE(h.n, 0)::INTEGER, + h.avg_price, + h.median_price::NUMERIC, + h.min_price, + h.max_price, + peers.peer_reference_max, + h.first_at + FROM rate.products pg + JOIN rate.merchants m ON m.id = pg.merchant_id + LEFT JOIN LATERAL ( + SELECT + COALESCE( + pg_catalog.min(o.price_current) FILTER (WHERE o.id <> pg.id AND om.is_official), + pg_catalog.min(o.price_current) FILTER (WHERE o.id <> pg.id) + ) AS peer_reference_min, + pg_catalog.min(o.price_current) FILTER (WHERE om.is_official) AS cluster_official_min, + COALESCE( + pg_catalog.max(o.price_current) FILTER (WHERE o.id <> pg.id AND om.is_official), + pg_catalog.max(o.price_current) FILTER (WHERE o.id <> pg.id) + ) AS peer_reference_max + FROM rate.products o + JOIN rate.merchants om ON om.id = o.merchant_id + WHERE pg.cluster_id IS NOT NULL + AND o.cluster_id = pg.cluster_id + AND o.origin = 'crawl' + AND o.price_current > 0 + AND o.availability = 'in_stock' + ) peers ON true + LEFT JOIN LATERAL ( + SELECT + pg_catalog.count(*) AS n, + pg_catalog.avg(ph.price) AS avg_price, + pg_catalog.percentile_cont(0.5) WITHIN GROUP (ORDER BY ph.price) AS median_price, + pg_catalog.min(ph.price) AS min_price, + pg_catalog.max(ph.price) AS max_price, + pg_catalog.min(ph.recorded_at) AS first_at + FROM rate.price_history ph + WHERE ph.product_id = pg.id + AND ph.origin = 'crawl' + AND ph.price > 0 + AND ph.recorded_at >= v_since + ) h ON true + WHERE pg.id = ANY (v_page) + ORDER BY pg.id; +END; +$$; + +REVOKE ALL ON FUNCTION rate.listing_price_context(UUID, INTEGER, INTEGER, UUID[]) FROM PUBLIC, anon, authenticated; +GRANT EXECUTE ON FUNCTION rate.listing_price_context(UUID, INTEGER, INTEGER, UUID[]) TO service_role; + +DO $$ +BEGIN + IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'rate_worker') THEN + GRANT EXECUTE ON FUNCTION rate.listing_price_context(UUID, INTEGER, INTEGER, UUID[]) TO rate_worker; + END IF; +END $$;
    • ModelGiá đọc vào / viết ra (USD mỗi 1 triệu token)Nhận xét
    Claude Haiku 5.5 (claude-haiku-5-5)0,10 / 0,50Rẻ nhất. Việc này đơn giản: đọc một đoạn ngắn, rút ý có trích dẫn. Đề xuất dùng mặc định.