diff --git a/CHANGELOG.md b/CHANGELOG.md index 9544c15..016929c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,7 +15,8 @@ Notable changes to Papilot, newest first. The format follows [Keep a Changelog]( - **Claude on your Claude plan.** With Claude Code installed and signed in, Claude tasks run `claude -p` on that login, so a Claude plan works without an API key. Edits in the worktree are accepted, commands run in Claude Code's sandbox, anything that would need approval is declined and noted, and each task resumes its session. Without Claude Code, Claude still runs through the Agent SDK with an API key. - **More agents, found automatically.** Cursor (through the Cursor SDK, with `CURSOR_API_KEY`), GLM (Claude Code on Zhipu's Anthropic-compatible endpoint, with `ZHIPU_API_KEY` or `ZAI_API_KEY`), DeepSeek (`dsh acp` from DeepSeek Harness), Qwen Code (`qwen --acp`) and OpenCode (`opencode acp`) join the agent menu. Papilot offers only the agents it finds on the host, looks again on **Reconnect**, and names the rest in Settings → Runtime & diagnostics. DeepSeek keeps its workspace sandbox, Qwen runs in Auto Edit mode, and OpenCode asks before commands and anything outside the worktree. `papilot doctor` checks each of them and lists agents that are not set up for information only. - +- **Bibliography checks.** A change to a `.bib` file is reviewed entry by entry, with every field and where the paper cites each entry; hunks of one entry are decided together. Papilot looks up added and modified entries in Crossref, DBLP and arXiv (DOIs through doi.org) and marks each as verified, differing from its record or not verified, with the record beside it. Accepting an entry that is not verified asks first, and **Accept all** leaves it for you. Removed entries the paper still cites, duplicate keys and new citations without an entry are flagged too, and agents are told never to invent references. Turn the lookups off in **Settings → Check references online**. +- **Reference check.** Under **More** in the sidebar, check the whole bibliography: citations without an entry, duplicate keys, the same work under two keys, uncited entries, and each entry against Crossref, DBLP and arXiv. Mark an entry no source knows as **I checked it myself**, which reviews then honour, and ask an agent to find out what happened with the rest, in a new task you can edit first; it reports and changes nothing. - **Kimi Code support.** Choose Kimi for paper-editing tasks using your local Kimi Code CLI login. The ACP adapter supports model and thinking-level selection, streamed replies, image inputs, tool approvals, cancellation and session recovery. Diagnostics include Kimi CLI availability; set `PAPILOT_KIMI_BIN` for a custom executable path. ### Changed diff --git a/docs/guide.md b/docs/guide.md index 9b71451..6b66ee1 100644 --- a/docs/guide.md +++ b/docs/guide.md @@ -117,6 +117,10 @@ Titles (the brand, paper names, task and page headings) use the manuscript's TeX The current review shows action bars and map markers only for pending changes. Accepted changes show the clean revised PDF; rejected changes show the clean original. Commented changes leave the pending list while their feedback waits to be sent. If one PDF region contains a mix of accepted and rejected edits, open **History** to inspect it; the viewer does not pretend the entire region was accepted. **History** shows the diffs and comments from any round. **Undo last decision** restores the most recent decision in the latest round, returning those changes to the current review. Earlier rounds are read-only. After sending feedback, the viewer shows the agent's progress; if the agent makes no new edits, the completed review stays clear of old Change bars. +**Bibliography changes** are reviewed entry by entry rather than as source lines. Each entry a change adds, modifies or removes in a `.bib` file appears whole under that change: every field with its value as written (a modified field shows its old and new value), the lines it occupies, and where the paper cites it. Hunks that touch the same entry make one change, so an entry is never half accepted; the raw source stays under **LaTeX**. Papilot then looks up each added or modified entry online: a DOI through doi.org, an arXiv identifier through arXiv, and otherwise the title and authors in Crossref, DBLP and arXiv. An entry is **Verified** when a record agrees in title, authors and year, **Differs from the record** when the same work has other details or its DOI names another work (the differing fields are listed against the record's, with a link to it), **Not verified** when no record matches, and **Could not check** when no source could be reached. Not verified is not proof that a work does not exist: theses, reports and many books are in none of these sources. A preprint's year before the published version's, or a subtitle, counts as a small difference only. A source that turns requests away, as DBLP's bot check sometimes does, is skipped, and the entry names it. The **Bibliography** line above the review counts the entries and lists those that need a look, together with removed entries the paper still cites, keys defined twice, entries in a `.bib` file the paper does not read, and new citations with no entry. Accepting (or staging) a change with such an entry asks first, and **Accept all** leaves those changes for you to decide one by one. An entry no source knows, such as a book or a thesis, can be marked **I checked it myself**: it then counts as checked in every later review, until its title, authors, year or identifiers change. Only an entry's title, authors, year and identifiers are sent; turn the lookups off with **Settings → Check references online**, which applies to this browser. Results are kept for a month, or a day when nothing matched, in the repository's `.git/latex-review/references.json`, and in the data folder for uncommitted changes. Agents are told never to write an entry from memory and to include its DOI or arXiv identifier. + +**Reference check**, under **More** below **Manuscript** in the sidebar, checks the paper's whole bibliography as it is on disk: the `.bib` files named by `\bibliography` or `\addbibresource`, and the citations in every file the paper inputs. Without looking anything up it lists citations of keys no entry defines, keys defined twice, the same work under two keys (same DOI or title), entries the paper never cites (unless it uses `\nocite{*}`) and entries BibTeX cannot read. **Check online** compares each entry with Crossref, DBLP and arXiv as reviews do; results are kept, so later checks look up only new or edited entries, and **Check again** retries those that found nothing. **Problems** lists what needs a look, **All entries** every entry; open one to see all its fields, where it is cited, and the record it was compared with. **I checked it myself** works here as in reviews. **Ask an agent to look into** opens the manuscript's new task with the ticked problems in its prompt, which you can edit before sending. The agent is asked to change nothing and to find out what happened with each: whether the work exists and what its published record says, where the entry or citation came from in the Git history, and what it would suggest, marking which findings it confirmed in a source. What to change stays your decision. **More** opens its tools beside the sidebar (a sheet on a phone); once the check has run, the menu shows how many problems it found. + Conversation drafts are saved per task in this browser. Sending review comments preserves an unsent draft, and text entered while a message is sending stays in the composer. Switching tasks preserves pending submissions and unfinished Codex answers. If browser storage is unavailable, the workspace remains usable and drafts stay in the current tab until it closes or reloads. Press Enter to send and Shift+Enter for a new line. On touch keyboards, Enter adds a new line; use the send button. **Command+Enter** on macOS or **Ctrl+Enter** elsewhere always sends. Input-method composition is never sent by Enter. The new-task prompt is also saved in this browser until the task is created. Agent replies render Markdown, including tables, task lists, nested lists, fenced code and LaTeX mathematics (`$…$`, `$$…$$`, `\(…\)` and `\[…\]`, typeset with KaTeX, which loads the first time a reply contains mathematics). Your own messages show what you did: a skill as its name with your words, sent review feedback as its decisions and comments, and PDF notes as the notes, never the prompt the agent receives. Local file references copy their full path. The **Add a paper** and delete dialogs keep keyboard focus inside them; **Escape** closes them when no submission is in progress. The mobile task drawer keeps keyboard focus inside it and closes with **Escape**, returning focus to its opener. Long conversations follow new messages until you scroll up; **Latest messages** returns to the bottom. Connection loss is shown above the workspace even when the sidebar is hidden. Failed conversation and PDF review loads offer their own retry controls, so your instruction draft stays in place. @@ -309,6 +313,6 @@ To release, set the version in `package.json` and `package-lock.json`, give it a ## Review engine -`vendor/latex-review` is based on the existing MIT-licensed `latex-review-skill`. Its upstream revision is recorded in `UPSTREAM_COMMIT`; its license and source are preserved. A small viewer integration honors Papilot's decision availability state. Papilot's copy also compares the whole PDFs to list the places an edit changes away from its own lines, as its reference describes. Papilot uses the engine's public HTTP interface and does not write its SQLite database. The embedded review gateway only exposes review assets and review decisions, not shutdown or task-deletion endpoints. +`vendor/latex-review` is based on the existing MIT-licensed `latex-review-skill`. Its upstream revision is recorded in `UPSTREAM_COMMIT`; its license and source are preserved. A small viewer integration honors Papilot's decision availability state. Papilot's copy also compares the whole PDFs to list the places an edit changes away from its own lines, as its reference describes. It also reads `.bib` changes entry by entry and checks added and modified entries online (`src/core/bib.ts`, `src/core/verify.ts`, and `GET /api/tasks/:name/rounds/:number/references`). Papilot uses the engine's public HTTP interface and does not write its SQLite database. The embedded review gateway only exposes review assets and review decisions, not shutdown or task-deletion endpoints. See [REQUIREMENTS.md](../REQUIREMENTS.md) for the initial scope and acceptance criteria. diff --git a/src/server/app.ts b/src/server/app.ts index 98c92b4..ae64386 100644 --- a/src/server/app.ts +++ b/src/server/app.ts @@ -11,6 +11,7 @@ import type { Workspace } from './workspace.ts'; import { reviewClient, allowedReviewRoute, rewriteReviewHtml, rewriteReviewJson, confirmedReviewResponse, annotateReviewState } from './review.ts'; import { fullPaperPdf } from './pdf.ts'; import { type ChangeAction, Changes } from './changes.ts'; +import { bibliography, confirmEntry } from './references.ts'; import { type Manuscripts, annotationTask, gitAbortMerge, gitCommit, gitDiff, gitFinishMerge, gitIgnore, gitRemote, gitStage, gitStatus } from './manuscript.ts'; import { gitOverview } from './git-overview.ts'; import { findSkill, listedSkill, loadSkills } from './skills.ts'; @@ -263,6 +264,14 @@ export async function buildApp(options: AppOptions) { // The manuscript preview compiles each paper's working tree and reports its Git status. app.get<{ Params: { id: string } }>('/api/projects/:id/manuscript', project(), request => manuscript(request.params.id)); app.get<{ Params: { id: string } }>('/api/projects/:id/git', project(), request => gitStatus(store.project(request.params.id).path)); + // The paper's bibliography: every entry, its problems and its online check. + app.get<{ Params: { id: string }; Querystring: { check?: string; retry?: string } }>('/api/projects/:id/references', project(), request => + bibliography(store.project(request.params.id).path, request.query.check === '1', request.query.retry === '1')); + app.post<{ Params: { id: string } }>('/api/projects/:id/references/confirm', project(), request => { + const b = request.body as any; + if (typeof b?.id !== 'string' || typeof b?.confirmed !== 'boolean') throw new Error('Choose an entry to confirm.'); + return confirmEntry(store.project(request.params.id).path, b.id, b.confirmed); + }); app.get<{ Params: { id: string } }>('/api/projects/:id/git/overview', project(), request => gitOverview(store.project(request.params.id).path, store.tasks().filter(task => task.projectId === request.params.id))); app.get<{ Params: { id: string }; Querystring: { path?: string } }>('/api/projects/:id/git/diff', project(), request => gitDiff(store.project(request.params.id).path, string(request.query.path, 'File path', 4096))); const paths = (value: unknown): string[] => Array.isArray(value) ? value.filter((p: unknown): p is string => typeof p === 'string').slice(0, 5000) : []; @@ -427,6 +436,16 @@ export async function buildApp(options: AppOptions) { app.get<{ Params: { projectId: string; number: string } }>('/review/:projectId/api/changes/rounds/:number', changesAccess, request => changes.detail(store.project(request.params.projectId).id, Number(request.params.number))); app.get<{ Params: { projectId: string; number: string; unit: string } }>('/review/:projectId/api/changes/rounds/:number/effects/:unit', changesAccess, request => changes.effects(store.project(request.params.projectId).id, Number(request.params.number), request.params.unit)); + app.get<{ Params: { projectId: string; number: string }; Querystring: { check?: string; retry?: string } }>('/review/:projectId/api/changes/rounds/:number/references', changesAccess, request => { + const project = store.project(request.params.projectId); + return changes.references(project.id, project.path, Number(request.params.number), request.query.check === '1', request.query.retry === '1'); + }); + app.post<{ Params: { projectId: string; number: string } }>('/review/:projectId/api/changes/rounds/:number/references/confirm', changesAccess, request => { + changesToken(request); + const project = store.project(request.params.projectId), b = request.body as any; + if (typeof b?.id !== 'string' || typeof b?.confirmed !== 'boolean') throw new Error('Choose an entry to confirm.'); + return changes.confirmReference(project.id, project.path, Number(request.params.number), b.id, b.confirmed); + }); app.get<{ Params: { projectId: string; number: string; file: string } }>('/review/:projectId/changes/files/:number/:file', changesAccess, (request, reply) => { const file = changes.file(store.project(request.params.projectId).id, Number(request.params.number), request.params.file); return reply.header('Cache-Control', 'private, max-age=31536000, immutable').type(file.endsWith('.svg') ? 'image/svg+xml' : 'image/png').send(fs.createReadStream(file)); @@ -475,7 +494,7 @@ export async function buildApp(options: AppOptions) { const task = store.tasks().find(t => t.projectId === project.id && t.name === route.split('/')[3]); if (task) store.updateTask(task.id, { error: 'Review action failed: ' + (value as any).error }); } - if (response.ok && request.method === 'POST') { + if (response.ok && request.method === 'POST' && !route.endsWith('/references/confirm')) { const task = store.tasks().find(t => t.projectId === project.id && t.name === route.split('/')[3]); if (task) { store.updateTask(task.id, { error: null }); diff --git a/src/server/changes.ts b/src/server/changes.ts index ac49e14..0889019 100644 --- a/src/server/changes.ts +++ b/src/server/changes.ts @@ -5,6 +5,7 @@ import { pathToFileURL } from 'node:url'; import { DATA, ENGINE } from './paths.ts'; import { mergeInProgress, workingTree } from './manuscript.ts'; import { ignorePattern } from '../commit.ts'; +import { checker, confirmations, engine as referenceEngine } from './references.ts'; // The manuscript's uncommitted changes as a PDF comparison: the last commit // against the working tree, rendered by the review engine like a task's round @@ -197,14 +198,36 @@ export class Changes { detail(projectId: string, number: number) { const round = this.comparison(projectId).rounds.find(item => item.number === number); if (!round) throw Object.assign(new Error('This comparison is out of date. Refresh the page.'), { statusCode: 404 }); - let render: { units: Record; sections: unknown[]; loose: unknown[] } = { units: {}, sections: [], loose: [] }; - try { render = JSON.parse(fs.readFileSync(path.join(round.folder, 'round.json'), 'utf8')); } catch { /* An empty comparison has no render. */ } + const render = this.rendered(round); return { task: 'changes', round: round.number, before: round.before, after: round.after, units: round.units.map(unit => ({ ...unit, ...render.units[unit.id] })), sections: render.sections, loose: render.loose, warnings: round.warnings, files: `/review/${projectId}/changes/files/${round.number}/`, + ...(render.references ? { references: render.references } : {}), }; } + private rendered(round: Round): { units: Record; sections: unknown[]; loose: unknown[]; references?: unknown } { + try { return JSON.parse(fs.readFileSync(path.join(round.folder, 'round.json'), 'utf8')); } catch { return { units: {}, sections: [], loose: [] }; } // An empty comparison has no render. + } + + // Online checks of the bibliography entries the uncommitted changes add or + // modify, as for a task's round; check asks for lookups to start. + async references(projectId: string, root: string, number: number, check: boolean, retry: boolean) { + const inputs = await this.entryInputs(projectId, number); + return { ...(await checker()).lookup(inputs, check, retry), confirmed: (await confirmations(root)).of(inputs) }; + } + // The author checked an entry by hand, or takes that back. + async confirmReference(projectId: string, root: string, number: number, id: string, on: boolean) { + const inputs = await this.entryInputs(projectId, number), input = inputs.find(item => item.id === id); + if (!input) throw Object.assign(new Error('This entry is no longer among the changes. Refresh the page.'), { statusCode: 404 }); + (await confirmations(root)).set(input, on); + return this.references(projectId, root, number, false, false); + } + private async entryInputs(projectId: string, number: number) { + const round = this.comparison(projectId).rounds.find(item => item.number === number); + if (!round) throw Object.assign(new Error('This comparison is out of date. Refresh the page.'), { statusCode: 404 }); + return (await referenceEngine()).checkInputs(this.rendered(round).references); + } // The places a change alters away from its own lines, past those the // comparison rendered, rendered into its folder when first asked for. diff --git a/src/server/references.ts b/src/server/references.ts new file mode 100644 index 0000000..b12d121 --- /dev/null +++ b/src/server/references.ts @@ -0,0 +1,70 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import { pathToFileURL } from 'node:url'; +import { DATA, ENGINE } from './paths.ts'; +import type { BibliographyReport, ReferenceCheck } from '../shared.ts'; + +// The paper's bibliography as it is on disk: every entry, where it is cited, +// what is wrong with it without looking anything up, and, when asked, what +// Crossref, DBLP and arXiv know of it. The review engine does the reading +// and the checking; this is where Papilot asks it for a whole paper. + +type Input = { id: string; type: string; fields: Record }; +interface Checker { lookup(inputs: Input[], start?: boolean, retry?: boolean): { checks: Record; pending: number } } +interface Confirmations { of(inputs: Input[]): Record; set(input: Input, on: boolean): void } +interface Engine { + bibliographyReport(doc: { read(file: string): string | null; main: string; outline: unknown[] }): Omit; + buildOutline(read: (file: string) => string | null, main: string): { nodes: unknown[] }; + loadProject(root: string): { main: string }; + checkInputs(references: unknown): Input[]; + entryInputs(entries: unknown[]): Input[]; + ReferenceChecker: new (file: string) => Checker; + Confirmations: new (file: string) => Confirmations; + Repo: new (root: string) => { gitDir(): string }; +} +let loading: Promise | undefined; +export function engine(): Promise { + const load = (file: string) => import(pathToFileURL(path.join(ENGINE, file)).href); + loading ??= Promise.all(['src/core/bib.ts', 'src/core/outline.ts', 'src/core/config.ts', 'src/core/verify.ts', 'src/core/git.ts'].map(load)) + .then(modules => Object.assign({}, ...modules) as Engine); + return loading; +} +// One checker for Papilot's process, so that the panel and the review of +// uncommitted changes share their results and their file. +let shared: Checker | undefined; +export async function checker(file = path.join(DATA, 'references.json')) { + const { ReferenceChecker } = await engine(); + return shared ??= new ReferenceChecker(file); +} +// Confirmations live beside the review engine's state in the repository, so +// a task's review, the uncommitted changes and this panel agree. +export async function confirmations(root: string) { + const { Confirmations, Repo } = await engine(); + return new Confirmations(path.join(new Repo(root).gitDir(), 'latex-review', 'references-confirmed.json')); +} + +// A file of the paper, read only from inside its folder. +function reader(root: string) { + const base = fs.realpathSync(root); + return (file: string) => { + const target = path.resolve(base, file); + if (target !== base && !target.startsWith(base + path.sep)) return null; + try { return fs.statSync(target).isFile() ? fs.readFileSync(target, 'utf8') : null; } catch { return null; } + }; +} +async function report(root: string) { + const mods = await engine(), read = reader(root), { main } = mods.loadProject(root); + const result = mods.bibliographyReport({ read, main, outline: mods.buildOutline(read, main).nodes }); + return { result, inputs: mods.entryInputs(result.entries) }; +} + +export async function bibliography(root: string, check: boolean, retry: boolean): Promise { + const { result, inputs } = await report(root); + return { ...result, ...(await checker()).lookup(inputs, check, retry), confirmed: (await confirmations(root)).of(inputs) }; +} +export async function confirmEntry(root: string, id: string, on: boolean) { + const { inputs } = await report(root), input = inputs.find(item => item.id === id); + if (!input) throw Object.assign(new Error('This entry is no longer in the bibliography. Refresh the panel.'), { statusCode: 404 }); + (await confirmations(root)).set(input, on); + return bibliography(root, false, false); +} diff --git a/src/server/review.ts b/src/server/review.ts index 8cc9a91..118a239 100644 --- a/src/server/review.ts +++ b/src/server/review.ts @@ -41,8 +41,8 @@ export function annotateReviewState(value: unknown, tasks: readonly Task[]): unk export function allowedReviewRoute(route: string, method: string): boolean { if (route.includes('..') || route.includes('\\')) return false; - if (method === 'GET') return route === '/' || /^\/(assets|fonts)\/[\w./-]+$/.test(route) || /^\/files\/[a-z0-9-]+\/\d+\/[\w.-]+\.(svg|png)$/.test(route) || /^\/api\/tasks(?:\/[a-z0-9-]+(?:\/(?:state|rounds\/\d+(?:\/effects\/c\d+)?))?)?$/.test(route); - return method === 'POST' && /^\/api\/tasks\/[a-z0-9-]+\/(decide|undo|note)$/.test(route); + if (method === 'GET') return route === '/' || /^\/(assets|fonts)\/[\w./-]+$/.test(route) || /^\/files\/[a-z0-9-]+\/\d+\/[\w.-]+\.(svg|png)$/.test(route) || /^\/api\/tasks(?:\/[a-z0-9-]+(?:\/(?:state|rounds\/\d+(?:\/effects\/c\d+|\/references)?))?)?$/.test(route); + return method === 'POST' && /^\/api\/tasks\/[a-z0-9-]+\/(decide|undo|note|rounds\/\d+\/references\/confirm)$/.test(route); } export function rewriteReviewJson(value: any, prefix: string): any { if (Array.isArray(value)) return value.map(v => rewriteReviewJson(v, prefix)); diff --git a/src/server/workspace.ts b/src/server/workspace.ts index 5a82394..93660ae 100644 --- a/src/server/workspace.ts +++ b/src/server/workspace.ts @@ -408,6 +408,7 @@ Original task: ${task.instructions} Only edit this isolated task worktree. Never edit the manuscript checkout, other task worktrees, or Papilot's own source/state. Never commit, push, stash, or accept/reject/undo review decisions. ${scope.length ? `The review covers only ${scope.join(', ')} in this worktree, and an edit to any other file stops the PDF build. Leave other files unchanged even when project instructions ask for a matching update; mention that update in your summary instead.\n` : ''}Papilot owns the LaTeX review lifecycle: it compiles and publishes automatically AFTER your turn finishes, and delivers review feedback in a new turn. Do not create another review task, invoke publish, or wait on feedback --wait. This application workflow replaces the standalone latex-review skill's publish-and-wait loop. End your turn after completing the requested edits, with a concise summary for the author of what you changed and why; do not describe Papilot's review workflow. Preserve meaning and mathematical correctness. Read relevant project instructions. Treat reviewer comments as instructions tied to their round, file and changes. Rejected edits must not be reintroduced unless explicitly requested. If there is a conflict with accepted text, explain it and propose a revision for a new review round; never overwrite the accepted manuscript. +Never invent a bibliography entry or any of its details. Add a .bib entry only for a work you can identify, take its authors, title, year and venue from the published record, and include its doi or arXiv eprint when it has one. Papilot looks up every added or changed entry in Crossref, DBLP and arXiv and shows the author any it cannot confirm. Name the entries you added or changed in your summary. ${this.agentOf(task) === 'antigravity' ? 'Shell commands are not available to you here and any attempt is refused, which ends your turn: read, search and edit files in the task worktree with your file tools only.' : 'You may run tools in the task worktree to inspect and validate your changes.'} ${viewPdf ? `When the look of the pages matters, such as line breaks, spacing, figures, tables or displayed formulas, use the ${VIEW_PDF} tool and open the pages it draws with your image viewer to see them as the author does. It compiles the paper, so do not call it for wording alone. ` : ''}If a decision requires the user, ask through the available user-input tool or explain it in your final response. Use English for your progress updates, questions, and final summaries unless the user explicitly requests another language. Preserve the manuscript language unless asked to translate it.`; } private async startRun(run: Run) { diff --git a/src/shared.ts b/src/shared.ts index 0e3ad8c..4cde124 100644 --- a/src/shared.ts +++ b/src/shared.ts @@ -136,3 +136,20 @@ export interface PaperImport { phase: 'choose' | 'building' | 'failed' | 'ready' | 'repairing'; error?: string; summary?: string; versioned?: boolean; } +// The paper's bibliography as the References panel reads it: the review +// engine's report of every entry, with online checks and the entries the +// author checked by hand. +export interface ReferenceCitation { file: string; line: number; section: string | null } +export interface ReferenceRecord { source: 'doi' | 'crossref' | 'arxiv' | 'dblp'; title: string; authors: string[]; year: number | null; venue: string | null; doi: string | null; url: string } +export interface ReferenceCheck { + status: 'verified' | 'mismatch' | 'unverified' | 'failed' | 'skipped'; + record?: ReferenceRecord; via: 'doi' | 'arxiv' | 'title' | null; + differences: { field: 'title' | 'author' | 'year' | 'doi' | 'arxiv'; bib: string; record: string; minor?: boolean }[]; + reason?: 'doi-not-found' | 'arxiv-not-found' | 'no-match' | 'nothing-to-check'; unavailable: string[]; checked: number; +} +export interface BibliographyEntry { id: string; file: string; key: string; type: string; fields: { name: string; value: string; resolved?: string }[]; lines: [number, number]; cited: ReferenceCitation[]; error?: string } +export interface BibliographyReport { + bibs: string[]; absent: string[]; entries: BibliographyEntry[]; + missing: (ReferenceCitation & { key: string })[]; duplicates: string[][]; similar: { ids: string[]; by: 'doi' | 'title' }[]; all_cited: boolean; + checks: Record; pending: number; confirmed: Record; +} diff --git a/src/web/action-menu.tsx b/src/web/action-menu.tsx index 7e7b9c8..26d501f 100644 --- a/src/web/action-menu.tsx +++ b/src/web/action-menu.tsx @@ -6,7 +6,9 @@ export interface MenuAction { id?: string; label: string; title?: string; detail // Native popovers escape scrolling containers and handle outside clicks without // making the rest of the workspace modal. Each menu keeps its own task identity. // The trigger defaults to a "more" button; pass children for a labelled trigger. -export function ActionMenu({ label, actions, children, className = 'icon-button more-button', menuClassName = '', align = 'end', disabled }: { label: string; actions: MenuAction[]; children?: ReactNode; className?: string; menuClassName?: string; align?: 'start' | 'end'; disabled?: boolean }) { +// A side menu opens beside its trigger, as a submenu does, and the arrow +// keys toward and away from it open and close it. +export function ActionMenu({ label, actions, children, className = 'icon-button more-button', menuClassName = '', align = 'end', disabled }: { label: string; actions: MenuAction[]; children?: ReactNode; className?: string; menuClassName?: string; align?: 'start' | 'end' | 'side'; disabled?: boolean }) { const id = useId(); const trigger = useRef(null); const menu = useRef(null); @@ -22,6 +24,12 @@ export function ActionMenu({ label, actions, children, className = 'icon-button }, [open]); function position() { const panel = menu.current!, rect = trigger.current!.getBoundingClientRect(); + if (align === 'side') { + const right = rect.right + 6, left = rect.left - panel.offsetWidth - 6; + panel.style.left = `${right + panel.offsetWidth <= window.innerWidth - 8 || left < 8 ? Math.max(8, Math.min(right, window.innerWidth - panel.offsetWidth - 8)) : left}px`; + panel.style.top = `${Math.max(8, Math.min(rect.top - 5, window.innerHeight - panel.offsetHeight - 8))}px`; + return; + } if (align === 'start') panel.style.minWidth = `${rect.width}px`; const left = align === 'start' ? rect.left : rect.right - panel.offsetWidth; panel.style.left = `${Math.max(8, Math.min(left, window.innerWidth - panel.offsetWidth - 8))}px`; @@ -41,6 +49,7 @@ export function ActionMenu({ label, actions, children, className = 'icon-button event.preventDefault(); if (menu.current?.matches(':popover-open')) close(); else show(); }} onKeyDown={event => { if (event.key === 'ArrowDown' || event.key === 'ArrowUp') { event.preventDefault(); show(event.key === 'ArrowUp'); } + else if (align === 'side' && event.key === 'ArrowRight') { event.preventDefault(); show(); } }}>{children || }
- {project && } + {project && } + {project && { setSelected(''); setPage('references'); setSidebar(false); } }, + ]}>{t('More')}{referencesOpen ? {t('Reference check')} : null}}
setTaskSearch(event.target.value)} onKeyDown={event => { if (event.key === 'Escape' && taskSearch) { event.preventDefault(); event.stopPropagation(); setTaskSearch(''); } }}/>{taskSearch && }
@@ -1398,13 +1415,13 @@ function App() {
} {state && owner && } - { e.preventDefault(); setSelected(''); if (mobile) setManuscriptView('pdf'); setSidebar(false); }}>Papilot{state && v{versionLabel(state.version)}} + { e.preventDefault(); setSelected(''); setPage('manuscript'); if (mobile) setManuscriptView('pdf'); setSidebar(false); }}>Papilot{state && v{versionLabel(state.version)}} {sidebar && narrow && } : !owner ?

{t('No papers yet')}

{t('No papers have been shared with you yet. Papers appear here as soon as {name} shares them.', { name: viewer?.owner || '' })}

:

{t('Add your first paper')}

} diff --git a/src/web/references-panel.css b/src/web/references-panel.css new file mode 100644 index 0000000..26e5de2 --- /dev/null +++ b/src/web/references-panel.css @@ -0,0 +1,72 @@ +/* The References panel beside the PDF: a summary with the online check, + the problems to act on, every entry on request, and a task for an agent. */ +.references-panel{flex:1;min-height:0;overflow:auto;display:flex;flex-direction:column;font-size:13px;line-height:1.5} +.references-summary{padding:12px 14px 10px;border-bottom:1px solid var(--line)} +.references-summary>p{display:flex;flex-wrap:wrap;align-items:baseline;gap:4px 10px;margin:0} +.references-stat{font-size:12px;color:var(--text-2)} +.references-stat.s-verified,.references-stat.s-confirmed{color:var(--green)} +.references-stat.s-mismatch{color:var(--red)} +.references-stat.s-unverified,.references-stat.s-failed{color:var(--amber-text)} +.references-actions{display:flex;align-items:center;gap:8px;margin-top:8px} +.references-progress{display:inline-flex;align-items:center;gap:8px;color:var(--text-2)} +.references-note{margin:8px 0 0;font-size:12px;color:var(--text-3)} +.references-filter{display:flex;gap:2px;padding:8px 10px;border-bottom:1px solid var(--line)} +.references-filter>button{display:inline-flex;align-items:center;gap:6px;height:28px;padding:0 10px;border-radius:var(--r);font-size:13px;font-weight:500;color:var(--text-2)} +.references-filter>button:hover{background:var(--bg-hover);color:var(--text)} +.references-filter>button[aria-selected=true]{background:var(--bg-muted);color:var(--text)} +.references-filter .count{font-size:11.5px;color:var(--text-3)} +.references-list{list-style:none;margin:0;padding:4px 0;flex:1} +.references-item{border-bottom:1px solid var(--line)} +.references-item[data-open=true]{background:var(--bg-subtle)} +.references-row{display:flex;align-items:flex-start;gap:8px;padding-left:12px} +.references-row>input{margin-top:11px;flex-shrink:0} +.references-head{display:flex;flex-wrap:wrap;align-items:baseline;gap:2px 8px;width:100%;min-width:0;padding:8px 12px;text-align:left;color:var(--text)} +.references-row .references-head{padding-left:0} +.references-head:hover:not(:disabled){background:var(--bg-hover)} +.references-head:disabled{opacity:1;cursor:default} +.references-head code{font-size:12.5px;font-weight:600;overflow-wrap:anywhere} +.references-state{font-size:11.5px;font-weight:600;padding:0 7px;border-radius:999px;background:var(--bg-muted);color:var(--text-2);white-space:nowrap} +.references-state.s-verified,.references-state.s-confirmed{background:var(--green-soft);color:var(--green)} +.references-state.s-mismatch{background:var(--red-soft);color:var(--red)} +.references-state.s-unverified,.references-state.s-failed,.references-state.s-skipped,.references-state.s-checking{background:var(--amber-soft);color:var(--amber-text)} +.references-problems{flex-basis:100%;color:var(--red);font-size:12.5px} +.references-item[data-state=unverified] .references-problems,.references-item[data-state=failed] .references-problems{color:var(--amber-text)} +.references-item.soft .references-problems{color:var(--text-3)} +.references-title{flex-basis:100%;color:var(--text);overflow-wrap:anywhere} +.references-meta{flex-basis:100%;font-size:12px;color:var(--text-3)} +.references-detail{padding:2px 14px 12px 14px} +.references-row+.references-detail{padding-left:36px} +.references-detail table{width:100%;border-collapse:collapse;font:12px/1.55 var(--mono,ui-monospace,SFMono-Regular,Menlo,monospace)} +.references-detail th{width:1%;padding:1px 10px 1px 0;text-align:left;vertical-align:top;font-weight:400;color:var(--text-3);white-space:nowrap} +.references-detail td{padding:1px 0;overflow-wrap:anywhere} +.references-resolved{color:var(--text-3)} +.references-where,.references-record,.references-confirmed{margin:8px 0 0;font-size:12px;color:var(--text-2);overflow-wrap:anywhere} +.references-record>span:first-child{margin-right:6px;color:var(--text-3)} +.references-record a{color:var(--accent)} +.references-differences{margin:8px 0 0;display:grid;gap:6px} +.references-differences>div{display:grid;grid-template-columns:auto 1fr;gap:0 10px} +.references-differences dt{font-weight:600;white-space:nowrap} +.references-differences dd{margin:0;display:grid;min-width:0;overflow-wrap:anywhere} +.references-differences dd>span:first-child{color:var(--red)} +.references-differences dd>span:last-child{color:var(--text-2)} +.references-detail>.button{margin-top:10px} +.references-confirmed{display:flex;align-items:center;gap:6px;color:var(--green)} +.references-empty{display:flex;align-items:center;gap:6px;margin:14px;color:var(--text-2)} +.references-fix{position:sticky;bottom:0;display:flex;flex-wrap:wrap;align-items:center;gap:6px 10px;padding:10px 14px;border-top:1px solid var(--line);background:var(--bg)} +.references-fix>span{font-size:12px;color:var(--text-3)} +@media(max-width:480px){.references-differences>div{grid-template-columns:1fr}} +/* More in the sidebar, under Manuscript: a row that opens its tools beside it. */ +.sidebar-actions>.menu-anchor{display:flex} +.more-link{min-height:32px;gap:8px;font-weight:400;color:var(--text-2)} +.more-link>svg:last-child{margin-left:auto;color:var(--text-3)} +.more-link.active{color:var(--text)} +.more-link .nav-meta+svg{margin-left:6px} +.more-menu{min-width:240px} +/* The reference check as a page: the panel, centred at a reading width. */ +.references-page{flex:1;min-height:0;display:flex;flex-direction:column;width:100%;max-width:820px;margin:0 auto;padding:0 8px} +.references-page-head{display:flex;align-items:baseline;gap:10px;padding:18px 14px 6px} +.references-page-head h2{margin:0;font:600 18px/1.3 var(--sans)} +.references-page-head span{font-size:13px;color:var(--text-3)} +.references-page .references-panel{border:1px solid var(--line);border-radius:12px;background:var(--bg);margin-bottom:16px} +.references-page .references-fix{border-radius:0 0 12px 12px} +@media(max-width:680px){.references-page{padding:0}.references-page .references-panel{border:0;border-radius:0;margin:0}.references-page-head{padding:12px 14px 4px}} diff --git a/src/web/references-panel.tsx b/src/web/references-panel.tsx new file mode 100644 index 0000000..3330488 --- /dev/null +++ b/src/web/references-panel.tsx @@ -0,0 +1,181 @@ +import { useCallback, useEffect, useMemo, useRef, useState } from 'react'; +import type { BibliographyEntry, BibliographyReport, ReferenceCheck } from '../shared.ts'; +import { Icon } from './icons.tsx'; +import { language, t, tn } from './i18n.ts'; +import { browserStorage } from './storage.ts'; +import './references-panel.css'; + +// The References panel checks the paper's whole bibliography as it is on +// disk: citations without an entry, keys used twice, the same work under two +// keys, entries nobody cites, and, when asked, what Crossref, DBLP and arXiv +// know of each entry. An entry no source knows can be marked as checked by +// hand, which the reviews of later changes honour. Problems go to an agent +// as a new task that finds out what happened; the author decides the edits. + +type Load = (route: string, body?: unknown) => Promise; +const online = () => browserStorage.getItem('latex-review:references') !== 'off'; +const SOURCE: Record = { doi: 'doi.org', crossref: 'Crossref', dblp: 'DBLP', arxiv: 'arXiv' }; +const fieldName = (field: string) => ({ title: t('Title'), author: t('Authors'), year: t('Year'), doi: 'DOI', arxiv: 'arXiv' } as Record)[field] || field; +const gap = () => (language() === 'zh-CN' ? '' : ' '); +const where = (place: { file: string; line: number; section?: string | null }) => `${place.section ? place.section + ' · ' : ''}${place.file}:${place.line}`; +const value = (entry: BibliographyEntry, name: string) => { const field = entry.fields.find(f => f.name === name); return field ? field.resolved ?? field.value : ''; }; +// Titles and names as they read, without TeX's braces. +const readable = (text: string) => text.replace(/\\[a-zA-Z]+\s*/g, '').replace(/[{}$]/g, '').replace(/\s+/g, ' ').trim(); + +/** The panel's data: read when the PDF is rebuilt, and checked online only + * when the author asks. */ +export function useBibliography(projectId: string, version: string, load: Load, enabled = true) { + const [report, setReport] = useState(null), [error, setError] = useState(''); + const [checking, setChecking] = useState(false), retry = useRef(false), [round, setRound] = useState(0); + useEffect(() => { setReport(null); setError(''); setChecking(false); }, [projectId]); + // Back from the editor, the files may have changed. + useEffect(() => { + if (!enabled) return; + const again = () => setRound(n => n + 1); + window.addEventListener('focus', again); + return () => window.removeEventListener('focus', again); + }, [enabled]); + useEffect(() => { + if (!projectId || !enabled) return; + let live = true, timer: ReturnType | undefined; + const poll = async () => { + const again = retry.current; retry.current = false; + try { + const next = await load(`/projects/${projectId}/references${checking && online() ? `?check=1${again ? '&retry=1' : ''}` : ''}`); + if (!live) return; + setReport(next); setError(''); + if (checking && next.pending) timer = setTimeout(() => void poll(), 1500); + else if (checking) setChecking(false); + } catch (e) { if (live) { setError(e instanceof Error ? e.message : String(e)); setChecking(false); } } + }; + void poll(); + return () => { live = false; clearTimeout(timer); }; + }, [projectId, version, checking, round, load, enabled]); + return { + report, error, checking, + check(again = false) { retry.current = again; setChecking(true); setRound(n => n + 1); }, + async confirm(id: string, on: boolean) { + try { setReport(await load(`/projects/${projectId}/references/confirm`, { id, confirmed: on })); } + catch (e) { setError(e instanceof Error ? e.message : String(e)); } + }, + }; +} +export type Bibliography = ReturnType; + +interface Item { id: string; key: string; label: string; entry?: BibliographyEntry; problems: string[]; soft: boolean; fix: string } +// Every problem of the bibliography, entry by entry, most pressing first. +function items(report: BibliographyReport): Item[] { + const list: Item[] = [], byId = new Map(report.entries.map(entry => [entry.id, entry])); + // One item for each key cited without an entry, with every place. + for (const key of [...new Set(report.missing.map(cite => cite.key))]) { + const places = report.missing.filter(cite => cite.key === key).map(cite => `${cite.file}:${cite.line}`).join('; '); + list.push({ id: `missing:${key}`, key, label: `\\cite{${key}}`, problems: [t('No entry has this key.') + gap() + places], soft: false, + fix: t('{cite} ({places}) has no entry with this key.', { cite: `\\cite{${key}}`, places }) }); + } + for (const entry of report.entries) { + const check = report.checks[entry.id], mine = entry.id in report.confirmed, problems: string[] = [], fixes: string[] = []; + if (entry.error) { problems.push(t('BibTeX cannot read it: {error}', { error: entry.error })); fixes.push(t('BibTeX cannot read it ({error}).', { error: entry.error })); } + if (report.duplicates.some(ids => ids.includes(entry.id))) { problems.push(t('Another entry has the same key.')); fixes.push(t('another entry has the same key.')); } + for (const group of report.similar.filter(g => g.ids.includes(entry.id))) { + const others = group.ids.filter(id => id !== entry.id).map(id => byId.get(id)?.key).join(', '); + problems.push(group.by === 'doi' ? t('Same DOI as {keys}.', { keys: others }) : t('Same title as {keys}.', { keys: others })); + fixes.push(group.by === 'doi' ? t('the same work as {keys} (same DOI).', { keys: others }) : t('possibly the same work as {keys} (same title).', { keys: others })); + } + if (check && !mine && check.status === 'mismatch') { + const major = check.differences.filter(d => !d.minor); + problems.push(major.map(d => t('{field}: {bib} here, {record} in {source}', { field: fieldName(d.field), bib: d.bib, record: d.record || '—', source: SOURCE[check.record?.source || ''] || '' })).join('; ')); + fixes.push(t('{differences} (record: {url}).', { differences: major.map(d => t('{field} is {bib} but {source} has {record}', { field: fieldName(d.field), bib: d.bib, record: d.record || t('nothing'), source: SOURCE[check.record?.source || ''] || '' })).join('; '), url: check.record?.url || '' })); + } + if (check && !mine && check.status === 'unverified') { problems.push(t('No record in Crossref, DBLP or arXiv matches it.')); fixes.push(t('no record in Crossref, DBLP or arXiv matches it.')); } + const soft = !problems.length && !report.all_cited && !entry.cited.length; + if (soft) { problems.push(t('Not cited in the paper.')); fixes.push(t('it is not cited.')); } + if (problems.length) list.push({ id: entry.id, key: entry.key, label: entry.key, entry, problems, soft, + fix: t('{key} ({place}): {problems}', { key: entry.key, place: `${entry.file}:${entry.lines[0]}`, problems: fixes.join(gap()) }) }); + } + return list; +} +function state(entry: BibliographyEntry, report: BibliographyReport, checking: boolean) { + const check = report.checks[entry.id]; + if (entry.id in report.confirmed && check?.status !== 'verified') return 'confirmed'; + return check?.status || (checking ? 'checking' : 'unchecked'); +} +const stateLabel = (status: string) => ({ verified: t('Verified'), mismatch: t('Differs'), unverified: t('Not found'), failed: t('Could not check'), skipped: t('Nothing to check'), confirmed: t('Checked by you'), checking: t('Checking…'), unchecked: t('Not checked') } as Record)[status] || status; + +/** How many problems the tab shows: those an author should act on. */ +export const problemCount = (report: BibliographyReport | null) => report ? items(report).filter(item => !item.soft).length : 0; + +export function ReferencesPanel({ data, onInvestigate }: { data: Bibliography; onInvestigate: (text: string) => void }) { + const { report, error, checking } = data; + const [filter, setFilter] = useState<'problems' | 'all'>('problems'), [open, setOpen] = useState(''); + // The author's own choices; by default every problem but an uncited entry goes to the agent. + const [choices, setChoices] = useState>({}); + const all = useMemo(() => report ? items(report) : [], [report]); + const included = (item: Item) => choices[item.id] ?? !item.soft; + const toggle = useCallback((item: Item) => setChoices(current => ({ ...current, [item.id]: !(current[item.id] ?? !item.soft) })), []); + if (error && !report) return

{error}

; + if (!report) return

{t('Reading the bibliography…')}

; + if (!report.bibs.length) return

{t('This paper names no .bib file with {bibliography} or {addbibresource}.', { bibliography: '\\bibliography', addbibresource: '\\addbibresource' })}

; + const count = (status: string) => report.entries.filter(entry => state(entry, report, checking) === status).length; + const unchecked = report.entries.filter(entry => !entry.error && !report.checks[entry.id]).length; + const chosen = all.filter(included); + const shown = filter === 'problems' ? null : report.entries; + // The agent finds out what happened; the author decides what to change. + const investigate = () => onInvestigate([t('Find out what happened with these bibliography problems and report back. Do not edit any file. For each, check whether the work exists and what its published record says (authors, title, year, venue, DOI), find where the entry or citation came from in the Git history of the .bib and .tex files, and say what you would suggest. Mark which findings you confirmed in a source and which are guesses.'), ...chosen.map(item => '- ' + item.fix)].join('\n')); + return
+
+

+ {tn(report.entries.length, '{n} entry', '{n} entries')} + {([['verified', t('{n} verified', { n: count('verified') })], ['confirmed', t('{n} checked by you', { n: count('confirmed') })], ['mismatch', t('{n} differ', { n: count('mismatch') })], ['unverified', t('{n} not found', { n: count('unverified') })], ['failed', t('{n} could not be checked', { n: count('failed') })]] as const).map(([status, label]) => count(status) > 0 && {label})} + {report.absent.length > 0 && {t('Missing: {files}', { files: report.absent.join(', ') })}} +

+
+ {checking ? {t('Checking {n}…', { n: report.pending || unchecked })} + : } +
+ {!online() &&

{t('Online checks are off in Settings, so only problems found in the files are shown.')}

} + {error &&

{error}

} +
+
+ + +
+ {filter === 'problems' && (all.length ?
    {all.map(item =>
  • +
    + toggle(item)}/> + +
    + {item.entry && open === item.id && void data.confirm(item.entry!.id, on)}/>} +
    • )}
:

{unchecked ? t('No problems in the files. Check online to compare the entries with their records.') : t('No problems found.')}

)} + {shown &&
    {shown.map(entry => { const status = state(entry, report, checking); return
  • + + {open === entry.id && void data.confirm(entry.id, on)}/>} +
    • ; })}
} + {filter === 'problems' && all.length > 0 &&
+ + {t('Opens a new task you can edit before sending. The agent reports what it finds and changes nothing.')} +
} +
; +} + +function EntryDetail({ entry, check, confirmed, onConfirm }: { entry: BibliographyEntry; check?: ReferenceCheck; confirmed: boolean; onConfirm: (on: boolean) => void }) { + return
+ {entry.fields.map(field => )}
{field.name}{field.value}{field.resolved && = {field.resolved}}
+

{entry.file}:{entry.lines[0]}–{entry.lines[1]}{gap() || ' · '}{entry.cited.length ? tn(entry.cited.length, 'Cited {n} time: {places}', 'Cited {n} times: {places}', { places: entry.cited.map(where).join('; ') }) : t('Not cited in the paper.')}

+ {check?.record &&

{check.status === 'unverified' ? t('Closest record') : t('Record')} · {SOURCE[check.record.source]} + {check.record.title}{[check.record.authors.join(', '), check.record.venue, check.record.year, check.record.doi && `doi:${check.record.doi}`].filter(Boolean).map((part, i) => · {part})}

} + {check && check.differences.length > 0 &&
{check.differences.map((d, i) =>
{fieldName(d.field)}{d.minor ? ` · ${t('small')}` : ''}
{d.bib}{SOURCE[check.record?.source || ''] || t('Record')}: {d.record || '—'}
)}
} + {check?.unavailable.length ?

{t('{sources} could not be reached.', { sources: check.unavailable.join(', ') })}

: null} + {!entry.error && (confirmed + ?

{t('You checked this entry yourself.')}

+ : check?.status !== 'verified' && )} +
; +} diff --git a/src/web/review-frame.tsx b/src/web/review-frame.tsx index 9407804..fdfad51 100644 --- a/src/web/review-frame.tsx +++ b/src/web/review-frame.tsx @@ -33,6 +33,10 @@ function compactReviewFrame(doc: Document, reserve: number, lead: number, change .source-code del{background:var(--red-soft);color:var(--red)} .source-code ins{background:var(--green-soft);color:var(--green)} .remark{background:var(--accent-soft)}.histories,.caption{background:var(--bg-subtle);color:var(--text-2)} + /* Bibliography entries and their checks follow the theme. */ + :root{--ok-soft:var(--green-soft);--bad-soft:var(--red-soft);--warn:var(--amber-text);--warn-soft:var(--amber-soft);--warn-line:var(--amber-line)} + .references,.reference-entry{background:var(--bg);color:var(--text)} + .reference-fields th,.reference-status,.reference-lines,.reference-cited,.reference-side,.reference-resolved{color:var(--text-3)} .toast.error{background:var(--red-soft);color:var(--red)} /* Papilot's commit step at the end of the completion card. */ .review-complete{align-items:center} diff --git a/src/web/settings.tsx b/src/web/settings.tsx index 668192e..10ce7df 100644 --- a/src/web/settings.tsx +++ b/src/web/settings.tsx @@ -4,6 +4,7 @@ import { ChipSelect } from './chip-select.tsx'; import { keepDialogFocus } from './focus.ts'; import { LANGUAGES, browserLanguage, setLanguage, t, useLanguage, type LanguageChoice } from './i18n.ts'; import { setNotifications, testNotification, useNotifications } from './notifications.ts'; +import { browserStorage } from './storage.ts'; import type { Theme } from './theme.tsx'; import { ShortcutSettings } from './shortcut-settings.tsx'; import { AboutSettings, LicenseSettings } from './about-settings.tsx'; @@ -14,6 +15,10 @@ import { SkillLibraryPanel, type SkillLibraryControls, type SkillLibraryOptions type SettingsPage = 'general' | 'shortcuts' | 'about' | 'licenses' | 'diagnostics' | 'skills' | 'updates'; +// The review page reads this key: "off" stops it from looking up the +// bibliography entries a change adds or modifies in Crossref, DBLP and arXiv. +const REFERENCE_CHECKS = 'latex-review:references'; + // Browser preferences and workspace management share one sidebar entry. export function SettingsDialog({ theme, onTheme, onClose, onPeople, onSignOut, accountName, diagnostics, skills, updates, initialPage = 'general' }: { theme: Theme; onTheme: (theme: Theme) => void; onClose: () => void; onPeople?: () => void; onSignOut?: () => void; accountName?: string; diagnostics?: DiagnosticsSettingsProps; skills?: SkillLibraryOptions; updates?: UpdateApi; initialPage?: 'general' | 'shortcuts' }) { const dialog = useRef(null); @@ -23,6 +28,7 @@ export function SettingsDialog({ theme, onTheme, onClose, onPeople, onSignOut, a const [page, setPage] = useState(initialPage); const previousPage = useRef(initialPage); const [asking, setAsking] = useState(false), [test, setTest] = useState<'' | 'sent' | 'failed'>(''); + const [checkReferences, setCheckReferences] = useState(() => browserStorage.getItem(REFERENCE_CHECKS) !== 'off'); useEffect(() => { const opener = document.activeElement as HTMLElement | null; const element = dialog.current!; @@ -82,6 +88,13 @@ export function SettingsDialog({ theme, onTheme, onClose, onPeople, onSignOut, a {test && {test === 'sent' ? t('Sent. If nothing appeared, allow notifications for this browser in your system settings.') : t('This browser could not show the notification.')}} } +
+
+ +

{t('When a change adds or edits a bibliography entry, look up its title, authors, year and DOI in Crossref, DBLP and arXiv. Nothing else from your paper is sent.')}

+
+ { setCheckReferences(event.target.checked); browserStorage.setItem(REFERENCE_CHECKS, event.target.checked ? 'on' : 'off'); }}/> +
{skills && } diff --git a/src/web/style.css b/src/web/style.css index ab3f378..d062ad9 100644 --- a/src/web/style.css +++ b/src/web/style.css @@ -822,7 +822,7 @@ body.resizing iframe{pointer-events:none} .segmented label.checked{background:var(--bg);color:var(--text);font-weight:500;box-shadow:var(--shadow-xs)} .segmented label:has(input:focus-visible){outline:2px solid var(--accent);outline-offset:1px} .settings-dialog .segmented input{position:absolute;inset:0;width:100%;height:100%;margin:0;opacity:0;cursor:pointer} -.settings-notifications{align-items:flex-start} +.settings-notifications,.settings-toggle{align-items:flex-start} .settings-text p{margin-top:4px;font-size:13px;line-height:1.55;color:var(--text-3)} .people-dialog{width:560px} .people-warning{display:flex;gap:8px;align-items:flex-start;margin:-8px 0 18px;padding:10px 12px;border:1px solid var(--red-line);border-radius:var(--r);background:var(--red-soft);color:var(--red);font-size:13px;line-height:1.55} @@ -860,7 +860,7 @@ body.resizing iframe{pointer-events:none} .settings-dialog input.switch:disabled{opacity:.45;cursor:not-allowed} .settings-test{display:flex;flex-wrap:wrap;align-items:center;gap:8px 12px;padding-bottom:4px;font-size:13px;color:var(--text-2)} .settings-test .button{display:inline-flex;align-items:center;gap:6px} -@media(max-width:480px){.settings-row:not(.settings-notifications){flex-direction:column;align-items:stretch;gap:8px}.segmented{display:flex}.segmented label{flex:1;justify-content:center}.settings-dialog select{width:100%}} +@media(max-width:480px){.settings-row:not(.settings-notifications):not(.settings-toggle){flex-direction:column;align-items:stretch;gap:8px}.segmented{display:flex}.segmented label{flex:1;justify-content:center}.settings-dialog select{width:100%}} /* Two-panel views */ @media(min-width:681px){ diff --git a/src/web/zh-CN.ts b/src/web/zh-CN.ts index 92d4aaa..dd38f53 100644 --- a/src/web/zh-CN.ts +++ b/src/web/zh-CN.ts @@ -344,6 +344,8 @@ export const zhCN: Record = { 'Appearance': '外观', 'Light': '浅色', 'Dark': '深色', 'System': '跟随系统', 'Language': '语言', 'Browser default ({language})': '浏览器默认（{language}）', 'Notifications': '通知', 'Notify me when a task needs my reply, its review is ready, or it stops with an error, while Papilot is in the background.': 'Papilot 在后台时，如果任务需要我回复、审阅已就绪或因错误停止，就通知我。', + 'Check references online': '联网核对参考文献', + 'When a change adds or edits a bibliography entry, look up its title, authors, year and DOI in Crossref, DBLP and arXiv. Nothing else from your paper is sent.': '改动新增或修改参考文献条目时，用它的标题、作者、年份和 DOI 在 Crossref、DBLP 和 arXiv 中核对。论文的其他内容不会发出去。', 'Send a test notification': '发送测试通知', 'Sent. If nothing appeared, allow notifications for this browser in your system settings.': '已发送。如果没有看到通知，请在系统设置中允许此浏览器发送通知。', 'This browser could not show the notification.': '此浏览器无法显示这条通知。', @@ -428,6 +430,70 @@ export const zhCN: Record = { 'Undo typing, a deleted note, or the last review decision, depending on focus.': '根据焦点位置，撤销输入、恢复删除的批注或撤销上一次审阅操作。', 'Send the review comment being written.': '发送正在填写的审阅意见。', 'Close the current editor or cancel region selection.': '关闭当前编辑器或取消画框。', + '{differences} (record: {url}).': '{differences}（记录：{url}）。', + '{field} is {bib} but {source} has {record}': '{field}写的是 {bib}，{source} 里是 {record}', + '{field}: {bib} here, {record} in {source}': '{field}：这里是 {bib}，{source} 里是 {record}', + '{key} ({place}): {problems}': '{key}（{place}）：{problems}', + '{n} checked by you': '{n} 个已人工确认', + '{n} could not be checked': '{n} 个未能核对', + '{n} differ': '{n} 个有出入', + '{n} entries': '{n} 个条目', + '{n} entry': '{n} 个条目', + '{n} not found': '{n} 个未找到', + '{n} verified': '{n} 个已核实', + '{cite} ({places}) has no entry with this key.': '{cite}（{places}）没有对应的条目。', + '{sources} could not be reached.': '无法访问 {sources}。', + 'All entries': '全部条目', + 'another entry has the same key.': '另一个条目用了同一个 key。', + 'Another entry has the same key.': '另一个条目用了同一个 key。', + 'Authors': '作者', + 'BibTeX cannot read it ({error}).': 'BibTeX 读不了这个条目（{error}）。', + 'BibTeX cannot read it: {error}': 'BibTeX 读不了：{error}', + 'Check {n} more online': '再联网核对 {n} 个', + 'Check online': '联网核对', + 'Checked by you': '已人工确认', + 'Checking {n}…': '核对中，还剩 {n} 个…', + 'Cited {n} time: {places}': '正文引用 {n} 处：{places}', + 'Cited {n} times: {places}': '正文引用 {n} 处：{places}', + 'Closest record': '最接近的记录', + 'Could not check': '未能核对', + 'Differs': '有出入', + 'I checked it myself': '我已人工确认', + 'Include {key} in the task': '把 {key} 放进任务', + 'it is not cited.': '正文没有引用它。', + 'Look the entries up in Crossref, DBLP and arXiv. Only their titles, authors, years and identifiers are sent.': '在 Crossref、DBLP 和 arXiv 中查找这些条目。只发送标题、作者、年份和编号。', + 'Missing: {files}': '缺少文件：{files}', + 'No entry has this key.': '没有条目用这个 key。', + 'No problems found.': '没有发现问题。', + 'No problems in the files. Check online to compare the entries with their records.': '文件里没有发现问题。联网核对可以把条目和正式记录逐项对照。', + 'No record in Crossref, DBLP or arXiv matches it.': 'Crossref、DBLP 和 arXiv 中都没有对应记录。', + 'Not checked': '未核对', + 'Not cited in the paper.': '正文没有引用。', + 'Nothing to check': '无从核对', + 'nothing': '没有', + 'Online checks are off in Settings, so only problems found in the files are shown.': '设置里关闭了联网核对，这里只显示从文件中查出的问题。', + 'Online checks are off in Settings.': '设置里关闭了联网核对。', + 'possibly the same work as {keys} (same title).': '可能和 {keys} 是同一篇文献（标题相同）。', + 'Problems': '问题', + 'Reading the bibliography…': '正在读取参考文献…', + 'Record': '记录', + 'Same DOI as {keys}.': '和 {keys} 的 DOI 相同。', + 'Same title as {keys}.': '和 {keys} 的标题相同。', + 'small': '小差别', + 'the same work as {keys} (same DOI).': '和 {keys} 是同一篇文献（DOI 相同）。', + 'This paper names no .bib file with {bibliography} or {addbibresource}.': '论文里没有用 {bibliography} 或 {addbibresource} 指定 .bib 文件。', + 'Title': '标题', + 'Verified': '已核实', + 'Year': '年份', + 'You checked this entry yourself.': '你已人工确认过这个条目。', + 'More': '更多', + 'Reference check': '引用检查', + 'Citations, entries and their records': '引用、条目及其正式记录', + 'Find out what happened with these bibliography problems and report back. Do not edit any file. For each, check whether the work exists and what its published record says (authors, title, year, venue, DOI), find where the entry or citation came from in the Git history of the .bib and .tex files, and say what you would suggest. Mark which findings you confirmed in a source and which are guesses.': '查清楚参考文献里这些问题是怎么回事，然后告诉我。不要修改任何文件。逐条确认：这篇文献是否存在，它的正式记录是什么（作者、标题、年份、出处、DOI）；在 .bib 和 .tex 文件的 Git 历史里找出这个条目或引用是怎么来的；再说说你建议怎么处理。注明哪些结论在来源中核实过，哪些只是推测。', + 'no record in Crossref, DBLP or arXiv matches it.': 'Crossref、DBLP 和 arXiv 中都没有对应记录。', + 'Ask an agent to look into {n} problem': '让 agent 调查 {n} 个问题', + 'Ask an agent to look into {n} problems': '让 agent 调查 {n} 个问题', + 'Opens a new task you can edit before sending. The agent reports what it finds and changes nothing.': '会打开一个新任务，发送前可以修改。agent 只汇报查到的情况，不改动文件。', 'Undefined reference': '未定义的引用', 'Undefined citation': '未定义的文献', 'Label defined more than once': '标签重复定义', 'Missing file': '缺少文件', 'Missing character': '缺少字符', 'Line too wide by {amount}pt': '行宽超出 {amount}pt', 'Errors': '错误', 'Warnings': '警告', diff --git a/tests/core.test.ts b/tests/core.test.ts index 5745c04..2d2755a 100644 --- a/tests/core.test.ts +++ b/tests/core.test.ts @@ -47,6 +47,10 @@ test('review proxy only exposes review routes and keeps assets on the remote ori assert.ok(allowedReviewRoute('/api/tasks/p-test/rounds/1', 'GET')); assert.ok(allowedReviewRoute('/api/tasks/p-test/decide', 'POST')); assert.ok(allowedReviewRoute('/api/tasks/p-test/rounds/1/effects/c001', 'GET')); + assert.ok(allowedReviewRoute('/api/tasks/p-test/rounds/1/references', 'GET')); + assert.equal(allowedReviewRoute('/api/tasks/p-test/rounds/1/references', 'POST'), false); + assert.ok(allowedReviewRoute('/api/tasks/p-test/rounds/1/references/confirm', 'POST')); + assert.equal(allowedReviewRoute('/api/tasks/p-test/rounds/1/references/confirm', 'GET'), false); assert.equal(allowedReviewRoute('/api/tasks/p-test/rounds/1/effects/../x', 'GET'), false); assert.equal(allowedReviewRoute('/api/tasks/p-test/rounds/1/effects/c001', 'POST'), false); assert.equal(allowedReviewRoute('/api/shutdown', 'POST'), false); diff --git a/tests/references.test.ts b/tests/references.test.ts new file mode 100644 index 0000000..908daff --- /dev/null +++ b/tests/references.test.ts @@ -0,0 +1,58 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { execFileSync } from 'node:child_process'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { bibliography, confirmEntry } from '../src/server/references.ts'; + +// A paper with one entry cited twice, one never cited, a key used twice, +// and a citation of a key no entry defines. Nothing is looked up online. +function paper() { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'papilot-references-')); + execFileSync('git', ['init', '-q'], { cwd: root }); + fs.mkdirSync(path.join(root, 'sections')); + fs.writeFileSync(path.join(root, 'main.tex'), String.raw`\documentclass{article} +\begin{document} +\section{Intro} +\input{sections/body} +\bibliography{refs} +\end{document} +`); + fs.writeFileSync(path.join(root, 'sections/body.tex'), String.raw`See \cite{knuth} and \citep{knuth,ghost}. +`); + fs.writeFileSync(path.join(root, 'refs.bib'), `@book{knuth, + author = {Donald E. Knuth}, + title = {The {\\TeX}book}, + year = 1984 +} +@misc{unused, title = {Never cited}} +@misc{unused, title = {Twice}} +`); + return root; +} + +test('the References panel reports the bibliography on disk and keeps confirmations in the repository', async () => { + const root = paper(); + try { + const report = await bibliography(root, false, false); + assert.deepEqual(report.bibs, ['refs.bib']); + assert.deepEqual(report.entries.map(entry => [entry.id, entry.cited.length]), [['refs.bib#knuth', 2], ['refs.bib#unused', 0], ['refs.bib#unused#2', 0]]); + assert.deepEqual(report.entries[0].cited[0], { file: 'sections/body.tex', line: 1, section: '1 Intro' }); + assert.deepEqual(report.missing.map(cite => cite.key), ['ghost']); + assert.deepEqual(report.duplicates, [['refs.bib#unused', 'refs.bib#unused#2']]); + assert.equal(report.pending, 0); + assert.deepEqual(report.confirmed, {}); + + const confirmed = await confirmEntry(root, 'refs.bib#knuth', true); + assert.ok(confirmed.confirmed['refs.bib#knuth'] > 0); + const file = path.join(root, '.git/latex-review/references-confirmed.json'); + assert.ok(fs.existsSync(file)); + // Editing the entry's looked-up fields asks again. + fs.writeFileSync(path.join(root, 'refs.bib'), fs.readFileSync(path.join(root, 'refs.bib'), 'utf8').replace('year = 1984', 'year = 1986')); + assert.deepEqual((await bibliography(root, false, false)).confirmed, {}); + await assert.rejects(confirmEntry(root, 'refs.bib#nothing', true), /no longer in the bibliography/); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } +}); diff --git a/vendor/latex-review/src/core/bib.ts b/vendor/latex-review/src/core/bib.ts new file mode 100644 index 0000000..155dc25 --- /dev/null +++ b/vendor/latex-review/src/core/bib.ts @@ -0,0 +1,692 @@ +import path from "node:path"; +import { deepestNode } from "./outline.ts"; +import { compact, doiOf } from "./verify.ts"; +import type { + Bibliography, + BibliographyEntry, + Heading, + MissingCitation, + ReferenceCitation, + ReferenceEntry, + ReferenceField, + ReferenceProblem, + References, + Span, + Unit, +} from "../types.ts"; + +// BibTeX entries as they are written, so that a review can show a .bib +// change entry by entry and check it. The reader follows BibTeX: text +// outside entries and @comment are ignored, @string macros resolve, and a +// value is {braced}, "quoted", a number or a macro name, joined with #. + +interface BibSource { + file: string; + lines: Span; +} +type Macros = Map; +export interface BibField { + name: string; + /** The value as written, without its outer braces or quotes. */ + value: string; + /** The value with macros expanded and parts joined. */ + resolved: string; + /** Source lines of @string definitions used here, including nested macros. */ + dependencies?: BibSource[]; +} +export interface BibEntry { + key: string; + type: string; + fields: BibField[]; + /** First and last line of the entry, counted from 1. */ + lines: Span; + /** Why the entry could not be read to its end. */ + error?: string; +} + +const MONTHS: Record = { + jan: "January", + feb: "February", + mar: "March", + apr: "April", + may: "May", + jun: "June", + jul: "July", + aug: "August", + sep: "September", + oct: "October", + nov: "November", + dec: "December", +}; +// Line breaks and indentation inside a value are one space when typeset. +export const collapse = (text: string) => text.replace(/\s+/g, " ").trim(); + +class Unreadable extends Error {} + +const defaultMacros = (): Macros => new Map(Object.entries(MONTHS).map(([name, resolved]) => + [name, { resolved, dependencies: [] }], +)); + +export function parseBib(text: string, file = "", macros = defaultMacros()): BibEntry[] { + const lineAt = lines(text), + entries: BibEntry[] = []; + let at = 0; + const space = () => { + while (at < text.length && /\s/.test(text[at])) at++; + }; + // From an opening brace to just after its matching close. + const braced = () => { + const start = at; + let depth = 0; + // BibTeX counts every brace, \{ included. + for (; at < text.length; at++) { + if (text[at] === "{") depth++; + else if (text[at] === "}" && --depth === 0) { + at++; + return text.slice(start + 1, at - 1); + } + } + throw new Unreadable("A brace is not closed."); + }; + const quoted = () => { + const start = ++at; + let depth = 0; + for (; at < text.length; at++) { + if (text[at] === "{") depth++; + else if (text[at] === "}") depth--; + else if (text[at] === '"' && depth === 0) { + at++; + return text.slice(start, at - 1); + } + } + throw new Unreadable("A quotation mark is not closed."); + }; + const name = () => { + const match = /^[^\s=,{}()"#%]+/.exec(text.slice(at, at + 200)); + if (!match) return ""; + at += match[0].length; + return match[0]; + }; + const value = (): Omit => { + const start = at, + parts: { text: string; resolved: string }[] = [], + dependencies: BibSource[] = []; + for (;;) { + space(); + const c = text[at]; + if (c === "{") { + const inner = braced(); + parts.push({ text: inner, resolved: inner }); + } else if (c === '"') { + const inner = quoted(); + parts.push({ text: inner, resolved: inner }); + } else { + const word = name(); + if (!word) throw new Unreadable("A field has no value."); + const macro = /^\d+$/.test(word) ? undefined : macros.get(word.toLowerCase()); + if (macro) dependencies.push(...macro.dependencies); + parts.push({ + text: word, + resolved: macro?.resolved ?? word, + }); + } + space(); + if (text[at] !== "#") break; + at++; + } + return { + value: collapse(parts.length === 1 ? parts[0].text : text.slice(start, at)), + resolved: collapse(parts.map((p) => p.resolved).join("")), + ...(dependencies.length ? { dependencies: uniqueSources(dependencies) } : {}), + }; + }; + // `name = value` pairs up to the closing delimiter. + const fields = (close: string) => { + const result: BibField[] = []; + for (;;) { + space(); + if (text[at] === close) { + at++; + return result; + } + if (text[at] === ",") { + at++; + continue; + } + const field = name().toLowerCase(); + if (!field) { + if (at >= text.length) throw new Unreadable("The entry is not closed."); + throw new Unreadable(`Unexpected “${text[at]}” in the entry.`); + } + space(); + if (text[at] !== "=") + throw new Unreadable(`The field “${field}” has no “=”.`); + at++; + result.push({ name: field, ...value() }); + space(); + if (text[at] !== "," && text[at] !== close) + throw new Unreadable(`A comma is missing after the field “${field}”.`); + } + }; + for (;;) { + const sign = text.indexOf("@", at); + if (sign < 0) break; + at = sign + 1; + space(); + const type = /^[A-Za-z]+/.exec(text.slice(at, at + 40))?.[0].toLowerCase(); + if (!type) continue; + at += type.length; + space(); + const open = text[at]; + if (open !== "{" && open !== "(") continue; + const close = open === "{" ? "}" : ")"; + if (type === "comment" || type === "preamble") { + try { + if (open === "{") braced(); + else at = Math.max(text.indexOf(")", at), at) + 1; + } catch { + at = text.length; + } + continue; + } + at++; + if (type === "string") { + try { + for (const field of fields(close)) macros.set(field.name, { + resolved: field.resolved, + dependencies: uniqueSources([{ file, lines: [lineAt(sign), lineAt(at - 1)] }, ...(field.dependencies ?? [])]), + }); + } catch { + /* A broken macro leaves its uses unresolved. */ + } + continue; + } + space(); + const keyStart = at; + while (at < text.length && !/[,\s]/.test(text[at]) && text[at] !== close) at++; + const entry: BibEntry = { + key: text.slice(keyStart, at), + type, + fields: [], + lines: [lineAt(sign), lineAt(sign)], + }; + try { + entry.fields = fields(close); + entry.lines[1] = lineAt(at - 1); + } catch (error) { + if (!(error instanceof Unreadable)) throw error; + entry.error = error.message; + // Read on from the next line that starts an entry. + const next = /\n[ \t]*@/g; + next.lastIndex = sign + 1; + const found = next.exec(text); + at = found ? found.index + 1 : text.length; + entry.lines[1] = lineAt(Math.max(at - 1, sign)); + } + entries.push(entry); + } + return entries; +} + +export interface EntryChange { + key: string; + oldKey?: string; + type: string; + oldType?: string; + status: "added" | "removed" | "modified"; + fields: ReferenceField[]; + /** Lines in the new file, or in the old one for a removed entry. */ + lines: Span; + oldLines?: Span; + /** Entry and macro definition lines that can change its effective values. */ + sources: BibSource[]; + oldSources?: BibSource[]; + error?: string; +} +const fieldMap = (entry: BibEntry) => + new Map(entry.fields.map((f) => [f.name, f])); +// Every field of an entry pair, those of the new entry in its order first. +function fieldChanges(old: BibEntry | null, next: BibEntry | null) { + const before = old ? fieldMap(old) : new Map(), + after = next ? fieldMap(next) : new Map(), + names = [...after.keys(), ...[...before.keys()].filter((n) => !after.has(n))]; + return names.map((name): ReferenceField => { + const a = before.get(name), + b = after.get(name); + return { + name, + old: a?.value ?? null, + new: b?.value ?? null, + ...(a && a.resolved !== a.value ? { oldResolved: a.resolved } : {}), + ...(b && b.resolved !== b.value ? { resolved: b.resolved } : {}), + }; + }); +} +const plainTitle = (entry: BibEntry) => + (fieldMap(entry).get("title")?.resolved ?? "") + .toLowerCase() + .replace(/[^\p{L}\p{N}]+/gu, ""); + +// The entries a change of one .bib file adds, removes or modifies. Entries +// pair by key; an added and a removed entry with the same title are one +// entry under a new key. +export function entryChanges(oldText: string, newText: string): EntryChange[] { + return compareEntries(parseBib(oldText), parseBib(newText), ""); +} +function compareEntries(before: BibEntry[], after: BibEntry[], file: string): EntryChange[] { + const unpaired = new Map(); + for (const entry of before) + unpaired.set(entry.key, [...(unpaired.get(entry.key) ?? []), entry]); + const pairs: [BibEntry | null, BibEntry | null][] = []; + for (const entry of after) { + const old = unpaired.get(entry.key)?.shift() ?? null; + pairs.push([old, entry]); + } + const removed = [...unpaired.values()].flat(); + for (const pair of pairs) { + if (pair[0]) continue; + const title = plainTitle(pair[1]!), + index = title + ? removed.findIndex( + (old) => plainTitle(old) === title && old.type === pair[1]!.type, + ) + : -1; + if (index >= 0) pair[0] = removed.splice(index, 1)[0]; + } + for (const old of removed) pairs.push([old, null]); + const changes: EntryChange[] = []; + for (const [old, next] of pairs) { + const fields = fieldChanges(old, next), + entry = (next ?? old)!; + if ( + old && + next && + old.key === next.key && + old.type === next.type && + old.error === next.error && + fields.every((f) => f.old === f.new && (f.oldResolved ?? f.old) === (f.resolved ?? f.new)) + ) + continue; + changes.push({ + key: entry.key, + ...(old && next && old.key !== next.key ? { oldKey: old.key } : {}), + type: entry.type, + ...(old && next && old.type !== next.type ? { oldType: old.type } : {}), + status: !old ? "added" : !next ? "removed" : "modified", + fields, + lines: entry.lines, + ...(old && next ? { oldLines: old.lines } : {}), + sources: entrySources(entry, file), + ...(old && next ? { oldSources: entrySources(old, file) } : {}), + ...(next?.error ? { error: next.error } : {}), + }); + } + return changes.sort( + (a, b) => + (a.status === "removed" ? 1 : 0) - (b.status === "removed" ? 1 : 0) || + a.lines[0] - b.lines[0], + ); +} + +// Line ranges [start, end) of each entry, counted from 0, for grouping a +// .bib diff's hunks by entry. +const uniqueSources = (sources: BibSource[]): BibSource[] => + [...new Map(sources.map((source) => [JSON.stringify([source.file, source.lines]), source])).values()]; +const entrySources = (entry: BibEntry, file: string) => + uniqueSources([{ file, lines: entry.lines }, ...entry.fields.flatMap((f) => f.dependencies ?? [])]); +export function entrySpans(text: string) { + return parseBib(text).flatMap((e) => entrySources(e, "").map(({ lines: span }) => ({ + key: e.key, + span: [span[0] - 1, span[1]] as Span, + }))); +} + +// TeX comments, blanked so offsets and line numbers stay. +export function uncomment(text: string) { + return text.replace(/(^|[^\\])((?:\\\\)*)%[^\n]*/g, (match, before, slashes) => + before + slashes + " ".repeat(match.length - before.length - slashes.length), + ); +} +// Ignore comments and literal examples in one pass: a percent sign inside +// \verb is data, not the start of a comment. Preserve offsets and newlines. +function texSource(text: string) { + const parts: string[] = [], + tokens = /\\([A-Za-z]+)\*?|\\[^A-Za-z]|%[^\n]*/g, + literal = /^(?:verbatim\*?|Verbatim\*?|BVerbatim|LVerbatim|SaveVerbatim|lstlisting|minted|comment)$/; + let from = 0; + for (let token; (token = tokens.exec(text)); ) { + let end = tokens.lastIndex; + const command = token[1]; + if (token[0].startsWith("%")) { + // The whole comment is already consumed. + } else if (command === "begin") { + const env = /^\s*\{([^}]+)\}/.exec(text.slice(end)); + if (!env || !literal.test(env[1])) continue; + const close = new RegExp(String.raw`\\end\s*\{${env[1].replace("*", "\\*")}\}`, "g"); + close.lastIndex = end + env[0].length; + const found = close.exec(text); + end = found ? close.lastIndex : text.length; + } else if (command === "verb" || command === "Verb" || command === "lstinline" || command === "mintinline") { + if (command !== "verb") end += /^\s*(?:\[[^\]]*\])?/.exec(text.slice(end))![0].length; + if (command === "mintinline") { + const language = /^\s*\{[^}]*\}\s*/.exec(text.slice(end)); + if (!language) continue; + end += language[0].length; + } + const delimiter = text[end]; + if (!delimiter || /\s/.test(delimiter)) continue; + if (delimiter === "{" && (command === "lstinline" || command === "mintinline")) { + let depth = 1; + while (++end < text.length && depth) { + if (text[end] === "{") depth++; + else if (text[end] === "}") depth--; + } + end = Math.min(end, text.length); + } else { + const close = text.indexOf(delimiter, end + 1), newline = text.indexOf("\n", end + 1); + end = close >= 0 && (newline < 0 || close < newline) + ? close + 1 : newline >= 0 ? newline : text.length; + } + } else if (token[0] !== "\\\\") { + continue; + } + parts.push(text.slice(from, token.index), text.slice(token.index, end).replace(/[^\n\r]/g, " ")); + from = end; + tokens.lastIndex = end; + } + return parts.join("") + text.slice(from); +} +// The line of each offset, counted from 1. +function lines(text: string) { + const starts = [0]; + for (let i = 0; i < text.length; i++) if (text[i] === "\n") starts.push(i + 1); + return (offset: number) => { + let low = 0, + high = starts.length - 1; + while (low < high) { + const mid = (low + high + 1) >> 1; + if (starts[mid] <= offset) low = mid; + else high = mid - 1; + } + return low + 1; + }; +} +// Commands that print a citation or add one to the bibliography, in the +// forms of LaTeX, natbib and biblatex, including \cites{a}{b}. +const CITE = /\\([A-Za-z]*[cC]ite[A-Za-z]*|nocite)\*?(?![A-Za-z])/g; +const NOT_CITES = new Set(["citestyle", "citesetup", "citeindextrue", "citeindexfalse", "citename"]); +export function citations(text: string, file: string) { + const source = texSource(text), + lineAt = lines(source), + found: { key: string; file: string; line: number }[] = []; + for (const match of source.matchAll(CITE)) { + if (NOT_CITES.has(match[1])) continue; + let at = match.index + match[0].length; + const groups = match[1].endsWith("cites") ? Infinity : 1; + for (let taken = 0; taken < groups; ) { + const rest = source.slice(at), + options = /^\s*(?:\([^)]*\)\s*)*(?:\[[^\]]*\]\s*)*/.exec(rest)![0], + body = /^\{([^{}]*)\}/.exec(rest.slice(options.length)); + if (!body) break; + const line = lineAt(at + options.length); + for (const key of body[1].split(",").map((k) => k.trim()).filter((k) => k && !/[\\#\s]/.test(k))) + found.push({ key, file, line }); + at += options.length + body[0].length; + taken++; + } + } + return found; +} + +// The document's files in reading order, from the main file through +// \input, \include, \subfile and \import, and the .bib files it names. +export function documentFiles(read: (file: string) => string | null, main: string) { + const base = path.posix.dirname(main), + files: string[] = [], + bibs: string[] = []; + const resolve = (name: string, dir = base, extension = ".tex") => { + name = name.trim(); + if (!name || /[\\#]/.test(name)) return null; + const file = path.posix.normalize(path.posix.join(dir, name)); + if (file.startsWith("../") || path.posix.isAbsolute(name)) return null; + return path.posix.extname(file) ? file : file + extension; + }; + const walk = (file: string) => { + if (files.includes(file)) return; + const text = read(file); + if (text == null) return; + files.push(file); + const source = texSource(text), + here = path.posix.dirname(file); + for (const m of source.matchAll( + /\\(?:input|include|subfile|InputIfFileExists)\s*\{([^}]+)\}|\\(sub)?(?:import|inputfrom|includefrom)\*?\s*\{([^}]*)\}\s*\{([^}]+)\}|\\bibliography\s*\{([^}]+)\}|\\add(?:bibresource|globalbib|sectionbib)\s*(?:\[[^\]]*\])?\s*\{([^}]+)\}/g, + )) { + if (m[5] || m[6]) { + for (const name of m[5] ? m[5].split(",") : [m[6]]) { + const bib = resolve(name, base, ".bib"); + if (bib && !bibs.includes(bib)) bibs.push(bib); + } + continue; + } + const target = m[1] + ? resolve(m[1]) + : resolve(m[4], path.posix.join(m[2] ? here : base, m[3])); + if (target) walk(target); + } + }; + walk(main); + return { files, bibs }; +} + +/** A built document as the reference check reads it. */ +export interface Document { + read(file: string): string | null; + main: string; + outline: Heading[]; +} +function citationsOf(doc: Document) { + const { files, bibs } = documentFiles((f) => doc.read(f), doc.main), + cites = files.flatMap((file) => citations(doc.read(file) ?? "", file)), + entries = bibliographyEntries(doc.read.bind(doc), bibs), + keys = new Map(); + for (const values of entries.values()) + for (const entry of values) + keys.set(entry.key, (keys.get(entry.key) ?? 0) + 1); + return { cites, bibs, entries, keys, all: cites.some((c) => c.key === "*") }; +} +// BibTeX reads databases in declaration order, retaining @string definitions +// for subsequent files. Keep the defining file with every dependency. +function bibliographyEntries(read: (file: string) => string | null, bibs: string[]) { + const macros = defaultMacros(); + return new Map(bibs.map((file) => [file, parseBib(read(file) ?? "", file, macros)])); +} +const sectionOf = (doc: Document, file: string, line: number) => { + const node = deepestNode(doc.outline, file, line, line); + return node ? `${node.number} ${node.title}`.trim() : null; +}; +const meets = (a: Span, b: Span) => a[0] <= b[1] && b[0] <= a[1]; +// A grouped macro edit can span unrelated entries. Match its actual edits, +// not the envelope between the first and last hunk. +function changesSources(unit: Unit, sources: BibSource[], side: "old" | "new") { + const own = sources.filter((source) => source.file === unit.file); + if (unit.whole_file) return own.length > 0; + const ranges: Span[] = unit.ops.length + ? unit.ops.map((op) => side === "old" ? [op[1], op[2]] : [op[3], op[4]]) + : [side === "old" ? [unit.old_start - 1, unit.old_end] : [unit.new_start - 1, unit.new_end]]; + return own.some(({ lines: [start, end] }) => ranges.some(([a, b]) => + a === b ? start - 1 < a && a < end : a < end && start - 1 < b, + )); +} + +// What a round changes in the bibliography: each added, removed or modified +// entry under the edit it belongs to, where the revised document cites it, +// and the keys it newly cites without an entry. +export function referenceChanges( + units: Unit[], + old: Document, + next: Document, +): References | undefined { + const bibUnits = units.filter((u) => u.file.endsWith(".bib") && u.kind === "text"); + const before = citationsOf(old), + after = citationsOf(next), + entries: ReferenceEntry[] = []; + for (const file of new Set([...bibUnits.map((u) => u.file), ...before.bibs, ...after.bibs])) { + const counts = new Map(), + a = before.entries.get(file) ?? parseBib(old.read(file) ?? "", file), + b = after.entries.get(file) ?? parseBib(next.read(file) ?? "", file); + for (const change of compareEntries(a, b, file)) { + const affected = bibUnits.filter((u) => + change.status === "removed" + ? changesSources(u, change.sources, "old") + : changesSources(u, change.sources, "new") || + (change.oldSources !== undefined && changesSources(u, change.oldSources, "old")), + ); + if (!affected.length) continue; + const n = (counts.get(change.key) ?? 0) + 1; + counts.set(change.key, n); + const cited = (key: string) => + after.cites + .filter((c) => c.key === key) + .map( + (c): ReferenceCitation => ({ + file: c.file, + line: c.line, + section: sectionOf(next, c.file, c.line), + }), + ), + problems: ReferenceProblem[] = []; + if (change.error) problems.push("unreadable"); + if (change.status !== "removed" && (after.keys.get(change.key) ?? 0) > 1) + problems.push("duplicate-key"); + const gone = change.status === "removed" ? change.key : change.oldKey; + if (gone && !after.keys.has(gone) && cited(gone).length) + problems.push("still-cited"); + if (after.bibs.length && !after.bibs.includes(file)) + problems.push("not-in-bibliography"); + entries.push({ + id: `${file}#${change.key}${n > 1 ? `#${n}` : ""}`, + unit: affected[0].id, + ...(affected.length > 1 ? { units: affected.map((u) => u.id) } : {}), + file, + key: change.key, + ...(change.oldKey ? { old_key: change.oldKey } : {}), + type: change.type, + ...(change.oldType ? { old_type: change.oldType } : {}), + status: change.status, + fields: change.fields, + lines: change.lines, + cited: cited(change.key), + ...(gone && gone !== change.key ? { old_cited: cited(gone) } : {}), + problems, + ...(change.error ? { error: change.error } : {}), + }); + } + } + // Keys the revised document cites with no entry in its bibliography, if + // it names one and did not already lack them. + const missing: MissingCitation[] = []; + if (after.bibs.length) { + // A key whose entry this round removes or renames is shown with it. + const lacking = new Set([ + ...before.cites.filter((c) => !before.keys.has(c.key)).map((c) => c.key), + ...entries.flatMap((e) => (e.status === "removed" ? [e.key] : e.old_key ? [e.old_key] : [])), + ]); + for (const cite of after.cites) { + if (cite.key === "*" || after.keys.has(cite.key) || lacking.has(cite.key)) + continue; + const unit = units.find( + (u) => + u.file === cite.file && + u.status !== "deleted" && + meets([cite.line, cite.line], [u.new_start, u.new_end]), + ); + missing.push({ + key: cite.key, + file: cite.file, + line: cite.line, + unit: unit?.id ?? null, + section: sectionOf(next, cite.file, cite.line), + }); + } + } + if (!entries.length && !missing.length) return undefined; + return { entries, missing }; +} + +// The paper's whole bibliography as it is, for a panel that checks it +// rather than a round's changes. +export function bibliographyReport(doc: Document): Bibliography { + const read = (file: string) => doc.read(file), + { files, bibs } = documentFiles(read, doc.main), + parsed = bibliographyEntries(read, bibs), + cites = files.flatMap((file) => citations(read(file) ?? "", file)), + place = (c: { file: string; line: number }): ReferenceCitation => ({ + file: c.file, + line: c.line, + section: sectionOf(doc, c.file, c.line), + }), + entries: BibliographyEntry[] = [], + absent: string[] = [], + counts = new Map(); + for (const file of bibs) { + const text = read(file); + if (text == null) { + absent.push(file); + continue; + } + for (const entry of parsed.get(file)!) { + const n = (counts.get(`${file}#${entry.key}`) ?? 0) + 1; + counts.set(`${file}#${entry.key}`, n); + entries.push({ + id: `${file}#${entry.key}${n > 1 ? `#${n}` : ""}`, + file, + key: entry.key, + type: entry.type, + fields: entry.fields.map((f) => ({ + name: f.name, + value: f.value, + ...(f.resolved !== f.value ? { resolved: f.resolved } : {}), + })), + lines: entry.lines, + cited: cites.filter((c) => c.key === entry.key).map(place), + ...(entry.error ? { error: entry.error } : {}), + }); + } + } + const keys = new Set(entries.map((e) => e.key)), + group = (value: (e: BibliographyEntry) => string) => { + const groups = new Map(); + for (const entry of entries) { + const v = value(entry); + if (v) groups.set(v, [...(groups.get(v) ?? []), entry]); + } + return [...groups.values()].filter((list) => list.length > 1); + }, + field = (e: BibliographyEntry, name: string) => { + const f = e.fields.find((x) => x.name === name); + return f ? (f.resolved ?? f.value) : ""; + }, + titled = (e: BibliographyEntry) => { + const title = compact(field(e, "title")); + return title.length >= 15 ? title : ""; + }, + distinct = (list: BibliographyEntry[]) => new Set(list.map((e) => e.key)).size > 1; + return { + bibs, + absent, + entries, + missing: bibs.length + ? cites.filter((c) => c.key !== "*" && !keys.has(c.key)).map((c) => ({ key: c.key, ...place(c) })) + : [], + duplicates: group((e) => e.key).map((list) => list.map((e) => e.id)), + similar: [ + ...group((e) => (doiOf(Object.fromEntries(e.fields.map((f) => [f.name, f.resolved ?? f.value]))) ?? "").toLowerCase()) + .filter(distinct) + .map((list) => ({ ids: list.map((e) => e.id), by: "doi" as const })), + ...group(titled) + .filter(distinct) + .map((list) => ({ ids: list.map((e) => e.id), by: "title" as const })), + ], + all_cited: cites.some((c) => c.key === "*"), + }; +} diff --git a/vendor/latex-review/src/core/changes.ts b/vendor/latex-review/src/core/changes.ts index 25231d2..71a13b7 100644 --- a/vendor/latex-review/src/core/changes.ts +++ b/vendor/latex-review/src/core/changes.ts @@ -1,5 +1,6 @@ import { Repo, within } from "./git.ts"; import { groups, opcodes } from "./diff.ts"; +import { entrySpans } from "./bib.ts"; import type { Scope, Unit, Op } from "../types.ts"; export const EXCERPT = 6000; export function textOf(data: Buffer | null): string | null { @@ -34,6 +35,8 @@ export function diffUnits( const left = binary ? [] : splitLines(a!), right = binary ? [] : splitLines(b!); let hunks = binary ? [] : groups(opcodes(left, right)); + if (!binary && file.endsWith(".bib")) + hunks = byEntry(hunks, entrySpans(a!), entrySpans(b!)); const whole = binary || status !== "modified" || isFigure(file) || !hunks.length; if (whole) hunks = [[["replace", 0, left.length, 0, right.length]]]; @@ -61,6 +64,42 @@ export function diffUnits( } return units; } +// A bibliography entry is decided whole: hunks that touch the same entry, +// before or after, or the same key on either side, become one change, so +// accepting part of an entry cannot mix its old and new fields. +export function byEntry( + hunks: Op[][], + before: { key: string; span: [number, number] }[], + after: { key: string; span: [number, number] }[], +) { + // An empty range touches an entry it falls strictly inside. + const touches = ([a, b]: [number, number], [s, e]: [number, number]) => + a === b ? s < a && a < e : a < e && s < b; + const root = hunks.map((_, i) => i), + find = (i: number): number => (root[i] === i ? i : (root[i] = find(root[i]))), + owner = new Map(); + hunks.forEach((hunk, i) => { + const old: [number, number] = [hunk[0][1], hunk.at(-1)![2]], + next: [number, number] = [hunk[0][3], hunk.at(-1)![4]], + keys = [ + ...before.filter((e) => touches(old, e.span)).map((e) => e.key), + ...after.filter((e) => touches(next, e.span)).map((e) => e.key), + ]; + for (const key of keys) { + const j = owner.get(key); + if (j === undefined) owner.set(key, i); + else root[find(i)] = find(j); + } + }); + const merged = new Map(); + hunks.forEach((hunk, i) => { + const r = find(i); + merged.set(r, [...(merged.get(r) ?? []), ...hunk]); + }); + return [...merged.values()].map((ops) => + ops.sort((p, q) => p[1] - q[1] || p[3] - q[3]), + ); +} function edits(units: Unit[], index: 1 | 3): Op[] { return units.flatMap((u) => u.ops).sort((a, b) => b[index] - a[index]); } diff --git a/vendor/latex-review/src/core/review.ts b/vendor/latex-review/src/core/review.ts index 0e6d25d..bf391b3 100644 --- a/vendor/latex-review/src/core/review.ts +++ b/vendor/latex-review/src/core/review.ts @@ -261,6 +261,7 @@ export class Review { loose: render.loose, warnings: round.summary.warnings ?? [], files: `/files/${name}/${number}/`, + ...(render.references ? { references: render.references } : {}), }; } history(name: string, round: Round): Record { diff --git a/vendor/latex-review/src/core/verify.ts b/vendor/latex-review/src/core/verify.ts new file mode 100644 index 0000000..47a86a6 --- /dev/null +++ b/vendor/latex-review/src/core/verify.ts @@ -0,0 +1,784 @@ +import { createHash } from "node:crypto"; +import fs from "node:fs"; +import path from "node:path"; +import type { + CheckDifference, + CheckRecord, + ReferenceCheck, + ReferenceChecks, + References, +} from "../types.ts"; + +// Checks bibliography entries against published records: a DOI through +// doi.org, an arXiv identifier through arXiv, and otherwise the title in +// Crossref, then DBLP, then arXiv. Only the entry's title, authors, year and +// identifiers leave the computer. A record agrees when its title, authors and +// year match; finding none is "unverified", not proof that the work does not +// exist, since theses, reports and many books are in none of these sources. + +export interface CheckInput { + id: string; + type: string; + fields: Record; +} +export type Fetch = (url: string, init?: RequestInit) => Promise; +interface Candidate { + record: CheckRecord; + families: string[]; + preprint: boolean; + /** Crossref's type of the work, such as journal-article or book. */ + kind?: string; +} +type Lookup = T | "unavailable"; + +const AGENT = "latex-review bibliography check (+https://github.com/cu1ch3n/papilot)"; +// Milliseconds between two requests to a host. arXiv asks for one request +// every three seconds; DBLP for restraint. +const PACE: Record = { + "doi.org": 200, + "api.crossref.org": 250, + "dblp.org": 1000, + "export.arxiv.org": 3000, +}; + +// --- Text ----------------------------------------------------------------- + +const LETTERS: Record = { + ss: "ss", o: "o", O: "O", aa: "a", AA: "A", ae: "ae", AE: "AE", + oe: "oe", OE: "OE", l: "l", L: "L", i: "i", j: "j", +}; +const GREEK: Record = Object.fromEntries( + "alpha α beta β gamma γ delta δ epsilon ε varepsilon ε zeta ζ eta η theta θ vartheta θ iota ι kappa κ lambda λ mu μ nu ν xi ξ pi π rho ρ sigma σ tau τ upsilon υ phi φ varphi φ chi χ psi ψ omega ω Gamma Γ Delta Δ Theta Θ Lambda Λ Xi Ξ Pi Π Sigma Σ Phi Φ Psi Ψ Omega Ω" + .split(" ") + .flatMap((word, i, all) => (i % 2 ? [] : [[word, all[i + 1]]])), +); +/** LaTeX as plain text: accents to letters, Greek letters to characters, + * other commands dropped, braces gone. HTML tags, which Crossref puts in + * titles, go too. */ +export function plain(text: string) { + return text + .replace(/<\/?(?:i|b|em|strong|sub|sup|scp|tt|span|mml:[a-z]+)\b[^>]*>/gi, "") + .replace(/\\([A-Za-z]+)(?![A-Za-z])/g, (match, name) => GREEK[name] ?? match) + .replace(/\\\\|\\[ ,;:!]/g, " ") + .replace( + /\\[`'^"~=.]\s*(?:\{\s*\\?([A-Za-z])\s*\}|\\?([A-Za-z]))/g, + (_, braced, bare) => braced ?? bare, + ) + // Accents named by a letter, such as \c{c} or \v s, unlike \rm. + .replace( + /\\[uvHtcdbkr](?:\s*\{\s*\\?([A-Za-z])\s*\}|\s+\\?([A-Za-z]))/g, + (_, braced, bare) => braced ?? bare, + ) + .replace(/\\(ss|aa|AA|ae|AE|oe|OE|o|O|l|L|i|j)(?![A-Za-z])\s*/g, (_, c) => LETTERS[c]) + .replace(/\\([&%$#_{}])/g, "$1") + .replace(/\\(LaTeX|TeX)(?![A-Za-z])/g, "$1") + .replace(/\\[A-Za-z]+\*?\s*/g, "") + .replace(/[{}$]/g, "") + .replace(/~/g, " ") + .replace(/-{2,3}/g, "-") + .replace(/\s+/g, " ") + .trim(); +} +const fold = (text: string) => + text.normalize("NFKD").replace(/\p{M}+/gu, "").toLowerCase(); +/** Letters and digits only, for comparing titles and names. */ +export const compact = (text: string) => + fold(plain(text)).replace(/[^\p{L}\p{N}]+/gu, ""); +function similarity(a: string, b: string) { + if (!a || !b) return 0; + if (a === b) return 1; + let previous = Array.from({ length: b.length + 1 }, (_, j) => j); + for (let i = 1; i <= a.length; i++) { + const row = [i]; + for (let j = 1; j <= b.length; j++) + row[j] = Math.min( + previous[j] + 1, + row[j - 1] + 1, + previous[j - 1] + (a[i - 1] === b[j - 1] ? 0 : 1), + ); + previous = row; + } + return 1 - previous[b.length] / Math.max(a.length, b.length); +} +/** 1 for the same title; 0.9 when one adds only a subtitle after a colon, + * full stop or dash. Words added without one make another title. */ +export function titleScore(a: string, b: string) { + const x = compact(a), + y = compact(b); + if (x && x === y) return 1; + const main = (title: string) => compact(plain(title).split(/\s*(?:[:.?!]|\s[-–—])\s+/)[0]), + subtitle = + (x.length >= 15 && main(b) === x && y !== x) || (y.length >= 15 && main(a) === y && x !== y) + ? 0.9 + : 0; + return Math.max(similarity(x.slice(0, 400), y.slice(0, 400)), subtitle); +} + +// --- Names ---------------------------------------------------------------- + +// Split at top-level " and ", as BibTeX does. +function splitNames(value: string) { + const names: string[] = []; + let depth = 0, + start = 0; + for (let i = 0; i < value.length; i++) { + if (value[i] === "{") depth++; + else if (value[i] === "}") depth--; + else if (depth === 0 && /\s/.test(value[i])) { + const match = /^\s+and\s+/i.exec(value.slice(i)); + if (match) { + names.push(value.slice(start, i)); + i += match[0].length - 1; + start = i + 1; + } + } + } + names.push(value.slice(start)); + return names.map((n) => n.trim()).filter(Boolean); +} +function topLevel(name: string, separator: RegExp) { + const parts: string[] = []; + let depth = 0, + current = ""; + for (const c of name) { + if (c === "{") depth++; + if (c === "}") depth--; + if (depth === 0 && separator.test(c)) { + parts.push(current); + current = ""; + } else current += c; + } + parts.push(current); + return parts.map((p) => p.trim()).filter(Boolean); +} +/** A BibTeX author list as names and family names; `others` marks a list + * cut short with "and others". */ +export function bibAuthors(value: string) { + const names: string[] = [], + families: string[] = []; + let others = false; + for (const name of splitNames(value)) { + if (name.toLowerCase() === "others") { + others = true; + continue; + } + let family: string; + if (/^\{[^{}]*\}$/.test(name)) family = name; + else { + const comma = topLevel(name, /,/); + if (comma.length > 1) family = comma[0]; + else { + const words = topLevel(name, /\s/), + von = words.findIndex((w, i) => i > 0 && i < words.length - 1 && /^[a-z]/.test(w)); + family = words.slice(von > 0 ? von : words.length - 1).join(" "); + } + } + names.push(plain(comma(name))); + families.push(compact(family)); + } + return { names, families, others }; +} +// "Knuth, Donald E." as "Donald E. Knuth" for display. +function comma(name: string) { + const parts = topLevel(name, /,/); + return parts.length === 2 ? `${parts[1]} ${parts[0]}` : name; +} +const recordFamily = (name: string) => + compact(name.replace(/\s+\d{4}$/, "").trim().split(/\s+/).at(-1) ?? ""); +const sameFamily = (a: string, b: string) => + a === b || (Math.min(a.length, b.length) >= 3 && (a.endsWith(b) || b.endsWith(a))); + +// --- Identifiers ---------------------------------------------------------- + +export function doiOf(fields: Record) { + const clean = (value: string) => + value + .replace(/\\_/g, "_") + .replace(/^\s*(?:https?:\/\/(?:dx\.)?doi\.org\/|doi:\s*)/i, "") + .replace(/[\s.,;]+$/, "") + .trim(); + if (fields.doi) { + const doi = clean(fields.doi); + if (/^10\.\d{4,9}\/\S+$/.test(doi)) return doi; + } + const inUrl = /doi\.org\/(10\.\d{4,9}\/[^\s}]+)/i.exec(fields.url ?? ""); + return inUrl ? clean(inUrl[1]) : null; +} +const ARXIV = /(\d{4}\.\d{4,5}|[a-z-]+(?:\.[A-Z]{2})?\/\d{7})(?:v\d+)?/; +export function arxivOf(fields: Record) { + const kind = (fields.archiveprefix ?? fields.eprinttype ?? "").toLowerCase(); + if (fields.eprint && (kind === "arxiv" || !kind)) { + const match = new RegExp(`^\\s*(?:arxiv:)?${ARXIV.source}\\s*$`, "i").exec(fields.eprint); + if (match) return match[1]; + } + if (fields.arxiv) { + const match = ARXIV.exec(fields.arxiv); + if (match) return match[1]; + } + const url = /arxiv\.org\/(?:abs|pdf)\/([^\s}?#]+)/i.exec(fields.url ?? ""); + if (url) { + const match = ARXIV.exec(url[1]); + if (match) return match[1]; + } + for (const name of ["journal", "note", "howpublished", "booktitle", "publisher"]) { + const match = new RegExp(`arXiv:\\s*${ARXIV.source}`, "i").exec(fields[name] ?? ""); + if (match) return match[1]; + } + const doi = /^10\.48550\/arxiv\.(.+)$/i.exec(doiOf(fields) ?? ""); + return doi ? (ARXIV.exec(doi[1])?.[1] ?? null) : null; +} +const yearOf = (fields: Record) => { + const match = /\b(\d{4})\b/.exec(fields.year ?? fields.date ?? ""); + return match ? Number(match[1]) : null; +}; + +// --- HTTP ----------------------------------------------------------------- + +interface Answer { + status: number; + type: string; + body: string; +} +const delay = (ms: number) => new Promise((r) => setTimeout(r, ms)); +/** One request at a time to each host, paced, with a short retry, and a + * host that keeps failing set aside for a few minutes. */ +export class Http { + private queues = new Map>(); + private last = new Map(); + private down = new Map(); + constructor( + private fetcher: Fetch = fetch, + private pace: Record = PACE, + private now: () => number = Date.now, + private sleep: (ms: number) => Promise = delay, + ) {} + get(url: string, accept: string): Promise> { + const host = new URL(url).hostname, + run = (this.queues.get(host) ?? Promise.resolve()).then(() => + this.request(host, url, accept), + ); + this.queues.set(host, run.catch(() => {})); + return run; + } + private async request(host: string, url: string, accept: string): Promise> { + if ((this.down.get(host) ?? 0) > this.now()) return "unavailable"; + for (let attempt = 0; attempt < 3; attempt++) { + const wait = (this.last.get(host) ?? 0) + (this.pace[host] ?? 500) - this.now(); + if (wait > 0) await this.sleep(wait); + this.last.set(host, this.now()); + try { + const response = await this.fetcher(url, { + headers: { Accept: accept, "User-Agent": AGENT }, + redirect: "follow", + signal: AbortSignal.timeout(15000), + }); + const type = response.headers.get("content-type") ?? "", + body = await response.text(); + // A page asking to prove the client is not a bot is not an answer. + if (response.status === 202 || (accept.includes("json") && response.ok && !type.includes("json"))) { + this.down.set(host, this.now() + 10 * 60000); + return "unavailable"; + } + if (response.status === 429 || response.status >= 500) { + const after = Number(response.headers.get("retry-after")); + if (attempt < 2) { + await this.sleep(Math.min(Number.isFinite(after) && after > 0 ? after * 1000 : 1500 * (attempt + 1), 10000)); + continue; + } + this.down.set(host, this.now() + 5 * 60000); + return "unavailable"; + } + return { status: response.status, type, body }; + } catch { + if (attempt < 2) { + await this.sleep(1000 * (attempt + 1)); + continue; + } + this.down.set(host, this.now() + 2 * 60000); + return "unavailable"; + } + } + return "unavailable"; + } +} + +// --- Sources -------------------------------------------------------------- + +const first = (value: unknown) => + (Array.isArray(value) ? value[0] : value) as string | undefined; +function cslCandidate(item: any, source: CheckRecord["source"]): Candidate | null { + const title = first(item?.title); + if (!title) return null; + const people = (Array.isArray(item.author) ? item.author : []) as any[], + authors = people.map((a) => a.literal ?? a.name ?? [a.given, a.family].filter(Boolean).join(" ")).filter(Boolean), + families = people.map((a) => compact(a.family ?? a.literal ?? a.name ?? "")).filter(Boolean), + year = (item.issued ?? item.published ?? item["published-print"] ?? item["published-online"])?.["date-parts"]?.[0]?.[0], + doi = typeof item.DOI === "string" ? item.DOI : null, + venue = first(item["container-title"]) || (typeof item.publisher === "string" ? item.publisher : null) || null; + return { + record: { + source, + title: plain(String(title)), + authors, + year: Number.isInteger(year) ? year : null, + venue, + doi, + url: doi ? `https://doi.org/${doi}` : (item.URL ?? ""), + }, + families, + preprint: item.type === "posted-content" || /arxiv/i.test(String(item.publisher ?? "")), + kind: typeof item.type === "string" ? item.type : undefined, + }; +} +async function byDoi(http: Http, doi: string): Promise> { + const url = "https://doi.org/" + doi.split("/").map(encodeURIComponent).join("/"), + answer = await http.get(url, "application/vnd.citationstyles.csl+json"); + if (answer === "unavailable") return answer; + if (answer.status === 404) return null; + if (answer.status !== 200) return "unavailable"; + try { + return cslCandidate(JSON.parse(answer.body), "doi") ?? "unavailable"; + } catch { + return "unavailable"; + } +} +// Crossref ranks the right record first far more often when the query +// carries the authors' names as well as the title. +async function searchCrossref(http: Http, title: string, names: string[]): Promise> { + const query = new URLSearchParams({ + "query.bibliographic": [title, ...names.slice(0, 4)].join(" "), + rows: "10", + select: "DOI,title,author,issued,container-title,type,publisher", + }), + answer = await http.get(`https://api.crossref.org/works?${query}`, "application/json"); + if (answer === "unavailable" || answer.status !== 200) return "unavailable"; + try { + const items: any[] = JSON.parse(answer.body)?.message?.items ?? []; + return items.map((item) => cslCandidate(item, "crossref")).filter((c): c is Candidate => c !== null); + } catch { + return "unavailable"; + } +} +async function searchDblp(http: Http, title: string): Promise> { + const query = new URLSearchParams({ q: title, format: "json", h: "5" }), + answer = await http.get(`https://dblp.org/search/publ/api?${query}`, "application/json"); + if (answer === "unavailable" || answer.status !== 200) return "unavailable"; + try { + const hits: any[] = JSON.parse(answer.body)?.result?.hits?.hit ?? []; + return hits.flatMap((hit): Candidate[] => { + const info = hit.info ?? {}, + people = info.authors?.author, + list = (Array.isArray(people) ? people : people ? [people] : []).map((a: any) => + String(a.text ?? a).replace(/\s+\d{4}$/, ""), + ); + if (!info.title) return []; + return [{ + record: { + source: "dblp", + title: String(info.title).replace(/\.$/, ""), + authors: list, + year: Number(info.year) || null, + venue: first(info.venue) ?? null, + doi: info.doi ?? null, + url: info.url ?? info.ee ?? "", + }, + families: list.map(recordFamily), + preprint: /corr/i.test(String(info.venue ?? "")), + }]; + }); + } catch { + return "unavailable"; + } +} +const decode = (text: string) => + text + .replace(/&#x([0-9a-f]+);/gi, (_, h) => String.fromCodePoint(parseInt(h, 16))) + .replace(/&#(\d+);/g, (_, d) => String.fromCodePoint(Number(d))) + .replace(/"/g, '"') + .replace(/'/g, "'") + .replace(/</g, "<") + .replace(/>/g, ">") + .replace(/&/g, "&"); +function atomCandidates(xml: string): Candidate[] { + return [...xml.matchAll(/([\s\S]*?)<\/entry>/g)].flatMap(([, entry]): Candidate[] => { + const tag = (name: string) => + decode(new RegExp(`<${name}[^>]*>([\\s\\S]*?)`).exec(entry)?.[1] ?? "").replace(/\s+/g, " ").trim(), + id = ARXIV.exec(tag("id"))?.[1], + title = tag("title"); + if (!id || !title) return []; + const authors = [...entry.matchAll(/\s*([\s\S]*?)<\/name>/g)].map((m) => decode(m[1]).trim()); + return [{ + record: { + source: "arxiv", + title, + authors, + year: Number(/^(\d{4})/.exec(tag("published"))?.[1]) || null, + venue: tag("arxiv:journal_ref") || "arXiv", + doi: tag("arxiv:doi") || null, + url: `https://arxiv.org/abs/${id}`, + }, + families: authors.map(recordFamily), + preprint: true, + }]; + }); +} +async function byArxiv(http: Http, id: string): Promise> { + const answer = await http.get( + `https://export.arxiv.org/api/query?id_list=${encodeURIComponent(id)}`, + "application/atom+xml", + ); + if (answer === "unavailable" || answer.status !== 200) return "unavailable"; + return atomCandidates(answer.body)[0] ?? null; +} +async function searchArxiv(http: Http, title: string): Promise> { + const words = fold(title).replace(/[^\p{L}\p{N}\s-]+/gu, " ").replace(/\s+/g, " ").trim(), + answer = await http.get( + `https://export.arxiv.org/api/query?search_query=${encodeURIComponent(`ti:"${words}"`)}&max_results=5`, + "application/atom+xml", + ); + if (answer === "unavailable" || answer.status !== 200) return "unavailable"; + return atomCandidates(answer.body); +} + +// --- Comparison ----------------------------------------------------------- + +interface Entry { + type: string; + title: string; + authors: ReturnType; + year: number | null; + doi: string | null; +} +function readEntry(type: string, fields: Record): Entry { + return { + type: type.toLowerCase(), + title: plain(fields.title ?? ""), + authors: bibAuthors(fields.author ?? fields.editor ?? ""), + year: yearOf(fields), + doi: doiOf(fields), + }; +} +const yearAgrees = (entry: Entry, c: Candidate) => + entry.year === null || + c.record.year === null || + entry.year === c.record.year || + (c.preprint && entry.year > c.record.year && entry.year - c.record.year <= 2); +function authorsAgree(entry: Entry, c: Candidate) { + const bib = entry.authors.families, + record = c.families; + if (!bib.length || !record.length) return { first: true, all: true, order: true }; + const found = bib.filter((f) => record.some((r) => sameFamily(f, r))), + counted = entry.authors.others ? bib.length <= record.length : bib.length === record.length; + return { + first: sameFamily(bib[0], record[0]), + all: found.length === bib.length && counted, + order: bib.every((f, i) => record[i] !== undefined && sameFamily(f, record[i])), + }; +} +function differences(entry: Entry, c: Candidate): CheckDifference[] { + const result: CheckDifference[] = [], + title = titleScore(entry.title, c.record.title); + if (entry.title && title < 1) + result.push({ field: "title", bib: entry.title, record: c.record.title, ...(title >= 0.9 ? { minor: true } : {}) }); + const authors = authorsAgree(entry, c); + if (!authors.all || !authors.first || !authors.order) + result.push({ + field: "author", + bib: [...entry.authors.names, ...(entry.authors.others ? ["others"] : [])].join("; "), + record: c.record.authors.join("; "), + ...(authors.all && authors.first ? { minor: true } : {}), + }); + if (entry.year !== null && c.record.year !== null && entry.year !== c.record.year) + result.push({ field: "year", bib: String(entry.year), record: String(c.record.year), ...(yearAgrees(entry, c) ? { minor: true } : {}) }); + if (entry.doi && c.record.doi && entry.doi.toLowerCase() !== c.record.doi.toLowerCase() && c.record.source !== "arxiv") + result.push({ field: "doi", bib: entry.doi, record: c.record.doi }); + return result; +} +function judge(entry: Entry, c: Candidate) { + const found = differences(entry, c); + return { + status: found.some((d) => !d.minor) ? ("mismatch" as const) : ("verified" as const), + record: c.record, + differences: found, + }; +} +// A journal article with a book's title is a review of the book. +const BOOKS = new Set(["book", "booklet", "manual", "collection", "mvbook", "proceedings", "reference"]); +const review = (entry: Entry, c: Candidate) => + BOOKS.has(entry.type) && (c.kind === "journal-article" || c.kind === "book-review"); +// How well a search result fits the entry, and whether it agrees. +function fit(entry: Entry, c: Candidate) { + const title = titleScore(entry.title, c.record.title), + authors = authorsAgree(entry, c), + year = yearAgrees(entry, c); + return { + title, + agrees: title >= 0.9 && authors.first && year, + value: title * 3 + (authors.first ? 1 : 0) + (authors.all ? 0.5 : 0) + (year ? 0.5 : 0), + }; +} + +export async function checkEntry(input: CheckInput, http: Http, now = Date.now): Promise { + const entry = readEntry(input.type, input.fields), + arxiv = arxivOf(input.fields), + unavailable = new Set(), + result = (check: Omit & { differences?: CheckDifference[] }): ReferenceCheck => ({ + differences: [], + ...check, + unavailable: [...unavailable], + checked: Math.floor(now() / 1000), + }); + if (!entry.doi && !arxiv && !compact(entry.title)) + return result({ status: "skipped", via: null, reason: "nothing-to-check" }); + let doiLost = false, + arxivLost = false, + answered = false; + // An identifier that names no record is itself a finding, also when the + // work turns up by its title. + const settle = (c: Candidate, via: ReferenceCheck["via"], agrees = true) => { + let found = judge(entry, c).differences; + if (doiLost) + found = [{ field: "doi", bib: entry.doi!, record: c.record.doi ?? "" }, ...found.filter((d) => d.field !== "doi")]; + if (arxivLost) + found = [{ field: "arxiv", bib: arxiv!, record: c.record.source === "arxiv" ? (ARXIV.exec(c.record.url)?.[1] ?? "") : "" }, ...found]; + return result({ + status: !agrees || found.some((d) => !d.minor) ? "mismatch" : "verified", + via, + record: c.record, + differences: found, + }); + }; + if (entry.doi && !/^10\.48550\//.test(entry.doi)) { + const found = await byDoi(http, entry.doi); + if (found === "unavailable") unavailable.add("doi.org"); + else if (found) return settle(found, "doi"); + else answered = doiLost = true; + } + if (arxiv) { + const found = await byArxiv(http, arxiv); + if (found === "unavailable") unavailable.add("arXiv"); + else if (found) return settle(found, "arxiv"); + else answered = arxivLost = true; + } + const reason = doiLost ? "doi-not-found" : arxivLost ? "arxiv-not-found" : "no-match"; + if (compact(entry.title)) { + let best: { c: Candidate; fit: ReturnType } | null = null; + const searches: [string, () => Promise>][] = [ + ["Crossref", () => searchCrossref(http, entry.title, entry.authors.names)], + ["DBLP", () => searchDblp(http, entry.title)], + ["arXiv", () => searchArxiv(http, entry.title)], + ]; + for (const [name, search] of searches) { + const found = await search(); + if (found === "unavailable") { + unavailable.add(name); + continue; + } + answered = true; + for (const c of found) { + if (review(entry, c)) continue; + const f = fit(entry, c); + // A result that agrees beats a closer one that does not. + if ( + f.title >= 0.6 && + (!best || (f.agrees === best.fit.agrees ? f.value > best.fit.value : f.agrees)) + ) + best = { c, fit: f }; + } + if (best?.fit.agrees) break; + } + // The same title with other authors or another year is the work with + // wrong details; a similar title, or one with a subtitle added, is only + // the closest record. + if (best && (best.fit.agrees || best.fit.title >= 0.95)) return settle(best.c, "title", best.fit.agrees); + if (best) + return result({ status: "unverified", via: "title", reason, record: best.c.record, differences: judge(entry, best.c).differences }); + } + return result({ + status: answered ? "unverified" : "failed", + via: entry.doi ? "doi" : arxiv ? "arxiv" : "title", + reason, + }); +} + +// --- Checker -------------------------------------------------------------- + +/** The added and modified entries of a round, as the check reads them. */ +export function checkInputs(references?: References): CheckInput[] { + return (references?.entries ?? []) + .filter((e) => e.status !== "removed" && !e.error) + .map((e) => ({ + id: e.id, + type: e.type, + fields: Object.fromEntries( + e.fields.filter((f) => f.new !== null).map((f) => [f.name, f.resolved ?? f.new!]), + ), + })); +} +/** Entries of the whole bibliography, as the check reads them. */ +export function entryInputs(entries: { id: string; type: string; fields: { name: string; value: string; resolved?: string }[]; error?: string }[]): CheckInput[] { + return entries + .filter((e) => !e.error) + .map((e) => ({ id: e.id, type: e.type, fields: Object.fromEntries(e.fields.map((f) => [f.name, f.resolved ?? f.value])) })); +} +// Bump when the comparison changes, so that saved results are redone. +const VERSION = 2; +const LOOKED_UP = ["title", "author", "editor", "year", "date", "doi", "eprint", "archiveprefix", "eprinttype", "arxiv", "url", "journal", "note", "howpublished", "booktitle", "publisher"]; +export const keyOf = (input: CheckInput) => + createHash("sha256") + .update(JSON.stringify([VERSION, input.type.toLowerCase(), LOOKED_UP.map((n) => input.fields[n] ?? "")])) + .digest("hex") + .slice(0, 32); +// How long a result stands before the entry is looked up again, in seconds. +const KEEP: Record = { + verified: 30 * 86400, + mismatch: 30 * 86400, + skipped: 30 * 86400, + unverified: 86400, + failed: 120, +}; + +/** Checks entries in the background and keeps the results in a file, so + * that a page can ask again until every entry has one. */ +export class ReferenceChecker { + private saved: Record | null = null; + private failed = new Map(); + private running = new Map>(); + private waiting: (() => void)[] = []; + private active = 0; + private now: () => number; + private concurrency: number; + private http: Http; + constructor( + private file: string | null, + options: { fetch?: Fetch; pace?: Record; now?: () => number; concurrency?: number } = {}, + ) { + this.now = options.now ?? Date.now; + this.concurrency = options.concurrency ?? 3; + this.http = new Http(options.fetch, options.pace, this.now); + } + private results() { + if (!this.saved) { + try { + const data = JSON.parse(fs.readFileSync(this.file ?? "", "utf8")); + this.saved = data?.version === VERSION && data.checks && typeof data.checks === "object" ? data.checks : {}; + } catch { + this.saved = {}; + } + } + return this.saved!; + } + private save() { + if (!this.file) return; + const kept = Object.entries(this.results()) + .sort((a, b) => b[1].checked - a[1].checked) + .slice(0, 3000); + this.saved = Object.fromEntries(kept); + try { + fs.mkdirSync(path.dirname(this.file), { recursive: true }); + const temporary = `${this.file}.${process.pid}.tmp`; + fs.writeFileSync(temporary, JSON.stringify({ version: VERSION, checks: this.saved })); + fs.renameSync(temporary, this.file); + } catch { + /* The results stay in memory. */ + } + } + private fresh(check: ReferenceCheck) { + return this.now() / 1000 - check.checked < KEEP[check.status]; + } + /** The results known for these entries. With `start`, entries without a + * fresh result are looked up; `retry` also redoes those that found nothing + * or could not be checked. */ + lookup(inputs: CheckInput[], start = false, retry = false): ReferenceChecks { + const checks: Record = {}; + let pending = 0; + for (const input of inputs) { + const key = keyOf(input), + known = this.failed.get(key) ?? this.results()[key], + redo = retry && known && (known.status === "failed" || known.status === "unverified"); + if (this.running.has(key)) { + pending++; + continue; + } + if (known && this.fresh(known) && !redo) { + checks[input.id] = known; + continue; + } + if (!start) { + if (known) checks[input.id] = known; + continue; + } + pending++; + this.run(key, input); + } + return { checks, pending }; + } + private run(key: string, input: CheckInput) { + const task = (async () => { + if (this.active >= this.concurrency) await new Promise((r) => this.waiting.push(r)); + this.active++; + try { + const check = await checkEntry(input, this.http, this.now); + if (check.status === "failed") this.failed.set(key, check); + else { + this.failed.delete(key); + this.results()[key] = check; + this.save(); + } + } catch { + this.failed.set(key, { + status: "failed", + via: null, + differences: [], + unavailable: [], + checked: Math.floor(this.now() / 1000), + }); + } finally { + this.active--; + this.waiting.shift()?.(); + this.running.delete(key); + } + })(); + this.running.set(key, task); + } + /** Settles when every lookup in progress has finished, for tests. */ + async idle() { + while (this.running.size) await Promise.all(this.running.values()); + } +} + +/** Entries the author has checked by hand, kept beside the repository's + * review state so that the panel, the review of uncommitted changes and + * every task's review share them. A confirmation is of the entry's looked-up + * fields: editing them asks again. */ +export class Confirmations { + constructor(private file: string) {} + private read(): Record { + try { + const data = JSON.parse(fs.readFileSync(this.file, "utf8")); + return data?.version === 1 && data.confirmed && typeof data.confirmed === "object" ? data.confirmed : {}; + } catch { + return {}; + } + } + /** When each of these entries was confirmed, in seconds. */ + of(inputs: CheckInput[]) { + const saved = this.read(), + result: Record = {}; + for (const input of inputs) { + const at = saved[keyOf(input)]; + if (typeof at === "number") result[input.id] = at; + } + return result; + } + set(input: CheckInput, on: boolean, now = Date.now()) { + const saved = this.read(), + key = keyOf(input); + if (on) saved[key] = Math.floor(now / 1000); + else delete saved[key]; + fs.mkdirSync(path.dirname(this.file), { recursive: true }); + const temporary = `${this.file}.${process.pid}.tmp`; + fs.writeFileSync(temporary, JSON.stringify({ version: 1, confirmed: saved })); + fs.renameSync(temporary, this.file); + } +} diff --git a/vendor/latex-review/src/render/round.ts b/vendor/latex-review/src/render/round.ts index 352f66e..64298ea 100644 --- a/vendor/latex-review/src/render/round.ts +++ b/vendor/latex-review/src/render/round.ts @@ -3,6 +3,7 @@ import path from "node:path"; import { readJson, writeJson } from "../core/files.ts"; import { within } from "../core/git.ts"; import { deepestNode } from "../core/outline.ts"; +import { referenceChanges } from "../core/bib.ts"; import { changesOf, placesOf, renderPlace } from "./effects.ts"; import { Build } from "./locate.ts"; import { renderSection, snap, Strips } from "./inline.ts"; @@ -54,8 +55,11 @@ export function classify( /\\(?:verb\b|lstinline\b|mintinline\b|begin\{(?:verbatim\*?|Verbatim|lstlisting|minted)\})/.test( (old.read(name) ?? "") + (next.read(name) ?? ""), ); + // A .bib file is not typeset where it is written, and % in a URL is not + // a comment there: its edits are compared in the PDF. if ( unit.kind !== "binary" && + !name.endsWith(".bib") && !whitespaceSensitive && !unit.truncated && unit.status === "modified" && @@ -286,6 +290,13 @@ export function renderRound( if (!placed.has(index)) result.loose.push(looseBlock(card, strips, old, next)); }); + try { + const references = referenceChanges(request.units, old, next); + if (references) result.references = references; + } catch (error) { + // The PDF review stands without the bibliography's entry view. + console.error("Could not read the bibliography changes:", error); + } fs.writeFileSync(path.join(out, "round.json"), JSON.stringify(result)); return result; } finally { diff --git a/vendor/latex-review/src/server.ts b/vendor/latex-review/src/server.ts index bb92cee..391e2d7 100644 --- a/vendor/latex-review/src/server.ts +++ b/vendor/latex-review/src/server.ts @@ -15,6 +15,7 @@ import { more, PublishError, Publisher } from "./core/publish.ts"; import { view, type ViewRequest } from "./core/view.ts"; import { acquireLock, now, readJson, writeJson } from "./core/files.ts"; import { stopAll } from "./core/process.ts"; +import { checkInputs, Confirmations, ReferenceChecker } from "./core/verify.ts"; import type { ActionKind } from "./types.ts"; export class App { root: string; @@ -29,6 +30,8 @@ export class App { review: Review; delivery: Deliveries; publisher: Publisher; + references: ReferenceChecker; + confirmations: Confirmations; private cachedProject?: Project; private stamp?: number; constructor(root: string) { @@ -40,6 +43,12 @@ export class App { this.review = new Review(this); this.delivery = new Deliveries(this); this.publisher = new Publisher(this); + this.references = new ReferenceChecker( + path.join(this.state, "references.json"), + ); + this.confirmations = new Confirmations( + path.join(this.state, "references-confirmed.json"), + ); this.review.recover(); this.delivery.recover(); } @@ -211,6 +220,55 @@ export function createServer(app: App) { } }, ); + // Checks of the bibliography entries a round adds or modifies. The page + // asks with check=1 to look up entries online, and keeps asking until + // none is pending; without it only results already known are returned. + server.get<{ + Params: { name: string; number: string }; + Querystring: { check?: string; retry?: string }; + }>("/api/tasks/:name/rounds/:number/references", (request) => { + const inputs = checkInputs( + app.review.roundDetail(request.params.name, Number(request.params.number)) + .references, + ); + return { + ...app.references.lookup( + inputs, + request.query.check === "1", + request.query.retry === "1", + ), + confirmed: app.confirmations.of(inputs), + }; + }); + // The author confirms, or takes back, that they checked an entry by hand. + server.post<{ + Params: { name: string; number: string }; + Body: { id: string; confirmed: boolean }; + }>( + "/api/tasks/:name/rounds/:number/references/confirm", + { + schema: { + body: { + ...{ type: "object" as const }, + required: ["id", "confirmed"], + properties: { id: { type: "string" }, confirmed: { type: "boolean" } }, + }, + }, + }, + (request, reply) => { + const inputs = checkInputs( + app.review.roundDetail(request.params.name, Number(request.params.number)) + .references, + ), + input = inputs.find((i) => i.id === request.body.id); + if (!input) return reply.code(404).send({ error: "No such entry in this round." }); + app.confirmations.set(input, request.body.confirmed); + return { + ...app.references.lookup(inputs), + confirmed: app.confirmations.of(inputs), + }; + }, + ); server.get<{ Params: { name: string; number: string; file: string } }>( "/files/:name/:number/:file", (request, reply) => { diff --git a/vendor/latex-review/src/types.ts b/vendor/latex-review/src/types.ts index 6026ce7..1f28688 100644 --- a/vendor/latex-review/src/types.ts +++ b/vendor/latex-review/src/types.ts @@ -239,6 +239,130 @@ export interface RenderedRound { units: Record; sections: { node: string; number: string; title: string; blocks: Block[] }[]; loose: ChangeBlock[]; + references?: References; +} +/** One field of a changed bibliography entry, before and after; null + * where the entry lacks it. `resolved` is the new value with @string + * macros expanded, when that differs; `oldResolved` is its old expansion. */ +export interface ReferenceField { + name: string; + old: string | null; + new: string | null; + resolved?: string; + oldResolved?: string; +} +export interface ReferenceCitation { + file: string; + line: number; + section: string | null; +} +/** still-cited: a removed key that the document still cites. + * not-in-bibliography: the document does not read this .bib file. */ +export type ReferenceProblem = + | "still-cited" + | "duplicate-key" + | "unreadable" + | "not-in-bibliography"; +/** A bibliography entry that a round adds, removes or modifies, shown whole + * under the edit it belongs to. */ +export interface ReferenceEntry { + id: string; + unit: string; + /** All edits affecting this entry when macros span multiple files. */ + units?: string[]; + file: string; + key: string; + old_key?: string; + type: string; + old_type?: string; + status: "added" | "removed" | "modified"; + fields: ReferenceField[]; + lines: Span; + /** Where the revised document cites the key. */ + cited: ReferenceCitation[]; + /** Where it still cites a renamed entry's old key. */ + old_cited?: ReferenceCitation[]; + problems: ReferenceProblem[]; + error?: string; +} +/** A key the revised document newly cites with no bibliography entry. */ +export interface MissingCitation { + key: string; + file: string; + line: number; + unit: string | null; + section: string | null; +} +export interface References { + entries: ReferenceEntry[]; + missing: MissingCitation[]; +} +/** A published record a bibliography entry was compared with. */ +export interface CheckRecord { + source: "doi" | "crossref" | "arxiv" | "dblp"; + title: string; + authors: string[]; + year: number | null; + venue: string | null; + doi: string | null; + url: string; +} +export interface CheckDifference { + field: "title" | "author" | "year" | "doi" | "arxiv"; + bib: string; + record: string; + /** Shown, but not a reason to doubt the entry: punctuation in a title, + * or a preprint's year before the published version's. */ + minor?: boolean; +} +/** verified: a record agrees. mismatch: a record of the same work differs, + * or the DOI names another work. unverified: no record was found. + * failed: a source could not be reached. skipped: nothing to look up. */ +export interface ReferenceCheck { + status: "verified" | "mismatch" | "unverified" | "failed" | "skipped"; + record?: CheckRecord; + differences: CheckDifference[]; + /** What the entry was looked up by. */ + via: "doi" | "arxiv" | "title" | null; + reason?: "doi-not-found" | "arxiv-not-found" | "no-match" | "nothing-to-check"; + /** Sources that could not be reached for this check. */ + unavailable: string[]; + checked: number; +} +export interface ReferenceChecks { + checks: Record; + /** Entries still being looked up. */ + pending: number; + /** Entries the author has checked by hand, with when (seconds). A + * confirmation holds while the entry's looked-up fields stay the same. */ + confirmed?: Record; +} +/** An entry of the paper's bibliography as it is now. */ +export interface BibliographyEntry { + id: string; + file: string; + key: string; + type: string; + fields: { name: string; value: string; resolved?: string }[]; + lines: Span; + cited: ReferenceCitation[]; + error?: string; +} +/** The whole bibliography of a paper: its entries, where each is cited, + * and what is wrong with them without looking anything up. */ +export interface Bibliography { + /** The .bib files the paper names, and those of them that do not exist. */ + bibs: string[]; + absent: string[]; + entries: BibliographyEntry[]; + /** Citations of keys no entry defines, each place. */ + missing: (ReferenceCitation & { key: string })[]; + /** Entries sharing a key. */ + duplicates: string[][]; + /** Entries under different keys with the same DOI or title. */ + similar: { ids: string[]; by: "doi" | "title" }[]; + /** \nocite{*} puts every entry in the bibliography. */ + all_cited: boolean; } export interface RenderRequest { main: string; @@ -273,6 +397,7 @@ export interface RoundDetail extends Omit { units: (Unit & Partial & { history?: History[] })[]; warnings: string[]; files: string; + references?: References; } export type TaskState = "editing" | "building" | "build_failed" | "reviewing" | "revise" | "done"; diff --git a/vendor/latex-review/src/web/references.tsx b/vendor/latex-review/src/web/references.tsx new file mode 100644 index 0000000..8c10f93 --- /dev/null +++ b/vendor/latex-review/src/web/references.tsx @@ -0,0 +1,499 @@ +import { Fragment, useEffect, useRef, useState } from "react"; +import type { + CheckDifference, + ReferenceCheck, + ReferenceChecks, + ReferenceCitation, + ReferenceEntry, + ReferenceProblem, + References, + RoundDetail, +} from "../types.ts"; +import { language, t, tn } from "./i18n.ts"; + +// Bibliography changes are shown whole, entry by entry, with what an online +// check found. An entry that is not verified is accepted only after the +// author confirms it, and batch acceptance leaves it for the author. + +/** The host can turn online checks off; a page that cannot read the + * setting checks online. */ +export const REFERENCE_STORAGE = "latex-review:references"; +export function onlineChecks() { + try { + return localStorage.getItem(REFERENCE_STORAGE) !== "off"; + } catch { + return true; + } +} + +const SOURCE: Record = { + doi: "doi.org", + crossref: "Crossref", + dblp: "DBLP", + arxiv: "arXiv", +}; +const FIELD: Record = { + title: "Title", + author: "Authors", + year: "Year", + doi: "DOI", + arxiv: "arXiv", +}; +const ENTRY_STATUS: Record = { + added: "New entry", + modified: "Changed entry", + removed: "Removed entry", +}; +const CHECK: Record = { + verified: "Verified", + mismatch: "Differs from the record", + unverified: "Not verified", + failed: "Could not check", + skipped: "Nothing to check", + checking: "Checking…", + off: "Not checked online", + confirmed: "Checked by you", +}; +const PROBLEM: Record = { + "still-cited": "The paper still cites this key, and no entry defines it now.", + "duplicate-key": "Another entry in the bibliography has the same key.", + unreadable: "BibTeX cannot read this entry.", + "not-in-bibliography": "The paper does not use this .bib file, so the entry will not appear.", +}; + +export interface CheckState { + round: number; + value: ReferenceChecks | null; + error: boolean; +} +/** Asks for the checks of a round's entries until none is pending. */ +export function useReferenceChecks( + detail: RoundDetail | null, + load: (route: string, body?: unknown) => Promise, +) { + const [state, setState] = useState(null), + retrying = useRef(false), + [attempt, setAttempt] = useState(0), + checked = Boolean(detail?.references?.entries.some((e) => e.status !== "removed" && !e.error)); + useEffect(() => { + if (!detail || !checked) { + setState(null); + return; + } + let cancelled = false, + timer: ReturnType | undefined; + const round = detail.round; + async function poll() { + const online = onlineChecks(), + retry = retrying.current; + retrying.current = false; + try { + const value = await load( + `/rounds/${round}/references${online ? `?check=1${retry ? "&retry=1" : ""}` : ""}`, + ); + if (cancelled) return; + setState({ round, value, error: false }); + if (online && value.pending) timer = setTimeout(poll, document.hidden ? 5000 : 1500); + } catch { + if (cancelled) return; + setState((s) => ({ round, value: s?.round === round ? s.value : null, error: true })); + timer = setTimeout(poll, 5000); + } + } + void poll(); + return () => { + cancelled = true; + clearTimeout(timer); + }; + }, [detail?.round, detail?.after, checked, attempt]); + const current = state && detail && state.round === detail.round ? state : null; + return { + checks: current?.value?.checks ?? {}, + pending: current?.value?.pending ?? (checked && onlineChecks() ? 1 : 0), + confirmed: current?.value?.confirmed ?? {}, + error: Boolean(current?.error), + retry() { + retrying.current = true; + setAttempt((n) => n + 1); + }, + async confirm(id: string, on: boolean) { + if (!detail) return; + const round = detail.round, + value = await load(`/rounds/${round}/references/confirm`, { id, confirmed: on }); + setState((s) => (s?.round === round ? { round, value: { ...value, pending: Math.max(value.pending, s.value?.pending ?? 0) }, error: false } : s)); + }, + }; +} + +/** What the author should know before accepting an entry; empty when it is + * verified and nothing is wrong with it. */ +export function concerns( + entry: ReferenceEntry, + check: ReferenceCheck | undefined, + pending: boolean, + confirmed = false, +): string[] { + const found = entry.problems.map((p) => t(PROBLEM[p])); + if (entry.status === "removed" || entry.error || confirmed) return found; + const state = checkState(check, pending); + if (state !== "verified") found.push(t(CHECK[state])); + return found; +} +function checkState(check: ReferenceCheck | undefined, pending: boolean) { + if (check) return check.status; + return pending && onlineChecks() ? "checking" : "off"; +} +/** Each change's entries that need a look before acceptance, and the + * citations it adds without an entry. */ +export function risks( + references: References | undefined, + checks: Record, + pending: number, + confirmed: Record = {}, +) { + const result = new Map(); + for (const entry of references?.entries ?? []) { + const reasons = concerns(entry, checks[entry.id], pending > 0, entry.id in confirmed); + if (reasons.length) + for (const unit of entry.units ?? [entry.unit]) + result.set(unit, [...(result.get(unit) ?? []), { label: entry.key, reasons }]); + } + for (const cite of references?.missing ?? []) + if (cite.unit) + result.set(cite.unit, [ + ...(result.get(cite.unit) ?? []), + { label: `\\cite{${cite.key}}`, reasons: [t("No bibliography entry has this key.")] }, + ]); + return result; +} + +// Chinese sentences follow each other without a space. +const gap = () => (language() === "zh-CN" ? "" : " "); +function where(place: ReferenceCitation) { + return `${place.section ? `${place.section} · ` : ""}${place.file}:${place.line}`; +} +function Cited({ entry }: { entry: ReferenceEntry }) { + const places = entry.status === "removed" ? [] : entry.cited; + if (entry.status === "removed") return null; + return ( +

+ {places.length + ? tn(places.length, "Cited {n} time: {places}", "Cited {n} times: {places}", { places: places.map(where).join("; ") }) + : t("Not cited in the paper, so it does not appear in the PDF.")} +

+ ); +} +function Value({ old, next, status }: { old: string | null; next: string | null; status: ReferenceEntry["status"] }) { + if (status !== "modified" || old === next) return <>{next ?? old}; + return ( + <> + {old !== null && {old}} + {old !== null && next !== null && " "} + {next !== null && {next}} + + ); +} +function Verdict({ check, state, entry, onRetry, confirmed, onConfirm }: { + check?: ReferenceCheck; + state: string; + entry: ReferenceEntry; + onRetry: () => void; + confirmed?: number; + onConfirm?: (on: boolean) => void; +}) { + if (entry.status === "removed" || entry.error) return null; + const confirmation = + onConfirm && (confirmed !== undefined || (state !== "verified" && state !== "checking")) ? ( +

+ {confirmed !== undefined ? ( + <> + {t("You checked this entry yourself.")}{" "} + + + ) : ( + + )} +

+ ) : null; + const source = check?.record ? SOURCE[check.record.source] : "", + unreachable = check?.unavailable.length + ? t("{sources} could not be reached.", { sources: check.unavailable.join(", ") }) + : ""; + const text = + state === "checking" + ? t("Looking this entry up in Crossref, DBLP and arXiv…") + : state === "off" + ? t("Online checks are off in Settings, so this entry was not looked up.") + : state === "verified" + ? check!.differences.length + ? t("Matches the record in {source}, apart from the small differences below.", { source }) + : t("Matches the record in {source}.", { source }) + : state === "mismatch" + ? t("{source} has this work with other details. Compare them below.", { source }) + : state === "unverified" + ? check!.reason === "doi-not-found" + ? t("The DOI is not registered, and no record matches the title.") + : check!.reason === "arxiv-not-found" + ? t("No arXiv paper has this identifier, and no record matches the title.") + : t("No record in Crossref, DBLP or arXiv matches this entry. Theses, reports and many books are in none of them, so check it yourself.") + : state === "failed" + ? t("The sources could not be reached.") + : t("The entry has no title, DOI or arXiv identifier to look up."); + return ( +
+

+ {text} + {unreachable && gap() + unreachable} + {(state === "failed" || state === "unverified" || (check?.unavailable.length ?? 0) > 0) && ( + <> + {" "} + + + )} +

+ {check && check.differences.length > 0 && ( +
+ {check.differences.map((d, i) => ( + +
+ {t(FIELD[d.field])} + {d.minor ? ` · ${t("small")}` : ""} +
+
+ + {t("Entry")} + {d.bib} + + + {SOURCE[check.record?.source ?? ""] ?? t("Record")} + {d.record || (d.field === "doi" ? t("(not registered)") : d.field === "arxiv" ? t("(no such paper)") : "—")} + +
+ + ))} +
+ )} + {check?.record && ( +

+ + {state === "unverified" ? t("Closest record") : t("Record")} · {source} + + + {check.record.title} + + {[ + check.record.authors.join(", "), + check.record.venue, + check.record.year, + check.record.doi ? `doi:${check.record.doi}` : null, + ] + .filter(Boolean) + .map((part, i) => ( + · {part} + ))} +

+ )} + {confirmation} +
+ ); +} + +/** A change's bibliography entries, every field shown. */ +export function ReferenceEntries({ entries, checks, pending, onRetry, confirmed = {}, onConfirm }: { + entries: ReferenceEntry[]; + checks: Record; + pending: boolean; + onRetry: () => void; + confirmed?: Record; + onConfirm?: (id: string, on: boolean) => void; +}) { + return ( +
+ {entries.map((entry) => { + const check = checks[entry.id], + state = entry.status === "removed" ? "removed" : entry.error ? "unreadable" : checkState(check, pending), + mine = confirmed[entry.id], + shown = mine !== undefined && state !== "verified" && state !== "removed" && state !== "unreadable" ? "confirmed" : state; + return ( +
+
+ + @{entry.old_type ? <>{entry.old_type}{entry.type} : entry.type} + {"{"} + {entry.old_key ? <>{entry.old_key}{entry.key} : entry.key} + {"}"} + + {t(ENTRY_STATUS[entry.status])} + {state !== "removed" && state !== "unreadable" && ( + + {t(CHECK[shown])} + + )} + + {entry.file}:{entry.lines[0]}–{entry.lines[1]} + +
+ {entry.problems.map((p) => ( +

+ {t(PROBLEM[p])} + {p === "unreadable" && entry.error ? gap() + entry.error : ""} + {p === "still-cited" && + gap() + [...(entry.status === "removed" ? entry.cited : []), ...(entry.old_cited ?? [])].map(where).join("; ")} +

+ ))} + + + {entry.fields.map((field) => ( + + + + + ))} + +
{field.name} + + {(field.resolved !== undefined || field.oldResolved !== undefined) && ( + = + )} +
+ + onConfirm(entry.id, on))} + /> +
+ ); + })} +
+ ); +} + +/** One line above the document: how many entries changed, how many need a + * look, and the citations without an entry. */ +export function ReferenceBanner({ references, checks, pending, confirmed = {}, changeOf, onSelect, onRetry }: { + references: References; + checks: Record; + pending: number; + confirmed?: Record; + changeOf: (unit: string | null) => number; + onSelect: (index: number) => void; + onRetry: () => void; +}) { + const entries = references.entries, + looked = entries.filter((e) => e.status !== "removed" && !e.error), + mine = (e: ReferenceEntry) => e.id in confirmed && checks[e.id]?.status !== "verified", + count = (status: string) => looked.filter((e) => checks[e.id]?.status === status && !mine(e)).length, + flagged = entries + .map((entry) => ({ entry, reasons: concerns(entry, checks[entry.id], pending > 0, entry.id in confirmed) })) + .filter((item) => item.reasons.length), + summary = [ + entries.length + ? tn(entries.length, "{n} entry changed", "{n} entries changed") + : tn(references.missing.length, "{n} citation without an entry", "{n} citations without an entry"), + count("verified") && tn(count("verified"), "{n} verified", "{n} verified"), + looked.filter(mine).length && tn(looked.filter(mine).length, "{n} checked by you", "{n} checked by you"), + count("mismatch") && tn(count("mismatch"), "{n} differs from its record", "{n} differ from their records"), + count("unverified") && tn(count("unverified"), "{n} not verified", "{n} not verified"), + count("failed") && tn(count("failed"), "{n} could not be checked", "{n} could not be checked"), + pending > 0 && onlineChecks() && t("checking {n}…", { n: pending }), + !onlineChecks() && looked.length > 0 && t("online checks are off"), + ].filter(Boolean); + const link = (unit: string | null, label: string) => { + const index = changeOf(unit); + return index >= 0 ? ( + + ) : ( + {label} + ); + }; + return ( +
+
0 || references.missing.length > 0}> + + {t("Bibliography")} · {summary.join(" · ")} + + {(flagged.length > 0 || references.missing.length > 0) && ( +
    + {flagged.map(({ entry, reasons }) => ( +
  • + {link(entry.unit, entry.key)} — {reasons.join(gap())} +
    • + ))} + {references.missing.map((cite, i) => ( +
  • + {link(cite.unit, `\\cite{${cite.key}}`)} — {t("No bibliography entry has this key.")}{" "} + {where(cite)} +
    • + ))} +
+ )} + {count("failed") > 0 && ( + + )} +
+
+ ); +} + +/** Asks before accepting entries that are not verified. */ +export function ConfirmReferences({ items, staging, onCancel, onConfirm }: { + items: { label: string; reasons: string[] }[]; + staging: boolean; + onCancel: () => void; + onConfirm: () => void; +}) { + const confirm = useRef(null); + useEffect(() => confirm.current?.focus(), []); + return ( +
+ e.stopPropagation()}> +

+ {staging ? t("Stage bibliography entries that are not verified?") : t("Accept bibliography entries that are not verified?")} +

+
    + {items.map((item, i) => ( +
  • + {item.label} — {item.reasons.join(gap())} +
    • + ))} +
+

+ {t("An agent can write an entry for a work that does not exist, or with the wrong authors, year or DOI. Check each one before it goes into your paper.")} +

+
+ + +
+ +
+ ); +} diff --git a/vendor/latex-review/src/web/review.tsx b/vendor/latex-review/src/web/review.tsx index 3cd760b..65f867f 100644 --- a/vendor/latex-review/src/web/review.tsx +++ b/vendor/latex-review/src/web/review.tsx @@ -19,7 +19,14 @@ import type { import { PdfStrip } from "./PdfStrip.tsx"; import { ChangeMap } from "./ChangeMap.tsx"; import { STATUS } from "./overview.tsx"; -import { locale, t, tn, withLanguage } from "./i18n.ts"; +import { language, locale, t, tn, withLanguage } from "./i18n.ts"; +import { + ConfirmReferences, + ReferenceBanner, + ReferenceEntries, + risks, + useReferenceChecks, +} from "./references.tsx"; import { bindingLabel, readShortcuts, resolveShortcuts, shortcutAction, SHORTCUT_EVENT, SHORTCUT_STORAGE } from "./shortcuts.ts"; import { STATE, @@ -137,6 +144,13 @@ export function ReviewApp({ Record >({}), [composer, setComposer] = useState(null), + // Accepting bibliography entries that are not verified waits for this. + [confirm, setConfirm] = useState<{ + kind: Kind; + chosen: string[]; + body: string; + items: { label: string; reasons: string[] }[]; + } | null>(null), [draft, setDraft] = useState(""), [help, setHelp] = useState(false), // A note can reveal a decided change without opening the history controls. @@ -191,6 +205,13 @@ export function ReviewApp({ undecided = Object.keys(decisions).filter( (id) => decisions[id] === "pending", ), + references = useReferenceChecks(detail, api), + hazards = useMemo( + () => risks(detail?.references, references.checks, references.pending, references.confirmed), + [detail, references.checks, references.pending, references.confirmed], + ), + // Batch acceptance leaves the changes whose entries need a look. + batch = undecided.filter((id) => !hazards.has(id)), latest = Boolean( detail && state && detail.round === state.latest && !state.closed, ), @@ -198,6 +219,9 @@ export function ReviewApp({ label = (kind: string) => t((changes ? CHANGE_ACTION : ACTION)[kind] ?? kind), stateLabel = (value: string) => t((changes ? CHANGE_STATE : STATE)[value] ?? value); + // Keyboard handlers read the newest checks, not those of their render. + const hazardsNow = useRef(hazards); + hazardsNow.current = hazards; // What each button acts on: undecided changes, or in changes mode the // staged ones for Unstage and every change still on disk for Revert. const targets = (units: string[], kind: Kind) => @@ -416,6 +440,7 @@ export function ReviewApp({ kind: Kind, chosen = targets(stops[current]?.units ?? [], kind), body = "", + confirmed = false, ) { if ( !chosen.length || @@ -425,6 +450,13 @@ export function ReviewApp({ blocked(kind) ) return; + if (kind === "accept" && !confirmed) { + const items = chosen.flatMap((id) => hazardsNow.current.get(id) ?? []); + if (items.length) { + setConfirm({ kind, chosen, body, items }); + return; + } + } epoch.current++; setBusy(true); snapshot.current.busy = true; @@ -538,7 +570,8 @@ export function ReviewApp({ typing = target.matches("input,textarea,select") || target.isContentEditable; if (event.key === "Escape") { - if (help) setHelp(false); + if (confirm) setConfirm(null); + else if (help) setHelp(false); else if (composer) closeComposer(); return; } @@ -551,7 +584,7 @@ export function ReviewApp({ void submitComposer(); return; } - if (typing || help || window.getSelection()?.toString()) return; + if (typing || help || confirm || window.getSelection()?.toString()) return; if ((event.metaKey || event.ctrlKey) && !event.altKey && !event.shiftKey && event.key.toLowerCase() === "z") { event.preventDefault(); void undo(); @@ -576,7 +609,7 @@ export function ReviewApp({ }; window.addEventListener("keydown", listener); return () => window.removeEventListener("keydown", listener); - }, [state, detail, current, busy, composer, draft, help, history, shortcuts]); + }, [state, detail, current, busy, composer, draft, help, confirm, history, shortcuts]); function onScroll() { if ( !paper.current || @@ -778,6 +811,10 @@ export function ReviewApp({ inline: boolean, ): ReactNode { const known = block.units.map((id) => units.get(id)!).filter(Boolean), + // A .bib edit is shown entry by entry, every field, rather than as + // the bibliography's typeset lines alone. + bib = detail!.references?.entries.filter((e) => (e.units ?? [e.unit]).some((id) => block.units.includes(id))) ?? [], + uncited = detail!.references?.missing.filter((m) => m.unit && block.units.includes(m.unit)) ?? [], status = stopState(block.units, decisions), reviewed = !pendingUnits(stops[index], decisions).length, expanded = browsing || index === linkedChange, @@ -793,15 +830,17 @@ export function ReviewApp({ (status === "commented" ? "after" : undefined), source = openSource.has(index) || - block.identical || - (!block.before.length && !block.after.length), + (!bib.length && + (block.identical || (!block.before.length && !block.after.length))), remarks = [ ...new Set( block.units.map((id) => state?.comments[id]).filter(Boolean), ), ], location = describeLocation(known, t), - kindLabel = block.reflow + kindLabel = bib.length + ? t("Bibliography") + : block.reflow ? t("Reflow") : [ ...new Set( @@ -913,7 +952,7 @@ export function ReviewApp({ block.before.length > 0 && images(block.before, "before")} {block.after.length > 0 && images(block.after, "after")} - {block.identical && ( + {block.identical && !bib.length && (

{t("No visible change in the PDF; see the LaTeX source.")}

@@ -933,6 +972,25 @@ export function ReviewApp({ ))}
)} + {uncited.length > 0 && ( +

+ {uncited + .map((m) => t("Cites {key}, which no bibliography entry defines.", { key: m.key })) + .join(language() === "zh-CN" ? "" : " ")} +

+ )} + {bib.length > 0 && ( + 0} + onRetry={references.retry} + confirmed={references.confirmed} + onConfirm={(id, on) => + void references.confirm(id, on).catch((e) => setToast((e as Error).message)) + } + /> + )} {source && (
{known.map((unit) => ( @@ -1044,7 +1102,7 @@ export function ReviewApp({ {/* Batch decisions cover only undecided changes; accepted, rejected and commented changes keep their state. A single undecided change is decided from its own bar. */} - {count.pending > 1 && ( + {count.pending > 1 && batch.length > 0 && ( )} {count.pending > 1 && !changes && ( @@ -1289,6 +1358,19 @@ export function ReviewApp({
) : null} + {detail?.references && + (detail.references.entries.length > 0 || + detail.references.missing.length > 0) && ( + (unit ? (firstStop.get(unit) ?? -1) : -1)} + onSelect={(index) => select(index)} + onRetry={references.retry} + /> + )} {changes && detail && !count.pending && (count.accepted > 0 || !stops.length) && (
@@ -1471,6 +1553,18 @@ export function ReviewApp({ {toast}
)} + {confirm && ( + setConfirm(null)} + onConfirm={() => { + const chosen = confirm; + setConfirm(null); + void decide(chosen.kind, chosen.chosen, chosen.body, true); + }} + /> + )} {help && (
setHelp(false)}> = { "Finish or abort the merge in the Git panel before staging or reverting changes.": "请先在 Git 面板中完成或中止合并，再暂存或还原修改。", "The last commit changed. The comparison is being updated.": "上次提交已变化，正在更新比较。", + + // Bibliography entries and their online checks + Bibliography: "参考文献", + Title: "标题", + Authors: "作者", + Year: "年份", + DOI: "DOI", + arXiv: "arXiv", + "New entry": "新增条目", + "Changed entry": "修改的条目", + "Removed entry": "删除的条目", + Verified: "已核实", + "Differs from the record": "与记录有出入", + "Not verified": "未核实", + "Could not check": "未能核对", + "Nothing to check": "无从核对", + "Checking…": "核对中…", + "Not checked online": "未联网核对", + "The paper still cites this key, and no entry defines it now.": + "正文仍在引用这个 key，但现在没有条目定义它。", + "Another entry in the bibliography has the same key.": "参考文献里还有一个条目用了同一个 key。", + "BibTeX cannot read this entry.": "BibTeX 读不了这个条目。", + "The paper does not use this .bib file, so the entry will not appear.": + "论文没有用到这个 .bib 文件，这个条目不会出现在参考文献里。", + "No bibliography entry has this key.": "没有参考文献条目用这个 key。", + "Cited {n} time: {places}": "正文引用 {n} 处：{places}", + "Cited {n} times: {places}": "正文引用 {n} 处：{places}", + "Not cited in the paper, so it does not appear in the PDF.": "正文没有引用，所以不会出现在 PDF 里。", + "{sources} could not be reached.": "无法访问 {sources}。", + "Looking this entry up in Crossref, DBLP and arXiv…": "正在 Crossref、DBLP 和 arXiv 中查找这个条目…", + "Online checks are off in Settings, so this entry was not looked up.": + "设置里关闭了联网核对，这个条目没有查找。", + "Matches the record in {source}, apart from the small differences below.": + "与 {source} 的记录一致，只有下面几处小差别。", + "Matches the record in {source}.": "与 {source} 的记录一致。", + "{source} has this work with other details. Compare them below.": + "{source} 里有这篇文献，但信息不一样，请对照下面几项。", + "The DOI is not registered, and no record matches the title.": "这个 DOI 没有注册，按标题也没有找到对应记录。", + "No arXiv paper has this identifier, and no record matches the title.": + "arXiv 上没有这个编号的论文，按标题也没有找到对应记录。", + "No record in Crossref, DBLP or arXiv matches this entry. Theses, reports and many books are in none of them, so check it yourself.": + "Crossref、DBLP 和 arXiv 中都没有与这个条目对应的记录。学位论文、技术报告和很多书都不在这些库里，请自己核对。", + "The sources could not be reached.": "无法访问这些数据源。", + "The entry has no title, DOI or arXiv identifier to look up.": "这个条目没有标题、DOI 或 arXiv 编号，无从查找。", + "Check again": "重新核对", + small: "小差别", + Entry: "条目", + Record: "记录", + "(not registered)": "（未注册）", + "(no such paper)": "（没有这篇论文）", + "Closest record": "最接近的记录", + "{n} entry changed": "{n} 个条目有改动", + "{n} entries changed": "{n} 个条目有改动", + "{n} citation without an entry": "{n} 处引用没有对应条目", + "{n} citations without an entry": "{n} 处引用没有对应条目", + "{n} verified": "{n} 个已核实", + "{n} differs from its record": "{n} 个与记录有出入", + "{n} differ from their records": "{n} 个与记录有出入", + "{n} not verified": "{n} 个未核实", + "{n} could not be checked": "{n} 个未能核对", + "checking {n}…": "核对中 {n} 个…", + "online checks are off": "联网核对已关闭", + "Stage bibliography entries that are not verified?": "暂存未核实的参考文献条目？", + "Accept bibliography entries that are not verified?": "接受未核实的参考文献条目？", + "An agent can write an entry for a work that does not exist, or with the wrong authors, year or DOI. Check each one before it goes into your paper.": + "agent 可能写出根本不存在的文献，或者写错作者、年份、DOI。放进论文之前请逐条确认。", + "Stage anyway": "仍然暂存", + "Accept anyway": "仍然接受", + "Cites {key}, which no bibliography entry defines.": "引用了 {key}，但没有参考文献条目定义它。", + "Leaves {n} change with bibliography entries to check.": "留下 {n} 处含待核对参考文献的改动。", + "Leaves {n} changes with bibliography entries to check.": "留下 {n} 处含待核对参考文献的改动。", + "Checked by you": "已人工确认", + "You checked this entry yourself.": "你已人工确认过这个条目。", + "I checked it myself": "我已人工确认", + "{n} checked by you": "{n} 个已人工确认", }; diff --git a/vendor/latex-review/tests/browser/references.spec.ts b/vendor/latex-review/tests/browser/references.spec.ts new file mode 100644 index 0000000..da121e4 --- /dev/null +++ b/vendor/latex-review/tests/browser/references.spec.ts @@ -0,0 +1,136 @@ +import { expect, test, type Page } from "@playwright/test"; +import type { PageState, ReferenceChecks, RoundDetail } from "../../src/types.ts"; + +// A round whose first change adds two bibliography entries, one of which no +// source knows, and whose second modifies a verified entry. The engine's +// answers are served by the page's own routes. +const unit = (id: string, file: string, start: number) => ({ + id, file, status: "modified" as const, kind: "text" as const, whole_file: false, ops: [], + old_start: start, old_end: start, new_start: start, new_end: start + 8, + old_text: "", new_text: "@misc{x}", truncated: false, visual: "unmapped" as const, pages: [], section: null, +}); +const field = (name: string, value: string, old: string | null = null) => ({ name, old, new: value }); +const detail: RoundDetail = { + task: "refs", round: 1, before: "b", after: "a", warnings: [], files: "/files/refs/1/", sections: [], + loose: ["c001", "c002", "c003"].map((id) => ({ kind: "change" as const, units: [id], identical: false, before: [], after: [] })), + units: [unit("c001", "refs.bib", 40), unit("c002", "refs.bib", 3), unit("c003", "main.tex", 12)], + references: { + missing: [], + entries: [ + { id: "refs.bib#real", unit: "c001", file: "refs.bib", key: "real", type: "article", status: "added", lines: [40, 45], problems: [], + cited: [{ file: "main.tex", line: 12, section: "2 Method" }], + fields: [field("author", "Donald E. Knuth and Michael F. Plass"), field("title", "Breaking paragraphs into lines"), field("year", "1981"), field("doi", "10.1002/spe.4380111102")] }, + { id: "refs.bib#made-up", unit: "c001", file: "refs.bib", key: "made-up", type: "article", status: "added", lines: [47, 51], problems: [], cited: [], + fields: [field("author", "Jian Li"), field("title", "A study that was never written"), field("year", "2022")] }, + { id: "refs.bib#knuth", unit: "c002", file: "refs.bib", key: "knuth", type: "book", status: "modified", lines: [3, 8], problems: [], cited: [], + fields: [field("title", "The TeXbook", "The TeXbook"), field("year", "1984", "1986")] }, + ], + }, +}; +const checks: ReferenceChecks = { + pending: 0, + checks: { + "refs.bib#real": { status: "verified", via: "doi", differences: [], unavailable: [], checked: 1, + record: { source: "doi", title: "Breaking paragraphs into lines", authors: ["Donald E. Knuth", "Michael F. Plass"], year: 1981, venue: "Software: Practice and Experience", doi: "10.1002/spe.4380111102", url: "https://doi.org/10.1002/spe.4380111102" } }, + "refs.bib#made-up": { status: "unverified", via: "title", reason: "no-match", differences: [], unavailable: ["DBLP"], checked: 1 }, + "refs.bib#knuth": { status: "verified", via: "title", differences: [], unavailable: [], checked: 1, + record: { source: "crossref", title: "The TeXbook", authors: ["Donald E. Knuth"], year: 1984, venue: null, doi: null, url: "https://example.org/texbook" } }, + }, +}; + +async function serve(page: Page, roundDetail = detail) { + const decisions: Record = { c001: "pending", c002: "pending", c003: "pending" }, + posted: string[][] = [], + state = (): PageState => ({ + csrf: "token", task: "refs", title: "Add references", closed: false, state: "reviewing", latest: 1, round: 1, + revision: "a", decisions: { ...decisions }, comments: {}, undo: null, job: null, + }); + await page.route("**/api/tasks/refs/**", async (route) => { + const url = new URL(route.request().url()); + if (url.pathname.endsWith("/state")) return route.fulfill({ json: state() }); + if (url.pathname.endsWith("/rounds/1")) return route.fulfill({ json: roundDetail }); + if (url.pathname.endsWith("/rounds/1/references")) { + expect(url.searchParams.get("check")).toBe("1"); + return route.fulfill({ json: checks }); + } + if (url.pathname.endsWith("/decide")) { + const body = route.request().postDataJSON(); + posted.push(body.changes); + for (const id of body.changes) decisions[id] = "accepted"; + return route.fulfill({ json: state() }); + } + return route.fulfill({ status: 404, json: { error: "Unexpected request" } }); + }); + return posted; +} + +test("a .bib change shows every entry and field with its online check", async ({ page }) => { + await serve(page); + await page.goto("/?task=refs"); + const added = page.locator("article.change").first(); + await expect(added.locator(".reference-entry")).toHaveCount(2); + await expect(added.locator('.reference-entry[data-check="verified"] .reference-key')).toHaveText("@article{real}"); + await expect(added.locator('.reference-entry[data-check="unverified"] .reference-check')).toHaveText("Not verified"); + await expect(added.locator(".reference-fields th")).toHaveText(["author", "title", "year", "doi", "author", "title", "year"]); + await expect(added.locator(".reference-record a")).toHaveAttribute("href", "https://doi.org/10.1002/spe.4380111102"); + await expect(added.locator(".source")).toHaveCount(0); + const modified = page.locator("article.change").nth(1); + await expect(modified.locator('tr[data-change="changed"] del')).toHaveText("1986"); + await expect(modified.locator('tr[data-change="changed"] ins')).toHaveText("1984"); + await expect(page.locator(".references-banner")).toContainText("3 entries changed"); + await expect(page.locator(".references-flagged li")).toHaveText([/made-up — Not verified/]); +}); + +test("Accept all leaves a change with an unverified entry, which is accepted only after confirming", async ({ page }) => { + const posted = await serve(page); + await page.goto("/?task=refs"); + await expect(page.locator(".reference-check").first()).toHaveText("Verified"); + const batch = page.locator(".accept-all"); + await expect(batch.locator(".batch-count")).toHaveText("2"); + await expect(batch).toHaveAttribute("title", /Leaves 1 change with bibliography entries to check/); + await batch.click(); + await expect.poll(() => posted).toEqual([["c002", "c003"]]); + // The change with the unverified entry is left, and accepting it asks first. + const current = page.locator("article.change.current"); + await expect(current.locator(".reference-entry")).toHaveCount(2); + await current.locator(".actions button.accept").click(); + const dialog = page.locator(".references-confirm"); + await expect(dialog).toContainText("made-up — Not verified"); + await dialog.getByRole("button", { name: "Cancel" }).click(); + await expect(dialog).toHaveCount(0); + expect(posted).toHaveLength(1); + await page.keyboard.press("a"); + await page.locator(".references-confirm").getByRole("button", { name: "Accept anyway" }).click(); + await expect.poll(() => posted).toEqual([["c002", "c003"], ["c001"]]); +}); + +test("changed macro expansions are visible and unverified entries still require confirmation", async ({ page }) => { + const round = structuredClone(detail), entry = round.references!.entries[1]; + entry.status = "modified"; + entry.fields = [{ name: "title", old: "work", new: "work", oldResolved: "Original work", resolved: "A study that was never written" }]; + const posted = await serve(page, round); + await page.goto("/?task=refs"); + const row = page.locator('.reference-entry[data-status="modified"][data-check="unverified"] tr'); + await expect(row).toHaveAttribute("data-change", "changed"); + await expect(row.locator("del")).toHaveText("Original work"); + await expect(row.locator("ins")).toHaveText("A study that was never written"); + await page.locator("article.change").first().locator(".actions button.accept").click(); + await expect(page.locator(".references-confirm")).toContainText("made-up — Not verified"); + expect(posted).toEqual([]); +}); + +test("cross-file macro dependencies appear on both edits and Accept all skips both", async ({ page }) => { + const round = structuredClone(detail), entry = round.references!.entries[1]; + entry.units = ["c001", "c002"]; + round.units[1].file = "strings.bib"; + const posted = await serve(page, round); + await page.goto("/?task=refs"); + await expect(page.locator('article.change').nth(1).locator('.reference-entry[data-check="unverified"] .reference-key')).toHaveText("@article{made-up}"); + const batch = page.locator(".accept-all"); + await expect(batch.locator(".batch-count")).toHaveText("1"); + await batch.click(); + await expect.poll(() => posted).toEqual([["c003"]]); + await page.getByRole("button", { name: "Next change", exact: true }).click(); + await page.locator('article.change').nth(1).locator('.actions button.accept').click(); + await expect(page.locator(".references-confirm")).toContainText("made-up — Not verified"); +}); diff --git a/vendor/latex-review/tests/ts/i18n.test.ts b/vendor/latex-review/tests/ts/i18n.test.ts index 6cac838..c9b7783 100644 --- a/vendor/latex-review/tests/ts/i18n.test.ts +++ b/vendor/latex-review/tests/ts/i18n.test.ts @@ -22,7 +22,7 @@ function keys() { for (const [, text] of call[1].matchAll(/"((?:[^"\\]|\\.)*)"/g)) found.add(text); for (const map of source.matchAll( - /const (?:KIND|PHASE|ACTION|CHANGE_ACTION|CHANGE_STATE)\b[^{]*\{([^}]*)\}/g, + /const (?:KIND|PHASE|ACTION|CHANGE_ACTION|CHANGE_STATE|FIELD|ENTRY_STATUS|CHECK|PROBLEM)\b[^{]*\{([^}]*)\}/g, )) for (const [, text] of map[1].matchAll(/:\s*"([^"]*)"/g)) found.add(text); diff --git a/vendor/latex-review/tests/ts/references.test.ts b/vendor/latex-review/tests/ts/references.test.ts new file mode 100644 index 0000000..50a7ad5 --- /dev/null +++ b/vendor/latex-review/tests/ts/references.test.ts @@ -0,0 +1,437 @@ +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { execFileSync } from "node:child_process"; +import { Fixture } from "./helpers.ts"; +import { + bibliographyReport, + citations, + documentFiles, + entryChanges, + parseBib, + referenceChanges, + uncomment, + type Document, +} from "../../src/core/bib.ts"; +import { applyUnits, diffUnits } from "../../src/core/changes.ts"; +import { buildOutline } from "../../src/core/outline.ts"; +import { checkInputs } from "../../src/core/verify.ts"; +import { risks } from "../../src/web/references.tsx"; +import type { Unit } from "../../src/types.ts"; + +const BIB = String.raw`% A comment line, ignored like any text outside entries. +@string{ acm = "ACM" } + +@article{knuth1981, + author = {Donald E. Knuth and Michael F. Plass}, + title = {Breaking paragraphs + into lines}, + journal = {Software: Practice and Experience}, + year = 1981, + month = nov, +} + +@inproceedings{ellis1989, + author = "C. A. Ellis and S. J. Gibbs", + title = "Concurrency control in {Groupware} systems", + booktitle = acm # " SIGMOD", + year = {1989} +} +`; +const hasTeX = (() => { + try { + for (const command of ["pdflatex", "bibtex"]) execFileSync(command, ["--version"], { stdio: "ignore" }); + return true; + } catch { return false; } +})(); + +describe("BibTeX entries", () => { + it("reads entries with their lines, values as written and macros expanded", () => { + const [knuth, ellis] = parseBib(BIB); + expect(knuth).toMatchObject({ key: "knuth1981", type: "article", lines: [4, 11] }); + expect(knuth.fields.map((f) => [f.name, f.value])).toEqual([ + ["author", "Donald E. Knuth and Michael F. Plass"], + ["title", "Breaking paragraphs into lines"], + ["journal", "Software: Practice and Experience"], + ["year", "1981"], + ["month", "nov"], + ]); + expect(knuth.fields.at(-1)!.resolved).toBe("November"); + expect(ellis).toMatchObject({ key: "ellis1989", type: "inproceedings", lines: [13, 18] }); + expect(ellis.fields[1].value).toBe("Concurrency control in {Groupware} systems"); + expect(ellis.fields[2]).toMatchObject({ value: 'acm # " SIGMOD"', resolved: "ACM SIGMOD" }); + }); + it("keeps reading after an entry it cannot read, and says why", () => { + const entries = parseBib( + "@article{broken,\n title = {Unclosed,\n year = 2020\n\n@book{fine, title={Fine}}\n", + ); + expect(entries.map((e) => e.key)).toEqual(["broken", "fine"]); + expect(entries[0].error).toBeTruthy(); + expect(entries[0].lines).toEqual([1, 4]); + expect(entries[1]).toMatchObject({ lines: [5, 5], fields: [{ name: "title", value: "Fine" }] }); + expect(parseBib("@misc{a, title = {A} year = 2020}")[0].error).toMatch(/comma/); + }); + it("ignores @comment and @preamble, and parenthesised entries work", () => { + const entries = parseBib( + "@comment{ @article{hidden, title={No}} }\n@preamble{ {\\newcommand{\\x}{y}} }\n@misc(paren, title = {P})\n", + ); + expect(entries.map((e) => e.key)).toEqual(["paren"]); + }); + it("pairs entries by key and lists every field of a changed one", () => { + const next = BIB.replace("year = 1981", "year = 1982") + .replace("@inproceedings{ellis1989", "@inproceedings{ellis89") + .concat("\n@misc{new2024,\n title = {A new one},\n note = {x}\n}\n"); + const changes = entryChanges(BIB, next); + expect(changes.map((c) => [c.status, c.key, c.oldKey])).toEqual([ + ["modified", "knuth1981", undefined], + ["modified", "ellis89", "ellis1989"], + ["added", "new2024", undefined], + ]); + expect(changes[0].fields.find((f) => f.name === "year")).toEqual({ name: "year", old: "1981", new: "1982" }); + expect(changes[0].fields.find((f) => f.name === "title")).toEqual({ + name: "title", + old: "Breaking paragraphs into lines", + new: "Breaking paragraphs into lines", + }); + expect(changes[1].fields.every((f) => f.old === f.new)).toBe(true); + expect(changes[2].fields.map((f) => f.old)).toEqual([null, null]); + expect(entryChanges(next, BIB).find((c) => c.key === "new2024")?.status).toBe("removed"); + // Reformatting an entry changes none of its values. + expect(entryChanges(BIB, BIB.replace("year = 1981,", "year = 1981,"))).toEqual([]); + }); +}); + +describe("Citations", () => { + it("ignores parameterized keys and literal examples while preserving real citations and lines", () => { + const text = String.raw`\newcommand{\mycite}[1]{\cite{#1}} +\verb|\cite{not-a-real-cite}| \mycite{real} +\verb*+100% \cite{fake}+ \cite{after-percent} +\begin{verbatim} +\cite{verbatim} \input{fake} \bibliography{fake} +\end{verbatim} +\begin{lstlisting}[language=TeX] +\cite{listing} +\end{lstlisting} +\begin{minted}{tex} +\cite{minted} +\end{minted} +\Verb|\cite{fancy}| \lstinline[language=TeX]|\cite{inline}| +\mintinline{tex}{\cite{mint-inline}} \cite{last} +\cite{\dynamic,known} \\cite{escaped} % \cite{comment} +\bibliography{refs}`; + expect(citations(text, "x.tex").map((c) => `${c.key}:${c.line}`)).toEqual([ + "real:2", "after-percent:3", "last:14", "known:15", + ]); + expect(documentFiles((file) => file === "x.tex" ? text : "", "x.tex")).toEqual({ + files: ["x.tex"], bibs: ["refs.bib"], + }); + }); + it("finds the keys of LaTeX, natbib and biblatex citations, without comments", () => { + const text = String.raw`As \cite{a} and \citep[p.~3]{b, c} show. +% \cite{commented} +\textcite[see][12]{d}\nocite{*} 100\% \citet*{e} +\cites[1]{f}[2]{g} \Citep{h} \citestyle{plain} \footcite{i}`; + expect(citations(text, "x.tex").map((c) => `${c.key}:${c.line}`)).toEqual([ + "a:1", "b:1", "c:1", "d:3", "*:3", "e:3", "f:4", "g:4", "h:4", "i:4", + ]); + expect(uncomment("a % b\nc \\% d % e")).toBe("a \nc \\% d "); + }); + it("follows inputs from the main file and finds its bibliography", () => { + const files: Record = { + "paper/main.tex": String.raw`\input{sections/a}\include{b}\import{parts/}{c} +% \input{ignored} +\bibliography{refs,more.bib}\addbibresource[label=x]{extra.bib}`, + "paper/sections/a.tex": String.raw`\subimport{deep/}{d}`, + "paper/sections/deep/d.tex": "", + "paper/b.tex": "", + "paper/parts/c.tex": "", + }; + expect(documentFiles((f) => files[f] ?? null, "paper/main.tex")).toEqual({ + files: ["paper/main.tex", "paper/sections/a.tex", "paper/sections/deep/d.tex", "paper/b.tex", "paper/parts/c.tex"], + bibs: ["paper/refs.bib", "paper/more.bib", "paper/extra.bib"], + }); + }); +}); + +const MAIN = String.raw`\documentclass{article} +\begin{document} +\section{Intro} +Text \cite{knuth1981}. +\section{Related} +See \cite{ellis1989}. +\bibliography{refs} +\end{document} +`; +function doc(files: Record): Document { + const read = (f: string) => files[f] ?? null; + return { read, main: "paper/main.tex", outline: buildOutline(read, "paper/main.tex").nodes }; +} +const unit = (over: Partial): Unit => ({ + id: "c001", file: "paper/refs.bib", status: "modified", kind: "text", whole_file: false, ops: [], + old_start: 1, old_end: 1, new_start: 1, new_end: 1, old_text: "", new_text: "", truncated: false, ...over, +}); + +describe("Bibliography changes of a round", () => { + it("does not flag valid citation wrappers or literal code as missing entries", () => { + const before = String.raw`\bibliography{refs}`, + next = String.raw`\newcommand{\mycite}[1]{\cite{#1}} +\verb|\cite{not-a-real-cite}| \mycite{real} +\bibliography{refs}`, + bib = "@misc{real, title={Real work}}", + old = doc({ "paper/main.tex": before, "paper/refs.bib": bib }), + revised = doc({ "paper/main.tex": next, "paper/refs.bib": bib }); + expect(referenceChanges([unit({ file: "paper/main.tex", new_end: 3 })], old, revised)).toBeUndefined(); + expect(bibliographyReport(revised).missing).toEqual([]); + expect(bibliographyReport(revised).entries[0].cited).toMatchObject([{ file: "paper/main.tex", line: 2 }]); + }); + it("checks explicit missing citations even when nocite includes every entry", () => { + const before = MAIN.replace("\\bibliography", "\\nocite{*}\n\\bibliography"), + next = before.replace("Text \\cite{knuth1981}", "Text \\cite{knuth1981,ghost}"), + result = referenceChanges( + [unit({ file: "paper/main.tex", old_start: 4, old_end: 4, new_start: 4, new_end: 4 })], + doc({ "paper/main.tex": before, "paper/refs.bib": BIB }), + doc({ "paper/main.tex": next, "paper/refs.bib": BIB }), + )!; + expect(result.missing).toEqual([ + { key: "ghost", file: "paper/main.tex", line: 4, unit: "c001", section: "1 Intro" }, + ]); + expect(result.entries).toEqual([]); + }); + it("shows each entry under its edit, where it is cited, and what is wrong with it", () => { + const next = BIB.replace("@inproceedings{ellis1989", "@inproceedings{ellis89") + .concat("\n@misc{knuth1981,\n title = {Twice}\n}\n"); + const result = referenceChanges( + [ + unit({ id: "c001", old_start: 13, old_end: 13, new_start: 13, new_end: 13 }), + unit({ id: "c002", old_start: 19, old_end: 19, new_start: 19, new_end: 22 }), + ], + doc({ "paper/main.tex": MAIN, "paper/refs.bib": BIB }), + doc({ "paper/main.tex": MAIN.replace("Text \\cite{knuth1981}", "Text \\cite{knuth1981,missing}"), "paper/refs.bib": next }), + )!; + expect(result.entries.map((e) => [e.unit, e.key, e.status, e.problems])).toEqual([ + ["c001", "ellis89", "modified", ["still-cited"]], + ["c002", "knuth1981", "added", ["duplicate-key"]], + ]); + expect(result.entries[0]).toMatchObject({ + id: "paper/refs.bib#ellis89", + old_key: "ellis1989", + cited: [], + old_cited: [{ file: "paper/main.tex", line: 6, section: "2 Related" }], + }); + expect(result.entries[1].cited).toEqual([{ file: "paper/main.tex", line: 4, section: "1 Intro" }]); + expect(result.missing).toEqual([ + { key: "missing", file: "paper/main.tex", line: 4, unit: null, section: "1 Intro" }, + ]); + }); + it("flags a .bib file the document does not read, and reports nothing without changes", () => { + const result = referenceChanges( + [unit({ file: "paper/other.bib", status: "added", whole_file: true })], + doc({ "paper/main.tex": MAIN, "paper/refs.bib": BIB }), + doc({ "paper/main.tex": MAIN, "paper/refs.bib": BIB, "paper/other.bib": "@misc{x, title={X}}" }), + )!; + expect(result.entries.map((e) => [e.key, e.status, e.problems])).toEqual([["x", "added", ["not-in-bibliography"]]]); + expect(referenceChanges([], doc({ "paper/main.tex": MAIN }), doc({ "paper/main.tex": MAIN }))).toBeUndefined(); + }); +}); + +describe("Bibliography hunks", () => { + let f: Fixture; + beforeEach(() => { + f = new Fixture(); + }); + afterEach(async () => f.close()); + it("checks unchanged entries using macros from earlier files and guards every affected edit", () => { + const main = String.raw`\cite{one,two}\bibliography{strings,nested,refs}`, + strings = '@string{work = "Real title"}\n', + nested = '@string{extended = work # " subtitle"}\n', + bib = `@article{one, title=extended, author={John Smith}, year=2020} +@article{two, title=work, author={Jane Doe}, year=2021} +`, + files = { "paper/main.tex": main, "paper/strings.bib": strings, "paper/nested.bib": nested, "paper/refs.bib": bib }; + for (const [file, text] of Object.entries(files)) f.write(file, text); + f.repo.run(["add", "."]); + f.repo.run(["commit", "-qm", "shared macros"]); + const before = f.repo.rev("HEAD"), + revised = { ...files, "paper/strings.bib": strings.replace("Real title", "Invented title") }; + f.write("paper/strings.bib", revised["paper/strings.bib"]); + const units = diffUnits(f.repo, before, f.repo.snapshot(["paper"])), + result = referenceChanges(units, doc(files), doc(revised))!; + expect(result.entries.map((e) => [e.file, e.key, e.unit])).toEqual([ + ["paper/refs.bib", "one", units[0].id], ["paper/refs.bib", "two", units[0].id], + ]); + expect(result.entries[0].fields.find((f) => f.name === "title")).toEqual({ + name: "title", old: "extended", new: "extended", oldResolved: "Real title subtitle", resolved: "Invented title subtitle", + }); + expect(checkInputs(result).map((e) => e.fields.title)).toEqual(["Invented title subtitle", "Invented title"]); + expect([...risks(result, {}, 0).keys()]).toEqual([units[0].id]); + expect(bibliographyReport(doc(revised)).entries[0].fields.find((f) => f.name === "title")?.resolved).toBe("Invented title subtitle"); + // Editing both the macro and a dependent entry must guard both files. + revised["paper/refs.bib"] = bib.replace("year=2020", "year=2022"); + f.write("paper/refs.bib", revised["paper/refs.bib"]); + const bothUnits = diffUnits(f.repo, before, f.repo.snapshot(["paper"])), + both = referenceChanges(bothUnits, doc(files), doc(revised))!; + expect(both.entries.find((e) => e.key === "one")?.units?.sort()).toEqual(bothUnits.map((u) => u.id).sort()); + expect([...risks(both, {}, 0).keys()].sort()).toEqual(bothUnits.map((u) => u.id).sort()); + // Removing a defining file must still find dependents through old sources. + f.write("paper/refs.bib", bib); + f.repo.run(["rm", "-f", "paper/strings.bib"]); + const removed = referenceChanges(diffUnits(f.repo, before, f.repo.snapshot(["paper"])), doc(files), doc({ ...files, "paper/strings.bib": "" }))!; + expect(removed.entries.map((e) => e.key)).toEqual(["one", "two"]); + }); + it.skipIf(!hasTeX)("agrees with a real BibTeX build for shared macros and literal citations", () => { + const main = String.raw`\documentclass{article} +\newcommand{\mycite}[1]{\cite{#1}} +\begin{document} +\verb|\cite{not-a-real-cite}| \mycite{real} +\bibliographystyle{plain} +\bibliography{strings,refs} +\end{document} +`, + files = { "paper/main.tex": main, "paper/strings.bib": '@string{work="Real title"}', + "paper/refs.bib": '@misc{real, title=work, author={John Smith}, year=2020}' }, + build = () => { + const run = (command: string, args: string[]) => execFileSync(command, args, { cwd: `${f.root}/paper`, stdio: "ignore" }); + run("pdflatex", ["-interaction=nonstopmode", "-halt-on-error", "main.tex"]); + run("bibtex", ["main"]); + for (let n = 0; n < 2; n++) run("pdflatex", ["-interaction=nonstopmode", "-halt-on-error", "main.tex"]); + expect(f.read("paper/main.log")).not.toMatch(/Citation .*undefined/); + return f.read("paper/main.bbl"); + }; + for (const [file, text] of Object.entries(files)) f.write(file, text); + expect(build()).toContain("Real title"); + const revised = { ...files, "paper/strings.bib": '@string{work="Invented title"}' }; + f.write("paper/strings.bib", revised["paper/strings.bib"]); + expect(build()).toContain("Invented title"); + const result = referenceChanges([unit({ file: "paper/strings.bib" })], doc(files), doc(revised))!; + expect(result.missing).toEqual([]); + expect(checkInputs(result)[0].fields.title).toBe("Invented title"); + expect(bibliographyReport(doc(revised)).missing).toEqual([]); + }); + it("checks macro changes and keeps each affected entry's edits together", () => { + const main = MAIN.replace("knuth1981", "one").replace("ellis1989", "two"), + bib = `@string{work = "Original title"} +@string{extended = work # " subtitle"} + +@article{one, + title = extended, + author = {John Smith}, + year = 2020 +} + +@article{unrelated, + title = {Other work}, + year = 2000 +} +${"\n".repeat(8)} +@article{two, + title = work, + author = {Jane Doe}, + year = 2021 +} +`; + f.write("paper/main.tex", main); + f.write("paper/refs.bib", bib); + f.repo.run(["add", "."]); + f.repo.run(["commit", "-qm", "macro bibliography"]); + const before = f.repo.rev("HEAD"), + next = bib.replace("Original title", "Changed title").replace("year = 2021", "year = 2022").replace("Other work", "Another work"); + f.write("paper/refs.bib", next); + const after = f.repo.snapshot(["paper"]), + units = diffUnits(f.repo, before, after), + result = referenceChanges(units, + doc({ "paper/main.tex": main, "paper/refs.bib": bib }), + doc({ "paper/main.tex": main, "paper/refs.bib": next }), + )!; + expect(units).toHaveLength(2); + const one = result.entries.find((e) => e.key === "one")!, + two = result.entries.find((e) => e.key === "two")!, + unrelated = result.entries.find((e) => e.key === "unrelated")!; + expect(one.unit).toBe(two.unit); + expect(unrelated.unit).not.toBe(one.unit); + expect(one.fields.find((f) => f.name === "title")).toEqual({ + name: "title", old: "extended", new: "extended", + oldResolved: "Original title subtitle", resolved: "Changed title subtitle", + }); + expect(checkInputs(result).find((e) => e.id === one.id)?.fields.title).toBe("Changed title subtitle"); + // Accepting the macro's change also accepts the dependent field edit, + // but leaves the unrelated entry alone, even though it sits between them. + const accepted = applyUnits(f.repo.blob(before, "paper/refs.bib"), f.repo.blob(after, "paper/refs.bib"), units.filter((u) => u.id === one.unit))!.toString(); + expect(accepted).toBe(next.replace("Another work", "Other work")); + // A macro edit alone still produces entries, not just a source-only diff. + const macroOnly = bib.replace("Original title", "Changed title"); + f.write("paper/refs.bib", macroOnly); + const macroUnits = diffUnits(f.repo, before, f.repo.snapshot(["paper"])); + const macroResult = referenceChanges(macroUnits, + doc({ "paper/main.tex": main, "paper/refs.bib": bib }), + doc({ "paper/main.tex": main, "paper/refs.bib": macroOnly }), + )!; + expect(macroResult.entries.map((e) => e.key)).toEqual(["one", "two"]); + expect(macroResult.entries.every((e) => e.unit === macroUnits[0].id)).toBe(true); + }); + it("makes one change of hunks that touch the same entry, so an entry is decided whole", () => { + const fields = Array.from({ length: 12 }, (_, i) => ` field${i} = {${i}},`).join("\n"), + bib = `@article{long,\n${fields}\n title = {Old}\n}\n\n@misc{other,\n title = {Other}\n}\n`; + f.write("paper/refs.bib", bib); + f.repo.run(["add", "."]); + f.repo.run(["commit", "-qm", "bib"]); + const head = f.repo.rev("HEAD"), + next = bib.replace("field0 = {0}", "field0 = {zero}").replace("title = {Old}", "title = {New}"); + f.write("paper/refs.bib", next); + const after = f.repo.snapshot(["paper"]), + units = diffUnits(f.repo, head, after); + expect(units).toHaveLength(1); + expect(units[0].ops.length).toBe(2); + expect(applyUnits(f.repo.blob(head, "paper/refs.bib"), f.repo.blob(after, "paper/refs.bib"), units)?.toString()).toBe(next); + // Distant edits of different entries stay separate. + const apart = bib.replace("field0 = {0}", "field0 = {zero}").replace("title = {Other}", "title = {Another}"); + f.write("paper/refs.bib", apart); + expect(diffUnits(f.repo, head, f.repo.snapshot(["paper"]))).toHaveLength(2); + }); +}); + +describe("The whole bibliography", () => { + it("resolves macros in declaration order, including resources declared around inputs", () => { + const files = { + "paper/main.tex": String.raw`\addbibresource{strings.bib}\input{section}\addbibresource{late.bib}`, + "paper/section.tex": String.raw`\addbibresource{refs.bib}`, + "paper/strings.bib": '@string{work="Earlier title"}', + "paper/refs.bib": '@misc{early,title=work}', + "paper/late.bib": '@string{work="Later title"}\n@misc{late,title=work}', + }; + expect(bibliographyReport(doc(files)).entries.map((e) => e.fields[0].resolved)).toEqual(["Earlier title", "Later title"]); + }); + it("lists every entry with its citations, and what is wrong without looking anything up", () => { + const bib = BIB + String.raw` +@misc{knuth1981, + title = {Again} +} +@article{plass81, + title = {Breaking paragraphs into lines}, + doi = {10.1002/spe.4380111102} +} +@book{other, + title = {Something else}, + doi = {https://doi.org/10.1002/SPE.4380111102} +} +`; + const report = bibliographyReport(doc({ "paper/main.tex": MAIN.replace("See \\cite{ellis1989}.", "See \\cite{ellis1989,ghost}."), "paper/refs.bib": bib })); + expect(report.bibs).toEqual(["paper/refs.bib"]); + expect(report.entries.map((e) => [e.id, e.cited.length])).toEqual([ + ["paper/refs.bib#knuth1981", 1], + ["paper/refs.bib#ellis1989", 1], + ["paper/refs.bib#knuth1981#2", 1], + ["paper/refs.bib#plass81", 0], + ["paper/refs.bib#other", 0], + ]); + expect(report.entries[1].cited).toEqual([{ file: "paper/main.tex", line: 6, section: "2 Related" }]); + expect(report.missing).toEqual([{ key: "ghost", file: "paper/main.tex", line: 6, section: "2 Related" }]); + expect(report.duplicates).toEqual([["paper/refs.bib#knuth1981", "paper/refs.bib#knuth1981#2"]]); + expect(report.similar).toEqual([ + { ids: ["paper/refs.bib#plass81", "paper/refs.bib#other"], by: "doi" }, + { ids: ["paper/refs.bib#knuth1981", "paper/refs.bib#plass81"], by: "title" }, + ]); + expect(report.all_cited).toBe(false); + }); + it("names .bib files that do not exist", () => { + const report = bibliographyReport(doc({ "paper/main.tex": MAIN })); + expect(report).toMatchObject({ bibs: ["paper/refs.bib"], absent: ["paper/refs.bib"], entries: [] }); + }); +}); diff --git a/vendor/latex-review/tests/ts/verify.test.ts b/vendor/latex-review/tests/ts/verify.test.ts new file mode 100644 index 0000000..37f6343 --- /dev/null +++ b/vendor/latex-review/tests/ts/verify.test.ts @@ -0,0 +1,286 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; +import { + arxivOf, + bibAuthors, + checkEntry, + checkInputs, + Confirmations, + doiOf, + Http, + keyOf, + plain, + ReferenceChecker, + titleScore, + type CheckInput, + type Fetch, +} from "../../src/core/verify.ts"; + +// Responses in the shapes doi.org, Crossref, DBLP and arXiv return them, +// cut down to the fields the check reads. +const KNUTH_CSL = { + type: "journal-article", + title: "Breaking paragraphs into lines", + author: [{ family: "Knuth", given: "Donald E." }, { family: "Plass", given: "Michael F." }], + issued: { "date-parts": [[1981, 11]] }, + "container-title": "Software: Practice and Experience", + DOI: "10.1002/spe.4380111102", +}; +const MYERS_CSL = { + type: "journal-article", + title: "AnO(ND) difference algorithm and its variations", + author: [{ family: "Myers", given: "Eugene W." }], + issued: { "date-parts": [[1986, 11]] }, + "container-title": ["Algorithmica"], + DOI: "10.1007/BF01840446", +}; +// Crossref answers a search for a famous preprint with later copies first. +const COPIES = { + message: { + items: [2025, 2025].map((year, i) => ({ + type: "posted-content", + title: ["Attention Is All You Need"], + author: [{ family: "Vaswani" }, { family: "Shazeer" }, { family: "Parmar" }], + issued: { "date-parts": [[year, 8, 23]] }, + DOI: `10.65215/copy${i}`, + })), + }, +}; +const ATTENTION_ATOM = ` + + + http://arxiv.org/abs/1706.03762v7 + Attention Is All You + Need + 2017-06-12T17:57:34Z + Ashish Vaswani + Noam Shazeer + Niki Parmar + +`; +const EMPTY_ATOM = `0`; +const BOT_PAGE = "Making sure you're not a bot!"; + +type Reply = { status?: number; type?: string; body: unknown } | "offline"; +function server(routes: [RegExp, Reply][]) { + const seen: string[] = []; + const fetcher: Fetch = async (url) => { + seen.push(url); + const reply = routes.find(([pattern]) => pattern.test(decodeURIComponent(url)))?.[1] ?? { status: 404, type: "text/html", body: "Not found" }; + if (reply === "offline") throw new TypeError("fetch failed"); + const body = typeof reply.body === "string" ? reply.body : JSON.stringify(reply.body); + return new Response(body, { + status: reply.status ?? 200, + headers: { "content-type": reply.type ?? (typeof reply.body === "string" ? "application/atom+xml" : "application/json") }, + }); + }; + return { fetcher, seen, http: new Http(fetcher, {}, Date.now, async () => {}) }; +} +const NO_PACE = { "doi.org": 0, "api.crossref.org": 0, "dblp.org": 0, "export.arxiv.org": 0 }; +const entry = (fields: Record, id = "refs.bib#x"): CheckInput => ({ id, type: "article", fields }); +const knuth = { + author: "Donald E. Knuth and Michael F. Plass", + title: "Breaking paragraphs into lines", + year: "1981", +}; + +describe("Reading entries for a check", () => { + it("turns LaTeX into plain text and reads names as BibTeX does", () => { + expect(plain(String.raw`J{\'e}r{\^o}me Laurens, Pregui{\c{c}}a, \v{S}ar\'{\i}k, {\TeX}book, \emph{An} {$O(ND)$} 1--2 \rm x`)).toBe( + "Jerome Laurens, Preguica, Sarik, TeXbook, An O(ND) 1-2 x", + ); + expect(bibAuthors(String.raw`Knuth, Donald E. and Ludwig van Beethoven and {World Health Organization} and others`)).toEqual({ + names: ["Donald E. Knuth", "Ludwig van Beethoven", "World Health Organization"], + families: ["knuth", "vanbeethoven", "worldhealthorganization"], + others: true, + }); + }); + it("finds DOIs and arXiv identifiers wherever entries keep them", () => { + expect(doiOf({ doi: "https://doi.org/10.1002/SPE.4380111102." })).toBe("10.1002/SPE.4380111102"); + expect(doiOf({ url: "https://dx.doi.org/10.1145/3290380" })).toBe("10.1145/3290380"); + expect(doiOf({ doi: "not a doi" })).toBeNull(); + expect(arxivOf({ eprint: "1706.03762v2", archiveprefix: "arXiv" })).toBe("1706.03762"); + expect(arxivOf({ journal: "arXiv preprint arXiv:2104.04692" })).toBe("2104.04692"); + expect(arxivOf({ url: "https://arxiv.org/pdf/hep-th/9711200v3" })).toBe("hep-th/9711200"); + expect(arxivOf({ doi: "10.48550/arXiv.1706.03762" })).toBe("1706.03762"); + expect(arxivOf({ eprint: "12345", eprinttype: "pubmed" })).toBeNull(); + }); + it("scores a title with other punctuation or a subtitle as the same", () => { + expect(titleScore("An {$O(ND)$} difference algorithm and its variations", "AnO(ND) difference algorithm and its variations")).toBe(1); + expect(titleScore("Conflict-free replicated data types", "Conflict-Free Replicated Data Types: an overview")).toBeGreaterThanOrEqual(0.9); + expect(titleScore("Deep residual learning", "Attention is all you need")).toBeLessThan(0.6); + // Words added to a real title, without a subtitle mark, make another title. + expect(titleScore("Attention is all you need for graphs", "Attention is all you need")).toBeLessThan(0.9); + expect(titleScore("Attention is not all you need", "Attention is all you need")).toBeLessThan(0.9); + }); +}); + +describe("Checking an entry", () => { + it("verifies an entry whose DOI names the same work", async () => { + const { http, seen } = server([[/doi\.org\/10\.1002\/spe/, { body: KNUTH_CSL }]]); + const check = await checkEntry(entry({ ...knuth, doi: "10.1002/spe.4380111102" }), http); + expect(check).toMatchObject({ status: "verified", via: "doi", differences: [], unavailable: [] }); + expect(check.record).toMatchObject({ source: "doi", year: 1981, authors: ["Donald E. Knuth", "Michael F. Plass"], url: "https://doi.org/10.1002/spe.4380111102" }); + expect(seen).toEqual(["https://doi.org/10.1002/spe.4380111102"]); + }); + it("calls an entry a mismatch when its DOI names another work", async () => { + const { http } = server([[/doi\.org/, { body: MYERS_CSL }]]); + const check = await checkEntry(entry({ ...knuth, doi: "10.1007/BF01840446" }), http); + expect(check.status).toBe("mismatch"); + expect(check.differences.map((d) => [d.field, d.minor ?? false])).toEqual([ + ["title", false], + ["author", false], + ["year", false], + ]); + }); + it("lists a wrong year and a missing author, and accepts reordered punctuation", async () => { + const { http } = server([[/doi\.org/, { body: KNUTH_CSL }]]); + const check = await checkEntry(entry({ author: "Donald E. Knuth", title: "Breaking Paragraphs Into Lines.", year: "1982", doi: "10.1002/spe.4380111102" }), http); + expect(check.status).toBe("mismatch"); + expect(check.differences).toEqual([ + { field: "author", bib: "Donald E. Knuth", record: "Donald E. Knuth; Michael F. Plass" }, + { field: "year", bib: "1982", record: "1981" }, + ]); + }); + it("reports a DOI that is not registered, with the work found by its title", async () => { + const { http } = server([ + [/doi\.org/, { status: 404, type: "text/html", body: "Error: DOI Not Found" }], + [/api\.crossref\.org\/works\?/, { body: { message: { items: [KNUTH_CSL] } } }], + ]); + const check = await checkEntry(entry({ ...knuth, doi: "10.1002/spe.999999" }), http); + expect(check).toMatchObject({ status: "mismatch", via: "title" }); + expect(check.differences).toEqual([{ field: "doi", bib: "10.1002/spe.999999", record: "10.1002/spe.4380111102" }]); + }); + it("finds a preprint on arXiv when Crossref has only later copies and DBLP turns clients away", async () => { + const { http, seen } = server([ + [/api\.crossref\.org/, { body: COPIES }], + [/dblp\.org/, { status: 202, type: "text/html", body: BOT_PAGE }], + [/export\.arxiv\.org.*search_query/, { body: ATTENTION_ATOM }], + ]); + const check = await checkEntry(entry({ author: "Ashish Vaswani and Noam Shazeer and Niki Parmar", title: "Attention is all you need", year: "2017" }), http); + expect(check).toMatchObject({ status: "verified", via: "title", unavailable: ["DBLP"] }); + expect(check.record).toMatchObject({ source: "arxiv", year: 2017, url: "https://arxiv.org/abs/1706.03762" }); + expect(seen.map((u) => new URL(u).hostname)).toEqual(["api.crossref.org", "dblp.org", "export.arxiv.org"]); + }); + it("accepts a conference year after the preprint's, and looks up arXiv identifiers directly", async () => { + const { http, seen } = server([[/export\.arxiv\.org.*id_list=1706\.03762/, { body: ATTENTION_ATOM }]]); + const check = await checkEntry(entry({ author: "Vaswani, Ashish and Shazeer, Noam and Parmar, Niki", title: "Attention is All you Need", year: "2018", eprint: "1706.03762", archiveprefix: "arXiv" }), http); + expect(check).toMatchObject({ status: "verified", via: "arxiv", differences: [{ field: "year", bib: "2018", record: "2017", minor: true }] }); + expect(seen).toHaveLength(1); + }); + it("leaves an entry no source knows unverified, with the closest record", async () => { + const { http } = server([ + [/api\.crossref\.org/, { body: { message: { items: [MYERS_CSL] } } }], + [/dblp\.org/, { body: { result: { hits: { hit: [{ info: { title: "Neural difference algorithms.", authors: { author: { text: "Ann Smith 0001" } }, year: "2021", venue: "NeurIPS", url: "https://dblp.org/rec/x" } }] } } } }], + [/export\.arxiv\.org/, { body: EMPTY_ATOM }], + ]); + const check = await checkEntry(entry({ author: "Ann Smith", title: "Neural difference algorithms and their variations", year: "2021" }), http); + // A real title with words added is not that work; the closest title is shown. + expect(check).toMatchObject({ status: "unverified", reason: "no-match", record: { source: "crossref", title: MYERS_CSL.title } }); + const none = await checkEntry(entry({ author: "Nobody", title: "A study that was never written", year: "2023" }), server([[/crossref/, { body: { message: { items: [] } } }], [/dblp/, { body: { result: { hits: {} } } }], [/arxiv/, { body: EMPTY_ATOM }]]).http); + expect(none).toMatchObject({ status: "unverified", reason: "no-match", differences: [] }); + expect(none.record).toBeUndefined(); + }); + it("does not take a review of a book for the book, and reads titles with markup and Greek letters", async () => { + const reviewOf = { message: { items: [{ type: "journal-article", title: ["Queueing Systems Volume 1: Theory."], author: [{ family: "Butterworth" }, { family: "Kleinrock" }], issued: { "date-parts": [[1976]] } }] } }; + const { http } = server([[/crossref/, { body: reviewOf }], [/dblp/, { body: { result: { hits: {} } } }], [/arxiv/, { body: EMPTY_ATOM }]]); + const book = await checkEntry({ id: "k", type: "book", fields: { author: "Leonard Kleinrock", title: "Queueing Systems, Volume 1: Theory", year: "1975" } }, http); + expect(book).toMatchObject({ status: "unverified", reason: "no-match" }); + expect(book.record).toBeUndefined(); + expect(titleScore(String.raw`A proof for the queuing formula: {$L = \lambda W$}`, "A Proof for the Queuing Formula: L = λ W")).toBe(1); + }); + it("fails, rather than doubting the entry, when no source can be reached, and skips entries with nothing to look up", async () => { + const { http } = server([[/./, "offline"]]); + const check = await checkEntry(entry(knuth), http); + expect(check).toMatchObject({ status: "failed", unavailable: ["Crossref", "DBLP", "arXiv"] }); + expect(await checkEntry(entry({ howpublished: "\\url{https://example.org}" }), http)).toMatchObject({ status: "skipped", reason: "nothing-to-check" }); + }); +}); + +describe("The checker", () => { + it("checks again when the entry type changes instead of reusing an article result for a book", async () => { + const { fetcher, seen } = server([ + [/crossref/, { body: { message: { items: [KNUTH_CSL] } } }], + [/dblp/, { body: { result: { hits: {} } } }], + [/arxiv/, { body: EMPTY_ATOM }], + ]), checker = new ReferenceChecker(null, { fetch: fetcher, pace: NO_PACE }), + article = entry(knuth), book = { ...article, type: "book" }; + checker.lookup([article], true); + await checker.idle(); + expect(checker.lookup([article]).checks[article.id].status).toBe("verified"); + const requests = seen.length; + expect(checker.lookup([book], true)).toEqual({ checks: {}, pending: 1 }); + await checker.idle(); + expect(checker.lookup([book]).checks[book.id].status).toBe("unverified"); + expect(seen.length).toBeGreaterThan(requests); + }); + it("discards cached verdicts written before entry types were part of the key", async () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "latex-review-cache-version-")); + try { + const file = path.join(dir, "references.json"), input = entry(knuth); + fs.writeFileSync(file, JSON.stringify({ version: 1, checks: { + [keyOf(input)]: { status: "verified", checked: Math.floor(Date.now() / 1000) }, + } })); + expect(new ReferenceChecker(file).lookup([input])).toEqual({ checks: {}, pending: 0 }); + } finally { fs.rmSync(dir, { recursive: true, force: true }); } + }); + it("checks in the background, keeps results in a file and redoes unverified entries on request", async () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "latex-review-verify-")), + file = path.join(dir, "references.json"), + { fetcher, seen } = server([[/doi\.org/, { body: KNUTH_CSL }], [/crossref/, { body: { message: { items: [] } } }], [/dblp/, { body: { result: { hits: {} } } }], [/arxiv/, { body: EMPTY_ATOM }]]), + inputs = [entry({ ...knuth, doi: "10.1002/spe.4380111102" }, "a"), entry({ title: "Unknown work" }, "b")]; + try { + const checker = new ReferenceChecker(file, { fetch: fetcher, pace: NO_PACE }); + expect(checker.lookup(inputs)).toEqual({ checks: {}, pending: 0 }); + expect(checker.lookup(inputs, true)).toEqual({ checks: {}, pending: 2 }); + expect(checker.lookup(inputs, true).pending).toBe(2); + await checker.idle(); + const done = checker.lookup(inputs, true); + expect(done.pending).toBe(0); + expect(done.checks.a.status).toBe("verified"); + expect(done.checks.b.status).toBe("unverified"); + const requests = seen.length; + // Another process reads the saved results instead of asking again. + const later = new ReferenceChecker(file, { fetch: fetcher, pace: NO_PACE }); + expect(later.lookup(inputs, true).pending).toBe(0); + expect(seen.length).toBe(requests); + expect(later.lookup(inputs, true, true)).toMatchObject({ pending: 1, checks: { a: { status: "verified" } } }); + await later.idle(); + expect(seen.length).toBeGreaterThan(requests); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + }); + it("checks the added and modified entries of a round with their new values", () => { + expect( + checkInputs({ + missing: [], + entries: [ + { id: "r#a", unit: "c001", file: "r", key: "a", type: "misc", status: "added", lines: [1, 3], cited: [], problems: [], fields: [{ name: "title", old: null, new: "acm # x", resolved: "ACM x" }, { name: "note", old: "n", new: null }] }, + { id: "r#b", unit: "c001", file: "r", key: "b", type: "misc", status: "removed", lines: [4, 5], cited: [], problems: [], fields: [] }, + ], + }), + ).toEqual([{ id: "r#a", type: "misc", fields: { title: "ACM x" } }]); + }); +}); + +describe("Confirmations", () => { + it("hold for an entry's looked-up fields, wherever the entry is, until they change or are taken back", () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), "latex-review-confirm-")), + file = path.join(dir, "latex-review", "references-confirmed.json"); + try { + const book = entry({ author: "Donald E. Knuth", title: "The TeXbook", year: "1984" }, "refs.bib#knuth"); + new Confirmations(file).set(book, true, 5000); + const later = new Confirmations(file); + expect(later.of([book, { ...book, id: "other.bib#tb" }])).toEqual({ "refs.bib#knuth": 5, "other.bib#tb": 5 }); + expect(later.of([{ ...book, fields: { ...book.fields, year: "1986" } }])).toEqual({}); + expect(later.of([{ ...book, type: "book" }])).toEqual({}); + later.set(book, false); + expect(new Confirmations(file).of([book])).toEqual({}); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + }); +});