fix: give the degrading paginator a caller, and check what the answer says it is - #130
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 37f92eab0d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # No PDF landed, so the inline answer the RS API served is all there is | ||
| # to check — and inline text carries the same defect. | ||
| self.verify_document_qno(rec) |
There was a problem hiding this comment.
When an RS row has no files URL, _rs_attach_pdf returns at line 524 before this new inline fallback; the fallback is also never reached under --no-download because the caller invokes _rs_attach_pdf only when download is true. In either case the API's available ans_text can contain a mismatched question number, but the emitted record lacks document_qno and document_qno_status; run inline verification whenever no PDF was actually read.
Useful? React with 👍 / 👎.
| if self.write_pdf(rec["pdf_url"], pdf_path, PDF_HEADERS): | ||
| rec["pdf_path"] = str(pdf_path.relative_to(self.out_dir)) | ||
| self.verify_document_qno(rec, pdf_path=pdf_path) |
There was a problem hiding this comment.
When download is enabled and a row has pdf_url, a failed write_pdf bypasses both this verification call and the outer else at line 1627. Historical LS rows can carry answer_text alongside questionsFilePath, so a transient or permanent PDF download failure causes an otherwise checkable mismatch to be emitted without any verdict; verify the inline answer whenever the PDF was not successfully written and read.
Useful? React with 👍 / 👎.
37f92ea to
bcaaa3c
Compare
bcaaa3c to
fc241b9
Compare
… says it is The page-size degradation two releases built had no production caller. The CLI enumeration ran its own loop, so the degrade, the floor retry, the skip and the climb-back reached nothing a user could invoke. Searching 630 portal shapes for the reported misalignment found none. It found two other defects: - a session that answers every page with a 5xx walked forever. An empty page is the only end-of-data signal and a skipped page makes no statement, so nothing stopped it: measured, it passed 4,000 requests and was still going. - a degrade re-served 25 rows, because a coarser page's boundary can lie behind the rows already yielded. Downstream dedupe by key hid it. After the fix: zero lost, zero duplicated, zero runaway across all 630 shapes. Also: sansad.in serves the wrong document under the right URL, live again today (AU2549 prints QUESTION NO. 2594, AU2594 prints 2549, both md5s unchanged since 2026-08-05). Records now carry document_qno and document_qno_status. The check reads inline answer text as well as attached PDFs, because the proven inline cases have no PDF at all. It costs a median 18.9 ms over 50 live answer PDFs, so it runs unconditionally. And answer_text_hindi now survives the typed reader: the raw manifest carried it, the schema never declared it, the dataclass never named it, and validate passed. Silent at all three layers.
fc241b9 to
7b1383d
Compare
1 participant
Two review findings on merged PRs, and two cross-repo requests.
The paginator had no caller
paginate_ls_question_list— the degrade, the floor retry, the skip and theclimb-back that 0.14.8 and 0.14.9 built — was reachable only from tests.
probe_ls_sessions, the path the CLI takes, ran its own page loop. Those tworeleases fixed a walk no user could invoke.
Searching 630 portal shapes for the reported misalignment found none. It found
two other defects:
After the fix, across all 630 shapes: zero lost, zero duplicated, zero runaway.
A session with a hole now leaves its window suspect and names the skipped
offsets, instead of reporting a total that quietly omits them.
The answer is checked against the number it prints
sansad.in serves the wrong document under the right URL. Re-fetched live during
this work, and unchanged since it was first recorded:
Re-fetching cannot repair a source-side swap, and downstream the record is
flawless — the key parses, the subject is right, the text is a real reply. So
records carry
document_qnoanddocument_qno_status, stamped where therequested number and the document are both in hand.
unreadable, nevermismatch—4.7% of LS answer PDFs print none, and a check that cries wolf gets disabled.
download is harder to notice than a flagged one.
The reading is copied from zero-hour's
corpus_integrity, with its citationhandling and both OCR traps, rather than re-derived one bug at a time.
answer_text_hindireached no typed consumerThe raw manifest carried it, the schema never declared it,
ManifestQaRecordnever named it, and
validatepassed because the schema sets noadditionalPropertiesbar. Silent at all three layers.Verification
1,300 passed, ruff clean.
test_pyproject_version_matches_package_versionfails here and on untouched
master: the venv is an editable install of adifferent worktree. Environmental, not from this branch.