feat: drive ASP.NET WebForms portals, and read CDN-backed SPA dashboards - #131
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3a3021f2c0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| def __init__(self, session: Any | None = None, rate_limit_sec: float = 1.0, | ||
| report_url: str = BIHAR) -> None: | ||
| self.session = session or make_session(rate_limit_sec=rate_limit_sec) |
There was a problem hiding this comment.
When Aangan() is constructed without an injected session, this creates a session with the package's commoner-probe/... User-Agent, even though this commit documents that the default Bihar deployment returns HTTP 500 for that identity. Consequently the constructor's initial GET fails before any cascade can be used; expose a User-Agent argument or deliberately construct the default client with the supported identity.
Useful? React with 👍 / 👎.
| def reset(self) -> None: | ||
| """Drop selections and refetch. Used to recover a poisoned session.""" | ||
| self._selected.clear() | ||
| self.page = self.session.get(self.report_url).content.decode("utf8", "replace") |
There was a problem hiding this comment.
In the stale-session scenario this recovery path is designed for, reset() reuses self.session and therefore preserves the poisoned session cookie. The module states that every request continues returning 500 until a fresh session is created, so the subsequent district re-selection will fail again and long crawls cannot recover after the first stale session; reseating must construct and assign a new session before refetching.
Useful? React with 👍 / 👎.
| verified rather than ignored: a silently-wrong key would otherwise surface | ||
| as a JSON parse error somewhere much further downstream. | ||
| """ | ||
| from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes |
There was a problem hiding this comment.
On a clean package installation, decrypt_envelope() fails here with ModuleNotFoundError because pyproject.toml declares no runtime dependencies and neither the development nor all extras include cryptography. This also makes places() unusable, since all reference data passes through this function; the dependency needs to be declared in an appropriate install extra or replaced with an available implementation.
Useful? React with 👍 / 👎.
| if resp.status_code in (403, 404): | ||
| return None |
There was a problem hiding this comment.
When harvesting outside an Indian egress region, CloudFront returns the geo-fence 403 described by this module, but this branch converts every such response to None. harvest() then skips every endpoint and emits no rows, making access denial look like a valid empty dataset; inspect the known geo-fence response and raise it rather than treating all 403s as unpublished periods.
Useful? React with 👍 / 👎.
| return [(n, v) for n, v in submit_buttons(page) | ||
| if any(h in (n + " " + v).lower() for h in WRITE_HINTS)] |
There was a problem hiding this comment.
For a Hindi-labelled submit control whose name is neutral (for example Button1), ASP.NET commonly emits the visible value as numeric HTML entities, so none of the literal Hindi entries in WRITE_HINTS matches here. The new test fixture itself uses this encoding but passes only because its name contains BtnSave; decoding v before matching is necessary or a crawler relying on this safety check can misclassify a live write control as harmless.
Useful? React with 👍 / 👎.
3a3021f to
4b9342a
Compare
skishchampi
changed the title
f9c60a3 to
cd4be4f
Compare
skishchampi
changed the title
cd4be4f to
9b0046d
Compare
9b0046d to
ce53d89
Compare
ce53d89 to
974fb5a
Compare
Two access patterns that recur across Indian government data sites, named for
the mechanics a caller must implement rather than for the scheme whose numbers
come out the other end. The scheme, the department and the host are context and
live in the comments.
aspnet.py WebForms per-request rules: the hidden state tokens, the
registered-value requirement behind an unexplained 500,
AJAX detection by registered panels rather than by script
presence, write-button refusal, and 500 diagnosis
aspnet_cascade.py what spans requests: walking a chain of dependent
dropdowns one postback at a time, resuming it, rebuilding
a session whose state expired, surviving a dead branch
gridview.py reading and auditing the rendered tables
cdn_dashboard.py a client-rendered dashboard whose figures are static CDN
JSON keyed by period and place, with a second transport —
an AES-256-CBC enveloped API — for the reference ids
Each failure these document returns a full, plausible response rather than an
error: a 500 for an unregistered dropdown value; a 200 whose second dropdown is
empty because two levels were set in one POST; a table whose header was not row
0, mislabelling every column while the totals still add up; a 403 that means
the wrong id system or the wrong country, not a missing period.
aspnet_cascade takes a control map and a level order, so a new portal is
configuration rather than a subclass, and cdn_dashboard gathers its instance in
one dict. The two worked instances — Bihar's ICDS "Aangan" WebForms deployment
and the MWCD Anganwadi dashboard — are data, with their hosts, vintage and
provenance beside them.
No key ships in the package. The dashboard's envelope key is in its own JS
bundle because the browser must hold it, but that does not make it ours to
publish, and it rotates. It comes from POSHAN_ENVELOPE_KEY, a missing one says
so by name, and a test fails if a key literal ever reappears in the module.
Writing the missing cascade tests found a real defect in both form builders:
they took the first <option> while the note above each said "never empty", so
on a page whose placeholder carries value="" they posted exactly the empty
value the note warns about. Only a truly empty value is skipped now — the
placeholder on these pages is value="0", which the server registered.
974fb5a to
96396ec
Compare
1 participant
Four modules named for the mechanics a caller must implement. The scheme, the
department and the host are context; they live in the comments.
aspnet.pyaspnet_cascade.pygridview.pycdn_dashboard.pyWhy these need writing down
Every failure they document returns a full, plausible response instead of an
error:
message. The usual cause is posting
""for the controls you are notchanging.
set in one POST. Identical to a working request that found no data.
which backoff and jitter cannot fix because the fault is not in the transport.
totals still add up.
codes beside them in the same record) or the wrong country (a geo-fenced
edge), never a missing period.
/6/, never/06/, which 403s.submits it.
Structure
aspnet_cascadetakes a control map and a level order, so a new portal isconfiguration rather than a subclass.
cdn_dashboardgathers its instance intoone dict. The two worked instances — a state ICDS WebForms deployment and the
MWCD Anganwadi dashboard — are data, with host, vintage and provenance beside
them.
No key ships in the package
The dashboard's envelope key was a literal. The site carries it in its own JS
bundle because the browser must hold it, but that does not make it ours to
publish, and it rotates. It now comes from
POSHAN_ENVELOPE_KEY, a missing oneraises by name rather than surfacing as a JSON parse error further down, and a
test fails if a key literal reappears. The literal is in no pushed commit.
Writing the missing tests found a defect
The cascade module had none. Adding them caught
form_fieldsand_formtaking the first
<option>while the note above each said "never empty" — on apage whose placeholder carries
value="", that posts exactly the empty valuethe note warns about. Only a truly empty value is skipped now; the placeholder
on these pages is
value="0", which the server registered and accepts, soskipping it would leave an un-populated select with nothing to send.
A blanket rename in the same pass rewrote a live host into one that does not
exist — visible only as a connection error at crawl time. A test now asserts it.
Verification
1,324 passed, ruff clean. The cascade walk was verified against the portal's
own report: one district enumerated to 2,806 centres, matching its
DstWiseTotAWCfigure exactly — two independent paths through the same systemagreeing.
test_pyproject_version_matches_package_versionfails here and on untouchedmaster: the venv is an editable install of a different worktree.Environmental.