feat: extract points from a WMS-only GeoServer, and reach OTP-gated microdata - #134
Conversation
…naive way is wrong Release 0.15.0. Adds commoner_probe.geoserver, for the case that most Indian state spatial-data infrastructures present: a GeoServer publishing WMS with WFS switched off, so there is no vector download and the only way to data is through GetFeatureInfo. The module exists mainly to defeat one trap. GetFeatureInfo does not query the data — it hit-tests the rendered symbol under the pixel you name, using whatever style the server has set as default. Where that style draws a small point marker, a query returns a feature only when it lands inside those few pixels, and the sweep quietly returns a fraction of the layer while reporting nothing wrong. Measured against Andhra Pradesh's APSAC school layer: the default style yielded 19,090 schools and the same sweep with a 200-pixel symbol via SLD_BODY yielded 58,301. Three times as many, with no error, no warning and no missing-data indicator on the first run. Every rate computed on that number would have been wrong with it. wfs_status() is meant to be called first: if WFS is enabled, use it and ignore this module, because it returns real geometry including the lines and polygons WMS extraction cannot honestly recover. big_symbol_sld() refuses non-point geometry for the same reason — a road line cannot be reconstructed from point hit-tests, and returning a style for one would invite a caller to sweep a road layer and believe what came back. Tile.offset() is the completeness test. Re-running an identical grid re-asks the same questions and would confirm any systematic miss; a grid offset by half a tile interrogates the ground between the original query points. On the APSAC school layer that returned 58,301 against 58,301 with zero new features, which is what turns a floor into a count. A capped response subdivides rather than being believed: reaching FEATURE_COUNT means "there are more here", never "there are this many". And a single failing tile no longer empties a layer — it used to raise out of the sweep and leave a run recording "0 rows", which in a results table cannot be distinguished from "this layer is empty". Deduplication across workspaces is deliberately absent. State portals republish one dataset under several workspaces, and agreement between two independently swept copies is the best completeness check available when no authoritative count exists; APSAC's anganwadi layer returns 53,682 under both gatishakti: and Andhra-. Verified against the live server as well as offline: sweeps returning 178 and 197 features on two layers where an independent extractor returned 178 and 197. Full suite 1,302 passed.
…aps on it commoner_probe.udise records how to get eight years of UDISE+ microdata out of the Data Sharing Portal — six datasets a year, 2018-19 to 2025-26 — because every step of that route has a trap that returns a plausible wrong answer rather than an error, and rediscovering them costs an afternoon each. The expensive one is the all-India sentinel. stateId=0 returns HTTP 200 with Content-Type: application/zip and a body that is actually a PDF: the schema document, not data. Every reportId except 1 then 404s, which reads convincingly as "only one report exists". With stateId=99 the same URL returns the real 30-70 MB archive and reportIds 2 through 7 all work. Nothing anywhere says 99. So the module carries ALL_INDIA=99 as a named constant, csv_url() refuses an unknown reportId rather than silently handing back the schema, and the docstring says to check the magic bytes are PK and never to trust the Content-Type. The others: the portal times out from non-Indian egress and answers 200 from ap-south-1, so a blanket timeout is an egress fact and not an outage; the API base is compiled into a single 2.25 MB Angular bundle as a constant with endpoints assembled from template literals, so grepping for quoted paths finds nothing and the search that works is for the interpolations; the district select is disabled by design when All States is chosen; and the per-year schema PDFs are only two distinct documents, so for six of the eight years the schema describes fields the data does not contain. Authentication is mobile OTP and the module deliberately makes it awkward to automate: request_otp() requires a caller-supplied solve() and ships no captcha solver, because the captcha is a control the portal is entitled to have. The OTP goes to the account holder's phone. verify_otp() carries the warning that a shell-quoting slip sends an empty mobile, at which point the portal answers mobile_invalid_strict rather than "expired" and the real OTP burns while the quoting is fixed. Also recorded: the terms accepted at download time forbid redistributing the data without consent and require the source to be acknowledged, which constrains what can go into an open data deposit; and the portal publishes a pseudocode rather than the real UDISE code and keys geography on village NAMES, so these rows answer "which schools are in village X" and cannot be joined to a school-level GIS layer. Full suite passes 1,302 with tests/test_sansad_pagination_degrade.py excluded. That file is untracked and imports _halve_to_multiple, which does not exist in commoner_probe.sansad — pre-existing in-flight work in this tree, unrelated to this change.
…inel The module landed as udise.py, one day before the naming sweep. UDISE+ is a programme of the Ministry of Education. The name told a developer nothing about what to implement. It is now otp_download_portal.py. The mechanism is a bulk download gated behind a mobile OTP. The docstring names the ministry, the programme, both hosts and the captcha constraint. No shim. The module has never been released, so no consumer can import the old name. It also had no tests. It has eight now, and they cover the trap that costs an afternoon: the all-India sentinel is 99, and 0 returns a schema PDF inside a 200 response labelled application/zip. The changelog now records the module, the sentinel, the India-egress requirement, the Angular-bundle API base and the OTP flow.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8d4b52944b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| r = sess.post(base + DSP_ENDPOINTS["send_otp"], | ||
| json={"mobile": mobile, "captcha": value, "captchaKey": key}, | ||
| timeout=timeout) |
There was a problem hiding this comment.
In a default installation without the optional requests dependency, make_session() returns StdlibSession, whose POST implementation reads only the data keyword and silently ignores json. Both this request and verify_otp() therefore send empty bodies, making the OTP flow unusable unless callers happen to install the HTTP extra or inject another session. Encode the JSON and set its content type, or add JSON support to the fallback client.
Useful? React with 👍 / 👎.
| shifted = Tile(west, south, east, north).offset(0.5) | ||
| got = self.sweep(layer, (shifted.west, shifted.south, shifted.east, shifted.north), |
There was a problem hiding this comment.
When the requested bounding box is larger than one grid cell, this shifts by half the full region rather than half of start_span; for example, a 4-degree box with 2-degree cells moves 2 degrees instead of 1. The verification pass consequently queries far outside the original region while leaving its leading portion untested, so it can report saturated=True even when the first pass systematically missed features there.
Useful? React with 👍 / 👎.
| if len(feats) >= self.feature_count and tile.span > min_span: | ||
| queue.extend(tile.quarter()) | ||
| continue |
There was a problem hiding this comment.
If a dense tile still returns feature_count features after reaching min_span, this condition falls through and accepts the capped response as complete. Because hitting the cap explicitly means that additional features may have been omitted, dense clusters are silently truncated without a failure or PARTIAL indication. Continue subdividing or record the leaf as incomplete instead of ingesting it as a complete result.
Useful? React with 👍 / 👎.
| "new": len(new), | ||
| "recall": (len(known & set(got)) / len(known)) if known else 0.0, | ||
| "saturated": not new, |
There was a problem hiding this comment.
sweep() tolerates tile errors by default and returns only the successful features, without exposing its failure list to verify(). If every offset request fails, got and new are empty and this still reports saturated=True; even one failed tile leaves a gap in the completeness check. Propagate partial status from the sweep and require zero failed or capped tiles before declaring saturation.
Useful? React with 👍 / 👎.
| for f in feats: | ||
| props = f.get("properties", {}) or {} | ||
| ident = str(props.get(key)) if key else str(f.get("id")) | ||
| if ident and ident not in ("None", ""): | ||
| found[ident] = props |
There was a problem hiding this comment.
For GeoJSON features whose coordinates exist only in the standard geometry member, this assignment stores only properties and discards the point location. The main recursive extraction API then returns identifiers and attributes but no coordinates, despite having fetched them and being presented as a point extractor. Preserve the feature or copy its geometry into the stored result.
Useful? React with 👍 / 👎.
1 participant
Two acquisition capabilities, plus the traps that make the naive version of each
job return a plausible wrong answer.
geoserver— point extraction where WFS is disabledState spatial-data infrastructures are GeoServer deployments. Many publish WMS
and disable WFS, so no vector download exists. This sweeps a bounding box with
GetFeatureInfoand subdivides wherever a response hits the feature cap.The trap it exists to defeat.
GetFeatureInfodoes not query the data. Ithit-tests the rendered symbol under the pixel, using the server's default
style. Measured against one state's school layer: the default style returned
19,090 schools. The same sweep with a 200-pixel symbol via
SLD_BODYreturned 58,301. That is 3.05x, with no error, no warning and no
missing-data indicator either time.
wfs_status()— call this first. If WFS works, use it and ignore this module.big_symbol_sld()refuses non-point geometry. A road line cannot be recoveredby hit-testing symbols, and returning a style for it would invite the attempt.
Tile.offset()— the verification pass. Re-running an identical grid asks thesame questions and confirms any systematic miss.
otp_download_portal— bulk microdata behind a mobile OTPSix CSV datasets for each academic year since 2018-19, from the Ministry of
Education's Data Sharing Portal.
The expensive trap is the all-India sentinel: 99, not 0.
stateId=0&districtId=0answers HTTP 200 withContent-Type: application/zipand a body that is actually
%PDF-1.7. EveryreportIdexcept 1 then 404s,which reads convincingly as "only one report exists". Check the payload begins
PK; never trust the header.Also recorded: the portal times out from a non-Indian connection; the API base
sits in a 2.25 MB Angular bundle as
Y3_apiBaseUrland is assembled withtemplate literals, so grepping for URL literals finds nothing; the auth flow is
captcha, send-OTP, verify-OTP.
A human reads the captcha. This module ships no solver, and a test fails if
one appears.
Naming and tests
The module landed as
udise.py, one day before the naming sweep. It is nowotp_download_portal.py. No shim: it has never been released, so no consumercan import the old name.
It also had no tests. It has eight now, aimed at the sentinel and the OTP flow.
Verification
1,428 passed, ruff clean. Rebased onto master after #130 to #133.