Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,68 @@ inside the page that had just failed and span on one offset until it was killed.
guessed into something a consumer will trust. There is a test asserting that
no unreadable value is ever returned in ISO shape.

**Take this one if you need point data out of any Indian state's GeoServer.**
It adds the module, and it documents the trap that makes the naive version of
that job silently return a fraction of the layer.

### Added

- **`commoner_probe.otp_download_portal` — bulk microdata from a portal that
gates it behind a mobile OTP.** The Ministry of Education's UDISE+ Data
Sharing Portal serves six CSV datasets for each academic year since 2018-19.
The module records the whole route, because each step has a trap that returns
a plausible wrong answer instead of an error.

**The expensive one is the all-India sentinel: it is 99, not 0.**
`stateId=0&districtId=0` answers HTTP 200 with `Content-Type: application/zip`
and a body that is actually `%PDF-1.7` — the schema document, not data. Every
`reportId` except 1 then 404s, which reads convincingly as "only one report
exists". Always check the payload begins `PK`. Never trust the header.

Also recorded: the portal times out from a non-Indian connection and answers
from ap-south-1; the API base sits in a 2.25 MB Angular bundle as
`Y3_apiBaseUrl`, assembled with template literals, so grepping for URL string
literals finds almost nothing; the auth flow is captcha, send-OTP, verify-OTP,
and the captcha needs a human, so this module ships no solver.

- **`commoner_probe.geoserver` — point extraction from a WMS-only GeoServer.**
State spatial-data infrastructures are GeoServer deployments and many publish
WMS while disabling WFS, so there is no vector download. This sweeps a bounding
box with `GetFeatureInfo`, subdividing wherever the response hits the feature
cap.

**The trap it exists to defeat.** `GetFeatureInfo` does not query the data — it
hit-tests the *rendered symbol* under the pixel, using the server's default
style. Where that style draws a small marker, a query only returns a feature
when it lands inside those few pixels. Measured against Andhra Pradesh's APSAC
school layer: the default style yielded **19,090** schools; the same sweep with
a 200-pixel symbol via `SLD_BODY` yielded **58,301** — 3.05x more. The first
number carried no error, no warning and no missing-data indicator. Use
`big_symbol_sld()`, and treat any extraction that did not override the style as
a lower bound of unknown tightness.

- `wfs_status()` — **call this first.** If WFS is enabled, use it and ignore
this module: it returns real geometry, including the lines and polygons WMS
extraction cannot honestly recover. APSAC answers every version with
`org.geoserver.platform.ServiceException: Service WFS is disabled`.
- `big_symbol_sld()` **refuses non-point geometry.** A road line cannot be
recovered by hit-testing symbols; returning a style for it would invite a
caller to sweep a road layer and believe the result.
- `Tile.offset()` — the verification pass. Re-running an identical grid asks
the same questions and "confirms" anything; an offset grid interrogates the
ground between the original query points. On the APSAC school layer this
returned 58,301 against 58,301 with zero new features, which is what turns a
floor into a count.
- **One failing tile no longer zeroes a layer.** A single bad tile used to
raise out of the sweep, and the run recorded "0 rows" — indistinguishable in
a results table from "this layer is empty", which is the expensive kind of
wrong. Failed tiles are collected and the sweep reports itself PARTIAL.
- Deduplication across workspaces is deliberately NOT done: state portals
republish one dataset under several workspaces, and agreement between two
independently-swept copies is the best completeness check available when no
authoritative count exists. APSAC's anganwadi layer returns 53,682 under both
`gatishakti:` and `Andhra-`.

## 0.14.6 (2026-08-14)

**Take this one if you enumerated any Lok Sabha session before August 2015 with
Expand Down
Loading
Loading