Skip to content

docs: update stale repo names in ARCHITECTURE.md #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
skishchampi merged 3 commits into from
Aug 10, 2026
Merged

docs: update stale repo names in ARCHITECTURE.md #39

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
19c2510
docs: update stale repo names in ARCHITECTURE.md
skishchampi Aug 8, 2026
92b8b5f
Merge branch 'main' into docs/stale-repo-names
skishchampi Aug 9, 2026
1773ab1
docs: catch the short-form sansad-crawler reference Codex flagged
skishchampi Aug 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 23 additions & 14 deletions docs/ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,16 @@
> CommonerLLP repo that touches corpora, search, crawling, or MCP
> serving. Not optional. Read before scoping any new feature.
>
> **Last updated: 2026-06-24**
> **Last updated: 2026-08-08** (repo names only)
>
> **Naming note (2026-08-08):** repo names in this doc were updated for two
> renames that happened after it was written — `budget-crawler` →
> `public-finance` (2026-06-20) and `sansad-semantic-crawler` →
> `commoner-analyse` (2026-07-06, v2.1.0). The layer diagrams otherwise
> predate the extraction of acquisition into `commoner-probe`: where this doc
> describes the analysis repo as owning the crawl library, that role now
> belongs to `commoner-probe`. For current org topology, `_org/architecture.md`
> is canonical.

---

Expand Down Expand Up @@ -92,8 +101,8 @@ You write ~200 lines. You get a production search stack.
| `CalibreAdapter` | Calibre library | In progress | `partial-recall` |
| `MarkdownNotesAdapter` | Markdown note directories | In progress | `partial-recall` |
| `CADAdapter` | Constituent Assembly Debates | **Planned** | `cad-mcp` |
| `SansadAdapter` | Lok Sabha Q&A | **Planned** | `sansad-semantic-crawler` |
| `BudgetAdapter` | OBI / state budget documents | **Planned** | `budget-crawler` |
| `SansadAdapter` | Lok Sabha Q&A | **Planned** | `commoner-analyse` |
Comment thread
skishchampi marked this conversation as resolved.
| `BudgetAdapter` | OBI / state budget documents | **Planned** | `public-finance` |

---

Expand Down Expand Up @@ -148,13 +157,13 @@ as Item metadata.
## The org topology

```
Layer 0 sansad-semantic-crawler shared crawl library, parliament sources
Layer 0 commoner-analyse shared crawl library, parliament sources
↓ pip install (pinned)
Layer 1 partial-recall engine: chunk, embed, index, search, serve
↓ CorpusAdapter instances
Layer 2 cad-mcp CAD corpus + session/context tools
sansad (as corpus) parliament Q&A corpus
budget-crawler OBI/budget corpus + forensic tools
public-finance OBI/budget corpus + forensic tools
↓ search results
Layer 3 theright2read public dashboard
academiaindia public dashboard
Expand All @@ -166,7 +175,7 @@ Full topology with capability registry: `_org/architecture.md`.

## History (for orientation, not re-litigation)

`sansad-semantic-crawler` was built first and discovered the need for
`commoner-analyse` was built first and discovered the need for
corpus search through practice. `partial-recall` was extracted from that
experience as the correct shared solution — a deliberate DRY move.

Expand Down Expand Up @@ -198,13 +207,13 @@ Repeating that extraction in every new domain repo is the wrong move.
║ ├── SPEAKER_RE.finditer(text) → (speaker, body) pairs ║
║ └── sqlite3.connect(cad.db) → sessions + speeches tables ║
║ ║
║ sansad-semantic-crawler ║
║ commoner-analyse ║
║ ├── http_client.StdlibSession urllib, no cache ║
║ ├── crawl() GET sansad.in/… ║
║ ├── parse_qa() → Question, Answer objects ║
║ └── classify(topic_profile) regex|embedding|llm|ensemble ║
║ ║
║ budget-crawler/budget_crawler/ ║
║ public-finance/publicfinance/ ║
║ ├── scrapping_utils.ScrappingUtils requests + HTTPAdapter retry ║
║ ├── obi_utils.OBIUtils (duplicate of above — debt) ║
║ ├── rbi_fetcher.fetch() GET RBI state finance PDFs ║
Expand All @@ -225,7 +234,7 @@ Repeating that extraction in every new domain repo is the wrong move.
║ ├── ZoteroAdapter.list_items() reads ~/Zotero/zotero.sqlite (RO) ║
║ ├── FolderAdapter.list_items() walks directory, reads files ║
║ ├── [CADAdapter] PLANNED — lives in cad-mcp ║
║ └── [SansadAdapter] PLANNED — lives in sansad-crawler ║
║ └── [SansadAdapter] PLANNED — lives in commoner-analyse║
║ ║
║ extract/pdf.py ║
║ └── extract_pdf_text(path) pypdf + pdfplumber ║
Expand Down Expand Up @@ -285,20 +294,20 @@ Repeating that extraction in every new domain repo is the wrong move.
╠══════════════════════════════════════════════════════════════════════╣
║ ║
║ theright2read ║
║ └── sansad-semantic-crawler (pip, pinned v0.2.0) ║
║ └── commoner-analyse (pip, pinned v0.2.0) ║
║ → corpus-refresh → assets/parliament_libraries.js ║
║ → static GH Pages site ║
║ ║
║ academiaindia ║
║ └── sansad-semantic-crawler (pip, pinned v0.2.0) ║
║ └── commoner-analyse (pip, pinned v0.2.0) ║
║ → make scrape → docs/data/*.json ║
║ → static GH Pages site + Vitest frontend tests ║
║ ║
╚══════════════════════════════════════════════════════════════════════╝

DEBT ITEMS (annotated inline above with ← DEBT):
1. cad-mcp search tools duplicate partial-recall [HIGH]
2. budget-crawler scrapping_utils ≈ obi_utils [MEDIUM]
3. budget-crawler/zotero_semantic_search.py [fossil — delete]
4. budget-crawler/sansad_vacancy_scraper.py [delete, import sansad]
2. public-finance scrapping_utils ≈ obi_utils [MEDIUM]
3. public-finance/zotero_semantic_search.py [fossil — delete]
4. public-finance/sansad_vacancy_scraper.py [delete, import sansad]
```
Loading