Economists crowdsource digitizing India's 1951-2011 district handbooks with AI parsing
paulnovosad · x · 2026-11-09
Development economist Paul Novosad's DDL project is crowdsourcing a complete digitized collection of India's District Census Handbooks from 1951–2011, using AI to turn scanned tables into usable data.
- The handbooks contain village-level socioeconomic detail beyond population: public goods, irrigated land, land use, trade and credit.
- A browsable/downloadable site is live with 75% of handbooks collected; coverage is weakest for 1991 (333 of 452 districts missing) and nearly complete for 2011.
- Key challenges: heavy state-to-state format variation, harmonizing district boundary changes and village name variants over time (leveraging @indiastatestory's district-change data).
- The team invites anyone with missing handbooks, parsing scripts, or scraped data to collaborate.
More from AGI Musings
- Thom Wolf on software's future: Everflux, translucent sources, and bespoke AI-built features — Thom_Wolf · 2026-11-10
- HN debate: permission-scoped subaccounts as the missing safety layer for AI agents — vboykis · 2026-11-10
- Sam Altman warns AI could concentrate too much power in one person, company or country — VraserX · 2026-11-10
- Engineers with AI experience are suddenly getting flooded with job offers — rickasaurus · 2026-11-10
- Dev's viral jab: artists rage at AI — will doctors rage when it cures disease? — ostrisai · 2026-11-10
- Armin Ronacher: AI is Irontown — progress with no villains — mitsuhiko · 2026-11-10