Government PDF ingestion
ETL pipelines that scrape, clean, transform, and structure government PDFs spanning 2000–2026 — on weekly + monthly cycles, into multiple stores.
01
At a glance
2000 — 2026
coverage
weekly + monthly
cycles
Multi-store
Postgres · Mongo · Wasabi
Schema-validated
every row
02
How it works
01ScrapeScrapy crawls portal, downloads new PDFs.
02ExtractParse, normalise tables and text.
03ValidatePydantic schema enforcement + quality checks.
04StoreWasabi (raw) + Postgres/Mongo (structured).
03
Architecture
04
Features
ScrapeScrapy crawlers per portal, with monthly + weekly publication cycles.
TransformPandas-driven ETL — clean, normalise, structure.
StoreRaw PDFs to Wasabi, structured to Postgres/MongoDB depending on shape.
ValidateReusable Pydantic schema layer across every pipeline.
05
Full stack
ScrapyPythonPandasWasabiPostgreSQLMongoDBPydanticArgo
next project →01SadhanAI↗