Building Scrapers
Scraper Types
The taxonomy of agency meeting sites, with a real spider for each type.
Repo
city-scrapers (core + consumer repos)city-scrapers-core and the per-city repos like city-scrapers-fortxScraper types
Spiders share a structure, but how they parse varies with how the agency renders meeting information. Meeting websites range from modern to ancient; these are the types that recur, each with a live example.
More worked examples - including GET+POST hybrids, authenticated requests, and Legistar - are in the link index.
Last updated on