As part of the Data Source Identification Pipeline, we want attach to discovered data sources the agency they are associated with. To aid in this process, we should utilize the Sitemap Scraper tool that was developed: https://github.com/Police-Data-Accessibility-Project/PDAP-Scrapers/tree/c7ecd224c2aab66666a96832bea6bc11b69de623/utils/sitemap_scraper
- Query the Supabase Agency table for all agencies with a URL and their name
- Modify Sitemap Scraper so the output retains the root URL and agency name
- Run the agency URLs through the sitemap scraper
As part of the Data Source Identification Pipeline, we want attach to discovered data sources the agency they are associated with. To aid in this process, we should utilize the Sitemap Scraper tool that was developed: https://github.com/Police-Data-Accessibility-Project/PDAP-Scrapers/tree/c7ecd224c2aab66666a96832bea6bc11b69de623/utils/sitemap_scraper