Looking for a developer to write a series of scripts to crawl web pages from various countries, scraping air quality data.
Most developed countries around the world have set up dozens or hundreds of air quality monitoring stations within their borders. These stations sample the air every hour and measure everything from PM2.5 (particulate matter smaller than 2.5 microns) to carbon monoxide. Most governments make this data available online in some form, but the situation isn't great:
- The data isn't in machine-readable form; i.e. there's rarely an API
- Only the current data or data from the last 24 hours is made available, so there's no way to do historical analysis
- As each country has its own system for publishing the data, and some countries have multiple systems, one does not simply write a crawler to go to each website and download the data.
Full documentation will be provided upon project commencement.