We have multiple sites that we need to scrape data from and we need all of that data to be saved within a database.
1. YellowPages.com - business contact information (website, phone, address, and email if available)
2. Google - website rankings, page rank, number of indexed pages,
3. Google Maps - business contact information
4. Yelp - business contact information
5. Moz.com - Domain Authority, Page Authority, number of inbound links.
6. Google My Business - Check to see whether profile is claimed.
We'd like to be able to append this information to the information we currently have in other databases.
We'd also like to be able to collect the following information from individual websites within the database:
1. email address (if available)
2. Phone number (if available)
3. Whether the site has a mobile version or not
4. Titles and metas
5. Whether the site has an XML sitemap and Robots.txt file
6. Whether the site has a blog
7. Site speed