I need a solution to scrape data from Linked..In with the following requirements. I will need the selected developer to propose a working solution to scrape large data sets and tell me which proxy service to use that is reliable. The details are as follows.
1. Write a program that will take in a list of LI profile URLs (about 20,000 at a time).
2. The program will extract, in CSV format, the profile's name, most recent job title, and company name.
3. The program should then browse to the company, and scrape the company profile for company size, location, industry, and company type, and add this to the same row in the CSV file.
I will need to be able to run this myself and need to return at least 500 results per hour.