NOTE ABOUT BUDGET
We don't know how much this will cost. Please give us your fixed bid.
Write an application that will scrape information from profiles on a major social media website. (Site will be revealed to the freelancer who gets the job.)
Your application must be written in a popular scripting language like Python or Ruby using standard libraries. It must be designed to run on a Unix platform. We prefer Python using the Scrapy library, but you may choose another language with our approval.
See attached flowchart for a diagram of execution flow.
The Scraper will be run from the command line by one User.
The User will enter their site username and password into a configuration file.
The User will set up search criteria on the Site using their web browser. Once the User has obtained the first page of search results, the User will enter that Result Page URL into the Scraper. There may be a queue of multiple Result Page URLs.
Once a day, the Scraper will begin with the next Result Page URL in the queue and scrape information from all the profiles on that page. The Scraper will then advance to the next page of results and continue scraping until the maximum number of scrapes for the day has been met.
Inbetween each profile scrape, the Scraper should pause for a random number of seconds.
The pause between scrapes, as well as limiting the number of scrapes per day, is to prevent the Site from identifying the Scraper as a bot.
The Scraper should output results to a CSV file.