Real-Time Scraping App (Long Term Contract)

Web, Mobile & Software Dev Web Development Posted 1 year ago

Hourly Job

Full Time
More than 6 months

Details

Hi guys,
We're looking for a talented Ruby/Rails programmer to build a rails application from scratch. The app will scrape and aggregate results in real-time from 8 different sources(referral websites) and store & display those results with referral linking back to the sources. We have a BRD document with more details we can share. Here's a brief overview, this is a long term engagement and we're interviewing to begin immediately.

Proposed Technology Stack:
  - Typhoeus (invoke parallel web requests)
  - Nokogiri  (used for scraping data from typhoeus responses)
  - Rails (for webapp)
  - Heroku (web host)

Phase 1: Website and Crawlers (3 weeks)
  - Write 8 crawlers using rake tasks to scrape, collect and store company data
     in db from up to 8 referral sources.

  - Create a Rails app to query the db for this data

   - When users search from our website for Companys or product-types, Rails should first preform a real time scrape against all 8 referrals again and should match the results against our pre-populated db, and only return the results that have a match grouped by Company having many referrals.




Phase 2: Additional Crawlers (4 weeks ongoing)
  - Write additional crawlers to search our db and find Companys which have missing information and then populate that Company with the info needed (store schedules, ratings and reviews and so on, we have a list).



CODE QUALITY:
-----------------------------------------------------------------------------------------------------
Code should be clean, well-thought out making use of new types of classes and modules and be extensible.

Skills Required:

Client Activity on this Job

Last Viewed: 1 year ago

Applicants: 15

Hired: 1


About the Client

(5.00) 4 reviews

United States
New York 12:42 AM

7 Jobs Posted
86% Hire Rate, 1 Open Job

Over $20,000 Total Spent
6 Hires, 2 Active

$22.16/hr Avg Hourly Rate Paid
909 Hours

Member Since Jun 21, 2013