Real-Time Scraping App (Long Term Contract)

Web, Mobile & Software Dev Web Development Posted 3 years ago


More than 30 hrs/week
More than 6 months


Hi guys,
We're looking for a talented Ruby/Rails programmer to build a rails application from scratch. The app will scrape and aggregate results in real-time from 8 different sources(referral websites) and store & display those results with referral linking back to the sources. We have a BRD document with more details we can share. Here's a brief overview, this is a long term engagement and we're interviewing to begin immediately.

Proposed Technology Stack:
  - Typhoeus (invoke parallel web requests)
  - Nokogiri  (used for scraping data from typhoeus responses)
  - Rails (for webapp)
  - Heroku (web host)

Phase 1: Website and Crawlers (3 weeks)
  - Write 8 crawlers using rake tasks to scrape, collect and store company data
     in db from up to 8 referral sources.

  - Create a Rails app to query the db for this data

   - When users search from our website for Companys or product-types, Rails should first preform a real time scrape against all 8 referrals again and should match the results against our pre-populated db, and only return the results that have a match grouped by Company having many referrals.

Phase 2: Additional Crawlers (4 weeks ongoing)
  - Write additional crawlers to search our db and find Companys which have missing information and then populate that Company with the info needed (store schedules, ratings and reviews and so on, we have a list).

Code should be clean, well-thought out making use of new types of classes and modules and be extensible.

  • Other Skills:

Activity on this Job

Last Viewed by Client: 3 years ago

Invites Sent: 7

Unanswered Invites: 0

Hired: 1

Other open jobs by this client ({{ jobsCount }})

About the Client

(5.00) 5 reviews

United States
New York 05:15 AM

11 Jobs Posted
64% Hire Rate, 1 Open Job

Over $20,000 Total Spent
7 Hires, 1 Active

$22.16/hr Avg Hourly Rate Paid
909 Hours

Member Since Jun 21, 2013