Developer Needed for Large-Scale Product Scraping and Database System

Posted yesterday

Worldwide

Summary

We are looking for an experienced developer to build and manage a large-scale product scraping and database system. The system should collect up to 1 million product listings daily from major retail websites and reputable eBay sellers. It must capture product titles, brands, UPCs, prices, availability, variations, product URLs and image URLs. The developer will also build and manage the database, prevent duplicate products, monitor price and inventory changes, and keep the scrapers working when websites change. The monthly operating budget for servers, databases, proxies and scraping services should remain close to $1,000. Please include: * Similar systems you have built * Your recommended setup * Estimated development cost * Expected monthly operating cost * How you would realistically handle this volume We are looking for someone experienced, practical and available for ongoing maintenance—not someone offering a basic scraper.

  • Less than 30 hrs/week
    Hourly
  • 1-3 months
    Duration
  • Intermediate
    Experience Level
  • Remote Job
  • Ongoing project
    Project Type
Skills and Expertise
Mandatory skills
Python
Data Scraping
API
Activity on this job
  • Proposals:50+
  • Interviewing:
    0
  • Invites sent:
    0
  • Unanswered invites:
    0
About the client
Member since Mar 4, 2013
  • United States
    Brooklyn1:03 AM
  • $71K total spent
    45 hires, 8 active
  • 10,770 hours

Explore similar jobs on Upwork

Artificial Intelligence
AI Agent Development
Automation
AI Development
Python
AI App Development
Document AI
Generative AI
AI Bot
AI Chatbot
Data Extraction
LLM Prompt Engineering
LangChain
OCR Algorithm
Prompt Engineering
Claude
n8n
AI Implementation
Retrieval Augmented Generation
AI Model Integration
Lead List w/ Personal Phone NumbersHourly‐ Posted 1 month ago
Lead Generation
B2B Lead Generation
List Building
Data Entry
Data Collection
Data Extraction
Data Management
Data Cleaning
Contact List
Company Research
Online Research
Contact Info Research
Prospect List
Data Scraping

How it works

  • Post a job icon
    Create your free profile
    Highlight your skills and experience, show your portfolio, and set your ideal pay rate.
  • Talent comes to you icon
    Work the way you want
    Apply for jobs, create easy-to-by projects, or access exclusive opportunities that come to you.
  • Payment simplified icon
    Get paid securely
    From contract to payment, we help you work safely and get paid securely.
Want to get started? Create a profile

About Upwork

  • Rating is 4.9 out of 5.
    4.9/5
    (Average rating of clients by professionals)
  • G2 2021
    #1 freelance platform
  • 49,000+
    Signed contract every week
  • $2.3B
    Freelancers earned on Upwork in 2020

Find the best freelance jobs

Growing your career is as easy as creating a free profile and finding work like this that fits your skills.

Trusted by

  • Microsoft Logo
  • Airbnb Logo
  • Bissell Logo
  • GoDaddy Logo