Merging large public data - millions of records

Posted 3 weeks ago

Worldwide

Summary

I need to merge several large public data files together, they are provided via csv or api. In total they have millions of rows of data - more than i can open in excel. I have instructions on how to merge it from ChatGPT and sample scripting but need someone who execute it. We are going to do it in a few parts The first part is taking the public files, and merging each sheet together, deleting a lot of data we don't need (columns/rows) and sending me back the completed file in parts that can open each in excel and then send a few segments out to me separately. Once that is complete we discuss part 2 which is merging many more files together to create a large database Part 3 would be to use the API and store the data on a cloud big data server so we can build something with it. Please tell about experience handling and merging together files with millions of rows. This is like 20mm rows and tell how you would go about it and store it. Then there are instructions we have for how to match the files and merge them together to get the exact file we need.

  • Less than 30 hrs/week
    Hourly
  • 1-3 months
    Duration
  • Intermediate
    Experience Level
  • $5.00

    -

    $12.00

    Hourly
  • Remote Job
  • One-time project
    Project Type
Skills and Expertise
Mandatory skills
Python
Data Extraction
Activity on this job
  • Proposals:50+
  • Last viewed by client:3 weeks ago
  • Interviewing:
    30
  • Invites sent:
    17
  • Unanswered invites:
    2
About the client
Member since Mar 30, 2020
  • United States
    Stamford4:13 AM
  • $53K total spent
    154 hires, 16 active
  • 4,697 hours
  • Individual client

Explore similar jobs on Upwork

Local Lead GenerationHourly‐ Posted 2 weeks ago
Web Scraping
Data Scraping
Data Extraction
Lead Generation
Data Entry
Data Mining
Data Collection
Data Processing
Web Scraping Framework
Web Crawler Framework
Web Scraping Software
Web Scraping Plugin
Web API
Search Tool
Search Engine
Microsoft Word
Data Entry
Administrative Support
Microsoft Excel

How it works

  • Post a job icon
    Create your free profile
    Highlight your skills and experience, show your portfolio, and set your ideal pay rate.
  • Talent comes to you icon
    Work the way you want
    Apply for jobs, create easy-to-by projects, or access exclusive opportunities that come to you.
  • Payment simplified icon
    Get paid securely
    From contract to payment, we help you work safely and get paid securely.
Want to get started? Create a profile

About Upwork

  • Rating is 4.9 out of 5.
    4.9/5
    (Average rating of clients by professionals)
  • G2 2021
    #1 freelance platform
  • 49,000+
    Signed contract every week
  • $2.3B
    Freelancers earned on Upwork in 2020

Find the best freelance jobs

Growing your career is as easy as creating a free profile and finding work like this that fits your skills.

Trusted by

  • Microsoft Logo
  • Airbnb Logo
  • Bissell Logo
  • GoDaddy Logo