Senior Backend Engineer for High-Concurrency Pipelines

Posted 4 weeks ago

Worldwide

Summary

Seeking an elite Backend Engineer to design and build Phase 1 of a proprietary data acquisition and analytics platform. The core objective is to establish a rock-solid, production-grade ingestion pipeline that processes real-time data streams from multiple third-party REST APIs into a highly optimized database architecture. The Pipeline Architecture: Ingestion Layer: Efficient connection handling and rate-limit management for 5 external REST API feeds. Data Normalization Layer: Robust validation and defensive parsing (e.g., Zod/TypeScript or equivalent) to map heterogeneous JSON responses into a unified internal schema, gracefully handling provider schema updates. Storage Layer: High-concurrency time-series database design utilizing PostgreSQL with TimescaleDB extensions, optimized with proper partitioning (hypertables) and indexing strategies (e.g., BRIN) for low-latency analytical queries and data integrity. Downstream Ready: Structuring the normalized historical data store to seamlessly feed downstream analytical and predictive logic layers. Requirements: Proven track record building high-concurrency data pipelines and time-series architectures. Deep expertise in relational databases, transaction isolation, connection pooling, and continuous aggregation. Experience with defensive parsing techniques to prevent pipeline breakage due to upstream API schema drift. Ability to write clean, maintainable, and highly documented code. Strict NDA and Work-for-Hire intellectual property agreements are mandatory before full technical specifications are released. Deliverable for Phase 1 Milestone ($150 Trial): A functional technical proof-of-concept demonstrating a single normalized API ingestion pipe feeding into an optimized time-series database structure to verify code quality, data integrity handling, and communication speed.

  • $150.00

    Fixed-price
  • Expert
    Experience Level
  • Remote Job
  • Complex project
    Project Type
Skills and Expertise
Mandatory skills
Python
Apache Hadoop
Nice-to-have skills
SQL
PostgreSQL Programming
Activity on this job
  • Proposals:20 to 50
  • Last viewed by client:4 weeks ago
  • Interviewing:
    0
  • Invites sent:
    0
  • Unanswered invites:
    0
About the client
Member since May 10, 2020
  • United States
    Houston5:57 AM
  • $6.2K total spent
    8 hires, 3 active
  • 65 hours

Explore similar jobs on Upwork

Database University AssignmentsHourly‐ Posted 9 months ago
SQL
Database
Microsoft Excel
Database Design
Database Management
SQL Server Integration Services
Excel Macros
Excel Formula
Microsoft Power BI
Microsoft Excel PowerPivot
Power Query
Data Entry
Data Cleaning
Data Analytics
Data Extraction
Data analysis and regression interpretationFixed-price‐ Posted 1 week ago
Python
Web Scraping
n8n
CRM Automation
Airtable
Automation
Selenium
API Integration
Retrieval Augmented Generation
Data Processing
LangChain
OpenAI API
Beautiful Soup
pandas
ETL Pipeline
Natural Language Processing
Data Extraction

How it works

  • Post a job icon
    Create your free profile
    Highlight your skills and experience, show your portfolio, and set your ideal pay rate.
  • Talent comes to you icon
    Work the way you want
    Apply for jobs, create easy-to-by projects, or access exclusive opportunities that come to you.
  • Payment simplified icon
    Get paid securely
    From contract to payment, we help you work safely and get paid securely.
Want to get started? Create a profile

About Upwork

  • Rating is 4.9 out of 5.
    4.9/5
    (Average rating of clients by professionals)
  • G2 2021
    #1 freelance platform
  • 49,000+
    Signed contract every week
  • $2.3B
    Freelancers earned on Upwork in 2020

Find the best freelance jobs

Growing your career is as easy as creating a free profile and finding work like this that fits your skills.

Trusted by

  • Microsoft Logo
  • Airbnb Logo
  • Bissell Logo
  • GoDaddy Logo