Senior Backend Engineer for High-Concurrency Pipelines
Worldwide
Seeking an elite Backend Engineer to design and build Phase 1 of a proprietary data acquisition and analytics platform. The core objective is to establish a rock-solid, production-grade ingestion pipeline that processes real-time data streams from multiple third-party REST APIs into a highly optimized database architecture. The Pipeline Architecture: Ingestion Layer: Efficient connection handling and rate-limit management for 5 external REST API feeds. Data Normalization Layer: Robust validation and defensive parsing (e.g., Zod/TypeScript or equivalent) to map heterogeneous JSON responses into a unified internal schema, gracefully handling provider schema updates. Storage Layer: High-concurrency time-series database design utilizing PostgreSQL with TimescaleDB extensions, optimized with proper partitioning (hypertables) and indexing strategies (e.g., BRIN) for low-latency analytical queries and data integrity. Downstream Ready: Structuring the normalized historical data store to seamlessly feed downstream analytical and predictive logic layers. Requirements: Proven track record building high-concurrency data pipelines and time-series architectures. Deep expertise in relational databases, transaction isolation, connection pooling, and continuous aggregation. Experience with defensive parsing techniques to prevent pipeline breakage due to upstream API schema drift. Ability to write clean, maintainable, and highly documented code. Strict NDA and Work-for-Hire intellectual property agreements are mandatory before full technical specifications are released. Deliverable for Phase 1 Milestone ($150 Trial): A functional technical proof-of-concept demonstrating a single normalized API ingestion pipe feeding into an optimized time-series database structure to verify code quality, data integrity handling, and communication speed.
$150.00
Fixed-price- ExpertExperience Level
- Remote Job
- Complex projectProject Type
Skills and Expertise
Activity on this job
- Proposals:20 to 50
- Last viewed by client:4 weeks ago
- Interviewing:0
- Invites sent:0
- Unanswered invites:0
About the client
- United StatesHouston5:57 AM
- $6.2K total spent8 hires, 3 active
- 65 hours
Explore similar jobs on Upwork
How it works
Create your free profileHighlight your skills and experience, show your portfolio, and set your ideal pay rate.
Work the way you wantApply for jobs, create easy-to-by projects, or access exclusive opportunities that come to you.
Get paid securelyFrom contract to payment, we help you work safely and get paid securely.
About Upwork
- 4.9/5(Average rating of clients by professionals)
- G2 2021#1 freelance platform
- 49,000+Signed contract every week
- $2.3BFreelancers earned on Upwork in 2020
Find the best freelance jobs
Growing your career is as easy as creating a free profile and finding work like this that fits your skills.
Trusted by