Data Engineer — Python,Airflow, SQL, APIs & Production Pipelines

Python Data Engineer — Airflow, SQL, APIs & Production Pipelines We are looking for a strong **Python Data Engineer** to build and improve the data infrastructure supporting a sports analytics and quantitative research platform. This is a hands-on engineering role. You will be working with real-world data from APIs, live feeds, files, and external providers—data that may be delayed, duplicated, incomplete, inconsistently formatted, or subject to unexpected schema changes. We are not looking for someone who only builds dashboards, runs notebooks, or writes one-off scraping scripts. We need an engineer who can build reliable, testable, production-grade pipelines. ## Responsibilities * Build and maintain Python-based data ingestion pipelines. * Orchestrate scheduled workflows using Apache Airflow. * Integrate REST APIs, websocket feeds, files, and third-party data providers. * Write and optimize complex SQL transformations. * Build clean, structured, and reusable data models. * Implement data validation, monitoring, logging, retries, and alerting. * Debug missing records, duplicates, schema changes, pipeline failures, and data inconsistencies. * Improve the reliability and performance of existing infrastructure. * Document pipelines, datasets, dependencies, and technical decisions. * Work directly with quantitative researchers and data scientists to turn data requirements into dependable systems. ## Required Experience * Strong Python development skills. * Strong SQL skills. * Hands-on experience building ETL or ELT pipelines. * Production experience with Apache Airflow. * Experience integrating APIs and handling pagination, authentication, rate limits, retries, and schema changes. * Experience with relational databases or cloud data warehouses. * Strong debugging and problem-solving ability. * Experience with Git, testing, logging, and maintainable software-development practices. ## Preferred Experience * Snowflake. * AWS or another cloud platform. * Docker. * CI/CD. * Streaming or near-real-time data. * Websockets, RabbitMQ, Kafka, or other message-queue systems. * Sports, betting, financial-market, or other event-driven data. * Data infrastructure supporting machine-learning or quantitative-research workflows. Sports knowledge is useful but not required. Engineering quality, reliability, and ownership matter more. ## What We Are Not Looking For Please do not apply if your experience is primarily limited to: * Data visualization or dashboards. * Jupyter notebooks without production deployment. * Basic web scraping. * No-code ETL tools. * Academic projects without meaningful hands-on pipeline ownership. * Managing engineers without personally writing and debugging code. ## How to Apply Begin your proposal with the words **“Reliable Pipelines”** so we know you read the full post. Then answer the following: 1. Describe the most technically difficult data pipeline you personally built. 2. What parts of the architecture and code did you own directly? 3. How have you used Apache Airflow in production? 4. Describe a pipeline failure or data-quality issue that was difficult to diagnose. How did you solve it? 5. What is your experience with Python, SQL, APIs, and Snowflake? 6. Are you comfortable completing a short paid technical assessment? Please include relevant GitHub repositories, code samples, architecture diagrams, or examples of production systems you have built. Generic or AI-generated proposals that do not answer the questions above will not be considered.