Backfill Orchestrator Engineer (Airflow + Daily Support)

Overview
We are looking for a skilled engineer to help us design and build a Backfill Orchestrator on Apache Airflow.
The build must validate serial backfill execution per ETL DAG, while still allowing parallel execution across different DAGs.

You will work closely with one of our engineers, providing daily 2–3 hours of support over a 2-week period to guide implementation, answer questions, review progress, and build key components yourself.


Project Goals
1. Implement a Backfill Generator

Accepts large date range (fromDate → toDate)
Splits into batches (daily/weekly/monthly)
Sends each batch to the orchestrator

2. Implement a Backfill Orchestrator

Receives batches from generator
Triggers ETL DAG (we call these “decks”—same as DAG)
Ensures only 1 run per ETL DAG at a time
When ETL DAG is busy → queue/wait
Other DAGs must remain unblocked
Provide clear logs



3. Concurrency Control

Evaluate Airflow features (max_active_runs, pools, sensors)
If needed, implement custom locking (DB or in-memory)

4. Logging & Monitoring

Full visibility of execution order
Clean logs for demo purposes

5. Deliverables

Complete source code

Documentation:
Architecture diagram

Backfill flow

Concurrency strategy
Setup + execution instructions

Required Skills
AWS, Python, Docker, Kubernetes
Strong experience with Airflow (preferred) or Dagster/Prefect
ETL orchestration and pipeline design
Concurrency control: locks, sensors, semaphores, queues
Python proficiency
Solid understanding of DAG scheduling, event-based flows
Experience building such systems

Nice-to-Have:
Experience with backfill systems
Distributed systems knowledge
Metadata DB querying experience
Familiarity with concurrency patterns in Airflow


Working Style
You’ll collaborate with our internal engineer, providing:
Daily sessions (2–3 hours) for support, explanations, guidance
Code reviews and architectural advice
Helping fix blocking issues
Hands-on coding when needed
Speed and correctness matter more than polish.
If the build is successful, we may engage for additional work.

Timeline
2 weeks, 9PM IST onwards.


How to Apply

Please include:
Short intro + relevant experience
Examples of orchestration/ETL/backfill work
Airflow or Prefect projects you’ve built if possible
Availability for daily 2–3 hour collaboration