Senior Data Engineer - AWS Lakehouse (Iceberg, Trino, Airflow, dbt)
Worldwide
We run a production data platform on AWS and are looking for an experienced data engineer to work on it with us on an ongoing basis. THE STACK YOU WOULD ACTUALLY TOUCH - Orchestration: Apache Airflow (managed / MWAA) - DAG authoring, scheduling, backfills - Lakehouse: Apache Iceberg on S3 with the Glue catalog - table maintenance, compaction, snapshot expiry, orphan-file cleanup - Query engines: Trino / Starburst, Athena, Snowflake - Transformation: dbt (Trino and Snowflake adapters) - Processing: Spark / PySpark, EMR - Infrastructure: Terraform on AWS, EKS + Helm, ArgoCD - Services: Python, FastAPI TYPICAL WORK - Build and fix Airflow DAGs, including hourly and backfill runs that must be safely re-runnable - Iceberg table maintenance and the retention / compaction jobs around it - dbt models against Trino and Snowflake - Terraform changes for the AWS resources these pipelines depend on - Debug pipeline failures end to end, from S3 and the Glue catalog through Iceberg and Trino out to the downstream API WHAT WE NEED - Real production experience with Iceberg and at least one of Trino / Starburst / Athena. This is the part most applicants do not have - please be honest about it. - Airflow in production, not just tutorials - dbt beyond the basics - Terraform and AWS IAM competence - You use Claude / Claude Code (or a comparable AI coding assistant) as a normal part of your daily workflow. We work this way and expect you to be genuinely fast with it - including knowing where these tools get things wrong and how you verify their output. This is not optional for us. - Clear written English, works independently, asks when something is unclear NICE TO HAVE Snowflake, Kubernetes/Helm, Grafana/Prometheus, German. HOW WE WORK We start with one small paid task so both sides can see whether it fits. If that works, ongoing part-time collaboration. Our budget for this role is up to $20/hour - please only apply if that works for you. PLEASE ANSWER THESE IN YOUR PROPOSAL (short, concrete answers - no marketing copy) 1. How do you handle orphan-file cleanup and snapshot expiry for Iceberg tables on S3 with the Glue catalog? What goes wrong if the retention is configured incorrectly? 2. Describe one Airflow DAG you made safely re-runnable (idempotent). What exactly did you change? 3. How do you use Claude (or a similar AI coding assistant) in your daily work? Give one concrete example where it was wrong and how you caught it. 4. Your hourly rate, and how many hours per week you can commit. Proposals that do not answer these four questions will not be reviewed.
- More than 30 hrs/weekHourly
- 6+ monthsDuration
- IntermediateExperience Level
$12.00
-
$20.00
Hourly- Remote Job
- Ongoing projectProject Type
Skills and Expertise
Activity on this job
- Proposals:20 to 50
- Last viewed by client:6 hours ago
- Interviewing:3
- Invites sent:3
- Unanswered invites:0
About the client
- GermanyBarby11:20 PM
- $12K total spent17 hires, 3 active
- 644 hours
Explore similar jobs on Upwork
How it works
Create your free profileHighlight your skills and experience, show your portfolio, and set your ideal pay rate.
Work the way you wantApply for jobs, create easy-to-by projects, or access exclusive opportunities that come to you.
Get paid securelyFrom contract to payment, we help you work safely and get paid securely.
About Upwork
- 4.9/5(Average rating of clients by professionals)
- G2 2021#1 freelance platform
- 49,000+Signed contract every week
- $2.3BFreelancers earned on Upwork in 2020
Find the best freelance jobs
Growing your career is as easy as creating a free profile and finding work like this that fits your skills.
Trusted by