Senior Data Engineer - AWS Lakehouse (Iceberg, Trino, Airflow, dbt)

Posted 5 hours ago

Worldwide

Summary

We run a production data platform on AWS and are looking for an experienced data engineer to work on it with us on an ongoing basis. THE STACK YOU WOULD ACTUALLY TOUCH - Orchestration: Apache Airflow (managed / MWAA) - DAG authoring, scheduling, backfills - Lakehouse: Apache Iceberg on S3 with the Glue catalog - table maintenance, compaction, snapshot expiry, orphan-file cleanup - Query engines: Trino / Starburst, Athena, Snowflake - Transformation: dbt (Trino and Snowflake adapters) - Processing: Spark / PySpark, EMR - Infrastructure: Terraform on AWS, EKS + Helm, ArgoCD - Services: Python, FastAPI TYPICAL WORK - Build and fix Airflow DAGs, including hourly and backfill runs that must be safely re-runnable - Iceberg table maintenance and the retention / compaction jobs around it - dbt models against Trino and Snowflake - Terraform changes for the AWS resources these pipelines depend on - Debug pipeline failures end to end, from S3 and the Glue catalog through Iceberg and Trino out to the downstream API WHAT WE NEED - Real production experience with Iceberg and at least one of Trino / Starburst / Athena. This is the part most applicants do not have - please be honest about it. - Airflow in production, not just tutorials - dbt beyond the basics - Terraform and AWS IAM competence - You use Claude / Claude Code (or a comparable AI coding assistant) as a normal part of your daily workflow. We work this way and expect you to be genuinely fast with it - including knowing where these tools get things wrong and how you verify their output. This is not optional for us. - Clear written English, works independently, asks when something is unclear NICE TO HAVE Snowflake, Kubernetes/Helm, Grafana/Prometheus, German. HOW WE WORK We start with one small paid task so both sides can see whether it fits. If that works, ongoing part-time collaboration. Our budget for this role is up to $20/hour - please only apply if that works for you. PLEASE ANSWER THESE IN YOUR PROPOSAL (short, concrete answers - no marketing copy) 1. How do you handle orphan-file cleanup and snapshot expiry for Iceberg tables on S3 with the Glue catalog? What goes wrong if the retention is configured incorrectly? 2. Describe one Airflow DAG you made safely re-runnable (idempotent). What exactly did you change? 3. How do you use Claude (or a similar AI coding assistant) in your daily work? Give one concrete example where it was wrong and how you caught it. 4. Your hourly rate, and how many hours per week you can commit. Proposals that do not answer these four questions will not be reviewed.

  • More than 30 hrs/week
    Hourly
  • 6+ months
    Duration
  • Intermediate
    Experience Level
  • $12.00

    -

    $20.00

    Hourly
  • Remote Job
  • Ongoing project
    Project Type
Skills and Expertise
Mandatory skills
Apache Airflow
dbt
Apache Spark
Activity on this job
  • Proposals:20 to 50
  • Last viewed by client:4 hours ago
  • Interviewing:
    3
  • Invites sent:
    3
  • Unanswered invites:
    0
About the client
Member since Dec 17, 2023
  • Germany
    Barby9:52 PM
  • $12K total spent
    17 hires, 3 active
  • 644 hours

Explore similar jobs on Upwork

Graph & Database EngineerHourly‐ Posted 5 days ago
Graph Database
Ontology
JSON-LD
GTM EngineerHourly‐ Posted 2 months ago
Clay
LinkedIn Campaign Manager
Lead Generation Strategy

How it works

  • Post a job icon
    Create your free profile
    Highlight your skills and experience, show your portfolio, and set your ideal pay rate.
  • Talent comes to you icon
    Work the way you want
    Apply for jobs, create easy-to-by projects, or access exclusive opportunities that come to you.
  • Payment simplified icon
    Get paid securely
    From contract to payment, we help you work safely and get paid securely.
Want to get started? Create a profile

About Upwork

  • Rating is 4.9 out of 5.
    4.9/5
    (Average rating of clients by professionals)
  • G2 2021
    #1 freelance platform
  • 49,000+
    Signed contract every week
  • $2.3B
    Freelancers earned on Upwork in 2020

Find the best freelance jobs

Growing your career is as easy as creating a free profile and finding work like this that fits your skills.

Trusted by

  • Microsoft Logo
  • Airbnb Logo
  • Bissell Logo
  • GoDaddy Logo