Azure Databricks & IBM Datastage Developer

Azure Python Spark Data Engineer — Remote, Part-Time NDIM Technologies Inc. is seeking an experienced Azure Data Engineer with strong hands-on expertise in Python, Apache Spark, PySpark, Azure Data Factory (ADF), and Azure Databricks. This is a remote, part-time, long-term opportunity for an experienced engineer who can independently design, build, troubleshoot, and optimize enterprise-scale data pipelines. 🎯 Key Responsibilities * Design, build, and maintain scalable ETL/ELT pipelines using Azure Data Factory and Azure Databricks as we are migrating from ibm datastage to Databricks so past IBM Datastage knowledge is more desirable * Develop and optimize data transformation logic using Python, Spark, and PySpark * Build and optimize data ingestion pipelines into Azure Data Lake Storage (ADLS) * Work with Delta Lake and Delta Tables * Perform data validation, reconciliation, and data-quality checks * Troubleshoot Spark jobs and optimize pipeline performance * Develop and maintain Databricks Jobs and Workflows * Collaborate with architects, analysts, and engineering teams to deliver reliable data solutions * Support Git, CI/CD, and DevOps processes * Ensure scalability, reliability, performance, and data quality across enterprise datasets 🧠 Mandatory Skills ✅ Strong hands-on Python development ✅ Strong practical Apache Spark & PySpark experience ✅ Azure Databricks expertise ✅ Azure Data Factory (ADF) experience ✅ Strong SQL development and query optimization skills ✅ Azure Data Lake Storage (ADLS) experience ✅ Delta Lake / Delta Tables ✅ Familiarity with Unity Catalog ✅ Databricks Jobs, Workflows, and APIs ✅ Understanding of Git, CI/CD, and DevOps principles ⚠️ Candidates without practical, hands-on Spark/PySpark experience will not be considered. ⏰ Commitment & Work Schedule 🕐 Part-Time: 20–30 hours per week 🌎 100% Remote — Open to qualified freelancers globally 🕘 Required Availability: Candidates must be available for agreed working hours overlapping with 8:00 AM–5:00 PM Central Time (CT), U.S. 📅 Duration: Initially 3–6 months, with potential for long-term extension We are seeking professionals who can consistently dedicate 20–30 hours per week, communicate effectively during the required U.S. Central Time working window, and take strong ownership of their deliverables. 💰 Compensation 💵 $8–$13USD per hour, based on experience and technical expertise. 📩 How to Apply Please submit your proposal through Upwork and include: * A brief summary of your Azure + Databricks + Python + PySpark experience * Your years of relevant data engineering experience * A brief description or portfolio example of a previously completed Azure/Databricks data pipeline project * Your experience with ADF, ADLS, Delta Lake, and SQL * Your proposed hourly rate * Your weekly availability * Confirmation that you can work during the required 8:00 AM–5:00 PM U.S. Central Time window 🌎 About NDIM Technologies Inc. NDIM Technologies Inc. is a Data Engineering & AI Consulting company specializing in modern cloud data platforms, Azure Databricks, enterprise data migration, transformation, and data-validation solutions. We are looking for a dependable engineer who can contribute quickly and help us build scalable, production-grade data platforms. Please apply only if you have genuine hands-on Azure Databricks and Spark/PySpark experience.