Hire the Best Apache Spark Engineers
in India

OBJECTIVE I aspire for a challenging position in a professional Organization where I can enhance my skills and strengthen them in conjunction with Organization's goals. A self-motivated achiever with an ability to plan and execute.

Hi, I’m Anita, managing this Upwork account. Our projects are led by Pankaj, a Principal Data Engineer with 19+ years of experience, supported by a team of skilled data engineers, cloud specialists, and analytics professionals. Together, we deliver end-to-end data engineering and analytics solutions — from design to deployment and training. Most of our experience lies in the banking, finance, and e-commerce domains, where we have built and optimized large-scale data platforms for risk management, fraud detection, customer analytics, and regulatory reporting — ensuring performance, scalability, and reliability. 🔹 Key Expertise Big Data & Spark: 10+ years of experience with PySpark, Spark SQL, Spark Scala, and Structured Streaming for large-scale and streaming data pipelines. Programming Languages: Proficient in Python and Scala for ETL, automation, and distributed data processing. Cloud & Platforms: Expertise across AWS (S3, Glue, EMR, Redshift, Kinesis), Azure (Data Factory, Synapse, Databricks), and GCP (BigQuery, Dataflow, Pub/Sub). Modern Data Stack: Hands-on with dbt, Snowflake, and Apache Airflow for ELT, orchestration, and data modeling. Domain Knowledge: Strong experience in banking data engineering, plus e-commerce analytics, recommendation engines, and customer insights. Team Capabilities: Our team can handle complete data projects — including architecture setup, pipeline development, performance tuning, and dashboard delivery. Training & Mentorship: We also provide corporate and individual training in Python, PySpark, and modern data engineering, using real-world projects like: “Designing a Kafka → PySpark → Snowflake → dbt → Airflow data pipeline.” We combine deep technical expertise, agile delivery, and strong business understanding to deliver scalable, cloud-native, and future-ready data platforms tailored to your goals. Let’s connect to discuss how our team can help you design, build, and deliver robust, modern data engineering solutions for your business. — Anita (Account Manager) & Pankaj (Principal Data Engineer & Delivery Lead)

Most businesses are drowning in data but starving for insights - pipelines that break, dashboards nobody trusts, reports built manually in Excel. I fix that. 5+ years building enterprise data platforms: 800M+ records processed daily 70% pipeline runtime reduction via dbt 50+ Power BI dashboards delivered 98% data trust through quality frameworks Stack: Snowflake, Databricks, dbt, Airflow, PySpark, Microsoft Fabric, ADF, AWS, Azure, Power BI, Python Beyond the warehouse layer, I've also built production backend systems integrating third-party APIs — WhatsApp Cloud API, Supabase, webhook orchestration — deployed on cloud infrastructure. Available 8 AM - 2 PM IST. Open to short and long term projects. Let's talk.

Helping companies build scalable, reliable, and cost-efficient data platforms. I'm a Principal Data Engineer with 11+ years of experience designing and implementing modern data engineering solutions for startups, fintech companies, healthcare organizations, and enterprise businesses. I've helped organizations migrate legacy systems, build cloud-native data platforms, optimize processing costs, and deliver production-ready analytics pipelines. My expertise includes designing end-to-end data architectures, building batch and streaming pipelines, implementing Data Lakes and Lakehouses, and automating infrastructure using Infrastructure as Code. What I can help you with ✔ Databricks Development & Optimization ✔ Apache Spark (PySpark & Scala) ✔ Azure Data Factory (ADF) ✔ Azure Data Lake Storage (ADLS) ✔ Delta Lake & Delta Live Tables ✔ AWS (EMR, Glue, Athena, Lambda, S3) ✔ Data Warehouse Design ✔ ETL / ELT Pipelines ✔ Data Migration ✔ Data Modeling ✔ Terraform & Infrastructure Automation ✔ SQL Performance Optimization ✔ Python Development ✔ CI/CD for Data Platforms ✔ Airflow Workflow Automation ✔ AI-powered Workflow Automation (Cursor, Claude, MCP, n8n) Recent accomplishments • Reduced operational costs by 90% by redesigning SCD implementation using Delta Live Tables. • Led the architecture and delivery of financial products including Loans and Credit Cards. • Migrated enterprise data warehouses to cloud-native lakehouse architecture. • Built scalable reconciliation frameworks using Databricks and Airflow. • Implemented Terraform-managed Databricks infrastructure for improved governance and scalability. • Designed enterprise-grade data platforms for healthcare, fintech, and retail organizations. My Skills Sets are: SQL, Apache Spark, Hive, Hadoop, Excel, Shell Scripting, AWS EMR, Ec2, S3, cloud formation, Clojure, MongoDB MySQL, Airflow.

Azure Data Factory, Databricks, Microsoft Fabric and Snowflake pipelines — plus the Power BI layer on top. SAP, Epicor, Salesforce and NetSuite already synced to Azure. 5+ years in Azure data engineering. I scope, build and hand over the whole path: source system → ingestion → ETL/ELT into a Medallion Lakehouse (Bronze, Silver, Gold) → semantic model → the dashboard your team actually opens. Most projects reach me one of four ways: - No data platform yet — spreadsheets, an app database, and numbers people need every week - A legacy platform that needs moving to Microsoft Fabric, Azure or Databricks - An ERP or CRM that won't talk to your warehouse — SAP, Epicor, Salesforce, NetSuite and QuickBooks are ones I've already wired to Azure - Month-end reporting that takes three days and still doesn't reconcile 🔧 What I build ✅ New data platforms from scratch — empty Azure subscription to first working dashboard ✅ Azure Data Factory and Microsoft Fabric pipelines — OneLake, Lakehouse, Warehouse, Notebooks ✅ Azure Databricks, Apache Spark and Delta Lake processing at scale ✅ ETL and ELT development, including incremental and metadata-driven patterns ✅ Data warehouses, Lakehouse architectures and dimensional data models ✅ API, ERP, CRM and SQL database integrations ✅ Legacy platform migrations to Microsoft Fabric, Azure and Databricks ✅ Power BI dashboards, semantic models and DAX ✅ Data quality, validation, monitoring, and cost and performance tuning 🛠️ Stack Microsoft Fabric · Azure Data Factory · Azure Databricks · Azure Synapse Analytics · Azure SQL Database · Snowflake · BigQuery · SAP Python · SQL · PySpark · DAX SQL Server · PostgreSQL · MySQL Power BI · Tableau · Looker Studio 🤝 How I work ✅ A written build plan before any code — scope, sequence, and what "done" means ✅ Pipelines ship with logging and failure alerts, not just a green run on my machine ✅ Handover documentation, so your team can run it without me ✅ 4+ hours of daily overlap with US business hours (I'm UTC+5:30) 📩 Send me your source list and the one report that keeps breaking. I'll come back with a pipeline plan, a rough timeline, and an honest answer about whether Microsoft Fabric is the right target for you.

Hands on Data & AI platforms architect & lead data engineer, storage architect & lead engineer, streaming platforms architect & lead engineer, with 14+ years of experience in designing & building end to end peta byte scale distributed systems, real time systems & batch data platforms from scratch on clouds & on-prem. I develop distributed & scalable back-end systems using using languages like goLang, Rust & python. Building AI platforms using Self hosted LLMs, Kserve on kubernetes, RAG, MCP Servers, Langfuse MLops Platforms using Kubeflow & MLFlow. Mordenize existing data platforms with AI first approach. Hybrid semantic mapping layers or unstructured and structured data using heuristics, memory and LLMs. Built & worked on peta byte scale streaming, batch data & AI platforms in top companies. An open source contributor to data technologies & products like Airbyte etc. Love working on database internals, performance and optimizations. I have experience working with telemetry data, payments data, video data, sports data, eCommerce data & affiliate marketing data, logs data, clickstream data. Skill Set: AI Technologies: Langraph, Langchain, Kserve, VLLM, Quadrant, Langfuse, Ollama. Big Data Technologies: Spark, Kafka, Flink, Presto, Dremio, Hudi, Deltalake Data warehouses: Snowflake, Druid, Clickhouse, Redshift, SingleStore(Memsql), Quest Databases: Postgres, Mysql, Cassandra, DynamoDB, DuckDB Programming languages: Golang, Python, Rust, Scala, Java Visualization: Tableau, Apache Superset, Zoomdata Data Technologies - Airbyte, Fivetran, Dagster, Airflow, Nifi, Kubeflow, ElasticSearch, OpenSearch Platforms: Databricks, Snowflake, Cloudera, Supabase, Aiven Ops: Kubernetes, Docker Cloud: AWS, GCP, Azure