Upwork home
Sign up
Secondary

Hire the best Apache Spark Engineers in Hyderabad, IN

Check out Apache Spark Engineers in Hyderabad, IN with the skills you need for your next job.
Hire freelancers
Clients rate Apache Spark Engineers
Rating is 4.7 out of 5.
4.7/5
based on 283 client reviews
  • $35 hourly

    • 5.0/5
    • (3 jobs)
    ════ Who Am I? ════ Hi, nice to meet you! I'm Ajay, a Tableau and SQL Specialist, Business Intelligence Developer & Data Analyst with half a decade of experience working with data. For the last few years I've been helping companies all over the globe achieve their Data Goals and making friends on the journey. If you're looking for someone who can understand your needs, collaboratively develop the best solution, and execute a vision - you have found the right person! Looking forward to hearing from you! ═════ What do I do? (Services) ═════ ✔️ Tableau Reports Development & Maintenance - Pull data from (SQL Servers, Excel Files, Hive etc.) - Clean and transform data - Model relationships - Calculate and test measures - Create and test charts and filters - Build user interfaces - Publish reports ✔️ SQL - Build out the data and reporting infrastructure from the ground up using Tableau and SQL to provide real time insights into the product and business KPI's - Identified procedural areas of improvement through customer data, using SQL to help improve the probability of a program by 7% - Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs and Scala. ═════ How do I work? (Method) ═════ 1️⃣ First, we need a plan; I will listen, take notes, analyze and discuss your goals, how to achieve them, determine costs, development phases, and time involved to deliver the solution. 2️⃣ Clear and frequent communication; I provide frequent project updates and will be available to discuss important questions that come up along the way. 3️⃣ Stick to the plan; I will deliver, on time, what we agreed upon. If any unforeseen delay happens, I will promptly let you know and provide a new delivery date. 4️⃣ Deliver a high-quality product. My approach aims to deliver the most durable, secure, scalable, and extensible product possible. All development includes testing, documentation, and demo meetings.
    Featured Skill Apache Spark
    Apache Hive
    Python Script
    Scala
    Machine Learning
    Hive
    SQL Programming
    Business Intelligence
    Microsoft Excel
    Microsoft Power BI
    Tableau
    SQL
    Python
  • $60 hourly

    • 4.9/5
    • (8 jobs)
    Nikhil is a Microsoft certified azure data engineer with 5+ years of experience in data engineering and big data. Have worked for couple of fortune 500 companies for developing and deploying their data solutions in azure and helped them find business insights out of their data. Coding: - SQL, Python, Pyspark Azure: - Azure Data Factory - Azure Databricks - Azure Synapse Analytics - Azure Datalake - Azure Functions and other azure services Reporting: - Power BI - Microsoft Office
    Featured Skill Apache Spark
    ETL
    Microsoft Azure
    Data Lake
    Data Warehousing
    Microsoft SQL Server
    Big Data
    PySpark
    Databricks Platform
    SQL
    Apache Hive
    Python
    Microsoft Excel
    Data Engineering
    Data Integration
  • $60 hourly

    • 4.9/5
    • (12 jobs)
    I am an expert python developer with 13 years of experience in building machine learning models , data visualization , web and rest api applications. I have worked with Investment Banks , Startups and consulted for Technology services companies. I Specialize in : - Data Analytics and Machine Learning with Apache Spark and TensorFlow - Web Scrapping and Data Mining - Data Visualization - Back end Development with Python / Django/ Flask - Mysql , MongoDB With Computer Science and Math Masters degree and extensive experience in agile development environment , I have the necessary skill set and problem solving abilities to get your job done and deliver on the expectations.
    Featured Skill Apache Spark
    MongoDB
    Django
    Flask
    Machine Learning
    Python
    Deep Learning
    Keras
    PyTorch
    TensorFlow
  • $40 hourly

    • 5.0/5
    • (8 jobs)
    Welcome to my Upwork profile! I’m an experienced Golang Developer with a proven track record of building high-performance, scalable backends and robust microservices. With over 8 years of expertise in crafting efficient, secure, and maintainable code, I’m ready to bring your project to life. Core Competencies: - API Development: Expert in designing and developing RESTful APIs that power modern web applications, integrating seamlessly with frontend technologies. - Microservices Architecture: Skilled in building microservices focusing on security, scalability, and reliability, using tools like Kafka/RabbitMQ/Kinesis for messaging and GoFiber/Gin for high-performance routing. - System Optimization: Strong background in optimizing backend performance, adhering to best practices such as Test-Driven Development (TDD), and implementing efficient workflows. Skills Summary: Languages: Golang, Scala, Python Data & Messaging: Apache Kafka, Kafka Streams, Apache Spark, NiFi Cloud & DevOps: AWS (S3, EKS, ECS, Lambda, EC2, Redshift, Glue ETL), Terraform, Ansible, Kubernetes, Docker, Airflow Architecture & Frameworks: REST APIs, gRPC, FIX Streams, ELK Stack Deployment & Orchestration: Docker, Kubernetes (ECS, EKS) CI/CD & Automation: Terraform, Ansible, Jenkins, GitLab CI, GitHub Actions Let’s work together to turn your vision into reality. Reach out, and let's discuss how I can add value to your project.
    Featured Skill Apache Spark
    Amazon Web Services
    HTTP
    Scala
    Golang
    Apache Kafka
    DevOps
    AWS Lambda
    RESTful Architecture
    Database Design
    Docker
    API
    Python
    Big Data
  • $50 hourly

    • 5.0/5
    • (3 jobs)
    With around 13 Years of IT experience on data driven applications.I excel in building robust data foundations for both structured and unstructured data from diverse sources. Additionally, I possess expertise in efficiently migrating data lakes and pipelines from on-premise to cloud environments. My skills include designing and developing scalable ETL/ELT pipelines using cutting-edge technologies such as Spark, kafka, Pyspark, Hadoop, Hive, DBT, Python, and leveraging cloud services like AWS, Snowflake, and DBT Cloud,Airbyte, BigQuery, Metabase and A good understanding of containerisation frameworks like Kubernetes and Docker is essential
    Featured Skill Apache Spark
    Apache Airflow
    Apache Hive
    Databricks Platform
    Python
    Apache Hadoop
    PySpark
    Snowflake
    Amazon S3
    dbt
    Database
    Oracle PLSQL
    Unix Shell
  • $10 hourly

    • 5.0/5
    • (1 job)
    PROFESSIONAL SUMMARY Having 3 Years 3 months of Total Experience in IT Industry with 3 Years of project experience in Data warehouse development and enhancement using Microsoft SQL Server, Azure Data Factory, Azure Data Bricks . Good Knowledge in analyzing data and writing SQL queries according to the requirement. Skilled in mapping clients' business requirements and translating the same into functional specifications, services, implementations and custom designing solutions. Experienced in resolution of production tickets related to Data Issues, Design changes, ETL issues and performance tuning on SQL server. Handled many Change Requests/BUGS which involves Data analysis, Problem identification, code change, Testing and finally deploying the code. Willing to accept responsibility and commit to working as part of a team to utilize my experience and skills in contributing
    Featured Skill Apache Spark
    Microsoft Azure
    Microsoft Azure SQL Database
    Microsoft SQL Server
    Microsoft SQL Server Programming
    SQL
    ETL Pipeline
  • $125 hourly

    • 5.0/5
    • (1 job)
    A Computer Science Engineering leader with 12+ yrs. of experience in gathering requirements and translating them into functional specifications and developing/designing code. I'm experienced in creating microservices/REST API's using spring boot. I can also create data engineer pipeline using apache spark, kafka. I can also create pipelines to analyze your data using OLAP.
    Featured Skill Apache Spark
    Docker Compose
    Snowflake
    Apache Kafka
    Data Engineering
    Microsoft Azure
    Kubernetes
    Apache Maven
    Spring MVC
    Docker
    Spring Framework
    Gradle
    Java
    Microservice
    Spring Boot
  • $39 hourly

    • 0.0/5
    • (0 jobs)
    I have 3 years of experience. Expert in Python and SQL. I have a strong grip on BigData Technologies. ADF, Databricks, Pyspark, Kafka, Data warehousing and many more. I build frameworks and ETL pipelines to consume data from all kinds of sources.
    Featured Skill Apache Spark
    RESTful API
    Algorithm Development
    Big Data
    Microsoft Azure
    Google Cloud Platform
    BigQuery
    PySpark
    Databricks Platform
    Apache Kafka
    SQL
    Python
    Data Engineering
  • $20 hourly

    • 5.0/5
    • (1 job)
    With a background steeped in Azure Data Engineering, I am a dedicated professional known for optimizing data processes and driving innovation in the field. As an Azure Data Engineer, I have honed my expertise in developing a streamlined framework tailored for the seamless migration from Teradata SQL to PySpark, achieving remarkable improvements in data transformation efficiency. My journey in data engineering is characterized by a relentless commitment to excellence. I excel in the management of CI/CD pipelines, orchestrating deployments for critical components like Synapse SQL, Azure Data Factory, and Databricks. My proficiency extends to leveraging a robust suite of Azure data services, including SQL Database, Data Lake Storage, Databricks, Data Factory, and Synapse Analytics, to craft and execute data solutions that empower organizations to harness the full potential of their data. Within the realm of ETL processes, I am recognized for my ability to efficiently extract data from diverse sources, apply sophisticated transformations with Azure Databricks, and load the refined data seamlessly into Azure SQL Database and Data Lake Storage. My approach places a strong emphasis on data quality, incorporating Key Performance Indicator (KPI) validations and comprehensive end-to-end pipeline monitoring to ensure that data remains accurate, reliable, and trustworthy throughout its journey. One of my standout achievements lies in the successful migration of a project to the Azure platform, where I orchestrated the optimization of the ETL process, resulting in an astonishing 365% improvement in execution time. Additionally, my project management skills shine through as I seamlessly coordinate tasks using Azure DevOps, aligning development efforts with the overarching project objectives. In essence, my career as an Azure Data Engineer is characterized by a steadfast dedication to efficiency, data integrity, and innovative data management. My passion for crafting elegant solutions to complex data challenges has made me a sought-after expert in the field, and I continue to push the boundaries of what is possible within the Azure ecosystem.
    Featured Skill Apache Spark
    Data Warehousing & ETL Software
    Data Scraping
    SQL Server Integration Services
    Microsoft Azure SQL Database
    Unix
    Microsoft Power BI
    Azure DevOps
    Microsoft Azure
    Databricks Platform
    ETL Pipeline
    Python
    PySpark
    SQL
    Data Integration
  • $10 hourly

    • 5.0/5
    • (2 jobs)
    • IT Professional around 6.1 years of experience in Software Development and Maintenance of Big Data projects • Possess in-depth working knowledge in all the areas of development of Big Data • Worked extensively on Technologies like Apache Spark, Databricks, Hive, Sqoop, Map Reduce, Apache Kafka applications.
    Featured Skill Apache Spark
    Sqoop
    Hive
    Apache Kafka
    SQL
    Python
    PySpark
  • $35 hourly

    • 0.0/5
    • (0 jobs)
    Specializing in full-stack web development, cloud infrastructure, and DevOps, I focus on AWS (EC2, S3, RDS, Lambda, SNS, SQS, SES, CloudFront) and GCP for designing and deploying scalable backend systems. I have expertise in Docker for containerization and Kubernetes for microservices orchestration. I develop dynamic frontends using React and TypeScript while managing data with SQLAlchemy, Snowflake, PostgreSQL, and BigQuery. My experience spans ETL pipelines, data engineering, and distributed computing with Apache Airflow, Spark, and Kafka for real-time data processing. I design scalable data architectures, optimize ETL workflows, and implement data lake and warehouse solutions. I architect cloud-native solutions using AWS Glue, Data Pipeline, and GCP’s Pub/Sub, Cloud Functions, and Dataflow. I automate infrastructure with Terraform and manage DevOps workflows, CI/CD pipelines, and infrastructure-as-code using Jenkins, AWS CodePipeline, and GitHub Actions. Additionally, I streamline monitoring and logging with Prometheus, Grafana, and ELK stack. I work with Git, GitHub, and Linux (Ubuntu) to enhance development efficiency. With expertise in Selenium and PyTest for testing and event-driven architectures, I ensure scalability, reliability, and performance in cloud-based environments.
    Featured Skill Apache Spark
    Cloud Run
    Amazon EC2
    Amazon S3
    Amazon Redshift
    Databricks Platform
    Snowflake
    Apache Kafka
    PySpark
    Apache Airflow
    Python
    Data Analysis
    ETL
    ETL Pipeline
    Data Extraction
  • $40 hourly

    • 5.0/5
    • (10 jobs)
    Hi, I am Srujan Alikanti, a seasoned ETL Developer and Data Engineer specializing in cloud platforms like AWS, Azure, and GCP. With over 18+ years of experience, I excel in building scalable ETL pipelines, data migrations, and advanced analytics solutions using Databricks, Python, and SQL. I have a strong background in integrating diverse data sources, optimizing data workflows, and delivering business intelligence solutions. Expertise: 1. ETL Development & Data Pipelines - Design and implement robust ETL pipelines using Databricks, AWS Glue, and Azure Data Factory - Optimize ETL workflows to ensure efficient data extraction, transformation, and loading across cloud platforms (AWS, Azure, GCP) - Develop end-to-end data ingestion frameworks using Python and SQL - Implement real-time and batch processing pipelines for structured and unstructured data 2. Data Engineering & Cloud Platforms - Cloud-Native Data Solutions: AWS (S3, Glue, Lambda, Athena), Azure (Data Factory, Synapse), and GCP (BigQuery, Dataflow) - Data Lake and Data Warehouse: Design and optimize data lakes and modern data warehouses (Snowflake, Databricks) - Migrate on-premise ETL systems to cloud-based data pipelines - Implement DataOps practices for CI/CD in data workflows 3. Data Migration - Platform Migration: Legacy ETL to modern cloud-based pipelines (AWS Glue, Azure Data Factory, Databricks) - Data Migration: Salesforce, HubSpot, Cloud, ERP (SAP, Oracle) - CRM & ERP Migration: Seamlessly transfer data between business-critical systems 4. Data Analytics & Business Intelligence - Data Strategy: Data modeling, integration, governance, and compliance - Business Insights: Build insightful dashboards and reports using Tableau, Power BI, and Google Data Studio - Implement advanced analytics solutions for e-commerce, healthcare, and digital marketing domains - Conduct data profiling, quality checks, and data reconciliation for accurate analytics 5. API Integration & Data Automation - Develop and maintain complex API integrations (Salesforce, Google Analytics, Shopify, Amazon MWS) - Automate data pipelines and workflows using Airflow and cloud-native services - Implement bi-directional sync and real-time data ingestion pipelines 6. Big Data & Machine Learning - Build and optimize big data workflows using Databricks and Spark - Enable data-driven decisions by deploying scalable ML models in cloud environments - Process and analyze petabyte-scale data using distributed computing frameworks 7. Software Development & Custom Solutions - Full-stack development using Python, SQL, Java, and Node.js - Design custom ETL frameworks and reusable data transformation libraries - Automate data processing tasks with Python scripts and serverless cloud functions Specialties: ETL Tools: Databricks, Talend, Matillion, Informatica, AWS Glue, Azure Data Factory Databases: Snowflake, PostgreSQL, DynamoDB, MSSQL, Neo4j, MongoDB Languages: Python, SQL, Java, Unix, HTML, Node.js, React.js Cloud Platforms: AWS (Glue, S3, Lambda, Athena), Azure (Data Factory, Synapse), GCP (BigQuery, Dataflow) Reporting Tools: Tableau, Power BI, Google Data Studio, Yellowfin BI Workflow Orchestration: Apache Airflow, AWS Step Functions, Azure Logic Apps You have the data? Great!! I can help you extract, transform, and load it using cutting-edge ETL tools like Databricks and AWS Glue. You have Big Data? Even Better!! I can build scalable, cloud-native pipelines for high-volume data processing on AWS, Azure, and GCP. You want to track KPIs? No Problem!! I can develop advanced BI dashboards and analytics reports to keep you ahead in your business. Expect integrity, excellent communication, technical proficiency, and long-term support.
    Featured Skill Apache Spark
    Amazon Web Services
    Apache Hadoop
    Microsoft Azure
    AWS Glue
    Akka
    Snowflake
    Looker Studio
    BigQuery
    Google Analytics
    Big Data
    Apache Hive
    Cloudera
    Scala
  • $10 hourly

    • 5.0/5
    • (3 jobs)
    In my dynamic 3+ year journey as an Azure Data Engineer, I've become a maestro of transformative solutions, wielding Azure's arsenal with finesse. From Synapse Analytics to Databricks, Data Factory to Power Automate, I've mastered the tools of the trade, seamlessly orchestrating data migrations and crafting workflows that redefine efficiency. Whether it's bridging the gap between MySQL, SQL Server, and Salesforce, or optimizing batch and streaming processes with Pyspark and Azure Data Factory, I thrive on turning complexity into clarity. But my impact doesn't end with data movement. I fervently advocate for automation, infusing unit-testing into Databricks workflows and championing DevOps practices that ensure resilience and agility. I'm a virtuoso in Power Platform, sculpting ecosystems where Power Apps and Automate converge, empowering teams to innovate at lightning speed. And when it comes to insights, I'm the maestro, sculpting KQL queries and crafting dashboards that illuminate the path forward. With a relentless commitment to transparency and a passion for driving cost-effective solutions, I'm poised to continue reshaping the Azure landscape, one ingenious solution at a time.
    Featured Skill Apache Spark
    Azure Cosmos DB
    Apache Kafka
    Scala
    Microsoft Azure
    Data Engineering
    pytest
    Azure DevOps
    Data Lake
    Apache Hadoop
    Microsoft Azure SQL Database
    PySpark
    Python
    Databricks Platform
    SQL
  • $15 hourly

    • 5.0/5
    • (1 job)
    I am a proactive and achievement-oriented professional offering nearly 4 years of experience in the IT industry, specializing in Data Warehousing, Data Modelling, Microsoft Azure, Python, Databricks and Business Intelligence tools with skills in designing and implementing systems to collect and ingest data from various sources, including databases, APIs, files, web scraping leveraging the cloud architectures. I am a forward thinking person with proficiency across Manufacturing, Private Equity, Health Care, Consumer Durables, Satellite Broadcasting, and SAAS companies; successfully provided comprehensive data management solutions utilizing platforms like Microsoft Azure, Snowflake, and Databricks.
    Featured Skill Apache Spark
    Amazon Web Services
    Databricks Platform
    PySpark
    Microsoft Azure
    API
    Snowflake
    Looker Studio
    Google Analytics
    Python
    SQL
    Data Analysis
    Data Interpretation
    Tableau
    Microsoft Power BI
  • $30 hourly

    • 0.0/5
    • (0 jobs)
    Passionate data engineer with more than a decade of experience on Hadoop, hive, spark, GCP and Azure
    Featured Skill Apache Spark
    Google Cloud Platform
    Microsoft Azure
    Apache Hadoop
    Databricks Platform
    Looker Studio
    Apache Hive
    BigQuery
    Big Data
  • $60 hourly

    • 0.0/5
    • (0 jobs)
    CAREER OBJECTIVE: I would describe myself as a hard-working person and a friendly individual. My motive is to learn & adapt to new technologies related to my profession, thus enhancing my innovation skills and making myself more profitable to the organization. SUMMARY: Deadline-oriented Software Tester with more than 3 years expertise in both manual and automation testing . And recent experiences of integrating the test cases and test suites of robot frame work scripts into CI/CD Pipelines . Solid history of discovering errors , resolving defects and meeting clients expectations .
    Featured Skill Apache Spark
    Apache Hive
    System Automation
    Amazon Web Services
    Data Ingestion
    Automation
    Testing
    Continuous Integration
    Apache Solr
    Apache Hadoop
    CI/CD
    Software Testing
    Software QA
    Test Results & Analysis
    Apache JMeter
  • $5 hourly

    • 0.0/5
    • (0 jobs)
    I am a data engineer with extensive experience in advanced data analysis, pipeline development, and cloud solutions. My technical proficiency includes Advanced SQL, MongoDB, Python, Linux, Tableau, C++, and R. I offer full project management from end to end, ensuring that every phase of the project is executed with precision. Regular communication is a priority for me, ensuring we stay connected throughout the project's lifecycle.
    Featured Skill Apache Spark
    Azure Cosmos DB
    Linux
    Machine Learning
    Data Analysis
    SQL
    Python Script
    Python
    Engineering & Architecture
    Data Engineering
    Apache Spark MLlib
  • $22 hourly

    • 0.0/5
    • (0 jobs)
    I am a developer experienced in Oracle database PL/SQL, Oracle SCM, Oracle Inventory Management, Oracle Order management. Also have experience as Azure data engineer with Azure data factory, Azure Databricks, Azure Synapse, Data lake, Spark, pyspark. Regular Communication is important to me so let's keep in touch
    Featured Skill Apache Spark
    Maerix Synapse
    Databricks Platform
    Database
    Data Analysis
    PySpark
    Oracle Applications
    Oracle Database
    Oracle E-Business Suite
    Oracle PLSQL
    Oracle
  • $10 hourly

    • 0.0/5
    • (0 jobs)
    1. Data Pipeline Design: Skilled in creating efficient, scalable data pipelines to transform raw data into valuable insights. 2. Big Data & Cloud Solutions: Experienced with Apache Spark, Hadoop, and cloud platforms (AWS, Google Cloud, Azure) to handle large datasets for analytics and BI. 3. Data Transformation & Integration: Proficient in SQL, Python, and tools like dbt and Talend to ensure data quality and accessibility across sources. 4. Automation & Workflow Orchestration: Skilled with Apache Airflow to automate workflows, reducing manual tasks and ensuring seamless data operations. With a technical background from BIT Mesra and a passion for computer science, I bring a results-driven approach to building reliable, performance-optimized data solutions. Let’s connect to unlock insights from your data!
    Featured Skill Apache Spark
    Data Analysis
    ETL
    Data Lake
    Python
    Scala
    Elasticsearch
    Apache Airflow
    Hive
    Apache Hadoop
  • $15 hourly

    • 0.0/5
    • (0 jobs)
    Few of the client for whome I worked are: Meta Apple JP Morgan Rakuten Reliance Telstra and many more I am a skilled Data Engineer with over 7+ years of experience in designing, building, and optimizing large-scale data solutions. My expertise lies in creating scalable data pipelines, managing big data infrastructures, and ensuring data quality for actionable business insights. I have hands-on experience with cutting-edge technologies like Apache Spark, Hadoop, Kafka, AWS, and Snowflake, enabling me to handle complex data challenges efficiently. Throughout my career, I’ve worked on high-impact projects, including e-commerce data platforms and analytics systems, where I integrated diverse data sources, optimized ETL workflows, and developed robust data models to support analytics and machine learning initiatives. My solutions have improved data processing efficiency by up to 40% and enhanced the accessibility and accuracy of data across organizations. I thrive in solving complex data problems and collaborating with cross-functional teams to deliver tailored solutions. Whether it's architecting a data pipeline, automating reporting systems, or setting up a secure and compliant data infrastructure, I bring a results-oriented approach to every project. If you’re looking for a data engineering expert to build scalable systems, streamline your data workflows, or support your analytics needs, I’d be excited to help. Let’s turn your data into actionable insights and drive business success together!
    Featured Skill Apache Spark
    Web Design
    AI Content Writing
    Content Writing
    Data Analysis
    Scala
    PySpark
    Python
    MySQL
    Oracle PLSQL
    AWS Glue
    Kubernetes
    ETL
    Apache Hadoop
    Apache Airflow
  • $50 hourly

    • 0.0/5
    • (0 jobs)
    Building mission critical Software Applications for 16 years. Solid Telematics and Connected Vehicle experience. Expertise in redesigning monolith apps using modern tools while keeping business as usual. Hands-on Programmer, Problem Solver and an Architect. 1. Full experience of SDLC with agile approach. 2. Redesign monolith apps using Event Driven + Microservices architecture 3. Introduce/Design/Develop scalable reports that allow customers to query thousands of assets analyzing millions of events. 4. Expert level Apache Kafka experience with Kafka steaming API expertise 5. Strong experience in Spring Boot and most of the Spring libraries 6. Strong experience in Apache Spark, Hive, Nifi and other Big Data tech stack 7. Strong experience normalizing and creating new RDMS designs/data models 8. Experience in creating a data lake architecture and thus implementing it 9. Experience in front end stack using React, Anuglar, NextJS and other JS based frameworks
    Featured Skill Apache Spark
    MQTT
    AWS Fargate
    React
    JavaScript
    Docker
    Grails
    Java
    Apache Kafka
    Spring Framework
    Spring Boot
    Microservice
    Software Design Pattern
    Software Design
    Software Architecture
  • $30 hourly

    • 0.0/5
    • (0 jobs)
     Expertise in writing end to end Data processing Jobs to analyze data using Spark and Hive.  Excellent knowledge in building data engineering pipelines, automating, and fine-tuning for both batch and real time data pipelines.  Have good understanding of Spark Architecture including Spark core, Spark SQL, Data frames, and Spark Streaming.  Hands on experience on Spark using Java, expertise in creating Spark RDD (Resilient Distributed Dataset), and performing transformations, actions.  Expertise in using Spark-SQL with various data sources like CSV, JSON files and apply transformation and saving into different file formats.  Hands-on experience with Amazon EC2, S3, RDS, IAM, Auto Scaling, CloudWatch, SNS, Athena, Glue, Kinesis, Lambda, EMR, Redshift, DynamoDB and other services of the AWS family.  Migrated an existing on-premises application to AWS. Used AWS services like EC2 and S3 for small data sets processing and storage, experienced in maintaining the Hadoop cluster on AWS EMR.  Built real time data pipelines by developing Kafka producers and streaming applications for consuming.  Design and develop Spark applications using Scala and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage pattern.  Good experience working on AWS-Bigdata/Hadoop Ecosystem in the implementation of Data Lake.  Strong Hadoop and platform support experience with all the entire suite of tools and services in major Hadoop Distributions - Cloudera, Amazon EMR, Azure HDInsight, and Hortonworks.  Expertise in loading and reading the data into Hive using Spark-Sql.  Developed Spark scripts by using Java shell commands as per the requirement.  Hands on experience in installing, configuration and using Hadoop Ecosystem components like HDFS, Map Reduce, HIVE, PIG, HBase, Sqoop, Flume.  Working experience in importing and exporting data using Sqoop, from HDFS to Relational Database Systems and vice-versa for further processing.  Experience on Migrating SQL database to Azure data Lake, Azure SQL Database and Azure SQL Data warehouse, GCP and controlling and granting database access and Migrating On premise databases to Azure Data Lake store using Azure Data factory.  Experience in GCP, Big Query, GCS bucket, G - cloud function, cloud dataflow, Pub/sub cloud shell, Data Proc, Stack driver.  Experience in writing programs using PySpark, Python in Azure Databricks.  Experience in writing REST APIs in Java for large-scale applications.  Extract, Transform and Load (ETL) source data into respective target tables to build Data Marts.  Experience data processing like collecting, aggregating, moving from various sources using Apache Flume and Kafka.  Experience in using the version control tools like GIT, Bit Bucket.  Experience on working with Power BI.  Good real time experience in SQL on ORACLE 11g Database.  Active involvement in all scrum ceremonies – Sprint Planning, Daily Scrum, Sprint Review and Retrospective meetings and assisted Product owner in creating and prioritizing user stories.  Extensive experience in Banking domain and Product development.  Excellent communication, interpersonal, analytical skills, and strong ability to perform as part of team.  Hard working and enthusiastic.  Excellent attitude towards learning new tools and technologies.
    Featured Skill Apache Spark
    ETL
    Sqoop
    Jenkins
    Scala
    MySQL
    Python
    Java
    Apache Impala
    Hive
    SQL
    Apache Hadoop
  • $25 hourly

    • 0.0/5
    • (0 jobs)
    Big Data Expert | Spark, Hadoop, ETL Solutions | Data Engineer Services Highly experienced Data Engineer offering robust Big Data solutions using Apache Spark, Hadoop, and ETL technologies. Proven track record of designing, developing, and deploying scalable data pipelines for various industries. **Services Include:** * **Big Data Engineering:** + Apache Spark (PySpark) + Hadoop (HDFS, YARN, MapReduce) + NoSQL databases (HBase) * **ETL Development:** * **Data Pipeline Design:** + Data warehousing solutions + Real-time data processing architectures + Data lake implementation * **Consulting and Support:** + Big Data strategy planning + Performance optimization + Troubleshooting and maintenance **Deliverables:** * Scalable Big Data solutions tailored to your business needs * Efficient ETL pipelines for data integration and transformation * Detailed documentation and knowledge transfer * Ongoing support and maintenance **Why Hire Me:** * 11+ years of experience in Big Data engineering * Proven expertise in Spark, Hadoop, and ETL technologies * Strong understanding of data warehousing and business intelligence concepts **Packages:** * **Basic:** ₹8,000 (5 hours of consulting/service) * **Standard:** ₹20,000 (20 hours of consulting/service) * **Premium:** ₹50,000 (50 hours of consulting/service) Shall I help you add this to your profiles, Guruji?
    Featured Skill Apache Spark
    Bash Programming
    Cloudera
    Hortonworks
    Apache Impala
    Oracle
    Python
    Microsoft SQL Server
    HDFS
    Hive
    PySpark
    Big Data
    Data Analysis
    ETL Pipeline
    ETL
  • $25 hourly

    • 4.6/5
    • (42 jobs)
    Core Expertise and offerings: Cloud Architiect with consulting experience to offer Operational Excellece, Security, Reliability, Performance Efficiency and Cost Optimization for Business IT. Expert in ETL tool Matillion and Cloud warehouse Snowflake. My role as Cloud Architect include to pick right architecture, evaluate cost, planning failover/High Availability/Disaster Recovery, designing models for End to End Migrations, Load Balancing for traffic management , Backup and Archival strategy for new and old infra, monitoring tools to keep track of infra health. Other Offerings: Project Management: Tracking of project activities and timely mis report to senior management with KPIs Spread Sheet/Grid Solutions: Expert in Excel VBA /Google Sheets App Scripts Certifications: I am a certified "Mastering Data Analyst" from Duke University. I am a certified "Software Engineer" From Indian Institute of Technology, Madras" in IOT, Cloud and Block Chain. Skills: Cloud: Architecting applications for the Cloud, Containers & Microservices, DevOps for the Cloud IOT: IoT Data Sources & Networking. IoT on Cloud, IoT Big Data Processing, IoT Analytics Block Chain: Fundamentals of Blockchain, Blockchain Development
    Featured Skill Apache Spark
    Data Extraction
    Spreadsheet Software
    English
    Blockchain
    Accuracy Verification
    ITIL
    Smart Contract
    Macro Programming
    MongoDB
    Amazon Web Services
    Google Cloud Platform
    Python
  • $35 hourly

    • 0.0/5
    • (0 jobs)
    Helical IT is a company specializing in data stack solutions. We do extensive work in implementation of Data Lake, Data Warehouse, Data Analytics, Data Pipeline, Business Intelligence and Generative AI Services. For providing these services we can make use of Open source tool stack (to help you reduce the licensing cost and vendor lockin), any of the most popular cloud vendors (like AWS, Azure and GCP) or using modern data stack and tools like Snowflake, Databricks, DBT, Airflow, Airbyte etc. We have experience in building the entire 3 generations of data stack and solutions which includes Traditional Data Stack - Canned Reports - BI - Designing Data Warehouse - Populating DW using ETL tools 2nd Gen Data Stack - Designing Data Lake - ETL - Data Warehouse - Business Intelligence - Data Science - ML Modern Data Stack - Data Lakehouse - ETL - Business Intelligence - Data Science - ML Some of the tools and technologies that we have experience on includes BI: Open Source [Helical Insight, Jaspersoft, Pentaho, Metabase, Superset], Proprietary [PowerBI, Tableau, Quicksight] DW Platforms: Redshift, Vertica, Big Query Data Lake / Lakehouse: Snowflake, Databricks, S3, AWS Lake, GCP, Dremio, Apache Iceberg, Hadoop Canned report: Jaspersoft, Pentaho, Helical Insight ETL/ELT: Talend, Kettle, Glue, Spark, Python Transformation: DBT, Airflow, Airbyte AI Services - Generative AI (Hugging face, Tensorflow, Pytorch, LangChain) - NLP & Chatbot development Owing to our strong technical expertise we have been technology partner of various tools which includes DBT, Snowflake, AWS, Jaspersoft, Pentaho etc. We have got multiple resources who are certified and having relevant skills. If you are looking for support or any new fatures in any of the legacy implementations or migratin to modern dat stack from one of the older generation tools or if you are a new greenfield implementation also, we at Helical can help you with the same. Over the last 10+ years we have worked with Fortune 500 clients, Government organizations, SMEs etc and have been part of 85+ DWBI implementations across various domains and geographies. - Fortune 500 - Unilever, CA Technologies, Tata Communications, Technip, Smithsdetection, Mutual of America - Unicorns: Mindtickle, Fractal Analytics, - Govt - Government of Micronesia, Government of Marshall Islands, Government of Kiribati Islands, INRA France - Energy - Vortecy, Wipro Ecoenergy - Education- University of Bridgeport, Envision Global, Nexquare, KidsXAP - Insurance - 4sightBI, Hcentive - Social Media Analytics - UnifiedSocial - HR - SyncHR, Sage Human Capital - Data Analytics - Numerify, Syntasa - Supply Chain- New Age Global, Canadian Bearings, Autoplant - FinTech- Wealthhub Solutions - Manufacturing- Unidesign Jewellery - Clinical Trial - Inductive Quotient, Radiant Sage, Reify Health Please reach out to us for learning more about our implementations
    Featured Skill Apache Spark
    Data Modeling
    GIS
    Talend Data Integration
    Snowflake
    Data Lake
    dbt
    Jaspersoft Studio
    Data Warehousing
    Big Data
    Talend Open Studio
    Pentaho
    Databricks Platform
    Apache Airflow
    Apache Hadoop
    Apache Hive
    Apache Cassandra
  • $10 hourly

    • 0.0/5
    • (0 jobs)
    With 6+ years of hands-on experience in data engineering, I specialize in building scalable, high-performance data solutions that drive business insights. My expertise spans across cloud platforms (AWS, Azure) and I have a strong focus on leveraging Databricks for big data processing, data pipelines, and advanced analytics. Key Skills & Expertise: Cloud Platforms: AWS, Azure Big Data & Analytics: Databricks, Delta Lake (data versioning & optimization), data lakes & warehouses Data Engineering: Pipeline design & optimization, data transformation, ETL processes Machine Learning Integration: Deploying ML models, batch/real-time pipelines, model serving & monitoring Collaboration: Worked with cross-functional teams (data scientists, analysts) to deliver data-driven solutions that empower machine learning initiatives Why Choose Me: Proven track record of building and optimizing end-to-end data pipelines on Databricks. Strong problem-solving skills, meticulous attention to detail, and ability to work in fast-paced, dynamic environments. Collaborative approach, ensuring your data science and analytics teams get reliable, high-quality data solutions that support strategic decision-making. I’m passionate about solving complex data challenges and helping businesses unlock the full potential of their data. If you need a skilled and reliable data engineer to take your projects to the next level, I’d love to discuss how I can help. Let’s connect and explore how I can contribute to your success! Tech Stack I use: Languages : Python, BigData, PySpark, Apache Spark Database : MySQL, Mongo DB, RDBMS, Hadoop, Dynamo DB Data Manipulation: Pandas, Numpy, PySpark, Excel, Regex, Selenium Data Visualisation: Power BI, Matplotlib, Seaborn, Excel Data Warehouse : Redshift, Snowflake, Bigquery Workflows: Airflow, Databricks Platforms : AWS, Bigdata, Databricks, Linux Versioning : Bitbucket, GitLab Certifications: Databricks Certified Data Engineer Professional Databricks Certified Data Engineer Associate
    Featured Skill Apache Spark
    Data Modeling
    Data Preprocessing
    Data Analysis
    Cost Estimate
    GitHub
    SQL Programming
    SQL
    Apache Airflow
    PySpark
    Databricks Platform
    Data Engineering
    Data Model
    Data Lake
    Python
  • $20 hourly

    • 0.0/5
    • (0 jobs)
    Project 1: Employee Time Tracking App Description: Developed a PowerApps-based time tracking app for a client to automate their employee time tracking process. The app features included clock-in and clock-out functionality, employee shift management, and real-time data tracking. The app helped the client save time and increase productivity. Project 2: Sales Dashboard App Description: Developed a PowerApps-based sales dashboard app for a client to track their sales performance. The app features included sales data visualization, drill-down functionality, and real-time data tracking. The app helped the client make data-driven decisions and improve their sales performance. Project 3: Inventory Management App Description: Developed a PowerApps-based inventory management app for a client to track their inventory levels. The app features included barcode scanning, inventory tracking, and real-time data tracking. The app helped the client reduce inventory errors and save time on manual inventory tracking.
    Featured Skill Apache Spark
    Microsoft SharePoint
    Data Engineering
    Data Science
    Data Entry
    Machine Learning
    PySpark
    Python
    Microsoft Power Automate
    Microsoft Power BI
    SQL
    Azure DevOps
    Microsoft PowerApps
  • Want to browse more freelancers?
    Sign up

How hiring on Upwork works

1. Post a job

Tell us what you need. Provide as many details as possible, but don’t worry about getting it perfect.

2. Talent comes to you

Get qualified proposals within 24 hours, and meet the candidates you’re excited about. Hire as soon as you’re ready.

3. Collaborate easily

Use Upwork to chat or video call, share files, and track project progress right from the app.

4. Payment simplified

Receive invoices and make payments through Upwork. Only pay for work you authorize.

Trusted by

How do I hire a Apache Spark Engineer near Hyderabad, on Upwork?

You can hire a Apache Spark Engineer near Hyderabad, on Upwork in four simple steps:

  • Create a job post tailored to your Apache Spark Engineer project scope. We’ll walk you through the process step by step.
  • Browse top Apache Spark Engineer talent on Upwork and invite them to your project.
  • Once the proposals start flowing in, create a shortlist of top Apache Spark Engineer profiles and interview.
  • Hire the right Apache Spark Engineer for your project from Upwork, the world’s largest work marketplace.

At Upwork, we believe talent staffing should be easy.

How much does it cost to hire a Apache Spark Engineer?

Rates charged by Apache Spark Engineers on Upwork can vary with a number of factors including experience, location, and market conditions. See hourly rates for in-demand skills on Upwork.

Why hire a Apache Spark Engineer near Hyderabad, on Upwork?

As the world’s work marketplace, we connect highly-skilled freelance Apache Spark Engineers and businesses and help them build trusted, long-term relationships so they can achieve more together. Let us help you build the dream Apache Spark Engineer team you need to succeed.

Can I hire a Apache Spark Engineer near Hyderabad, within 24 hours on Upwork?

Depending on availability and the quality of your job post, it’s entirely possible to sign up for Upwork and receive Apache Spark Engineer proposals within 24 hours of posting a job description.

Looking to hire for long-term or full-time assignments? See how Full-time hiring on Upwork simplifies the process.