Apache Spark Engineer job description template

An effective description can help you hire the best fit for your job. Check out our tips to provide details that skilled professionals are looking for.

Hire Apache Spark Engineers

Looking to find work instead?

Trusted by

Example of Apache Spark Engineer job description

Apache Spark is an open-source framework that supports both data streaming and batch processing. The engine interprets multiple programming languages, and its users can harness its power to improve data engineering, data science, and machine learning. With over 1,000 contributing software engineers and developers from hundreds of organizations, it has been leveraged by businesses across all industries to process data of any size.

Recognized as one of the world’s largest data processing clusters, even engineers with just a few years of experience are capable of learning, building, and leveraging the framework. Experienced engineers bring vast knowledge to support various functions, from enhancing processing speed to rebuilding and monitoring data pipelines. The value they add can benefit many facets of an organization.

Copy to Clipboard

The job overview

We're looking to hire a new Apache Spark engineer for our team who can help us develop and evolve a large real-time data processing system. As an expert software engineer, your problem-solving and scripting skills will help us manage business requirements to support data scientists. You'll work closely with our system designers and software developers to collaborate on interface development and data pipelines.

Responsibilities of an Apache Spark Engineer

Below are the responsibilities an Apache Spark team member:

Design and implement Spark jobs to define, schedule, monitor, and control processes
Develop and test algorithms for large-scale machine learning
Optimize Spark jobs to maximize speed and scalability while remaining data-use compliant
Manage data pipelines and acquisition processes
Perform data processing and analysis
Build machine learning models using Spark or MapReduce to visualize and present results
Work with other Spark developers and back-end data engineers to design interactive Spark pipelines
Develop REST APIs for Spark jobs

Job qualifications for an Apache Spark Engineer

Below are the qualifications for an Apache Spark engineer:

Expertise building data and processing pipelines
Familiarity with Spark engine syntax modules, including Spark SQL
Familiarity with APIs including RDD, DataFrame, Dataset, and PySpark
Fluency in programming languages including Python, Java, and Scala
Knowledge of Spark internals and streaming technology (Kafka, KSQL, etc.)
Expertise in SQL and big data processing (Hadoop ecosystems, Hive, Impala, Druid, etc.)
Familiarity with machine learning algorithms and foundations such as PyTorch
Experience with an ETL tool and expertise in managing the post-loading data
Expert in one or more distributed file systems, such as HDFS, S3, and Ceph
Familiarity with visualization tools
Familiarity with Amazon's AWS for building Apache Spark clusters

A bachelor's degree in data science, software development, and computer science isn't required for Apache Spark jobs. But having a higher certification is highly encouraged (specifically from Cloudera, MapR, or Hadoop).

Copy to Clipboard

4 steps to creating an Apache Spark Engineer job description that fits your needs

Now that you’ve become familiar with a sample job description, it’s time to make sure that it suits your own needs. Understand your business requirements and consider the following tips as you begin your search for the right Apache Spark engineer:

1. Determine what type of Apache Spark Engineer you need

Apache Spark engineers are specialists who support various functions. Some are data scientists with a goal of faster data processing, while others are responsible for monitoring, integrating, and maintaining the data pipelines. These engineers offer critical services by ensuring that your Spark jobs are well-managed, accurate, and cost-efficient.

2. Employee vs. independent contractor vs. agency

Apache Spark specialists can be hired in-house, contracted through an agency, or independently as a freelancer. Independent contractors and freelancers can fill the role of a software developer or data scientist to fulfill resourcing for short-term projects. Freelancers are responsible for their withholdings and benefits as well as training, so you should be able to find a freelancer with the skills you need, especially if you're looking for a specialist in Microsoft or AWS. Agencies have developers who can support various aspects of any job, and they'll be eager to serve as subject matter experts in any way they can. If you need an Apache Spark team member for a long-term project, it might make sense to hire a full-time employee. This commitment offers the employer rate efficiencies and usually provides the employee benefits and training.

3. Experience

The more experienced an Apache Spark engineer is, the more complex tasks they can manage for your business. Entry-level Apache Spark engineers may provide your business with a simple data stream aggregation, while a senior-level consultant will be knowledgeable at building a modular data pipeline. Some prominent data engineers may create custom integrations or data streams if you need a competitive edge.

4. Industry

Spark streaming and data science specialists have become prevalent in most industries. A skilled Apache Spark engineer can help your business optimize the world’s leading data and AI/ML tools.

Next steps

Whether you're hiring for an entry-level position or recruiting the next big data engineer, you're ready to attract qualified applicants to fill the role. Start by writing a great job description, and check out our list of top Apache Spark engineers on Upwork by exploring job boards. When you're ready to speak with candidates, be prepared with great interview questions.

Upwork is not affiliated with and does not sponsor or endorse any of the tools or services discussed in this section. These tools and services are provided only as potential options, and each reader and company should take the time needed to adequately analyze and determine the tools or services that would best fit their specific needs and situation.

Apache Spark Engineer Hiring Resources

Explore talent to hire

Learn about cost factors

4.8/5

clients rate Apache Spark Engineers based on 775 reviews

Hire Apache Spark Engineers

Apache Spark Engineers you can meet on Upwork

$80 hourly
Amar K.
- 5.0
- (28 jobs)
Bengaluru, KA
Apache Spark
API Development
Flask
Google App Engine
Software Development
Big Data
Google Cloud Platform
Amazon Web Services
BigQuery
PySpark
Apache Airflow
Data Engineering
SQL
Python
Java

𝟭𝟬+ 𝘆𝗲𝗮𝗿𝘀 𝗼𝗳 𝗘𝘅𝗽𝗲𝗿𝗶𝗲𝗻𝗰𝗲 | 𝗘𝘅𝗽𝗲𝗿𝘁-𝗩𝗲𝘁𝘁𝗲𝗱 (𝗧𝗼𝗽 𝟭%) 𝗳𝗿𝗲𝗲𝗹𝗮𝗻𝗰𝗲𝗿 | 𝗪𝗼𝗿𝗸𝗲𝗱 𝘄𝗶𝗵 𝗚𝗼𝗹𝗱𝗺𝗮𝗻 𝗦𝗮𝗰𝗵𝘀, 𝗠𝗼𝗿𝗴𝗮𝗻 𝗦𝘁𝗮𝗻𝗹𝗲𝘆, 𝗞𝗠𝗣𝗚, 𝗢𝗿𝗮𝗰𝗹𝗲 𝗲𝘁𝗰. I take pride in maintaining a 𝗽𝗲𝗿𝗳𝗲𝗰𝘁 𝗿𝗲𝗰𝗼𝗿𝗱 𝗼𝗳 𝟱-𝘀𝘁𝗮𝗿 𝗿𝗮𝘁𝗶𝗻𝗴𝘀 𝗮𝗰𝗿𝗼𝘀𝘀 𝗮𝗹𝗹 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀. My expertise is strongly backed by 𝗳𝘂𝗹𝗹-𝘀𝘁𝗮𝗰𝗸 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁 and 𝗰𝗹𝗼𝘂𝗱 𝗱𝗮𝘁𝗮 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝘀𝗸𝗶𝗹𝗹𝘀, honed through work with leading institutions. With over 10+ years of experience in Data Engineering and Programming, I bring a commitment to excellence and a passion for perfection in every project I undertake. My approach is centered around delivering not just functional, but 𝗵𝗶𝗴𝗵𝗹𝘆 𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 𝗮𝗻𝗱 𝗼𝗽𝘁𝗶𝗺𝗶𝘇𝗲𝗱 code, ensuring top-quality outputs that consistently impress my clients. My expertise combined with extensive experience on both GCP and AWS Cloud platforms, allows me to provide solutions that are not only effective but also innovative and forward-thinking. I believe in going beyond the basics, striving for excellence in every aspect of my work, and delivering results that speak for themselves. 𝗖𝗵𝗼𝗼𝘀𝗲 𝗺𝗲 𝗶𝗳 𝘆𝗼𝘂 𝗽𝗿𝗶𝗼𝗿𝗶𝘁𝗶𝘇𝗲 𝘁𝗼𝗽-𝗻𝗼𝘁𝗰𝗵 𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝗶𝗻 𝘆𝗼𝘂𝗿 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀 𝗮𝗻𝗱 𝗮𝗽𝗽𝗿𝗲𝗰𝗶𝗮𝘁𝗲 𝗮 𝗳𝗿𝗲𝗲𝗹𝗮𝗻𝗰𝗲𝗿 𝘄𝗵𝗼 𝗮𝘂𝘁𝗼𝗻𝗼𝗺𝗼𝘂𝘀𝗹𝘆 𝗺𝗮𝗸𝗲𝘀 𝗼𝗽𝘁𝗶𝗺𝗮𝗹 𝗱𝗲𝗰𝗶𝘀𝗶𝗼𝗻𝘀, 𝘀𝗲𝗲𝗸𝗶𝗻𝗴 𝗰𝗹𝗮𝗿𝗶𝗳𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀 𝗼𝗻𝗹𝘆 𝘄𝗵𝗲𝗻 𝗮𝗯𝘀𝗼𝗹𝘂𝘁𝗲𝗹𝘆 𝗻𝗲𝗰𝗲𝘀𝘀𝗮𝗿𝘆. ❝ 𝗥𝗲𝗰𝗼𝗴𝗻𝗶𝘇𝗲𝗱 𝗮𝘀 𝗨𝗽𝘄𝗼𝗿𝗸'𝘀 𝗧𝗼𝗽 𝟭% 𝗧𝗮𝗹𝗲𝗻𝘁 𝗮𝗻𝗱 𝗮𝗻 𝗲𝘅𝗽𝗲𝗿𝘁-𝘃𝗲𝘁𝘁𝗲𝗱 𝗽𝗿𝗼𝗳𝗲𝘀𝘀𝗶𝗼𝗻𝗮𝗹 ❞ 𝗔𝗿𝗲𝗮𝘀 𝗼𝗳 𝗘𝘅𝗽𝗲𝗿𝘁𝗶𝘀𝗲: - 𝗖𝗹𝗼𝘂𝗱: GCP (Google Cloud Platform), AWS (Amazon Web Services) - 𝗣𝗿𝗼𝗴𝗿𝗮𝗺𝗺𝗶𝗻𝗴 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲: Java, Scala, Python, Ruby, HTML, Javascript - 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴: Spark, Kafka, Crunch, MapReduce, Hive, HBase, AWS Glue, PySpark, BiqQuery, Snowflake, ETL, Datawarehouse, Databricks, Data Lake, Airflow, Cloudwatch 𝗖𝗹𝗼𝘂𝗱 𝗧𝗼𝗼𝗹𝘀: AWS Lambda, Cloud Functions, App Engine, Cloud Run, Datastore, EC2, S3, - 𝗗𝗲𝘃𝗢𝗽𝘀: GitHub, GitLab. BitBucket, CHEF, Docker, Kubernetes, Jenkins, Cloud Deploy, Cloud Build, - 𝗪𝗲𝗯 & 𝗔𝗣𝗜: SpringBoot, Jersey, Flask, HTML & JSP, ReactJS, Django 𝗥𝗲𝘃𝗶𝗲𝘄𝘀: ❝ Amar is a highly intelligent and experienced individual who is exceeding expectations with his service. He has very deep knowledge across the entire field of data engineering and is a very passionate individual, so I am extremely happy to have finished my data engineering project with such a responsible fantastic guy. I was able to complete my project faster than anticipated. Many thanks.... ❞ ❝ Amar is an exceptional programmer that is hard to find on Upwork. He combines top-notch technical skills in Python & Big Data, excellent work ethic, communication skills, and strong dedication to his projects. Amar systematically works to break down complex problems, plan an approach, and implement thought-out high-quality solutions. I would highly recommend Amar! ❞ ❝ Amar is a fabulous developer. He is fully committed. Is not a clock watcher. Technically very very strong. His Java and Python skills are top-notch. What I really like about him is his attitude of taking a technical challenge personally and putting in a lot of hours to solve that problem. Best yet, he does not charge the client for all those hours, He still sticks to the agreement. Very professional. It was a delight working with him. and Will reach out to him if I have a Java or Python task. ❞ With 10+ years of experience and recognition as an Expert-Vetted (Top 1%) freelancer, I’ve delivered exceptional results for top organizations like Goldman Sachs, Morgan Stanley, and KPMG. I’m confident I can be the perfect fit for your project—let’s connect to discuss how I can help achieve your goals!
$150 hourly
Thomas T.
- 5.0
- (12 jobs)
Los Angeles, CA
Apache Spark
Data Management
Business Intelligence
API Development
Amazon Redshift
Amazon Web Services
MongoDB
Data Warehousing
ETL
Node.js
Docker
AWS Glue
Apache Airflow
SQL
Python

I am a professional cloud architect, data engineer, and software developer with 18 years of solid work experience. I deliver solutions using a variety of technologies, selected based on the best fit for the task. I have experience aiding startups, offering consulting services to small and medium-sized businesses, as well as experience working on large enterprise initiatives. I am an Amazon Web Services (AWS) Certified Solutions Architect. I have expertise in data engineering and data warehouse architecture as well. I am well versed in cloud-native ETL schemes/scenarios from various source systems (SQL, NoSQL, files, streams, and web scraping). I use Infrastructure as Code tools (IaC) and am well versed in writing continuous integration/delivery (CICD) processes. Equally important are my communication skills and ability to interface with business executives, end users, and technical personnel. I strive to deliver elegant, performant solutions that provide value to my stakeholders in a "sane," supportable way. I have bachelor's degrees in Information Systems and Economics as well as a Master of Science degree in Information Management. I recently helped a client architect, develop, and grow a cloud-based advertising attribution system into a multi-million $ profit center for their company. The engagement lasted two years, in which I designed the platform from inception, conceived/deployed new capabilities, led client onboardings, and a team to run the product. The project started from loosely defined requirements, and I transformed it into a critical component of my client's business.
$50 hourly
Akshaykumar T.
- 5.0
- (3 jobs)
Pune, MAHARASHTRA
Apache Spark
Apache Cordova
Cloud Services
Analytics
PySpark
Data Science
Python
Machine Learning

Having a hands on experience on developing Analytics and Machine Learning, Data Science, Big Data and AWS Solutions.

$80 hourly
Amar K.
- 5.0
- (28 jobs)
Bengaluru, KA
Apache Spark
API Development
Flask
Google App Engine
Software Development
Big Data
Google Cloud Platform
Amazon Web Services
BigQuery
PySpark
Apache Airflow
Data Engineering
SQL
Python
Java

𝟭𝟬+ 𝘆𝗲𝗮𝗿𝘀 𝗼𝗳 𝗘𝘅𝗽𝗲𝗿𝗶𝗲𝗻𝗰𝗲 | 𝗘𝘅𝗽𝗲𝗿𝘁-𝗩𝗲𝘁𝘁𝗲𝗱 (𝗧𝗼𝗽 𝟭%) 𝗳𝗿𝗲𝗲𝗹𝗮𝗻𝗰𝗲𝗿 | 𝗪𝗼𝗿𝗸𝗲𝗱 𝘄𝗶𝗵 𝗚𝗼𝗹𝗱𝗺𝗮𝗻 𝗦𝗮𝗰𝗵𝘀, 𝗠𝗼𝗿𝗴𝗮𝗻 𝗦𝘁𝗮𝗻𝗹𝗲𝘆, 𝗞𝗠𝗣𝗚, 𝗢𝗿𝗮𝗰𝗹𝗲 𝗲𝘁𝗰. I take pride in maintaining a 𝗽𝗲𝗿𝗳𝗲𝗰𝘁 𝗿𝗲𝗰𝗼𝗿𝗱 𝗼𝗳 𝟱-𝘀𝘁𝗮𝗿 𝗿𝗮𝘁𝗶𝗻𝗴𝘀 𝗮𝗰𝗿𝗼𝘀𝘀 𝗮𝗹𝗹 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀. My expertise is strongly backed by 𝗳𝘂𝗹𝗹-𝘀𝘁𝗮𝗰𝗸 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁 and 𝗰𝗹𝗼𝘂𝗱 𝗱𝗮𝘁𝗮 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝘀𝗸𝗶𝗹𝗹𝘀, honed through work with leading institutions. With over 10+ years of experience in Data Engineering and Programming, I bring a commitment to excellence and a passion for perfection in every project I undertake. My approach is centered around delivering not just functional, but 𝗵𝗶𝗴𝗵𝗹𝘆 𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 𝗮𝗻𝗱 𝗼𝗽𝘁𝗶𝗺𝗶𝘇𝗲𝗱 code, ensuring top-quality outputs that consistently impress my clients. My expertise combined with extensive experience on both GCP and AWS Cloud platforms, allows me to provide solutions that are not only effective but also innovative and forward-thinking. I believe in going beyond the basics, striving for excellence in every aspect of my work, and delivering results that speak for themselves. 𝗖𝗵𝗼𝗼𝘀𝗲 𝗺𝗲 𝗶𝗳 𝘆𝗼𝘂 𝗽𝗿𝗶𝗼𝗿𝗶𝘁𝗶𝘇𝗲 𝘁𝗼𝗽-𝗻𝗼𝘁𝗰𝗵 𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝗶𝗻 𝘆𝗼𝘂𝗿 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀 𝗮𝗻𝗱 𝗮𝗽𝗽𝗿𝗲𝗰𝗶𝗮𝘁𝗲 𝗮 𝗳𝗿𝗲𝗲𝗹𝗮𝗻𝗰𝗲𝗿 𝘄𝗵𝗼 𝗮𝘂𝘁𝗼𝗻𝗼𝗺𝗼𝘂𝘀𝗹𝘆 𝗺𝗮𝗸𝗲𝘀 𝗼𝗽𝘁𝗶𝗺𝗮𝗹 𝗱𝗲𝗰𝗶𝘀𝗶𝗼𝗻𝘀, 𝘀𝗲𝗲𝗸𝗶𝗻𝗴 𝗰𝗹𝗮𝗿𝗶𝗳𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀 𝗼𝗻𝗹𝘆 𝘄𝗵𝗲𝗻 𝗮𝗯𝘀𝗼𝗹𝘂𝘁𝗲𝗹𝘆 𝗻𝗲𝗰𝗲𝘀𝘀𝗮𝗿𝘆. ❝ 𝗥𝗲𝗰𝗼𝗴𝗻𝗶𝘇𝗲𝗱 𝗮𝘀 𝗨𝗽𝘄𝗼𝗿𝗸'𝘀 𝗧𝗼𝗽 𝟭% 𝗧𝗮𝗹𝗲𝗻𝘁 𝗮𝗻𝗱 𝗮𝗻 𝗲𝘅𝗽𝗲𝗿𝘁-𝘃𝗲𝘁𝘁𝗲𝗱 𝗽𝗿𝗼𝗳𝗲𝘀𝘀𝗶𝗼𝗻𝗮𝗹 ❞ 𝗔𝗿𝗲𝗮𝘀 𝗼𝗳 𝗘𝘅𝗽𝗲𝗿𝘁𝗶𝘀𝗲: - 𝗖𝗹𝗼𝘂𝗱: GCP (Google Cloud Platform), AWS (Amazon Web Services) - 𝗣𝗿𝗼𝗴𝗿𝗮𝗺𝗺𝗶𝗻𝗴 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲: Java, Scala, Python, Ruby, HTML, Javascript - 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴: Spark, Kafka, Crunch, MapReduce, Hive, HBase, AWS Glue, PySpark, BiqQuery, Snowflake, ETL, Datawarehouse, Databricks, Data Lake, Airflow, Cloudwatch 𝗖𝗹𝗼𝘂𝗱 𝗧𝗼𝗼𝗹𝘀: AWS Lambda, Cloud Functions, App Engine, Cloud Run, Datastore, EC2, S3, - 𝗗𝗲𝘃𝗢𝗽𝘀: GitHub, GitLab. BitBucket, CHEF, Docker, Kubernetes, Jenkins, Cloud Deploy, Cloud Build, - 𝗪𝗲𝗯 & 𝗔𝗣𝗜: SpringBoot, Jersey, Flask, HTML & JSP, ReactJS, Django 𝗥𝗲𝘃𝗶𝗲𝘄𝘀: ❝ Amar is a highly intelligent and experienced individual who is exceeding expectations with his service. He has very deep knowledge across the entire field of data engineering and is a very passionate individual, so I am extremely happy to have finished my data engineering project with such a responsible fantastic guy. I was able to complete my project faster than anticipated. Many thanks.... ❞ ❝ Amar is an exceptional programmer that is hard to find on Upwork. He combines top-notch technical skills in Python & Big Data, excellent work ethic, communication skills, and strong dedication to his projects. Amar systematically works to break down complex problems, plan an approach, and implement thought-out high-quality solutions. I would highly recommend Amar! ❞ ❝ Amar is a fabulous developer. He is fully committed. Is not a clock watcher. Technically very very strong. His Java and Python skills are top-notch. What I really like about him is his attitude of taking a technical challenge personally and putting in a lot of hours to solve that problem. Best yet, he does not charge the client for all those hours, He still sticks to the agreement. Very professional. It was a delight working with him. and Will reach out to him if I have a Java or Python task. ❞ With 10+ years of experience and recognition as an Expert-Vetted (Top 1%) freelancer, I’ve delivered exceptional results for top organizations like Goldman Sachs, Morgan Stanley, and KPMG. I’m confident I can be the perfect fit for your project—let’s connect to discuss how I can help achieve your goals!
$150 hourly
Thomas T.
- 5.0
- (12 jobs)
Los Angeles, CA
Apache Spark
Data Management
Business Intelligence
API Development
Amazon Redshift
Amazon Web Services
MongoDB
Data Warehousing
ETL
Node.js
Docker
AWS Glue
Apache Airflow
SQL
Python

I am a professional cloud architect, data engineer, and software developer with 18 years of solid work experience. I deliver solutions using a variety of technologies, selected based on the best fit for the task. I have experience aiding startups, offering consulting services to small and medium-sized businesses, as well as experience working on large enterprise initiatives. I am an Amazon Web Services (AWS) Certified Solutions Architect. I have expertise in data engineering and data warehouse architecture as well. I am well versed in cloud-native ETL schemes/scenarios from various source systems (SQL, NoSQL, files, streams, and web scraping). I use Infrastructure as Code tools (IaC) and am well versed in writing continuous integration/delivery (CICD) processes. Equally important are my communication skills and ability to interface with business executives, end users, and technical personnel. I strive to deliver elegant, performant solutions that provide value to my stakeholders in a "sane," supportable way. I have bachelor's degrees in Information Systems and Economics as well as a Master of Science degree in Information Management. I recently helped a client architect, develop, and grow a cloud-based advertising attribution system into a multi-million $ profit center for their company. The engagement lasted two years, in which I designed the platform from inception, conceived/deployed new capabilities, led client onboardings, and a team to run the product. The project started from loosely defined requirements, and I transformed it into a critical component of my client's business.
$50 hourly
Akshaykumar T.
- 5.0
- (3 jobs)
Pune, MAHARASHTRA
Apache Spark
Apache Cordova
Cloud Services
Analytics
PySpark
Data Science
Python
Machine Learning

Having a hands on experience on developing Analytics and Machine Learning, Data Science, Big Data and AWS Solutions.
$35 hourly
Hassan U.
- 5.0
- (12 jobs)
Karachi, SD
Apache Spark
Amazon RDS
Apache Airflow
Amazon S3
Amazon Redshift
dbt
Python
SQL
Data Engineering

As a professional Data Engineering enthusiast with a Bachelor's degree in Computer Science, I am an expert in data extraction, modeling, reporting, and database backups and restoration processes. My technical skills include: Experienced in PySpark and Python for data manipulation and analysis Proficient in Python libraries such as Numpy, Pandas, BeautifulSoup4, requests, Pymongo, and Plotly Experienced in Hadoop Imapla for writing complex SQL queries Skilled in SQL databases such as Postgre SQL, MariaDB, SparkSQL,MySQL Server with experience in MariaDB replication and server installations Experience with NoSQL databases such as MongoDB with experience in MongoDB aggregation pipelines, MongoDB replication and server installations Proficient in MS Excel
$150 hourly
Dan S.
- 5.0
- (17 jobs)
Corvallis, OR
Apache Spark
API
Data Analysis
Database
Amazon Web Services
Business Analysis
Snowflake
Databricks Platform
ETL Pipeline
Python
Apache Airflow
Dashboard
Tableau
SQL

As a Data and Business Intelligence Engineer, I strive to deliver consulting and freelance data engineering services, with a focus on overseeing and executing projects in alignment with customer needs. With services encompassing the full data journey, I create and implement robust data foundations that streamline process development and enable leaders to execute rapid business decisions. Three categories of service include: • Consulting: Data Strategy Development, Data & Reporting Solution Architecture, and Process Development. • Data Products/Engineering: Dashboard Development & Reporting, Data Pipelines (ETL), Process Automation, and Data Collection. • Analytics: Key Performance Indicators (KPIs), Metrics, Data Analysis, and Business Process Analysis. Leveraging over eight years of experience in business intelligence, data visualization, business analysis, and requirements analysis, I build data pipelines and translate data into actionable insights that provide a competitive edge. Tools of choice include Amazon Web Services (AWS), Databricks, Snowflake, Kafka, Snowpipe Streams, Airflow, Tableau/PowerBI, SQL, NoSQL, APIs, Python, and Spark/PySpark. Let me know what I can do for YOU!
$60 hourly
Azamat A.
- 5.0
- (2 jobs)
Kenosha, WI
Apache Spark
Jakarta EE
Android SDK
Android App Development
Data Lake
Data Modeling
Amazon Web Services
Microsoft Azure
AWS Lambda
AWS Glue
PySpark
ETL
Data Engineering
Machine Learning
Databricks Platform
SQL
Java
Python

ABOUT ME: I am Lead Data Engineer with strong software development background. I have over 10 years of professional experience in IT, 7 years of which in Data Engineering. I have MS in Software Engineering from DePaul University (Chicago, IL USA) WHAT I CAN DO FOR YOU: Having worked as a Lead Data Engineer in Fortune 500 big enterprises, I can help startups with with *developing comprehensive data governance and security strategies, *designing and implementing cloud data platforms (Azure, AWS, Databricks) * data warehouse modelling * data lake/data lakehouse modelling *cost optimization of data and ML pipelines *performance optimization of data and ML pipelines TECHNICAL SKILLS Python| Java| Scala| PySpark| Apache Spark| Apache Airflow| Databricks| AWS| Azure| AWS EMR| AWS GLUE | Azure Datafactory | Azure Synapse
$175 hourly
Joshua S.
- 5.0
- (4 jobs)
Taylor, TX
Apache Spark
YARN
Apache Hadoop
Big Data
Apache Zookeeper
TensorFlow
Apache NiFi
Apache Kafka
Artificial Neural Network
Artificial Intelligence

Mr. Joshua B. Seagroves is a seasoned professional having served as an Enterprise Architect/Senior Data Engineer for multiple Fortune 100 Companies. With a successful track record as a startup founder and CTO, Mr. Seagroves brings a wealth of experience to his role, specializing in the strategic design, development, and implementation of advanced technology systems. Throughout his career, Mr. Seagroves has demonstrated expertise in architecting and delivering cutting-edge solutions, particularly in the realm of data engineering and sciences. He has successfully spearheaded the implementation of multiple such systems and applications for a diverse range of clients. As part of his current responsibilities, Mr. Seagroves actively contributes to the prototyping and research efforts in the field of data engineering/data science, specifically in the development of operational systems for critical mission systems. Leveraging his extensive background in architecture and software modeling methodologies, he has consistently led and collaborated with multidisciplinary teams, successfully integrating various distributed computing technologies, including Hadoop, NiFi, HBase, Accumulo, and MongoDB. Mr. Seagroves' exceptional professional achievements and extensive experience make him a highly sought-after expert in his field. His comprehensive knowledge and hands-on expertise in advanced technology systems and big data make him a valuable asset to any organization.
$50 hourly
POLYCHRONIS A.
- 5.0
- (45 jobs)
Athens, ATTICA
Apache Spark
ETL Pipeline
API
Web Crawling
Big Data
Data Scraping
Machine Learning
Python

I have great experience in web scraping and ETL, mainly using Python and the panda's library. I am familiar with proxies and many scraping techniques. Also, running them on the cloud is my forte, as I am familiar with many cloud services. Finally, I also have experience in Big Data and machine learning using Apache Spark (both Scala and Python). I have acquired that from my job as a freelancer, in which I applied machine learning algorithms to economic data (cryptocurrency).
$70 hourly
Aleksandr B.
- 5.0
- (2 jobs)
Alvsjo, AB
Apache Spark
Data Quality Assessment
Big Data
Software Testing
React
TypeScript
Python
Robot Framework
Selenium WebDriver
Automated Testing
Functional Testing

- Lead QA Automation professional with 8+ years of experience in test process optimization in GCP and AWS environments. - Enhanced CI pipeline performance, advocated for TestCases-as-a-code, and achieved high automation coverage. - Proficient in TypeScript, Python, Groovy, Java, Scala, and tools like WebdriverIO, Mocha, Allure, and Robot Framework. - Specialized in Big Data and Machine Learning with a focus on Data Quality, using AWS, Deeque, Great Expectations, Hadoop, Spark, Airflow, and Kubernetes.
$80 hourly
Arooj Ahmed Q.
- 4.9
- (32 jobs)
Hamilton, ON
Apache Spark
Android Studio
PostgreSQL Programming
Artificial Intelligence
IBM Watson
SQLite Programming
PySpark
Django
Deep Neural Network
Flask
Tableau
Python
Data Science
Java
Machine Learning Model

I have been working on variety of projects which involves project management, coding, machine learning, neural networks and data presentation. I am well versed with ML tools, Cloud based applications and data exploration.
$80 hourly
Alexander D.
- 5.0
- (9 jobs)
Haifa, HA
Apache Spark
Java
DataTables
Data Management
Apache Hadoop
Pentaho
BigQuery
Apache Airflow
ETL Pipeline
Python
SQL
Scala
ETL

Development experience in information management solutions, ETL processes, database design and storage systems; Responsible, able to work and solve problems independently. Software Developer, Integration process Architect Envion Software Creating a Hadoop cluster system to process heterogeneous data (ETL, Hadoop cluster, RDF/SparQL, NoSQL DB, IBM DashDB) ETL processes for big amount of database DataWarehouses creation and support Database Developer and Data Scientist A software development company Programming Analytics Stream processing Associate Professor Saint-Petersburg State University Member of the Database and Information Management Research Group
$35 hourly
Huanqing Z.
- 4.9
- (48 jobs)
Shenzhen, GUANGDONG
Apache Spark
React
Java
JavaScript
Scala
Elasticsearch
Web Development
Docker
OpenLayers
D3.js
Rust
Spring Boot
Apache Flink
Apache Kafka
Apache Hadoop

I'm Fusion Zhu, with over 10 years of experience in Java development, including 5 years focusing on Big Data Processing and Visualisation using Java, Scala, JavaScript, HTML5, Apache Spark, Apache Hadoop, Apache Hive, Apache Flume, Apache Hbase, Storm, Kafka, DataX, and ECharts. Throughout my career: I've assisted employers in data ingestion from various sources such as RDBMS, NoSQL databases, and files by developing utilities on OSS platforms like DataXServer (open-source on GitHub) and Realtime Page Click Statistical System (refer to Portfolio section) as a Big Data Developer. I've played a key role in building Big Data Platforms using technologies like Hadoop, Spark, Hive, HBase, Flink, Kafka, and ElasticSearch as a Big Data Architect. I've designed and developed Web Applications including e-commerce and Report Systems using Java, Scala, HTML5, JavaScript, CSS, Spring, Akka, Mybatis, D3JS, ExtJS, JQuery, ReactJs, ECharts, and Bootstrap CSS as a Java & Front-end Developer. I've managed full-stack teams (Java, Front-end, QA, and Operation) effectively as a Team Leader. Furthermore, I possess extensive skills and experience in Microservice design & architecture, Container Cloud (Docker, Kubernetes), Rust, and Linux. If you're seeking a reputable and reliable professional who consistently delivers results, I'm the one you're looking for. Thank you for visiting my profile, and I look forward to hearing from you!
$45 hourly
Girish P.
- 5.0
- (10 jobs)
Coimbatore, TAMIL NADU
Apache Spark
Keras
AWS Lambda
Microsoft Azure
Natural Language Processing
Flask
TensorFlow
Bash Programming
Machine Learning
Snowflake
Elasticsearch
Apache Kafka
Python
Scala

I have 7+ years of experience as a Software engineer. I love working with big data, building ML pipelines, and data engineering in general. I have designed and developed scalable cloud services on Azure as well as AWS. I also contribute to open source projects (I have co-authored a python package called nbQA). I am open to exploring exciting long-term projects. My hourly rate could vary depending on the project. If you feel my skills match your requirement feel free to invite me for the job. Languages ========= Python, Scala, C#, Typescript, Bash Frameworks ========= Apache Spark, Scikit-learn, Pandas, Keras, Tensorflow, FastAPI, Spacy, Scrapy Cloud ===== AWS Lambda, AWS Fargate, AWS Sagemaker, Azure Functions, Azure Data Factory, Azure Databricks, and various other Azure services MISC ==== ElasticSearch, Postgres, Redis, Kafka, Docker, Docker Swarm I am experienced in mentoring engineers to pick up and apply new technology stacks. Feel free to reach out to me if you need mentoring/help with any of the technologies listed below.
$60 hourly
Joaquim V.
- 5.0
- (4 jobs)
Amora, SETÚBAL
Apache Spark
ETL Pipeline
Amazon Web Services
Web Scraping
API
Kubernetes
Terraform
PySpark
AWS Lambda
Apache Hadoop
Python
pandas
Apache Hive

Over the past years I have been gathering knowledge of all things data. Throughout my career I have successfully merged the concerns of data processing with those of software development, delivering datasets and tools with immense added value for my employers. As of late I have increasingly adopted the philosophy of DevOps, not only managing data transformation pipelines, but also their life-cycle and that of their supporting infrastructure, most notably by the use of Terraform in combination with AWS. I am hoping to capitalize on my accumulated expertise in a way that would not be possible on a long term job, delivering great value to individuals and companies that are willing to invest in order to reap excellent results. I am looking for projects that require a wide range of expertise and a capacity to think outside the box, projects with hard and challenging projects. I am also a fan of automation so projects that aim at a software solution for repetitive tasks (either for increased efficiency or scale) are also welcome. I hope my profile fits your requirements and I'm looking forward to hearing from interesting clients.
$350 hourly
Michael M.
- 5.0
- (34 jobs)
Brigham City, UT
Apache Spark
Large Language Model
Visual Basic for Applications
Modeling
Forecasting
ChatGPT
Natural Language Processing
Machine Learning
Python Scikit-Learn
Microsoft Excel
SQL
TensorFlow
Python

"Michael is just FANTASTIC. He is by far the best freelancer I have worked with over the past four years. He makes the process so seamless." Ranked in the top 1% of freelancers, member of the Upwork vetted expert program, and over 12 years experience. Please reach out to me for any of your AI/ML & Data Science Needs. Please see modelforge.ai for more information.
$50 hourly
Phuoc N.
- 5.0
- (3 jobs)
Ho Chi Minh City, HO CHI MINH CITY
Apache Spark
Back-End Development
Microsoft Azure
Microsoft Power BI
D3.js
ETL
Data Visualization
SQL
Keras
R
TensorFlow
Python

Hello, I'm Phuoc (pronounced /fuəʊk/), a seasoned Full-stack Data Analyst with 7 years of expertise in crafting creative, user-friendly dashboards that tell a story. My specialization lies in Data Visualization, Data Warehouse Architecture, ETL processes, and particularly in leveraging AI and Machine Learning to drive data insights. In my capacity as the Director of a tech company in the US, I've honed strong capabilities in AI and ML, applying these technologies to solve complex business challenges. Academically, I am furthering my data science knowledge by pursuing a Master's in Data Science at Harvard University, complemented by a Micromaster in Data Science from MIT. While my Upwork history may appear modest, I have been actively engaged in full-time roles and consultation, offering advanced data analysis and training to top-tier clients and companies worldwide. My experience spans various sectors, including real estate, healthcare, technology, and consulting. My approach is not just to execute tasks, but to provide insightful, AI-driven consultation tailored to your specific needs. This methodology is rooted in best practices developed through years of collaboration with leading companies and a deep understanding of AI and ML technologies. I am confident in my ability to help you achieve your business objectives through insightful dashboards, actionable insights, and sophisticated AI and ML applications. Snapshot of My Expertise: Experience: 7+ years as a Full-stack Data Analyst; Director at a US Tech Company. AI and ML Expertise: Strong capability in applying AI and ML for innovative data solutions. Academic Credentials: Pursuing a Master's in Data Science at Harvard; Micromaster in Data Science from MIT. Clientele: Multi-million-dollar corporations. Commitment: 100% dedication to project development; 4 hours free trial. Core Competencies: Data Transformation Data Visualization Data Warehouse Architecture AI & Machine Learning Solutions Business & Power Apps, Form Checklists Technological Proficiency: Power BI, Tableau, Google Data Studio Power Platform (Automate, Apps, etc.) Azure SQL, Synapse, Function; other data solutions Open-source platforms: Apache Spark, Kafka Python: Pandas, Tensorflow, Fast API SQL: Snowflake, Postgres, BigQuery I am committed to continual skill enhancement and am eager to discuss how my expertise aligns with your project needs. Let's Connect! Please feel free to reach out for more information or to initiate collaboration. Warm regards, Phuoc
$75 hourly
Jason M.
- 5.0
- (5 jobs)
Little Rock, AR
Apache Spark
R
Serverless Stack
React
Apache Hadoop
Java
Cloudera
AWS Lambda
Apache Impala
R Hadoop
Bash Programming
PostgreSQL
Python
AWS Development
Apache Hive

Tool-oriented data science professional with extensive experience supporting multiple clients in Hadoop and Kubernetes environments, deployed with Cloudera Hadoop on-premise and Databricks in AWS. My passion is client adoption and success, with a focus on usability. With my computer science and applied math background, I have been able to fill the gap between platform engineers and users, continuously pushing for product enhancements. As a result, I have continued to create innovative solutions for clients in an environment where use-cases continue to evolve every day. I find fulfillment in being able to drive the direction of a solution in a way that allows both client and support teams to have open lanes of communication, creating success and growth. I enjoy working in a diverse environment that pushes me to learn new things. I'm interested in working on emerging solutions as data science continues to evolve.
$40 hourly
Muhammad U.
- 4.6
- (14 jobs)
Lahore, PB
Apache Spark
React Native
Mobile App
Flutter
Spring Framework
React
TypeScript
Angular
Spring Boot
Node.js

I am a diligent and passionate full stack developer with extensive experience in developing web and mobile applications using different technology stacks including Java/Angular, MEAN, React and React native. I am skilled in following tools/technologies/frameworks: Front-end : Angular, React, React native, Angular Material, Bootstrap, jQuery, AJAX, Html, JavaScript, CSS Back-end : Nodejs, Spring Boot Databases: MongoDB, PostgreSQL, Oracle, SQL Server, MySQL Unit/integration tests: JUnit, Mockito, Jasmine, Karma, NUnit Others Skills: 508 compliance/web accessibility, internationalization/localization.
$35 hourly
Rakesh D.
- 5.0
- (12 jobs)
Pune, MAHARASHTRA
Apache Spark
C++
Java
Scala
Apache Hadoop
Python
Apache Cassandra
Oracle PLSQL
Apache Hive
Cloudera
Google Cloud Platform

I have 18+ years of experience in software development in Telecom, Banking, and Healthcare domains. Primary skillsets include Big Data eco-systems (Apache Spark, Hive, Map Reduce, Cassandra), Scala, Core Java, Python, C++. I am well versed in designing and implementing Big data solutions, ETL and Data Pipelines, Serverless and event-driven architectures on Google Cloud Platform (GCP), and Cloudera Hadoop 5.5. I like to work with organizations to develop sustainable, scalable, and modern data-oriented software systems. - Keen eye on scalability, sustainability of the solution - Can come up with maintainable & good object-oriented designs quickly - Highly experienced in seamlessly working with remote teams effectively - Aptitude for recognizing business requirements and solving the root cause of the problem - Can quickly learn new technologies Sound experience in following technology stacks: Big Data: Apache Spark, Spark Streaming, HDFS, Hadoop MR, Hive, Apache Kafka, Cassandra, Google Cloud Platform (Dataproc, Cloud storage, Cloud Function, Data Store, Pub/Sub), Cloudera Hadoop 5.x Languages: Scala, Python, Java, C++, C, Scala with Akka and Play frameworks Build Tools: Sbt, Maven Databases: Postgres, Oracle, MongoDB/CosmosDB, Cassandra, Hive GCP Services: GCS, DataProc, Cloud functions, Pub/Sub, Data-store, BigQuery AWS Services: S3, VM, VM Auto-scaling Group, EMR, S3 Java APIs, Redshift Azure Services: Blob, VM, VM scale-set, Blob Java APIs, Synapse Other Tools/Technologies: Dockerization, Terraform Worked with different types of Input & Storage formats: CSV, XML, JSON file, Mongodb, Parquet, ORC
$35 hourly
Gideon A.
- 4.9
- (5 jobs)
Ile-Ife, OS
Apache Spark
Selenium
Amazon Web Services
Data Analysis
BigQuery
Data Extraction
AWS Glue
Web Crawling
Data Engineering
ETL Pipeline
Scrapy
Microsoft Power BI
SQL
Data Science
Python

Highly skilled and experienced data professional with over three years of industry experience in automating data-driven reports, data extraction, cleaning, transformations, visualizations, and application development. I have a proven track record of delivering high-quality data solutions for companies in various industries with a successful project background writing ETL pipelines for operational and analytic purposes. Need to confirm my fitness for your project? I provide free 45 minutes, no-obligation consultations. Service Summary: ✔️ Data analytics using Python and R. Examples include - Building machine learning models for academic research - Providing technical assistance in research ✔️ Data processes Automation. Examples include: - Building ETL pipeline and pipeline optimization - Creating R and Python scripts to extract and scrape data from APIs - Performing Statistical Analysis and scientific programming - Automating data entry forms to promote data quality and efficiency - Writing SQL queries to manipulate and extract your data ✔️ Interactive Data Visualization and Dashboards. Examples include: - KPI dashboards, - Tableau dashboards - PowerBI dashboards - email dashboards ✔️ Development of data-driven desktop and web applications using Streamlit and Tkinter I also suggest additional and best ways to execute your project. In addition to my technical skills, I am a strong communicator and enjoy interacting with executives, managers, and subject matter experts professionally, whether in large meetings or one-on-one interactions, to understand their needs and deliver solutions that meet their goals. I am a quick learner and always eager to take on new challenges. I am confident that my skills and experience will help to deliver measurable results that meet and exceeds your expectations. Thank you for reading and I look forward to working with you.
$125 hourly
Chisom E.
- 4.8
- (14 jobs)
Dallas, TX
Apache Spark
Java
Apache Hadoop
Amazon Web Services
Snowflake
Microsoft Azure
Google Cloud Platform
Database Management
Linux
ETL
API Integration
Scala
SQL
Python

🏆 Achieved Top-Rated Freelancer status (Top 10%) with a proven track record of success. Past experience: Twitter, Spotify, & PwC. I am a certified data engineer & software developer with 5+ years of experience. I am familiar with almost all major tech stacks on data science/engineering and app development. If you require support in your projects, please do get in touch. Programming Languages: Python | Java | Scala | C++ | Rust | SQL | Bash Big Data: Airflow | Hadoop | MapReduce | Hive | Spark | Iceberg | Presto | Trino | Scio | Databricks Cloud: GCP | AWS | Azure | Cloudera Backend: Spring Boot | FastAPI | Flask AI/ML: Pytorch | ChatGPT | Kubeflow | Onnx | Spacy | Vertex AI Streaming: Apache Beam | Apache Flink | Apache Kafka | Spark Streaming SQL Databases: MSSQL | Postgres | MySql | BigQuery | Snowflake | Redshift | Teradata NoSQL Databases: Bigtable | Cassandra | HBase | MongoDB | Elasticsearch Devops: Terraform | Docker | Git | Kubernetes | Linux | Github Actions | Jenkins | Gitlab
$35 hourly
Vignesh I.
- 5.0
- (32 jobs)
Chennai, TAMIL NADU
Apache Spark
SQL
AWS Glue
PySpark
Apache Cassandra
ETL Pipeline
Apache Hive
Apache NiFi
Apache Kafka
Big Data
Apache Hadoop
Scala

Seasoned data engineer with over 11 years of experience in building sophisticated and reliable ETL applications using Big Data and cloud stacks (Azure and AWS). TOP RATED PLUS . Collaborated with over 20 clients, accumulating more than 2000 hours on Upwork. 🏆 Expert in creating robust, scalable and cost-effective solutions using Big Data technologies for past 9 years. 🏆 The main areas of expertise are: 📍 Big data - Apache Spark, Spark Streaming, Hadoop, Kafka, Kafka Streams, Trino, HDFS, Hive, Solr, Airflow, Sqoop, NiFi, Flink 📍 AWS Cloud Services - AWS S3, AWS EC2, AWS Glue, AWS RedShift, AWS SQS, AWS RDS, AWS EMR 📍 Azure Cloud Services - Azure Data Factory, Azure Databricks, Azure HDInsights, Azure SQL 📍 Google Cloud Services - GCP DataProc 📍 Search Engine - Apache Solr 📍 NoSQL - HBase, Cassandra, MongoDB 📍 Platform - Data Warehousing, Data lake 📍 Visualization - Power BI 📍 Distributions - Cloudera 📍 DevOps - Jenkins 📍 Accelerators - Data Quality, Data Curation, Data Catalog
$60 hourly
Lakshmi R.
- 5.0
- (7 jobs)
Chennai, TAMIL NADU
Apache Spark
Functional Programming
React
Big Data
Apache Kafka
Akka
Apache Cassandra
Amazon DynamoDB
Databricks Platform
Machine Learning
Python
Scala
JavaScript

Senior Software Engineer with 7 years of experience in functional programming, machine learning, AI & BigData. Also got front-end experience building websites and tools.
$35 hourly
Junaid A.
- 5.0
- (15 jobs)
Islamabad, IS
Apache Spark
Generative AI
ETL Pipeline
PySpark
Neural Network
ChatGPT
Data Science
SQL
Deep Learning
Machine Learning
Computer Vision
Python
TensorFlow
PyTorch
Natural Language Processing

"Junaid was the perfect choice for my project, and excellent to work with. He is a strong ML developer, efficient, and his productivity is wonderful. He takes the time to explain things to me in a way I can understand, and is the type to not only double, but triple check his work to ensure it is high quality. His work ethic speaks volumes. I am well pleased with his work, and look forward to the long lasting business relationship we have cultivated. I couldn't recommend him enough!" Are you looking to speed up the business decisions and leverage data to do so? Are you looking to identify objects in pictures? Are you looking to generate automated responses for your customers with a chatbot? --- Keep reading --- Core Services I offer are: ✔️A/B Testing ✔️Image Classification ✔️Object Detection with state-of-the-art models like YOLOV5 ✔️Recommendation Systems ✔️Natural Language Processing ✔️Natural Language Translation ✔️Chatbots ✔️Data Analysis, Dashboards, Data Visualization All you do is explain to me your goal and I will do the heavy lifting. The process: 1️⃣Clearly define and understand your goal (Your involvement will be here) 2️⃣Data Collection and Integration 3️⃣Data Cleaning, Data Analysis and extracting useful Insights 4️⃣Feature Engineering 5️⃣Build the model and optimize it to meet your goals 6️⃣Deploy the model Models and Tools to solve your problems: ✔️LLM Fine-Tuning ✔️Convolutional Neural Networks ✔️Generative AI ✔️Long Short-Term Memory (LSTM) ✔️Gradient Boosters -- XGBoost ✔️Support Vector Machine ✔️Principal Component Analysis ✔️Python Language (Pandas, NumPy, sklearn, TensorFlow, Keras, matplotlib) ✔️SQL (Microsoft SQL Server) ✔️Microsoft Power BI (For Dashboards and clickable reports) Here's why we should be starting right now ✅Self-initiative: This has been my core strength for the past 5 years to start projects without much guidance and finish them successfully. ✅Understanding: I will thoroughly understand your goals and problems and make sure that you know that I understand them. ✅Commitment: to achieving your business goals, to solving your problems. ✅Communication: Prompt, smooth and fluent communication helps me keep my clients informed and updated. I am committed to updating my clients twice a day. ✅Working with your team: I can be an effective team member and make sure stakeholders and other people are on board when solving the problem. Technologies I am experienced in Python - SQL - sklearn - pytorch - tensorflow - keras - Apache Spark - Apache Kafka - Apache Flink - AWS SageMaker Studio - GCP Vertex AI - Azuer ML Studio - Django - Docker - nginx I can be a great investment for your money if I work on your project. I am willing to put whatever it takes to deliver you "100%" results to satisfy you and earn a 5-star review from you.
$45 hourly
Eniko V.
- 5.0
- (7 jobs)
London, ENGLAND
Apache Spark
AWS Lambda
Terraform
Snowflake
Data Ingestion
Grafana
SQL
AWS Glue
Amazon ECS
Python
dbt
CI/CD
Data Modeling
Apache Hadoop

Data engineer with extensive commercial experience in designing and building cloud native data solutions. Experience & Skills: Python application development ✅ Developing ETL and ELT applications ✅ Reading and writing files ✅ Data manipulation using Pyspark and Pandas ✅ Developing applications on Docker ✅ Writing pytest test cases SQL/Data Warehousing ✅ Snowflake, BigQuery, RDS (PostgreSQL, MySQL, Aurora) ✅ Data warehousing and modelling ✅ Writing complex queries and metrics ✅ Creating dbt models Infrastructure ✅ Serverless architecture design ✅ Event driven architecture design using SNS and SQS ✅ AWS Glue applications with event trigger or cron schedule, including crawlers and Athena table integration ✅ AWS ECS tasks and services to run dockerized applications ✅ AWS Batch jobs to run dockerized applications ✅ AWS Lambda functions with event trigger or cron schedule ✅ Static website hosting on S3 with CDN ✅ AWS EMR to run Apache Spark applications ✅ All infrastructure is provisioned using Terraform Monitoring and alerting ✅ Job monitoring and alerting using Cloudwatch metrics and Grafana CI/CD ✅ CircleCI, GoCD, GitLab CI/CD Version Control ✅ GitHub, GitLab
$50 hourly
Abha K.
- 5.0
- (8 jobs)
Bhilwara, RJ
Apache Spark
Apache NiFi
PySpark
Databricks Platform
ETL Pipeline
Big Data
Grafana
Kibana
Apache Kafka
PostgreSQL
Microsoft Azure
MongoDB
Scala
Python
Elasticsearch
Google Cloud Platform
Amazon Web Services

"She is very good in coding. She is the best and to go person for any hadoop or nifi requirements." "Abha is a star; have successfully handed the project in a very professional manner. I will definitely be working with Abha again; I am very happy with the quality of the work. 🙏" "Abha Kabra is one of the most talented programmers I have ever meet in Upwork. Her communication was top-notch, she met all deadlines, a skilled developer and super fast on any task was given to her. Perfect work is done. Would re-hire and highly recommended!!" Highly skilled and experienced Bigdata engineer with over 6 years of experience in the field. With a strong background in Analysis, Data Migration, Design, and Development of Big Data and Hadoop based Projects using technologies like following: ✅ Apache spark with Scala & python ✅ Apache NiFi ✅ Apache Kafka ✅ Apache Airflow ✅ ElasticSearch ✅ Logstash ✅ Kibana ✅ Mongodb ✅ Grafana ✅ Azure data factory ✅ Azure pipelines ✅ Azure databricks ✅ AWS EMR ✅ AWS S3 ✅ AWS Glue ✅ AWS Lambda ✅ GCP ✅ cloud functions ✅ PostgreSql ✅ MySql ✅ Oracle ✅ MongoDB ✅ Ansible ✅ Terraform ✅ Logo/Book Cover Design ✅ Technical Blog writing A proven track record of delivering high-quality work that meets or exceeds client expectations. Deep understanding of Energy-Related data, IoT devices, Hospitality industry, Retail Market, Ad-tech, Data encryptions-related projects, and has worked with a wide range of clients, from Marriott, P&G, Vodafone UK, eXate UK etc. Able to quickly understand client requirements and develop tailored solutions that address their unique needs. Very communicative and responsive, ensuring that clients are kept informed every step of the way. A quick learner and is always eager to explore new technologies and techniques to better serve clients. Familiar with Agile Methodology, Active participation in Daily Scrum meetings, Sprint meetings, and retrospective meetings, know about working in all the phases of the project life cycle. A strong team player and a leader with good interpersonal and communication skills and ready to take independent challenges.
$40 hourly
Teoman Y.
- 5.0
- (12 jobs)
Ankara, ANKARA
Apache Spark
Ansible
Red Hat Administration
Apache NiFi
DevOps Engineering
Kubernetes
Docker
Scripting
Python
Bash

Hi! I'm Teoman. I currently work as a full time DevOps Engineer for the Ministry of Interior. My main responsibilities include: - Managing Kubernetes Clusters that vary from development, staging to production. I also hold the CKA certificate. The applications run on the cluster are Java microservices, infrastructure related applications such as internal packaging systems (plugins, image registries), deployment related applications such as ArgoCD and GitLab runners and so on. I manage Big Data Engineering Kubernetes clusters that host Spark Applications, NiFi clusters, Trino backends, etc. - GitLab CI/CD pipelines where as a DevOps team, managing more than 50 projects and tracing every pipeline 24/7, creating smooth deployments that are being used by the whole country. - Managing infrastructure as code where I make calls affecting hundreds of Linux servers including production servers, tracking changes with Git. - Monitoring the running infrastructure where this many servers need to be intervened immediately in case of any failure, in which I rely on Grafana Prometheus Loki stack which again deployed on Kubernetes and bare metal, with many instances running on many networks collecting logs and metrics. I'm a Linux user since 16, and a professional administrator for 2 years now. Would be glad to be of your service, Thanks
Want to browse more talent?
Sign up