KC
Kalyan C.
May 17, 2024
Create and Set up pyspark jobs on EMR
You will get Data Engineering and Deep Learning using Spark & Python on Azure/AWS/Google
You will get Data Engineering and Deep Learning using Spark & Python on Azure/AWS/Google
- Delivery Time 4 days
- Number of Revisions 2
4 days
delivery —
May 23, 2024
Revisions may occur after this date.
Upwork Payment Protection
Fund the project upfront. Gaurav gets paid once you are satisfied with the work.
Project details
I am a Databricks Certified Data Engineering professional who has progressive experience in delivering business insights through data-driven methods. I am adept at gathering and analyzing data, using machine learning, deep learning.
Here are my key achievements in data engineering so far :
• Derived actionable insights from massive data sets using statistical analysis, SQL queries, real-time data ingestion.
• Improved data pipeline completion time using Apache Spark Caching and Databricks DBIO caching.
• Removed the overheads of a parquet data format by tunning of spark configuration.
• Improved Spark-Streaming Jobs performance by implementing Back-Pressure in the stream.
• Expertise in Big Data Eco-system.
Here are my key achievements in data engineering so far :
• Derived actionable insights from massive data sets using statistical analysis, SQL queries, real-time data ingestion.
• Improved data pipeline completion time using Apache Spark Caching and Databricks DBIO caching.
• Removed the overheads of a parquet data format by tunning of spark configuration.
• Improved Spark-Streaming Jobs performance by implementing Back-Pressure in the stream.
• Expertise in Big Data Eco-system.
Project Type
Data Analysis, Development, Cybersecurity, Data Protection, IT, File Conversion, QA, User TestingWhat's included
| Service Tiers |
Starter
$30
|
Standard
$40
|
Advanced
$50
|
|---|---|---|---|
| Delivery Time | 4 days | 4 days | 4 days |
Number of Revisions | 2 | 2 | 2 |
13 reviews
(12)
(0)
(1)
(0)
(0)
This project doesn't have any reviews.
AS
Apeksha S.
May 17, 2024
Azure Synapse and databricks expert
Gaurav executed the project precisely with seamless implementation of the code. He has handled the situations where multi-tasking was involved very well. I currently rely on Gaurav's experience and expertise to complete several Data Science and Data Engineering related tasks independently.
SH
Simo H.
Aug 7, 2023
spark scala
FV
Ferdinand V.
Jun 1, 2023
30 minute consultation
jd
joel d.
Jan 9, 2023
30 minute consultation
About Gaurav
Data Engineer|Data Architect| Warehousing|Python|SQL|ETL|Azure|AWS|GCP
100%
Job Success
Jaipur, India - 8:41 pm local time
Hello, I am Data Architect and Big Data Engineer with extensive experience building large-scale Analytics solutions starting from Solution Architecture design to Implementation and Subsequent maintenance. The solution includes building and managing cloud Infrastructure.
EXPERIENCE
9+ years working in data warehousing, ETL, Cloud Computing (Google Cloud Platform & AWS), and Real-time streaming.
MY TOP SKILLS
- Python, Java, Scala, SQL, TSQL, HQL
- Apache Spark, Flink, Kafka, NiFi, Hive, Presto, Apache Beam (DataFlow)
- Azure: Azure Databricks, Azure Data factory, Azure Synapse, Azure Datawarehouse
- GCP: Google Dataproc, BigQuery, BigTable, Cloud storage, Cloud Pub/Sub
- AWS: EMR, Redshift, DynamoDB, AWS Glue, AWS Athena, Kinesis Streams, S3
- File Format: Parquet, AVRO, CSV, JSON
- Other: Data Migration, Snowflake, Pandas, Pyarrow, Delta Lake
- Cloud Infra: Kubernetes, GKE, Azure Kubernetes services, EC2, Lambda functions
House of Apache Spark:
- Spark Jobs tunning: Executors, Core, Memory, Shuffle Partitions, Data Skewness
- Spark-SQL: Catalyst Optimizer and Tungsten Optimizer
- Spark-MLlib: Machine Learning with Pyspark
- Streaming : Spark Structured Streaming(Dataframes), Spark Streaming(RDD)
Data Store:
- SQL: PostgreSQL, MySQL, Oracle, Azure SQL, DynamoDB
- No-SQL: Cassandra, Elasticsearch ILM, OpenSearch ISM, Mongo DB, Hbase
- File system: HDFS, Object Storage, Block Storage(Azure Blob, AWS S3)
Data Orchestrator:
- Apache Airlfow, Apache Oozie Workflow, Azkaban
Authentication:
- Azure Active Directory
- LDAP
- Kerberos
- SAML
Next Steps 👣
Requirements Discussion + Prototyping + Visual Design + Backend Development + Support = Success!
Steps for completing your project
After purchasing the project, send requirements so Gaurav can start the project.
Delivery time starts when Gaurav receives requirements from you.
Gaurav works on your project following the steps below.
Revisions may occur after the delivery date.
Understanding of Domain/Business of the Data
How data is helpful to the client and what all business logic needs to be implemented on top of the Process layer.