Upwork home
Sign up
Project Catalog
DesignDesign
Video & AudioVideo & Audio
Development & ITDevelopment & IT
MarketingMarketing
Writing & TranslationWriting & Translation
MoreMore

You will get a streaming solution in pyspark to process data from kafka to S3 deltalake

Mandar G.Status: Offline
Mandar G.
Preview image

Let a pro handle the details

Buy Other Data Analysis & Reports services from Mandar, priced and ready to go.
Mandar G.Status: Offline
Mandar G.
Preview image

Let a pro handle the details

Buy Other Data Analysis & Reports services from Mandar, priced and ready to go.

Project details

Real-Time Kafka to Delta Lake Pipeline using PySpark | A PySpark streaming solution, that ingests JSON data from Apache Kafka, performs transformations and validations, and writes to Delta Lake on AWS S3, using Spark Structured Streaming.

Key Features:
1. Application logging (log4j) enabled
2. Ready to use Kafka messages producer
3. Test case sample included

What's included $20

These options are included with the project scope.

$20
  • Delivery Time 15 days
  • Number of Revisions 3
    • Source Code
Mandar G.Status: Offline

About Mandar

Mandar G.Status: Offline
Databricks | PySpark | Delta Lake | ETL | Streaming App | SQL
Bengaluru, India - 11:17 am local time
I’m a seasoned data engineer experienced in architecting and delivering end-to-end data pipelines—both batch and real-time. I have a proven track record using Databricks, Spark, AWS services, and Kafka, designing systems that are scalable, fault-tolerant, optimized, and cost-effective. I excel at performance tuning, cloud-native architectures, and building observability tools for production readiness. Passionate about solving complex data challenges, I help businesses turn raw data into actionable insights. In short: I build data platforms that work reliably at scale—and make them easy to maintain.

Steps for completing your project

After purchasing the project, send requirements so Mandar can start the project.

Delivery time starts when Mandar receives requirements from you.

Mandar works on your project following the steps below.

Revisions may occur after the delivery date.

Understand the exact requirement

Will need to hear the exact requirement based on which the existing template pipeline can we enhanced.

Review the work, release payment, and leave feedback to Mandar.