Hire the Best Apache Flume Developers in Chennai, IN

Overall 6 years of experience in IT industry with 4 years of relevant experience in Big Data Engineer, handling and transforming heterogeneous data into Key information using Hadoop ecosystem. - Expertise with the tools in Hadoop Ecosystem – HDFS, Hive , Sqoop, Spark, Kafka, Nifi. - Experience working with Elastic Search, Kibana and good knowledge on Oozie, Hbase, Phonix. - Good understanding of distributed systems, HDFS architecture, internal working details of MapReduce, Yarn and Spark processing frameworks. - More than two year of hands on experience using Spark framework with Scala. - Expertise in Inbound and Outbound (importing/exporting) data form/to traditional RDBMS using ApacheSQOOP. - Extensively worked on HiveQL, join operations, writing custom UDF’s and having good experience in optimizing Hive Queries. - Experience in data processing like collecting, aggregating, moving from various sources using Apache Nifi and Kafka. - Worked with various formats of files like delimited text files , JSON files, XML Files - Having basic knowledge on Amazon Web Services.

Seasoned data engineer with over 11 years of experience in building sophisticated and reliable ETL applications using Big Data and cloud stacks (Azure and AWS). TOP RATED PLUS . Collaborated with over 20 clients, accumulating more than 2000 hours on Upwork. 🏆 Expert in creating robust, scalable and cost-effective solutions using Big Data technologies for past 9 years. 🏆 The main areas of expertise are: 📍 Big data - Apache Spark, Spark Streaming, Hadoop, Kafka, Kafka Streams, Trino, HDFS, Hive, Solr, Airflow, Sqoop, NiFi, Flink 📍 AWS Cloud Services - AWS S3, AWS EC2, AWS Glue, AWS RedShift, AWS SQS, AWS RDS, AWS EMR 📍 Azure Cloud Services - Azure Data Factory, Azure Databricks, Azure HDInsights, Azure SQL 📍 Google Cloud Services - GCP DataProc 📍 Search Engine - Apache Solr 📍 NoSQL - HBase, Cassandra, MongoDB 📍 Platform - Data Warehousing, Data lake 📍 Visualization - Power BI 📍 Distributions - Cloudera 📍 DevOps - Jenkins 📍 Accelerators - Data Quality, Data Curation, Data Catalog

Data Integration & Big Data Specialist | Talend, Spark, Airflow, AWS Expert I am a seasoned Data Integration and Big Data Specialist with over 9 years of experience at Sopra Steria, delivering cutting-edge solutions using Hadoop, Talend, Spark, Airflow, and cloud technologies. My expertise lies in designing, implementing, and optimizing robust data pipelines and systems to handle complex business requirements. Key Strengths & Skills Data Engineering Expertise: Proficient in processing large datasets, both structured and semi-structured, with advanced Spark performance tuning to maximize efficiency. Cloud Specialist: Extensive experience in AWS services, including Redshift, S3, EMR, EC2, and Lambda, ensuring scalable and secure data processing. ETL Mastery: Designed and developed highly efficient Talend jobs for ETL processes, with a focus on performance optimization and scheduling in EC2. Orchestration & Automation: Expert in Apache Airflow for orchestration, designing complex DAGs, and automating data validations using shell scripting. CI/CD Implementation: Skilled in implementing CI/CD pipelines using Jenkins, streamlining deployment processes for enhanced efficiency. Cross-System Data Sync: Expertise in synchronizing data between AWS Redshift and Netezza. Notable Achievements Spark Optimization: Successfully tuned Spark jobs for large-scale data processing on Amazon EMR clusters, achieving significant performance improvements. Automation Initiatives: Designed and automated complex data validation frameworks, reducing manual intervention and increasing accuracy. Client Collaboration: Participated in sprint planning meetings, design discussions, and user story clarifications, ensuring smooth development and testing phases. Production Deployment: Led the deployment of Talend and Spark jobs in production clusters with meticulous configuration and monitoring. Education & Certifications B.Tech - Information Technology With a passion for solving data challenges and a proven track record of delivering impactful solutions, I am ready to contribute to your project with efficiency and precision. Let’s collaborate to turn your vision into reality!

Data is powerful — but only when it’s usable. I help businesses design and build AI-ready data platforms — from raw data ingestion to real-time analytics and intelligent insights. With 11+ years of experience in Data and Cloud ecosystems, I specialize in: ✔ Scalable Spark & Databricks pipelines ✔ Lakehouse architectures (Azure / AWS) ✔ Real-time streaming (Kafka, Spark Streaming) ✔ ETL modernization (Talend, Informatica → Cloud) ✔ ML-ready data engineering foundations Whether you need: A new data platform built from scratch Migration to Databricks or Cloud Performance optimization Data pipelines for AI/ML use cases I focus on writing clean, production-grade, scalable solutions — not quick hacks. Clients describe me as reliable, detail-oriented, and easy to work with. If you're looking for someone who understands both engineering depth and business outcomes, let's connect.

Extensive experience (16 years) in Oracle, MySQL, SQL Server, Postgres, Snowflake and Azure Data Lake methodically identifying and resolving complex underlying causes of problems, using a vast variety of techniques and tools. I have solved numerous show-stopping Performance problems, as well as often re-engineered immensely complex designs and code, providing significantly improved performance. Have designed data warehouse and implemented ETL pipeline strategies on AWS and Azure. Extensive experience as a Data Engineer with multiple work experience in data extraction, data transformation, data load, data protection, data cleansing, data migration, data mergers etc. Expert level ability with Data Architecture, Data Modelling and Data Analysis. I can conceptualize, design and develop long-term, automated, integrated solutions with a focus on increasing your competitive advantage. Have also held roles where I had to work with business end users to provide Solution Architecture and drive the solution from discussing with the users to delivering the final solution product to the end users. Have around 6 years of experience in Python (ETL and Data Analytics), 4 years in Tableau, 2.5 years in DBT and 2 years in Matillion (ETL). Domain expertise: Freights and Logistics (Quotation, Pricing, Booking, Reservation, Inventory management, Operations, Control products for Accenture Freights and Logistics Solutions) - 7 years Finance - Capital Market (Backend book-keeping for different instruments for BNP Paribas) - 3. years Digital Marketing (Generic product for Call Center and Website for a Digital Marketing company called Candid Maven) - 2 years Merchant Data Platform (Creating the automated framework from scratch to bring in data that was managed in Smartsheet to the centralized database using Python and PL/SQL to serve the analytic needs - for a company called Woflow that holds 96% of the market in this domain) - 6 months Pharmaceutical (Data Lake and Data Warehouse setup for Precision for Medicine - integrated systems like CRM (Saalesforce), ERP (Infor CloudSuite, PSA), HR (Greenhouse, ADP, PSA, SAP Success Factor), RM (Smartsheet, WorkFront), Clinical vendor data (Citeline, KMR) - 3 years At Precision for Medicine, I have worked on integrating the data of all the above mentioned disparate data platforms in 1 single dimension model (ERP, HR, RM, CRM) for creating the centralized enterprise analytics platform. I am eager to work with you, to provide reliable, consistent, and High-Level solutions to design/development challenges. Please contact me, so we can discuss how we can work together to FULLY meet your Business Needs!