Hire the Best Apache Kafka Developers

Clients rate our Apache Kafka Developers
Rating is 4.8 out of 5.
4.8/5
Based on 311 client reviews
Anatoliy K.

Bratislava, Slovakia

$55/hr
5.0
6 jobs

Software engineer, Linux enthusiast - Leading tech decisions, architecture design; - Multi-model AI orchestration: OpenAI (GPT), Perplexity (Sonar-Pro), Google Gemini; function-calling agents with massive context - Real-time WebSocket infrastructure; Socket-io with Redis adapter for horizontal scaling - AWS-native architecture: ECS Fargate, Lambda, S3, SQS, EventBridge, CloudFront, Cognito - Infrastructure as Code; Terraform, GitHub Actions CI/CD, zero-downtime deployments - High-load API development; NestJS, Node.js, TypeScript - PostgreSQL (Prisma ORM), Redis/ElastiCache

  • Apache Kafka
  • Python
  • React
  • Node.js
  • WebRTC
  • TypeScript
  • CI/CD
  • Terraform
  • AWS Lambda
  • Kubernetes
  • Amazon ECS
  • Hyperledger Fabric
  • PostgreSQL
  • NestJS
  • NestJS Development
  • Websockets
  • Socket.io
  • Product Concept
Basant K.

Gurgaon, India

$15/hr
5.0
6 jobs

๐Ÿš€ ๐—ฆ๐—ฒ๐—ป๐—ถ๐—ผ๐—ฟ ๐—ฃ๐˜†๐˜๐—ต๐—ผ๐—ป & ๐—ก๐—ผ๐—ฑ๐—ฒ.๐—ท๐˜€ ๐——๐—ฒ๐˜ƒ๐—ฒ๐—น๐—ผ๐—ฝ๐—ฒ๐—ฟ | ๐—š๐—ฒ๐—ป๐—ฒ๐—ฟ๐—ฎ๐˜๐—ถ๐˜ƒ๐—ฒ ๐—”๐—œ ๐—˜๐—ป๐—ด๐—ถ๐—ป๐—ฒ๐—ฒ๐—ฟ | ๐—™๐—ฎ๐˜€๐˜๐—”๐—ฃ๐—œ | ๐—Ÿ๐—Ÿ๐—  | ๐—ฅ๐—”๐—š | ๐—”๐—œ ๐—”๐—ด๐—ฒ๐—ป๐˜๐˜€ Hi, Iโ€™m Basant, a ๐—ฆ๐—ฒ๐—ป๐—ถ๐—ผ๐—ฟ ๐—ฆ๐—ผ๐—ณ๐˜๐˜„๐—ฎ๐—ฟ๐—ฒ ๐—˜๐—ป๐—ด๐—ถ๐—ป๐—ฒ๐—ฒ๐—ฟ and ๐—š๐—ฒ๐—ป๐—ฒ๐—ฟ๐—ฎ๐˜๐—ถ๐˜ƒ๐—ฒ ๐—”๐—œ ๐——๐—ฒ๐˜ƒ๐—ฒ๐—น๐—ผ๐—ฝ๐—ฒ๐—ฟ with ๐Ÿญ๐Ÿฌ+ ๐˜†๐—ฒ๐—ฎ๐—ฟ๐˜€ ๐—ผ๐—ณ ๐—ฒ๐˜…๐—ฝ๐—ฒ๐—ฟ๐—ถ๐—ฒ๐—ป๐—ฐ๐—ฒ building scalable backend systems, SaaS platforms, APIs, automation solutions, full-stack applications, and AI-powered products. I specialize in ๐—ฃ๐˜†๐˜๐—ต๐—ผ๐—ป, ๐—™๐—ฎ๐˜€๐˜๐—”๐—ฃ๐—œ, ๐—ก๐—ผ๐—ฑ๐—ฒ.๐—ท๐˜€, ๐—ง๐˜†๐—ฝ๐—ฒ๐—ฆ๐—ฐ๐—ฟ๐—ถ๐—ฝ๐˜, ๐—ก๐—ฒ๐˜…๐˜.๐—ท๐˜€, ๐—ฃ๐—ผ๐˜€๐˜๐—ด๐—ฟ๐—ฒ๐—ฆ๐—ค๐—Ÿ, ๐—”๐—ช๐—ฆ, ๐—ฎ๐—ป๐—ฑ ๐—š๐—ฒ๐—ป๐—ฒ๐—ฟ๐—ฎ๐˜๐—ถ๐˜ƒ๐—ฒ ๐—”๐—œ, helping businesses turn ideas into secure, scalable, production-ready applications. I can help you build a new AI product, integrate AI into an existing application, develop scalable APIs, modernize your backend, or automate business workflows using LLMs and AI Agents. ๐Ÿง  ๐—š๐—ฒ๐—ป๐—ฒ๐—ฟ๐—ฎ๐˜๐—ถ๐˜ƒ๐—ฒ ๐—”๐—œ & ๐—Ÿ๐—Ÿ๐—  ๐——๐—ฒ๐˜ƒ๐—ฒ๐—น๐—ผ๐—ฝ๐—บ๐—ฒ๐—ป๐˜ LLM application development AI chatbot development RAG-based applications Document Q&A systems PDF/document processing Prompt engineering Embeddings & semantic search Vector databases AI Agents Tool & function calling Database AI Agents Intelligent automation Structured LLM outputs LangChain & LangGraph OpenAI API integrations AI-powered business applications ๐Ÿ ๐—ฃ๐˜†๐˜๐—ต๐—ผ๐—ป & ๐—™๐—ฎ๐˜€๐˜๐—”๐—ฃ๐—œ Python FastAPI Pydantic SQLAlchemy 2.x Alembic PostgreSQL MySQL Redis REST API development Async programming WebSockets OAuth 2.0 & JWT RBAC API integrations Background processing Microservices Docker โšก ๐—ก๐—ผ๐—ฑ๐—ฒ.๐—ท๐˜€ & ๐—™๐˜‚๐—น๐—น ๐—ฆ๐˜๐—ฎ๐—ฐ๐—ธ ๐——๐—ฒ๐˜ƒ๐—ฒ๐—น๐—ผ๐—ฝ๐—บ๐—ฒ๐—ป๐˜ Node.js Express.js TypeScript React.js Next.js Redux REST APIs WebSockets Real-time applications Third-party API integrations โ˜๏ธ ๐—–๐—น๐—ผ๐˜‚๐—ฑ, ๐—”๐—ฟ๐—ฐ๐—ต๐—ถ๐˜๐—ฒ๐—ฐ๐˜๐˜‚๐—ฟ๐—ฒ & ๐——๐—ฒ๐˜ƒ๐—ข๐—ฝ๐˜€ AWS EC2 S3 Lambda CloudWatch Docker Nginx Kafka RabbitMQ Redis & caching CI/CD Microservices Distributed systems Event-driven architecture API performance optimization ๐Ÿงฉ ๐—ช๐—ต๐—ฎ๐˜ ๐—œ ๐—–๐—ฎ๐—ป ๐—•๐˜‚๐—ถ๐—น๐—ฑ โœ… AI-powered SaaS applications โœ… RAG knowledge assistants โœ… AI chatbots and custom AI assistants โœ… Document/PDF Q&A platforms โœ… AI Agents with database & API tools โœ… LLM-powered business automation โœ… AI-enabled enterprise applications โœ… FastAPI AI backends โœ… Node.js AI applications โœ… Next.js AI applications โœ… AI APIs and microservices โœ… Database AI Agents โœ… Real-time applications โœ… Existing application modernization with AI ๐Ÿ—๏ธ ๐—œ๐—ป๐—ฑ๐˜‚๐˜€๐˜๐—ฟ๐˜† & ๐—˜๐—ป๐—ด๐—ถ๐—ป๐—ฒ๐—ฒ๐—ฟ๐—ถ๐—ป๐—ด ๐—˜๐˜…๐—ฝ๐—ฒ๐—ฟ๐—ถ๐—ฒ๐—ป๐—ฐ๐—ฒ I have 10+ years of software engineering experience working on enterprise applications, SaaS platforms, analytics platforms, real-time systems, HRMS, payroll, ATS, fintech, logistics, travel, gaming, and business automation solutions. I don't just build AI prototypes. My focus is on building production-ready software with scalable architecture, clean APIs, secure authentication, database design, performance optimization, maintainable code, and cloud deployment. ๐Ÿ“ˆ ๐—ช๐—ต๐˜† ๐—ช๐—ผ๐—ฟ๐—ธ ๐—ช๐—ถ๐˜๐—ต ๐— ๐—ฒ? 10+ years of software engineering experience Strong Python + Node.js backend expertise FastAPI & REST API development Generative AI & LLM application development RAG & AI Agent development Strong React.js & Next.js experience PostgreSQL, MongoDB, MySQL & Redis Kafka & microservices AWS & Docker API integrations & automation End-to-end product development Experience modernizing existing applications Strong system design and architecture experience Technical leadership and team management experience ๐Ÿค ๐—Ÿ๐—ฒ๐˜'๐˜€ ๐—•๐˜‚๐—ถ๐—น๐—ฑ ๐—ฆ๐—ผ๐—บ๐—ฒ๐˜๐—ต๐—ถ๐—ป๐—ด Whether you need a Python/FastAPI Developer, Node.js Developer, Full Stack Engineer, or Generative AI Developer, I can help you take your project from architecture โ†’ development โ†’ AI integration โ†’ testing โ†’ deployment. If you have an AI product idea, need to integrate an LLM into an existing application, want to build a RAG/AI Agent system, or need a scalable Python/Node.js backend, let's discuss your project. ๐—•๐—ฒ๐˜€๐˜ ๐—ฟ๐—ฒ๐—ด๐—ฎ๐—ฟ๐—ฑ๐˜€, ๐—•๐—ฎ๐˜€๐—ฎ๐—ป๐˜ ๐—ž๐˜‚๐—บ๐—ฎ๐—ฟ ๐—ฆ๐—ถ๐—ป๐—ด๐—ต

  • Apache Kafka
  • Python
  • FastAPI
  • Generative AI
  • Node.js
  • Next.js
  • PostgreSQL
  • MongoDB
  • Vector Database
  • AI Agent Development
  • LangChain
  • Docker
  • CI/CD
  • ExpressJS
  • SQLAlchemy
Abel A.

Addis Ababa, Ethiopia

$10/hr
5.0
10 jobs

Clients come to me for one of two things: building an enterprise-grade system the right way from the start, or rescuing an existing codebase fixing bugs, resolving integration issues, and shipping features other developers left unfinished. I'm Abel, a full-stack developer with 5 years building production systems โ€” from a government justice case management platform to a real-time tax risk engine processing live data through Kafka. I pair Java/Spring Boot microservices with Angular or React frontends, and I turn messy requirements into stable, scalable products. WHAT I HELP WITH โœ” Enterprise & government systems โ€” case management, revenue/tax platforms, complex backend-heavy applications โœ” Microservices & API integration โ€” event-driven architecture, Kafka, REST APIs that hold up under real load โœ” SaaS & MVPs โ€” end-to-end builds with Spring Boot + Angular/React โœ” Rescuing broken projects โ€” messy auth, unstable APIs, half-finished features โœ” Auth & security โ€” Keycloak, OAuth2, JWT done properly โœ” Deployment & monitoring โ€” Docker, Kubernetes, Grafana, Loki MY STACK Backend: Java, Spring Boot, Kotlin, Node.js, Apache Kafka Frontend: Angular, React, TypeScript, Tailwind CSS Databases: PostgreSQL, Supabase DevOps: Docker, Kubernetes, Grafana, Loki, GitHub Actions Auth: Keycloak, OAuth2, JWT Clients tell me I bring enterprise-level discipline to fast-moving projects โ€” clean architecture, clear documentation, and systems that hold up under real load. If your project involves real users, real transactions, or real consequences โ€” let's talk. Send me a message with your project details or requirements, and let's bring your platform to life.

  • Apache Kafka
  • Java
  • Spring Boot
  • Microservice
  • REST API
  • Spring Security
  • Software Architecture & Design
  • Software Integration
  • API Integration
  • Enterprise Architecture
  • Database Design
  • Database Caching
  • Grafana
  • React
  • SaaS
Rushabh A.

Pune, India

$25/hr
5.0
3 jobs

I'm a Full-Stack Java Developer with over 8+ years of industry experience. I excel in all aspects of Java application development, from design and development to deployment. I have a proven track record of building robust and Scalable applications across diverse domains. My Expertise Full-Stack Development: Java: Core Java, multithreading, OOP, Spring, Hibernate, Struts, J2EE. Frameworks & Libraries: Spring, Hibernate, JavaFX, Apache Struts. Microservices Architecture: Spring Cloud, Netflix OSS, service discovery, load balancing, tolerance mechanisms. Web Development: JSP, Servlets, RESTful APIs, Angular, React, HTML, CSS, JavaScript. Databases: MySQL, PostgreSQL, MongoDB, Redis. Testing & Debugging: JUnit, Mockito. I also use GitHub Copilot and Claude Code to deliver the best solutions. โœ” LLM integrations (OpenAI, Claude, etc.) โœ” AI-assisted workflow automation โœ” Prompt engineering & optimization โœ” RAG based enterprise search solutions โœ” AI code acceleration using tools like Cursor โœ” Designing AI-ready Microservices architecture Java Solution Architect: Designing scalable solutions with Java, Spring, Hibernate, and microservices. DevOps: Cloud setup, infrastructure management, CI/CD. Proficient in AWS (Lambda, EC2, S3, Redshift, SageMaker, Cognito) and GCP (VMs, Google cloud storage). Domain Experience: Insurance, finance, healthcare, e-learning, e-commerce, travel, real estate, logistics, supply chain, social networking, and education. CRM and ERP systems.

  • Apache Kafka
  • Java
  • Spring Boot
  • Spring Batch
  • TypeScript
  • Microservice
  • MySQL
  • PostgreSQL
  • API
  • Google Analytics
  • ETL
  • Angular
  • Data Engineering
  • Angular 5
  • React
  • Hibernate
  • RESTful API
  • Spring Framework
Nova A.

Petukangan, Indonesia

$7/hr
5.0
6 jobs

8+ years of experience and 1,000+ hours delivered. I design and scale end-to-end full-stack systems using Node.js, TypeScript, Go, and event-driven microservices (RabbitMQ, Redis, Docker). Core Expertise: - Architecture: Building production-ready, scalable solutions for high-traffic applications. - Integrations: Specialized in integrating global payment platforms (Stripe, Paypal, Ryftpay, GoCardless, and NMI) and Indonesian marketplaces Tokopedia, Tiktok Shop, Lazada, BliBli, Shopee, WooCommerce, Shopline, and Shopify). - Efficiency: Proven track record of improving engineering workflows and meeting demanding timelines. I translate business requirements into reliable technical solutions. Letโ€™s discuss how I can help scale your next project.

  • Apache Kafka
  • PostgreSQL
  • MySQL
  • ExpressJS
  • Node.js
  • Docker
  • Golang
  • Next.js
  • NestJS
  • CI/CD
  • MongoDB
  • Microservice
  • Redis
  • Kubernetes
  • RabbitMQ
  • Stripe API
  • PayPal Integration
Sanjoy K.

Dhaka, Bangladesh

$25/hr
4.1
18 jobs

Java & Spring Boot Architect | AI-Powered Backend Systems | 14+ Years Experience I am a Java & Spring Boot architect with 14+ years of hands-on experience building production-grade enterprise systems โ€” from core banking platforms to AI-integrated microservices. I hold a BSc in Computer Science & Technology and now work as a dedicated full-time freelancer on Upwork, bringing the same rigour I applied in large fintech institutions directly to client projects. My core expertise spans Spring Boot, Spring Framework, and microservices architecture, with deep practical knowledge of SSO & Keycloak, Docker, DevOps pipelines, and GitLab CI/CD. I design systems that are secure, observable, and built to scale โ€” whether that means integrating Kafka for event-driven flows, Flowable BPMN for workflow automation, or AWS for cloud infrastructure. Over the past year I have been heavily invested in the AI/LLM frontier: building Spring AI applications, wiring up MCP (Model Context Protocol) servers, running local models via Ollama, and implementing advanced patterns like tool calling, RAG pipelines, and AI agent workflows. I see AI not as a separate layer but as a first-class citizen inside enterprise Java backends. If you need someone who understands both the old-school reliability of enterprise Java and the new possibilities of AI-augmented architecture โ€” let's talk. CORE STACK: Java ยท Spring Boot ยท Spring Framework ยท Microservices ยท REST API ยท Keycloak/SSO ยท Docker ยท GitLab CI/CD ยท DevOps ยท Kafka ยท Flowable BPMN ยท Oracle ยท PostgreSQL ยท AWS ยท Angular AI & EMERGING TECH: Spring AI ยท MCP ยท Ollama ยท Tool Calling ยท AI Agents ยท RAG Pipelines ยท pgvector ยท LLM Integration ยท OpenAI API ยท Prompt Engineering DOMAINS: Banking & Fintech ยท SWIFT/ISO 8583 ยท AML/CFT Compliance ยท NPSB/RTGS ยท ATM Switch ยท eKYC ยท Healthcare/EMR ยท Document Management

  • Apache Kafka
  • Spring Security
  • Spring Boot
  • Java Persistence API
  • Microservice
  • Hibernate
  • JasperReports
  • Java
  • RESTful Architecture
  • GitLab
  • DevOps
  • Android
  • Object-Relational Mapping
  • Angular 6
  • Google Sheets

How it works

Post a job for freePost a job

Tell us what you need. Create your own job post or generate one with AI then filter talent matches.

Hire top talent fast

Consult, interview, and hire quickly, so you can meet the freelancers you're excited about.

Collaborate easily

Use Upwork to chat or video call, share files, and track project progress right from the app.

Payment simplified

Manage payments in one place with flexible billing options. Only pay for approved work, hourly or by milestone.

Don't just take our word for it

How to Hire Top Apache Kafka Developers

How to hire Apache Kafka developers

Looking for a high-throughput, fault-tolerant, data streaming solution for processing large volumes of messages? An Apache Kafka developer can help.

So how do you hire Apache Kafka developers? What follows are some tips for finding top Apache Kafka consultants on Upwork.

How to shortlist Apache Kafka professionals

As youโ€™re browsing available Apache Kafka consultants, it can be helpful to develop a shortlist of the professionals you may want to interview. You can screen profiles on criteria such as:

  • Technology fit. You want a developer who understands how to integrate Apache Kafka with the rest of your technology stack.
  • Project experience. Screen candidate profiles for specific skills and experience (e.g., building a website activity tracking pipeline).
  • Feedback. Check reviews from past clients for glowing testimonials or red flags that can tell you what itโ€™s like to work with a particular Apache Kafka developer.

How to write an effective Apache Kafka job post

With a clear picture of your ideal Apache Kafka developer in mind, itโ€™s time to write that job post. Although you donโ€™t need a full job description as you would when hiring an employee, aim to provide enough detail for a contractor to know if theyโ€™re the right fit for the project.

Job post title

Create a simple title that describes exactly what youโ€™re looking for. The idea is to target the keywords that your ideal candidate is likely to type into a job search bar to find your project. Here are some sample Apache Kafka job post titles:

  • Need help building low-latency log aggregation solution with Apache Kafka
  • Seeking Java developer with Kafka Pepper-Box and JMeter expertise
  • Developing a Change Data Capture (CDC) agent with Kafka

Apache Kafka project description

An effective Apache Kafka job post should include:

  • Scope of work: From message brokers to real-time analytics feeds, list all the deliverables youโ€™ll need.
  • Project length: Your job post should indicate whether this is a smaller or larger project.
  • Background: If you prefer experience with certain industries, software, or developer tools, mention this here.
  • Budget: Set a budget and note your preference for hourly rates vs. fixed-price contracts.

Apache Kafka developer responsibilities

Here are some examples of Apache Kafka job responsibilities:

  • Design and develop data pipelines
  • Manage data quality
  • Implement data integration solutions
  • Troubleshoot and debug data streaming processes

Apache Kafka developer job requirements and qualifications

Be sure to include any requirements and qualifications youโ€™re looking for in an Apache Kafka developer. Here are some examples:

  • Proficiency in Java and/or Scala
  • Data streaming
  • CDC
  • Data engineering
>

Apache Kafka Developers FAQ

Frequently asked questions

What is Apache Kafka?

Apache Kafka is an open-source stream-processing solution developed by LinkedIn and later donated to the Apache Software Foundation. The software platform aims to provide a low-latency, high-throughput solution for processing real-time data feeds.

Apache Kafka uses the publish/subscribe messaging pattern common in distributed systems. Kafka instances typically exist as clusters of nodes called brokers that can receive messages from multiple producers (any apps sending data to the cluster) and deliver them to multiple consumers (any apps receiving data from the cluster). Producers publish messages to Kafka topics (i.e., categories of messages), while consumers subscribe to Kafka topics. It is through this topic categorization that the brokers are able to determine where messages need to be delivered.

Apache Kafka is a popular choice among developers looking to build message brokers, website activity trackers, and analytics pipelines that must deal with large volumes of real-time data from disparate sources.

How much does it cost to hire an Apache Kafka developer?

The first step to determining the cost to hire an Apache Kafka developer will be to define your needs. Rates can vary due to many factors, including expertise and experience, location, and market conditions.

Cost factor #1: project scope

The first variable to consider when determining scope is the nature of the work that needs to be completed. Not all Apache Kafka development projects are created equal. Creating a simple log aggregator to collect log files off different servers into a central place for processing will typically take less time than building out a multistage data streaming pipeline for your SaaS (software-as-a-service) product.

Tip: The more accurately your job description describes the scope of your project, the easier it will be for talent to give you accurate cost estimates and proposals.

Cost factor #2: Apache Kafka developer experience

Choosing the right level of expertise for the job is closely tied to how well you determined the scope of your project. You wouldnโ€™t need an advanced Apache Kafka developer to create your own custom site analytics dashboard using Kafka. On the other hand, building a large-scale enterprise messaging system will require the skills of a seasoned Apache Kafka developer.

Beyond experience level, you need to consider the type of experience the talent possesses. The following table breaks down the rates of the typical types of Apache Kafka developers you can find on Upwork.

Rates charged by Apache Kafka developers on Upwork

Level of Experience Description Hourly Rate
Beginner Familiarity across the technology stack. Data engineering fundamentals (e.g., data streaming, data quality, data integration). Can use Kafka for basic website tracking, messaging, and data streaming. $40-70+
Intermediate Professional full-stack developers or data engineers. Experience working with high-throughput data needs, microservices architectures, and multistage data streaming pipelines. $70-100+
Expert Advanced full-stack developers or data engineers with years of experience in big data. Capable of managing teams of developers and engineers. Advanced knowledge of application architectures, data streaming technologies, and data processing solutions. $100-130+

Cost factor #3: location

Location is another variable that can impact an Apache Kafka developerโ€™s cost. Itโ€™s no secret that you can leverage differences in purchasing power between countries to gain savings on talent. But itโ€™s also important to factor in hidden costs such as language barriers, time zones, and the logistics of managing a remote team. The real advantage to sourcing talent remotely on Upwork is the ability to scan a global talent pool for the best possible person for the job. Location is no longer an obstacle.

Cost factor #4: independent contractor vs. agency

The final variable regarding talent cost is hiring an independent contractor vs. an agency. An agency is often a โ€œone size fits allโ€ model, so youโ€™ll often have access to a designer, a project manager, an engineer, and more. When hiring individuals you have total autonomy regarding who is responsible for which part of the project, but youโ€™ll need to source each of those skills separately.

The trade-off between hiring individuals vs. hiring an agency is the level of administrative overhead you incur personally in coordinating tasks among all members of the team. Project scope and personal preference will determine which style is a better fit for your needs.

Apache Kafka developer tips and best practices

Understand your partition data rate limitations

In Kafka, messages are organized into topics that can be divided into a number of smaller partitions. Partitions allow your Kafka cluster to process the data in a particular topic in parallel across multiple brokers. This capacity for parallel processing is what enables Kafka to deliver high-throughput messaging.

Of course, even high-throughput systems are going to have their limitations. Messages sent to a partition exist in a log for a configurable period of time or until a configurable size limit is reached. Exceed that retention limit prematurely, and itโ€™s possible you can start losing messages before consumers can pull them from the topic partition.

Thatโ€™s why itโ€™s important to understand the data rate of your topic partitions. Simply multiply the average message size times the number of messages per second to calculate your average retention rate. This will enable you to figure out how much retention space is required to guarantee data is retained for the desired period of time.

Widen those consumer socket buffers for high-speed ingestion

The default settings for consumer socket buffers tend to be around 100 KB (Kafka 2.4.x), which is too small for high-throughput environments. For low-latency, high-bandwidth networks (10 Gbps or higher), it might be necessary to bump those values up to 8 or 16 MB. You can tune the socket buffer setting for consumers with the โ€œsocket.receive.buffer.bytesโ€ parameter.

Tune your memory buffer and batch sizes for high-throughput producers

On the producer side of the equation, high-throughput environments will likely require a change to the default memory sizes for your โ€œbuffer.memoryโ€ and โ€œbatch.sizeโ€ parameters. These values are trickier to set than your consumer socket buffers as they depend on a number of factors, including producer data rate, number of partitions, and the total memory you have available. Larger buffers arenโ€™t necessarily always better, because having too much data buffered on-heap can lead to increased garbage collectionโ€”a process that will compete for resources and affect your importance. Best practices should be established based on the unique configuration and settings of your Kafka data streaming system.