Backend & Integration tech.Conference50min
From Kafka Misconfigurations to High Performance - A Practical Guide to Keeping Your Producer Sane
This talk offers practical insights into optimizing Kafka producers for high throughput and low latency. It covers understanding and updating configurations, avoiding common misconfigurations, and adjusting settings like linger.ms and batch.size. Attendees will learn to enhance performance and ensure data consistency with transaction IDs, transforming their Kafka experience.
Nupur AgarwalING
talkDetail.whenAndWhere
Thursday, June 19, 10:15-11:05
Linux
Ever felt like your Kafka producer is more chaos than control? I have. Join me as I share how I tuned my Kafka producer from a nightmare into a smooth, efficient system handling high throughput and low latency without a hitch. Whether you're a seasoned Kafka user or just starting out, this talk will provide you with practical insights and best practices to optimize your Kafka producer and avoid common pitfalls.
In this talk, I'll dive deep into the technical intricacies of Kafka producers. We'll start by exploring need of understanding and updating default producer configurations, and then common misconfigurations that can lead to timeout exception while tuning producer for high throughput, with some tips on how to avoid them. You'll learn what all happens at broker end while you call kafkaTemplate.send() method in your Kafka producer, which will help in adjusting settings like linger.ms, batch.size, and request.timeout.ms for batching your kafka records. Additionally, we'll cover importance of transaction IDs for ensuring data consistency while using batching.
By the end of this talk, you'll have a solid understanding of how to configure and optimize your Kafka producer for peak performance and avoid common pitfalls. You'll walk away with practical tips to avoid common misconfigurations, you'll be equipped with the knowledge to keep your Kafka producer running efficiently. Join me to transform your Kafka experience and take your data streaming capabilities to the next level
In this talk, I'll dive deep into the technical intricacies of Kafka producers. We'll start by exploring need of understanding and updating default producer configurations, and then common misconfigurations that can lead to timeout exception while tuning producer for high throughput, with some tips on how to avoid them. You'll learn what all happens at broker end while you call kafkaTemplate.send() method in your Kafka producer, which will help in adjusting settings like linger.ms, batch.size, and request.timeout.ms for batching your kafka records. Additionally, we'll cover importance of transaction IDs for ensuring data consistency while using batching.
By the end of this talk, you'll have a solid understanding of how to configure and optimize your Kafka producer for peak performance and avoid common pitfalls. You'll walk away with practical tips to avoid common misconfigurations, you'll be equipped with the knowledge to keep your Kafka producer running efficiently. Join me to transform your Kafka experience and take your data streaming capabilities to the next level
Nupur Agarwal
Nupur is a Senior Backend Engineer working in ING primarily with Java applications. She has worked across various industries, including finance, banking, telecom and supply chain.
With a passion for data, she is working on ensuring data consistency and completeness across various systems in ING which are exchanging data via kafka in Lending domain.
With a passion for data, she is working on ensuring data consistency and completeness across various systems in ING which are exchanging data via kafka in Lending domain.
comments.speakerNotEnabledComments