ArchitectureConference50min
What Can You Do with an Apache Kafka "Queue?"
This talk introduces KIP-932, enabling a new consumer group model in Kafka for parallel event processing beyond partition limits, ideal when ordering isn't required. It covers how shared consumer groups work, use cases, code samples, and advantages over traditional messaging systems, helping developers adopt "queues the Kafka way."
Sandon JacobsConfluent
talkDetail.whenAndWhere
Thursday, October 9, 11:50-12:40
Room 9
Introducing KIP-932, the long-awaited “Queues for Kafka” functionality. With typical consumer group coordination in Apache Kafka, each partition of a topic is assigned to a member of a consumer group. But this constrains parallel processing to the number of partitions of the target topic.
KIP-932 addresses those use cases where ordering is not the priority, and our events represent individual units of work. Also of note: it's not actually a "queue" at all, rather a different consumer group model. And yet every time I speak to developers about this functionality, I get the same questions: “When do I use it? Can you share a use case?”
So let’s take some time to identify and explore a couple of use cases. Our talk will start with how shared consumer groups work. Then we’ll walk through code samples of how and when we would use this consumption model. This is much more than just a JMS substitute or replacement. We'll discuss what queues “the Kafka way” gives us above and beyond transitional messaging systems.
When we wrap up, you’ll have a better idea of how and where to get started with queues the Kafka way.
KIP-932 addresses those use cases where ordering is not the priority, and our events represent individual units of work. Also of note: it's not actually a "queue" at all, rather a different consumer group model. And yet every time I speak to developers about this functionality, I get the same questions: “When do I use it? Can you share a use case?”
So let’s take some time to identify and explore a couple of use cases. Our talk will start with how shared consumer groups work. Then we’ll walk through code samples of how and when we would use this consumption model. This is much more than just a JMS substitute or replacement. We'll discuss what queues “the Kafka way” gives us above and beyond transitional messaging systems.
When we wrap up, you’ll have a better idea of how and where to get started with queues the Kafka way.
Sandon Jacobs
Sandon Jacobs is a Developer Advocate at Confluent, based in Raleigh, NC. Sandon has two decades of experience designing and building applications, primarily on the JVM. His data streaming journey began while building data pipelines for real-time bidding on mobile advertising exchanges—and Apache Kafka was the platform to meet that need. Later work in television media and the energy sector included Kafka Streams, Kafka Connect, and provisioning Kafka infrastructure with various infrastructure as code frameworks.
Outside of work, Sandon is actively involved in his Indigenous tribal community. He serves on the NC American Indian Heritage Commission, and also as a powwow singer and emcee at many celebrations around North America. Follow Sandon on Twitter @SandonJacobs or Instagram @_sandonjacobs, where he posts about his powwow travels, family, golf, and more.
Outside of work, Sandon is actively involved in his Indigenous tribal community. He serves on the NC American Indian Heritage Commission, and also as a powwow singer and emcee at many celebrations around North America. Follow Sandon on Twitter @SandonJacobs or Instagram @_sandonjacobs, where he posts about his powwow travels, family, golf, and more.
talkDetail.transcript.loginToSeeTakeaways
talkDetail.transcript.loginToSeeInsights
comments.speakerNotEnabledComments