Data & AIConference40min
Apache Kafka as a Queue: Bridging Event Streaming and Point-to-Point Messaging
This session introduces KIP-932, adding native queue semantics to Apache Kafka through the ShareConsumer API and cooperative consumption. It covers message management, long-running tasks, observability features, and code examples, enabling seamless point-to-point messaging with Kafka’s durability, scalability, and fault tolerance—modernizing messaging architectures without changing producer behavior.
Sandon JacobsConfluent
talkDetail.whenAndWhere
Thursday, April 23, 10:55-11:35
Skalkotas
talks.roomOccupancytalks.noOccupancyInfo
Apache Kafka has established itself as the leading platform for event streaming, yet architects and engineers have often turned to other systems to fulfill point-to-point messaging needs. With KIP-932—Queues for Kafka—Kafka now natively supports queue-like semantics.
In this session, we’ll dive into the cooperative consumption model that brings message-level acknowledgments to Kafka via the new ShareConsumer API. We’ll trace the journey of a message and demonstrate how developers can programmatically manage message state. Queue-driven tasks are often long-running, with a non-deterministic return time - and the API has this use case covered, as well. Observability is essential in distributed systems - so we’ll also explore the metrics and configuration options that provide essential visibility. Finally, we’ll walk through some sample code - because slides never make it to production.
Queues for Kafka give developers a seamless way to meet messaging requirements using the Kafka protocol—without altering how messages are produced. This solution delivers point-to-point messaging with all the core guarantees of Apache Kafka, including durable storage, horizontal scalability, and fault tolerance. Discover how “queues the Kafka way” can elevate your messaging architecture.
In this session, we’ll dive into the cooperative consumption model that brings message-level acknowledgments to Kafka via the new ShareConsumer API. We’ll trace the journey of a message and demonstrate how developers can programmatically manage message state. Queue-driven tasks are often long-running, with a non-deterministic return time - and the API has this use case covered, as well. Observability is essential in distributed systems - so we’ll also explore the metrics and configuration options that provide essential visibility. Finally, we’ll walk through some sample code - because slides never make it to production.
Queues for Kafka give developers a seamless way to meet messaging requirements using the Kafka protocol—without altering how messages are produced. This solution delivers point-to-point messaging with all the core guarantees of Apache Kafka, including durable storage, horizontal scalability, and fault tolerance. Discover how “queues the Kafka way” can elevate your messaging architecture.
Sandon Jacobs
Sandon Jacobs is a Developer Advocate at Confluent, based in Raleigh, NC. Sandon has two decades of experience designing and building applications, primarily on the JVM. His data streaming journey began while building data pipelines for real-time bidding on mobile advertising exchanges—and Apache Kafka was the platform to meet that need. Later work in television media and the energy sector included Kafka Streams, Kafka Connect, and provisioning Kafka infrastructure with various infrastructure as code frameworks.
Outside of work, Sandon is actively involved in his Indigenous tribal community. He serves on the NC American Indian Heritage Commission, and also as a powwow singer and emcee at many celebrations around North America. Follow Sandon on Twitter @SandonJacobs or Instagram @_sandonjacobs, where he posts about his powwow travels, family, golf, and more.
Outside of work, Sandon is actively involved in his Indigenous tribal community. He serves on the NC American Indian Heritage Commission, and also as a powwow singer and emcee at many celebrations around North America. Follow Sandon on Twitter @SandonJacobs or Instagram @_sandonjacobs, where he posts about his powwow travels, family, golf, and more.
talkDetail.shareFeedback
talkDetail.feedbackNotYetAvailable
talkDetail.feedbackAvailableAfterStart
talkDetail.signInRequired
talkDetail.signInToFeedbackDescription
occupancy.title
occupancy.votingNotYetAvailable
occupancy.votingAvailableBeforeStart
talkDetail.signInRequired
occupancy.signInToVoteDescription
comments.speakerNotEnabledComments