Blob-stream rethinks Kafka around object storage and stateless brokers
A practical redesign of durable streaming around object storage, stateless brokers, and simpler Kubernetes operations, aimed at lowering the cost and maintenance burden of high-volume data systems.
Logged at IST: 2026-10-05 14:38 IST
What it is: Matt Klein’s introduction to blob-stream, an open-source high-volume streaming system designed as a Kafka alternative.
Gist: Blob-stream keeps familiar topics, partitions, producer batches, consumer groups, and durable acknowledgements, but changes the operating model. It writes data to S3 rather than broker disks and aims for stateless brokers, no separate cluster manager, and no cross-availability-zone data traffic. Kubernetes service discovery and DynamoDB coordinate the system. The tradeoff is a new system and client ecosystem rather than drop-in Kafka compatibility.
Why it matters: Kafka’s operational costs are not just compute. Storage management, rebalancing, cross-zone traffic, and scaling all add up. Blob-stream is a useful case study in changing the architecture, not merely tuning the Kafka deployment.
Source note: The design and goals are from Matt Klein’s article. Cost and operational improvements are project goals, not independently benchmarked claims here.