Shawn GordonWelcome to the fourth issue of the WarpStream newsletter. A lot has happened since our last newsletter: we’ve released five new blogs, made a bunch of product updates, and added new social channels (like Facebook and YouTube. Connect with us on social media and other platforms to stay updated via the links in the social footer at the bottom of this email.
Lots of New Blog Posts
Introducing WarpStream Managed Data Pipelines for BYOC clusters
For WarpStream BYOC clusters, Managed Data Pipelines provide a fully-managed SaaS user experience for Bento, a lightweight stream processing framework that offers much of the functionality of Kafka Connect, without sacrificing any of the cost benefits, data sovereignty, or deployment flexibility of the BYOC deployment model and comes with version control.
Pixel Federation Powers Mobile Analytics Platform with WarpStream, saves 83% over MSK
Pixel Federation’s mobile games have millions of users, so you can imagine how many events and Kafka topics they have. By swapping MSK for WarpStream, they not only drastically reduced their costs, but were able to ditch complex VPC peering in favor of simpler agent groups.
Interested in Learning More About WarpStream?
Zero Disks is Better (for Kafka)
In a prior blog, we discussed how tiered storage won’t fix Kafka. The end goal is not some disks but zero disks. We cover how WarpStream’s Zero Disk Architecture (ZDA) allows you to do things like trivial or dead-simple auto-scaling of Kafka brokers (“agents” in WarpStream terminology), isolate workloads with agent groups, and easily run your entire data pipeline in your virtual private cloud (VPC) without the need for custom code or additional services.
Secure by default: How WarpStream’s BYOC deployment model secures the most sensitive workloads
WarpStream’s BYOC model is a hybrid approach that balances the two common cloud deployment models (fully self-managed and fully hosted SaaS). By splitting the software into discrete data and control planes, it ensures data privacy and sovereignty, compliance, cost optimization, and control.
Try WarpStream With $400 in Free Credits
Multiple Regions, Single Pane of Glass
A common problem when building infrastructure-as-a-service products is the need to provide highly available and isolated resources in many different regions while also having the overall product present as a “single pane of glass” to end-users. We review the options available to solve this and what we ultimately used (pushed-based replication).
Recent Product Updates
Managed Data Pipelines
BYOC customers can now use Managed Data Pipelines. These combine the power of WarpStream’s control plane with Bento, an open-source streaming processing platform.
This provides much of the same functionality as Kafka Connect and additional stream processing functionality like single message transforms, aggregations, multiplexing, enrichments, and native support for WebAssembly (WASM).
Pipelines run in your VPC and on your VMs, and data is processed in your buckets. WarpStream has zero access to this data. WarpStream provides a helpful UI for creating and editing pipelines, the ability to pause and resume pipelines dynamically, and version control.
Lots of New Metrics
We’ve added new metrics (and deprecated unnecessary ones) with nearly every release. We’ve recapped some of these new metrics below. You can check out our official changelog to get the full list.
- warpstream_consumer_group_generation_id = This metric indicates the generation number of the consumer group, incrementing by one with each rebalance. It serves as an effective indicator for detecting occurrences of rebalances.
- warpstream_agent_kafka_fetch_uncompressed_bytes = Tracks the total uncompressed bytes fetched, replacing warpstream_agent_kafka_fetch_bytes_sent metric.
- warpstream_consumer_group_generation_id = Uses the consumer_group tag. This metric indicates the generation number of the consumer group, incrementing by one with each rebalance. It serves as an effective indicator for detecting occurrences of rebalances.
Coming Soon: Kafka Transactions
As we announced in our previous newsletter, the team is working on building in support for Kafka Transactions and expects to finish this work soon. If you want to use WarpStream for a workload requiring Transactions, please contact us! We would love to chat.
Try WarpStream With $400 in Free Credits
WarpStream is free to try. After you create your account, it will be loaded with $400 in free credits so you can test how easy it is to set up and use WarpStream.
