What I Wish I’d Known When I Started With Apache Kafka®

This age of connected devices in every nook and cranny of our lives has driven an appetite for data insights in real time. As consumers, we want real-time status updates on our ride shares, our pizza orders, and our home thermostats. For tech-driven organizations, the real value lies in insights into understanding customer buying trends, supply chain changes, and financial market movements.

As the de facto platform for event streaming, Apache Kafka® has proven time and again that it meets the requirements for these use cases and many others. Just ask 80% of the Fortune 100 that trust Kafka as the backbone of their data planes.

Given the time I’ve spent both developing with Kafka as well as operating it, I’d like to share some of what I’ve learned along the way—the good, the bad, and the ugly. Here’s a brief overview of what I wish I’d known when I started.

My Early Journey With Kafka

In a previous life, I worked for a mobile advertising startup using batch processing to manage campaigns and reconcile ad spend. Our advertisers needed to know how their ad dollars were spent as they configured their ads. Meanwhile publishers were paid on campaign performance, getting attributions for the impressions, clicks, and other user events from the ads they displayed. This literally ran our business, and without it, we couldn’t bill the advertisers or pay the publishers without directly affecting our bottom line.

Our team wasted a lot of time and shed many tears repeatedly fixing and rerunning the same brittle ETL pipelines. The pain only increased as the data volumes grew, until the skyrocketing infrastructure costs that were required to process this data forced the C-level executives to take notice.

As we made our foray into real-time bidding (RTB), closing the feedback loop on ad performance was the key factor in measuring success. Our batch-driven approach meant it could be hours before we gained any insights from the data—and delayed insights aren’t exactly useful when servicing more than 100,000 requests per second.

With the increasing costs and need for real-time feedback, we needed a new paradigm. After a “bake-off,” we quickly landed on Kafka as the backbone of our data plane. Initially, my team focused on simple use cases such as producing events from our applications to build an audit log. As we proved the viability of Kafka and built our understanding of what it could do, we moved to more complex processes. We closed the loop between analytics and stateful stream processing by joining our bids with the impressions served and user interactions with those ads.

Ultimately the RTB business took a more targeted approach, using fresher data about campaign performance. We could now focus our ad spend on areas where the data directed a higher probability of success.

As our data pipelines became more reliable, we began to bring new data sources into our streaming ecosystem. Development teams added new functionality as our application ecosystem became more decoupled, untangling the infamous data mess.

Along the way, we learned not just about Kafka but also about building a resilient, scalable data architecture. So with that background, here are my top tips and things I wish I’d known before starting this Kafka journey.

Top Tips for Getting Started With Kafka

• Start small: Don’t start trying to boil the ocean. Consider a scope of low-risk, simple use cases that help you master the core concepts of Kafka and event streaming patterns. Use cases like log streaming or basic event processing can give you a solid foundation. These early wins will help improve buy-in from interested parties.

• Focus on the basics: Learning key concepts like data governance, developer education, and an established approach to the software delivery life cycle are essential building blocks for succeeding with Kafka.

• Define events thoughtfully: Kafka can exponentially benefit an organization by allowing teams to independently use the same source data. To make this work well, carefully plan the data you’ll share with other teams. Design schemas for those events and use industry-standard serialization formats such as Avro or Protobuf. For bonus points, embrace the broader principles of data contracts.

• Key design and partitioning—learn it and live it: For each Kafka-based use case, take great care in planning your key strategy. Concentrate on factors like expected data volume. Design the keys with the goal of distributing the data as evenly as possible across the cluster. Take into account your processing service level agreements to help determine the number of partitions as you create topics.

• Infrastructure as code (IaC): Your Kafka infrastructure configuration will evolve as you refine and expand its use. Apply established DevOps principles such as IaC with tools like Terraform from an early stage. Your future self will thank you.

• Avoid the premature optimization trap: Knowing how to optimize and tune your cluster (and your client code) is almost as important as knowing when to do it. Leverage performance testing and observability tools to understand which knobs and levers to tune. The Kafka documentation goes into detail about related settings.

• Consider a managed solution: If your team doesn’t have the right experience or resources, a managed and hosted Kafka solution can help. Confluent’s complete data streaming platform has multiple offerings to meet your team’s event streaming needs so that your developer and operations resources can focus on building your application logic.

Sounds Great! Where Do I Start?

Always consult the experts in this space. I encourage you to bookmark Confluent Developer, a  curated collection of courses and examples for all things data streaming. Kafka 101 is a great starting point for the Kafka-curious developer. There’s also YouTube for a variety of informative and fun videos unpacking a number of topics.

The Confluent Blog has many how-to guides and thought leadership posts from our team of data streaming professionals. You can also join the Confluent Community Slack channel, where you can ask your burning questions on a variety of topics.

We can continue the conversation on social media. Find me on LinkedIn, X, and BlueSky.

Apache®, Apache Kafka®, and Kafka® are registered trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by the Apache Software Foundation is implied by using these marks. All other trademarks are the property of their respective owners.