Two weeks ago, I had front-row seats as an analyst at the Confluent Current 2024 event in Austin, TX. Unlike previous events where the Confluent executive team layered multiple announcements about various new innovations and products, this year was a little different. The announcements seemed very strategic about their next set of moves in the market related to growth and expansion. Don’t get me wrong; we saw a few exciting product announcements, too, but a lot of them have been in the news cycle before.

Competitive Landscape

From its humble beginnings in LinkedIn, where Apache Kafka was born, Confluent has risen into an undeniable strength of force in the data streaming market. But in the last couple of years, several smaller players have started nibbling at its market share, compellingly sustained it, and are growing, too. 

Cost as a differentiator — The above-said competitive pressure could be easily ignored for the kind of brand and market share that Confluent has built up. But, there has been one crucial factor that the competition has been playing on well, which has not been working well in Confluent’s favor – Costs! Particularly cloud networking costs. Kafka can scale exceptionally well, but when you start running even a medium-sized set of clusters on the cloud, the costs will start adding up over time. During the event, I spoke to a Confluent customer at my lunch table. She worked for one of the large financial institutions in the country, and they have been using Confluent Cloud for the last 3.5 years. She did say that running Kafka on Confluent Cloud is becoming very expensive. She didn’t say that she was looking for alternatives, but if you read the room, you will see that the alternatives are waiting to grab an opportunity like this.

Feature catch-up — Not that any product has come truly competitive in terms of scale and speed when it comes to Kafka, but the reality is that not all use cases are the same either. So, when the answer is “it depends,” everyone has a fair shot at this game. Alternative solutions in the market have built up a truly competitive set of features and capabilities that would encourage buyers to set up a PoC in order to determine what solution is right for them. There are actual Kafka alternatives in the market today, which wasn’t the case a few years ago. There are vendors that are already supporting deployments like BYOC (Bring Your Own Cloud), and Confluent is left playing catch-up with them.

Level playing field — One common point of contention in the industry is the number of open-source Kafka deployments in the market. There are analyst firms that quote them to be in the tens of thousands, and there are some that say it is 100K. At the event, I heard the number to be 120K. Out of which, Confluent supposedly supports only 5,500. This means that Confluent still has a significant market (low-hanging fruit) on which to capitalize. However, the same is true for the alternative vendors as well. 

Current 2024 Announcements

So, with all that said, the question begs what Confluent should do to retain its market dominance and continue to expand and grow across all types of deployment models. The answer is, hopefully, in the series of announcements at Current 2024 –

Warpstream acquisition — This was probably the most significant news at the event. Jay Kreps, CEO of Confluent, used a good visual in his presentation to show how Confluent has been able to offer solutions for two ends of the deployment spectrum — full-managed and self-managed. Now, with the Warpstream acquisition, they are addressing the middle of the spectrum for customers who want to deploy Confluent on their own clouds, aka BYOC. BYOC uses a shared responsibility framework where the customer and the vendor are jointly responsible for the operations and health of the system.Warpstream
  • New OEM program — Confluent announced the new Confluent OEM program, which enables MSPs, CSPs, and ISVs to offer their customers the full Confluent streaming platform with Kafka and Flink. They will have the license to redistribute or embed Confluent’s enterprise-grade platform globally. Given that such partners stand to benefit from a lot of revenue from implementation and maintenance, this is a win-win. Confluent has committed itself to offering expert implementation guidance and certifications to help partners launch such offerings.
  • Commitment to partners — Confluent is launching its first-ever Partner Advisory Board this year. In an event touted as “Growth Kick Off,” Confluent plans to encourage its sales teams to network with its partners and boost joint GTM motions. With its partner enablement teams, Confluent plans to nurture more “Data Streaming Engineers” across the partner ecosystem. Confluent’s CMO, Stephanie Buscemi, has also reportedly set goals to have 75% of their overall marketing pipeline attached to partners. On a related note, they are also incentivizing some of their partners to push the Confluent Migration Accelerator program to help companies move from other messaging platforms to Confluent.
  • New developer tools and features for Flink on Confluent Cloud — Confluent’s latest support of Table API makes Apache Flink available to Java and Python developers. Confluent Cloud now offers private networking support for Flink, providing a critical layer of security for businesses that need to process data within strict regulatory environments. There is also a new extension for Visual Studio Code that offers a seamless development environment for building Kafka applications. You can manage Kafka topics, code and debug Kafka clients with features like code complete, and seamlessly manage cloud resources within VS Code. Confluent has also introduced client-side field-level encryption to protect sensitive data and adhere to compliance requirements.
  • A few other announcements, like support for Apache Iceberg with Tableflow, were more like reiterations of past announcements.

Data Streaming and AI

Lately, there has not been a single event in the industry where the company doesn’t talk about AI and where they are in the journey with their AI innovations. And, Confluent was no exception to this. Jay Kreps, in his keynote address, set up the point very well as to how today’s AI applications or LLMs have significant challenges in terms of getting trained on enterprise data that is not fresh, the cost and time associated with such training exercises, the inherent hallucinations that make them far less trustworthy, and more. He brought home the point that data streaming brings in fresh contextual data to this setup, making such AI models more relevant and well-informed. He emphasized the fact that a data streaming architecture is more relevant in today’s AI world than ever before. Personally, I cannot agree more with this assessment. We are seeing practical examples of how streaming data brings so much freshness and context into an AI conversation.Jay Kreps

Customer Testimonials

At the event, I heard and met some of Confluent’s customers, and their testimonials were truly a testament to Kafka’s sheer superiority and dominating presence in today’s enterprise architectures.

  • Seema Acharya, Software Engineering Manager at Mercedes-Benz R&D NA, spoke during the keynote. She shared that they run over 800 TB of data per month and over 180 billion events per day through Confluent Cloud to deliver hyper-personalized experiences in their cars.
  • Kushal Khandelwal, Head of Data Platforms, Viacom18 and JioCinema, spoke about how they view managing Kafka infrastructure as not being the best use of their developers, and so they lean on Confluent Cloud. They would rather have them focus on customer experiences. Their mobile app is the most downloaded app in India. During the 2024 IPL cricket season, they had 26 billion views or 350 billion minutes of watch time in 2024. They have over 100 M+ users. All of this runs on Kafka at JioCinema. They receive heartbeat and clickstream data from millions of smartphones from across the country. They are able to handle over 30M – 50M views concurrently, which translates to 2M messages/second – built entirely on their Kafka concurrency system.
  • Vitaly Shoykhet, SVP of Engineering, Audacy – A one-stop shop for podcasts, radio stations, live streams, and other types of audio streams. They support over 5.4 million active listeners with 200K-400K concurrent listeners. They migrated from AWS MSK into Confluent Cloud.
  • Chandra Kuchi, Engineering Manager, Data Platforms, Robinhood Financial – To them, Kafka is the central nervous system between all systems like their central brokerage system, clearing house system etc. They use Confluent platform across 15 on-premises clusters and not the cloud. However, they feel that Warpstream would be highly useful to them, given their internal security policies about moving into a vendor cloud.

Conclusion

With the audience’s pervasive excitement about Kafka and Flink, you can safely say that Confluent is here to stay. However, for its growth and market expansion, it needs to deliver more on the product side faster. For example, the Flink integration and feature set have taken quite a while. Confluent’s current moves on the partner ecosystem, the new OEM program, the acquisition of Warpstream, etc., are all spot on. But the key is to deliver on them successfully. There is also a lot of clarity that can be offered in the field about cost benefits between the various deployment options and how a customer stands to benefit from choosing one over the other. As customers start to realize that there are other products in the market that can offer similar, if not superior, streaming and stream processing capabilities, this space will become commoditized, and the pricing wars will start. Confluent’s focus on AI should be delivered in such a way that it doesn’t move far away from its core competency but rather educates the market on the importance of data streaming in the context of AI today.