How Singapore Embraces Data Streaming Across Finance, Air Travel & More

Real-time data has become an essential asset for today’s businesses, and the Asia-Pacific market is no exception. Across competitive industries like finance, travel, and more, data streams and event-driven architectures have become table stakes for leading organizations in the region.

Today, having access to real-time insights and event-driven operations not only lays the foundation for responsive business processes, increased efficiency, and better customer experiences, but it also paves the way for artificial intelligence (AI) use cases with real business and revenue impact. Learning how it plays out in an actual business setting—and how it integrates with a company’s specific architecture—is the way to truly understand the potency of real-time data streams.

At a recent Data in Motion Tour (DIMT) in Singapore, Cathay Pacific and Endowus participated in fireside chats to share how their organizations derive value from real-time data. The event also featured a panel of data streaming experts discussing the importance of quality real-time data to future generative AI (GenAI) efforts and product development. Here are the most significant takeaways from this event.

Interested in how global IT leaders are using and benefiting from data streaming? Read the 2024 Data Streaming Report.

How airline Cathay Pacific uses data products to enable productivity and innovation

As a premium airline in the Asian market, Cathay Pacific is steadily growing into a leading brand that includes other travel and tourism services as well. In light of this growth, the engineering team wanted the ability to treat data as a product, share trustworthy data products across different teams and departments, and thereby enable more streaming use cases that would advance the business.

The company’s data streaming journey actually began about seven years ago when Cathay’s IT team started to build on microservices, APIs, and messaging services. Quickly, it became obvious that they needed the ability to support high throughput with low latency so they could scale better and more efficiently. As Cathay’s engineering team began to work with more and more microservices off the shelf, Apache Kafka® became an essential part of the solution.

Ultimately, this move led Cathay’s engineering team to migrate to Confluent’s fully managed cloud-native Apache Kafka service. Using Confluent Cloud meant the engineering team could use private, dedicated clusters to maintain data security standards while also granting access to the fully managed connectors available out of the box. These advantages added up to a reduced time to market for any new products and services that required data streaming.

As a critical part of the data strategy, Cathay now relies on Schema Registry for data governance when implementing Kafka use cases. Integration Architect Lead Marc Keng said, “We want to make sure all the data we ingest adheres to a certain level of data quality so it becomes a standard schema. That way, there aren’t any surprises when we start to consume this data down the line.”

With a complete data streaming platform, companies like Cathay Pacific can shift data governance and processing left to build highly trustworthy and infinitely reusable data products.

Today, Cathay teams use Confluent’s governance capabilities to define quality rules for each attribute, configure data queues out of the box, and conduct more policy enforcement. This approach allows various business units to quickly and consistently derive value from correlated data or raw data—whatever suits their needs.

“It's not just a one-time story,” Keng said. “This is really how we see the evolution across the organization if you provide data in good quality and with good access control into the different units.”

And with the underlying Kafka architecture, it’s easy to travel back in time and replay historical data that has not been processed. But more importantly, data does not need to be processed every single time, which helps the company save on compute costs. As Cathay teams integrate more types of consumer offerings—not just flights but shopping opportunities and more—the underlying data products will have to scale to match. By using Confluent Cloud, engineering and business teams will have reliable access to the data and microservices they need without having to worry about Kafka operations or management responsibilities.

How Endowus overcame production bottlenecks with Confluent

Endowus is a high-profile financial consultant based in Singapore—and Asia’s leading fee-only wealth platform, purpose-built to meet a wide range of investment needs, including investment in public markets, private public markets, equities, fixed income, private markets (like hedge funds), and multi-currency. This range of investment types requires data to flow between microservices in a highly scalable way.

At the Singapore DIMT tour stop, Deepak Sarda, chief technology officer (CTO) at Endowus, talked about how the company marries its business vision with its technology strategy in a highly regulated industry. Their microservices-based architecture relies heavily on domain-driven design. The company uses roughly 20 microservices for different domains of the wealth management platform, using Cassandra for persistent storage of transaction data and PostgreSQL as its read-only database.

“Kafka,” Sarda explains, “is the technology that holds all of these services together because we rely heavily on asynchronous event-based communication between them.”

Despite how resilient Kafka initially proved in this model, self-managing open source Kafka required a lot of manual upgrading and calibration. As the number of projects and engineering teams using data streaming grew, many were relegated to sharing the same few software testing environments, which created bottlenecks and stumped agility.

Sarda told the DIMT audience, “We knew that we wanted to build a purpose-built digital investment platform that would scale and scale in many dimensions… We needed the ability to spin up our entire digital platform on demand, in an ephemeral environment, with the data we need, with the services we need, and with the application code we need.”

Migrating to Confluent delivers three primary benefits for Endowus when it comes to scaling and hardening its event-driven architecture: 

1. Core production databases are managed well, with the security resilience they need from their core infrastructure component.

2. Relying on Confluent’s fully managed service has not only reduced production bottlenecks and downtime but also increased cost efficiency and returns on the company’s investment in data streaming.

3. When building new products, they can use real Kafka in dynamic environments, which enables far more product innovation, more efficiently. They can provision environments quickly and take them down just as quickly within the production environment. When Endowus launched the Hong Kong platform, it only took six months to go live—unprecedented.

Notably, Sarda called the cost-efficiency benefits “an underappreciated aspect of scaling that leads to stronger innovation and is ultimately cost efficient with pay-as-you-go clusters.” 

Experts from Google Cloud, IDC, banking, and the Singapore government discuss real-time AI

After Keng and Sarda spoke of how their companies are using data in motion right now to innovate and elevate their businesses, a panel of industry experts had a roundtable discussion on how the future of AI depends on real-time data streams.

Dr. Chris Marshall from IDC pointed out that the conversation around GenAI use cases may be fairly new, but plenty of enterprises have been engaged in predictive AI projects for a good decade. In fact, predictive AI is still where the bulk of AI spending sits. He says, “The discrepancy between the hype of GenAI and the reality of predictive AI is slowly being brought together as expectations set by the board are being tempered by the reality of availability of quality data.”

Quality and also quantity of real-time data is where data streaming comes into the AI conversation. It’s important to note as well that for companies with any sort of regulatory burden, as several of our speakers represent, there’s also a risk component that has to be minded. For this reason, these companies are making large investments in how to structure unstructured data in data lakes and warehouses.

But before any company can truly capitalize on GenAI in a meaningful, business-critical way, it needs to first revisit its data architecture. Marshall put it bluntly: “Data streaming and event brokering are fundamental parts of the GenAI capability.”

His belief is that the only way forward is to have data consumers able to subscribe to data streams across the enterprise in an unfettered but still secure and compliant way: “I think data streaming is just such a fundamental part of that story, and I think maybe we've realized a little too late, frankly, that AI is conditioned on having that data fabric story in place before it can be successful.”

Different challenges, same solution

One of the reasons we host DIMT tour stops is to give a diversity of storytellers a platform to describe how data in motion is revolutionizing not just the world of business, but their specific businesses, enabling enterprise organizations from finance to defense to travel to get ahead of their competitors and defray the costs of innovation.

The challenges that brought Cathay Pacific, Endowus, and our panel participants to partner with Confluent were very different. Their stories, though, converge in one way: the importance of data streaming to enabling business-critical technology initiatives.

Watch the full session recordings for Cathay Pacific, Endowus, or this thought leadership panel from the APAC Data in Motion Tour and catch up on other topics covered. For more insights and information on how business leaders are seeing real-world benefits from data streaming, read the 2024 Data Streaming Report.

Apache®, Apache Kafka®, and Kafka® are registered trademarks of Apache Software Foundation.