Data sits at the heart of business competitiveness in the current market landscape, but it seems businesses still have a long way to go before getting a handle on their data, with close to 70% of IT leaders citing the inconsistency of data sources as a major issue or frequent challenge.
This is according to new research by Confluent, which reveals that 65% of IT leaders surveyed believe that the uncertain timeliness or quality of data is also a major issue or frequent business challenge, as is data spread across separate silos throughout the organisation, a factor flagged by 64% of survey respondents.
Overall, 76% of IT leaders cite five or more data-related challenges impacting their organisations, including siloed data and governance-related challenges. This should come as no big surprise, given the role that digital infrastructure plays in creating and maintaining competitive differentiation in today’s business landscape.
Confluent’s 2024 Data Streaming Report, which captured the perspectives of over 300 local IT leaders and more than 4,000 globally, also found that 94% of respondents cite five or more areas where data streaming platforms (DSPs) can help combat such challenges in most and many situations.
Indeed, it turns out that DSPs — software platforms that empower businesses to stream, process, connect and govern real-time data streams — are seen by 51% of those surveyed as being able to enable organisations to be truly nimble. They do this by providing real-time visibility into operations and customer interactions.
Having access to the right data at the right time is crucial to gaining timely insights into everything from business operations to market trends, customer needs and more. Against this backdrop, data streaming platforms are emerging as the key solution for businesses to reach their goals and get a competitive edge in multiple areas.
Generating AI leadership
Artificial intelligence (AI) has been a major driver of efficiencies in business for years. However, with the emergence of easily-accessible platforms like ChatGPT, generative AI is swiftly being used by more organisations in many new and innovative ways.
With generative AI front of mind across the enterprise landscape, 63% of IT leaders surveyed say DSPs are extensively or significantly fuelling AI progress by building the real-time data foundation needed to propel such initiatives. This is because for AI models to generate accurate and relevant results, the data must be trustworthy, in the right format, and as real-time as possible.
Retrieval augmented generation (RAG) has emerged as a common pattern for building GenAI-enabled applications by connecting general-purpose LLM models to domain-specific data while allowing granular access controls and minimising hallucinations. Data streaming enriches RAG-enabled workloads with trustworthy and contextual data by tapping into a continuous stream of real-time data from systems that power the business. It also becomes a competitive differentiator by contributing to things such as real-time customer service chatbots, real-time business intelligence, instant financial transactions and more.
With businesses facing such competitive demands, DSPs are increasingly playing a key role in informing AI models aimed at underpinning many real-time business tools. A full 90% of IT leaders surveyed said that data streaming enabled product and service innovation around AI and machine learning (ML), according to Confluent’s research.
Solving business security
While we’re seeing rapid GenAI adoption, the market has also seen fresh growth in the cyber security arena. Cyber security has long been a board-level issue in Australia, a factor that is reflected in the investment priorities of businesses.
According to Confluent’s research, cyber security continues to be one of the top strategic priorities or an important priority for IT investments, cited by 96% of research respondents, sitting above other big investment categories in terms of popularity, including data governance (87%), data streaming (86%) and even AI and ML (81%).
It should be noted that data governance, data streaming and AI or ML are technologies that are typically employed to underpin cyber security solutions. Real-time threat analysis is an essential element in the protection of businesses today, and DSPs make the real-time data analysis needed for such solutions possible, with 95% of those surveyed saying their data streaming activities have improved cyber security and digital risk management.
With DSPs, businesses can combine the best real-time data streaming infrastructure with their cyber security platforms for a more proactive detection and prevention against malicious acts, enabling more effective cyber security protections in a business environment that needs them.
Unpacking unified insight
The key element that ties together all of an organisation’s digital differentiators is data. Yet over six out of ten of those surveyed in the Confluent research indicated that data spread across separate silos is a challenge — the top such challenge cited. And without unified data, businesses can’t leverage their most valuable asset.
However, a whopping 93% respondents said DSPs help them break down data silos in most and many situations. Not only that, but DSPs give them access to data in motion, meaning that the tools underpinned by data, such as AI-powered customer service chatbots, can be real-time in nature, using instant, up-to-date and reliable data at any time. Such real-time, unified data can drive differentiating customer experiences.
For instance,Kmart Australia Limited uses data streaming to power the OnePass digital loyalty program, which sits within its parent company, Wesfarmers, granting the company an opportunity to study the customer behaviour of OnePass users on a grand scale and then personalise offers on an individual scale.
No surprise, then, that 51% of surveyed IT leaders cite data streaming as a top strategic priority for IT investments in 2024 compared to 44% in 2023. What’s more, 68% expect the use of this technology to continue to grow over the next two years — a clear indicator of where businesses should be looking to solve their top challenges and create an advantage against their competitors.
Regional Director of Australia and New Zealand (A/NZ) at Confluent
