Confluent: Data architecture gap impairing AI ambitions

Australia's Consumer Data Right (CDR) reforms are exposing significant gaps in the underlying data architecture of organisations, with businesses that have modernised their infrastructure better positioned to respond to regulatory changes and the growing demands of artificial intelligence.

Regulation itself is relatively straightforward for organisations to address.

The more difficult challenge is ensuring data is structured, accessible and available quickly enough to support new use cases, according to Country Manager ANZ at Confluent, Simon Laskaj.

"Regulation is the easy part," Laskaj said.

"The architecture - that's the really difficult part."

Architecture encompasses the state, scale and accessibility of an organisation's data infrastructure.

Companies that have already invested in modernising their data environments are ready to respond to CDR requirements, while those operating on older infrastructure face a potentially significant modernisation effort.

"If organisations have modernised or improved the state of their data infrastructure, they're more ready for the reform," Laskaj said.

"Those that haven't are way behind, and there's a lot of work that needs to be done."

The issue is becoming increasingly important as businesses move towards systems that require access to information in real time.

Traditional enterprise systems have often been built around batch processing, where data is collected and processed periodically rather than continuously.

That process can create considerable lag between an event occurring, and a company being able to act on the information.

Businesses need to think about the 'freshness' of their data - how quickly information needs to become available for a particular use case.

"Organisations run on batch, but in life, we run on continuous information. We want it now, we want it available, and we want it constantly informing us of different information," Laskaj said.

Instead of treating real-time data as a binary concept, organisations must determine how fresh their data needs to be for individual applications.

That could mean milliseconds for some applications, while others may only require information every few minutes.

The shift towards fresher data is also being accelerated by artificial intelligence, particularly the emergence of agents capable of taking actions on behalf of organisations.

AI systems are only as useful as the information available to them, making timely access to relevant data increasingly important.

Laskaj explained that context will become one of the most important elements of enterprise data infrastructure as organisations deploy more AI agents.

"Context is absolutely going to be king of the data," he said.

The challenge for companies is moving beyond simply transporting large volumes of information and instead ensuring systems can understand what that information means at the time it is being processed.

Confluent's technology has traditionally been built around data streaming, with Apache Kafka - the open-source event streaming platform - designed to move large amounts of data at scale.

The next phase is all about adding context to that data as it moves throughout an organisation.

"You can move it, but in what state does it land, and what do you do with that when it lands?" Laskaj said.

Having the ability to capture contextual information while data is moving could allow organisations to make decisions much earlier than they have previously been able to.

Laskaj cited use cases including fraud detection and preventative maintenance in manufacturing, where delays in accessing crucial information can directly affect an organisation's ability to respond.

In traditional environments, some decisions could be made 24 hours or several days after the relevant event has occurred.

Real-time data infrastructure combined with AI could, instead, allow organisations to identify patterns and act while an event is still unfolding.

The rise of the agentic age is expected to further increase that demand, with agents requiring up-to-date information to make decisions and perform tasks.

Bendigo Bank's early move 

One financial institution that is well-positioned for this shift is Bendigo Bank.

The bank has been a Confluent customer since 2020 and was among a smaller group of Australian financial institutions that chose to adopt Confluent Cloud rather than initially managing the infrastructure themselves.

That decision put them among early adopters of cloud-based data infrastructure at a time when many Australian banks remained cautious about moving workloads into the cloud, partly because of regulatory considerations in the tightly regulated Australian finance sector.

Bendigo Bank's early investment in modernising its architecture has also helped prepare it for changes such as open banking and the broader CDR reforms.

A key component of that modernisation has been moving towards a more decoupled architecture.

In traditional tightly connected systems, applications can be heavily dependent on one another, meaning the failure of one system can potentially affect multiple applications.

A decoupled, event-driven architecture can instead allow individual applications to fail without necessarily bringing down the broader business process.

"If there's a failure, one application can come down," Laskaj said.

"But the business use case or the outcome is still able to be achieved through an event-driven architecture."

That architecture is becoming increasingly relevant as companies expose more data to third parties and consumers under frameworks such as CDR.

Australia's approach to data governance also creates particular requirements for organisations operating in the market.

Australia is relatively advanced in how it approaches data governance and consumer data protections, pointing to reforms across industries including banking and energy.

Governance requires organisations to understand where data is held, who can access it and how it moves through an organisation.

However, stronger regulation also increases the pressure on businesses to ensure their underlying infrastructure is capable of meeting those requirements.

"If the organisation doesn't have an optimal structure of that data, consumer rights could be challenged," Laskaj said.

For businesses preparing for the next stage of Australia's data reforms, the implication is that compliance cannot be considered separately from technology infrastructure.

As CDR expands and organisations increasingly deploy AI and automated systems, the ability to make data available in the right form, with the right context and at the right time will become a competitive differentiator.

The organisations that invested early in modernising their data architecture are therefore at an advantage - not simply because they can comply with regulation more easily, but because they are better positioned to build the real-time and AI-driven applications increasingly being demanded by customers.