Confluent unveils Streamhouse & Lightning Tables for AI

Confluent has introduced a Streamhouse architecture and added new query tools to its Data Streaming Platform, including the general availability of Lightning Tables on AWS.

The architecture is designed to give AI agents, applications and analytics systems a governed view of the current state of a business. It brings streaming, connection, processing, governance and query functions together in one platform.

Confluent, which sits within IBM, is positioning Streamhouse as a layer alongside data warehouses, data lakes and lakehouses. It argues that older architectures were built mainly for reporting and batch processing, while AI systems increasingly need current operational data from multiple systems at the moment decisions are made.

A central part of the update is Lightning Tables, a read-only serving layer in Confluent Cloud that stores and materialises Apache Kafka data into tables and keeps them updated as data changes. Customers can query that data through SQL or a direct API.

According to Confluent, Lightning Tables reduces the need to move streaming data into separate databases to serve applications. The service also applies governance controls, including schemas, lineage and access policies, from data capture through to serving.

The wider platform now groups its functions into five areas: Stream, Connect, Process, Govern and Query. Stream is based on Apache Kafka and Confluent's Kora engine, while Process uses Apache Flink to combine and enrich data close to the source. Connect includes more than 150 managed and custom connectors for moving data in and out of enterprise systems.

Within the new Query layer, Confluent is also offering Tableflow to make streaming data available to lakehouses and warehouses in open table formats, and a Real-Time Context Engine to serve business context to AI agents through the Model Context Protocol.

To illustrate the business case, Confluent pointed to an airline disruption scenario in which separate systems for flight operations, scheduling and reservations each hold different versions of events. In that case, an automated rebooking system could propose an itinerary that appears valid in isolation but fails once aircraft location, departure time and seat availability are considered together.

Confluent said Streamhouse is intended to avoid that problem by combining updates from source systems into a shared, continuously updated operational picture. It described the architecture as current, mission-critical and decentralised, with data remaining where it already resides rather than being consolidated first.

Some customers are already using similar approaches. Accenture said it has adopted a Streamhouse architecture based on Confluent across its global business.

"Operating across more than 120 countries, our global enterprise was previously constrained by legacy batch and point-to-point architectures where duplicative data extractions drove up operational overhead and analytics data lagged. Moving to a Streamhouse architecture powered by Confluent decouples data producers from consumers, transforming high-volume event streams into governed, reusable real-time data products that flow seamlessly across operational and analytical systems. With Confluent handling infrastructure automation and providing seamless enterprise connectivity, our developers operate in a standardised environment without the friction of building custom integrations or micro-managing data movement. This continuous data foundation has dropped our time-to-insight from months to days and slashed reporting times, empowering our teams to focus on delivering high-value AI for our 800,000 employees," said Thomas Mahaffey, SAI Chief Data Strategist, Accenture.

Another customer, Extend, said it moved away from traditional message queues and static data lakes as it built systems for pricing, fraud detection and merchant onboarding.

"As an AI-native commerce platform delivering personalised shopper experience and operational and financial advantage to merchants, Extend requires exceptional adaptability across rapid merchant onboarding, real-time pricing adjustments, and instant fraud detection. We initially struggled with traditional message queues and static data lakes that lacked event replay capabilities, persistence, and strict schema enforcement at scale. Implementing a 100% serverless Streamhouse architecture anchored by Confluent Cloud turns Kafka into our operational source of truth, enabling downstream microservices to asynchronously consume and query real-time data without breaking contracts. This inherently decoupled streaming foundation directly provides the flexibility and responsiveness required to scale our operations, adapt to live market signals, and power real-time agentic AI workflows," said Jonathan Kropp, Director of AI and Architecture, Extend.

Confluent said internal testing showed Lightning Tables can serve newly created data in about 100 milliseconds, with query latency of 10 to 100 milliseconds and throughput of more than 1,000 queries per second, although actual performance depends on configuration and workload. It also said the service can support machine learning use cases by storing features computed in Flink for tasks such as real-time personalisation.

"The next era of enterprise AI will be won by the agents that know what's happening in the business the moment they act. That's what a Streamhouse can deliver. With Lightning Tables, our Data Streaming Platform gives each agent and application a live, governed view of the business. This means companies can act on fraud signals sooner, serve customers in the moment and run operations on what's true now versus what was true yesterday," said Shaun Clowes, General Manager, Confluent and Data, IBM.