Video: 10 Minute IT Jams - An update from Materialize

Materialize says it wants to change the way we think about data.

The company, which describes itself as a fast distributed SQL database built on streaming internals, aims to make real-time analytics accessible to all businesses, not just the big technology players. Speaking to Terminal It Jams, Arjun Narayan, CEO of Materialize, explained how streaming databases could transform the landscape.

"Materialize is a streaming database that looks like a regular SQL database except both on streaming internals," Narayan began. "Traditionally, people are familiar with largely two classes of databases. They have their transactional databases – your Postgres, your MySQL, your Oracles – where you put data in, you get data out. And it's sort of the workhorse of your application stack. Folks typically also have an analytics database – your Snowflakes, your BigQueries, your Redshifts – what people use to data-mine, analyse and inspect, and get complex answers to complex questions."

He explained that typically, organisations "move data from the transactional database to the analytics database but once a day at best, maybe once every few hours if they're running this continuously, but oftentimes a lot slower than that too – maybe once every few days or once a week." This, Narayan said, means that analytics are always "a lot more out of date than they would ideally like."

This situation leads to a costly trade-off for companies who need up-to-date information. Building a real-time stack – using stream processors like Apache Kafka or Apache Flink – is possible, but complex and resource intensive. "People have built complex microservices using a pile of Java code and Kafka and Flink and stream processors like that to build a parallel pipeline that is real-time," he said. "When building something complicated like this, you have to put in a ton of resources."

This is precisely the problem Materialize wants to address. Narayan posed the question at the core of the company: "What if it was as easy to build real-time, always up-to-date experiences using a database that people are already familiar with? If it was as easy to use as the analytics databases that people use today in batch, except that it was always up to date, in real time, live as of milliseconds. What would you build?"

The answer, he argued, is that you could not only have always-fresh analytics but also build entirely new types of experiences. "It's not just that you would have your analytics always fresh, which is of course a place where a lot of our customers and users start. It's oftentimes the first place that they think of deploying a streaming database – to just have their batch answers live and real time – but what are the new experiences that you would build? How would you roll this analytics information and machine learning back into your application when it can inspect all of the data that you've already seen through your transactional, analytical systems but always be up to date?"

Narayan made clear that delivering this capability is currently a significant challenge. "Today it's really complicated and hard to build real-time experiences. You have to use Kafka, Flink, and all these complex microservices. The word 'streaming' oftentimes elicits a bunch of groans from developers who go, 'Oh, I've got to build and maintain a lot of infrastructure just to get that first bit of real-time experience back in my application.' We want to avoid all of that. We want to make it really easy to build real-time experiences using the SQL that people know and already love."

Describing Materialize's approach, he said, "It looks and feels like Postgres… It's built on top of foundationally different streaming internals than traditional databases, so it does have a fully state-of-the-art stream processor under the hood, but from the user experience it is just the same SQL. It looks and feels like Postgres."

Asked how streaming data could enable new types of real-time applications, Narayan said, "We think that users may begin with taking traditional batch analytics and making it real time, but really, it changes the kinds of things that you would want to build, or expands the possibility space of your application." He cited examples from Silicon Valley's tech leaders. "You could think of the dynamic pricing that Uber does when supply and demand conditions change in a given city – that is all gone today with streaming data that has been continuously recomputed."

But the ambition, he said, is about democratising this technology. "In order to move beyond just the few companies that are able to build and deploy hundreds of micro streaming microservices at scale, to sort of the entire economy of every single company taking advantage and building the same real-time experiences, is going to require tools like Materialize that make it simple and easy to build without really changing and ripping apart your whole enterprise architecture and rebuilding on top of a new technology."

Materialize's main users are data teams, said Narayan – "primarily data teams that today are using a Snowflake or DBT, building and deploying analytics models" – but existing batch analytic stacks have limitations. "When they really want to move that to production, when they have an insight… deploying that into production becomes a real hurdle because your traditional batch analytics stack is unable to surface that back into application the same day, even when really you need to deploy that back within milliseconds or seconds. That's where something like Materialize can really help you because you take that exact same SQL and put it into production on top of the live, changing data without really having to rebuild or re-architect your entire enterprise architecture."

Looking to the future, Narayan is optimistic. "We have a wonderful cloud product that looks and feels really like the cloud analytics data warehouses that you already know and love, and our hope is that every business will become a real-time business." He argued that with real-time analytics available at the same ease as current batch-based systems and, in some cases, even cheaper, "Why wouldn't you opt to have this ubiquitous through your organisation servicing the most up-to-date data?"

Ultimately, Narayan's vision is a world where every business can harness the power of streaming data, simply and cost-effectively. "Our hope is to make every business be a real-time, up-to-date business," he said.