New research focuses on how enterprises benefit from the increasing amount of streaming data.

Swim, the developer of the industries first open core platform for continuous intelligence at scale, conducted the research in partnership with Virtual Intelligence Briefing (ViB). The survey found almost half (48%) of participants continuously generate insights from streaming data in-stream without storing the data first. The survey found the number one business outcome organisations expect when using that approach is improved customer experience.

The survey included over 200 IT professionals from a cross-section of mid-to-large companies, with more than half from companies with 5,000 employees or more. All the organisations use event-driven architectures, along with other streaming technologies.

On average, the surveyed organisations use three different open source streaming technologies, with Apache Kafka being the most popular, used by 87%, and Apache Pulsar used by 17% of organisations. Other popular open source projects used for streaming data solutions include Apache Cassandra, Apache ActiveMQ and Apache Spark Streaming.

"Customer perceptions and decisions, driven by a complex set of variables, are formed in real-time, in a competitive dynamic that changes moment to moment, from the unique buyer level, up to the market as a whole," says ViB director of research, Tom Riddle.

"Creating the highest quality customer experience requires real-time, in-stream analysis of immense volumes of data, driving real-time adaptation. Swim's solution democratises this process, greatly reducing the time, complexity, and resources required to build applications that create the most compelling customer experiences."

Key findings include:

• More than a third of organisations surveyed are building streaming applications. Event-driven architectures are used on average for more than four purposes, with the most important being for data pipelines, messaging, microservices, data integration and stream processing.
• Custom-built streaming applications dominate, with 70% of the surveyed organisations building their own custom streaming data solutions, while 30% choose to source commercial applications or cloud services when available. Most organisations (71%) are building their own custom solutions by integrating custom logic with available infrastructure components, while the remainder use a pre-integrated commercial platform to benefit from consistency and reusability.
• Python and Java skills are most prevalent. When it comes to the technical skills used to develop streaming applications, Python (60%), Java (56%), and Kafka Stream/KStream (50%) took the lead among organisations, followed by Javascript (39%), SQL (37%) and C#/C++ (35%).

"Streaming applications that continuously generate insights, drive actions and help build always-on, situational awareness of the current state of the business, are the new frontier for modern, data-driven organisations," says Swim CEO, Ramana Jonnala.

"This research shows, among other things, how organisations can use massive, boundless flows of information to improve the customer experience."