itb-au logo
Story image

Interview: Why log management is crucial to modern business

08 Mar 2019

Recently I had the chance to sit down for an interview with Balázs Scheidler, an industry veteran in IT security and software engineering.

In 2000 he co-founded Balabit because he was tired of using the old syslog protocol and not getting valuable data fast enough from log messages for security analytics and IT operations.

In just a few years his syslog-ng log management solution became the de facto industry standard for log collection in Linux-based installations, used by millions of corporations around the world. Having established a name for itself, the company then delved into the Privileged Access Management (PAM) solution space.

Last year Balabit was acquired by One Identity, with Scheidler’s role now focused on shaping the future strategy for its PAM and log management solutions.

Scheidler says syslog was originally created in the early 1980s to debug software as part of the Sendmail project.

"Basically, it was designed to be a fire and forget approach to log management and logging in general. It was very simple as they didn't care too much about how reliable it was or even security in general,” says Scheidler.

Their point was to make it possible to get activity data from the mail servers off those hosts and to their separate repository where they could run their own statistics and reporting.

"Vendors of network gear, applications and OSes started to adapt syslog for their own use, which expanded the scope of data considerably. It was very difficult to extract value from log messages because of the huge amount of unstructured data, making it a bit like finding a needle in a haystack," says Scheidler.

Despite this, businesses still saw the huge value in log messages as if there was an incident or some kind of outage, the only data source was this data. Even if an incident was recognised years after it happened, you could still go back and find the details and root causes.

However, Scheidler says there are still organisations in this age relying on syslog over UDP, which doesn’t guarantee delivery and doesn’t use encryption in the logging, which means they’re being deprived of the many benefits of the latest technology to support their log management

“As I mentioned, the adoption of syslog-ng was significant early on, but today still more and more devices are becoming capable and the demand to integrate them into logging infrastructure is always increasing. Furthermore, log data is needed by more and more downstream systems. A SIEM solution is a natural consumer of log messages, but these days things like UBA or business intelligence systems also tap into logs. Using something that can connect log-producing devices with downstream processing systems is needed and syslog-ng is one of the biggest implementations of this idea,” says Scheidler.

“Fixing the reliability and security issues while adopting the growing set of devices that do logging and be able to feed more downstream systems - those are the main reasons for improving log management practices today.”

In terms of how syslog-ng differs from other log management solutions, Scheidler says the original reason he started the project was to improve reliability.

“Originally it was using UDP only, and then I introduced the TCP based-logging system and we have constantly improved that ever since. So now in a distributed enterprise-wide and global log management system we are capable of never losing messages within the log management infrastructure. We are also very good at security - when it comes to data at rest and data in transit we solve both of these issues,” Scheidler says.

“And we do this at a high scale. So I mentioned the relationship between log-producing devices and SIEMS, and feeding multiple teams which naturally means that we have terabytes of data to transport daily, and scale is a key feature of a system like that. To put it into perspective, syslog-ng can process hundreds of thousands of messages a second, and it does that efficiently. What other systems would do on 10 computers, it can do on one.”

Scheidler stressed that while there is an overlap, syslog-ng isn’t a SIEM solution. Often they can meet all the requirements for SMEs that are unable to afford a complete SIEM system or deployment (because they can tend to be quite expensive), but he generally positions syslog-ng as the ‘glue’ between log-producing and log-consuming devices.

“Effectively it can run on end systems and ensure that the collection, centralising, storage, archival, indexing, and searching functions are all performed on our side,” says Scheidler.

“You don't have to send everything into the SIEM, only those things that you really need to, which enhances the performance of your existing SIEM.”

Looking to the future, Scheidler says they’re moving with the demands of their customers.

“Our priorities now lie in the cloud space. Centralised system functions or log management functions are usually an on-premise system because you have to collect everything that you already have on-premise. Customers now expect us to fold logs coming from Cloud systems into the same repository. For example, they want to see Office 365 logs in their Splunk installation, or they want to see AWS access logs in their SIEM, to talk about it broadly,” says Scheidler.

“And that means we have to integrate with those systems as well and provide solutions for customers to solve this hybrid issue. ‘I have a lot of on-premise systems that produce logs; I have a lot of cloud-based systems that produce logs; I want to see them in the same place be that either in the cloud or on-premise’ - that is the opportunity we’re looking to capitalise on.”

In terms of implementation of the log management solution, Scheidler says it is a very simple process.

The syslog-ng Store Box (SSB) for example can be deployed in just a few days, offering a high-performance, high-reliability log management appliance that enables you to search logs, secure sensitive information with granular access policies, generate reports to demonstrate compliance and forward log data to third-party analysis tools.

Click here to learn how to adopt log management best practices.

Story image
Databricks partners with Google Cloud, enables multicloud for the enterprise
“This is a pivotal milestone that underscores our commitment to enable customer flexibility and choice with a seamless experience across cloud platforms."More
Story image
Women in tech: Equality journey not over
The idea of gender equality represents more than just physical bodies through doors. It is also the notion of perceptions, feelings, stereotypes and opportunity.More
Story image
Video: 10 Minute IT Jams - Who is Logmore?
Logmore is a Finnish company specialising in shipment condition monitoring using an in-house cloud service and QR code tags. With its specialisation in temperature monitoring for food and pharmaceuticals, the company is keeping busy with COVID-19 vaccine shipments.More
Story image
Fergus grows staff by 50%, bolsters C-suite after bumper growth year
Chris Stevens has been appointed as the chief technology and product officer and is tasked with supporting the company’s technical development and product roadmap. More
Story image
Frost & Sullivan breaks down biggest changes in the world of UIs
“UI technologies will be an important component in the shift from the office to a virtual workplace that is mobile and more flexible, while interactive interfaces have expanded the scope of businesses to engage with their customers and build deep relationships."More
Story image
Verizon and Cisco expand partnership and add new SD-WAN offerings
“These new services reflect the significant ongoing joint Cisco and Verizon research and development investments which aim to help customers accelerate change.” More