Grafana Labs buys Logline to speed Loki log searches

Grafana Labs has acquired Logline, adding search technology for log queries involving highly unique values.

Logline specialises in full-text search and what Grafana Labs calls "needle-in-the-haystack" queries. The acquisition is intended to improve how Loki, its log aggregation system, searches very large datasets for specific identifiers such as request IDs or job IDs.

Loki uses a label-based indexing model designed to keep storage costs low and simplify operations at scale. That approach works well for many common logging tasks, but searches for highly unique values can take longer as data volumes grow.

Logline adds a different indexing method built for high-cardinality attributes stored over object storage. In practice, that should let users find specific values more quickly without changing Loki's underlying design.

Founded by Jason Nochlin, Logline emerged from discussions about the limits of existing approaches to log indexing, according to Grafana Labs. Nochlin previously led Teleport Data, which Fivetran acquired.

He described the origin of the technology in his own words: "After that conversation, I started thinking about new ways to do indexing over object storage. It took awhile, but eventually I had a breakthrough and thought, 'wow, I may be onto something here-maybe Grafana Labs will be interested.'"

Grafana Labs said the new indexing approach is designed to reduce the amount of data scanned during certain searches. In early benchmark figures it cited, a query for a universally unique identifier, or UUID, in Loki previously scanned 3.5 TB of data without returning a result. With Logline, that fell to 8 GB, a 99.7% reduction.

Search focus

The issue is especially relevant for engineering teams operating at large scale that need to search logs for rare events, failed jobs or individual request traces. These lookups can become slow and expensive when they depend on scanning broad datasets rather than using an index built for unique values.

Grafana Labs said Logline is meant to address that problem without relying on more computationally intensive methods. "We want to drive down the time it takes to perform these searches without having to introduce techniques that are much more computationally expensive," Nochlin said. "So Logline is the best of both worlds, where we can accelerate those needle-in-the-haystack searches with much simpler indexing than anything else that's on the market today."

The acquisition also reflects a broader effort by Grafana Labs to extend Loki beyond its original strengths in cost-conscious log management. Combined with other architectural changes to Loki, the Logline technology is intended to support faster large-scale scans, reduce the impact of stream cardinality and improve performance for analytical workloads.

Open source link

Grafana Labs linked the deal to its broader open source strategy. Loki is an open source project and also underpins Grafana Cloud Logs, its managed logging service.

The new search feature is already available in limited private preview in Grafana Cloud Logs, according to Grafana Labs. The company is also working to bring the functionality to users of the open source Loki software in a later major release.

For Nochlin, part of the appeal was that model. "The mission of making observability ubiquitous while also keeping costs under control really stood out to me about Grafana Labs," he said. "You don't usually see that in this space."