Grafana Labs boosts log search as AI telemetry surges

Grafana Labs has added faster full-text search to Grafana Cloud Logs and introduced Adaptive Logs Archive in public preview, as it looks to manage rising telemetry volumes linked to AI use.

The search improvements for Grafana Cloud Logs are now generally available, while Adaptive Logs Archive has entered public preview. Grafana Labs also announced broader updates to its Adaptive Telemetry tools for metrics and traces.

The changes are aimed at a familiar problem for engineering teams as software output rises and AI systems generate more operational data. Grafana Labs cited survey findings showing that 78% of 300 IT and engineering leaders increased observability spending in the past year, while 73% expect to do so again.

The faster search in Grafana Cloud Logs is built on technology from Logline, a business Grafana Labs acquired earlier this year. The feature is designed to help users identify low-frequency values such as request IDs and customer IDs in object storage more quickly, at no extra charge.

That matters during operational investigations, when engineers may need to locate a specific log line quickly after receiving an alert. Grafana Labs added that software agents are now sending more queries than human users, increasing pressure on search performance.

Jen Villa, Senior Director of Product at Grafana Labs, outlined the company's view of the cost challenge facing observability teams.

"The economics of observability only work if you keep the data worth keeping and can dial that up or down as your system's state changes, and that's more true than ever with the explosion in telemetry driven by AI," said Jen Villa, Senior Director of Product at Grafana Labs. "But even the data worth keeping is still massive, so you still need to be able to find what you're looking for in it, fast. Adaptive Telemetry handles the first, and the query performance enhancements we've made in Grafana Cloud Logs handle the second."

Search speed

Customer DATEV said it had seen a marked reduction in the amount of log data scanned after enabling the Logline-based function.

"When I ran our first query on newly submitted logs after enabling Logline, the improvement was clear. We went from processing nearly a terabyte of logs in three minutes to around 100 gigabytes in under three seconds," said Timo Kirzdörfer, OpenTelemetry & Grafana Cloud Admin at DATEV eG. "That represents an 80-90 percent reduction in the volume of logs searched for newly submitted logs, and we saw the results immediately."

Alongside the search changes, Grafana Labs expanded the scope of its Adaptive Telemetry software, which recommends which telemetry data should be sampled, aggregated or kept. Customers using several components of the suite have reduced telemetry spending by 30% to 50% on average, according to the company.

Adaptive Logs Archive lets teams send logs to their own object storage bucket in an open format instead of storing that data in Grafana Cloud. The aim is to let companies retain infrequently needed logs at lower cost while preserving access for investigations, audits and compliance requests.

Grafana Labs also moved Adaptive Metrics Auto-Apply to general availability. The tool automatically applies cardinality-reduction recommendations without requiring manual review of each one, while a setting called Cost Mode is intended to prevent recommendations from increasing the total number of active series.

For traces, users can now query traces dropped by Adaptive Traces for up to 24 hours using a trace ID. This is intended to let teams sample more aggressively while still being able to retrieve a specific trace during an investigation.

Cost pressure

The broader backdrop is a sustained rise in observability spending as companies add more services, software and AI-driven workflows. Engineering teams have long faced a trade-off between retaining enough telemetry for troubleshooting and controlling storage and query costs.

Rapid7 said the adaptive tools had reduced the amount of manual analysis needed to identify waste in logs and metrics.

"Before, finding waste meant someone manually digging through metric cardinality and log volume, and the job never ended because new services kept adding more. Adaptive Metrics and Adaptive Logs now do that analysis for us," said Carl Eastman, Principal Software Engineer at Rapid7. "They show which metrics and log patterns are never queried and recommend what to aggregate or drop, and we apply those recommendations instead of hunting for them. Our engineers spend their time on the few rules that need human judgment, like dropping debug logs we know are noise, rather than the whole long tail."