Dell launches Warp Speed to boost Data Lakehouse performance

Dell Technologies has announced Warp Speed to the Dell Data Lakehouse, a new feature designed to enhance the performance of its S3-compatible storage solution.

This development comes in the wake of the launch of Dell Data Lakehouse earlier this year, aiming to provide high-performance and high-concurrency access to distributed data irrespective of the data source.

Dell Data Lakehouse is underpinned by Dell's S3-compatible storage, offering a high-performance and highly available storage layer capable of storing and querying data in open formats such as Iceberg. According to Dell, the new Warp Speed feature autonomously learns query patterns and identifies frequently accessed data, creating optimal indexes and caches whilst keeping infrequently accessed data in place.

This autonomous feature is set to boost query performance significantly, potentially running data lake queries up to five times faster without necessitating any changes on the user's end. Additionally, this improvement could lead to a reduction in cluster sizes by as much as 40 per cent. Essentially, organisations can run a greater volume of queries on large clusters or maintain the same query volume on smaller clusters.

Warp Speed achieves these performance improvements through several key technologies. Autonomous Indexing generates suitable index types—bitmap, dictionary, or tree—tailored to each data block, thus speeding up operations such as joins, filters, and searches. These indexes are stored on SSDs in compute nodes for swift access. Meanwhile, Smart Caching, a proprietary SSD columnar block caching system, optimises performance based on the frequency of data usage by eliminating unnecessary table scanning and maximising data reuse between queries, ultimately saving compute costs.

According to the press release, IT and data leaders are grappling with the challenge of accelerating analytics and AI while managing costs. Dell posits that the Data Lakehouse model already offers performance advantages at lower costs, but Warp Speed further alleviates the burden by autonomously deciding which data to optimise and cache, thereby improving both speed and cost-efficiency.

For those building high-performance dashboards, Warp Speed enables faster drill-downs on terabytes to petabytes of data without any modification to the end-user experience. The same queries are simply executed more rapidly. This performance boost is likely to be particularly advantageous for organisations with extensive data analytics needs, enabling more efficient and effective data handling.

Performance benchmarks conducted by Dell in July 2024 illustrate Warp Speed's efficacy. Tests using TPC-DS 1TB and 10TB datasets on Dell ECS S3-Compatible Object Storage across various Dell Data Lakehouse cluster sizes revealed performance improvements across a range of query scenarios. Top 20 per cent of queries showed improvements between three to five times faster execution. Compute savings estimates, derived from comparative tests of different cluster sizes with and without Warp Speed, indicated how the feature could yield significant efficiency gains.

The introduction of Warp Speed positions Dell Data Lakehouse as a robust solution for organisations aiming to accelerate their data analytics and AI initiatives while maintaining budgetary constraints. By autonomously optimising data handling processes and significantly enhancing query performance, Dell Data Lakehouse with Warp Speed is geared towards facilitating faster, more efficient data insights in the AI era.