Alluxio launches version 3.5, boosting AI data management

Alluxio has announced enhancements to its Alluxio Enterprise AI platform with the release of version 3.5, designed to accelerate AI model training and ease the management of expansive datasets.

Version 3.5 introduces several features, including a new Cache Only Write Mode, state-of-the-art cache management, and enhanced integration with Python SDK, to optimise AI workload performance.

Haoyuan (HY) Li, Founder and CEO of Alluxio, stated: "Our customers are training AI models with enormous datasets that often span billions of files. Alluxio Enterprise AI 3.5 was built to ensure workloads perform at peak performance while also simplifying management and operations of AI infrastructure."

The release's flagship CACHE_ONLY Write Mode is designed to enhance the speed of AI checkpoint operations by writing data purely to the Alluxio cache instead of any underlying file system. This experimental feature aims to bypass potential bottlenecks linked to storage systems, thereby improving write performance.

In addition, the upgrade introduces advanced cache eviction policies, which grant administrators detailed control over data cached within Alluxio. The time-to-live (TTL) Cache Eviction Policies ensure that data not accessed frequently is automatically removed from cache following predefined policies. Moreover, priority-based cache eviction allows critical data to be retained in cache over less important data, prioritising consistent low-latency access for essential datasets. Both these eviction policy features are now generally available.

Further enhancing Alluxio's offering, the integration of its Python SDK with primary AI frameworks such as PyTorch, PyArrow, and Ray aims to simplify interactions with diverse storage backends. The Python SDK provides a unified interface, making it easier for Python-based applications to handle data-intensive tasks and AI model training. This, too, is available as an experimental feature.

Notable upgrades to Alluxio's S3 API are also part of this release. These enhancements include support for HTTP persistent connections to streamline data requests, reduce latency, and improve overall performance. Security has been bolstered with the introduction of TLS encryption for secure communication between various components of the platform.

The system now accommodates multipart upload processes, significantly accelerating the transmission of large files by segmenting them for upload.

Other enhancements to version 3.5 are aimed at improving overall efficiency. The Alluxio Index Service, still in an experimental stage, is set to hasten directory listing processes by fetching cached directory details, promising a 3-5 times faster processing speed for directories containing significant volumes of files and subdirectories.

Alluxio has also introduced a generally available UFS Read Rate Limiter, allowing administrators to control the maximum bandwidth allocated to individual Alluxio Workers when interacting with underlying file systems. This helps to optimise resource usage while maintaining system reliability.

Finally, support for heterogeneous worker nodes is available, providing flexibility in resource configuration amongst cluster nodes, allowing admins to better manage diverse environments with varied requirements.