As the enterprise transitions from disk to flash, the focus shifts from optimising for performance to optimising for capacity. Such is the popularity of flash that analyst group IDC is predicting that more than 60 percent of primary storage spend will be on flash by 2019.
Flash is truly disruptive. We've been using hard disks for almost 60 years, and both storage system design and storage management have revolved around them. Fundamentally, with hard disks we can assume capacity is cheap and performance is expensive. However in a flash world, the equation is inverted: Capacity is expensive and performance is cheap.
Cost per workload
With all current storage media (flash included), you're limited in the amount of performance you can get for a given amount of storage.
When you purchase a solid-state drive or disk drive, you get a fixed amount of capacity and performance. While there are faster and slower disk drives and flash drives, they sit within a fairly small range.
The cost of storage for a given workload is either the cost to buy the necessary IO (input/output) or the cost to buy the necessary capacity, whichever is greater. For a given combination of storage and workload, your cost will be determined by only one of these factors.
If the cost for the workload's performance is low, then the capacity cost will be most important; if the cost for the workload's capacity is low, then the performance cost will be most important.
Limitations of disks and flash
In disk systems, IO per second (IOPS) are very important because while we think of disk as cheap, disk IOPS are actually very expensive. As disk IOPS are extremely expensive, many enterprise workloads are actually sized not for capacity, but for performance.
While a workload may need only a few drives' worth of capacity, it may need a huge number of drives for performance. Flash inverts this equation. While flash capacity is much more expensive than disk, flash IOPS are actually very cheap.
This is because flash no longer has the mechanical systems that have limited disk performance. This means that in flash-based storage, even for many intensive enterprise workloads, the number of IOPS is no longer the dominant factor in system cost; it is now dominated by capacity.
Because of this shift, in many cases flash-based systems (especially systems mixing flash and disk) can actually be cheaper than disk-based systems.
Hybrid storage: Going for the best of both worlds
One obvious way to improve the cost of flash is to combine flash and disk. By keeping hot data in flash and putting cold data on disk, a hybrid array can potentially combine the per IOPS cost of flash and the per-gigabyte cost of disk, leading to a reduced overall cost.
This works only if some of your data is accessed frequently, and a lot of it is not. But this pattern continues to be true for most workloads, as it has in the past.
Workloads with large working sets (they access a large set of data frequently) are likely to be a good fit for all-flash platforms because they do not lend themselves to a hybrid model. Workloads that have extremely good compression or deduplication are also likely to be good all-flash candidates, since high compaction mediates the high-capacity cost of flash.
From physical isolation to quality of service
The unpredictability of disk drive performance extends to workload interactions. In contrast, flash drives have much more predictable performance when used correctly.
The impact of one workload on another is much easier to understand and control from the software layer. This opens the possibility of true quality of service where the system keeps workloads from interfering with one another without requiring them to live on different drives.
Density: The incredible shrinking data center
Because flash dramatically improves the IO density (more performance in the same drive form factor) and mitigates the need to physically separate workloads, data center designers can create storage that is extremely dense by carefully combining high-performance flash and high-capacity disk.
Combined with virtualisation, flash-based storage is leading to a new wave of data center consolidation, where multiple data centers collapse down into one.
IDC reports that as the storage industry continues to evolve, both flash-based array market – including all-flash arrays (AFAs) and hybrid flash arrays (HFAs) – is showing explosive growth.
This consolidation will certainly have a number of interesting impacts on the industry. We will start to see a rise in pod-based architectures where companies scale in discrete units that include storage, networking, and compute.
This may very well lead to a rise in remote office technologies, perhaps even to alternatives to public cloud for small businesses. For now one thing is clear: It is certainly going to save those who run data centers a lot of money.