0.007: virtual provisioning catch-22
There's been a lot of discussion lately about what I will henceforth refer to as virtual provisioning (as the only appropriate use of the term "virtual" in relationship to storage - see here for my reasons). I've seen many a blogger and blog commenter discuss the implementations, implications and merits of this so-called thin provisioning technology, and for the most part, I think people have got the basics figured out.
Put simply, virtual provisioning technology presents hosts/applications/file systems the illusion that they have more physical storage than is physically allocated, and allocates physical storage only when it is used (written). The technology thus improves storage utilization and simplifies the tasks of storage administration.
But does it really?
From my vantage point, I see a few things about virtual storage provisioning that seem to have been overlooked - paradoxes that may well prove to limit the utility and value of this technology to a subset of the storage domain that is smaller than most think.
Will virtually provisioned storage fit in your environment?
If you've read the book or watched the movie, you know that the concept of "Catch-22" is a trap of circular logic: that which you do actually prevents you from doing it(or something like that).
I assert that virtual provisioning provides a Catch-22 paradox along a few different dimensions. In fact, EMC has over a year of practical experience with virtual provisioning on our Celerra platform, and some of the things we've learned can be very enlightening.
New chores for the storage administrator. With virtual provisioning, storage administration is supposed to be simplified. The idea is that storage admins can simply over-provision every request for storage without having to bother with the traditional issues of capacity planning, locating unused space, balancing performance, or the inevitable expansion/relocation when the initial allocation is outgrown.
Couldn't make the storage admin's job any easier - point & click for another (virtual) terabyte - just set it, and forget it!
But as many are beginning to understand, it's not quite that simple. Sure, the process of allocation can be dumbed down, but a new problem is created to fill the void (so to speak):
Bad things happen if you ever run out of space!
A runaway application suddenly writes gigabytes of unexpected data. The alarms and alerts go unanswered for one I/O too many. Somebody decides to dump his entire MP3 collection to share drive. The purchase request for additional storage gets hung up, and the drives aren't ordered or don't arrive in time to meet the demand.
Bad things start to happen.
So to avoid this, the storage admin now has a new use for the time freed up by fast-and-simple allocation: monitoring the use of the virtual provisioned devices and their supporting pools. Adding more storage to pools that are getting full. Moving fast-growing virtual devices into new pools and away from critical applications that can't be allowed to EVER get a "disk full" error.
Deleting files doesn't return storage to the un-allocated pool. That's right - for virtually every file system in use today, when you delete a file, the space it uses is not freed up for use by another virtual volume. In many cases (such as NTFS), the file system doesn't even actually delete the file data at all - it simply marks the file name "deleted" so that it no longer appears to be in your folder or directory. Further frustrating the situation, NTFS (and other file systems) will in fact avoid re-using any space used by deleted files so as to improve the success rate of future undelete requests. And Windows Vista actually keeps multiple previous versions of files around, just in case.
Apparently nobody told Microsoft that disks (and disk administrators) still aren't cheap.
Net-net, that the file system on your virtually provisioned device can appear to use far less storage than actually has been physically allocated to it. This means that when your storage admin (or more likely, when your server admin) discovers the runaway application or improper use of corporate resources to store those MP3 files, there's no easy way they can reclaim the space. Deleting the extraneous data doesn't do anything to solve the allocation problem - after the delete, the blocks will remain allocated to the file system, and the storage pools supporting the virtual devices won't be a single byte less full
The only option is to copy only the undeleted files into a fresh, clean new volume. And since the storage array has no clue which blocks are good data, and which are allocated to deleted files, you can't even use array-based migration tools to relocate the volume - you have to use a host-based approach that can see the data through the file system (e.g. RoboCopy or EMC's Open Migrator).
Performance is can be significantly reduced.This one could be the most significant of all. I/O performance is often overlooked in discussions of virtual provisioning. In fact, the benefits of improved utilization are usually assumed to come at no cost in performance at all.
But in fact, as the Symmetrix Performance Engineering team pointed out to me just this week, improved capacity utilization means (by definition) fewer disk drives have to support more data - and more I/O!
Sure, you can stripe the storage pools out over a lot of spindles to spread out the load, but unless you never allocate all of the space in the pools or you leave unallocated space on these drives, sooner or later you're going to hit the IOPS limits of the spindles, and then the performance of every application on them will suffer.
People tend to overlook the fact that a single disk drive can support relatively few I/O operations per second - typically in the range of 120-150 IOPs, and maybe as high as 200 IOPS with the assistance of external intelligence to optimize the order I/O requests. This IOPS ceiling is generally irrespective of the drive size. The spindle speed does have some impact, e.g., 15K drives usually support more IOPS than 7200 rpm ones, but it's not anywhere near twice as many.
"Wait a minute," you say, "My array is routinely delivering thousands and thousands of IOPS to multiple applications!" And this is true. Cached disk arrays deliver improved throughput through a variety of techniques that leverage the raw performance advantage of cache memory to mask the limitations of rotating mechanical storage.
But in the end, a cache miss requires physical a disk I/O, and a single disk drive can only support so many I/O requests per second.
So the paradox is this: when you lay out normal "fat" devices (I guess I should call them "real" devices, huh?) the performance load on each spindle is reduced by the overallocated-but-unused storage space. But with virtually provisioned devices, there is less unallocated-and-unused space on each drive, forcing each drive to support a bigger load.
Surely there are applications that won't suffer at all from this performance paradox. Our Celerra experience shows this to be true - software development, network shares for office documents, print queues - all work pretty well when virtually provisioned.
Fact is, storage admins are going to have to take performance into consideration when doing their virtual provisioning, and from an entirely different perspective (see catch-22 #1)
And if they don't get it right up front, they may have to spend time to (manually) relocate the virtual volumes to meet performance SLAs.
Don't you just love circular logic?