3.020: reality check - vsp vaai support

I've seen lots of bluster lately from the Hitachi PR machine about VSP being the first virtualization platform to support VMware's vStorage API for Array Integration (VAAI).

When you're next to last delivering something, I guess you gotta try something (I note that IBM has yet to deliver VAAI on either DS8K or on XIV – not surprising, since both seem to be on life support, if for different reasons).

Hitachi have spared no blather in their messaging. If you were to believe their PR proclamations, you would expect to gain all the benefits of VAAI without waiting for your existing storage platform to be upgraded with VAAI support. Just tuck it behind a spanking new VSP and forget all your troubles, they seem to say.

Reality Check time.

As Stephen Foskett essentially explains in his post VMware VAAI Storage Array Support in Plain English, VAAI was developed by VMware in cooperation of industry storage suppliers to address TWO issues:

1. Copy and Erase operations place a huge load on the servers, network and storage arrays
2. The SCSI reservation locking mechanism does not scale efficiently for large LUNs nor for large number of hosts sharing the same LUN(s)

What the Hitachi PR machine fails to mention is that moving the Bulk Zero and Bulk Copy workloads off of the server CPU is not the only benefit of a good VAAI implementation. In fact, with Done Right implementations like VMAX, moving these operations into the array allows the array to optimize the operations to further reduce the overhead and impact.
 

inside vaai

Take the Bulk Zero operation: prior to VAAI, the ESX server had to write all the zeros down the wire to the array. This adds a lot of traffic to the SAN, and creates a lot of cache-invalidating write operations on the target array, which must ultimately write the zeros down to the disk.

With VMAX's VAAI implementation, not only is the server and SAN overhead significantly reduced, but so too is the impact on the array. As I've discussed before, VMAX maintains the notion of "should be zero" and "never written by host" for each track. So, when it receives the VAAI Bulk Zero command, VMAX can actually avoid writing zeros to the disks – it simply needs to mark the metadata for the appropriate tracks as being all zeros, invalidate any cached data, and return back to the host. Chop-chop, all done.

Now, insert a VSP into this I/O stream. While it will support the VAAI command from the ESX server, it must still write the zeros to the array(s) behind it. And since Hitachi recommends cache write-through mode to ensure data consistency and to prevent data getting "pinned" into its cache, the overhead of writing the zeros still exists. That overhead is just now running on a different CPU (one where the MIPS are a tad bit more expensive than on most ESX servers, I suspect). And, all those zeros are still being sent over a SAN, albeit likely a separate (added cost) SAN that's behind the VSP and the arrays it is front-ending. AND, the array(s) behind the VSP must still actually write all those zeros, since they really can't differentiate them from other data.

See, Hitachi is proposing to suck you into a shell game that moves the overhead, but doesn't eliminate it.

The situation with Bulk Copy is similar, but potentially even worse.

Arrays that support the VAAI command will natively employ internally-optimized copy engines to effect the requested copy. With VMAX, Bulk Copy leverages the same internal mechanisms that support TimeFinder Snaps and Clones, which can logically copy data (by manipulating internal metadata). Thus, VMAX can complete the VAAI request without the delay of physically copying the data (it will do so in the background, but it can properly present the "copy" even before it has been fully copied).

Now, put VSP into the path, and you lose any possible benefits of optimizations in the arrays behind the VSP. While the VSP may also be able to make a "logical" copy of external storage, it will ultimately have to read the source data from the array behind it, and then write it back to the new location. The old array that the VSP is fronting will have to do the same amount of work as without VAAI+VSP.

vmware + vaai + vsp + old external storage = one bad idea

Couple all this with the fact that the VSP adds at least 1ms of overhead to every I/O request made to external storage, and you begin to realize that there is no practical benefit of using VSP's VAAI to front-end storage connected to VMware ESX clusters. It is probably more cost-effective to buy faster hardware for your ESX servers, or to upgrade your storage arrays to a platform that has native VAAI support.

At least, that's the way I see Reality.
 

technorati tags: EMC,VMAX,Symmetrix,VAAI,VMware,vStorage API for Array Integration,Hitachi,HDS,VSP

Comments

[Stephen Foskett]

Thanks for the shout-out! I find it amazingly hard to locate VAAI support information, and just wanted to consolidate it somehow.

I also am amazed at how each VAAI implementation is different based on array capabilities. The VMAX and 3PAR arrays seem much better-suited to block zeroing than others, including the HDS line and EMC's own Clariion. I wonder how the VSP interacts with bulk copy and hardware-assisted locking. Gotta ask Hu...

[the storage anarchist]

Just for clarification... CLARiiON and VNX both reclaim zero blocks in 8KB increments out of the 1GB pages that we allocate out of the VP pools. This reclaimed space can then be utilized by any other LUNs using the same pool, and will be filled before new 1GB pages are claimed.