« 5.004: the cloud gets big. rreeaallyy big! | Main

May 22, 2012

5.005: who said it couldn't be done?

They said "it" couldn't be done. They said nobody else's array could do "it" – that only their array architecture could handle "it." They said all kinds of things about how "it" was going to bring the demise of Symmetrix, because Symmetrix would never do "it." Even if we could do “it,” they said we wouldn’t – but they said we can’t. 

But they were wrong. VERY wrong.

Today EMC announced "it" is now available on VMAX. And then EMC went one better than they ever imagined – EMC took "it" further than they have been able to, even after all the (8+) years they have been shipping "it."

And of course, they will try to undermine the fact that they now have DIRECT competition from another array vendor who has implemented "it" - highlighting the history of EMC bashing "it", as if that matters any more. As I have noted before, being "first" is only important until there is a second - then all that matters is which implementation is better. And so they will childishly act like first means best perpetually.

Have you guess what "it" is yet?

More importantly, do you know who “they” are?


Read on to see what they never expected…and should have feared...


VMAX Federated Tiered Storage

Yup. THOSE guys tried to convince users that EMC would never implement what they call “array-based virtualization.” They made crazy videos featuring zombies and even Mr. T in their lame attempts to discredit EMC’s network-based virtualization approach instead of “it.” Humorous though they may have been, word is that these marketing shenanigans ultimately led to the replacement of at least one generation of their marketing organization.

But the news this week isn’t simply that EMC has introduced array-based VMAX Federated Tiered Storage (a descriptive name we prefer over simple “storage virtualization” for reasons I will explain in this post).

No, the news is that we’ve upped the ante on what it means to leverage external capacity “behind” an array, in several key ways:

  1. Federated Tiered Storage (FTS) is implemented as a native data service of Enginuity 5876, and is provided to customers to support basic I/O to external capacity at no extra charge. Licensed features like FAST VP, SRDF, TimeFinder (etc.) will include external capacity as if it were internal Nearline drives – that is, the license charge will be for only 50% of the external capacity. But if you just want to VLUN existing LUNs into VMAX, no additional licenses are required. None. Nada. Zip. (And this is true no matter who the vendor of the external storage).
  2. FTS Implements proactive data integrity validation for all data written to external arrays, so as to protect against possible silent data corruption within them. This is a feature that, to my knowledge, nobody else provides today.
  3. FTS adds significantly less than 1ms of latency to most I/O operations directed to external capacity. This averages around 0.5ms or so, depending upon request size and workload. Competitive solutions add 1ms of latency or more…once again demonstrating the superiority of the Symmetrix VMAX architecture.
  4. Unlike the other guys’ so-called “universal” storage virtualization, FTS is specifically designed to utilize external capacity as a tier within FAST VP – they strongly recommend against using external storage with their automated tiering.
  5. Also unlike the competitors, FTS does not require external arrays to present “Windows” personality LUNs only – the VMAX implementation is far more robust and agile, able to encapsulate pretty much any LUN personality currently in use.

And finally, while VMAX FTS has not yet been qualified with as many external arrays as they currently support, EMC innovation is already at work. EMC is encouraging users to self verify FTS interoperability with their external arrays and to submit the results to EMC’s eLab for ratification. The expectation is that FTS will “just work” with most storage – the product has been architected around common FC personality models. Working with our users, we expect to round out the support matrix quite rapidly over the coming months.

silent data corruption

Of this list of differentiated features, I assert that item #2 above is the most important.

Long the Prime Objective of Symmetrix, Data Integrity should never be taken for granted. This is why EMC implemented a more robust variant of T10DIF in the VMAX – to verify that the data blocks we get back from a disk or flash drive have not gotten corrupted (the VMAX DIF also verifies that the block actually comes from the specified location (LBA)).

As I have noted before, silent data corruption is a reality of modern disk drives that most people aren’t aware of. It can happen on any drive, at any time, and unless the array implements checks and validations, these drive errors can be passed on to applications totally without notice.

So, for FTS, VMAX calculates CRC checksums for the data blocks that we write, and we keep them with the global metadata maintained within the VMAX. Although we do not write additional bytes to the external array, the checksum implementation can detect a huge percentage of what would be otherwise overlooked errors. If the data read is bad, FTS will attempt to reread the data multiple times before returning the standard SCSI error code for a failed read.

Now, they have been known to argue that the FC spec includes a per-frame CRC that does the same thing, but that’s misleading. The FC frame CRC does in fact ensure that the data received from the source is in fact unmodified in transit. But if the drive returns bad data to the array and it goes undetected by the array, the FC frame CRC will serve to ensure that the bad data is still bad…it cannot correct for silent data corruption if it happens before it is transmitted.

why federated, and not virtualized?

“Storage Virtualization” is a very general term. Taken literally, it simply describes what is the foundation of all storage arrays – take the various storage components controlled by the array(disks, DRAM, NAND, etc.), and present them to hosts as LUNs that behave as if they were in fact real, physical SCSI drives. This is pretty much what EMC invented over 23 years ago with its first external storage array.

So, rather than co-opt the term for yet another variation of “storage component” (i.e., external arrays), EMC chose to name the feature descriptively. Federated Tiered Storage is what you get when you put storage arrays behind a VMAX. “Federated” because the arrays now are working together to support the storage needs of the hosts and “Tiered” because the external storage has different operational, performance and reliability characteristics from the internal storage – which is pretty much how the Flash SSD, Enterprise HDD and Nearline HDD are differentiated within the array.

This notion of Federation – the cooperation and aggregation of disparate resources towards a common objective (from the BG dictionary Nerd) – this is the foundation of how EMC is tackling the demands of the hybrid cloud.

FTS joins Federated Live Migration (FLM), a feature introduced last year to enable seamless and non-disruptive migrations of LUNs during Symmetrix-to-Symmetrix tech refreshes. Prior to the introduction of FLM, they would try to position storage virtualization as the only non-disruptive answer to tech refresh – when in fact, it still requires a disruption to insert the virtualization layer in between the hosts and the “old” storage…not to mention the fact that an outage is also required to tech refresh their array-based virtualization implementation.

Fact is, FLM is currently a Symmetrix-to-Symmetrix solution; FTS may thus be attractive to some for heterogeneous environments, and it does in fact support non-disruptively moving LUNs between any arrays within the host VMAX’s domain of capacity. VPLEX is another solution that is non-disruptive after the initial insertion into the I/O stream.

Like FTS, VPLEX delivers far more than simple migrations. As the only solution that delivers a true high-availability active/active LUN presentation that remains HA through a site outage or failure. But more on that in another EMC World-week related post Cool.

The immediate focus of FTS is not migrations, but to support the utilization of external storage as another tier under the control of VMAX. FTS can be a powerful component for improving both operational and capital efficiency in customer environments. Being able to standardize on the Symmetrix robust suite of industry-defining storage services is very attractive, especially given the tremendous improvements that have been made in simplicity and ease of use in the latest release of Enginuity 5876.

Oh, and there will be more value-add build upon the foundation of FTS coming in the future.

I just can’t talk about that quite yet.

transformational federation

Admittedly, FTS represents an expansion of the approach EMC took to integrating storage resources before. I think of it as transformational – an evolution of strategy towards a much broader objective than simple migration challenges. In fact, FLM and FTS present two of the first steps towards the seamless federation of storage assets into a coherent, scalable, distributed and manageable pool of information storage resources. We at EMC see this notion of federation to be the requisite foundation the hybrid cloud.

In closing, I must admit that my persona (the storage anarchist) has long argued against the very notion of storage virtualization. In retrospect, I disagreed mostly with the limited focus (migrations) and the lack of data integrity in most everyone’s implementation. FTS addresses these concerns and expands the use cases, so I now have to admit…

…I have been transformed.

Have you?


TrackBack URL for this entry:

Listed below are links to weblogs that reference 5.005: who said it couldn't be done?:


Feed You can follow this conversation by subscribing to the comment feed for this post.

Tach Kiejun

Storage Anarchist, this must be the funniest article I've read in a long time. Not only does EMC have great marketing, they do also employ comedians like yourself. You might was well include something like: "FTS, the iPad killer"

Scott Welland

I read this article and must say that I find it amusing that EMC has changed it's approach to storage virtualization going from San based technologies (Invista / VPlex) to a controller based model such as used for the last 5 years by HDS.

I was at one of your EBC presentations in 2010 and you yourself said that controller based virtualization was not the way EMC was going to use this technology. Your direction was VPLEX.

I too was excited about VPlex but after my experiences with it in several customer sites found it to lack the scaling needed to virtualize and entire DC. It was more in line with SVC and creates Islands of virtualized storage.

Today the VSP supports EMC, XIV, and other storage vendors and HDS RECOMMENDS using HDT with the virtualized storage. It can support up to 255 PB of virtualized storage.

I have worked for both EMC and HDS and I applaud EMC for following HDS and recognizing that they had it right from day one.

As you can imagine from the tone of my comments I spend most of my consulting hours with HDS these days and focus the bulk of my efforts on virtualizing HDS and third party storage behind the VSP.

I just find it very humorous that your are trying to take shots at a company who's technology you are trying to copy.

You made the right move, you now just have to catch up to them, and they have a 5 year head start.

Good luck.



Tach Kiejun

Well said, Scott!

Barry, do you remember the EBC for Deutsche Bank in 2009, where both Dave Donatelli and yourself were bashing HDS virtualization strategy and even depicted a USP-V with external arrays virtualized with a big red circle and line through it?

the storage anarchist

Tach - indeed I remember, and I am glad you find my post so hilarious. Coincidentally, I met with DB yesterday to explore FTS, and they didn't seem to care that we had changed our strategy 180 degrees...

But you know, what is pretty funny to ME is that the Cult of HItachi's responses to FTS are so childishly focused on EMC's historical anti-array-based virtualization position. Yet the Cult offers no position on the abject lack of data integrity validation in UVM.


Oh, and for the record, Dave D. Went to HP shortly after that meeting with DB. Our strategy changed to Federated Storage using both VPLEX and VMAX later in 2009, after he left.

Just sayin...


Its stuff like this that I point to when people ask why I call the entire Tier 1 storage Industry a multi-billion dollar barking Carnival.

When I heard EMC was going to do this I swung by the blog hoping you were coming out swinging as always (if you had settled for a crow pie it wouldn't have been the same old crazy Barry, I expected). While I honestly don't do a huge amount of storage virtualization I find it confusing that Checksumming is viewed as the magic bullet that EMC brings to the table here? Given a choice between Is there a reason why you wouldn't just trust the virtualized array's block level verification? If you virtualize a VMAX or Netapp behind a VSP why wouldn't you just offload/have the virtualized array do its own checksumming? Is there some giant performance advantage from doing that centrally rather than on the 3rd party array? If your virtualizing Arrays were the SP's are overloaded I guess this could help, but I'm not really seeing the smoking gun here, especially when your arguing that it is superior to have this than a Full HCL . - Eagerly awaiting more rants, marketing, and hilarious back and froths with HU.

Ken Steinhardt

As Barry implies, it doesn't matter who was "first", it only matters who is "best" in the present tense at any given point in time. If anyone doubts that the silent data corruption issue is both real and critically important (and if you really want to scare yourself!!!), please use your favorite search engine and look for "CERN" plus "silent data corruption" and you will find both what I consider to be the definitive independent and credible white paper on the subject, as well as independent analyst and press commentary on what CERN found in their actual physical tests and analysis. The reason that this matters is that while EMC storage products (like Symmetrix VMAX and VNX) have always had features that provide deep and persistent data integrity protection, amazingly today ther are some MAJOR storage products from some MAJOR storage vendors that either don't provide persistent data integrity protection at all, or don't provide full end-to-end integrity protection (which is logically the same as NOT providing integrity checking). Simple positioning - EMC Symmetrix VMAX Federated Tiered Storage provides this level of persisent integrity protection even for those (non-EMC) storage products that don't do it themselves, and the "other" storage-based virtualization technology does not. So the question is - do you want confidence that the data that you write is always going to be the data that you retrieve, or are you OK with the very-real and probable risk (read the CERN paper!) of data corruption?


Ken, Good read on that paper However I've always seen storage virtualization useful for a few things. Archival (where generally I've got some type of verification or checksum's built ito whatever's throwing the data there) or the big reason being for data migrations (Migrating 2PB of data into a frame without FTS would be less than exciting). I'm not sure of the utility of pushing Tier 1, or long term primary storage for storage virtualization (and never was on any of the other platforms I've used it on) as the hardware support costs start to get ridiculous with most vendors. The market will decide this and 24 months from now it will be interesting to see how many people are really stretching 4-6 year old frames along behind anyone's engines (Maybe HDS/IBM/Netapp could talk about their experiences with people keeping old power hungry slower clunkers around). Adding all the whizz bang features (VAAI/Replication) is nice, but Until someone has a means to make Netapp and HP renewals on 5 year old arrays not ridiculous, I'm curious how far virtualization will go beyond the traditional uses of Migrate, keep old system around as VTL/Backup Target/Tier 2 till budget free's up.

Doesn't Netapp's V-Series do checksum's and verification on 3rd party Arrays? I Thought Datacore did this also with their NMV volumes on external Tiers.

Barry Whyte

Congrats on finally seeing the light, but I guess the U turn is due to the fact that both Invista and VPlex haven't been anywhere near as successful as hope. Bundle it with the big iron and maybe someone will use it - a bit like HDS ...

Anyway, good luck with that interop catchup - being first (and industry leading) does have one advantage 10+ years of interop testing you need to catch up on - unless you plan to use SVC as the gateway to your support as I've seen in the very rare case where I've come across Vplex.


the storage anarchist

Thanks, Barry - I think ;-)

Seriously, VPLEX is doing quite well, thanks for asking - growing at a rate faster than did SVC in its first years. Winning lots of deals, in fact, where SVC doesn't work - like providing Active/Active HA over distance. Seems the SVC split cluster is no longer HA when either of the nodes fail, while VPLEX Metro remains fully HA at the surviving site. It's just one of those Achilles's heel deficiencies of SVC that gets people looking for a better solution.

And it's not like we have to reinvent Interop testing - we're just redeploying what we've learned over the past 10 years into two platforms instead of one.


Barry Whyte

"quite" is the keyword. Not seeing it come up as a real competitor in the field. Lots of "the roadmap looks good" style hype, but when bit comes to byte... Active active HA at distance, is it really that active? watch this space...

Interop. Good luck with that. Learning, and real customer use are very different things as I;m sure you'd agreed.

Anyway, maybe we can concentrate on differentiators, now that you've realised/agreed we had the right architecture in the first place, and for once you are playing catch up ;) Look forward to sparring in the future, been quiet out here with your absence.

the storage anarchist

Barry -

"Quite" - as I said, growing faster than SVC did in its early days. You guys where pretty boistrous back then about your growth rates...well, we're quietly exceeding them today.

Interop - VPLEX already supports 42+ different platforms. For FTS, just a matter of running everything we already know through the paces. Customer qualifications have added 17+ arrays since GA, so not to worry. Oddly, the current hottest field request is for us to qualify SVC behind FTS - seems that more than a few customers are looking for an easy way to migrate OFF of your product. You probably don't see those unhappy customers who have learned the realities of your not-so-active/active, definitively non-HA solution and realized that they need more.

And that both VMAX and VPLEX can deliver what SVC cannot.

Jus' sayin'

Catch-up? In numbers deployed, maybe, but not in features or capabilities. With your 8 year lead, you've still not delivered end-to-end data integrity OR truly Active/Active. The lack of serious competition has bred complacency in both IBM SVC and Hitachi UVM.

Here's your wake-up call.

As to differentiators for FTS? Umm...where should I begin?

TimeFinder, SRDF, FAST VP with an 8MB granularity over 128,000 devices and 4PB+ usable capacity, VMware+HyperV+XEN integration, RecoverPoint, Federated Live Migration, 53+ GB/s cache miss bandwidth, 2M+ IOPS, less than 45 microsecond I/O overhead for both FTS and VPLEX to external storage?

Oh, and did I mention that VPLEX is the *ONLY* solution certified by Oracle to enable stretch Oracle RAC clusters to over 100KM, with full HA and witness coordination?

Consider the ante raised by an order of magnitude (or more).


I really love this post. It is so typical EMC style - lots of words and very less sense. With Invista, you guys said a switch based virtualization is the way to go forward. I am really wondering what Invista customers must be going thru right now. Then you guys said V-Plex is the way to go forward, and going by your past, I am sure already it is on it's deathbed. Now with FTS, again you guys are saying that this is the way to go. But then the 40K just seems to be a box which can also do virtualization. You have to offer it for free cause you are late in the game. Coming to your differentiators - lets talk about the performance numbers ou have mentioned - 53GB/s and 2M IOPs. Just glance at all your competitiors specs and you will realise that they already have more than the cache miss bandwidth you have mentioned. As regards 2M IOPs, I guess we just have to believe you as EMC does not do any public benchmarks. Then you again put a meaningless figure of 45 microseconds from FTS/VPLEX to external storage. When will EMC start talking about numbers that really matter to a customer - for example, the overall response time/IOPs/throughput in customer relevant environments and for specific configurations???

Barry Whyte


I think you'll find SVC/V7000 has been true active/active since 2009. And with 6.4 we now have the ability to migrate volumes between IO groups without disruption, and the ability to present volumes through all IO groups, so true N-way access.

Meanwhile, 53+GB/s miss bandwidth - thats some serious number of SAS ports (or are you still stuck on FC-AL disks) - thats a marketing number, much like 296PB HDS... whats the real miss (from disk read IOPs number) or will EMC continue to hide behind those amazing performance graphs that show no scale on the y-axis - gotta love how you get away with that:)

the storage anarchist

Barry -

Active/Active maybe, but lose a node in an I/O group, and you are no longer HA - unlike VPLEX.

Non-disruptive migration across I/O groups, except according to your latest customer bulletins, only supported on two variants of Linux, and not on Windows, AIX, Solaris, HP/UX, nor VMware or Xen.

Nice try...

And 53GB/s miss is actually an engineering number - real, live measured by your equivalent performance specialists in our labs...stacked up against similar claims by Hitachi and IBM (we actually tested MORE miss bandwidth on the VSP than Hitachi claims in their documentation).

the storage anarchist

Raghavaprakash21 -

Thanks for your comments, but I have to admit I am not aware of any competitor claiming 56Gb/s of 100% cache miss bandwidth or 2M+ cache miss I/O per second. And indeed, we routinely demonstrate our superior response times vs. VSP.

But then again, since VSP enables silent data corruption for externally virtualized capacity, it really doesn't matter who is faster, now does it?

Barry Whyte

You've only got 4x 2 way Intel Westmere(?) CPU in your Vmax, So at about 6GB/s per DDR3 channel, and limited by the QPI on that generation, I can see about 48GB/s cache hit... at 100% memory efficiency...

As for migration, its quite sad that (IBM included) the open source multi-path OS implementations work, and those you pay for don't always - with AIX we found an edge case that we want fixed before releasing full GA support - support for those other OS will be coming when they fix their problems... ;)

the storage anarchist

Barry -

You need to read the specs a little closer: VMAX 40K has 2x 6 core Westmere 2.887GHz (3.066 Turbo) on a PCIe Gen2 per node/director. 4x6 core per engine, 32x6core (and 8 PCIe2's) in a full system. And the numbers are cache MISS.

2 simple points about migrating across IO Groups: 1) Neither VMAX nor VPLEX need anything special to do this - ANY LUN can be exported on ANY port (or EVERY port) on EVERY engine. No need to hack around with multi-pathing or brandy-new SVC features.

2) It is an IBM tech note that states that ONLY the 2 Linux versions (and nothing else) listed on the published qualification matrix are currently supported by SVC non-disruptive movement.

VMAX Federated Live Migration allows non-disruptive move between separate arrays, and doesn't require ANY changes to host multi-path drivers. Instead, our arrays are smart enough to understand which multi-pathing each host is using, and then to adapt itself to handle the particular implementation. Currently supports PowerPath on ANY host, plus DMP and native MPIO on almost all hosts. Even supports SCSI-2 & SCSI-3 reservation clusters...

If you require hosts to update their software in order to use your non-disruptive features, it's not really non-disruptive, is it?

Ours works; yours has quite a ways to go before you start bragging about it.

Barry Whyte

How much does PowerPath cost per port?

the storage anarchist

PowerPath for path failover (the pre-requisite for Federated Live Migration) is free with the purchase of a VMAX or VNX.

As I said, FLM also supports native MPIO and Veritas DMP, thus PowerPath is not required for non-disruptive relocations.

(For its advanced functions, like dynamic load balancing across up to 16 ports, PowerPath is licensed per host, and is usually sold under Enterprise License Agreements for volume discounts).

Meanwhile, relocating LUNs from one IO Group to another non-disruptively is supported only on two host operating systems, and not even IBM's own enterprise server platforms.

And it is not even possible to have a host connected to the same LUN(s) on two separate IO groups - a native feature of both VMAX and VPLEX for maximum HA and resiliency.

This all just demonstrates once again the significant benefits that best-of-breed solution suppliers (such as EMC) deliver. We only do storage, and we do it better than anyone else.

So much for Tony's one-stop-shopping argument...

The comments to this entry are closed.

anarchy cannot be moderated

the storage anarchist

View Barry Burke's profile on LinkedIn Digg Facebook FriendFeed LinkedIn Ning Other... Other... Other... Pandora Technorati Twitter Typepad YouTube


I am unabashedly an employee of EMC, but the opinions expressed here are entirely my own. I am a blogger who works at EMC, not an EMC blogger. This is my blog, and not EMC's. Content published here is not read or approved in advance by EMC and does not necessarily reflect the views and opinions of EMC.

search & follow

search blogs by many emc employees:

search this blog only:

 posts feed
      Subscribe by Email
 comments feed

 visit the anarchist @home
follow me on twitter follow me on twitter

TwitterCounter for @storageanarchy

recommended reads

privacy policy

This blog uses Google Ads to serve relevant ads with posts & comments. Google may use DoubleClick cookies to collect information (not including your name, address, email address, or telephone number) about your visits to this and other websites in order to provide ads about goods and services of interest to you. If you would like more information about this practice and your options for not having this information used by Google, please visit the Google Privacy Center.

All comments and trackbacks are moderated. Courteous comments always welcomed.

Email addresses are requested for validation of comment submitters only, and will not be shared or sold.

Use OpenDNS