1.039: don't miss the amazing vendor flash dance
UPDATED: 17 Feb 2009 - changes in green
That description pretty much sums up what Sun, HP, Hitachi, IBM and NetApp have been doing (and saying) about Flash Storage over the past couple of weeks. Some are tap dancing around their continual delays in getting product to market, while others have resorted to high-wire theatrics to cover up the fact that they’re still nowhere near ready to integrate flash tech.
And almost all of them have finally realized that EMC was right over a year ago – the first place we’re going to see benefits from flash technology is indeed as a new tier in high performance storage arrays. That’s right, after a year of excuses and a cacophony of claims that EMC’s introduction of Enterprise Flash Drives (EFDs) wasn’t innovative, today we find virtually every storage vendor (with one major exception) having announced that they, too, will soon be shipping EFDs in their arrays.
And every one of them has chosen the very supplier (STEC) and the same drive (ZeusIOPS) that EMC introduced to the world over a year ago.
To be honest, I’ve expected all along that this is where we’d be at this point in time, but I surely didn’t think it would take them this long to
admit figure out that array-based EFDs is where they should start.
Where we are today is remarkable, and no one can argue that we’d be here were it not for EMC’s vision and investment in bringing the game-changing NAND technology to market ahead of all expectations.
But though the road we’ve travelled to get where we are today is relatively short, it has been littered with some remarkable Flash Dancing (and FUD) from the competition.
Let’s take a look at each of these vendors journeys on this Road to Flash, shall we?
WARNING: this one’s long – probably the longest ever. My apologies…I had lots to say
HDS started their flash journey in a state of denial, trotting out their CTO Hu Yoshida to proclaim that flash drives were a figment of EMC’s imagination. Hu dutifully spread the FUD far and wide, telling anyone who’d listen that they couldn’t work because flash wears out and Hitachi wouldn’t support them because customers would lose data (seems Hu overlooked the importance of RAID protection at the time). Hu got a lot of press coverage for his anti-EFD stance, but eventually Hitachi Japan shut him down by leaking that yes, indeed, the world would have EFDs in the USP-V.
And Hu has been pretty much silent on flash ever since. Personally, I know it’s hard to admit when you were wrong, especially when you’re the CTO and supposedly an expert on storage technologies. So when Hu reappeared, he fell back into his tried and true “Virtualization Solves Everything” theme, repeated ad nauseam. I figure he’s hoping that if he replays that tune long enough, his reader’s minds will turn to mush and they’ll forget how he showed the world that he was out of touch with both Flash technology and Hitachi Japan for most of last year.
Of course, like many flame-throwers out there, ole’ Hu can’t leave things well enough alone – in his recent post, 10 Trends for 2009, he includes this little ditty:
Flash-based SSDs will continue to be a niche market. Due to current economic conditions adoption of expensive flash-based SSD disk assemblies will be limited.
What EMC sales teams know (and Hu clearly doesn’t) is that Flash drives actually save people money! And as you can imagine, anything that saves money is hot right now, and adoption is growing faster than you’d (or Hu’d) imagine, (among EMC’s customers at least).
Now, being as I don’t want to help any competitor to get up to speed on Flash any quicker (why accept a 12-month lead if you can stretch it out even longer), I’m not going to explain the various ways that EMC customers are reducing their storage and server acquisition and operational costs using EFDs. But believe me, they are, and your local EMC rep will be most happy to show you how you can too..
Of course, Hu didn’t stop there – he always finds a way to hook in the term “virtualization” (a term which is rapidly becoming so…yesterday).
Virtualization 2.0 with thin provisioning and dynamic tiered services will be needed to maximize the utilization of SSDs.
And that’s what I meant by “flash dancing.” Translated, he’s saying: “we haven’t figured flash out yet, but we’re sure to have something interesting in the future, just as soon as we find it and copy it.”
Not to worry, Hu, EMC is still leading the way on the Flash highway, and many customers have already seen (under NDA) that EMC is alone in the fast lane. And just to one up your 2.0, I’m thinking “Virtualizing in 3D” might be a catchy name for whatever’s next.
Interestingly, Fellow Blogger Claus Mikkelsen was recently revived to add to HDS’ blogging volume. Though his first two posts this year were blatant rip-offs of my past XIV exposes (plagiary is the sincerest form of flattery, so I’m honored), his most recent post, What’s a Flash Drive?, explores the practical application of non-mirrored DRAM SSDs and flash in an almost-coherent manner. So I think the HDS manhunt for a believable flash spokesperson may be over (although I admit I was shocked that Claus would actually acknowledge that he agrees with EMC!)
ibm: (he’s) a dream
As transparent as Hu’s denials of flash were in 2008, IBM inarguably did HDS one better last year.
IBM got off to a bad start with flash, jumping in all tongue-tied early on when they actually suggested that TAPE was the more appropriate alternative. Never in a million years would I have predicted that one, but it happened. At the same time, nearly a year ago, IBM also acknowledged that customers were asking for Thin Provisioning, a feature they thought they indeed would need on the DS8000 in 2008. And here we are, 2009 and IBM’s first platform preview behind us, and still no thin provisioning.
Now that I think of it, I wonder if that little tete-a-tete that had me rotflmao had anything to do with Sir Monshaw’s recent exile back to the selling ranks over in under-performing and economically-torn Japan?
Of course, with 380-something thousand employees world-wide, the smart ones over there at the It’s Better Manually Corp quickly stepped forward to pick up the banner of flash, and they obviously took it very seriously.
At least, they started out with a bang. Who can forget their Frankenstorage science experiment packing Fusion-IO Flash Cards behind an unsupported configuration of SVC nodes to create an honest-to-goodness monster (that’s right, Chuck, you too are a plagiarist). BarryW was all proud, and hasn’t missed an opportunity since to claim whatever mind-share he can, whenever he can (watch – he’ll be one of the first to comment on this post). Heck – just this week he challenged my assumption that they’d be putting PCIe-based Flash inside of the SVC.
Of course, BarryW also spent most of last year denigrating the very notion of putting flash drives into a storage array. His assertion was that storage arrays can’t get all the IOPS out of a fast EFD, and thus he implied it foolhardy to make the investment. What the world needs instead, he asserted, is a storage platform optimized to deliver high bandwidth, low-latency, IOPS optimized I/O. Not surprisingly, he asserts that the SVC is just that platform, and the aforementioned science experiment was proof positive that all other applications of flash were a waste of money.
If you’ve seen how much IBM is charging for the newly announced flash drives in the DS8K, you’d have to agree: the minimum purchase of 16 146GB EFDs for your DS8K will set you back a cool 1.6 MILLION dollars! Hey – that’s probably more than a brandy-new DS8K system itself will cost you (field upgrades of EFDs are not supported). And while these are list prices, translated to the typically aggressive discounts IBM customers will see,
the expected street price of IBM’s EFDs are 4-5 times more
than the exact same drives are from EMC
That’s A Dream you shouldn’t let them get away with. No way, no how.
So, although BarryW spent all last year proclaiming the lunacy of EFDs in enterprise arrays, here now IBM’s gone and done exactly that. And now that the shoe is on the other foot, BarryW won’t answer the FUD question he flung at me all last year - How
many few of the IOPS in those flash drives is the DS8K actually able to deliver to applications?
So, if BarryW won’t answer the question, then BarryB has to take a stab at it. Let’s see, now:
STEC rates their ZeusIOPS drives at something north of 50,000 read IOPS each, but as I have explained before, this is a misleading number because it’s for 512-byte blocks, read-only, without the overhead of RAID protection. A more realistic expectation is that the drives will deliver somewhere around 5-6000 4K IOPS (4K is a more typical I/O block size).
In the IBM DS8K you buy EFDs in groups of 16, of which 14 are usable and the remaining 2 are hot spares. So 14 EFDs should be able to support 70-84,000 4K read miss IOPS. Except that in a DS8K, all of these drives have to be installed behind a single back-end I/O controller pair, and a DS8K controller pair maxes out around 12,000 total 4K read miss IOPS (and somewhere around 3-4000 4K write IOPS, which explains why IBM engineers have been frantically trying to improve their write cache destage algorithms). Suffice to say there’s a sizable gap between the IOPS you pay for and the IOPS you get, even excluding the exorbitant prices IBM is trying to extract from customers for flash.
And for the curious, a DMX4-4500 today supports more than twice as many back-end IOPS than a DS8K while delivering better overall read/write response times than the current DS8K hw & sw, thanks in no small part to the larger cache and already-optimized algorithms. It will be interesting to see how much I'BM’s new cache logic will change things – if at all.
a new king on the ibm hill?
As an aside (that’s not a song from the movie), IBM bloggers positioned this week’s DS8000-related announcements as “proof” that the platform isn’t dead. As if adding Flash Drives, SATA drives and new cache algorithms to a platform that hasn’t been significantly refreshed in over 4 years was enough to bring it back to life. And while there are rumors of an upcoming DS8500 and DS8700 announcement (probably before EMC World, I’d guess), I still think the platform is indeed on its deathbed.
EDIT 17 Feb 2009: Tony has apparently retracted his original, offensive responses to my questions on his blog, and replaced his answer to my questions about DS8K’s RAID 5 support for SATA with a more polite -– and comprehensive -- answer. The following is edited in light of his changes. New text in green.
- As TonyP confirmed in his responses to my clarifying questions about IBM’s storage announcements last week, IBM isn’t supporting RAID 5 for SATA drives on the DS8K as a standard feature. While TonyP says they’ll accept RPQ’s for such support, citing “professional malpractice” for using RAID 5 with large SATA drives, he also freely admits that RAID 5 –is– supported on almost all of the IBM storage arrays that they OEM from other suppliers, including the DS4000 and DS5000. Now, I find it interesting that IBM considers it “professional malpractice” to risk SATA drives to RAID 5 on the DS8K, but not on other platforms – apparently to IBM, information stored on a DS8000 SATA is somehow more important than data stored on DS5000 SATA. It seems counter-intuitive that DS8K owners are required to submit an RPQ for a feature that DS5000 customers aren’t – especially since the DS8K shops are usually the ones with a storage staff. I suspect that the real reason for the restriction is that the DS8K engineers at IBM haven’t yet figured out how to improve SATA reliability (and drive rebuild speeds) sufficiently for RAID 5 (that, or IBM just doesn’t trust their DS8K customers to make good decisions).
Given a 3-year head start in working with SATA drives (first shipped on CLARiiON in 2005) EMC engineers have significantly improved SATA reliability and shortened rebuild times up and down both the Symmetrix and CLARiiON product lines through a combination of hardware and software improvements. While not “proof positive” that the DS8K is dead, it is yet another indication that IBM isn’t investing enough in this platform to keep it competitive (along with no thin provisioning, insufficient write cache for flash drives, and the like).
- BarryW is also gleefully commenting (and twittering) about how the SVC is going to get enough embedded flash so as to offset the paltry real-world performance of Moshe’s heralded XIV platform. Now, while I suspect that will be a huge blow to Moshe’s ego, I also figure he has less than 12 months left on his contract with IBM, with a sizable pay-out at the end, so he’ll probably swallow his pride and watch it happen from the sidelines. But I sincerely doubt that the SVC can save XIV, now that people understand that SATA read-miss performance is never going to be fast enough to run production applications on, and the reliability profile isn’t adequate for a true private cloud platform. Still IBM’s pouring money into both SVC and XIV, and it has to be coming out of somewhere, so expect any new DS8Ks to be lipstick on the pig while IBM irons out the kinks on their next attempt at mastering the art of storage.
Just don’t hold your breath.
OK, this is getting long, so I’ll trim back a little here.
Actually, that’s not hard, since NetApp really haven’t done ANYTHING with Flash yet. Oh sure, they’ve qualified their vFiler gateway in front of a Texas Memory Systems RamSAN 500 – news that bored everyone (fortunately, there was a good fight on at the time, so we all didn’t fall asleep). And they also spotlighted Yet Again their SDRAM-based Performance Acceleration Module (PAM) in the same announcement, even though it truly has nothing to do with flash – it’s a non-persistent read cache that in reality is little more than an attempt to catch up with the massive global memories found in Symmetrix DMX and Hitachi USP-V arrays.
What is perhaps most remarkable, though, is what NetApp hasn’t yet announced – 13 months and counting, and still no support for actual flash drives. Now, I have it on reasonably good authority that indeed they have been trying to integrate flash drives into their systems, but not a peep out of them about that. But NetApp is never one to shy away from the acrobatics, and their Flash Dance asks us all to use our Imagination to foresee how WAFL will work better with flash than anything else on the planet. Given their essentially content-free flash announcement a couple of weeks ago, I have to wonder if they’re experiencing conflicts between WAFL and flash drives – could it be that an architecture designed for slow disk drives is struggling to leverage the performance advantages of flash?
I don’t know, but one notoriously Flashy Dancing NetApp blogger went so far as to assert that it was WAFL that was doing the wear-leveling for the RamSAN 500, as if TMS’s product would be useless without NetApp’s vFiler. Being as he’s no stranger to throwing partners under the bus (Symantec was another of his victims), I wasn’t surprised when I got no response from my suggestion that he have a TMS spokesperson explain exactly how that worked.
You see, I am indeed learning not to believe most of what that particular Valentine says.
sun: seduce me tonight
Sun jumped into the World o’ Flash with the “less filling” excuse – Flash belongs in the server, not in the storage (hmm…reminiscent of the Mr. T viral video that got at least one HDS marketing team an early retirement, isn’t it?). We heard all about the seductive wonders of ZFS and Write-Zillas and Read-Zillas…and how putting flash in the storage sacrifices all the performance.
As we see by the growing number of vendors who have announced that they will in fact be adding Flash drives to their storage arrays, we now know that all that “only in the server” walk was just Balderdash. When you come to grips with the facts that the best a 15K drive can deliver is 6ms response, a 10K is 9ms, and a 7200rpm SATA drive is around 12ms, getting less than 1ms consistently from a Flash drive is a HUGE benefit to application performance – even if you do have to tolerate a few microseconds of transport protocol overhead.
But when you step back and realize that those are BEST CASE numbers for spinning rust, obtained only when head movement and contention is virtually non-existent, then it really starts to sink in. A drive under load can quickly degrade to 30-50 milliseconds average response time, while that same workload will stay around 1ms on a Flash drive – that’s when you realize you don’t really have to embed Flash in your servers where it is going to be only fractionally faster than external, yet captive to that single server.
And in an external array, Flash drives have added benefits: you can share a RAID-set of EFDs across multiple applications and even different servers; you can use Virtual Provisioning to maximize utilization; you can replicate the data and the data can be included in multi-volume, multi-host consistency groups. Not to mention, you don’t have to wait around for all the server vendors (and the lone mainframe vendor) to get around to providing host-based flash.
And according to Sun, you may have a very long wait if you want server-based Flash.
Last week I found this white paper on SearchStorage: Solid State Storage in the Enterprise. In this paper (which was clearly sponsored by Sun) we find a NEW excuse for why we don’t yet see Flash storage embedded in servers - a lack of standards.
Yup – it seems that given the premise that the well-known Disk Drive Interface isn’t appropriate for embedded Flash, and since PCIe doesn’t define a standardized manner for talking to low-latency block-oriented storage devices, well…it appears that server vendors don’t want to commit themselves to any approach until there is a common interface standard.
That’s probably not good news for folks like Fusion-IO.
This could also be what’s behind BarryW’s suggestion to me that SVC might not be using Fusion-IO for its internal flash also. Double-bad for Fusion-IO, if true.
On the other hand, the argument probably has a lot of merit – heck, we’ll probably see the likes of Intel or AMD define and implement a standard flash interface along with the appropriate logic to handle wear-leveling and bad block remapping…and they’ll probably put it right on the motherboard, eliminating the need for the whole PCIe card form factor for storage.
I guess I can see why Sun might want to wait before going full-bore into server-side flash…
Last, and indeed arguably least, is HP. As you may have seen, in an article titled HP Lays Out SSD Datacenter Ambitions, there are apparently people that HP allows to talk to the press who have been living under a rock for the past year. According to these crazy HP’ers, flash won’t be ready for the enterprise until 2012.
And you know, if that’s what they want to think, who am I to correct them?
But seriously, don’t these Maniacs realize that they ALREADY ship enterprise flash drives? As Storagezilla explains, since HP OEM’s the Hitachi USP-V (as the HP XP24000), they already support flash. And the 73GB EFD for the XP24000 is already in their price book!
And while Virtual Geek is understanding of the confusion over at HP (which is made up of mostly printer, server and laptop geeks), he took the time this week to explain why the server guys (at least) ought to be paying more attention to the realities of Enterprise Flash Drives. Turns out that EFDs bring a lot of value to the world of virtualized server farms!
Unfortunately, the HP article is littered with Flash Dancing and misinformation. Perhaps the author misunderstood, or perhaps, indeed, it was an uninformed HP representative who provided the mistaken perceptions. For the record, I’ll correct a few of the most blatant mistakes here:
- Enterprise-ready flash, as we all know, has been shipping from EMC for a year, and today it is pretty clear that every major external storage vendor will be soon shipping the same drives from the same supplier that EMC has been using.
- These drives didn’t start-out in laptops – they were in fact purpose-built to meet the performance, reliability, and data integrity requirements of the world’s most mission-critical applications for the world’s most demanding customers of the world’s most trusted storage vendor (EMC).
- The STEC ZeusIOPS drives provide the necessary support for hot-swap, including on-board reserve power sufficient to de-stage data from the drive’s internal SDRAM to the persistent NAND Flash storage.
- The ZeusIOPS drive can deliver as much as 30x the IOPS (I/O’s per second) than a 15K rpm enterprise-class fibre channel drive. The article used a 5400rpm SATA drive as a reference point; the ZeusIOPS can deliver nearly 100x the IOPS of that class drive.
- The ZeusIOPS drives everyone is standardizing on is available from STEC in standard sizes of 73GB, 146GB and 300GB (although I don’t think any storage vendor has announced using the largest drive yet).
- There’s no need for faster storage controllers to capitalize on Flash drives – servers have been receiving performance far faster than a raw disk drive can deliver for more than a decade. An intelligently cached external disk arrays routinely deliver hundreds of thousands of IOPS at average latencies well below that of even the fastest Flash drive. While servers may require multiple HBAs to generate and receive this much I/O, there’s no need for new hardware or even host software to reap the benefits of fast storage
Oddly, I know personally that there are people at HP that already know all this. One of them, Jieming Zhu, is the Treasurer of SNIA’s Solid State Storage Initiative, for example. But HP’s misinformation campaign is obviously well-funded – spend a few minutes on their HP On Solid State Storage Technology web page and you’ll find even more evidence of the confusion that seemingly runs rampant in the hallowed halls of HP.
emc: here where the heart is
Having long ago passed the recommended word length for a blog post, I’ll close out for now with a simple observation:
EMC started the Flash Revolution in hopes of ushering in a new age of Information Technology, where storage efficiency and IO performance are accelerated to cost-effectively meet the demands of exponential growth in processing power being deployed in our world.
The Enterprise Flash Drive is but the first step in this new era, and it is an important one. Over the coming years, we will see many new and exciting uses of solid-state storage across the entire computing landscape, born of wide and varied requirements and innovations, developed by the collective wisdom of all of us who touch this technology as suppliers and consumers.
In the grand scheme of things, we are at the very beginning of this journey.
And EMC is leading the way.
Stay tuned for more solid-state storage announcements from EMC throughout the year – we are indeed Taking Our Passion, And Making It Happen!
Please, don’t support the Flash Dancers…encourage them to (try to) Catch Up, or just Get Out Of The Way!