« 0.075: iomega joins emc storage division | Main | 0.077: ...priceless! »

April 10, 2008

0.076: oops!... i(bm) did it again!

oops!... i did it againI know that many of you are getting tired of me pointing out the frequent faux-pas made by competitor's executives. To you, I apologize in advance for today's post, and I'll understand completely if you skip this entry or unsubscribe from my feed in protest.

Especially those of you from IBM, on both sides of the pond. I seem to get more hits from the ibm.com domain on these articles than from anywhere else!

I am sincerely trying to stop, honest I am. But just I can't. At least not until these guys stop feeding me material.

Before I continue, though...a note about today's theme.

About a year ago, when I was first thinking of starting up this blog, Chuck Hollis told me one of his super-secret tricks for attracting hits to his blog. He said he would include "Britney Spears" as one of the keywords for every one of his posts, and that he'd get a sizable percentage of hits from search engines like Google and Yahoo! as a result. Not that I need the hits, but the song title fits my topic, so I figured I might try it to see what happens.

So, if you're a Britney fan who accidentally got lured here by this little ploy, my apologies to you as well. This probably isn't what you were looking for.

But if you're both a storage geek and a Britney fan - Welcome! You'll probably recognize the subtitles below...


what u see (is what u get)

Today's poster children for the hoof-in-mouth charity are none other than IBM's general manager of storage, Andy Monshaw himself, and Clod Barrera, IBM engineer and chief technical strategist.

As Dave Raffo points out in his article over on SearchStorage (Blues Clues: IBM's storage roadmap...), Andy apparently waxed enthusiastically about "emerging technologies" in the storage space during his talk at SNW this week.

What was it that so excited Andy? Thin Provisioning, Deduplication, Flash Drives and MAID.

ALL of which have been available from multiple other vendors for a couple of years now. (OK, ALL except for Flash - but more on that later).

None of which IBM is shipping today.

And none of which Andy actually "announced." Or even implied was coming soon.

And then Andy disappeared, leaving poor Clod Barrera to suffer the slings and arrows of the audience.

And with that little interaction, I think Andy made it clear that IBM has proudly claimed sole possession of last place in the storage technology race.

But al least they're thinking about this exciting "new" stuff that others have already implemented.

where are you now?

That's probably what Clod was thinking after Andy left him to answer all the tough questions.

I won't rehash Dave's entire coverage of the "tough questions", but some of the responses did hit my LMAO! button again:

  • Dedup - "a great idea!" yeah - that's why just about every VTL on the planet supports it already (or soon)
  • Thin Provisioning - apparently only available on IBM's re-branded NetApp gear. Oh, and on the XIV platform, but "we've not yet released that as an IBM logo product," he said.
  • Thin provisioning on the DS8000? - "a good expectation"

Last time I checked, you can't store all that much data on an "expectation." Reportedly access times are extremely elongated...

(i can't get no) satisfaction

Actually, *I* can. And did.

Because Clod pretty much confirmed what I've been saying for months - that the DS6000 is all but dead.

OK, what I actually said is that it WAS dead, but when TonyP piped in to assert that I had misinterpreted the IBM Alert, I (almost) believed him.

But Clod made it clear that my premonition was correct, if perhaps a bit temporally ahead of reality (emphasis mine):

Barrera also indicated that reports of the demise of IBM's midrange DS6000 arrays are not exaggerated. "There has been some demand, but not a large amount," he said. "We don't have big expectations." He added that the XIV Ltd. product would likely replace the DS6000 as IBM's preferred open system in high, midrange and lower enterprise shops.

Finally, Nextra positioning that makes some sense - a replacement for the MIDRANGE OPEN SYSTEMS space previously occupied (poorly) by the DS6000.

Fitting for an architecture that has a higher probability of losing data due to a dual drive failure than does RAID 5. And being built on 7200 RPM SATA drives, with a BEST CASE random read miss response time of about 12 milliseconds, it's clearly not a viable solution for cache-hostile, large block, high performance applications. Or mainframe workloads, for that matter.

don't let me be the last to know

Speaking of high performance applications, Andy did drop some choice tidbits about IBM's focus on flash.

No, he didn't say "Tape" again (and I'm sure there are many IBMers who are very thankful for that).

No, what he did was offer an excuse for why IBM hasn't been updating its premier storage platform (the DS8000) on a regular basis (again, emphasis mine):

Monshaw gave some details on IBM's plans [around flash SSDs] during his talk. He said "the biggest buzz in the industry now" is around SSDs. But first, "systems need to be redesigned," he added. "It will happen in high-performance computing first, then I/O-intense applications in financial services. It's coming into the mainstream. However, latency in the system will not let you take advantage of all its capabilities yet."

Seems that Andy is confirming what I've suspected all along: that the DS8000 can't get much, if ANY, benefit from Flash Drives. The two-node clustered architecture simply can't handle it, and unlike what EMC was able to do with the existing DMX architecture, IBM needs a redesign. There's simply too much latency and overhead, and not enough write cache, to be a viable player in high-performance storage arena.


Truth is, it is very possible to leverage a tremendous amount of performance out of Flash Drives without massively overhauling your architecture.

If you have a strong and versatile architecture to work with.

And Symmetrix DMX most certainly does.

That's why it is able to deliver at least an order of magnitude faster response times (consistently) and up to 30x more IOPS than 15k rpm hard disk drives. Today. Without a massive redesign - just a important few tweaks and tuning here and there. It's times like this that having a robust and flexible architecture really pays off.

And so, after spending a couple of years working on Flash technology, it turns out that the Symmetrix DMX architecture is a rather ideal complement to the inherent idiosyncrasies of flash SSDs. For example, true dynamic cache partitioning (instead of only 2, hard-coded, 50/50 LPARs) allows for precise cache tuning to match the exact write pending throughput of the drives. With this set appropriately, the cache fast-write buffering and write coalescence that Symmetrix does has a HUGE benefit to the expected lifetime of the drive while not unnecessarily wasting expensive SDRAM for buffering reads. And this can be translated into not only improved lifetime for flash drives, but also better overall system performance, without penalizing the hard drive-based applications.

Architectures that can't dynamically partition cache resources or effectively buffer writes for more than a few milliseconds will by definition wear flash drives out faster - and I'm specifically talking about architectures like the DS8000 (with it's infinitesimal write cache) and the SVC (which essentially avoids buffering altogether in order to minimize the inherent overhead it adds to every single I/O - a costly trade-off for flash write endurance).

Without write caching, there can be no doubt that you'd be playing Russian Roulette with your data if you stored on most any of today's IBM architectures. Especially using the SVC, since it has no end-to end DIF checking to ensure that the drive actually returned the bits as they were previously written. You may recall that the smart folks at CERN were surprised to learn just how pervasive such silent bit errors can be with disk drives -- and the situation could be much worse with unverified flash I/O.

Yup - architectural changes are required over in the land of Big Blue, to be sure.

dear diary

Time to start busting some of the FUD...

Early testers of Flash Drives in DMX are learning that the system is able to take advantage TODAY of far more of the IOPS and Response Time potential offered up by the STEC drives than I'm sure Barry Whyte would like you to believe. He's been harping on the rated 512-byte IOPS of the STEC drive, and he's even asserted that he has validated these claims. And he's trying to imply that the Symm can't take advantage of all the IOPS in the drive.

What he hasn't yet admitted is that those numbers are "specs" and not real-world expectations for the drive. He knows full-well that the drives won't deliver those sorts of Read *or* Write IOPS under normal random I/O workloads...in fact, you can get those Read IOPS only by doing 100% Reads of 512 byte blocks for an extended time - long enough for the drive to adapt its internal tuning parameters to favor reads 100%. Same for writes - the drive can self-optimize to maximize IOPS based on "pure" read or write workloads.

Admittedly, that's great for the spec wars. But meaningless in the real world, where there are ALWAYS reads mixed with writes, and practically always dynamically changing the bias to one side vs. the other over time.

More significant is the fact that the vast majority of applications, file systems, and databases never do ANY 512-byte I/O's! For example, Oracle and Exchange 2007 always do 8Kb reads and writes. In fact, I have seen traces of Symmetrix systems where 40% of the I/O's are 8K - and the rest are larger than that!

And guess what - no flash drive that you can buy today can do 50,000 8K random I/O's per second. NONE of them. And I seriously doubt that there will be any that even approach that rate in the next 2-3 years, either.

Add to that the fact that the faster you write to the drive, the faster it will wear out. And, the smaller the block size, the more writes are "amplified" by the drive to cause writes to an area larger than 512 bytes. Meaning small writes will wear out a flash drive faster than big ones will. In fact, if Sir Whyte were to have his testbed randomly write 512 byte blocks as fast as possible to the drive (in its default configuration), he'd probably be surprised by how fast it wears out.

But then, he's probably not all that concerned with reliability - he's in performance engineering, after all.

Oh - wanna know what the optimal random write size for the ZeusIOPS drive is?

You guessed it: 8 Kilobytes - exactly.

(Conveniently aligned to the DMX architecture, by the way).

i can't make you love me

(Not that I'm trying, that is. It's just a song title, and I needed a transition.)

But, one last point on performance. Andy mentions flash as first benefiting "high performance computing." To that claim, I advise caution - reducing the read miss I/O response time (is the primary benefit of Flash Drives) does NOT necessarily mean your application will run faster - even if your application has a high read miss component. If the application is multi-threaded and isn't pending (idle) waiting on the I/O, speeding up the I/O won't make much of a difference. Or, if your server CPU is fully utilized (like, say, you might be when running multiple instances on a server with VMware) - speeding up the I/O probably won't help the net throughput of the server - in fact, it could make OTHER applications on that server run slower.

So yes - what early adopters are learning is that they may in fact have to re-architect...

...their applications, that is. Or at least, they may have to re-think their server consolidation strategies.

Because believe it, or not, a LOT of server consolidation has actually been enabled by the fact that applications often have to wait on I/O. It's during those wait times that the processor complex can be "time-shared" by VMware (et al).

Take away the wait, and you probably can run fewer applications on the same hardware.

At least, if you want to take advantage of the faster I/O.

Just something you should be considering, so you're not surprised if your application doesn't seem to go faster when you introduce flash - you really have to make sure your infrastructure can take advantage of the performance that Flash can deliver.


I'd like to think that you are the lucky one; if you stuck through this post to the end, perhaps you learned a little more about the realities of Flash performance, rated specs and FUD vs. realities, and some considerations you may have to factor in should you decide to move to flash.

And if you're already a DMX-4 customer, you're even luckier - because you can start taking advantage of this new technology long before IBM customers will get the chance. If you're not sure whether Flash Drives would benefit your application, have a chat with your local Symmetrix SPEED Guru - they can help you identify the performance signatures that indicate whether read miss response times are holding back your application.

I'll bet you're probably surprised at the opportunities.

Better yet - why don't you call up your EMC sales rep and see if they can get you a few drives in for an evaluation? I'm not saying they can: availability is limited, and you may have to put up some collateral on the loan. But there's no better way to know how much you can gain than trying them yourself.

And the sooner you get in line, the sooner you start leaving your competition in the dust.

don't go knockin' on my door

Until we get to EMC World, that is...see you there!

DISCLAIMER: the storage anarchist is most definitely not a Britney Spears fan. In fact, I don't think I have ever intentionally listened to any of her music until today, and then only to link the MP3 file. I'll say nothing pro nor con - I'm just not a fan. Of course, that could change if my subscription base goes up by 25% or more over the weekend.


TrackBack URL for this entry:

Listed below are links to weblogs that reference 0.076: oops!... i(bm) did it again!:


Feed You can follow this conversation by subscribing to the comment feed for this post.

Tony Pearson (IBM)

I am rolling my eyes. IBM ships deduplication and thin provisioning in the various IBM System Storage N series filers and gateways. SSD Flash Drives in our H21-XM.
No MAID, but we are shipping three of the four items that you claim we are not shipping at all! Need to check your facts next time....

Barry Whyte


I'm sorry but thats what I thought when I read - "and not enough write cache, to be a viable player in high-performance storage arena"

Why the heck would you need write cache????

You really are trying to make it sound like sticking a few drive that each one would outperform a DMX into a big box is the answer.

What Andy was saying is that all of today's storage products are built around hiding the sequential nature of HDD. Those things like write cache just aint needed for flash itself.

Now if you can't see that and if you think you can hide behind your attacks rather than give up on the monolith then good luck to you!

Barry Whyte

PS. Its not just about the storage, its about the whole stack. You need to make use of the improved latency, the higher IOPs/GB and change the whole application, server, and storage stack to make the most of flash. Who stands the better chance of making the most of it, a specialist monolithic storage company, or an IT supermarket that supplies the whole food chain to its customers...

the storage anarchist

You two guys are funny - you oughta take your act on the road!

On one hand, you want to be the WalMart of IT - everything anyone needs in one place.

Unfortunately you've overlooked the fact that your house brand enterprise storage goods are way past their "sell by" dates. All you have to offer is the generic brand ("N").

Then on the other hand, you're clearly pining for the proprietary days of old, when IBM servers only worked with IBM storage and IBM applications.

News Flash: the Open Systems revolution is over, but the fear of single-vendor lock-in isn't. Freedom of choice is a liberty that even the new generation of customers isn't going to sacrifice - and the smart ones choose best-of-breed for everything in their IT shop. I am thus pretty sure that your servers and applications will work quite well with EMC's storage - better than with the house brand, even.

Hey - I have an idea! Why not stop all this bickerin' and competin' for business? If you're the WalMart of IT, why don't you start carrying EMC kit in your superstore? I'm pretty sure Joe would agree to that. Think about it - you could be leveraging the innivation and creativity of the more than 5000 engineers that EMC has working on storage platforms in its Storage Division, instead of wastin' your shareholders' money playing catch-up all the time.

As to "why write cache for flash?" - you'll figure it out soon enough, I reckon - once you have more experience with a wider range of flash technologies in real-world applications.

You'll figure it out eventually. Maybe.

Barry Whyte

Maybe if the subsystem itself is still a bottleneck, then yes you need cache. Or if the flash device has curious mixed workload characteristic for example... but the right subsystem with the right flash devices....

the storage anarchist

And with the right architecture, you don't have to redesign the subsystem or be super-selective about the devices you support in order to capitalize on new technologies.

Thanks to EMC's 100% focus on storage, customers don't have to wait years for the latest in storage technology to show up on the shelves down at the superstore.

Brandon Whitmore

Hi. Where's the Britney stuff?


Barry, I'm with you most of the time, but I disagree on the last "marketing" sentence "Thanks to EMC's 100% focus on storage, customers don't have to wait years .." in fact I did for SAN virtualisation and even more for NAS virtualisation, we are somehow on the right road on the first item, but you're still missing the target with the last one! But that's probably a different story!


The comments to this entry are closed.

anarchy cannot be moderated

the storage anarchist

View Barry Burke's profile on LinkedIn Digg Facebook FriendFeed LinkedIn Ning Other... Other... Other... Pandora Technorati Twitter Typepad YouTube


I am unabashedly an employee of EMC, but the opinions expressed here are entirely my own. I am a blogger who works at EMC, not an EMC blogger. This is my blog, and not EMC's. Content published here is not read or approved in advance by EMC and does not necessarily reflect the views and opinions of EMC.

search & follow

search blogs by many emc employees:

search this blog only:

 posts feed
      Subscribe by Email
 comments feed

 visit the anarchist @home
follow me on twitter follow me on twitter

TwitterCounter for @storageanarchy

recommended reads

privacy policy

This blog uses Google Ads to serve relevant ads with posts & comments. Google may use DoubleClick cookies to collect information (not including your name, address, email address, or telephone number) about your visits to this and other websites in order to provide ads about goods and services of interest to you. If you would like more information about this practice and your options for not having this information used by Google, please visit the Google Privacy Center.

All comments and trackbacks are moderated. Courteous comments always welcomed.

Email addresses are requested for validation of comment submitters only, and will not be shared or sold.

Use OpenDNS