« 0.038: how much is too much information? | Main | 0.040: where's the spc disclaimer? »

October 02, 2007

0.039: ibm and spc vs. hitachi math

Hitachi dropped another shoe Monday with its announcement of the best-ever SPC-1 benchmark results for an Enterprise Storage System.

I'm sure the "thud" was pretty deafening over in Blue-ville, especially given this recent reiteration of IBM superiority by Tony Pearson - partially on the back of the DS8000's (now defunct) claims to the top spot on the SPC stepladder.

I pretty much established my position as a disbeliever in the real-world value of benchmarks such as the SPC in my prior post entitled the case against standardized benchmarking, so I won't rehash my arguments of irrelevance here.

But I will take the opportunity to add to my continuing expose of Hitachi math, both as a service to my readers, and in support my pals Tony & BarryW over at Poor Old Big Blue. I'm sure they'll have their own spin soon, but I figured I might be able to jump-start their responses.

But before you read on, I'm curious: Were YOU able to recognize the Hitachi math in this announcement?
 

mysteriously macabre marketing

First, let's break down this announcement from the top, starting with the headline, subtitle and first sentence:

Hitachi Universal Storage Platform V Blasts through SPC-1 Benchmark; Delivers Highest Performance of Any Enterprise Storage System Ever

USP V Breaks Records, Annihilates Competitors, and Delivers Superior Enterprise Storage Economics to Customers

Further expanding its industry lead in enterprise storage system performance while crushing through the limitations of aging monolithic architectures, Hitachi today announced that its flagship Universal Storage Platform™ V achieved the highest Storage Performance Council (SPC-1) benchmark result in enterprise storage system history, eclipsing the results of every single enterprise storage system ever tested using this widely recognized industry-standard benchmark.

Despite the fact that terms like "Blasts through," "Annihilates Competitors" and "crushing [...] aging [...] architectures" are generally avoided in the post-9/11 civilized world, for some unknown reason Hitachi's marketing miscreants continue to devolve everything they write into a distasteful parody of Halo 3. Hopefully its not a cultural thing, but I don't recall other competitors harkening up blood and guts imagery in their press announcements.

Nor resorting to blowing up cars or kicking down walls as in the two most recent Mr. T YouTube-o-mercials (original, "T" for Trucker and mid-TEEr T). I mean, really, can't we all just get along?
 

But I digress.

The early-warning indicator of impending Hitachi Math comes in the qualifier in the second phrase of the title, where it is admitted that the results aren't really The Best, only the Best Ever Tested For An Enterprise Storage System. The fact that only the IBM DS8300 Turbo  and the Sun 9960 are the only other "enterprise storage systems" ever tested probably goes unrecognized by most readers.

But here's the real rub:

There are in fact other storage systems that have posted better SPC-1 results!

Posted right alongside Hitachi's results on the SPC web site, I was able to find several other platforms that posted certified results that were "better" than the Hitachi USP-V results. I didn't bother checking them all, but here's what I found:

Platform SPC-1 IOPS Tested Data Capacity Price of Tested Configuration $'s per SPC-1 IOP $'s per GB of Tested Data
Hitachi USP-V 200,245.73 26,000.000 GB $3,525,389 $17.61 $139.59
IBM DS8300 Turbo 123,033.40 9,103.360 GB $2,336,626.45 $18.99 $256.68
IBM SVC 4.2 272,505.19 24,433.589 GB $3,284,767 $12.05 $134.44
Fujitsu Eternus8000 115,090.06 10,854.400 GB $1,855,100 $16.12 $170.91
DataCore SANmelody 9,298.56 200.000 GB $45,145.70 $4.86 $225.73
3PAR InServ S800 100,045.74 16,467.672 GB $1,482,977 $14.81 $90.05
Texas Memory RamSan 320 112,491.34 68.719 GB $168,776 $1.50 $2,456.07

Now, maybe it's just me, but when I look at these results, I'm not so sure the USP-V is "the best" - by any comparison. We can argue over the definition of "enterprise" I guess, but the benchmark doesn't really measure "enterprise" - it measures performance. And the results are pretty clear - the SVC seems to hold the land speed record, and it beats the USP-V in cost per SPC-1 IOP as well. And if $/SPC-1 IOP really is most important, nothing seems to beat Texas Memories - heck, you could buy 20 of those RamSans for the price of a single USP-V, and you'd have more than 11 times the SPC-1 IOPS.

But then look at what the effective cost per gigabyte of actual data is for each of these platforms, and the picture gets even more confused. In a world where a 300GB SATA drive costs less than $1/GB down at Best Buy, these are pretty whacky numbers. I doubt that anyone would ever consider paying this kind of change for storage - even if their application DID match the SPC benchmark profile. I mean, it would be NICE if you could actually charge this much and get away with it, but I think those days are long behind us - wouldn't you agree?
 

So you tell me - who's really the "best" here?

I mean seriously - can you surmise from this collection of data that a USP-V will actually perform your specific workload better than any of the other kit?

And if you answered yes, I encourage you to do a little research into the actual SPC-1 benchmark. Although it claims to be representative of an OLTP workload, did you know that it originated as a model of a DEC Alpha-based email server? Fact is, the benchmark itself is over 7 years old, built in a bygone era before Exchange and Notes became the world's predominant enterprise email infrastructures.  Suffice to say that those close to both will readily attest that the workload of each is radically different from each other, and neither bears any real resemblance to the typical modern-day OLTP workloads - that despite the reassurances from the SPC membership and participants.

Did you also know that SPC-1 is intentionally cache unfriendly? That it intentionally works on a widely scattered working set specifically to defeat any benefits of cache, prefetch and optimizations? This is why small memory systems appear to do so well - and also why one vendor once resorted to disabling cache protection in order to optimize their results (in a submission that the SPC now refers to as "Withdrawn," I guess).

And another fact about the SPC-1 benchmark that most who have never seen/touched/run it don't know is that while it does distinctly model 3 different Application Storage Units (different workloads), these three are run sequentially and not concurrently. This in effect asks the array to support one thing at a time. Now, I don't know about you, but I honestly don't know of ANY enterprise class arrays that ever operate under the luxury of supporting only one type of workload at a time. And while I understand that it is difficult and complicated to model multiple concurrent workloads, I stand by my assertion that this benchmark is unrealistically artificial and thus meaningless as either a prediction or a comparison tool.

But I promised not to get into all that again - so I won't. Except to note that the definition of "enterprise" storage typically means a healthy workload of both local and remote replication concurrent to the operational workload. Which neither SPC-1 nor SPC-2 include, and which I know first-hand causes most every tested system fits (if they even support array-based local and remote replication, that is - not all do!).
 

more hitachi math

Now if that isn't enough, deeper scrutiny reveals some apparent Hitachi shenanigans in how the systems were configured. Firstly, the USP-V used 1024 146GB 15K rpm disk drives to get these results. The DS8300 Turbo attained its results using only 512 drives - 256 73GB 15Ks and 256 146GB 15Ks. From a performance perspective, benchmarks like the SPC-1 are often what's called "spindle limited" - to get the best results, you need as many spindles as possible. So the USP-V has an undeniable advantage over the IBM because of having 2x as many spindles. (Granted, back with IBM ran the tests on the DS8300, the max standard config was 640 drives - but today you can get 1024 drives on your DS8300, so one wonders how that might change the results).

On top of that, Hitachi's benchmark results document that a whopping total of 150,528 GB of physical capacity was deployed to support only 26,000 GB of benchmark data - an incredible over-provisioning of nearly 6x more storage than the application required. And while that's really only 3x excess if you allow for the mirrored data protection, you have to wonder why the Hitachi configuration didn't also include any hot spare drives, while the DS8300 Turbo configuration did.
 

So, is this really an apples-to-apples comparison?

More importantly, do you really think you can conclude that the USP-V will be faster than the DS8300 Turbo if both systems were configured to cost exactly the same amount? Virtually every negotiation I've seen starts with the premise that the kit from each vendor should cost approximately the same, and in today's world I doubt ANYONE would get away with either:

  1. buying more 3-6x capacity than required by the application
  2. paying vendor A significantly more for the same amount of capacity from vendor B

So, if you held price/cost constant, used the same number of disk drives, the same number of ports, and the same amount of global memory...would the results still be the same?
 

I think not.

Now, the Storage Performance Council specifically intended to limit this sort of shell game by requiring vendors to quote the actual prices for the tested configurations. But there is an obvious flaw in this: the continuing disk drive price erosion gives a measurable advantage to vendors who test last. This allows Hitachi to use 2x as many drives as IBM without the end system costing 2x as much.

Oddly, even with twice as many drives, the USP-V isn't twice-as-fast as the DS8300 Turbo. And you know, I'm not quite sure why. Maybe the DS8300 Turbo is able to deliver more SPC-1 IOPS/spindle than the USP-V.

Or maybe this is yet another example of why the SPC-1 isn't really a meaningful benchmark. I mean seriously, if you have to dig this deep into the configurations and pricing and cost erosions and such, how the heck is anyone supposed to be able to make a reasonable judgement of "better?"
 

the case of the missing iops

Perhaps the biggest the most blatant slap in the face was the liberal intermixing of SPC-1 IOPS and "cache hit IOPS" in the Hitachi press release. So deftly done that more than one naive reporter has mixed the two numbers up in their coverage.

Brilliant!

And on the other hand, some have joked that we need a "where's Waldo" search for the missing 3.3 million IOPS.

Cute!

But the point is that Hitachi's SPC-1 results spotlight the ridiculousness of their "3.5 million cached IOPs" claims. Granted, the SPC-1 is intentionally cache-hostile, but there's more than an order of magnitude difference in the Nosebleed Numbers of "cached IOPS" and the Impractical Reality of SPC IOPs.

Not me. And probably not BarryW - we've recognized all along that there was no way the USP-V could deliver that much throughput. The SPC results just bring that whole inflated claim back down to earth.

Too bad the press and analysts are so easily mislead.
 

now what?

Well, hats off to Hitachi, I guess, for finally completing this test (which I just happen to know they started back in 2005 with the original USP - no explanation why they never published USP results though). At least there's one less thing for TonyP to crow about.

But to BarryW and the rest of you who will inevitably ask, I say don't expect this to change EMC's stance one iota. As evidenced by the confusion I've described above, the significance of the SPC benchmark remains in the category of useless and misleading. The workload isn't easily aligned with anything in the real world, and there is insufficient analytic understanding of how the workload scales with ports, spindles, cache or CPU power to be able to correlate the results across different systems from different vendors, much less to predict the results of different configurations of the same system from the same vendor.

And I personally remain convinced that storage performance is far too complex to tag with a simple rating system like the SPC.
 

so don't wait for emc to run the SPC tests

And not just because I personally don't think they're meaningful.

No, you don't have to wait, because I can tell you the answer right now: I assure you that whatever the SPC-1 results might say, DMX would come out The Best.

Yes, of course, marketing will spin the results, just like the Hitachi machine has done.

But more importantly, the world will not change. Customers who were happy with their DMX performance the day before EMC published the results would still be getting that same level of performance the day after. Those that wanted to buy a DMX beforehand will find a way to do so, even if the DMX results weren't greater than the USP-V...as evidence, I sincerely doubt that USP-V sales will be impacted significantly by the fact that the SVC is indeed not only faster, but cheaper for better performance.

In my mind, the SPC Just Doesn't Matter.

So I have to ask myself - why bother? Running this benchmark isn't cheap (although I do note that Hitachi has gone for the 3-fer, listing the single test results under all three names/brands that the USP-V is sold under - I wonder, did they all three split the costs?). And it's not easy, as BarryW lamented that just getting a complete run done without incurring a drive failure is a major challenge.

And given that EMC's performance gurus around the world would spend just as much time explaining why the SPC isn't representative of any customer workload whether the DMX results won, lost or weren't there, I find myself asking the age-old ROI question:

Why bother?


TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d834c659f269e200e54f027cac8834

Listed below are links to weblogs that reference 0.039: ibm and spc vs. hitachi math:

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

CMP

Barry, thanks for being so predictable.

Newbie

Barry, you love to poke Hitachi right?
You notice more disks in USPV than in DS8300...
You notice more SPC IOPS and less $/IOPS in SVC...

Somehow you miss over 1.5K disks in SVC which in cache hostile config _will_ give more IOPS.
Somehow you miss those disks installed on cheap DS4700s behind SVC which _will_ give a better $/IOPS.

Selective approach? Well, normal for blogketing...

the storage anarchist

Newbie -

Thanks for the added insight - indeed, it appears that the more disks you use in this benchmark, the more IOPS you get...kinda goes hand-in-hand with the concept of "cache hostile," don't you think? And using cheap 4700s instead of a USP-V can indeed lower the $/IOPS.

You're just underscoring my points: The USP-V isn't "best" and the SPC doesn't even attempt to differentiate the value/benefits of "enterprise" vs. other classes of storage.

Snig

First off, by not speaking the way Americans always have in our culture by saying things like "Blasts Through", etc., etc. all I have to say is "Terrorists Win!" Give me a break and get over yourself with all the PC crap. Americans need to be Americans and not let some weak group of people that prey on innocent lives shape or determine how we live ours. Maybe it's the former soldier coming out in me, let's not give them what they want and be free. The ESPN guys and football announcers say those things every week on TV. Oh and "The Bourne Ultimatum" is out in movies and guess what? They blow things up in NYC. Go figure.

I couldn't agree more that the SPC is garbage, but that's all we have right now. Why not, at the next SNW, have a bake off in the lab under real world customer load. Run Oracle, Exchange, and SQL all on the same frame and let's see who wins.

Are you saying that the SVC is an "Enterprise Storage System" even though it doesn't have any addressable disk in it? Isn't the SVC just a virtualization engine? Plus they had 8 frickin engines running the benchmark. Oh, and here is a quote straight from the SVC SPC results "Each EXP810 and DS4700 has 16 disks (total of 1536). Disks are 73 GB, 15K RPM." So I'm note sure where you were getting your numbers from.

Also, look at the response time comparison. At 200K IOPs the USP-V is just below 5ms and the SVC is around 6-7ms. Reads on the SVC are over 10ms! That just doesn't work!

The problem with you writing something like this is that you can't, or won't, put your money where your mouth is. All you can do is make unsubstantiated claims.

Han Solo

I agree that they are doing some silly wording of their press release to make it sound better than it is.

But part of me has to give them credit for at least participating with SPC. That makes EMC the lone holdout.

Imagine an industry were you can actually have some sort of semi-real comparison between products instead of smoke mirrors.

Without SPC, which you seem to not like, we are left with the bogus 'aggregate throughput of the array' type bogus numbers which we ALL KNOW is fake.

Chuck Hollis

I've tried to illustrate by analogy the importance of relevance in industry testing.

Just to be extreme, I propose establishing the Pencil Drop and Bounce Test for storage arrays.

Simply drop a pencil (eraser side down) from 1 meter above your array, and measure the rebound. Higher numbers are deemed better, but that's up for debate.

I will shortly announce the formation of the PDB Council to make sure that all tests are conducted in an open and fair manner. No special erasers, etc.

Sounds pretty silly, doesn't it?

So does the SPC.

Barry Whyte

Some interesting points as always. I hadn't noticed the 3x short-stroking, at least with the SVC tests we used mirroring and had about 47% capacity usage, so almost all the capacity available. Yes, while SVC did use 1536 DDM, this was needed to max out SVC not the benchmark.

As for 'best enterprise' - I agree with Tony, SVC has greater than 5 9's, infact 7 9's the last few months availability. Maybe HDS should have said 'fastest monolithic controller' I'd also be interested in their External Attach IOPs numbers for a true virtualization comparison.

One point though, the $/GB number includes the host, fabric, cables etc - so its not just the price of the storage. Given we have to use a highend p595 style host to generate enough IOPs - this does add to the cost considerably.

Greg Schulz

Ok, we can all wait and hold our breath (Apply oxygen or scuba mask and breathing apparatus now) while listening to why EMC should get involved in SPC (They should) and why they don’t (some reasons may have some validity) however that’s a different discussion.

However, to that point about validity of the SPC tests and their applicability to real world workloads that changed over time. Rather than turn blue in the face waiting for EMC to show up for SPC-1, SPC-2 or SPC-n (unless you applied breathing apparatus while holding your breath ;) ) all one needs to do is look elsewhere for comparisons such as MSFT ESRP (http://technet.microsoft.com/en-us/exchange/bb412165.aspx) where you will find EMC DMX as well as CLARiiON tested systems that you can do your own comparisons and conclusions from, or if you cant call ghost busters or myth busters…

Cheers
GS

Newbie

Re Barry White:

Almost all capacity available? The report (http://www.storageperformance.org/results/a00052_IBM-SVC4.2_SPC1_executive-summary.pdf) states a bit different - 112TB configured and 24TB used (even with mirror it gives well over 60 TB unused space).
Also, the same report kind of defeats your point about all SAN infrastructure and high-end servers included $/IOPS calculations - just take a minute to look through, it's only the storage that counted...

Re Barry Burke:

Barry, you know what makes me somewhat sad when I read your blog? I'll try to explain it while keeping short:

Why don't you guys (this applies to yourself, your EMC colleagues, HDS bloggers and people from IBM to the same extent) just get over marketing and start being frank to your readers? I know it sounds naive but c'mon, your blog's disclaimer says it's not sponsored by EMC and is even hosted outside emc.com...

So why do you have to fight any feature competitors have and you don't? Why not tell once "Hey, Hitachi (replace with other vendor of choice) guys implemented really neat feature which we don't have! I gonna talk to our R&D folks to implement it in DMX4 and possibly even improve it!"

Why each of your posts has to look as an attempt to make other look worse than they are? Why do I have to dig for rare really nice ideas in piles of FUD?

Remember your response to HDS Power Saving stuff in Nigel's blog? Did you really have to be all-negative? Is it really dictated by a passion to storage technologies as you like to state? And when you speak about “things they did not tell in press-release” – when was the last time you’ve seen something like “The following things are far from being perfect and some might even be simply poorly implemented in DMX family” in EMC press-releases? Oh, c’mon…

Sorry if this sounds a bit offensive, it didn't intend to...

the storage anarchist

Newbie - thanks for the insightful suggestion. I honestly have intended to do more of what you suggest. I probably HAVE spent a little too much time trying to counter-balance the misleading messaging and posturing going on out there - in part because nobody else was before I started blogging.

But I do feel that we have a more even keel going now, and my posts have served to rally bloggers and lurkers alike to share a broader perspective - often going even deeper into the issues than I. And that's A Good Thing.

So I think I will try to give some more introspective perspective on things like our strategy and our focus. As an employee of EMC I can't disclose certain things, even though my blog is independent, but I agree that there is more perspective that I can provide.

Thanks again for the feedback.

Barry Whyte

Newbie, I stand corrected. I had mis-remembered the % of usage capacity. I also see that since previous runs and involvement the p595 host is no longer included, however the SAN itself, HBA's etc are included. The majority of the cost is the actual DDM's themselves. However, your points are noted for future reference.

Heeki Park

It was mentioned previously but I agree that if there are to be any type of tests out there, they should be set on equal footing, whether financially or technically, rather than having some general number with unrealistic configurations/tests. I just worked with a customer recently who was looking to make the jump from a CX to a DMX, and I was tasked with providing some technical justifications for that (I blogged some of my findings). However, while the customer was concerned about overall IOPS delivery, the project was dictated by the price. To get a true apples-to-apples comparison between vendors, you need to take one of two approaches:

1) Equal configurations: same drive counts/speed/capacity, same cache, same front-end counts/speed, same volume configurations. That and then specify the price of each configuration.

2) Equal price: start at a set budget amount and allow each vendor to build the best box possible, at list cost. No funny discounting. Use the list cost that is quoted to customers.

Performance is too complex a thing to dumb down to a single number. There's a story and that story needs to be told. Context is important. Appreciate the blog.

the storage anarchist

First, welcome Heeki to the wonderful world of blogging. Looking forward to what you have to say on your blog (it's at http://sanartisan.wordpress.com/ for anyone who's interested).

Although your guidance is sensible (I've argued the same elsewhere), one has to be careful not to be too proscriptive in defining the comparison configuration. Ideally, you'd like to say "I need this specific application to run at XXX speed - configure your cheapest solution to meet my SLA."

Otherwise, you may be forcing a less-than-optimal solution. For example, a customer recently required a quote for a DMX solution using all 300GB 10K rpm drives, even though the capacity requirements could be more cost-effectively supported using 500GB 7.2K drives - AND without sacrificing the required performance SLA. But since Hu-Know-Who's system doesn't support 500GB SATA drives, the customer chose to dumb down the comparison to the least common denominator instead of seeking the most optimal solution.

(We ultimately won the business, but it stretched out the sales cycle a bit).

But I agree wholeheartedly that a benchmark without cost limits is, well, silly.

The comments to this entry are closed.

anarchy cannot be moderated

about
the storage anarchist


View Barry Burke's profile on LinkedIn Digg Facebook FriendFeed LinkedIn Ning Other... Other... Other... Pandora Technorati Twitter Typepad YouTube

disclaimer

I am unabashedly an employee of EMC, but the opinions expressed here are entirely my own. I am a blogger who works at EMC, not an EMC blogger. This is my blog, and not EMC's. Content published here is not read or approved in advance by EMC and does not necessarily reflect the views and opinions of EMC.

search & follow

search blogs by many emc employees:

search this blog only:

 posts feed
      Subscribe by Email
 
 comments feed
 

 visit the anarchist @home
 
follow me on twitter follow me on twitter

TwitterCounter for @storageanarchy

recommended reads

privacy policy

This blog uses Google Ads to serve relevant ads with posts & comments. Google may use DoubleClick cookies to collect information (not including your name, address, email address, or telephone number) about your visits to this and other websites in order to provide ads about goods and services of interest to you. If you would like more information about this practice and your options for not having this information used by Google, please visit the Google Privacy Center.

All comments and trackbacks are moderated. Courteous comments always welcomed.

Email addresses are requested for validation of comment submitters only, and will not be shared or sold.

Use OpenDNS