0.039: ibm and spc vs. hitachi math
Hitachi dropped another shoe Monday with its announcement of the best-ever SPC-1 benchmark results for an Enterprise Storage System.
I'm sure the "thud" was pretty deafening over in Blue-ville, especially given this recent reiteration of IBM superiority by Tony Pearson - partially on the back of the DS8000's (now defunct) claims to the top spot on the SPC stepladder.
I pretty much established my position as a disbeliever in the real-world value of benchmarks such as the SPC in my prior post entitled the case against standardized benchmarking, so I won't rehash my arguments of irrelevance here.
But I will take the opportunity to add to my continuing expose of Hitachi math, both as a service to my readers, and in support my pals Tony & BarryW over at Poor Old Big Blue. I'm sure they'll have their own spin soon, but I figured I might be able to jump-start their responses.
But before you read on, I'm curious: Were YOU able to recognize the Hitachi math in this announcement?
mysteriously macabre marketing
First, let's break down this announcement from the top, starting with the headline, subtitle and first sentence:
Hitachi Universal Storage Platform V Blasts through SPC-1 Benchmark; Delivers Highest Performance of Any Enterprise Storage System Ever
USP V Breaks Records, Annihilates Competitors, and Delivers Superior Enterprise Storage Economics to Customers
Further expanding its industry lead in enterprise storage system performance while crushing through the limitations of aging monolithic architectures, Hitachi today announced that its flagship Universal Storage Platform™ V achieved the highest Storage Performance Council (SPC-1) benchmark result in enterprise storage system history, eclipsing the results of every single enterprise storage system ever tested using this widely recognized industry-standard benchmark.
Despite the fact that terms like "Blasts through," "Annihilates Competitors" and "crushing [...] aging [...] architectures" are generally avoided in the post-9/11 civilized world, for some unknown reason Hitachi's marketing miscreants continue to devolve everything they write into a distasteful parody of Halo 3. Hopefully its not a cultural thing, but I don't recall other competitors harkening up blood and guts imagery in their press announcements.
But I digress.
The early-warning indicator of impending Hitachi Math comes in the qualifier in the second phrase of the title, where it is admitted that the results aren't really The Best, only the Best Ever Tested For An Enterprise Storage System. The fact that only the IBM DS8300 Turbo and the Sun 9960 are the only other "enterprise storage systems" ever tested probably goes unrecognized by most readers.
But here's the real rub:
There are in fact other storage systems that have posted better SPC-1 results!
Posted right alongside Hitachi's results on the SPC web site, I was able to find several other platforms that posted certified results that were "better" than the Hitachi USP-V results. I didn't bother checking them all, but here's what I found:
|Platform||SPC-1 IOPS||Tested Data Capacity||Price of Tested Configuration||$'s per SPC-1 IOP||$'s per GB of Tested Data|
|Hitachi USP-V||200,245.73||26,000.000 GB||$3,525,389||$17.61||$139.59|
|IBM DS8300 Turbo||123,033.40||9,103.360 GB||$2,336,626.45||$18.99||$256.68|
|IBM SVC 4.2||272,505.19||24,433.589 GB||$3,284,767||$12.05||$134.44|
|Fujitsu Eternus8000||115,090.06||10,854.400 GB||$1,855,100||$16.12||$170.91|
|DataCore SANmelody||9,298.56||200.000 GB||$45,145.70||$4.86||$225.73|
|3PAR InServ S800||100,045.74||16,467.672 GB||$1,482,977||$14.81||$90.05|
|Texas Memory RamSan 320||112,491.34||68.719 GB||$168,776||$1.50||$2,456.07|
Now, maybe it's just me, but when I look at these results, I'm not so sure the USP-V is "the best" - by any comparison. We can argue over the definition of "enterprise" I guess, but the benchmark doesn't really measure "enterprise" - it measures performance. And the results are pretty clear - the SVC seems to hold the land speed record, and it beats the USP-V in cost per SPC-1 IOP as well. And if $/SPC-1 IOP really is most important, nothing seems to beat Texas Memories - heck, you could buy 20 of those RamSans for the price of a single USP-V, and you'd have more than 11 times the SPC-1 IOPS.
But then look at what the effective cost per gigabyte of actual data is for each of these platforms, and the picture gets even more confused. In a world where a 300GB SATA drive costs less than $1/GB down at Best Buy, these are pretty whacky numbers. I doubt that anyone would ever consider paying this kind of change for storage - even if their application DID match the SPC benchmark profile. I mean, it would be NICE if you could actually charge this much and get away with it, but I think those days are long behind us - wouldn't you agree?
So you tell me - who's really the "best" here?
I mean seriously - can you surmise from this collection of data that a USP-V will actually perform your specific workload better than any of the other kit?
And if you answered yes, I encourage you to do a little research into the actual SPC-1 benchmark. Although it claims to be representative of an OLTP workload, did you know that it originated as a model of a DEC Alpha-based email server? Fact is, the benchmark itself is over 7 years old, built in a bygone era before Exchange and Notes became the world's predominant enterprise email infrastructures. Suffice to say that those close to both will readily attest that the workload of each is radically different from each other, and neither bears any real resemblance to the typical modern-day OLTP workloads - that despite the reassurances from the SPC membership and participants.
Did you also know that SPC-1 is intentionally cache unfriendly? That it intentionally works on a widely scattered working set specifically to defeat any benefits of cache, prefetch and optimizations? This is why small memory systems appear to do so well - and also why one vendor once resorted to disabling cache protection in order to optimize their results (in a submission that the SPC now refers to as "Withdrawn," I guess).
And another fact about the SPC-1 benchmark that most who have never seen/touched/run it don't know is that while it does distinctly model 3 different Application Storage Units (different workloads), these three are run sequentially and not concurrently. This in effect asks the array to support one thing at a time. Now, I don't know about you, but I honestly don't know of ANY enterprise class arrays that ever operate under the luxury of supporting only one type of workload at a time. And while I understand that it is difficult and complicated to model multiple concurrent workloads, I stand by my assertion that this benchmark is unrealistically artificial and thus meaningless as either a prediction or a comparison tool.
But I promised not to get into all that again - so I won't. Except to note that the definition of "enterprise" storage typically means a healthy workload of both local and remote replication concurrent to the operational workload. Which neither SPC-1 nor SPC-2 include, and which I know first-hand causes most every tested system fits (if they even support array-based local and remote replication, that is - not all do!).
more hitachi math
Now if that isn't enough, deeper scrutiny reveals some apparent Hitachi shenanigans in how the systems were configured. Firstly, the USP-V used 1024 146GB 15K rpm disk drives to get these results. The DS8300 Turbo attained its results using only 512 drives - 256 73GB 15Ks and 256 146GB 15Ks. From a performance perspective, benchmarks like the SPC-1 are often what's called "spindle limited" - to get the best results, you need as many spindles as possible. So the USP-V has an undeniable advantage over the IBM because of having 2x as many spindles. (Granted, back with IBM ran the tests on the DS8300, the max standard config was 640 drives - but today you can get 1024 drives on your DS8300, so one wonders how that might change the results).
On top of that, Hitachi's benchmark results document that a whopping total of 150,528 GB of physical capacity was deployed to support only 26,000 GB of benchmark data - an incredible over-provisioning of nearly 6x more storage than the application required. And while that's really only 3x excess if you allow for the mirrored data protection, you have to wonder why the Hitachi configuration didn't also include any hot spare drives, while the DS8300 Turbo configuration did.
So, is this really an apples-to-apples comparison?
More importantly, do you really think you can conclude that the USP-V will be faster than the DS8300 Turbo if both systems were configured to cost exactly the same amount? Virtually every negotiation I've seen starts with the premise that the kit from each vendor should cost approximately the same, and in today's world I doubt ANYONE would get away with either:
- buying more 3-6x capacity than required by the application
- paying vendor A significantly more for the same amount of capacity from vendor B
So, if you held price/cost constant, used the same number of disk drives, the same number of ports, and the same amount of global memory...would the results still be the same?
I think not.
Now, the Storage Performance Council specifically intended to limit this sort of shell game by requiring vendors to quote the actual prices for the tested configurations. But there is an obvious flaw in this: the continuing disk drive price erosion gives a measurable advantage to vendors who test last. This allows Hitachi to use 2x as many drives as IBM without the end system costing 2x as much.
Oddly, even with twice as many drives, the USP-V isn't twice-as-fast as the DS8300 Turbo. And you know, I'm not quite sure why. Maybe the DS8300 Turbo is able to deliver more SPC-1 IOPS/spindle than the USP-V.
Or maybe this is yet another example of why the SPC-1 isn't really a meaningful benchmark. I mean seriously, if you have to dig this deep into the configurations and pricing and cost erosions and such, how the heck is anyone supposed to be able to make a reasonable judgement of "better?"
the case of the missing iops
Perhaps the biggest the most blatant slap in the face was the liberal intermixing of SPC-1 IOPS and "cache hit IOPS" in the Hitachi press release. So deftly done that more than one naive reporter has mixed the two numbers up in their coverage.
And on the other hand, some have joked that we need a "where's Waldo" search for the missing 3.3 million IOPS.
But the point is that Hitachi's SPC-1 results spotlight the ridiculousness of their "3.5 million cached IOPs" claims. Granted, the SPC-1 is intentionally cache-hostile, but there's more than an order of magnitude difference in the Nosebleed Numbers of "cached IOPS" and the Impractical Reality of SPC IOPs.
Not me. And probably not BarryW - we've recognized all along that there was no way the USP-V could deliver that much throughput. The SPC results just bring that whole inflated claim back down to earth.
Too bad the press and analysts are so easily mislead.
Well, hats off to Hitachi, I guess, for finally completing this test (which I just happen to know they started back in 2005 with the original USP - no explanation why they never published USP results though). At least there's one less thing for TonyP to crow about.
But to BarryW and the rest of you who will inevitably ask, I say don't expect this to change EMC's stance one iota. As evidenced by the confusion I've described above, the significance of the SPC benchmark remains in the category of useless and misleading. The workload isn't easily aligned with anything in the real world, and there is insufficient analytic understanding of how the workload scales with ports, spindles, cache or CPU power to be able to correlate the results across different systems from different vendors, much less to predict the results of different configurations of the same system from the same vendor.
And I personally remain convinced that storage performance is far too complex to tag with a simple rating system like the SPC.
so don't wait for emc to run the SPC tests
And not just because I personally don't think they're meaningful.
No, you don't have to wait, because I can tell you the answer right now: I assure you that whatever the SPC-1 results might say, DMX would come out The Best.
Yes, of course, marketing will spin the results, just like the Hitachi machine has done.
But more importantly, the world will not change. Customers who were happy with their DMX performance the day before EMC published the results would still be getting that same level of performance the day after. Those that wanted to buy a DMX beforehand will find a way to do so, even if the DMX results weren't greater than the USP-V...as evidence, I sincerely doubt that USP-V sales will be impacted significantly by the fact that the SVC is indeed not only faster, but cheaper for better performance.
In my mind, the SPC Just Doesn't Matter.
So I have to ask myself - why bother? Running this benchmark isn't cheap (although I do note that Hitachi has gone for the 3-fer, listing the single test results under all three names/brands that the USP-V is sold under - I wonder, did they all three split the costs?). And it's not easy, as BarryW lamented that just getting a complete run done without incurring a drive failure is a major challenge.
And given that EMC's performance gurus around the world would spend just as much time explaining why the SPC isn't representative of any customer workload whether the DMX results won, lost or weren't there, I find myself asking the age-old ROI question: