1.028: benchmarketing. badly.
OK, so my regular readers know where I stand on so-called "standardized" storage benchmarks: they are bad for our industry, and they lead customers (and vendors) to do dumb things.
For context, newcomers are encouraged to revisit my earlier post on the subject: 0.021 the case against standardized (performance) testing.
- IBM's Quicksilver science experiment to attain 1M IOPS using jury-rigged unreleased kit
- Texas Memory Systems' inevitable response to Quicksilver using equally jury rigged kit
- IBM's "enterprise" benchmark of the new DS5000, commissioned to ESG
IMHO, as I'll explain below, this sort of benchmarketing isn't helping consumers to make informed decisions. In fact, if anything, these are nothing more than carefully architected marketing ploys masked as "scientifically representative tests" intended to influence the relative naïveté of the masses who truly have no real understanding of how to measure or compare performance.
UPDATED Oct 31, 2008 with corrections provided by BarryW (IBM) and Woody Hutsell (RamSAN).
quicksilver: not so quick
As I discussed in my afore-linked "flash-y science experiment" post, what BarryW and his band of merry men and women put together to attain 1M IOPS was indeed nothing more than a science experiment.
What I've found interesting, however, is that IBM have never come right out and explained the actual configuration they used to attain those numbers - even when I've asked, BarryW has dodged the question.
The reason for the dodge was unknown, until earlier this week when a reader (who wishes to remain anonymous) sent me the results of his sleuthing. As this reader explained to me:
IBM’s 1.1 million IOPS claim “broken down” lays out in the following manner:
71 SAN Volume Controller cluster was required; the cluster was made up of twoten nodes, each with 4 FC ports. Total 10 nodes and 40 FC ports, meaning each node in this unsupported and nonstandard SVC configuration pushed a paltry 78571110,000 average IOPS from requester to storage and back.
- 29 host or target mode systems were required for the back-end storage, each with either one or two 250GB Fusion-io IO adapters. Each one of these servers conceptually would be equivalent to an enterprise RAID controller or head. This means that the average IOPS per server/controller was only 37,931 IOPS—not exactly impressive, even by DS5000 standards (more on that later).
- 41 total Fusion-io adapters were used. This equates to 26,829 IOPS per adapter. This is approximately 25% of the Fusion I/O stated performance of this adapter.
- We have no facts as to the actual workload that was used, the number of servers that were required to run this workload, nor the number of FC target or initiator ports that were involved. Judging from the numbers, though, it's probably safe to assume that the workload was nothing more multi-threaded IOmeter with very small block sizes - it most definitely wasn't anything even remotely representative of real-world.
- And no, there was no error detection, correction or protection applied to the Fusion-io storage adapters, and even the adapters themselves do almost no error correction/recovery. Net-net: you'd better mirror every write and T10DIF all your data if you really want to use these as persistent storage devices.. Otherwise there's nothing to protect against software errors, microcode bugs or SEUs (undetected errors caused by cosmic radiation - don't laugh, it happens more than you know). Net-net: good data in, garbage out. Do you feel lucky?
So basically, IBM striped (RAID 0) a bunch of servers and Fusion I/O adapters together to realize lackluster results at best. They then made up for this by throwing 29 target servers and 10 initiator servers at the problem until they hit the magic million IOPS milestone.
Seem's the lure of "One MEELLLYUN" caused Big Blue to do Bad Things - cobble together a kludge of a kit to meet a marketing number and then make the implementation as opaque as possible to avoid comparative evaluation. BarryW asserts that Quicksilver has garnered tons of interest from IBM customers and prospects, and I don't doubt that this is true.
But I suspect few of them understand how meaningless this whole science experiment really is - it proves nothing other than how well IBM can obfuscate facts.
And the fact is, while this science project proved it was possible to deliver 1M IOPS by aggregating multiple flash storage arrays/controllers/servers/adapters, the experiment needs a boatload of server hardware and yet still fails to deliver more than 1/4th the rated IOPS of the flash devices themselves.
Hardly revolutionary, much less innovative or awe-inspiring.
It is, however, Benchmarketing.
everything's bigger down in texas
Including the Benchmarketing.
Not to be outdone, the ladies and gentlemen at Texas Memories have responded to QuickSilver with their own cobbled-together 1-MEELLLYUN IOPS concoction. To their credit, they at least they had the decency to do this using generally available, currently shipping products.
But it's no less of a science experiment, IMHO.
See, what the TMS folks did was take 10 of their RamSAN 500's, each delivering up to 100,000 IOPS, and put them all into a single rack. Re-label it as the RamSAN 5000 (clever, that), and viola, a 1M IOPS flash SSD in-a-box.
Except that all those IOPS don't really come out of the flash. See, the RamSAN 500 employs up to 64GB of DDR SDRAM to get the IOPS, and up to 4TB of flash. So you'll only get 100,000 IOPS if your working set doesn't greatly exceed 64GB, but it still must fit within 4TB.
UPDATE: Woody sent me screen shots showing that the 100,000 IOPS were indeed cache miss, served up from the flash and not SDRAM. He also sent me a screen shot of something doing over 580,000 IOPS cache hit from the SDRAM. Unfortunately, he provided no explanation as to why they didn't just use TWO RamSAN 500's to demonstrate the 1MEELLYUN IOPS.
That's a lot of IOPS to drive into a relatively small amount of storage -
1500 IOPS per GB of RAM, or 25,000 per TB of flash (dependent upon how you look at it). That's roughly 3570 IOPS per 146GB Flash SSD (in a 7+1 RAID configuration), if you were to run this on say, a DMX-4 with EFDs.
And to get to 1,000,000 IOPS, TMS simply has you stripe your LUNs across 10 RamSANs.
Interesting. Feasible, even, as evidenced by there referenced (but unnamed) customer running an unspecified workload.
The performance comes out of 640GB of DRAM (10 x 64GB), while the total capacity is only 40TB - All for a cool $1.5M US Dollars.
But hardly compelling proof without the evidence of how the 1M IOPS was measured. Like IBM, TMS offers no information about the configuration, server or application specifics required to achieve this 1M IOPS.
The audience is left to assume that they could get 1M IOPS out of virtually any application they threw at the box.
Which we all know isn't true.
But I'm positive it's exactly what TMS marketing wants you to think.
Come on guys, I know you can do better than that - give us the data behind the claims. And please don't succumb into behaving like IBM with misleading misrepresentations!
of ibm and the ds5000 enterprise wanna-be
Back on IBM and benchmarketing,
I was pleasantly surprised to learn that both IBM and ESG agree with me about the relevance and importance of the Storage Performance Council benchmarks.
That is, SPC's are a meaningless tool by which to measure or compare
enterprise storage arrays.
As you might expect, I'll stop here and outright challenge the assertion that the DS5000 is an "enterprise storage array" - anything with a dual controller is in no way "enterprise class", and I don't care if it can dynamically rebalance workloads (more on the hilarity of that assertion by Hitachi in a future post).
No, the DS5000 isn't "enterprise", but it - like most ALL arrays of any significant capacity, will in fact be used to support multiple concurrent and non-orderly applications. Which is why I make the above assertion: the SPCs do not simulate multiple concurrent applications - they model A SINGLE application, and the relevance of THAT application to anything that you might run in your own environment is both indeterminate and irrelevant unless you happen to run SPCs as the only application on your array 24x7x365 (which BarryW might do, but none of the rest of us in this planet would find that fun or productive, I'm sure).
But rather than me rehashing my position, allow me to quote from the ESG DS5000 report that IBM commissioned, from the section entitled A Mixed Real-world Benchmark Methodology:
"Conventional server benchmarks were designed to measure the performance of a single application running on a single operating system inside a single physical computer. SPEC CPU2000 and CPU2006 are well known examples of this type of server benchmarking tool. Much like traditional server benchmarks, conventional storage system benchmarks were designed to measure the performance of a single storage system running a single application workload. The SPC-1 benchmark, developed and managed by the Storage Performance Council with IBM playing a key role, is a great example. SPC-1 was designed to assess the performance capabilities of a single storage system as it services an on-line interactive database application.
Traditional benchmarks running a single application workload can’t help IT managers understand what happens when a mix of applications are deployed together in a virtual server environment. [Emphasis mine]
I couldn't have said it better myself.
And you know as well as I that this isn't limited to a "virtual" server environment - traditional benchmarks like the SPC do nothing to help you understand how the system under test is going to perform under multiple and dynamic workloads on either real or virtual servers, each competing for resources and bandwidth in an unpredictable manner.
So finally, an "independent" third party, commissioned by one of the SPC's staunchest supporters, admitting publicly that the SPC benchmarks aren't good enough to simulate the real-world performance characteristics of anything but single-application workloads.
And as we all know, the best sprinter isn't necessarily the one who'll win the Decathlon.
Now, I know I should rest my case right here, and send my compatriots at IBM Thank You cards.
but I won't
And do you know why?
Because the solution to the problem that ESG hath concocted is no better than the SPC.
Now ESG isn't necessarily to fault with this, because it's hard to create a repeatable, real-world emulating multi-application benchmark that can place the identical workload across a broad variety of target systems.
I know - I've actually tried several times in my own career.
The problem with the ESG test is that although it uses several reasonably-well understood workloads, it uses them in a manner that can't be repeated. There are several reasons behind my assertion, but the biggest is that the amount of time the benchmark runs is limited by how long it takes JetStress to complete its entire cycle - once it's done, everything else is stopped, and the results gathered about how much work each got done.
So, if run on a different server/storage configuration that could complete JetStress in 1/2 the time, all the rest of the tests would only run 1/2 as long. Do know know how to interpret the results in that case? Is IOPS the real relevant metric, or would perhaps MB/s be more important. Peak performance, or total amount of work done?
And most importantly, given that the benchmark runs multiple instances of Exchange against a single array alongside several other load generators, how closely does THAT match your own world? Can you determine from the results whether the system under test will run multiple instances of your Oracle workload as fast or as well?
See...there are still too many questions that can't be answered from the results, which puts this approach to benchmarking in the same sinking ship as the SPCs - meaningless results with indeterminate implications.
The very definition of Benchmarketing.
P.S. - I am at EMEA Customer Council in Madrid, ES today (as StorageBod has already leaked - he looks nothing like his caricature, BTW) and a customer here asked me if EMC would be willing to participate in an initiative to get multiple storage vendors to collaborate on truly representative real-world "enterprise-class" benchmarks, and I reassured him that I would personally sponsor active and objective participation in such an effort - IF he could get the others to join in with similar intent.
And in fact, Dr. Kartik and I tried to spin this very proposal up as a community development project last year, but it didn't get very far due to widespread lack of participation and interest. Could it be that nobody really wants to create a truly representative benchmark, for fear that they'd lose their mythical SPC advantage?
In any event, Kartik's offer still stands - so if you're truly interested in getting us beyond the benchmarketing (rather than just harping on why EMC should be running the SPC's that now both IBM and ESG admit are meaningless), well, we'd love to hear from you. You can catch up on what progress has been made over on dotConnector's blog, starting here and reading forward. Drop Kartik or I a note if you're ready to help take the next steps.