0.046: spc-1 results for symmetrix dmx

This is your SPC-1 benchmark. And this is your SPC-1 benchmark on a calculator.

If you have lots of time and money on your hands (and something to prove, I guess), you can collect a boatload of hardware, create a totally unrealistic (and prohibitively expensive) configuration, execute the benchmark, independently validate your results, and wrap yourself in the banality of standardized testing to claim world dominance. (Oh, and hope no drives fail during your testing, because the added overhead of rebuilding a drive would skew the results, if you in fact included hot spares in your configuration in the first place - most don't).

Alternatively, now that EMC CTO and Distinguished Engineer Dr. Subramanian Kartik (aka dotConnector) has decoded the Q-factor for the SPC-1 benchmark, you can take the less expensive approach, and just run the SPC-1 on your handy-dandy calculator. Doesn't even have to be one of those fancy scientific calculators you just had to buy for your son or daughter - no heavy duty calculus required.

No, since Dr. Kartik has demonstrated beyond any doubt that the SPC-1 is nothing more than a measurement of the number of IOPS you can get per spindle, it's really simple. Plot out the slope of the line, plug in the number of drives you plan to use, and viola! SPC-1 IOPS, calculated within a few fractions of a percentage point of all historical measurements to date.

Heck, even the minor tweaking of the benchmark over the years doesn't seem to have made much of a difference - all the results fall on the same line, irrespective of version, platform or tested configuration!

Go figure! 
 

dmx-4 spc-1 results

So...I know you're all dying to know the answer to this question: how many SPC-1 IOPS will a DMX do?

Simple: With a 2400 drive configuration, the DMX-3 and the DMX-4 will both do around 396,332 SPC-1 IOPS.

dmx wins again!!!

For those of you playing the home game, that's almost 2x the IOPS a USP-V can do with 1024 drives (look closely, and you'll note that the slope isn't 1:1, that's why 2x more drives doesn't deliver 2x more SPC-1 IOPS, as is evidenced by the two SVC results).

a picture's worth a couple thousand drives

dotConnector clearly summarized his findings by plotting all the results on a single chart, and I've taken the liberty to edit that chart to include the DMX-4 calculated results here (click the pic for a clearer image)

And in keeping with the precedent of most every other SPC-1 participant, we'll over-provision the DMX's for this test. Let's assume we've used the latest-greatest 300GB 15K rpm drives that are just now coming into the market. At 2400 drives, that's 720TB of raw storage - but everyone does the SPC-1 mirrored, so we're really talking 360TB of usable capacity. And to optimize response time, we really can't afford to fill all those drives up, so I'm thinking we'll use the same 26TB of test data that Hitachi (et al) did for the USP-V testing (they used 26TB of test data on 150TB of raw storage). That'll mean that each spindle will only have to deliver up about 10.8GB of data (about 3.33% of the total drive capacity), since Symmetrix DMSP (Dynamic Mirror Service Policy) will split the I/O's pretty evenly between the two mirrors so as to further minimize head movement.

Clearly not real-world, especially with today's focus on Green IT. But heck, this is just the SPC-1, nobody said it had to bear any relationship to reality!

With that sort of short-stroking of the drives, I'm guessing we should see the effective response times in the 1.2-1.5ms range - significantly better than the USP-V, since we're asking each drive to do even less than they did. (Maybe dotConnector can actually calculate that for us...I'm sure it's just more simple math - I mean, we're all using the same drives here, aren't we?)

done and done

I'm pretty sure that this is the closest you'll ever come to seeing SPC-1 results for a Symmetrix array, for all the reasons that I and others have discussed elsewhere, and because there really isn't anything new that this benchmark can prove. More spindles give you more SPC-1 IOPS, and that's that.

Architecturally and mathematically there's no reason to believe that real-world testing would reveal any different results, except that we may not be able to find a single server that can generate this level of I/O. Unfortunately, unlike the Real World that we live in, the SPC benchmarks are all single-server workloads trying to mimic a multi-server environment. And they have to be in order to coordinate the workflow and consistently measure the response times and throughput, at least until someone re-implements it as a distributed, platform independent, multi-node benchmark. Until then, the top-end limits are really whatever a single server can generate, and not necessarily the limits of the array itself.

But you should of course take these results with a grain of salt (which is just about as much as actually running the configurations is worth).

And this approach is far more cost-effective, so EMC's shareholders can applaud the wise use of capital on more important endeavors.

Personally, I'm looking forward to dotConnector's analysis of the SPC-2 workload, which seems less about spindles and IOPS, and more about bulk sequential I/O bandwidth (MB/s). I'm wondering if the real limiting factor of that test is even the storage at all - it could well be the single-host that's trying to simulate the workload has more to do with the results. Of course, it might be an embarrassment if we learn that the SPC tests are really a benchmark of server throughput and not the storage.

----

Oh, and BarryW, by the way - can you explain why the older version of the SVC (v3.1) gets more SPC-1 IOPS per spindle than the latest submission (v4.2)?

Comments

[Storagezilla]

You're no fun.

I would have asked people to guess what they think the DMX-4 could do on SPC and give a crummy prize for the closest number. ;)

[Barry Whyte]

Given that SVC supports 4096 mdisks, then by your very very very sketchy assumption SVC would do double that of DMX4. So SVC wins again.

However, disks are NOT the limiting factor here, the box itself is.

I'm preparing a full reply to Dr Kartik's over simplified statistical analysis - it may take a while as the full analysis needs to be done to draw any kind of sensible conclusion.

[the storage anarchist]

You might want to double check your math...adding 1696 drives to 2400 won't even come close to doubling your SPC-1 IOPS. :)

Given that the results from every single reported architecture running the SPC-1 all fall along the same IOPS/Spindle line, you'll have a difficult time proving that the benchmark is anything other than spindle limited. If architecture made any real difference in this benchmark, we should be seeing significant variations in the IOPS/spindle results. Instead, as OSSG points out, SPC-1 is really just a cache-hostile workload, and the only thing it tests is how fast the disks can service I/O requests.

Generally, and as intended by the originators of the SPC-1, such a workload minimizes any benefits of cache-centric storage arrays. That Intelligent Cached Disk Arrays (ICDAs) do in fact perform marginally better than mid-tier ones is largely due to write buffering and the minimalist benefits of prefetch/read-ahead that they are able to sqeak out.

But since there is no stand-out winner or loser in the IOPS/spindle race clearly indicates that the benchmark designers were successful in their attempt to neutralize any ICDA advantage - even though this class of storage inarguably delivers real-world performance advantage for multipel concurrent large/complex/mixed/variable workloads and where local & remote replication is a prerequisite.

Alas, SPC-1 clearly doesn't represent this sort of real-world workload.

But you didn't answer my question - why is the newer SVC slower than the old one?

[Open Systems Guy]

The architecture will determine how many spindles you can add before the controller can no longer saturate the drives- I commented again on .connector's post to this effect. Essentially, most companies will value features and reliability over performance, but there should be some sort of benchmark to support a vendor's claim about if and when their controller will stop scaling its performance as drives are added.

[the storage anarchist]

We are all 100% in agreement that features and availability are most important (along with reputation, service, value, etc.)

But this benchmark doesn't do anything to prove how far a storage array will scale - not for the artificial workload it mimics, and much less for any real-world work-load.

Knowing how many IOPS an array can service with a "perfectly optimized" configuration that uses only 3.3% of a disk drive is totally meaningless. Since nobody runs this sort of configuration in the real world, it would be MUCH MORE VALUABLE to know how these products handle the normal contention for resources, bandwidth and the inherent overhead of drive latencies. Only then will architectural differences such as cache, interprocessor communications and optimization algorithms be tested and quantified.

I contend that the system that handles "perfect conditions" best is not necessarily the one that will handle the unpredictable chaos of a real-world environment. I have no idea how to create a repeatable test to prove this, but I do know that the SPC is nowhere near filling this need (and by design, actually).

[dotconnector]

The "simplicity" of the analysis is what I found amazing, BarryW. For a good benchmark, I would have expected that it could distinguish between the DS8300, SVC and the USP V given fixed spindle count. It doesn't.

I am not even claiming this is an analysis in the true sense of the term. What I presented was an observation - one that my 7-year old son can make. One look at the data and anyone can tell - its a very straight line. It doesn't get any simpler than that.

OSSG pointed out that eventually the controller itself may be the limiting factor (front-end, cache bandwidth, architecture. True in theory - but honestly, I really believe that for high-end systems of today, spindles are it.

Take a look at the maxed out USP V. Spindles are still the limiting factor - there is no sign of other component saturation, and the box is full!

I would love to see your more detailed and sophisticated analysis of this straight line, BarryW.

[Snig]

What you guys aren't taking into account is the additional latency added as you add more drives to a box. The SVC is a perfect example. It's latency is way out of whack when you look at IBM's results. Who wants to run an application that won't get a response back for half a second or more?

[Dan]

It may be an older article, but 'nuff said on the SPC testing ...
http://www.theregister.co.uk/2002/12/11/hp_comes_clean_in_latest/

[the storage anarchist]

Snig - how right you are about the SVC...since it is literally "blind" to the physical storage devices, it really can't do any sort of optimization (other than configure LUNs so small as to only use the outer tracks of each drive). \

Although I'll admit I don't understand why BarryW says the SVC is limited to 4096 drives - shouldn't the actual drive count be opaque?

"Real" storage arrays, on the other hand, do all kinds of optimizations to order I/O requests in accordance to where heads are positioned and to compensate for rotational latency. The SVC is at the mercy of the back-end storage to handle this minutia, and while it does pretty well for the total IOPS, it scores very poorly on the response time metric.

I'm pretty sure Kartik is going to explain the math behind the response times in a future blog...

By the way, most people don't realize that the response time metric for the SPC-1 isn't measured under full load, but under a prescribed fraction of the workload. And there are lots of other wrinkles like these that make translating the results into real-world expectations difficult, if not impossible.

Dan - thanks for the find. It's sad to admit that not much has changed or improved with the SPC or its proponent participants since 2002.

The more people understand about this benchmark, the more I expect will ignore it.

[Barry Whyte]

Its not the straight line thats of interest, its the ones WAY off that are of interest.

See my response at the link below.

[Barry Whyte]

Snig - "half a second or more" - am I missing something - the average worst case at full loading is around 15ms - even in the detailed full disclosure report I don't see any peak response time measurements?

[Snig]

I guess I should have qualified that statement. You are right in that the SVC doesn't have any numbers in your report that shows .5 seconds for the response time. Based on the latency curve though I can imagine it won't take too long for it to get there when you continue adding more drives.

This would happen to any array including the DMX, USP-V, etc., etc. You have to continue to add front end processing as you add more disk. While the DMX may be able to beat everything with the mathematical calculations done here, it still comes down to the front end and I still don't believe that the DMX can run all 2400 at the full IOPS rating of the drives. EMC is going to have to prove it to me rather than using some silly math calculation.

[the storage anarchist]

Snig - Next time you (or anyone else, for that matter) is in the market for a new high-end storage array, ask your EMC Sales Rep to hook you up with his or her SPEED Guru. They'll work with you to understand your real workload, model it with their capacity/performance planning tools, and then they will share with you the IOPS you can expect the front-end and back-end of the DMX-4 to handle, and the disk and cache config you'll need to make it happen.

And I can guarantee you that discussion will be far more beneficial (and relevant) than any of these SPC gyrations.

Unless you happen to work for someone who doesn't care how much the storage costs or how much energy it wastes. In that case, the SPC results table might be all you need to make your decisions :)

[Dave Vellante]

It seems benchmarks have played a role in advancing comparative metrics in database, networking, server and other worlds...why not storage? Here's a perspective that says if you don't like the benchmarks...help make them better so users ultimately will benefit:
http://www.wikibon.org/Even_bad_benchmarks_can_lead_to_good_standards

[the storage anarchist]

Dave (& Peter) - Methinks you are being a bit naive about the real use of benchmarks, and the difficulty of making them "real world."

Having been a performance specialist for both large and small companies, I know first-hand that the primary purpose of benchmarks is for the "little guy" to prove that he's as good or better than the "big guys." And the SPC is no different - if you ask the original membership (and if they answer truthfully) they will tell you that the whole idea of the SPC was to define benchmarks that demonstrated that the "little guys" could beat EMC's Symmetrix. This is of course why the workloads are carefully crafted to minimize the role that cache and pre-fetch can play, since the "little guys'" products don't include the CPU power or global memory necessary to implement performance-assisting algorithms. Thus these benchmarks "top out" at whatever the disk drives can deliver in totally unrealistic configurations (e.g., single server workloads against only 3-10% physical storage utilization). If anything, the SPCs measure only the effeciency of the host-to-disk I/O path (which is why JBOD configurations do so well in the SPC tests).

You assert that imperfect benchmarks are still beneficial, and that "it seems" benchmarks have played a role everywhere else but storage. I'll counter that there isn't a performance benchmark that's useful in comparing a z9 to a p6 or a BladeCenter complex...and (more importantly) that people don't buy those platforms based in any way on any benchmarks you might cite. And if anyone makes a purchase decision based in any part on an "imperfect benchmark," they are likely to face repurcussions later if the imperfection happens to align poorly with their own real-world workload.

Systems as complex and configurable as the z9, a DMX4, a USP-V, or any blade server are nearly impossible to benchmark, because the workloads they will support are far too diverse and random to characterize in a manner that is sufficiently repeatable and controlled to meet the rigors of scientific method.

And I contend that it is much more realistic to suggest that customers should ask their vendors to propose configurations that will meet their SLA requirements, and for them to have the vendors explain how they arrived at the recommended configuration. Given the impossibility for most people to translate spec sheets or the simplistic workloads of SPC into realistic expectations for their own particular environments, they should enlist the help of professionals trained in the science of performance and capacity planning.

At EMC, these are known as SPEED Gurus - get them involved whenever performance of an EMC product is a consideration.

[the storage anarchist]

Dave Vellante and I continued this conversion via email, and he suggested I should post my response here as well. So here's where I left it with him:

---
Dave-

I'm sorry, but I grow weary of all the wise folks who advise "if it's broken, then fix it or stop complaining." The thing is, it is very easy for just about anyone to recognize that benchmarks like the SPC are truly meaningless. But that doesn't make the solution easy or simple. Nor is it fair to challenge the ones who point out that the emperor has no clothes to also be the ones to fix the problem - to me, that's just an excuse to justify the "anything is better than nothing" stance that Peter espouses. Thankfully the American Pioneers didn't harbor that notion, or these United States would only reach from New York to the Mississippi, instead of all the way to California.

So I hereby challenge you and the rest of the storage community to come up with an alternative to the Storage Performance Council, with the objective to define MEANINGFUL BENCHMARKS for enterprise storage environments instead of ones optimized to favor the simplistic, under-cached RAID controllers. EMC probably can't sponsor such a thing by itself, for obvious reasons. But if a group emerges with a defined charter that sets out to create realistic benchmarks that help modern enterprise IT architects make better informed decisions, I'll do everything in my power to get EMC to join in.

Not that I really think it is even possible, mind you, to create a truly representative benchmark - if it was easy, it would have already been done.

Nor do I think it even necessary - there are really only 2 viable enterprise-class storage vendors (and one dual-controller wanna be) that can realistically provide the scalability, performance, availability, replication and environmental efficiency to support exa-scale consolidation. And while any benchmark between only 2 players is marginally interesting (in a macho sense), you've got to admit, the benchmark will rarely (if ever) be the deciding factor. I mean seriously, when all the "drag race" proves is that we're all within a few percentage points of what the hard drives can deliver, does it really matter who's faster?

And seriously - do YOU really thing DMX-4 is significantly slower than the USP or USP-V? Or that the predominant number of customers who choose DMX over USP are stoopid? Symmetrix hasn't been #1 in market share for over 17 years by being second best, I can assure you. If the product really was all that bad, our revenues and market share wouldn't still be growing today (and trust me, it is).

I think it better if we all spend our time and money making this stuff simpler, more autonomic and more energy efficient, instead of running mindless benchmarks. My estimation of the cost to run SPC-1 is over $2M - given that the World's Fastest (commerical) Server is required to even come CLOSE to loading up the storage. Note that the p5 that was used for the DS8000, the SVC and USP-V tests costs more than $1M alone (discounted). Even if the storage is free (it isn't, since EMC doesn't actually make disk drives), that's a lot of coin to prove - ABSOLUTELY NOTHING.

Proving that the DMX can (or can't) do X-ty-hundred-thousand SPC-1 IOPS is undeniably meaningless, except that the SVC and DS8000 wanna-bees can claim to be "just as efficient" or even "more efficient" on this unrealistic benchmark.

And this all presumes that one p5 can simulate and generate the workload of hundreds or thousands of hosts. Yet we have customers HAPPILY running multiple p5 595's against a single DMX. Methinks we need a more comprehensive benchmark.

Like I've said on every thread related to this topic - define a benchmark that can't be optimized by throwing unrealistic amounts of underutilized hardware at the problem, and we can talk. It's gotta be real-world - multiple hosts, variable workloads, unexpected "random" spikes in performance demands, and all the while including the overhead of local & remote replication, drive rebuilds, greater than 60% capacity utilization, and the like. Come up with that, THEN everyone will be able to see who's really done the best job of designing a system to deliver Predictable Performance for an Unpredictable World (that's been a tag-line for DMX since 2003).

Anybody can optimize for the perfect cast - EMC has spent the last 17 years optimizing for the real-world worst case. And #1 high-end market share for 15+ years is the only benchmark we really need to prove it.

[Subramanian Kartik]

Hi Barry,
In the quest for a more meaningful benchmark, I've started the process (hopefully!) in a blog post in http://dotconnector.typepad.com

Check it out!
Cheers, K.

[proslaviy]

Hi, how I can send PM?

[the storage anarchist]

Use the "email the anarchist" link in the top section of the sidebar.