« 1.027: unprotected: no thanks, michael... | Main | 1.029: atmos. with, and without, the sphere »

October 30, 2008

1.028: benchmarketing. badly.

OK, so my regular readers know where I stand on so-called "standardized" storage benchmarks: they are bad for our industry, and they lead customers (and vendors) to do dumb things.

For context, newcomers are encouraged to revisit my earlier post on the subject: 0.021 the case against standardized (performance) testing. 

imageAs if  to underscore my point, there have recently been three separate applications of what I'll call Bad Benchmarketing that have caught my attention:

  1. IBM's Quicksilver science experiment to attain 1M IOPS using jury-rigged unreleased kit
  2. Texas Memory Systems' inevitable response to Quicksilver using equally jury rigged kit
  3. IBM's "enterprise" benchmark of the new DS5000, commissioned to ESG

IMHO, as I'll explain below, this sort of benchmarketing isn't helping consumers to make informed decisions. In fact, if anything, these are nothing more than carefully architected marketing ploys masked as "scientifically representative tests" intended to influence the relative naïveté of the masses who truly have no real understanding of how to measure or compare performance.

Benchmarketing personified.

UPDATED Oct 31, 2008 with corrections provided by BarryW (IBM) and Woody Hutsell (RamSAN).

quicksilver: not so quick

As I discussed in my afore-linked "flash-y science experiment" post, what BarryW and his band of merry men and women put together to attain 1M IOPS was indeed nothing more than a science experiment.

What I've found interesting, however, is that IBM have never come right out and explained the actual configuration they used to attain those numbers - even when I've asked, BarryW has dodged the question.

The reason for the dodge was unknown, until earlier this week when a reader (who wishes to remain anonymous) sent me the results of his sleuthing. As this reader explained to me:

IBM’s 1.1 million IOPS claim “broken down” lays out in the following manner:

  1. 7 1 SAN Volume Controller cluster was required; the cluster was made up of two ten nodes, each with 4 FC ports. Total 10 nodes and 40 FC ports, meaning each node in this unsupported and nonstandard SVC configuration pushed a paltry 78571 110,000 average IOPS from requester to storage and back.
  2. 29 host or target mode systems were required for the back-end storage, each with either one or two 250GB Fusion-io IO adapters. Each one of these servers conceptually would be equivalent to an enterprise RAID controller or head. This means that the average IOPS per server/controller was only 37,931 IOPS—not exactly impressive, even by DS5000 standards (more on that later).
  3. 41 total Fusion-io adapters were used.  This equates to 26,829 IOPS per adapter.  This is approximately 25% of the Fusion I/O stated performance of this adapter. 
  4. We have no facts as to the actual workload that was used, the number of servers that were required to run this workload, nor the number of FC target or initiator ports that were involved. Judging from the numbers, though, it's probably safe to assume that the workload was nothing more multi-threaded IOmeter with very small block sizes - it most definitely wasn't anything even remotely representative of real-world.
  5. And no, there was no error detection, correction or protection applied to the Fusion-io storage adapters, and even the adapters themselves do almost no error correction/recovery. Net-net: you'd better mirror every write and T10DIF all your data if you really want to use these as persistent storage devices.. Otherwise there's nothing to protect against software errors, microcode bugs or SEUs (undetected errors caused by cosmic radiation - don't laugh, it happens more than you know). Net-net: good data in, garbage out. Do you feel lucky?

So basically, IBM striped (RAID 0) a bunch of servers and Fusion I/O adapters together to realize lackluster results at best.  They then made up for this by throwing 29 target servers and 10 initiator servers at the problem until they hit the magic million IOPS milestone.

Seem's the lure of "One MEELLLYUN" caused Big Blue to do Bad Things - cobble together a kludge of a kit to meet a marketing number and then make the implementation as opaque as possible to avoid comparative evaluation. BarryW asserts that Quicksilver has garnered tons of interest from IBM customers and prospects, and I don't doubt that this is true.

But I suspect few of them understand how meaningless this whole science experiment really is - it proves nothing other than how well IBM can obfuscate facts.

And the fact is, while this science project proved it was possible to deliver 1M IOPS by aggregating multiple flash storage arrays/controllers/servers/adapters, the experiment needs a boatload of server hardware and yet still fails to deliver more than 1/4th the rated IOPS of the flash devices themselves.

Hardly revolutionary, much less innovative or awe-inspiring.

It is, however, Benchmarketing.

everything's bigger down in texas

Including the Benchmarketing.

Not to be outdone, the ladies and gentlemen at Texas Memories have responded to QuickSilver with their own cobbled-together 1-MEELLLYUN IOPS concoction. To their credit, they at least they had the decency to do this using generally available, currently shipping products.

But it's no less of a science experiment, IMHO.

See, what the TMS folks did was take 10 of their RamSAN 500's, each delivering up to 100,000 IOPS, and put them all into a single rack. Re-label it as the RamSAN 5000 (clever, that), and viola, a 1M IOPS flash SSD in-a-box.

Except that all those IOPS don't really come out of the flash. See, the RamSAN 500 employs up to 64GB of DDR SDRAM to get the IOPS, and up to 4TB of flash. So you'll only get 100,000 IOPS if your working set doesn't greatly exceed 64GB, but it still must fit within 4TB.

UPDATE: Woody sent me screen shots showing that the 100,000 IOPS were indeed cache miss, served up from the flash and not SDRAM. He also sent me a screen shot of something doing over 580,000 IOPS cache hit from the SDRAM. Unfortunately, he provided no explanation as to why they didn't just use TWO RamSAN 500's to demonstrate the 1MEELLYUN IOPS.

That's a lot of IOPS to drive into a relatively small amount of storage - 1500 IOPS per GB of RAM, or 25,000 per TB of flash (dependent upon how you look at it). That's roughly 3570 IOPS per 146GB Flash SSD (in a 7+1 RAID configuration), if you were to run this on say, a DMX-4 with EFDs.

And to get to 1,000,000 IOPS, TMS simply has you stripe your LUNs across 10 RamSANs.

Interesting. Feasible, even, as evidenced by there referenced (but unnamed) customer running an unspecified workload.The performance comes out of 640GB of DRAM (10 x 64GB), while the total capacity is only 40TB - All for a cool $1.5M US Dollars.

But hardly compelling proof without the evidence of how the 1M IOPS was measured. Like IBM, TMS offers no information about the configuration, server or application specifics required to achieve this 1M IOPS.

The audience is left to assume that they could get 1M IOPS out of virtually any application they threw at the box.

Which we all know isn't true.

But I'm positive it's exactly what TMS marketing wants you to think.

Come on guys, I know you can do better than that - give us the data behind the claims. And please don't succumb into behaving like IBM with misleading misrepresentations!

of ibm and the ds5000 enterprise wanna-be

Back on IBM and benchmarketing,

I was pleasantly surprised to learn that both IBM and ESG agree with me about the relevance and importance of the Storage Performance Council benchmarks.

That is, SPC's are a meaningless tool by which to measure or compare
enterprise storage arrays.

As you might expect, I'll stop here and outright challenge the assertion that the DS5000 is an "enterprise storage array" - anything with a dual controller is in no way "enterprise class", and I don't care if it can dynamically rebalance workloads (more on the hilarity of that assertion by Hitachi in a future post).

No, the DS5000 isn't "enterprise", but it - like most ALL arrays of any significant capacity, will in fact be used to support multiple concurrent and non-orderly applications. Which is why I make the above assertion: the SPCs do not simulate multiple concurrent applications - they model A SINGLE application, and the relevance of THAT application to anything that you might run in your own environment is both indeterminate and irrelevant unless you happen to run SPCs as the only application on your array 24x7x365 (which BarryW might do, but none of the rest of us in this planet would find that fun or productive, I'm sure).

But rather than me rehashing my position, allow me to quote from the ESG DS5000 report that IBM commissioned, from the section entitled A Mixed Real-world Benchmark Methodology:

"Conventional server benchmarks were designed to measure the performance of a single application running on a single operating system inside a single physical computer. SPEC CPU2000 and CPU2006 are well known examples of this type of server benchmarking tool. Much like traditional server benchmarks, conventional storage system benchmarks were designed to measure the performance of a single storage system running a single application workload. The SPC-1 benchmark, developed and managed by the Storage Performance Council with IBM playing a key role, is a great example. SPC-1 was designed to assess the performance capabilities of a single storage system as it services an on-line interactive database application.

Traditional benchmarks running a single application workload can’t help IT managers understand what happens when a mix of applications are deployed together in a virtual server environment. [Emphasis mine]

I couldn't have said it better myself.

And you know as well as I that this isn't limited to a "virtual" server environment - traditional benchmarks like the SPC do nothing to help you understand how the system under test is going to perform under multiple and dynamic workloads on either real or virtual servers, each competing for resources and bandwidth in an unpredictable manner.

imageSo finally, an "independent" third party, commissioned by one of the SPC's staunchest supporters, admitting publicly that the SPC benchmarks aren't good enough to simulate the real-world performance characteristics of anything but single-application workloads.

And as we all know, the best sprinter isn't necessarily the one who'll win the Decathlon.

 

Now, I know I should rest my case right here, and send my compatriots at IBM Thank You cards.

but I won't

And do you know why?

Because the solution to the problem that ESG hath concocted is no better than the SPC.

Now ESG isn't necessarily to fault with this, because it's hard to create a repeatable, real-world emulating multi-application benchmark that can place the identical workload across a broad variety of target systems.

I know - I've actually tried several times in my own career.

The problem with the ESG test is that although it uses several reasonably-well understood workloads, it uses them in a manner that can't be repeated. There are several reasons behind my assertion, but the biggest is that the amount of time the benchmark runs is limited by how long it takes JetStress to complete its entire cycle - once it's done, everything else is stopped, and the results gathered about how much work each got done.

So, if run on a different server/storage configuration that could complete JetStress in 1/2 the time, all the rest of the tests would only run 1/2 as long. Do know know how to interpret the results in that case? Is IOPS the real relevant metric, or would perhaps MB/s be more important. Peak performance, or total amount of work done?

And most importantly, given that the benchmark runs multiple instances of Exchange against a single array alongside several other load generators, how closely does THAT match your own world? Can you determine from the results whether the system under test will run multiple instances of your Oracle workload as fast or as well?

See...there are still too many questions that can't be answered from the results, which puts this approach to benchmarking in the same sinking ship as the SPCs - meaningless results with indeterminate implications.

The very definition of Benchmarketing.

 

P.S. - I am at EMEA Customer Council in Madrid, ES today (as StorageBod has already leaked - he looks nothing like his caricature, BTW) and a customer here asked me if EMC would be willing to participate in an initiative to get multiple storage vendors to collaborate on truly representative real-world "enterprise-class" benchmarks, and I reassured him that I would personally sponsor active and objective participation in such an effort - IF he could get the others to join in with similar intent.

And in fact, Dr. Kartik and I tried to spin this very proposal up as a community development project last year, but it didn't get very far due to widespread lack of participation and interest. Could it be that nobody really wants to create a truly representative benchmark, for fear that they'd lose their mythical SPC advantage?

In any event,  Kartik's offer still stands - so if you're truly interested in getting us beyond the benchmarketing (rather than just harping on why EMC should be running the SPC's that now both  IBM and ESG admit are meaningless), well, we'd love to hear from you. You can catch up on what progress has been made over on dotConnector's blog, starting here and reading forward. Drop Kartik or I a note if you're ready to help take the next steps.

 


TrackBack

TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d834c659f269e2010535cadb98970c

Listed below are links to weblogs that reference 1.028: benchmarketing. badly.:

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Barry Whyte

So I think you are tarnishing the information about quicksilver with an unfair slant.

We were quite clear - it was too much technical information to put in the press release - hence why it all appeared in my detailed blog post. This was a mixed 70/30 workload running 4K blocks across the entire capacity of the attached SSD storage. This was using my daily tool that I use to benchmark SVC code releases in the lab, which is a simple I/O generation tool using standard C based APIs for doing read10, write10 I/O to an opened file object.

We would have liked to use something like SPC-1 so it it was a known benchmark (despite your dislike) but since this is not a shipping product you cannot publish any benchmark figures. Hence why we ran the same 70/30 4K all miss workload on the 8 node SVC cluster we used for our normal SPC-1 benchmark and used this in the numerical comparisons used for the press release.

Before I move on to correct some errors above - I think you are missing *the whole point* of what quicksilver showed. Its primary objective was to show that a multi-node scale-out single system image was a much better fit for SSD technology than a monolithic scale-up approach. Remember that the 1Million mixed read and write workload sustained a response time of less than 700 microseconds. So its not all about the IOPs number, its about sustaining a sub-millisecond response time over a huge number of IOPs. From what I've seen of your DMX numbers, you'd need at least as many DMX controller frames to match the same kind of response time and IOPs rates. Maybe even as many as 10 say.

So thats 10 entire frames to perform the same kind of workload... where is the power and cooling saving there???

OK, so factually there are a few errors above.

It was a single SVC cluster, not 7 as your informant told you. There were 12 active nodes in the cluster at the time of benchmarking, but only 10 would have been needed to provide the 1M. 12 were used to leave headroom to internally benchmark additional functions. Remeber its a single management instance, and every node in the cluster has access to all of the storage in the cluster - just as SVC does today. We did not discuss exact details of this because we do not provide support for more than 8 nodes with SVC today. This is purely a customer requirement and testing statement, there is no architectural reason why SVC cannot (and indeed has been proved to work) at beyond 32 node cluster.

With the 10 node number, it therefore means some of your numbers need adjusting. And remember this is cache miss numbers, so its 110K 4K 70/30 cache miss per node. How many can you do of this workload per DMX quadrant for example? I'm sure you have the number - and still maintain 700 microsecond response time?

The 29 controller units were needed due to only have 1 or 2 spare PCIe slots for the cards. You could build a larger system with more PCIe slots, but this is what we had available and further demonstrates the commodity appliance approach behind SVC, and quicksilver. I don't see the IOPs per controller as anything interesting. Its purely driven by the number of ioDrives we could put in each pizza box server.

The performance per ioDrive depends on the workload. I'm not going to get into marketing figures from other companies, but most SSD vendors are quoting read performance in the headline figures. And we have indeed proved the >100K iops when reading. Those numbers drop off with writes and mixed workloads. However ioDrives do not suffer the same "bathtub" mixed performance issues as seen with your EFDs. Based on your 30x the performance of enterprise HDD, you are only promising ~9K IOPs per EFD? Correct?

I think all the facts of the workload have been presented in my blog, and again here. Multithreaded, yes, per vdisk created (two per ioDrive) we were running 16 concurrent workloads. As for the number of hosts, not sure that really matters, but we used two IBM Power5 p590 systems, well one and a 1/4 really. We had each system runnig 4 LPARs, only 1 was used on the second system, and eahc LPAR had 8 4Gbit FC ports.

The error protection for this system is not representative of any resulting product that comes from this technology demonstration work. We could for example use our Vdisk Mirroring function today. I would not suggest to anybody to ever run in a RAID-0 solution, if you care about your data that is.

Again I'd sum up our work as a technology demonstration, showing that a scale-out single system image model (i.e. a cluster) is a much better investment and indeed investment protection going forward than simply plugging super fast disks into a old-school HDD orientated monolith.

PS. When can we actually buy some EFDs for our CX4 - we tried and were told that they weren't available to order yet...

I love the way people talk about EMC shipping EFDs since Janurary in DMX and today in CX4, when it was infact (May?) for DMX, and N/A yet for CX4.

marc farley

Wow, another terrific post Barry, with your usual in-depth analysis. (For transparency's sake, I work for 3PAR and we use SPC benchmarks.)

My interpretation is that you see 3 main problems:

1) "Sci-Fi" configurations that do not represent anything a customer could buy or engineer. The IBM Frankenstein is a great example.

2) Wild-ass claims that cannot be verified. The bogus TMS announcement fits this class as it has nothing at all to do with a known benchmark. Hey, take our word for it!

3) Workloads that don't represent anything resembling a customer's reality. This is what you are suggesting gets fixed with a new benchmark initiative.

I'd suggest another problem: increasing the transparency of configuration, cost, setup and tuning information.

But I don't understand why EMC has it in for the SPC so badly. Sure there are imperfections in their system, but they do provide a vendor-independent workload to measure and 3rd party verification of the results. Those are two important steps towards objective, meaningful results. My personal complaint is that the information in Appendix C where the details of the configuration are listed is tough plowing and takes more effort than many customers can afford to give it. Heck, even you didn't have time to chase it down - somebody had to dig into it an email you what they found. That's obviously not working. My guess is that the SPC would like to figure out how to make this info more transparent and easier to digest, but that is very difficult considering the multiple configuration and tuning options used by all the various vendors.

So, I understand your call for an overhaul, but I'm not sure that starting from scratch is the best way. From your perspective, what's keeping the SPC from being more effective and useful for EMC?

Also, do you really think the problem of defining a "representative" workload will ever be resolved? I have a hard time believing vendors agreeing on something like this. That's the value of having an independent entity do it - warts and all.

the storage anarchist

BarryW -

You seem to have missed both of my points about quicksilver and SPC.

First, the fact is, your little Science Experiment demonstrated nothing useful at all. The cobbled together kit is nothing you would sell, by your own admission. And your little I/O driver is representative of exactly ZERO known real-world application environment, for NOTHING actually generates 1,100,000 IO requests per second in a precise 70/30 R/W mix using standard C I/O API's across 16 instances with no intervening processing between I/O requests.

Nothing except your I/O generator, that is. Which makes the "demonstration" meaningless to anyone but you.

As to whether the SVC can replace a DMX, well, sir, I guess if your life is spent centered only on measuring and improving performance day in and day out, you inevitably would come to believe that performance is the only purchase criteria for storage.

Alas, beyond the walls of your laboratory lies a world where applications require more than just performance. Enterprise storage serves more than just IOPS, and (more importantly) the system with the fastest benchmarks is not necessarily the most efficient and effective solution in the real world.

The other point you missed entirely is that by IBM's own admission, the SPC benchmark is meaningless for measuring or comparing anything other than a single workload being run on a given configuration. Out there in real-world-land, customers don't run single workloads on most of their mid-tier nor on virtually any of their enterprise-class storage platforms. It just doesn't happen. More importantly, the SPC results have an unknown and undefinable relationship to how well a given kit will actually perform in the real world. As with your preciuos I/O driver, the SPCs do not (and can not) reflect the real-world I/O patterns of anything but the simplest of applications - despite your continued assertions to the contrary.

The bottom line is that your employer paid ESG to come up with a more real-world representative test for precisely these reasons.

Finally, despite your ad nauseum assertions of SVC superiority, customers continue to find your platform to be complex, unreliable, undercached, overly expensive, poorly fault tolerant, and insufficiently scalable. Being "fast" simply can't solve any of those issues or concerns.

Furthermore, the SVC makes for a quite silly solution when bid in a bundled configuration in front of the XIV. In fact, that very configuration got kicked out on its a** earlier this week in a rather significant competitive debate in the States, due in no small part to the numerous shortcomings pointed out here in my blog (the IBM team referenced me by name, mind you, although I'm sure it wasn't in a very polite context).

There's an old saying about lipstick and pigs, but I'll save that joust for another day.

the storage anarchist

Marc -

I think you've netted out my points pretty accurately.

As to the SPC, the problem isn't only that they bear no demonstrable relevance to real-world workloads that customers routinely run on their mid-tier and high-end storage platforms, which both IBM and ESG now admit.

It is also that the tests have been specifically and intentionally defined to perform I/Os in a manner that is truly unlike that of any application typically run on enterprise-class storage. Referential and temporal locality of LBAs have been carefully tuned to obviate the value of large global cache and to measure ONLY the raw IOPS capabilties of the back-end I/O engines and drives.

The result is a workload unlike any seen in the real world - and EMC have nearly 2 decades of I/O traces of real, live customer workloads to prove it.

More importantly, as I have pointed out to the Mad Scientist above (no offense intended, Sir Whyte), the system with the fastest back-end I/O processing isn't necessarily the best solution. Nor is it even necessarily true that the fastest cache miss back-end I/O engine will also be the fastest system in a more realistic workload - the tests prove exactly nothing more than the efficiency of the back-end in a purely artificial environment.

That they're called "transaction processing workloads" is entirely misrepresenting what OLTP workloads really look like.

As to whether or not SPC wants to improve the benchmarks to be more representative of the real world, I can't say, other than to note that I have seen nothing that even remotely suggests this is the case. To the contrary, the newer workloads are even less applicable to enterprise-class storage as evidenced by Hitachi's Mickey Mouse setup with 4 drives per back-end channel pair - how real-world is THAT configuration for a video streaming server?

NOT!!!

As to the impartiality and peer reviews, the numerous shenanegans that have been played with intentionally improper configurations and unreleased software over the past several years demonstrates that the review process is insufficient to keep everyone "honest" - it is inarguably impossible for the reviewers to understand or police all of every products' configuration and tuning parameters, much less comprehend the implications of the various settings.

Finally, changing the organization's rules on the evening of a competitive attack based on SPC benchmarks doesn't make the organization a very attractive partner candidate...

I will note, however, that there's nothing stopping SPC members from participating in the development of a better benchmark, such as Dr. Kartik has proposed. His invitation is open to anyone and everyone who wants to participate, so it doesn't have to be started from scratch.

That no SPC members have stepped forward to assist in the effort says to me that the so-called "independent" organization has something to fear from a truly real-world workload.

And I'm pretty sure I know what it is: that in a real-world benchmark, the differences between mid-tier and enterprise-class storage platforms will be demonstrated, documented and transparently available to everyone - clearly not in the interests of 99% of the membership who try to use SPC's to compete against the enterprise-class storage platforms.

open systems storage guy

Wow, you guys sure can turn out walls of text!

Okay, for the record, I believe that comparisons between systems are important, and that not every company is going to be able to build their environment twice the same way to really benchmark two options. I find myself drawn to some sort of benchmark.

You say SPC doesn't take your specialized cache algorithm into account (and in fact tries to negate the advantages it gives you). Fine. Just allow anyone to benchmark your systems and publish their results and configuration. If you believe they are misconfiguring it to deliberately disadvantage you, ask for a neutral third party to call them out and fix their config.

If they're flat out misrepresenting the results they get, then their benchmark wouldn't be repeatable.

Basically, we can re-use the best parts of how scientists come to agree with each other on things. If someone who's not biased (or is biased against you) can repeat the results you postulate, your results are good.

marc farley

Anarchist, the link at the end of your post to Dr. Kartik's blog doesn't work. No big deal, its easy to google.

My concern with pursuing this is that it could certainly have an EMC bias, seeing as Dr. Kartik is an EMC employee. How do you propose fairness in this endeavor considering that it any sane person would view it as some sort of trap?

BTW, How's Madrid? I spent several days there after Sept 11, 2001 (and SNW Europe). While the situation was stressful, Madrid wasn't the worst place to hang out waiting for airplanes.

James3678

Disclaimer: My organization is an SPC associate member and is a storage end-user, not a vendor.

Sure there are problems with the single workload of SPC-1 and how vendors can "game" the system. Our problem is there is no alternative. What other storage benchmark is as widely used, with reasonable run rules and documentation requirements?

We use SPC-1 in-house to verify and validate our own configurations. You don't see vendors running SPC-1 on a 200 SATA drive subsystem and publishing the results. We run that test because that is a configuration we use. Published benchmarks are optimized results. We want to know what we can achieve in the "real world". SPC-1 at least gives us a common point of comparison.

Sure SPC could make their benchmark better, but recognize one reason they don't is they don't have to. There is no alternative. It is what it is.

I think the real question is why someone doesn't come up with something better?

Admiring the problem is easy. Solving it is a different animal.

the storage anarchist

Marc - thanks, link fixed.

Dr. Kartik is indeed an EMC employee, but all he has done is offer to host the developmental debate for the community - sort of an open social-media approach to creating a benchmark. No hidden meetings or secret handshakes - all done 100% in the open, without restrictions. If there's a bias, it will be visible to all, and it inevitably will be quashed.

At least, that's the plan.

James - thanks for your comments. Clearly, a customer running the tests on realistic configurations isn't benchmarketing. But unless you fully understand exactly how the benchmark is constructed and (more importantly) how it relates to your own target real-world applications, I firmly assert you aren't getting any useful insights from your efforts. In fact, you may be being totally misled by the results, perhps without even reaalizing it.

That said, I'm not admiring the problem. Almost a year ago, Dr. Kartik and I called for an open source, publicly developed benchmark, developed using social media as the interaction platform. To date, noone other than Kartik and I have shown interest, and clearly we can't build the benchmark on our own without it being considered proprietary.

If you'd like to help, here's the starting link on Kartik's site: http://dotconnector.typepad.com/dotconnectors_blog/2007/11/what-should-a-r.html

James3678

Anarchist, I stumbled across dot Connectors blog almost a year ago. Unfortunately, it seems to be inactive. Too bad there hasn't been more follow-up on a great idea like a better storage benchmark.

Thanks for keeping the topic alive!

marc farley

There is a wiki on wikibon that I assume was started as a result of the discussion on Kartik's site:

http://wikibon.org/?c=wiki&m=v&title=Benchmark_storage&comments

There hasn't been much done with this wiki. Do you think this is still a good vehicle for this work?

sharbu

Disclaimer - I am a customer and a current user of SVC.

I, and all the other storage guys I have ever worked with, don't believe anything a vendor tells us anyway. Having an independent body to give us some feel that the numbers are wildly inaccurate, or reasonably close is a good thing. Is it perfect? No - not by a long shot. As you say, the fine print behind the SPC numbers is normally very dense and complex, and very few people read them. But just saying that the example IO pattern isn't what users will see so it is meaningless is crazy - you could take that argument and say that until you put the real stuff in place and run it with the live applications, or replicate the whole environment there is no real data [ which is sort of true actually, but most customers can't afford that kind of testing ].

I totally agree about the IBM propensity to try to bundle SVC in front of any spinning rust they happen to be selling - it is crazy - but my experience is that it is the marketing guys who do this, and BarryW is probably just too polite (or likes his job too much) to say much about that policy.

And - ignoring all the other SVC bashing - complex? I don't think that is a realistic accusation!

Finally - directed at most vendor bloggers - please stop rabidly slagging each other off. It is very annoying, and serves no purpose that I can see (like i said above, we think you are all liars anyway!)

Kartik

Hey guys, I'm back and ready to keep going on!

Just posted on my blog, so let continue the dialog. I am a proud EMC employee, but my grounds for opposing the SPC-1 are not based on that - they are more aligned with the reasons that Marc and Barry B. have shared.

That being said, lets work to a common end that goes beyond vendor judo. Cheers. K.

The comments to this entry are closed.

anarchy cannot be moderated

about
the storage anarchist


View Barry Burke's profile on LinkedIn Digg Facebook FriendFeed LinkedIn Ning Other... Other... Other... Pandora Technorati Twitter Typepad YouTube

disclaimer

I am unabashedly an employee of EMC, but the opinions expressed here are entirely my own. I am a blogger who works at EMC, not an EMC blogger. This is my blog, and not EMC's. Content published here is not read or approved in advance by EMC and does not necessarily reflect the views and opinions of EMC.

search & follow

search blogs by many emc employees:

search this blog only:

 posts feed
      Subscribe by Email
 
 comments feed
 

 visit the anarchist @home
 
follow me on twitter follow me on twitter

TwitterCounter for @storageanarchy

recommended reads

privacy policy

This blog uses Google Ads to serve relevant ads with posts & comments. Google may use DoubleClick cookies to collect information (not including your name, address, email address, or telephone number) about your visits to this and other websites in order to provide ads about goods and services of interest to you. If you would like more information about this practice and your options for not having this information used by Google, please visit the Google Privacy Center.

All comments and trackbacks are moderated. Courteous comments always welcomed.

Email addresses are requested for validation of comment submitters only, and will not be shared or sold.

Use OpenDNS