0.040: where's the spc disclaimer?

There's been lots of feedback and response to my post earlier this week on Hitachi's USP-V SPC-1 results. Most of it has been very insightful, including additional observations by several people that I only scratched the surface on the inconsistencies between the various tested platforms, etc. From the responses, I am encouraged that many of you are actually taking the time to more thoroughly research and understand exactly what the SPC does, and does not, actually mean.

Importantly, the SPC benchmarks are not a test of whether a platform is "enterprise" or not. Nor is it appropriate to use the results as the basis claim superiority within any such category. In fact, the SPC tests are intentionally agnostic to the "class" of storage, even while they are intentionally cache-hostile benchmarks (noting that most "enterprise" arrays do typically have lots of cache and do derive performance benefits from large cache, which the SPC tests try to factor out of the results).

So since the SPC doesn't define a criteria or measurement of "enterprise" any such claims of "superiority within class" are merely  examples of Bad Blogketing, in the same category as Hitachi Math- unsubstantiated, unverifiable, and bearing no real relationship to reality.

In fact, the SPC disclaimer, taken directly from the specifications of the SPC-1 tests, says exactly this.
 

spc disclaimer

Quoting specifically from the SPC BENCHMARK-1 (SPC-1) OFFICIAL SPECIFICATION Revision 1.10.1 - Effective 27 September 2006, page 14:

0.4 Disclaimer

While this workload models a rich multi-user environment that emulates a broad range of server applications, it neither represents the entire range of I/O requirements for server systems nor precisely mimics any particular application. In addition, the extent to which anyone is capable of achieving the results reported by a vendor is highly dependent upon how closely the customer’s application maps to the SPC-1 workload. The extrapolation of SPC-1 results to other environments is therefore not recommended. [emphasis mine - tsa]

Actual system performance is highly dependent upon specific workload characteristics, platform configuration, and application-specific tuning. Relative system performance will vary as a result of these and other factors. Thus, SPC-1 should not be used as a substitute for customer application benchmarking when critical performance requirements are called for. [emphasis mine - tsa]

SPC-1 uses terminology and metrics that are similar to other benchmarks. This similarity does not imply that results from this benchmark are comparable with other benchmarks. [emphasis mine - tsa]

The SPC-2 definition document includes a similar disclaimer.

So why is this missing from all the hype and hoopla?

Reading the original Hitachi press release, the press and analyst coverage, and the blogs of Claus and TonyP, it seems clear that these writers have misunderstood, accidentally overlooked, or intentionally ignored these disclaimers.

Although I strongly suspect the reason is the latter, I admit that it could be one of the first two reasons. By printing it here, I've solved the "didn't know" problem (for my readership, at least). And for those that haven't spent years in the midst of the "it depends" scientific method of performance profiling, please allow me to simplify this disclaimer:

The SPC-1 (nor any other benchmark, for that matter) cannot be used to predict the performance of any environment other than one identical to that tested, where the environment includes everything used in the test: the specific application and host software configurations & versions, server configurations, interconnect infrastructure, storage configurations, etc.

Comparisons across different test configurations cannot be made, because relative performance will vary due to numerous unquantified factors.

Even though the terminology is similar to other benchmarks, there is no correlation - an SPC IOP is not the same as any other IOP.

Clearly the Storage Performance Council never intended for these tests to be used as a foundation of the sorts of misleading and misrepresentative claims that Claus and TonyP are making.

And that's just Bad Blogketing, plain and simple.