3.021: spec sfs wars
I won't be the first to observe that most benchmarks are unrelated to anything anyone would see in the real world – no matter how hard we try, it borders on impossible to build truly representative artificial workloads. And even if we could, the second order challenge with benchmarking in the IT world is that the desire to win leads us all to often to create test configurations that would never exist in the real world.
Yet it is a dominant gene of some sort that drives us carbon-based life forms to use such artificial comparisons as the basis of what we hope are informed decisions.
The latest two entries in the SPECsfs benchmark comparisons, both of which were released within the past 24 hours or so, provide an interesting comparison of performance. In this machine-vs-machine battle, we have IBM's SONAS pitted against EMC's new VNX platform, each boasting to have shattered the SPECsfs benchmark performance records.
Now, so far as I am aware, nobody runs SPECsfs as a production workload, so the direct meaning of these results is debatable. All they really tell us is which system configuration is better at running this specific benchmark – I personally would not want to extract anything more than that from the results.
But if you look closely at the results, there is perhaps another story…
First, take a look at this excerpt from the IBM SONAS SPECsfs results
IBM Corporation : IBM Scale Out Network Attached Storage, Version 1.2
SPECsfs2008_nfs.v3 = 403326 Ops/Sec (Overall Response Time = 3.23 msec)
EMC Corporation : EMC VNX VG8 Gateway/EMC VNX5700, 5 X-Blades (including 1 stdby)
SPECsfs2008_nfs.v3 = 497623 Ops/Sec (Overall Response Time = 0.96 msec)
On the surface, you see that IBM's product did just over 400,000 SPECsfs ops/second, while the EMC product did just under 500,000. So, you'd probably assume that the EMC product is about 25% faster than IBM's
But look a little closer, and focus on that dotted line. That dotted line is the average response time of all observations. This is an important metric of SPEC sfs – so important, that the results actually call out this average response time in the header of the results.
- IBM SONAS average response time: 3.23 milliseconds
- EMC VNX average response time: 0.96 milliseconds
Folks, that says that the EMC solution is more than 3 times faster than IBM's.
That's pretty amazing, but not totally unexpected, because the EMC config used almost 100% solid state drives which can deliver a pretty consistent response time of less than 1 millisecond (as little as 0.4 milliseconds in fact, as we see above), while IBM used 15K rpm hard disk drives which have an average response time of around 6msec under nominal workloads.
Apples vs. oranges, the naysayers will say, and I will not argue with that – if we wanted to have a fair comparison, we should ask IBM to rerun their SONAS product using all SSDs instead of the 1,475 HDDs they used.
But…we don't necessarily have to wait for them to do that (I doubt they ever will, actually).
SPECsfs is a somewhat cache-friendly workload – that is, up to a certain point, most of the I/O's will be cache-hit I/Os, especially for a cache-rich product like IBM's SONAS which was configured with over 1.4 TERABYTES of DRAM cache, all told. Looking at the results chart, we can guess that most of the benchmark I/O's were being serviced out of cache, at least up until the next to the last observation.
This is probably a safe guess, because we know that a 15K rpm disk drive's average response time under nominal load is about 6msec. Since all but the last observation was LESS than 6msec, a decreasing majority of those I/Os were likely cache hits. And as soon as the benchmark working set exceeded the cache size, however, the workload swung over to being more predominately cache-miss, and thus the I/Os presented to the HDDs was beyond "nominal", and the measured response time almost doubled.
different architectures, different approaches
Now, the VNX is not a large-cache storage platform – so comparing heads up to the cache-rich SONAS product isn't necessarily apples-to-apples either (did I forget to mention that DRAM is expensive?).
VNX does, however, natively support solid state flash drives, and SSDs support a much higher "nominal" workload – that's why the VNX configuration used only 436 SSDs vs. IBM's 1,475 HDDs. As a result, the benchmark working set doesn't have to fit into cache to deliver great response times on the VNX, and we see by the gently rising slope of the VNX results that the response time elongates on a much more gentle curve.
It's not apples to apples, but where IBM effectively throws cache at the problem, VNX uses flash. It's a tale of two architectures, each using its differentiated capabilities to deliver the best possible results.
What else would you expect?
I ask you to consider this: look back at how many IOPS each platform was able to support when delivering 3.3msec response time:
- IBM SONAS: 234,043 SPECsfs ops/sec (halfway between the 3.1 & 3.5ms observations)
- EMC VNX: 497,623 SPECsfs ops/sec
Remembering (as you must) that this is an artificial benchmark, run on entirely different hardware configurations with markedly different costs, and that the results bear no definitive corollary to any known real-world workload…can you answer this question:
Which product is faster?