1.005.2 hitachi hits new lows (reposted)

Notice: I withdrew this post yesterday afternoon after receiving a challenge from a commenter who insisted that I was mixing up Hitachi's disk drive results with their storage array business. After carefully reviewing Hitachi's published results, I am convinced that the revenue growth numbers I used for Hitachi's storage business are correct (and exclude HGST). Therefore, I have reinstating my post intact, with the addition of a new "Hitachi Math" section (in blue) below.

Hitachi announced their earnings this past Tuesday (May 13th), and their Storage Solutions results (among others) were particularly gloomy. Not as bad as the free-fall in plasma TV sales in the US that they experienced, maybe, but dismal nonetheless.

In what is their fiscal Q4, once high-flying Hitachi only managed to eek out storage revenues that were down 1% from a year ago and down 3% from last quarter, while both EMC and IBM (if you include Tape) actually grew revenues double digits Y/Y in the same period. It marks a notably downward trend in Hitachi's Storage Solutions revenue growth over the past couple of years, as can be seen in this chart:

And Hitachi's projections for the future was for even more revenue contraction for this quarter and next - shrinking perhaps another 5% before they expect a turn-around, they said.

more hitachi math

As I noted above, the above results were challenged yesterday with an assertion that the "decline" was due to HGST. Fact is, these numbers are taken directly from Hitachi's earnings supplement, where they report "Storage Solutions" (array hardware, software and services) separately from "Hard Disk Drives" (see the top of page 2).

You'll note that Hitachi reports by halves (wouldn't want to make it easy to figure quarterly results now, would you?) - but if you go back to last quarters' results, you can do the math to verify that their Storage Solutions revenues were up 2% Y/Y in FQ3'07 and down 1% Y/Y and 3% Q/Q in FQ4'07 - just like I said.

In researching the accuracy of the numbers used in the chart, I was also allowed to see a couple of financial analyst's reports that included additional revenue insights provided them by HDS executives. While I cannot reprint specifically what these analysts published, I must say that there is something that smells an awful lot like Hitachi Math in their reports. Perhaps they just misunderstood what the HDS execs told them, but the numbers the published in their reports simply don't add up.

More importantly, since HDS only sees revenues excluding Japan, their perspective undoubtedly skews any possible analysis - especially since it's not clear whether HP and/or Sun storage revenues are reported through HDS or if they go directly to Hitachi Ltd. (I'm pretty sure they go directly to Japan).

And if revenues really weren't shrinking, then why the heck would HDS execs be trying to spin the story with Wall Street in the first place?

Bottom line: Hitachi Headquarters reported (and documented) that FQ4'07 Storage Solutions (ex-hard disk) was down 3% Q/Q and down 1% Y/Y - just like I said.

And on top of declining revenues, word on the street is that morale in Hitachi's US field operations is at an all-time low. Nobody seems to know if morale is suffering from the recent out-sourcing of customer service, the collapsing of the former solution/consulting business with the former Hitachi Data Systems subsidiary, or the new Japanese management that are running the new US holding company now. Or maybe it's something else?

Given that the flagship USP-V is nearly a year old, it seems very odd that revenues would be shrinking at a time when the new system should be really starting to gain traction.

It all makes me wonder...
 

...what's really causing hitachi's storage solutions revenue decline?

FUD ALERT - - - FUD ALERT - - - FUD ALERT - - - FUD ALERT - - - FUD ALERT - - - FUD ALERT
It is pretty much impossible to propose theories behind Hitachi's decline without crossing over the FUD line.
For whatever it's worth, I acknowledge that this will be considered FUD, although this isn't my intent.
- You Have Been Warned -
You don't have to read this if you don't want to be subjected to my opinions and observations.

My first thought was that maybe there is actually something to those continuing rumors of product quality problems, on both the USP-V and AMS platforms. I've heard as much from several different sources, although I'll admit it's not something I've directly experienced. If true, however, customers could be voting with their feet instead of cashing in those near-worthless discount coupons they receive when Hitachi's products fail to fulfill their so-called availability guarantee.

On the other hand, I personally have spoken to several customers who have complained about poor performance of SATA drives in Hitachi arrays. The problems are especially bad for write IOPS, which suffer because the Hitachi arrays don't use DIBs to protect against silent data corruption by the drives. Instead they perform a read after every write (on the AMS at least) to verify that the intended bits were actually stored on the disk. Not only is this sssslllllooooowwww, it is also totally ineffective, as data corruption that occurs days or weeks later will never be detected or corrected.

Poor SATA performance and silent data corruption are problems that specifically do NOT exist in either Symmetrix or CLARiiON, due to the data integrity checksums that both platforms have employed since, well, forever. If at any time in the future what's read doesn't match its checksums, the data is simply rebuilt from the RAID protection and rewritten - much more efficient than read-and-compare-after-every-write.

Another thing that has become increasingly clear to a growing community of enterprise storage consumers is that Hitachi's "virtualize everything" strategy is wearing thin (no pun intended). First there is the inherent risk of undetected data corruption introduced by in-band regenerative I/O virtualization - a write can fail or be corrupted to the back-end storage after it has already been acknowledged to the host, resulting in irrecoverable data loss.

Customers are also beginning to realize that adding at least 1ms of overhead to every I/O routed through a USP-V or USP-VM literally means the difference between "tier 1" and "tier 2" (or even "tier 3") performance. And if you turn off the USP-V caching to reduce the latency and data corruption risks, you also sacrifice most of the utility of the array, including local and remote replication of external storage for all intents and purposes.

And while that doesn't necessarily mean Hitachi virtualization is useless (I know it isn't, so please don't flame me again, Nigel), it does mean that it's more appropriate for migrations into the USP-V than it is for any practical tier 1 production deployments (other than nearline repositories and archives). Since Hitachi is basically making UVM "free with every system," more customers seem be learning about the limitations on their own, effectively neutralizing Hitachi's only "silver bullet" feature.

Free isn't always a good thing, I guess.

I also suspect that the lack of reference-able customers using Hitachi virtualization in any tier 1 production environment is starting to hurt business. I do know that Hitachi has been flat-out unable to deliver when prospects insist on talking to Hitachi references with real live production tier 1 deployments similar to their own intended use cases. And I know of several EMC account teams who have won sizable deals after the references that Hitachi did provide proved irrelevant to the prospects.

Finally, I'm sure that EMC's introduction of Enterprise Flash Drives (and Hitachi's "we-don't-get-it" response) has had a negative impact on their sales. In fact, flash drives have opened a lot of doors for the DMX-4, and customer interest has been growing steadily since the announcement. We're finding more and more applications that benefit from Flash every day, and the IOPS and response time economics are causing them to be considered for applications that hadn't been imagined at the time of the launch. One interesting example: more than one customer is considering replacing their entire Exchange storage farm of short-stroked 73GB 15k rpm disk drives with something like 1/30th as many 146GB flash drives. This is not so much for the inherent response time benefits; it is really being driven by the economics of significantly reduced power, cooling and floor space requirements.

Cheaper, faster and smaller - a three-fer that is just hard to ignore!

Commercial Break: Flash drive performance and applications was one of the focus topics for last week's SPEED Guru Summit EMC held for it's pre- and post-sales technical community. I also expect that there will be a lot of flash-related customer discussions next week at EMC World -be sure to stop by my session and say hi if you're there.

But no matter the reason (or reasons), declining revenue growth can't be good news for Hitachi customers or prospects (or salespeople and shareholders, for that matter).

Fortunately, EMC is ready, willing and able to pick up the slack.

By the way, I got a chuckle from an email I received Tuesday night from one of my co-workers, in response to Hitachi's earnings report. In it he said (paraphrased):

HDS' field has lost their confidence.
Hitachi customers are losing their data.
and
HDS executives are losing their jobs.

What's not to like?

 

technorati tags: Hitahi Math,Hitachi,Hitachi Data Systems,HDS,Hitachi Storage Solutions,earnings,virtualization,reference accounts,enterprise storage,UVM,USP-V,USP-VMware,AMS,SATA storage,data corruption,DIB,customer references,tiered storage,enterprise flash drives,EFD,EMC,DMX-4,Symmetrix

Comments

[Snig]

One has to wonder why you only have EMC numbers back to Q106? Maybe because EMC was in the negatives Y/Y for the few years before that?

[the storage anarchist]

Snig -

Wonder no more:

I only plotted the numbers that have been made public - if EMC had made earlier apples-to-apples numbers publicly available, I would have plotted them, and you'd not have had to wonder.

The first sentence of this quarter's earnings release pretty much says it all, though:

"...today announced all-time record first-quarter revenue and its 19th consecutive quarter of double-digit year-over-year revenue growth."

19 consecutive quarters = double-digite revenie growth every quarter since Q2'2003. Although the company didn't break out "storage" as a separate revenue stream until Q1'2006, pretty much ALL of it's revenues prior to then were driven by the storage businesses (Symm+CLARiiON+Celerra+Centera, etc.).

[Nigel]

Hi Barry, my turn to wonder now…… can you please clarify the following statement – “…lack of reference-able customers using Hitachi virtualization in any tier 1 production environment….”.

Are you saying that HDS cannot provide any reference sites that use UVM on a USP that also runs tier 1 storage?

Or are you saying they have no reference sites where external storage is used as tier 1 storage (mapped through UVM)?

Or do you mean something totally different?

I may or may not have some other comments to make but Im way too tired now.

[the storage anarchist]

Nigel - worry not.

I've not heard of any customer or prospect that was able to get references from Hitachi, HP or Sun that were using externally virtualized storage for what said customer/prospect would have defined as "Tier 1" applications.

Now, definitions may vary, but with UVM adding 1-3 milliseconds of overhead to every single I/O, I doubt ANYONE would consider it useful for Tier 1. Maybe for SATA the overhead is relatively small, but compared to a DMX-4 that can return a read hit I/O in 250 microseconds or less, UVM is a non-starter....useful only for near-line storage, backups and maybe some Tier 3 file sharing apps...

[Gianluca]

I am intrigued by your ability to make HDS state what you like. HDS never states that you can run Tier1 applications on externally virtualized storage without any performance loss. HDS states that you can run most applications on external storage, move most Tier1 App volumes to external storage and keep inside the USP V only the most critical volumes of Tier1 apps. this leads to a solution where you can offer applications the same performance as with non virtualized storage (and actually give applications that were running on non performing storage a performance increase), but with lower costs, especially compared to EMC's notoriously costly "solutions". I can submerge you with references of very happy customers who have implemented virtualized architectures and run Tier1 aplications in the way I have outlined.I understand your salary is payed by EMC, but a bit of objectivity would be welcome. I find this attitude quite unprofessional and typical of EMC. a malicious mind would think that this is due to EMC's inability to get any kind of decent virtualization working (InVista, or rather who's seen it as it is better known), therefore resorting to the usual EMC slandering tactics. nothing new.

[the storage anarchist]

I take your point, but I guess it's all up to the users' definition of "tier 1", and whether there are references whose use cases match that definition.

As to adding performance to external storage while costing less, I to admit that I haven't seen any actual data to back that up - just the marketing claims (and HHSNBN's blog).

I get that just adding the cost of incremental ports, cache, control store and SW licenses required to connect up the external storage to the USP-V might be cheaper than buying new disks. But what about the indirect/hidden costs of extended warranty and reduced availability (more things in the I/O path to go wrong), plus the added power and cooling. When you consider the complete TCO, every analysis I've seen says buying brand new big and fat SATA drives for inside the array is significantly less expensive, more available/reliable AND faster than the externally virtualized alternative.

At the very least, I encourage everyone to run the complete TCO before jumping in - you might be surprised to find that you can get better savings by replacing rather than demoting your old storage!

[obServer]

Hitachi Data Systems doing well particularly against Symmetrix

http://www.blocksandfiles.co.uk/article/6265

HDS/HGST vs EMC Sym + Clarion?

Should be worth some more EMC FUD... Douglas

[the storage anarchist]

Through the alchemy of Hitachi Math, a 2% Y-Y rise in revenue magically becomes an astonishing 19% increase if converted from Yen to US Dollars.

Could it be that the Yen buys more dollars this year than last?

Pay no intention to the man behind the curtain...