3.008: shame on all of us

Sometimes we in the storage industry misbehave.

Sometimes Badly.

The most recent example surrounds the reports early this week about how a bank was unable to service its ATM customers as a result of a vendors' process mistake. Apparently an operator used an out-of-date procedure to execute a routine service operation during a planned outage and the result was an protracted unplanned outage. To their credit, the vendor publicly owned up to the mistake and is certainly taking steps to avoid similar occurrences in the future.

All fine and good, if we could have just left it there.

But no, it seems this is not to be the case. Sales reps from the vendor-at-fault's competitors are gleefully emailing these reports to every customer and prospect, in hopes of creating sufficient Fear, Uncertainty and Doubt (FUD) about the competitor in the minds of these potential sources of revenue. I personally have had over a dozen emails in my inbox linking to the reports.

I find this deplorable, childish behavior.

And yes, that is directed at folks from my own company as well as those from competitors.

hey, we're all human here!

Everyone makes mistakes, and whether the fault of a single person or a process or a whole communications chain, none of us are perfect. I know at EMC we work very hard to eliminate such errors, and I'm sure that our competitors try hard as well. But whether we are more successful at avoiding mistakes or not, this does not give any of us the right to exploit human failure – for to be sure, there inevitably will be mistakes made by all of us that have similarly serious impact on a customer.

Don't get me wrong, though. If a competitors product has a deficiency that causes a problem or if the product simply isn't fit-for-purpose (e.g., Apple's iPhone 4), then I say it is fair game. But to imply that a competitors' products are lessened by a human mistake – No sir, it ain't right, and I don't like it.

Hence the title of this post. Shame on us all for this behavior, and let us not wish similar mistakes on anyone. We all work and live in glass houses.

preventative measures

That said, I would like to highlight a couple of things that EMC does to help reduce the probability of human error causing problems. Up front I'll admit these aren't necessarily foolproof, and they don't guarantee 100% success. But our internal metrics indicate that they are helping to minimize the risks and reduce the occurrences:

• Procedure Generators: perhaps the most relevant tool in the Symmetrix arsenal is the automated procedure generators that we provide to our global customer services and professional services representatives – and to our customers! Available today for Symmetrix, Invista and VPLEX, these generators will create the procedure checklist for a variety of operational, management and repair tasks for the various audiences. To ensure that the latest procedures are used, the Generators automatically check in to EMC's centralized servers every time they are run (these run on the users' laptop or PC).
• Uptime Bulletin: The Uptime Bulletin is a customer-facing periodical that documents EMC's experiences related to system and data availability. In it, customers find not only statistics, but also recommendations on how best to configure, deploy and maintain their arrays to maximize their availability. Originally pioneered by CLARiiON, it is now also published regularly for Symmetrix, Invista, VPLEX and Celerra.
• Regulator: Mentioned in an earlier post, the Regulator is capable of automatically stopping the execution of CLI commands that have been identified as problematic or that are operating in an improper order. The database of risk events is updated regularly, and the Regulator checks in via the EMC Secure Remote Service (ESRS) IP gateway used for remote support of virtually all of EMC's storage platforms.

As I've noted, customers, partners and EMC representatives all have access to these tools, and they are key components of the best practices for operating and maintaining EMC's products. I have no idea whether competitors offer any or all of these utilities, mainly because we don't typically employ these as competitive differentiators. They exist because customers have asked us to provide them, and they are used because they have positive impact.

My advice to those that are wont to throw stones – remember, we are all humans, we all make mistakes, and when those mistakes have an impact on customers, it is not a good thing for any of us…there is no joy, no victory, no competitive advantage in another vendors' mishaps.

What is most important is what we do to avoid them happening again…

 

technorati tags: EMC,availability,VMAX,VPLEX,Procedure Generator,Uptime Bulletin,Regulator,competitive advantage

Comments

[Niketown588]

Barry,

A powerful post. This needed to be brought to light in this fashion. Your message of how we in the industry need to act as grown men and women is spot on. Professionalism is the standard.
/r
Thomas

[Alex]

Barry, well said.

But as a counter-point to the contents of your inbox, mine contained zero on this subject. NetApp's competitive mailing list contained zero too.

[greg schulz]

Nice post Barry and well said.

It’s not if technology will fail, rather, when, where, why and how not to mention what steps are taken to isolate, contain as well as prevent from recurring.

We have all heard claims from vendors or their partners or channel of how their technology has never nor can it fail. It is probably pretty safe to say that is FUD as all vendors have had some failure or issue at some point in time however the good ones a) learn from it, b) handle or address the situation, c) take steps to eliminate or minimize the chances of it occurring again, d) address hardware, software as well as service not to mention educating internal, external personal along with customers on best practices.

However, sure as the sun usually rises in the east and sets in the west, FUD will continue. With technology as long as humans are involved either designing, programming, building, testing, installing, configuring, servicing or using it, you will have at least one if not more single points of failure.

As to email inbox, I have not seen any of the fud reports yet, however they might be trapped in my fud detection mailbox along with a bunch of fud laden press releases.

Cheers gs