3.005: transparency as a competitive advantage

Several years ago, Symmetrix customers let us Symmetrix developers know that they wanted more transparency from us about code bugs issues. They wanted to know if anyone else had seen problems like they were seeing, and (more importantly) what the solution was. They wanted to know the extent of our testing, and (more importantly) they wanted to know whether their environments fit inside the standard test/regression scope that a release went through before GA. They wanted a way to identify things in their environments that weren't in step with the EMC Support Matrix. They wanted to know the bugs issues we fixed in each release, even if it had nothing to do with their environment. And they wanted us to protect them from issues that we knew about, even if THEY weren't aware of the issue.

And they made it quite clear they weren't going to take "no, sorry" for an answer.

And so began the virtuous cycle of transparency. For most of the last decade we have had teams focus on providing the tools and information that customers were demanding. Driven by customer feedback, we have expanded this transparency far beyond the original "baby-steps" into what today is at least a differentiator, if not a huge competitive advantage.

Just some of the things we've done:

• We changed the EMC eLab Support Matrix from a printout of hundreds of pages into an on-line database that supports ad-hoc and template inquiries;
• We provided customers the means to create templates of their environments that could automatically be used to validate against the on-line support matrix;
• We tied our test matrix into these systems so customers could see the scope of configuration testing that was applied to each release prior to GA;
• We provided customers with on-line access to all customer-reported issues, and empowered them to track progress, work-arounds and solutions to the problems as they were identified and implemented;
• We provided them with an automated interface into the issues database that can block scripts and management interfaces (GUI, SMI-S, etc.) from performing operations that are known to invoke potential issues – in real time, as they are discovered;
• And we document every potential DU/DL issue we fix in New Releases, Service Releases and Maintenance Releases – even if the issue has never been seen by a customer.

This last one is perhaps to most transparent thing we've ever done; more importantly, customers tell us that they really like what we've done. And all of this transparency is a foundational component of our overall commitment to TCE – maximizing the Total Customer Experience of our installed base.

informed decision making

Part of what makes this whole effort unique is that we actually document bugs issues that we find that could potentially cause Data Unavailability or Data Loss (DU/DL for short). That's right – we proactively inform our customers that we have found and fixed issues that have the potential to cause damage to their data – and we pro-actively notify customers of these.

Now, before everyone piles on, you should know that to date, almost every single one of these DU/DL-class issues have been found in QA and not in a customer site. And in fact, the vast majority of them would never (or perhaps even COULD never) occur outside of our QA Labs. The reason is simple: we test-to-break our products, pushing the system beyond the limits, often by disabling preventative checks within the code or injecting errors unlike any that could be seen outside of a lab environment. We do these things to find out where the outer boundaries are, and to make sure that the preventatives are appropriately configured to protect the data. We find issues, we proactively notify customers if their data is at risk, we fix the issues, and we document the fixes in the release notes for the Service Releases and Maintenance releases so that customers can see what we've done.

And although you might think this approach would scare customers away from the product, they tell us exactly the opposite. Armed with the awareness of the problems we've fixed, they are empowered to make an informed decision about if and when to schedule a code upgrade. If they have a concern about a specific issue, they can ask us for more details, and we'll happily discuss any implications to their environment with them. As a result, customers tell us that they operate under less fear of issues that might be lurking – the transparency has earned Symmetrix a certain degree of trust.

Which is the real Why behind Transparency: we want customers to trust Symmetrix and to trust the people who stand behind it. As I've noted here several times before, Symmetrix is used to store some of the most mission-critical data in the world, and if it can't be trusted, that data would soon live elsewhere.

In other words, our motivation is simple- we want the following statement always to be true:

Symmetrix is the most trusted storage platform.

transparency has a price

Unfortunately, it turns out that what is good for our customers can also be used against us. Or rather, *IS* being used against us by at least one of our competitors.

(I actually blogged about this a couple of months ago, as you may recall).

Not that I'm surprised, especially given the strength of VMAX in the market over the past year…it is increasingly clear that not only was VMAX unexpected by competition, its impact was underestimated by them as well. Customer adoption has been brisk, and the market share data I've seen probably means that competitive sales teams are scrambling to slow the success of the product in every way imaginable.

One tactic has been to attack the quality of VMAX by showing prospects a semi-official looking table of DU/DL fixes in each of the various Service and Maintenance releases that EMC has shipped for VMAX over the past year. The competitor will tell the prospect that VMAX quality is poor, that EMC has had all kinds of difficulty with the port over to Intel, and that there are all kinds of issues lurking that could cause data loss if they were to put their data on EMC's flagship platform.

It's a common tactic: when you're losing in every dimension, try spreading Fear, Uncertainty and Doubt about the competitor's products.

What these desperate purveyors of FUD always omit is that the source of these assertions about DU/DL is in fact EMC's own transparency – the counts are lifted directly from the release notes of the software releases that I just described. And by obfuscating the source of these counts, and indeed the very context under which they are made available, this competitor is trying to convince prospects that EMC is trying to keep secret that there are bugs in its code, when exactly the opposite is the case!

Sigh!

Such is the price of transparency, I guess. But our commitment to earning customer trust of our products, and to delivering the best possible Total Customer Experience to the Symmetrix installed base will not be dampened by such tactics. If anything, we are driven to provide more information to our customers, not less.

the roi of transparency

As aggravating as the competitive FUD created from our transparency may be, the return on our investment in transparency is proven by the continued market share gains that Symmetrix has earned ever since we began investing in providing our customers with the visibility they require.

And in fact, this little game of FUD has apparently cost that competitor many deals – deals that in the end have gone to VMAX. In one, the customer actually called out our transparency and the competitors misrepresentations as one of the reasons he chose VMAX.

Here, quoting from the account team's summary after winning the deal, is what they said:

After an eleven (!) month fight against for our life at , I am happy and proud to announce that our team won a contract worth $M for 's mainframe VMAX production plant.

The solution includes production VMAXs, test VMAXs, SRDF, SRDF/A, and STAR/GDDR. Since July 2009, I was told at least twice that EMC would lose our footprint due to 's improved technology & support, plus superior pricing based on larger contract volumes with that competitor.

Why did we win? For the below reasons:

• 's migration plan did not meet the customer's requirements, EMC's did. With 's plan, the customer would have been out of compliance for too long.  EMC's ability to migrate a subsystem at a time and add it to STAR on the weekend once it’s migrated is an EMC advantage.
• EMC's technology is proven in the customer's production environment and 's is not.  In addition, proposed an unproven solution. Compared to EMC's solution, 's proposal was unproven in any environment.
• EMC's local and global support team and model is superior to 's.  The customer has told me that our sense of urgency, quickness, and quality of personnel are a competitive edge.
• EMC stuck to the truth. played dirty pool by going into our release notes and creating false information that EMC had 80 customer instances of lost data with VMAXs.  They also classically oversold. The customer's words "one of the reasons you won this is the transparency and trust I have with you, [your manager], and [the head of Symmetrix engineering]." We followed a "I'd rather disappoint you now with the truth then disappoint you later with a surprise" principal. 

I can think of no more compelling excuse for us to stay the course of transparency than to earn and maintain the trust of customers like these.

thanks!

Though I want very much to chastise this competitor, I actually offer thanks and appreciation instead.

For you see,

• Transparency breeds Trust.
• Trust leads customers (and prospects) to question things that would seem to contradict that Trust.
• While Trust does not necessarily grant EMC the benefit of the doubt, Trust does insure that EMC is given the opportunity to respond.

And that opportunity to respond is all we need to underscore the differentiation and customer value of EMC's approach to quality, data integrity and transparency.

Such is the power of transparency.

And the why…

technorati tags: EMC,Symmetrix,VMAX,transparency,Issue Tracker,Symmetrix Regulator,EMC Support Matrix,FUD,trust,TCE,quality

Comments

[Ken Steinhardt]

Barry, Thanks for posting this, as curiously, I have run into the very situation that you describe three different times while meeting with customers over the last two weeks. In each case it was the same competitor that was mentioned by these customers that had provided them with the information, and it was fun taking the conversation to the truth.

[MauroAyala]

Thanks Barry, a friend of mine is doing what you call test-to-break it is awesome! Many brilliant EMC engineers trying to break the Enginuity code, disabling the preventive test, I do not know if other companies are doing that at the level that EMC does, but I am sure this is one of the important things that makes EMC a trusted company.

[polly pearson]

Nice post, Barry.

Makes me think of the research we did some time ago with customers on why they did business with EMC -- and what the EMC brand meant to them.

The #1 take away was that our customers felt EMC was a "promise keeper." The word "trust" came out again and again.

-- Polly