0.010: operating at exa-scale

You've seen the IDC/EMC report on the staggering rate of information growth the world faces. At something like 56% CAGR, we collectively today manage over 161 exabytes of data storage. If you have read the press release or the complete study, you'll have noted that the bulk of this growth is in data that resides outside of the typical IT data center - digital video, photography and audio. And unless you work within the industries that are building value on or around these new storage-hungry data types, you might harbor the impression that this growth is really somebody else's problem.

And you'd be wrong.

Many Individual IT organizations around the globe are dealing with the same levels of compounded growth in their storage requirements as is the entire global information economy cited in this report. And many of these IT departments are already dealing with 10's or even 100's of petabytes of on-line storage, not to mention perhaps a couple of exabytes of off-line or near-line capacity.

I've seen a few of these environments up close, and it literally boggles the mind. Off the top of my head, some of the larger ones include:

• the cellular communications company in Japan that services real-time "debit card" type transactions from millions of customers' cell phones and the cell phone company in China that has more customers than the United States has residents;
• multiple different global financial institutions that handle the billions of concurrent stock, currency and credit transactions that make up the global economy;
• the airline reservation and scheduling systems that keep pilots, planes and passengers moving in relatively organized chaos;
• the processed foods and consumer goods companies that constantly analyze point-of-sale and shelf-life information for its products in near-real-time so that it can maintain just-in-time manufacturing and delivery in response to geographical and seasonal  demand shifts;
• the DoD and national security organizations that um...well...ahem... do whatever it is they do with all that information that they've intercepted collected over the years (and it's a lot, or so I've been told d:^).

Bottom line - there are literally thousands of IT organizations operating at a scale that most of us can't even imagine.

And whether by vendor intent or sheer customer need, EMC supports the storage requirements of (dare I say it) the majority of these largest IT organizations in the world. Dealing with the evolving (and exploding) information demands of these customers has largely defined EMC's product and services portfolio, from the early days of Symmetrix through the latest server virtualization, data-dedup and information security additions. We've learned a lot by supporting these bleeding edge consumers of storage, including one very sobering reality.

And you know what that is?

Operating at exa-scale is extra-hard.

the economic challenges of exa-scale

For many readers, I'm sure that their storage environments are defined with perhaps a few centralized "tier 1" storage arrays (from EMC, Hitachi or IBM), surrounded by perhaps dozens or even hundreds of mid-tier arrays (again from EMC, IBM, HP, NetApp and a slew of others). And no doubt these environments are large, complex and growing - in fact, collectively these IT shops represent the vast majority of the external storage market.

But the IT shops I'm thinking of here deal with much larger numbers. Many have more than 100 EMC Symmetrix arrays and 1,000's of CLARiiONs. Some have taken the path of large numbers of "small" arrays (small is relative - one customer I know has standardized on 9TB arrays, another on 42TB arrays). The data warehouse customers tend towards the largest arrays (one has multiple 1500+ drive DMX-3's). And many in the financial community tend to segment applications onto dedicated, distributed array pairs or trios for isolation, redundancy and availability.

While there is no real consistent approach to these exa-scale data centers, they share one critical characteristic with virtually every other data center of any scale: change is constant. Only in the exa-scale IT shops, the rate and impact of change is often larger than is typical in smaller shops. Some of this has to do with the industry's requirements on its membership (e.g., compliance, security, etc.). And a lot just has to do with the complexity of operating at scale - the larger the process environment, the more things can go wrong - you have to look no further than the Space Station or the Space Shuttle to know what I mean.

At the core of the exa-scale IT organization is a massive challenge: consolidation is supposed to reduce costs faster than expansion is supposed to increase them. And while virtually every IT director or CIO is seeking to cut costs (or hold them constant), the head of exa-scale is faced with the daunting task of saving literally billions of dollars (or euros). When the numbers get that big, it's rarely realistic to think of implementing cost-cutting measures in the increments of $100's or even $1,000's - it is truly hard to imagine saving a billion dollars one penny at a time.

some not-so-obvious operational challenges of exa-scale

But taking this down to a more hands-on level, imagine the impact of simple storage allocation requests on the exa-scale data center. Where a smaller shop might be dealing with perhaps only dozens of outstanding storage allocation requests, I know many of the aforementioned environments where the typical outstanding backlog of storage requests numbers literally in the hundreds and represent easily several petabytes of storage. This is not only a lot of work that will have to get done, but think about it for a minute: the shop either has to have petabytes of unused storage already installed and waiting, or it has to extend the "allocate" process to include acquisition and installation of new storage. And where a smaller shop might need only to add a dozen or two more disk drives to an existing array, many times these exa-customers are actually scheduling weekly delivery of additional storage arrays. I've seen storage allocation processes that span literally months for approvals, planning and scheduling, even when the actual "create and allocate" can be completed in literally minutes.

Several of these exa-scale customers face a related challenge - the inevitable technology refresh cycle. Again, most every IT shop rotates or retires servers, storage and communications infrastructure on some frequency - usually associated with lease rotations or capitalization/depreciation schedules. But in the exa-scale environment, such technology refresh cycles can be constant - I know of several customers who literally have at least one storage array being retired/replaced every single week of the year!

Consider drive failures and replacements. Fact is, drives do fail, and the more drives you have, the more that will fail in a given period of time - the simple realities of mechanical devices. And if you look back at the representative list of customers I provided above, you'll notice many are in industries where information security is paramount. So, what happens to all the failed drives? Stick them into a secure room may work for a while, but for the exa-scale environments, with years of failed drives from hundreds of different arrays - we could be talking major land-fill required here, if there were such a things as a secure land fill (that isn't already full of nuclear waste, that is!).

And the list goes on. How to encrypt the SRDF traffic of literally thousands of dedicated dark-fibre pipes between hundreds of data centers spread around the globe in a manageable and cost effective manner. How to back up 10s of thousands of application data every night, and then how to handle the hundreds of recovery requests every day. Inventing a naming scheme for more than a hundred thousand servers, storage, arrays, switch ports, LUNs and even FC cables. Recovering from not one runaway application, data corruption bug or environmentally-fueled data center outage a year, but more like on a monthly, weekly or even daily frequency.

These my-information-infrastructure-is-bigger-than-everyone-else's customers and their exa-scale environments demand information infrastructures that are exponentially larger and more complex than the average. They require the ultimate in availability, and can tolerate only the minutest of outages in the event of a disaster. And their staffs and budgets are rapidly shrinking, at least in contract to the CAGR of their infrastructure.Do more with less seems to be the economic battle cry these days - yet the complexities of dealing with "more" often requires more than the presumed and appropriated "less." But for these exa-scale enterprise IT organizations, failure is not an option. Not when they're the ones making the world go 'round.

And Chuck Hollis has the audacity to suggest that simply becoming an Informationist is the solution. HAH!

some economic benefits of exa-scale

But all is not necessarily doom-and-gloom. Server virtualization such as that enabled by VMware has made it possible to get more done with less hardware (unlike the misnamed storage virtualization, as I explained in an earlier blog entry). Massive consolidation of storage into a single array as enabled by the 2400-drive DMX-3 has changed the economics of not just tier 1 storage, but for tier 2 & 3 applications as well. Our recently announced (and now shipping) 1.8 petabyte virtual tape library changes the economics of backups and (more importantly) recovery. Even the 500GB low-cost Fibre Channel disk drives we've supported in Symmetrix for more than a year now help the exa-scale customer to keep costs low.

No, we didn't call the Symmetrix DMX-3 "The Consolidator" for nothing.

so what's my point?

Fact is nothing scales as large or as effectively as does Symmetrix DMX when you require enterprise-class performance and functionality. At the high-end (where my job is focused), we intentionally designed the DMX-3 to allow customers to consolidate more storage into a single array than anyone else; but that same DMX-3 is nearly as cost effective in smaller configurations, since the control logic scales along with the capacity. We have developed array-based functionality that provide unparalleled local and distance information availability with a minimum of application impact and overhead. We have simplified the allocation and prioritization of critical resources like cache and internal queues so that the array can dynamically reallocate resources in order to deliver the performance where its needed, without requiring manual intervention or costly downtime to reconfigure. And we have introduced technologies and solutions like Open Replicator, Open Migrator and PowerPath Migration Enabler (PPME) that simplify the challenge of data migration for even the largest of environments - all without the need for adding any new hardware into the data path. Dave Donatelli discussed how these tools, and others, are helping these exa-scale customers to be more productive in his recent EMC World keynote entitled Making Life Easier in Your Exabyte World (a must-see presentation that was in fact the inspiration for this blog entry).

Meanwhile, one of our competitors is focused on trying to virtualize everything behind a single controller in an attempt at dis-aggregation - almost the exact opposite of EMC's consolidated tiered storage strategy. Where EMC embeds low-cost storage into a large array to enable single-array tiered storage for reduced cost, power, heat and footprint, they propose to virtualize the multiple hardware boxes behind a USP-V. Surrounded by lots of big-sounding numbers and backed by, they'd have you believe that a single USP-V is sufficient to meet the demands of the exa-scale customers, yet so "green" as to offset the inherent incremental impact of using multiple different arrays to do the job of just one.

But the reality is that the exa-scale club's requirements are today far more than a single USP-V can accommodate, bigger even than multiple Invistas, far too complex for a hundred SVCs and yes, bigger even than a single DMX-3. Heck, the way these guys consume storage, probably even a Symmetrix-99 won't be big enough!

So , ultimately, the technology answer to the challenges of the exa-scale customer is almost by definition going to be a distributed solution. Nothing else can, or likely will ever, scale to encompass the entire information infrastructure into a single, holistic utility. Sure, maybe one day we will find a way to make these massive storage infrastructures work seamlessly together and look like the energy grid or the telephone network: with operational simplicity at the end-points masking the inherent complexity of massively distributed switching and generation plants behind layers of resiliency and automation. That's clearly the vision that the exa-scale customer is driving us towards.

But nobody can do this today, at least not at exa-scale.

For the sake of the exa-scale enterprises I outlined above, I hope we (collectively) are up to the challenge. If their information infrastructures ever collapse under the weight of growth and complexity, the resulting chaos is virtually unimaginable.

But I bet it won't be pretty.

Comments

[Chuck Hollis]

Hah!

And "hah!" again!

No, seriously, you paint a great picture of large-scale requirements that most of the industry doesn't even contemplate -- yet!

And, as we have both seen, these uber-sites wrestle with problems that -- sooner or later -- find their way into more moderate environments.

But, I think you'd agree, at some point, even the largest organizations have to have some sort of clue as to what they're storing, and why.

Loved the link to the Donatelli PDF. Lots of great stuff in there for those who missed it.

Keep up the good work, Barry!

[Nigel]

Barry

A few months ago I did some work for the MoD in the UK who obviously had to securely destroy data on failed disks etc. So all failed disks remained on site and when a few had stacked up, a contracted shredding company came on site and took them away to securely dispose of them. The thing was, this company charged based on weight and not the number of disks being destroyed. As a result the MoD had us physically open up each failed spindle and remove the relatively light-weight platter. It was only the platter that was then handed over to the shredding company, thus keeping costs low (doing more with less).

The thing that might not be immediately obvious is how hard it is to get into modern disk drives! These were Seagate Cheetah’s, mainly 73GB 15K. Anyway there were 40 something screws that needed removing before you could remove the platter. No word of a lie, 40 something screws! And the first time you did it took an age because some of the screws are hidden under stickers etc…..

It was fun for one or two drives, but were this a massive environment, the type of which you mention, this would have been a full time job for at least one person.

Oh and you said “……operational simplicity at the end-points masking the inherent complexity of massively distributed switching and generation plants behind layers of resiliency and automation”

You seem to be describing hanging a lot of arrays etc off the back of a USP ;-)

[the storage anarchist]

Hardly.

If only a single USP could virtualize an entire exa-scale enterprise. Reality is, like the phone and power networks, these enterprises are far too large for any single point of centralization. The only approach that scales AND provides near-absolute availability is a global network of distributed, interoperable, cooperative and fault-resilient nodes with no centralized controller, bottelneck, or single point of failure.

In fact, methinks USP virtualization is really the antithesis of what's required to operate at exa-scale. Forcing everything through a single monolithic intelligence while dumbing down the functionality of the subservient storage devices...IMHO this "controller-centric" approach is very reminiscent of the Borg.

Except that resistance doesn't necessarily have to be futile ;^)

Thanks!

[Nigel]

Ah, I never said anything about a single USP virtualising exa-scale environments. I was simply saying that it can provide operational simplicity masking complexity behind it. It was also said very much tongue in cheek ;-)

[the storage anarchist]

Point taken - I must have missed the smiley. All in good fun!