1.059: fully automated storage tiering (fast)

This is the sixth in a series of posts on EMC's Overtake the future launch on 14 April 2009.

Today's announcement is chock-full with exciting news.

First, there is the breakthrough Virtual Matrix Architecture, combining the best of Scale Up and Scale Out to revolutionize enterprise storage.

And then, the Symmetrix V-Max itself, integrating the proven power and functionality of the Enginuity storage OS on a new industry standard platform to deliver cost-effective flexibility and a new definition for ease-of-use in enterprise storage.

Next up? Perhaps the most exciting (and unexpected) announcement of all.:

Fully Automated Storage Tiering (FAST)

As Chief Strategy Officer for the Symmetrix Product Group, my role in today's global Virtual Launch is to describe and answer questions about FAST in one of the half-dozen or so "break out" sessions available to all participants. Given the high level of interest in the topic, I thought I'd take a few moments and discuss FAST here as well.

So, what exactly is FAST?
 

solving the challenges of tiering storage

Put simply, EMC's FAST technology is designed to automate the allocation and relocation of data across two or more storage tiers based upon the performance requirements of the applications.

The objective of tiered storage is to minimize the cost of storage by putting the right data, on the right tier, at the right time. In a practical sense, IT shops often translate this into storage tiering standards, such as:

• Performance-sensitive applications go on mirrored 15K rpm drives
• Bulk data and backups go on large SATA drives, ideally protected by RAID 6
• Anything we aren't sure of goes on 10K rpm drives, possibly using RAID 5

At a high level, this makes empirical sense, and many customers today have deployed tiered storage to reduce their overall storage spend, especially by pushing more and more data down to the SATA tier. It is generally pretty easy to identify classes of data that really don't justify the expense of more expensive spinning disk media. Backups, photographs, documents, presentations, spreadsheets, software development, etc. are just a few examples. In fact today, you'll find development and even production database applications running on SATA drives, especially in arrays (like the DMX) that can offset the relative slow performance with intelligent caching and pre-fetch.

The introduction last year of Enterprise Flash Drives for Symmetrix DMX-4 opened up a whole new frontier for tiering.

Yes, the performance advantage of EFDs over spinning rust is huge, and indeed EMC has almost single-handedly driven the cost/GB of EFDs vs. HDDs down from about 40x a year a go to around 8x today. But because flash still costs much more than HDDs, customers tend to be very surgical in what they put on these drives, aiming to get the most possible benefit from the smallest possible capacity.

Generally speaking (as I've proposed on these pages before), the best practices for using Flash as a tier (put the busiest LUNs on flash) is more than sufficient to justify the cost. And with the ability of Symmetrix and CLARiiON Virtual LUN to non-disruptively relocate a LUN to a different tier of storage, this is exactly what many customers have done.

However, as many of my readers have pointed out over the past year, it is not always that easy to capitalize on the performance of EFDs. Finding the busiest devices can be difficult, although EMC tools, white papers and the Symmetrix Management Console offer a lot of assistance.

But even when you can find the "hot" LUN, it may not be "hot" tomorrow (or next week).

EMC's FAST technology is designed to help customers get better value out of their tiered storage, both fast EFDs and huge SATA drives, as well as their 15K and 10K rpm media. And FAST will do this by automating the allocation and utilization of available resources.

Under development for several years, EMC will be delivering FAST technology across all of its storage platforms (Symmetrix, CLARiiON and Celerra) beginning in 2009 with Symmetrix.

building the foundation of fast

FAST has not happened overnight – in fact, the foundation of FAST has been developed and delivered to customers in stages over the past several years. And that foundation is the collective of features and enhancements that EMC has delivered on Symmetrix DMX over the past 5 years.

In order to automate storage tiering effectively, you first need an array architecture that is adept at handling multiple different tiers of storage within a single array. Symmetrix was the first array to support slower drives (the 183 GB 7200rpm SCSI drive back in 2001); DMX was the first with LC-FC (aka FATA) and then native SATA drive support.

And we all know which array was first to support Enterprise Flash Drives.

But being able to plug in lots of different drive types into an array isn't enough: the array also has to be able to adjust its performance to maximize the benefit of each drive type, while not degrading the performance of important applications. And this isn't as simple an you might think. In fact, you need look no further than the excuses postured by both IBM and Hitachi over the last couple of years for why they took 1-2 years longer to support Flash and SATA in their arrays than it did EMC…they had to learn what EMC had already solved, for without proper precautions, slow SATA drives can slow everything down. And Flash drives are so fast that they can effectively starve the rest of the system – if you don't know how to protect against that like Symmetrix and CLARiiON do.

The FAST foundation doesn't stop with storage tiers, either. Dynamic Cache Partitioning and Priority Controls in Symmetrix are also key enablers to the ultimate objective of automated tiering – being able to adjust cache allocation and relative priorities for different I/O streams are crucial.

And let's not forget the 18 years of cache optimizations that have gone into Symmetrix. Automation requires analysis and prediction of what data will be required "soon," and the cache management algorithms in Enginuity are second to none. In fact, Symmetrix today can maintain very high cache hit rates, even though the servers connected to an array have more collective local cache than the entire Symmetrix! Uncanny, but true!

The final foundational technology is Virtual LUNs – the ability to non-disruptively relocate a LUN (or CKD volume, for that matter) from one tier of storage to another. Symmetrix DMX3 and DMX4 today support this feature, and customers use it widely to promote and demote data to different tiers as their needs change (CLARiiON and Hitachi USP-V also support this feature; IBM DS8K does not).

Symmetrix V-Max takes Virtual LUNs to a whole new level – beyond what any current platform offers:

First, you can change RAID protection types as you relocate LUNs and volumes;

Second, you can relocate more LUNs and volumes concurrently (up to 128 times as many as a USP-V can move simultaneously)

Third, you can relocate data faster (moving data 2.8 times faster – or more – than a USP-V).

Fourth, you can relocate data with less performance impact on the system overall and on the applications being moved

and Fifth, you can relocate LUNs and volumes without disrupting replication sessions (local or remote), maintaining compliance as you rebalance your resources.

Not a bad place to start implementing automation now, is it?

fast roll out

In a very real sense, Virtual LUNs on V-Max are the first phase of FAST – you get the "automated" quick relocation without disruption, but you have to make the decision of what to move and when yourself. Still, given the simplicity of the user interface to perform Virtual LUN relocations, it will seem almost automatic.

At least, that's the feedback EMC got from the beta sites. Most were simply amazed at how easy it was to use Virtual LUNs.

Later in 2009, EMC will deliver the first FAST add-on option for V-Max customers. Essentially this will be the automation engine for Virtual LUNs, and it will monitor performance and demands on individual LUNs to make recommendations of moves that could improve performance. As with today's Symmetrix Optimizer, we expect people to first utilize FAST in this "recommendation" mode, but over time they will probably let it do its job without any intervention. The system will just move data (and change RAID types) as workloads change, in accordance with administrator defined policy.

Following the automation of full-LUN relocation, a second version of FAST will be offered; this second FAST option will bring dynamic relocation to Symmetrix Virtual Provisioning volumes.

That's right, Martin. EMC has indeed been listening:

FAST will automatically and dynamically relocate data across multiple storage tiers, at a sub-LUN granularity, based on performance and cost requirements.

EMC's FAST for Virtual Provisioning has been designed to dynamically relocate subsets of a VP LUN to higher tiers of storage for better performance, or to lower tiers to reduce storage costs and make room for other performance-sensitive bits. And indeed, Virtual Provisioning for V-Max (in Enginuity 5874) already implements the requisite infrastructure to support FAST for VP; in fact, some of the meta-data enhancements I mentioned in an earlier post today are specifically to support utilization tracking that FAST will require with VP.

All in preparation for FAST.

and so the light dawns

With today's announcement of FAST, EMC's true intent for in-the-box storage tiering and both Flash and SATA technologies should now be much clearer.

It may surprise you to know, even, that it was partially in anticipation of the upcoming FAST option that EMC named its implementation of thin provisioning Virtual Provisioning in the first place – VP was designed from the outset to support sub-LUN level storage tiering that FAST will deliver.

And though Flash Drives were initially positioned as Tier 0, the plan all along has been that EMC's FAST will allow customers to save money through the automated utilization of tiered storage.

And now, with FAST revealed, we can envision future arrays that will employ a small amount Flash capacity to store the I/O-intensive subsets of data, plus a large amount of cost-effective SATA capacity to house infrequently accessed data. Some applications might also benefit from a sort of  middle tier of large 10/15K rpm drives for the data that isn't quite idle enough for SATA, and we might imagine adding other tiers as well (FAST won't be limited to 2 or 3 tiers, by the way).

Customers will thus require fewer 15K rpm drives than today to meet their IOPS objectives by leveraging the 10-30x or better IOPS capabilities of Flash for the busiest parts of each LUN. And they will also save storage costs by being able to more effectively leverage the low-cost capacity of SATA drives for data that is rarely touched. And combined effectively through FAST, the result will be large capacity systems that require far total fewer drives than systems built with disk drives alone.

Fewer drives, less power and cooling, smaller footprint, and better overall response times owing to the power of Enterprise Flash Drives and efficiency of SATA. The automation of FAST coupled with the management simplicity of Virtual Provisioning, Symmetrix V-Max will be even easier to operate and manage, since the system itself will

put the right CHUNK of data, on the right tier, at the right time!

This is the world for which the Virtual Matrix Architecture and Symmetrix V-Max were designed.

and for the record,

I'm excited to be able to explain EMC's real plans for leveraging Flash Drives and Tiered Storage, because I think it will truly change the nature of not only solid state, but it will revolutionize storage in general.

It is because of the FAST technology development that I haven't actually argued that Flash as a Tier 0 was the only (or best) way to leverage solid state storage technology. If you look back, you'll note that I always just pointed out that EMC was alone in delivering Flash as a tier for more than a year.

And I never denied that sub-LUN allocation of data on Flash drives was a good idea – because I've always known that sub-LUN tiering is in fact a BRILLIANT idea, since EMC engineers were hard at work on exactly that. But of course, I couldn't discuss that development until EMC announced the planned capabilities publicly. My pen was tied, so to speak.

As to using Flash as cache? Well, I think I will still have to tiptoe around that topic for a while longer, until EMC starts publicly discussing the actual FAST implementations. But one could imagine that FAST could have similar benefits as a dedicated Flash cache, only at a lower cost…so stay tuned!

Given the inevitable "me too" or "nobody needs THAT!" that we will soon be hearing from the competition, I look forward to a the future that FAST will enable…

It should be interesting, don't you think?

technorati tags: EMC,FAST,Fully Automated Storage Tieringi,Enterprise Flash Drivesr,EFD,sub-lun,Symmetrix,DMX,V-Max,Virtual Matrix Architecture,Overtake the future,Dynamic Cache Partitioning,Priority Controls,Virtual LUNs,Virtual Provisioning,SATA,fine-grained tiered storage

Comments

[Fabio Rapposelli]

Nice to know that EMC has finally recognized and implemented (altough not yet shipped) a technology that Compellent delivered since 2004 :-)

[the storage anarchist]

Fabio -

Isn't it great - the market leader is integrating every practical new feature from every single-feature new market entrant.

More telling is the fact that these small start-ups aren't coming anywhere close to closing the feature/function gap with the market leaders.

Fortunately for Symmetrix customers, they now know they'll be able to get the benefits of FAST without having to risk their information to an unproven and untested startup.

Case in point, with the suggested 5-year head start, Compellent is notoriously SLOW, while EMC's approach is FAST :)

[Barry Whyte]

When you say, 'later this year' and 'second stage' - any pointer as to actual GA timeframes?

EMC likes to pre-announce things well before they are available, then play the "we were first" card... Compellent were first and you could say we are all going to be doing a "me too" - EMC included.

I assume your sub-lun chunk size will be the same as your thin provisioned chunk size - 768MB ?

[the storage anarchist]

Sorry, BarryW, but I can't be more specific about dates at this time.

And actually, EMC rarely pre-announces anything - especially given the post-Enron revenue recognition rules that American corporations must operate under these days. FAST is a carefully executed exception.

As I've been pointing out to the Clan HDS this week, being first is only important until there is a second. Then first becomes meaningless, and it becomes a matter of whose offering is best.

Now that (almost) everybody is shipping Flash drives, EMC is just moving the game along to its inevitable next step.

I'm sure there will be others, but at least we can now stop imagining how flash cold be better utilized, and start explaining it instead.

Oh, and by the way: even with their head start, I assure you that Compellent have not set a very high bar to surpass. V-Max has quite a bit more CPU power, memory and available bandwidth with which to dynamically optimize layouts, to be sure.

That's why I've taken to saying "EMC=FAST, Compellent=slow."

And the VP chucnk size is 768 KILObytes, not MEGAbytes. It's Hitachi that has Flabby Provisioning, remember?