1.024: something you should know (about xiv)

Here in the States there is a radio show hosted by Mike Carruthers called "Something You Should Know." Each weekday on the show, which is broadcast by 150 radio stations across the most of mainland US, Mike interviews interesting people who have information about something you should know.

I think he needs to interview an IBM "Top Gun" customer service engineer soon.

Why? Because they know something that YOU should know -  something that IBM marketing, bloggers and your IBM XIV salesperson probably have been neglecting to discuss honestly with you.

It's the answer to this simple question:

"What happens if a second drive fails before my XIV array has completed rebuilding the first failed drive?"

Now, I'm sure that you've been told that the XIV can rebuild the data on a failed 1TB SATA drive in something like 20-30 minutes. And you probably understand that this makes for a very short window of opportunity that a second drive might fail on its own.

But the probability isn't zero - it can't be. Especially when you factor in human error (wrong drive pulled) or adjacent failures (a node dies). But let's not argue the math - let's just explore the results of such a double failure for the moment, irrespective of the probability.

And lest I be accused of spreading FUD, I won't even tell you the answer (until after the break).

Go call your IBM customer service engineer and ask him or her the question (I'd suggest not asking your sales rep - at least, not if you want an honest and complete answer). Note that you may have to ask specifically to speak with a "Top Gun" storage CSE - not all of IBM's service engineers have been trained on XIV service yet. But the Top Guns have.

You might want to be seated for the answer...the chairs are still available.

 

the problem is this

One of the so-called "innovative" features of the XIV array is that it doesn't use traditional approaches to RAID, where individual RAID Groups are created from 2 or more drives with parity/recovery data shared across them. RAID 1 uses a 1+1 pair of drives, RAID 5 uses N+1 drives, and RAID 6 uses N+2. These RAID groups form an interdependent set of drives, and do long as you don't lose more than the +1 or +2 drives, your data is safely protected. But with RAID 5 (for example), if a second drive fails before the first failed drive is entirely rebuilt, you'll lose data.

XIV's "RAID-X" is different - in fact, it really isn't RAID at all. Instead of using a small RAID group, RAID-X actually makes two copies of each 1MB chunk of data on different drives (and nodes). These two chunks can be placed on virtually any one of the 180 drives in the one-size-fits-all XIV array. Actually, if one of the chunks is stored on Module 1, then its copy has to be stored on a different module entirely (to protect against module failure), so there are really only 160 or so drives that the other chunk can go on.

No matter that, though, because ultimately every drive contains chunks that are mated to 160+ other drives, and thus the interdependent set of drives becomes the entire 180-drive configuration. And when a drive fails (not IF, but WHEN), ALL of the remaining drives will get involved in recovering the lost data, each identifying the matching chunks it contains for the now-missing drive and independently creating a second copy on some OTHER drive/node. The theory is that this allows for up to 1TB of lost data to be recovered in less than 30 minutes or so.

So far so good.

But....before I get to the answer to the question, there's something else about XIV you should know:

You can't segment data on a subset of the drives in the array.

That's right - every LUN you create gets spread across ALL of the drives in the array. And you can't stop that from happening! Even though you can create "domains" with capacity limits, these are entirely virtual constructs - they allow you only to control the amount of storage a group of LUNs consumes, but not where the capacity is allocated.

have you figured out the answer on your own?

Let's simplify:

• All LUNs are spread across all of the drives in an XIV array, so some part of every LUN is (probably) on every single drive in the array.
• When a drive is lost, the remaining drives have to regenerate mirror copies of any data that was on the failed drive
• The probability of a second drive failure during the rebuild of the first is non-zero

So the Top Gun answer to the question:

"What happens if a second drive fails before my XIV array has completed rebuilding the first failed drive?"

is inarguably:

You lose ALL of the LUNs on that XIV array!

Losing a second drive means that there are some number of chunks on that drive whose only other copy was on the first failed drive. Lose two, and you WILL lose data.

In fact, the fuller you run your XIV, the more data you are going to lose - it will both take longer to rebuild the first failure and there will be more blocks on the second that were dependent upon the first.

And while yes, indeed, there should be SOME LUNs that were fully recovered, the unfortunate truth is apparently that the XIV array has no way to know which LUNs were fully recovered, and which ones are missing some (unknown) number of 1MB chunks.

I know this is going to be hard to believe, so - please - don't believe me. Ask your XIV or Top Gun customer service engineer for yourself.

(Or, if you're going to Montpelier next week, ask IBM there - I'm sure they'd love to entertain the question from the floor).

why traditional raid is better than raid-x

Yes, indeed, it's true that RAID rebuilds will take longer than XIV RAID-X. But the dependent set of drives is smaller - out of 180 drives made up of 3+1 RAID 5 sets, a second drive failure is not very likely to occur within the 3 remaining drives of the impacted RAID group - even within the longer rebuild time.

But setting probabilities aside again, the impact of a second failure is far, far smaller - at worst, you'll lose data ONLY on those LUNs that were on that RAID group - not every LUN in the system. And with both Symmetrix and CLARiiON, you'll probably only lose a subset of those, thanks to the intelligence of the rebuild process that seeks to rebuild all of one LUN or Hyper stripe on the RAID group before starting the next. So with a smart RAID platform, you're probably only going to have to recover a few LUNs from backup or the remote replica.

Not so with XIV...since you can't tell which LUNs are impacted, you'll have no choice but to wipe the entire array clean and recover ALL of your lost data from backup (or the remote replica). And of course, the more data you had on the array, the longer the recovery process. (You might want to ask your IBM customer service engineer about that too - it could take a long time to recover 60 or 80 TB of data).

and That's something you should know!

I actually raised this issue months ago when IBM first acquired XIV, in my obligatory "ibm buys xiv" post and in of blind men and an elephant. TonyP tried to discredit my assertions in his (distasteful an irrespective) EMC Electrocutes the Elephant, Spreading out the Rereplication Process, More Questions about IBM XIV Nextra Architecture, and later in Cleaning up the Circus Gold.

Turns out I was right all along, and TonyP was spewing misinformation. (It's OK Tony, I know you weren't intentionally misleading your audience - you just didn't know any better).

It's a fact:

With RAID-X, some XIV arrays sooner or later will suffer a double-drive failure, and the owner will lose ALL of their data as a result.

But you don't have to take my word for it. Please - go ask your IBM Top Gun customer service engineer for yourself.

Or ask the IBM storage execs that will be in France next week.

 

technorati tags: IBM, XIV, RAID-X, data integrity, drive rebuild, data loss, probability of data loss

Comments

[Open Systems Storage Guy]

Trust me when I say that the first massive data loss in XIV will be because of a catastrophic firmware failure, not a long shot hardware failure. As an IBMer recently told me: all hardware eventually breaks, all software eventually works".

People irrationally protect themselves against statistically less likely hardware events (like a double drive failure) more than they do against frighteningly common software/firmware problems that are often far more disruptive (like that little VMWare issue we had the other day). It's a fact of life I've struggled to understand since as long as I've worked in this industry. I assume it's the same as what the venerable Bruce Schneier calls "security theater"- peoples' sense of risk rarely reflects reality.

[Storagezilla]

So to survive a double disk failure I need two copies of every chunk.

It's like it started out as an object based system and was then repurposed for block storage.

[Scott Waterhouse]

So a bit of quick math:

- If a hypothetical 4+1 RAID 5 group with 1 TB drives takes 10 hours to rebuild, lets say the risk of double disk failure is = 4 TB x 10 hours = 40 TB/hours of risk.

- If a hypothetical 180 drive RAID X group takes 30 minutes to rebuild, then the risk of double disk failure is 80 TB x .5 hours = 40 TB/hours of risk.

So it is really no better than RAID-5?

What I wonder is this: when it scales, will it spread 1 MB chunks across multiple nodes? In other words, would a single drive failure expose 360 or 720 drives? Meaning that risk scales linearly with capacity?

[Martin G]

Okay, so what happens if I have a double disk on a Sym with a RAID-5 protected Virtual Provisioned pool (when you allow me to do RAID-5 and if I was rash enough to do it?)?

Or, perhaps a triple disk failure on a RAID-6 protected Virtual provisioned pool?

These are statistically unlikely events but they could happen. And as far as I understand it from the Sym experts I have spoken to, these could be fairly catastrophic and I could end in a fairly substantial data recovery scenario.

It won't stop me deploying the technology however! Hopefully, I've been sensible enough to consider my RTO/RPOs and have worked out the appropriate place to utilise it.

[the storage anarchist]

Scott - the math is actually much worse...RAID-X is something like 7 times more likely to experience data loss than is RAID 5 3+1. Perhaps I'll publish the whole spreadsheet - but the point wasn't really the probability, but the impact when it (inevitably) occurs.

[the storage anarchist]

Martin - the difference between a Symm and Virtual Provisioning and an XIV is that with the Symm you get to CHOOSE if you're willing to expose yourself to the risk.

With XIV you don't get any choices.

One-size-fits-all, everything in the same giant pool, no tiered performance, everyone shares the same risk, exactly one and only one RPO/RTO choice...truly "one tier storage", just as advertised.

And considering the XIV is designed for ultimate simplicity of storage allocation with a minimum of expertise required, it is quite likely that nobody will understand the risks until all of the data is lost.

[stewey]

Anarchist, you're wrong! It's no worse then a raid 5 (of any emc supported config) on a CLARiiON or DMX. In fact, your internal marketing fluff comparing xiv and nextra even says so!!

Also, I'd like to bring up the deja vu of this. Does this not remind you of the days when NetApp was finding holes in the hash algorithm used in the Centera? Just for the fun of it, what are the chances of hitting a duplicate hash and a double drive failure in a Nextra? What's worse?

Also, let's not forget that a Nextra has built-in replication software. If you're so paranoid about a statistcly rediculous statement, then you could just replcate the data. Buying two Nextra's is still cheaper then one CLARiiON.

[stewey]

I have one more scenario. What's the statistical chance that an EMC CE pulls out the wrong drive (in a mirrored or RAID 5 set) and crashes a RAID group. What are the chances that all my large servers are all touching that RAID group because they all have TB's of data assigned and I lose all my data? Granted, he can also just plug it back in, but boy does that make for a REALLY bad day.

By the way, that's never going to happen on an XIV, because the drive will be in a full redundant state (< 30 mins) before the IBM CE even gets on site. And, even if he pulls a drive while the XIV is doing a 'sparing' operation, he can just plug the thing back in.

By the way, how long does it take a Symm or CLARiiON to spare a 1TB drive? How much does it cost to license SRDF-S and/or MirrorView-S to protect against those types of failures? IF if want to migrate off of an XIV and onto a CLARiiON, how much does SAN Copy cost?

[the storage anarchist]

Stewey -

Geez, you sure are riled up. Seems I hit a nerve, huh?

But you're missing the point - I purposely haven't gotten into the whole probability of occurrence thing here. The objective of this post is to expose what some may consider a serious issue with the current XIV RAID-X implementation - that if (or when) the system experiences a double-drive failure during a rebuild, ALL OF THE DATA STORED ON THE ARRAY IS LOST OR DAMAGED BEYOND REPAIR.

The fact is, on a more traditional array that uses RAID Groups, only the unrecovered LUNs that dependent upon that RAID group are damaged.

AND, if that concerns you, you can use RAID6 to further reduce the probability...this is probably the reason why IBM is introducing RAID 6 simultaneously with supporting 1TB SATA drives on the DS8000, by the way.

Now, the probability math makes this issue worse as compared to truly mirrored or RAID groups, but the only important probability is that the probability of a double-drive failure in an XIV array IS NOT ZERO. It will happen, and nobody can prove it won't.

You throw a lot of stuff in response...if you would like to present your reasoned case, along with the references and price lists you are citing, please do.

[william Bishop]

Barry, you do realize that you're giving IBM more marketing than they themselves are distributing....don't you?

[Barry Whyte]

Just a quick counterpoint here.

Whats the most common reason for a double drive failure?

The extra work suddenly requested from the system when a rebuild starts. Read xN and write for several hours...

So RAID-6 helps, but you are still hitting those drives and should you have 2 drives fail, its likely a 3rd fails too.. (Same age, same lifestyle etc)

Spreading the work over 180 drives means minimal work for each one, over a much smaller period of time, therefore mitigating the chance of a 2nd drive failure.

You can't open up this kind of discussion without looking at the probability/statistical side.

[the storage anarchist]

BarryW -

Figured it would be you who would open this angle of the discussion. I imagine you have lots of experience with the SVC waiting for drive rebuilds by various 3rd party arrays.

And it's true - many poorly optimized systems (including older generations of EMC's arrays mind you) create a very stressful environment during rebuilds.

Interstingly, though, one could actually make the counter argument - that it's the XIV drives that have to work harder on a rebuild.

Clearly, rebuild is an added load on ANY array, and every drive involved will have to work harder for at least some period of time. Thus, your "same age, same lifestyle" argument is just as applicable to XIV as to a 3+1 RAID set...except that there are more drives being worked harder on the XIV, increasing the probability of failure.

On "RAID-set" oriented systems, rebuilds often start with copying all the "good" data off of the failing drive onto the spare - possible because pre-emptive logic can determine a drive is failing before it actually ceases to function. This pre-fail copy is a purely sequential operation that places little additional strain on the failing drive while not effecting the other members of the RAID set. I haven't been able to find any reference that the XIV does something similar, so it is likely that a "full" XIV array will actually have to rebuild more data for a single drive failure than will a well optimized RAID-set array.

Finally, the XIV rebuild operations are at a 1MB chunk level, but these chunks are spread randomly across all the remaining drives. This makes the rebuild operation significantly more "random" than normal operation on the XIV, again increasing the workload on each drive. On a RAID-set array, especially those with significant amounts of cache, the cache and pro-active drive command reordering can further smooth out the I/O pattern to the disks involved to minimize seek distances (which is what is hardest on a disk drive).

No we're talking lots of hypotheticals here, and not all arrays are well optimized for rebuilds, so it's hard to model the probabilities of the different approaches to perfection.

But so far nobody has argued my point:

* double drive failures WILL happen, even on an XIV
* when they DO occur on an XIV, the ENTIRE content of the array are lost and the only recourse is to recover EVERYTHING from a backup.

Probably not the kind of marketing that IBM or XIV folks would prefer, but it's FACT, not FUD.

[Chas Hockenbarger]

I find it interesting that the argument here is not that the XIV won't lose the data, but over math that hasn't been done.

No one denies the fact that there is now an array being touted as Enterprise class that has no real ability to limit data loss in the case of a very minor hardware failure.

Very interesting.

[RZ]

Could be a lot worse. What happens when one or two data modules fail at the same time. These contain 16 disks driven by a single commercial grade controller.

[Steven Schwartz]

Great comments, the Anarchist makes some very valid points. Some things I disagree with in the comments to this post revolve around "drive failures". Most drive failures when sent back to HD manufacturers were in fact recoverable. Pre-copying data to a Hot Spare is becoming much more popular, however, many vendors are a long way off from implementing this. One of the reasons for second drive failure in a storage system has to do with drive hardware failure (not bit parity error, or soft errors) where the original failed drive causes vibration in the system and disrupts the drives next to it.

I think XIV has an interesting architecture, and I know a few people who have implemented it for specific applications. I think the idea of storage pooling, and thin stripe/high spindle count provisioning will continue to grow in this industry, but protecting from hardware failure is something that needs to continue to be a TOP priority.

I can remember the BladeStor system from STK, where they connected in serial 5x 250GB ATA drives per "disk blade" behind the LSI controller. Each "blade" was seen by the controller as a single drive. Rebuilding 1.25TB drives in a RAID failure would take days on an active system. Those were tenuous days waiting for a Hot Spare to rebuild!

[Gary Watson]

Hi Barry,
Doesn't the EMC Centera distribute data the same way that xiv does?

[the storage anarchist]

Gary -

Yes, indeed, Centera has been using a similar "mirrored chunk" strategy since its introduction, albeit with smaller chunks (256KB I think).

But Centera quickly found that customers were unwilling to purchase 2x the raw storage than they needed, even for a compliance platform. So back in 2003-2004 they implemented a RAID-5-like scheme, where each N chunks are protected by an additional parity chunk.

Another way Centera differs from XIV is that each of the N+1 chunks are distributed across a subset of the back-end nodes - and not across the entire system. This helps availability by reducing size of the dependent set of drives to say 16 or 32 drives (across 4 or 8 nodes perhaps), the probability of a second drive failure during rebuild is significantly reduced (as compared to XIV's dependent set of 162 drives).

Hmm...I wonder - perhaps there are some Centera patents that pre-date XIV's very existence...

[Han Solo]

WAIT!!

Doesn't the HP EVA pretty much have the exact same design as the XIV? Where there are no traditional raid groups, the data is all spread out across all the drives in the array?

HP has been doing these designs since the old HP SCSI Autoraid's in the early 90's and they have all been successful products which the marketplace has generally considered pretty reliable at this point.

It seems like maybe XIV should get a little bit of the benefit due to the large number of installations and long successful track record of the HP products?

[the storage anarchist]

I'm no HP EVA expert, but my understanding is that there are significant differences between the EVA and the XIV approaches, and not just the maturity of the implementations.

I'm sure there are some HP lurkers here who can elaborate on the differences in far more detail than I.

But from what I think I know, EVA isn't limited to mirroring data - it does support parity RAID protection. Also, the EVA supports multiple independent drive groups, whereas XIV places all the drives in the array into a single large pool. The larger the interdependent set of drives, the more frequent the failures and the broader the impact.

These not-so-subtle differences will result in both different probabilities of dual-drive failure and in different scope of damage caused by such failures.

I never meant to say that the idea of wide striping was inherently flawed - more that the XIV implementation seems to have some very large gaps.

[Martin G]

I'm no EVA expert either yes you are right, EVA does support multiple disk groups and yes it also supports other parity RAID protection.

I'm no statistician either but surely you will see no more failures but the impact of those failures could be worse in the XIV implementation.

However in almost any array, due to the way that people often lay disks out, presenting LUNs from different raid-sets to a server's volume group (often due to performance reasons), a multiple failure on a single RAID set can take out a huge amount of services.

And if you are running automated optimisation software and I'm not just talking 'Spindle Randomizers'; you might not know which LUNs are being presented from which raid-sets and your potential exposures might come as a bit of a shock.

Blimey, I've just scared myself! I think I'm going to go to redundant arrays and have all the disk in them RAID-6 protected. (Did consider mirroring them, decided that might be overkill tho')

Actually, I know a number of companies who have their redundant data-centers within synchronous mirroring distance who are using host based mirroring instead of array based mirroring for two reasons...the licensing costs of array based mirroring are astronomical but it also gives them precisely this; protection against double disk failures in an array raid-set.

[- jl -]

I would ask those who manage or support large storage arrays for an HONEST showing of hands to the following question: how many of you have seen 2 drives in an array fail independantly of each other in a relatively short timeframe? Of those honest souls with hands raised upward, how many of you lost data?

The breeze caused by the enthusiastic raising and sudden dropping of arms and hands sure felt nice on this warm on muggy morning in our Nations Capital...

Barry is absolutely right and noone with any credibility can refute the basic premise of the discussion -- a double drive failure renders the XIV array to the role of being a target of a FULL restore of the hundreds of terabytes of data it used to store. It's a shame because of those hundres of Terabytes lost only 2TB had to go belly up. I don't care how fast a 1TB drive rebuilds - there are 2 indisputable truths here. One Barry has disclosed - that no matter how Pascalian your math may be, you cannot put a zero on the right side of the probability equation. The second truth - of which I am certain and offer here for those not thinking ahead - is that WHEN this happens (how's the rotator cuff feeling from that rapid up and down move with your arm?) - your servers are DOWN. If you have deployed this marvel of technology in a true enterprise environment, major functions within the business are also down. Offline. Caput! No orders, no shipments, no email, no invoices. No joke. There are some enterprise applications whereby 1 minute of downtime affects the company's stock price or network, 10 mins of downtime affects a Nation's economy or communications infrastructure, and 11 or more minutes affects the global markets. Still think a 30 minute rebuild window is safe? Then the next part is for you.

Let's consider the operational pragmatics of this scenario. WHEN this happens, tape restore may be the only way to repopulate the array. By the same show of hands, how many of you backup data on an array basis and not by servers and filesystems throughout much of the 24 hours in a day? Are those crickets I hear? It sure got quiet and the air became still. A bead of sweat is trickling down my forehead as I've managed to scare myself! So our critical servers are down and our fate is in the hands of our last FULL tape backup ... and subsequent incrementals... For each and every server on the array. It is not uncommon to see 400 servers on an array of this capacity in the enterprise data center.

This process, if we are being completely honest, is likely going to take DAYS. What's more, this process has not likely been tested. Remember, we are being honest. Even if there is a tape based recovery site, test windows are typically too short to try a FULL restore for an entire array. More often a select few application-based restores are practiced at that site. What's the initial success rate on those firedrills? Do these emergency restore tests typically work the first time? Honesty, remember. Only the reader that has experienced a grueling weekend of a "DR test" knows the answer within himself. More sweat.

So, go ahead and argue whatever points you wish to make - they are irrelevant to the TRUTH about this array. This isn't marketting FUD or doomsday blogging - I've been in the field or engineering supporting enterprise customers for 24 years at a server, a database, and now an information management company. I'd say Barry is just trying to help you from making a grave mistake and being wowed by "I've Bought Moshe"'s latest "enterprise" array.

So go get some ice for that rotator cuff. And if you haven't practiced restoring an entire array lately, you may want to polish up that resume - I hear IBM is hiring.

- jl -

[williamwbishop]

Okay. New info. Off the books of course(NDA for fellow admins), but I got confirmation from an IBM guy yesterday for something I was told two weeks ago from a friend who ended up buying one of these after the demo period. My buddy tells me that he pulled multiple drives out without error during testing. It mirrors(raid-like 1), but does not mirror the same data to duplicate disks so if one disk fails, another disk failing will not have the same data on it, and unless you're at 100% full(I'm guessing, it's probable that there is still buffer space to compensate for even that), you do not risk your data if more than one disk fails, even during rebuild. So, it seems that risk is now removed. The only thing remaining is the power consumption(I also got confirmation that it does not idle disks EVER)....but that's not really a big deal to most of us, because to get FC performance from sata disks(imagine 80TB of FC and how much it would draw)you'd expect to draw more juice anyway.

[williamwbishop]

As to whether the patents pre-date XIV, they do. But I'm pretty certain EMC does not own those patents. Wild guess who does?

[the storage anarchist]

William - I've posted your comment, even though I know for a fact that your "friend's" experiment doesn't disprove my assertion. There is no way that data isn't lost - and more importantly, no way to verify that it WASN'T lost, in the experiment you describe.

There are always two copies of every 1MB chunk; the second copy is NEVER written to the same storage drawer as the first, and they are indeed randomized. XIV likes to pull this Barnum and Bailey stunt for customers - the "slight of hand" is that they always pull multiple drives from the same storage drawer. OR, they do the stunt with an array that has barely any data on it, and pull the drives with just enough time between drive pulls to allow the first pulled drive's content to be recovered.

Once you understand the trick, and the implications, you'll realize you can only win 3-card-monty if the dealer WANTS you to win. Dual drive failure in a reasonably full XIV array in two separate storage drawers, and you WILL lose data - like playing Russian Roullette with a 6-shot revolver and 6 bullets.

And nobody will be able to tell you WHICH data you lost.

[william bishop]

Ah, but then we're moving the goal posts aren't we?

You and others very strongly advertised that if you lose two drives...that's the show.

But to allay your fears, IBM didn't do it, he did. I'm unaware that he had IBM tell him how to do it(I doubt it, he's more than a bit crusty)....He had it populated to about 70% and pulled drives from all over the place because I had pointed him to this very site.

He was satisified with the results, and since I've known him a long time and trust his abilities, I by extension trust the reliability of the box. If he said he pulled a "bunch" of drives....You can bet he had a cart to hold all the ones he pulled. Maybe he was exponentially luckier than the rest of the world. But even as you described it prior, with your new data(Now it has to be two drives in separate drawers in a "reasonably full" array(I need the definition of reasonable, he had about 50T on his) to determine where my fear threshold is. Even on my symm, which I trust almost with my life, there are circumstances I could lose data...

[the storage anarchist]

William -

No, I haven't moved the goalposts - the point all along has been that indeed there are dual drive failures that will cause data loss, and that the XIV can't tell you which data was lost WHEN it happens.

I'm glad your freind is satisfied - I gain nothing by helping him understand how he has been lulled into a false sense of security. The math is inarguable: the architecture HAS to lose data - there are only 2 copies of data, they each are on two separate disks, meaning almost every disk contains SOMETHING that was only on the first failed drive. And my sources for my assertion are infallible - let's just say they get paid to know about things like this.

Question: how did your friend verify that no data was lost while all the drives were pulled? The fact the array kept running is meaningless - did he run a data integrity check across all of the "70% full" array? That probably took a day or two to verify the 60-some terabytes were 100% intact - did he leave all the drives out for the entire duration of the check?

We may not know HOW the trick was done, but we do know for a fact that it was a trick.

[storagedude]

Has anyone bought an XIV? What's been your experience? Has anyone used it for Oracle or any other large database?

[yfeefy]

With the EVA, all drives in a disk group (a disk group could be the whole array, depending on how you set it up) are divided into RSSes (redundant storage sets), probably precisely to avoid this sort of scenario. All the redundant data to recover a failed drive resides within that RSS (generally and optimally 8 drives, but can be from 6-11 I think)... So if you lose another drive, it has to be within the same RSS to cause data loss for vraid5 LUNs, and has to be the 'married pair' of the failed drive to lose vraid1 data.

[billy bathgates]

Oh yeah, I think the XIV remembers what blocks have been actually written with data, so merely 'allocating' 70% of the array without populating it with data, is still for the purposes of a rebuild, an almost empty array. (This is obviously a good feature)