Guide · RAID & NAS

RAID won't rebuild? Stop here first.

Your array had a failure of a disk, you replaced it with another disk, but your rebuild isn’t completing: it’s stalled, has some error, or dropped the entire array. This is one of the most hazardous times in the existence of a RAID, and the actions taken during this very short period of time will be greatly important. There’s an almost overwhelming urge to retry the rebuild. Most of the time, that impulse is incorrect.

Free 48-hour diagnostic
Handled in-house
Only pay if we recover your data, on most jobs
// in short

Stop rebuilding.

If your rebuild fails then your array is fragile. Attempting to run a rebuild again puts additional stress on already weak disks and could also result in further failures. Power your array off, do not attempt to force it, and instead image all of your disks.

Stop
Don’t retry the rebuild
Fragile
Disks under strain
Never
Recreate the array
Image
Then reconstruct
// why they fail

Why rebuilds fail.

In order to understand why a rebuild is so dangerous, you need to realise exactly what a rebuild does. Once you’ve placed a replacement disk into the place of a failed disk, the array reads each and every other disk in the array completely and intensively to recreate all missing data from those other disks onto your new drive. This creates a significant amount of work for disks which have often been functioning together continuously over long periods of time.

And here’s the issue: those remaining disks are often old themselves and already have weakened sectors. Then add a rebuild process to the equation, and a second disk can hit a bad sector, or even completely fail. When there isn’t any additional redundancy in place during this rebuild (especially RAID 5), that is enough for you to lose your rebuild, and therefore the array will fail. Disks purchased at the same time tend to deteriorate at the same rate, which is why one failure so often triggers another.

// the danger

The danger of trying again.

Once your rebuild has failed, you want to try again; maybe it’ll work this time. Do not do this. The reason your rebuild has failed is because your RAID system is currently in a weakened and degraded state. Each attempt you make to redo the rebuild puts just as much stress on your RAID and on the weak disks as did the first, raising your chances of another failure and moving you farther away from recovery.

Worse still is to try to “fix” this condition by re-creating or re-initialising the array, an action that some controllers may present as a way of recovering your array. This process can overwrite the configuration of the array, which in turn makes it much harder if not impossible to recover your data. The array is telling you that it cannot safely rebuild itself. Attempting to force it to do so usually does not end well.

// what to do

What to do instead.

In case of failure during an array rebuild, the safe recovery process is short and deliberate:

01

Stop and power down

Do not attempt to rebuild again. Shut the system down now to remove all stress on any working disk(s). If you don’t, you could cause them to fail as well.

02

Don’t recreate the array

Under no circumstances should you ever attempt to re-initialise, re-create or reconfigure your array in an attempt to “fix” it. In doing so you may be overwriting the exact same information you need to successfully recover your data.

03

Keep the disks and their order

Mark each disk with its slot position and retain even those that have been labelled as “failed.” They often still contain readable data that helps in your recovery efforts.

04

Get it imaged and reconstructed

Have an image made of each disk and the array created virtually from the images, in a safe environment; avoid putting stress on the original disks, which can be damaged easily.

// why professional

Why this needs a lab.

Why a rebuild failure is a job for experts is simply because rebuilding from live, degraded disks is what is causing the failure. A lab has its own way of doing things: each disk (even the ones that “failed”) gets copied to healthy storage first, using methods that read gently around bad sectors. The array is then created virtually from these images. This way all the dangerous stuff happens with the images, and the original disks are only read one time and never get rebuilt.

This is why a rebuild failure is usually recoverable if done right, but will often not be recoverable if forced; this just depends upon how well you protected your weak disks versus how far you tried to push them.

// prevention

Avoiding it next time.

There are two valuable lessons in every rebuild that fails. The first lesson is that a RAID is not a backup; it is uptime protection, and when there is a rebuild failure it is the perfect time to use a separate backup. The second lesson from this experience is that all the disks in an array get older at the same rate. Therefore, it’s worth replacing your ageing array proactively, before the first drive fails and exposes how worn the rest are, and monitoring your disks to determine their remaining life.

Redundancy gives you time to take action, but redundancy will never remove the need to back up and maintain your array.

// faq

Common questions.

These are the questions that people call us about most when they have a failed RAID rebuild.

No. A failed rebuild means the array will be unstable (fragile). If you try again, this will perform the exact same high-impact disk read on already weak disks, increasing the risk of another failure and additional data loss. Shut the array down instead. Do not attempt to recreate or reinitialise it. Keep all disks in their original positions. Have the disks imaged and then rebuilt by a professional. Repeated rebuild attempts are one of the most common ways that recoverable arrays become unrecoverable.

A rebuild causes a long-lasting, heavy read across each of the rest of your disks to replace the failed one. These disks are often old, have weak sectors, and have been running together for years. Thus, when under this stress, a second disk frequently fails or produces an unreadable sector, causing the rebuild to fail. Rebuilds are most dangerous for configurations where there is zero redundancy available during a rebuild.

Yes, usually, if you stop in time. Recovery works with all disks, including the ones that have failed: an image (or copy) of each disk is created onto healthy disks, and a virtual version of the array is built using these images, so no stress is put on the fragile original disks. The most likely way for a recoverable array to become unrecoverable is when repeated rebuilds are forced or the array is recreated, so your safest option is to shut off power and not force it.

// rebuild failed?

Rebuild won’t complete? Power down now.

Don’t retry or re-create the array: this is when you lose your information. Turn off the power source of the device, remove the hard drives, label each drive by its bay, and get a quote first via telephone or on our quote request form. Then send only the disks to us (with a NAS, the disks if you can take them out, otherwise the NAS). We will image them and safely restore your files from those copies. You may post your disks to us from anywhere within the UK or hand deliver them to us in person in Belfast, with no appointment required.

Call us — 028 9002 0144
Mon–Fri · 9am–5:30pm · No fix, no fee
Start a free diagnostic →
028 9002 0144