A business’s 16-disk HPE ProLiant RAID 10 lost a drive, and the rebuild onto its replacement failed part-way — because other disks in the array had quietly developed bad sectors, and two had failed outright. An in-place rebuild can’t cope with that; pushed further, it makes things worse. We imaged all sixteen disks, used the mirror copies to fill each other’s gaps, and rebuilt the array in software.
A long-standing business client ran their core file and database server on a 16-disk HPE ProLiant RAID 10. A disk failed — routine for an array built to tolerate it — and a replacement was fitted so the controller could rebuild. But the rebuild ran for a while and then stalled, the array would not come back healthy, some files were unreadable, and operations ground to a halt. An in-house attempt with standard tools couldn’t get past the errors, and every further attempt risked making the loss permanent, so the whole array was powered down and brought to us.
In RAID 10 the disks are mirrored in pairs and those pairs are striped together. When one disk fails, its partner still holds a complete copy, so a rebuild simply copies the surviving mirror onto the new disk. That works perfectly — as long as the surviving partner can be read from end to end. The trouble is that large disks accumulate latent bad sectors: areas that have quietly gone unreadable but aren’t noticed until something tries to read them. A rebuild reads the entire surviving disk, and the moment it hits an unreadable sector on that mirror it cannot get the data it needs — and the rebuild fails. On this array the problem was compounded: four disks had developed bad sectors and two had failed completely, so several mirrors could not be read cleanly from either side by the controller alone.
It rarely feels fair that several disks fail at once, but it’s common and predictable. The disks in an array are usually from the same batch, the same age, and have done the same work, so they wear towards the end of their lives together. A rebuild is exactly when latent faults surface, because it is the single most read-intensive thing an array ever does — hours of flat-out reading across every remaining disk. That heavy load on tired disks is why a rebuild so often triggers a second (or third) failure, and why continuing to retry a failing rebuild is the most dangerous thing you can do: it hammers marginal disks that a recovery lab would instead read once, gently, and set aside.
Every disk was handled individually and imaged to healthy storage, write-blocked, using tools built to read around bad sectors: retries capped so a bad sector is skipped rather than pounded, healthy regions secured first, weak zones revisited in gentler passes afterwards. The two fully failed disks couldn’t simply be cloned — they needed physical work on the clean bench first (recovering a disk to the point where it can be imaged at all) before their data could be read. Patiently imaging all sixteen, rather than trusting the controller to read them live, is what made the difference — because it meant we had, on stable images, every readable sector from every disk in the array.
This is where a lab reconstruction beats an in-place rebuild. In RAID 10 every block exists on two disks. When one disk has a bad sector at a given location, its mirror almost always holds a good copy of that exact block — but a live controller rebuild, reading one disk at a time, can’t take advantage of that once it stalls. Working from images of both members of each mirror, we merged them: wherever one image had an unreadable gap, the corresponding block was taken from its partner. Combined across all eight mirror pairs, the readable data from the surviving sides reconstructed nearly the entire array even though no single disk was complete. The RAID 10 geometry — stripe size, disk order and mirror pairing — was re-derived by analysis rather than trusting the partially-corrupt metadata, and the array was assembled in software, read-only.
With the array reconstructed, the file system was repaired on the image and the business’s data extracted — the file server’s shares and the database files — then checked for integrity, with database files validated against expected structures and record counts before sign-off. A small amount of data sat on blocks that were unreadable on both mirror members at once, which no reconstruction can conjure back, leaving a 98% recovery with the client’s critical systems restored. Everything was delivered on fresh media inside a priority turnaround of around 72 hours. The lesson we passed on: RAID redundancy buys you time, not immunity — monitor disk health so latent bad sectors are caught early, and keep a real backup, because the day a rebuild fails is the day you find out whether you had one.
Clean-bench recovery of failed disks to imageable state · per-disk imaging with bad-sector handling (16 disks) · mirror-merge across RAID 10 pairs to fill unreadable gaps · RAID geometry analysis · software array reconstruction, read-only · file-system repair and database validation. All work in-house at our Belfast lab.
Stop the rebuild, power the array down, and send us the disks for a free, no-obligation diagnostic. Retrying a failing rebuild is what most often turns a recoverable array into a lost one. We’ll tell you what can be recovered and put a fixed price in writing before any work starts, and we handle priority RAID recoveries for businesses right across the UK.
Usually, yes. A rebuild fails when a surviving disk has unreadable sectors, but the data on the array is still largely there. In the lab we image every disk, then use the mirror or parity copies to fill each other’s gaps — something a live controller can’t do once it stalls. The important thing is to stop retrying the rebuild, because that’s what risks a further disk failure.
It’s common. Disks in an array are usually the same age and have done the same work, so they wear out together, and a rebuild’s heavy read load is exactly what surfaces latent faults on the remaining disks. That’s why multiple failures so often appear during a rebuild — and why imaging the disks gently in a lab is far safer than pushing the array to try again.
No. Each retry means hours of intensive reading across disks that may already be marginal, which is the most likely way to cause another failure and lose more data. Power the array down, keep the disks in order, and get a diagnostic — reconstructing from careful images gives a far better outcome than a repeatedly failing rebuild.