RAR / Every level and layout / RAID-Z1, Z2 and Z3

RAID-Z1 · Z2 · Z3 · vdev · uberblock · labels · TrueNAS · Proxmox · zpool import -F

RAID-Z data recovery. Not a fixed map: variable stripes, parity per block, and a transaction history the pool can be rewound through.

RAID-Z is ZFS's own parity layout, and it does not work like a controller's RAID 5. There is no fixed stripe size: each block is written across the vdev's members with its own parity, one, two or three blocks of it, in a width that depends on the block's size. There is no controller metadata at the end of the drives: each member carries four ZFS labels, two at the start and two at the end, and each label holds a ring of uberblocks pointing at the pool's transaction groups. A pool is lost when any top-level vdev is lost, and a vdev is lost when more members fail than its parity allows. What arrives here is TrueNAS, Proxmox and home-built pools with a vdev one failure past its parity, pools that will not import after a power cut, and pools somebody rewound with zpool import -F on the originals. A RAID-Z of two to four members is £500 + VAT upwards after the free look; larger pools from £1,250 + VAT.

Free first lookOne fixed figure in writingNo data, no bill on most jobsReturn postage paid

Rather talk it through? An engineer answers the bench line
0800 6890668

Power down. Label every slot before a drive comes out. Do not rebuild, do not force a drive online, do not import or clear a foreign configuration, do not initialise, and do not run chkdsk, fsck, zpool import -F or mdadm --create on the members. A rebuild reads every sector of every survivor and writes to the replacement; each of the commands writes to the drives. On images, all of them are reversible. On the originals, none of them is.

ZFS symptoms, and what each means.

Not listed? Describe it on the form →

Pool DEGRADEDA vdev has lost a member and still reads. Image the weak drives before resilvering.Do not delay

Pool FAULTED: insufficient replicasA vdev past its parity. Power down; every member imaged.Do not delay

cannot import: one or more devices is currently unavailableA member missing, or its labels damaged. Find it; do not force.Do not delay

Pool UNAVAIL or SUSPENDED after a power cutUberblocks or labels inconsistent. Read-only import on images.Do not delay

zpool import -F already runThe pool was rewound and written. Assessed for what was discarded.Do not delay

Corrupted data on a healthy poolChecksum failures on both copies. A per-file job.Everyday job

Packing it and posting it: power down, photograph the controller's screen or export its log, and write the slot number on each drive with a marker before it comes out of its carrier. Send every member of the set, including the one that failed first; it holds the stripes the rebuild never reached. Each drive travels in an anti-static bag inside its own padding, in a box with nothing able to move. Send the controller if it was replaced or holds an encryption key; otherwise it stays. Insure the parcel for what the data is worth, use a tracked service, and know that the posting address is not printed anywhere on this site; it arrives by email in reply to the form, with a booking sheet to print; the sheet inside the parcel is what matches it to your enquiry when it is opened. Or hand the sealed parcel in at the nearest of ten drop-off points, your name on the outside and the sheet inside; say where you are on the form and it comes by email. We pay the postage home either way. The whole of it is written up on the guide to packing and posting.

What it tolerates, what fails it, and what the bench has to find.

Labels and uberblocksEach member carries four copies of its label, two at the start and two at the end, and each label holds 128 uberblocks pointing at successive transaction groups. The bench reads all four on every image and chooses the newest uberblock that every member agrees on.

Variable stripesA 128KB block on a six-drive RAID-Z2 is spread across the members with two parity blocks; a 4KB block is one data block and two parity blocks. There is no stripe size to find; there is a block pointer tree to walk, from the uberblock down.

zpool import -FRewinds the pool to an earlier transaction group, discarding the most recent writes, and on the originals the discarded groups are gone. On images it is reversible and often the right move. The FAULTED page says when.

The OpenZFS 2.2.0 bugOpenZFS 2.2.1, released 22 November 2023, disabled block cloning by default after a data-corruption bug was found; 2.2.2 and 2.1.14 on 30 November fixed the underlying dirty-dnode check, which The Register reported might go back to 2006. A pool that ran 2.2.0 with block cloning on may hold silently corrupt files.

What a reconstruction has to determine.

Describe yours to us →

What the bench reads	Where it lives	What goes wrong when it is forced
Labels	Four per member: two at the start, two at the end	A member with damaged labels is treated as absent
Uberblocks	128 per label, one per transaction group	The wrong one opens an old or inconsistent pool
vdev membership	The label's vdev tree and GUIDs	A member assigned to the wrong vdev
ashift and offsets	The label; the device's own sector size	Blocks read from the wrong place
Transaction groups	The uberblock ring	A rewind on the originals discards them

From the parcel arriving to your files going back.

Work we have closed →

Logged the day it lands, and the first look costs nothing Free

A case number goes on the parcel and a number on every member the day it is opened, matched to the slot you wrote on it. Each member goes on the imager its interface needs, never on a controller, and its metadata is read before a sector is: the level, the order, the stripe size, the event counts that say which member is current and which dropped out first. An engineer settles what has happened to the set and how much of it can honestly be read back. Back to you come two things together: a straight note of what is liftable and what is not, plus one figure, fixed and written down. Accept it, or decline and owe us nothing.

Nothing to pay for lookingA single figure, put in writingNo rebuilds, no imports, no initialise

Every member imaged, including the one that failed first

Every drive in the set is imaged sector by sector, weak areas last, on hardware that controls every retry, with a map of what could not be read kept for each. Members with failed heads go to the clean bench first; that is the drive site's work and the same bench does it. The drive that dropped out first is imaged too, because it still holds every stripe written before it dropped, and a rebuild that stalled part-way never reached them.

Sector by sector, weak areas lastThe first-dropped member included

The geometry, from the metadata or from the parity

A ZFS pool is opened from images, read-only, by reading all four labels on every member, choosing the newest uberblock that every member agrees on, and walking the block pointer tree from it with checksums verified at every step. Where the newest transaction groups are inconsistent, earlier uberblocks are tried on the images until one opens cleanly, which is what zpool import -F does destructively on the originals.

Metadata first, parity secondOrder, stripe, rotation, offset

Assembled in software, and repaired on the virtual volume

The set is put together from the images in software, with nothing written to any of them: the current members in, the stale member used only to fill holes a survivor could not give. The file system is checked and repaired on a copy of the virtual volume, VMFS and CSV volumes opened and the virtual machines' disks extracted, databases repaired where they need it. The originals are not touched again.

Nothing written to the imagesVirtual machines and databases opened

You see the file list before you pay

What was recovered is listed for you first, and only then does a bill exist. Approve the list and it is invoiced; turn it down and it is not — and where nothing has come back, most jobs carry no charge at all. Recovered data travels home on fresh media bought in for your job, with the postage at our end. Your case is not closed until you have opened the files on a machine of your own.

No charge until you accept the figureFresh media, supplied with the job5–10 days at the bench

What arrives most often

Do not run zpool import -F, -X or -T on the originals. Each rewinds the pool by writing. On images the same rewind is reversible.
Do not resilver a degraded pool with a survivor showing checksum errors. A resilver reads every allocated block of every survivor.
Send every member, and say the pool's version. Feature flags and the ZFS version decide which tools open it.
TrueNAS, Proxmox and Ubuntu pools are the same pools. The host is the NAS site's or the servers' subject; the vdev is this page's.

One job, followed all the way through.

UK · RAR-2026-0607JOB LOGGED ✓

An eight-drive RAID-Z2 on a TrueNAS SCALE box, two members failed a week apart, a third throwing checksum errors during the resilver, and the pool FAULTED with a firm's shares on it

All eight drives came in. The two failed members were imaged, one after a head swap; the checksum-error drive was imaged with its weak areas last; the five survivors directly. All four labels were read on every image, the newest uberblock every member agreed on chosen, and the pool opened read-only from the images at that transaction group. The datasets were copied out with every checksum verified.

100% of the datasets recovered9 days here, and back by post

Illustrative example — replace with a genuine case

What helps, and what harms.

Do this much first

Power down when the vdev reaches its parity limit
Send every member, labelled by slot
Copy out zpool status and zpool import output before touching anything
Say the ZFS version and the host

What sets us back

zpool import -F, -X or -T on the originals
Resilvering with a survivor showing checksum errors
zpool clear to make the error go away
Replacing a drive and letting the resilver read a weak survivor

Questions answered before you commit.

My pool says FAULTED, insufficient replicas. Is the data gone?

Not usually. A vdev has lost more members than its parity allows, and those members are usually imageable. The pool is opened from images at the newest transaction group every member agrees on.

Should I try zpool import -F?

Not on the originals. It rewinds the pool by discarding the newest transaction groups, and on the originals they are gone. On images the same rewind is reversible and the bench does it that way.

Is RAID-Z harder to recover than RAID 5?

Different. There is no stripe size to find, but there is a block pointer tree to walk and checksums to honour. When enough members read, it is routine.

What does it cost?

£500 + VAT upwards for two to four members after the free look; larger pools from £1,250 + VAT, fixed in writing.

How long does it take?

5–10 days at the bench for a small pool; 10–15 days at the bench for a large one.

Begin here if yours is doing the same thing.

ZFS pool FAULTED: insufficient replicas →ZFS and TrueNAS →RAID 6 →Btrfs and Unraid →

Nothing gets worse while it is powered down.

Looking at it is free. Back comes a list of what opened and what did not, together with a single price to finish, set down in writing while you are still free to say no. On most jobs an invoice only follows the data. Until that list reaches you, leave the server off and the drives in their slots.

0800 6890668

Price my job Packing and posting

Every level and layout

Manufacturers

Whatever it is saying now

Servers

What it costs, and what follows