RAR / Every level and layout / mdadm and LVM

mdadm · md0 · superblock 0.90 · 1.0 · 1.1 · 1.2 · data offset · left-symmetric · LVM · --examine

mdadm and LVM software RAID recovery. The most honest RAID there is, and the one most often re-created over itself.

Linux's md driver writes a superblock on every member, records the array's UUID, level, chunk size, layout and each member's role, and counts events on every write, so that at assembly it can tell which members are current. It is the RAID inside most NAS units, most home servers and a good many rack servers, often with LVM on top and ext4, XFS or Btrfs on that. It fails honestly, with log lines people paste into Google: kicking non-fresh, not enough devices to start the array, no superblock. And it is ruined in one way above all: the forum's fix for an array that will not assemble is mdadm --create over the existing members, which writes new superblocks, new UUIDs and, with a newer mdadm, a different data offset, and with --assume-clean leaves a set whose geometry no longer matches its data. Where the superblock lives depends on the version: 0.90 and 1.0 at the end of the device, 1.1 at the start, 1.2 four kilobytes from the start. A set of two to four members is £500 + VAT upwards after the free look, fixed in writing, 5–10 days at the bench.

Free first lookOne fixed figure in writingNo data, no bill on most jobsReturn postage paid

Rather talk it through? An engineer answers the bench line
0800 6890668

Power down. Label every slot before a drive comes out. Do not rebuild, do not force a drive online, do not import or clear a foreign configuration, do not initialise, and do not run chkdsk, fsck, zpool import -F or mdadm --create on the members. A rebuild reads every sector of every survivor and writes to the replacement; each of the commands writes to the drives. On images, all of them are reversible. On the originals, none of them is.

mdadm symptoms, and what each means.

Not listed? Describe it on the form →

md: kicking non-fresh sdX from arrayThat member's events are behind. Image it; assemble without it.Everyday job

Not enough devices to start the arrayToo few current members agreed. Read every superblock from images.Do not delay

no superblock on /dev/sdXOverwritten, or the wrong version looked for. The data is usually intact.Do not delay

--create already runNew superblocks written; the old geometry a reconstruction.Do not delay

Array inactive in /proc/mdstatAssembly refused. Do not force.Do not delay

LVM: volume group not foundThe md device is down, or the PV header overwritten.Everyday job

Packing it and posting it: power down, photograph the controller's screen or export its log, and write the slot number on each drive with a marker before it comes out of its carrier. Send every member of the set, including the one that failed first; it holds the stripes the rebuild never reached. Each drive travels in an anti-static bag inside its own padding, in a box with nothing able to move. Send the controller if it was replaced or holds an encryption key; otherwise it stays. Insure the parcel for what the data is worth, use a tracked service, and know that the posting address is not printed anywhere on this site; it arrives by email in reply to the form, with a booking sheet to print; the sheet inside the parcel is what matches it to your enquiry when it is opened. Or hand the sealed parcel in at the nearest of ten drop-off points, your name on the outside and the sheet inside; say where you are on the form and it comes by email. We pay the postage home either way. The whole of it is written up on the guide to packing and posting.

What it tolerates, what fails it, and what the bench has to find.

Four versions, four places0.90 keeps its superblock near the end of the device, which is why old arrays survive an accidental partitioning of the start; 1.0 also at the end; 1.1 at the very start; 1.2, the modern default, 4K from the start. A --create with the wrong version writes a new superblock somewhere the old one was not, and the old one may survive to be read.

The data offsetMetadata 1.2 leaves a gap between the superblock and the data, and the size of that gap has changed with mdadm versions, to 128MiB on modern ones. Recreate an old array with a new mdadm and the data starts in a different place; the superblock says one thing and the data another.

Layoutsleft-symmetric is the RAID 5 default; left-asymmetric, right-symmetric and right-asymmetric exist, as do parity-first and parity-last, and RAID 10 has near, far and offset. The superblock records it, and --examine prints it.

Event countsEvery write increments the array's event counter on every member. A member that dropped out has a lower count and is stale; mdadm kicks it as non-fresh. On the bench the counts are read from the images and decide which members are current.

What a reconstruction has to determine.

Describe yours to us →

Unknown	Where it is read from	What goes wrong when it is guessed
Superblock version and location	--examine on each image, at all four locations	The wrong version finds nothing and --create writes a new one
Chunk size and layout	The superblock	Every stripe past the first reads wrongly
Data offset	The superblock's Data Offset field; the partition table beyond it	Volume start missed after a recreate
Roles and event counts	The superblock	A stale member assembled as current
LVM segments	The PV header and VG metadata on the md device	Logical volumes joined in the wrong order

From the parcel arriving to your files going back.

Work we have closed →

Logged the day it lands, and the first look costs nothing Free

A case number goes on the parcel and a number on every member the day it is opened, matched to the slot you wrote on it. Each member goes on the imager its interface needs, never on a controller, and its metadata is read before a sector is: the level, the order, the stripe size, the event counts that say which member is current and which dropped out first. An engineer settles what has happened to the set and how much of it can honestly be read back. Back to you come two things together: a straight note of what is liftable and what is not, plus one figure, fixed and written down. Accept it, or decline and owe us nothing.

Nothing to pay for lookingA single figure, put in writingNo rebuilds, no imports, no initialise

Every member imaged, including the one that failed first

Every drive in the set is imaged sector by sector, weak areas last, on hardware that controls every retry, with a map of what could not be read kept for each. Members with failed heads go to the clean bench first; that is the drive site's work and the same bench does it. The drive that dropped out first is imaged too, because it still holds every stripe written before it dropped, and a rebuild that stalled part-way never reached them.

Sector by sector, weak areas lastThe first-dropped member included

The geometry, from the metadata or from the parity

Every superblock is read from the images, at all four locations, before anything is assembled: version, UUID, chunk, layout, role, data offset and event count on each member. The current members are chosen by count and the set assembled in software from them, with the stale member used only where the others cannot read. Where --create has overwritten the superblocks, the original geometry is recovered from the parity and the data offset from where the partition table actually sits.

Metadata first, parity secondOrder, stripe, rotation, offset

Assembled in software, and repaired on the virtual volume

The set is put together from the images in software, with nothing written to any of them: the current members in, the stale member used only to fill holes a survivor could not give. The file system is checked and repaired on a copy of the virtual volume, VMFS and CSV volumes opened and the virtual machines' disks extracted, databases repaired where they need it. The originals are not touched again.

Nothing written to the imagesVirtual machines and databases opened

You see the file list before you pay

What was recovered is listed for you first, and only then does a bill exist. Approve the list and it is invoiced; turn it down and it is not — and where nothing has come back, most jobs carry no charge at all. Recovered data travels home on fresh media bought in for your job, with the postage at our end. Your case is not closed until you have opened the files on a machine of your own.

No charge until you accept the figureFresh media, supplied with the job5–10 days at the bench

What arrives most often

Copy out mdadm --examine for every member, and dmesg, before anything else. They are the case.
Do not --create. Not even with --assume-clean. It writes new superblocks and possibly a new data offset, and the old geometry becomes a reconstruction.
Do not --assemble --force until the stale member has been identified and imaged. It promotes old data to current.
An overwritten 1.2 superblock often leaves a 0.90 or 1.0 one intact at the end of the device from an earlier life. The bench looks in all four places.

One job, followed all the way through.

UK · RAR-2026-0610JOB LOGGED ✓

A six-drive mdadm RAID 5 on an Ubuntu server, two members kicked as non-fresh after a power cut, --assemble --force tried, then --create with the wrong chunk size, and an XFS volume that would not mount

All six drives came in. The --create had written new 1.2 superblocks with a 512K chunk and a 128MiB data offset over an array built with a 64K chunk and an older offset. The original geometry was recovered from parity across the images and the true data offset from where the XFS superblock actually sat; the two stale members were identified by their old event counts, still readable at the end of the devices in a 0.90 superblock from the array's first life. The set was assembled from the four current images and the XFS volume mounted read-only.

99.9% of the volume recovered9 days here, and back by post

Illustrative example — replace with a genuine case

What helps, and what harms.

Do this much first

Copy out --examine, mdstat and dmesg for every member
Power down and label the device names or slots
Send every member
Tell us every command that was run, in order

What sets us back

mdadm --create over existing members
--assemble --force before the stale member is known
Adding a spare and letting it resync
Running fsck on a degraded or mis-assembled array

Questions answered before you commit.

I ran mdadm --create over my array. Is it gone?

Not the data; --create writes superblocks, not the data area. What changed is the description, and possibly the data offset. The original geometry is recovered from the parity and the true offset from where the file system actually starts. Do not run anything else.

What does kicking non-fresh mean?

That the member's event count is behind: it dropped out and missed writes. mdadm assembles without it. Image it; do not force it back in.

Where is the mdadm superblock?

It depends on the version. 0.90 and 1.0 near the end of the device, 1.1 at the start, 1.2 four kilobytes from the start. The bench looks in all four places on every image.

What does it cost?

£500 + VAT upwards for two to four members after the free look, fixed in writing; larger sets from £1,250 + VAT.

How long does it take?

5–10 days at the bench.

Begin here if yours is doing the same thing.

mdadm: not enough devices, array inactive →Linux mdadm →SHR and Synology RAID F1 →RAID 5 →

Nothing gets worse while it is powered down.

Looking at it is free. Back comes a list of what opened and what did not, together with a single price to finish, set down in writing while you are still free to say no. On most jobs an invoice only follows the data. Until that list reaches you, leave the server off and the drives in their slots.

0800 6890668

Price my job Packing and posting

Every level and layout

Manufacturers

Whatever it is saying now

Servers

What it costs, and what follows