Re: raid10 issues after reorder of boot drives.

[NeilBrown <neilb@xxxxxxx>]

On Fri, 27 Apr 2012 16:04:22 -0400 likewhoa <likewhoa@xxxxxxxxxxxxxxxx> wrote:

> I have a strange issue on my raid10 8x400GB array. I cannot assemble my
> array anymore and it's gotten to a point that I don't know what to do to
> recover my data. I was hoping I could get some advice on it. Below is
> some info which I hope will help troubleshoot this issue.
> 
> > mdadm -Esv
> ARRAY /dev/md/1 metadata=1.0 num-devices=0
> UUID=828ed03d:0c28afda:4a636e88:7b29ec9f name=Darkside:1
>    spares=7  
> devices=/dev/sdg3,/dev/sdf3,/dev/sde3,/dev/sdd3,/dev/sdc3,/dev/sda3,/dev/sdb3
> ARRAY /dev/md/1 level=raid10 metadata=1.0 num-devices=8
> UUID=828ed03d:0c28afda:4a636e88:7b29ec9f name=Darkside:1
>    devices=8:131

I'm afraid you've been bitten by a rather nasty bug which is present in 3.3
and got back ported to some -stable kernel.  The fix has been submitted and
should be appearing in -stable kernels soon (maybe already).

The effect of this bug is to remove key data from the metadata.  It makes the
devices look like a spare for an array of unknown level with no devices and
no chunk size.

It seems that you have one device which wasn't corrupted:  8:131 which should
be /dev/sdi3.  The others are messed up.

What you will need to do is re-create the array.  You can look at /dev/sdi3
to see what the chunksize and layout is:
  mdadm --examine /dev/sdi3

and then run a command like:

  mdadm --stop /dev/md1
  mdadm --create /dev/md1 --metadata=1.0 -l10 -n8 --chunk=XXX --layout=YY \
  --assume-clean \
    /dev/sdg3 /dev/sda3 /dev/sdi3 /dev/sdb3 /dev/sdf3 /dev/sde3 /dev/sdc3 /dev/sdh3
(probably have some device names wrong)

Note that the order of the devices is *very* important, and that is
information that has been lost.  However some of it can be recovered.
If this is an 'n2' or 'near=2' array, which is the default, then adjacent
devices will be identical.  You can find identical pairs with:

 for i in a b c d e f g h i
 do for j in a b c d e f g h i
  do if [ $i = $j ] ; then continue fi
     if cmp -s -n 32768 /dev/sd${i}3 /dev/sd${j}3
     then echo /dev/sd${i}3 and /dev/sd${j}3 seem to match
     fi
  done
 done

That should help you find pairs.  Then you have at most 16 possible orderings
of those 4 pairs to test.

Note that the "--create" command  doesn't touch the data.  It just writes new
metadata and assembles the array.  Then you can run "fsck -n" to check if the
array looks good.

Good luck, and please ask if anything here isn't clear.

NeilBrown

Attachment: signature.asc
Description: PGP signature