Re: RAID6 fails to assemble after unclean shutdown

[NeilBrown <neilb@xxxxxxx>]

On Wed, 25 Apr 2012 11:35:36 +0100 Brian Candler <B.Candler@xxxxxxxxx> wrote:

> I have a storage box (currently under test) which has two 12-drive RAID6
> arrays, /dev/md/data1 and /dev/md/data2.
> 
> The box crashed for an unrelated reason, and when I brought it back up, only
> one of the arrays assembled:
> 
>   root@storage1:~# cat /proc/mdstat
>   Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
>   md126 : active raid6 sdj[8] sdk[9] sdd[2] sde[3] sdi[7] sdm[11] sdg[5] sdc[1] sdb[0] sdl[10] sdh[6] sdf[4]
>         29302650880 blocks super 1.2 level 6, 1024k chunk, algorithm 2 [12/12] [UUUUUUUUUUUU]
>         
>   md127 : inactive sdq[3](S) sdx[10](S) sdu[6](S) sdt[5](S) sds[4](S) sdv[8](S) sdp[2](S) sdy[11](S) sdo[1](S) sdn[0](S) sdw[9](S) sdr[7](S)
>         35163186720 blocks super 1.2
>          
>   unused devices: <none>
> 
> So it looks like 12 of the disks have all become spares (S)!

The '(S) is a bit misleading there.  When the array is 'inactive', everything
claims to be spare.  Once the array is actually started it all would become
more sensible.


> 
> An attempt to manually assemble the array failed:
> 
>   root@storage1:~# mdadm --stop /dev/md127
>   mdadm: stopped /dev/md127
>   root@storage1:~# mdadm --assemble /dev/md/disk2 /dev/sd{n..y}
>   mdadm: /dev/md/disk2 assembled from 4 drives - not enough to start the array.

Adding "--verbose" here would help a lot.
Possibly adding "--force" would make it all work.

> 
> Since this is currently under test system I just forcibly recreated the
> array, but I'm a bit worried about how I would handle this problem when I go
> into production.
> 
> Here is how I recreated the array:
> 
>   root@storage1:~# mdadm --create /dev/md/disk2 -n 12 -c 1024 -l raid6 /dev/sd{n..y}
>   mdadm: /dev/sdn appears to be part of a raid array:
>       level=raid6 devices=12 ctime=Mon Mar 19 11:52:55 2012
>   mdadm: /dev/sdo appears to be part of a raid array:
>       level=raid6 devices=12 ctime=Mon Mar 19 11:52:55 2012
>   mdadm: /dev/sdp appears to be part of a raid array:
>       level=raid6 devices=12 ctime=Mon Mar 19 11:52:55 2012
>   mdadm: /dev/sdq appears to be part of a raid array:
>       level=raid6 devices=12 ctime=Mon Mar 19 11:52:55 2012
>   mdadm: /dev/sdr appears to be part of a raid array:
>       level=raid6 devices=12 ctime=Mon Mar 19 11:52:55 2012
>   mdadm: /dev/sds appears to be part of a raid array:
>       level=raid6 devices=12 ctime=Mon Mar 19 11:52:55 2012
>   mdadm: /dev/sdt appears to be part of a raid array:
>       level=raid6 devices=12 ctime=Mon Mar 19 11:52:55 2012
>   mdadm: /dev/sdu appears to be part of a raid array:
>       level=raid6 devices=12 ctime=Mon Mar 19 11:52:55 2012
>   mdadm: /dev/sdv appears to be part of a raid array:
>       level=raid6 devices=12 ctime=Mon Mar 19 11:52:55 2012
>   mdadm: /dev/sdw appears to be part of a raid array:
>       level=raid6 devices=12 ctime=Mon Mar 19 11:52:55 2012
>   mdadm: /dev/sdx appears to be part of a raid array:
>       level=raid6 devices=12 ctime=Mon Mar 19 11:52:55 2012
>   mdadm: /dev/sdy appears to be part of a raid array:
>   # /etc/fstab: static file system information.
>       level=raid6 devices=12 ctime=Mon Mar 19 11:52:55 2012
>   Continue creating array? y
>   mdadm: Defaulting to version 1.2 metadata
>   mdadm: array /dev/md/disk2 started.
> 
> So it seems like all the disks were known to be part of an array, but mdadm
> was still unable to assemble more than 4.

I would need to see the "--examine" output of each disk (Before you
recreated) to be able to explain.


> 
> Platform: Ubuntu 11.10 server x86_64, stock kernel:
> 
>   Linux storage1 3.0.0-16-server #29-Ubuntu SMP Tue Feb 14 13:08:12 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
> 
> Unfortunately I saw the same problem once before on a different test system,
> and also had to forcibly rebuild the array.
> 
> So my questions are:
> 
> * Have I built the RAID array correctly in the first place? Are there some
> options I could have given to mdadm to make it more robust?

Yes, you have built the array correctly.


> 
> * What should I have done when presented with an array which would not
> assemble, to attempt to recover without losing data?

 --verbose
and maybe
 --force

> 
> * Any ideas why mdadm only thought 4 of the drives were usable?

Presumably something when wrong during shutdown.  However without more
details (--examine) I cannot guess.


NeilBrown

Attachment: signature.asc
Description: PGP signature