Re: mdadm-3.2.5: segfault in "--grow --continue"

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]


Hi!

Am 10.07.2012 um 15:02 schrieb NeilBrown:
> On Tue, 10 Jul 2012 12:46:19 +0200 Sebastian Hegler <sebastian.hegler@xxxxxxxxxxxxx> wrote:
>> I had to shut down a server with a raid grow operation (cleanly). After some hassles I got the array assembled again (manually), but it would not continue to grow. When trying 
>> 
>> mdadm --grow --continue /dev/md127 
>> 
>> (as per the manpage) I receive a segfault. Compiling from source, and using gdb, I see:
>> 
>> root@kuiper:~/mdadm-3.2.5# gdb ./mdadm 
>> [SNIP]
>> Reading symbols from /root/mdadm-3.2.5/mdadm...done.
>> (gdb) set args --grow --continue /dev/md127 
>> (gdb) run 
>> Starting program: /root/mdadm-3.2.5/mdadm --grow --continue /dev/md127 
>> 
>> Program received signal SIGSEGV, Segmentation fault.
>> Grow_continue_command (devname=0x7fffffffe8cc "/dev/md127", fd=7, backup_file=0x0, verbose=0) at Grow.c:4118
>> 4118		if (verify_reshape_position(content,
>> (gdb) bt
>> #0  Grow_continue_command (devname=0x7fffffffe8cc "/dev/md127", fd=7, backup_file=0x0, verbose=0) at Grow.c:4118
>> #1  0x0000000000407ac2 in main (argc=4, argv=0x7fffffffe678) at mdadm.c:1701
>> 
>> 
>> 
>> The same bug is present in the git repository, but another location:
>> 
>> root@kuiper:~/mdadm.git# gdb ./mdadm 
>> [SNIP]
>> Reading symbols from /root/mdadm.git/mdadm...done.
>> (gdb) set args --grow --continue /dev/md127 
>> (gdb) run 
>> Starting program: /root/mdadm.git/mdadm --grow --continue /dev/md127 
>> 
>> Program received signal SIGSEGV, Segmentation fault.
>> Grow_continue_command (devname=0x7fffffffe8d0 "/dev/md127", fd=7, backup_file=0x0, verbose=0) at Grow.c:4086
>> 4086		if (verify_reshape_position(content,
>> (gdb) bt
>> #0  Grow_continue_command (devname=0x7fffffffe8d0 "/dev/md127", fd=7, backup_file=0x0, verbose=0) at Grow.c:4086
>> #1  0x0000000000406c39 in main (argc=4, argv=0x7fffffffe678) at mdadm.c:1447
>> 
>> 
>> I also filed a bug against Debian's BTS, but I did not receive a bug number yet. In the meantime, I'd be very happy about any information on how to get my RAID array back into growing. I'm not on the list, please CC me.
> 
> --grow --continue should only be needed if the array was assembled with --freeze-reshape (I think).
> 
> Normally, the reshape should continue automatically.
Well, it did not.

> cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] 
md127 : active raid6 sdi[14] sdh[22] sdt[11] sdp[16] sdq[13] sdo[17] sdn[18] sdu[12] sdm[19] sdl[20] sdk[21] sdj[15]
      17581629312 blocks super 1.2 level 6, 128k chunk, algorithm 2 [12/12] [UUUUUUUUUUUU]

> mdadm -E ....
root@kuiper:~# for i in /sys/block/md127/md/dev-sd* ; do mdadm -E /`basename $i |tr - / ` ; done 
/dev/sdh:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : 42683d6d:6479d415:492ede77:6fe9437b
           Name : kuiper:scratch2G  (local to host kuiper)
  Creation Time : Fri Apr 16 12:26:18 2010
     Raid Level : raid6
   Raid Devices : 12

 Avail Dev Size : 3907028736 (1863.02 GiB 2000.40 GB)
     Array Size : 19535143680 (18630.17 GiB 20003.99 GB)
    Data Offset : 432 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : f281deb2:b5de8070:487c1900:0078e6e1

  Reshape pos'n : 6220766720 (5932.59 GiB 6370.07 GB)
  Delta Devices : 1 (11->12)

    Update Time : Tue Jul 10 16:10:45 2012
       Checksum : 16d3efce - correct
         Events : 6245853

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 11
   Array State : AAAAAAAAAAAA ('A' == active, '.' == missing)
/dev/sdi:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : 42683d6d:6479d415:492ede77:6fe9437b
           Name : kuiper:scratch2G  (local to host kuiper)
  Creation Time : Fri Apr 16 12:26:18 2010
     Raid Level : raid6
   Raid Devices : 12

 Avail Dev Size : 3907028736 (1863.02 GiB 2000.40 GB)
     Array Size : 19535143680 (18630.17 GiB 20003.99 GB)
    Data Offset : 432 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : b67b1847:cb68ac69:7ab0b4ee:894a268b

  Reshape pos'n : 6220766720 (5932.59 GiB 6370.07 GB)
  Delta Devices : 1 (11->12)

    Update Time : Tue Jul 10 16:10:45 2012
       Checksum : 3c147a5b - correct
         Events : 6245853

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 0
   Array State : AAAAAAAAAAAA ('A' == active, '.' == missing)
/dev/sdj:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : 42683d6d:6479d415:492ede77:6fe9437b
           Name : kuiper:scratch2G  (local to host kuiper)
  Creation Time : Fri Apr 16 12:26:18 2010
     Raid Level : raid6
   Raid Devices : 12

 Avail Dev Size : 3907028736 (1863.02 GiB 2000.40 GB)
     Array Size : 19535143680 (18630.17 GiB 20003.99 GB)
    Data Offset : 432 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 5c4e03cc:e2fe014e:bd3d17a9:517d34f5

  Reshape pos'n : 6220766720 (5932.59 GiB 6370.07 GB)
  Delta Devices : 1 (11->12)

    Update Time : Tue Jul 10 16:10:46 2012
       Checksum : c9c5a324 - correct
         Events : 6245852

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 1
   Array State : AAAAAAAAAAAA ('A' == active, '.' == missing)
/dev/sdk:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : 42683d6d:6479d415:492ede77:6fe9437b
           Name : kuiper:scratch2G  (local to host kuiper)
  Creation Time : Fri Apr 16 12:26:18 2010
     Raid Level : raid6
   Raid Devices : 12

 Avail Dev Size : 3907028736 (1863.02 GiB 2000.40 GB)
     Array Size : 19535143680 (18630.17 GiB 20003.99 GB)
    Data Offset : 432 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : e276dc2b:048e6135:2364e8d0:00566eed

  Reshape pos'n : 6220766720 (5932.59 GiB 6370.07 GB)
  Delta Devices : 1 (11->12)

    Update Time : Tue Jul 10 16:10:46 2012
       Checksum : 310959e7 - correct
         Events : 6245852

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 2
   Array State : AAAAAAAAAAAA ('A' == active, '.' == missing)
/dev/sdl:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : 42683d6d:6479d415:492ede77:6fe9437b
           Name : kuiper:scratch2G  (local to host kuiper)
  Creation Time : Fri Apr 16 12:26:18 2010
     Raid Level : raid6
   Raid Devices : 12

 Avail Dev Size : 3907028736 (1863.02 GiB 2000.40 GB)
     Array Size : 19535143680 (18630.17 GiB 20003.99 GB)
    Data Offset : 432 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 547956df:6be81839:89424cbc:664bc6d6

  Reshape pos'n : 6220766720 (5932.59 GiB 6370.07 GB)
  Delta Devices : 1 (11->12)

    Update Time : Tue Jul 10 16:10:46 2012
       Checksum : bcf68a8b - correct
         Events : 6245852

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 3
   Array State : AAAAAAAAAAAA ('A' == active, '.' == missing)
/dev/sdm:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : 42683d6d:6479d415:492ede77:6fe9437b
           Name : kuiper:scratch2G  (local to host kuiper)
  Creation Time : Fri Apr 16 12:26:18 2010
     Raid Level : raid6
   Raid Devices : 12

 Avail Dev Size : 3907028736 (1863.02 GiB 2000.40 GB)
     Array Size : 19535143680 (18630.17 GiB 20003.99 GB)
    Data Offset : 432 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 72a0a33b:faa5ecd4:7b543389:f49798a2

  Reshape pos'n : 6220766720 (5932.59 GiB 6370.07 GB)
  Delta Devices : 1 (11->12)

    Update Time : Tue Jul 10 16:10:46 2012
       Checksum : 4dd0cdb7 - correct
         Events : 6245852

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 4
   Array State : AAAAAAAAAAAA ('A' == active, '.' == missing)
/dev/sdn:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : 42683d6d:6479d415:492ede77:6fe9437b
           Name : kuiper:scratch2G  (local to host kuiper)
  Creation Time : Fri Apr 16 12:26:18 2010
     Raid Level : raid6
   Raid Devices : 12

 Avail Dev Size : 3907028736 (1863.02 GiB 2000.40 GB)
     Array Size : 19535143680 (18630.17 GiB 20003.99 GB)
    Data Offset : 432 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 11c4c3ac:ac1be81a:d4750037:45a85a60

  Reshape pos'n : 6220766720 (5932.59 GiB 6370.07 GB)
  Delta Devices : 1 (11->12)

    Update Time : Tue Jul 10 16:10:46 2012
       Checksum : 707b98b0 - correct
         Events : 6245852

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 6
   Array State : AAAAAAAAAAAA ('A' == active, '.' == missing)
/dev/sdo:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : 42683d6d:6479d415:492ede77:6fe9437b
           Name : kuiper:scratch2G  (local to host kuiper)
  Creation Time : Fri Apr 16 12:26:18 2010
     Raid Level : raid6
   Raid Devices : 12

 Avail Dev Size : 3907028736 (1863.02 GiB 2000.40 GB)
     Array Size : 19535143680 (18630.17 GiB 20003.99 GB)
    Data Offset : 432 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 76692433:04aab129:7f46b4f6:d2d93ffd

  Reshape pos'n : 6220766720 (5932.59 GiB 6370.07 GB)
  Delta Devices : 1 (11->12)

    Update Time : Tue Jul 10 16:10:46 2012
       Checksum : 623ecea5 - correct
         Events : 6245852

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 7
   Array State : AAAAAAAAAAAA ('A' == active, '.' == missing)
/dev/sdp:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : 42683d6d:6479d415:492ede77:6fe9437b
           Name : kuiper:scratch2G  (local to host kuiper)
  Creation Time : Fri Apr 16 12:26:18 2010
     Raid Level : raid6
   Raid Devices : 12

 Avail Dev Size : 3907028736 (1863.02 GiB 2000.40 GB)
     Array Size : 19535143680 (18630.17 GiB 20003.99 GB)
    Data Offset : 432 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : f62ba0d8:17345ece:55a5d1bf:39540cc5

  Reshape pos'n : 6220766720 (5932.59 GiB 6370.07 GB)
  Delta Devices : 1 (11->12)

    Update Time : Tue Jul 10 16:10:46 2012
       Checksum : 3d50f475 - correct
         Events : 6245852

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 9
   Array State : AAAAAAAAAAAA ('A' == active, '.' == missing)
/dev/sdq:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : 42683d6d:6479d415:492ede77:6fe9437b
           Name : kuiper:scratch2G  (local to host kuiper)
  Creation Time : Fri Apr 16 12:26:18 2010
     Raid Level : raid6
   Raid Devices : 12

 Avail Dev Size : 3907028736 (1863.02 GiB 2000.40 GB)
     Array Size : 19535143680 (18630.17 GiB 20003.99 GB)
    Data Offset : 432 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : a4091eff:68556278:3a22b428:aa59ea46

  Reshape pos'n : 6220766720 (5932.59 GiB 6370.07 GB)
  Delta Devices : 1 (11->12)

    Update Time : Tue Jul 10 16:10:46 2012
       Checksum : f89375c5 - correct
         Events : 6245852

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 8
   Array State : AAAAAAAAAAAA ('A' == active, '.' == missing)
/dev/sdt:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : 42683d6d:6479d415:492ede77:6fe9437b
           Name : kuiper:scratch2G  (local to host kuiper)
  Creation Time : Fri Apr 16 12:26:18 2010
     Raid Level : raid6
   Raid Devices : 12

 Avail Dev Size : 3907028736 (1863.02 GiB 2000.40 GB)
     Array Size : 19535143680 (18630.17 GiB 20003.99 GB)
    Data Offset : 432 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : f8425467:c7533842:df4c4da6:424e107f

  Reshape pos'n : 6220766720 (5932.59 GiB 6370.07 GB)
  Delta Devices : 1 (11->12)

    Update Time : Tue Jul 10 16:10:46 2012
       Checksum : e05eccb3 - correct
         Events : 6245852

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 10
   Array State : AAAAAAAAAAAA ('A' == active, '.' == missing)
/dev/sdu:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x4
     Array UUID : 42683d6d:6479d415:492ede77:6fe9437b
           Name : kuiper:scratch2G  (local to host kuiper)
  Creation Time : Fri Apr 16 12:26:18 2010
     Raid Level : raid6
   Raid Devices : 12

 Avail Dev Size : 3907028736 (1863.02 GiB 2000.40 GB)
     Array Size : 19535143680 (18630.17 GiB 20003.99 GB)
    Data Offset : 432 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : acde8544:51d21dcd:e6b6827e:c258f274

  Reshape pos'n : 6220766720 (5932.59 GiB 6370.07 GB)
  Delta Devices : 1 (11->12)

    Update Time : Tue Jul 10 16:10:46 2012
       Checksum : 168d5b7a - correct
         Events : 6245852

         Layout : left-symmetric
     Chunk Size : 128K

   Device Role : Active device 5
   Array State : AAAAAAAAAAAA ('A' == active, '.' == missing)

> mdadm -D ...
root@kuiper:~# mdadm -D /dev/md127 
/dev/md127:
        Version : 1.2
  Creation Time : Fri Apr 16 12:26:18 2010
     Raid Level : raid6
     Array Size : 17581629312 (16767.15 GiB 18003.59 GB)
  Used Dev Size : 1953514368 (1863.02 GiB 2000.40 GB)
   Raid Devices : 12
  Total Devices : 12
    Persistence : Superblock is persistent

    Update Time : Tue Jul 10 16:11:23 2012
          State : clean 
 Active Devices : 12
Working Devices : 12
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 128K

  Delta Devices : 1, (11->12)

           Name : kuiper:scratch2G  (local to host kuiper)
           UUID : 42683d6d:6479d415:492ede77:6fe9437b
         Events : 6245852

    Number   Major   Minor   RaidDevice State
      14       8      128        0      active sync   /dev/sdi
      15       8      144        1      active sync   /dev/sdj
      21       8      160        2      active sync   /dev/sdk
      20       8      176        3      active sync   /dev/sdl
      19       8      192        4      active sync   /dev/sdm
      12      65       64        5      active sync   /dev/sdu
      18       8      208        6      active sync   /dev/sdn
      17       8      224        7      active sync   /dev/sdo
      13      65        0        8      active sync   /dev/sdq
      16       8      240        9      active sync   /dev/sdp
      11      65       48       10      active sync   /dev/sdt
      22       8      112       11      active sync   /dev/sdh

> would be helpful, maybe together with kernel logs if there are any.
Please see the attached files. I truncated the file up until the first segfault of mdadm. Short wrapup of the contents:
* Bad interaction: "mpt-sas ioc1" is too slow for md. md is assembling RAID arrays from the disks it sees, creating some MD devices, one partially broken and recovering where there is no need (if md had waited), and two incomplete arrays actually belonging to one array (timestamps: 35.655213, 34.528229). Some disks are also completely missing thanks to failed slots in the enclosure.
* I then removed all drives belonging to the array I interrupted in the Grow procedure (via "/sys/block/sd*/delete"), re-inserted them into the enclosure, and re-assembled the RAID array manually. Finish is at "1397.498460": "md/raid:md127: reshape will continue"
* At "1565.278839" The RAID array is in "auto-read-only" mode. I mount the FS, wondering about not seeing the enclosure LEDs blink. After checking "/proc/mdstat", I see that everything is quiet, although I expected a progress bar.
* At "2612.464768" (fourth-to-last line) mdadm segfaults with the command "mdadm --grow --continue /dev/md127"

After yet another reboot, and careful manual re-assembly of the RAID array in question, everything works fine. Grow operation continued after mounting the file system. 

The issue was assigned bug number 681056 in the Debian BTS in the meantime: CCing the BTS.

Yours,
Sebastian

Attachment: kern.mdcrash.log.xz
Description: Binary data


[ATA RAID]     [Linux SCSI Target Infrastructure]     [Managing RAID on Linux]     [Linux IDE]     [Linux SCSI]     [Linux Hams]     [Device-Mapper]     [Kernel]     [Linux Books]     [Linux Admin]     [Linux Net]     [GFS]     [RPM]     [git]     [Photos]     [Yosemite Photos]     [Yosemite News]     [AMD 64]     [Linux Networking]

Add to Google Powered by Linux