Hello, TL;DR == btrfs 3x500GB RAID 5 - One device failed. Added a new device (btrfs device add) and tried to remove the failed device (btrfs device delete). I tried to mount the array in degraded mode, but that didn't work either. After multiple attempts (including adding back the failed HDD), I finally ran the btrfs rescue chunk-recover command on the primary member /dev/sdb. This ran for about 4 hours, and then failed with "floating point exception (core dumped)". == I am testing out btrfs to gain familiarity with it. I am quite amazed at it's capabilities and performance. However, I am either not able to understand or implement RAID5 fault tolerance. I understand from the wiki that RAID56 is experimental. The data I am working with is backed up elsewhere and for all intents and purposes, discard-able. I have set up a btrfs RAID5 with 3x500GB Seagate HDDs, with a mount point of /storage. Booting is off a fourth HDD (ext4, lubuntu 64bit) that is not involved in the RAID. Everything was working amazingly well, until one HDD failed and was quietly offlined. For a couple of days, the RAID was running off 2 HDDs and I didn't notice. When I DID realize, I shut down the system, bought a new HDD (2TB), which took a couple of days to arrive. When I powered up the system again, the failed 500GB was back. Everything loaded fine, and looked good. To be on the safe side, I ran a badblocks test (ro) on the failing HDD. Halfway through the test, the HDD disappeared again. After a cold reboot, it was loaded fine again. At this point, I decided to replace the failed HDD. I shutdown, plugged in the new HDD in place of the boot HDD, booted up with Lubuntu live, mounted (/storage) and added the device to the RAID. After adding the device successfully, I gave a device delete command for the failed HDD. Partway through the process, the failing HDD (/dev/sdc) disappeared again, and after waiting a couple of hours, I hard-reset the system, and removed the failing HDD, assuming that the RAID will re-build on the existing devices. Now, the RAID (/storage) refused to mount. I got a c_tree error (please see enclosed logs below). I tried to mount the array in degraded mode, but that didn't work either. After multiple attempts (including adding back the failed HDD), I finally ran the btrfs rescue chunk-recover command on the primary member /dev/sdb. This ran for about 4 hours, and then failed with "floating point exception (core dumped)". Can I recover the array or should I start again? The data is not important, but I would like to know the recovery process, or any misconceptions in my thinking that RAID5 with 3 devices is enough for SOHO-level fault tolerance? Any advice, pointers, etc, much appreciated. Tech level: medium-high (RHCE). Relevant system information: === uname -a Linux lubuntu 4.2.0-16-generic #19-Ubuntu SMP Thu Oct 8 15:35:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux == btrfs --version btrfs-progs v4.0 == btrfs fi show warning, device 2 is missing Label: 'storage' uuid: 5a3d6590-df08-4520-b61b-802d350849c7 Total devices 4 FS bytes used 176.91GiB devid 1 size 465.76GiB used 90.03GiB path /dev/sdb devid 3 size 465.76GiB used 90.01GiB path /dev/sdc devid 4 size 1.82TiB used 10.00GiB path /dev/sda *** Some devices missing == dmesg info ... Jan 5 01:45:22 lubuntu kernel: [ 10.338295] Btrfs loaded Jan 5 01:45:22 lubuntu kernel: [ 10.338899] BTRFS: device label storage devid 4 transid 969 /dev/sda Jan 5 01:45:22 lubuntu kernel: [ 10.340448] BTRFS info (device sda): disk space caching is enabled Jan 5 01:45:22 lubuntu kernel: [ 10.340454] BTRFS: has skinny extents Jan 5 01:45:22 lubuntu kernel: [ 10.343395] BTRFS: failed to read the system array on sda Jan 5 01:45:22 lubuntu kernel: [ 10.352137] BTRFS: open_ctree failed Jan 5 01:45:22 lubuntu kernel: [ 10.382199] BTRFS: device label storage devid 1 transid 969 /dev/sdb Jan 5 01:45:22 lubuntu kernel: [ 10.383740] BTRFS info (device sdb): disk space caching is enabled Jan 5 01:45:22 lubuntu kernel: [ 10.383744] BTRFS: has skinny extents Jan 5 01:45:22 lubuntu kernel: [ 10.384469] BTRFS: failed to read the system array on sdb Jan 5 01:45:22 lubuntu kernel: [ 10.392116] BTRFS: open_ctree failed Jan 5 01:45:22 lubuntu kernel: [ 10.423075] BTRFS: device label storage devid 3 transid ... // after btrfs rescue chunk for about 4 hours Jan 5 06:01:45 lubuntu kernel: [15404.828156] traps: btrfs[3016] trap divide error ip:4211a0 sp:7ffd7dbb03a8 error:0 in btrfs[400000+73000] ... == some output from btrfs rescu chunk -vv ... Stripes list: [ 0] Stripe: devid = 3, offset = 21484273664 [ 1] Stripe: devid = 2, offset = 21484273664 [ 2] Stripe: devid = 1, offset = 21504196608 Chunk: start = 45134905344, len = 2147483648, type = 81, num_stripes = 3 Stripes list: [ 0] Stripe: devid = 3, offset = 22558015488 [ 1] Stripe: devid = 2, offset = 22558015488 [ 2] Stripe: devid = 1, offset = 22577938432 Chunk: start = 47282388992, len = 2147483648, type = 81, num_stripes = 3 Stripes list: [ 0] Stripe: devid = 3, offset = 23631757312 [ 1] Stripe: devid = 2, offset = 23631757312 [ 2] Stripe: devid = 1, offset = 23651680256 ... Device extent: devid = 4, start = 5369757696, len = 1073741824, chunk offset = 201901211648 Device extent: devid = 4, start = 6443499520, len = 1073741824, chunk offset = 204048695296 Device extent: devid = 4, start = 7517241344, len = 1073741824, chunk offset = 206196178944 Device extent: devid = 4, start = 8590983168, len = 1073741824, chunk offset = 208343662592 // floating point error Regards, PRShah -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
