Hiya,
Before setting up a new RAID10 btrfs array with 6 drives, I
wanted to check how good it behaved in case of disk failure.
I've not been too impressed. Is RAID10 btrfs support only
meant for reading performance improvement?
My test method was:
Use the device-mapper to have devices mapped (linear) to loop
devices
zsh syntax:
# l=({1..4})
# mv /dev/loop$^l .
# truncate -s1T $l
# s=$(blockdev --getsize /dev/loop1)
# for f ($l) losetup loop$f $f
# for f ($l) echo 0 $s linear loop$f 0 | dmsetup create hd$f
# mkfs.btrfs -m raid10 -d raid10 /dev/mapper/hd$^l
# d=(device=/dev/mapper/hd$^l)
# mount -o ${(j:,:)d} /dev/mapper/hd1 /mnt/3
Then write some data, and then use DM's error target to simulate
a failing drive (all I/O ends up in error):
# dmsetup suspend hd3; echo 0 $s error | dmsetup reload hd3; dmsetup resume hd3
Then write some more data. The FS doesn't become degraded
automatically. If I restore the drive:
# echo 0 $s linear loop3 0 | dmsetup create hd3
More funny things occur of course as btrfs doesn't seem to have
registered it being broken.
If I do a scrub with the failing drive, it "BUGs ON":
[13960.286464] ------------[ cut here ]------------
[13960.286484] kernel BUG at /home/blank/debian/kernel/release/linux-2.6/linux-2.6-3.1.0/debian/build/source_amd64_none/fs/btrfs/volumes.c:2891!
[13960.286496] invalid opcode: 0000 [#1] SMP
[13960.286507] CPU 0
[13960.286510] Modules linked in: vboxnetadp(O) vboxnetflt(O) vboxdrv(O) ip6table_filter ip6_tables ebtable_nat acpi_cpufreq mperf ebtables cpufreq_powersave cpufreq_userspace cpufreq_conservative cpufreq_stats ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables bridge stp parport_pc ppdev lp parport rfcomm bnep binfmt_misc uinput deflate ctr twofish_generic twofish_x86_64 twofish_common camellia serpent blowfish cast5 des_generic cbc xcbc rmd160 sha512_generic sha256_generic sha1_generic hmac crypto_null af_key fuse nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_crypt coretemp loop kvm_intel kvm uvcvideo bcm5974 videodev media v4l2_compat_ioctl32 cryptd snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm aes_x86_64 aes_generic snd_seq_midi ecb btusb bluetooth rfkill snd_rawmidi nouveau ttm snd_seq_midi_event drm_kms_helper snd_seq drm i2c_algo_bit mxm_wmi snd_timer wmi snd_seq_device joydev video battery snd ac apple_bl power_supply soundcore snd_page_alloc applesmc pcspkr input_polldev i2c_i801 i2c_core evdev button processor thermal_sys ext4 mbcache jbd2 crc16 btrfs zlib_deflate crc32c libcrc32c raid10 raid456 async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx raid1 raid0 multipath linear md_mod nbd dm_mirror dm_region_hash dm_log dm_mod sg sr_mod sd_mod cdrom crc_t10dif hid_apple usbhid hid ata_generic uhci_hcd firewire_ohci sata_sil24 ata_piix firewire_core crc_itu_t libata sky2 ehci_hcd scsi_mod usbcore [last unloaded: scsi_wait_scan]
[13960.287012]
[13960.287016] Pid: 12681, comm: btrfs-scrub-0 Tainted: G W O 3.1.0-1-amd64 #1 Apple Inc. MacBookPro4,1/Mac-F42C86C8
[13960.287037] RIP: 0010:[<ffffffffa020fc8a>] [<ffffffffa020fc8a>] __btrfs_map_block+0xfd/0x629 [btrfs]
[13960.287061] RSP: 0018:ffff880078c87cb0 EFLAGS: 00010282
[13960.287067] RAX: 0000000000000042 RBX: ffff880078c87d68 RCX: 00000000000054ee
[13960.287076] RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000246
[13960.287085] RBP: ffff8800378f6380 R08: 0000000000000002 R09: 00000000fffffffe
[13960.287093] R10: 0000000000000001 R11: ffff88007e395c90 R12: 0000000000000000
[13960.287102] R13: 0000000000000008 R14: 0000000000000000 R15: 0000000000000001
[13960.287114] FS: 0000000000000000(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000
[13960.287126] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[13960.287136] CR2: 00000000f76990a0 CR3: 0000000104319000 CR4: 00000000000006f0
[13960.287147] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[13960.287159] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[13960.287169] Process btrfs-scrub-0 (pid: 12681, threadinfo ffff880078c86000, task ffff880021698280)
[13960.287184] Stack:
[13960.287189] 0000000000000000 ffff880100000001 0000004000000000 ffff88008a1f40f8
[13960.287207] ffff88008a1f4100 ffff880078c87d60 0000000000000001 0000000000000001
[13960.287223] ffff880078c87cf0 ffff880109671000 ffff880123416400 0000000000000000
[13960.287242] Call Trace:
[13960.287259] [<ffffffffa022aa1f>] ? scrub_recheck_error+0x105/0x29b [btrfs]
[13960.287280] [<ffffffffa022ac2a>] ? scrub_checksum+0x75/0x372 [btrfs]
[13960.287288] [<ffffffff810394c7>] ? check_preempt_wakeup+0x122/0x18b
[13960.287297] [<ffffffff81036e0c>] ? set_next_entity+0x32/0x52
[13960.287304] [<ffffffff8100d031>] ? load_gs_index+0x7/0xa
[13960.287312] [<ffffffff8100d6a8>] ? __switch_to+0x15a/0x20e
[13960.287331] [<ffffffffa02141f1>] ? worker_loop+0x16a/0x45d [btrfs]
[13960.287341] [<ffffffff8132bcd4>] ? __schedule+0x5ac/0x5c3
[13960.287360] [<ffffffffa0214087>] ? btrfs_queue_worker+0x25b/0x25b [btrfs]
[13960.287371] [<ffffffff8105e5f5>] ? kthread+0x76/0x7e
[13960.287379] [<ffffffff81334134>] ? kernel_thread_helper+0x4/0x10
[13960.287388] [<ffffffff8105e57f>] ? kthread_worker_fn+0x139/0x139
[13960.287395] [<ffffffff81334130>] ? gs_change+0x13/0x13
[13960.287403] Code: 7c 24 20 48 89 04 24 e8 29 ea ff ff 48 8b 0c 24 48 85 c9 75 18 48 8b 13 48 8b 74 24 10 48 c7 c7 0c 63 23 a0 31 c0 e8 a8 6f 11 e1 <0f> 0b 48 8b 41 18 48 3b 44 24 10 77 0f 4c 8b 49 20 49 8d 14 01
[13960.287595] RIP [<ffffffffa020fc8a>] __btrfs_map_block+0xfd/0x629 [btrfs]
[13960.287618] RSP <ffff880078c87cb0>
[13960.305670] ---[ end trace 99b51efe6d872029 ]---
In which way has btrfs raid10 been tested? For reliability,
should we stick with md raid for now, or btrfs raid0 on md
raid1?
How would one go about replacing a failing drive without
unmounting the FS?
Other things I've seen in dmesg:
[11859.945412] btrfs_readpage_end_io_hook: 9614 callbacks suppressed
[11859.945417] btrfs csum failed ino 257 off 0 csum 2566472073 private 397849880
[11859.945451] btrfs csum failed ino 257 off 4096 csum 2566472073 private 2631731436
[11859.945481] btrfs csum failed ino 257 off 8192 csum 2566472073 private 2640556530
[11859.945513] btrfs csum failed ino 257 off 12288 csum 2566472073 private 27602266
[11859.945544] btrfs csum failed ino 257 off 16384 csum 2566472073 private 321745155
[11859.945570] btrfs csum failed ino 257 off 8192 csum 2566472073 private 2640556530
[11859.945574] btrfs csum failed ino 257 off 20480 csum 2566472073 private 1458484598
[11859.945605] btrfs csum failed ino 257 off 12288 csum 2566472073 private 27602266
[11859.945611] btrfs csum failed ino 257 off 24576 csum 2566472073 private 2602336016
[11859.945642] btrfs csum failed ino 257 off 28672 csum 2566472073 private 1915815056
[11882.115754] btrfs_readpage_end_io_hook: 23 callbacks suppressed
[11882.115757] btrfs csum failed ino 257 off 12288 csum 2566472073 private 27602266
[11882.115775] btrfs csum failed ino 257 off 16384 csum 2566472073 private 321745155
[11882.115790] btrfs csum failed ino 257 off 20480 csum 2566472073 private 1458484598
[11882.115806] btrfs csum failed ino 257 off 24576 csum 2566472073 private 2602336016
[11882.115823] btrfs csum failed ino 257 off 28672 csum 2566472073 private 1915815056
[11882.115839] btrfs csum failed ino 257 off 32768 csum 2566472073 private 1866379989
[11882.115854] btrfs csum failed ino 257 off 36864 csum 2566472073 private 2312891658
[11882.115869] btrfs csum failed ino 257 off 40960 csum 2566472073 private 3982095500
[11882.115912] btrfs csum failed ino 257 off 0 csum 2566472073 private 397849880
[11882.115933] btrfs csum failed ino 257 off 4096 csum 2566472073 private 2631731436
[...]
[13769.913828] Buffer I/O error on device dm-8, logical block 268435440
[13769.913835] Buffer I/O error on device dm-8, logical block 268435440
[13769.913843] Buffer I/O error on device dm-8, logical block 268435454
[13769.913847] Buffer I/O error on device dm-8, logical block 268435454
[13769.913854] Buffer I/O error on device dm-8, logical block 0
[13769.913858] Buffer I/O error on device dm-8, logical block 0
[13769.913865] Buffer I/O error on device dm-8, logical block 1
[13769.913871] Buffer I/O error on device dm-8, logical block 268435455
[13769.913966] Buffer I/O error on device dm-8, logical block 268435455
[13769.913971] Buffer I/O error on device dm-8, logical block 268435455
[13792.465869] btrfs_end_buffer_write_sync: 2 callbacks suppressed
[13792.465873] lost page write due to I/O error on dm-8
[13792.465891] lost page write due to I/O error on dm-8
[13792.465908] lost page write due to I/O error on dm-8
[13801.762573] btrfs: unable to go below four devices on raid10
[13883.465898] ioctl32(btrfs:12629): Unknown cmd fd(3) cmd(5000940b){t:ffffff94;sz:4096} arg(ffc4d1cc) on /home/stephane/install/cvs/btrfs-progs-unstable-mason
[13960.286090] scrub_fixup: 138 callbacks suppressed
[13960.286093] btrfs: unable to fixup at 65536
[13960.286135] ------------[ cut here ]------------
[13960.286195] WARNING: at /home/blank/debian/kernel/release/linux-2.6/linux-2.6-3.1.0/debian/build/source_amd64_none/fs/btrfs/scrub.c:236 scrub_fixup_check+0xab/0xc1 [btrfs]()
[13960.286198] Hardware name: MacBookPro4,1
[13960.286199] Modules linked in: vboxnetadp(O) vboxnetflt(O) vboxdrv(O) ip6table_filter ip6_tables ebtable_nat acpi_cpufreq mperf ebtables cpufreq_powersave cpufreq_userspace cpufreq_conservative cpufreq_stats ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables bridge stp parport_pc ppdev lp parport rfcomm bnep binfmt_misc uinput deflate ctr twofish_generic twofish_x86_64 twofish_common camellia serpent blowfish cast5 des_generic cbc xcbc rmd160 sha512_generic sha256_generic sha1_generic hmac crypto_null af_key fuse nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_crypt coretemp loop kvm_intel kvm uvcvideo bcm5974 videodev media v4l2_compat_ioctl32 cryptd snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm aes_x86_64 aes_generic snd_seq_midi ecb btusb bluetooth rfkill snd_rawmidi nouveau ttm snd_seq_midi_event drm_kms_helper snd_seq drm i2c_algo_bit mxm_wmi snd_timer wmi snd_seq_device joydev video battery snd ac apple_bl power_supply soundcore snd_page_alloc applesmc pcspkr input_polldev i2c_i801 i2c_core evdev button processor thermal_sys ext4 mbcache jbd2 crc16 btrfs zlib_deflate crc32c libcrc32c raid10 raid456 async_raid6_recov async_pq raid6_pq async_xor xor async_memcpy async_tx raid1 raid0 multipath linear md_mod nbd dm_mirror dm_region_hash dm_log dm_mod sg sr_mod sd_mod cdrom crc_t10dif hid_apple usbhid hid ata_generic uhci_hcd firewire_ohci sata_sil24 ata_piix firewire_core crc_itu_t libata sky2 ehci_hcd scsi_mod usbcore [last unloaded: scsi_wait_scan]
[13960.286299] Pid: 12681, comm: btrfs-scrub-0 Tainted: G O 3.1.0-1-amd64 #1
[13960.286302] Call Trace:
[13960.286310] [<ffffffff81045795>] ? warn_slowpath_common+0x78/0x8c
[13960.286323] [<ffffffffa022a904>] ? scrub_fixup_check+0xab/0xc1 [btrfs]
[13960.286337] [<ffffffffa022aaca>] ? scrub_recheck_error+0x1b0/0x29b [btrfs]
[13960.286350] [<ffffffffa022ac2a>] ? scrub_checksum+0x75/0x372 [btrfs]
[13960.286354] [<ffffffff810394c7>] ? check_preempt_wakeup+0x122/0x18b
[13960.286357] [<ffffffff81036e0c>] ? set_next_entity+0x32/0x52
[13960.286362] [<ffffffff8100d031>] ? load_gs_index+0x7/0xa
[13960.286365] [<ffffffff8100d6a8>] ? __switch_to+0x15a/0x20e
[13960.286379] [<ffffffffa02141f1>] ? worker_loop+0x16a/0x45d [btrfs]
[13960.286384] [<ffffffff8132bcd4>] ? __schedule+0x5ac/0x5c3
[13960.286399] [<ffffffffa0214087>] ? btrfs_queue_worker+0x25b/0x25b [btrfs]
[13960.286403] [<ffffffff8105e5f5>] ? kthread+0x76/0x7e
[13960.286408] [<ffffffff81334134>] ? kernel_thread_helper+0x4/0x10
[13960.286411] [<ffffffff8105e57f>] ? kthread_worker_fn+0x139/0x139
[13960.286414] [<ffffffff81334130>] ? gs_change+0x13/0x13
[13960.286416] ---[ end trace 99b51efe6d872028 ]---
[13960.286418] btrfs: unable to fixup at 67108864
[13960.286425] unable to find logical 274877906944 len 4096
regards,
Stephane
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
