Re: fio reports data corruption with btrfs

[Alex Lyakas <alex.bolshoy.btrfs@xxxxxxxxx>]

Hi Josef,
Mount options were noatime, nodatacow.
So you say that fio might have received ENOSPC, but didn't abort the test?

I will compile your branch and let you know.

I did not see any error messages from the kernel, except from:
Jun 25 10:04:28 vc kernel: [  436.730890] btrfs: setting nodatacow
Jun 25 10:04:28 vc kernel: [  436.744139] btrfs: no dev_stats entry
found for device /dev/sdb2 (devid 1) (OK on first mount after mkfs)
Jun 25 10:13:12 vc kernel: [  960.844149] INFO: task
flush-btrfs-2:3349 blocked for more than 120 seconds.
Jun 25 10:13:12 vc kernel: [  960.846600] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 25 10:13:12 vc kernel: [  960.847507] flush-btrfs-2   D
ffffffff8180ca80     0  3349      2 0x00000000
Jun 25 10:13:12 vc kernel: [  960.847515]  ffff8801186337a0
0000000000000046 0000000013e332ba ffffffff81c1d780
Jun 25 10:13:12 vc kernel: [  960.847520]  ffff880118633fd8
ffff880118633fd8 ffff880118633fd8 0000000000013840
Jun 25 10:13:12 vc kernel: [  960.847525]  ffffffff81c13020
ffff8801176f5b80 ffff880118633790 ffff88011fc140e8
Jun 25 10:13:12 vc kernel: [  960.847530] Call Trace:
Jun 25 10:13:12 vc kernel: [  960.847554]  [<ffffffff8166c239>]
schedule+0x29/0x70
Jun 25 10:13:12 vc kernel: [  960.847558]  [<ffffffff8166c30f>]
io_schedule+0x8f/0xd0
Jun 25 10:13:12 vc kernel: [  960.847574]  [<ffffffff812f0a3f>]
get_request_wait+0xef/0x240
Jun 25 10:13:12 vc kernel: [  960.847587]  [<ffffffff81073a80>] ?
add_wait_queue+0x60/0x60
Jun 25 10:13:12 vc kernel: [  960.847592]  [<ffffffff812f191f>]
blk_queue_bio+0x7f/0x3a0
Jun 25 10:13:12 vc kernel: [  960.847596]  [<ffffffff812ee784>]
generic_make_request.part.50+0x74/0xb0
Jun 25 10:13:12 vc kernel: [  960.847600]  [<ffffffff812eef18>]
generic_make_request+0x68/0x70
Jun 25 10:13:12 vc kernel: [  960.847603]  [<ffffffff812eefa7>]
submit_bio+0x87/0x110
Jun 25 10:13:12 vc kernel: [  960.847649]  [<ffffffffa006f8c7>]
btrfs_map_bio+0x167/0x210 [btrfs]
Jun 25 10:13:12 vc kernel: [  960.847669]  [<ffffffffa00428ad>]
btrfs_submit_bio_hook+0x7d/0x140 [btrfs]
Jun 25 10:13:12 vc kernel: [  960.847691]  [<ffffffffa00609fa>]
submit_one_bio+0x6a/0xa0 [btrfs]
Jun 25 10:13:12 vc kernel: [  960.847713]  [<ffffffffa0061059>]
flush_epd_write_bio+0x39/0x50 [btrfs]
Jun 25 10:13:12 vc kernel: [  960.847734]  [<ffffffffa00662c0>]
extent_writepages+0x50/0x60 [btrfs]
Jun 25 10:13:12 vc kernel: [  960.847754]  [<ffffffffa0045ba0>] ?
btrfs_submit_direct+0x1e0/0x1e0 [btrfs]
Jun 25 10:13:12 vc kernel: [  960.847759]  [<ffffffff81073654>] ?
bit_waitqueue+0x14/0xc0
Jun 25 10:13:12 vc kernel: [  960.847779]  [<ffffffffa00436d8>]
btrfs_writepages+0x28/0x30 [btrfs]
Jun 25 10:13:12 vc kernel: [  960.847793]  [<ffffffff81128191>]
do_writepages+0x21/0x40
Jun 25 10:13:12 vc kernel: [  960.847805]  [<ffffffff811a5462>]
writeback_single_inode+0x112/0x380
Jun 25 10:13:12 vc kernel: [  960.847809]  [<ffffffff811a5886>]
writeback_sb_inodes+0x1b6/0x270
Jun 25 10:13:12 vc kernel: [  960.847813]  [<ffffffff811a59de>]
__writeback_inodes_wb+0x9e/0xd0
Jun 25 10:13:12 vc kernel: [  960.847816]  [<ffffffff811a5c9b>]
wb_writeback+0x28b/0x340
Jun 25 10:13:12 vc kernel: [  960.847823]  [<ffffffff810125c7>] ?
__switch_to+0x137/0x410
Jun 25 10:13:12 vc kernel: [  960.847833]  [<ffffffff81197d02>] ?
get_nr_dirty_inodes+0x52/0x80
Jun 25 10:13:12 vc kernel: [  960.847837]  [<ffffffff811a5def>]
wb_check_old_data_flush+0x9f/0xb0
Jun 25 10:13:12 vc kernel: [  960.847842]  [<ffffffff811a72c9>]
wb_do_writeback+0x149/0x1d0
Jun 25 10:13:12 vc kernel: [  960.847848]  [<ffffffff8105f610>] ?
usleep_range+0x50/0x50
Jun 25 10:13:12 vc kernel: [  960.847852]  [<ffffffff811a73db>]
bdi_writeback_thread+0x8b/0x290
Jun 25 10:13:12 vc kernel: [  960.847855]  [<ffffffff811a7350>] ?
wb_do_writeback+0x1d0/0x1d0
Jun 25 10:13:12 vc kernel: [  960.847860]  [<ffffffff81072fe3>]
kthread+0x93/0xa0
Jun 25 10:13:12 vc kernel: [  960.847868]  [<ffffffff81676be4>]
kernel_thread_helper+0x4/0x10
Jun 25 10:13:12 vc kernel: [  960.847873]  [<ffffffff81072f50>] ?
kthread_freezable_should_stop+0x70/0x70
Jun 25 10:13:12 vc kernel: [  960.847877]  [<ffffffff81676be0>] ?
gs_change+0x13/0x13

Thanks,
Alex.



On Mon, Jun 25, 2012 at 10:26 PM, Josef Bacik <jbacik@xxxxxxxxxxxx> wrote:
> On Mon, Jun 25, 2012 at 12:30:34PM -0600, Alex Lyakas wrote:
>> Greetings everybody,
>>
>> I am running a fio test on btrfs compiled from
>> git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git,
>> up to commit:
>> cb77fcd88569cd2b7b25ecd4086ea932a53be9b3 Btrfs: delay iput with async extents
>> including this commit.
>>
>> Below is a fio configuration file, and later fio textual output.
>> Here:
>> https://docs.google.com/folder/d/0B1AuaIB8xZtbNTRuSW1zVGozWFE/edit
>> are "expected" vs "received" mismatch reports. Strangely, when I read
>> the mismatched block from the file reported as corrupted by fio, I
>> receive data different both from "expected" and "received" blocks that
>> fio reports. I added one such file (job0.1.0.88576.now) to the
>> pastebin as well.
>>
>> If you think that my fio configuration file is faulty, please let me
>> know. fio version is 1.59. The idea is to run 10 io processes in
>> parallel.
>>
>
> So we think it may be enospc, so try btrfs-next
>
> git://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git
>
> which has an enospc fix related to creating a crapptone of files.  Let me know
> if that helps.  Thanks,
>
> Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html