Re: raid10 make_request failure during iozone benchmark upon btrfs

[Kerin Millar <kerframil@xxxxxxxxx>]

Hi,

On 03/07/2012 02:39, NeilBrown wrote:

[snip]

>>> Could you please double check that you are running a kernel with
>>>
>>> commit aba336bd1d46d6b0404b06f6915ed76150739057
>>> Author: NeilBrown<neilb@xxxxxxx>
>>> Date:   Thu May 31 15:39:11 2012 +1000
>>>
>>>       md: raid1/raid10: fix problem with merge_bvec_fn
>>>
>>> in it?
>>
>> I am indeed. I searched the list beforehand and noticed the patch in
>> question. Not sure which -rc it landed in but I checked my source tree
>> and it's definitely in there.
>>
>> Cheers,
>>
>> --Kerin
>
> Thanks.
> Looking at it again I see that it is definitely a different bug, that patch
> wouldn't affect it.
>
> But I cannot see what could possibly be causing the problem.
> You have a 256K chunk size, so requests should be limited to 512 sectors
> aligned at a 512-sector boundary.
> However all the requests that a causing errors are 512 sectors long, but
> aligned on a 256-sector boundary (which is not also 512-sector).  This is
> wrong.

I see.

>
> It could be that btrfs is submitting bad requests, but I think it always uses
> bio_add_page, and bio_add_page appears to do the right thing.
> It could be that dm-linear is causing problem, but it seems to correctly after
> the underlying device for alignment, and reports that alignment to
> bio_add_page.
> It could be that md/raid10 is the problem but I cannot find any fault in
> raid10_mergeable_bvec - performs much the same tests that the
> raid01 make_request function does.
>
> So it is a mystery.
>
> Is this failure repeatable?

Yes, it's reproducible with 100% consistency. Furthermore, I tried to
use the btrfs volume as a store for the package manager, so as to try
with a 'realistic' workload. Many of these errors were triggered
immediately upon invoking the package manager. In case it matters, the
package manager is portage (in Gentoo Linux) and the directory structure
entails a shallow directory depth with a large number of distributed
small files. I haven't been able to reproduce with xfs, ext4 or reiserfs.

>
> If so, could you please insert
>     WARN_ON_ONCE(1);
> in drivers/md/raid10.c where it prints out the message: just after the
> "bad_map:" label.
>
> Also, in raid10_mergeable_bvec, insert
>     WARN_ON_ONCE(max<  0);
> just before
> 		if (max<  0)
> 			/* bio_add cannot handle a negative return */
> 			max = 0;
>
> and then see if either of those generate a warning, and post the full stack
> trace  if they do.

OK. I ran iozone again on a fresh filesystem, mounted with the default
options. Here's the trace that appears, just before the first
make_request_bug message:

WARNING: at drivers/md/raid10.c:1094 make_request+0xda5/0xe20()
Hardware name: ProLiant MicroServer
Modules linked in: btrfs zlib_deflate lzo_compress kvm_amd kvm sp5100_tco i2c_piix4
Pid: 1031, comm: btrfs-submit-1 Not tainted 3.5.0-rc5 #3
Call Trace:
[<ffffffff81031987>] ? warn_slowpath_common+0x67/0xa0
[<ffffffff81442b45>] ? make_request+0xda5/0xe20
[<ffffffff81460b34>] ? __split_and_process_bio+0x2d4/0x600
[<ffffffff81063429>] ? set_next_entity+0x29/0x60
[<ffffffff810652c3>] ? pick_next_task_fair+0x63/0x140
[<ffffffff81450b7f>] ? md_make_request+0xbf/0x1e0
[<ffffffff8123d12f>] ? generic_make_request+0xaf/0xe0
[<ffffffff8123d1c3>] ? submit_bio+0x63/0xe0
[<ffffffff81040abd>] ? try_to_del_timer_sync+0x7d/0x120
[<ffffffffa016839a>] ? run_scheduled_bios+0x23a/0x520 [btrfs]
[<ffffffffa0170e40>] ? worker_loop+0x120/0x520 [btrfs]
[<ffffffffa0170d20>] ? btrfs_queue_worker+0x2e0/0x2e0 [btrfs]
[<ffffffff810520c5>] ? kthread+0x85/0xa0
[<ffffffff815441f4>] ? kernel_thread_helper+0x4/0x10
[<ffffffff81052040>] ? kthread_freezable_should_stop+0x60/0x60
[<ffffffff815441f0>] ? gs_change+0xb/0xb

Cheers,

--Kerin
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html