Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Fri, 2012-07-13 at 06:47 -0400, Chris Mason wrote: 
> On Fri, Jul 13, 2012 at 04:26:26AM -0600, Thomas Gleixner wrote:
> > On Fri, 13 Jul 2012, Mike Galbraith wrote:
> > > On Fri, 2012-07-13 at 11:52 +0200, Thomas Gleixner wrote: 
> > > > On Fri, 13 Jul 2012, Mike Galbraith wrote:
> > > > > On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: 
> > > > > > Bingo, that makes it more likely that this is caused by copying w/o
> > > > > > initializing the lock and then freeing the original structure.
> > > > > > 
> > > > > > A quick check for memcpy finds that __btrfs_close_devices() does a
> > > > > > memcpy of btrfs_device structs w/o initializing the lock in the new
> > > > > > copy, but I have no idea whether that's the place we are looking for.
> > > > > 
> > > > > Thanks a bunch Thomas.  I doubt I would have ever figured out that lala
> > > > > land resulted from _copying_ a lock.  That's one I won't be forgetting
> > > > > any time soon.  Box not only survived a few thousand xfstests 006 runs,
> > > > > dbench seemed disinterested in deadlocking virgin 3.0-rt.
> > > > 
> > > > Cute. It think that the lock copying caused the deadlock problem as
> > > > the list pointed to the wrong place, so we might have ended up with
> > > > following down the wrong chain when walking the list as long as the
> > > > original struct was not freed. That beast is freed under RCU so there
> > > > could be a rcu read side critical section fiddling with the old lock
> > > > and cause utter confusion.
> > > 
> > > Virgin 3.0-rt appears to really be solid.  But then it doesn't have
> > > pesky rwlocks.
> > 
> > Ah. So 3.0 is not having those rwlock thingies. Bummer.
> >  
> > > > /me goes and writes a nastigram^W proper changelog
> > > > 
> > > > > btrfs still locks up in my enterprise kernel, so I suppose I had better
> > > > > plug your fix into 3.4-rt and see what happens, and go beat hell out of
> > > > > virgin 3.0-rt again to be sure box really really survives dbench.
> > > > 
> > > > A test against 3.4-rt sans enterprise mess might be nice as well.
> > > 
> > > Enterprise is 3.0-stable with um 555 btrfs patches (oh dear).
> > > 
> > > Virgin 3.4-rt and 3.2-rt deadlock gripe.  Enterprise doesn't gripe, but
> > > deadlocks, so I have another adventure in my future even if I figure out
> > > wth to do about rwlocks.
> > 
> > Hrmpf. /me goes to stare into fs/btrfs/ some more.
> 
> Please post the deadlocks here, I'll help ;)

This is the one from top of thread.  Below that is without the deadlock
detector.

[  692.857165] ============================================
[  692.863963] [ BUG: circular locking deadlock detected! ]
[  692.869264] Not tainted
[  692.871708] --------------------------------------------
[  692.877008] btrfs-delayed-m/1404 is deadlocking current task dbench/7937
[  692.877009] 
[  692.885183] 
[  692.885184] 1) dbench/7937 is trying to acquire this lock:
[  692.892149]  [ffff88014d6aea80] {&(&eb->lock)->lock}
[  692.897102] .. ->owner: ffff880175808501
[  692.901018] .. held by:   btrfs-delayed-m: 1404 [ffff880175808500, 120]
[  692.907657] 
[  692.907657] 2) btrfs-delayed-m/1404 is blocked on this lock:
[  692.914797]  [ffff88014bf58d60] {&(&eb->lock)->lock}
[  692.919751] .. ->owner: ffff880175186101
[  692.923672] .. held by:            dbench: 7937 [ffff880175186100, 120]
[  692.930309] 
[  692.930309] btrfs-delayed-m/1404's [blocked] stackdump:
[  692.930310] 
[  692.938504]  ffff880177575aa0 0000000000000046 ffff88014bf58d60 000000000000fb00
[  692.938507]  000000000000fb00 ffff880177575fd8 000000000000fb00 ffff880177574000
[  692.938509]  ffff880177575fd8 000000000000fb00 ffff88017662f240 ffff880175808500
[  692.960635] Call Trace:
[  692.963085]  [<ffffffff814c68e9>] schedule+0x29/0x90
[  692.963087]  [<ffffffff814c745d>] rt_spin_lock_slowlock+0xfd/0x330
[  692.963090]  [<ffffffff814c7a39>] __rt_spin_lock+0x9/0x10
[  692.963092]  [<ffffffff814c7b27>] rt_read_lock+0x27/0x40
[  692.963096]  [<ffffffff812550cf>] btrfs_clear_lock_blocking_rw+0x6f/0x180
[  692.963099]  [<ffffffff811fabd2>] btrfs_clear_path_blocking+0x32/0x70
[  692.963102]  [<ffffffff81200342>] btrfs_search_slot+0x6b2/0x810
[  692.963105]  [<ffffffff812148da>] btrfs_lookup_inode+0x2a/0xa0
[  692.963107]  [<ffffffff814c7312>] ? rt_mutex_lock+0x12/0x20
[  692.963111]  [<ffffffff8126d0bc>] btrfs_update_delayed_inode+0x6c/0x160
[  692.963113]  [<ffffffff814c7ab9>] ? _mutex_unlock+0x9/0x10
[  692.963116]  [<ffffffff8126e142>] btrfs_async_run_delayed_node_done+0x182/0x1a0
[  692.963119]  [<ffffffff8124ed5f>] worker_loop+0xaf/0x430
[  692.963121]  [<ffffffff8124ecb0>] ? btrfs_queue_worker+0x1e0/0x1e0
[  692.963123]  [<ffffffff8124ecb0>] ? btrfs_queue_worker+0x1e0/0x1e0
[  692.963127]  [<ffffffff8105850e>] kthread+0xae/0xc0
[  692.963129]  [<ffffffff814c68e9>] ? schedule+0x29/0x90
[  692.963133]  [<ffffffff810015bc>] ? __switch_to+0x14c/0x410
[  692.963137]  [<ffffffff81061e44>] ? finish_task_switch+0x54/0xd0
[  692.963140]  [<ffffffff814ceca4>] kernel_thread_helper+0x4/0x10
[  692.963143]  [<ffffffff81058460>] ? __init_kthread_worker+0x50/0x50
[  692.963145]  [<ffffffff814ceca0>] ? gs_change+0x13/0x13
[  692.963146] 
[  692.963147] dbench/7937's [current] stackdump:
[  692.963147] 
[  693.098724] Pid: 7937, comm: dbench Not tainted 3.4.4-rt13 #25
[  693.104544] Call Trace:
[  693.106993]  [<ffffffff8108b436>] debug_rt_mutex_print_deadlock+0x176/0x190
[  693.106995]  [<ffffffff814c74ec>] rt_spin_lock_slowlock+0x18c/0x330
[  693.106998]  [<ffffffff814c7a39>] __rt_spin_lock+0x9/0x10
[  693.107000]  [<ffffffff814c7b27>] rt_read_lock+0x27/0x40
[  693.107002]  [<ffffffff8125538c>] btrfs_try_tree_read_lock+0x4c/0x80
[  693.107004]  [<ffffffff812001bd>] btrfs_search_slot+0x52d/0x810
[  693.107007]  [<ffffffff812027ba>] btrfs_next_leaf+0xea/0x440
[  693.107010]  [<ffffffff81238ed8>] ? btrfs_token_dir_data_len+0x58/0xd0
[  693.107012]  [<ffffffff81238ed8>] ? btrfs_token_dir_data_len+0x58/0xd0
[  693.107016]  [<ffffffff81222e17>] btrfs_real_readdir+0x247/0x610
[  693.107020]  [<ffffffff81131350>] ? sys_ioctl+0xa0/0xa0
[  693.107022]  [<ffffffff81131350>] ? sys_ioctl+0xa0/0xa0
[  693.107024]  [<ffffffff81131660>] vfs_readdir+0xb0/0xd0
[  693.107026]  [<ffffffff81131840>] sys_getdents64+0x80/0xe0
[  693.107030]  [<ffffffff814cd9b9>] system_call_fastpath+0x16/0x1b
[  693.107032] [ turning off deadlock detection.Please report this trace. ]
[  693.107033] 


[  679.476016] SysRq : Show Blocked State
[  679.479781]   task                        PC stack   pid father
[  679.485708] btrfs-endio-wri D ffffffff81605920     0  1314      2 0x00000000
[  679.492785]  ffff880172939810 0000000000000046 ffff8801774ca538 000000000000f7c0
[  679.492789]  000000000000f7c0 ffff880172939fd8 ffff880172938000 000000000000f7c0
[  679.492792]  ffff880172939fd8 000000000000f7c0 ffff88017a4caa60 ffff8801744a88e0
[  679.514922] Call Trace:
[  679.517374]  [<ffffffff814ca569>] schedule+0x29/0x90
[  679.517378]  [<ffffffff814cb525>] rt_spin_lock_slowlock+0xd5/0x2bf
[  679.517382]  [<ffffffff814cb821>] __rt_spin_lock+0x21/0x30
[  679.517384]  [<ffffffff814cb927>] rt_read_lock+0x27/0x40
[  679.517388]  [<ffffffff81257d5d>] btrfs_tree_read_lock+0x5d/0x160
[  679.517392]  [<ffffffff812e46c6>] ? cpumask_next_and+0x36/0x50
[  679.517397]  [<ffffffff811fd863>] btrfs_read_lock_root_node+0x33/0x40
[  679.517400]  [<ffffffff81202919>] btrfs_search_slot+0x459/0x810
[  679.517404]  [<ffffffff8110dd60>] ? poison_obj+0x30/0x50
[  679.517409]  [<ffffffff81215957>] btrfs_lookup_file_extent+0x37/0x40
[  679.517412]  [<ffffffff811fd375>] ? btrfs_alloc_path+0x15/0x20
[  679.517417]  [<ffffffff81231bce>] btrfs_drop_extents+0xfe/0xa70
[  679.517421]  [<ffffffff814cf69d>] ? sub_preempt_count+0x9d/0xd0
[  679.517424]  [<ffffffff81066bb4>] ? migrate_enable+0xf4/0x1d0
[  679.517429]  [<ffffffff8110eb81>] ? cache_alloc_debugcheck_after+0x101/0x1a0
[  679.517432]  [<ffffffff81110545>] ? kmem_cache_alloc+0x1d5/0x200
[  679.517436]  [<ffffffff81224add>] insert_reserved_file_extent.clone.0+0x7d/0x270
[  679.517440]  [<ffffffff812218ab>] ? start_transaction+0x8b/0x290
[  679.517443]  [<ffffffff8122847e>] btrfs_finish_ordered_io+0x32e/0x3b0
[  679.517451]  [<ffffffff81047d1b>] ? try_to_del_timer_sync+0x6b/0xa0
[  679.517455]  [<ffffffff81228515>] btrfs_writepage_end_io_hook+0x15/0x20
[  679.517459]  [<ffffffff812446b4>] end_extent_writepage+0x64/0x100
[  679.517463]  [<ffffffff8124478b>] end_bio_extent_writepage+0x3b/0xa0
[  679.517468]  [<ffffffff81151af8>] bio_endio+0x18/0x30
[  679.517470]  [<ffffffff81219ba0>] end_workqueue_fn+0x40/0x50
[  679.517473]  [<ffffffff81251883>] worker_loop+0xc3/0x450
[  679.517476]  [<ffffffff814ca17f>] ? __schedule+0x2df/0x640
[  679.517480]  [<ffffffff812517c0>] ? btrfs_queue_worker+0x220/0x220
[  679.517483]  [<ffffffff812517c0>] ? btrfs_queue_worker+0x220/0x220
[  679.517486]  [<ffffffff81059586>] kthread+0x96/0xa0
[  679.517490]  [<ffffffff81062fb4>] ? finish_task_switch+0x54/0xd0
[  679.517494]  [<ffffffff814d2d24>] kernel_thread_helper+0x4/0x10
[  679.517498]  [<ffffffff810594f0>] ? __init_kthread_worker+0x50/0x50
[  679.517501]  [<ffffffff814d2d20>] ? gs_change+0x13/0x13
[  679.517509] btrfs-transacti D ffffffff81605920     0  1320      2 0x00000000
[  679.725747]  ffff880172d07a50 0000000000000046 ffff8801774ca538 000000000000f7c0
[  679.725750]  000000000000f7c0 ffff880172d07fd8 ffff880172d06000 000000000000f7c0
[  679.725753]  ffff880172d07fd8 000000000000f7c0 ffff88017a4929e0 ffff880176320920
[  679.747879] Call Trace:
[  679.750326]  [<ffffffff814ca569>] schedule+0x29/0x90
[  679.750328]  [<ffffffff814cb525>] rt_spin_lock_slowlock+0xd5/0x2bf
[  679.750331]  [<ffffffff814cb821>] __rt_spin_lock+0x21/0x30
[  679.750333]  [<ffffffff814cb927>] rt_read_lock+0x27/0x40
[  679.750335]  [<ffffffff81257d5d>] btrfs_tree_read_lock+0x5d/0x160
[  679.750338]  [<ffffffff814cf69d>] ? sub_preempt_count+0x9d/0xd0
[  679.750340]  [<ffffffff81066bb4>] ? migrate_enable+0xf4/0x1d0
[  679.750343]  [<ffffffff811fd863>] btrfs_read_lock_root_node+0x33/0x40
[  679.750345]  [<ffffffff81202919>] btrfs_search_slot+0x459/0x810
[  679.750349]  [<ffffffff81270d13>] btrfs_delete_delayed_items+0x63/0x100
[  679.750352]  [<ffffffff812710d2>] btrfs_run_delayed_items+0x112/0x160
[  679.750355]  [<ffffffff81220da2>] btrfs_commit_transaction+0x322/0xa70
[  679.750357]  [<ffffffff8122021a>] ? join_transaction+0x35a/0x3a0
[  679.750360]  [<ffffffff81059d30>] ? wake_up_bit+0x40/0x40
[  679.750362]  [<ffffffff81219a33>] transaction_kthread+0x273/0x2f0
[  679.750364]  [<ffffffff812197c0>] ? btrfs_congested_fn+0xb0/0xb0
[  679.750367]  [<ffffffff812197c0>] ? btrfs_congested_fn+0xb0/0xb0
[  679.750369]  [<ffffffff81059586>] kthread+0x96/0xa0
[  679.750371]  [<ffffffff81062fb4>] ? finish_task_switch+0x54/0xd0
[  679.750374]  [<ffffffff814d2d24>] kernel_thread_helper+0x4/0x10
[  679.750377]  [<ffffffff810594f0>] ? __init_kthread_worker+0x50/0x50
[  679.750379]  [<ffffffff814d2d20>] ? gs_change+0x13/0x13
[  679.750401] dbench          D ffffffff81605920     0  7812      1 0x00000004
[  679.886585]  ffff880174d99a98 0000000000000086 000000000000b380 000000000000f7c0
[  679.886587]  000000000000f7c0 ffff880174d99fd8 ffff880174d98000 000000000000f7c0
[  679.886590]  ffff880174d99fd8 000000000000f7c0 ffffffff81816020 ffff8801738be700
[  679.908712] Call Trace:
[  679.911158]  [<ffffffff810c87e0>] ? __lock_page+0x70/0x70
[  679.911160]  [<ffffffff814ca569>] schedule+0x29/0x90
[  679.911163]  [<ffffffff814ca657>] io_schedule+0x87/0xd0
[  679.911165]  [<ffffffff810c87e9>] sleep_on_page+0x9/0x10
[  679.911167]  [<ffffffff814c97a7>] __wait_on_bit+0x57/0x80
[  679.911170]  [<ffffffff810c983f>] ? find_get_pages_tag+0xcf/0x190
[  679.911172]  [<ffffffff810c8a0e>] wait_on_page_bit+0x6e/0x80
[  679.911175]  [<ffffffff81059d70>] ? autoremove_wake_function+0x40/0x40
[  679.911177]  [<ffffffff810d58b0>] ? pagevec_lookup_tag+0x20/0x30
[  679.911180]  [<ffffffff810c900e>] filemap_fdatawait_range+0xee/0x190
[  679.911183]  [<ffffffff812460bc>] ? extent_writepages+0x4c/0x60
[  679.911185]  [<ffffffff81226190>] ? btrfs_submit_direct+0x1d0/0x1d0
[  679.911188]  [<ffffffff811113e6>] ? kfree+0x1a6/0x2e0
[  679.911190]  [<ffffffff812240b2>] ? btrfs_writepages+0x22/0x30
[  679.911192]  [<ffffffff810d4dbf>] ? do_writepages+0x1f/0x40
[  679.911195]  [<ffffffff810c9e30>] filemap_write_and_wait_range+0x70/0x80
[  679.911198]  [<ffffffff81230587>] btrfs_sync_file+0x37/0x1b0
[  679.911201]  [<ffffffff8114ba50>] generic_write_sync+0x50/0x70
[  679.911203]  [<ffffffff812316dc>] btrfs_file_aio_write+0x31c/0x370
[  679.911207]  [<ffffffff811208da>] do_sync_write+0xda/0x120
[  679.911210]  [<ffffffff810ec752>] ? handle_mm_fault+0x162/0x220
[  679.911213]  [<ffffffff8104bc99>] ? kill_something_info+0x49/0x160
[  679.911216]  [<ffffffff812b45a3>] ? apparmor_file_permission+0x13/0x20
[  679.911219]  [<ffffffff8128ef27>] ? security_file_permission+0x27/0xb0
[  679.911222]  [<ffffffff811211c6>] vfs_write+0xc6/0x180
[  679.911224]  [<ffffffff81121672>] sys_pwrite64+0xa2/0xb0
[  679.911227]  [<ffffffff814d1a39>] system_call_fastpath+0x16/0x1b
[  679.911229] dbench          D ffffffff81605920     0  7813      1 0x00000004
[  680.074937]  ffff880174445908 0000000000000086 ffff8801744458a8 000000000000f7c0
[  680.074940]  000000000000f7c0 ffff880174445fd8 ffff880174444000 000000000000f7c0
[  680.074942]  ffff880174445fd8 000000000000f7c0 ffff88017a4e4ae0 ffff880174eb62c0
[  680.097069] Call Trace:
[  680.099514]  [<ffffffff814ca569>] schedule+0x29/0x90
[  680.099516]  [<ffffffff814cb525>] rt_spin_lock_slowlock+0xd5/0x2bf
[  680.099519]  [<ffffffff810d083a>] ? prep_new_page+0x12a/0x190
[  680.099521]  [<ffffffff814cb821>] __rt_spin_lock+0x21/0x30
[  680.099523]  [<ffffffff814cb927>] rt_read_lock+0x27/0x40
[  680.099525]  [<ffffffff81257d5d>] btrfs_tree_read_lock+0x5d/0x160
[  680.099529]  [<ffffffff812c21c6>] ? chksum_update+0x16/0x30
[  680.099531]  [<ffffffff811fd863>] btrfs_read_lock_root_node+0x33/0x40
[  680.099533]  [<ffffffff81202919>] btrfs_search_slot+0x459/0x810
[  680.099536]  [<ffffffff812148da>] btrfs_lookup_xattr+0x7a/0xc0
[  680.099539]  [<ffffffff8123e447>] __btrfs_getxattr+0x77/0x150
[  680.099542]  [<ffffffff8123e9cd>] btrfs_getxattr+0x7d/0x80
[  680.099544]  [<ffffffff8128d618>] cap_inode_need_killpriv+0x28/0x40
[  680.099547]  [<ffffffff8128ee41>] security_inode_need_killpriv+0x11/0x20
[  680.099549]  [<ffffffff810c85cb>] file_remove_suid+0x4b/0xc0
[  680.099551]  [<ffffffff810c88b5>] ? unlock_page+0x25/0x30
[  680.099556]  [<ffffffff810e7931>] ? __do_fault+0x431/0x530
[  680.099559]  [<ffffffff81231550>] btrfs_file_aio_write+0x190/0x370
[  680.099562]  [<ffffffff810eb437>] ? handle_pte_fault+0xe7/0x200
[  680.099565]  [<ffffffff811208da>] do_sync_write+0xda/0x120
[  680.099567]  [<ffffffff810ec752>] ? handle_mm_fault+0x162/0x220
[  680.099570]  [<ffffffff8104bc99>] ? kill_something_info+0x49/0x160
[  680.099572]  [<ffffffff812b45a3>] ? apparmor_file_permission+0x13/0x20
[  680.099574]  [<ffffffff8128ef27>] ? security_file_permission+0x27/0xb0
[  680.099577]  [<ffffffff811211c6>] vfs_write+0xc6/0x180
[  680.099579]  [<ffffffff81121672>] sys_pwrite64+0xa2/0xb0
[  680.099582]  [<ffffffff814d1a39>] system_call_fastpath+0x16/0x1b
[  680.099584] dbench          D ffffffff81605920     0  7814      1 0x00000004
[  680.263730]  ffff880176169be8 0000000000000086 ffff88017759a9b8 000000000000f7c0
[  680.263733]  000000000000f7c0 ffff880176169fd8 ffff880176168000 000000000000f7c0
[  680.263735]  ffff880176169fd8 000000000000f7c0 ffff88017a476960 ffff8801756ec1c0
[  680.285863] Call Trace:
[  680.288309]  [<ffffffff814ca569>] schedule+0x29/0x90
[  680.288311]  [<ffffffff814cb525>] rt_spin_lock_slowlock+0xd5/0x2bf
[  680.288314]  [<ffffffff814cb821>] __rt_spin_lock+0x21/0x30
[  680.288316]  [<ffffffff814cb927>] rt_read_lock+0x27/0x40
[  680.288319]  [<ffffffff81257eac>] btrfs_try_tree_read_lock+0x4c/0x80
[  680.288321]  [<ffffffff812029ed>] btrfs_search_slot+0x52d/0x810
[  680.288324]  [<ffffffff812256bf>] btrfs_real_readdir+0x1af/0x5f0
[  680.288326]  [<ffffffff81133690>] ? sys_ioctl+0xa0/0xa0
[  680.288329]  [<ffffffff81133690>] ? sys_ioctl+0xa0/0xa0
[  680.288331]  [<ffffffff811339a0>] vfs_readdir+0xb0/0xd0
[  680.288333]  [<ffffffff81133b80>] sys_getdents64+0x80/0xe0
[  680.288336]  [<ffffffff814d1a39>] system_call_fastpath+0x16/0x1b
[  680.288338] dbench          D ffffffff81605920     0  7815      1 0x00000004
[  680.362625]  ffff880174ea7ad8 0000000000000086 ffff88017759abf8 000000000000f7c0
[  680.362627]  000000000000f7c0 ffff880174ea7fd8 ffff880174ea6000 000000000000f7c0
[  680.362630]  ffff880174ea7fd8 000000000000f7c0 ffff88017a4fab60 ffff880174d5ca60
[  680.384756] Call Trace:
[  680.387202]  [<ffffffff814ca569>] schedule+0x29/0x90
[  680.387204]  [<ffffffff814cb525>] rt_spin_lock_slowlock+0xd5/0x2bf
[  680.387207]  [<ffffffff814cb821>] __rt_spin_lock+0x21/0x30
[  680.387209]  [<ffffffff814cb927>] rt_read_lock+0x27/0x40
[  680.387211]  [<ffffffff81257eac>] btrfs_try_tree_read_lock+0x4c/0x80
[  680.387214]  [<ffffffff812029ed>] btrfs_search_slot+0x52d/0x810
[  680.387216]  [<ffffffff812148da>] btrfs_lookup_xattr+0x7a/0xc0
[  680.387219]  [<ffffffff8123e447>] __btrfs_getxattr+0x77/0x150
[  680.387222]  [<ffffffff814cf69d>] ? sub_preempt_count+0x9d/0xd0
[  680.387225]  [<ffffffff8127b060>] btrfs_get_acl+0xf0/0x250
[  680.387227]  [<ffffffff8122d33d>] ? btrfs_new_inode+0x2bd/0x360
[  680.387230]  [<ffffffff8127b2e1>] btrfs_init_acl+0x81/0x150
[  680.387232]  [<ffffffff8122258c>] btrfs_init_inode_security+0x2c/0x60
[  680.387235]  [<ffffffff8122ea51>] btrfs_mkdir+0x121/0x1f0
[  680.387237]  [<ffffffff8112dfc8>] vfs_mkdir+0xb8/0x130
[  680.387240]  [<ffffffff81131273>] sys_mkdirat+0xf3/0x100
[  680.387242]  [<ffffffff81131294>] sys_mkdir+0x14/0x20
[  680.387244]  [<ffffffff814d1a39>] system_call_fastpath+0x16/0x1b
[  680.387246] dbench          D ffffffff81605920     0  7818      1 0x00000004
[  680.496083]  ffff8801790c98e8 0000000000000082 ffff8801790c9888 000000000000f7c0
[  680.496086]  000000000000f7c0 ffff8801790c9fd8 ffff8801790c8000 000000000000f7c0
[  680.496088]  ffff8801790c9fd8 000000000000f7c0 ffff88017a536be0 ffff880176072700
[  680.518214] Call Trace:
[  680.520658]  [<ffffffff814ca569>] schedule+0x29/0x90
[  680.520660]  [<ffffffff814cb525>] rt_spin_lock_slowlock+0xd5/0x2bf
[  680.520663]  [<ffffffff814cb821>] ? __rt_spin_lock+0x21/0x30
[  680.520665]  [<ffffffff814cb821>] __rt_spin_lock+0x21/0x30
[  680.520667]  [<ffffffff814cb927>] rt_read_lock+0x27/0x40
[  680.520669]  [<ffffffff81257bef>] btrfs_clear_lock_blocking_rw+0x6f/0x180
[  680.520672]  [<ffffffff811fd422>] btrfs_clear_path_blocking+0x32/0x70
[  680.520674]  [<ffffffff81202ba2>] btrfs_search_slot+0x6e2/0x810
[  680.520676]  [<ffffffff812c21c6>] ? chksum_update+0x16/0x30
[  680.520679]  [<ffffffff81214bd6>] btrfs_lookup_dir_item+0x76/0xc0
[  680.520682]  [<ffffffff8122c8db>] btrfs_lookup_dentry+0x9b/0x370
[  680.520685]  [<ffffffff810659c1>] ? get_parent_ip+0x11/0x50
[  680.520687]  [<ffffffff814cf69d>] ? sub_preempt_count+0x9d/0xd0
[  680.520690]  [<ffffffff8122cbc8>] btrfs_lookup+0x18/0x70
[  680.520693]  [<ffffffff8112bba8>] __lookup_hash+0x58/0x120
[  680.520695]  [<ffffffff8112e59f>] do_lookup+0x2af/0x330
[  680.520697]  [<ffffffff8112f174>] path_lookupat+0x134/0x750
[  680.520700]  [<ffffffff8110eb10>] ? cache_alloc_debugcheck_after+0x90/0x1a0
[  680.520702]  [<ffffffff8111048a>] ? kmem_cache_alloc+0x11a/0x200
[  680.520705]  [<ffffffff8112f7bc>] do_path_lookup+0x2c/0xc0
[  680.520707]  [<ffffffff810659c1>] ? get_parent_ip+0x11/0x50
[  680.520709]  [<ffffffff81130d74>] user_path_at_empty+0x54/0xa0
[  680.520712]  [<ffffffff81049380>] ? check_kill_permission+0x100/0x190
[  680.520714]  [<ffffffff8104baad>] ? group_send_sig_info+0x3d/0x80
[  680.520716]  [<ffffffff8104bc23>] ? kill_pid_info+0x53/0x80
[  680.520718]  [<ffffffff81130dcc>] user_path_at+0xc/0x10
[  680.520721]  [<ffffffff81125f1f>] vfs_fstatat+0x3f/0x80
[  680.520723]  [<ffffffff81125f96>] vfs_stat+0x16/0x20
[  680.520725]  [<ffffffff811260df>] sys_newstat+0x1f/0x40
[  680.520727]  [<ffffffff81122d6d>] ? fput+0x1d/0x30
[  680.520730]  [<ffffffff8111edd1>] ? filp_close+0x61/0x90
[  680.520732]  [<ffffffff8111eea8>] ? sys_close+0xa8/0x110
[  680.520735]  [<ffffffff814d1a39>] system_call_fastpath+0x16/0x1b
[  680.520737] rm              D ffffffff81605920     0  7820      1 0x00000004
[  680.714842]  ffff880176411a08 0000000000000086 ffff8801774ca538 000000000000f7c0
[  680.714844]  000000000000f7c0 ffff880176411fd8 ffff880176410000 000000000000f7c0
[  680.714847]  ffff880176411fd8 000000000000f7c0 ffff88017a4e4ae0 ffff880175892d20
[  680.736972] Call Trace:
[  680.739420]  [<ffffffff814ca569>] schedule+0x29/0x90
[  680.739422]  [<ffffffff814cb525>] rt_spin_lock_slowlock+0xd5/0x2bf
[  680.739426]  [<ffffffff810659c1>] ? get_parent_ip+0x11/0x50
[  680.739428]  [<ffffffff814cb821>] __rt_spin_lock+0x21/0x30
[  680.739430]  [<ffffffff814cb927>] rt_read_lock+0x27/0x40
[  680.739432]  [<ffffffff81257d5d>] btrfs_tree_read_lock+0x5d/0x160
[  680.739435]  [<ffffffff81246a34>] ? free_extent_buffer+0x34/0x70
[  680.739437]  [<ffffffff812002f1>] ? read_block_for_search+0x161/0x210
[  680.739440]  [<ffffffff811fd863>] btrfs_read_lock_root_node+0x33/0x40
[  680.739442]  [<ffffffff81202919>] btrfs_search_slot+0x459/0x810
[  680.739445]  [<ffffffff8121716a>] btrfs_lookup_inode+0x2a/0xa0
[  680.739448]  [<ffffffff81224e1e>] btrfs_read_locked_inode+0x7e/0x3a0
[  680.739450]  [<ffffffff81223eb0>] ? btrfs_permission+0x60/0x60
[  680.739453]  [<ffffffff8122c33f>] btrfs_iget+0x9f/0x100
[  680.739455]  [<ffffffff8122cad0>] btrfs_lookup_dentry+0x290/0x370
[  680.739458]  [<ffffffff8122cbc8>] btrfs_lookup+0x18/0x70
[  680.739461]  [<ffffffff8112bba8>] __lookup_hash+0x58/0x120
[  680.739463]  [<ffffffff8112faf7>] ? user_path_parent+0x47/0x80
[  680.739466]  [<ffffffff8112bc84>] lookup_hash+0x14/0x20
[  680.739468]  [<ffffffff8112fbc6>] do_unlinkat+0x96/0x1d0
[  680.739470]  [<ffffffff81133987>] ? vfs_readdir+0x97/0xd0
[  680.739472]  [<ffffffff81133baf>] ? sys_getdents64+0xaf/0xe0
[  680.739474]  [<ffffffff811312dd>] sys_unlinkat+0x1d/0x40
[  680.739477]  [<ffffffff814d1a39>] system_call_fastpath+0x16/0x1b
[  680.739479] rm              D ffffffff81605920     0  7822      1 0x00000004
[  680.883473]  ffff880172e0fa28 0000000000000086 ffff8801774ca538 000000000000f7c0
[  680.883475]  000000000000f7c0 ffff880172e0ffd8 ffff880172e0e000 000000000000f7c0
[  680.883478]  ffff880172e0ffd8 000000000000f7c0 ffff88017a476960 ffff8801791a4500
[  680.905605] Call Trace:
[  680.908050]  [<ffffffff814ca569>] schedule+0x29/0x90
[  680.908052]  [<ffffffff814cb525>] rt_spin_lock_slowlock+0xd5/0x2bf
[  680.908055]  [<ffffffff814cb821>] __rt_spin_lock+0x21/0x30
[  680.908057]  [<ffffffff814cb927>] rt_read_lock+0x27/0x40
[  680.908059]  [<ffffffff81257d5d>] btrfs_tree_read_lock+0x5d/0x160
[  680.908062]  [<ffffffff814cf69d>] ? sub_preempt_count+0x9d/0xd0
[  680.908065]  [<ffffffff811fd863>] btrfs_read_lock_root_node+0x33/0x40
[  680.908067]  [<ffffffff81202919>] btrfs_search_slot+0x459/0x810
[  680.908069]  [<ffffffff81066bb4>] ? migrate_enable+0xf4/0x1d0
[  680.908072]  [<ffffffff8122aefc>] btrfs_truncate_inode_items+0x13c/0x880
[  680.908074]  [<ffffffff814cf69d>] ? sub_preempt_count+0x9d/0xd0
[  680.908077]  [<ffffffff810659c1>] ? get_parent_ip+0x11/0x50
[  680.908079]  [<ffffffff814cf69d>] ? sub_preempt_count+0x9d/0xd0
[  680.908082]  [<ffffffff812218ab>] ? start_transaction+0x8b/0x290
[  680.908085]  [<ffffffff8122bf54>] btrfs_evict_inode+0x194/0x2c0
[  680.908087]  [<ffffffff814cf69d>] ? sub_preempt_count+0x9d/0xd0
[  680.908090]  [<ffffffff8113b875>] evict+0xb5/0x1d0
[  680.908092]  [<ffffffff8113ba83>] iput_final+0xf3/0x220
[  680.908095]  [<ffffffff8113bbe9>] iput+0x39/0x50
[  680.908097]  [<ffffffff8112fc87>] do_unlinkat+0x157/0x1d0
[  680.908099]  [<ffffffff81133987>] ? vfs_readdir+0x97/0xd0
[  680.908101]  [<ffffffff81133baf>] ? sys_getdents64+0xaf/0xe0
[  680.908104]  [<ffffffff811312dd>] sys_unlinkat+0x1d/0x40
[  680.908106]  [<ffffffff814d1a39>] system_call_fastpath+0x16/0x1b
[  680.908108] rm              D ffffffff81605920     0  7827      1 0x00000004
[  681.051241]  ffff880172dffa98 0000000000000086 ffff8801774ca538 000000000000f7c0
[  681.051244]  000000000000f7c0 ffff880172dfffd8 ffff880172dfe000 000000000000f7c0
[  681.051246]  ffff880172dfffd8 000000000000f7c0 ffff88017a476960 ffff8801761e4340
[  681.073378] Call Trace:
[  681.075823]  [<ffffffff814ca569>] schedule+0x29/0x90
[  681.075825]  [<ffffffff814cb525>] rt_spin_lock_slowlock+0xd5/0x2bf
[  681.075827]  [<ffffffff814cb821>] __rt_spin_lock+0x21/0x30
[  681.075829]  [<ffffffff814cb927>] rt_read_lock+0x27/0x40
[  681.075832]  [<ffffffff81257d5d>] btrfs_tree_read_lock+0x5d/0x160
[  681.075834]  [<ffffffff810659c1>] ? get_parent_ip+0x11/0x50
[  681.075837]  [<ffffffff814cf69d>] ? sub_preempt_count+0x9d/0xd0
[  681.075840]  [<ffffffff811fd863>] btrfs_read_lock_root_node+0x33/0x40
[  681.075842]  [<ffffffff81202919>] btrfs_search_slot+0x459/0x810
[  681.075844]  [<ffffffff812c21c6>] ? chksum_update+0x16/0x30
[  681.075847]  [<ffffffff81214bd6>] btrfs_lookup_dir_item+0x76/0xc0
[  681.075850]  [<ffffffff8122c8db>] btrfs_lookup_dentry+0x9b/0x370
[  681.075852]  [<ffffffff810659c1>] ? get_parent_ip+0x11/0x50
[  681.075855]  [<ffffffff814cf69d>] ? sub_preempt_count+0x9d/0xd0
[  681.075857]  [<ffffffff8122cbc8>] btrfs_lookup+0x18/0x70
[  681.075860]  [<ffffffff8112bba8>] __lookup_hash+0x58/0x120
[  681.075862]  [<ffffffff8112faf7>] ? user_path_parent+0x47/0x80
[  681.075864]  [<ffffffff8112bc84>] lookup_hash+0x14/0x20
[  681.075866]  [<ffffffff8112fbc6>] do_unlinkat+0x96/0x1d0
[  681.075869]  [<ffffffff81133987>] ? vfs_readdir+0x97/0xd0
[  681.075871]  [<ffffffff81133baf>] ? sys_getdents64+0xaf/0xe0
[  681.075873]  [<ffffffff811312dd>] sys_unlinkat+0x1d/0x40
[  681.075875]  [<ffffffff814d1a39>] system_call_fastpath+0x16/0x1b
[  681.075878] Sched Debug Version: v0.10, 3.4.4-rt13 #37


--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[Index of Archives]     [Linux Ext4 Filesystem]     [Union Filesystem]     [Filesystem Testing]     [Ceph Users]     [Ecryptfs]     [AutoFS]     [Kernel Newbies]     [Share Photos]     [Security]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux Cachefs]     [Reiser Filesystem]     [Linux RAID]     [Samba]     [Device Mapper]     [CEPH Development]
  Powered by Linux