Having an issue with CentOS7.1 + brtfs w/ docker where /var/lib/docker is btrfs This appears to be an issue related to brtfs so posting here. We have not seen the stack traces (below) on a similar server upgraded to kernel 4.1.1, but since 3.10.0-229 is LTS, we wanted to report it to get it patched in the distribution. We've tried the same set up on kernel 4.1.1 but we also get uninterruptible processes. In this case, we did not observe any call traces from the kernel in dmesg or journalctl. On that machine, we attempted a full PS, but only got this: [root@eg-mesos-jenkins-003 ~]# ps aux | grep D USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 238 0.0 0.0 0 0 ? DN Jul02 0:45 [khugepaged] Below is the relevant system info for the base kernel machine plus the dmesg.log as an attachment. % cat /etc/redhat-release CentOS Linux release 7.1.1503 (Core) % uname -a Linux server-001 3.10.0-229.4.2.el7.x86_64 #1 SMP Wed May 13 10:06:09 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux % btrfs --version Btrfs v3.16.2 % sudo btrfs fi show Label: 'docker' uuid: 4d41939a-099d-4868-b692-c62ddf8eb1b2 Total devices 1 FS bytes used 15.14GiB devid 1 size 1.07TiB used 71.04GiB path /dev/sda5 Btrfs v3.16.2 % sudo btrfs fi df /var/lib/docker Data, single: total=62.01GiB, used=14.61GiB System, DUP: total=8.00MiB, used=16.00KiB System, single: total=4.00MiB, used=0.00 Metadata, DUP: total=4.50GiB, used=534.55MiB Metadata, single: total=8.00MiB, used=0.00 GlobalReserve, single: total=192.00MiB, used=0.00 Call Trace: Jul 06 23:40:35 server-001 kernel: INFO: task kworker/u65:9:31973 blocked for more than 120 seconds. Jul 06 23:40:35 server-001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 06 23:40:35 server-001 kernel: kworker/u65:9 D ffff881fffdb3680 0 31973 2 0x00000080 Jul 06 23:40:35 server-001 kernel: Workqueue: writeback bdi_writeback_workfn (flush-btrfs-1) Jul 06 23:40:35 server-001 kernel: ffff8816e1a87738 0000000000000046 ffff8816e1a87fd8 0000000000013680 Jul 06 23:40:36 server-001 kernel: ffff8816e1a87fd8 0000000000013680 ffff8810e1b7cfa0 ffff881fffdb3f48 Jul 06 23:40:36 server-001 kernel: ffff8816e1a877c0 0000000000000002 ffffffff81156330 ffff8816e1a877b0 Jul 06 23:40:36 server-001 kernel: Call Trace: Jul 06 23:40:36 server-001 kernel: [<ffffffff81156330>] ? wait_on_page_read+0x60/0x60 Jul 06 23:40:36 server-001 kernel: [<ffffffff8160a4dd>] io_schedule+0x9d/0x140 Jul 06 23:40:36 server-001 kernel: [<ffffffff8115633e>] sleep_on_page+0xe/0x20 Jul 06 23:40:36 server-001 kernel: [<ffffffff816083db>] __wait_on_bit_lock+0x5b/0xc0 Jul 06 23:40:36 server-001 kernel: [<ffffffff81156458>] __lock_page+0x78/0xa0 Jul 06 23:40:36 server-001 kernel: [<ffffffff81098390>] ? autoremove_wake_function+0x40/0x40 Jul 06 23:40:36 server-001 kernel: [<ffffffffa07fe715>] lock_delalloc_pages+0x1e5/0x1f0 [btrfs] Jul 06 23:40:36 server-001 kernel: [<ffffffffa0800f13>] find_lock_delalloc_range.constprop.43+0x153/0x200 [btrfs] Jul 06 23:40:36 server-001 kernel: [<ffffffffa080104b>] writepage_delalloc.isra.33+0x8b/0x180 [btrfs] Jul 06 23:40:36 server-001 kernel: [<ffffffffa0801cba>] __extent_writepage+0xca/0x2b0 [btrfs] Jul 06 23:40:36 server-001 kernel: [<ffffffffa08021ea>] extent_write_cache_pages.isra.28.constprop.48+0x34a/0x420 [btrfs] Jul 06 23:40:37 server-001 kernel: [<ffffffffa08040dc>] extent_writepages+0x5c/0x90 [btrfs] Jul 06 23:40:37 server-001 kernel: [<ffffffffa07e6e30>] ? btrfs_submit_direct+0x6c0/0x6c0 [btrfs] Jul 06 23:40:37 server-001 kernel: [<ffffffffa07e4738>] btrfs_writepages+0x28/0x30 [btrfs] Jul 06 23:40:37 server-001 kernel: [<ffffffff81162fae>] do_writepages+0x1e/0x40 Jul 06 23:40:37 server-001 kernel: [<ffffffff811f0670>] __writeback_single_inode+0x40/0x220 Jul 06 23:40:37 server-001 kernel: [<ffffffff811f136e>] writeback_sb_inodes+0x25e/0x420 Jul 06 23:40:37 server-001 kernel: [<ffffffff811f15cf>] __writeback_inodes_wb+0x9f/0xd0 Jul 06 23:40:37 server-001 kernel: [<ffffffff811f1e13>] wb_writeback+0x263/0x2f0 Jul 06 23:40:37 server-001 kernel: [<ffffffff811f32a5>] bdi_writeback_workfn+0x115/0x460 Jul 06 23:40:37 server-001 kernel: [<ffffffff8108f1eb>] process_one_work+0x17b/0x470 Jul 06 23:40:37 server-001 kernel: [<ffffffff8108ffbb>] worker_thread+0x11b/0x400 Jul 06 23:40:37 server-001 kernel: [<ffffffff8108fea0>] ? rescuer_thread+0x400/0x400 Jul 06 23:40:37 server-001 kernel: [<ffffffff8109739f>] kthread+0xcf/0xe0 Jul 06 23:40:37 server-001 kernel: [<ffffffff810972d0>] ? kthread_create_on_node+0x140/0x140 Jul 06 23:40:38 server-001 kernel: [<ffffffff81614d3c>] ret_from_fork+0x7c/0xb0 Jul 06 23:40:38 server-001 kernel: [<ffffffff810972d0>] ? kthread_create_on_node+0x140/0x140 Jul 06 23:40:38 server-001 kernel: INFO: task git:8697 blocked for more than 120 seconds. Jul 06 23:40:38 server-001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 06 23:40:38 server-001 kernel: git D ffff881fffc93680 0 8697 8695 0x00000084 Jul 06 23:40:38 server-001 kernel: ffff881d50c43498 0000000000000082 ffff881d50c43fd8 0000000000013680 Jul 06 23:40:38 server-001 kernel: ffff881d50c43fd8 0000000000013680 ffff883488958b60 ffff881fffc93f48 Jul 06 23:40:38 server-001 kernel: ffff88207ffa6ee8 0000000000000002 ffffffff81156330 ffff881d50c43510 Jul 06 23:40:38 server-001 kernel: Call Trace: Jul 06 23:40:38 server-001 kernel: [<ffffffff81156330>] ? wait_on_page_read+0x60/0x60 Jul 06 23:40:38 server-001 kernel: [<ffffffff8160a4dd>] io_schedule+0x9d/0x140 Jul 06 23:40:38 server-001 kernel: [<ffffffff8115633e>] sleep_on_page+0xe/0x20 Jul 06 23:40:38 server-001 kernel: [<ffffffff816082a0>] __wait_on_bit+0x60/0x90 Jul 06 23:40:38 server-001 kernel: [<ffffffff811560c6>] wait_on_page_bit+0x86/0xb0 Jul 06 23:40:39 server-001 kernel: [<ffffffff81098390>] ? autoremove_wake_function+0x40/0x40 Jul 06 23:40:39 server-001 kernel: [<ffffffff8116a1b2>] shrink_page_list+0x6c2/0xad0 Jul 06 23:40:39 server-001 kernel: [<ffffffff813f9b80>] ? scsi_request_fn+0x50/0x570 Jul 06 23:40:39 server-001 kernel: [<ffffffff8116ac7a>] shrink_inactive_list+0x1ea/0x560 Jul 06 23:40:39 server-001 kernel: [<ffffffff8116b73d>] shrink_lruvec+0x36d/0x730 Jul 06 23:40:39 server-001 kernel: [<ffffffff8116bb76>] shrink_zone+0x76/0x1a0 Jul 06 23:40:39 server-001 kernel: [<ffffffff8116c080>] do_try_to_free_pages+0xf0/0x4e0 Jul 06 23:40:39 server-001 kernel: [<ffffffff8115d90a>] ? __rmqueue+0x8a/0x460 Jul 06 23:40:39 server-001 kernel: [<ffffffff8116c6ba>] try_to_free_mem_cgroup_pages+0xca/0x160 Jul 06 23:40:39 server-001 kernel: [<ffffffff811bc9ce>] mem_cgroup_reclaim+0x4e/0xe0 Jul 06 23:40:39 server-001 kernel: [<ffffffff811bceb9>] __mem_cgroup_try_charge+0x459/0xbe0 Jul 06 23:40:39 server-001 kernel: [<ffffffffa07e4dd5>] ? btrfs_split_extent_hook+0x35/0x40 [btrfs] Jul 06 23:40:39 server-001 kernel: [<ffffffffa07c6055>] ? block_rsv_release_bytes+0x95/0x180 [btrfs] Jul 06 23:40:40 server-001 kernel: [<ffffffff811bdd69>] mem_cgroup_charge_common+0x59/0xc0 Jul 06 23:40:40 server-001 kernel: [<ffffffff811bf9ba>] mem_cgroup_cache_charge+0x8a/0xb0 Jul 06 23:40:40 server-001 kernel: [<ffffffff811571f2>] __add_to_page_cache_locked+0x52/0x260 Jul 06 23:40:40 server-001 kernel: [<ffffffff81157457>] add_to_page_cache_lru+0x37/0xb0 Jul 06 23:40:40 server-001 kernel: [<ffffffff811577de>] find_or_create_page+0x5e/0xa0 Jul 06 23:40:40 server-001 kernel: [<ffffffffa07f3b00>] prepare_pages.isra.19+0xc0/0x180 [btrfs] Jul 06 23:40:40 server-001 kernel: [<ffffffffa07f472c>] __btrfs_buffered_write+0x1dc/0x5c0 [btrfs] Jul 06 23:40:40 server-001 kernel: [<ffffffff810a0898>] ? __wake_up_common+0x58/0x90 Jul 06 23:40:40 server-001 kernel: [<ffffffffa07f4d5b>] btrfs_file_aio_write+0x24b/0x5a0 [btrfs] Jul 06 23:40:40 server-001 kernel: [<ffffffff811c650d>] do_sync_write+0x8d/0xd0 Jul 06 23:40:40 server-001 kernel: [<ffffffff811c6cad>] vfs_write+0xbd/0x1e0 Jul 06 23:40:40 server-001 kernel: [<ffffffff811c76f8>] SyS_write+0x58/0xb0 Jul 06 23:40:40 server-001 kernel: [<ffffffff81614de9>] system_call_fastpath+0x16/0x1b Jul 06 23:42:41 server-001 kernel: INFO: task kworker/u65:9:31973 blocked for more than 120 seconds. Jul 06 23:42:41 server-001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 06 23:42:41 server-001 kernel: kworker/u65:9 D ffff881fffdb3680 0 31973 2 0x00000080 Jul 06 23:42:41 server-001 kernel: Workqueue: writeback bdi_writeback_workfn (flush-btrfs-1) Jul 06 23:42:41 server-001 kernel: ffff8816e1a87738 0000000000000046 ffff8816e1a87fd8 0000000000013680 Jul 06 23:42:41 server-001 kernel: ffff8816e1a87fd8 0000000000013680 ffff8810e1b7cfa0 ffff881fffdb3f48 Jul 06 23:42:41 server-001 kernel: ffff8816e1a877c0 0000000000000002 ffffffff81156330 ffff8816e1a877b0 Jul 06 23:42:41 server-001 kernel: Call Trace: Jul 06 23:42:41 server-001 kernel: [<ffffffff81156330>] ? wait_on_page_read+0x60/0x60 Jul 06 23:42:41 server-001 kernel: [<ffffffff8160a4dd>] io_schedule+0x9d/0x140 Jul 06 23:42:41 server-001 kernel: [<ffffffff8115633e>] sleep_on_page+0xe/0x20 Jul 06 23:42:41 server-001 kernel: [<ffffffff816083db>] __wait_on_bit_lock+0x5b/0xc0 Jul 06 23:42:41 server-001 kernel: [<ffffffff81156458>] __lock_page+0x78/0xa0 Jul 06 23:42:41 server-001 kernel: [<ffffffff81098390>] ? autoremove_wake_function+0x40/0x40 Jul 06 23:42:42 server-001 kernel: [<ffffffffa07fe715>] lock_delalloc_pages+0x1e5/0x1f0 [btrfs] Jul 06 23:42:42 server-001 kernel: [<ffffffffa0800f13>] find_lock_delalloc_range.constprop.43+0x153/0x200 [btrfs] Jul 06 23:42:42 server-001 kernel: [<ffffffffa080104b>] writepage_delalloc.isra.33+0x8b/0x180 [btrfs] Jul 06 23:42:42 server-001 kernel: [<ffffffffa0801cba>] __extent_writepage+0xca/0x2b0 [btrfs] Jul 06 23:42:42 server-001 kernel: [<ffffffffa08021ea>] extent_write_cache_pages.isra.28.constprop.48+0x34a/0x420 [btrfs] Jul 06 23:42:42 server-001 kernel: [<ffffffffa08040dc>] extent_writepages+0x5c/0x90 [btrfs] Jul 06 23:42:42 server-001 kernel: [<ffffffffa07e6e30>] ? btrfs_submit_direct+0x6c0/0x6c0 [btrfs] Jul 06 23:42:42 server-001 kernel: [<ffffffffa07e4738>] btrfs_writepages+0x28/0x30 [btrfs] Jul 06 23:42:42 server-001 kernel: [<ffffffff81162fae>] do_writepages+0x1e/0x40 Jul 06 23:42:42 server-001 kernel: [<ffffffff811f0670>] __writeback_single_inode+0x40/0x220 Jul 06 23:42:42 server-001 kernel: [<ffffffff811f136e>] writeback_sb_inodes+0x25e/0x420 Jul 06 23:42:43 server-001 kernel: [<ffffffff811f15cf>] __writeback_inodes_wb+0x9f/0xd0 Jul 06 23:42:43 server-001 kernel: [<ffffffff811f1e13>] wb_writeback+0x263/0x2f0 Jul 06 23:42:43 server-001 kernel: [<ffffffff811f32a5>] bdi_writeback_workfn+0x115/0x460 Jul 06 23:42:43 server-001 kernel: [<ffffffff8108f1eb>] process_one_work+0x17b/0x470 Jul 06 23:42:43 server-001 kernel: [<ffffffff8108ffbb>] worker_thread+0x11b/0x400 Jul 06 23:42:43 server-001 kernel: [<ffffffff8108fea0>] ? rescuer_thread+0x400/0x400 Jul 06 23:42:43 server-001 kernel: [<ffffffff8109739f>] kthread+0xcf/0xe0 Jul 06 23:42:43 server-001 kernel: [<ffffffff810972d0>] ? kthread_create_on_node+0x140/0x140 Jul 06 23:42:43 server-001 kernel: [<ffffffff81614d3c>] ret_from_fork+0x7c/0xb0 Jul 06 23:42:43 server-001 kernel: [<ffffffff810972d0>] ? kthread_create_on_node+0x140/0x140 Jul 06 23:42:43 server-001 kernel: INFO: task kworker/u65:22:27037 blocked for more than 120 seconds. Jul 06 23:42:43 server-001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 06 23:42:43 server-001 kernel: kworker/u65:22 D ffff881fffc33680 0 27037 2 0x00000080 Jul 06 23:42:44 server-001 kernel: Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs] Jul 06 23:42:44 server-001 kernel: ffff88001ab8fb80 0000000000000046 ffff88001ab8ffd8 0000000000013680 Jul 06 23:42:44 server-001 kernel: ffff88001ab8ffd8 0000000000013680 ffff8812414571c0 ffff88001ab8fca8 Jul 06 23:42:44 server-001 kernel: ffff88001ab8fcb0 7fffffffffffffff ffff8812414571c0 0000000000000000 Jul 06 23:42:44 server-001 kernel: Call Trace: Jul 06 23:42:44 server-001 kernel: [<ffffffff8160a1d9>] schedule+0x29/0x70 Jul 06 23:42:44 server-001 kernel: [<ffffffff81608119>] schedule_timeout+0x209/0x2d0 Jul 06 23:42:44 server-001 kernel: [<ffffffff8108d126>] ? __queue_work+0x136/0x320 Jul 06 23:42:44 server-001 kernel: [<ffffffff8108d3da>] ? __queue_delayed_work+0xaa/0x1a0 Jul 06 23:42:44 server-001 kernel: [<ffffffff8160a6e6>] wait_for_completion+0x116/0x170 Jul 06 23:42:44 server-001 kernel: [<ffffffff810a9650>] ? wake_up_state+0x20/0x20 Jul 06 23:42:44 server-001 kernel: [<ffffffff811f09ee>] writeback_inodes_sb_nr+0x8e/0xd0 Jul 06 23:42:44 server-001 kernel: [<ffffffffa07c9ea8>] flush_space+0x458/0x4f0 [btrfs] Jul 06 23:42:44 server-001 kernel: [<ffffffffa07c9530>] ? btrfs_get_alloc_profile+0x30/0x40 [btrfs] Jul 06 23:42:44 server-001 kernel: [<ffffffffa07c9a04>] ? can_overcommit+0xa4/0xf0 [btrfs] Jul 06 23:42:45 server-001 kernel: [<ffffffffa07ca0d4>] btrfs_async_reclaim_metadata_space+0x194/0x210 [btrfs] Jul 06 23:42:45 server-001 kernel: [<ffffffff8108f1eb>] process_one_work+0x17b/0x470 Jul 06 23:42:45 server-001 kernel: [<ffffffff8108ffbb>] worker_thread+0x11b/0x400 Jul 06 23:42:45 server-001 kernel: [<ffffffff8108fea0>] ? rescuer_thread+0x400/0x400 Jul 06 23:42:45 server-001 kernel: [<ffffffff8109739f>] kthread+0xcf/0xe0 Jul 06 23:42:45 server-001 kernel: [<ffffffff810972d0>] ? kthread_create_on_node+0x140/0x140 Jul 06 23:42:45 server-001 kernel: [<ffffffff81614d3c>] ret_from_fork+0x7c/0xb0 Jul 06 23:42:45 server-001 kernel: [<ffffffff810972d0>] ? kthread_create_on_node+0x140/0x140 Jul 06 23:42:45 server-001 kernel: INFO: task git:8697 blocked for more than 120 seconds. Jul 06 23:42:45 server-001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 06 23:42:45 server-001 kernel: git D ffff881fffc93680 0 8697 8695 0x00000084 Jul 06 23:42:46 server-001 kernel: ffff881d50c43498 0000000000000082 ffff881d50c43fd8 0000000000013680 Jul 06 23:42:46 server-001 kernel: ffff881d50c43fd8 0000000000013680 ffff883488958b60 ffff881fffc93f48 Jul 06 23:42:46 server-001 kernel: ffff88207ffa6ee8 0000000000000002 ffffffff81156330 ffff881d50c43510 Jul 06 23:42:46 server-001 kernel: Call Trace: Jul 06 23:42:46 server-001 kernel: [<ffffffff81156330>] ? wait_on_page_read+0x60/0x60 Jul 06 23:42:46 server-001 kernel: [<ffffffff8160a4dd>] io_schedule+0x9d/0x140 Jul 06 23:42:46 server-001 kernel: [<ffffffff8115633e>] sleep_on_page+0xe/0x20 Jul 06 23:42:46 server-001 kernel: [<ffffffff816082a0>] __wait_on_bit+0x60/0x90 Jul 06 23:42:46 server-001 kernel: [<ffffffff811560c6>] wait_on_page_bit+0x86/0xb0 Jul 06 23:42:46 server-001 kernel: [<ffffffff81098390>] ? autoremove_wake_function+0x40/0x40 Jul 06 23:42:46 server-001 kernel: [<ffffffff8116a1b2>] shrink_page_list+0x6c2/0xad0 Jul 06 23:42:46 server-001 kernel: [<ffffffff813f9b80>] ? scsi_request_fn+0x50/0x570 Jul 06 23:42:46 server-001 kernel: [<ffffffff8116ac7a>] shrink_inactive_list+0x1ea/0x560 Jul 06 23:42:46 server-001 kernel: [<ffffffff8116b73d>] shrink_lruvec+0x36d/0x730 Jul 06 23:42:46 server-001 kernel: [<ffffffff8116bb76>] shrink_zone+0x76/0x1a0 Jul 06 23:42:46 server-001 kernel: [<ffffffff8116c080>] do_try_to_free_pages+0xf0/0x4e0 Jul 06 23:42:47 server-001 kernel: [<ffffffff8115d90a>] ? __rmqueue+0x8a/0x460 Jul 06 23:42:47 server-001 kernel: [<ffffffff8116c6ba>] try_to_free_mem_cgroup_pages+0xca/0x160 Jul 06 23:42:47 server-001 kernel: [<ffffffff811bc9ce>] mem_cgroup_reclaim+0x4e/0xe0 Jul 06 23:42:47 server-001 kernel: [<ffffffff811bceb9>] __mem_cgroup_try_charge+0x459/0xbe0 Jul 06 23:42:47 server-001 kernel: [<ffffffffa07e4dd5>] ? btrfs_split_extent_hook+0x35/0x40 [btrfs] Jul 06 23:42:47 server-001 kernel: [<ffffffffa07c6055>] ? block_rsv_release_bytes+0x95/0x180 [btrfs] Jul 06 23:42:47 server-001 kernel: [<ffffffff811bdd69>] mem_cgroup_charge_common+0x59/0xc0 Jul 06 23:42:47 server-001 kernel: [<ffffffff811bf9ba>] mem_cgroup_cache_charge+0x8a/0xb0 Jul 06 23:42:47 server-001 kernel: [<ffffffff811571f2>] __add_to_page_cache_locked+0x52/0x260 Jul 06 23:42:47 server-001 kernel: [<ffffffff81157457>] add_to_page_cache_lru+0x37/0xb0 Jul 06 23:42:47 server-001 kernel: [<ffffffff811577de>] find_or_create_page+0x5e/0xa0 Jul 06 23:42:47 server-001 kernel: [<ffffffffa07f3b00>] prepare_pages.isra.19+0xc0/0x180 [btrfs] Jul 06 23:42:48 server-001 kernel: [<ffffffffa07f472c>] __btrfs_buffered_write+0x1dc/0x5c0 [btrfs] Jul 06 23:42:48 server-001 kernel: [<ffffffff810a0898>] ? __wake_up_common+0x58/0x90 Jul 06 23:42:48 server-001 kernel: [<ffffffffa07f4d5b>] btrfs_file_aio_write+0x24b/0x5a0 [btrfs] Jul 06 23:42:48 server-001 kernel: [<ffffffff811c650d>] do_sync_write+0x8d/0xd0 Jul 06 23:42:48 server-001 kernel: [<ffffffff811c6cad>] vfs_write+0xbd/0x1e0 Jul 06 23:42:48 server-001 kernel: [<ffffffff811c76f8>] SyS_write+0x58/0xb0 Jul 06 23:42:48 server-001 kernel: [<ffffffff81614de9>] system_call_fastpath+0x16/0x1b Jul 06 23:42:48 server-001 kernel: INFO: task tar:10489 blocked for more than 120 seconds. Jul 06 23:42:48 server-001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 06 23:42:48 server-001 kernel: tar D ffff881fffd93680 0 10489 10479 0x00000080 Jul 06 23:42:48 server-001 kernel: ffff883439adf8d0 0000000000000086 ffff883439adffd8 0000000000013680 Jul 06 23:42:48 server-001 kernel: ffff883439adffd8 0000000000013680 ffff88339d5396c0 ffff883439adf9f8 Jul 06 23:42:49 server-001 kernel: ffff883439adfa00 7fffffffffffffff ffff88339d5396c0 0000000000000000 Jul 06 23:42:49 server-001 kernel: Call Trace: Jul 06 23:42:49 server-001 kernel: [<ffffffff8160a1d9>] schedule+0x29/0x70 Jul 06 23:42:49 server-001 kernel: [<ffffffff81608119>] schedule_timeout+0x209/0x2d0 Jul 06 23:42:49 server-001 kernel: [<ffffffff8108d126>] ? __queue_work+0x136/0x320 Jul 06 23:42:49 server-001 kernel: [<ffffffff8108d3da>] ? __queue_delayed_work+0xaa/0x1a0 Jul 06 23:42:49 server-001 kernel: [<ffffffff8160a6e6>] wait_for_completion+0x116/0x170 Jul 06 23:42:49 server-001 kernel: [<ffffffff810a9650>] ? wake_up_state+0x20/0x20 Jul 06 23:42:49 server-001 kernel: [<ffffffff811f09ee>] writeback_inodes_sb_nr+0x8e/0xd0 Jul 06 23:42:49 server-001 kernel: [<ffffffffa07c9ea8>] flush_space+0x458/0x4f0 [btrfs] Jul 06 23:42:49 server-001 kernel: [<ffffffffa07c9530>] ? btrfs_get_alloc_profile+0x30/0x40 [btrfs] Jul 06 23:42:49 server-001 kernel: [<ffffffffa07c9a04>] ? can_overcommit+0xa4/0xf0 [btrfs] Jul 06 23:42:49 server-001 kernel: [<ffffffffa07ca31e>] reserve_metadata_bytes+0x1ce/0x540 [btrfs] Jul 06 23:42:49 server-001 kernel: [<ffffffff81295718>] ? crypto_shash_update+0x38/0x100 Jul 06 23:42:49 server-001 kernel: [<ffffffffa07cac40>] btrfs_block_rsv_add+0x30/0x60 [btrfs] Jul 06 23:42:50 server-001 kernel: [<ffffffffa07e2ee3>] start_transaction+0x453/0x5a0 [btrfs] Jul 06 23:42:50 server-001 kernel: [<ffffffffa07b8b25>] ? btrfs_release_path+0x25/0xb0 [btrfs] Jul 06 23:42:50 server-001 kernel: [<ffffffffa07e304b>] btrfs_start_transaction+0x1b/0x20 [btrfs] Jul 06 23:42:50 server-001 kernel: [<ffffffffa07f08ea>] btrfs_create+0x4a/0x230 [btrfs] Jul 06 23:42:50 server-001 kernel: [<ffffffff8126986c>] ? security_inode_permission+0x1c/0x30 Jul 06 23:42:50 server-001 kernel: [<ffffffff811d30ed>] vfs_create+0xcd/0x130 Jul 06 23:42:50 server-001 kernel: [<ffffffff811d632f>] do_last+0xb8f/0x1270 Jul 06 23:42:50 server-001 kernel: [<ffffffff811d6ad2>] path_openat+0xc2/0x490 Jul 06 23:42:50 server-001 kernel: [<ffffffffa07faf12>] ? btrfs_removexattr+0x72/0xd0 [btrfs] Jul 06 23:42:50 server-001 kernel: [<ffffffff811d829b>] do_filp_open+0x4b/0xb0 Jul 06 23:42:50 server-001 kernel: [<ffffffff811e5a4f>] ? mnt_drop_write+0x1f/0x30 Jul 06 23:42:50 server-001 kernel: [<ffffffff811e4d07>] ? __alloc_fd+0xa7/0x130 Jul 06 23:42:50 server-001 kernel: [<ffffffff811c5f83>] do_sys_open+0xf3/0x1f0 Jul 06 23:42:50 server-001 kernel: [<ffffffff811c609e>] SyS_open+0x1e/0x20 Jul 06 23:42:51 server-001 kernel: [<ffffffff81614de9>] system_call_fastpath+0x16/0x1b Jul 06 23:44:51 server-001 kernel: INFO: task khugepaged:252 blocked for more than 120 seconds. Jul 06 23:44:51 server-001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 06 23:44:51 server-001 kernel: khugepaged D ffff881fffcb3680 0 252 2 0x00000000 Jul 06 23:44:51 server-001 kernel: ffff881fd058bc98 0000000000000046 ffff881fd058bfd8 0000000000013680 Jul 06 23:44:51 server-001 kernel: ffff881fd058bfd8 0000000000013680 ffff883fd1e05b00 ffff883fd1e05b00 Jul 06 23:44:51 server-001 kernel: ffff881fced21fb8 ffff881fced21fc0 ffffffff00000000 ffff881fced21fc8 Jul 06 23:44:51 server-001 kernel: Call Trace: Jul 06 23:44:51 server-001 kernel: [<ffffffff8160a1d9>] schedule+0x29/0x70 Jul 06 23:44:51 server-001 kernel: [<ffffffff8160bad5>] rwsem_down_write_failed+0x115/0x220 Jul 06 23:44:51 server-001 kernel: [<ffffffff811bbf12>] ? __mem_cgroup_commit_charge+0x152/0x390 Jul 06 23:44:51 server-001 kernel: [<ffffffff812e3493>] call_rwsem_down_write_failed+0x13/0x20 Jul 06 23:44:51 server-001 kernel: [<ffffffff816095dd>] ? down_write+0x2d/0x30 Jul 06 23:44:52 server-001 kernel: [<ffffffff811b4485>] khugepaged_scan_mm_slot+0x415/0xca0 Jul 06 23:44:52 server-001 kernel: [<ffffffff811b4f6f>] khugepaged+0x25f/0x4a0 Jul 06 23:44:52 server-001 kernel: [<ffffffff81098350>] ? wake_up_bit+0x30/0x30 Jul 06 23:44:52 server-001 kernel: [<ffffffff811b4d10>] ? khugepaged_scan_mm_slot+0xca0/0xca0 Jul 06 23:44:52 server-001 kernel: [<ffffffff8109739f>] kthread+0xcf/0xe0 Jul 06 23:44:52 server-001 kernel: [<ffffffff810972d0>] ? kthread_create_on_node+0x140/0x140 Jul 06 23:44:52 server-001 kernel: [<ffffffff81614d3c>] ret_from_fork+0x7c/0xb0 Jul 06 23:44:52 server-001 kernel: [<ffffffff810972d0>] ? kthread_create_on_node+0x140/0x140 Jul 06 23:44:52 server-001 kernel: INFO: task mesos-slave:1559 blocked for more than 120 seconds. Jul 06 23:44:52 server-001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 06 23:44:52 server-001 kernel: mesos-slave D ffff88407fdf3680 0 1559 1 0x00000080 Jul 06 23:44:52 server-001 kernel: ffff881fc2a07cc8 0000000000000086 ffff881fc2a07fd8 0000000000013680 Jul 06 23:44:53 server-001 kernel: ffff881fc2a07fd8 0000000000013680 ffff881fd0894fa0 ffff881fd0894fa0 Jul 06 23:44:53 server-001 kernel: ffff881fced21fb8 ffffffffffffffff ffff881fced21fc0 000000000000015c Jul 06 23:44:53 server-001 kernel: Call Trace: Jul 06 23:44:53 server-001 kernel: [<ffffffff8160a1d9>] schedule+0x29/0x70 Jul 06 23:44:53 server-001 kernel: [<ffffffff8160bcd5>] rwsem_down_read_failed+0xf5/0x165 Jul 06 23:44:53 server-001 kernel: [<ffffffff812e3464>] call_rwsem_down_read_failed+0x14/0x30 Jul 06 23:44:53 server-001 kernel: [<ffffffff816095a0>] ? down_read+0x20/0x30 Jul 06 23:44:53 server-001 kernel: [<ffffffff81183a41>] __access_remote_vm+0x51/0x1f0 Jul 06 23:44:53 server-001 kernel: [<ffffffff81184880>] access_process_vm+0x50/0x70 Jul 06 23:44:53 server-001 kernel: [<ffffffff8122fc1a>] proc_pid_cmdline+0x8a/0x120 Jul 06 23:44:53 server-001 kernel: [<ffffffff8123107f>] proc_info_read+0x8f/0xe0 Jul 06 23:44:53 server-001 kernel: [<ffffffff811c6b1c>] vfs_read+0x9c/0x170 Jul 06 23:44:53 server-001 kernel: [<ffffffff811c7648>] SyS_read+0x58/0xb0 Jul 06 23:44:53 server-001 kernel: [<ffffffff81614de9>] system_call_fastpath+0x16/0x1b Jul 06 23:44:53 server-001 kernel: INFO: task atop:17585 blocked for more than 120 seconds. Jul 06 23:44:54 server-001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 06 23:44:54 server-001 kernel: atop D ffff881fffcd3680 0 17585 1 0x00000084 Jul 06 23:44:54 server-001 kernel: ffff8816e1677cc8 0000000000000086 ffff8816e1677fd8 0000000000013680 Jul 06 23:44:54 server-001 kernel: ffff8816e1677fd8 0000000000013680 ffff881e441b6660 ffff881e441b6660 Jul 06 23:44:54 server-001 kernel: ffff881fced21fb8 ffffffffffffffff ffff881fced21fc0 000000000000015c Jul 06 23:44:54 server-001 kernel: Call Trace: Jul 06 23:44:54 server-001 kernel: [<ffffffff8160a1d9>] schedule+0x29/0x70 Jul 06 23:44:54 server-001 kernel: [<ffffffff8160bcd5>] rwsem_down_read_failed+0xf5/0x165 Jul 06 23:44:54 server-001 kernel: [<ffffffff812e3464>] call_rwsem_down_read_failed+0x14/0x30 Jul 06 23:44:54 server-001 kernel: [<ffffffff816095a0>] ? down_read+0x20/0x30 Jul 06 23:44:54 server-001 kernel: [<ffffffff81183a41>] __access_remote_vm+0x51/0x1f0 Jul 06 23:44:54 server-001 kernel: [<ffffffff81184880>] access_process_vm+0x50/0x70 Jul 06 23:44:54 server-001 kernel: [<ffffffff8122fc1a>] proc_pid_cmdline+0x8a/0x120 Jul 06 23:44:54 server-001 kernel: [<ffffffff8123107f>] proc_info_read+0x8f/0xe0 Jul 06 23:44:55 server-001 kernel: [<ffffffff811c6b1c>] vfs_read+0x9c/0x170 Jul 06 23:44:55 server-001 kernel: [<ffffffff811c7648>] SyS_read+0x58/0xb0 Jul 06 23:44:55 server-001 kernel: [<ffffffff81614de9>] system_call_fastpath+0x16/0x1b Jul 06 23:44:55 server-001 kernel: INFO: task kworker/u65:9:31973 blocked for more than 120 seconds. Jul 06 23:44:55 server-001 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jul 06 23:44:55 server-001 kernel: kworker/u65:9 D ffff881fffdb3680 0 31973 2 0x00000080 Jul 06 23:44:55 server-001 kernel: Workqueue: writeback bdi_writeback_workfn (flush-btrfs-1) Jul 06 23:44:55 server-001 kernel: ffff8816e1a87738 0000000000000046 ffff8816e1a87fd8 0000000000013680 Jul 06 23:44:55 server-001 kernel: ffff8816e1a87fd8 0000000000013680 ffff8810e1b7cfa0 ffff881fffdb3f48 Jul 06 23:44:55 server-001 kernel: ffff8816e1a877c0 0000000000000002 ffffffff81156330 ffff8816e1a877b0 Jul 06 23:44:55 server-001 kernel: Call Trace: Jul 06 23:44:55 server-001 kernel: [<ffffffff81156330>] ? wait_on_page_read+0x60/0x60 Jul 06 23:44:55 server-001 kernel: [<ffffffff8160a4dd>] io_schedule+0x9d/0x140 Jul 06 23:44:56 server-001 kernel: [<ffffffff8115633e>] sleep_on_page+0xe/0x20 Jul 06 23:44:56 server-001 kernel: [<ffffffff816083db>] __wait_on_bit_lock+0x5b/0xc0 Jul 06 23:44:56 server-001 kernel: [<ffffffff81156458>] __lock_page+0x78/0xa0 Jul 06 23:44:56 server-001 kernel: [<ffffffff81098390>] ? autoremove_wake_function+0x40/0x40 Jul 06 23:44:56 server-001 kernel: [<ffffffffa07fe715>] lock_delalloc_pages+0x1e5/0x1f0 [btrfs] Jul 06 23:44:56 server-001 kernel: [<ffffffffa0800f13>] find_lock_delalloc_range.constprop.43+0x153/0x200 [btrfs] Jul 06 23:44:56 server-001 kernel: [<ffffffffa080104b>] writepage_delalloc.isra.33+0x8b/0x180 [btrfs] Jul 06 23:44:56 server-001 kernel: [<ffffffffa0801cba>] __extent_writepage+0xca/0x2b0 [btrfs] Jul 06 23:44:56 server-001 kernel: [<ffffffffa08021ea>] extent_write_cache_pages.isra.28.constprop.48+0x34a/0x420 [btrfs] Jul 06 23:44:56 server-001 kernel: [<ffffffffa08040dc>] extent_writepages+0x5c/0x90 [btrfs] Jul 06 23:44:56 server-001 kernel: [<ffffffffa07e6e30>] ? btrfs_submit_direct+0x6c0/0x6c0 [btrfs] Jul 06 23:44:56 server-001 kernel: [<ffffffffa07e4738>] btrfs_writepages+0x28/0x30 [btrfs] Jul 06 23:44:57 server-001 kernel: [<ffffffff81162fae>] do_writepages+0x1e/0x40 Jul 06 23:44:57 server-001 kernel: [<ffffffff811f0670>] __writeback_single_inode+0x40/0x220 Jul 06 23:44:57 server-001 kernel: [<ffffffff811f136e>] writeback_sb_inodes+0x25e/0x420 Jul 06 23:44:57 server-001 kernel: [<ffffffff811f15cf>] __writeback_inodes_wb+0x9f/0xd0 Jul 06 23:44:57 server-001 kernel: [<ffffffff811f1e13>] wb_writeback+0x263/0x2f0 Jul 06 23:44:57 server-001 kernel: [<ffffffff811f32a5>] bdi_writeback_workfn+0x115/0x460 Jul 06 23:44:57 server-001 kernel: [<ffffffff8108f1eb>] process_one_work+0x17b/0x470 Jul 06 23:44:57 server-001 kernel: [<ffffffff8108ffbb>] worker_thread+0x11b/0x400 Jul 06 23:44:57 server-001 kernel: [<ffffffff8108fea0>] ? rescuer_thread+0x400/0x400 Jul 06 23:44:57 server-001 kernel: [<ffffffff8109739f>] kthread+0xcf/0xe0 Jul 06 23:44:57 server-001 kernel: [<ffffffff810972d0>] ? kthread_create_on_node+0x140/0x140 Jul 06 23:44:57 server-001 kernel: [<ffffffff81614d3c>] ret_from_fork+0x7c/0xb0 Jul 06 23:44:57 server-001 kernel: [<ffffffff810972d0>] ? kthread_create_on_node+0x140/0x140
Attachment:
dmesg.log.gz
Description: dmesg.log.gz
