I've got more hung tasks. Log is attached. Several empty lines
in the file mean that there was a reboot in between.
btrfs scrub -Bdr says there are no errors on disks.
btrfs device stats also shows no errors.
Jul 19 11:47:36 prod-dbsnap-01 systemd: Started PostgreSQL snapshot service for ardcimicrofinance_6432.
Jul 19 11:47:36 prod-dbsnap-01 systemd: Reloading.
Jul 19 11:47:36 prod-dbsnap-01 systemd: Binding to IPv6 address not available since kernel does not support IPv6.
Jul 19 11:47:36 prod-dbsnap-01 systemd: Binding to IPv6 address not available since kernel does not support IPv6.
Jul 19 11:50:01 prod-dbsnap-01 systemd: Created slice User Slice of root.
Jul 19 11:50:01 prod-dbsnap-01 systemd: Started Session 241 of user root.
Jul 19 11:50:01 prod-dbsnap-01 systemd: Removed slice User Slice of root.
Jul 19 11:52:01 prod-dbsnap-01 kernel: INFO: task postgres:20710 blocked for more than 122 seconds.
Jul 19 11:52:01 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 11:52:01 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 11:52:01 prod-dbsnap-01 kernel: postgres D 0 20710 20672 0x00004000
Jul 19 11:52:01 prod-dbsnap-01 kernel: Call Trace:
Jul 19 11:52:01 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 11:52:01 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: __x64_sys_fsync+0x14/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 11:52:01 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 11:52:01 prod-dbsnap-01 kernel: RIP: 0033:0x7fc16f67bd00
Jul 19 11:52:01 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 11:52:01 prod-dbsnap-01 kernel: RSP: 002b:00007ffee0d14198 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
Jul 19 11:52:01 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 00000000000000f9 RCX: 00007fc16f67bd00
Jul 19 11:52:01 prod-dbsnap-01 kernel: RDX: 0000000000000120 RSI: 00007fc150c5c580 RDI: 00000000000000f9
Jul 19 11:52:01 prod-dbsnap-01 kernel: RBP: 00007fc16fa877f0 R08: 00007fc150c5c690 R09: 00007fc150c5c588
Jul 19 11:52:01 prod-dbsnap-01 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000019abc10bce0
Jul 19 11:52:01 prod-dbsnap-01 kernel: R13: 0000000000000001 R14: 000000000206dba0 R15: 0000000000000f9d
Jul 19 11:52:01 prod-dbsnap-01 kernel: INFO: task postgres:28814 blocked for more than 122 seconds.
Jul 19 11:52:01 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 11:52:01 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 11:52:01 prod-dbsnap-01 kernel: postgres D 0 28814 28787 0x00004000
Jul 19 11:52:01 prod-dbsnap-01 kernel: Call Trace:
Jul 19 11:52:01 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 11:52:01 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: __x64_sys_fsync+0x14/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 11:52:01 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 11:52:01 prod-dbsnap-01 kernel: RIP: 0033:0x7f1e421e4d00
Jul 19 11:52:01 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 11:52:01 prod-dbsnap-01 kernel: RSP: 002b:00007ffc18ebe718 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
Jul 19 11:52:01 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 00000000000003e0 RCX: 00007f1e421e4d00
Jul 19 11:52:01 prod-dbsnap-01 kernel: RDX: 0000000000000120 RSI: 00007f1e237c5580 RDI: 00000000000003e0
Jul 19 11:52:01 prod-dbsnap-01 kernel: RBP: 00007f1e425f07f0 R08: 00007f1e237c5690 R09: 00007f1e237c5588
Jul 19 11:52:01 prod-dbsnap-01 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000006053e93948
Jul 19 11:52:01 prod-dbsnap-01 kernel: R13: 0000000000000001 R14: 0000000000e21f20 R15: 0000000000003042
Jul 19 11:52:01 prod-dbsnap-01 kernel: INFO: task postgres:31379 blocked for more than 122 seconds.
Jul 19 11:52:01 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 11:52:01 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 11:52:01 prod-dbsnap-01 kernel: postgres D 0 31379 28787 0x00004004
Jul 19 11:52:01 prod-dbsnap-01 kernel: Call Trace:
Jul 19 11:52:01 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 11:52:01 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: io_schedule+0x16/0x40
Jul 19 11:52:01 prod-dbsnap-01 kernel: wait_on_page_bit+0x1c0/0x270
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? file_fdatawait_range+0x30/0x30
Jul 19 11:52:01 prod-dbsnap-01 kernel: wait_on_page_writeback+0x48/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: extent_write_cache_pages+0x214/0x4c0 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: extent_writepages+0x4f/0x90 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? tcp_push+0xde/0x110
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_writepages+0xe/0x10 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_writepages+0x1f/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: __filemap_fdatawrite_range+0xc6/0x100
Jul 19 11:52:01 prod-dbsnap-01 kernel: filemap_fdatawrite_range+0x13/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_fdatawrite_range+0x20/0x50 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: start_ordered_ops+0x4c/0x90 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __sys_sendto+0x107/0x1a0
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_sync_file+0xce/0x410 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: __x64_sys_fdatasync+0x17/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 11:52:01 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 11:52:01 prod-dbsnap-01 kernel: RIP: 0033:0x7f1e3fd58310
Jul 19 11:52:01 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 11:52:01 prod-dbsnap-01 kernel: RSP: 002b:00007ffc18ebf888 EFLAGS: 00000246 ORIG_RAX: 000000000000004b
Jul 19 11:52:01 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000006053 RCX: 00007f1e3fd58310
Jul 19 11:52:01 prod-dbsnap-01 kernel: RDX: 0000000000000027 RSI: 0000000000006053 RDI: 0000000000000004
Jul 19 11:52:01 prod-dbsnap-01 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jul 19 11:52:01 prod-dbsnap-01 kernel: R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000004
Jul 19 11:52:01 prod-dbsnap-01 kernel: R13: 0000000000e208e9 R14: 00007f1e2fbb9280 R15: 0000000000001e00
Jul 19 11:52:01 prod-dbsnap-01 kernel: INFO: task postgres:31526 blocked for more than 122 seconds.
Jul 19 11:52:01 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 11:52:01 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 11:52:01 prod-dbsnap-01 kernel: postgres D 0 31526 20672 0x00004000
Jul 19 11:52:01 prod-dbsnap-01 kernel: Call Trace:
Jul 19 11:52:01 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 11:52:01 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: __x64_sys_fdatasync+0x17/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 11:52:01 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 11:52:01 prod-dbsnap-01 kernel: RIP: 0033:0x7fc16d1ef310
Jul 19 11:52:01 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 11:52:01 prod-dbsnap-01 kernel: RSP: 002b:00007ffee0d15308 EFLAGS: 00000246 ORIG_RAX: 000000000000004b
Jul 19 11:52:01 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000019abc RCX: 00007fc16d1ef310
Jul 19 11:52:01 prod-dbsnap-01 kernel: RDX: 0000000000000027 RSI: 0000000000019abc RDI: 0000000000000004
Jul 19 11:52:01 prod-dbsnap-01 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jul 19 11:52:01 prod-dbsnap-01 kernel: R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000004
Jul 19 11:52:01 prod-dbsnap-01 kernel: R13: 000000000208a5d9 R14: 00007fc15d050280 R15: 0000000000012000
Jul 19 11:52:01 prod-dbsnap-01 kernel: INFO: task postgres:36718 blocked for more than 122 seconds.
Jul 19 11:52:01 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 11:52:01 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 11:52:01 prod-dbsnap-01 kernel: postgres D 0 36718 36697 0x00004000
Jul 19 11:52:01 prod-dbsnap-01 kernel: Call Trace:
Jul 19 11:52:01 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 11:52:01 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: __x64_sys_fsync+0x14/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 11:52:01 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 11:52:01 prod-dbsnap-01 kernel: RIP: 0033:0x7f577cf4cd00
Jul 19 11:52:01 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 11:52:01 prod-dbsnap-01 kernel: RSP: 002b:00007ffdc0a03fb8 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
Jul 19 11:52:01 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007f577cf4cd00
Jul 19 11:52:01 prod-dbsnap-01 kernel: RDX: 0000000000000120 RSI: 00007f575e52d580 RDI: 0000000000000003
Jul 19 11:52:01 prod-dbsnap-01 kernel: RBP: 00007f577d3587f0 R08: 00007f575e52d690 R09: 00007f575e52d588
Jul 19 11:52:01 prod-dbsnap-01 kernel: R10: 00000000000032c8 R11: 0000000000000246 R12: 000000930ca33770
Jul 19 11:52:01 prod-dbsnap-01 kernel: R13: 0000000000000001 R14: 0000009308e80f30 R15: 0000000000000001
Jul 19 11:52:01 prod-dbsnap-01 kernel: INFO: task postgres:41263 blocked for more than 122 seconds.
Jul 19 11:52:01 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 11:52:01 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 11:52:01 prod-dbsnap-01 kernel: postgres D 0 41263 36697 0x00004004
Jul 19 11:52:01 prod-dbsnap-01 kernel: Call Trace:
Jul 19 11:52:01 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 11:52:01 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: __x64_sys_fdatasync+0x17/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 11:52:01 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 11:52:01 prod-dbsnap-01 kernel: RIP: 0033:0x7f577aac0310
Jul 19 11:52:01 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 11:52:01 prod-dbsnap-01 kernel: RSP: 002b:00007ffdc0a03bc8 EFLAGS: 00000246 ORIG_RAX: 000000000000004b
Jul 19 11:52:01 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 000000000000930c RCX: 00007f577aac0310
Jul 19 11:52:01 prod-dbsnap-01 kernel: RDX: 0000000000000027 RSI: 000000000000930c RDI: 0000000000000004
Jul 19 11:52:01 prod-dbsnap-01 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jul 19 11:52:01 prod-dbsnap-01 kernel: R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000004
Jul 19 11:52:01 prod-dbsnap-01 kernel: R13: 00000000015392c9 R14: 00007f576a921280 R15: 00000000000000e0
Jul 19 11:52:01 prod-dbsnap-01 kernel: INFO: task kworker/u256:5:118303 blocked for more than 122 seconds.
Jul 19 11:52:01 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 11:52:01 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 11:52:01 prod-dbsnap-01 kernel: kworker/u256:5 D 0 118303 2 0x80004000
Jul 19 11:52:01 prod-dbsnap-01 kernel: Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: Call Trace:
Jul 19 11:52:01 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 11:52:01 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 11:52:01 prod-dbsnap-01 kernel: start_transaction+0x316/0x4a0 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_join_transaction+0x1d/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_finish_ordered_io+0x45a/0x7c0 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: finish_ordered_fn+0x15/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: normal_work_helper+0xc5/0x320 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? pwq_activate_delayed_work+0x42/0xb0
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_endio_write_helper+0x12/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: process_one_work+0x179/0x3a0
Jul 19 11:52:01 prod-dbsnap-01 kernel: worker_thread+0x4f/0x3e0
Jul 19 11:52:01 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? max_active_store+0x80/0x80
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
Jul 19 11:52:01 prod-dbsnap-01 kernel: INFO: task kworker/u256:4:6869 blocked for more than 122 seconds.
Jul 19 11:52:01 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 11:52:01 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 11:52:01 prod-dbsnap-01 kernel: kworker/u256:4 D 0 6869 2 0x80004000
Jul 19 11:52:01 prod-dbsnap-01 kernel: Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: Call Trace:
Jul 19 11:52:01 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 11:52:01 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 11:52:01 prod-dbsnap-01 kernel: start_transaction+0x316/0x4a0 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_join_transaction+0x1d/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_finish_ordered_io+0x45a/0x7c0 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x40/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x40/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? x86_gsbase_write_task+0x20/0x30
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: finish_ordered_fn+0x15/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: normal_work_helper+0xc5/0x320 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? pwq_activate_delayed_work+0x42/0xb0
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_endio_write_helper+0x12/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: process_one_work+0x179/0x3a0
Jul 19 11:52:01 prod-dbsnap-01 kernel: worker_thread+0x4f/0x3e0
Jul 19 11:52:01 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? max_active_store+0x80/0x80
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
Jul 19 11:52:01 prod-dbsnap-01 kernel: INFO: task kworker/u256:14:8117 blocked for more than 122 seconds.
Jul 19 11:52:01 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 11:52:01 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 11:52:01 prod-dbsnap-01 kernel: kworker/u256:14 D 0 8117 2 0x80004000
Jul 19 11:52:01 prod-dbsnap-01 kernel: Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: Call Trace:
Jul 19 11:52:01 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 11:52:01 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 11:52:01 prod-dbsnap-01 kernel: start_transaction+0x316/0x4a0 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_join_transaction+0x1d/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_finish_ordered_io+0x45a/0x7c0 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x40/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x40/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? x86_gsbase_write_task+0x20/0x30
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 11:52:01 prod-dbsnap-01 kernel: finish_ordered_fn+0x15/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: normal_work_helper+0xc5/0x320 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? pwq_activate_delayed_work+0x42/0xb0
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_endio_write_helper+0x12/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: process_one_work+0x179/0x3a0
Jul 19 11:52:01 prod-dbsnap-01 kernel: worker_thread+0x4f/0x3e0
Jul 19 11:52:01 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? max_active_store+0x80/0x80
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
Jul 19 11:52:01 prod-dbsnap-01 kernel: INFO: task kworker/u256:6:10389 blocked for more than 122 seconds.
Jul 19 11:52:01 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 11:52:01 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 11:52:01 prod-dbsnap-01 kernel: kworker/u256:6 D 0 10389 2 0x80004000
Jul 19 11:52:01 prod-dbsnap-01 kernel: Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: Call Trace:
Jul 19 11:52:01 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 11:52:01 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 11:52:01 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 11:52:01 prod-dbsnap-01 kernel: start_transaction+0x316/0x4a0 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_join_transaction+0x1d/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_finish_ordered_io+0x45a/0x7c0 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? syscall_return_via_sysret+0xf/0x7f
Jul 19 11:52:01 prod-dbsnap-01 kernel: finish_ordered_fn+0x15/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: normal_work_helper+0xc5/0x320 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? pwq_activate_delayed_work+0x42/0xb0
Jul 19 11:52:01 prod-dbsnap-01 kernel: btrfs_endio_write_helper+0x12/0x20 [btrfs]
Jul 19 11:52:01 prod-dbsnap-01 kernel: process_one_work+0x179/0x3a0
Jul 19 11:52:01 prod-dbsnap-01 kernel: worker_thread+0x4f/0x3e0
Jul 19 11:52:01 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? max_active_store+0x80/0x80
Jul 19 11:52:01 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 11:52:01 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
Jul 19 12:20:13 prod-dbsnap-01 kernel: INFO: task postgres:1840 blocked for more than 122 seconds.
Jul 19 12:20:13 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 12:20:13 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 12:20:13 prod-dbsnap-01 kernel: postgres D 0 1840 1796 0x00000000
Jul 19 12:20:13 prod-dbsnap-01 kernel: Call Trace:
Jul 19 12:20:13 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 12:20:13 prod-dbsnap-01 kernel: ? __switch_to_asm+0x40/0x70
Jul 19 12:20:13 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 12:20:13 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 12:20:13 prod-dbsnap-01 kernel: rwsem_down_write_failed+0x1f3/0x380
Jul 19 12:20:13 prod-dbsnap-01 kernel: ? new_sync_read+0x11a/0x1c0
Jul 19 12:20:13 prod-dbsnap-01 kernel: down_write+0x3e/0x40
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_file_llseek+0x4c/0x2b0 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: ? vfs_read+0x8e/0x140
Jul 19 12:20:13 prod-dbsnap-01 kernel: ksys_lseek+0x8a/0xb0
Jul 19 12:20:13 prod-dbsnap-01 kernel: __x64_sys_lseek+0x1a/0x20
Jul 19 12:20:13 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 12:20:13 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 12:20:13 prod-dbsnap-01 kernel: RIP: 0033:0x7f49e7ff0d60
Jul 19 12:20:13 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 12:20:13 prod-dbsnap-01 kernel: RSP: 002b:00007ffd950c1358 EFLAGS: 00000246 ORIG_RAX: 0000000000000008
Jul 19 12:20:13 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000390000 RCX: 00007f49e7ff0d60
Jul 19 12:20:13 prod-dbsnap-01 kernel: RDX: 0000000000000000 RSI: 0000000000390000 RDI: 0000000000000003
Jul 19 12:20:13 prod-dbsnap-01 kernel: RBP: 0000019ac1390000 R08: 0000000002c29758 R09: 0000000002c2050c
Jul 19 12:20:13 prod-dbsnap-01 kernel: R10: 0000000002c2b730 R11: 0000000000000246 R12: 0000000000000003
Jul 19 12:20:13 prod-dbsnap-01 kernel: R13: 0000000000390000 R14: 0000000002c2050c R15: 0000000000000026
Jul 19 12:20:13 prod-dbsnap-01 kernel: INFO: task postgres:2037 blocked for more than 122 seconds.
Jul 19 12:20:13 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 12:20:13 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 12:20:13 prod-dbsnap-01 kernel: postgres D 0 2037 1801 0x00004000
Jul 19 12:20:13 prod-dbsnap-01 kernel: Call Trace:
Jul 19 12:20:13 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 12:20:13 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 12:20:13 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 12:20:13 prod-dbsnap-01 kernel: __x64_sys_fsync+0x14/0x20
Jul 19 12:20:13 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 12:20:13 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 12:20:13 prod-dbsnap-01 kernel: RIP: 0033:0x7f866ab96d00
Jul 19 12:20:13 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 12:20:13 prod-dbsnap-01 kernel: RSP: 002b:00007ffdc52426c8 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
Jul 19 12:20:13 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000000341 RCX: 00007f866ab96d00
Jul 19 12:20:13 prod-dbsnap-01 kernel: RDX: 0000000000000120 RSI: 00007f864c177580 RDI: 0000000000000341
Jul 19 12:20:13 prod-dbsnap-01 kernel: RBP: 00007f866afa27f0 R08: 00007f864c177690 R09: 00007f864c177588
Jul 19 12:20:13 prod-dbsnap-01 kernel: R10: 959bd756f2ca9fd3 R11: 0000000000000246 R12: 00000093121d6de0
Jul 19 12:20:13 prod-dbsnap-01 kernel: R13: 0000000000000001 R14: 0000000000000058 R15: 0000000000001eae
Jul 19 12:20:13 prod-dbsnap-01 kernel: INFO: task postgres:2061 blocked for more than 122 seconds.
Jul 19 12:20:13 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 12:20:13 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 12:20:13 prod-dbsnap-01 kernel: postgres D 0 2061 1808 0x00004000
Jul 19 12:20:13 prod-dbsnap-01 kernel: Call Trace:
Jul 19 12:20:13 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 12:20:13 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 12:20:13 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 12:20:13 prod-dbsnap-01 kernel: __x64_sys_fsync+0x14/0x20
Jul 19 12:20:13 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 12:20:13 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 12:20:13 prod-dbsnap-01 kernel: RIP: 0033:0x7fb113087d00
Jul 19 12:20:13 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 12:20:13 prod-dbsnap-01 kernel: RSP: 002b:00007fffba1597c8 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
Jul 19 12:20:13 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 00000000000003e0 RCX: 00007fb113087d00
Jul 19 12:20:13 prod-dbsnap-01 kernel: RDX: 0000000000000120 RSI: 00007fb0f4668580 RDI: 00000000000003e0
Jul 19 12:20:13 prod-dbsnap-01 kernel: RBP: 00007fb1134937f0 R08: 00007fb0f4668690 R09: 00007fb0f4668588
Jul 19 12:20:13 prod-dbsnap-01 kernel: R10: 959bd756f2ca9fd3 R11: 0000000000000246 R12: 0000006055e08180
Jul 19 12:20:13 prod-dbsnap-01 kernel: R13: 0000000000000001 R14: 000000000000126f R15: 000000000000158d
Jul 19 12:20:13 prod-dbsnap-01 kernel: INFO: task postgres:2135 blocked for more than 122 seconds.
Jul 19 12:20:13 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 12:20:13 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 12:20:13 prod-dbsnap-01 kernel: postgres D 0 2135 1801 0x00004004
Jul 19 12:20:13 prod-dbsnap-01 kernel: Call Trace:
Jul 19 12:20:13 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 12:20:13 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 12:20:13 prod-dbsnap-01 kernel: io_schedule+0x16/0x40
Jul 19 12:20:13 prod-dbsnap-01 kernel: wait_on_page_bit+0x1c0/0x270
Jul 19 12:20:13 prod-dbsnap-01 kernel: ? file_fdatawait_range+0x30/0x30
Jul 19 12:20:13 prod-dbsnap-01 kernel: wait_on_page_writeback+0x48/0xa0
Jul 19 12:20:13 prod-dbsnap-01 kernel: extent_write_cache_pages+0x214/0x4c0 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: extent_writepages+0x4f/0x90 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: ? tcp_push+0xde/0x110
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_writepages+0xe/0x10 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: do_writepages+0x1f/0x70
Jul 19 12:20:13 prod-dbsnap-01 kernel: __filemap_fdatawrite_range+0xc6/0x100
Jul 19 12:20:13 prod-dbsnap-01 kernel: filemap_fdatawrite_range+0x13/0x20
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_fdatawrite_range+0x20/0x50 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: start_ordered_ops+0x4c/0x90 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: ? __sys_sendto+0x107/0x1a0
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_sync_file+0xce/0x410 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 12:20:13 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 12:20:13 prod-dbsnap-01 kernel: __x64_sys_fdatasync+0x17/0x20
Jul 19 12:20:13 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 12:20:13 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 12:20:13 prod-dbsnap-01 kernel: RIP: 0033:0x7f866870a310
Jul 19 12:20:13 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 12:20:13 prod-dbsnap-01 kernel: RSP: 002b:00007ffdc5243748 EFLAGS: 00000246 ORIG_RAX: 000000000000004b
Jul 19 12:20:13 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000009312 RCX: 00007f866870a310
Jul 19 12:20:13 prod-dbsnap-01 kernel: RDX: 0000000000000027 RSI: 0000000000009312 RDI: 0000000000000004
Jul 19 12:20:13 prod-dbsnap-01 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jul 19 12:20:13 prod-dbsnap-01 kernel: R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000004
Jul 19 12:20:13 prod-dbsnap-01 kernel: R13: 0000000002c2a009 R14: 00007f865856b280 R15: 0000000000002430
Jul 19 12:20:13 prod-dbsnap-01 kernel: INFO: task postgres:2165 blocked for more than 122 seconds.
Jul 19 12:20:13 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 12:20:13 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 12:20:13 prod-dbsnap-01 kernel: postgres D 0 2165 1796 0x00004000
Jul 19 12:20:13 prod-dbsnap-01 kernel: Call Trace:
Jul 19 12:20:13 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 12:20:13 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 12:20:13 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 12:20:13 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 12:20:13 prod-dbsnap-01 kernel: __x64_sys_fdatasync+0x17/0x20
Jul 19 12:20:13 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 12:20:13 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 12:20:13 prod-dbsnap-01 kernel: RIP: 0033:0x7f49e5b64310
Jul 19 12:20:13 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 12:20:13 prod-dbsnap-01 kernel: RSP: 002b:00007ffd950c4088 EFLAGS: 00000246 ORIG_RAX: 000000000000004b
Jul 19 12:20:13 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000019ac1 RCX: 00007f49e5b64310
Jul 19 12:20:13 prod-dbsnap-01 kernel: RDX: 0000000000000027 RSI: 0000000000019ac1 RDI: 0000000000000004
Jul 19 12:20:13 prod-dbsnap-01 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jul 19 12:20:13 prod-dbsnap-01 kernel: R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000004
Jul 19 12:20:13 prod-dbsnap-01 kernel: R13: 0000000002c370c9 R14: 00007f49d59c5280 R15: 000000000000abb0
Jul 19 12:24:18 prod-dbsnap-01 kernel: INFO: task postgres:1961 blocked for more than 122 seconds.
Jul 19 12:24:18 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 12:24:18 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 12:24:18 prod-dbsnap-01 kernel: postgres D 0 1961 1807 0x00004000
Jul 19 12:24:18 prod-dbsnap-01 kernel: Call Trace:
Jul 19 12:24:18 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 12:24:18 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 12:24:18 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 12:24:18 prod-dbsnap-01 kernel: __x64_sys_fsync+0x14/0x20
Jul 19 12:24:18 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 12:24:18 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 12:24:18 prod-dbsnap-01 kernel: RIP: 0033:0x7f99ee448d00
Jul 19 12:24:18 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 12:24:18 prod-dbsnap-01 kernel: RSP: 002b:00007ffdb772d008 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
Jul 19 12:24:18 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000000389 RCX: 00007f99ee448d00
Jul 19 12:24:18 prod-dbsnap-01 kernel: RDX: 00007f99df2a8070 RSI: 000000000a00000e RDI: 000000000000038f
Jul 19 12:24:18 prod-dbsnap-01 kernel: RBP: 000000000a00000e R08: 000000000007cc24 R09: 00007ffdb77810b0
Jul 19 12:24:18 prod-dbsnap-01 kernel: R10: 000000000000010c R11: 0000000000000246 R12: 0000000000000009
Jul 19 12:24:18 prod-dbsnap-01 kernel: R13: 00000000010ee588 R14: 00000000010a7190 R15: 00000000010a7190
Jul 19 12:24:18 prod-dbsnap-01 kernel: INFO: task postgres:2032 blocked for more than 122 seconds.
Jul 19 12:24:18 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 12:24:18 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 12:24:18 prod-dbsnap-01 kernel: postgres D 0 2032 1796 0x00004000
Jul 19 12:24:18 prod-dbsnap-01 kernel: Call Trace:
Jul 19 12:24:18 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 12:24:18 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 12:24:18 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 12:24:18 prod-dbsnap-01 kernel: __x64_sys_fsync+0x14/0x20
Jul 19 12:24:18 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 12:24:18 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 12:24:18 prod-dbsnap-01 kernel: RIP: 0033:0x7f49e7ff0d00
Jul 19 12:24:18 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 12:24:18 prod-dbsnap-01 kernel: RSP: 002b:00007ffd950c3008 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
Jul 19 12:24:18 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 00000000000003e0 RCX: 00007f49e7ff0d00
Jul 19 12:24:18 prod-dbsnap-01 kernel: RDX: 0000000000000120 RSI: 00007f49c95d1580 RDI: 00000000000003e0
Jul 19 12:24:18 prod-dbsnap-01 kernel: RBP: 00007f49e83fc7f0 R08: 00007f49c95d1690 R09: 00007f49c95d1588
Jul 19 12:24:18 prod-dbsnap-01 kernel: R10: 959bd756f2ca9fd3 R11: 0000000000000246 R12: 0000019ac2a59b20
Jul 19 12:24:18 prod-dbsnap-01 kernel: R13: 0000000000000001 R14: 00000000000002de R15: 0000000000000307
Jul 19 12:24:18 prod-dbsnap-01 kernel: INFO: task postgres:2061 blocked for more than 122 seconds.
Jul 19 12:24:18 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 12:24:18 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 12:24:18 prod-dbsnap-01 kernel: postgres D 0 2061 1808 0x00004000
Jul 19 12:24:18 prod-dbsnap-01 kernel: Call Trace:
Jul 19 12:24:18 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 12:24:18 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 12:24:18 prod-dbsnap-01 kernel: lock_and_cleanup_extent_if_need+0x1b7/0x230 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_buffered_write.isra.25+0x22e/0x710 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_file_write_iter+0x20d/0x580 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: new_sync_write+0x120/0x1a0
Jul 19 12:24:18 prod-dbsnap-01 kernel: __vfs_write+0x29/0x40
Jul 19 12:24:18 prod-dbsnap-01 kernel: vfs_write+0xb2/0x1b0
Jul 19 12:24:18 prod-dbsnap-01 kernel: ksys_write+0x61/0xd0
Jul 19 12:24:18 prod-dbsnap-01 kernel: __x64_sys_write+0x1a/0x20
Jul 19 12:24:18 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 12:24:18 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 12:24:18 prod-dbsnap-01 kernel: RIP: 0033:0x7fb113087680
Jul 19 12:24:18 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 12:24:18 prod-dbsnap-01 kernel: RSP: 002b:00007fffba159788 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
Jul 19 12:24:18 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 00007fb1133e4ad0 RCX: 00007fb113087680
Jul 19 12:24:18 prod-dbsnap-01 kernel: RDX: 0000000000002000 RSI: 00000000014bb758 RDI: 00000000000003a7
Jul 19 12:24:18 prod-dbsnap-01 kernel: RBP: 0000000000002000 R08: 0000000000000009 R09: 0000000000000000
Jul 19 12:24:18 prod-dbsnap-01 kernel: R10: 959bd756f2ca9fd3 R11: 0000000000000246 R12: 0000000000002000
Jul 19 12:24:18 prod-dbsnap-01 kernel: R13: 00000000014bb758 R14: 000000000a000010 R15: 00007fb1134937f0
Jul 19 12:24:18 prod-dbsnap-01 kernel: INFO: task postgres:2444 blocked for more than 122 seconds.
Jul 19 12:24:18 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 12:24:18 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 12:24:18 prod-dbsnap-01 kernel: postgres D 0 2444 1796 0x00004004
Jul 19 12:24:18 prod-dbsnap-01 kernel: Call Trace:
Jul 19 12:24:18 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 12:24:18 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 12:24:18 prod-dbsnap-01 kernel: io_schedule+0x16/0x40
Jul 19 12:24:18 prod-dbsnap-01 kernel: wait_on_page_bit+0x1c0/0x270
Jul 19 12:24:18 prod-dbsnap-01 kernel: ? file_fdatawait_range+0x30/0x30
Jul 19 12:24:18 prod-dbsnap-01 kernel: wait_on_page_writeback+0x48/0xa0
Jul 19 12:24:18 prod-dbsnap-01 kernel: extent_write_cache_pages+0x214/0x4c0 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: extent_writepages+0x4f/0x90 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: ? tcp_push+0xde/0x110
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_writepages+0xe/0x10 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: do_writepages+0x1f/0x70
Jul 19 12:24:18 prod-dbsnap-01 kernel: __filemap_fdatawrite_range+0xc6/0x100
Jul 19 12:24:18 prod-dbsnap-01 kernel: filemap_fdatawrite_range+0x13/0x20
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_fdatawrite_range+0x20/0x50 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: start_ordered_ops+0x4c/0x90 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: ? __sys_sendto+0x107/0x1a0
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_sync_file+0xce/0x410 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 12:24:18 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 12:24:18 prod-dbsnap-01 kernel: __x64_sys_fdatasync+0x17/0x20
Jul 19 12:24:18 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 12:24:18 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 12:24:18 prod-dbsnap-01 kernel: RIP: 0033:0x7f49e5b64310
Jul 19 12:24:18 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 12:24:18 prod-dbsnap-01 kernel: RSP: 002b:00007ffd950c4088 EFLAGS: 00000246 ORIG_RAX: 000000000000004b
Jul 19 12:24:18 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000019ac2 RCX: 00007f49e5b64310
Jul 19 12:24:18 prod-dbsnap-01 kernel: RDX: 0000000000000027 RSI: 0000000000019ac2 RDI: 0000000000000004
Jul 19 12:24:18 prod-dbsnap-01 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jul 19 12:24:18 prod-dbsnap-01 kernel: R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000004
Jul 19 12:24:18 prod-dbsnap-01 kernel: R13: 0000000002c208f9 R14: 00007f49d59c5280 R15: 0000000000003d20
Jul 19 12:24:18 prod-dbsnap-01 kernel: INFO: task postgres:2445 blocked for more than 122 seconds.
Jul 19 12:24:18 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 12:24:18 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 12:24:18 prod-dbsnap-01 kernel: postgres D 0 2445 1801 0x00004000
Jul 19 12:24:18 prod-dbsnap-01 kernel: Call Trace:
Jul 19 12:24:18 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 12:24:18 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_start_ordered_extent+0x104/0x140 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_wait_ordered_range+0xca/0x110 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: btrfs_sync_file+0x1f8/0x410 [btrfs]
Jul 19 12:24:18 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 12:24:18 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 12:24:18 prod-dbsnap-01 kernel: __x64_sys_fdatasync+0x17/0x20
Jul 19 12:24:18 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 12:24:18 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 12:24:18 prod-dbsnap-01 kernel: RIP: 0033:0x7f866870a310
Jul 19 12:24:18 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 12:24:18 prod-dbsnap-01 kernel: RSP: 002b:00007ffdc5243748 EFLAGS: 00000246 ORIG_RAX: 000000000000004b
Jul 19 12:24:18 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000009312 RCX: 00007f866870a310
Jul 19 12:24:18 prod-dbsnap-01 kernel: RDX: 0000000000000027 RSI: 0000000000009312 RDI: 0000000000000004
Jul 19 12:24:18 prod-dbsnap-01 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jul 19 12:24:18 prod-dbsnap-01 kernel: R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000004
Jul 19 12:24:18 prod-dbsnap-01 kernel: R13: 0000000002c30571 R14: 00007f865856b280 R15: 0000000000000038
Jul 19 13:02:23 prod-dbsnap-01 kernel: INFO: task btrfs-cleaner:1400 blocked for more than 122 seconds.
Jul 19 13:02:23 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 13:02:23 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 13:02:23 prod-dbsnap-01 kernel: btrfs-cleaner D 0 1400 2 0x80004000
Jul 19 13:02:23 prod-dbsnap-01 kernel: Call Trace:
Jul 19 13:02:23 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 13:02:23 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 13:02:23 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 13:02:23 prod-dbsnap-01 kernel: start_transaction+0x377/0x4a0 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: btrfs_start_transaction+0x1e/0x20 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: btrfs_drop_snapshot+0x500/0x960 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: btrfs_clean_one_deleted_snapshot+0xcb/0x120 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: cleaner_kthread+0xeb/0x140 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 13:02:23 prod-dbsnap-01 kernel: ? btree_invalidatepage+0xd0/0xd0 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 13:02:23 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
Jul 19 13:02:23 prod-dbsnap-01 kernel: INFO: task postgres:3423 blocked for more than 122 seconds.
Jul 19 13:02:23 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 13:02:23 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 13:02:23 prod-dbsnap-01 kernel: postgres D 0 3423 3269 0x00004004
Jul 19 13:02:23 prod-dbsnap-01 kernel: Call Trace:
Jul 19 13:02:23 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 13:02:23 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 13:02:23 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 13:02:23 prod-dbsnap-01 kernel: start_transaction+0x377/0x4a0 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: btrfs_start_transaction+0x1e/0x20 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: btrfs_sync_file+0x335/0x410 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 13:02:23 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 13:02:23 prod-dbsnap-01 kernel: __x64_sys_fdatasync+0x17/0x20
Jul 19 13:02:23 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 13:02:23 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 13:02:23 prod-dbsnap-01 kernel: RIP: 0033:0x7fb03c2f2310
Jul 19 13:02:23 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 13:02:23 prod-dbsnap-01 kernel: RSP: 002b:00007ffdbacb2988 EFLAGS: 00000246 ORIG_RAX: 000000000000004b
Jul 19 13:02:23 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 000000000000931e RCX: 00007fb03c2f2310
Jul 19 13:02:23 prod-dbsnap-01 kernel: RDX: 0000000000000027 RSI: 000000000000931e RDI: 0000000000000004
Jul 19 13:02:23 prod-dbsnap-01 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jul 19 13:02:23 prod-dbsnap-01 kernel: R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000004
Jul 19 13:02:23 prod-dbsnap-01 kernel: R13: 0000000000ee19f1 R14: 00007fb02c153280 R15: 0000000000000f08
Jul 19 13:02:23 prod-dbsnap-01 kernel: INFO: task postgres:4039 blocked for more than 122 seconds.
Jul 19 13:02:23 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 13:02:23 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 13:02:23 prod-dbsnap-01 kernel: postgres D 0 4039 4036 0x00004000
Jul 19 13:02:23 prod-dbsnap-01 kernel: Call Trace:
Jul 19 13:02:23 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 13:02:23 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 13:02:23 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 13:02:23 prod-dbsnap-01 kernel: start_transaction+0x377/0x4a0 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: btrfs_start_transaction+0x1e/0x20 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: btrfs_sync_file+0x335/0x410 [btrfs]
Jul 19 13:02:23 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 13:02:23 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 13:02:23 prod-dbsnap-01 kernel: __x64_sys_fsync+0x14/0x20
Jul 19 13:02:23 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 13:02:23 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 13:02:23 prod-dbsnap-01 kernel: RIP: 0033:0x7f31f038bd00
Jul 19 13:02:23 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 13:02:23 prod-dbsnap-01 kernel: RSP: 002b:00007ffff1c80508 EFLAGS: 00000246 ORIG_RAX: 000000000000004a
Jul 19 13:02:23 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007f31f038bd00
Jul 19 13:02:23 prod-dbsnap-01 kernel: RDX: 0000000000000180 RSI: 0000000000000002 RDI: 0000000000000003
Jul 19 13:02:23 prod-dbsnap-01 kernel: RBP: 0000000000000000 R08: 4231303030303030 R09: 00007f31ede5720d
Jul 19 13:02:23 prod-dbsnap-01 kernel: R10: 00007f31ee1d1c8c R11: 0000000000000246 R12: 00007ffff1c80ed0
Jul 19 13:02:23 prod-dbsnap-01 kernel: R13: 0000000000000000 R14: 0000000000000014 R15: 00007ffff1c80ed0
Jul 19 13:04:26 prod-dbsnap-01 kernel: INFO: task btrfs-cleaner:1400 blocked for more than 245 seconds.
Jul 19 13:04:26 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 13:04:26 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs-cleaner D 0 1400 2 0x80004000
Jul 19 13:04:26 prod-dbsnap-01 kernel: Call Trace:
Jul 19 13:04:26 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 13:04:26 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 13:04:26 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 13:04:26 prod-dbsnap-01 kernel: start_transaction+0x377/0x4a0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_start_transaction+0x1e/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_drop_snapshot+0x500/0x960 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_clean_one_deleted_snapshot+0xcb/0x120 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: cleaner_kthread+0xeb/0x140 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? btree_invalidatepage+0xd0/0xd0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 13:04:26 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
Jul 19 13:04:26 prod-dbsnap-01 kernel: INFO: task kworker/u256:3:2678 blocked for more than 122 seconds.
Jul 19 13:04:26 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 13:04:26 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 13:04:26 prod-dbsnap-01 kernel: kworker/u256:3 D 0 2678 2 0x80004000
Jul 19 13:04:26 prod-dbsnap-01 kernel: Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: Call Trace:
Jul 19 13:04:26 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 13:04:26 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 13:04:26 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 13:04:26 prod-dbsnap-01 kernel: start_transaction+0x316/0x4a0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_join_transaction+0x1d/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_finish_ordered_io+0x45a/0x7c0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? x86_gsbase_write_task+0x20/0x30
Jul 19 13:04:26 prod-dbsnap-01 kernel: finish_ordered_fn+0x15/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: normal_work_helper+0xc5/0x320 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? pwq_activate_delayed_work+0x42/0xb0
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_endio_write_helper+0x12/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: process_one_work+0x179/0x3a0
Jul 19 13:04:26 prod-dbsnap-01 kernel: worker_thread+0x4f/0x3e0
Jul 19 13:04:26 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? max_active_store+0x80/0x80
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 13:04:26 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
Jul 19 13:04:26 prod-dbsnap-01 kernel: INFO: task kworker/u256:8:2683 blocked for more than 122 seconds.
Jul 19 13:04:26 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 13:04:26 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 13:04:26 prod-dbsnap-01 kernel: kworker/u256:8 D 0 2683 2 0x80004000
Jul 19 13:04:26 prod-dbsnap-01 kernel: Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: Call Trace:
Jul 19 13:04:26 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 13:04:26 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 13:04:26 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 13:04:26 prod-dbsnap-01 kernel: start_transaction+0x316/0x4a0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_join_transaction+0x1d/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_finish_ordered_io+0x45a/0x7c0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: finish_ordered_fn+0x15/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: normal_work_helper+0xc5/0x320 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? pwq_activate_delayed_work+0x42/0xb0
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_endio_write_helper+0x12/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: process_one_work+0x179/0x3a0
Jul 19 13:04:26 prod-dbsnap-01 kernel: worker_thread+0x4f/0x3e0
Jul 19 13:04:26 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? max_active_store+0x80/0x80
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 13:04:26 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
Jul 19 13:04:26 prod-dbsnap-01 kernel: INFO: task kworker/u256:9:2684 blocked for more than 122 seconds.
Jul 19 13:04:26 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 13:04:26 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 13:04:26 prod-dbsnap-01 kernel: kworker/u256:9 D 0 2684 2 0x80004000
Jul 19 13:04:26 prod-dbsnap-01 kernel: Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: Call Trace:
Jul 19 13:04:26 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 13:04:26 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 13:04:26 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 13:04:26 prod-dbsnap-01 kernel: start_transaction+0x316/0x4a0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_join_transaction+0x1d/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_finish_ordered_io+0x45a/0x7c0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? __switch_to_asm+0x40/0x70
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? __switch_to_asm+0x40/0x70
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? x86_gsbase_write_task+0x20/0x30
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? __switch_to_asm+0x34/0x70
Jul 19 13:04:26 prod-dbsnap-01 kernel: finish_ordered_fn+0x15/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: normal_work_helper+0xc5/0x320 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? pwq_activate_delayed_work+0x42/0xb0
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_endio_write_helper+0x12/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: process_one_work+0x179/0x3a0
Jul 19 13:04:26 prod-dbsnap-01 kernel: worker_thread+0x4f/0x3e0
Jul 19 13:04:26 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? max_active_store+0x80/0x80
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 13:04:26 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
Jul 19 13:04:26 prod-dbsnap-01 kernel: INFO: task kworker/u256:13:3321 blocked for more than 122 seconds.
Jul 19 13:04:26 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 13:04:26 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 13:04:26 prod-dbsnap-01 kernel: kworker/u256:13 D 0 3321 2 0x80004000
Jul 19 13:04:26 prod-dbsnap-01 kernel: Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: Call Trace:
Jul 19 13:04:26 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 13:04:26 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 13:04:26 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 13:04:26 prod-dbsnap-01 kernel: start_transaction+0x316/0x4a0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_join_transaction+0x1d/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_finish_ordered_io+0x45a/0x7c0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: finish_ordered_fn+0x15/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: normal_work_helper+0xc5/0x320 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? pwq_activate_delayed_work+0x42/0xb0
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_endio_write_helper+0x12/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: process_one_work+0x179/0x3a0
Jul 19 13:04:26 prod-dbsnap-01 kernel: worker_thread+0x4f/0x3e0
Jul 19 13:04:26 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? max_active_store+0x80/0x80
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 13:04:26 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
Jul 19 13:04:26 prod-dbsnap-01 kernel: INFO: task postgres:3423 blocked for more than 245 seconds.
Jul 19 13:04:26 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 13:04:26 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 13:04:26 prod-dbsnap-01 kernel: postgres D 0 3423 3269 0x00004004
Jul 19 13:04:26 prod-dbsnap-01 kernel: Call Trace:
Jul 19 13:04:26 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 13:04:26 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 13:04:26 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 13:04:26 prod-dbsnap-01 kernel: start_transaction+0x377/0x4a0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_start_transaction+0x1e/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_sync_file+0x335/0x410 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: vfs_fsync_range+0x3f/0x80
Jul 19 13:04:26 prod-dbsnap-01 kernel: do_fsync+0x3d/0x70
Jul 19 13:04:26 prod-dbsnap-01 kernel: __x64_sys_fdatasync+0x17/0x20
Jul 19 13:04:26 prod-dbsnap-01 kernel: do_syscall_64+0x60/0x1c0
Jul 19 13:04:26 prod-dbsnap-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 19 13:04:26 prod-dbsnap-01 kernel: RIP: 0033:0x7fb03c2f2310
Jul 19 13:04:26 prod-dbsnap-01 kernel: Code: Bad RIP value.
Jul 19 13:04:26 prod-dbsnap-01 kernel: RSP: 002b:00007ffdbacb2988 EFLAGS: 00000246 ORIG_RAX: 000000000000004b
Jul 19 13:04:26 prod-dbsnap-01 kernel: RAX: ffffffffffffffda RBX: 000000000000931e RCX: 00007fb03c2f2310
Jul 19 13:04:26 prod-dbsnap-01 kernel: RDX: 0000000000000027 RSI: 000000000000931e RDI: 0000000000000004
Jul 19 13:04:26 prod-dbsnap-01 kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jul 19 13:04:26 prod-dbsnap-01 kernel: R10: 0000000000004000 R11: 0000000000000246 R12: 0000000000000004
Jul 19 13:04:26 prod-dbsnap-01 kernel: R13: 0000000000ee19f1 R14: 00007fb02c153280 R15: 0000000000000f08
Jul 19 13:04:26 prod-dbsnap-01 kernel: INFO: task kworker/u256:1:3787 blocked for more than 122 seconds.
Jul 19 13:04:26 prod-dbsnap-01 kernel: Not tainted 5.2.1-1.el7.elrepo.x86_64 #1
Jul 19 13:04:26 prod-dbsnap-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 13:04:26 prod-dbsnap-01 kernel: kworker/u256:1 D 0 3787 2 0x80004000
Jul 19 13:04:26 prod-dbsnap-01 kernel: Workqueue: btrfs-endio-write btrfs_endio_write_helper [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: Call Trace:
Jul 19 13:04:26 prod-dbsnap-01 kernel: __schedule+0x2a9/0x650
Jul 19 13:04:26 prod-dbsnap-01 kernel: schedule+0x38/0xa0
Jul 19 13:04:26 prod-dbsnap-01 kernel: wait_current_trans+0xc5/0x100 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? remove_wait_queue+0x60/0x60
Jul 19 13:04:26 prod-dbsnap-01 kernel: start_transaction+0x316/0x4a0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_join_transaction+0x1d/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_finish_ordered_io+0x45a/0x7c0 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: finish_ordered_fn+0x15/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: normal_work_helper+0xc5/0x320 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? pwq_activate_delayed_work+0x42/0xb0
Jul 19 13:04:26 prod-dbsnap-01 kernel: btrfs_endio_write_helper+0x12/0x20 [btrfs]
Jul 19 13:04:26 prod-dbsnap-01 kernel: process_one_work+0x179/0x3a0
Jul 19 13:04:26 prod-dbsnap-01 kernel: worker_thread+0x4f/0x3e0
Jul 19 13:04:26 prod-dbsnap-01 kernel: kthread+0x105/0x140
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? max_active_store+0x80/0x80
Jul 19 13:04:26 prod-dbsnap-01 kernel: ? kthread_bind+0x20/0x20
Jul 19 13:04:26 prod-dbsnap-01 kernel: ret_from_fork+0x35/0x40
