On 2017年12月07日 09:19, Taibai Li wrote: > thanks for the quick response, I tired to test this on 4.4.100 > kernel, disabled quota : > # btrfs qgroup show /data/ > ERROR: can't list qgroups: quotas not enabled > > But seems it still OOM after about 7 hours copyed 144G files, any > other ideas? Maybe I will try to test by disable quota on 4.14 kernel > too. Trying latest kernel is always a good idea. Despite qgroup, I am not pretty sure which can be the cause. Is there any special mount option used? Thanks, Qu > > thanks. > > On Wed, Dec 6, 2017 at 2:25 PM, Qu Wenruo <quwenruo.btrfs@xxxxxxx> wrote: >> >> >> On 2017年12月06日 14:22, taibai li wrote: >>> Hi Guys, >>> >>> I hit the OOM issues with as Box running 4.4.x kernel, so I tried to >>> build a 4.14.3 kernel to try that. The testbed is : >>> NAS box with 2G memory, and a single disk raid , I setup a nfs server >>> with sync mode, add the storage on ESXi servers 6.0 and backup all >>> the VMs on it by the ghettoVCB script , after about 14 hours, >>> Inpot/Output error happened, >>> checked the box , found OOM. >>> # cat /etc/exports >>> "/data/Videos" *(insecure,insecure_locks,no_subtree_check,crossmnt,anonuid=99,anongid=99,root_squash,rw,sync) >>> # uname -a >>> Linux lzx-314-desk 4.14.3.x86_64.1 #1 SMP Fri Dec 1 01:31:25 UTC 2017 >>> x86_64 GNU/Linux >>> # btrfs fi show /data/ >>> Label: '43f611ae:data' uuid: 6fefb319-a21d-476e-9642-565e0600a049 >>> Total devices 1 FS bytes used 292.78GiB >>> devid 1 size 1.81TiB used 296.02GiB path /dev/md127 >>> >>> The stack is : >>> Dec 04 23:47:43 lzx-314-desk kernel: nfsd: page allocation stalls for >>> 621031ms, order:0, mode:0x14000c0(GFP_KERNEL), nodemask=(null) >>> Dec 04 23:47:43 lzx-314-desk kernel: nfsd cpuset=/ mems_allowed=0 >>> Dec 04 23:47:43 lzx-314-desk kernel: CPU: 0 PID: 3376 Comm: nfsd Not >>> tainted 4.14.3.x86_64.1 #1 >>> Dec 04 23:47:43 lzx-314-desk kernel: Hardware name: NETGEAR ReadyNAS >>> 314/To be filled by O.E.M., BIOS 4.6.5 11/05/2013 >>> Dec 04 23:47:43 lzx-314-desk kernel: Call Trace: >>> Dec 04 23:47:43 lzx-314-desk kernel: dump_stack+0x4d/0x6a >>> Dec 04 23:47:43 lzx-314-desk kernel: warn_alloc+0xe3/0x180 >>> Dec 04 23:47:43 lzx-314-desk kernel: __alloc_pages_nodemask+0xb1e/0xed0 >>> Dec 04 23:47:43 lzx-314-desk kernel: svc_recv+0x99/0x900 >>> Dec 04 23:47:43 lzx-314-desk kernel: ? svc_process+0x241/0x690 >>> Dec 04 23:47:43 lzx-314-desk kernel: nfsd+0xd2/0x150 >>> Dec 04 23:47:43 lzx-314-desk kernel: kthread+0x11a/0x150 >>> Dec 04 23:47:43 lzx-314-desk kernel: ? nfsd_destroy+0x60/0x60 >>> Dec 04 23:47:43 lzx-314-desk kernel: ? kthread_create_on_node+0x40/0x40 >>> Dec 04 23:47:43 lzx-314-desk kernel: ret_from_fork+0x22/0x30 >>> Dec 04 23:47:43 lzx-314-desk kernel: readynasd invoked oom-killer: >>> gfp_mask=0x14200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, >>> oom_score_adj=-1000 >>> Dec 04 23:47:43 lzx-314-desk kernel: readynasd cpuset=/ mems_allowed=0 >>> Dec 04 23:47:43 lzx-314-desk kernel: CPU: 3 PID: 3307 Comm: readynasd >>> Not tainted 4.14.3.x86_64.1 #1 >>> Dec 04 23:47:43 lzx-314-desk kernel: Hardware name: NETGEAR ReadyNAS >>> 314/To be filled by O.E.M., BIOS 4.6.5 11/05/2013 >>> Dec 04 23:47:43 lzx-314-desk kernel: Call Trace: >>> Dec 04 23:47:43 lzx-314-desk kernel: dump_stack+0x4d/0x6a >>> Dec 04 23:47:43 lzx-314-desk kernel: dump_header+0x9a/0x21b >>> Dec 04 23:47:43 lzx-314-desk kernel: ? pick_next_task_fair+0x1d5/0x4b0 >>> Dec 04 23:47:43 lzx-314-desk kernel: ? security_capable_noaudit+0x40/0x60 >>> Dec 04 23:47:43 lzx-314-desk kernel: oom_kill_process+0x216/0x430 >>> Dec 04 23:47:43 lzx-314-desk kernel: out_of_memory+0xf9/0x2e0 >>> Dec 04 23:47:43 lzx-314-desk kernel: __alloc_pages_nodemask+0xd6c/0xed0 >>> Dec 04 23:47:43 lzx-314-desk kernel: __read_swap_cache_async+0x11d/0x190 >>> Dec 04 23:47:43 lzx-314-desk kernel: read_swap_cache_async+0x17/0x40 >>> Dec 04 23:47:43 lzx-314-desk kernel: swapin_readahead+0x1f1/0x230 >>> Dec 04 23:47:43 lzx-314-desk kernel: ? find_get_entry+0x19/0xf0 >>> Dec 04 23:47:43 lzx-314-desk kernel: ? pagecache_get_page+0x27/0x210 >>> Dec 04 23:47:43 lzx-314-desk kernel: do_swap_page+0x432/0x590 >>> Dec 04 23:47:43 lzx-314-desk kernel: ? do_swap_page+0x432/0x590 >>> Dec 04 23:47:43 lzx-314-desk kernel: ? poll_select_copy_remaining+0x120/0x120 >>> Dec 04 23:47:43 lzx-314-desk kernel: __handle_mm_fault+0x33e/0xa20 >>> Dec 04 23:47:43 lzx-314-desk kernel: handle_mm_fault+0x14a/0x1d0 >>> Dec 04 23:47:43 lzx-314-desk kernel: __do_page_fault+0x212/0x440 >>> Dec 04 23:47:43 lzx-314-desk kernel: page_fault+0x22/0x30 >>> Dec 04 23:47:43 lzx-314-desk kernel: RIP: >>> 0010:copy_user_generic_unrolled+0x89/0xc0 >>> Dec 04 23:47:43 lzx-314-desk kernel: RSP: 0000:ffffc90000cdbd70 EFLAGS: 00010202 >>> Dec 04 23:47:43 lzx-314-desk kernel: RAX: 0000000000000000 RBX: >>> 0000000000000008 RCX: 0000000000000001 >>> Dec 04 23:47:43 lzx-314-desk kernel: RDX: 0000000000000000 RSI: >>> ffffc90000cdbdd8 RDI: 00007ff45e7fba80 >>> Dec 04 23:47:43 lzx-314-desk kernel: RBP: ffffc90000cdbee8 R08: >>> 0000000000000000 R09: 0000000000000104 >>> Dec 04 23:47:43 lzx-314-desk kernel: R10: ffffc90000cdbd78 R11: >>> 0000000000000104 R12: 0000000000000000 >>> Dec 04 23:47:43 lzx-314-desk kernel: R13: ffffc90000cdbdc0 R14: >>> 00007ff45e7fba80 R15: ffffc90000cdbdc0 >>> Dec 04 23:47:43 lzx-314-desk kernel: ? core_sys_select+0x208/0x2a0 >>> Dec 04 23:47:43 lzx-314-desk kernel: ? __handle_mm_fault+0x4fc/0xa20 >>> Dec 04 23:47:43 lzx-314-desk kernel: ? ktime_get_ts64+0x44/0xe0 >>> Dec 04 23:47:43 lzx-314-desk kernel: SyS_select+0xa6/0xe0 >>> Dec 04 23:47:43 lzx-314-desk kernel: entry_SYSCALL_64_fastpath+0x13/0x94 >>> Dec 04 23:47:43 lzx-314-desk kernel: RIP: 0033:0x7ff488f05893 >>> Dec 04 23:47:43 lzx-314-desk kernel: RSP: 002b:00007ff45e7fb9f0 >>> EFLAGS: 00000293 ORIG_RAX: 0000000000000017 >>> Dec 04 23:47:43 lzx-314-desk kernel: RAX: ffffffffffffffda RBX: >>> 0000000000000018 RCX: 00007ff488f05893 >>> Dec 04 23:47:43 lzx-314-desk kernel: RDX: 0000000000000000 RSI: >>> 00007ff45e7fba80 RDI: 0000000000000019 >>> Dec 04 23:47:43 lzx-314-desk kernel: RBP: 00007ff48fc4b0f0 R08: >>> 00007ff45e7fba70 R09: 00007ff4480008c0 >>> Dec 04 23:47:43 lzx-314-desk kernel: R10: 0000000000000000 R11: >>> 0000000000000293 R12: 0000000000000000 >>> Dec 04 23:47:43 lzx-314-desk kernel: R13: 0000000000000000 R14: >>> 0000000001999670 R15: 00007ff4480008c0 >>> Dec 04 23:47:43 lzx-314-desk kernel: Mem-Info: >>> Dec 04 23:47:43 lzx-314-desk kernel: active_anon:0 inactive_anon:0 >>> isolated_anon:0 >>> active_file:322 inactive_file:461978 isolated_file:352 >>> unevictable:0 dirty:136 writeback:0 unstable:0 >>> slab_reclaimable:15780 slab_unreclaimable:5246 >>> mapped:1 shmem:0 pagetables:1165 bounce:0 >>> free:13867 free_pcp:60 free_cma:0 >>> ...... >>> >>> I tried to fomat the md device to ext4, then it's fine to backup all >>> the VMs, And if I use async option for nfs, it works too, so seems >>> like btrfs is more memory consuming sometimes. >>> >>> I attatched the full logs. >>> >>> Any one hit similar issue or have any ideas ? >> >> Are you using btrfs qgroups (quota)? >> >> It's known qgroup will take extra memory and may cause OOM if there are >> a lot of extents modified in current transaction. >> >> Thanks, >> Qu >> >>> >>> thanks so much. >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >>
Attachment:
signature.asc
Description: OpenPGP digital signature
