Very slow writing and low I/O throuput

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi there,

I'm a maintainer of a linux mirror site, and we use btrfs snapshots provide better service. The
workflow of our mirror job is :
1. create a working snapshot of current mirror
2. rsync from upstream to the working snapshot
3. after rsync done, remove the original subvolume and mv working snapshot to production subvolume

Everything was fine until several days ago, every `btrfs subvolume snapshot` command takes more than
1 minute to finish, and during this 1 minitue, every write to the disk is extremly slow (< 1MB/s).

Also, when the disk is totally idel, if I tried to  `dd if=/dev/zero of=test bs=1M count=100 conv=fdatasync`,
this `dd` just hangs, and iostat told me the disk is reading at speed of ~5MB/s, not a bit was
writtien to the disk.

Smartctl showed the disk is fine, and btrfs device stat reports no error. This server is a KVM
virtual machine and btrfs is on two 4T qcow2 images, with virtio cache=none and io=native.

Uname -a is
Linux debian-opensource 3.16.0-0.bpo.4-amd64 #1 SMP Debian 3.16.7-ckt9-3~deb8u1~bpo70+1 (2015-04-27) x86_64 GNU/Linux

Btrfs fi show is
Label: none  uuid: 179cb1ff-fe45-4e02-996a-a9ad02f766b9
Total devices 2 FS bytes used 5.46TiB
devid    1 size 4.00TiB used 3.08TiB path /dev/vdb
devid    2 size 4.00TiB used 2.41TiB path /dev/vdc

Btrfs v3.14.1

SysRq+w reports: http://pastebin.com/ZRPwZiUQ

Mount Options: (rw,noatime,space_cache,autodefrag)

No dmesg errors.

So I wanna know:
1. Why is it slow?
2. Is it bad to use btrfs inside KVM with qcow2?
3. Everything was fast before, can I somehow "reset" the filesystem without losing data?


Cheers,
-- 
Justin Wong

Blog: https://bigeagle.me/
Fingerprint: 15CC 6A61 738B 1599 0095  E256 CB67 DA7A 865B AC3A

Attachment: signature.asc
Description: PGP signature


[Index of Archives]     [Linux Filesystem Development]     [Linux NFS]     [Linux NILFS]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux