Re: RAID extremely slow
|[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Thank you very much for taking the time to look into this. On 07/25/2012 06:00 PM, Phil Turmel wrote:
Piles of small reads scattered across multiple drives, and a concentration of queued writes to /dev/sda. What's on /dev/sda? It's not a member of the raid, so it must be some other system task involved.
/dev/sda1 is the root filesystem. The writes were most likely by MySQL, but I would have to run iotop to be sure.
[ The output of "lsdrv"  might be useful here, along with "mdadm -D /dev/md0" and "mdadm -E /dev/[b-j]" ]
Here you go: http://pastebin.ca/2174740
MythTV is trying to flush recorded video to disk, I presume. Sync is known to cause stalls--a great deal of work is on-going to improve this. How old is this kernel?
After rebooting, MythTV is currently recording two shows, and the resync is running at full speed.
# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4]md0 : active raid6 sdh1 sdd1 sde1 sdb1 sdi1 sdc1 sdf1 sdg1 sdj1 6837311488 blocks super 1.2 level 6, 512k chunk, algorithm 2 [9/9] [UUUUUUUUU] [=>...................] resync = 9.3% (91363840/976758784) finish=1434.3min speed=10287K/sec
unused devices: <none>atop shows the avio of all the drives to be less than 1ms, where before they were much higher. It will run for a couple days under load just fine, and then it will come to a halt.
It's a 3.2.21 kernel. I'm running Debian Testing, and the exact Debian package version is:
ii linux-image-3.2.0-3-686-pae 3.2.21-3 Linux 3.2 for modern PCs
[51000.672258] [<c12c409f>] ? sysenter_do_call+0x12/0x28 [51000.672261] [<c12b0000>] ? quirk_usb_early_handoff+0x4a9/0x522 Here is some other possibly relevant info: # cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid6 sdh1 sdd1 sde1 sdb1 sdi1 sdc1 sdf1 sdg1 sdj1 6837311488 blocks super 1.2 level 6, 512k chunk, algorithm 2 [9/9] [UUUUUUUUU] [==========>..........] resync = 51.3% (501954432/976758784) finish=28755.6min speed=275K/secIs this resync a weekly check, or did something else trigger it?
This is not a scheduled check. It was triggered by, I believe, an unclean shutdown. An unclean shutdown will trigger a resync. I don't think it used to do this, but I could be remembering wrong.
unused devices:<none> # cat /proc/sys/dev/raid/speed_limit_min 10000MD is unable to reach its minimum rebuild rate while other system activity is ongoing. You might want to lower this number to see if that gets you out of the stalls. Or temporarily shut down mythtv.
I will try lowering those numbers next time this happens, which will probably be within the next day or two. That's about how often this happens.
# cat /proc/sys/dev/raid/speed_limit_max 200000 Thanks in advance! -- KevinHTH, Phil  http://github.com/pturmel/lsdrv
Thanks! -- Kevin -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html