On 12/21/2014 08:32 AM, Charles Cazabon wrote:
Hi, Robert,
Thanks for the response. Many of the things you mentioned I have tried, but
for completeness:
Have you taken SMART (smartmotools etc) to these disks
There are no errors or warnings from SMART for the disks.
Do make sure you are regularly running the long "offline" test. [offline
is a bad name, what it really should be called is the long idle-interval
test. sigh] about once a week. Otherwise SMART is just going to tell you
the disk just died when it dies.
I'm not saying this is relevant to the current circumstance. But since
you didn't mention a testing schedule I figured it bared a mention
Have you tried segregating some of your system memory for to make
sure that you aren't actually having application performance issues?
The system isn't running out of memory; as I say, about the only userspace
processes running are ssh, my shell, and rsync.
The thing with "movablecore=" will not lead to an "out of memory"
condition or not, its a question of cache and buffer evictions.
I figured that you'd have said something about actual out of memory errors.
But here's the thing.
Once storage pressure gets "high enough" the system will start
forgetting things intermittently to make room for other things. One of
the things it will "forget" is pages of code from running programs. The
other thing it can "forget" is dirent (directory entries) relevant to
ongoing activity.
The real killer can involve "swappiness" (e.g. /proc/sys/vm/swapiness ::
the tendency of the system to drop pages of program code, do not adjust
this till you understand it fully) and overall page fault rates on the
system. You'll start geting evictions long before you start using _any_
swap file space.
So if your effective throughput is low, the first thing to really look
at is if your page fault rates are rising. Variations of sar, ps, and
top may be able to tell you about the current system and/or per-process
page fault rates. You'll have to compare your distro's tool set to the
procedures you can find online.
It's a little pernicious because it's a silent performance drain. There
are no system messages to tell you "uh, hey dude, I'm doing a lot of
reclaims lately and even going back to disk for pages of this program
you really like". You just have to know how to look in that area.
However, your first suggestion caused me to slap myself:
Have you tried increasing the number of stripe buffers for the
filesystem?
This I had totally forgotten. When I bump up the stripe cache size, it
*seems* (so far, at least) to eliminate the slowest performance I'm seeing -
specifically, the periods I've been seeing where no I/O at all seems to
happen, plus the long runs of 1-3MB/s. The copy is now staying pretty much in
the 22-27MB/s range.
That's not as fast as the hardware is capable of - as I say, with other
filesystems on the same hardware, I can easily see 100+MB/s - but it's much
better than it was.
Is this remaining difference (25 vs 100+ MB/s) simply due to btrfs not being
tuned for performance yet, or is there something else I'm probably
overlooking?
I find BTRFS can be a little slow on my laptop, but I blame memory
pressure evicting important structures somewhat system wide. Which is
part of why I did the moveablecore= parametric tuning. I don't think
there is anything that will pack the locality of the various trees, so
you can end up needing bits of things from all over your disk in order
to sequentially resolve a large directory and compute the running
checksums for rsync (etc.).
Simple rule of thumb, if "wait for I/O time" has started to rise you've
got some odd memory pressure that's sending you to idle land. It's not
hard-and-fast as a rule, but since you've said that your CPU load (wich
I'm taking to be the user+system time) is staying low you are likely
waiting for something.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
