Hi, Robert, Thanks for the response. Many of the things you mentioned I have tried, but for completeness: > Have you taken SMART (smartmotools etc) to these disks Yes. The disks are actually connected to a proper hardware RAID controller that does SMART monitoring of all the disks, although I don't use the RAID features of the controller. By using mdadm, if the controller fails I can slap the disks in another machine, or a different controller into this one, and still have it work without needing to worry about getting a replacement for this particular model of controller. There are no errors or warnings from SMART for the disks. > Try experimentally mounting the filesystem read-only Actually, I'd already done that before I mailed the list. It made no difference to the symptoms. > Have you changed any hardware lately in a way that could de-optimize > your interrupt handling. No. > I have a vague recollection that somewhere in the last month and a > half or so there was a patch here (or in the kernel changelogs) > about an extra put operation (or something) that would cause a > worker thread to roll over to -1, then spin back down to zero before > work could proceed. I know, could I _be_ more vague? Right? Try > switching to kernel 3.18.1 to see if the issue just goes away. I tend to track linux-stable pretty closely (as that seems to be recommended for btrfs use), so I already switched to 3.18.1 as soon as it came out. That made no difference to the symptoms either. > When was the last time you did any of the maintenance things (like > balance or defrag)? Not that I'd want to sit through 15Tb of that > sort of thing, but I'm curious about the maintenance history. I don't generally do those at all. I was under the impression that balance would not apply in my case as btrfs is on a single logical device, but I see that I was wrong in that impression. Is this something that is recommended on a regular basis? Most of the advice I've read regarding them is that it's no longer necessary unless there is a particular problem that these will fix... > Does the read performance fall off with uptime? No. I see these problems right from boot. > I _imagine_ that if your filesystem huge and your server is modest by > comparison in terms of ram, cache pinning and fragmentation can start > becoming a real problem. What else besides marshaling this filesystem is > this system used for? This particular server is only used for holding backups of other machines, nothing else. It has far more CPU and memory (2x quad-core Xeon plus hyperthreading, 16GB RAM) than it needs for this task. So when I say the machine is doing nothing other than this copy/rsync I'm currently running, that's practically the literal truth - there are the normal system processes and my ssh/shell running and that's about it. > Have you tried segregating some of your system memory for to make > sure that you aren't actually having application performance issues? The system isn't running out of memory; as I say, about the only userspace processes running are ssh, my shell, and rsync. However, your first suggestion caused me to slap myself: > Have you tried increasing the number of stripe buffers for the > filesystem? This I had totally forgotten. When I bump up the stripe cache size, it *seems* (so far, at least) to eliminate the slowest performance I'm seeing - specifically, the periods I've been seeing where no I/O at all seems to happen, plus the long runs of 1-3MB/s. The copy is now staying pretty much in the 22-27MB/s range. That's not as fast as the hardware is capable of - as I say, with other filesystems on the same hardware, I can easily see 100+MB/s - but it's much better than it was. Is this remaining difference (25 vs 100+ MB/s) simply due to btrfs not being tuned for performance yet, or is there something else I'm probably overlooking? Thanks, Charles -- ----------------------------------------------------------------------- Charles Cazabon GPL'ed software available at: http://pyropus.ca/software/ ----------------------------------------------------------------------- -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
