Nicolas Ross wrote:
Get me right, there are millions of files, but no more than a few hundreds per directory. They are spread out splited on the database id, 2 caracters at a time. So a file name 1234567.jpg would end up in a directory 12/34/5/, or something similar.
OK, the way you wrote it looked like flat directory spacing.We see appreciable knee points in GFS directory performance at 512, 4096 and 16384 files/directory, with progressively worse performance deterioration between each knee pair. (It's a 2^n type problem)
Yes it is a GFS specific, our backup server is on ext3 and rsyncing can be made in a couple of hours, without eating cpu at all (only memory), and without bringing the server on it's knees.
Have you tuned dentry/inode hashes? Have you got enough memory?Bear in mind that rsync has to (at least) stat() every single file it looks at, which causes multicast locking traffic between the nodes if the FS is mounted on multiple machines - even mounted on a single node, it's slow.
If you can remount the FS with localflock then you'll see performance akin to your ext3 results, but on a single node mount with appropriate network/memory tuning you can at least double the rsync speed over vanilla configuration if there are a few million files involved.
We've experienced numerous cases where the filesystem hangs after a service migration due a node (or service) failover. These hangs all seem to be related to quota or NFS issues, so this may not be an issue in your environment.While we do not use nfs on top of the 3 most important directories, it will be used on some of those volumes...
nfs(v2,3) is old, crufty, non-cluster/multitask aware(*), doesn't play nice with anything else accessing the disk and seems to be the root cause of most of our stability problems.
I can't talk about pNFS (NFSv4) stability as that requires bind mounts which aren't supported in a failover environment - it seems to work on individual nodes but I've never managed to have it working properly on a cluster.
(*) BEWARE if you have multiple services with NFS exports in them, the exportfs commands can play a nasty race game and scribble over the export list in an unpredictable manner. We fixed this with flocking in nfsclient.sh but redhat haven't rolled it into their distribution yet.
=> would be failed and need to be manually restarted. What would be the => consequence if the filesystem happens to be mounted on 2 nodes ? Most likely, filesystem corruption.Other responses led me to beleive that if I let the cluster manage the filesystem, and never mount it myselef, it's much less likely to happen.
Correct... But human factors being what they are added with other possibilities (such as failure to unmount, etc) mean that the chance is significantly higher than zero for my liking on any important FS
-- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster