Re: Ext3/ext4 in a clustered environement
|[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]|
I've seen significant a performance drop with ext3 (and other) filesystems with 10s to 100s of thousands of files per directory. Make sure that the"directory hash" option is enabled for ext3. With ~1M files per directory, I'ddo some performance tests comparing rsync under ext3, ext4, and gfs beforchanging filesystems...while ext3/4 do perform better than gfs, the directorysize may be such an overwhelming factor that the filesystem choice is irrelevent.
Get me right, there are millions of files, but no more than a few hundreds per directory. They are spread out splited on the database id, 2 caracters at a time. So a file name 1234567.jpg would end up in a directory 12/34/5/, or something similar.
Is this a GFS issue strictly, or an issue with rsync. Have you set up a similar environment under ext3/4 to test jus the rsync part? Rsync is known for being a memory & resource hog, particularly at the initial stage of building the filesystem tree. I would strongly recommend benchmarking rsync on ext3/4 before making the switch. One option would be to do several 'rsync' operations (serially, not in parallel!), each operating on a subset of the filesystem, while continuing to use gfs.
Yes it is a GFS specific, our backup server is on ext3 and rsyncing can be made in a couple of hours, without eating cpu at all (only memory), and without bringing the server on it's knees.
Spliting on subdirectories might be an option, but that would be more like a band-aid... I'll try to avoid that.
=> <fs device="/dev/VGx/documentsA" force_unmount="1" fstype="ext4" => mountpoint="/GFSVolume1/Service1/documentsA" name="documentsA" => options="noatime,quota=off"/> => => So, first, is this doable ? Yes. We have been doing something very similar for the past ~2 years, except not mounting the ext3/4 partition under a GFS mountpoint.
I will be doing some expiriment with that...
=>=> Second, is this risky ? In the sens of that with force_unmont true, I assume => that no other node would mount that filesystem before it is unmounted on the => stopping service. I know that for some reason umount could hang, but it's => not likely since this data is mostly read-only. In that case the serviceWe've experienced numerous cases where the filesystem hangs after a service migration due a node (or service) failover. These hangs all seem to be related to quota or NFS issues, so this may not be an issue in your environment.
While we do not use nfs on top of the 3 most important directories, it will be used on some of those volumes...
=> would be failed and need to be manually restarted. What would be the => consequence if the filesystem happens to be mounted on 2 nodes ? Most likely, filesystem corruption.
Other responses led me to beleive that if I let the cluster manage the filesystem, and never mount it myselef, it's much less likely to happen.
-- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster