Re: builder io isue

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]


On 12/25/2011 06:16 AM, Brendan Conoboy wrote:
On 12/24/2011 09:25 PM, Dennis Gilmore wrote:
Just throwing out what im seeing. lets see what ideas we can come up
with. performance is better than before. and seneca has done a great
job and put a lot of hard work into the reorg and im thankful for that.
We just have another bottleneck to address.

Allocating builders to individual rather than a single raid volume will
help dramatically.

Care to explain why?

Since hfp/sfp F15 builds happen at the same time,
having 2 hfp disks and 2 sfp disks is a good start. Split it up again by
some other criteria- you want to try to ensure that any one builder only
causes one spindle to be used and any 2 builders each only cause 2
spindles to be used (1-to-1), and so forth. As long as any single
spindle isn't doing multiple simultaneous mock inits it should scale
pretty well.

You can largely avoid this by making sure your RAID and FS are aligned. Most mock IOPS are small, and provided layers are aligned right, should scale very well on RAID 0/10 (RAID5/6 will inevitably cripple just about anything when it comes to writes regardless of what you do, unless the vast majority of your writes is a multiple of your stripe-width, which is unlikely).

Allocate the large files the builders are using all at once
(no sparse files) to ensure large read/writes don't require seeks.

Is this a "proper" SAN or just another Linux box with some disks in it? Is NFS backed by a SAN "volume"?

Use the async nfs export option if it isn't already in use.

+1

Disable fs
journaling (normally dangerous, but this is throw-away space).

Not really dangerous - the only danger is that you might have to wait for fsck to do it's thing on an unclean shutdown (which can take hours on a full TB scale disk, granted).

Speaking of "dangerous" tweaks, you could LD_PRELOAD libeatmydata (add to a profile.d file in the mock config, and add the package to the buildsys-build group). That will eat fsync() calls which will smooth out commits and make a substantial difference to performance. Since it's scratch space anyway it doesn't matter WRT safety.

Use
noatime mounts if that won't break package builds checking filesystem
conformance.

Build of zsh will break on NFS whateveryou do. It will also break on a local FS with noatime. There may be other packages that suffer from this issue but I don't recall them off the top of my head. Anyway, that is an issue for a build policy - have one builder using block level storage with atime and the rest on NFS.

Once all that is done, tweak the number of nfsds such that
there are as many as possible without most of them going into deep
sleep. Perhaps somebody else can suggest some optimal sysctl and ext4fs
settings?

As mentioned in a previous post, have a look here:
http://www.altechnative.net/?p=96

Deadline scheduler might also help on the NAS/SAN end, plus all the usual tweaks (e.g. make sure write caches on the disks are enabled, if the disks support write-read-verify disable it, etc.)

Gordan
_______________________________________________
arm mailing list
arm@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/arm



[Linux ARM (Vger)]     [Linux ARM]     [ARM Kernel]     [Fedora User Discussion]     [Older Fedora Users Discussion]     [Fedora Advisory Board]     [Fedora Security]     [Fedora Maintainers]     [Fedora Devel Java]     [Fedora Legacy]     [Fedora Desktop]     [ATA RAID]     [Fedora Marketing]     [Fedora Mentors]     [Fedora Package Announce]     [Fedora Package Review]     [Fedora Music]     [Fedora Packaging]     [Centos]     [Fedora SELinux]     [Fedora Triage]     [Deep Creek Hot Springs]     [Coolkey]     [Yum Users]     [Tux]     [Big List of Linux Books]     [Yosemite News]     [Yosemite Photos]     [Linux Apps]     [KDE Users]     [Fedora Tools]     [Fedora Art]     [Fedora Docs]     [Asterisk PBX]

Powered by Linux