Re: builder io isue

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Mon, Jan 2, 2012 at 6:55 PM, Dennis Gilmore <dennis@xxxxxxxx> wrote:
> El Sat, 24 Dec 2011 23:25:56 -0600
> Dennis Gilmore <dennis@xxxxxxxx> escribió:
>> When we were testing build were happening really fast, once we loaded
>> up the build jobs things have become really slow
>>
>> http://arm.koji.fedoraproject.org/koji/taskinfo?taskID=238672 started
>> at 1:19 utc and at 5:18 utc  four hours later the buildrequires are
>> still being installed. australia is completely io bound. i think we
>> need to see how we can spread the io load. ~30 hosts reading and
>> writing to the 4 spindles just saturates the disk io.  i guess options
>> are find a way to add more spindles. move /var/lib/mock to sdcard, see
>> if we can get some kind of san that has alot of smaller fast disks.
>> get some 2.5" usb drives one for each builder. some other idea?  is
>> there anyway we could add 4-8 more disks to australia. the size of the
>> matters little. gaining more iops by adding more spindles would help.
>>
>> Just throwing out what im seeing. lets see what ideas we can come up
>> with.  performance is better than before. and seneca has done a great
>> job and put a lot of hard work into the reorg and im thankful for
>> that. We just have another bottleneck to address.
>> _______________________________________________
>> arm mailing list
>> arm@xxxxxxxxxxxxxxxxxxxxxxx
>> https://admin.fedoraproject.org/mailman/listinfo/arm
>
> http://arm.koji.fedoraproject.org/koji/taskinfo?taskID=245410 has been
> running 10 hours now. its inited the chroot and installing the srpm. it
> is a big srpm but still. performance on the hfp builders while not
> great is better. when looking at a sfp builder the mock process is
> using 100% cpu and in a D state. there is one of two issues here. the
> hfp builders are running a slightly older kernel. so it could be a
> kernel ext3 nfs or loop device issue, upgrading a hfp builder for
> testing would confirm it. or its some issue with decompresion of the
> rpms. I do think that disk io on the nfs server is partly to blame. the
> box sits at a load of 9 all the time with 30% or so io wait. Id like to
> take a layer or 2 of the complexity out. switching over to nfsv4 we
> should be able to run mock on it. im going to do some compression tests
> with the latest 2.6.41.6 kernel.

Nearly two days after I submitted this build its still not actually to
the point of building.

This is a dire situation and doesn't give me much hope of being able
to follow closely behind mainline at this rate.

Peter
_______________________________________________
arm mailing list
arm@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/arm



[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Linux ARM (Vger)]     [Linux ARM]     [ARM Kernel]     [Fedora User Discussion]     [Older Fedora Users Discussion]     [Fedora Advisory Board]     [Fedora Security]     [Fedora Maintainers]     [Fedora Devel Java]     [Fedora Legacy]     [Fedora Desktop]     [ATA RAID]     [Fedora Marketing]     [Fedora Mentors]     [Fedora Package Announce]     [Fedora Package Review]     [Fedora Music]     [Fedora Packaging]     [Centos]     [Fedora SELinux]     [Coolkey]     [Yum Users]     [Tux]     [Yosemite News]     [Linux Apps]     [KDE Users]     [Fedora Tools]     [Fedora Art]     [Fedora Docs]     [Asterisk PBX]

Powered by Linux