[updated / mostly solved] 100% cpu load with network traffic

[Steve Mickeler <steve@neptune.ca>]

First, thanks to everyone who replied with suggestions.

After doing what felt like a thousand netpipe tcp tests across every
possible link and finding everything to be good I headed back to the linux
box as the culprit.

The difference between the netpipe test traffic and the real traffic
coming from the 7507 was that the 7507 is sending a lot of traffic with
tonnes of different source and destination IP's because we're using this
linux box as a router in an asyncronous route setup.

The CPU hog culprit turns out to be the ip_conntrack table.

I had /proc/sys/net/ipv4/ip_conntrack_max set to 32768 and the CPU was
spending all its time dealing with the tens of thousands of UNREPLIED
entries due to the async routing.

I changed /proc/sys/net/ipv4/ip_conntrack_max to 3072 and now the box is
currently pushing about 25Mbps @ 21:05 EST. It should get up over 70Mbps
tomorrow.

Now the question I have is concerning the ip_conntrack table.

# wc -l /proc/net/ip_conntrack
   2919 /proc/net/ip_conntrack

# grep -v UNREPLIED /proc/net/ip_conntrack | wc -l
    318

Since the ip_conntrack will simply drop any UNREPLIED entries it needs to
to make room for more entries, and I really only have just over 300 real
connections that need tracking, am I hurting myself in any way by keeping
the ip_conntrack_max setting at 3072 ?

I'm seeing tonnes of messages like this:

NET: 752 messages suppressed.
ip_conntrack: table full, dropping packet.

But I am not actually seeing any dropped packets or data loss.



On Wed, 4 Sep 2002, Steve Mickeler wrote:

>
>
> I'm trying to implement a linux based router which will take the traffic
> from one of the interfaces on our cisco 7507 and then pass it off to
> another network.
>
> The problem is that when we start sending data from the cisco 7507 to the
> linux box, the linux box starts to go up to 100% CPU Utilization at around
> 12Mbps.
>
> Running mpstat I only see around 4000 Intr/s which is nothing for this
> box.
>
> If I do a simple NetPipe test between another linux box and the linux
> router, I am able to get over 450Mbps across the link which is why I am
> puzzled as to why the box is choking at only 12Mbps when the data is being
> sent from the cisco 7507.
>
> I also noticed that ksoftirqd_eth0 is taking up loads of CPU when the
> traffic is coming from the cisco 7507 but not when the traffic is coming
> from the other linux box.
>
> # cat /proc/interrupts | grep eth
>   5:     339565          XT-PIC  eth2
>  11:   31063475          XT-PIC  eth0
>  15:   39496363          XT-PIC  eth1, eth3
>
> Traffic is coming from the cisco 7507 in on eth1 and being sent out on
> eth2
>
>
> Any ideas / advice ?
>
>
> Hardware is as follows:
>
> Make & Model: IBM x342
> CPU: 1 x 1.2Ghz PIII
> Mem: 1GB Ram
> NICs: 3 x 3Com Corporation 3c985 1000BaseSX (acenic driver)
>
> Software:
>
> Distro: Debian Woody (3.0)
> Kernel: 2.4.19 ( no patches applied )
>
>
>
>
> [-] Steve Mickeler [ steve@neptune.ca ]
>
> [|] Todays root password is brought to you by /dev/random
>
> [+] 1024D/9AA80CDF = 4103 9E35 2713 D432 924F  3C2E A7B9 A0FE 9AA8 0CDF
>
>
>
>
>
> ____________  The ISP-ROUTING Discussion List  ____________
> To Join: mailto:join-isp-routing@isp-routing.com
> To Remove: mailto:remove-isp-routing@isp-routing.com
> Archives: http://isp-lists.isp-planet.com/isp-routing/archives/
>



[-] Steve Mickeler [ steve@neptune.ca ]

[|] Todays root password is brought to you by /dev/random

[+] 1024D/9AA80CDF = 4103 9E35 2713 D432 924F  3C2E A7B9 A0FE 9AA8 0CDF
_______________________________________________
LinuxManagers mailing list - http://www.linuxmanagers.org
submissions: LinuxManagers@linuxmanagers.org
subscribe/unsubscribe: http://www.linuxmanagers.org/mailman/listinfo/linuxmanagers