Re: PPPoE performance regression

[David Woodhouse <dwmw2@xxxxxxxxxxxxx>]

 On Sun, 2012-06-10 at 10:50 +1000, Nathan Williams wrote:
> > When using iperf with UDP, we can get 20Mbps downstream, but only about
> > 15Mbps throughput when using TCP on a short ADSL line (line sync at
> > 25Mbps).  Using iperf to send UDP traffic upstream at the same time
> > doesn't affect the downstream rate.
>
> ...
>
> I found the change responsible for the performance problem and rebuilt
> OpenWrt with the patch reversed on kernel 3.3.8 to confirm everything
> still works.  So the TX buffer is getting full, which causes the netif
> queue to be stopped and restarted after some skbs have been freed?

The *Ethernet* netif queue, yes. But not the PPP netif queue, I believe.
I think the PPP code keeps just blindly calling dev_queue_xmit() and
throwing away packets when they're not accepted.

> commit 137742cf9738f1b4784058ff79aec7ca85e769d4
> Author: Karl Hiramoto <karl@xxxxxxxxxxxx>
> Date:   Wed Sep 2 23:26:39 2009 -0700
> 
>     atm/br2684: netif_stop_queue() when atm device busy and
> netif_wake_queue() when we can send packets again.

Nice work; well done finding that. I've added Karl and DaveM, and the
netdev@ list to Cc.

(Btw, I assume the performance problem also goes away if you use PPPoA?
I've made changes in the PPPoA code recently to *eliminate* excessive
calls to netif_wake_queue(), and also to stop it from filling the ATM
device queue. That was commit 9d02daf7 in 3.5-rc1, which is already in
OpenWRT.)

I was already looking vaguely at how we could limit the PPP queue depth
for PPPoE and implement byte queue limits. Currently the PPP code just
throws the packets at the Ethernet device and considers them 'gone',
which is why it's hitting the ATM limits all the time. The patch you
highlight is changing the behaviour in a case that should never *happen*
with PPP. It's suffering massive queue bloat if it's filling the ATM
queue, and we should fix *that*.

I was looking to see if we could (ab)use the skb->destructor somehow so
that we get *notified* when the packet is actually sent (or dropped),
and then that would allow us to manage the queue 'downstream' of PPP
more sanely. But I haven't really got very far with that yet.

I was planning to find some time to look into it a bit better, and then
send mail to netdev@ asking for more clue. But since you're now falling
over it and it isn't just a theoretical problem, this mail will have to
suffice for now...

-- 
dwmw2

<<attachment: smime.p7s>>