Re: root_lock vs. device's TX lock

[Eric Dumazet <eric.dumazet@xxxxxxxxx>]

Le jeudi 17 novembre 2011 à 17:34 +0100, Eric Dumazet a écrit :
> Le jeudi 17 novembre 2011 à 08:10 -0800, Tom Herbert a écrit :
> > From sch_direct_xmit:
> > 
> >         /* And release qdisc */
> >         spin_unlock(root_lock);
> > 
> >         HARD_TX_LOCK(dev, txq, smp_processor_id());
> >         if (!netif_tx_queue_frozen_or_stopped(txq))
> >                 ret = dev_hard_start_xmit(skb, dev, txq);
> > 
> >         HARD_TX_UNLOCK(dev, txq);
> > 
> >         spin_lock(root_lock);
> > 
> > This is a lot of lock manipulation to basically switch from one lock
> > to another and possibly thrashing just to send a packet.  I am
> > thinking that if the there is a 1-1 correspondence between qdisc and
> > device queue then we could actually use the device's lock as the root
> > lock for the qdisc.  So in that case, we would need to touch any locks
> > from sch_direct_xmit (just hold root lock which is already device lock
> > for the duration).
> > 
> > Is there any reason why this couldn't work?
> 
> But we have to dirty part of Qdisc anyway ?
> (state, bstats, q, ...)
> 

Also we want to permit other cpus to enqueue packets to Qdisc while our
cpu is busy in network driver ndo_start_xmit()

For complex Qdisc / tc setups (potentially touching a lot of cache
lines), we could eventually add a small ring buffer so that the cpu
doing the ndo_start_xmit() also queues the packets into Qdisc.

This ringbuffer could use a lockless algo. (we currently use the
secondary 'busylock' to serialize other cpus, but each cpu calls qdisc
enqueue itself.)



--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html