Re: Difficulties to get 1Gbps on be2net ethernet card

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



2012/6/6 Jean-Michel Hautbois <jhautbois@xxxxxxxxx>:
> 2012/6/6 Jean-Michel Hautbois <jhautbois@xxxxxxxxx>:
>> 2012/6/6 Jean-Michel Hautbois <jhautbois@xxxxxxxxx>:
>>> 2012/6/6 Eric Dumazet <eric.dumazet@xxxxxxxxx>:
>>>> On Wed, 2012-06-06 at 12:04 +0200, Jean-Michel Hautbois wrote:
>>>>
>>>>> Well, well, well, after having tested several configurations, several
>>>>> drivers, I have a big difference between an old 2.6.26 kernel and a
>>>>> newer one (I tried 3.2 and 3.4).
>>>>>
>>>>> Here is my stream : UDP packets (multicast), 4000 bytes length, MTU
>>>>> set to 4096. I am sending packets only, nothing on RX.
>>>>> I send from 1Gbps upto 2.4Gbps and I see no drops in tc with 2.6.26
>>>>> kernel, but a lot of drops with a newer kernel.
>>>>> So, I don't know if I missed something in my kernel configuration, but
>>>>> I have used the 2.6.26 one as a reference, in order to set the same
>>>>> options (DMA related, etc).
>>>>>
>>>>> I easily reproduce this problem and setting a bigger txqueuelen solves
>>>>> it partially.
>>>>> 1Gbps requires a txqueulen of 9000, 2.4Gbps requires more than 20000 !
>>>>>
>>>>> If you have any idea, I am interested, as this is a big issue for my use case.
>>>>>
>>>>
>>>> Yep.
>>>>
>>>> This driver wants to limit number of tx completions, thats just wrong.
>>>>
>>>> Fix and dirty patch:
>>>>
>>>>
>>>> diff --git a/drivers/net/ethernet/emulex/benet/be.h b/drivers/net/ethernet/emulex/benet/be.h
>>>> index c5c4c0e..1e8f8a6 100644
>>>> --- a/drivers/net/ethernet/emulex/benet/be.h
>>>> +++ b/drivers/net/ethernet/emulex/benet/be.h
>>>> @@ -105,7 +105,7 @@ static inline char *nic_name(struct pci_dev *pdev)
>>>>  #define MAX_TX_QS              8
>>>>  #define MAX_ROCE_EQS           5
>>>>  #define MAX_MSIX_VECTORS       (MAX_RSS_QS + MAX_ROCE_EQS) /* RSS qs + RoCE */
>>>> -#define BE_TX_BUDGET           256
>>>> +#define BE_TX_BUDGET           65535
>>>>  #define BE_NAPI_WEIGHT         64
>>>>  #define MAX_RX_POST            BE_NAPI_WEIGHT /* Frags posted at a time */
>>>>  #define RX_FRAGS_REFILL_WM     (RX_Q_LEN - MAX_RX_POST)
>>>>
>>>
>>> I will try that in a few minutes.
>>> I also have a mlx4 driver (mlx4_en) which has a similar behaviour, and
>>> a broadcom (bnx2x).
>>>
>>
>> And it is not really better, still need about 18000 at 2.4Gbps in
>> order to avoid drops...
>> I really think there is something in the networking stack or in my
>> configuration (DMA ? Something else ?)...
>> As it doesn't seem to be driver related as I said...
>>
>
> If it can help, on a 3.0 kernel a txqueuelen of 9000 is sufficient in
> order to get this bandwith on TX.
>
> JM

All,

I made some tests, and I didn't mention it : I am using the bonding
driver over my ethernet drivers (be2net/mlx4 etc.).
When I am using bonding, I need a big txqeuelen in order to send 2.4Gbps.
When I disable bonding, and use directly the NIC then I don't see any
drops in qdisc and it works well.
So, I think there is something between 2.6.26 and 3.0 in the bonding
driver which causes this issue.

Any help would be appreciated :).
Regards,
JM
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html


[Index of Archives]     [Linux Kernel Discussion]     [TCP Instrumentation]     [Ethernet Bridging]     [Linux Wireless Networking]     [Linux WPAN Networking]     [Linux Host AP]     [Linux WPAN Networking]     [Linux Bluetooth Networking]     [Linux ATH6KL Networking]     [Linux Networking Users]     [Linux Coverity]     [VLAN]     [Git]     [IETF Annouce]     [Linux Assembly]     [Security]     [Bugtraq]     [Yosemite Information]     [MIPS Linux]     [ARM Linux Kernel]     [ARM Linux]     [Linux Virtualization]     [Linux IDE]     [Linux RAID]     [Linux SCSI]