RE: Is SCTP throughput really this low compared to TCP?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



I started the bisection process - again this will take some time and I (unfortunately) can't dedicate all my time to this at work (other fires for me to put out here as well).  However on a hunch, based on TIPC work I had previously done that seemed to suggest that the 3.10.x kernel stream was the last stream before significant changes were made to the underlying net API, I tested the latest kernel in this stream, namely 3.10.36.

As I suspected, this kernel is still 'good' - that is, there is no erratic behaviour with SCTP/IPv4 and the throughput.  However the throughput is still better with the 3.4 2 kernel (2.1 Gbps as opposed to 1.75 Gbps).

SUMMARY so far for SCTP+IPv4:

3.4.2 kernel:  smooth throughput 2.1 Gbps
3.10.36 kernel: smooth throughput 1.75 Gbps
3.14 kernel: highly erratic, terrible throughput




-----Original Message-----
From: Daniel Borkmann [mailto:dborkman@xxxxxxxxxx] 
Sent: April-11-14 2:40 PM
To: Butler, Peter
Cc: Vlad Yasevich; linux-sctp@xxxxxxxxxxxxxxx
Subject: Re: Is SCTP throughput really this low compared to TCP?

On 04/11/2014 08:22 PM, Butler, Peter wrote:
> The difference between 3.14 and 3.4.2 is staggering.  An order of 
> magnitude or so.  For example,
 > using the precisely same setup as before, whereas I get about 2.1 Gbps throughput with 3.4 2, I  > can only manage between 70-150 Mbps with 3.14 - a staggering difference.
>
> Moreover, the SCTP throughput seems to 'choke' itself with 3.14, such 
> that it is always trying to
 > recover.  For example, with 3.4.2 the 2.1 Gbps throughput is quite consistent from one second to  > the next (as you would expect):
>
> [root@Lab200slot2 ~]# iperf3 --sctp -4 -c 192.168.241.3 -V -l 1452 -t 
> 60 iperf version 3.0.1 (10 January 2014) Linux Lab200slot2 
> 3.4.2-1.fc16.x86_64 #1 SMP Thu Jun 14 20:17:26 UTC 2012 x86_64
> Time: Fri, 11 Apr 2014 18:19:15 GMT
> Connecting to host 192.168.241.3, port 5201
>        Cookie: Lab200slot2.1397240355.069035.0d5b0f
> [  4] local 192.168.241.2 port 56030 connected to 192.168.241.3 port 
> 5201 Starting Test: protocol: SCTP, 1 streams, 1452 byte blocks, omitting 0 seconds, 60 second test
> [ ID] Interval           Transfer     Bandwidth
> [  4]   0.00-1.00   sec   255 MBytes  2.14 Gbits/sec
> [  4]   1.00-2.00   sec   253 MBytes  2.12 Gbits/sec
> [  4]   2.00-3.00   sec   255 MBytes  2.14 Gbits/sec
> [  4]   3.00-4.00   sec   255 MBytes  2.14 Gbits/sec
> [  4]   4.00-5.00   sec   255 MBytes  2.14 Gbits/sec
> [  4]   5.00-6.00   sec   257 MBytes  2.15 Gbits/sec
> [  4]   6.00-7.00   sec   253 MBytes  2.13 Gbits/sec
> [  4]   7.00-8.00   sec   254 MBytes  2.13 Gbits/sec
> [  4]   8.00-9.00   sec   255 MBytes  2.14 Gbits/sec
> [  4]   9.00-10.00  sec   252 MBytes  2.12 Gbits/sec
> (etc)
>
> but with 3.14 the numbers as all over the place:
>
> [root@Lab200slot2 ~]# iperf3 --sctp -4 -c 192.168.241.3 -V -l 1452 -t 
> 60 iperf version 3.0.1 (10 January 2014) Linux Lab200slot2 3.14.0 #1 
> SMP Thu Apr 3 23:18:29 EDT 2014 x86_64
> Time: Fri, 11 Apr 2014 17:56:21 GMT
> Connecting to host 192.168.241.3, port 5201
>        Cookie: Lab200slot2.1397238981.812898.548918
> [  4] local 192.168.241.2 port 38616 connected to 192.168.241.3 port 
> 5201 Starting Test: protocol: SCTP, 1 streams, 1452 byte blocks, omitting 0 seconds, 60 second test
> [ ID] Interval           Transfer     Bandwidth
> [  4]   0.00-1.09   sec  20.8 MBytes   161 Mbits/sec
> [  4]   1.09-2.13   sec  10.8 MBytes  86.8 Mbits/sec
> [  4]   2.13-3.15   sec  3.57 MBytes  29.5 Mbits/sec
> [  4]   3.15-4.16   sec  4.33 MBytes  35.7 Mbits/sec
> [  4]   4.16-6.21   sec  10.4 MBytes  42.7 Mbits/sec
> [  4]   6.21-6.21   sec  0.00 Bytes  0.00 bits/sec
> [  4]   6.21-7.35   sec  34.6 MBytes   253 Mbits/sec
> [  4]   7.35-11.45  sec  22.0 MBytes  45.0 Mbits/sec
> [  4]  11.45-11.45  sec  0.00 Bytes  0.00 bits/sec [  4]  11.45-11.45  
> sec  0.00 Bytes  0.00 bits/sec [  4]  11.45-11.45  sec  0.00 Bytes  
> 0.00 bits/sec
> [  4]  11.45-12.51  sec  16.0 MBytes   126 Mbits/sec
> [  4]  12.51-13.59  sec  20.3 MBytes   158 Mbits/sec
> [  4]  13.59-14.65  sec  13.4 MBytes   107 Mbits/sec
> [  4]  14.65-16.79  sec  33.3 MBytes   130 Mbits/sec
> [  4]  16.79-16.79  sec  0.00 Bytes  0.00 bits/sec [  4]  16.79-17.82  
> sec  5.94 MBytes  48.7 Mbits/sec
> (etc)
>
> Note: the difference appears to be SCTP-specific, as I get exactly the 
> same TCP
 > throughput in both kernels.

Hmm, okay. :/ Could you further bisect on your side to narrow down from which kernel onwards this behaviour can be seen?

Thanks,

Daniel

> -----Original Message-----
> From: Daniel Borkmann [mailto:dborkman@xxxxxxxxxx]
> Sent: April-11-14 11:22 AM
> To: Butler, Peter
> Cc: Vlad Yasevich; linux-sctp@xxxxxxxxxxxxxxx
> Subject: Re: Is SCTP throughput really this low compared to TCP?
>
> On 04/11/2014 05:07 PM, Butler, Peter wrote:
>> Yes indeed this is ixgbe and I do have SCTP checksum offloading 
>> enabled.  (For what
>   > it's worth, the checksum offload gives about a 20% throughput gain 
> - but this is,  > of course, already included in the numbers I posted 
> to this thread as I've been using  > the CRC offload all along.)
>
> Ok, understood.
>
>> I re-did all the tests with TSO/GSO/LRO/GRO disabled (on both sides 
>> of the association -
>   > i.e. on both endpoint nodes), and using 1452-byte messages instead of 1000-byte  > messages.  With this new setup, the TCP performance drops significantly, as expected,  > while the SCTP performance is boosted, and the playing field is somewhat more 'level'.
>   > (Note that I could not use 1464-byte messages as suggested by 
> Vlad, as anything above  > 1452 cut the SCTP performance in half - 
> must have hit the segmentation limit at this  > slightly lower message 
> size.  MTU is 1500.)
>>
>> So comparing "apples to apples" now, TCP only out-performs SCTP by 
>> approximately 40-70%
>   > over the various range of network latencies I tested with (RTTs of 0.2 ms, 10 ms, 20 ms,  > and 50 ms).  40-70% is still significant, but nowhere near the 200% better (i.e. 3  > times the throughput) I was getting before.
>>
>> Does this value (i.e. 40-70%) sound reasonable?  Is this the 
>> more-or-less accepted
>   > performance difference with the current LKSCTP implementation?
>
> Yes, that sounds reasonable to me. There are still a lot of open todos in terms of performance that we need to tackle over time, e.g. the way chunks are handled, imho, and copies involved in fast path, also that we're heavily using atomic reference counting, and other open issues.
>
>> Also, for what it's worth, I get better SCTP throughput numbers with 
>> the older kernel
>   > (3.4.2) than with the newer kernel (3.14)...
>
> Interesting, a lot of things happened in between; were you able to bisect/identify a possible commit that causes this? How big is the difference?
>
--
To unsubscribe from this list: send the line "unsubscribe linux-sctp" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html




[Index of Archives]     [Linux Networking Development]     [Linux OMAP]     [Linux USB Devel]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]

  Powered by Linux