SOLVED Low iperf and SMB speeds with 10Gbit link (Scale)

[Stoffern]

Hello,

It is as the title says. I'm experiencing way lower speeds than anticipated with my TrueNAS Scale server. Both my desktop and Scale server have X520-DA2 network cards. I have change the MTU to 9000 on both my server and desktop, with no noticeable change. Changing from PCI-E gen 4 to 3 on my server made no difference in speed either. I have no idea what might cause this, or why SMB speeds are faster than iperf?

Any help would be greatly appreciated!

iperf:

SMB transfer (NVME gen 4 SSD's on both ends, fully capable of far more than 10Gbits):

 

[sretalla]

Did you run iperf with the switches:
-P 8 -bidir

Basically you're seeing that you only get 3Gbits with a single thread (so probably a CPU limit somewhere).

why SMB speeds are faster than iperf?

SMB is (or can be) multi-threaded, so can get around that limit of a single thread.

 

[Stoffern]

Did you run iperf with the switches:
-P 8 -bidir

Basically you're seeing that you only get 3Gbits with a single thread (so probably a CPU limit somewhere).


SMB is (or can be) multi-threaded, so can get around that limit of a single thread.

I tried running it like this now (completely new to iPerf, never used it before this):

iperf3.exe -P 8 -bidir -c 192.168.1.147 on my desktop
and just iperf3 -s on my TrueNAS server, which yielded this result:

Show : iPerf3 results

Code:

[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   401 MBytes   336 Mbits/sec                  sender
[  4]   0.00-10.00  sec   401 MBytes   336 Mbits/sec                  receiver
[  6]   0.00-10.00  sec   349 MBytes   293 Mbits/sec                  sender
[  6]   0.00-10.00  sec   349 MBytes   293 Mbits/sec                  receiver
[  8]   0.00-10.00  sec   525 MBytes   441 Mbits/sec                  sender
[  8]   0.00-10.00  sec   525 MBytes   441 Mbits/sec                  receiver
[ 10]   0.00-10.00  sec   445 MBytes   373 Mbits/sec                  sender
[ 10]   0.00-10.00  sec   445 MBytes   373 Mbits/sec                  receiver
[ 12]   0.00-10.00  sec   412 MBytes   346 Mbits/sec                  sender
[ 12]   0.00-10.00  sec   412 MBytes   345 Mbits/sec                  receiver
[ 14]   0.00-10.00  sec   404 MBytes   339 Mbits/sec                  sender
[ 14]   0.00-10.00  sec   404 MBytes   339 Mbits/sec                  receiver
[ 16]   0.00-10.00  sec   363 MBytes   305 Mbits/sec                  sender
[ 16]   0.00-10.00  sec   363 MBytes   305 Mbits/sec                  receiver
[ 18]   0.00-10.00  sec   523 MBytes   439 Mbits/sec                  sender
[ 18]   0.00-10.00  sec   523 MBytes   439 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  3.34 GBytes  2.87 Gbits/sec                  sender
[SUM]   0.00-10.00  sec  3.34 GBytes  2.87 Gbits/sec                  receiver

iperf Done.

Running it with iperf3.exe -P 24 -bidir -c 192.168.1.147 just to try, yielded these results:

Show : iPerf3 results #2

Code:

[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   243 MBytes   204 Mbits/sec                  sender
[  4]   0.00-10.00  sec   243 MBytes   204 Mbits/sec                  receiver
[  6]   0.00-10.00  sec   307 MBytes   258 Mbits/sec                  sender
[  6]   0.00-10.00  sec   307 MBytes   258 Mbits/sec                  receiver
[  8]   0.00-10.00  sec   222 MBytes   186 Mbits/sec                  sender
[  8]   0.00-10.00  sec   222 MBytes   186 Mbits/sec                  receiver
[ 10]   0.00-10.00  sec   336 MBytes   282 Mbits/sec                  sender
[ 10]   0.00-10.00  sec   336 MBytes   282 Mbits/sec                  receiver
[ 12]   0.00-10.00  sec   268 MBytes   225 Mbits/sec                  sender
[ 12]   0.00-10.00  sec   268 MBytes   225 Mbits/sec                  receiver
[ 14]   0.00-10.00  sec   184 MBytes   154 Mbits/sec                  sender
[ 14]   0.00-10.00  sec   183 MBytes   154 Mbits/sec                  receiver
[ 16]   0.00-10.00  sec   285 MBytes   239 Mbits/sec                  sender
[ 16]   0.00-10.00  sec   284 MBytes   239 Mbits/sec                  receiver
[ 18]   0.00-10.00  sec   191 MBytes   160 Mbits/sec                  sender
[ 18]   0.00-10.00  sec   191 MBytes   160 Mbits/sec                  receiver
[ 20]   0.00-10.00  sec   227 MBytes   190 Mbits/sec                  sender
[ 20]   0.00-10.00  sec   226 MBytes   190 Mbits/sec                  receiver
[ 22]   0.00-10.00  sec   265 MBytes   222 Mbits/sec                  sender
[ 22]   0.00-10.00  sec   265 MBytes   222 Mbits/sec                  receiver
[ 24]   0.00-10.00  sec   210 MBytes   176 Mbits/sec                  sender
[ 24]   0.00-10.00  sec   210 MBytes   176 Mbits/sec                  receiver
[ 26]   0.00-10.00  sec   142 MBytes   119 Mbits/sec                  sender
[ 26]   0.00-10.00  sec   142 MBytes   119 Mbits/sec                  receiver
[ 28]   0.00-10.00  sec   187 MBytes   157 Mbits/sec                  sender
[ 28]   0.00-10.00  sec   187 MBytes   157 Mbits/sec                  receiver
[ 30]   0.00-10.00  sec   206 MBytes   173 Mbits/sec                  sender
[ 30]   0.00-10.00  sec   206 MBytes   173 Mbits/sec                  receiver
[ 32]   0.00-10.00  sec   201 MBytes   169 Mbits/sec                  sender
[ 32]   0.00-10.00  sec   201 MBytes   168 Mbits/sec                  receiver
[ 34]   0.00-10.00  sec   132 MBytes   111 Mbits/sec                  sender
[ 34]   0.00-10.00  sec   132 MBytes   110 Mbits/sec                  receiver
[ 36]   0.00-10.00  sec   375 MBytes   315 Mbits/sec                  sender
[ 36]   0.00-10.00  sec   375 MBytes   315 Mbits/sec                  receiver
[ 38]   0.00-10.00  sec   231 MBytes   194 Mbits/sec                  sender
[ 38]   0.00-10.00  sec   231 MBytes   194 Mbits/sec                  receiver
[ 40]   0.00-10.00  sec   223 MBytes   187 Mbits/sec                  sender
[ 40]   0.00-10.00  sec   223 MBytes   187 Mbits/sec                  receiver
[ 42]   0.00-10.00  sec   228 MBytes   191 Mbits/sec                  sender
[ 42]   0.00-10.00  sec   227 MBytes   191 Mbits/sec                  receiver
[ 44]   0.00-10.00  sec   140 MBytes   118 Mbits/sec                  sender
[ 44]   0.00-10.00  sec   140 MBytes   118 Mbits/sec                  receiver
[ 46]   0.00-10.00  sec   231 MBytes   194 Mbits/sec                  sender
[ 46]   0.00-10.00  sec   231 MBytes   194 Mbits/sec                  receiver
[ 48]   0.00-10.00  sec   210 MBytes   176 Mbits/sec                  sender
[ 48]   0.00-10.00  sec   210 MBytes   176 Mbits/sec                  receiver
[ 50]   0.00-10.00  sec   140 MBytes   118 Mbits/sec                  sender
[ 50]   0.00-10.00  sec   140 MBytes   118 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  5.26 GBytes  4.52 Gbits/sec                  sender
[SUM]   0.00-10.00  sec  5.25 GBytes  4.51 Gbits/sec                  receiver

iperf Done.

Speeds seem to vary a lot as well. After going back to the previous settings I'd ran it with, which got me the results in my original post, simply iperf3.exe -c 192.168.1.147, I only now got ~700Mbits instead of the ~3Gbits I'd gotten previously. Not sure what to make of it all to be honest.

CPU on server side is a Ryzen 9 5950X, and 5900X in my desktop.

 

[mat4226]

This sounds very similar to something I was dealing with a few months ago in getting better throughput with higher speed NICs. Here's a link to the discussion, but the TLDR is the NIC placement was even more important than anticipated. Once moving the NIC to a PCIe lane that went through the CPU instead of the chipset, speeds tripled. Also, my NAS servers aren't using nearly as powerful CPU's as yours (Ryzen 3200G and 4700G), so there's got to be something else as the bottleneck.

 

[Stoffern]

This sounds very similar to something I was dealing with a few months ago in getting better throughput with higher speed NICs. Here's a link to the discussion, but the TLDR is the NIC placement was even more important than anticipated. Once moving the NIC to a PCIe lane that went through the CPU instead of the chipset, speeds tripled. Also, my NAS servers aren't using nearly as powerful CPU's as yours (Ryzen 3200G and 4700G), so there's got to be something else as the bottleneck.

You were spot on.

I didn't get around to changing the configuration of my server before yesterday, but changing the X520-DA2 to the PCI-E slot above, presumably connected to the CPU rather than the chipset, solved the issue.

Running: iperf3.exe -P 8 -bidir -c 192.168.1.147

I got:

9.47 Gbits/sec is about what I expected to begin with.

Without the parameters @sretalla provided, it was stuck at ~5 Gbits even after the PCI-E slot change.

It seems you both solved my issue. Thanks a ton!