100Mbps on a 10Gbps connection, bottleneck?

[bradburns]

(admin, this was moved from the wrong location. please del: https://www.truenas.com/community/threads/100mbps-on-a-10gbps-connection-bottleneck.110255/ )

Thanks in advance for responding. I am building a TrueNAS system for the first time, and would appreciate any recommendations you have. My issues is that I am only able to transfer at 100mbps (12.5MBps) using SMB to Windows on a 10Gbps connection. I have not tried any other method. Below is my configuration, please note I wont be able to run any tests until Tuesday, so any thorough suggestions you have are greatly appreciated.

NETWORK: Cat6a cables with Ubiquiti 10Gbps switch
Ethernet Status from ifconfig: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)

HARDWARE:
NIC: 10Gbps https://www.amazon.com/gp/product/B008F6ID0U/
OS Drive: SSD https://www.amazon.com/gp/product/B07DNLY1R2/
PSU: Corsair HX1200
Mainboard: ASUS ROG Strix B450-F II https://www.amazon.com/gp/product/B08KH1M1H4/
RAM: 64GB DDR4
CPU: AMD RYZEN 7 5700G

HBA: LSI 9207-8e (LSI SAS2308)
https://www.serversupply.com/products/part_search/query.asp?q=9207-8E

Expander: HPE 3Gb 24-Port SAS
https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c01733557

*Expander upgrade soon: HP 12Gb 36-Port SAS-3*
https://www.ebay.com/itm/165984546437

Drives (18 of them): Seagate Exos 20TB Sata
https://www.amazon.com/Seagate-Exos-20TB-SATA-ST20000NM007D/dp/B09MWKXR2T

OS: TrueNAS
Config: 3 vDevs, 6 drives in each vDev, each vDev on RAIDZ1
Everything else is default

 

[bradburns]

bump

 

[joeschmuck]

I would recommend that you clearly define how you tested and what files/size you tried to transfer, the direction of transfer. Also the configuration of your shares. Have you searched the forums for this kind of problem? We ask for all this kind of basic information so we can provide you a proper response. Additionally you may find your answer in the forums as well.

Have you read this?

10GBase-T: Best to avoid it if you can

This resource was originally created by user: jgreco on the TrueNAS Community Forums Archive. Please DM this account or comment in this thread to claim it. I recently wrote a resource on high speed network performance tuning, and in it, I commented “Do not try to use copper 10GBase-T”. @Elliot...

www.truenas.com

 

[bradburns]

Thanks for your response. Can you explain what you meant by configurations of my shares? I use SMB and I have tried transferring everything from multiple 1kb files to 10gb files.

I have searched the forum, or else I would not have posted here. I did read that post, but I didn't find it helpful as the networking we have is professionally installed and works correctly.

Thanks again for your continued help

 

[MrGuvernment]

Can you test to any other device from the TrueNAS with the same files to see how it performs?

Have you tested internally with in truenas the performance of your pools to see how they are?

 

[somethingweird]

Could the network card be fake?

 

[joeschmuck]

Can you explain what you meant by configurations of my shares?

Do you have encryption, do you have compression? These things will effect the transfer speed.

I have tried transferring everything from multiple 1kb files to 10gb files.

In which direction? To the NAS from the PC? To the PC form the NAS? What are the speeds for each?

Maybe you should take a step back and remove the 10Gbps adapter and just use the built in Intel LAN connection. Test out it's speed and ensure it works correctly. Once this is all working properly and you can verify you have good throughput, then move forward with the 10Gbps connection.

Why do this? To ensure something else is not the cause of the slowdown. If you can get ~100MB/sec transfer speeds, then you have proven the NAS is not the issue, the 10Ggps NIC or your network is the issue.

A good tool to check throughput is 'iperf'. 'iperf' is also built into TrueNAS so you can use it and report the results it provides.

Other suggestions...

the networking we have is professionally installed and works correctly.

Are you sure? How do you know it works correctly? We have no idea on how your network is actually configured so we are still guessing, this is why my next suggestion is so important.
One test is to directly connect the TrueNAS machine to the PC, one ethernet cable, nothing more. Test the transfer speeds.

Could the network card be fake?

It could be.

 

[danb35]

A good tool to check throughput is 'iperf'. 'iperf' is also built into TrueNAS so you can use it and report the results it provides.

...and it's available for Windows as well:

iPerf - Download iPerf3 and original iPerf pre-compiled binaries

iPerf3 binaries for Windows, Linux, MacOS X

iperf.fr

@bradburns, the first thing to do would be to check that you actually can transfer data over the network at something close to 10 Gbit/sec, and iperf3 is a good way to do that. Run that test in both directions--with the NAS as both the server and the client--using the docs at the link above. If you're getting somewhere close to 10 Gbit/sec, you've ruled out the network as the problem.

If not, time to start narrowing things down. This won't be an exhaustive list, but some things to check include:

• What link speed is reported? On the NAS, on the Windows machine, and in the management interface for the switch?
• Do you have another 10G-capable client machine you can test with?
• As @joeschmuck suggests above, can you try over a direct cable connection from one machine to the other?
• Try different ports on the switch
• Try different cables for each system

If you can get ~10Gbit/sec using iperf, then your network is fine--suboptimal in that you're using 10G over copper, but it isn't the immediate problem. So then it's time to look elsewhere. My suspicion would be your pool layout and the SAS1 expander.

 

[bradburns]

Very very good info. thank you all for your help. I'll get back to you tomorrow with the results.