TrueNAS Scale and 25/40GbE (Mellanox ConnectX-3/4) Setup, Benchmark and Tuning ...

[BloodyIron]

Oh also I just noticed this is in the SCALE section... not CORE... I'm aiming to do mine with CORE...

 

[pixelwave]

@Glitch01 are your ConnectX-4 cards in IB or ETH mode? I for one would like to know a lot more about your configuration and set-up specific to the IB cards please! Also, why does your iperf only go up to 24gpbs? That seems low... (from my armchair, since I'm starting the IB adventure myself hehe).

@Glitch01 is using ConnectX-3 cards (QSFP) in ETH Mode. I am using ConnectX-4 cards (SFP28/SFP+) ... also in ETH Mode.

 

[BloodyIron]

Thanks! :) I keep getting ConnectX-3 and -4 muddled haha.

 

[Glitch01]

I am wondering how you achieve 4000 MB/s read and write to the TrueNAS system? Especially since they are mostly HDDs?

I currently have issues with my ConnectX-4 cards since I have a weird behaviour not achieving full 10GbE down/up speed:

Upgrade to 25GbE (Mellanox ConnectX-4)

I am in the market to upgrade my network infrastructure to 25GbE. Therefore I would like to purchase three PCIe cards (Workstation, Server, Backup Node) and after reading the STH article (https://www.servethehome.com/mellanox-connectx-4-lx-mini-review-ubiquitous-25gbe/) I guess price-wise the...

www.truenas.com

Have TrueNAS in ZFS which buffers directly to RAM before the actual read/writes to the HDD's. Since I have 32 GB on the NAS it just dumps it as fast as possible to RAM as completed before the actual writes to HDD. Similarly, I have an app for RAM caching on Windows (4 GB allocated) for reads/writes to the local drives that function similarly to ZFS. MTU for the testing was also set to 9000 over a direct connection from the client to TrueNAS, but I'm finding that Windows does not like the value of 9000 as it frequently resets the connection on this card when running through a switch. I have MTU set for 9k on a 10 Gbe RJ45 connection that works on Windows, but it just seems to hate it on the Mellanox. 9k seems to be fine on TrueNAS to the switch which is also set to 9k.

 

[Ericloewe]

The Mellanox drivers have historically been a bit dodgy on FreeBSD. It's why they're not super popular around these parts.

 

[Glitch01]

@Glitch01 are your ConnectX-4 cards in IB or ETH mode? I for one would like to know a lot more about your configuration and set-up specific to the IB cards please! Also, why does your iperf only go up to 24gpbs? That seems low... (from my armchair, since I'm starting the IB adventure myself hehe).

Ran some testing on 10 Gbe from Windows box to router and NAS to router. I'm thinking the throughput limitation is related to Windows. Windows to router is almost half the bitrate of TrueNAS to the router.

From TrueNAS to router iperf3

From Windows to router iperf3

 

[pixelwave]

Ran some testing on 10 Gbe from Windows box to router and NAS to router. I'm thinking the throughput limitation is related to Windows. Windows to router is almost half the bitrate of TrueNAS to the router.

From TrueNAS to router iperf3
View attachment 68230

From Windows to router iperf3
View attachment 68231

I just ran into a super weird behaviour ... testing win11 and mac machines connected to my 10GbE switch accessing the TrueNAS machine:

In Windows the Read from TrueNAS is limited and for Mac it is the Write to TrueNAS Server. Same SMB target folder on NAS (NVMe drive), connected to same router?!?

Windows to Proxmox

Mac to Proxmox

 

[Ericloewe]

Mac it is the Write to TrueNAS Server

Mac OS issuing sync writes? It's known to do that over SMB, unlike Windows which does basically none.

 

[Glitch01]

I just ran into a super weird behaviour ... testing win11 and mac machines connected to my 10GbE switch accessing the TrueNAS machine:

In Windows the Read from TrueNAS is limited and for Mac it is the Write to TrueNAS Server. Same SMB target folder on NAS (NVMe drive), connected to same router?!?

Windows to Proxmox

Mac to Proxmox

Not sure if this applies to your situation. Depending on how your proxmox is configured and how many devices are sharing the connection may also have an impact.

SMB with Mac​

Between OS X 10.11.5 and macOS 10.13.3, Apple enforced packet signing when using SMB. This security feature was designed to prevent "man in the middle" attacks, but it turned out to have major performance implications for 10Gb SMB connections.

The policy was lifted in 10.13.4, and should not apply to subsequent releases.

More information about the packet signing policy can be found in Apple's knowledge base: https://support.apple.com/en-us/HT205926

 

[pixelwave]

Not sure if this applies to your situation. Depending on how your proxmox is configured and how many devices are sharing the connection may also have an impact.

SMB with Mac​

Between OS X 10.11.5 and macOS 10.13.3, Apple enforced packet signing when using SMB. This security feature was designed to prevent "man in the middle" attacks, but it turned out to have major performance implications for 10Gb SMB connections.

The policy was lifted in 10.13.4, and should not apply to subsequent releases.

More information about the packet signing policy can be found in Apple's knowledge base: https://support.apple.com/en-us/HT205926

I am on 13.4 newest update.

 

[Glitch01]

Additional testing from Linux on my main pc. Mellanox adapter reached 36 Gbps in Linux while 10 Gbe reached 5.7 Gbps. The 10 Gbe nic was originally on a pcie 4.0 x4 bus, but I moved it to a pcie 3.0 x8 bus with no noticeable difference. Speeds performed better under Linux OS for both 40 Gbe and 10 Gbe though I'm still scratching my head about the bottleneck for 10 Gbe. I guess there's something in the mb bios that I'm missing that is preventing the nic from reaching max performance.

From TrueNAS to Linux iperf3 over Mellanox to Mellanox

From TrueNAS to Linux iperf3 over 10 Gbe to 10 Gbe

 

[pixelwave]

Additional testing from Linux on my main pc. Mellanox adapter reached 36 Gbps in Linux while 10 Gbe reached 5.7 Gbps. The 10 Gbe nic was originally on a pcie 4.0 x4 bus, but I moved it to a pcie 3.0 x8 bus with no noticeable difference. Speeds performed better under Linux OS for both 40 Gbe and 10 Gbe though I'm still scratching my head about the bottleneck for 10 Gbe. I guess there's something in the mb bios that I'm missing that is preventing the nic from reaching max performance.

From TrueNAS to Linux iperf3 over Mellanox to Mellanox
View attachment 68266

From TrueNAS to Linux iperf3 over 10 Gbe to 10 Gbe
View attachment 68267

TrueNAS and Linux are directly connected or via a Switch? If so what Switch...?

When I speed test both my proxmox systems to each other (via switch) directly from CLI shows full speed, I tested both devices in server/client mode:

 

[Glitch01]

TrueNAS and Linux are directly connected or via a Switch? If so what Switch...?

When I speed test both my proxmox systems to each other (via switch) directly from CLI shows full speed, I tested both devices in server/client mode:

The 10 Gbe connection is through a Zyxel semi-managed switch. Both the TrueNAS and the client machine are connected through a 10GBase-T SFP+ to RJ45 connection. A direct connection from the client machine to TrueNAS faired a bit better for 10 Gbe. Windows gets about 6 Gbps through the direct connection in iperf, so I'm guessing Linux will get about 7-8 Gbps respectively. But I digress, the Mellanox connection is working well over a direct connection right now and well exceeds the NVMe cache write speeds to the pool (~2.8 GB/s). I'm currently looking into a custom-built switch solution that will host the 40 Gbe connection from the NAS to the rest of the 10 Gbe clients. This is more for future proofing as the current NAS hardware can host up to 26 sata connections. I'll probably cap the system out at 14x 14 tb hdd's to a single pool as end-of-life for the hardware.

More testing from a 30 GB RAM disk to/from TrueNAS 6x 14 tb hdd's in RaidZ2 array for more realistic expectations of direct-to-disk read/writes over 40 Gbe Mellanox direct connection.

Direct connection 10 Gbe from TrueNAS to Windows client

30 GB RAM disk used for testing benchmarks

26.8 GB file transfer from TrueNAS to RAM disk

26.8 GB file transfer from RAM disk to TrueNAS

 

[Glitch01]

Just to confuse you more Guess the changes in speeds from the previous copy was a single 26.8 GB file whereas the 60 GB vm folder is multiple files written in parallel.

4 tb NVMe writes to TrueNAS 6x disk hdd raidz2 60 GB folder, Mellanox direct connection

TrueNAS to 4tb NVMe, same 60 GB folder, Mellanox direct connection

 

[Glitch01]

One more mention I wanted to share. I decided to bridge 1 gbe, 2x 10 gbe, & the two Mellanox 40 gbe ports for testing. 1 gbe performed at 1 gbe speeds, 10 gbe ports performed at 10 gbe speeds, and the 40 gbe only performed at 10 gbe speeds. Not sure if there is a software cap in TrueNAS or something, but when the 40 gbe are bridged with the rest of the ports, they will not operate at 40 gbe speds.

 

[simonj]

Hi. Did you make any progress on this? We have two servers (1 core, 1 scale) with 10G and 40G nics and gave up on the 40G connections as the transfer speed from our clients (mostly Mac) to the TrueNas servers is horrible (less than 200MB/s). Read speeds are fine. Same time the 10G connections work as expected with almost full bandwidth.
We were recently in contact with a server vendor about purchasing a prebuilt and configured TrueNas system and they were also telling us that 40G is broken in general on TrueNAS.

 

[Glitch01]

Hi. Did you make any progress on this? We have two servers (1 core, 1 scale) with 10G and 40G nics and gave up on the 40G connections as the transfer speed from our clients (mostly Mac) to the TrueNas servers is horrible (less than 200MB/s). Read speeds are fine. Same time the 10G connections work as expected with almost full bandwidth.
We were recently in contact with a server vendor about purchasing a prebuilt and configured TrueNas system and they were also telling us that 40G is broken in general on TrueNAS.

The direct 40G connection works fine and the 10G connection works fine on their own on the same box. I'm only having an issue when bridging the two connections, but bridging requires a lot of cpu power to deal with the overhead. My plan is to just have the 40G on TrueNAS and have a dedicated switch to deal with the bridging issue for the rest of the network.

 

[mat4226]

I've been following this and several other threads over the last year, and after picking up a handful of Mellanox ConnectX-3 cards earlier this Fall I wanted to share my findings. Here are the specific cards used:

3x MCX314A-BCCT - all came pre-flashed to IT Mode and ETH Mode from eBay seller

The plan was to direct connect my workstation PC to my two TrueNAS machines which are racked up in a different room. The workstation is a 12700K w/ 64GB DDR5 all NVME storage running Win 10 Pro, the Edit NAS is a 4700G w/ 64GB DDR4 (rust & SATA SSD) running TrueNAS Core 13.0-U5.3, and the Backup NAS 3200G w/ 32GB DDR4 (rust & SATA SSD) running TrueNAS Scale 22.12. The TrueNAS servers were connected via Mellanox DAC (from above eBay seller) and the two longer runs to the workstation PC needed fiber so I got two 10m AOC cables from 10Gtek.

Setup was painless in the Windows machine and checking firmware was as easy as right clicking the interface and getting to hardware configuration (and a quick reboot). Both TrueNAS machines recognized the cards right away, and nothing more needed to be done other than get testing. All three cards were given dedicated IP's for each port, and were setup for direct connection to SMB shares on each TrueNAS machine. For the first 4-5 weeks of running on my Windows and Editing NAS, things were stable but max transfers capped out around 9 Gbps. I tried throwing in some of the tunables mentioned in the High Speed Networking Primer and managed to eek out an extra 1-1.5Gbps in iperf3 but very little real-world performance gain. Seeing more promising results from this thread and various YT resources, I tried to see if I could get that transfer a little closer to 25Gb.

For those of you that are more experienced with networking hardware this might be a no-brainer, but folks not all PCIE lanes are created equal when transferring over the network. Get those NIC's into a slot that runs to the CPU, preferably the top slot. Even though I had ample lane bandwidth, both my TrueNAS boxes had the Mellanox cards in a slot running to the chipset and that was the bottleneck. After doing nothing more than swapping the Mellanox cards to the top slot of each machine, speeds had nearly tripled:

Speeds to the Windows workstation are a tad slower since the top slot is occupied by the GPU, but still consistently above 25Gb. This was far better than I had expected based on the discussions I've seen here and on other forums. For now my biggest bottleneck is the sustained reads/writes due to the drive makeup of my TrueNAS machines. The Editing NAS is currently in the process of being transferred to a new chassis and upgraded to all-flash storage to get closer to saturating the NIC.

Overall, getting my little homelab upgraded to fiber for < $250 on three machines was a success. This setup will be more than adequate for the next several years, and if I decide to do something crazier like 100Gbe, at least the cables (the cheapest part) already support it!

 

[-cj-]

One more mention I wanted to share. I decided to bridge 1 gbe, 2x 10 gbe, & the two Mellanox 40 gbe ports for testing. 1 gbe performed at 1 gbe speeds, 10 gbe ports performed at 10 gbe speeds, and the 40 gbe only performed at 10 gbe speeds. Not sure if there is a software cap in TrueNAS or something, but when the 40 gbe are bridged with the rest of the ports, they will not operate at 40 gbe speds.

Aggregating multiple links (I assume LACP/Etherchannel) won't take advantage of the total bandwidth unless you have the ability to play around with the hashing method.. and even then, you can't get a single connection to span the multiple ports. I believe this is one of the reasons people use Infinitiband over Ethernet.. it's supposed to scale over multiple links much better. Plus the lower latency and higher speeds..

FYI, I have a Mellanox ConnectX-5 and it took a LOT of tinkering with BIOS (Power states, PCI power and speed, etc) to get my iperf3 to max out the links. With Jumbo frames, LRO and TSO enabled via ethtool, I can hit 40+ Gbps with a single iperf3 connection, Linux to Linux. Prior to all this tweaking the max I could get was 20-24 Gbps and for some strange reasons, those speeds would drop overnight. That no longer happens.