MCX354A-FCBT TrueNAS Scale

[Jacoub]

Greetings All
I'm running latest release of TrueNAS Scale Version 22.02-RC.2 which is Debian 11 based.
lspci | grep Mellanox
0b:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
Downloaded Debian 10.9 Driver from

Linux Ethernet Drivers

NVIDIA offers a robust and full set of protocol software and driver for Linux with the ConnectX® Ethernet family cards.

www.mellanox.com

but installing it shows it is only for 10.9
I'm using the same model on TrueNAS Core as 40GbEthernet "Not IB" & it is working fine no issues
any idea how to configure it

Thanks in advance

 

[engineerdj]

Having the same situation with a MT27800 ConnectX-5 card.

 

[jgreco]

TrueNAS is an appliance OS and you are not supposed to be downloading and installing drivers from random sources (even if it's a manufacturer).

The Mellanox cards are not particularly recommended for use with TrueNAS. They're kind of fiddly, firmware and driver dependent, configuration may be needed, and performance has been problematic for some people.

You are encouraged to make use of cards recommended in the 10 Gig Networking Primer. The Chelsio and Intel options listed therein are both strong choices for highly performant systems.

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...

www.truenas.com

You can always put in a request in Jira for Mellanox driver updates, but this won't happen until at least(!) the next OS release.

 

[Jacoub]

Im using supermicro with built in 10G Intel 550 but i'm looking to use 40Gb since I'm transfereeing a huge amount of file size between a 24x14TB TrueNAS SCALE & same configuration but TrueNAS CORE. I find it odd these cards are supported in FreeBSD & Debian 10.9 but not Debian 11. I,m hoping the release version on Feb 22, 22 will have support if not I will go back to TrueNAS Core which will be painfull since I already upgraded Pools accordingly

 

[jgreco]

painfull since I already upgraded Pools accordingly

There's a lesson in there. Don't just click upgrade because a button appears. Some of us only upgrade pools as we are getting ready to abandon a version for a newer one, which means we can always roll back.

 

[engineerdj]

TrueNAS is an appliance OS and you are not supposed to be downloading and installing drivers from random sources (even if it's a manufacturer).

The Mellanox cards are not particularly recommended for use with TrueNAS. They're kind of fiddly, firmware and driver dependent, configuration may be needed, and performance has been problematic for some people.

You are encouraged to make use of cards recommended in the 10 Gig Networking Primer. The Chelsio and Intel options listed therein are both strong choices for highly performant systems.

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...

www.truenas.com

You can always put in a request in Jira for Mellanox driver updates, but this won't happen until at least(!) the next OS release.

Would this be the appropriate ticket to add the request to?

https://jira.ixsystems.com/browse/NAS-114296

We're also using 100GB Mellanox on our Supermicro chassis with SCALE and need the throughput to our transcoding farm.

So far I've managed to load the modules following @RapidScampi post below:

InfiniBand 40Gb Mellanox Installation on TrueNAS Scale?

Hi every one. Roberto here trying to get my Mellanox InfiniBand dual port setup with my new TrueNAS Scale box. I am just testing the platform with the Idea to use same server to as Plesk server or even JellyFin. Me and my wife have an small photo and video studio. Currently we have a TrueNAS 12...

www.truenas.com

But I haven't been able to get it to work completely. We're building out a 25GB network around this to the workstations and transcoding farm to see if better support for these cards comes around, hopefully in an upcoming release. I've seen some additional configuration and work-arounds to get this working, but I'd rather not have to hack-up the OS to do that, and risk stability issues down the line on a production system.

We did engage sales to evaluate the appliance, M-Series, but unfortunately out of budget for us. Nice sales guy, though. Good product.

 

[Jacoub]

There's a lesson in there. Don't just click upgrade because a button appears. Some of us only upgrade pools as we are getting ready to abandon a version for a newer one, which means we can always roll back.

To be honest I did that on purpose thinking compatibility issues may affect data integrity especially it is RC hence, I sacrificed throughput for Data integrity

 

[Jacoub]

Updated to TrueNAS SCALE 22.02.0.1 Then Installed TrueNAS Core 12.08 I was able to read all Datasets

Can someone advice if this is acceptable or it can be problematic in the future??
Thanks in advance