SMB is faster than ISCSI. Help!

[mbosman]

Hello Everyone!

First post on TrueNAS, so very much a newbie!

I currently have a system that I am building as a general storage server. I am planning on using it to store my Media Library, and use it as a Steam Library.
Currently I am in a "testing" phase where I have connected the TrueNAS server to my Gaming pc using a Mellanox ConnectX-3 CX311A in both. And the link seems to be working correctly because Windows says 10Gbit speeds.

In TrueNAS I set a static IP for just that card (dhcp for the 1 Gigabit NIC to my router) and in Windows I have done the same to make sure they communicate.

In my testing, I have set up a 256GB SPCC M.2 NVME by itself in a VDEV. Then I have create a dataset for the ISCSI and created a ZVOL inside that dataset of about 200GB.
I did the same with the Samba, except I left it at the dataset level.

I was able to mount the ISCSI drive to my Windows and the Samba drive as well.
When I tested a copy of a file that is 7GB to the ISCSI I have worse performance than copying it to the Samba.

Specs of server:
CPU: i3-12100F
MOBO: ASROCK B660M Pro RS
RAM: Corsair LPX 32GB 3200
NIC: Mellanox ConnectX-3 CX311A
Add-on: LSI 9207-8i (IT Mode)
OS installed on a Teamgroup 500GB CX2 (just laying around, and temporary)
Other ssd: 256GB SPCC M.2 NVME


Would anyone know what could cause this? I was expecting the other way around if anything?
Any help would be appriciated!

Thanks

 

[mbosman]

Also wanted to mention that the IPERF3 results were as expected

 

[jgreco]

When I tested a copy of a file that is 7GB to the ISCSI I have worse performance than copying it to the Samba.

Yes, of course. Why is it surprising that having your NAS do a much more complex task when it is under-resourced would cause it to be slower than when you do a task the way it is intended?

Resource - Why iSCSI often requires more resources for the same result

iSCSI is a SAN protocol. NFS, CIFS, etc., are NAS protocols. For a NAS protocol, the client sends a command to the filer, such as "open this file", or "read ten blocks", or "remove this file." On the filer, the local NAS protocol daemon...

www.truenas.com

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

 

[mbosman]

Yes, of course. Why is it surprising that having your NAS do a much more complex task when it is under-resourced would cause it to be slower than when you do a task the way it is intended?

I see! That makes a whole lot more sense now. I appreciate the quick reply.

I just wanted to make sure I wasn't missing anything, I already knew the RAM might be an issue, and the articles you mentioned clearly state that it will be.

Future upgrades are in sight already!

Thanks again!

 

[jgreco]

It's not just the RAM (ARC). It's really the whole thing of serving iSCSI is inherently less efficient because ZFS has no idea what it is storing or why. This makes it less efficient. You can "fix" this by throwing more resources at the problem, but it can get pricey to do so.

 

[mbosman]

Yeah, I was thinking of instead of using RAM, maybe add an Optane m.2 drive and set it up as a CACHE L2ARC to help speed it up a little bit.

Currently I am still getting sufficient enough speeds for Steam Games, which is the only thing I would be using the iSCSI for. Everything else I would use SMB and NFS.

 

[Ericloewe]

I have my doubts that would help, but you can drill down into the ARC statistics and see whether the deadlists have meaningful hit rates.

Thing is, any meaningful improvement here will be more expensive and slower than just adding a local NVMe SSD.