Configuration and performance testing

[Kosta]

Hello,

I am not new to TrueNAS, have been running it for some time now, more or less testing period. Now after couple of months, I would like to go productive, and put all my data on it. Before I do however, I would like to know which configuration would give me best usage and performance.

My system revolves around an ESXi host with various VMs, including TrueNAS, which has 2 SAS2008 HBAs passed through for 10 2TB disks in the server. Up until now, I had a single VDEV RAID-Z2 configured, all standard settings, nothing changed from default. I believe lz4 was on, 128k record size. I deleted that VDEV and starting fresh now. The VDEV will be used solely for either LUNs or fileshares (NFS and/or SMB).

What I'd like to know are couple of things:
1) single disk read/write speed for each of the drives
2) zfs-pool read/write speed
3) whether the speed I am getting is what I should expect

So here is where I need help, how would I go about this? Which commands do I use?

What I did myself:
diskinfo -t /dev/da(x)
I get numbers similar to what tech specs of disks provide.

Were I on Linux, I would mount the disk and use parted to create a partition, then use dd to perform write and read tests like this:
dd if=/dev/zero of=/mnt/xxxx/tmp.dat bs=2048k count=50k
dd if=/mnt/xxxx/tmp.dat of=/dev/null bs=2048k count=50k
(or something like this, didn't try it yet, as I am neither Linux pro nor do I have Linux controlling the HBAs)

How would I go about this under TrueNAS Core, which I currently have running?

Thanks
Kosta

PS. Also been contemplating if to ugprade to Scale, but since I merely want to use LUN, NFS or SMB, I see no benefit.

 

[joeschmuck]

When you test, make a dataset with no compression (compression is the default). Also you need to transfer more data than you have RAM or it will be cached and also provide invalid results.

There are several threads on this forum about how to benchmark your throughput. I find it easier to do a Google search on for example "truenas benchmark" or maybe "truenas speed test" and you should be able to locate several threads that support what you are trying to do.

TrueNAS Core is based on FreeBSD so if you know the command in Linux, Google that same command but add "FreeBSD man" and you will likely find something.

For Core: (My results from 13 years ago)
dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 336.745898 secs (318858175 bytes/sec)

dd of=/dev/zero if=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 287.611204 secs (373331014 bytes/sec)

As for using Scale instead of Core, I'd say this is a personal preference. Core is definitely more mature than Scale but Scale is actually doing pretty good. You can also migrate to Scale at a later date. Actually I flip/flop between the two for testing on my machine. I never upgrade my pool if/when asked so I can continue to do this, plus the new features in ZFS do not provide any benefit to me for my home system.

 

[Kosta]

I followed this to disable caching:

Testing Benchmark ZFS Pool

Does anyone know how to test the benckmark on a ZFS-Pool? And how can I test it on a Linux VM and Windows VM? In this PDF I saw the command has been used: https://www.proxmox.com/de/downloads/item/proxmox-ve-zfs-benchmark-2020 command: fio --ioengine=libaio --filename=/dev/sdx --direct=1...

forum.proxmox.com

I get around 400/400 MB/s read write result with the above dd-command.

Diskinfo -tv tests show around 110-180MB/s, depending on the drive.

Does the above result mirror what I should expect?

 

[joeschmuck]

Did you use a dataset that had no compression? If not, I'd be curious if there is a difference for the dd command.

 

[Kosta]

Yes, there are no self created datasets, only the one created by the Pool-creation itself and there I turned off LZ4. Otherweise I get close to 4000MB/s, tested before turning it off.
Also I set zfs set primarycache=metadata YourPoolName, according to this:

Testing Benchmark ZFS Pool

Does anyone know how to test the benckmark on a ZFS-Pool? And how can I test it on a Linux VM and Windows VM? In this PDF I saw the command has been used: https://www.proxmox.com/de/downloads/item/proxmox-ve-zfs-benchmark-2020 command: fio --ioengine=libaio --filename=/dev/sdx --direct=1...

forum.proxmox.com

However, even if I then set it to all (not any, this is wrong), I get "source=local", instead default. No idea what that does however.

I think the next meaningful scenario would be to create a VMDK on my Samsung NVME, add it to the TrueNAS, mount it and then test with dd?

 

[joeschmuck]

I think the next meaningful scenario would be to create a VMDK on my Samsung NVME, add it to the TrueNAS, mount it and then test with dd?

That is fine if it's just a test to satisfy your curiosity however using VMDK's to store data using ZFS is risky and not advised. I only use a VMDK as my boot drive and pass through the HBA controller to the VM for the hard drives.

 

[Kosta]

It was late yesterday. While yes, just curiosity, I actually started creating LUNs and Shares, both SMB and NFS, to see which scenario delivers better performance. Those shares or target(s) will then become a storage space (NFS, SMB) or LUNs for datastores, which will hold VMDKs. This whole thing started because I had quite poor performance and before troubleshooting, I just started new and now going step by step.

 

[joeschmuck]

I understand now. I could be wrong but I believe a block format (iSCSI) would deliver the best performance in serving up VMDK files. @grecko would be able to answer this better than I could.

 

[Kosta]

That is what I’ve been reading all over and would be my much preferred configuration, however tests on the previous configuration have shown much better NFS performance. I do not count out bad or wrong configuration.

 

[Kosta]

Hmm, next thing:
I am now in Windows, created two things on TrueNAS: lun0 for iSCSI bind to ESXi directly and an NFS share, so that I can create two VMDKs respectively on my windows server, and then test the speed with Crystal Disk.

However, testing shows "real" numbers for Reads, something like 200-300MB/s, and writes are abysmally high, something like 1600MB/s. I believe there is still some caching here. How do I remediate that?

 

[joeschmuck]

If you have tried very large test file size (larger than you have RAM, you can go up to 64GiB), then I suspect you are seeing your NVME performance. Of course a real hard drive will not give you that kind of performance.

Also, make sure you have random test file data selected (Settings -> Test Data -> Random) in Crystal Disk Mark.

I think you need to benchmark using the real hardware otherwise your results will not be accurate.

 

[Kosta]

I am doing only real hardware tests now. Basically have two datastores set up, one iSCSI LUN (which is not between VM and TrueNAS, but ESXi and TrueNAS of course), one NFS. On the other side is a windows server, which is in the end-phase going to use those to create main shares for the data. Those drives are simply VMDKs that have been assigned to this windows server in ESXi.
Currently my TrueNAS has 32GB RAM, but I will lower it to 8GB, and do as you said, in CM set Random and use 64GB tests.

 

[Kosta]

Well OK, this is my result with LUN in place. LZ4 is disabled, primarycache is Metadata for this specific LUN only. 8GB RAM.

Sounds realistic to me, considering the speed of my drives (all 5400, some "newer" WD Red 2TB, and some older Seagates):

What do you think?

Oh and NFS performance?
Abysmal. Not sure why. It is NFS3 only yes, so single path, but still, getting under 100MB/s for both reads and writes, as usualy, writes lower than reads. Don't have any conclusive tests from before, just some trials with FIO, but those were also around Read 90 / Write 80 MB/s, which loosely resembles current result.

In any case, current LUN result is quite good. I will proceed with some data copying going from datastore on my NVME in the server itself and to TrueNAS LUN.

 

[joeschmuck]

What do you think?

More realistic values for the configuration.

 

[Kosta]

LZ4 totally hurts performance of the LUN over iSCSI? I have tested couple of times back and forth, enabling and disabling LZ4 on a LUN, and copying from my internal NVME to the LUN in windows (single 40GB file) transfers at about 110MB/s with LZ4 on vs over 250MB/s with LZ4 off.
When I disable LZ4 it takes a while until the speed goes back to 200+, but it will go there.

Does that make sense?

Copying from the LUN to NVME goes at about 350MB/s.

 

[joeschmuck]

Does that make sense?

I honestly do not know but I'm sure this topic has been covered before.

Have you read this?

Some differences between RAIDZ and mirrors, and why we use mirrors for block storage

ZFS is a complicated, powerful system. Unfortunately, it isn't actually magic, and there's a lot of opportunity for disappointment if you don't understand what's going on. RAIDZ (including Z2, Z3) is good for storing large sequential files. ZFS will allocate long, contiguous stretches of disk...

www.truenas.com

 

[NickF]

Also,
This:

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

 

[Kosta]

Many thanks. No, I did not, but did now.
Considering the fact that my VDEV will not, ever, hold any VMs (and by that, I mean System-Partitions, like boot drives), I am not overly concerned. My (single) VDEV holds things like documents, simple files, media (videos, photos, music), and the only thing done with these is either being copied to the VDEV or deleted. Everything productive is being done either on my Macbook or on the NVME in the server (which mostly holds all VMs). It's most likely something like 90% archive, but being used the whole time mostly to read.
My intent is to put one or more LUNs on the TrueNAS via iSCSI to ESXi, and then put VMDKs on those, and then create shares on a windows server.
The main reason actually is because I want to take TrueNAS out of the windows domain, and control access to the shares through the domain and NTFS permissions. All storage or backup devices must be outside of domain for security reasons.
And finally, space is more important than performance.
And since I've seen better performance with block than NFS, I think I should stay with it. Currently with RAIDZ2, which seems like a logical choice.

And yes, I was wrong about LZ4. There was something else amiss... I have very good copy-times and speed from NVME to the TrueNAS now, even with LZ4 on..

 

[Kosta]

I slept a night... and actually, an idea with mirrored setup sounds quite attractive. The reason is twofold:
- much easier space expansion, as I understand it - I have 24-bay 4U server, and going 2-way mirrors is a nice way of keeping it flexible (and for now, I would keep large static data still on my Synology, while using TrueNAS for more lively files), and later expand moving everything to the TrueNAS
- disks are only going to get larger, and with that my whole pool can grow, including expanding the 5-by-2 VDEV to for instance 10-by-2 VDEV

The benefit of increased speed is actually quite large, I didn't expect that - I think I can it's around double, both reads and writes.

With single RAIDZ2 array, I can achieve sustained 300 MB/s on the empty VDEV. Which will only go down with time. With mirrored I achieve transfer speeds of 500-600 MB/s, and according to documents, for LUNs a way to go. Expanding server memory to 256GB is an option, that would give me ability to give TrueNAS 64GB of RAM.

What I also noticed, with gstat -dp, all drives are actually being used, going to 80%+. With RAIDZ2, actually only two drives were high usage, while others had very low usage.

 

[Davvo]

Make sure you understand the impact on resiliency with such a pool.

ZFS Storage Pool Layout

This resource was originally created by user: @Davvo on the TrueNAS Community Forums Archive. https://www.truenas.com/community/resources/zfs-storage-pool-layout.201/download [1] This amazing document, created by iXsystems in February 2022 as a “White Paper”, cleanly explains how to qualify...

www.truenas.com