TrueNAS RaidZ2 Poor Read Performance 4X4TB Drive

[OpenSpeedTest]

This is my first TrueNAS build.
I got 2X 4TB WD Blue and 2X4TB Segate Drive (Normal Compute Drive)

This is RaidZ1

This is RaidZ2

Installed TrueNAS on top of EsXI
PassThrough MotherBoard SATA Controller

When i put 2 Drives in Mirror both pair working very well
When i put 4 Drives in RaidZ1 it's performance is fine but when copying large iso speed come down to 40MBps
but on RaidZ2 is is 18 to 21MBps
RaidZ2 is very very slow.
I don't know why is that.
Tested this on 10GbE MacMini and from windows from Esxi
So It's not the network issue.

 

[ChrisRJ]

Please read the forum rules (in bold read at the top) and post your hardware details

 

[OpenSpeedTest]

I can't see the Edit options

 

[OpenSpeedTest]

CPU : 3900XT
RAM : 32Gb DDR4
HDD 1 : Seagate Barracuda 4TB 5400 RPM Desktop Internal Hard Drive (ST4000DM004)
HDD 2 : Western Digital Blue 4TB 5400 RPM Desktop Hard Drive (WD40EZAZ)
HDD 3 : Seagate Barracuda 4TB 5400 RPM Desktop Internal Hard Drive (ST4000DM004)
HDD 4 : Western Digital Blue 4TB 5400 RPM Desktop Hard Drive (WD40EZAZ)
Intel 1GbE, tp-link 10GbE, Both are working fine.

1Tb Samsung Evo 970
25Gb TrueNas Core on Top of VMware ESXi
Client
1 ) M1 MacMini with 10GbE
2 ) Window 10 from Esxi

 

[Heracles]

When i put 2 Drives in Mirror both pair working very well
When i put 4 Drives in RaidZ1 it's performance is fine but when copying large iso speed come down to 40MBps
but on RaidZ2 is is 18 to 21MBps
RaidZ2 is very very slow.
I don't know why is that.

Hi,

When you write to Raid Z1, you have 3 data disks and only 1 parity to calculate.
When you write to RaidZ2, you have 2 data disks and 2 parity to calculate.

For RaidZ1: Take your original ISO, split it in 3 parts and that is the amount of data you need to write to each disk. You also need to process and write one set of parity equal to that size (total 1/3)
For RaidZ2: Take your original ISO, split it in 2 parts and that is the amount of data you need to write to each disk. You then need to process 2 sets of parity for that size (total = 2 * 1/2 ; so 3 times more)

50% more data ( 3/6 of the original size instead of 2/6) and 300% more parity...

Does it make more sense that it takes more time to write on RaidZ2 compared to RaidZ1 ?

 

[Yorick]

These are SMR drives. https://www.truenas.com/community/resources/list-of-known-smr-drives.141/

They’ll be slow. Slower in raidz2 than raidz1, maybe: Not sure I’ll be able to get to the bottom of that. But slow in general because SMR: Yes. Particularly when writing larger files.

If you have the option to replace these via RMA, do. Drive manufacturers acknowledge that SMR is not well suited for ZFS.

If that is not an option and you have the option to not run them in ZFS, do.

If that is not an option, plan to replace the drives with CMR drives when you are upgrading capacity or when they fail.

 

[OpenSpeedTest]

@Yorick Maybe CMR is the issue here. But i did check the drive i can find any CMR mention or in specs. But i did noticed the Seagate 4TB drive is significantly slim and light weight when compared to 4TB WD Blue.

@Heracles Read Performance is too bad for me.

Here is my test. i have a folder with 5 Windows 10 iSO [Same file slightly different name] That is 27.1 Gb

2 Drives in Each VDevs as Mirror (7Tb usable storage) SPEED = R/W 260 MBps

260 MBps for Read and Write Consistent Performance. And i am happy with the performance. [Windows Estimate time was 1.30 Minutes ]

4 Drives in RaidZ2 (7Tb usable storage) SPEED = W 140 MBps R 20 MBps
Copying Data to NAS (Write Performance)
Speed is not Consistent shows between 140 to 250 MBps Average is around 140 i believe for Write [Windows Estimate time was 2.30 Minutes]
Copying Data FROM NAS (READ Performance)
Speed is not Consistent shows between 20 to 60 MBps Average is around 20 i believe for Write [Windows Estimate time was 16 to 20 Minutes]

4 Drives in RaidZ1 (10Tb usable storage) SPEED = W 290 MBps R 60 MBps
Copying Data to NAS (Write Performance)
280 to 290 MBps [Windows Estimate time was 1.30 Minutes]
Copying Data FROM NAS (READ Performance)
Started at 400 to 700MBps for few seconds and came to 50 to 70 MBps and average was around 60MBps.


I don't want to deal with RMA etc so i am going to keep using this drives is 2x Mirror VDev. I will keep a copy of my imp data in another drive.
Also i can tolerate 1 drive failure on each mirror right?

How to identify and avoid CMR drives in future? If that is the root cause of this issue.

Thanks

 

[OpenSpeedTest]

My 10Gb Nic is connected directly to M1 MacMini
Every time when i reboot the IPAddress will be 0.0.0.0
i need to set it back to 192.168.2.1
is there is a way to set it permanently?

 

[Heracles]

Read Performance is too bad for me.

As you wish. It is considered that a pool as the performance of as many drives as it has vdevs. Also, mirrors are considered better at IOPS and random access while RaidZ is considered better for large sequential reads / writes.

2 mirrors vdev :
Each mirror receives 50% of the volume and write it twice as is.
For reads, each one of the 4 drives can contribute a different part of the data, so all 4 of them are put to the job for the read. That is why they are very fast.

1 RaidZ1 vdev :
Each drive receives only 33% of the data and the last drive is parity. Each drive has less to write when compared to mirrors, so RaidZ1 ends up even a little faster. Also, your test being a few large files, that is where RaidZ1 is good at when compared to mirrors.
But for read, every single access requires all the drives to work together on the same request. As such, they do not work in parallel as with mirror. Read drops to much lower. Basically, the 3 RaidZ1 as the read speed of a single drive, so is about 4 times slower than your 2 mirrors that read with all 4 drives at once.

RaidZ2 is worst than RaidZ1 on all points : with one less data disk, the drives need to write more and read more for everything. To go up to 3 data drives instead of 2, that is 50% more. It is significant.

Finally, your test are probably affected by caching. For a more representative test, you should use files that contain random data. A good way to produce that is to encrypt your files. Encryption is designed to produce an output as random as possible. But even that, if you do with the same files test after test, leftovers from one test may very well remains in cache and affect the objective result.

 

[OpenSpeedTest]

I don't know it's affected by caching or not. but the results where same when i copy over 100Gb or random files that contain text, html, photos, mp4 , disk images etc random data each files are unique.

In my case 2X Mirror 2X Vdev giving 260+ For Write and 280+ for Read so i am getting 2.5 to 3Gbps of Data Transfer speed over my 10GbE.
Which is more than enough for me.

I was thinking like i Must use Z1 or Z2 and that was the only option.
But that 2X Mirror 2X Vdev was really Awesome and serve the purpose.

 

[Heracles]

But that 2X Mirror 2X Vdev was really Awesome and serve the purpose.

Now that you have selected your pool design, don't forget to plan your backups. Remember that TrueNAS server, no matter how robust it is, can be more than a single point of failure. Backups are always required.

 

[ChrisRJ]

How to identify and avoid CMR drives in future?

It is SMR drives you want to avoid like the plague with ZFS. CMR drives is what you want

 

[OpenSpeedTest]

My 10Gb Nic is connected directly to M1 MacMini
Every time when i reboot the IPAddress will be 0.0.0.0
i need to set it back to 192.168.2.1
is there is a way to set it permanently?

@ChrisRJ Ok, It is SMR, Better choose NAS drives right. I am not running 24x7 that is why i choose normal HDD.
How to add a static IP to my 10GbE interface?
No DHCP.

 

[Heracles]

Better choose NAS drives right. I am not running 24x7 that is why i choose normal HDD

NAS drives are not only meant for 24x7. They have many other characteristics that you are looking for. Ex: should a sector be hard to read back, a normal drive will fight it for as long as possible before giving up. It does so because it considers that it is the only one holding that data and should it not be able to get it back, it will be lost. A NAS drive being meant to be in a redundant structure, it will give up way faster and will rely on the system to compensate for it. ZFS will do exactly that and recover the data from other drives before working on fixing that sector after.

Vibration is another item. When they are all spinning together, the drives make much more vibration. NAS drive are designed to sustain that while normal drives are not. Vibration is one more thing that will damage drives...

I would still recommend NAS drives instead of regular desktop drives...

 

[OpenSpeedTest]

How to add a static IP to my 10GbE interface?

 

[NugentS]

 

[OpenSpeedTest]

View attachment 49077

OpenSpeedTest said:
My 10Gb Nic is connected directly to M1 MacMini
Every time when i reboot the IPAddress will be 0.0.0.0
i need to set it back to 192.168.2.1
is there is a way to set it permanently?

 

[NugentS]

So you have a direct connection M1 to TN NIC, so switch or anything?
If so - the address should stay there once set
Is there another card that gives TN access to the internet and whats its IP address

 

[OpenSpeedTest]

For internet I have a 1GbE NIC. DHCP enabled and Mac address bind to a static IP on the router. Everything fine.

But my 10GbE which is connected directly to M1 Mini and When M1 is offline or after a restart my ip address for 10GbE shows 0.0.0.0 .. everytime I need to manually set 192.168.2.1

For 10GbE no router or switch.

 

[NugentS]

Copper or fibre?
If copper can both ends run at 1Gb or are they fixed to 10Gb?
If they can run at 1Gb can you get a cheap switch to put in the middle and see if the same problem occurs - this may help pinpoint the trouble (not a permanent solution). I wonder if TrueNAS is resetting the IP address when there is no link. Seems daft though