Hight 'iowait' on data transfer (read/write) causes TrueNAS Scale to freeze

[m.dolensek]

Hello, first off I would like to say that I'm quite a new to the whole TrueNAS system and that my hardware is not the recommended type but still task like file copy in my opinion should not result in a system freeze like the one I'm having (I think).

So to get to the point... When I'm starting any kind of file transfer to my storage pools (or while app is doing library scan), read or write, using SMB share or with Syncthing my CPU usage jumps to around 30-50% and the rest is used by 'iowait'. Because of that the whole TrueNAS system becomes unresponsive and it completely freezes. When trying to login to WebUI I receive message 'Connecting to TrueNAS ...' which after quite a while it gets connected and shows login page but I'm still unable to login. Whats interesting to me that apps installed in FreeNAS are still working while copy is in progress, I can login and use them, some do freeze up on times but becomes operational again in a few minutes. SSH is also still working, slow to respond but usable.

I first thought that the problem was encryption on disks and because of much fiddling around the system already I decided to do a fresh install of the TrueNAS system without any encryption on pools or datasets but to no avail. The high 'iowait' is still present.

Below are screenshots taken while Syncthing copy was in progress (writing data, last trasfer below):

The graphs from report page look like this (taken after the copy, since FreeNAS was unresponsive while copying):​

I/O statistic ('iostat') like this:​

And the CPU usage ('htop') like this:​

Hardware:
HP EliteDesk 800 G2 (all firmware/BIOS is up to date, stress test done w/o errors)

M.B.: HP with Intel Q170 chipset​

CPU: Intel Core i5 - 6500 3,2GHz 4 cores​

RAM: 2x4GB DDR4 - 2133 Crucial​

NIC: Intel I219LM Gigabit​

SATA controller: Intel (onboard), HP SmartArray P222 (PCIe)​

HDD's

4x Segate Ironwolf 4TB = RAIDZ1 (HP SmartArray P222)​

- sda, sdb, sdc, sdd​

3x WD Red 2TB = RAIDZ1 (onboard controller)​

- sdf, sdg, sdh​

1x Segate 750GB = boot pool (onboard controller)​

- sde​

​

Software:
System

TrueNAS SCALE 23.10.0.1​

Apps, services

Syncthing, Sonarr, Plex, SMB, NFS​

- App storage connected with host paths as well as NFS, no difference.​

​

I have tried transferring to/from the disks on my TrueNAS connected to the onboard controller as well as to the disks connected to HP SmartArray controller but the results are the same. The source/destination of files also doesn't make a difference, have tried transfers from mine and some other laptop and from Ubuntu server set up on Raspberry Pi. Nothing makes any difference.

So I'm turning to to you guys if maybe someone had a similar issue or is familiar with something like that to help me pinpoint the issue... I have tried and searched everything I could but did not find any viable solution. Many thanks to everyone in advance....

While writing this and while the TrueNAS was frozen I managed to see some critical error that i received (in case it has anything to to with this, but probably just a cause of a system frezze):
Failed to check for alert ScrubPaused...(rest is attached)

 

[Arwen]

Their are a few things. Hardware RAID controllers don't work well, (and in some cases, not at all), with ZFS. Someone else may have to check, but you should see if this device is a HW RAID controller;

HP SmartArray P222 (PCIe)

Next, what is the exact model / firmware of the WD Red 2TB disks?

Western Digital damaged the WD Red NAS disk line by changing the recording methodology. Some of the WD Reds, including the 2TB model, now use SMR recording technology. This is in-compatible with ZFS, though can work, delays and other bad behavior can occur. When caught, Western Digital had to move the original WD Reds to WD Red Plus line, which don't use SMR recording technology. If your WD Red 2TB disks are old enough, they may not be SMR. Thus, the request for the exact model & firmware.

Last, the amount of Apps with the absolute minimum RAM of 8GBs won't help performance. Consider adding RAM or removing some or all Apps. This is shown because you have SWAP in use. Generally TrueNAS SCALE does not need SWAP, but will resort to using SWAP if running low on RAM.

 

[PhilD13]

The HPE Smart Array P222 Controller is a "low profile, 6 Gb/s, PCIe 3.0, Serial Attached SCSI (SAS) RAID controller that provides
entry-level, second generation storage performance, scalability, and data protection for select HPE ProLiant Gen8 rack servers and
tower servers." I don't think it has a JBOD or pass through mode, but if you wish to try using it with Teuenas then look in in the cards bios (not the computers) during boot and set the card to use JBOD or pass through mode. If those options are not available then it can't be used with Truenas

As Arwen outlined, You would be better off in the long run replacing the P222 with a LSI card that has already been updates, and flashed to IT mode for Truenas use.

You also need more then the minimum RAM is you want any performance, and you will need even more for any VM's

The WD Reds are likely SMR drives which is very likely causing the freezing but without an exact model number no one can tell.. In a nut shell; The issue with SMR drives (no matter who makes them) is SMR buffer on each drive fills up due to the SMR system on each drive trying to rewriting ajacent tracks which causes massive I/O waits (10's of seconds) leading to data corruption, disk kickout of pools due to non response, (marking drives failed) and ultimate failure. This SMR rewite occures on every write so with a copy on write system such as ZFS it is totally incompatible.

 

[m.dolensek]

Hello first off thank you both for the reply :)

Yeah the HP Smart Array P222 is a hardware RAID controller but I've put it to HBA mode so the RAID is done by software since the drives are passed through. I've also just came across a good post on this forum about the RAID controllers and their modes. And according to this, the controller could be the problem since it has a some kind of memory installed and it is stated that the ZFS can flood this memory with I/O. So that could be the reason why in Windows environment everything was working fine before.

WD Red's are a bit older had these laying around so I thought I would throw them into a system...was aware of this issue with WD that's why I've bought IronWolf's few years back....anyway the three WD drives used are CMR.

- WD Red 1TB (WD10EFRX)​

....and yeah they are 1TB, had mistakenly put 2TB in the original post...oops.​

So I guess the first part with the HP P222 controller is somehow a dead end until I get myself a new LSI card like PhilD 13 pointed out and according to the informaton from the post I've linked.

But the part with the onboard SATA controller and it's high I/O wait is still a mistery since the drives are not SMR. Will get myself a few sticks of RAM, and see if that solves it. Because was not that focused on the SWAP usage before Arwen pointed it out. Cuz I thought that the system had few bits left according to the FreeNAS dashboard, but when looking the usage with 'htop' I now see that even when the system is idle it uses almost 1GB of SWAP.

Will post an update when I do some more testing after the small upgrade.

 

[m.dolensek]

So I have received my two 16GB sticks of RAM yesterday and am glad to let you know that that was the only missing resource Now I have 40GB of available RAM (2x8 & 2x16GB), a bit strange config but now that I see how much the ZFS loves RAM I'll stick with that. Now I have zero of extreme 'iowait' values like before. While I was doing some similar tests and after even some more intensive ones the highest 'iowait' value was around 35 (before it was around 90 while idling after transfers).

I also have to point out that the HP Smart Array P222 (on its latest firmware) with its HBA mode turned on is also working fine now even though it has the onboard cache module. Possibly because it's bypassed in HBA mode?...but someone else should confirm this. Did not find any documentation on this, juts my thoughts....

So anyone with similar problem and without the cache VDEV for the storage pool just grab some more RAM and the 'iowait' will go away

 

[Arwen]

I checked the WD10EFRX, and they are CMR, so good.

Glad you have the RAM part taken care of. In the old days, (>6 years ago), the minimum RAM was raised to 8GBs because of irregular performance and occasional crash issues. But, with Apps / VMs on SCALE, or VMs / Jails on Core, the minimum is just that, minimum without extras.

Yes, while 2x4GB & 2x16GB is un-even, it won't generally cause any problems.


Please be aware that while your HP Smart Array P222 in "HBA" mode may work, and work for years, without trouble, their are known problems with RAID controllers. The biggest is that they may re-order writes for performance reasons, aka elevator seeks / writes. This out of order writing is not what ZFS wants. It can work fine, as long as an out of order write does not occur during a crash or un-expected power cycle. Then, the pool can be damaged, potentially beyond repair.

ZFS was designed as a Copy On Write, (COW). This means it writes the following in this order:

1. Data to free space
2. Metadata for that Data, (aka directory entry updates), to free space
3. Top level Metadata that makes the previous changes permanent, to oldest ring buffer entry for Uber blocks

By doing it this way, any crash or un-expected power cycle that occurs in the middle of a write, leaves the ZFS pool and Datasets as they were before the write started. No corrupt data, no file system check at boot, import or mount. True, the write in progress is gone for good, (unless you know what exactly was happening). Yet this applies to every file system.

But, if either step 2 or 3 is completed before step 1, and a crash or un-expected power cycle occurs, the pointers to the data now point to empty space, (which is likely filled with garbage). This is not completely fatal, just that data / file is corrupt and needs to be restored from backups.

However, if 3 is written before 2, and we get a crash or un-expected power cycle, more serious pool or dataset corruption can occur. Possibly even to the point of not being able to get rid of it. Except copy all usable data off and re-creating the pool.

ZFS was written to survive crashes and un-expected power cycles, even hundreds, or hundreds of thousands, without data loss, (of existing data). Of course hardware faults can occur. And even hardware lying about having written data, (like a HW RAID controller can do).

So, the general recommendation is never to use RAID controllers with ZFS. And if using a hardware RAID controller capable chip, use it in clearly defined pass through mode, (aka LSI IT mode, Initiator Target).

Lecture done .

PS - This is also for others that may read this...

 

[m.dolensek]

Arwen, thank you very much on sharing your knowledge and taking your time to write this...really appreciate it!

Will keep in mind that I change out this HP controller as soon as possible to avoid any problems in the future. Because I really fell in love with the TrueNAS system I'm planing to make this as my primary server at home. But yeah if this "recycled" RAID controller is posing as a potential thereat to the safety of my data it is necessary to change out...since there is much that can go wrong even in a perfect build.