Docs: SSD Boot Device

[sgroesz]

The docs state: "You do not need an SSD boot device, but we discourage using a spinner or a USB stick for obvious reasons." SCALE: Minimum HW Requirements

To me, this is not obvious. I have not used Free/TrueNAS, so maybe it is completely obvious to those who know better. Based on my knowledge, I would think the boot device would be the least important and slowest media. The only time a fast boot device should matter is when, well, booting up. Which, should occur very infrequently. Also, if the boot device fails, you can reinstall and recover your system pretty easily. Compared to data drives, the boot device should be the least important concern, right?

 

[Samuel Tai]

USB sticks have very poor reliability, and are often hacked to appear larger than they actually are. They're discouraged from use as TrueNAS boot media because they cause too many failures, especially if you configure your system dataset (which includes the logs and the system configuration database) to reside on the boot pool.

Spinners aren't susceptible to the same problems as USB sticks, but you end up wasting a lot of capacity using a spinner as a boot device. Whereas, you can find small M.2 SSDs (16/32/64 GB) very easily.

 

[sgroesz]

Thanks for the fast response. I think the docs can be improved by briefly explaining what are the preferred characteristics of the boot device. Endurance? Write speed? Read speed? Maybe a link that provides more in depth detail about the considerations when selecting a boot device.

When I read "You do not need an SSD boot device, but we discourage using a spinner or a USB stick for obvious reasons", my first question is "HDD and USB not recommended, SSD not REQUIRED... well, what other options are there?"

I do understand wasting capacity on spinning drives. As well as potentially wasting watts. Is an 80gb SATA drive preferred over a cheap 16GB SATA SSD? I have the 80gb SATA available; I would need to order a 16GB SSD. I have 2 very high endurance 400GB SAS SSDs, but I don't want to use them where they are not needed.

Nowadays, there are reliable SD media available with high endurance that are able to act as OS media.

 

[WI_Hedgehog]

That's something I wondered for a long time, until I knew enough to know better, as it turns out. Even other heads of IT whom I asked didn't know. Maybe that's one of those things that when you're familiar enough with these system it becomes apparent...like a rite of passage. Maybe it's "the secret handshake" of the Elite Server Admins. Makes for good lore anyway...

 

[NickF]

The problem is that the amount of endurance on the NAND on a USB flash drive is pitiful. By default TrueNAS scrubs (and for good reason) the boot drive every month, and this accelerates the deterioration. I have used a pseudo workaround for this for years now that allows me to have my cake and eat it too.

Some history: I've had several USB thumbdrives run CORE (and FreeNAS) that have died. I keep a backup of my configs, so it's never a tremendous problem but it is annoying. The most common thing I've seen, is that the boot drive dies, you see a general I/O failure in the logs and some stuff breaks. I've had web interfaces stop working because of a boot drive failure, but the NAS continues to serve files and I didn't know there was an issue. I've had systems lock up and boot back up and work as if there was never a problem. I've had others that never boot back up. Others still where I am able to boot up the system and attempt to mirror the boot drive to a new one, and it died in the process.

My solution to not using a drive bay or SATA DOM and not use a USB thumb drive? An 11 dollar cable

https://www.amazon.com/StarTech-com-SATA-USB-Cable-USB3S2SAT3CB/dp/B00HJZJI84/ref=sr_1_3?keywords=usb+to+sata+adapter&qid=1681663731&sr=8-3

And a cheap 120GB SSD You can even install two and mirror them.
Never had a failure yet, with the exception of someone unplugging it by accident,and then it was just a reboot to fix it.

To your point @sgroesz about performance. A hard drive can absolutely be a boot drive in terms of speed, it's not about the performance though. It's about reliability. NAND can be read from a nearly infinite amount of times, and only wears on reads. Hard drives are constantly spinning at several thousand rotations per minute and wear out with time, regardless of the load on them. The main issue with thumb drives is that they are only designed to be over written a couple of times before they die. Even for a boot drive in TrueNAS, there are enough transactions to kill them in a years time.

 

[WI_Hedgehog]

The problem is that the amount of endurance on the NAND on a USB flash drive is pitiful. By default TrueNAS scrubs (and for good reason) the boot drive every month, and this accelerates the deterioration.

In my estimation a scrub (reading the USB drive) should not harm a USB drive. There are exceptions: high-speed USB on poorly manufactured USB drives using MLC chips without proper heat dissipation could cook the flash memory or on-board controller; re-writing bad memory blocks (which should be a rare occurrence on USB thumb drives) could push a failing drive over the edge.

I would guess the major issue is small writes by TrueNAS to MLC flash is wearing out the flash on low-end drives.

https://blogs.oracle.com/oracle-systems/post/disk-scrub-why-and-when

zpool-scrub.8 — OpenZFS documentation

openzfs.github.io

I've had several USB thumbdrives run CORE (and FreeNAS) that have died. I keep a backup of my configs, so it's never a tremendous problem but it is annoying. The most common thing I've seen, is that the boot drive dies, you see a general I/O failure in the logs and some stuff breaks. .... To your point @sgroesz about performance. A hard drive can absolutely be a boot drive in terms of speed, it's not about the performance though. It's about reliability. NAND can be read from a nearly infinite amount of times, and only wears on reads. Hard drives are constantly spinning at several thousand rotations per minute and wear out with time, regardless of the load on them. The main issue with thumb drives is that they are only designed to be over written a couple of times before they die. Even for a boot drive in TrueNAS, there are enough transactions to kill them in a years time.

I would think a better option for USB boot reliability is to use a USB expander and triple-mirror setup, which is better with your USB-to-SATA cable. "Better," not necessarily "great," and if hardware is that constraining TrueNAS is probably not the best choice of OS, Ubuntu Server & LVM is much more flexible in that regard, Gentoo moreso.

My reasoning is TrueNAS has non-trivial overhead in protecting data which is fine for datacenters, but if using not-ultimately-reliable consumer hardware such as USB thumb drives it's better to eliminate that overhead and use something suited to better working with said hardware.