Terribly slow boot from USB thumb drive

[David Dyer-Bennet]

FreeNAS has annoyed me for years with how slowly it boots in the default single USB thumb drive configuration. (I'm using all 6 disk controller ports on data drives; this is a small home server, a mere 36 TB raw, run on a severe budget. So adding an SSD disk to boot from is physically impossible currently.)

By "terribly slow", I mean it takes well in excess of half an hour to do a restart. Power fail recovery, software update, I can't avoid restarting forever. (Yeah, system is on a UPS, but sometimes the power outages exceed the UPS capacity. Does mean the restarts are from a clean shutdown, though.)

That means, among other things, that if I were in a corporate environment where people monitored availability with automated stopwatches, I'd be down several 9s from where I otherwise should be :-( .

Is this an inherent FreeNAS problem, or a problem of the thumb drive boot, or something weird in my configuration? Any ideas? It's not exactly urgent, I'm *not* in a corporate environment where we demand "5 nines" of uptime, and I can generally plan my reboot times to avoid personal inconvenience. But still; I can't really believe this is anywhere near what I should expect.

 

[Evertb1]

As far as I know that slow is in no way normal. I must admit that I stopped using USB sticks somewhere around FreeNAS 9.x and FreeNAS 11.3 is from a time that USB sticks where not recommanded anymore. But still your boot should take far less time. I don't think that I ever had a boot time more then 10 minutes or so on my USB sticks. I assume that you already tried a new USB stick?

 

[Jailer]

My buddy recently switched over to a SSD boot drive for this very same reason. I've been trying to convince him to do so for a couple of years. USB boot drives are convenient but really not a great idea in practice.

 

[David Dyer-Bennet]

Interesting, I missed USB boot not being recommended. That was the strongly-recommended standard when I switched over to FreeNAS (when Oracle took away the free Solaris).

As I say, I don't have any free disk controller ports, so there's nothing I can do about that easily. Hmmm; I suppose an add-on card could actually work, and I do have a small pile of SATA SSD drives so I wouldn't have to buy that. An add-on card is not unaffordable on my budget (a whole new server is; at least it's too much to spend just to reduce boot time).

 

[Evertb1]

A couple of years ago I bought a couple of Dell Perc H310's for about 30 dollars a piece (used of course). They are more expensive now, but I can still find them for 50 euro's (60 dollars). I flashed them to IT mode with SLI firmware and use them with pleasure. The only bad thing about them is that they get hot if you don't cool them sufficiently. You should be able to find one or a simular card for an reasonable price. Here in Europe those things are a bit more expensive then in the USA as I can see when I look at the prices on ebay.

 

[elvisimprsntr]

You can always use an external USB enclosure with SSD or spinning rust drive. I used a Crucial X8 M.2 SSD until I replaced the OEM 512MB eUSB DOM with a 16GB SLC eUSB DOM.

 

[David Dyer-Bennet]

You can always use an external USB enclosure with SSD or spinning rust drive. I used a Crucial X8 M.2 SSD until I replaced the OEM 512MB eUSB DOM with a 16GB SLC eUSB DOM.

Sure, but my experience with those is that they're not notably faster than the USB thumb drives I'm using. I guess there are some super-high-end expensive ones that use an M.2 inside that ought to be faster, but my experiences with random USB 3 external hard drives vs. USB thumb drives aren't that different. (Ah, I notice at least one of the names you cite sounds like it has an M.2 in it.)

I could experiment but it would have to be booting actual FreeNAS I guess (since my experience booting Linux from thumb drives is that it's reasonably fast).

I've also been wondering if, if I had an additional disk interface and an SSD drive, whether most of the time it would be better used as a log drive, anyway.

 

[elvisimprsntr]

Sure, but my experience with those is that they're not notably faster than the USB thumb drives I'm using.

It's not about speed for the boot drive, it's about longevity. Consumer USB flash drives just don't survive repeated writes.

The Crucial X8 is a M.2 SSD in a USB3 enclosure. I found out it's QLC flash, which is why I switched to a SLC eUSB DOM for better wear durability. The eUSB DOM is only rated for 34 MB/s read, 22 MB/s write. It doesn't need to be any faster. It's just a boot drive, writing some system logs. A few KB/s every now any then.

 

[David Dyer-Bennet]

It's not about speed for the boot drive, it's about longevity. Consumer USB flash drives just don't survive repeated writes.

My problem, and this thread, is entirely about boot speed. If I were having write longevity problems, I'd be getting completely different symptoms (errors relating to that, rather than a successful but very slow boot).

More than half an hour to boot the server!

 

[elvisimprsntr]

You do know that consumer flash drives degrade in both read and write performance as they wear out?
How long have your current flash drives been in service?
What manufacturer, capacity? What type of flash? SLC, MLC, TLC, QLC, etc.?
My slow eUSB DOM's take less than 2 mins to complete boot up.

 

[David Dyer-Bennet]

You do know that consumer flash drives degrade in both read and write performance as they wear out?

No, actually. I know they stop working after enough writes, but I know nothing of performance degradation of any significance.

 

[NugentS]

I boot my QNAS from USB, takes a minute or so

 

[elvisimprsntr]

No, actually. I know they stop working after enough writes, but I know nothing of performance degradation of any significance.

Have you considered replacing your current USB flash drives, for example industrial SLC flash drives?

 

[David Dyer-Bennet]

Have you considered replacing your current USB flash drives, for example industrial SLC flash drives?

I may well try just copying the boot code onto a new ordinary flash drive; that should, according to your explanation, produce a huge short-term improvement, and then degrade back to the current level, right? If it does, then a longer-term solution is a better flash drive, but if it does not, then there's no point. (Amazon shows very little for SLC flash drives, and many of the examples advertise so hard about being specialized for Tesla dashcams that I'd have to ask if they were usable for general-purpose tasks! Also they cost 10x as much.) (I've got essentially new ordinary flash drives sitting around; so this test costs nothing, whereas buying a new special expensive drive just to test does.)

 

[elvisimprsntr]

It's one possible explanation. I don't think simply copying the existing flash drive to a new one is recommended.
1. You will want to save your current config (exporting secret seed)
2. Backup your existing data to separate NAS or media (RAID is not backup, RAID is resiliency)
3. If you are migrating from a very old FreeNAS version, there may be additional steps. Suggest reading up a bit first.

How To Upgrade FreeNAS to TrueNAS - Lawrence Technology Services

Lawrence SystemsSat, October 24, 2020 6:03pmURL:Embed:Amazon Affiliate Store ➡️ https://www.amazon.com/shop/lawrencesystemspcpickup Gear we used on Kit (affiliate Links) ➡️ https://kit.co/lawrencesystems Try […]

lawrencesystems.com

Reddit - The heart of the internet

www.reddit.com

4. Create a TrueNAS installer flash drive
5. Boot off TrueNAS installer and install on new boot device, preferably a SSD.

 

[Electr0]

There must be something wrong with your USB drive, too old, too slow, too degraded.

I've been booting my server off a SanDisk Ultra Fit 64GB drive for the past six months or so, and it only ever take about 3 minutes until it's up and running and I can access the Web GUI.

I would recommend buying a new USB (3.1) flash drive, move your boot/system files to that, and give it a go.
If you still have a massive wait time, then there must be something broken in your boot process.

As a side note, I was mucking around with some different installations and tried installing TrueNAS to an older USB drive I had laying around. It took forever to install the distro and was super buggy during boot. I bought a brand new USB 3.1 drive and it installed way quick, with zero boot issues.

 

[David Dyer-Bennet]

Do USB 3 drives work now? They didn't used to, you had to avoid them (or at least not put them in USB 3 ports; the current drive is a 3).

I wasn't proposing copying the existing drive, I was saying just put a new boot image on the drive. All the config is on the disks, it should just come up in place.

 

[Evertb1]

I wasn't proposing copying the existing drive, I was saying just put a new boot image on the drive. All the config is on the disks, it should just come up in place.

I am not sure you have the right idea. You should save your config file to a secure place. You can do that with the GUI. Then upload it after a fresh install. I have done that a couple of times when I lost my boot device (and yes, USB sticks). It's an easy and not time consuming proces. By the way: it's not a bad habit to save your config file regularly. And for sure before each system change being it hardware or software.

 

[joeschmuck]

USB 3 drives work fine functionally, the issue is heat and premature death, this is the reason USB 2.x was preferred. But you certainly can use a USB 3.x flash drive.

All the config is on the disks, it should just come up in place.

Yea, taking what you said strickley is incorrect, it won't just come up in place. You will need to backup your configuration file first (it is located on the hard drives but more steps to make it work) then build your boot device to the same version you are currently using (your choice how to build it) and lastly restore your configuration file after you boot up. It should take only a few minutes maximum.

Lastly, I take it that this is your FreeNAS 11.3 machine? I was running that version for a long time, never any hint of a problem like this so I doubt it's FreeNAS causing it.

 

[Evertb1]

You will find the save config and upload config buttons on the System --> General tab of the GUI (Paragraph 7.1 of the manual)