11.3 u1 -> 11.3 u2 upgrade terribly slow?

[David Dyer-Bennet]

It's been sitting for over 45 minutes at 62.5% on step 1/4, "installing base-OS".

I seem to remember the last few upgrades being unreasonably slow, but not this bad!

I triggered this upgrade from the GUI by clicking the "apply pending update" button.

 

[DarrylM]

I experienced the same phenomenon. The first of the four steps took a very long time (about an hour). The second through fourth steps went much quicker. I am experiencing another problem, however, and have a system that will not boot. Guidance on the thread suggests that I may have a BIOS problem, which I believe may be true. I'm trying to figure that out now.

 

[DarrylM]

Follow-on to my earlier reply: I just realized that you and I running same or similar Mobos and CPUs. I definitely want to know how your upgrade goes and which level of BIOS you are running! :)

 

[David Dyer-Bennet]

Hey, progress! It's now over 90 minutes, but I'm at 100% on step 4/4 (but still showing the progress box in the GUI). But it's not looking absolutely stuck any more.

DarrylM, I'll check BIOS version for you; I think I'll need to force a boot for that, but I can do that, if it does get finished (which now looks possible!).

That's a camera in your profile picture, I see; that's why *I* need that much storage space (in addition to my own photos it's currently housing an archive of photos for the local science-fiction club including the full collection of one member who died and left us his photos).

 

[DarrylM]

@David Dyer-Bennet - I've got lots of archived photos and videos. But most of them aren't yet on the NAS, still in separate hdds. That's the next step. The NAS currently backs up my working papers (Windows and Apple). Also stores security cam footage (NFS). FreeNAS (when working) allows me to process security cam files on a Windows machine running NFS. Picks em up from the FreeNAS share.

As soon as things stabilize, my plan is to next backup my photos and videos. Right now, I've just had a mini-breakthrough and system is up. But I think its just temporary and I don't server is working right yet - took over 20 mins to boot and I still got some odd messages.

When you do check the BIOS, I'd be interested in settings used - particularly for SATA configuration. After reading guidance from @SweetAndLow I'm reading about BIOS settings and quickly getting into the weeds on CMOS reset and other things I didn't know existed. Maybe there is an easy way to export BIOS settings ... I don't know.

 

[David Dyer-Bennet]

Okay, so it did complete the upgrade and come back up.
It was starting at 15:00, when I went out for a walk. So far as I can tell from the logs, it completed the upgrade and reboot and was back in service at 17:45.
That's almost 3 hours!!!!!!!!
I trust that's not ordinary performance? That's a LONG time to have to have a server out of service for a routine software upgrade!
Any ideas? What kind of thing should I look for in the logs, say?

 

[Apollo]

Okay, so it did complete the upgrade and come back up.
It was starting at 15:00, when I went out for a walk. So far as I can tell from the logs, it completed the upgrade and reboot and was back in service at 17:45.
That's almost 3 hours!!!!!!!!
I trust that's not ordinary performance? That's a LONG time to have to have a server out of service for a routine software upgrade!
Any ideas? What kind of thing should I look for in the logs, say?

I remember my first attempt to updating to earlier 11.3 release, I had to wait a singnificant amount of time and pretty much in the same upgrade path as you.
I believe Freenas is updating/upgrading the underlying iocage jails and most likely download all the packages.
I think the download of the Freenas update package is only the visible portion of the update.

 

[DarrylM]

Could well explain the long installation time, doing package updates, iocage, etc.

I am still experiencing extremely long (25 minute) boot ups. Initial portion of boot seems normal, but then hangs after displaying the message "middlewared: loaded plugin middlewared.plugins.cache. " It takes at least 20 minutes after that point. Any ideas on what might be causing that?

 

[Apollo]

Could well explain the long installation time, doing package updates, iocage, etc.

I am still experiencing extremely long (25 minute) boot ups. Initial portion of boot seems normal, but then hangs after displaying the message "middlewared: loaded plugin middlewared.plugins.cache. " It takes at least 20 minutes after that point. Any ideas on what might be causing that?

Early version of 11.3 would take lnger to start because of the iocage jail issues with snapshots, I believe. This is no longer the case.
However, you system indicates RAM: 8GB which is pretty much the minimum Freenas (ZFS) need to run. ARC and such are not going to be stored in RAM and iocage jails may have limited resources. I doubt this is the issue though.

During the update/upgrade process, and impending reboots, the system will/may take longer to start.
Once system is finally operational, it shouldn't take too much time for the system to be up and running.

Until all the iocage jails are up and running, the network/bridge connection could be lagging behind if there are some conflicting jails causing the network to crash and be reinitialized. Just hypothesis on my part.

The more obvious source of the issue could be your boot disk. Are you on USB or SSD?

 

[DarrylM]

Thanks for quick reply. I may have co-morbid factors here, not sure. I am presently booting from 32 Gb USB. I had tried installing to brand new HP 500 SSD but could not get it to boot. In another thread, my only other post), it was suggested that I had a BIOS problem. I flashed the latest for my board and that did not resolve seeing the SSD. So back to the USB I went.

In the interim, I've had several SMART messages on one of my 3TB drives, so just pulled and replaced it and initiated the resilvering process. Praying that, too, isn't slowed down (else it will not finish). Currently in the Scanning phase, with ETA extending, not decreasing (saying 130 seconds, but similar status 10 minutes ago).

I had not been able to boot from the USB for several days. Only thing I did different was change the BIOS settings from IDE to AHCI. I honestly don't think this has much to do with the issue (I could be wrong), but it did allow reliable albeit extremely slow reboots When system does boot, normal operation is observed, access from web browser and from Windows file share all seem to be normal. No indication of high CPU, memory or IO. But I agree with you that having more RAM would be beneficial and booting off the SSD would be preferable. I hope to accomplish both of those soon - after the resilvering completes.

 

[DarrylM]

By the way I don't run any VMs, jails or plugins on this server.

 

[David Dyer-Bennet]

Yeah, I was wondering about "iocage jails". These are a regular part of the system? That is, they aren't present only if I've installed jails through the GUI? Because I haven't; I have on VM (which doesn't start on boot, so it can't explain too much of the slowness; and it's just a ubuntu virtual to give me a place to work).

Dunno what the IxSystems server can support, but I've got gigabit fiber into the house (Ookla speedtest routinely shows download throughput at 900+ Mbits/sec, often 950). So unless it's server-side, the whole install can't be very many seconds of download; maybe a few minutes if the server is slow.

Any suggestions about log things to look for? And, I'll try to get a reboot in tonight or tomorrow morning; checking the Bios version for DarrylM, plus timing a clean reboot, when an upgrade hasn't been made.

 

[Apollo]

Thanks for quick reply. I may have co-morbid factors here, not sure. I am presently booting from 32 Gb USB. I had tried installing to brand new HP 500 SSD but could not get it to boot. In another thread, my only other post), it was suggested that I had a BIOS problem. I flashed the latest for my board and that did not resolve seeing the SSD. So back to the USB I went.

In the interim, I've had several SMART messages on one of my 3TB drives, so just pulled and replaced it and initiated the resilvering process. Praying that, too, isn't slowed down (else it will not finish). Currently in the Scanning phase, with ETA extending, not decreasing (saying 130 seconds, but similar status 10 minutes ago).

I had not been able to boot from the USB for several days. Only thing I did different was change the BIOS settings from IDE to AHCI. I honestly don't think this has much to do with the issue (I could be wrong), but it did allow reliable albeit extremely slow reboots When system does boot, normal operation is observed, access from web browser and from Windows file share all seem to be normal. No indication of high CPU, memory or IO. But I agree with you that having more RAM would be beneficial and booting off the SSD would be preferable. I hope to accomplish both of those soon - after the resilvering completes.

Good thing to postpone and test until resilvering is complete.
AHCI is the way to go, but IDE shouldn't be an issue.
You do need to boot with UEFI and this is asked upon install from the ISO file, if not you can decide to boot non-UEFI and set the BIOS to Legacy (it may be refered to a different acronym).

If you can boot UEFI from the USB key, then BIOS compatibility isn't really the problem.
In the BIOS, you should be able to set the boot device to HDD or SSD and under the HDD/SSD section point to the HP 500 SSD.

If this still doesn't work, it is possible boot option (INT 13) or similar is disabled on external RAID/HBA and will not allow booting from the card.
If the HP 500 SSD was already formatted somehow, you could try the following once you have completed resilvering.

You can boot from USB. and you don't really need to have your existing pool plugged in, to check everything. With HP 500 SSD installed, you can wipe it's content through the Disk Management in GUI and then create a volume on the SSD to see if ZFS is working (just to eliminate SSD controller issues).
Then you can detach the HP 500 SSD and wipe its content again.
Install Freenas on it using a different USB key flashed with the ISO image.
Install and make sure you select booting with UEFI. See if you are able to boot and make changes to BIOS settings if you don't.

If all fails, try installing on HP 500 SSD using the non UEFI solution and set BIOS to non UEFI and see how it goes.

 

[Apollo]

Yeah, I was wondering about "iocage jails". These are a regular part of the system? That is, they aren't present only if I've installed jails through the GUI? Because I haven't; I have on VM (which doesn't start on boot, so it can't explain too much of the slowness; and it's just a ubuntu virtual to give me a place to work).

Dunno what the IxSystems server can support, but I've got gigabit fiber into the house (Ookla speedtest routinely shows download throughput at 900+ Mbits/sec, often 950). So unless it's server-side, the whole install can't be very many seconds of download; maybe a few minutes if the server is slow.

Any suggestions about log things to look for? And, I'll try to get a reboot in tonight or tomorrow morning; checking the Bios version for DarrylM, plus timing a clean reboot, when an upgrade hasn't been made.

From what I have seen so far (on my end), the middleware process during upgrade seems to only affect the Iocage jails that are installed on the pool, and the iocage jails are not being updated from IxSystem, but from whatever ports they are based on, FreeBSD Git or the like, and usually do not have high throughput.
I think, if you don't have any iocage jails nor plugins, then there shouldn't be any update to the iocage jails system.

 

[DarrylM]

Good explanation and suggestions; very much appreciated! I am currently booting BIOS and Legacy; did so after watching the YouTube installation video recommending same. But, I can definitely give UEFI a try though! I didn't save my BIOS settings before flashing to latest rev, so I have no idea how it used to be set. But I am going to do some more experimenting tomorrow and also try adding additional RAM. Good news, resilvering finished quickly and pool appears to be healthy again. Initiated a Scrub and will do SMART tests in the morning. Thanks again!

 

[DarrylM]

Update on my situation. There are no more SMART alerts. I am reliably able to boot, power down and reboot 11-3. It is still taking an exorbitantly long time to do so.

I installed an additional 8Gb RAM today (bringing server total to 16Gb). There is no improvement in boot time or change in messaging. The system still requires over 20 minutes to boot (whereas before it took only a 2). And the first login to the Web UI and painting of the dashboard is also proportionally slow. Once that is over, the system hums just fine and is as responsive as ever, which is a great thing.

I'll try the UEFI boot option next, and then booting from the SSD device. Hope one or both of those helps!

 

[David Dyer-Bennet]

Okay, got a reboot in this morning finally. Actually a cold boot; from a cold start, it took 17.5 minutes for the server to be up for ssh connects, gui connects, and serving files (they seemed to happen at about the same time, to the limits of very casual measurement).

So I feel like that's amazingly slow. But I feel like FreeNAS has *always* been hugely slow to boot. I don't boot it much, and ran Solaris and original ZFS for quite a few years before moving to FreeNAS, so my impressions may be confused (I'm bad at attaching memories to points in time usually).

I'm booting from a thumb drive; I generally don't have any spare SATA connectors (true in this system; 4+2 pool using all 6 SATA interfaces) so changing that to something else is not likely any time soon. If it's a normal consequence of USB boot maybe it should be made clear in the manual that booting will be slow? But that's jumping ahead, we don't know it's not some local problem of mine.

And here's the BIOS version info



Nothing to SATA configuration, just the list of drives. I'm not doing any over-clocking or anything scary like that :) .

 

[Apollo]

Okay, got a reboot in this morning finally. Actually a cold boot; from a cold start, it took 17.5 minutes for the server to be up for ssh connects, gui connects, and serving files (they seemed to happen at about the same time, to the limits of very casual measurement).

So I feel like that's amazingly slow. But I feel like FreeNAS has *always* been hugely slow to boot. I don't boot it much, and ran Solaris and original ZFS for quite a few years before moving to FreeNAS, so my impressions may be confused (I'm bad at attaching memories to points in time usually).

I'm booting from a thumb drive; I generally don't have any spare SATA connectors (true in this system; 4+2 pool using all 6 SATA interfaces) so changing that to something else is not likely any time soon. If it's a normal consequence of USB boot maybe it should be made clear in the manual that booting will be slow? But that's jumping ahead, we don't know it's not some local problem of mine.

And here's the BIOS version info

View attachment 38147

Nothing to SATA configuration, just the list of drives. I'm not doing any over-clocking or anything scary like that :) .

It is slow.
I would look at screen output when system is booting to see the progress. If you are trying to boot via PXE, it will add delays.
You can check how much time Freenas takes when importing the pool, it can take some time there too.
If you have jails and VM's it will cause the network to drop until all the jails are up and running.
Usually when section about middleware is reached or completed, the web GUI should be available (some time before you get the Freenas Console menu.

If you can break the boot process into sections until Freenas becomes responsive, it should give you a clue to what the problem could be.
Another thought, with Freenas 11.3, I believe using USB3 sticks could prevent booting. Booting with USB2 will work but at a lower loading speed.

 

[David Dyer-Bennet]

No PXE, straight local boot from thumb drive. Not actually sure if it's 2 or 3 -- it works, so if it's 3, 3 is working :) .

I remember a lot more info going by on the console than I find in the log; I was hoping I was going to be able to go back through the log (/var/log/messages) and break down the times for the various stages, but that doesn't seem to get logged.

Also--something weird with the timestamps. I booted starting at 10am sharp, and the first log entry, noting syslog comming up, is timestamped 10:13am. Not sure if the log entries are entirely missing, or what. Dmesg reports somewhat overlapping entries but without timestamps. I guess maybe taking video of the boot with my phone is the way to capture something here? Sheesh!

 

[Apollo]

If you had iocage Jails, VM or pluggins running, the boot up time could be accounted for as there were issues with iocage jail snapshots causing system bootup to be in the 15-20 minutes boot time range. And this is on SSD with a decent system.