Possible Boot Pool failure following upgrade?

[NumberSix]

Hi
Following a tricky upgrade to TRUENAS 12.0-U8 yesterday, which finally appeared to be resolved, I left the system running overnight in case more gremlins appeared. Sure enough, today the system sent me two emails containing the following;
==========
New alert:
* Boot pool status is DEGRADED: One or more devices has experienced an error resulting in data corruption. Applications may be affected..

The following alert has been cleared:
* Boot pool status is ONLINE: One or more devices has experienced an error resulting in data corruption. Applications may be affected..

Current alerts:
* Boot pool status is DEGRADED: One or more devices has experienced an error resulting in data corruption. Applications may be affected..
==========

My boot pool is a USB 'thumb drive'. I've never experienced the failure of a thumb drive under TRUENAS before so I have tried to remedy this myself, with no success. I found the closest-to-relevant tutorial available on Youtube, which, notwithstanding it's for 'FreeNAS' I was able to follow the principals outlined here: https://www.youtube.com/watch?v=JoCeOjDNAUk However, once I got to the section at around 3:24 where he nominates which new USB is to become the new boot pool, things went very sideways. I received the following error:

CallError​

[EZFS_BADDEV] /dev/da1p2 is busy, or device removal is in progress

That was followed by a couple of thousand words worth of detail, which I wont post here since I suspect those more knowledgeable than I will likely already know where I went wrong. So, bereft of any other good ideas, I closed down, rebooted, and tried again. This time I got a new error at the same point:

CallError​

[EFAULT] newfs_msdos -F 16 /dev/da1p1 failed: newfs_msdos: /dev/da1p1: Operation not permitted

So it seems I can still boot from the USB although it is marked as 'Degraded' by the system. Can it be 'Scrubbed' and made healthy again? A long shot I know. How can I make myself a copy, or a brand new 12.0-U8 boot USB, before it's too late?

Any assistance would be wonderful please!


Thank you.
PS, yes, after this, I WILL be asking someone how to move from USB to SSD boot (given the way my system is already set up and needing to protect the existing data pools above all else), but for immediate security, I'd be happy just to fix this Degraded USB issue.

 

[Alecmascot]

Have you a copy of your saved config ?
If not get one now if possible.
Shut down the server and boot from an pre-prepared install usb and using a new boot device, install 12.0-U8 on it and boot from that and restore your config.
Be careful NOT to install the system on a DATA drive !!!

 

[winnielinnie]

My boot pool is a USB 'thumb drive'.

This is the issue. It’s discouraged to rely on a USB stick to house your OS (or even logs and System Dataset).

An alternative is to use an SSD or NVMe within a USB enclosure / adapter. This way you can leave internal ports freed up, but still enjoy the benefits of a more reslient storage media for your boot, logs, and System Dataset.

 

[NumberSix]

Have you a copy of your saved config ?
If not get one now if possible.
Shut down the server and boot from an pre-prepared install usb and using a new boot device, install 12.0-U8 on it and boot from that and restore your config.
Be careful NOT to install the system on a DATA drive !!!

Hi
Thank you yes, I saved a config file!
I was hesitating on the brink of doing as you suggest because the chief menu item says it will 'Install TruNAS'. Having an install already, I feared it might go ahead and write to my Data drives/pools and that would kill me. To explain, here's my layout currently:

* HDD (2, mirrored HDDs) - various Data Pools
* SSD - System Pool
* USB drive - Boot Pool

So, since the fresh instal .iso that I'd put onto another USB didn't explicitly say 'Create Boot Pool USB' or the like, I shied away from it - erring on the side of caution, of course. Are you saying that if I go down that route, I'll get an option to create a boot USB, without doing anything destructive to my Data pools along the way?

And by extension, if I go along with that dialogue, will I get an option to install the Boot Pool onto my existing SSD - and if so, what about it being the current system device? Will I get options to add both Boot and System to the SSD?

Thank you.

 

[NumberSix]

This is the issue. It’s discouraged to rely on a USB stick to house your OS (or even logs and System Dataset).

An alternative is to use an SSD or NVMe within a USB enclosure / adapter. This way you can leave internal ports freed up, but still enjoy the benefits of a more reslient storage media for your boot, logs, and System Dataset.

Hi Winnie
I've touched on this question in my reply to Alecmascot (above) but I welcome your input too. Will I be able to create both System and Boot pools on my SSD, from the TrueNAS.iso administered via a fresh USB, and can I do that despite my existing install and not loose any Data, Jails etc?

Yes I have a copy of Config. Will I definitely need to re-apply it if I do the change to SSD, or is that just in case something breaks in the process?

On tenterhooks about this to be honest, but I realise I should make this change.

Thank you.

 

[winnielinnie]

Will I be able to create both System and Boot pools on my SSD

Yes. You should be able to boot from a USB (of which you burned the ISO to make a bootable installation media).

During installation, you select your SSD to install the OS (i.e, TrueNAS), and if it's large enough, you can let it create a 16GB swap on the SSD boot drive.

After successfully rebooting into the freshly installed system, you can import your config and data pool, and then go to System > System Dataset, to tell it to place the System Dataset and logs on your boot pool (i.e, your SSD).

UPDATE: The installer should be smart enough not to offer to install TrueNAS to an existing data pool drive. But if you are unsure or worried, you can unplug the data pool drives during installation to prevent any potential accident.

 

[NumberSix]

Hi
Thank you for this.
I know where to go to import my config, but where do I go to import the all important Data Pool?

Thank you for your thoughts.

 

[Alecmascot]

Hi
Thank you for this.
I know where to go to import my config, but where do I go to import the all important Data Pool?

Thank you.

Storage - Pools - Add

 

[NumberSix]

Yes. You should be able to boot from a USB (of which you burned the ISO to make a bootable installation media).

During installation, you select your SSD to install the OS (i.e, TrueNAS), and if it's large enough, you can let it create a 16GB swap on the SSD boot drive.

After successfully rebooting into the freshly installed system, you can import your config and data pool, and then go to System > System Dataset, to tell it to place the System Dataset and logs on your boot pool (i.e, your SSD).

And a related question. My NAS motherboard is nothing special and relatively old. Of course I'll need to change the Bios/UEFI boot order, but taking a peep just now, I see nothing that hints at an SSD as a boot option. Here's a snap - what choice might it be?

Thank you!!

 

[Alecmascot]

I cannot see your attachment....
I assume the SSD is attached via USB so it is a USB boot.

 

[winnielinnie]

Is that a photo from before or after you installed TrueNAS to the SSD?

 

[NumberSix]

Is that a photo from before or after you installed TrueNAS to the SSD?

Photo from Before the install - but this just became academic.

As I write this - the Gods are smiling on me! I finished the (re)install of TrueNAS as described, and the PC rebooted itself - into TrueNAS, - via it's SSD, all without me doing anything to the Bios/UEFI!! I think it must have been already configured to be the fail-over boot source if no USB stick were present. Lucky day!

And then - the fickle Gods change their minds again.

* Re-importing CONFIG went seamlessly.
* Telling the System data set to use the boot-pool seemed to go ok too.
*However after that - not so much. I was trying to do Storage/Pools, and it resisted letting me add anything. Then I get to this screen:

(An aside - I see it's become aware of my Data pools, even though I never got to complete the 'Add' dialogue).

The real problem here is this "System OFFLINE" message. I've told System it's with the boot-pool, so why is it saying it's offline I wonder??

Note, I called the system 'System' back when it was the only thing on my SSD. I'm guessing that having over-written the SSD in the fresh install, and told it to have the boot and swap and logs there too, has made this section of the GUI unable to recognise what was formerly just my 'System Pool' anymore? The options when I click on the big red button are:

Could it be simply that I hit 'Delete configuration of shares that use this pool' to remove this anomalous entry? Obviously I tend to avoid hitting buttons marked 'Delete' where all my data is concerned!

What should I do?

Please help!

Thank you.
P.S. Afterthought; 'System' was previously a data set and would be listed in the 2nd screenshot above, but no longer, because I've moved it to the boot pool. Doesn't explain why it just hasn't silently de-listed itself here, or re-listed itself as the new System' pool though.

 

[winnielinnie]

Photo from Before the install - but this just became academic.

Because your system is using UEFI. Before you installed the TrueNAS OS to the SSD, there was no EFI partition / EFI bootloaders, and hence your motherboard has nothing to list in the bootup menu. Unlike the traditional legacy/MBR boot method, UEFI boot menu will not list, willy nilly, all attached disks. It will scan each disk for an EFI partition and search for EFI bootloaders.

Could it be simply that I hit 'Delete configuration of shares that use this pool' to remove this anomalous entry? Obviously I tend to avoid hitting buttons marked 'Delete' where all my data is concerned!

To be clear, this SSD used to be a dedicated pool for the System Dataset? But now this very same SSD is what you're using as your boot-pool, which also now houses the System Dataset and syslogs?

What likely happened is that when you re-imported your config, it automatically imported the pools you were using at the time you backed up your config file: NAS and System. However, since the vdevs for System are missing (obviously), it errors out when trying to automatically re-import the defunct System pool.

You should be able to safely disconnect/export it, as it no longer exists anyways. (Including tasks / shares that rely on this obsolete, nonexistent pool.)

I suppose it would have been "more proper" to first disconnect/export the no-longer-needed System pool prior to the upgrade/install and config import.

 

[NumberSix]

Because your system is using UEFI. Before you installed the TrueNAS OS to the SSD, there was no EFI partition / EFI bootloaders, and hence your motherboard has nothing to list in the bootup menu. Unlike the traditional legacy/MBR boot method, UEFI boot menu will not list, willy nilly, all attached disks. It will scan each disk for an EFI partition and search for EFI bootloaders.

Thank you for explaining this slightly off-topic detail. So, even though the SSD was previously home to the System pool, because that pool was non-bootable, it wasn't showing up in the UEFI? I think that's what you're saying. So that raises a question for me (academic now, but one day it won't be), if you inserted a brand new SSD that would of course have no bootable characteristics, and, if you wanted to tell the UEFI you plan to boot from it, and it was invisible because of the no boot characteristics yet - how would one ever escape that catch-22 and create a bootable SSD from it? Are you saying you'd always need to write something bootable to it first - as we have in this case - and only then will it appear in the UEFI list?

To be clear, this SSD used to be a dedicated pool for the System Dataset? But now this very same SSD is what you're using as your boot-pool, which also now houses the System Dataset and syslogs?

Yes, that's absolutely correct - thank you for determining that so exactly.

What likely happened is that when you re-imported your config, it automatically imported the pools you were using at the time you backed up your config file: NAS and System. However, since the vdevs for System are missing (obviously), it errors out when trying to automatically re-import the defunct System pool.

You should be able to safely disconnect/export it, as it no longer exists anyways. (Including tasks / shares that rely on this obsolete, nonexistent pool.)

I think events bear out that explanation.
I did as you suggested and it has successfully removed the defunct System pool, with no immediately apparent ill effects. Thank you!

One final question I think. I seem to recall seeing boot and system pools listed under Storage/Pools, but now I don't see either. This is a little worrying. For one thing, I used to set up an automatic backup of the system pool to a data set on one of the HDDs. If I can't see the Boot Pool (as it is now, combined) then I can't target it as an entity to be copied to the HDD periodically. Or is it folly to back it up, given that I'd need to reinstall anyway if/when it goes corrupt? And should I need to see the Boot Pool listed under 'Pools' for any other reason anyway - like setting a periodic scrub task for example? Right now the new Boot pool seems inaccessible to any of the usual 'data care/management' operations.

Thanks!

 

[winnielinnie]

if you inserted a brand new SSD that would of course have no bootable characteristics, and, if you wanted to tell the UEFI you plan to boot from it, and it was invisible because of the no boot characteristics yet - how would one ever escape that catch-22 and create a bootable SSD from it?

It becomes bootable (EFI bootloader created within) when you install something to it. There's no catch-22, because you're using another bootable media, such as an ISO, USB stick, CD, network source, etc, to install to the new, fresh SSD. (Think about your USB stick. You used the TrueNAS ISO and made the USB stick a bootable device, whose sole purpose is to install TrueNAS to another device.)

Or is it folly to back it up, given that I'd need to reinstall anyway if/when it goes corrupt?

This. Why do you need to make regular backups of your System Dataset and boot-pool? You can always re-install and re-import your config. (Make sure to often backup your config file.) Besides, TrueNAS automatically saves a boot environment of the previous version any time you upgrade the system from within the GUI.

And should I need to see the Boot Pool listed under 'Pools' for any other reason anyway - like setting a periodic scrub task for example?

It's not displayed for a reason. It's exclusively used to house the OS (and logs / System Dataset, if so desired.) It's not user data. Any such management of the boot-pool is done through System -> Boot, and nowhere else.

 

[NumberSix]

Thank you! That was a full and complete account of everything I wanted to know. I am really immensely grateful to you for your thoughts on this.

Thank you Winnie!

 

[NumberSix]

Sorry about this late catch, but I've just realised that both of my Plugins have vanished. I had Syncthing (which took an age to set up) and Grafana (which took seven ages of man to setup) running fine, and now both are absent. I went to 'Plugins' and it asked me for a Pool to create plugins in - offering me an option of one - the only pool I have, 'NAS'. Creating it didn't fix anything, and neither plugin appeared.

The Syncthing data set is still appears as listed in my 'NAS' (one and only) pool. Will I get that back at least? Grafana never required/generated it's own data set.

Please tell me I haven't lost everything there, despite saving a Config file?!!

Thank you Winnie.

 

[winnielinnie]

Did you intentionally place the iocage dataset on your makeshift "System" pool previously, rather than leave it on the data pool?

 

[NumberSix]

My knowledge and understanding of TrueNAS isn't sufficient for me to knowingly alter any default behavior of the iocage (or any other system element), so it was wherever it would have been by default. I guess that's why I see 'Syncthing' as it's own dataset in my Data Pool - it must have wanted to be there by default. As I said previously, to my best recollection, I don't think Grafana ever had/needed it's own Dataset.

 

[winnielinnie]

Plugins and Jails require the "iocage" dataset to reside. Upon creation of your first Plugin or Jail, it will prompt you where you want to hold the iocage dataset.

My hunch tells you you chose (or left it at) "System". Because for all intents and purposes, the dedicated "System" pool you created is a data pool, according to TrueNAS. (In your setup, "System" is just the name you gave it, but from TrueNAS's perspective it's "data pool number two".)

TrueNAS doesn't know your intention is to only use it for the System Dataset and syslogs.