Disk gone from pool after upgrade

[mtommy84]

Dear all.
I just perform an upgrade on my truenas scale to TrueNAS-SCALE-23.10.1.3 (previously .1.1).
After the upgrade, a pool is not available. System tells that no VDEVs have been assigned and in fact there is no dev in the Pool. (attach images) .

In fact, there is an unassigned disk in the storage section.

If I try to add it, from GUI, system tells me that Name not added/ al least 1 dada VDEV is required, despite having added the right pool name and disk.

If I run zpool status, the pool does not exist anymore

if I run zpool import I get

Code:

no pools available to import

if I run zpool list I get

Code:

NAME          SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
Main Pool     928G   306G   622G        -         -    10%    32%  1.00x    ONLINE  /mnt
TimeMachine   230G   152G  78.4G        -         -    47%    65%  1.00x    ONLINE  /mnt
boot-pool     216G  8.75G   207G        -       16G     0%     4%  1.00x    ONLINE  -

Secondary Pool has disappeard. Also in the syslog I see no indication upon booting.
Any help?

 

[mtommy84]

Update:
I tried to

Code:

zpool import -d /dev/nvme3n1

, where nvme3n1 is the disk assigned to the pool, but output is

Code:

no pools available to import

 

[joeschmuck]

zpool import -d /dev/nvme3n1

Why this command?

My advice is:
1) Import your previous configuration file that you hopefully have from before your "upgrade".
2) If step 1 fails, roll back to your previous version and import your previous configuration file. Since you upgraded from SCALE to SCALE, you may be able to just select the previous boot environment, this would be the easiest thing.

One thing, very important, Do Not Upgrade the ZFS Feature Set, this will prevent you from rolling back.

 

[mtommy84]

I have the backup done. Thanks for the input.
Will try and report here.

 

[mtommy84]

Why this command?

My advice is:
1) Import your previous configuration file that you hopefully have from before your "upgrade".
2) If step 1 fails, roll back to your previous version and import your previous configuration file. Since you upgraded from SCALE to SCALE, you may be able to just select the previous boot environment, this would be the easiest thing.

One thing, very important, Do Not Upgrade the ZFS Feature Set, this will prevent you from rolling back.

Option 1 did not work.

How do I perform a rollback? Do I need to download the image and perform a manual installation?

 

[mtommy84]

Option 1 did not work.

How do I perform a rollback? Do I need to download the image and perform a manual installation?

NVM found it.
rollback in process

 

[mtommy84]

Ok rollback to previous version and reupload of the config was unsuccessfull. Still disk unassigned, pool only visible in the GUI and not via zpool commands.

 

[mtommy84]

Ok there is clearly an issue/bug with the upgrade process.

on .1.1 I destroied the visible pool and created a new one from scratch, binding the nvme disk.
I then proceded to change the boot to .1.3 and now I am in the previous situation where the disk has been taken out from the pool and the pool is just a ghost.

Is there a way to debug this properly?

 

[joeschmuck]

The rollback is in the GUI. It is not reinstalling anything. You just needed to select the previous boot environment and reboot. It should bring you right back to before you tried the upgrade.

 

[mtommy84]

The rollback is in the GUI. It is not reinstalling anything. You just needed to select the previous boot environment and reboot. It should bring you right back to before you tried the upgrade.

I did exactly that and the situation remained the same after the upgrade. I then recreated the pool from zero and upgraded again. Problem reapperad exatly as the first time.

 

[joeschmuck]

I have seen a few (two) reported issues with SCALE upgrades with respect to the pools. Your problem is a bit different. I highly recommend you submit a trouble ticket and include the debug log, they will ask you for it anyway.

SCALE sure is not going as well as it should. It reminds me of a FreeNAS version which shall not be named that was just a piece of crap. It was rushed out the door and people paid the pain for it.

I'm glad you have a backup of your data, sorry to hear about the issues.

Question: For the upgrade, did you upgrade using the GUI upgrade process or did you download the update image and upgrade that way? Or maybe you did both. The fact that you can repeat it is a good thing, this means hopefully the developers will be able to recreate it as well.

 

[NugentS]

@mtommy84 can you post your hardware please - so we have some idea of what we are dealing with here

 

[mtommy84]

So the luck is that is had impact only on a, let's say, test pool, running only 1 drive. Fortunately the main pool did not have any problem. I anyway have an extranal additional backup + an S3 glacier bucket :)

I did run the upgrade using the GUI and clicking on "install pending updates". I haven't tried loading manually.

It will be quite scary to run an upgrade in the future.

Could you link me where I can open an official ticket?

Thank you!

 

[joeschmuck]

Could you link me where I can open an official ticket?

at the top of each forum page, look up.

 

[Jailer]

SCALE sure is not going as well as it should.

I've been thinking the same thing. We sure have had a lot of posts lately of Scale users pools going "poof" for seemingly innocuous reasons.

 

[mtommy84]

I've been thinking the same thing. We sure have had a lot of posts lately of Scale users pools going "poof" for seemingly innocuous reasons.

the scare bit here is that there is no existance of that excpect that in the GUI

 

[mtommy84]

@mtommy84 can you post your hardware please - so we have some idea of what we are dealing with here

Sure, here they are
CPU: Ryzen 4300G
Motherboard: MSI B450 Pro
Ram: 16GB
Boot Disk: Kingston NV2 250GB
Main Pool Disks: WD Red SN700 1TB
Secondary Pool: Verbatim 7000G 4 TB

 

[joeschmuck]

SMART Testing is not currently supported in TrueNAS CORE or SCALE for NVMe drives, but will be I'm sure. You can run the test and check the results in SCALE using these commands (Not all NVMe drives support SMART testing, especially the older ones). This is some of what my Multi-Report script does.

NOTE: 'nvme0' is the name of your NVMe, example: nvme id-ctrl /dev/nvme0 -H | grep -i "self-test"
You are looking for something like:

[4:4] : 0x1 Device Self-test Supported

Does your NVMe Support SMART Self-tests?
nvme id-ctrl /dev/nvme0 -H | grep -i "self-test"

SHORT Test (Should see Short Device self-test started)
nvme device-self-test /dev/nvme0 -s 1

LONG Test (Should see Long Device self-test started)
nvme device-self-test /dev/nvme0 -s 2

Results (0=PASS, anything else is probably not good)
nvme self-test-log --output-format=json /dev/nvme0 | grep -m 1 "Self test result" | cut -d ':' -f2 | cut -d ',' -f1

Results (Last self-test type run: 1=Short, 2=Long)
nvme self-test-log --output-format=json /dev/nvme0 | grep -m 1 "Self test code" | cut -d ':' -f2 | cut -d ',' -f1

This may help you at least feel a little better about your NVMe drives.

 

[mtommy84]

All disk came back clean

Code:

"List of Valid Reports":[
    {
      "Self test result":0,
      "Self test code":2,

 

[mtommy84]

Ticket has been opened

Log in with Atlassian account

Log in to Jira, Confluence, and all other Atlassian Cloud products here. Not an Atlassian user? Sign up for free.

ixsystems.atlassian.net