[WORKAROUND] After upgrade to TrueNAS-SCALE-23.10.0.1 pool is missing until I import (problem comes back every boot)

[justryan]

After upgrading to TrueNAS-SCALE-23.10.0.1 I have no pools on boot any more.

I can run this:
zpool import -f -F -R /mnt datatank
and they come back, but then they are gone again on boot.

 

[Kris Moore]

Can we get a bug ticket and debug file to review? There isn't any real information here for us to even begun troubleshooting what the problem could be...

 

[justryan]

Sure! How can I produce the debug file? Can I make the ticket private?

 

[justryan]

System > Advanced and click SAVE DEBUG

Doing it now.

 

[justryan]

[NAS-125144] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

 

[Kris Moore]

Thank you. We'll investigate. In meantime, can you try importing the pool via the UI properly? Does that present any errors?

 

[simdim]

I had an issue that looked similar. Pool would need to be imported every boot. and would not import properly from GUI. The solution was: import it from CLI, go into GUI, export it from GUI, import if rom GUI again. It had been surviving reboots so far.

 

[Kris Moore]

That's what I was hinting at, wondering if the DB entry is just missing or bad, and re-importing might fix it here. Not sure the history of this pool or how it would have gotten into this state. Was it an upgrade from CORE or some other Linux at some point?

 

[justryan]

It was originally an upgrade from CORE, maybe a year ago. It worked flawlessly until now. Only after the upgrade to Cobia did it have this problem.

I will try the GUI import once I have some downtime, depending on what happens with that bug report. If there is a bug it would be better not to get rid of it while there is still a chance of fixing it.

 

[Sean Hafeez]

So I had this same issue a few releases ago also. I posted in the forums and opened a support case. In the end the method @simdim proposed fixed the issue for me. No idea if IX ever sorted out what it was.

 

[OneHungryPoboy]

run "midclt call pool.query | jq" in shell and see if your pool has a GUID. I had the same issue persisting in reboot and it was due to not having a GUID. Exporting then reimporting from the GUI fixed it.

 

[justryan]

run "midclt call pool.query | jq" in shell and see if your pool has a GUID. I had the same issue persisting in reboot and it was due to not having a GUID. Exporting then reimporting from the GUI fixed it.

It has a guid

 

[justryan]

I am unsure if there is a solution to prevent this happening again, but there is a workaround. The same as @OneHungryPoboy said, for nearly the same issue:

you need to use the webUI to export the pool (DO NOT DELETE the data/configuration). Then simply re-import your zpool. The problem is that you have an entry in the table for a zpool with the same name but has a different GUID.

The easiest solution is to export and re-import. This will resolve your problem.

from https://ixsystems.atlassian.net/browse/NAS-125144

 

[aionica]

I had the same problem, meaning my main storage pool was missing after upgrading from Bluefin to Codia and the solution mentioned here (import from CLI, export from web UI, import back using web UI) fixed the problem.
My pool was missing the uuid and this was an old pool, which went through many updates of TrueNas Core and then later upgraded from Core to Scale Bluefin.
I'd bet that this specific problem is for people which have previously upgraded from TrueNas Core.

P.S. please don't just blindly run `zpool import -f -F -R /mnt $MY_STORAGE_POOL` and instead read up on what `-f -F -R` parameters do and asses if they make sense for you (for me neither -f or -F made sense).
Also now running `midclt call pool.query | jq` shows that my pool has a `guid` .

Thank you.