All apps gone after upgrade to Cobia

[dynamicnotion]

Hello everyone, I just upgraded to TrueNAS-SCALE-23.10.0 and everything seemed to go fine. The upgrade went smoothly and I waited a bit for everything to come up again. I got a warning for the new ZFS version which I think is expected but I did not upgrade my pools or anything yet.

However, as I went to navigate to the apps section I noticed that all my installed apps were suddenly gone. (see picture 1)

Another weird thing is that my datasets seem to be duplicated. Normally only the 3 datsets under vault were there and now they are listed as extra items below. (see picture 2) Is this something weird or does this have to do with the ZFS update?

I can still access the NAS normally as a network share but all apps are gone and they are also not working as I have tried to connect to them. Any help on this is greatly appreciated.

 

[Kris Moore]

Can you please file a bug ticket with a debug file attached? Also perhaps try a hard-refresh of the UI, I.E. clearing browser cache.

 

[asap2go]

There is already a ticket for this.

[NAS-124776] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

Problem seems to be the encrypted dataset under which ix-applications resides.

 

[dynamicnotion]

Ok thank you, I just did. Hard refreshing did not help, unfortunately. There just appeared a "Initializing Apps Service" message when I navigate to Apps.

Another interesting thing: When I go to Apps - Settings - Choose pool and I select "vault" and click "Choose", I get "Error: [EINVAL]
kubernetes_update.force: Missing
'/mnt/vault/ix-applications/config.json' configuration file. Specify
force to override this and let system re-initialize applications."
So my assumption would be that Truenas maybe can't find the ix-applications dataset. Or maybe the datasets are messed up right now because they show up two times? I am not entirely sure what to do from here. Everything worked great before the update. It would be great if there is some way to fix this without just nuking the datasets and installing and setting up all apps again.

Thank you @asap2go that seems to be the same issue. Is there any workaround right now?

 

[asap2go]

No. I tried restoring snapshots.
I tried creating the app folder on another unencrypted pool and putting the old snapshot data there. Nothing worked for restoring.
But maybe I am just to stupid.

 

[yomamasnerd]

Hi, I have had a similar issue.
This post helped me a lot. Somehow TrueNAS failed to set a mount point and I had to mount the zfs volume manually.

 

[Hussie]

Can someone of IXsystems give us a ETA of when the release will be available https://ixsystems.atlassian.net/browse/NAS-124776 ?

 

[blastkiller]

Please publish the release of 23.10.1 before the date stated in the official schedule. 19.12.2023 is too far away.

 

[Hussie]

I hope IXsystems will release the fix asap!
This is not a low impact bug. At the moment i can’t use my nas at all.
My whole system is based on apps.

 

[sfatula]

If your system is that important, I would recommend in the future not updating to a .0 release. I always wait for a .2 release myself. Risk tolerance.

 

[blastkiller]

I don't know what got into my finger that I pressed the update button.
Are the commits included in the nightly updates or can I simply maintain the changes manually and then receive the regular updates as usual?

 

[toyman]

I also have same issue and I tried to install all of Apps and set up again but when I restarted NAS, everything was just gone again….. this is production version but it seems to be still beta or even RC one.

 

[blastkiller]

This is a great Product and as spatula said x.0 Version is not a production version. Every software contains bugs and as production environment the first final release should never be used.

 

[toyman]

Anyway I have reverted back to 22.12.4.2 and I totally agree with you, this is good product and it is normal to have bugs but if the testing can cover backward compatibility before final release would be much better then.

 

[Hussie]

This is not just a bug it breaks down the apps.

According to the ticket issue is solved in the code. Is there a way to get this fix?

 

[Mugiwaraya]

Anyway I have reverted back to 22.12.4.2 and I totally agree with you, this is good product and it is normal to have bugs but if the testing can cover backward compatibility before final release would be much better then.

Reverting back (by changing boot env back) didn't do anything and all my apps are delisted. The ix-applications folder is still there, but it is whining about config.json or w/e which isn't in the folder (anymore?)... I just deleted the ix applications folder to set everything from the ground up, but the funny thing is that when I choose a pool for the apps, it creates the ix-applications folder. Then I go on installing some apps, which it does fine, but after reboot it loses access to the apps again and gets stuck at "Initializing Apps Service". Does anyone have a solution for this?

 

[toyman]

Reverting back (by changing boot env back) didn't do anything and all my apps are delisted. The ix-applications folder is still there, but it is whining about config.json or w/e which isn't in the folder (anymore?)... I just deleted the ix applications folder to set everything from the ground up, but the funny thing is that when I choose a pool for the apps, it creates the ix-applications folder. Then I go on installing some apps, which it does fine, but after reboot it loses access to the apps again and gets stuck at "Initializing Apps Service". Does anyone have a solution for this?

Yes, before I reverted back to the previous version, the apps were always gone after I restarted the NAS and it indicated “initializing Apps Service” as what you mentioned, and I had to unset pool and removed ix-applications folders with shell and then selected pool again, which means everything needs to be set up one more time.

I did revert back to the previous version as what you did, but I completely removed application pool before that so now I have to create new pool in the previous version and set up/import data again, it takes time to finish but at least it works at the moment.

 

[ABain]

23.10.0.1 is out today with a fix for apps on encrypted pools.

 

[Protopia]

This is a great Product and as spatula said x.0 Version is not a production version. Every software contains bugs and as production environment the first final release should never be used.

If no one used the first .0 release, then bugs would never be found and a .1 release would never happen.

ixSystems have been quite clear that the 12.23.0 release is not for everyone to use, just a brave few. I would personally suggest that it should be used by those people installing TrueNas Scale on a fresh system, and who won't mind waiting for a month or two if they discover a problem - on the basis that you won't experience upgrade issues, and as you configure it up, if you find a problem then you won't yet have your data on the system and therefor no longer available.

Certainly I am not planning to upgrade from Bluefin for several (4-6?) months in order to allow bugs and common user errors to be found and fixed.

 

[Joe-freenas]

If no one used the first .0 release, then bugs would never be found and a .1 release would never happen.

ixSystems have been quite clear that the 12.23.0 release is not for everyone to use, just a brave few. I would personally suggest that it should be used by those people installing TrueNas Scale on a fresh system, and who won't mind waiting for a month or two if they discover a problem - on the basis that you won't experience upgrade issues, and as you configure it up, if you find a problem then you won't yet have your data on the system and therefor no longer available.

Certainly I am not planning to upgrade from Bluefin for several (4-6?) months in order to allow bugs and common user errors to be found and fixed.

Disagree somehow as TrueNAS has had extremely well tested releases for the last 5-7 years?

UNTIL THIS ONE!

Are you tell us that some/all of those pretty basic scenarios could not (WERE NOT) being tested!?

I have a QA background, so I know that it's not easy to test everything, but in this *particular* case, the scenarios that did fail, were not complicated or unusual... can we at least agree on that!?