TrueNAS died, wont boot

[JohnFLi]

During the night, my TrueNas had a issue and was no longer responding, or even on the network (wouldn't respond to ping)
Did a power cycle, and still doesn't come up to the point of even getting back on the network.

I have included a couple of images in hopes somebody might know what to do.

The pool named "Sharepoint backup" did have a failed drive yesterday, and it was in process of re-silvering. I do not know if it completed before it had a stroke.

Any ideas on what I can do? I am by no means an expert, so if anybody has any ideas....explain them like I know nothing. ;)

 

[Whattteva]

Are those messages that it prints out during boot or before you rebooted?

 

[JohnFLi]

Are those messages that it prints out during boot or before you rebooted?

Those are during reboot. The screenshot that ends with 4496...that is where it stops. been sitting at that screen for about 2.5 hrs.
if I could get it to where it skips, or even drops the storage pool named 'Sharepoint Backup', that would be fine. ANy data I want is in a different pool

 

[Whattteva]

You can probably try to disconnect the drives that are allocated to Sharepoint Backup and see if it boots.

 

[JohnFLi]

You can probably try to disconnect the drives that are allocated to Sharepoint Backup and see if it boots.

I have no idea which drives those are. THis unit has about 40 or so hard drives in it

 

[Whattteva]

Another thing you can try to do. Hopefully, you have the config file backed up. Reinstall a fresh copy of TrueNAS and restore the config file on it. If it's a boot device corruption, this should fix it. You can do this without a config file backup too, but it's a bit annoying as you will have to restore all the configuration manually. Also, don't do this if your pools are encrypted because the keys are on the boot device... hopefully you also have those backed up.

 

[JohnFLi]

Another thing you can try to do. Hopefully, you have the config file backed up. Reinstall a fresh copy of TrueNAS and restore the config file on it. If it's a boot device corruption, this should fix it. You can do this without a config file backup too, but it's a bit annoying as you will have to restore all the configuration manually. Also, don't do this if your pools are encrypted because the keys are on the boot device... hopefully you also have those backed up.

One thing for sure, the pools are NOT encrypted.
What is the generic file name for the config file? I am sure i have it, just don't remember the name.

Is there any documentation for how to do this?
I did see on another post somethign about getting 2 usb. one with an install file for truneas, the other blank. run the install file, and install TrueNas onto the blank USB, then 'import' that way........ are there any insturctions for doing that?

 

[Whattteva]

One thing for sure, the pools are NOT encrypted.

That's a good thing for times like these. Lot of people encrypt their pools only to end up with a self-imposed ransomware.

What is the generic file name for the config file? I am sure i have it, just don't remember the name.

Should be something like name_of_server-TrueNAS-13.0-2023.....db

Is there any documentation for how to do this?

Normally, you'd do this from the web GUI when it was functional. But if you could somehow get to the shell, it's located in /var/db/system/configs-.../TrueNAS-13.0/2023....db

I did see on another post somethign about getting 2 usb. one with an install file for truneas, the other blank. run the install file, and install TrueNas onto the blank USB, then 'import' that way........ are there any insturctions for doing that?

I mean... how did you install TrueNAS in the first place? It's the same exact process. You just "burn" the ISO image onto a USB stick and reinstall that to your boot drive or another USB stick (for emergency, shouldn't be long term) and reupload your config. If you don't have a backup of that config, you can just reimport your pools and regenerate your config (users/groups/shares, etc.). It's a total pain in the neck, but something that's very doable, albeit tedious.

 

[JohnFLi]

That's a good thing for times like these. Lot of people encrypt their pools only to end up with a self-imposed ransomware.


Should be something like name_of_server-TrueNAS-13.0-2023.....db


Normally, you'd do this from the web GUI when it was functional. But if you could somehow get to the shell, it's located in /var/db/system/configs-.../TrueNAS-13.0/2023....db


I mean... how did you install TrueNAS in the first place? It's the same exact process. You just "burn" the ISO image onto a USB stick and reinstall that to your boot drive or another USB stick (for emergency, shouldn't be long term) and reupload your config. If you don't have a backup of that config, you can just reimport your pools and regenerate your config (users/groups/shares, etc.). It's a total pain in the neck, but something that's very doable, albeit tedious.

When I got this system....it was Still called FreeNas (Sept. of 2016) The OS was pre-installed.

I did put the ISO of the version on the system (12 -U8)...it did install with zero issue. Got to the point to upload the config. (I did find one a year or so old that I had previously downloaded.
It claimed to have uploaded just fine.....then it wanted a reboot. It stoped st the same place as before (see my inital screenshot) compleining about one of the storage pools.

I am going to try again here in a few min with a new install...just to make sure I didn't screw anythign up.

Once I get the OS installed a new harddrive (it will be a regular spinning disk harddrive but connected via USB)...how would I use the command line to see the existing hardware? I hardly know anyhting about Linux type stuff

.

 

[JohnFLi]

Made some progress!!
Ok, so after getting back into the interface, and after loading old config, and getting back in teh system.....
I go to storage Pools, (of cource nothing is there, but I was able to import 1)

Out of what I sees, I would really like to get the VideoStorage back.....even for a few hours, but when I try to import that one, I don't see the option

I go through the process for teh storage pool....I click Add, I select "import from existing pool"
After it does is searching, it come back with somethign different

It sees teh pool I want when going to 'storage pools', put doesn't see it when I try to import.
Any ideas?

One thing I have also noticed.... WHen I go to 'Storage' then 'disks' it is only showing 17 disks..... I have about 40 or so disks.....which I would guess might explain why i can't import the pool....

 

[Whattteva]

Seems like your disks are not being detected. What does geom disk list return? Verify that it returns all 40 or so of your disks. You may have a hardware problem here (backplane, cables, etc.).

 

[JohnFLi]

It only returns 17 disks

 

[Whattteva]

At this point, I think your best course of action is to reseat cables and check all your connections, backplane, HBA, etc. Any of the hardware involved in the chain that may need to be replaced. Your pool is very likely not at all the problem and just won't mount due to hardware issues. Conceivably, assuming all the disks are good and the problem is more cables or controller, once you get those replaced, the pool should mount again.

 

[JohnFLi]

At this point, I think your best course of action is to reseat cables and check all your connections, backplane, HBA, etc. Any of the hardware involved in the chain that may need to be replaced. Your pool is very likely not at all the problem and just won't mount due to hardware issues. Conceivably, assuming all the disks are good and the problem is more cables or controller, once you get those replaced, the pool should mount again.

Hopefully.....when trying to boot normally, it always seems to die when it gets to the Sharepoint Pool.

 

[MrGuvernment]

maybe one of the Rocket 750 cards went bad? Are you able to enter the config of the rocket cards before truenas boots to see if they show all drives connected?

 

[Redcoat]

Rocket 750 has a bad rep here - search on it and see the early hopeful history then conclusions to the contrary ref drivers, etc.

 

[JohnFLi]

At this point, I think your best course of action is to reseat cables and check all your connections, backplane, HBA, etc. Any of the hardware involved in the chain that may need to be replaced. Your pool is very likely not at all the problem and just won't mount due to hardware issues. Conceivably, assuming all the disks are good and the problem is more cables or controller, once you get those replaced, the pool should mount again.

Here was a fun little discovery.....
I reinstalled teh OS again (to a different disk I attaced to USB)....before loading a year old config, I looked at teh disks.... All 49 disks were detected and displayed.

So I went to storage/Pools, and clicked on ADD. I told it to 'import an existing pool"after a bit, it did come up with the pools I had before. I picked teh pool I cared about the most, and it said it brought it back. I turned on SMB, joined the Truenas box to teh domain, and poof, I have access to my pool on the network again.

Currently in process of copying everything to another location. 1 gig connection is much slower than the original 20 gig connection. I dont want to mess with things to much to try to get the faster connection reset. True nas isn't even seeing the twinax connectors at all.

Thank you for all your help in trying to get this working

 

[Whattteva]

Here was a fun little discovery.....
I reinstalled teh OS again (to a different disk I attaced to USB)....before loading a year old config, I looked at teh disks.... All 49 disks were detected and displayed.

It sounds like maybe whatever disk you were installing to was dying. Did you not use a different disk for the subsequent installs?

 

[JohnFLi]

It sounds like maybe whatever disk you were installing to was dying. Did you not use a different disk for the subsequent installs?

Same USB stick. I tried using a spinning disk that connected to USB, but either the adapter wasn't working right or whatever.... SO i ended up reinstalling Truenas on the USB stick again. And just tried recovering the pools without uploading the truenas config.db file.

 

[Whattteva]

The USB stick sounds suspect. I mean it initially rendered your system unbootable, then it exhibits these strange properties once you upload the config. I would probably replace that with a cheap SSD.