Recovering accidentally removed truenas config and keys

[mangof60]

So I blundered today. I reset the config of my truenas server wanting to only reset the network config without realizing it would reset everything (yes I know, it says that in the manual). The problem is that my dataset is encrypted and the keyfile I saved isn't working for some reason. I took out the boot drive and I want to try to recover the removed config data.

Just for clarity, my pools are safe and the data is there, they're just encrypted and my backup keyfile is invalid (perhaps I changed the key and didn't update my file). I understand there is no way to decrypt the data without the keyfile, so my only objective is to get the config db and keyfile from the boot drive. I don't know what process truenas does when it removes the old configs, if it does a secure erase or something like that.. So, is it even feasible to do some data forensics on this drive to recover the deleted files? If so, what tools can I use? I have access to a Windows and Ubuntu machine for this

Thanks

 

[winnielinnie]

In the boot-pool, check under the directory /data/ for the following files:

• freenas-v1.db
• freenas-v1.db.bak
• pwenc_secret
• pwenc_secret.bak

Make a backup of the above files in the meantime, such as copying them to your laptop, just in case they get overwritten.

You can find old configs under the directory /var/db/system/configs-XXXXXXXX, assuming that your .system dataset is not encrypted. However, these do not contain the secret seed.

You might be able to "tar" the *.bak versions of the DB file + secret seed into a single tarball, and then use that you "restore" your config from the GUI.


But really, you should have had up-to-date backups of your encryption keyfiles and TrueNAS configs. There is no substitute for this.

 

[mangof60]

I honestly don't know why my keyfile isn't working, but I will set up an automated backup system for that on an offsite machine just in case.

I will search for those files now! Since it was the most recent config, that means it should be the .bak files in /data, correct?

 

[winnielinnie]

Since it was the most recent config, that means it should be the .bak files in /data, correct?

Hopefully.

The .bak files may in fact be from the original install date, or the last time you upgraded the TrueNAS Core system.

 

[mangof60]

Hopefully.

The .bak files may in fact be from the original install date, or the last time you upgraded the TrueNAS Core system.

I used ddrescue to copy the boot drive from my Ubuntu machine, but I'm having trouble mounting it so I can look at the files. I used zpool import and it found boot-pool but I'm not able to find it in my filesystem.

 

[winnielinnie]

I used ddrescue to copy the boot drive from my Ubuntu machine, but I'm having trouble mounting it so I can look at the files. I used zpool import and it found boot-pool but I'm not able to find it in my filesystem.

Why are you physically removing the boot-pool drive(s) to be used in a separate Ubuntu system? You can use ssh / scp to copy them directly from the server to your client computer.

I used zpool import and it found boot-pool

If you use zpool import without specifying any pools, it will simply list what is "importable". It doesn't actually import anything.

I'd be careful trying to import a boot-pool dataset (from TrueNAS) into another live system. Unlike the datasets on your "data pools", it doesn't have a unique mountpoint set, and in fact your current environment (i.e, "13.0-U5") outright mounts to /

 

[Apollo]

Just a thought, @mangof60 did you update TrueNAS/FreeNAS at some point with that same boot drive? If so, it might be simpler to activate the previous boot environment and boot from it.

 

[winnielinnie]

Can't believe I overlooked that. I guess it goes to show you that how you ask a question influences the proposed solutions.

 

[mangof60]

Why are you physically removing the boot-pool drive(s) to be used in a separate Ubuntu system? You can use ssh / scp to copy them directly from the server to your client computer.

I removed the drive so that I could do data recovery on it, but if the files are backed up then you're right I don't need to. I'll put the drive back in and boot it up then transfer with SSH.

Just a thought, @mangof60 did you update TrueNAS/FreeNAS at some point with that same boot drive? If so, it might be simpler to activate the previous boot environment and boot from it.

I'm not sure I understand what this means. I don't have the original usb key, all I have is the boot SSD

 

[Apollo]

I'm not sure I understand what this means. I don't have the original usb key, all I have is the boot SSD

I had my mind set on USB as your Boot drive ( I realized and corrected my post).
What I meant, is that your boot SSD is the one you need to use as it may contains previous boot environement.
Each Update/Upgrade of FreeNAS/TrueNAS will create a snapshot of the boot drive environment which you can activate is something went wrong at some point.

 

[mangof60]

Ok I logged in through ssh to browse the files, but I don't see the .bak files here

Code:

root@truenas[/data]# l
total 150
-rw-r--r--  1 root  wheel        0 Dec 14 16:19 .ses_firmware_update
drwxrwxr-x  2 root  wheel        2 Jul 10  2022 crash
-rw-r--r--  1 root  wheel      424 Jul 10  2022 dhparam.pem
-rw-r--r--  1 www   www    1097728 Jul 10  2022 factory-v1.db
-rw-r--r--  1 root  wheel   720896 Jun  2 16:43 freenas-v1.db
-rw-r--r--  1 root  wheel      634 Dec 14 16:21 hba_firmware_update.log
-rw-r--r--  1 root  wheel     1610 Dec 14 16:13 manifest
drwxr-xr-x  2 root  www          3 Dec 14 16:13 pkgdb
-rw-------  1 www   www         32 Jun  1 15:06 pwenc_secret
drwxr-xr-x  2 root  wheel        3 Jun  1 15:08 sentinels
drwxr-xr-x  2 root  wheel        4 Jun  2 16:41 zfs

I also checked the configs path you mentioned above and it's empty as well

Code:

root@truenas[/var/db/system]# ls
configs-a6924d0eea7543099bc278a3576ed212        rrd-a6924d0eea7543099bc278a3576ed212            update
cores                                           samba4                                          webui
nfs-stablerestart                               services
nfs-stablerestart.bak                           syslog-a6924d0eea7543099bc278a3576ed212
root@truenas[/var/db/system]# cd configs-a6924d0eea7543099bc278a3576ed212
root@truenas[...figs-a6924d0eea7543099bc278a3576ed212]#

 

[mangof60]

I had my mind set on USB as your Boot drive ( I realized and corrected my post).
What I meant, is that your boot SSD is the one you need to use as it may contains previous boot environement.
Each Update/Upgrade of FreeNAS/TrueNAS will create a snapshot of the boot drive environment which you can activate is something went wrong at some point.

I don't know how recently, but I do remember updating my system. Where would that be stored?

 

[Apollo]

I don't know how recently, but I do remember updating my system. Where would that be stored?

Best is if your system is up and running, you can access the boot environments within the GUI => Left menu => System => Boot.

 

[mangof60]

The latest one I have is from 2022-12-14 which is the same date as the backup config and keyfiles I have (which are the ones I'm trying to use to decrypt but aren't working)

 

[Apollo]

Normally, if you go through multiple updates/upgrades, then you will get more selection amongst your list of Boot environment.
However, seeing what you have, Default would have been a good ooption, but I douobt it contains much data as it doesn't use a lot of space. You can try the "Default" and see if you can reboot and have the drives already imported.

Otherwise, I would try checking if you have any snapshot that could exist on your boot drive. (I might have been wrong about the creation of snapshots during update/upgrade.
However the Boot environment seem to be stored under "freenas-boot/ROOT" folder structure.
If you have snapshots, which you can list with:

zfs list -t snapshot -r freenas-boot

Then you might be able to clone the most recent one to look for the keys and such.

 

[joeschmuck]

You might take a look here to see if this might work, but I'm not sure at all. I would make a backup of your current configuration files before attempting this. Be safe, not sorry.

Restoring Configuration after OS Drive Failure and TrueNAS upgrade

Newbie here - USB OS drive failed recently so I installed TrueNAS onto to mirrored SSDs (should have done this the first time.) Imported the ZFS pool which can be seen in the GUI with original data folder and jails showing the near terabyte of data in the pool. But no jails or users shares are...

www.truenas.com

Also, once you get yourself out of this pickle, you might try the Multi-Report script (link below) which can be configured to automatically email you a copy of your config and password files. I get mine every Monday and it's fully automatic.

Best of luck to you.

 

[winnielinnie]

The latest one I have is from 2022-12-14 which is the same date as the backup config and keyfiles I have (which are the ones I'm trying to use to decrypt but aren't working)

But you have some from earlier dates (according to your screenshot.) Though they might be from before you made any pools.

 

[winnielinnie]

-rw------- 1 www www 32 Jun 1 15:06 pwenc_secret

-rw-r--r-- 1 root wheel 720896 Jun 2 16:43 freenas-v1.db

Go ahead and copy these somewhere safe, just in case.

Are you sure the configs folder is empty?

root@truenas[/var/db/system]# cd configs-a6924d0eea7543099bc278a3576ed212
root@truenas[...figs-a6924d0eea7543099bc278a3576ed212]#

You didn't list the directory's contents?

 

[winnielinnie]

Just for clarity, my pools are safe and the data is there, they're just encrypted and my backup keyfile is invalid (perhaps I changed the key and didn't update my file).

You can open up this .json file in a text editor, then manually copy+paste the HEX string to unlock the dataset. See if that works.

 

[mangof60]

You can open up this .json file in a text editor, then manually copy+paste the HEX string to unlock the dataset. See if that works

I did do that when the keyfile itself failed, but I get the same "invalid key" error that way as well.
What I think happened is that sometime between 2022-12-14, I reconfigured the encryption and didn't copy the keyfile, and then all this time I thought the 'old' keyfile was my current one and didn't think to make another copy.

You didn't list the directory's contents?

My fault, I forgot to copy, it is indeed empty

Code:

root@truenas[...figs-a6924d0eea7543099bc278a3576ed212]# l
total 0
root@truenas[...figs-a6924d0eea7543099bc278a3576ed212]#