HELP, All of a sudden my drive pools are offline?! I can't reach them, however nothing has changed on our end.

[Woozy Face]

Hi my true NAS scale has stopped working, at least we cannot connect the web page only shows a generic error. But no connection can be made even over ssh or anything else. Then did a reboot of the server to find out that, all of a sudden the pools have disappeared. IDK what happened. I can assure you that I changed NO settings or whatever else, before this happened. One day it worked fine, the other day it did not. I hope someone can help us since this means a complete data loss of my company at this point. Like 17 years of work through the drain… Had nothing like this when I used my win server for SMB. Now I feel awful about switching to True NoAnythingServer it should make things redundant not destroy my data…

Please refer to the screenshots for the information.

​

 

[Samuel Tai]

Per the Forum Rules, please supply details of your hardware. Your pool may be recoverable, but having this information is necessary for forum members to help you.

However, this may be a simple case of the TrueNAS SCALE middleware crashing, and thus spuriously displaying a blank pool status. This can be remedied by running service middlewared restart. If this doesn't work, try rebooting your server.

If this resolves your issue, great! However, regardless of your storage platform, Windows or TrueNAS, any platform redundancy is NO substitute for having a solid backup strategy. You've just seen what could happen without having backups. Don't delay in backing up your data off your server once your pool is back.

 

[Woozy Face]

Per the Forum Rules, please supply details of your hardware. Your pool may be recoverable, but having this information is necessary for forum members to help you.

However, this may be a simple case of the TrueNAS SCALE middleware crashing, and thus spuriously displaying a blank pool status. This can be remedied by running service middlewared restart. If this doesn't work, try rebooting your server.

If this resolves your issue, great! However, regardless of your storage platform, Windows or TrueNAS, any platform redundancy is NO substitute for having a solid backup strategy. You've just seen what could happen without having backups. Don't delay in backing up your data off your server once your pool is back.

I know i read the forum rules, but this makes me so angry and took a lot of time already that I couldn't take anytime to do so, however I thought that i did attach that screenshot. But not, sorry did not even have time to check it ;-). Here you go.

 

[Woozy Face]

If this resolves your issue, great! However, regardless of your storage platform, Windows or TrueNAS, any platform redundancy is NO substitute for having a solid backup strategy. You've just seen what could happen without having backups. Don't delay in backing up your data off your server once your pool is back.

I was about to make a back-up but yep time is a concern normally we do a 3 phase back-up so don't need to learn me that lesson. And of course i have some data still but it's like 1-month-old! Do you know how many changes occur in 30 days…?

 

[Samuel Tai]

I know i read the forum rules, but this makes me so angry and took a lot of time already that I couldn't take anytime to do so, however I thought that i did attach that screenshot. But not, sorry did not even have time to check it ;-). Here you go.

That's only somewhat helpful. Can you describe your system in text? Click the blue button in my signature for an example of the level of detail we're looking for.

 

[Woozy Face]

Per the Forum Rules, please supply details of your hardware. Your pool may be recoverable, but having this information is necessary for forum members to help you.

However, this may be a simple case of the TrueNAS SCALE middleware crashing, and thus spuriously displaying a blank pool status. This can be remedied by running service middlewared restart. If this doesn't work, try rebooting your server.

If this resolves your issue, great! However, regardless of your storage platform, Windows or TrueNAS, any platform redundancy is NO substitute for having a solid backup strategy. You've just seen what could happen without having backups. Don't delay in backing up your data off your server once your pool is back.

I also smashed the “service middlewared restart” into the terminal but did not take an effect so-far so I try another reboot now.

 

[Woozy Face]

I also smashed the “service middlewared restart” into the terminal but did not take an effect so-far so I try another reboot now.

Next time I will add text if needed ;-)

 

[Woozy Face]

Still an offline pool, even after reboot.

 

[Samuel Tai]

What does zpool status -v ECZ-FS-01 show?

 

[Woozy Face]

What does zpool status -v ECZ-FS-01 show?

root@truenas[~]# zpool status -v ECZ-FS-01
cannot open 'ECZ-FS-01': no such pool
root@truenas[~]#

Weird...

 

[Samuel Tai]

OK, does zpool import show an unimported pool?

 

[Woozy Face]

OK, does zpool import show an unimported pool?

Code:

# zpool import
  pool: ECZ-FS-01  id: 6594310499435478175
 state: UNAVAIL
status: One or more devices are faulted.
action: The pool cannot be imported due to damaged devices or data.
config:
        ECZ-FS-01                               UNAVAIL  insufficient replicas
          c9be9e88-ab6c-4d22-9f44-a392c3dfe55f  ONLINE
          179175a4-92b5-4e79-9950-2438db4f42c9  FAULTED  too many errors

Wait what, but that drive was new.

 

[Samuel Tai]

Ouch. Your pool is constructed as a stripe over 2 drives, instead of a mirror over 2 drives, which could've survived the loss of any single drive. Your only option is to rebuild from backups, and possibly rebuild your pool as a mirror next time.

Wait what, but that drive was new.

Did you run a burn-in test before putting any drive into production? What does smartctl -a show for the faulted drive?

 

[Woozy Face]

Ouch. Your pool is constructed as a stripe over 2 drives, instead of a mirror over 2 drives, which could've survived the loss of any single drive. Your only option is to rebuild from backups, and possibly rebuild your pool as a mirror next time.


Did you run a burn-in test before putting any drive into production? What does smartctl -a show for the faulted drive?

Yes for the time being, i did that but only cause of the fact that I did not have two 4 drives of the same size at hand. I was going to redo that.

 

[Woozy Face]

Ouch. Your pool is constructed as a stripe over 2 drives, instead of a mirror over 2 drives, which could've survived the loss of any single drive. Your only option is to rebuild from backups, and possibly rebuild your pool as a mirror next time.


Did you run a burn-in test before putting any drive into production? What does smartctl -a show for the faulted drive?

root@truenas[~]# 2022 Jun 21 18:21:55 truenas Device: /dev/sdc [SAT], 49 Currently unreadable (pending) sectors
2022 Jun 21 18:21:55 truenas Device: /dev/sdc [SAT], 49 Offline uncorrectable sectors
2022 Jun 21 18:21:55 truenas Device: /dev/sdc [SAT], ATA error count increased from 6135 to 6153

 

[Samuel Tai]

OK, unfortunately, that drive's not recoverable, so the only possibility I see for recovering your data is to use a ZFS recovery utility, like Klennet. You ran into a drive's infant mortality, which could've been avoided by burning it in for a week or so before putting it under production stress.

 

[Woozy Face]

OK, unfortunately, that drive's not recoverable, so the only possibility I see for recovering your data is to use a ZFS recovery utility, like Klennet. You ran into a drive's infant mortality, which could've been avoided by burning it in for a week or so before putting it under production stress.

That sucks, but okay then I'm going to shut down the system for now and continue my journey tomorrow. Since It's now 3:30. Also is there any similar software as klennet for mac or Linux, windows sucks, so we smashed that OS out of our windows a while ago now.

 

[Woozy Face]

That sucks, but okay then I'm going to shut down the system for now and continue my journey tomorrow. Since It's now 3:30. Also is there any similar software as klennet for mac or Linux, windows sucks, so we smashed that OS out of our windows a while ago now.

a google search doesn't make me wiser, it only throws windows software to me.

 

[Samuel Tai]

Also is there any similar software as klennet for mac or Linux, windows sucks, so we smashed that OS out of our windows a while ago now.

I'm not aware of anything else, and klennet has a known track record. For example, see this thread: https://www.truenas.com/community/threads/update-11-3-zfs-pool-empty.85299/

 

[Woozy Face]

I'm not aware of anything else, and klennet has a known track record. For example, see this thread: https://www.truenas.com/community/threads/update-11-3-zfs-pool-empty.85299/

I found about Disk Drill, normally we use Ease US but. Both are paid solutions, we will find out tomorrow. Thanks so far!