my nas randomly deleted 90% of my data

[artemisery]

hello everyone,

I just ran into a bit of a problem and i dont know how to proceed. I have 2 storage pools made up of 4 14tb drives (2 in each pool for redundancy). I just went to access a video file stored on my server and one of my pools is gone and it now saves i have 23tb of free space. I only use the server for an smb share, and the only things that I have done with it today is watch a video this morning, and moved a pdf from my computer onto the server (that was about an hour ago). I went to access a video on it and most everything is missing. the only file structure that remains is the folder path to the video that i watched this morning.

I dont know if it may be the cause, but my boot drive for the server is failing ( it has 8 unreadable sectors), and needs replaced (the os is the only thing stored on it). does anyone have any suggestions as to how to proceed. about 90% of the data on the server is backed up. It will be a headache if the data is not recoverable (the missing 10% is not crucial data), but it wont be the end of the world by any means.

any advice on how to proceed is greatly appreciated.

 

[artemisery]

hello everyone,

I just ran into a bit of a problem and i dont know how to proceed. I have 2 storage pools made up of 4 14tb drives (2 in each pool for redundancy). I just went to access a video file stored on my server and one of my pools is gone and it now saves i have 23tb of free space. I only use the server for an smb share, and the only things that I have done with it today is watch a video this morning, and moved a pdf from my computer onto the server (that was about an hour ago). I went to access a video on it and most everything is missing. the only file structure that remains is the folder path to the video that i watched this morning.

I dont know if it may be the cause, but my boot drive for the server is failing ( it has 8 unreadable sectors), and needs replaced (the os is the only thing stored on it). does anyone have any suggestions as to how to proceed. about 90% of the data on the server is backed up. It will be a headache if the data is not recoverable (the missing 10% is not crucial data), but it wont be the end of the world by any means.

any advice on how to proceed is greatly appreciated.

im not sure how to edit a post on here, but i need to make a couple of corrections to this post. i have one storage made up of two arrays. one of my arrays went down.

also I am using TrueNAS-13.0-U5.3

 

[sretalla]

please share the outputs (in code tags please) from:

zpool status -v

zfs list

What I'm expecting to see is what has happened to your pool and the data in it.

If you lost an "array"... maybe you mean VDEV... it's possible that your pool (what you're calling storage) is more-or-less dead.

 

[artemisery]

please share the outputs (in code tags please) from:

zpool status -v

zfs list

What I'm expecting to see is what has happened to your pool and the data in it.

If you lost an "array"... maybe you mean VDEV... it's possible that your pool (what you're calling storage) is more-or-less dead.

sorry for the mix up on the terminology, and thank you for your help. im getting a monitor hooked up to the server now to get those outputs.

POSSIBLE IRRELEVANT INFO JUST FOR THE SAKE OF CLARIFICATION:
the storage pool still shows 7% being used and the remaining 23tb free. the second vdev was set up (and the pool was expanded) two months ago and very little has been added since then. sothe second vdev has its data and the first one appears to be wiped.

I will reply to your post in a minute once I have the requested information.

 

[artemisery]

zpool status -v

RESULT IN SYSTEM FAILURE. root@truenas[~]# zpool status -v pool: boot-pool state: ONLINE scan: scrub repaired 0B in 00:00:32 with 0 errors on Wed Jan 10 03:45:32 2024 config: NAME STATE READ WRITE CKSUM boot-pool ONLINE 0 0 0 ada2p2 ONLINE 0 0 0 errors: No known data errors pool: general file server state: ONLINE config: NAME STATE READ WRITE CKSUM general file server ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gptid/8137114d-fb06-11ed-984e-001fbc08f42b ONLINE 0 0 0 gptid/811997f2-fb06-11ed-984e-001fbc08f42b ONLINE 0 0 0 mirror-1 ONLINE 0 0 0

zfs list

gptid/22e82f78-4eaa-11ee-883c-001fbc08f42b ONLINE 0 0 0 errors: No known data errors root@truenas[~]# zfs list NAME USED AVAIL REFER MOUNTPOINT boot-pool 3.03G 881G 96K none boot-pool/ROOT 3.03G 881G 96K none boot-pool/ROOT/Initial-Install 8K 881G 1.51G legacy boot-pool/ROOT/default 1.52G 881G 1.51G legacy boot-pool/ROOT/default-20230911-213158 1.51G 881G 1.51G legacy general file server 1.68T 23.6T 96K /mnt/general file server general file server/.system 899M 23.6T 854M legacy general file server/.system/configs-b85043590a434da692c02c2416a40e36 6.59M 23.6T 6.59M legacy general file server/.system/configs-bd26c8fd36fd4618a75dffa52c04f828 2.10M 23.6T 2.10M legacy general file server/.system/cores 96K 1024M 96K legacy general file server/.system/rrd-b85043590a434da692c02c2416a40e36 18.7M 23.6T 18.7M legacy general file server/.system/rrd-bd26c8fd36fd4618a75dffa52c04f828 13.4M 23.6T 13.4M legacy general file server/.system/samba4 376K 23.6T 376K legacy general file server/.system/services 96K 23.6T 96K legacy general file server/.system/syslog-b85043590a434da692c02c2416a40e36 1.63M 23.6T 1.63M legacy general file server/.system/syslog-bd26c8fd36fd4618a75dffa52c04f828 1.86M 23.6T 1.86M legacy general file server/.system/webui 96K 23.6T 96K legacy general file server/File Share 1.68T 23.6T 1.68T /mnt/general file server/File Share general file server/iocage 8.69M 23.6T 8.13M /mnt/general file server/iocage general file server/iocage/download 96K 23.6T 96K /mnt/general file server/iocage/download general file server/iocage/images 96K 23.6T 96K /mnt/general file server/iocage/images general file server/iocage/jails 96K 23.6T 96K /mnt/general file server/iocage/jails

 

[artemisery]

please share the outputs (in code tags please) from:

zpool status -v

zfs list

What I'm expecting to see is what has happened to your pool and the data in it.

If you lost an "array"... maybe you mean VDEV... it's possible that your pool (what you're calling storage) is more-or-less dead.

forgot to click reply. it has been posted.

 

[NugentS]

@artemisery can you post your complete hardware list please - so we have some idea what we are delaing with - as per forum rules

Thanks

 

[artemisery]

@artemisery can you post your complete hardware list please - so we have some idea what we are delaing with - as per forum rules

Thanks

yes, sorry about that. I read through the rules, but it didnt was during the my server broke panic phase, before the hey wait this is "almost all backed up" peace of mind kicked in. so thank you for the reminder.

hardware is as follows:
cpu: i7 870
motherboard: EVGA P55 ftw
ram: 8gb
boot drive: Seagate Barracuda 1tb
storage drives: 2X wd 14tb 7200rpm drives (wd element external drives that were shucked bought this year) [these are the missing vdev drives]
storage drives: 2X wd 14tb refurbished ultrastar drives.

 

[artemisery]

yes, sorry about that. I read through the rules, but it didnt was during the my server broke panic phase, before the hey wait this is "almost all backed up" peace of mind kicked in. so thank you for the reminder.

hardware is as follows:
cpu: i7 870
motherboard: EVGA P55 ftw
ram: 8gb
boot drive: Seagate Barracuda 1tb
storage drives: 2X wd 14tb 7200rpm drives (wd element external drives that were shucked bought this year) [these are the missing vdev drives]
storage drives: 2X wd 14tb refurbished ultrastar drives.

gpu is a an evga gtx 470

 

[wsoteros]

Friendly ping that I moved your thread to the TrueNAS CORE section for better visibility :)

 

[artemisery]

Friendly ping that I moved your thread to the TrueNAS CORE section for better visibility :)

thank you :D

 

[NugentS]

Can you see the two missing disks in the system BIOS?

 

[artemisery]

Can you see the two missing disks in the system BIOS?

only 3 drives show up when booting and in bios. i dont know if that is because 2 of the drives are redundant of if that means two drives arent recognized. as i mentioned to another user, i have 2 vdevs made of 2 drives each.

in the web gui it says I have 23tbs free, so all of the drives are detected there.

 

[artemisery]

only 3 drives show up when booting and in bios. i dont know if that is because 2 of the drives are redundant of if that means two drives arent recognized. as i mentioned to another user, i have 2 vdevs made of 2 drives each.

in the web gui it says I have 23tbs free, so all of the drives are detected there.

sorry for the sideways picture. i didnt know it was sideways, otherwise i wouldve fixed it before uploading.

 

[NugentS]

This is not a TrueNAS issue.
Based on your hardware spec all drives should appear in the BIOS. If they aren't you have a hardware issue of some kind

 

[artemisery]

This is not a TrueNAS issue.
Based on your hardware spec all drives should appear in the BIOS. If they aren't you have a hardware issue of some kind

that's what I was thinking after responding to you. i went ahead and got a new boot drive since mine is bad. im gonna do a fresh install on the new boot drive and see if i can get everything to work on a new drive. if that doesnt help, im gonna try pulling the drives from the wiped vdev out and see if i can read them on ghostbsd and see if the data is still accessible.

thank you for your time and all of you help today, I appreciate it.

 

[artemisery]

This is not a TrueNAS issue.
Based on your hardware spec all drives should appear in the BIOS. If they aren't you have a hardware issue of some kind

I resolved my hardware issue, but the problem persists. turns out my pcie to sata expansion card went bad (which was why 2 drives were not detected in the bios) now all the drives are detected, but unfortunately the drives were indeed wiped.

 

[NugentS]

And this is why we need specific hardware