Pool healthy but no data in mount

[JohnyR]

Hi all,

We've run into a strange issue where all of a sudden, all of the data in our pool is gone yet the pool is reporting healthy.

I've tried to scrub the pool, but it came back with 0 repairs and 0 errors. I then tried to export the pool and reimport it -- still no go. Not quite sure what happened.

Here are some screens:

In the screenshot above, you can see ZFS list shows the size of the ESX-VM folder while df -h doesn't have ESX-VM.

Details
- FreeNAS-11.3-U5
- Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
- 98247 MiB

Any tips on where to begin would be great appreciated!

Thank you

 

[JohnyR]

Forgot to post zpool status:

 

[Heracles]

In the screenshot above, you can see ZFS list shows the size of the ESX-VM folder while df -h doesn't have ESX-VM.

Hey @JohnyR,

Your pool looks just fine.

ESX-VM is not a dataset in which you can manipulate files and folders. It is a ZVOL. A ZVOL is kind of a virtual hard drive. It is a bunch of blocks that have no sense one towards the other. That ZVOL is presented to an operating system that will use it as a hard drive. That other OS will format the "drive" and use it its own way. FreeNAS will not be aware of anything at file level in that volume. For that reason, FreeNAS can not run any file level command toward that ZVOL.

I also have a ZVOL in my FreeNAS and I too use it for hosting an ESXi datastore and VMs. I too can see it as a ZFS structure (zfs list) but not as a file level structure (no df -h).

 

[sretalla]

You should normally find your zvols under /dev/zvol/<poolname>/<tree>/zvol

As they aren't part of a mount, you won't find them in /mnt.

Also noteworthy is your messy pool design...

Your RAIDZ2 VDEV will be hampering your pool's performance for block storage.

Read this if you're interested in understanding that: https://www.truenas.com/community/threads/the-path-to-success-for-block-storage.81165/

 

[RickP]

You should normally find your zvols under /dev/zvol/<poolname>/<tree>/zvol

As they aren't part of a mount, you won't find them in /mnt.

Also noteworthy is your messy pool design...

Your RAIDZ2 VDEV will be hampering your pool's performance for block storage.

Read this if you're interested in understanding that: https://www.truenas.com/community/threads/the-path-to-success-for-block-storage.81165/

Thanks for the article. Yea we were aware of that, it was a mistake that honestly I was too lazy to go back and fix given what we use it for lol

Anything you have on removing / swapping it would be appreciated but I didn't see an easy(ish) way to go about it.

 

[RickP]

You should normally find your zvols under /dev/zvol/<poolname>/<tree>/zvol

As they aren't part of a mount, you won't find them in /mnt.

Also noteworthy is your messy pool design...

Your RAIDZ2 VDEV will be hampering your pool's performance for block storage.

Read this if you're interested in understanding that: https://www.truenas.com/community/threads/the-path-to-success-for-block-storage.81165/

Also, I did find ESX-VM under the zvol as you suggested but the size is 1KB. Is that expect or should I be seeing something the full size we would expect? ~10TB?

 

[JohnyR]

Hey @JohnyR,

Your pool looks just fine.

ESX-VM is not a dataset in which you can manipulate files and folders. It is a ZVOL. A ZVOL is kind of a virtual hard drive. It is a bunch of blocks that have no sense one towards the other. That ZVOL is presented to an operating system that will use it as a hard drive. That other OS will format the "drive" and use it its own way. FreeNAS will not be aware of anything at file level in that volume. For that reason, FreeNAS can not run any file level command toward that ZVOL.

I also have a ZVOL in my FreeNAS and I too use it for hosting an ESXi datastore and VMs. I too can see it as a ZFS structure (zfs list) but not as a file level structure (no df -h).

Hey @Heracles,

Thank you for the swift response, and also the detailed explanation! :) I was not aware that FreeNAS wouldn't be aware of anything at file level, but it certainly makes sense.

We managed to get everything back online. Appreciate your help and guidance.

You should normally find your zvols under /dev/zvol/<poolname>/<tree>/zvol

As they aren't part of a mount, you won't find them in /mnt.

Also noteworthy is your messy pool design...

Your RAIDZ2 VDEV will be hampering your pool's performance for block storage.

Read this if you're interested in understanding that: https://www.truenas.com/community/threads/the-path-to-success-for-block-storage.81165/

Ah yes, that makes sense.

Agreed, and as Rick had stated, we kind of misconfigured and realized after it was too late. Whoops.. Is there an easy fix to our misconfiguration? I assume not.

Thanks for your help as well!

 

[Heracles]

Agreed, and as Rick had stated, we kind of misconfigured and realized after it was too late. Whoops.. Is there an easy fix to our misconfiguration? I assume not.

Hey @JohnyR,

Unfortunately, there is no easy way to do that. You can not remove a vDev from a pool. You will have to destroy the pool and re-create it. For that, you need a temporary pool big enough for you to zfs send / zfs receive your data. Once done, you destroy the first pool and re-create it properly. When that new pool is back online and properly configured, you zfs send / zfs receive your data back from your temporary pool.

To have complete and functional backups is a hard requirement for anything valuable. If you do not already have such a complete backup, it may be time to think about it...

 

[sretalla]

Also, I did find ESX-VM under the zvol as you suggested but the size is 1KB. Is that expect or should I be seeing something the full size we would expect? ~10TB?

You're right, it's effectively just the pointer.