Pool not showing correct available space after updating to 11.2 U2

[shiphtfour]

I am not very experienced with FreeNAS, so any help would be appreciated. I installed the recent 11.2 U2 updated on my system and now my ZFS storage pool is not showing the correct available space. Previous to the update I had over 6TB of available space and now it shows less than 10MB available and is extremely slow when trying to access it. Otherwise it shows the pool and drives are healthy. Aside from unsuccessfully trying to fix my Plex jail, nothing else has changed.

 

[danb35]

What's the output of zpool list?

 

[shiphtfour]

Code:

Shell
[root@freenas ~]# zpool list                                                   
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALT
H  ALTROOT                                                                     
banaNAS       32.5T  31.5T  1.02T        -         -    56%    96%  1.00x  ONLIN
E  /mnt                                                                         
freenas-boot    14G   759M  13.3G        -         -      -     5%  1.00x  ONLIN
E  -  

 

[danb35]

That's not anywhere close to either the 6 TB you thought you had, or the 10 MB you're saying it's now reporting. Where is it showing that number?

 

[shiphtfour]

It's showing ~12MB on the dashboard and under pools

 

[danb35]

What about zfs list?

 

[shiphtfour]

Used 21.0T Avail 10.7M Refer 14.9T

 

[danb35]

Well, since you've removed all the useful information that zfs list gave you, all I can say is that FreeNAS is reporting correctly that something (you've deleted anything that would identify what) has 10.7 MB available. Find what's using the space and delete it.

 

[shiphtfour]

Code:

NAME                                                       USED  AVAIL  REFER  MOUNTPOINT                                           
banaNAS                                                   21.0T  7.70M  14.9T  /mnt/banaNAS                                         
banaNAS/.system                                            212M  7.70M   208K  legacy                                               
banaNAS/.system/configs-02e83c25ec534b63a4d545000fc9a19d   176K  7.70M   176K  legacy                                               
banaNAS/.system/configs-0998234dd47b4c5e822375e363c0aafd  1.01M  7.70M  1.01M  legacy                                               
banaNAS/.system/configs-09be4b32ac104611b42b864ed584361b  39.3M  7.70M  39.3M  legacy                                               
banaNAS/.system/configs-76c11d7f8a944b3d8e42fe35420dbaa3  56.6M  7.70M  56.6M  legacy                                               
banaNAS/.system/configs-9ce90385f0194601ac7d946075cd6b54   176K  7.70M   176K  legacy                                               
banaNAS/.system/configs-e2eccb3703ad46d2b19f2e4809443384  25.5M  7.70M  25.5M  legacy                                               
banaNAS/.system/cores                                     9.94M  7.70M  9.94M  legacy                                               
banaNAS/.system/rrd-02e83c25ec534b63a4d545000fc9a19d      7.70M  7.70M  7.70M  legacy                                               
banaNAS/.system/rrd-0998234dd47b4c5e822375e363c0aafd       192K  7.70M   192K  legacy                                               
banaNAS/.system/rrd-09be4b32ac104611b42b864ed584361b       192K  7.70M   192K  legacy                                               
banaNAS/.system/rrd-76c11d7f8a944b3d8e42fe35420dbaa3      37.8M  7.70M  37.8M  legacy                                               
banaNAS/.system/rrd-9ce90385f0194601ac7d946075cd6b54      21.9M  7.70M  21.9M  legacy                                               
banaNAS/.system/rrd-e2eccb3703ad46d2b19f2e4809443384       192K  7.70M   192K  legacy                                               
banaNAS/.system/samba4                                    1.08M  7.70M  1.08M  legacy                                               
banaNAS/.system/syslog-02e83c25ec534b63a4d545000fc9a19d    296K  7.70M   296K  legacy                                               
banaNAS/.system/syslog-0998234dd47b4c5e822375e363c0aafd   1.14M  7.70M  1.14M  legacy                                               
banaNAS/.system/syslog-09be4b32ac104611b42b864ed584361b   5.42M  7.70M  5.42M  legacy                                               
banaNAS/.system/syslog-76c11d7f8a944b3d8e42fe35420dbaa3   1.16M  7.70M  1.16M  legacy                                               
banaNAS/.system/syslog-9ce90385f0194601ac7d946075cd6b54    408K  7.70M   408K  legacy                                               
banaNAS/.system/syslog-e2eccb3703ad46d2b19f2e4809443384   1.09M  7.70M  1.09M  legacy                                               
banaNAS/.system/webui                                      176K  7.70M   176K  legacy                                               
banaNAS/.warden-template-standard                         2.76G  7.70M  2.64G  /mnt/banaNAS/.warden-template-standard               
banaNAS/banaNAS                                           6.08T  7.70M  6.08T  /mnt/banaNAS/banaNAS                                 
banaNAS/iocage                                            2.24G  7.70M  4.25M  /mnt/banaNAS/iocage                                 
banaNAS/iocage/download                                    272M  7.70M   176K  /mnt/banaNAS/iocage/download                         
banaNAS/iocage/download/11.2-RELEASE                       272M  7.70M   272M  /mnt/banaNAS/iocage/download/11.2-RELEASE           
banaNAS/iocage/images                                      176K  7.70M   176K  /mnt/banaNAS/iocage/images                           
banaNAS/iocage/jails                                       435M  7.70M   176K  /mnt/banaNAS/iocage/jails                           
banaNAS/iocage/jails/plex                                  435M  7.70M   376K  /mnt/banaNAS/iocage/jails/plex                       
banaNAS/iocage/jails/plex/root                             434M  7.70M  1.49G  /mnt/banaNAS/iocage/jails/plex/root                 
banaNAS/iocage/log                                         184K  7.70M   184K  /mnt/banaNAS/iocage/log                             
banaNAS/iocage/releases                                   1.55G  7.70M   176K  /mnt/banaNAS/iocage/releases                         
banaNAS/iocage/releases/11.2-RELEASE                      1.55G  7.70M   176K  /mnt/banaNAS/iocage/releases/11.2-RELEASE           
banaNAS/iocage/releases/11.2-RELEASE/root                 1.55G  7.70M  1.45G  /mnt/banaNAS/iocage/releases/11.2-RELEASE/root       
banaNAS/iocage/templates                                   176K  7.70M   176K  /mnt/banaNAS/iocage/templates                       
banaNAS/jails                                             17.6G  7.70M   248K  /mnt/banaNAS/jails                                   
banaNAS/jails/.warden-template-pluginjail-11.0-x64         678M  7.70M   678M  /mnt/banaNAS/jails/.warden-template-pluginjail-11.0-x
64                                                                                                                                 
banaNAS/jails/.warden-template-standard-11.0-x64          2.93G  7.70M  2.92G  /mnt/banaNAS/jails/.warden-template-standard-11.0-x64
banaNAS/jails/TS3                                         2.11G  7.70M  3.50G  /mnt/banaNAS/jails/TS3                               
banaNAS/jails/plexmediaserver_1                           11.9G  7.70M  12.5G  /mnt/banaNAS/jails/plexmediaserver_1                 
freenas-boot                                               758M  12.8G    64K  none                                                 
freenas-boot/ROOT                                          757M  12.8G    29K  none                                                 
freenas-boot/ROOT/Initial-Install                            1K  12.8G   755M  legacy                                               
freenas-boot/ROOT/default                                  757M  12.8G   755M  legacy  

Sorry, I had to switch to legacy interface to copy text from shell, I can't figure out how to do that from the new interface. When I am in the legacy interface, the alert system shows the following:

• CRITICAL: Feb. 25, 2019, 10:21 p.m. - Device: /dev/ada5, 16 Currently unreadable (pending) sectors
• CRITICAL: Feb. 25, 2019, 10:21 p.m. - Device: /dev/ada5, 2 Offline uncorrectable sectors
• WARNING: Feb. 25, 2019, 10:21 p.m. - New feature flags are available for volume banaNAS. Refer to the "Upgrading a ZFS Pool" subsection in the User Guide "Installing and Upgrading" chapter and "Upgrading" section for more instructions.
• CRITICAL: Feb. 25, 2019, 10:21 p.m. - The capacity for the volume "banaNAS" is currently at 96%, while the recommended value is below 80%.

 

[danb35]

So you've got about 6 TB in the banaNAS dataset, but that still leaves 14+TB used in the banaNAS pool, not in any dataset. At this point, I'm not sure of a better way to track down where that's being used other than use of du -sh and ls--you don't seem to be using datasets for your shares, which makes it a bit harder to sort out. But you'll want to be looking for data in /mnt/banaNAS that isn't in /mnt/banaNAS/banaNAS, /mnt/banaNAS/iocage, or /mnt/banaNAS/jails. Start with du -sh /mnt/banaNAS/* and go from there.

For CLI access, you're really much better off using SSH.

Don't worry about the feature flags at this point. But it looks like ada5 is dying.

 

[shiphtfour]

Thanks for your help danb35. I can't think of any reason that there would suddenly be new data on the drive making it show 98% capacity when all I did was update. The only thing that comes to mind is that I was trying to use PMS_Updater to update the Plex server in the legacy jail and somehow that created snapshots that I don't know about or something like that, but it doesn't show the data being used in /mnt/banaNAS/jails...

 

[shiphtfour]

/mnt/banaNAS/banaNAS is showing 6.1T used, but that is not where I have stored any of my files. I thought that folder was created by FreeNAS since I have a user with the same name. Could that be the culprit? How do I go about deleting the unwanted data once I find it, just through Windows explorer?

 

[danb35]

Could that be the culprit?

It could be, though you'd need to look at what's in there. If that's a shared folder, the easiest way would be using one of your client computers. And then, yes, if there's data that shouldn't be there, delete it.

 

[shiphtfour]

When I look at the /mnt/banaNAS/banaNAS folder over an SMB share, I do not see any files in there except a handful of 1KB files associated with the FreeNAS user. I have no idea where the 6.1T is coming from... any ideas?

 

[danb35]

I have no idea where the 6.1T is coming from... any ideas?

Possibly snapshots, though nothing should have changed there during an upgrade. Try zfs list -t snapshot | grep banaNAS/banaNAS.

 

[shiphtfour]

zfs list -t snapshot produced

Code:

NAME                                                          USED  AVAIL  REFER                                      MOUNTPOINT
banaNAS/.warden-template-standard@clean                       128M      -  2.64G                                      -
banaNAS/iocage/releases/11.2-RELEASE/root@teamspeak3-server   272K      -  1.45G                                      -
banaNAS/jails/.warden-template-pluginjail-11.0-x64@clean      272K      -   678M                                      -
banaNAS/jails/.warden-template-standard-11.0-x64@clean        464K      -  2.92G                                      -
freenas-boot/ROOT/default@2019-02-26-05:06:49                2.38M      -   760M                                      -

I couldn't get zfs list -t snapshot | grep banaNAS/banaNAS to return anything.

Since this all happened, my disks seem to be constantly running when the server is on, and access to the share is extremely slow. I am not sure if this is due to the max capacity or something else, but it sounds like they're consistently writing or reading something.
I rebooted the server earlier and now I do see extra folders in the banaNAS/banaNAS folder. One of the folders was named "media" and contained duplicates of large media files stored elsewhere on the pool that I am currently deleting. No idea why or how that folder was created, but it must have had something to do with my attempt at updating both the legacy Plex jail and installing the iocage Plex plugin.