Storage Usage Question - 36 TiB installed, 24TiB available

[SweetAndLow]

Looks correct to me. Why do you think it's wrong?

You need to subtract 2 drives worth of space for parity. Then subtract the 2GB of swap on each drive. Then convert from TB to TiB. Should be about 24TiB I think.

 

[jde]

Looks correct to me. Why do you think it's wrong?

Freenas reports the gross storage space of his disks as 32.5 TiB. Accounting for raidz2 parity by multiplying that value by 10/12 yields 27.08 TiB. This result is in line with Bidule0hm's calculator results. However, freenas only reports 24 TiB of space on the OP's pool. It looks like he's missing ~3 TiB of space.

 

[Roger Mahler]

36 TB = 32.7 TiB
6 Tb = 5.45 TiB
20 G = .11 TiB

I would have thought i would have 27.14 (not exact but very close)

jde has it, i am missing approximately 3TB/2.72 TiB worth of storage. the concern is i have/system has a problem somewhere and may be a unstable, Just trying to get it right. but i think we are getting close, thank you very much for the guidance.

roger...

 

[Mlovelace]

Check the sector count for each disk and make sure they are all the same. The pool will use the lowest common denominator for space so if one of the drives is smaller that might explain why the pool is short. Maybe something more complex but I'd look there first.

 

[Mlovelace]

This is interesting shows the padding is about 2.9TiB for your pool setup. Might explain the missing space.

https://web.archive.org/web/2014040...s.org/ritk/zfs-4k-aligned-space-overhead.html

 

[Roger Mahler]

wow, very interesting. Thank you...

I need to start using the system, but am only going to put recoverable content on it for now. I will see how this works, and if the results are consistent. I am about to build a 6 bay array in the next month or so using FreeNAS. I will report on outcome.

Thank you everyone who guided me in this, i greatly appreciate all your support and information.

 

[Bidule0hm]

You may be interested by this thread: https://forums.freenas.org/index.php?threads/misaligned-pools-and-lost-space.40288/ ;)

 

[topping]

Hi guys, this is a very interesting thread as I am having the same problem with a significantly smaller setup. I read the content here and the content Bidule0hm linked from in the last message. Recordsize property is set to 128K and ashift is 12.

My zpool status:

Code:

  pool: freenas-boot
state: ONLINE
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      da0p2     ONLINE       0     0     0

errors: No known data errors

  pool: tank
state: ONLINE
  scan: resilvered 1.08T in 4h1m with 0 errors on Thu Feb  4 18:52:06 2016
config:

    NAME                                            STATE     READ WRITE CKSUM
    tank                                            ONLINE       0     0     0
      raidz1-0                                      ONLINE       0     0     0
        gptid/840c8b44-cadd-11e5-8946-b05ada8759c0  ONLINE       0     0     0
        gptid/405bbeb0-cb17-11e5-b302-b05ada8759c0  ONLINE       0     0     0
        gptid/4599f69b-cb89-11e5-9d7d-b05ada8759c0  ONLINE       0     0     0
        gptid/c6c6081e-cb58-11e5-a6b7-b05ada8759c0  ONLINE       0     0     0

errors: No known data errors

camcontrol devlist:

Code:

<ST6000DX000-1H217Z CC48>          at scbus0 target 0 lun 0 (ada0,pass0)
<ST6000DX000-1H217Z CC48>          at scbus1 target 0 lun 0 (ada1,pass1)
<ST6000DX000-1H217Z CC48>          at scbus2 target 0 lun 0 (ada2,pass2)
<ST6000DX000-1H217Z CC48>          at scbus3 target 0 lun 0 (ada3,pass3)
<Kingston DataTraveler 2.0 1.00>   at scbus7 target 0 lun 0 (pass4,da0)

According to Bidule0hm's math, a raidz1 with four disks, ashift=12 and 128kb records should be looking at 40 sectors for a data block, which is a clean multiple of raidz1 + 1. The volumes view shows 17.5TiB available for the volume, but then drops to 12.2 available for the only dataset on the volume:

Apologies if I am missing something obvious here, but why would the only dataset be missing over 5TB of space that is available in the volume? Are there other elements that I can check?

 

[Bidule0hm]

According to Bidule0hm's math, a raidz1 with four disks, ashift=12 and 128kb records should be looking at 40 sectors for a data block, which is a clean multiple of raidz1 + 1.

Nop, actually you'll have 42.667 sectors so ZFS will add padding until you reach 44 sectors and (44 - 42.667) / 42.667 = 0.0312 so you have a 3.12 % overhead.

You start with 18 (+ 6 of parity) TB raw. Then there's the TB -> TiB conversion so you have 16.37 TiB. Then there's 0.51 TiB if misalignment overhead (3.12 %) so you have 15.86 TiB. Then there's 0.26 TiB of metadata overhead (1.56 %) so you have 15.60 TiB usable (well, in practice you need to add the 80 % rule to have the real usable space). The GUI says 3.1 TiB used and 12.2 TiB available so it's a 15.3 usable space. There's only 0.3 TiB difference and some of it is probably because of the rounded values of the GUI so everything is fine ;)

The volumes view shows 17.5TiB available for the volume, but then drops to 12.2 available for the only dataset on the volume:

It's normal, please read this post: https://forums.freenas.org/index.php?threads/the-faq.30209/#post-194030 ;)

 

[topping]

Excellent, thank you! I think I was in the same boat of wanting to make sure that I wasn't building on an unstable foundation. I'm really grateful for you doing the math on my numbers, it obviously helps immensely to be able to apply it properly.

It's normal, please read this post: https://forums.freenas.org/index.php?threads/the-faq.30209/#post-194030 ;)

This FAQ says "The first one (pool) includes the parity space but the second one (root dataset) only refers to the data space.". I think what you wrote above is a far better explanation, I had to re-read both a few times to recognize I simply didn't understand what that meant, for whatever that's [not] worth. :)

 

[Bidule0hm]

Yep, but the FAQ is meant to be kept (is that proper english?) short. However I admit it can probably be better worded than it is, I just need time... more time... :)

Yep, I'll include the calculation in my calculator so you don't have to do it manually but, as always, the time...

 

[Ericloewe]

(is that proper english?)

Yup.

 

[Rick Henriksen]

Configuration
Server: IBM DX360-M3
Disks: 12+3TiB
RaidZ2
LSI SAS9210-8i
Memory: 24g

Install was almost seamless, struggled a little with configuration but mostly misunderstanding on my side.

Question: the top freenas mount shows 32.5 TiB, a little high from past experiences but ok, however when i get down to the data drives it shows available of 24TiB - quite a loss, is this normal. Wanted to know before loading up with data, i will be moving close to 15 TiB almost immediately.

Thank you.

Hi Roger. I just received the same IBM server and SAS card that you have. Using a USB hub, I was able to connect a USB keyboard, install image USB stick, and install destination USB stick. I set the BIOS boot order to boot from the USB install image stick. The installation image boots perfectly and then I installed onto the other USB stick from there. Then, I removed everything and re-insrted the newly installed stick into the usb slot and rebooted. At that point, Freenas does not boot successfully. It complains about an unrecognized device in GRUB. I was wondering if you would be able to give a quick summary of the steps you used to get Freenas installed. F.Y.I. I have tried both the UEFI and BIOS boot option selections during the install. Thanks

 

[SweetAndLow]

Hi Roger. I just received the same IBM server and SAS card that you have. Using a USB hub, I was able to connect a USB keyboard, install image USB stick, and install destination USB stick. I set the BIOS boot order to boot from the USB install image stick. The installation image boots perfectly and then I installed onto the other USB stick from there. Then, I removed everything and re-insrted the newly installed stick into the usb slot and rebooted. At that point, Freenas does not boot successfully. It complains about an unrecognized device in GRUB. I was wondering if you would be able to give a quick summary of the steps you used to get Freenas installed. F.Y.I. I have tried both the UEFI and BIOS boot option selections during the install. Thanks

Double check your boot order didn't change.

Sent from my Nexus 5X using Tapatalk