3 Drive Configuration

[McNinjaTNT]

I plan to have 3 16TB drives, 2 of which I want to be mirrors. What's the best config for this? RAIDZ2 is a option, but somehow that's not ideal (If someone could explain that to me that would be nice, because it seems to me as if that's optimal, I mean logically if you have a 16TB pool and want to mirror it twice isn't 2 extra 16TB drives perfect for that?). Also getting more than three drives is not ideal financially.

 

[Heracles]

Hi,

With only 3 drives, RaidZ2 is clearly not an option... If you are ready to have the storage space of a single drive, better to do a 3 way mirrors. By doing so, read access will be a lot faster.

The other option would be RaidZ1 to have the storage capacity of 2 drives but with such large drives, it is not recommended. Whenever a resilver will be needed, both of the drives and the total of 32TB will HAVE to be completely error-free because there will be no way to fix any error on the fly once down by 1 disk.

So here, I would do either a 3 way mirrors or do the extra mile for a 4th drive (or more if possible). You would be better with 4x 8 or 10TB instead of 3x 16TB. Also, be sure not to end up with SMR drives...

 

[McNinjaTNT]

Hi,

With only 3 drives, RaidZ2 is clearly not an option... If you are ready to have the storage space of a single drive, better to do a 3 way mirrors. By doing so, read access will be a lot faster.

The other option would be RaidZ1 to have the storage capacity of 2 drives but with such large drives, it is not recommended. Whenever a resilver will be needed, both of the drives and the total of 32TB will HAVE to be completely error-free because there will be no way to fix any error on the fly once down by 1 disk.

So here, I would do either a 3 way mirrors or do the extra mile for a 4th drive (or more if possible). You would be better with 4x 8 or 10TB instead of 3x 16TB. Also, be sure not to end up with SMR drives...

The drive prices for 16TB drives (specifically Exos) are pretty decent compared to 10TB I think, thus why I want 16TB.

Are there any downsides to a 3 way mirror? Would it allow two out of the three drives fail without incident? Also if you know any documentation on it that would help a lot.

Also with RAIDZ, adding drives to a pool later requires resilvering, does a mirror/3 way mirror require that or no?

Thank you for the help, I'm very new to all this, sorry if I'm asking dumb questions.

 

[Heracles]

Would it allow two out of the three drives fail without incident?

Indeed, a 3-way mirror can loose 2 of drives out of its 3. It would then drop to no redundancy but will survive.

Also with RAIDZ, adding drives to a pool later requires resilvering,

No it does not... You can create a brand new vDev and add it to the pool. That process does not involve or require a resilver. This is the same thing with mirrors : you can create a new mirror vDev and add it to the pool at a latter time. To add a 3rd disk to a 2 disks mirror, turning it to a 3-way mirror, that would require a re-silver. Of course, do not add a single drive as a single drive vDev because that would jeopardize your entire pool and would be fixable only by destroying the pool and re-creating it. You need to add another RaidZ or mirror vDev when doing that.

The drawback of a 3-way mirror is that you have the usable space of only 1 drive out of your 3. With only 1 vDev, you have the write speed of only 1 drive. The speed gain is for the reads only.

The price tag per storage unit is one thing. The possibility from the number of drive is another. With 3 disks, you can not do much. Not enough drives for RaidZ2 and too large drives for RaidZ1. As such, you are down to the 3-way mirrors.

Should you be able to add a 4th drive, you can then go RaidZ2 and have 2 usable drives instead of 1. With 5 or 6 drives, that would be even better : 4 usable drives out of 6 (2/3) with still a 2 drives redundancy solution. That is the double when compared to the 3-way mirror. That is why you may be better with 6x 8TB instead of 3x 16TB.

With more drives, you also have the option to create more vDev for more IOPS, so more performance.

 

[Etorix]

To avoid any misunderstanding, @Heracles means "adding" a 4th (5th, 6th, etc.) drive at the time of creating the pool. You cannot create a pool and then add drives later or magically change from mirror to raidzN; you can only add full vdevs of the same kind as the initial vdev(s), so you need to get design right the first time.

 

[McNinjaTNT]

Indeed, a 3-way mirror can loose 2 of drives out of its 3. It would then drop to no redundancy but will survive.



No it does not... You can create a brand new vDev and add it to the pool. That process does not involve or require a resilver. This is the same thing with mirrors : you can create a new mirror vDev and add it to the pool at a latter time. To add a 3rd disk to a 2 disks mirror, turning it to a 3-way mirror, that would require a re-silver. Of course, do not add a single drive as a single drive vDev because that would jeopardize your entire pool and would be fixable only by destroying the pool and re-creating it. You need to add another RaidZ or mirror vDev when doing that.

The drawback of a 3-way mirror is that you have the usable space of only 1 drive out of your 3. With only 1 vDev, you have the write speed of only 1 drive. The speed gain is for the reads only.

The price tag per storage unit is one thing. The possibility from the number of drive is another. With 3 disks, you can not do much. Not enough drives for RaidZ2 and too large drives for RaidZ1. As such, you are down to the 3-way mirrors.

Should you be able to add a 4th drive, you can then go RaidZ2 and have 2 usable drives instead of 1. With 5 or 6 drives, that would be even better : 4 usable drives out of 6 (2/3) with still a 2 drives redundancy solution. That is the double when compared to the 3-way mirror. That is why you may be better with 6x 8TB instead of 3x 16TB.

With more drives, you also have the option to create more vDev for more IOPS, so more performance.

I see, this all makes much more sense now, thank you for the detailed explanation and help.

To avoid any misunderstanding, @Heracles means "adding" a 4th (5th, 6th, etc.) drive at the time of creating the pool. You cannot create a pool and then add drives later or magically change from mirror to raidzN; you can only add full vdevs of the same kind as the initial vdev(s), so you need to get design right the first time.

Gotcha, so if I were to add drives after creation, I would have to get 3 more, and add another vdev identical to the first correct? (1 primary, 2 mirroring)

 

[Etorix]

Correct. 3-way mirrors are expensive but quite safe, and pools with multiple mirrors ("stripes of mirrors" in ZFS terminology, raid10 outside of ZFS) perform very well, in particular for block storage such as VMs.
If your use case is data storage for family use (limited number of clients), raidz2 would be a better choice cost-wise with adequate performance but that would require 4 drives or more (say 4 to 8, the more drives, the more cost-effective). You may want to check and rate your options in terms of $/usable TB before committing to 3*16TB (16 TB as mirror) vs. n*10-12TB in raidz2 (space of n-2 drives).

 

[Heracles]

you can only add full vdevs of the same kind

Technically, this is not exact. It is possible to add a mirror vDev in a pool that contains a RaidZ vDev. It is not recommended but it is possible.

Usually, it is recommended to keep the same type and sizing for all your vDev : all mirrors, all Raid-Z2 with the same number of drives, .... If you have a first 5 drives RaidZ2 vDev, you can add a second 5 drives RaidZ2 vdev anytime, without resilvering and still be in a good enough situation. The ideal will always be to build the pool at once / rebuild it when modifying it, but such increases are possible.

 

[McNinjaTNT]

Technically, this is not exact. It is possible to add a mirror vDev in a pool that contains a RaidZ vDev. It is not recommended but it is possible.

Usually, it is recommended to keep the same type and sizing for all your vDev : all mirrors, all Raid-Z2 with the same number of drives, .... If you have a first 5 drives RaidZ2 vDev, you can add a second 5 drives RaidZ2 vdev anytime, without resilvering and still be in a good enough situation. The ideal will always be to build the pool at once / rebuild it when modifying it, but such increases are possible.

Hmm, so if I did a 3 way mirror, then added another 3 drives and did another 3 way mirror with those that would NOT work then? And would only work if the original setup was RAIDZ2 correct?

 

[Heracles]

if I did a 3 way mirror, then added another 3 drives and did another 3 way mirror with those that would NOT work then?

It will work. You can create a pool with only a single 3-way mirror and at a later date add another 3-way mirror to the very same pool. The pool will then extend according to the size of that new mirror. Such a pool will be and remain safe and stable.

It is not ideal because at first, all your data will be on your first mirror. Once the second mirror is added, all new data will go to that second mirror until its % usage is re-balanced with the first vDev. Once the 2 vDevs are about the same %, only then will the load be divided between the 2. If you have both mirrors from the beginning, everything will be balance between the 2 from the beginning.

What is possible but not recommended is to start with a mirror and add a RaidZ vDev at a later date.

What you can not do is start with a 3 way mirror and then convert it to a RaidZ. That is possible only by destroying the pool completely and rebuilding it from scratch.