Disk size and type

[TDi39]

Hello,

I'm on the way to build my system, it's gonna look in this way:
- CPU 8/16 threads (2018 XEON probably)
- 128GB ECC DDR4
- The rest will be accordingly
- Disks | I have a few options:
- 3 disks in RAID 1 (I don't mean RAID 1, but the equivalent in ZFS, I just need to get familiar with the names)
- 4 disks in RAID 6 (2 drives failure)
- 6 disks in RAID 5 +0 (2 drives failure)
- There are also other combinations...
- Each disk will be minimum 12 TB, but I'll try to get 16 TB if I find a good price.

I'll use my NAS for projects, media etc, but also VM backups, basically data that I don't want to lose.
A lot of writing every day, let's say 100-150GB a day but big blocks, not a big drama for an HDD...
If it's better, I can just take snapshots on daily basis, but backups once a week for example... Something tells me that this is better... I just need to find a proper strategy.

I'll use VMs on the same systems, but they'll run on the M2, only the backups will be on the NAS.

Cost wise, more than 4 disks will become too expensive, so the doubt is between RAID 1 or RAID 6.
I think that RAID 1 it's absolutely fine for me, but RAID 6 would give me more reliability and availability. What do I mean with this? I mean that if one disk brakes when it's writing, the last writing job (as a whole) is loss and it should restart from scratch, this doesn't happen in a RAID 6 if I'm not wrong, you would lose only part of it.
Let's be clear, it's not crucial to me, I don't run an enterprise, but it's dangerous because I'm not a storage expert, so more I stay away from these issues, better it is.
This could be a problem if I rely on many snapshots and from different machines, if they all run at the same time, this issue increases the risk of data "loss" or corruption/incomplete, but I should also break the other machine at the same time, quite rare... In any case, with daily backups/snapshots I will be "fine".

Where's my real backup?
On a secondary disk, powered by the NAS anyway which is powered by an UPS.
Plus my cloud account, but I think I should use another service for TrueNAS just to make it easy...

So, how many disks you suggest?
What config?
What disk types?
I have experience mainly in consumer class HDDs, example: one brand never breaks (I've seen at least 30 disks), another one is able to break the entire machine :D , even though people keep saying that it's great, well probably in enterprise.......... It's always the same brand that breaks in consumer class :D , it can't be a coincidence...
I wonder what brands are reliable in enterprise, considering that I'll buy the enterprise level HDDs 7200 rpm (CMR).

--------
Off topic
The cluster will cost me 22 pounds of energy per month average (with the current prices estimated for 90W average 24/7), any suggestion on how to lower down consumption? :D
I know something about UEFI config, but I wonder if the XEON could have better C States or similar options...
Unfortunately, I decided to build only one machine, instead of having one NAS and one cluster (used for VMs for development), it's simply cheaper (on the day 1), but not that much more expensive if I had 2 machines instead, even on the long run, so I thought it was a better idea.
Plus the energy prices should go down in future, probably as before at some point, which could lower the cost at 13 pounds monthly.

 

[Davvo]

Oh gosh... Those layout names. Please read the following resources.

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to...

www.truenas.com

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com

https://www.ixsystems.com/wp-content/uploads/2022/02/ZFS_Storage_Pool_Layout_White_Paper_February_2022.pdf

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

For the HDDs, the sweet spot should be around 14TB if I am not wrong. Either mirrors or RAIDZ2 should be your layout of choice.
Anyway, you want CMR drives and not SMR ones. Do pay attention to this since some brands try to sneak you SMR drives as NAS-worthy.
WD and Seagate are common choices on this forum as manifacturers. For Enterprise-grade drives I am not too sure about the line of drives you should go for.
Also usually lower RPM drives are preferred since they produce less heat/noise and consume less.

 

[TDi39]

Oh gosh... Those layout names. Please read the following resources.

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to...

www.truenas.com

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com

https://www.ixsystems.com/wp-content/uploads/2022/02/ZFS_Storage_Pool_Layout_White_Paper_February_2022.pdf

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

For the HDDs, the sweet spot should be around 14TB if I am not wrong. Either mirrors or RAIDZ2 should be your layout of choice.
Anyway, you want CMR drives and not SMR ones. Do pay attention to this since some brands try to sneak you SMR drives as NAS-worthy.
WD and Seagate are common choices on this forum as manifacturers. For Enterprise-grade drives I am not too sure about the line of drives you should go for.
Also usually lower RPM drives are preferred since they produce less heat/noise and consume less.

Thanks, I'll read those guides.

Why did you suggest disks with less RPMs? I wasn't gonna take something beyond 7200 RPMs, is that even too much?

 

[NugentS]

Less RPM = less heat, less wear and tear and less electrickery (at least in theory)

Apparently makes very little difference to performance

 

[TDi39]

Less RPM = less heat, less wear and tear and less electrickery (at least in theory)

Apparently makes very little difference to performance

Thanks, but are you suggesting disks with 5400 RPMs?
I've never seen one so far with double digits TB...

 

[NugentS]

Honestly, I have never looked/ I just buy the cheapest enterprise/NAS drives I can whilst tending to avoid Western Digital

 

[artlessknave]

zfs is easy
stripe is no parity, similar to raid0
mirrors is 1/2 parity, similar to raid1
raidz1 is 1 parity disk, similar to raid5
raidz2 is 2 parity disk, similar to raid6
raidz3 is 3 parity disk, i don't think there is a raid equivalent, but maybe raid7?

neither raid5 nor RAIDz1 are recommended, anywhere, with drives over ~2TB, with data you care about. if you don't already know where and when you can use them, you shouldn't be using them.

I do not believe raidz expansion in in yet, so if you can fit 6 drives, and afford 6 drives, get all 6 drives and do raidz2 (~75% usable space, can't change number of drives per vdev) or mirrors (50% usable space, CAN change the number of drives per vdev).

 

[TDi39]

zfs is easy
stripe is no parity, similar to raid0
mirrors is 1/2 parity, similar to raid1
raidz1 is 1 parity disk, similar to raid5
raidz2 is 2 parity disk, similar to raid6
raidz3 is 3 parity disk, i don't think there is a raid equivalent, but maybe raid7?

neither raid5 nor RAIDz1 are recommended, anywhere, with drives over ~2TB, with data you care about. if you don't already know where and when you can use them, you shouldn't be using them.

I do not believe raidz expansion in in yet, so if you can fit 6 drives, and afford 6 drives, get all 6 drives and do raidz2 (~75% usable space, can't change number of drives per vdev) or mirrors (50% usable space, CAN change the number of drives per vdev).

Thanks for the explanation, but I do have a question on the last part.

What do you mean with "can't change the number of driver per vdev"?
Do you mean that I can't add disks when using RAIDZ2?

Also, with "mirrors" as you mentioned, I'd expect less than 50% capacity, like only one disk totally used and all the others in parity (5 in parity).
But it seems that it works in another way, that is, half of the disks is a sort of RAID0, then the others are mirrored to the rest of the 3 drives, but this is risky...

Anyway, 6 drives will become seriously expensive, probably 1800 pounds only for the disks, no way...

 

[artlessknave]

Also, with "mirrors" as you mentioned, I'd expect less than 50% capacity, like only one disk totally used and all the others in parity (5 in parity).
But it seems that it works in another way, that is, half of the disks is a sort of RAID0, then the others are mirrored to the rest of the 3 drives, but this is risky...

I changed my original wording, as mirrors aren't "parity"; they are identical, each disk holding the same data. a set of mirrors can be split, givnig you 2 pools of the same data, but thats advanced CLI.
mirrors are exactly that. mirrors. each disk in the vdev is made identical. 2 ways mirrors is 50% used on redundancy. 3 way mirrors is 66%. technically you can do as many mirrors as you want but over 3 is just silly.

basic zfs reading required, but a summary:
a vdev is the base group of disks. a pool must have at least one vdev. zfs redundancy is done at the vdev level. a pool stripes data across all vdevs in the pool.

stripe, mirror, and raidz are all vdev types.
a stripe can be turned into a mirror by attaching disks, or a mirror turned into a stripe by detaching disks

a raidz vdev, once made, cannot be changed (yet - in development), so if you make a raidz2 out of 4 disks, it will always be a 4 disks raidz2
you can, however, expand a pool by adding new vdevs. so you can have a pool of 4 raidz2 vdevs of 4 disks.
you can also replace all the disks in the vdev with larger (cannot be smaller) disks and the vdev can expand in size to the size of the smallest drive
(a raidz2 with 4TB|4TB|4TB|2TB will be 2TB until that last disk is upgraded)


oh, i forgot, as well, that multiple mirrors is similar to RAID10 (a stripe across mirrors)
raidz with mulptiple vdevs would be similar to RAID 50, or RAID 60. raid70?

 

[jgreco]

I wasn't gonna take something beyond 7200 RPMs, is that even too much?

I don't know if anyone is even making 10K or 15K RPM HDD's anymore. Those workloads have usually been shifted to SSD.

 

[Davvo]

It would be better for you to gain a comprehensive understanding of how TrueNAS and zfs work. In order to do so, please do read the resources I linked earlier.

At least the Introduction.
Mistakes here can be very expensive and/or painful.