hdd drives / backblaze reliability / NAS design

[phier]

Hello,
I am running old freenas with raid5 (4TB) and 3x 8 tb single drives. The setup is not good at all as in case of any 8tb drive failure data is gone, also raid5 is not good setup these days.

I was thinking to have 3-2-1 backup solution, ie Primary NAS (A) , secondary NAS (B for snapshots) and replica of A/B on different geo location.

Does it make any sense to have drives in NAS A (mirrored raid 1)?

Also there are backblaze reports but I am bit confused from them... is there some good advice which drives (brand/size) are the best to go for in NAS?

Appreciate!

 

[Nick2253]

Before I get into your question, a couple terminology points: in ZFS land, there is no RAID1 or RAID5. ZFS has mirrors, stripes, and RAIDZ1, RAIDZ2, and RAIDZ3. Part of why this matters is to avoid confusion between running ZFS on hardware RAID (generally a no-no), and to help be more precise so our language matches up with our tools and software. And finally, using the correct language helps us better understand what's going on behind the scenes, so we can make better architecture decisions to help protect our data.

So, with that being said, I'm still a little confused on your setup. On your current "old freenas" box, do you have multiple zpools? Or do you have a single zpool with 4 vdevs (1x RAIDZ1, and 3x 8TB single drives)?

You definitely want to re-do this setup. Depending on your workload, for NAS A I would recommend RAIDZ2 (space) or striped mirrors (performance). For NAS B, you can probably get by with RAIDZ1. Presumably, NAS B is rather easy to access, so RAIDZ1 allows it to continue operating while you take the few short days of sourcing and installing a replacement disk. If NAS B is more remote, then RAIDZ2 may be necessary.

Of course, it all depends on your risk tolerance. If you're dealing with a lot of bulk data that would be easy to re-create, then RAIDZ1 might be unnecessary. Or if you're dealing with lots of high-value photos and videos, then you might be more inclined to go for RAIDZ3.

3-2-1 backup is the gold standard. But it's not the end-all, be-all. If you are unable to validate your backups, then you could easily propagate bad data, invalidating your backups. And if all your backups are online, then a single attacker could theoretically destroy all your data at once, so you have to make sure that at least one copy of your backup is secure against these kind of attacks (whether that copy is offline, or otherwise inaccessible via two-factor authentication or similar).

Of course, for a home user, 3-2-1 is probably overkill. You have to understand your own risk factors. Personally, I have only 2 copies of my data: my NAS and a cloud backup provider. I use the built-in cloud backup from TrueNAS, so if my TrueNAS were ever compromised, the attacker could access my online cloud backup, and delete that as well. I only keep one copy of my data in the cloud, and it is daily synced to match exactly what's on my NAS, so I rely entirely on snapshots of my data to restore in the event of corruption of accidental deletion. My personal primary risk factor is physical loss of my NAS (fire, hardware damage, etc), and my backup strategy is geared to address that risk.

Backblaze's studies are interesting, but not entirely applicable for a home user. Backblaze puts their drives through load and vibration conditions that only a few home users would ever achieve. As such, what works well for you may not work well for Backblaze. My suggestion for drives is to pick drives that have 5yr+ warranties from "Enterprise" lines, like Seagate's Exos or WD's Gold. NAS-type drives should work well, but enough shenanigans have happened in that space that I would personally steer clear (Google "WD Red SMR" to learn more). In any event, avoid SMR like the plague.

 

[phier]

hi @Nick2253,
appreciate so much your reply.

Before I get into your question, a couple terminology points: in ZFS land, there is no RAID1 or RAID5. ZFS has mirrors, stripes, and RAIDZ1, RAIDZ2, and RAIDZ3. Part of why this matters is to avoid confusion between running ZFS on hardware RAID (generally a no-no), and to help be more precise so our language matches up with our tools and software. And finally, using the correct language helps us better understand what's going on behind the scenes, so we can make better architecture decisions to help protect our data.

So, with that being said, I'm still a little confused on your setup. On your current "old freenas" box, do you have multiple zpools? Or do you have a single zpool with 4 vdevs (1x RAIDZ1, and 3x 8TB single drives)?

Currently I have 4x separate zpools, bc as far as i can remember you cant dynamically extend (add new zpool to the existing setup).

You definitely want to re-do this setup. Depending on your workload, for NAS A I would recommend RAIDZ2 (space) or striped mirrors (performance). For NAS B, you can probably get by with RAIDZ1. Presumably, NAS B is rather easy to access, so RAIDZ1 allows it to continue operating while you take the few short days of sourcing and installing a replacement disk. If NAS B is more remote, then RAIDZ2 may be necessary.

The workload for A is not much, music, movies, files (photos), backup of laptops
B - mirror from A
C - offsite location

A/B might be in the same phy location

I thought that in case one is using B/C there is no need to have raidz2, or B with raidz1 as B,C is basically mirror of A
But bot sure about that thinking.

Of course, it all depends on your risk tolerance. If you're dealing with a lot of bulk data that would be easy to re-create, then RAIDZ1 might be unnecessary. Or if you're dealing with lots of high-value photos and videos, then you might be more inclined to go for RAIDZ3.

What do you mean by bulk data?
All data is high-value - thats why there will NAS B and offsite C.

3-2-1 backup is the gold standard. But it's not the end-all, be-all. If you are unable to validate your backups, then you could easily propagate bad data, invalidating your backups. And if all your backups are online, then a single attacker could theoretically destroy all your data at once, so you have to make sure that at least one copy of your backup is secure against these kind of attacks (whether that copy is offline, or otherwise inaccessible via two-factor authentication or similar).

Understood, yes agree - here how to prevent propagation of bad data - i have no idea here.

Regarding the security - the offsite NAS C will be accessible via VPN (and data will be there encrypted borgbackup or similar) so it should be secure.

Of course, for a home user, 3-2-1 is probably overkill. You have to understand your own risk factors. Personally, I have only 2 copies of my data: my NAS and a cloud backup provider. I use the built-in cloud backup from TrueNAS, so if my TrueNAS were ever compromised, the attacker could access my online cloud backup, and delete that as well. I only keep one copy of my data in the cloud, and it is daily synced to match exactly what's on my NAS, so I rely entirely on snapshots of my data to restore in the event of corruption of accidental deletion. My personal primary risk factor is physical loss of my NAS (fire, hardware damage, etc), and my backup strategy is geared to address that risk.

Why overkill? the reason why i was thinking about 3-2-1 is that i wont need any raid solution on any of these NAS boxes - as data will be replicated from A to B and from B to C. So basically B will be something as raid of A (but on different HW). Again maybe my reasoning is wrong.

What exactly is that built-in cloud backup?

Yes, bc of physical loss thats also why I want to have NAS C.

Backblaze's studies are interesting, but not entirely applicable for a home user. Backblaze puts their drives through load and vibration conditions that only a few home users would ever achieve. As such, what works well for you may not work well for Backblaze. My suggestion for drives is to pick drives that have 5yr+ warranties from "Enterprise" lines, like Seagate's Exos or WD's Gold. NAS-type drives should work well, but enough shenanigans have happened in that space that I would personally steer clear (Google "WD Red SMR" to learn more). In any event, avoid SMR like the plague.

Thats correct, also thats a reason why i was a bit skeptic to pick a drive based on their reports.

Currently I do use WD RED (old ones) i think these are not SMR, but i read that new WD Red are smr, not sure.
I think current models WD RED PLUS are CMR.

So basically truenas doesnt recommend some specific drives for a home solution?
Practically there is no difference between Seagate Exos, WD Gold or WD Red PLUS?

Also not sure about that warranty - if you have to apply to the shop that sell you drive or in case of issue you somehow apply directly to manufacturer.

edit: smr/crm>

https://www.westerndigital.com/en-ie/products/internal-drives/wd-red-pro-sata-hdd#WD161KFGX

Recording Technology: CMR


Also i dont thnik RaidZ1 is good idea these days? Maybe its better to have drives in mirror; ie 2x8TB (mirror), 2x8TB mirror which is total size 16TB. Or maybe instead of these 2 mirrors use raidz2? I am bit confused :(

Also i can see that you use Steel/Iron/Aluminum nas'es for a what purpose 3 setups? Thanks

 

[Nick2253]

Currently I have 4x separate zpools, bc as far as i can remember you cant dynamically extend (add new zpool to the existing setup).

This isn't quite right. You can't expand a vdev, but you definitely can add new vdevs to a pool. If you run the command "zpool status" on your NAS, you'll get a printout of the pools and vdevs.

I thought that in case one is using B/C there is no need to have raidz2, or B with raidz1 as B,C is basically mirror of A
But bot sure about that thinking.

RAID is not backup, and backup is not RAID. Both serve different purposes (though there is some overlap for sure). RAID is about resiliency in the face of hardware failure. Backup is about data protection. As such, RAID helps keep your system up and working when a hard drive fails. This can be the catastrophic loss of a disk, or as small as a bad sector (URE). If any such failure happens and you don't have any kind of RAID, then your NAS effectively goes offline, and you must restore from backup to get your system going again.

For most users, this kind of failure is common enough that it's worth protecting against.

Regarding the security - the offsite NAS C will be accessible via VPN (and data will be there encrypted borgbackup or similar) so it should be secure.

If NAS A or B can access NAS C via stored credentials, then a compromise of NAS A or B is as good as a compromise of NAS C.

Why overkill? the reason why i was thinking about 3-2-1 is that i wont need any raid solution on any of these NAS boxes - as data will be replicated from A to B and from B to C. So basically B will be something as raid of A (but on different HW). Again maybe my reasoning is wrong.

What exactly is that built-in cloud backup?

Yes, bc of physical loss thats also why I want to have NAS C.

Again, RAID is not backup, backup is not RAID.

If you're talking about NAS A and B being real-time mirrors of each other, then that could serve much the same purpose as RAID. But then NAS B really isn't a backup. Furthermore, from a home user perspective, running a NAS cluster really doesn't make sense from a performance or cost-benefit analysis. You don't need the performance benefits of a cluster (most likely), and why run a whole bunch of disks and a second system when you can put just one or two drives in NAS A?

Really, though, what you need to analyze is your failure modes, and acceptable risks. Can you survive your NAS being down for a day? A week? What are the costs, consequences, and likelihood of: hardware failure? HDD failure? Regional catastrophe? Local fire? Etc.

For most home users, a NAS with some kind of HDD redundancy (e.g. RAIDZ2) and a single daily offsite backup meets their needs. This allows the NAS to continue working reliably in the face of a HDD failure (single most common hardware failure), and protects against data loss caused by natural disaster or fire. For users who need extra data protection or to address concerns about corruption, the adding some kind of monthly or yearly offsite/offline backup is usually the third copy needed.

So basically truenas doesnt recommend some specific drives for a home solution?
Practically there is no difference between Seagate Exos, WD Gold or WD Red PLUS?

Also not sure about that warranty - if you have to apply to the shop that sell you drive or in case of issue you somehow apply directly to manufacturer.

In my experience, there is no meaingful difference. I've met people who have had great and terrible experiences with every brand, so I believe that getting a good batch is much more important than picking a good brand or model. Certain drive models have been lemons, for sure, but it's usually not obvious until the drives have been in circulation for a while.

Warranty, if you buy from an authorized seller, comes from the manufacturer.

Also i dont thnik RaidZ1 is good idea these days? Maybe its better to have drives in mirror; ie 2x8TB (mirror), 2x8TB mirror which is total size 16TB. Or maybe instead of these 2 mirrors use raidz2? I am bit confused :(

You've probably seen the "RAID5 is Dead" article. The math used in that article isn't quite right, but the general point is sound: as drives get larger, rebuild times get longer, and the possibility of a URE during rebuild increases. So, the takeaway is that double redundancy is necessary in some cases. However, single redundancy configurations, like mirrors, RAID5, and RAIDZ1, still have their time and place. If you want to be able to fix UREs during a scrub, you need some kind of redundancy. If you use a checksumming filesystem like ZFS, and regularly scrub your data, then the likelihood that you will run into both a URE and a drive failure at the same time substantially decreases. Also, ZFS is pretty resilient in the face of a checksum error, and can tell you exactly which data is corrupt, so you can restore from backups.

In general, a mirror isn't great for a NAS from a cost-benefit perspective: a full 50% of your available capacity is taken up in redundancy. And since most NASes are measured in cost per usable TB, you usually see NASes with 6+ drives, and because resiliency is important, you usually see double redundancy like RAIDZ2.

Also i can see that you use Steel/Iron/Aluminum nas'es for a what purpose 3 setups? Thanks

Iron is currently my main NAS. Steel will replace Iron; I'm in the process of migrating the many jails I was running on Iron (which is why it's still on 9.10) to containers on my Proxmox server. Once that's done, then Iron will be used for a different purpose (likely set up for a friend or family). Aluminum is the destination for my local server and PC backups. A weekly snapshot of these backups is replicated to Iron.

Since FreeNAS 9.10 doesn't have the cloud backup tool, I'm currently using Steel for this function. Daily snapshots from Iron are replicated to Steel, and then Steel pushes these to the cloud. Once Iron is decommissioned, then all the programs that currently "talk" to Iron will go directly to Steel, and I won't have a "middleman" in this role.

Once Steel is running everything, then I'll probably decommission Aluminum as well, and do the PC/server backups directly to Steel. Aluminum is very under-powered for TrueNAS and ZFS, and it struggles in it's current role.

 

[phier]

This isn't quite right. You can't expand a vdev, but you definitely can add new vdevs to a pool. If you run the command "zpool status" on your NAS, you'll get a printout of the pools and vdevs.

@Nick2253
well i think i messed it a bit ... current setup looks like
[root@freenas] ~# zpool status
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0h1m with 0 errors on Fri Apr 22 03:46:06 2022
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
da0p2 ONLINE 0 0 0

errors: No known data errors

pool: storage
state: ONLINE
scan: scrub repaired 0 in 49h47m with 0 errors on Tue Mar 29 02:47:05 2022
config:

NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/6f9e603c-15bd-11e6-940f-0cc47ab3250a ONLINE 0 0 0
gptid/70c0215e-15bd-11e6-940f-0cc47ab3250a ONLINE 0 0 0
gptid/71d80541-15bd-11e6-940f-0cc47ab3250a ONLINE 0 0 0
gptid/72cb0955-15bd-11e6-940f-0cc47ab3250a ONLINE 0 0 0
gptid/73c3c94b-15bd-11e6-940f-0cc47ab3250a ONLINE 0 0 0

errors: No known data errors

pool: storage2
state: ONLINE
scan: scrub repaired 0 in 17h51m with 0 errors on Sun May 1 17:51:59 2022
config:

NAME STATE READ WRITE CKSUM
storage2 ONLINE 0 0 0
gptid/54e8a441-5593-11e7-b888-0cc47ab3250a.eli ONLINE 0 0 0

errors: No known data errors

pool: storage3
state: ONLINE
scan: scrub repaired 0 in 15h14m with 0 errors on Sun Mar 27 16:14:23 2022
config:

NAME STATE READ WRITE CKSUM
storage3 ONLINE 0 0 0
gptid/ae864c15-247f-11e8-acd7-0cc47ab3250a.eli ONLINE 0 0 0

errors: No known data errors

pool: storage4
state: ONLINE
scan: scrub repaired 0 in 11h53m with 0 errors on Sun Apr 3 11:53:38 2022
config:

NAME STATE READ WRITE CKSUM
storage4 ONLINE 0 0 0
gptid/373e2a6b-cdd2-11e9-b7e5-0cc47ab3250a.eli ONLINE 0 0 0

errors: No known data errors

RAID is not backup, and backup is not RAID. Both serve different purposes (though there is some overlap for sure). RAID is about resiliency in the face of hardware failure. Backup is about data protection. As such, RAID helps keep your system up and working when a hard drive fails. This can be the catastrophic loss of a disk, or as small as a bad sector (URE). If any such failure happens and you don't have any kind of RAID, then your NAS effectively goes offline, and you must restore from backup to get your system going again.

For most users, this kind of failure is common enough that it's worth protecting against.

Now got your point so I think i should be protected that way ie raidz1/2 instead of A and B machines which wont bring any benefit.

If NAS A or B can access NAS C via stored credentials, then a compromise of NAS A or B is as good as a compromise of NAS C.

Yeah, assuming here these will be secured ie AB/C will be protected.

Again, RAID is not backup, backup is not RAID.

If you're talking about NAS A and B being real-time mirrors of each other, then that could serve much the same purpose as RAID. But then NAS B really isn't a backup. Furthermore, from a home user perspective, running a NAS cluster really doesn't make sense from a performance or cost-benefit analysis. You don't need the performance benefits of a cluster (most likely), and why run a whole bunch of disks and a second system when you can put just one or two drives in NAS A?

I was thinking to have lets say media/documents etc data primarily on A, and B would hold backup of A; and data as backup of local laptops.
C would act as offsite - ie backup of A and B

But the thing is as you mentioned in case my A or B dies - there will be outage and I will have to restore from C which can take a time.

Also Why I was thinking about B is that my line is 1000/40 so basically I am not sure about offsite from A on daily basis to C; thats why i wanted to have it partially (and locally via LAN on B). Maybe that thinking is incorrect.

Really, though, what you need to analyze is your failure modes, and acceptable risks. Can you survive your NAS being down for a day? A week? What are the costs, consequences, and likelihood of: hardware failure? HDD failure? Regional catastrophe? Local fire? Etc.

For most home users, a NAS with some kind of HDD redundancy (e.g. RAIDZ2) and a single daily offsite backup meets their needs. This allows the NAS to continue working reliably in the face of a HDD failure (single most common hardware failure), and protects against data loss caused by natural disaster or fire. For users who need extra data protection or to address concerns about corruption, the adding some kind of monthly or yearly offsite/offline backup is usually the third copy needed.

What do you mean by extra data protection, or how do you want to protect against the corruption?

So in the backup pattern
3-2-1 ...
1-2 is online replica/mirror physically separated
3 is offsite?

In my experience, there is no meaingful difference. I've met people who have had great and terrible experiences with every brand, so I believe that getting a good batch is much more important than picking a good brand or model. Certain drive models have been lemons, for sure, but it's usually not obvious until the drives have been in circulation for a while.

Warranty, if you buy from an authorized seller, comes from the manufacturer.

I will have to find out somehow if they are authorized sellers.
If warranty comes from manufacturer it means in case of failure you send drive to manufacturer for a replacement?

I saw some sellers selling drive with standard warranty and then option to extend that by 2-3y... but that extension contract is between me and seller so no clue if manufacturer will be able to step-in.


I can see your "Steel" is using HGST Ultrastar drives, which you didnt mention, for a some specific reason?

You've probably seen the "RAID5 is Dead" article. The math used in that article isn't quite right, but the general point is sound: as drives get larger, rebuild times get longer, and the possibility of a URE during rebuild increases. So, the takeaway is that double redundancy is necessary in some cases. However, single redundancy configurations, like mirrors, RAID5, and RAIDZ1, still have their time and place. If you want to be able to fix UREs during a scrub, you need some kind of redundancy. If you use a checksumming filesystem like ZFS, and regularly scrub your data, then the likelihood that you will run into both a URE and a drive failure at the same time substantially decreases. Also, ZFS is pretty resilient in the face of a checksum error, and can tell you exactly which data is corrupt, so you can restore from backups.

Yeah, exactly that article.
In that case is RAIDZ1(RAID5) enough for a home use? Using ZFS and ECC, or do I have to go for 2 parity drives ie raid6(z2)?

In general, a mirror isn't great for a NAS from a cost-benefit perspective: a full 50% of your available capacity is taken up in redundancy. And since most NASes are measured in cost per usable TB, you usually see NASes with 6+ drives, and because resiliency is important, you usually see double redundancy like RAIDZ2.


Iron is currently my main NAS. Steel will replace Iron; I'm in the process of migrating the many jails I was running on Iron (which is why it's still on 9.10) to containers on my Proxmox server. Once that's done, then Iron will be used for a different purpose (likely set up for a friend or family). Aluminum is the destination for my local server and PC backups. A weekly snapshot of these backups is replicated to Iron.

Since FreeNAS 9.10 doesn't have the cloud backup tool, I'm currently using Steel for this function. Daily snapshots from Iron are replicated to Steel, and then Steel pushes these to the cloud. Once Iron is decommissioned, then all the programs that currently "talk" to Iron will go directly to Steel, and I won't have a "middleman" in this role.

Got your point, but still not clear what cloud do you mean or what kind of tool/sw/mechanism is used (in truenas v12) for such a push to the cloud, as clouds like dropbox, backblaze dont provide good interface for offsite-backups (When i did research... )

So apart of the NAS boxes, you set up a separate HW/box for Proxmox server? Just asking bc I am also running some virtual machines on bhyve but have to move/migrate all of the off the NAS if possible.

Once Steel is running everything, then I'll probably decommission Aluminum as well, and do the PC/server backups directly to Steel. Aluminum is very under-powered for TrueNAS and ZFS, and it struggles in it's current role.

Also there is another thing ... to pickup reliable tool for win/linux/osx that can backup to NAS.


More about specific setup / scenario
I have:
9x 3TB drives
3x 8TB drives
so I was thinking to do following setup

RAIDZ2: 7x 3TB total 15TB
RAIDZ1: 3x 8TB total 16 TB
to buy 1x 16TB (not important data ie backups from laptops, movies/music etc - that can be restored from offsite within 1 week and i can live without)
Z2 and Z1 will store the critical data like VM images, documents, photos ... , data from various working projects.
Here I am not sure which pool suits better (more integrity, resilient etc) for *the most critical data* ie. 15TB or 16TB volume?


Then Offsite machine (was thinking about HW without ECC memories - which mean some check to prevent mentioned data corruption has to be done, on SW level? Or maybe design is wrong and that machine has to come with ECC also)

8TB smr (have the old one) + 2x 3TB - to partialy backup 15TB
2x new 16TB - to backup RAIDZ1, and single 16TB



Again, maybe i am completely missing something here, so happy to get tips/ideas.

Appreciate so much.


Theory>https://calomel.org/zfs_raid_speed_capacity.html
Be Safe, Not Sorry. When choosing a raid configuration you may look at raidz or RAID5 and see the speed and capacity benefits and decide it is a good choice. From real world experience we highly suggest NOT using raid5. It is simply not safe enough. We recommend using RAID1 mirroring or RAID6 (double parity) or even RAID7 (triple parity). The problem with raid5 is you only have one drive with parity. When one drive dies the raid5 array is degraded and you can not loose another drive before you lose the entire array; i.e. all your data is lost. What happens most of the time is one drive dies, you replace the drive and the array starts resilvering or rebuilding. This is when the other disks are being stressed and during this rebuild is when you have a very good chance (around 8%) that you you will lose another drive. For more information please take some time to read NetApp Weighs In On Disks which was written in 2007, but is still valid today since drive technology has not changed much.

 

[Nick2253]

We're getting a little unwieldy in the thread, so I'll try to condense and try to answer the important points. If you feel like I've not touched on something that you would like addressed, please let me know and I'll comment:

RAIDZ1 vs RAIDZ2 - Short answer, in my opinion, that the cost of an extra drive is usually pretty insignificant against the cost of a build. As such, I'd almost always recommend at least RAIDZ2 for data storage. The only time I'd ever support anything different is:

• You have an overriding need for performance, and so you go with striped mirrors.
• You have done a full analysis of your own risk factors, data backup plan, and recovery plan, and have *affirmatively* decided that more risk is acceptable.

Client/Server Backups - I use UrBackup. The interface is a bit clunky, but it works very well. I do image backups of my clients so I can quickly restore them if their drive dies, and I do file backups of my servers so I can restore key configuration files or DB files if needed.

Remote Backups - TrueNAS comes with a built-in cloud sync tool that makes online backups a breeze. I personally use the Backblaze B2 cloud because of its low cost/TB. Your network speed of 1000/40 should be plenty to do offsite backups. Your initial backup will take a while (assuming ~10TB at 40Mbps, you're looking at 23 days), but once you have everything at the destination server (or in the cloud), then all you'll need to do is incremental backups, which (depending on your data use) should be doable in the course of a few hours every night. For a point of comparison, I have 200/10 internet, and have nightly incremental backups of about 10 GB, which takes around 2 hours to upload.

Data Integrity - There are three sources of corruption: changes to data at rest (aka UREs, bitrot), changes to data in transit (controller issues, RAM issues), or unauthorized changes (virus, user error). ZFS is extremely resilient against the first one, moderately resilient against the second, and offers no protection against the third. ECC memory and quality hardware is the best way to build resilience against corruption due to data in transit. The only way to protect against the third is to use some kind of data validation, which is a tricky problem. Or, have lots of backups so that you can return to a point-in-time before the unauthorized changes were made.

Hypervisor - I have a cluster of 3 ProxMox servers for all of my VMs. The hardware that powers these machines is dual-socket Dell T7500 workstations.

---

You asked about my HGST drives: there are lots of other manufacturers of drives. Those were in no way recommendations for those specific models or brands, just easy-to-google brand/models that will help you identify the correct class. HGST, Toshiba, Samsung, and others make fantastic hard drives.

For your specific setup, we haven't really talked about how much data you need to store. The quantity and type of data will influence the optimal pool design. However, in any case, I'd strongly recommend that you:

• BURN ALL OF YOUR SMR DRIVES WITH FIRE.