SOLVED Advice regarding ZFS: number of disks and RAID level

[zamana]

Hi!

About 6 months ago I went from the "previous" to the "current" scenario:

I'm considering to backup all my data (13TB right now) and do some refactory on my storage. My intention with this refactory is to increase the resilience by one way or another. Also, I'm not considering to increase the data volume indefinitely. At some point I'll discard some very old data.

I know that there is no unique, final, scientifc answer to my question, but I would like to know what you would do in this particular situation.

For example: if I downgrade to 8 disks, I can have 2 disks as spare, in the case of failure. If I keep all the 10 disks but increase the level to RAID-Z3, I can keep all disks working and increase the storage's relilience to 3 failures instead of 2.

And before you say, I don't like the MIRROR scenario (RAID10). The possibility to lose 2 disks, one of each side, scares me; I prefer to take the risk to lose "any" 2 disks (or 3).

My motherboard is a Supermicro, model X11SSL-CF (6 sata ports and 8 sas ports), and the case is an NZXT H-440 (with room for 10 HDDs of 3.5' and 2 of 2.5'). All disks are Seagate, 4TB.

Thanks.
Regards.

 

[Chris Moore]

I would suggest that you have a cold spare or two, but the current state looks good to me.

 

[Constantin]

Many folk here seem to like Z2 redundancy for local installations. The argument being that losing more than a disk at once is very rare and every user is expected to keep a proven spare for the eventuality. I.e. you have a spare drive on a shelf that you've previously burned in. That spare drive allows you to get the array back up / resilvered in a hurry.

The more paranoid Z3 option (which I use) isn't even available via the regular pool creation menu, you have to go manual. I chose that layout with my old set of 3TB drives, ditto with my current set of 10TB drives. I also mirrored my SLOG drive.

It comes down to preferences and how confident you are re: your backups. I'd rather sacrifice the added capacity of a drive for Z3 redundancy, but that is my preference. You may decide that Z2 is good enough and gives you more capacity. Additionally, you always have the option to increase the capacity of your pool over time by replacing extant drives with bigger ones, resilvering, and then doing it with the next one. Once the drive cohort for the whole pool has been replaced, it will expand automatically.

Either way, always have burned-in, proven spare(s) on hand. I try to keep at least one drive in reserve.

 

[Chris Moore]

The more paranoid Z3 option (which I use) isn't even available via the regular pool creation menu,

I am sure that was once available in the menu because I built a RAIDz3 pool some time back and I am sure I did not need to use manual creation to do it. It might be a bug of sorts, that it is not in the standard pool creation tool. Have you submitted a bug report?

 

[Chris Moore]

I will add this to the discussion between RAIDz2 and RAIDz3.

I have a server at work where the storage drives are 6TB WD RedPro drives and the drives are around half full of data (I can't recall exactly) but that should be around 3TB of data. When I have had to replace one of those drives, the resilver has taken around four days and I have not understood why it was so slow. Here is why... The comparison I have is, on my system at home, where my pool is around 40% capacity on 4TB drives; meaning there is just a bit less than 2TB of data on any individual disk. On my server, a resilver takes just about four hours. So, obviously there is a difference in the amount of data, 3TB vs 2TB, but one is taking 4 hours while the other takes 4 DAYS. The only rational I can find for the difference is the type of data I have at home is very different from the type of data that we store at work.

So, depending on details we do not know, you may want to use RAIDz3 if you are concerned that you might have three drive failures before a resilver can be completed. For myself, in my home system, when I had drives that were old (over five years) I did have two drives fail within minutes of one another. I removed both failed drives and installed in their place the replacement drives, cold spares from my supply, and resilvered both new drives into the pool at the same time. Admittedly, this was a risk because another drive failure would have lost the entire pool, but I have a good backup. I run two FreeNAS servers in parallel that maintain sync from primary to backup and I have an additional backup on top of that.
For me, I don't see RAIDz3 as a requirement because It is so unlikely that I will suffer even two drive failures at once and having three at once is just too far outside the envelope to worry about. If I have three at once, I have a backup.

Keep in mind that no kind of RAID at any level is a backup on it's own. A backup is a separate copy of the data on different media.

Not sure if I conveyed my meaning adequately.

 

[danb35]

The more paranoid Z3 option (which I use) isn't even available via the regular pool creation menu,

You've posted this a couple of times now, and it simply isn't correct:

What makes you think it's the case? "Manual setup" isn't even a thing any more (it was in the old GUI, but not in the new).

 

[Constantin]

Sorry, I'm away from the NAS right now. If I can illustrate it when I get back, I'll post some screenshots. Maybe it's because I was building a 8-drive array that the options presented were different?

I was able to build the Z3 array within the new GUI and being able to assign the disk drives to their roles all at once (i.e. pool vs. L2ARC vs. SLOG) is a great feature.

 

[Constantin]

I am sure that was once available in the menu because I built a RAIDz3 pool some time back and I am sure I did not need to use manual creation to do it. It might be a bug of sorts, that it is not in the standard pool creation tool. Have you submitted a bug report?

Look, it's entirely possible that I misunderstood the GUI and hence had greater difficulty finding the Z3 option than I expected to. It might be a good idea for me to detach the new pool, reinstall 8 out of 9 disks from the old pool and see if I can re-create the issue. That's the problem with only having 8 slots for a pool vs. 48. :)

But first I have to go through the rest of the burn-in, SMART checking, etc.

 

[danb35]

That's the problem with only having 8 slots for a pool vs. 48. :)

That's where VMs are so helpful. I have lots of spare bays in my FreeNAS box, but not a lot of spare disks to put in them (if I did, they'd probably be part of my pool)--but I can create a VM under VirtualBox (or Proxmox, which I used for this test), assign it a half-dozen virtual disks, and test that way.

The screen shot I posted is in 11.2-RELEASE, using the new (i.e., default) GUI, the only place I know of to create a pool: storage -> pools -> + -> select "new pool". If you click the "suggest layout" link in that window, it will look just like what I posted except that it will say "Raid-z2" below the disk list. But that's a drop-down and you can change it to "Raid-z3".

 

[Constantin]

So far, VM's are mythical creatures to me... I've heard about them but I've never seen or set one up in real life. :)

 

[danb35]

They'd be worth your looking into, particularly if you want to test stuff without devoting hardware to it. I don't use the bhyve virtualization built into FreeNAS, as I don't find that it does anything I need done, better than other tools available to me* (and it seems much less user-friendly, at least for casual use). But VirtualBox is free (libre, not just no-cost); runs on Windows, Mac, and Linux (and can be persuaded to run on FreeBSD if needed); and will let you run just about any OS you want virtually, without interfering with the rest of what you're doing on your computer.

Most virtualization software (including VirtualBox) also gives you the ability to take snapshots of your VMs, which is very handy for testing: build your environment as you like it, take a snapshot, and then do the thing you're trying to test. If you don't like the result, roll back to the snapshot and try again.

* Edit: I really probably should, though--I've got an awful lot of CPU horsepower sitting idle most of the time.

 

[zamana]

If someone is still interested in my original question... I decided to keep my current setup.

Thanks.
Regards.

 

[Chris Moore]

If someone is still interested in my original question... I decided to keep my current setup.

Thanks.
Regards.

We were waiting to see if you had more questions but I guess you didn't