Unsure if RAIDZ is a viable option in a large pool of disks (72TB)

[viniciusferrao]

Hello Dudes,

I need some advices about ZFS and RAID-Z with a real large storage.

I'm building a Machine with 72TB in total. 24 disks of 3TB. And I'm really unsure of what RAID schema I should use. I'm aware that ZFS is a RAM eater. But with 72TB, I really don't know the RAM requirements of this monster.

Because of this problem I was considering a single RAID1. Even without ZFS...

I just want some opinions on what I should do.

Thanks in advance,

PS: The storage will be exported as an iSCSI device.

 

[cyberjock]

Personally, I have 24 disks(18x2TB and 6x3TB) and I have a RAIDZ3 of 2TB and a RAIDZ2 of the 3TB.

I'd probably do 12x3TB on RAIDZ3 on 2 vdevs if you want one very large zpool. I know, it doesn't fall on the 4k boundary. But I haven't had any problems with non-4k boundary either.

 

[viniciusferrao]

And what about the RAM requirements for ZFS? The basic rule of 1GB of RAM per TB of disk storage.

 

[ProtoSD]

And what about the RAM requirements for ZFS? The basic rule of 1GB of RAM per TB of disk storage.

You're asking about the combined total storage of the disks versus usable storage of your pool(s)?

There been some debate about which is the correct value to use.

 

[viniciusferrao]

You're asking about the combined total storage of the disks versus usable storage of your pool(s)?

There been some debate about which is the correct value to use.

Yeah... but let's consider a RAID-Z2; it will render 66TB of usable data, so I will need 66GB of RAM; even in the best case. The worse, obviously would be 72GB only for the pool.

My question is: due this situation; it wouldn't be better to have RAID1 or RAID10? Even without ZFS? That's my point...

And I'm little worried about the performance penalty of a RAID-Z setup in this schema.

 

[cyberjock]

The RAM requirements are just like the CPU requirements, they change based on what kind of speeds you want and what settings you choose.

If you want Dedup, you'd better have a buttload of RAM.

If you want compression, you'd better have a very powerful CPU.

If you want encryption, you'd better have a CPU that supports AES-NI.

There's like 10 "ifs", so YOU have to know what YOU want to figure out what YOU need. That's the best advice that can be accurately given.

I have 32TB of data space and I needed 20GB of RAM for good performance. 12GB was terrible performance wise. It serves only 1 user though, and typically to only one machine. If I wanted to serve 100 users I'd better have at least 32GB of RAM, and perhaps as much as 64GB.

I told you what I would do. If you have one of a bunch of those "ifs" then it may have to change. It's important that you know what you need to build for. Since you provided no other information at all aside from the hard drives I have to make assumptions about everything else.

And before you start telling yourself RAID1 is better than RAIDZ(x) you should understand why 6 drives in a RAIDZ2 is better than two sets of 3 drives mirrored. One is better than the other for reliability.

 

[viniciusferrao]

The RAM requirements are just like the CPU requirements, they change based on what kind of speeds you want and what settings you choose.

If you want Dedup, you'd better have a buttload of RAM.

If you want compression, you'd better have a very powerful CPU.

If you want encryption, you'd better have a CPU that supports AES-NI.

There's like 10 "ifs", so YOU have to know what YOU want to figure out what YOU need. That's the best advice that can be accurately given.

I have 32TB of data space and I needed 20GB of RAM for good performance. 12GB was terrible performance wise. It serves only 1 user though, and typically to only one machine. If I wanted to serve 100 users I'd better have at least 32GB of RAM, and perhaps as much as 64GB.

I told you what I would do. If you have one of a bunch of those "ifs" then it may have to change. It's important that you know what you need to build for. Since you provided no other information at all aside from the hard drives I have to make assumptions about everything else.

And before you start telling yourself RAID1 is better than RAIDZ(x) you should understand why 6 drives in a RAIDZ2 is better than two sets of 3 drives mirrored. One is better than the other for reliability.

I thought this information was unnecessary... Sorry.

I understand correctly the differences of RAID-Z(x) and RAID-0/1/5/10. My point here is only about the memory issue... But let's post everything to help solving the problem!

This storage will handle backup data. So, write performance is preferred, since it will receive data in a systemic way: cron and rsync. I don't need versioning.

I know that RAID-Z(x) will suffer from write penalty and a L2ARC/ZIL (don't remember now the differences or even if the names are correctly) would be necessary to speedup write.

I know that a big zpool consisting of only one vdev is more realiable than splitted vdevs; since we can have random disk losts without bigger issues or complex schemes.

My point here is about the RAM. Because we need different motherboards in case of extra RAM. As example, an Ivy-Bridge Xeon can only handle 32GB of ECC RAM. And a Dual Processed Sandy-Bridge Xeon can handle a lot more...

I don't care about encryption and deduplication. Compression can be an option; but the compression will be directly linked to the RAM issue.

Hope this helps...

Thanks in advance,

 

[JaimieV]

ZFS uses most of its RAM as a read-ahead cache to increase performance for multiple reads. I therefore suspect that a system which is primarily to be used for *writing* to will need much less RAM to perform okay, and will probably benefit from a fast ZIL a lot more.

But take this with a pinch of salt as I haven't ever experimented in that direction, my post is more to point you at another idea for investigation than anything else.

What sort of data rates are you expecting, btw? Full gigE, more?

(BTW - compression is only CPU-bound, it's done blockwise so RAM doesn't really figure)

 

[viniciusferrao]

ZFS uses most of its RAM as a read-ahead cache to increase performance for multiple reads. I therefore suspect that a system which is primarily to be used for *writing* to will need much less RAM to perform okay, and will probably benefit from a fast ZIL a lot more.

But take this with a pinch of salt as I haven't ever experimented in that direction, my post is more to point you at another idea for investigation than anything else.

What sort of data rates are you expecting, btw? Full gigE, more?

(BTW - compression is only CPU-bound, it's done blockwise so RAM doesn't really figure)

Hello Jaimie; thanks for your reply. The clarification about the RAM usage solved the problem. And I was considering a pair of RAIDed SSD's to use as ZIL.

But answering your questions... I was expecting some differential updates; 1GB to 2GB of data per day. We gonna use two trunked GigE to get a total of 2 Gbits.

I don't know if this storage will have a lot of read access in the future; but I hope not.

And about the CPU; I explained bad! If we need a lots of RAM we gonna need a dual processed board with a lot of RAM slots. So this way CPU is clearly linked to RAM usage.

Thanks a lot for your reply.