Vdev Layout

[Fastline]

Hello,

I'm using 6x16TB Seagate Exos X16 connected via onboard SATA. What would be the best vdev layout for maximum performance and data safety? I also want that my drives are not so much stressed during rebuilding or resilvering.

 

[ChrisRJ]

I would go for 2 mirrors (3-way)

 

[Fastline]

I would go for 2 mirrors (3-way)

You mean RAIDZ2?

 

[ChrisRJ]

No, I mean 2 vdevs with each of them consisting of a 3-way mirror. RAIDZ2 would conflict with your stated requirement about performance.

 

[Etorix]

No, @ChrisRJ just means what he said: Two vdevs of three disks each, configured as mirrors (one drive replicated to two other drives). Mirrors can have as many member drives as you want.
Space = 2 drives.
IOPS = 2 drives. This is the "performance" part.
Throughput = 2 drives write / 6 drives read.
Resiliency: Can at least safely sustain the complete loss of 1 drive (2 drives if spread across vdev). This is the "data safety" part, the killer being the sheer size of the drives, and some ZFS-using paranoids (including yours truly) not being confident with the prospect of resilvering a large amount of non-redundant data from a single drive—hence, 2-way mirror is not regarded as "safe enough".

Compare with:
RAIDZ2 space=4 IOPS=1 throughput=4 resiliency=at least 1
RAIDZ3 space=3 IOPS=1 throughput=3 resiliency=at least 2

 

[Fastline]

No, I mean 2 vdevs with each of them consisting of a 3-way mirror. RAIDZ2 would conflict with your stated requirement about performance.

Oh, i see. Would RAIDZ3 good to go then?

I want a single pool for easier management.

 

[ChrisRJ]

I think there is a misunderstanding about the relationship of vdev vs. pool. Please check out the ZFS basics from the recommended readings in my signature.

As to RAIDZ3, that would conflict with performance just like RAIDZ2. If you want performance there is only mirrors. What are your performance requirements after all?

 

[Heracles]

Oh, i see. Would RAIDZ3 good to go then?

No.

A pool's performance is directly related to the number of vDev it contains. A single vDev will give you the performance of a single drive. 2 vDev will give you the performance of 2 drives and so on.

A single Raid-Z2 vDev will give you the performance of a single drive.
A single Raid-Z3 vDev will also give you the performance of a single drive.
@ChrisRJ recommendation of 2 vDev of 3 drives each will give you the performance of 2 drives.
3 mirrors of 2 drives each would give you even more performance but then your safety would be limited because once a first drive failed, you end up vulnerable to the last and only redundant copy of that drive on its sole mirror. With a 3-way mirror like suggested, your safety is better.

So @ChrisRJ's recommendation is best performance and safety at a cost of reduced available space but that one was not one of your requirement. As such, his recommendation is very good.

3 mirrors of 2 drives would increase performance compared to that but reduce the safety a little bit. It would also increase capacity despite this is not something you asked for.

Single Raid-Z2 or 3 vDev would be safe and would offer larger space but are not compatible with your requirement of performance.

 

[Etorix]

I also want that my drives are not so much stressed during rebuilding or resilvering.

This also reads directly as "mirrors".
Raidz, by design, is stressful on disks during resilver—with raidz3 being worst of all.

Choose your poison! With limited budget and a limited number of large drives it's: Capacity, Safety, Performance—Pick ANY TWO.

 

[Morris]

I use 2x RAIDZ and it is very fast for throughput. Should a single drive fail, 3 drives will be involved in the rebuild. Another option you should consider is 3x mirror. In this scenario a resilver after a single drive failure involves 2 drives. There are also 3 VDEV so using the speed formula above it is the fastest. It's not that simple though and you might get better throughput with the 2x RAIDZ.

 

[ChrisRJ]

I use 2x RAIDZ and it is very fast for throughput.

Do you mean for sequential read operation? Or something else?

 

[Etorix]

2-way mirror or raidz1 are adequate if a drive reports SMART errors but can be replaced with the failing drive in place. In case of complete drive failure there's no redundancy during resilver, which @ChrisRJ and I would consider insufficiently safe. One should always have backups, of course, but with large drives in raidz1 or 2-way mirrors, one should have backups and be quite ready to restore from said backups in the event of failure.

 

[Morris]

Do you mean for sequential read operation? Or something else?

Sequential read and sequential write. Random read and write are a little faster yet if you need IOPS, and SSD array and/or lots of RAM and possibly taking advantage of the various SSD caches avail are better ways to go.

 

[Morris]

2-way mirror or raidz1 are adequate if a drive reports SMART errors but can be replaced with the failing drive in place. In case of complete drive failure there's no redundancy during resilver, which @ChrisRJ and I would consider insufficiently safe. One should always have backups, of course, but with large drives in raidz1 or 2-way mirrors, one should have backups and be quite ready to restore from said backups in the event of failure.

One should have backups even if RAIDZ3 as there is always risk and it's not just hardware failure. If you look at the risk of a double disk failure, it's incredibly low. In my 30 + years of IT experience and much of it involving data centers, I saw two double disk failures. Bot of the double disk failures were caused by somone pulling the wrong drive when attempting to replace the failed disk in an array. One time was with the failed disk having a flashing pull me light! A nice feature of RAIDZx is that if one pulls the wrong drive, reinserting it is recoverable.

 

[Fastline]

Hello @ChrisRJ Got back today. Been busy. Hope you're doing well. So, i read few and understood that more the vdevs in a pool, more the performance right?

So, according to this, let's say i have 15x10TB in another machine. So, if i configure

5 vdevs of 3 Drives each, that would give me a speed of 5 drives?

 

[ChrisRJ]

That should be the case. I have never personally dealt with multiple vdevs, but IIRC it is like you said. I assume you are thinking about 3-way mirrors and not RAIDZ1. Correct?

 

[Fastline]

That should be the case. I have never personally dealt with multiple vdevs, but IIRC it is like you said.

I see. So, lets say i don't care about data safety/redundancy, then 10 vdevs of 1 drive each, that would give me a speed of 10 drives? Or am i wrong here?

 

[Patrick M. Hausen]

With all usual caveats about approximations, pathological edge cases, etc. yes. One VDEV == IOPS of one drive.

 

[Fastline]

With all usual caveats about approximations, pathological edge cases, etc. yes. One VDEV == IOPS of one drive.

OMG. Then, why don't TrueNAS simply can understand that even if there is one vdev but it has a number of drives in it and it should combine the speed according to the vdev layout? Does TrueNAS fails to understand this logic?

 

[Patrick M. Hausen]

You might want to read some fundamentals on ZFS ...

ZFS Primer

Background information about the Zettabyte File System (ZFS).

www.truenas.com