ZFS Best Practices?

[Wronglebowski]

Hey all, just built out my second TrueNAS Pool and I'm looking for some tips. My pool is just a general bulk storage for Plex, documents, photos, few game servers.

Pool - Main

mirror-1

6TB

6TB

mirror-2

6TB

6TB

mirror-3

8TB

8TB

mirror-4

8TB

8TB

mirror-5

8TB

8TB

mirror-6

12TB

12TB

mirror-6

12TB

12TB

Few things I'm curious about

1. Is this the right way to do mirrors? I want to be able to swap out the 6TBs for larger drives down the road. If I put all 4 of them in one mirror it seems it would be more difficult to remove them. Is this accurate?
2. Even if it's not necessary for my use case, I do have 4 SATA ports free for 2.5 inch drives. I wanted to setup mirrors for Log and Cache as well. Any recommendations on the best way to do this?
3. Dedupe, when is this ever useful? I'm not sure how it would actually function.
4. Any recommendations on protection from controller failure? My last Pool was wrecked by one controller suddenly throwing I/O errors in a mirror I had. The drives were fine but it added severe metadata corruption to the pool and it was lost. Would snapshots protect against this?
5. If snapshots do help here, are they full snapshots and will take up the entirety of the space used? Or is it just a diff?

 

[sretalla]

If snapshots do help here, are they full snapshots and will take up the entirety of the space used? Or is it just a diff?

Confused re Replication

I sort of understand Snapshots. Essentially change files that "sit alongside" the mothership. Replication takes these and copies to another location.... What I can't get my head around is - if the original dataset etc gets destroyed by, lets say an idiot admin :-) or a local fire, if the...

www.truenas.com

They won't stop pool metadata corruption (and are sometimes part of/impacted by it)

Is this the right way to do mirrors? I want to be able to swap out the 6TBs for larger drives down the road. If I put all 4 of them in one mirror it seems it would be more difficult to remove them. Is this accurate?

It's one of the possible ways to do them... if it's actually right for you, I can't say other than that you seem to be using VMs, so it should be OK for block storage and won't be terrible for bulk storage (other than the 50% capacity lost to mirroring).

Even if it's not necessary for my use case, I do have 4 SATA ports free for 2.5 inch drives. I wanted to setup mirrors for Log and Cache as well. Any recommendations on the best way to do this?

Don't do SLOG (unless you want to buy a proper SLOG device) https://www.truenas.com/community/threads/slog-benchmarking-and-finding-the-best-slog.63521/

For cache (L2ARC), don't bother unless you have more than 64GB of RAM and your working set is much bigger than 64GB.

Dedupe, when is this ever useful? I'm not sure how it would actually function.

When you have a bunch of content that's going to be the same (things like multiple clones of the same operating system or backups of things that are mostly the same).

TrueNAS will (use a huge amount of resources to) look at the blocks which would be written to disk and will avoid writing the same exact block to the disk more than once even if it's in another file (just pointing to the first copy of it written in the file table instead).

This has traditionally been so resource intensive that it mostly result in you spending more on RAM than it would cost you to just buy more storage and allow duplicates. Recently it became a little more feasible with the special VDEV types, but it's still something requiring a lot of thought and configuration. I wouldn't recommend it in most cases.

Any recommendations on protection from controller failure? My last Pool was wrecked by one controller suddenly throwing I/O errors in a mirror I had. The drives were fine but it added severe metadata corruption to the pool and it was lost. Would snapshots protect against this?

Use a good quality controller (some Motherboard onboard chips are Marvell, which is known to be terrible).

An LSI HBA running in IT mode is the known and tested option that the forum continues to recommend based on millions of hours of runtime in the FreeBSD drivers and in FreeNAS/TrueNAS systems.

 

[Etorix]

If you want two-way mirrors, that is the way to do it. Putting four identical drives in a single mirror would result in a four-way mirror: One drive worth of usable space but as resilient as RAIDZ3.

Be aware, though, that 2-way mirrors are not secure storage with large HDDs. If one drive fails there is a non-insignificant risk that the remaining drive will post an URE while resilvering. HDDs are rated for (less than) one URE in 1E14 or 1E15 bits. 12 terabytes is 0.96E14 bits…
RAIDZ2 would be safer for bulk storage, plus a SSD mirror for your VMs.

As for ZFS "sexy features" (SLOG, L2ARC, dedup,…) there is a simple rule: If you don't fully understand what it does, you don't need it. (And you may still not need if you undertsand.)