Is a single 1TB NVMe enough for the system?

[jonmills]

Hello TrueNAS... I'm new to the forum, but I've been lurking for awhile. I've been accumulating parts for the better part of a year, and I'm just about ready to build. I just am seeking some advice on use of SSD/NVMe disks.

If you see my signature, you'll see I've got five 16TB Ironwolf Pro drives, which I plan to use in a RAIDZ2. Despite the slightly awkward geometry, I've read plenty of forum posts that suggest this config works just fine....so I'm not too worried about that. I'm worried about the other stuff....

There's the boot drive, for the OS....there's ZFS metadata, there ZFS logs, there's L2ARC, and maybe other things I'm not familiar with. Right now I just have a single 1TB M.2 Samsung 990 NVMe drive. 1.5 Million MTBF. Is this good enough for all those? Should I forego L2ARC until doing some testing with the 64GB RAM utilization? Should I get a separate SSD just for the boot drive, and separate out the ZFS metadata/logs/etc?

Truthfully I'm pretty good at building systems....I'm just new to ZFS and spec'ing hardware for ZFS.....so I'm seeking advice on how to use my hardware to optimize ZFS according to best practices!

Cheers,

Jon

 

[bcat]

TrueNAS assumes you're using a separate device (or mirrored devices, if you want a redundant boot pool, though the design of TrueNAS makes a boot pool failure more an issue of downtime than of data loss) for the system itself to boot off us. The docs recommend this be at least 20 GB, but 1 TB is way overkill. (I use a 256 GB SSD, which is still overkill, but smaller consumer SSDs weren't really significantly cheaper, so....)

There's a resource with some rationale on this. Note that there may be methods to manually partition the boot device and get TrueNAS to use just a small partition for booting, but this is an unsupported config and (IMO) is worth avoiding. Why invite trouble on a NAS that should Just Work (TM)? :)

As for L2ARC and other special vdevs, this is well outside my area of expertise, but from what I've generally seen/read, if you actually have a use case for them, you'll know it. For what sounds like a home NAS/lab use case, you're way better off making sure you have adequate RAM for (regular) ARC. (Anecdata, but I have a small RAID-Z2 HDD pool and it performs just fine for media storage and other light home use. No L2ARC or fancy metadata vdevs etc. needed.)

That said, if you intend to run SCALE apps (Helm charts) and/or VMs on the same machine, a mirrored SSD pool is likely a good idea. Something like 1) small SSD for boot, 2) larger mirrored SSDs for apps/VMs, 3) larger RAID-Z pool for media storage is what I ended up on personally, though your requirements may different, of course.

 

[Arwen]

As @bcat mentioned, if you don't know you need a L2ARC device or Metadata vDev, you don't need one. Either or both can be added later, though a Special Metadata vDev won't automatically populate from pre-existing data.

Further, ZFS Special Metadata devices are critical to ZFS pools, (if you have one). This means they really SHOULD have the same level of redundancy as the main data vDevs, RAID-Z2 in your case. But, the way this works, is you would have;

RAID-Z1 data pool = 2 way Mirror Special Metadata vDev
RAID-Z2 data pool = 3 way Mirror Special Metadata vDev
RAID-Z3 data pool = 4 way Mirror Special Metadata vDev

A loss of a Special Metadata vDev means total loss of the ZFS pool. So, using a single NVMe drive for ZFS Special Metadata vDev is highly discouraged.

 

[jonmills]

Well @Arwen and @bcat I may have gotten myself into a bit of a pickle here. I only have 6 SATA ports on my motherboard. And because it's MiniITX, I don't have additional PCIe slots. The one PCIe slot I do have is occupied by a 10GbE card. And since I have 5 HDDs....I only have one free SATA port. Plus, a single M.2 NVMe slot. So in other words, I don't have the SATA or M.2 slots I need to build any sort of redundant Special Metadata vDev.

Supposing I go ahead and just use my 1 TB NVMe drive, in the M.2 slot, as my boot device (overkill or not), can I just not use any Special Metadata vDevs at all? I assume then that the Metadata would just live inside the zpool, on spinning disk, and be safe there? Interested in your opinions...

 

[Patrick M. Hausen]

Supposing I go ahead and just use my 1 TB NVMe drive, in the M.2 slot, as my boot device (overkill or not)

That will work just splendidly, overkill yes, problem definitely no.

Can I just not use any Special Metadata vDevs at all? I assume then that the Metadata would just live inside the zpool, on spinning disk, and be safe there? Interested in your opinions...

What gave you the impression special vdevs are in any way mandatory? Of course you can build a pool from 5 or 6 spinning disks and assuming your system has got enough memory, that will run just fine.

Special vdevs are a way to speed up e.g. directory operations in an environment where they are a bottleneck. E.g. me at work: SMB file server for 20 people, all simultaneously, all time machine backups and regular file sharing - I did not bother to do some real benchmarks but the performance is perceived as "snappy" by all users and according to the ZFS stats the special vdev gets exercised quite a bit, so I probably designed right. If yours is a home system you will probably be fine with just spinning disks and RAIDZ2. Keep in mind: 16 G is recommended minimum, 32 G is better.

HTH
Patrick

 

[ChrisRJ]

@jonmills , you did not specify your use-case. Without that information it is difficult to know whether or not a given setup (we don't know your motherboard, CPU, and RAM either) is suitable.

In addition to what @Patrick M. Hausen wrote, let me emphasize that running a non-redundant metadata VDEV is a recipe for disaster. But as Patrick said, you likely don't need it anyway.

If you want to reduce the risk of unpleasant surprises, please spend considerable(!) time reading up on ZFS and TrueNAS. It's great that you have experience with building systems. But ZFS is very particular in some aspects. So even if you are an expert on e.g. EMC or NetApp storage systems, that knowledge would not help on its own.

Please refrain from YouTube videos for TrueNAS. These are mostly crap and somewhere between dangerous and misleading. Instead read the resources here on the forum. A starting point for the latter would be the "Recommended readings" in my signature.

Good luck!

 

[Etorix]

I only have 6 SATA ports on my motherboard. And because it's MiniITX, I don't have additional PCIe slots. The one PCIe slot I do have is occupied by a 10GbE card. And since I have 5 HDDs....I only have one free SATA port. Plus, a single M.2 NVMe slot.

5 HDDs, one free SATA port to replace drives when needed and boot from the smallest and cheapest NVMe drive you can find (using a 1 TB drive will work… but waste 0.99 TB). All fine for a home system.
If you're not satisfied with the experience of browsing directories, you might add a persistant L2ARC for metadata: This will speed up reads (browsing) but not writes. No redundancy is required because the L2ARC just holds a fast access copy of the metadat which is safely stored in the pool.

 

[Constantin]

This is a great example of how some additional planning up front would have saved some heartache later on. Until NVMEs become more ubiquitous, anyone contemplating even a moderate TrueNAS build should choose a motherboard with 8+ SATA ports.

The AsRock C2758 (used in the old Mini and Mini-XL) was a great little board whose quirks and deficiencies iXsystems allegedly spent considerable software resources to work around.

Synology, ReadyNAS, and QNAP have also designed a lot of boards whose guts are optimized around file storage, attaching large numbers of hard drives, and low cost.

I remain a huge fan of the Intel D-15xx series of CPUs for motherboards that are intended for file-storage applications on account of all the CPU lanes that are available for storage, PCIe cards, m2 interfaces, or whatever. The price point and heat generation is also more compatible with that application than the later D-2xxx series. Curiously, the price point is also the same as the atom boards used by iXsystems today.

Obsessing over the smallest possible form factor for a given motherboard, case, etc likely will hit the user in the pocketbook later even if the aesthestics are more pleasing in the short run.