How are these hardware choices for my TrueNAS Scale build?

[zachlovescoffee]

Hello! Thanks for all of the help recently. I'm planning to slowly acquire some parts of the next few months as prices fluctuate. My priority is to get MOBO and PSU first so I can test components as they come in. I think I'll be shooting for 10 drives eventually but 4 to start and somewhere upwards of 30-40TB in total, but it may expand over time as I continue to take my media library from physical to digital and continue to acquire content. Right now I have 10Tb in RAID1/SHR1 with roughly 35% usage. I anticipate I'll be out of space in < 6 months.

The intent of this build is (case #1) media storage/redundancy and file serve to an Intel NUC11THNi5 that will handle streaming via Plex (lifetime PlexPass). This is mostly because I've already purchased the NUC and I'll probably have it also run pfSense, adGuard, Gluetun and some other services via Docker on Ubuntu server. It also means I can get a cheaper CPU that what I have listed, I believe. Case #2 would be the to use this NAS to do all of that. I'm leaning toward case #1 since I have already invested in the NUC.

Motherboard: Supermicro X11SCL-F (purchased new for $150 all in)

PSU: Seasonic Focus PLUS 850 (purchased ~$80 all in)

CPU1: Intel Xeon 2126g or 2146g (depending on price / avail; prices vary but so far > $100)

CPU2: Intel Core i3-8300 (acquired new for $64 all in)

RAM: 32 of ECC (UDIMM) DDR4 (purchased for $90 all in)

Chassis: Fractal Node 804

Storage HDD: Seagate IronWolf or WD Red (SMR only CMR only) got 4 x 8tb for $60 each, looking to acquire 4 more.

Boot SSD/NVME: I have a handful of Samsung and Intel NVMEs (250GB) laying around

L2ARC: 250gb NVME
SLOG: Still researching
HBA: still researching

Does this look like I'm headed in the right direction?

 

[danb35]

Storage HDD: Seagate IronWolf or WD Red (SMR only) - still considering sizing

You definitely don't want SMR disks. Also consider your pool layout before you buy, because you can't expand vdevs. "Ten disks eventually but four to start" doesn't really work that well, unless you're going to be using mirrors--which work well, but at ten disks you're using five for redundancy, and if the wrong two disks fail you still lose your entire pool. Six or eight disks in RAIDZ2 will give you nearly as much (in the case of six) or more (in the case of eight) capacity, burn fewer watts, and have arguably better redundancy.

SLOG: Still researching

Nothing about your stated use case indicates that you'd benefit from SLOG at all.

 

[zachlovescoffee]

You definitely don't want SMR disks. Also consider your pool layout before you buy, because you can't expand vdevs. "Ten disks eventually but four to start" doesn't really work that well, unless you're going to be using mirrors--which work well, but at ten disks you're using five for redundancy, and if the wrong two disks fail you still lose your entire pool. Six or eight disks in RAIDZ2 will give you nearly as much (in the case of six) or more (in the case of eight) capacity, burn fewer watts, and have arguably better redundancy.

Nothing about your stated use case indicates that you'd benefit from SLOG at all.

Thank you! 'SMR' was an error. I meant 'CMR'. Got it on the SLOG, consider that canned. Thanks for the heads up on mirroring. I'm still researching/learning ZFS and trying to make sure I get the lingo down. I'm not going to buy disks until I really under the best strategy for my use case (pure storage in case 1; transcoding/streaming/services in case 2).

I'm thinking I'll get more bang for my buck if I keep it just to case 1 - storage/serving to the NUC.

 

[Arwen]

We have lots of resources, see top of any forum page, Resources. Each Resource has a discussion thread, which you can make suggestions. If the original author is not around any more, you can "report" the Resource and explain what you wanted.

Here are some that help new users to ZFS;

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to...

www.truenas.com

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com

ZFS Storage Pool Layout

This resource was originally created by user: @Davvo on the TrueNAS Community Forums Archive. https://www.truenas.com/community/resources/zfs-storage-pool-layout.201/download [1] This amazing document, created by iXsystems in February 2022 as a “White Paper”, cleanly explains how to qualify...

www.truenas.com

Welcome to TrueNAS SCALE - Beginners Intro

Since mid 2021 we have had many new users of TrueNAS wanting to try SCALE. Here are some things that are helpful for new TrueNAS SCALE users to know; SCALE is a specialized & targeted OS based on a Linux distro. However, SCALE is NOT a Linux distro, and updates are by full replacement with a...

www.truenas.com

Hard Drive Burn-in Testing

@jgreco did a nice system build/test/burn-in guide here, but I (and many others) found the details a bit lacking in the hard drive section. He mentions S.M.A.R.T. tests, but doesn't go over how to run them, or how to view the results, etc. and...

www.truenas.com

 

[danb35]

Chassis: Still considering options but noise/space and environmental temps aren't an issue (very dry/cold basement).

Something I've found with my NAS is that data storage space is much like physical storage space, in that "stuff" rapidly expands to fill it. If you're going to max out at ten disks, a desktop-like case/chassis will likely do, but that's about the most you can expect to hold in one. To store more disks, you're going to want something rack-mountable, whether you actually put it in a rack or not. This would also likely take care of the PSU issue. For example, here's a Supermicro 2U chassis. Twelve 3.5" bays, SAS expander backplane (greatly simplifying cabling, though it'd need to move the HBA up on your list), dual redundant 920W PSUs:

Supermicro CSE-826BE16-R920LPB 2U Server Chassis 2x920W 12x 3.5" BPN-SAS2-826EL1 | eBay

Expander support SAS2 6.0Gb/s. SupermicroCSE-826BE16-R920LPB 12x 3.5" Drive Bays 2U rackmount chassis. Clean and tested by qualified technician. Sold as pictured, no other accessories are included. Rail kit is not included.

www.ebay.com

Or if you want room for expansion, this one gives you 36 bays in 4U and is pretty much what I'm using. Also includes PSUs, 2x 1400W units in this case:

Supermicro 4U Server 36 Caddy Bay LFF E ATX Storage Chassis 6Gbs 847E16-R1400LPB | eBay

Supermicro 4U 36x 3.5" Drive Bays. Chassis only. Can fit ATX or E-ATX (supports for maximum motherboard sizes: 13.68" x 13" ). Drive bay: 36x 3.5" Supermicro caddy. Server Chassis/ Case: CSE-847E16-R1400LPB.

www.ebay.com

These are just representative; I have no affiliation with either of the sellers nor am I particularly recommending for or against them--they're just listings for the sorts of chassis I think would be worth considering. Unfortunately, they tend to be big and heavy, and therefore fairly expensive to ship.

 

[zachlovescoffee]

Something I've found with my NAS is that data storage space is much like physical storage space, in that "stuff" rapidly expands to fill it. If you're going to max out at ten disks, a desktop-like case/chassis will likely do, but that's about the most you can expect to hold in one. To store more disks, you're going to want something rack-mountable, whether you actually put it in a rack or not. This would also likely take care of the PSU issue. For example, here's a Supermicro 2U chassis. Twelve 3.5" bays, SAS expander backplane (greatly simplifying cabling, though it'd need to move the HBA up on your list), dual redundant 920W PSUs:

Supermicro CSE-826BE16-R920LPB 2U Server Chassis 2x920W 12x 3.5" BPN-SAS2-826EL1 | eBay

Expander support SAS2 6.0Gb/s. SupermicroCSE-826BE16-R920LPB 12x 3.5" Drive Bays 2U rackmount chassis. Clean and tested by qualified technician. Sold as pictured, no other accessories are included. Rail kit is not included.

www.ebay.com

Or if you want room for expansion, this one gives you 36 bays in 4U and is pretty much what I'm using. Also includes PSUs, 2x 1400W units in this case:

Supermicro 4U Server 36 Caddy Bay LFF E ATX Storage Chassis 6Gbs 847E16-R1400LPB | eBay

Supermicro 4U 36x 3.5" Drive Bays. Chassis only. Can fit ATX or E-ATX (supports for maximum motherboard sizes: 13.68" x 13" ). Drive bay: 36x 3.5" Supermicro caddy. Server Chassis/ Case: CSE-847E16-R1400LPB.

www.ebay.com

These are just representative; I have no affiliation with either of the sellers nor am I particularly recommending for or against them--they're just listings for the sorts of chassis I think would be worth considering. Unfortunately, they tend to be big and heavy, and therefore fairly expensive to ship.

Thank you! These are great examples of where I may get in the future. I think 6-8 drives is a good start for my needs.

 

[zachlovescoffee]

We have lots of resources, see top of any forum page, Resources. Each Resource has a discussion thread, which you can make suggestions. If the original author is not around any more, you can "report" the Resource and explain what you wanted.

Here are some that help new users to ZFS;

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to...

www.truenas.com

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com

ZFS Storage Pool Layout

This resource was originally created by user: @Davvo on the TrueNAS Community Forums Archive. https://www.truenas.com/community/resources/zfs-storage-pool-layout.201/download [1] This amazing document, created by iXsystems in February 2022 as a “White Paper”, cleanly explains how to qualify...

www.truenas.com

Welcome to TrueNAS SCALE - Beginners Intro

Since mid 2021 we have had many new users of TrueNAS wanting to try SCALE. Here are some things that are helpful for new TrueNAS SCALE users to know; SCALE is a specialized & targeted OS based on a Linux distro. However, SCALE is NOT a Linux distro, and updates are by full replacement with a...

www.truenas.com

Hard Drive Burn-in Testing

@jgreco did a nice system build/test/burn-in guide here, but I (and many others) found the details a bit lacking in the hard drive section. He mentions S.M.A.R.T. tests, but doesn't go over how to run them, or how to view the results, etc. and...

www.truenas.com

Thank you! Great reading that I keep coming back to

 

[NugentS]

Thought I would add my 2p's worth
L2ARC: With 128GB of ARC - you can support L2ARC in either its configuration options:
1. MetadataOnly - this will help accelerate access to a Pool. It makes it feel more snappy as all the metadata is cached on the SSD. This doesn't require that much space as as with all L2ARC is non pool critical. It speeds up pool responsiveness, but doesn't really effect downloading data from the pool. The more folders you have the more effective this will be
2. Full L2ARC - this will also cache metadata. Before implementing this use the NAS for a few weeks and then check the L2ARC Hit Ratio. If its >90% then L2ARC (full) probably won't help that much.

L2ARC (either) can be added to an existing pool with no issues - it will take a while to become useful though as its a cache so stuff has to be read first

 

[zachlovescoffee]

Thought I would add my 2p's worth
L2ARC: With 128GB of ARC - you can support L2ARC in either its configuration options:
1. MetadataOnly - this will help accelerate access to a Pool. It makes it feel more snappy as all the metadata is cached on the SSD. This doesn't require that much space as as with all L2ARC is non pool critical. It speeds up pool responsiveness, but doesn't really effect downloading data from the pool. The more folders you have the more effective this will be
2. Full L2ARC - this will also cache metadata. Before implementing this use the NAS for a few weeks and then check the L2ARC Hit Ratio. If its >90% then L2ARC (full) probably won't help that much.

L2ARC (either) can be added to an existing pool with no issues - it will take a while to become useful though as its a cache so stuff has to be read first

Thank you! I’m really excited to demo the NAS on a couple of old test drives I have that will volunteer their lives to my go at this before I go live.

 

[zachlovescoffee]

Something I've found with my NAS is that data storage space is much like physical storage space, in that "stuff" rapidly expands to fill it. If you're going to max out at ten disks, a desktop-like case/chassis will likely do, but that's about the most you can expect to hold in one. To store more disks, you're going to want something rack-mountable, whether you actually put it in a rack or not. This would also likely take care of the PSU issue. For example, here's a Supermicro 2U chassis. Twelve 3.5" bays, SAS expander backplane (greatly simplifying cabling, though it'd need to move the HBA up on your list), dual redundant 920W PSUs:

Supermicro CSE-826BE16-R920LPB 2U Server Chassis 2x920W 12x 3.5" BPN-SAS2-826EL1 | eBay

Expander support SAS2 6.0Gb/s. SupermicroCSE-826BE16-R920LPB 12x 3.5" Drive Bays 2U rackmount chassis. Clean and tested by qualified technician. Sold as pictured, no other accessories are included. Rail kit is not included.

www.ebay.com

Or if you want room for expansion, this one gives you 36 bays in 4U and is pretty much what I'm using. Also includes PSUs, 2x 1400W units in this case:

Supermicro 4U Server 36 Caddy Bay LFF E ATX Storage Chassis 6Gbs 847E16-R1400LPB | eBay

Supermicro 4U 36x 3.5" Drive Bays. Chassis only. Can fit ATX or E-ATX (supports for maximum motherboard sizes: 13.68" x 13" ). Drive bay: 36x 3.5" Supermicro caddy. Server Chassis/ Case: CSE-847E16-R1400LPB.

www.ebay.com

These are just representative; I have no affiliation with either of the sellers nor am I particularly recommending for or against them--they're just listings for the sorts of chassis I think would be worth considering. Unfortunately, they tend to be big and heavy, and therefore fairly expensive to ship.

I've decided to go with a Fractal Node 804 as it fits my current space restrictions quite well. It should support 8 x 3.5 HDDs and another 4 x 2.5 SSDs, so I should be quite good for a while :) Thank you for your fantastic suggestions. I took those and measured my space and the racks just wouldn't work for me at this time.