16-Drive New Threadripper Server Build

[niviro]

Hi all! I'm here to ask for some recommendations for my new server build. The hardware will be as follows:
- AMD Threadripper 3960X
- Asus STRIX TRX-40-e
- 4 modules of Crucial Micron 16GB-3200mhz ECC (1Rx8)
- Broadcom LSI 9305-16i connected to a Sata (6G) backplate
- Intel X550-T2
- 16 WD Ultrastar 16TB drives.
- 3 Samsung 870 EVO 250GB SSD (two for redundant boot and one for SLOG)
- 2 Samsung 870 Pro 2TB SSD (Cache)
- Nvidia GTX1050 (for Plex encoding)

It's a beefy machine but it'll need to be FAST in both read and write speed as I'll be changing my workflow from using the data from/on local machine to stream/read/write this data to the NAS. As a music producer I'm mostly dealing with filesizes over 1MB. Mostly, redundancy and speed don't go that well hand in hand, so I'd like to ask your opinion on the configs which I have in mind:
1. 3 vdevs of each having 5 drives in a RaidZ2 config + 1 hot spare (preferred)
2. 2 vdevs of each having 8 drives in a RaidZ2 config.
Luckily the disks have enough space to sacrifice a lot of room for redundancy. And because of that I'm also not a fan of mirrored disks, as to me, it seems like a it's a less viable option because of the 1-to-1 drive redundancy. But feel free to change my mind!

Also, the new Scale looks really interesting to future proof the machine, but I've also read the performance is rather slow compared to Core. Any thoughts on that?

If you have any other suggestions to increase the read/write speed, or better configurations, please let me know!

Thank you already for the help!

 

[NugentS]

Essentially the more vdevs you have - the better performance (mirrors are the better performant configuration). Also what do you mean by "fast" - its a term that describes a very subjective view.

1. SLOG - No - just No - the Samsung EVO is not an appropriate SLOG. Its also unlikely that you need it anyway - it probably won't get used. VMWAre (iSCSI) and NFS are the usual things that require a SLOG and then you must use an appropriate SLOG device which a Samsung EVO is definately not
2. L2ARC. 2TB of L2ARC on a 64GB machine is way too much. 64GB of ARC is really the minimum to use L2ARC and at most you should have 512GB of L2ARC on 64GB of ARC. Better to use the SLOG 870 EVO as L2ARC than the 870 PRO.

Most performance tests put Core > Scale - I hope this will change at Scale gets more dev time.

 

[Arwen]

You mention using the NAS as remote storage. But, you don't specify protocol.

Are you planning to use SMB / CIFS / Samba?

If so, as @NugentS said, a SLOG would not help here.


Further, it would help to list the application(s) you plan to run on the desktop, that would use the NAS for storage.

 

[ChrisRJ]

Further, it would help to list the application(s) you plan to run on the desktop, that would use the NAS for storage.

In addition, the crucial question is whether we are talking about sequential or random I/O. Many people (not necessarily you) do not pay attention to this, but focus on sequential I/O only. But the two are night and day in various ways. So to be clear on this is mandatory for sizing.

 

[niviro]

You mention using the NAS as remote storage. But, you don't specify protocol.

Are you planning to use SMB / CIFS / Samba?

If so, as @NugentS said, a SLOG would not help here.


Further, it would help to list the application(s) you plan to run on the desktop, that would use the NAS for storage.

I indeed didn't specify the protocol. It will be SMB on a couple of Windows 11 machines. The programs used will mainly need to get quick access to audio and videofiles. However these audio files (download) can be small in size compared to what video editing systems use (200kb is nothing special). Being able to stream those live would be a great plus.

When rendering/exporting the audio (thus upload), it's writing in realtime WAV files to the NAS, but approximately 2500 mono PCM tracks at once at a resolution of 96khz/24bit (if that gives you any clue). That translates into roughly a constant write of 6Gbps to the NAS when exporting (which happens at real time).

Essentially the more vdevs you have - the better performance (mirrors are the better performant configuration). Also what do you mean by "fast" - its a term that describes a very subjective view.

1. SLOG - No - just No - the Samsung EVO is not an appropriate SLOG. Its also unlikely that you need it anyway - it probably won't get used. VMWAre (iSCSI) and NFS are the usual things that require a SLOG and then you must use an appropriate SLOG device which a Samsung EVO is definately not
2. L2ARC. 2TB of L2ARC on a 64GB machine is way too much. 64GB of ARC is really the minimum to use L2ARC and at most you should have 512GB of L2ARC on 64GB of ARC. Better to use the SLOG 870 EVO as L2ARC than the 870 PRO.

Most performance tests put Core > Scale - I hope this will change at Scale gets more dev time.

Thanks a lot for this information! In short you're saying I'll be better off investing slightly more in RAM than anything else? I'll make sure to use the 870 EVO for the cache and forget about SLOG! I assume other "specials" should not be used in this case (like Dedup and Metadata)? Apologies for my newbie-level.

 

[NugentS]

6 Gbps???

Each vdev provides the approximate performance of a single disk which you can assume is about 150MB/s (assuming a sequential write, which it probably is in your case, at least when the pool has lots of space empty)
150MB/s = 1.2Gbps so you will need 5 vdevs minimum to hope for the performance you want - when the pool is empty. As pools fill up they get slower which means you will need more vdevs

See where I am going with this?

BTW - on the L2ARC - test for a couple of weeks of real useage before adding the L2ARC. After a couple of weeks check your ARC Hit rate. If its 90%+ then L2ARC will not help. Also add more memory before L2ARC - you seem to have plenty of slots.

There may be some tuning you can do to hold more write transactions in memory, if you have a lot more memory - but others will know a lot more about that than me.

Dedupe - No - you need specific hardware to make dedupe work. It can work well, it normally just crushes the NAS into uselessness.
Metadata / Special - probably not. And certainly not to start with

 

[NugentS]

I think you are going to want a pool of SSD's to use for that sort of consistent performance.