Building Nas for video editing

[Karibe Kulture]

Hi everyone I've recently signed up to the forum but longtime lurker, its finally time for me to build a truenas machine, I'm part of a video production company and we are looking for a system to store our content. We want to be able to edit the footage directly of the truenas system, 4k-6k footage and also be able to store important footage for access later. Im thinking of a two tier system with nvme drives to edit off and 3.5" drives for longterm storage. To start with I don't mind building a system that can only handle 1080p editing until I swap out the network card as the one that comes with the motherboard I have is currently 1gb.

Im looking to start this project from an old computer with and i3 9th gen quad core as this is a spare computer I already have access to, it can take to NVME ssd's so I was thinking to use that add 4x 3.5" drives, then change the network card and a more powerful cpu if needed. Can someone explain how much of a difference the cpu will make in terms of serving the dating up to the computers and also I've got 16gb ram

 

[NugentS]

Honestly throw all that equipment away and do it properly.
You will want a ton of RAM to edit off the NAS, along with a decent CPU (not Bronze), ECC and a proper server grade motherboard
SMB is CPU limited - so you will want a reasonbly fast CPU.

How many editors at any one time?

 

[Karibe Kulture]

Ok well if I'm starting from scratch I'd want 4-6 editors in future but currently two, any recommendations to start with?

 

[NugentS]

How big are the files?

 

[Karibe Kulture]

well for the fast storage im gonna need at least 8tb possibly increasing in the future then slower storage can probably work with 40tb ish

 

[NugentS]

I mean the individual files - not the total storage, roughly

 

[NugentS]

In general terms however you have a number of pain points.
1. Network Speed
2. RAM Quantity
3. CPU Speed
4. Disk Speed
5. Disk Sizing

So in order (above) - note that this is my opinion, others may disagree.

1. Network Speed - a minimum of 10Gb, preferably higher. Either multiple 10Gb in LAGG or (my preferred route) 40Gb or 100Gb. Obviously you will need suitable switching as well

2. RAM. Take you typical file size, multiply by 6 (for editors) and then double for safety, round up. Make sure you can expand from there

3. CPU Speed. Fewer Cores, Faster Cores

4. Disks. I agree with your two pools unless you want to follow LTT's "New New New Whonnock" expensive setup and go all NVMe. One big pool for bulk storage. Z2/Z3 with a couple of vdevs. For NVMe use enterprise disks, not consumer - they are more reliable, more consistent and don't lie about their speeds. Plug these into PCIe boards - so you will need lots of PCIe lanes. I would suggest that you use mirrors for IOPS rather than RAID. Of course to do this you will need PCIe lanes so you will need server grade CPU's and a server grade motherboard that supports bifurcation, and some PCIe to NVMe / U.2 / U.3 adapters. You may need to use a dusl socket board to get the extra lanes. Its probably best to start with a dual socket, even if you only start with 1 cpu.

5. Disk Sizing. The bulk doesn't matter so much. Just keep the vdevs reasonably small (6-8 disks) so you can add a new vdev every so often. It won't be balanced - but if that becomes an issue you can always trash the pool and restore from backups. NVMe sizing depends on how careful you are using it - bear in mind that people will always try to use the fast storage and leave stuff there "for later". So, using mirrors, not RAIDZn take you typical file size, multiple by 6 (editors) and then by 10 to get an idea of useable space required (so double that in raw storage). Again enterprise disks for consistency and reliability. Given that your initial editors is 2 then use that as a multiplyier initially - but make sure you have enough NVMe slots for enough drives to move that multiplier to 6

Don't forget backups either - maybe you can use your existing hardware for a backup NAS

 

[Karibe Kulture]

ok as we shoot a vast range of projects along with live streams we have video sizes from a few gb to 100 plus every so often. What type of motherboards do you recommend me looking at? Second hand, Im trying to keep the price as affordable as possible without compromising the performance we'll require. Currently I don't have a budget but want to figure out what's required would potentially cost. Thanks for your information so far NugentS

 

[Bikerchris]

Only because I very recently posted this website, but here may assist relating to SSD speeds (no mention of NVME): https://calomel.org/zfs_raid_speed_capacity.html

Search for "Solid State (Pure SSD) raids"

 

[NugentS]

ok as we shoot a vast range of projects along with live streams we have video sizes from a few gb to 100 plus every so often. What type of motherboards do you recommend me looking at? Second hand, Im trying to keep the price as affordable as possible without compromising the performance we'll require. Currently I don't have a budget but want to figure out what's required would potentially cost. Thanks for your information so far NugentS

Again my opinion.

Don't buy a second hand motherboard or PSU. But you can buy 2nd hand CPU & Memory.

My maths won't work for RAM size so start with 256GB (actually I would start with 128GB in as few sticks as possible, as you want to be able to expand easily) but be prepared to need more - you want as much ARC as possible so you can work off RAM where you can. Look at the spec in my sig - Primary TrueNAS as that might be a starter for 10. Lots of slots and lanes to use for bifurcation including 2*16 slots. There are better (I think) boards for your case with more X16 slots for more bifurcation - but even on Xeon PCIe lanes are limited (consider Epyc maybe - about which I know nothing) 2 Xeons = more lanes, and you don't need expensive ones.

I buy Supermicro - X10 or better. You don't need latest and greatest for a NAS. The X10's do support (based on the ones I have) bifurcation. Do not cheap out on a workstation board, if you are serious about this you will need PCIe lanes, so server CPU's and server boards.

 

[Karibe Kulture]

Honestly throw all that equipment away and do it properly.
You will want a ton of RAM to edit off the NAS, along with a decent CPU (not Bronze), ECC and a proper server grade motherboard
SMB is CPU limited - so you will want a reasonbly fast CPU.

How many editors at any one time?

Would something like this be suitable?

Gigabyte MU92-TU1, Intel C621A Express, LGA 4189, 7x PCIe x16 (Gen4 ), DDR4, SATA3/SlimSAS, M.2, 2x10GbE, USB 3.0, E-ATX

or

https://www.scan.co.uk/products/open-box-asus-ws-c621e-sage-intel-c621-s-3647-x2-ddr4-sata3-quad-u2-m2-4-way-sli-crossfire-dual-gbe

if not could you suggest something that is great bang for buck and highly expandable?

 

[NugentS]

Do they support bifurcation?
You can do without it - but things will get more expensive if they don't

Otherwise they look good.
Lots of slots - tick
IPMI - tick
Onboard video - tick
ECC - tick

The Gigabyte has 7*x16 Slots = 112 Lanes on a single CPU board that only has I thought 64 Lanes - how does that work. Something smells fishy there

The Asus has 2 sockets of say 48 lanes = 96 lanes. Its only PCIe3 but uses (5*16+2*8) = 96 lanes - so seems more honest.
An HBA (which you will need) will fit nicely into one of the X16 (x8 electrical) slots leaving 5*16 for NVMe of some kind. Not as fast as the Gigabyte PCIe4 - but you can use slightly cheaper NVMe's

I prefer the Asus. It seems more honest

As an alternative goto a Supermicro server dealer and ask them to put together a "to spec" server for you. You might find it cheaper than you think, and you get a warranty and a suitable case. Or go to ixsystems and ask them.

To be honest this is a big ask to DIY, there is a lot that can go wrong with this sort of system.

 

[Jessep]

First step is always budget.
Second step is frequently going back to "management" for increased budget.

This is for business use? (production not dev?)
Value of data?
Cost of downtime?
Project timeline? (how long until it needs to be in production?)
Who will be supporting this? (dedicated IT? Internal video editor?)

Offline questions would include how reasonable is management? (budgets, timelines, support/dev time, etc.)

 

[Dice]

Im looking to start this project from an old computer with and i3 9th gen quad core as this is a spare computer I already have access to, it can take to NVME ssd's so I was thinking to use that add 4x 3.5" drives, then change the network card and a more powerful cpu if needed. Can someone explain how much of a difference the cpu will make in terms of serving the dating up to the computers and also I've got 16gb ram

I just want to point out that the described usecase, and the suggested hardware from the community is ...well, let's say, quite far up in the atmosphere compared to what you are suggesting as a starting point.

First step is always budget.
Second step is frequently going back to "management" for increased budget.

I cannot stress how much I want to upvote this comment.

With the type of "wants" OP has, the "limit" of money to spend is no longer merely "the sky" - it goes beyond.

 

[NugentS]

Its why I suggested talking to IX. This is beyond a prosumer / enthusiast setup. This has the potential to be an cheap waste of time or an expensive screwup

 

[Karibe Kulture]

thanks for all the advice, I can give a budget without drives as we can build that out later or the different pools, tiers so to speak. As we are a small company we want something that can grow as our needs grow for the moment we just need safe storage of our files and by next year we need to be able to have multiple editors editing off a central location.

So to build the framework for this system not including drives would be 1k-2k or 2.5 if we absolutely need to, we can add a better network card later if needed because of the amount of editors so I would leave a 40gbe type network card out of the initial build. Our small company is growing fast but not fast enough too sink 10k into something like this just at this moment. So as I was saying maybe for now we can build a system to safely store our footage while we edit of external ssds and at a later date add u.2 or m.2 ssds to the system to edit off directly. Im hoping that if we start with the right motherboard and cpu we can add more ram and ssds over time

 

[NugentS]

The buy a motherboard that supports bifurcation on multiple slots, probably with 2 CPU's for the extra PCIe lanes. Use server grade equipment, lots of memory, with room for expansion. Dual PSU with lots of power (2*1000Watt say) and a suitable case with lots of cooling and then hope.

BTW - how are you going to back it up?

 

[Karibe Kulture]

Ok well you said and then hope at the end, do you feel like its a risky project, im looking at this motherboard does it support bifurcation

https://www.scan.co.uk/products/open-box-asus-ws-c621e-sage-intel-c621-s-3647-x2-ddr4-sata3-quad-u2-m2-4-way-sli-crossfire-dual-gbe

and two of these procesors

Intel Xeon Platinum P-8136 38.5MB 2.0GHz Processor for sale online | eBay

Find many great new & used options and get the best deals for Intel Xeon Platinum P-8136 38.5MB 2.0GHz Processor at the best online prices at eBay! Free delivery for many products!

www.ebay.co.uk

I will back up to another computer that I will repurpose and we also use dropbox with unlimited storage

 

[NugentS]

CPU is too slow. SMB is single threaded so you need faster cores, rather than more of them. Thats a 28 Core CPU Way more than you will ever need. Try 6-8 Cores or so per CPU with 3Ghz+ base clock if you can

You will need to check the manual for bifurcation

 

[Dice]

I second the general opinion presented by @NugentS in the thread.

Here's how I'd like to frame the design choices of OP.
The build seems to be of a '2 stage' type.
The first is to get up and running with a NAS storage solution to sovle backing up the current SSD-external solution for projects in a accessible and centralized manner.
The second stage is to feature a couple of editors to edit live off the NAS.

These two requirements are vastly different in terms of how to configure the hardware.
The way I see the problem, it is best to first focus on the <basics> which is the first requirement.
While doing so, it is key to select a platform that allows you to grow into the second stage by merely adding additional hardware.

The 'motherboard' is fittingly named, as it is the most crucial component of the build, it is the <mother of all things to come> on your horizon of the build.

There are as mentioned several 'pain points', but let's focus on the one's that are required for you to solve Stage-1.
- CPU that gets you higher single core performance than the "many slow cores". This is, as pointed out, important for CIFS.
- Plenty of PCIe and Bifurcation. This is key to get you ready for Stage-2.
- A PSU and case solution that supports enough drives to make you happy in the long run.

My vision here is a long the lines of, Stage-1
Supermicro 24bay box with PSUs solves cooling, and PSU, and drive housing at once. It gives you 2-3 vdevs of raidz2. Make sure to get one with a SAS2(or 3) expander backplane.
A HBA will connect nicely.

Stage-2 is handled by adding nVME drives, in PCIe 'holder cards' using bi-furcation, for 2 up 4 drives per compatible PCIe slot.
And of course networking to your liking.

Experience shows that in the long run, as soon as you surpass 6-8 drives (ie your first vdev raidz2), you'll greatly benefit from getting a purpose built box, even second hand.

I'd jump on ebay. Rarely you'll find the exact spec you want. Most of the time, some parts are easily changed and upgraded - as long as you get the basics in the right path / generation.
I would not go older than E5 15/26xx v3.

A yolo search on ebay : This machine
- ticks a few boxes (but may have fundamental other problems I missed)
New enough CPU (E5-2683v3)
...slower than perferred.
Onboard 4x10GBe is fantastic for your use case- Basically a few editors could get their own 10GBe to work off(!).
SAS3 expander backplane: BPN-SAS3-826EL1
Proper PSUs: 2x 1000Watt PWS-1K02A-1R
Looks to already have connectivity to the backplanes onboard.

The finniqy parts here the PCI slots, those needs to be carefully examined in the manual to see how they behave.
- Do they allow bifurcation, and at what height/angle can you fit NVMe Pcie boards?

Hopefully some help along the way,

Cheers, Dice