BUILD Building a server for FreeNas primary for Plex

Status
Not open for further replies.

bestboy

Contributor
Joined
Jun 8, 2014
Messages
198
I observed 4 users at one time connected to plex and my cpu usage fluctuated from 28-40%. Memory usage for the Plex was around 700MB.
That's a good sign, but the expected load is really depending on the target device capabilities. If you have to serve a device that needs video and audio transcoding, then things might look differently.

- Local (internal) will be (2) Roku 3 and (1) wireless device at a time, either a Fire HDX or Appe iPad Air.
I guess the tablet will need transcoding, but the Rokus might be ok with direct streaming most of the time.
According to this the Roku can handle H264 in MKV, MOV and MP4 containers. That's great as these will probably make up for almost all of your HD content. That should only leave AVI containers to be processed by Plex. But this might not be a problem, if the AVI containers contain SD content as they usually do. Furthermore it might be sufficient to just repackage video streams of unsupported containers instead of transcoding them. That should be very gentle on the CPU.
Judging the need for audio transcoding with the Roku is a bit harder. It depends more or less on how the Roku is set up. The Roku is ok with the usual 2 channel audio formats, but it doesn't know what to do with the encoded multi channel formats. So if the Roku is not set up for audio pass through, then audio transcoding and down mixing is needed for DD and DTS. Luckily this audio processing is not too hard on the CPU either.
 
Last edited:

AltecBX

Patron
Joined
Nov 3, 2014
Messages
285
That's what they say. http://www.qnap.com/i/en/product/model.php?II=151


Screen Shot 2014-11-11 at 12.14.57 PM.png
Screen Shot 2014-11-11 at 12.14.57 PM.png
 

AltecBX

Patron
Joined
Nov 3, 2014
Messages
285
If I would consider ESXi or WMware Workstation 10, which one would you pick?
Also what would my underlying OS be?
 

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874
If you pick ESXi, that would be the OS.
Workstation would require a Windows OS.

If the box is dedicated to virtualization, go with ESXi as you'll get better usage of resources and performance.
 

krikboh

Patron
Joined
Sep 21, 2013
Messages
209
Notice qnap specifies up to 5 devices. That is a maximum and doesn't specify resolutions.


Sent from my iPhone using Tapatalk
 

bestboy

Contributor
Joined
Jun 8, 2014
Messages
198
It allows for up to 5 devices to simultaneously view different videos stored on the TS-853 Pro with on-the-fly hardware accelerated transcoding.
Yes, but you will have to use their own streaming service "video station" and their special mobile apps "QFile" and "QVideo".
If you want to use Plex on the QNAP as stated by the topic "Building a server for Plex", then you are limited by what the QNAP can do with software transcoding.
If Plex is not a must, and you can get those QNAP apps running on your Rokus and various end devices, then you might be able to benefit from the hardware transcoding.
 

mjws00

Guru
Joined
Jul 25, 2014
Messages
798
Best case I can see on the 853 is that qnap has actually leveraged quicksync so they get hardware transcoding vs only cpu. Anandtech gave em props on that front. But mainly they are bragging celeron is faster than atom. I went digging for rational specs and benchmarks but really didn't find much. The only serious transcoding testing I did was on my i5 and e3, both transcode 1080p at ~12x. Turn on debugging and Plex shows EXACTLY what's up. It's super linear based on similar clocks and cores so estimates based on benchmark ratios should be sane. 2-3 streams on i3, 7-10 on e3 makes sense to me.

There have been folks swap i5-i7 into the 870 series QNAPS.

It's all mindset and preference, imho. The first post suggested enthusiast tastes for fast gear. The most powerful and scalable solutions need a similar measure of technical skill, or the desire to put in the time and learn. The upside is far more flexibility and power, the downside is it isn't plug and play.

If you grab the 853, make sure and let us know how it does. Not every requirement demands a server and ZFS.
 

AltecBX

Patron
Joined
Nov 3, 2014
Messages
285
mjws00 thanks for the advice. I'm still on the fence about all this. I've been putting in the time by reading as much as I can everyday for the past 2 weeks. I'm on the fence since I already have 10 6TB WD Reds 3.0 and are sitting there ready to get put in use. I set aside $2k for a built of some sort. I was just going to built another machine with the specs I stated in my first post, but after some digging, I came to a conclusion that I need to get server grade hardware. So if you can stream 7 streams of 1080p on your e3, then I won't have any problem building a rig with a e5. So after researching, this is what I have come up with.

SUPERMICRO MBD-X10SRI-F Server Motherboard LGA 2011 R3
Intel Xeon E5-2630 v3 Haswell-EP 2.4GHz LGA 2011-3 85W Server Processor BX80644E52630V3
Crucial 64GB (4 x 16GB) 288-Pin DDR4 SDRAM DDR4 2133 (PC4-17000) ECC Server Memory Model CT4K16G4RFD4213
Fractal Design Arc XL (FD-CA-ARC-XL-BL-W) Black Steel ATX Full Tower Computer Case

My total comes out to $2,020 on Newegg.

My dilema is that after reading all this, I don't know if to built a rig on ESXi 5.5 and then run an OS on top to just run Plex. Or should I built the FreeNas 9.3 for Plex as well. I also don't mind paying for the best processor without throwing away money on unused power for what I'm doing. I also want to run ECC o this setup as well.
I have another 2 weeks to buy this hardware, as I'm waiting for Black friday to buy the pieces on sale.

Question: If money wasn't an issue without throwing away money on unnecessary hardware, what will the ultimate machine to serve 10 users look like while running a few more jails on FreeNas.
Motherboard:
Processor:
Memory:
Case: Fractal Design Arc XL
Hard Drives: 10x WD Red 6TB 3.0
L2ARC SSD Drive: ?
LSI controller: ?
 
Last edited:

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874
OK. So looking at this: https://support.plex.tv/hc/en-us/ar...kind-of-CPU-do-I-need-for-my-Server-computer- it appears that to handle just the load of 15 concurrent transcoding sessions of 1080p (2000 passmark each stream) will require a total CPU capacity capable of 30,000 passmark (roughly). You would need dual E5-2690's ($4,600 total for both CPUs). http://www.cpubenchmark.net/multi_cpu.html

So now you need to realistically figure out the transcoding needs and how to scale. As someone else pointed out, if you are local and can play direct, you can reduce the transcoding drain on the CPU.

Now to the real question: One box for everything (storage and Plex) or 2.

My suggestion is to have 2. Size the FN box to handle serving the 15 streams, backups and replication. Which honestly isn't that strenuous.

Then point your existing Plex box at the data on FreeNas and see how it performs. Once you understand the true performance needs of Plex as things expand, then assess either upgrading it's MB/CPU or buying a faster box for Plex.
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
Here is some more info for you:
I tested out 6-7 streams on my E3-1230v3 again. This is fine unless you try to transcode something like Avatar which for some reason wasn't playable with 5 other streams going on. Who knows if this information is valuable but maybe someone will use it. System load was 25:19:10
Code:
[root@FreeNAS] ~# zpool iostat 2
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
tubby       9.60T  6.65T     33     20  3.81M   359K
tubby       9.60T  6.65T    256    125  31.9M  5.75M
tubby       9.60T  6.65T    255      0  31.4M      0
tubby       9.60T  6.65T    128    181  16.0M  6.31M
tubby       9.60T  6.65T    255      0  32.0M      0
tubby       9.60T  6.65T    161     40  19.9M  5.04M
tubby       9.60T  6.65T    222     85  27.5M   979K
tubby       9.60T  6.65T    128      0  16.0M      0
tubby       9.60T  6.65T    504    203  62.4M  5.77M
tubby       9.60T  6.65T     34      0  4.25M      0
tubby       9.60T  6.65T    273      0  34.1M      0
tubby       9.60T  6.65T     92    129  11.6M  5.30M
tubby       9.60T  6.65T    384      0  47.6M      0
tubby       9.60T  6.65T    127    191  16.0M  5.69M
tubby       9.60T  6.65T      0      0      0      0
tubby       9.60T  6.65T    594      0  73.7M      0
tubby       9.60T  6.65T     47    134  5.93M  5.32M
tubby       9.60T  6.65T    256      0  31.8M      0
tubby       9.60T  6.65T    127    178  15.6M  6.08M
tubby       9.60T  6.65T    255      0  32.0M      0
tubby       9.60T  6.65T    256     43  31.6M  5.43M
tubby       9.60T  6.65T      0     83      0   589K
tubby       9.60T  6.65T    384      0  48.0M  64.0K
tubby       9.60T  6.65T    127    218  15.4M  5.78M
 

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874
SnL: Any chance of a "top" or "htop" capture during the 6-7 streams?
 

mjws00

Guru
Joined
Jul 25, 2014
Messages
798
That's about the most cost effective board for a trust worthy ddr4 build. The only tricky bit is whether you want a lower core count higher clock proc which will help with all the single threaded things, or more cores for when it gets busy. For cores and clocks it is not cheap. Depasseg's build shows that. His board is cool as well if you wanted to consolidate your other nas's or pass through that extra controller to esxi.

The simplest and safest build is just run FreeNAS, configure jails etc for your services. That is the data-centric conservative approach, and it is solid.

However, some of us folks die a little inside when we see such kick-ass hardware sitting mostly idle. I'm comfortable with vt-d and esxi set up as per jgreco's great threads. The problem is you are unlikely to get any hand-holding if you screw up and nuke your data. If you set things up right, it is trivial to switch between virtual and baremetal configurations for troubleshooting or tweaking. But there is another level of complexity and moving parts. So you have to evaluate your own position.

The other board I'd look at is the X10DRi-T. Gets you that extra slot for a proc and more cores. PLUS 10GB Networking. My e5 will be around a while so I want that. $140 extra for 2 10Gb ports is a steal, imho.

You can always start with a Baremetal FN install and get comfortable, then migrate later without touching the pool.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
That's about the most cost effective board for a trust worthy ddr4 build. The only tricky bit is whether you want a lower core count higher clock proc which will help with all the single threaded things, or more cores for when it gets busy. For cores and clocks it is not cheap. Depasseg's build shows that. His board is cool as well if you wanted to consolidate your other nas's or pass through that extra controller to esxi.

The simplest and safest build is just run FreeNAS, configure jails etc for your services. That is the data-centric conservative approach, and it is solid.

However, some of us folks die a little inside when we see such kick-ass hardware sitting mostly idle. I'm comfortable with vt-d and esxi set up as per jgreco's great threads. The problem is you are unlikely to get any hand-holding if you screw up and nuke your data. If you set things up right, it is trivial to switch between virtual and baremetal configurations for troubleshooting or tweaking. But there is another level of complexity and moving parts. So you have to evaluate your own position.

The other board I'd look at is the X10DRi-T. Gets you that extra slot for a proc and more cores. PLUS 10GB Networking. My e5 will be around a while so I want that. $140 extra for 2 10Gb ports is a steal, imho.

You can always start with a Baremetal FN install and get comfortable, then migrate later without touching the pool.

You can't get any non-knockoff controller new for 140 bucks, much less two of them, so that sounds like an excellent deal.
 

AltecBX

Patron
Joined
Nov 3, 2014
Messages
285
depasseg I see your point. Built a box with all the storage needs and all the services I need with FreeNas. Then built a machine with Plex with the fast hardware to build and scale bigger if needed in the future...But I really want to built one, so I don't have to worry about 2 machine being up at one time. My current machine is not 100% reliable due to other things I do that sometimes freezes my machine.

So I looked at the cpu link, I need a CPU with at 13,000 passmark. I believe I will have no more than 6-720p streams (9,000), 2-1080p (4,000) external streams at a given time. My local connections with be direct, so need to transcode. So if the maximum amount of users all connect at one time, which I 99.8% doubt, then the most I can see this machine stress is up to 13,000 passmark. I think I will never see this but want a processor up to the task if needed those uncertain .02%.
So if i built just the ONE server, then this gives the server enough power to run more things at one time. I realistically foresee just 3-4 external connections on a sat night, with 1 local at a given time mostly streaming 720p content, as 90% of my media is 720p. So given that 4-720p * (1,500) = 6,000. Lets add another 1080p (2,000) to the mix, I then see the passmark at 8,000 at peak 99.8% of the time when serving media through Plex.

The E5-2630 v3 I mentioned above measures at 13,253 which gives me plenty of more head room for other things. Not to mention its' above the 13k mark as well, which will serve all users 100% of the time if it ever gets to it. Now this brings up another question then, Is there a passmark number range for other jails in FreeNas? Plex kind of gave a range with 720p & 1080p(1,500 & 2,000), has anyone measured there Jails or services to calculate the processor power needed to run them?

SweetandLow, I can't break down your data to make sense to me.

mjws00, Depasseg built has a E5-2637 v3 processor and I'm looking into the E5-2630 v3 processor. The E5-2637 v3 has a passmark of 10,519 and the E5-2630 v3 has a passmark of 13,253. This gives the E5-2630 v3 an advantage of 2,734 which is equivalent to serve another 2 720p transcode connections.
The only thing I see is that the E5-2637 v3 runs at 3.5Ghz with 4 cores(8-threads) and the E5-2630 v3 runs at 2.4Ghz with 8 cores(16-threads).
-The X10DRi-T doesn't mention anything about 10GB ports. says it has 3 10/100/1000Mbps ports.
Based on what you said, I think I won't be going ESXI then. It will be my first time doing it, and I currently have over 30TB of data I don't want to lose.
I also notice the TDP on the E5-2637 v3 is 135W and the E5-2630 v3 is 85W if that makes a big difference?

And if I go with a dual cpu MB, I can go with 2 E5-2630 v3 which will cost me $1,300 and yield me 26,506 passmark. Thats 16 cores(32-threads).
{This comes out to 18 streams transcoding 720p contents at one time}
The E5-2637 v3 will cost me $2,000 and only yield me 21,038 passmark. Thats 8 cores(16-threads).
{This comes out to 14 streams transcoding 720p contents at one time}
Either one will serve me just fine if I go with dual cpu but the E5-2637 will cost me $700 more. At to be frank, I think they both will be a overkill for what I'm foreseeing in my near future. But I like the idea of knowing that I can add another processor down the road to scale higher if needed.
So my question: Is the higher clock Ghz(3.5Ghz vs 2.4Ghz) with the less cores & threads worth it for my setup, Or the more cores(8 vs 4) & threads better suited for my setup?

They are both new processors for Q3 of 2014
E5-2637 v3
3.5 Ghz
4 cores (8-threads)
15Mb cache
TDP - 135W
$1,000

E5-2630 v3
2.4 Ghz
8 cores (16-threads)
20MB cache
TDP - 85W
$650
 
Last edited:

mjws00

Guru
Joined
Jul 25, 2014
Messages
798
Heh. Looks like I linked the wrong board. Model number was right. Sorry. It did feel for a second like that price was too good to be true and it was. Still a good deal on the x540's but $240 delta not $140. Proper link ($529)

The passmark number is interesting but not the whole story. The slow clock (2.4Ghz) may impact potential speeds on single threaded processes like samba (though not on 1GBe). That detail nags me for my own use. You are sharing "the love" more than I am so overall power is more useful. For me the potential bottleneck is meh. Which leaves only mega dollar procs, or a ~1620 which kills onboard supermicro 10GBe boards and limits me. I want a fast e3 with ddr4 and 64GB+. It doesn't exist.

I do kind of like the scale out options in your scenario. You need raw cycles, not ECC and data integrity for plex. So multiple servers could be good. I mentioned that earlier as well. Use the new server and add your current server to the mix if demand necessitates it. In all cases the data is still safe on your zfs system.
 

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,874
The single threaded issue with samba is why I went with the highest frequency cpu. I'm eventually going to have 10 gig.
 

mjws00

Guru
Joined
Jul 25, 2014
Messages
798
Yeah. You have the right answer, but try as I might I'm having a hard time justifying the $$$. I actually want the board with the 10Gb and the 3108, MBD-X10DRC-T4+ ($1050). But add a couple procs like yours to that, and a whack of RAM and I am WAY out there for a "toy". :) Sure is pretty, though.
 
Last edited:

bestboy

Contributor
Joined
Jun 8, 2014
Messages
198
Instead of getting one uber machine with expensive hardware to host several VMs with ESXi, I'd suggest an alternative approach for a scalable multilayer solution with commodity hardware.

The layers:
  • access layer (optional)
  • transcoding layer
  • storage layer
wave 1: Storage first
As a foundation I'd build a reliable storage layer with FreeNAS. For this box I'd focus on mass storage and build a nice home for the existing drives of the old NAS boxes as well as the future drives to come. Having so much data at hand (~120TB IIRC) it is mandatory to have a sound storage system. I'd spend about 1500 bucks of the 2k budget on such an enterprise grade storage machine.
  • Xeon E3
  • 32 gig ECC RAM
  • proper mainboard
  • HBA controller
  • case with good airflow that can house a bunch of disks (~20?)
I'd then create a Plex server in a jail on the storage machine as a first incarnation of the trancoding layer.
And that's all for wave 1. I'd just run it like that and see what's what. Because of the uncertainties about the actual transcoding requirements, it might well be the case that this setup is already sufficent.

However, if it turns out that this 2 layer setup is not sufficient, then I'd initiate wave 2.

wave 2: Enable scaling
I'd introduce an access layer on top of the trancoding and storage layer. It would consist of another machine build with the rest of budget (~500 bucks). It doesn't have to be too powerful. A normal consumer grade desktop PC with a cheap multicore CPU and some RAM should do it. First I'd outsource all network services onto that box: VPN, web server, dyndns, bittorrent, DHCP, firewall and QoS/AQM. And secondly I'd put an haproxy instance as a TCP load balancer on it.
With the load balancer it's then possible to scale the transcoding layer horizontally. In order to do so, I would add the current/old Plex server to the transcoding layer as well. The load balancer would then have to distribute Plex requests between the Plex instance in the FreeNAS jail and the old Plex server.
Both Plex servers are backed by the FreeNAS in the storage layer and offer the same content. It should not really matter which of the 2 Plex instances serves a particular user as long as it is guaranteed that all user requests are routed to the same instance. I'm quite positive that haproxy can be configured to use such sticky sessions based on the user IPs.

I would assume that such a transcoding layer can handle 10 - 12 transcoding jobs.
The really nice thing about this setup is that it is extremely flexible. I can stay at wave 1 and do nothing more. Or I do the wave 2 upgrade, repurpose old hardware and get horizontal as well as vertical scaling. And if I still need more power after wave 2 (e.g. because of 4K videos or even more users), then I can just launch a 3rd wave: Either replace the old Plex server with a more powerful system or just buy another cheap box with commodity hardware and add it as a 3rd Plex instance to the transcoding layer...
 
Last edited:

AltecBX

Patron
Joined
Nov 3, 2014
Messages
285
My current backbone are 2 gigabit switches and 1 gigabit modem through verizon fios. I also have 2 Netgear R8000(AC3200) wireless AP. All hardwire devices are 1Gb. Most wireless devices are AC 1300, and with both AP it's really hard to saturate 1 AP at one time especially with the processor in them. My main speed is usually 450Mb/s through wireless and 875Mb/s on wired.
Are you saying that 1GB is not sufficient for samba file transfer? If so, the E5-2630 v3 won't be sufficient?

bestboy, I like your idea of wave 1. I don't see myself ever needing more in the near future so no need for a second machine. Why would you recommend a E3 over a E5?
I building the machine with 10-6TB (~60TB). This will hold me for at least the next 2 years, unless I start putting in 4k stuff which i don't think I'll do in this coming year yet. It took 4-5 years to fill 28TB of space. I think 48TB will be sufficient after Raid2, I just need a case to handle the extra drives if needed down the road. I just don't want a rack case as they are too noisy to put in my office. Basically looking for a case to sit beside my trading workstation attached to the APS and sit there quietly mostly serving plex and files to the network. Also you mention a HBA controller, wouldn't the built in controller on the MB be sufficient? The DHCP, DNS, Firewall, QOS is all handled by the router.

mjws00, I think I can spring for that board. At the moment I have Gigabit switches, but might invest in a 10Gb switch in the upcoming year when they go down in price a bit. I like the idea in having 10Gb nics in place so I can benefit from the extra bandwidth. Will this MB support just one cpu at the moment, and then if I need the extra power just throw in a second cpu? Will it have to be the same CPU?
I'm looking to built a machine with DDR4, so that's why I'm looking at 2011 v3 processors.
 

mjws00

Guru
Joined
Jul 25, 2014
Messages
798
You can start with 1 cpu and throw in the second whenever, or never. You do need to match em up. You'll have 3 pcie slots dedicated to each proc. For me this is just cheap 10Gbe, expansion potential, and if this board is kicking around in (x) years up to 1TB RAM. The switches will get cost effective, but for now I can make do with an uberspeed connection to my workstation or a server. I'm the greedy one that gets pissed off at slow 1Gb connections and it's my money ;)

The E3 answer is all about growing in to your needs, imho. There is still a premium on DDR4 solutions, and you haven't legitimately tested your minimum requirements. Two cheap fast E3's or an E3 + i7 can hang with or beat an e5 for transcoding especially at 3.5Ghz vs 2.4Ghz. But if you compare apples to apples, and utilize the cheaper ddr4 board and a 1600 series proc, the premium is pretty reasonable.

The E3 solution also has the advantage of buying time for speeds to increase while prices drop. If it serves you for two years as your data and users scale... at that time scaling further will cost less. Plus WAY less upfront costs. :)

I maxed out my e3 on the first day I had it, and find the 32GB limit pathetic. I intended it as a test platform, and its inability to scale let me down. That is a BIAS. Grab an SM X10SL7-f, 32Gb, 1230v3 is boring as fsck. I also like analyzing new options, so it's fun to find an interesting challenge and someone not afraid to adopt early. Those are my people ;).

The first board you mentioned. The x10SRi-F, add a 1620V3, and 32GB is very comparable to the e3 solution, but you could drop 512GB RAM in it and 18 cores if you chose to. To me that is FUN. I'm ready to punt my E3 down the line or stick it on backup duty. (See BIAS up above)
 
Status
Not open for further replies.
Top