How should I utilize my drives? L2ARC, Log...

[kd7vea]

I have just acquired a new (new to me) server and quickly decided It would be a great TrueNAS machine. I'm trying to figure out the best setup to give the best performance with the part that are already in the server. The server has a single Xeon D-1541 with 8 cores, 64GB Ram, SA-9211-8I HBA flashed to IT mode, Dual 10Gb SFP+ NIC, 8 8TB Spinning drives, 6 1Tb Micron M600 SSDs, and a single 256GB m.2 NVME. About the only Piece that I'm really set on is running RAID-Z2 with the 8 spin up drives, and I will set up some or all of the 6 1TB SSDs for running VMs/jails.any thoughts on this are appreciated. The part that I'm trying to figure out is what would be the best utilization of the rest. L2ARC, a log dive, are there options I'm not thinking of. This system will be used for my Plex media server backups, and general SMB shares. I run several apps/Jails and VMs on my current TrueNAS machines and they will all be migrated to this new server. I do a lot of testing on VMs, everything from GrayLog to Zabbix to DNS servers down to basic script testing. There will be somethings that are read intensive, some will be some that are write intensive. I'm not sure if I want to let the RAM act as the ZFS cache and not set up another option because I never really know what the final configuration may be. Any recommendations re appreciated.

 

[NugentS]

How are you running the VM's. Via KVM on TrueNAS or externally using (Proxmox, KVM, ESX, Other)?

 

[kd7vea]

All of my VM‘s are running on KVM right now. I’ve honestly never looked a lot into Proxmox, but if I were to do anything else, it would probably be VMware just based on the fact that we use VMware at work, so I like to replicate things just to give me more hands-on experience outside of a critical production environment. I don’t expect a lot of performance out of most of my VM‘s because when I set them up, I’m generally not testing for performance specs, I am trying to work through the details so that when it’s deployed in our production environment. things go smoothly. Right now for instance, I have three Ubuntu VM’s set up on one of my current TrueNAS systems because I am creating some scripts to help move long-term archive back ups onto a secondary storage server. I have replicated some of the data and the file structure from the actual systems that this will be implemented on but all I am actually testing is the Scripts ability to perform the function correctly, not necessarily looking for performance specs. That’s not to say that I don’t care about the overall performance of my TrueNAS machine, I just don’t know if I will wind up running 30 VM’s on this at some point.

 

[NugentS]

From the sounds of it.
Then it seems that using the SSD's as a pool of mirrored vdevs seems like a sensible idea.

The M600 could potentially be used as a SLOG as they do have PLP apparently, but are only SATA - so IMHO not fast enough.
As for L2ARC, you won't know if that will help until you have built and run the thing for a while.

I would probably:
1. Try to work out a way of leaving the M.2 free, booting off USB attached SSD's (not thumb drives) or something else (motherboard SATA if you have a spare port)
2. Pool of mirrored SSD vdevs for VM's. Possible L2ARC on this pool using M.2 - but add more memory if you can first
3. RAIDZ2 of HDD's

 

[kd7vea]

@NugentS. Thanks for the info.
1. I think I have free USB interfaces on the motherboard, so I will look into using one of my spare 128GB SSDs as a boot device.
2. I won't be adding memory any time soon, there are only 4 memory slots on the motherboard and each one has a 16GB module in it. If I had available room, I would look at adding more 16GB sticks, but the cost of upgrading them all to 32BG or 64GB is going to drive the cost of that pretty high. I assume the Mirrored pool of SSD VDEVs just allows for speed increases because of the striping and a little protection being able to lose potentially all 3 drives if it were the correct 3 drives that died (Correct me if I am wrong on that).
3. RaidZ-2 was the plan. I hate losing storage space, but I hate losing data even more.

 

[NugentS]

SSD's are fairly reliable - so I am happy with mirrored pairs - and you get better performance (IOPS) and less write multiplication in mirrors
Do not use the M.2 initially - run first without any L2ARC for a few weeks and check your ARC hit ratio to see if L2ARC would help.

The problem that Scale has currently is that ARC only uses up to 50% of RAM, when in Core it uses everything it can. This is something that IX are working on. It does mean that (simplistically) a 64GB Scale Server is equivalent to a 32GB Core - I did say simplistically

 

[kd7vea]

Currently, my M.2 drive is being used as my boot drive, If arc hits becomes an issue, I will go back and change that so that I can utilize that drive better. The issue is I have the server in the rack, and the chassis has about 50 tiny screws to take the cover off, so going inside the chassis to add a different boot drive takes a bit of work. I am going to figure out what RAM is needed because I see some options around $150-$200 for 128GB, so I may order ram for it also just because If I'm going to pull the server out, I might as well do everything all at the same time. As I am going through this, I'm wondering, is there a reason to not set up L2ARC and Slog on different drives? can that be done, is it something that should be done?

 

[NugentS]

Using an M.2 to boot from is a waste - unless you have several not being used.

SLOG only works with sync writes - that might be something you want to do with a VM Store. Sync writes are slow, sync writes backed by a decent SLOG are better, but not nearly as fast as async writes.

Note I said a decent SLOG. It needs the following:
1. Low Latancy (in comparison to the pool its on)
2. High Speed of writes to the drive
3. Very High Endurance as ALL writes go to the SLOG
4. Power Loss Protection.
Note a SLOG does not need to be large. Not much space is used

Your M600's pass test 4 I believe, but not nessesarily the others

An L2ARC hardware requirement is a lot less restrictive. Basically as quick as possible as its (sort of) substituting for RAM

 

[kd7vea]

Thanks for the explanation. I will look into a good/affordable M.2 SLOG device, and like I stated earlier, the server has about 40 screws holding the cover on, so Ill try to upgrade the RAM and add a USB SSD boot drive to reclaim the m.2 slot. I really think this is all overkill for what I will be using it for anyways, but I figure I might as well beef it up a bit. I'm thinking I may remove the dual SFP+ Nic that's in the server and replace it with the MikroTik PCIE router. That would let me utilize my 10Gb fiber across my network and allow me to run my backups of my physical machines a bit quicker, or at least as fast as the drives can handle..

 

[NugentS]

Overkill - possibly - but thats better than having to do it again thinking "I wish I had..."

MikroTik PCIE router - whats that?
Is it : https://mikrotik.com/product/ccr2004_1g_2xs_pcie
In which case - I would wonder if there are any drivers for this thing built in to TN. TN is NOT geared up for the user to add drivers

 

[kd7vea]

Yes, the ccr2004_1g_2xs_pcie is what I was thinking. i have been searching and haven't seen anything about others using this card in their TrueNAS machine, but I understand it would be an experimental project. If it doesn't work on that server, I have some other options I Know it would work in. I'm still researching this to see if it will do what I want it to.

 

[samarium]

3. Very High Endurance as ALL writes go to the SLOG

Are you really sure that all writes go thru the SLOG? Or just sync writes to the ZIL.

 

[NugentS]

I guess that comes down to hwo you interpret what I wrote. However you are correct. All (and only) Sync writes goto the SLOG.