best way to benchmark NVMe pool?

[digity]

I want to create a fast, high throughput TrueNAS NVMe pool to store my ESXi datastore (via NFS and 40 GbE NICs). I have U.2 and M.2 NVMe SSDs (and necessary controllers/HBAs, everything PCIe 3.0) and I need to test them to see which will give me the fastest speeds and most breathing room for gaming VMs, general office workstation VMs and whatever else we homelabbers may get into. How do I properly set up and benchmark an all NVMe pool?

P.S. - TrueNAS server is XEON 3.0 GHz 20c/40t LGA2011 v2 platform

 

[NickF]

Some notes I have on this. These are general recommendations based on my experience and not hard and fast rules.

Ensure L2 adjacency to your VM hosts, crossing L3 boundaries will increase latency in almost every circumstance zx

Use mirrors not RAIDZ

Trade your CPU in for 2667 v2 IPC will be more important than cores.

Install only 1 dimm per channel per cpu in your motherboard, use fastest RAM your motherboard supports. You will need to review your motherboard’s manual to find what slots to use to do this. RAM LATENCY MATTERS! And I don’t mean in the gaming sense of min maxing CAS, I mean you have to use the optimal configuration above.

only cache metadata in ARC.

Code:

zfs set primarycache=metadata <dataset>

If you are going to use NFS, you’re going to want something really fast as a SLOG, but no matter what you choose this will be your bottleneck for writes. Alternatively you can set sync=never.
Of course there is inherent risk to data in motion by doing this, but when you’re trying to hot rod an 8 year old server, you gotta have some give and take.


Benchmarks? Spin up some VMs and simulate your workload as best you can and monitor key performance indicators. CPU usage, disk activity, network bandwidth.