What performance metrics to look at when troubleshooting

[azzurro]

Hi

so I have two TrueNAS boxes and I'm copying from one to the other which is a bit unexpectedly slow. The setup is rather complex, involving them being iSCSI targets for ESXi servers.
Hardware recommendations aside, what are the performance metrics I should be looking at, when I'm trying to figure out which of the two boxes is maxing out?

I'm talking moving VMs between two VMFS datastores where each datastores resides on one of the two TrueNAS boxes.

When there are no easy rule of thumb metrics, I'll be happy to dive in into the hardware specs and design of the boxes.

Disclaimer: this is a homelab playground, experimenting and playing around with stuff to see how capable it would be for my purposes.

Thanks!

 

[HoneyBadger]

Welcome,

Assuming a sufficient network connection, the most likely bottleneck is at the disk level of one of the arrays.

Check the Reporting -> Disk panel, and check the Busy, Latency, and Pending I/O metrics.

Describing the hardware being used will also help identifying any potential bottlenecks. Note that using iSCSI with ESXi defaults to asynchronous writes for VMFS datastores, which can be "unsafe" in case of sudden system or power failure:

Resource - Sync writes, or: Why is my ESXi NFS so slow, and why is iSCSI faster?

This post is not specific to ESXi, however, ESXi users typically experience more trouble with this topic due to the way VMware works. When an application on a client machine wishes to write something to storage, it issues a write request. On a...

www.truenas.com

 

[azzurro]

Thanks for the quick response and for welcoming me.
Could have figured that by myself, of course the four spinning drives (NL-SAS 7k2rpm 3TB) are 100% busy during writes.
I'm getting around 150 MB/s sustained write speed when moving a VM to that TrueNAS box.

Box 1:
i5 6500
16GB DDR4
5x PM863 1.92TB SSDs (RAIDz) on onboard SATA controller (Intel)

Box 2:
Xeon E5-2680v3
128GB DDR3
4x 3TB NL-SAS 7k2 HDDs in a stripe of two mirrors
2x 600GB Intel 320 G3 as Metadata mirror
1x 600GB Intel 320 G3 as log
1x 600GB Intel 320 G3 as cache
LSI 9211-8i based Perc H310 HBA

Both boxes have 2x10GbE (Intel X520-DA2) connections to separate switches (iSCSI-A and iSCSI-B) and the ESXi host has 2x 10GbE as well, multipathing is set to round robin, all paths are loaded equally.
Jumbo frames are not in place (yet), but I imagine they won't make THAT much of a difference.

Funny enough, reading from Box 2 and writing to Box 1, I get around 6 Gbit/s throughput but reading from Box 1 and writing to Box 2 i get only about 1.5 Gbit/s throughput, because the spinning disks seem to be bottlenecking.

Regarding the vdev layout on Box 2, I was just experimenting, not really knowing what I'm doing. I'm aware of the fact that a single (SLOG it was?) disk can break the pool if it fails with a power outage.

Cool and funny that the kinda underpowered Box 1 can write that fast, despite being around 70% full.
Sad, that the spinning hard drives can't really be compensated for with some ZFS magic.

Or can they?

 

[azzurro]

Note that using iSCSI with ESXi defaults to asynchronous writes for VMFS datastores, which can be "unsafe" in case of sudden system or power failure:

Yes, I read about that earlier somewhere. Although weirdly, turning sync write off for a zvol, increased sustained write speed for a HDD pool which was being accessed via iSCSI when I was playing around earlier.

 

[jgreco]

Sad, that the spinning hard drives can't really be compensated for with some ZFS magic.

Well, that's not true. If you have lots of free space, writes will remain fast, and with sufficient ARC and L2ARC, reads will also. This is all documented in the block storage resource.

Resource - The path to success for block storage

ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle bits modified after creation. This is optimal for RAIDZ. It's what most...

www.truenas.com

 

[azzurro]

150 MB/s i do not consider to be "fast" for two striped mirrors... That was on a completely empty pool with cache, metadata and log SSDs as described initially.

 

[HoneyBadger]

LSI 9211-8i based Perc H310 HBA

Did you crossflash official LSI IT firmware on this? The PERC H310 specifically has a queue depth issue on the Dell OEM FW that limits it to 25 pending IO. (Mini cards can also be flashed, but require some specific guidance and steps to not become paperweights.)

2x 600GB Intel 320 G3 as Metadata mirror
1x 600GB Intel 320 G3 as log
1x 600GB Intel 320 G3 as cache

Fine SSDs for meta and cache, but a bit long in the tooth as a log device these days unfortunately as you mention below:

Although weirdly, turning sync write off for a zvol, increased sustained write speed for a HDD pool which was being accessed via iSCSI when I was playing around earlier.

Check on the SLOG benchmark thread linked in my signature for some details on what's popular. (Spoiler: "Optane" is popular.)

SLOG benchmarking and finding the best SLOG

I'd like to take a few minutes to talk about SLOG devices and what makes good ones versus bad ones. I have no doubt that this will be a controversial topic since this is not well understood by many people. In short, there's 3 things that you need for a "great" SLOG: 1. Fast throughput 2...

www.truenas.com

Optane NVMe PCIe or M.2 cards (albeit mounted internally, so no hot-swap) are a good option for cost-conscious performance seekers. If hot-swap is mandatory on your setup then you'll need to scour for a performance SATA/SAS device which might be just as much money, as most of the vendors have leaned heavily into NVMe for performance options. Aim for a "write-intensive" unit for the highest level of endurance and lowest write latency - the Western Digital DC SS530 "WUSTM" model line might be a good option here.

 

[azzurro]

Did you crossflash official LSI IT firmware on this? The PERC H310 specifically has a queue depth issue on the Dell OEM FW that limits it to 25 pending IO. (Mini cards can also be flashed, but require some specific guidance and steps to not become paperweights.)

I THINK I did not. The fact that it says Dell 6Gbps HBA at boot seems to support that assumption. Which is weird because I had that HBA in an ESXi previously and I'm always checking the queue depths...

Check on the SLOG benchmark thread linked in my signature for some details on what's popular. (Spoiler: "Optane" is popular.)

OK, nice :-D but I won't be buying that for my playground What I was referring to, was the fact that async writes should be the default for iSCSI but turning sync writes off increased write speed on my pool nonetheless.
I now understand that sync write speed has to do with the speed of your SLOG device, though.

Also, still weird that I only get about 150 MB/s sustained writes on an empty striped mirror of 4 NL-SAS HDDs. But I guess that's just how things are.

 

[HoneyBadger]

I THINK I did not. The fact that it says Dell 6Gbps HBA at boot seems to support that assumption. Which is weird because I had that HBA in an ESXi previously and I'm always checking the queue depths...

I'm not sure if the "Dell 6Gbps HBA" has the same poor queue depth but it's likely worth attempting the crossflash anyways to rid it of the Dell OEM code.

OK, nice :-D but I won't be buying that for my playground What I was referring to, was the fact that async writes should be the default for iSCSI but turning sync writes off increased write speed on my pool nonetheless.
I now understand that sync write speed has to do with the speed of your SLOG device, though.

Check out the Optane P1600X cards, I'm sure one of them can find a place in your system without much fuss. They don't have the ridiculous endurance of the higher-spec DC series but still rate at 6 DWPD which is sufficient for most "playground" setups.

Explicitly disabling sync might offer small improvements over standard but it will also force everything to go async including writes that were asked by the client to by sync (including metadata/locking commands) so it's a further towards the "risk" side of the risk:reward ratio.

Also, still weird that I only get about 150 MB/s sustained writes on an empty striped mirror of 4 NL-SAS HDDs. But I guess that's just how things are.

Try creating a new LUN with an increased volblocksize of 32KB vs the default of 16KB. I'm able to peak at around 550-600MB/s doing svMotion on an eight-drive NL-SAS setup if I run async.

 

[azzurro]

Thanks for your suggestions and inputs, I'll continue playing around with it and trying various configs just for fun and out of curiousity but currently I'm a bit in the mood of ditching TrueNAS/ZFS and reverting to hardware RAID controllers. All the fuzz about pool fullness and raidz performance not being good for VMs and stuff made me question my plans and intentions.
I've been running for years without an issue with RAID 6 on LSI 9260/9271/9361 based controllers and really good performance even on spinning hard drives. I only swapped them out for SSDs because of power consumption, noise and heat.
Now seeing a raidz1 being ridiculously slow with sync writes forced with iSCSI really doesn't make me confident that this could be my future path to go.

I do not want to have to waste 64GB of memory or more + SLOG SSDs + processing power + half of my disk space (for mirrors instead of raidz) if my main goal/idea in the first place was to earn power savings from ditching the hardware RAID controller.

If you already HAVE a decent HW RAID controller, it doesn't seem to be viable to go for ZFS based solutions, if you're trying to maximize what you're getting out of your hardware. Correct me if I'm wrong/feel free to split the thread to a random discusison about whether to use or not to use ZFS based solutions...