Vmware too slow on NFS

[Vinicius Coelho]

Ive read a few topics about that...
Already set zfs set sync=disabled, and made a ZIL.

The first time I activated the sync disabled, the write and read latency went to 0. But after sometime (1 month) the latency started to increase. What else I should do? Change everything to ISCSI? Will the performance get better? There is any other setting that I can do?

 

[rs225]

Is this a raidz or mirrors? What are the new latency numbers?

 

[Vinicius Coelho]

Im using raidz, all my disks are 2tb ssd.
The average is 50, but sometimes it gets to 1000ms. The transfer rate is 20mbps on the nfs share (inside the virtual machine, or esx console). Using ssh, I get 700mbps on FreeNAS.

 

[rs225]

What model are the SSD?

 

[Vinicius Coelho]

Crucial MX300 2TB.

 

[rs225]

I see that Crucial released a firmware update for the drives a couple weeks ago. Have you applied that? Do you have a backup of the pool?

 

[Vinicius Coelho]

Yes.. I do, but it seems that the problem is something related with the nfs protocol and vmware.
The test that I've made inside the FreeNAS gave me a nice results. I have another physical linux, and made a test. got 200mbps.

 

[rs225]

I would agree, but the question is why the performance degraded after a period of time. The storage devices can cause that, or the pool filling. The only network issues that might cause that would be glitchy hardware, cables, interference or something like that.

Is sync still set to disabled?

 

[Vinicius Coelho]

Yes, its disabled. I tried to enable, a disable to see the behavior... When I enabled, the latency jumped to 2000ms... but when I disabled returned to 50ms. I will try ISCSI to see what happens.

 

[toadman]

If it's the SSDs iSCSI probably won't help.

I'm still trying to understand the storage topology. You said you have Crucial MX300 2TB drives in RAIDZ and you made a ZIL (I assume you meant SLOG). How many drives in the RAIDZ? And that is the only vdev I assume? Last, what are you using for the SLOG? The fact you said you have a ZIL (SLOG) and you are seeing high latency numbers when sync=enabled tells me your SLOG isn't a performance solution. (And therefore you should probably remove it as it's not doing anything useful.)

I'm not familiar with the Crucial drives, but there are times when even an SSD can get into a non-optimal situation where the internal controller has to do a bunch of changes. Beyond garbage collection. The entire drive can slow down because of it. Samsung had this issue a couple years back where 840s were dropping in performance over time. So definitely apply the firmware update mentioned.

In my experience NFS with sync=disabled (with all of the appropriate "you could lose your VMs and/or your pool" warnings) performs better than iSCSI. However, it's gonna depend on workload and the rest of the hardware setup. Others find the opposite. If you decide to give iSCSI a try, do it simultaneously with NFS. i.e. set up both and test side by side.

 

[Vinicius Coelho]

Hi toadman, I meant SLOG, sorry. Im using 8 disks, and another as slog (all of them are 2tb ssd, for the SLOG ive tried full disk, and small partition of 30gb). I know that is useless have this slog partition, but was a try.
What is really annoying is that when use file copies, writes, reads, etc, outside the NFS, I dont get those crappy results...
Im using the sync as disabled. But a simple tests that ive made:
Vmware NFS: IP://mnt/NAS01/NAS01 - 2MBPS
Direct connect (console) using dd: /mnt/NAS01/NAS01 - 700 MBPS
Remotly connect, trought scp: scp ip:/mnt/NAS01/NAS01/.vmdk /local/ - 80MBPS

I understand that firmware update can increase my performance, and I will try it, but there is another problem behind all of this. The performance for NFS is terrible.

 

[toadman]

Well if you are getting good results with the storage locally, but any networked storage is under performing I would conclude you have a networking issue.

On my (very full) freenas system (running as a vm on an exsi host) I get 125 MB/s writing and 240 MB/s reading directly from the pool (from inside the VM). When I access my NFS datastore (shared by that VM) from the exsi server over a distributed vswitch I get 111 MB/s writing and 146 MB/s reading. All that traffic is local to the esxi host as the VM is running on the host.

I can't see how you cant get 700 MB/s local and only 2 MB/s via NFS (with sync=disabled) unless you have a networking problem. You said you were able to get 200 MB/s from a linux box. (I'm not sure what you were expecting and whether or not that is a good result.) So I assume you are using 10G networking.

Are you sure the networking is setup properly on the vmware side?