All NFS write seem to be synchronous

[fa2k]

I'm using NFS v4.2 on Ubuntu Linux clients. I am expecting with this new NFS version that the client can tell the server what writes are synchronous, and what it can buffer in memory. However, my log device is being hit hard with all the traffic, even when I'm doing things like copying huge files with cp.

If I set sync=disabled, it of course stops using the log device.

Mount options on client are pretty standard - example fstab:

192.168.1.4:/mnt/xpool/data/nobackup /data/nobackup nfs defaults

I'd really like to keep sync enabled for the things that need it, but that it doesn't sync everything. Any ideas?

 

[winnielinnie]

I am expecting with this new NFS version that the client can tell the server what writes are synchronous, and what it can buffer in memory.

I'd really like to keep sync enabled for the things that need it, but that it doesn't sync everything. Any ideas?

Not to sound blunt, but since when was that a thing with NFS? It will decide what transfers require sync and async on-the-fly?

I've never heard of that before.

 

[fa2k]

Not to sound blunt, but since when was that a thing with NFS? It will decide what transfers require sync and async on-the-fly?

I've never heard of that before.

Since NFSv3 (so my comment about "new" nfs version was a bit misleading)

Linux NFS faq

NFS Version 3 introduces the concept of "safe asynchronous writes." A Version 3 client can specify that the server is allowed to reply before it has saved the requested data to disk, permitting the server to gather small NFS write operations into a single efficient disk write operation. A Version 3 client can also specify that the data must be written to disk before the server replies, just like a Version 2 write. The client specifies the type of write by setting the stable_how field in the arguments of each write operation to UNSTABLE to request a safe asynchronous write, and FILE_SYNC for an NFS Version 2 style write.

 

[jgreco]

Not to sound blunt, but since when was that a thing with NFS? It will decide what transfers require sync and async on-the-fly?

The client is allowed to make determinations for certain write operations; see RFC1813. It basically involves allowing async writes followed by a COMMIT. This has certain side effects such as implying that the client must maintain precious data in some sort of buffer on its side.

The Linux exports options "async" and "sync" can specify server-side behaviour. "sync" would mean server must always sync the data even if client didn't ask for it (see discussion near end of section 3.3.7 and the discussion of COMMIT at 3.3.21). You then specify "sync" on the client side to get the "safe async writes" behaviour.

In the real world, I typically decide whether a mount needs sync, and if it does not, I just make it async. This seems to be a rather esoteric implementation detail in my opinion.

 

[fa2k]

The client is allowed to make determinations for certain write operations; see RFC1813. It basically involves allowing async writes followed by a COMMIT. This has certain side effects such as implying that the client must maintain precious data in some sort of buffer on its side.

Thanks, this finally explains it, great link! I was reading a good blog from 2008 (https://utcc.utoronto.ca/~cks/space/blog/solaris/SlowNFSWritesToZFS) and saw this was and understood problem, and then I had a debate with ChatGPT about why Linux *had* to issue COMMIT before clearing its write cache. Now I see it's in the protocol.

I didn't see a way to set the "async" option in TrueNAS SCALE , but I think it would have a similar effect as zfs's `sync=disabled`.(Edit: I see, "async" is the default on the server; and "sync" is not very smart on the server so no point in exposing it in the UI)

In my case, the solution was the good old zfs property `logbias=throughput`. It achieves better performance for big file writes by using the pool disks instead of the log device (which in my case is limited to about 250MB/s and I'm slightly worried about wear). The solution only works on my "big file" dataset, but that was the main problem.

 

[winnielinnie]

but I think it would have a similar effect as zfs's `sync=disabled`.

The issue is that this "bombastic" approach will outright disable sync writes for metadata.

 

[winnielinnie]

The client is allowed to make determinations for certain write operations; see RFC1813. It basically involves allowing async writes followed by a COMMIT. This has certain side effects such as implying that the client must maintain precious data in some sort of buffer on its side.

I misunderstood the original OP question.

This is done "on-the-fly"?

I guess my question now is (if it's even the right question, as this goes over my head), what would a typical NFS client use to "determine" a write operation? The "size" of the file (or total stream) of what's being transferred over the network?

Assuming you leave the client's mount at the default (async), and the server's export at its default.

 

[jgreco]

I guess my question now is (if it's even the right question, as this goes over my head), what would a typical NFS client use to "determine" a write operation? The "size" of the file (or total stream) of what's being transferred over the network?

No, clearly not. Certain operations, such as those in ZFS that are always done synchronously, should generally be done sync across NFS as well. And just like ZFS, operations such as long stream writes may not gain any value from being written synchronously. Open a file for create, that's sync. Write 1GB of data, that doesn't need to be sync in most cases. But also writing 1KB of data, that probably doesn't need to be sync either. Close the file, there's your sync. ZFS does something similar.