Simultaneous CIFS/NFS, CIFS with passthrough acls, NFS on windows, and other forbidden subjects

[redoak42]

Bear with me here before you throw me overboard, and even then try to do it gently

One of the groups I support is a small (four person) group of scientific programmers. For about five years they ran a series of Synology NAS, and simultaneously exported NFS to their Linux boxes and CIFS to their Windows boxes on the same data sets, and they were happy. When it came time to replace the Synology, we went with a FreeNAS mini for a whole variety of reasons, and we also switched to CIFS only exporting, as simultaneous CIFS/NFS is dangerous unless there is some secret sauce, which I'm pretty sure the Synology didn't have, and I'm sure FreeNAS doesn't.

Over the next few months we've discovered that Linux, or, more accurately, the programs and programming languages they use, really don't like CIFS very much because chmod, chown and other similar permissions related commands and calls don't work the way the programs expect. This leaves us scratching our heads about what to do.

One option is to switch to NFS only exports, buy them Win 10 Enterprise (which supports NFS) and run that way. If anyone has any experience with doing this, I'm all ears.

I'm well aware the next option is probably not a good idea, and may result in corruption if people aren't VERY cautious and disciplined, but I want to check if there is anything about FreeNAS, FreeBSD, and ZFS that makes it a non starter.

The only way I can see to get them back to the multi-protocol nirvana they remember from their Synology experience on the FreeNAS mini is to do the following: remove the CIFS shares from the UI, switch the ZFS acls to pass through, export the the data sets as NFS through the UI, and then simultaneously export the data sets as CIFS shares with POSIX acls as described on the SAMBA wiki https://wiki.samba.org/index.php/Shares_with_POSIX_ACLs , meaning very simple permissions with no management of ACLs from windows. The exports would be put in file, and then into the samba.conf file through an include statement in the aux parameters of the cifs config area of the FreeNAS gui.

So, if I did the above, and my users have superhuman control about opening files simultaneously (and very good backups), is there any reason why it wouldn't give them what they had before? Any reason it would not work at all?

Besides running NFS on Windows, or multi-protocol export from the NAS, what option am I missing that could fix this?

 

[anodos]

Samba on Linux supports kernel oplocks. This allows the samba shares on a synology device to happily coexist with NFS shares. I don't believe freebsd supports these, and so you should disable oplocks on any shares you plan to also share out with NFS. For more information see here: https://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/locking.html

Share auxiliary parameter - "oplocks = no"

Since you're going to need to look at other methods of access control in samba, I recommend looking into using "valid users", "write list", and "force group". These will be added as auxiliary parameters to your shares.
Syntax is roughly as follows:

Code:

valid users = +group1 +group2 bob
write list = +group1 bob

The above auxiliary parameters will allow read-write access for the group "group1" and the user "bob" and read-only access for the group "group2".

In addition to setting the aclmode to "passthrough", you may also want to disable the zfsacl vfs module by adding the auxiliary parameter "vfs objects = zfs_space streams_xattr" to your shares.

 

[redoak42]

Thank you for the advice! Today, I tried this in a test environment, and it seems to be working.

 

[anodos]

Thank you for the advice! Today, I tried this in a test environment, and it seems to be working.

Glad to hear that! Some workloads actually get performance benefit from turning off oplocks so it's not exactly a negative thing.

By the way, there are significant performance improvements in the linux kernel CIFS client in more recent kernels. You may significant benefit in moving from CentOS 6 -> CentOS 7. Or debian whatever to debian new.
Use "smbstatus" at the command line to make sure your clients are negotiating SMB2 or SMB3 connections.

 

[cyberjock]

Few things:

- Synology DOES have secret sauce to make NFS/CIFS work together without corruption. I know this as someone that has had to troubleshoot stuff on Synology systems.
- In FreeNAS 10 you will NOT be able to share the same places with NFS and CIFS simultaneously. FreeNAS' CIFS ACLs and NFS Unix perms aren't compatible (yes, your workaround works, but the decision has been made), and the corruption issue only provides even more justification to code FreeNAS to flat out prevent his situation.
- Even with super-human control of what files they open, there is no guarantee that Windows won't go looking at things it shouldn't be (everyone's had to deal with thumbs.db issues before). Trying to prevent corruption with "super-human control" is a recipe for disaster. From people who have thought they could do this, they only figured out files were corrupt long after the backups were expired and they had no good copies.
- Yes, this will "work" if/when you test this. All will look well. But there's no formula to do this that is safe (hence it is being deliberately blocked in FreeNAS 10).

My advice would be to decide on NFS or CIFS right now, and go with that and give up on the other. Anything else is going to bring you file corruption, issues with deployment both now and in the future, or both.

Another option is to install Linux on the Mini, mount the zpool on Linux (I believe the zpool is compatible, but I'm not 100% sure) and then setup NFS and CIFS on Linux.

 

[anodos]

Few things:

- Synology DOES have secret sauce to make NFS/CIFS work together without corruption. I know this as someone that has had to troubleshoot stuff on Synology systems.

They probably achieve this by setting "kernel oplocks = yes".

Another option is to install Linux on the Mini, mount the zpool on Linux (I believe the zpool is compatible, but I'm not 100% sure) and then setup NFS and CIFS on Linux.

As samba transitions to using SMB2 leases / durable handles, it may not be sufficient to set "oplocks = no" to ensure that samba locks are visible to the NFS server. It might be a good idea to explicitly set:

Code:

oplocks = no
posix locking = yes
smb2 leases = no #Not a valid parameter on earlier versions of samba

In case the defaults in samba change in the future. But this does highlight some challenges in using samba in this way. Samba is pursuing becoming a drop-in replacement for a Windows server. This means pushing the boundaries of what you can do in a posix environment. You can see this in the way that samba stores ACLs, metadata, and handles different oplock types. I believe samba team members are trying to push changes in various other projects to make things work together better (like richacls in Linux), but I expect that the feasibility of simultaneously sharing a dataset with NFS and samba (with all the fancy features enabled) will start to die off.

You can get by with the above setup. The two problems you have to prevent are (1) file corruption due to oplocks (2) permissions getting out of whack.

Per the samba documentation:

Local UNIX and NFS clients access files without a mandatory file-locking mechanism. Thus, these client platforms are incapable of initiating an oplock break request from the server to a Windows client that has a file cached. Local UNIX or NFS file access can therefore write to a file that has been cached by a Windows client, which exposes the file to likely data corruption.

If files are shared between Windows clients and either local UNIX or NFS users, turn oplocks off.

Once you turn off oplocks you should be golden.

That said, it might be better to migrate to one of the Linuxes and set "kernel oplocks = yes" for better compatibility and improved client-side caching. It just depends on your use case as to whether you'll see benefit from this. If you decide to go Linux, you'll want a stable Linux with the samba 4.3 branch in its repositories.

Even better idea: switch to an all-samba environment :D

 

[bestboy]

I was just reading up on this old article on smallnetbuilders.com. If I understand it correctly, then oplocks should only be used on read-only shares where they do nothing, or single writer shares where the client lock has never to be broken.
For shares with multiple writers it advises to disable oplocks even in a CIFS-only environment with no simultaneous NFS sharing. Having the server revoke the client oplocks has negative impact on the performance.
Given that the article is from 2007, do you think this is still good advise today?

 

[anodos]

I was just reading up on this old article on smallnetbuilders.com. If I understand it correctly, then oplocks should only be used on read-only shares where they do nothing, or single writer shares where the client lock has never to be broken.
For shares with multiple writers it advises to disable oplocks even in a CIFS-only environment with no simultaneous NFS sharing. Having the server revoke the client oplocks has negative impact on the performance.
Given that the article is from 2007, do you think this is still good advise today?

Samba has changed dramatically since 2007. I'd suspect it doesn't apply any longer, but the only way to be certain is to do your own testing.

 

[cyberjock]

Unless I'm mistaken, setting oplocks in Samba still won't ensure that the locks are visible to NFS. The iXsystems dev team and I have discussed NFS and CIFS to death, and the short and skinny is that you'd have to write a new daemon to make the transition both ways, and nobody has the resources, time, and motivation to have done that (yet). There's been talk that once FreeNAS 10 is out and stable and not bug-prone then maybe iXsystems will consider championing it. But for now its basically impossible to make Samba and NFS talk to each other. :(

 

[anodos]

Unless I'm mistaken, setting oplocks in Samba still won't ensure that the locks are visible to NFS. The iXsystems dev team and I have discussed NFS and CIFS to death, and the short and skinny is that you'd have to write a new daemon to make the transition both ways, and nobody has the resources, time, and motivation to have done that (yet). There's been talk that once FreeNAS 10 is out and stable and not bug-prone then maybe iXsystems will consider championing it. But for now its basically impossible to make Samba and NFS talk to each other. :(

Yeah. From what I understand that's basically right. "oplocks = no" kills all oplocks in samba for the share you set it on. No fancy caching client-side, but I could be totally wrong.

 

[bestboy]

@anodos: Yes, that's why I'm not sure the article is still valid. Most of the samba advise on the internet is just outdated.

@cyberjock: I found a thread on the Samba mailing list from 2001 Implemented OPLOCK for FreeBsd with participation of Jordan. It seems there was already a working kernel patch, but I guess something went wrong...