Lots of files, SMB folder list slow

[IonutZ]

Hey guys, I've been dealing with some issues on my CIFS shares. Listing folder contents from Windows takes a very long time. I have folders with over 30-50k files and it will take up to 10 seconds to open them at times.

I've turned AutoTune on and added the regular CIFS Auxiliary settings:

socket options = TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE SO_RCVBUF=98304 SO_SNDBUF=98304
​

I also created a script that will do a recursive ls on all my folders every hour in an attempt to keep them in cache - and that works (sort of) but to me that's a quick fix if anything.

I noticed that my ARC isn't being used very much. I have around 4TB of files and my arc is only being used @ 2GB. I attached a copy of the arc_summary.py output.

vfs.zfs.arc_max is set at 29gb by AutoTune (I have 32gb of ram)

I set net.inet.tcp.recvbuf_max and net.inet.tcp.sendbuf_max and kern.ipc.maxsockbuf to 131072 (from a bigger number as per the recommendation of an user from here)

SO_SNDBUF from what I've read can be set as high as you want because it'll only yield an increase in performance up until a certain point (but doesn't negatively impact performance the higher you go).
​

I'm currently at a loss... any help would be very much appreciated. Also my Windows clients (that are seeing slow folder lists) are connected on SMB3.0 protocol.

Also turned atime off on my volumes.​

 

[dlavigne]

Any headway with this?

 

[IonutZ]

None so far unfortunately... it has something to do with the CIFS (SMB) cache. The easiest "patch" is to create a bash script that runs as a cronjob, performing a recursive 'ls' through smbclient on the share; this way the files are all in smb's cache. This isn't a solution, just a quickfix.

 

[cyberjock]

Few things:

1. Get rid of those settings. Those settings are the defaults already for the first 3, and the last 2 are actually worse than the default.

2. SMB 3.0 support in Samba isn't 100%. I'd recommend you set the max protocol on the FreeNAS machine back to SMB2. There's a reason it's the default. ;)

3. 10 seconds for 50k files isn't bad. That's 5000 files/second.

It was polite to include the arc_summary, but that doesn't tell me a whole lot as those numbers might be normal if you rebooted 5 minutes before getting that summary since a reboot wipes out the ARC. Can you post a debug file of your system?

 

[Norleif]

Do you know for a fact you wouldn't have the same problems if the same files where shared from a Windows machine?

 

[IonutZ]

I totally would cyberjock but it has too much personal information to redact everything, those logs are just ridiculous lol. If there's anything specific that you wanna take a look at I will gladly upload.

Norleif, yes, I had the same files in a LSI RAID under a Windows box which provided instant access.

Also, it makes sense that once they are cached, access is instant... but somehow that cache doesn't persist unless I have that script running (that does a recursive ls on the share using smbclient)

 

[cyberjock]

Yeah, if you can't post a debug I really don't have much interest in looking further. Trying to redact stuff just means stuff won't make sense and as a volunteer here I'm not about to spend that kind of time and energy on this. Sorry.

If you want support you might want to try contact iXsystems. They do a 1-off support option (it isn't free), but they might be able to look at it. Do be warned they *will* ask for a debug. ;)

 

[pernils]

Sorry for hijacking this thread but I didn't want to start a new one.

Background :

The company I work for is growing out of storage. After some reading I deiced to move to zfs store. Bought a freenas mini with 32GB memory and 4 red nas 4 TB drives.

Did a raid 10 for best performance (for what I can understand) and begun some testing.

I have around 25+k files and I'm measure against a win2008 with raid1. Folder browsing will be the same with

Code:

store dos attributes = no
ea support = no

CIFS Auxiliary settings:

With freenas 9.3 you will not see this when testing with testparm

This I think have to do that store dos attributes = no is default for freenas 9.3 version of samba

https://www.samba.org/samba/docs/man/manpages-3/smb.conf.5.html#STOREDOSATTRIBUTES

So far so good. I Have the same experience when browsing the file tree on the nas compare to win2008 box.

My other goal with this purchase (freenas mini) is to move the AD to the freenas. A bit annoyed by MS survey now and then over the license for win2008.

The sad thing is that samba's domain controller needs store dos attributes = yes to work.

It doesn't matter if you specify = no it will be turned on anyway.

(For newbies like me you can reload smb settings with samba reload. It's a bit annoying do this in the GUI because you must first turn Domain controller to off before turning the CIFS off/on)

From this thread...

https://forums.freenas.org/index.ph...-to-improve-samba-browsing-performance.24906/

Is there a chance that I/we can see some other speed improvements on this when Freenas is moved to FreeBSD 10 ?

I want my users to have as pleasant experience as possible with the new data store.

edit --
Some numbers.

On win2008 it takes around 3 sec to list 24 k files

On Freenas it takes the same when dos attribute = no (default)
With it = yes (acting as domain controller) it takes more then 8 sec.

As our cad software is scanning the current folder when you open a part for editing it's a key factor to have fast folder listening.

--edit

 

[IonutZ]

I can tell you that the only way I've figured it out so far (and I messed with a lot of settings), was to have a recursive ls script run in the background at a given interval of time. This worked in the beginning when I had less files to look through, but as the number of files increased in size, it became less and less efficient because I'm pretty sure the cache is limited. So you'd run it on one share, and the other one would be pushed out of cache...

However if you only have a few (as opposed to a lot) of folders that you need constant fast access too, the solution above might be good for you.

 

[pernils]

From my findings ls recursive will not help me. I get the same timing when I browse the same folder over and over again.
The overhead is coming from store dos attribute = yes

 

[IonutZ]

You need that param to run AD?

 

[pernils]

I think I don't need it. I will pro belly not use any backup software instead take the advantage of zfs send / recive.

Nerveless this is my findings .

From this page https://wiki.samba.org/index.php/Shares_with_Windows_ACLs

ACL support on member server
The following is only required on Domain Member Servers and not on Domain Controllers!

• Add the following to your [global] section of your smb.conf:

vfs objects = acl_xattr
map acl inherit = Yes
store dos attributes = Yes

These options are required on Member Servers, to enable the possibility for real windows ACL's. Domain Controllers have ACL support enabled globally by default!

When I make the freenas member of the win2008 server domain the folder browsing is slow. It takes around 7-8 sec to list a folder with 24k files. On win2008 share it's around 2-3 sec for the same.

A test with testparm reveals

Code:

acl allow execute always = Yes
    create mask = 0666
    directory mask = 0777
    ea support = Yes
    directory name cache size = 0
    kernel change notify = No
    store dos attributes = Yes
    strict locking = No
    dos filemode = Yes

Okay ...

In CIFS settings -> Auxiliary parameters I put in

Code:

  store dos attributes = no
  ea support = no

Test with testparm again ..

Code:

  
    acl allow execute always = Yes
    create mask = 0666
    directory mask = 0777
    directory name cache size = 0
    kernel change notify = No
    strict locking = No
    dos filemode = Yes

As store dos attribute = No and ea support = No are the default they will not be shown.

And the folder browsing will become almost as snappy as the win2008 share (3 sec).

But when I make the Freenas as a Domain Controller I'm not able to set store dos attribute = No.


It's boils down to ...

Making your Freenas member of a domain you can set the store dos attribute = No

Making your Freenas as the Domain Controller you are sort of out of luck.

But from what I understand from this thread ..

https://forums.freenas.org/index.ph...-to-improve-samba-browsing-performance.24906/

This slow folder browsing could be addressed by Freenas 10.x

 

[pernils]

Did some more testing this morning.

As stated before the store dos attribute = No will have no impact on CIFS Settings -> Auxiliary parameters if you config Freenas to be a Domain Controller. Witch is not what the wiki say.

If you set it there you will end up anyway with it set to =Yes. If you make the Freenas to be Member of a Domain it will be set according to your wish. Witch is the opposite of what the wiki say.

But if you set it on the smb share it will work even if you config your Freenas to be a Domain Controller.

Bellow readout from testparm.

Code:

..........
       server string = FreeNAS Server
        server role = active directory domain controller
        map to guest = Bad User
        obey pam restrictions = Yes
        passdb backend = samba_dsdb
        max log size = 51200
..........

[data]
        path = /mnt/Vol_1/data
        read only = No
        guest ok = Yes
        ea support = No
        veto files = /.snapshot/.windows/.mac/.zfs/
        store dos attributes = No
        vfs objects = zfs_space, zfsacl, aio_pthread, streams_xattr
        zfsacl:acesort = dontcare
        nfs4:chown = true
        nfs4:acedup = merge
        nfs4:mode = special

Huh ?

Did some googling ... pasting the relevant links that I found.

http://comments.gmane.org/gmane.network.samba.general/141993

https://lists.samba.org/archive/samba-technical/2013-May/092084.html

http://pig.made-it.com/samba-file-rights.html

http://wiki.nas4free.org/doku.php?i...r_guide:services_cifs_smb_samba&do=export_pdf

https://www.abclinuxu.cz/data/prilohy/8/9/139898-smb-41642.conf

https://daniel.washburn.at/howtos/freebsd-samba4-zfs-recipe

https://forums.freebsd.org/threads/xattr-sa-effect-on-samba4-directory-browsing-performance.47904/

http://www.sysadmin-cookbook.net/samba/

So for correct use of windows ACL you have to set store dos attribute = Yes. But this will slow down your folder browsing.


This store dos attribute is related to windows ACL but I can change security settings in AD even if I have store dos attribute = No on the share.

But if I try to change the attribute on the file I get "You will need to provide administrator permission to change these attributes"

Should I just disobey all the guides and set it to No just to have fast folder browsing. Or could I end up with borked files if I proceed ?

My conclusion is that nothing will happen but you will not be able to use backup software that relay on the archive bit.

 

[pernils]

For more reading about the Dos attribute and what this store dos attribute means you can read from page 162 on this link

ftp://ftp.iwf.oeaw.ac.at/pub/unischel/Using_Samba_Third_Edition.pdf

One thing I don't get my head around is with this smb.cfg

Code:

....
        server string = FreeNAS Server
        server role = active directory domain controller
        map to guest = Bad User
        obey pam restrictions = Yes
        passdb backend = samba_dsdb
......
        acl allow execute always = Yes
        create mask = 0666
        directory mask = 0777
        ea support = Yes
        directory name cache size = 0
        kernel change notify = No
        map archive = No
        map readonly = no
        store dos attributes = Yes
        strict locking = No
        dos filemode = Yes
        vfs objects = dfs_samba4, acl_xattr
......
[data]
        path = /mnt/Vol_1/data
        read only = No
        guest ok = Yes
        ea support = No
        veto files = /.snapshot/.windows/.mac/.zfs/
        store dos attributes = No
        vfs objects = zfs_space, zfsacl, aio_pthread, streams_xattr
        zfsacl:acesort = dontcare
        nfs4:chown = true
        nfs4:acedup = merge
        nfs4:mode = special
.....

* Freenas is acting as a Domain Controller

* That means that you will have automatic the settings store dos attribute and ea support = Yes in the global section.
I have set those to No in the share section.

The result is that I have a snappy file browser experience and also the ability to set permission on user basis in the AD (permit or deny user from deleting files etc..)

But the Domain administrator can't set the simpler legacy DOS bits (read only/ hidden/ archive). You get prompted that you don't have permission.

But when I again remove the parameters from the share section (see bellow)

Code:

......
[data]
        path = /mnt/Vol_1/data
        read only = No
        guest ok = Yes
        veto files = /.snapshot/.windows/.mac/.zfs/
        vfs objects = zfs_space, zfsacl, aio_pthread, streams_xattr
        zfsacl:acesort = dontcare
        nfs4:chown = true
        nfs4:acedup = merge
        nfs4:mode = special
.....

The Domain administrator can set those DOS bits again. But the folder browsing is ones more slow again.

Odd .....

Tips when you want to fiddling with this is that you must go to Service -> Domain Controller and click the edit symbol (you don't have to flip the "switch"). After that you just click Ok at the bottom. Freenas will take care of the rest. You don't have to manual reload samba etc.


Are there any powers here that can en light me in my suggestion that you can disable the store dos attributes and ae support on the local share section for faster folder browsing even if you are dealing with AD if you don't care about the archive bit ?

I need to take this box into production and I want it to be as pleasant as possible.

 

[cyberjock]

Honestly, your best bet is to submit a ticket at bugs.freenas.org and see if John Hixson has the answers. :P

I'm guessing there is some reason for this behavior, even if you and I don't know what it is.

 

[pernils]

Done.

A better write up on how samba is using *nix bits for mapping to MS bits could be found on page 251 (chapter 8:2)

ftp://ftp.bupt.edu.cn/pub/Documents/so-many-notsorted/new/oreilly/download/Oreilly-Using.Samba.pdf

 

[cyberjock]

What is the bug #?

 

[pernils]

I don't know if this should be called a bug.. but it ended up on the bug section. I leave it to the moderators to move it where it belongs.

https://bugs.freenas.org/issues/11288

 

[anodos]

I think I don't need it. I will pro belly not use any backup software instead take the advantage of zfs send / recive.

Nerveless this is my findings .

From this page https://wiki.samba.org/index.php/Shares_with_Windows_ACLs

ACL support on member server
The following is only required on Domain Member Servers and not on Domain Controllers!

• Add the following to your [global] section of your smb.conf:

vfs objects = acl_xattr
map acl inherit = Yes
store dos attributes = Yes

These options are required on Member Servers, to enable the possibility for real windows ACL's. Domain Controllers have ACL support enabled globally by default!

No, those are not required by AD member servers to enable "real windows ACLs". Windows ACLs are handled by the "zfsacl" vfs module. Also, you should not enable the "acl_xattr" vfs object on a FreeNAS server unless you enjoy tanking performance on your server.

When I make the freenas member of the win2008 server domain the folder browsing is slow. It takes around 7-8 sec to list a folder with 24k files. On win2008 share it's around 2-3 sec for the same.

A test with testparm reveals

Code:

acl allow execute always = Yes
    create mask = 0666
    directory mask = 0777
    ea support = Yes
    directory name cache size = 0
    kernel change notify = No
    store dos attributes = Yes
    strict locking = No
    dos filemode = Yes

Okay ...

In CIFS settings -> Auxiliary parameters I put in

Code:

  store dos attributes = no
  ea support = no

Test with testparm again ..

Code:

 
    acl allow execute always = Yes
    create mask = 0666
    directory mask = 0777
    directory name cache size = 0
    kernel change notify = No
    strict locking = No
    dos filemode = Yes

As store dos attribute = No and ea support = No are the default they will not be shown.

And the folder browsing will become almost as snappy as the win2008 share (3 sec).

But when I make the Freenas as a Domain Controller I'm not able to set store dos attribute = No.


It's boils down to ...

Making your Freenas member of a domain you can set the store dos attribute = No

Making your Freenas as the Domain Controller you are sort of out of luck.

But from what I understand from this thread ..

https://forums.freenas.org/index.ph...-to-improve-samba-browsing-performance.24906/

This slow folder browsing could be addressed by Freenas 10.x

You should not have the same samba instance act as a Domain Controller and file server simultaneously. If you need to have the same server fulfill both roles, you should install samba41 in a jail with vimage enabled, and configure the jailed samba instance to be the AD DC. Then join FreeNAS to the domain. This will allow you to set "store dos attributes = no" as a global parameter on your FreeNAS server.

 

[pernils]

No, those are not required by AD member servers to enable "real windows ACLs". Windows ACLs are handled by the "zfsacl" vfs module. Also, you should not enable the "acl_xattr" vfs object on a FreeNAS server unless you enjoy tanking performance on your server.

Those suggestion was cut and pasted from the samba wiki.

I don't follow you in the word "tanking" (=taking performance of ?) But indeed freenas will fill in "acl_xattr" in the global section. I seem there is no way I can revert it either.

Code:

[global]
....
        server string = FreeNAS Server
        server role = active directory domain controller
        map to guest = Bad User
        obey pam restrictions = Yes
        passdb backend = samba_dsdb
        max log size = 51200
        server max protocol = SMB2
        deadtime = 15
        max open files = 942176
        hostname lookups = Yes
        load printers = No
        printcap name = /dev/null
        disable spoolss = Yes
        lm announce = Yes
        dns proxy = No
        panic action = /usr/local/libexec/samba/samba-backtrace
        dns forwarder = 192.168.100.1
        nsupdate command = /usr/local/bin/samba-nsupdate -g
        rpc_server:tcpip = no
        rpc_daemon:spoolssd = embedded
        rpc_server:spoolss = embedded
        rpc_server:winreg = embedded
        rpc_server:ntsvcs = embedded
        rpc_server:eventlog = embedded
        rpc_server:srvsvc = embedded
        rpc_server:svcctl = embedded
        rpc_server:default = external
        idmap_ldb:use rfc2307 = yes
        idmap config *: range = 90000001-100000000
        idmap config * : backend = tdb
        acl allow execute always = Yes
        create mask = 0666
        directory mask = 0777
        ea support = Yes
        directory name cache size = 0
        kernel change notify = No
        map archive = No
        map readonly = no
        store dos attributes = Yes
        strict locking = No
        dos filemode = Yes
        vfs objects = dfs_samba4, acl_xattr

....

[data]
        path = /mnt/Vol_1/data
        read only = No
        guest ok = Yes
        ea support = No
        veto files = /.snapshot/.windows/.mac/.zfs/
        store dos attributes = No
        vfs objects = zfs_space, zfsacl, aio_pthread, streams_xattr
        zfsacl:acesort = dontcare
        nfs4:chown = true
        nfs4:acedup = merge
        nfs4:mode = special

(I haven't tried what happens when I dont have Freenas acting like a Domain Controller).

Maybe this is something that should be changed in the Freenas code base ?

You should not have the same samba instance act as a Domain Controller and file server simultaneously. If you need to have the same server fulfill both roles, you should install samba41 in a jail with vimage enabled, and configure the jailed samba instance to be the AD DC. Then join FreeNAS to the domain. This will allow you to set "store dos attributes = no" as a global parameter on your FreeNAS server.

Perhaps that's the best way to go. But I liked the web gui.