Slow directory listing and hickups

[no_connection]

Dell T20 with G3220 and 12GB RAM (4+8)
Four 4TB WD RED in Z2
FreeNAS-9.2.1.5-RELEASE-x64 (80c1d35)

I have run it a while and it seems to work well.
Depending on source I get 70MB+ write speed.

However it is slow when browsing directories. Once in cache it seems to browse the same directory just fine for a little while.
The problem is that it causes dropouts when streaming music or movies that is quite noticeable.
File browser also happily locks up while waiting.

CIFS share:
Taking one second to access a directory with 15 directories and one file in it.
Two seconds for 128 files.
Ten seconds for 354 files and 117 directories.
Eight seconds for 1190 files.
Forty seconds for 4296 directories.

Most of these takes a maximum of one sec on V7 with 6x 7200rpm drives in Z1 and 4GB RAM. (the NAS I'm replacing so I have the same data in two places to test with)
I know RED have worse access time and fewer spindles but transfers to and from the NAS at the same time does not seems to impact things too much and the other machines don't notice the hickups.

I noticed it being slow in 9.2.1.3 but with all the bugs and stuff I hoped it was temporary, so with most fixed in 9.2.1.5 I had high hopes.

I suppose that performance should not be this bad so something must be wrong somewhere.
Throwing more RAM at the problem might do something but I'm not sure that is the problem.
I will see if I can test with FTP or something if the access time is the same.

*edit* I created the dataset with atime set to inherit and didn't think about it at the time. Will change it once I stop transferring data to see if it helps.

 

[cyberjock]

There's a long thread dealing with this. There's some fixes and some not-so-fixes. The problem, in my opinion, stems from the fact that people often customize the heck out of their NAS thinking they can get something extra that the defaults won't provide. Just off the top of my head I can do 4 different settings changes and any one of them alone will cause a performance nightmare for listing directories. I have no doubt I could think up even more if I was paid to make it happen. Even with extremely powerful hardware too!

I will tell you I have directories with 50,000 files and I've had them load in just a few seconds(like less than 5 seconds). I'm even running FreeNAS virtualized which causes it's own performance problems. Literally when I open the file in Windows it counts up every second with about 10000 or so files in the directory.

So I know it *can* perform well. Unfortunately, unless you know what you did wrong(assuming you actually did something wrong and this isn't a hardware problem) there's little hope we'll be able to solve your problem. It's also why the thread is still open more than 2 years later. There's lots of potential causes, and unless you know what you are looking for you might never be able to figure it out.

If you are interested in paying someone to solve the problem, PM me. Of course, there's no guarantee I will be able to find the problem either, but I've been pretty good with optimizing servers in the past. ;)

 

[no_connection]

Thanks for your reply!

I noticed that thread a little after I posted, so atime is the likely culprit. (hope so at least).
FN is pretty much stock when it comes to settings. No deliberate tweaks to do anything yet.

Changing ZFS settings might be a little risky while I am transferring data so I will wait a little before I can test.

Funny thing is that I remember reading that thread quite a while ago thinking that I don't need atime so one less problem.

I'm probably too stubborn to pay someone to fix what I *should* be able to get working. There is always a reinstall with ZFS import possible.

 

[cyberjock]

Actually, reinstalling FreeNAS doesn't necessarily fix the problem. One person I worked with had to actually recreate his pools because of bad ZFS settings from his attempt to solve another issue.

 

[no_connection]

Unchecking "Support DOS File Attributes:" seems to have solved it.
Will I loose any important function? Using guest access I guess I don't need to change permissions that way. And even if I did I could do it through FTP right?

Disable atime did nothing for speed. Although I guess it saves some IO. Guess I could enable it if I ever need it.

Browsing feels snappy enough now.
New NAS is finally faster at calculating used disc space of folders while it was beaten quite badly before.

Is there any specific reason for DOS attributes to cause this or is it a bug?

 

[cyberjock]

Not a clue. My test system has it enabled and I just ran a script that created 50000 files and the directory loaded in about 5 seconds. :/

 

[no_connection]

Great, after upgrade to 9.2.1.7 it seems to have come back to stuttering and slow directory listing.
Problem is that I can't find "Support DOS File Attributes:" to uncheck it, which is likely checked by default now.
How do I solve this?

 

[anodos]

Off the top of my head the variable for your smb.conf is 'store dos attributes: no' you'll need to verify on your own. If the option isn't presented in the GUI then you can try manually adding it via the 'auxiliary parameters' field in your config.

As for why disabling these works- as far as I can tell there is some sort of performance problem related to CIFS+ZFS+extended attributes+long and complex file trees. Disabling DOS attributes cuts out the XATTR part of the problem but has negative ramifications in multiuser environments. Increasing RAM does help the problem significantly.

 

[no_connection]

Thanks. Unfortunately adding the parameter to auxiliary parameters did nothing.
Even when RAM isn't used up by cache it's slow.

So I have to go back to 9.2.1.6 to be able to use the NAS at all. At least we know that updating Samba version does not fix the bug.

 

[no_connection]

Thankfully writing 9.2.1.6 to a different stick and importing config made the system usable again.
It maybe feels a little less snappy than before upgrade but I am VERY happy that I can use the NAS again.

Please find a way to fix this.

 

[anodos]

The variable I was trying to think of was
store dos attributes = no

You will probably want to upgrade to 9.2.1.7 because 9.2.1.6 is vulnerable to a remote code execution attack on samba as reported here: https://www.samba.org/samba/security/CVE-2014-3560

 

[no_connection]

I did try that variable and it made no difference. Although I did not reboot the server but I did restart CIFS service (even if it did restart by itself).
Problem is I can't use 9.2.1.7 due to the performance problems above, browsing directories with sanity intact simply will not happen, it's that bad.

The advantage of buying two USB sticks is that I can just plug 9.2.1.7 back in if you think a reboot was required for it to work.

*edit* It seems to have done something when doing it this time around after reboot.
*edit2* it seems to work ok for now. (you have no idea how relieved I am about that)

 

[no_connection]

Bug #5752 closed as they can't reproduce.

Which means there is a problem with my setup so that toggling this single parameter causes directories to take obscene amount of time so list and browsing files to be slooow.
I use only quest access and separate dataset for CIFS with windows share set. I have no Idea what I could have done wrong.

 

[cyberjock]

Well, slow directory listings are commonly caused by 3 problems:

1. Failing disks (or at least major problems in the storage subsystem).
2. Insufficient RAM for the pool size.
3. Pool was filled so full it is fragmented to hell and back. (The only fix for this is destroying the pool.) This typically means filling it >80%.

Try adding more RAM to your system.

 

[no_connection]

1: Disks are fine thankfully, no sign of a problem.
2: 12GB for 8TB usable space should be enough, question then is why it's not using the RAM as I's rather have a few more % miss ratio from for cache than unusable performance.
Bringing it up to 16GB is definitely on the wish/planning list but I doubt performance should be this bad due to insufficient RAM. If it needs headroom that it is not going to use then something is wrong.
3: It's indeed starting to get filled up but still showed issued during the filling stage.
My previous NAS was filled to 99% and never had a problem listing directories, and that is using the same files. Not to mention 4GB RAM which disabled cache. (oh and Realtek NIC)
Not sure what ZFS version 7.2 used. Yes they are completely different but I doubt iX deliberately crippled performance just to mess with me. Old NAS did cap out a little bit more than 200Mb though.

IIRC you are using V28 of ZFS created by earlier FN version right?
Maybe V5000 was a mistake to use in my case.

I doubt performance should drop significantly by using newer hardware together with newer software.

What I am wondering is why this single setting causes this significant difference.
With store dos attributes = no I get acceptable access time for four slow spinning 4TB drives, caching of directories seems to work fine as well as relisting a directory is snappy enough.
With store dos attributes = yes access time gets horrible, caching seems non existent for directory data as going up then down one level makes no difference from the first time.

There should be no noticeable difference between the two from what I understand, but something happens.

Here is the result from the built in performance test in case someone understands it.

Code:

perftest/                                                                                           000755  000000  000000  00000000000 12372121024 013003  5                                                                                                    ustar 00root                            wheel                           000000  000000                                                                                                                                                                         perftest/iozone.txt                                                                                 000644  000000  000000  00000002644 12372121024 015055  0                                                                                                    ustar 00root                            wheel                           000000  000000                                                                                                                                                                         	Iozone: Performance Test of File I/O
	        Version $Revision: 3.420 $
		Compiled for 64 bit mode.
		Build: freebsd 

	Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
	             Al Slater, Scott Rhine, Mike Wisner, Ken Goss
	             Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
	             Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
	             Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
	             Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
	             Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer,
	             Vangel Bojaxhi, Ben England, Vikentsi Lapa.

	Run began: Mon Aug 11 12:37:32 2014

	Record Size 128 KB
	File size set to 41943040 KB
	Command line used: /usr/local/bin/iozone -r 128 -s 41943040k -i 0 -i 1
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                            random  random    bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
        41943040     128  139358  137054   134398   110457                                                                          

iozone test complete.

 

[cyberjock]

Yeah, iX didn't cripple it. The problem is if you need more room to cache stuff and you don't have enough RAM then performance tanks. Gee, and you happen to be complaining about poor performance. See the hardware requirements part of the manual that says something like 'the single best way to make a pool faster is more RAM'. It's the nature of the beast. Trying to argue that 12GB of RAM is sufficient for 16TB of capacity is a circular argument. The 1GB per TB is a thumbrule. Some people need less, others need 10x that. My 6TB pool needs 48GB of RAM to run my VMs. Why? Because that's what it needs. To break down how much RAM you need for agiven pool size remember this simple Q&A:

Is ZFS slow?

If yes, add more RAM.

If no, don't touch it.

It really is that simple.

 

[no_connection]

Not sure if ZFS is slow or not as I use CIFS to access it, but I don't think it is the cause of the problems. I did a ChrystalDiskMark from within a VM when testing before and performance there seems ridiculous, SSD seemed unnecessary.
But as the above is a samba setting it should not cause a huge increase of data that needs to be retrieved from the pool that would explain the performance difference. It could potentially be accessing data more spread out on the disks, but then it would reflect in the disks actually doing more IO, which they don't seem to be doing.
It could need more room to cache, problem it it is just as slow with *nothing* in the cache, meaning problem starts from first directory it lists after boot.

Performance goes down the drain from a setting that shouldn't affect it and that does not seem odd?

 

[cyberjock]

Yep... benchmarking won't help you identify this problem bro. In fact, I basically ignore all benchmark results, especially when people complain about poor performance. Mostly because there really is no benchmark tool out there that really does a good job of benchmarking ZFS.

 

[no_connection]

In this case they indicated that ZFS performed well through the VM layer as a VMDK file holding a NTFS file system housing a XP OS running under virtualbox. Or something like that. The use of cache was apparent.
But if you want to tell me that ZFS is performing badly even when benchmark (to me) gave good numbers then it's up to you.

 

[cyberjock]

Ok, I'll say it again...

Yep... benchmarking won't help you identify this problem bro. In fact, I basically ignore all benchmark results, especially when people complain about poor performance. Mostly because there really is no benchmark tool out there that really does a good job of benchmarking ZFS.

Just because you got good results does NOT mean ZFS is performing. We got people getting 4GB+/sec with 3 disks. You really gonna tell me that each of their spinning rust disks exceeded the theoretical 6Gbps speeds?

No, Like I said, no benchmarking tool benchmarks ZFS properly. Your results are invalid.

Edit: Just as a simple example, check out this guy's benchmark results. In a twist of fate, look at what I mentioned and look at how much the numbers changed. http://forums.freenas.org/index.php...folder-in-the-same-dataset.22696/#post-135936