SOLVED ZFS cache consuming nearly 40Gb of RAM

[KevDog]

I have a question regarding zfs cache. It's nearly consuming 40Gb of RAM and bringing the system to a crawl. I wasn't aware ZFS cache consumed this much memory. Is there a way to limit the amount of RAM it requires since its basically making the system unusable.

I have the following tunable:
vfs.zfs.arc_max
61632000000
sysctl
Generated by autotune
yes

 

[Fredda]

My understanding was the ZFS cache always uses the memory which is not used by applications.
So ideally there should be no free memory. Of course applications always have higher priority as cache.
I think there is something else wrong.

 

[KevDog]

@Fredda

You may be right -- I'm running FreeNAS 11.3U1. I updated to the release last week and things worked very well for the first week. Now my Middleware GUI is a crawl. The system is running really slow. I'm not sure what's going on or even where to look. I've rebooted the installation and still the same experience.

 

[jgreco]

It should not be bringing the system to a crawl, ZFS is designed to make use of all unused memory as ARC, which should make your disk I/O faster. When the userland calls for more memory, ZFS will release appropriate amounts of ARC. About the only time this seems to be a problem is when userland suddenly allocates gobs of memory and the system might get a little bogged down while ZFS has to do a ton of ARC evictions, and my recollection is that they did some stuff to "fix" even that.

I'm not saying your system isn't bogged down, but it's not likely to be due to ARC.

 

[KevDog]

The middleware is running at a crawl and the ssh connections keep breaking. I'm on a wired LAN. More and more its sounding potentially like a network issue (possibly).

 

[HoneyBadger]

I admit I haven't played much with 11.3 but seeing "Services" at 23GB of RAM tells me something is very wrong from that perspective, that's nearly triple the recommended minimum RAM for FreeNAS. 11.3U2 is out and had a pretty long list of bugs squashed.

 

[danb35]

11.3U2 is out and had a pretty long list of bugs squashed.

...including a memory leak in the middleware, IIRC.

 

[SweetAndLow]

You should probably also remove everything that auto tune has set. Usually auto tune is not something people use.

 

[KevDog]

Honestly -- I have no idea what's going on -- all my ssh connections now to the VM's running within freenas now become broken after about 1 minute. The middleware like freezes for about a minute or two and then starts working. It's really hard to use the system now.

 

[KevDog]

Honestly -- I have no idea what's going on -- all my ssh connections now to the VM's running within freenas now become broken after about 1 minute. The middleware like freezes for about a minute or two and then starts working. It's really hard to use the system now.

Thanks for your recommendations -- I'm think I have this pinned to a network issue rather than a RAM issue. I'm going to mark this a solved for now.

 

[Apollo]

Are you playing with LAGG?

Intermittency freezing trouble, what could it be.

I've been vexed by a intermittency issue with my VM Ubuntu 18.04 headless server, For about a year now freezing the transmission both in the app/web user interface and on the command line ssh connection. It works well its not showing any errors in the syslog only this output looks strange...

www.ixsystems.com

 

[KevDog]

Are you playing with LAGG?

Intermittency freezing trouble, what could it be.

I've been vexed by a intermittency issue with my VM Ubuntu 18.04 headless server, For about a year now freezing the transmission both in the app/web user interface and on the command line ssh connection. It works well its not showing any errors in the syslog only this output looks strange...

www.ixsystems.com

No I was playing with OpenVPN and although the VPN was technically down -- the network adapters the vpn created were not completely destroyed. I believe this may have been the cause of my problems.