Added an ssd as cache for zfs but it still eats up nearly all memory ???

[Jeff63]

Hello :)

as the zfs cache was eating all my memory I've tried to put an 128 GB SSD as cache, succeeded to do but it seems not be used at all and mem is still eaten instead ?
why ?

Jeff

 

[NugentS]

Thats good. ZFS will (well Scale is limited to 50%) use all thge memory it can

 

[somethingweird]

Understanding ZFS behaviour regarding disk writes and RAM usage!

I know this isnt technically a FreeNAS Qeustion, but it relates to zfs. Im currently running Linux Mint tara 19.1, but i packed a custom ubuntu kernel with 5.1.9 kernel and zfs 0.8.1, which i install it. Thus i have the latest zfs under the latest linux kernel. System consists of 32core epyc cpu...

www.truenas.com

 

[somethingweird]

Thats good. ZFS will (well Scale is limited to 50%) use all thge memory it can

Doesn't seem to hold true in TN13-U4 Core - does it?

 

[sretalla]

Doesn't seem to hold true in TN13-U4 Core - does it?

The statement specifically called out SCALE (to contrast to how CORE works) and indeed CORE does use almost all available RAM for ZFS cache (where it's needed).

 

[winnielinnie]

and indeed CORE does use almost all available RAM for ZFS cache (where it's needed).

But Core 13.0-U4 introduces a change (from upstream ZFS) that vomits ARC out of RAM because of some "too smart for its own good" algorithm.

Luckily, a fix is coming for U5.

And unlike what might have been implied in the relevant thread, it wasn't just noticeable because of "reporting" and "graphs". I experienced slowdowns consistent with a small ARC (suddenly dropping from 24 GiB to less than a few GiB); I noticed way more disk activity; I could hear the HDD heads moving more than normal; and browsing my directories was notably slower, as were rsync tasks.

 

[somethingweird]

The statement specifically called out SCALE (to contrast to how CORE works) and indeed CORE does use almost all available RAM for ZFS cache (where it's needed).

Yes - in theory SCALE is up to 50% and CORE everything it get, it will take!. As you said "Free RAM is wasted MONEY".

 

[jgreco]

Hello :)

as the zfs cache was eating all my memory I've tried to put an 128 GB SSD as cache, succeeded to do but it seems not be used at all and mem is still eaten instead ?
why ?

Jeff

Let's put this a different way. Why do you care? ZFS performance is driven by its caching. You WANT it to use all free memory for caching. When FreeBSD runs out of memory for user processes and needs more, it asks ZFS to release some of the memory, so the use of the memory for ZFS cache is not a problem at all. The design of Linux by comparison is sucktacular and it cannot do that. Either way, ZFS needs to categorize data in its ARC prior to placing it in the L2ARC, which is why it is important to have a significant amount of ARC (generally 64GB works well) available so that ZFS can make good choices about what to push out to your "cache SSD". So it is in your interest to give it as much ARC as possible.

 

[Jeff63]

I care 'cause there is only 16 GB of memory in that NAs... and I can't afford to double it (the max supported by the mobo) yet.
with that short mem I get sometimes the NAS 'disconnecting' from working :( plus that using heavily the swap part on the sys disk will wear that disk faster :(

 

[danb35]

I care 'cause there is only 16 GB of memory in that NAs

Then you shouldn't be using a cache device at all. Your money would have been better spent on more RAM.

 

[somethingweird]

I care 'cause there is only 16 GB of memory in that NAs... and I can't afford to double it (the max supported by the mobo) yet.
with that short mem I get sometimes the NAS 'disconnecting' from working :( plus that using heavily the swap part on the sys disk will wear that disk faster :(

What is your setup? Is just for storage or it is running plugins/jails/VM/containers?

 

[jgreco]

I care 'cause there is only 16 GB of memory in that NAs...

You don't have enough ARC, and your L2ARC is unlikely to be getting populated with good choices. You can check the stats on the off chance that it might be doing you some good.

It's like towing a large camper with a car. Just because the car is all you have available does not mean that the car must therefore be able to do the job.

 

[Jeff63]

My config is that :

TrueNAS Scale v22.12-1 on
Asus ROG Maximus IV Gene Z
i7-2600 @ 3.40GHz
16 GB ram
no graphic card it's in the CPU
Melanox 10 Gb/s fiber network card
SAS2308 HBA card in IT mode
Z2 pool with 6x10 TB + 1 spare (just seen that I need 4 more disks at a time minimum to expand the pool... ouchbtw I have room for 10 more in the tower ;))
128 GB Cache SSD

for now it's mainly for distributing and stocking files and all my systems saves accross the 10 systems I have here, btw I want it grow up and use a lot of stuff (including serving VMs instead of setting up vms on each machine).

I had that SSD laying alone in a drawer for some years, and I just thinked ZFS would use it as cache instead of ram... was wrong :(

I will see to update the hardware to something more recent when I can, btw just now it's not a bad NAS at all and appart that 'problem' it works well :)

 

[somethingweird]

Unfortunately you simply need more RAM - like everyone said.