l2Arc Growing Bigger Than Disk?

[Stikc]

Hey i have tried several different disks and my reporting wizard always shows after heavy use that the l2arc has grown largely more than it should be capable of. The system still runs fine but I would like to fix the issue. I had been running a 250G Samsung EVO Pro and it showed up to 330GB on the reporter. I am currently running a 750 NVMe 400GB partitioned to 250GB but i am showing 318GB used.. Arc hits are currently 81% and usually higher but I have been moving a ton of data today. L2arc hits usually below 3-5%. I am running hyper v vms through isci and a 10gb connection and they dont run too hard but i still like the l2 cache for if they need to kick it up abit. Any info on whats happening would be great. I'm running 9.3 stable.

 

[m0nkey_]

I think what you're seeing is the combination between what's in RAM and what is on your L2ARC device.

Now, I noticed in your signature: 2Arc (250GB) and Zil (120GB) Shared (I know) Intel 750 NVMe

This is very bad practice and the GUI (as far as I know) does not allow this. Besides, you have 80GB RAM there should be no requirement to have a L2ARC device. You're way above the 1GB RAM to 1TB ratio. If you can throw more RAM in the box, that would be way better.

L2arc hits usually below 3-5%.

Pretty much says it all. You don't need a L2ARC device, especially with hits that low.

 

[Stikc]

Ok, I wasn't sure about that because I have heard a lot of people say that with VM storage you should be using l2arc. I have no problems adding more ram but it always says I have 16GB free so I haven't yet, do you still think I should? I will also remove the l2arc as per your recommendation :) just a note I had to partition through the cli to add them on tre same disk. Seems to work great with this device, but I would never do it on an standard ssd. It seems weird that it reports both the ram and the HDD space ..

 

[Sakuru]

My current theory is this has something to do with compression. I think the L2ARC stores blocks as they exist in the pool, but the reporting graph shows their uncompressed size. I'm not an expert, so this is just a guess.

Here is a post from a few months ago with a picture showing my 512 GB L2ARC getting as high as 585 GB: https://forums.freenas.org/index.php?threads/l2-arc-issue.43393/#post-286043

 

[Stikc]

Hmm that's an interesting theory. I'm not sure if it is a combination of ram in the graph too or the uncompressed data but from studying my own system before I removed the l2arc I would guess that you have the closest theory I have heard. Just curious if this is shown like this for everyone or is it just a select few, maybe based on the data type?

 

[kalloritis]

Hmm that's an interesting theory. I'm not sure if it is a combination of ram in the graph too or the uncompressed data but from studying my own system before I removed the l2arc I would guess that you have the closest theory I have heard. Just curious if this is shown like this for everyone or is it just a select few, maybe based on the data type?

I have a L2ARC only on a 6TB WD Red mirror that seems to have exhibited something a bit stranger yet where the L2ARC "size" is more than double the drive's size even (32GB SSD). This is serving the jails and as well as a iSCSI target for two ELK instances ingesting logging data from web servers for several sites. The RAM is 64GB of ECC DDR4-2133 paired with a E5-2620v4.

The pillars are the endpoints backing up to this same system at night on a different encrypted pool that does not have L2ARC (<10% read, so very little use of cache, 10x10TB ZFS2 pool).

 

[Ericloewe]

I have a L2ARC only on a 6TB WD Red mirror that seems to have exhibited something a bit stranger yet where the L2ARC "size" is more than double the drive's size even (32GB SSD). This is serving the jails and as well as a iSCSI target for two ELK instances ingesting logging data from web servers for several sites. The RAM is 64GB of ECC DDR4-2133 paired with a E5-2620v4.

The pillars are the endpoints backing up to this same system at night on a different encrypted pool that does not have L2ARC (<10% read, so very little use of cache, 10x10TB ZFS2 pool).

View attachment 23732

It's probably compression, as stated above.

Also, what's the point of having an L2ARC that is smaller than your ARC?

 

[kalloritis]

It's probably compression, as stated above.

Also, what's the point of having an L2ARC that is smaller than your ARC?

This was just a "early days" SSD that was kicking around that I was using as a prove if even that pool could make use of the L2Arc.

Edit:
While I may have supported the compression argument earlier, It now seems that FreeNAS has lost its mind or is compressing the data on there by some huge margin (over 3x). Is it possible there is deduplication going on too?: