L2ARC filling before ARC is full?

[CJRoss]

I have a spare NVMe SSD and only one place to stick it so I figured I'd try it out as an L2ARC for a bit. My server is properly sized so I expect to see no benefit from this but I figured I'd let it run for a couple of months as a test.

What I found interesting and odd was that immediately after I started TrueNAS Core and added the L2ARC it began populating with data even though the ARC wasn't even close to using all of my free memory. Is there a reason the ARC started evicting things instead of growing?

Since then my ARC has grown to it's usual size and everything is running as expected but I'd like to know what was going on during that initial ARC population.

 

[Nick2253]

I'm not sure exactly what the parameters are, but once a block in ARC has dropped below a hit-ratio threshold, it is written out into L2ARC in anticipation of being saved there. In that way, ZFS can fill the L2ARC with useful data before it has run out of space in the ARC.

If it didn't do this pre-write, then you'd have an delay when the ARC got full, needed to take on a new block, but had to first flush that block out to L2ARC. By "pre-flushing", the ARC can immediately take on new data, replacing the now "less hot" data with your newly accessed block, and still have access to that "less hot" block by way of L2ARC.

 

[CJRoss]

Interesting.

 

[Nick2253]

Just to clarify, there is always some degree of overlap between the contents of the L2ARC and the ARC. So it's incorrect to assume that data has been "evicted" from ARC if it shows up in L2ARC. Data will only be "evicted" from ARC when there is "better" data to replace it with.

 

[HoneyBadger]

I'm not sure exactly what the parameters are, but once a block in ARC has dropped below a hit-ratio threshold, it is written out into L2ARC in anticipation of being saved there. In that way, ZFS can fill the L2ARC with useful data before it has run out of space in the ARC.

Expanding on this a little, there is no "threshold" that it has to drop below in terms of hitrate. The feed thread picks up from the tail end of the ARC, going a certain depth (tunables involved are the l2arc_headroom and l2arc_write family) in to look for candidates to load into L2ARC.

 

[Nick2253]

Ahh, thank you for that clarification. That makes a lot more sense, and helps me better understand those tunables.

 

[HoneyBadger]

Ahh, thank you for that clarification. That makes a lot more sense, and helps me better understand those tunables.

The defaults also try to fill L2ARC aggressively if it's empty/cold - controlled by l2arc_feed_again - so @CJRoss that's why you were seeing it taking in data.

As mentioned by @Nick2253 there is overlap, so just because the L2ARC was filling doesn't mean that same data was definitely being evicted from primary ARC.

 

[CJRoss]

Thanks all.