I have a 30 GB SSD for an L2ARC. According to zpool status -v the drive always seems to have approximately 3.5 GB space used and the rest is free.
Shouldn't the cache drive be mostly full? Especially after running 10 VMs for months on end?
Thanks,
-A
Not necessarily. Getting an L2ARC to work properly requires all other aspects of the system to be
properly resourced.
Here's just one small example. I don't know what your configuration is, so you'll have to decide if this applies to you or not:
Say you built a FreeNAS box with 8GB of RAM, then decided to put in an L2ARC. We'll say 30GB since that's your size (30GB of RAM is horribly oversized for your system, but go with it for now.. I'll explain in a bit).
Data must be read into RAM, then accessed enough times to meet the threshold for the L2ARC. I believe it has to be accessed 5 times. If it is not accessed 5 times before it is flagged for removal from the ARC, the L2ARC will not cache it at all. It is not even considered for the L2ARC. So because you under-resourced one aspect of your system (ARC) you've actually made the L2ARC mostly non-functional. This behavior, along with multiple others I won't go into detail on in this post, is just one of many reasons why we say you need lots of RAM
before you consider an L2ARC. Not after. Not during some testing phase. Not because you can. Before and only before. Otherwise you'll buy that nice shiny SSD and it will provide you with no useful caching. (Keyword: useful).
Now why is oversizing the L2ARC bad? Back to our example.
So as the L2ARC starts filling with data the L2ARC index must be stored in RAM. So the more L2ARC you have, the less RAM you have for actual zpool data. This means that as you cache more info in the L2ARC you actually hurt the performance of the ARC. At some point the ARC begins to churn through data so quickly that you not only will never use the data even twice (so the L2ARC effectively becomes a read-only cache device), but the ARC is so small that performance of the zpool takes a nosedive.
Now to go in detail on the "keyword: useful" aspect of my comment.
People like you will read the data like you did, see that a couple of GB of data is in the L2ARC, and then make the conclusion that your L2ARC isn't oversized and is being used properly because it *is* storing information. So clearly this Cyberjock guy doesn't know what he is talking about because *I* just looked at my zpool info and I got ABC GB of data in the L2ARC. These numbers are accurate, but the conclusion is probably false. ZFS metadata is used so frequently it is trivially easy to end up with metadata in the L2ARC, even on an under-resourced system. Seems like a win-win, but it's not. Metadata is often composed of smaller blocks. Since the L2ARC index stores its entries per block, the best scenario is to cache blocks that are as big as possible. Having 300 bytes of RAM cache 128KB of data in the L2ARC is better than having 300 bytes of RAM cache 512 bytes of data in the L2ARC. ZFS normally allocates a percentage of the ARC for metadata and a percentage for file data (typically 25%/75% if I remember correctly). You'll break this relationship because L2ARC index counts as metadata, even if the index entry points to zpool file data. Can you try to fix this relationship yourself with tuning. Sure can, but that doesn't fix other aspects of ZFS.
Now there is also a performance penalty with retrieving data from the L2ARC. After all, SATA/SAS isn't nearly as fast as RAM. Even PCIe isn't as fast as RAM in terms of latency and throughput. But L2ARCs are useful because, despite the fact that they are slower than your RAM, they are almost certainly faster than your zpool. But, because of all of the other aspects of how ZFS works, this performance gain from an L2ARC can be more than offset by the starved ARC, which can (and is documented plenty of times in the forum by various users that argued that they knew better) hurt the zpool's overall performance.
When it comes to ZFS, L2ARCs and slogs, the flowchart is not simple. It would look something like this...
Unless you are about to dive into ZFS and have the knowledge to create the flowchart yourself, you're stuck listening to the somewhat generalized experience of this forum. For this reason, the obvious answer to 99.9% of people's questions is to just blindly do what we recommend around here. Unfortunately a large percentage ignore that and do what they want because they think they know better. They don't, and they are simply throwing money and performance away when they don't want to listen. Sad, but also true.
So what do I know about your system based on just this one example and the output you provided? Since you have 2GB used in your L2ARC on a system with VMs that would make me question your system's resources. I'd almost bet money you don't have at least 64GB of RAM, and probably have 16GB or less, and as a result you have unknowingly broke the design of the L2ARC thereby making it somewhat useless. :P
There are, of course, exceptions. Some setups will not have a lot of data in the L2ARC because of what the VM itself does internally. If you run a SQL database in a VM and SQL is configured to cache the whole database in RAM (or at least the vast majority of the database that is used) you may find very little data is being read multiple times, so the L2ARC won't get much data put into it. So yes, there are exceptions. But what's the chances of every one of your VMs behaves in exactly this manner? Pretty darn close to zero. If you were that level of 'pro' with this stuff you wouldn't be going to a forum to get assistance, you'd be the one writing the guides and manual edits. ;)
See, when I ask for a debug file, that thing gives me massive amounts of information on a person's system. More than they realize. It is generally super-easy to figure out if someone tried to properly resource their system or is being a selfish jerk and deliberately ignored our recommendations and now expect us to help them tune ZFS for their system. Guess which group of people ends up on my ignore list quickly? ;) I don't have time for people that try to hide the fact that they're being selfish and won't at least try to make things work for themself before asking someone to help them out.. for free.
I have no pity or respect for those that ignore our recommendations and cry because they can't hit some performance metric that matters to them.
I'm not saying you are ignoring our recommendations or anything. I don't know. You also aren't crying because your performance is poor. You are just curious as to how this stuff all works. Hopefully now you understand that, with just the one small example I gave, that ZFS is far far more complex than most people realize and can't be taken lightly. Just looking at a couple of numbers won't tell you good or bad things, it's a lot of numbers and what the workflow of ZFS is doing based on those numbers. ;)
