Issue with SWAP

[Lansing123]

Info on system:
TrueNAS-SCALE-22.12.4

Hardware:
Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz
32GB none ECC
8x8TB raidz2 configuration

Hello, i am having issues with truenas scale using SWAP after it's been up and running a while. This causes lag in the UI for the dashboard and apps for example. As i understand it the OS is not really supposed to use SWAP unless it really needs it. I have free RAM but it still chooses to use SWAP. Does anyone have a clue on what i can try to troubleshoot this?I will attach printouts on RAM usage and whats using up my RAM. Most of the RAM is used by ZFS Cache, so even if my system ran out it ZFS should release some RAM before starting to use SWAP if i am not mistaken?

Edit: I might also be wrong and SWAP is not the issue. But after booting up and running for a day my UI slows down alot. Instead of loading in a secound or two it takes around 20+ secounds.

 

[Lansing123]

I guess i am the only one with this issue. I will try to look for information elsewhere!

 

[chuck32]

I had a similiar issue, here are two helpful replies I got:

Swap gets uses although there is unused RAM

Hello again, I recently installed truenas scale as my new hypervisor. Specs: Mainboard asrock b560m-itx/ac CPU i3 10100 32 GB RAM M2 (internal) - 500 GB as boot drive SATA (internal) - 4x WD Red Plus 4 TB directly connected via, raidz2 Delock 5 port SATA PCI Express x4 Card - 512GB and...

www.truenas.com

Swap gets uses although there is unused RAM

Hello again, I recently installed truenas scale as my new hypervisor. Specs: Mainboard asrock b560m-itx/ac CPU i3 10100 32 GB RAM M2 (internal) - 500 GB as boot drive SATA (internal) - 4x WD Red Plus 4 TB directly connected via, raidz2 Delock 5 port SATA PCI Express x4 Card - 512GB and...

www.truenas.com

Maybe you can give some more information on what else you use on your system, VMs, Apps etc..

The reason I ask is, maybe you allocated too much memory to Vms:

I think the motto here is, unused RAM is wasted RAM but you are going even lower in free RAM than I did. And for a fun fact: even with the 64 GB upgrade over time i still hit times where SWAP got used. But usually that wasn't a problem performance wise.

Maybe someone else could enlighten us more here, but personally I think even with some Gib left on your memory you probably already hit a certain treshold and the system tries to free RAM by using more swap. Did you try to increase the swap size?

 

[Lansing123]

I have not tried to increase SWAP, i will try and see if that changes anything. I did check those posts but i don't really feel like they give any information about why the system is using the SWAP for no reason. I have no running VM's just a few dockers i run. My RAM is no where close to being used up, the ZFS uses over 60% of RAM and services take only around 20%, so the system has 80% of total RAM to avoid using SWAP.

I have not come across a lot of others with the problem, seems like an issue with the OS tbh. Perhaps i should open a JIRA ticket with IX systems.

 

[sfatula]

There are actually a lot of reasons why it might use swap. First, unless you monitor it subsecond, you have no idea if all memory was ever consumed. Linux tries to use all memory, there are filesystem buffers too. But you say it should reduce those if under memory pressure, sure, it will, but, there are limitations to this. Look up "swapiness" as one example, swappiness affects tendency to free cache memory. . Another example is ZFS, while zfs tries to reduce the arc if under temporary memory pressure, it's not always fast enough to do so and the extra memory might temporarily go to swap before the arc is reduced. Just 2 examples. There are others. It's not that unusual. My system which has been up for a few weeks now shows swap usage, not concerned about it. I have as of this instant 27GB free. But that means nothing, in the middle of the night maybe backup, maybe syncs, who knows what changes the memory usage. Something does, as that's why it used some swap.

Linux memory management is not always so obvious or cut and dry.

If you search the internet (not this forum) for something like "linux swap usage even though memory not full" you'll find endless questions about it.

 

[jgreco]

Another example is ZFS, while zfs tries to reduce the arc if under temporary memory pressure, it's not always fast enough to do so and the extra memory might temporarily go to swap before the arc is reduced.

The real problem is that there was basically no effort put into integration with the Linux memory management system when ZFS was ported to Linux. The FreeBSD porters spent a lot of time integrating and restructuring so that ZFS and the FreeBSD kernel would cooperatively work together to manage memory, and have polished that integration even further for more than a decade. I'm told that the Linux memory management stuff is a trainwreck and that Torvalds was obstructionist towards any concessions to accommodating ZFS (Torvalds apparently hates ZFS), so I don't want you to come away with the idea that the Linux porters for ZFS were lazy or somehow failed to do their best, but it has been clear that Linux is poorly positioned to be a top tier ZFS platform and that no one is interested in polishing that horse apple -- and who can blame them.

That's part of why the recommendation has been to have twice as much physical memory as the amount of ARC you wish to have. I believe it is still preallocated space in the kernel memory area, which is clearly suboptimal as this does not allow that memory to be used when the system is under memory pressure. You get swapping instead. Way to go Torvalds.

 

[sfatula]

The arc will still shrink, it shows so in my memory usage (both memory graphs and zfs graphs show decreases and increases. Been running zfs on ubuntu for years as well, and it also shrinks when needed. It is not pre-allocated in kernel.

 

[winnielinnie]

it has been clear that Linux is poorly positioned to be a top tier ZFS platform

Well, yeah, but... "containers" and "apps"... and hardware support...

 

[sfatula]

I don't know what is meant by "top tier" specifically, but, if you follow zfs forums / rediit / most anywhere you can discuss it, virtually all the users are Linux. But you are correct as far as how it works somewhat better on freeBSD when it comes to memory management.

Back on topic, I don't believe the OPs issue has anything to do with swap space. 20 seconds is a long time! The top screen looks good, don't see any issues there. Load is fine, wait for IO is fine. Not sure if this is during a slow UI time or not. Memory also looks fine, I see no issues with any of that.

What client are you using OP to run the UI on? Is the top capture during a time when you are trying to bring the UI up on some client? Could be a toughie to pin down as there could be all sorts of causes.

 

[jgreco]

Well, yeah, but... "containers" and "apps"... and hardware support...

Well the "hardware support" mainly concerns trash grade hardware you shouldn't be using on a server anyways. Containers and apps are probably best run on a Linux VM that you can run on CORE under bhyve. But what do I know. The Linux folks like to talk trash about anything non-Linux, much as any other fanboi group, often with little-to-no real knowledge. I'm happy to talk trash about Linux but I back my trash talk up with facts.

if you follow zfs forums / rediit / most anywhere you can discuss it, virtually all the users are Linux.

Not shocking. When people are having trouble with something, they go to those discussion forums. TrueNAS has been based on BSD for more than a decade, and there hadn't this huge run of people with weird memory swapping issues or other weird Linux-related issues until the powers that be decided they needed more converged platform support for competitive reasons and decided it was best to embrace K8s which meant Linux. Hopefully this means that there will be pressure on King Torvalds to allow improvements in the Linux memory management system. This would be a good thing for Linux and also for ZFS. I am definitely in favor of open source software even if it is Linux so I do want this to happen... but I'm kinda

about it.

I don't believe the OPs issue has anything to do with swap space. 20 seconds is a long time! The top screen looks good, don't see any issues there. Load is fine, wait for IO is fine.

I've seen this happen too, but I didn't look too hard at it. I have some small SCALE VM's that I've used for development purposes, and sometimes after a period of inactivity (day++) they would seem like they had swapped the shell and other UI components out, taking 10-20 seconds to "wake up". These aren't busy; just an HBA with eight disks in a RAIDZ with nothing on them except for an array testing script when I was developing it. 16-32GB RAM and nothing running, no protocols, no containers, just SSHD and maybe a vim process. *shrug*

 

[sfatula]

I am not sure what to guess either. I'm running Scale and just don't see it, ever. Which doesn't help the OP of course. Just was trying to come up with a reason.

I am all for improvements as well. It's been a while, but I am an ex freeBSD user. Didn't have any particular problem with it, worked fine.

 

[Lansing123]

I don't know what is meant by "top tier" specifically, but, if you follow zfs forums / rediit / most anywhere you can discuss it, virtually all the users are Linux. But you are correct as far as how it works somewhat better on freeBSD when it comes to memory management.

Back on topic, I don't believe the OPs issue has anything to do with swap space. 20 seconds is a long time! The top screen looks good, don't see any issues there. Load is fine, wait for IO is fine. Not sure if this is during a slow UI time or not. Memory also looks fine, I see no issues with any of that.

What client are you using OP to run the UI on? Is the top capture during a time when you are trying to bring the UI up on some client? Could be a toughie to pin down as there could be all sorts of causes.

The PC i run the UI on is a gaming PC, like Radeon 7900xtx with a massive CPU and 64GB ram. The reason i assume it's the server is that after a restart it runs fine. It's only after 1-2 days that the UI slows down, the only thing i notice that corresponds is that SWAP usage increases.

Honestly i have checked around, no one seems to have this issue. I'm super confused haha

 

[Patrick M. Hausen]

Use top or htop to find out which process(es) is consuming all that swap space.

 

[Davvo]

Might be worth opening a bug report and getting troubleshooting instructions from iX engineers.
At least the guesswork will be done by those who later will attempt to fix/better integrate Linux with ZFS.

 

[Lansing123]

Use top or htop to find out which process(es) is consuming all that swap space.

Hello, thanks for the tip. In my original post i did check and post everyting that uses SWAP. It seems the OS is.

 

[Patrick M. Hausen]

Can you sort that top display by swap size? FreeBSD top can, on Linux you might need htop for that.

 

[Lansing123]

Can you sort that top display by swap size? FreeBSD top can, on Linux you might need htop for that.

The picture already has it sorted by SWAP size. If you check the left side you can see SWAP at the top with the largest SWAP contributer at the top and showing them in size of SWAP usage with lower swap usage being further down the list.

 

[Patrick M. Hausen]

Got it. So it seems k3s and lots of containerd instances are to blame ...

 

[Lansing123]

Got it. So it seems k3s and lots of containerd instances are to blame ...

The large issue is that both of those are run by the OS. I can't really fix that, and everyone should have the same problem in that case. Also this does not really answer the question on why the system even uses the SWAP in the first place. There is plenty of RAM available for it too use. I will have to open a ticket with IX i guess. It's very strange.

 

[Davvo]

Got it. So it seems k3s and lots of containerd instances are to blame ...

Kubernets having resources utilization issues? That's not a bug, it's a feature!
Seriously, the more I look around the more I love jails.